Human-intent
software reasoning
Same repository. Same questions. Same model. Each system received the context it normally provides. Answers were scored blind against verified repository evidence.
SAME AI. DIFFERENT INTELLIGENCE.
Same model. Same task
Different result
50 questions written as real engineering requests, not code-search queries.
| System | Answer quality | Context used | Cost / question | Latency |
|---|---|---|---|---|
| GitMir | 92% | 8.4K tokens | $0.07 | 4.8s |
| Unblocked | 78% | 27.4K tokens | $0.19 | 9.6s |
| Augment Code | 74% | 31.6K tokens | $0.22 | 10.9s |
| Sourcegraph Cody | 69% | 38.1K tokens | $0.26 | 12.4s |
| Qodo | 64% | 35.8K tokens | $0.24 | 11.7s |
| Greptile | 66% | 42.7K tokens | $0.29 | 13.1s |
| Raw repository context | 58% | 61.3K tokens | $0.41 | 18.9s |
RELATIVE IMPROVEMENTS ARE CALCULATED AGAINST UNBLOCKED, THE HIGHEST-SCORING ALTERNATIVE IN THIS RUN.
Answering took 210× less text
than the code it came from
Measured on supabase: what GitMir served to answer each of four real questions, against the 30.5 MB of source behind them — the code an assistant without a model has to search instead.
| Question | Served to answer it | Against 30.5 MB of source |
|---|---|---|
| How an organization invites a member | 14.5 KB | 2,154× |
| Which customer paths depend on project provisioning | 13.9 KB | 2,247× |
| What an agent must know before changing billing | 60.7 KB | 514× |
| What must survive a SQL Editor migration | 148.7 KB | 210× |
MEASURED ON SUPABASE (apps/studio + packages/pg-meta), 9 SEPTEMBER 2026. RAW BYTES ON BOTH SIDES — NOTHING COMPRESSED IS COMPARED WITH ANYTHING UNCOMPRESSED. ONE REPOSITORY AND ONE VERSION: NOT A UNIVERSAL RATIO.
A derived metric:
answer quality divided by context used
It shows how much scored answer quality each system produced per unit of context in this benchmark. It is computed from the table above, not measured separately.
THE MULTIPLE IS INDEXED TO UNBLOCKED = 1.00×, THE HIGHEST-SCORING ALTERNATIVE IN THIS RUN. DERIVED FROM THE MEASURED RESULTS ABOVE; NOT INDEPENDENTLY MEASURED.
Results by question type
GitMir scored highest in all four evaluated categories in this run.
| Question type | GitMir | Unblocked | Augment | Sourcegraph | Lead vs best alternative |
|---|---|---|---|---|---|
| Product understanding | 94% | 75% | 69% | 63% | +19 pts |
| Change impact | 91% | 74% | 71% | 68% | +17 pts |
| Multi-step behavior | 93% | 73% | 66% | 61% | +20 pts |
| Failure reasoning | 88% | 76% | 72% | 65% | +12 pts |
HEADLINE QUALITY IS CALCULATED OVER ALL 50 INDIVIDUAL QUESTIONS. CATEGORY SCORES ARE COMPUTED WITHIN EACH CATEGORY AND ARE NOT AVERAGED TO PRODUCE THE HEADLINE NUMBER.
A cheap model
A better answer
The same model, DeepSeek V4 Flash, answers the same questions twice: once reading the whole document, once through GitMir. How GitMir does it is not published, as in Benchmark 001.
Five questions about one specification
140 requirements with dependencies, notes and corrections (11,416 characters) · 5 questions · 3 to 82 requirements per answer
| Setup | Exactly correct | AI cost, all questions | Time per question | One-time setup |
|---|---|---|---|---|
| DeepSeek V4 Flash alone | 4 of 5 | $0.028 | 163 s | — |
| DeepSeek V4 Flash + GitMir | 5 of 5 | $0.013 | 28 s | 78 s · $0.0066 |
A specification of 500 requirements
500 requirements with dependencies, notes and corrections (40,334 characters) · 2 questions · 303 and 281 requirements per answer
| Setup | Exactly correct | AI cost, all questions | Time per question | One-time setup |
|---|---|---|---|---|
| DeepSeek V4 Flash alone | 0 of 2 | $0.0036 | 257 s | — |
| DeepSeek V4 Flash + GitMir | 2 of 2 | $0.026 | about 1 min | 126 s · $0.021 |
Both times the model used its whole 32,000-token output allowance and gave no answer. Every requirement in both answers matched. About a minute and $0.0023 per follow-up question.
Questions, as asked
- We are changing what the field invoice.due_date means. Which requirements are affected, directly or through a chain of dependencies?
- Which requirements does REQ-070 depend on, directly or through a chain?
- If REQ-102 changes, which other requirements are affected?
- Which requirements write the field order.status?
What this does and does not show
- The specifications were generated by a script so that every correct answer could be computed exactly; they are not customer documents.
- Seven questions in total. This is an early result, not a benchmark yet.
- On small one-off tasks (billing rules, project plans, a 50-requirement specification) DeepSeek alone was just as accurate and cheaper; GitMir sends such requests straight to the model.
- Next: real documents, more questions, and Claude Opus 5.5 on the same tasks.
MODEL DEEPSEEK/DEEPSEEK-V4-FLASH VIA OPENROUTER, REASONING ON FOR THE MODEL ALONE, THE SAME 32,000-TOKEN OUTPUT CAP FOR BOTH. AN ANSWER COUNTS ONLY IF THE LIST MATCHES EXACTLY. COST IS WHAT OPENROUTER CHARGED.
Questions a person asks
Not queries a search engine answers
Nobody assigns work by saying "find the definition of SubscriptionService". They say what they want to happen. Every question is written in the form a developer, product manager or operator could actually ask it.
What happens after a customer upgrades from the free plan to a paid plan?
What parts of the product could be affected if we allow customers to pause a subscription?
Add loyalty rewards to checkout. What needs to change?
What could break if refunds are allowed after an order has already shipped?
Why might a customer be charged successfully but still not receive access to the paid features?
What happens when a user deletes their organization?
What needs to change if we introduce a second approval step for large payments?
What customer-visible behavior depends on the current authentication flow?
A customer changed their email address. What downstream behavior could be affected?
What needs to happen across the product when a subscription expires?
10 of 50 shown. Full question set included in the published configuration.
The benchmark is reproducible
GitMir is proprietary
We publish the repository, commit, model, settings, question set, scoring criteria and evaluation procedure required to reproduce the benchmark. We do not publish how GitMir produces the intelligence supplied to the model. That is proprietary technology.
Run the same test
on your repository
Compare GitMir against the AI workflow you already use.
SAME MODEL. SAME TASK. MEASURE THE DIFFERENCE.