How to Benchmark AI Coding Tools on Your Own Repository
A practical protocol for testing AI code tools yourself: designing the question set, holding the model constant, blind scoring, and the four things worth measuring.
Every vendor in this market publishes a number. Almost none of them publish a method you could use to check it. That asymmetry is not an accident — a benchmark without a method is a marketing asset, and a benchmark with one is a liability if the result does not hold.
This is a guide to running your own. Not because vendor numbers are always wrong, but because your repository is the only one whose answer matters to you, and the test is cheaper to run than a bad procurement decision is to unwind.
The mistake almost every comparison makes
Most published comparisons of AI code tools are built on lookups: find the definition of PaymentService, list the callers of this function, where is this constant used. These questions are easy to score, easy to automate, and completely useless as a discriminator.
They are useless because every serious tool in this market passes them. Symbol lookup is a solved problem — it was solved by ctags decades ago and has been getting incrementally better ever since. A benchmark made of lookups measures how well each vendor implements a commodity.
Worse, it plays on the vendor's home ground. Tools built around code search are optimised for exactly that shape of question. Testing them on it tells you they are good at the thing they were built for, which you already knew.
The questions that discriminate are the ones a person actually asks. Nobody assigns work by saying "find the definition of SubscriptionService". They say what they want to happen, and the interesting property of a tool is whether it can answer in those terms.
Designing the question set
Write the questions before you look at any tool. This matters more than it sounds — questions written while holding a product in mind will unconsciously flatter it.
Aim for fifty. Fewer than about thirty and one lucky answer moves the result more than a percentage point; many more than fifty and the scoring becomes the bottleneck. Draw them from four kinds:
- Product understanding — how does this part actually work, and why does it behave this way?
- Change impact — what would break if we changed X?
- Multi-step behaviour — what happens end to end when a customer does Y?
- Failure reasoning — how can this outcome occur, given the code?
Phrase every one of them the way it would arrive in a ticket or a Slack message. "What could break if refunds become available after shipment?" — not "trace the refund call graph". If a question contains a class name, rewrite it.
Then write reference answers, from the code, before running anything. This is the unglamorous half of the work and it is where the benchmark's credibility comes from. A reference answer should state what is true and cite the file it came from. If you cannot write it, drop the question — an unanswerable question measures nothing.
Holding everything else constant
The claim you are testing is this tool changes the answer. So everything that could also change the answer has to be pinned:
- One model, one version, for every system in the test.
- Temperature 0, or as close as the tools allow.
- Same maximum output length.
- Same repository, same commit. Record the commit hash; a benchmark against a moving branch is not repeatable.
- The same question text, verbatim, to each system. No per-tool prompt tuning — the moment you tune, you are measuring your prompt-writing, not the tool.
- Several runs per question. Three is a reasonable minimum. Sampling noise on a single run is larger than most of the differences you are trying to detect.
Then give each system the context it normally provides, and nothing more. This is the part people get wrong out of a misplaced sense of fairness: do not hand every tool the same files. The whole product of a context tool is deciding what context to supply. Overriding that measures nothing.
Scoring without fooling yourself
Score blind. Strip the system names, shuffle the answers, and grade them against your reference answers without knowing which came from where. Everything else in a benchmark is technique; this is the part that protects you from yourself.
Weight the components explicitly and publish the weights. A defensible split:
| Component | Weight | What it asks |
|---|---|---|
| Correctness | 40% | Is what it says true of this repository? |
| Completeness | 30% | Did it cover the consequences that matter? |
| Relevance | 15% | Did it answer the question that was asked? |
| Evidence reliability | 15% | Can each claim be traced to the source? |
That last row does real work. An answer that is right by luck and an answer that is right with its evidence attached are not the same asset, because only one of them can be checked by the next person.
Measure four things, not one
Answer quality alone is a bad summary, because a system can buy quality with context you have to pay for on every request forever. Record all four:
- Answer quality, from the blind scoring above.
- Context used per question, in tokens. This is the line item that scales with your usage.
- Cost per question, at the model's published rates.
- Latency to a usable answer.
When we ran this on a public repository, the four moved together in a way that was informative on its own: the systems that used the most context scored the lowest. The best-scoring alternative needed 27.4K tokens per question at $0.19; the raw repository needed 61.3K at $0.41 and produced the worst answers of the set. More material did not mean better reasoning — it meant more reconstruction.
It is also worth deriving one ratio: answer quality per thousand tokens of context. It exposes something the raw numbers hide, which is how much of what you are paying for actually contributed to the answer. Say plainly that it is derived from the other measurements rather than measured separately — a derived metric presented as a measurement is the most common quiet dishonesty in this genre.
Publishing it
If you publish, publish the configuration: repository, commit, model and version, temperature, output limit, the full question set, the scoring criteria, and the evaluation procedure. Anyone should be able to reproduce your number without talking to you.
And name what the comparison is relative to. "The strongest system available" is a market verdict you have not earned; "the highest-scoring alternative in this run" is what you actually measured. The second phrasing is less impressive and much harder to attack, which is the trade you want.
What a good result looks like
A benchmark that comes out unanimous should make you suspicious. Real differences are uneven — a tool is usually much better at some question types than others, and the pattern of where it wins tells you more than the average does.
Our own run led on all four categories but by different margins: widest on questions about how the product behaves, narrowest on reasoning about failures. That narrow margin is more informative than the wide one. It says where the remaining work is, and a result that admits that is one you can act on.
Run it on your own repository. It takes a day, and it settles an argument that vendor pages cannot.
Put a number on your own workflow
Bring us the recurring workflow where AI still needs expensive people to supervise, review and correct. We baseline what it costs and put the target in writing before we build. Or connect a repository and see it on your own code first — 1,000 credits free, no card.
Build My Business Case → Start free →Frequently asked questions
How many questions do I need to benchmark AI coding tools?
About fifty. Below roughly thirty, one lucky answer moves the result by more than a percentage point; much above fifty and writing reference answers becomes the bottleneck. Spread them across product understanding, change impact, multi-step behaviour and failure reasoning, and run each question at least three times to average out sampling noise.
Why shouldn't I use "find the definition of X" questions in a benchmark?
Because every serious tool passes them. Symbol lookup is a commodity, so a benchmark built on it measures nothing that separates products, and it plays on the home ground of tools built around code search. Questions phrased the way work is actually assigned — "what breaks if we change refunds" — are what discriminate.
Should every tool in the benchmark get the same context?
No. Deciding what context to supply is the entire product of a context tool, so overriding it measures nothing. Hold the model, temperature, output limit, repository, commit and question text constant, then let each system supply the context it normally would.
How do I score AI answers fairly?
Blind. Strip the system names, shuffle the answers and grade against reference answers you wrote from the code before running anything. Publish the weights you used — correctness, completeness, relevance and whether each claim can be traced to source — so somebody else can apply the same standard and get the same result.
Related articles
Credits or Seats: How AI Dev Tools Actually Charge
Seat pricing broke when every request started costing the vendor money. A survey of what the market charges for now — agent compute, credits, per-action — and how to read a bill before you sign it.
Evaluate the Result, Not the Architecture Diagram
Vendor architecture slides are unfalsifiable, and treating them as evidence is the most common mistake in AI procurement. What counts as evidence, and six questions for a vendor call.
The AI Tooling Market in 2026: An Honest Map
Adoption is over at around 90% and proved little. The four positions in the market, what pricing reveals about cost structure, and buying advice for this quarter.