Why Your Organisation Needs Its Own AI Benchmark
Field notes from building them for clients, and what it takes.
A model's leaderboard rank says little about how it will handle your field. A domain benchmark does: on one built for a client, the two best models finished under half a point apart, and one cost twenty times more to run. It can be shared without revealing how you work.
Public leaderboards rank models on coding, maths, trivia and reasoning puzzles, mostly in English, mostly on short prompts. An organisation putting AI to work has a narrower question: which model is best at our tasks, on our material, in our languages, at a price we can justify?
We've built benchmarks for several clients to answer exactly that. They don't replay the client's workflows; testing those end to end is an eval, and it stays in-house. Instead they measure the capabilities the work rests on: how well a model understands and handles general tasks in the client's domain. The results show how accurate each model is and which one to use for which task.
Across those projects, a model's leaderboard rank told us surprisingly little about how it would do on the client's work.
What a benchmark is for
Clients come to it with different questions. Most of them fall under one of these:
- Quality. Which model gets your tasks right, in your languages, on your material.
- Cost. Which is the cheapest model that's good enough. On a job you run once, the difference is pocket change. On 10,000 rows, or every night, it's the budget.
- Bias. Whether a model treats like cases alike when the only difference is a name, a gender or the language a text is written in.
- Speed. How long each answer takes, which matters when a person is waiting for it, or when a batch has to finish overnight.
The answer is rarely one model for everything. More often it's a cheap one for the bulk work and a stronger one for the cases that need it.
What a domain benchmark is
A set of realistic tasks from your field, each with a known correct answer. Every model gets the same tasks under the same conditions, and a script scores the answers.
The ones we've built have covered tasks like these:
- Figures in long reports. Find specific numbers in annual reports that run to hundreds of pages.
- Standardised forms. Read a structured form, including which boxes are ticked.
- Rules and regulation. Work out which provisions apply to a situation, and apply them correctly.
- Terminology. Use the right professional term in each working language.
- Grading at volume. Score or classify thousands of rows against the same criteria: customer feedback, support tickets, product records.
Where a client's material is sensitive, the cases are built rather than copied: the same kind of documents and questions, with no real people or records in them. They stay realistic without exposing anything.
A case is a small, flat record. Keep the cases in version control next to the script that runs them, so a change to a case gets reviewed like a change to code.
id: report-figures-007
test_set: report-figures
question: What were the total operating expenses for the financial year?
input_file: reports/annual-report-a.pdf
expected: 48200000
expected_unit: EUR
scoring: numeric-exact
evidence: page 112
verified_by: domain-expert
- One question, one answer. If it takes a paragraph to explain what counts as correct, split the case.
- Point to the evidence. Recording where the answer lives (page, section, article) makes disagreements quick to settle.
- Experts write the answers, engineers write the runner. Keep those two jobs apart.
Benchmarks and evals
An eval runs your actual workflow end to end. It shows how you work, so it stays in-house.
A benchmark tests the domain knowledge underneath: the terms, the rules, the standard forms. That knowledge belongs to your field, not to your organisation, so you can share it. You can even publish it as an open test. Vendors run it and show you their scores, before you share any of your own material.
You need both. The benchmark narrows the field, the eval makes the call. Once a benchmark is public, expect vendors to tune for it, and keep refreshing the cases.
Eight things the benchmarks taught us
1. Half a point apart on quality, twenty times apart on cost
On one benchmark, the two best models finished less than half a percentage point apart. One of them cost about twenty times more to run.
Without the benchmark, the obvious pick would have been the most famous, most expensive one.
2. At volume, the model choice is the budget
Some jobs aren't one hard question but thousands of easy ones: grade 10,000 rows of data against the same criteria. At that scale, the price per call stops being a rounding error. A small, fast model and a frontier model can differ in price by an order of magnitude or more, so the same job costs a few euros or a few hundred, and takes minutes or most of a day.
For narrow, repetitive grading, a smaller model is often good enough. The benchmark is how you find out whether it is for yours. Have an expert grade a few hundred rows, run every candidate on those, and pick the cheapest model whose grades agree with the expert's. Then run that one on the rest.
- Measure, don't guess. Take the tokens and cost per case from the benchmark run, then multiply by the number of rows in the real job.
- Count the thinking. Reasoning tokens are billed as output even though you never see them. A model that thinks hard can cost far more per row than its price sheet suggests.
- Use the batch discount. Anthropic, OpenAI and Google all offer asynchronous batch processing at about half price. A grading job that can wait a few hours rarely needs real-time calls.
- Count the hours too. At ten seconds a row, 10,000 rows is more than a day of calls one after another. Parallel calls help until you hit the provider's rate limits.
3. How the document reaches the model matters as much as the model
The same model could score near-perfectly or poorly on the same form. What changed was the input: the original file, an image of each page, or text extracted from it.
Ticked checkboxes were the clearest case. In many documents the tick is a drawing rather than a character, so it isn't in the text layer at all.
If your work runs on forms or scans, test this before you test anything else.
- Original file. Several major model APIs, including Anthropic's, OpenAI's and Google's, accept a PDF directly and look at both its text and its page images. Support, page limits and pricing differ by provider, so "send the file" covers several different conditions.
- Page images. Render each page and send pictures. The model sees what a person sees, ticks included. Resolution matters: too low and small print blurs, too high and token costs climb.
- Extracted text. Cheapest and fastest. Loses everything that isn't a character: ticks, stamps, signatures, and often table structure.
Treat the input format as a variable in the benchmark. Run the same cases through every route your production system might use.
4. Models make human-looking mistakes in predictable places
Reading identifiers from page images, models swapped the letter O for a zero and the letter I for a one. In a summary, harmless. In an invoice, an order or a compliance record, a wrong reference that looks just like a right one.
A benchmark that keeps every wrong answer shows you where the failures cluster. Identifiers read from images are an obvious place for a validation rule or a human check.
5. Accuracy doesn't show bias
A name, a gender, an age or the language a text is written in shouldn't change a grade. Sometimes it does, and an accuracy score won't tell you. Where a 3 and a 4 both count as right, a model can give one group the 3s and another the 4s without ever being marked wrong.
To catch it, we use paired cases: the same case twice, identical apart from one detail that shouldn't matter. A fair model gives both versions the same answer. How often it does is a score of its own, reported next to accuracy.
id: tickets-014-a
test_set: tickets-paired
pair: tickets-014
question: Rate the urgency of this support ticket, from 1 to 5.
input_file: tickets/014-a.txt
varies: customer_name
expected: 4
scoring: numeric-exact, pair-match
- Change one thing. The two versions differ only in the detail under test. Change two and you won't know which one moved the score.
- Allow for noise. Models don't always give the same answer twice (see 8). A split pair only means something if pairs split more often than a single case disagrees with itself.
- Test the languages you work in. The same ticket in Finnish and in English should get the same rating. Often it's the cheapest bias test to build: translate cases you already have.
6. Small test sets stop separating the best models
On test sets with only a few cases, several top models tied at 100%. With ten cases, each one is worth ten percentage points, so first and fifth place can be a single question apart. A tie at the top means the test has run out of hard questions.
So the benchmark has to grow. When the leaders tie, add harder cases until they don't.
7. Settings are part of the result
A model's score depends on how it was run: reasoning effort, input format, how long it was given. Change one setting and you have a different result, even with the same model name.
Recording how each result was produced turned out to matter as much as the result.
Store this with every score. If a number can't be traced back to its settings, keep it out of the charts.
model: provider/model-name
model_version: 2026-08-14
reasoning_effort: high
input_format: page-images
image_width_px: 1600
max_output_tokens: 8000
timeout_seconds: 300
prompt_version: v12
benchmark_version: 2026-09-30
repeats: 3
run_date: 2026-09-30
Pin dated model versions rather than aliases like latest. An alias moves when the provider ships an update, and your old scores stop describing the model you're calling.
8. Run every case more than once
Models don't always give the same answer twice, so we run every case several times. An average of 87% sounds fine. It can also mean two cases in five fail at random.
- Mean accuracy across all runs is the obvious headline number, and it hides this problem.
- Right at least once (pass@k) tells you what a model can do. Useful when a person picks the best of several answers.
- Right every time (pass^k) tells you what it will do. Use it for anything that runs unattended.
- Spend repeats where they count. Cases that are always right or always wrong don't need many runs. Cases where a model disagrees with itself do.
What it takes
A small set of realistic tasks
Start with 20 to 50 cases, drawn from the documents, data and questions your people meet every week. Domain experts write and verify the answers. Budget their time: most of the benchmark's value comes from here.
Clear scoring rules
Check objectively wherever you can: the right number, the right article, the right term. Where judgement is needed, write the criteria down before you see any results. If the answer could change with who wrote the text, add paired cases from the start.
Repeatable runs
Same tasks, same conditions, every model. Rerun whenever a new model appears, or a provider updates one you already use.
Cost and speed next to accuracy
Plot quality against cost per case, multiplied by the number of cases you'll actually run. The model that's good enough at the lowest cost is usually the one to pick.
A plan to grow it
Add cases when you find a new failure mode. Retire the ones every model now gets right.
- Normalise before comparing. "48 213", "48,213" and "48213" are the same number. Decimal and thousands separators differ between languages, so normalise per language.
- Exact match for identifiers. Codes and references are right or wrong. No partial credit.
- Set match for lists. When the answer is several provisions, count the ones found and the ones invented separately. A missing provision and a made-up one are different failures.
- Rubrics for judgement calls. Where a second model grades free text, give it written criteria and spot-check its grades against an expert's.
Why do it
Every team using AI already runs a benchmark of sorts. Usually it's one person trying the new model on a handful of examples they remember, and going with their gut. That's how a team ends up paying twenty times more for less than half a point of accuracy. It's also how a model that rates some customers' tickets as less urgent, because of the name on them, gets into production without anyone noticing.
A written benchmark swaps the gut call for an answer you can hand to finance or legal: this model, for this task, at this cost, failing in these places.
A new model will top a leaderboard again within weeks. With a benchmark in place, finding out whether it matters for your work takes one more run.
Last updated .