How to Choose an LLM: Testing Models on Your Own Data
Benchmarks tell you which model is good in general. Your product needs to know which one is good at your task, at your cost, on your data.
A new language model is announced almost every week, each with a chart showing it ahead on some benchmark. For a team building a product, those charts answer the wrong question. You do not need the best model in general. You need the model that does your task well enough, at a cost and speed your product can sustain, and that keeps doing so when the provider ships an update.
The only reliable way to find that model is to test candidates on your own data. This post describes how we set up those comparisons for clients, from building the evaluation set to running side-by-side tests in production.
The cost of choosing badly is rarely dramatic. It shows up as a feature that is slightly too slow, slightly too expensive or slightly too unreliable to roll out more widely, and a team that cannot say whether a different model would fix it. An evaluation process answers that question in days rather than leaving it to opinion.
Why public benchmarks mislead
Benchmarks measure broad capabilities on fixed datasets. Your workload is narrower and messier: domain vocabulary, specific output formats, documents in several languages, users who phrase things badly. A model that leads on general reasoning can still struggle with your extraction schema, and a smaller, cheaper model can be entirely adequate for classification. Benchmark data may also have leaked into training sets, which inflates scores without improving real-world behaviour.
Build an evaluation set first
Everything rests on a set of test cases that represent what the system will actually see. It does not need to be large to be useful; a well-chosen set of a hundred or two examples beats thousands of random ones.
- Real inputs. Pull from production logs, support tickets or documents, anonymised where needed.
- Known good outputs or clear grading criteria for each case, written by people who understand the domain.
- Hard cases on purpose. Ambiguous requests, missing information, long inputs, other languages, attempts to make the model misbehave.
- Tagged by category, so you can see that a model is strong on summaries but weak on tables rather than just reading one average.
Write the evaluation set before you look at any model's output. If you build it after seeing what a favourite model does well, the set will quietly favour that model. Keep a held-out portion that you do not use while tuning prompts, so you have an honest final check.
Decide how you will grade
Grading is where most evaluations get sloppy. Match the method to the task:
- Exact or programmatic checks for structured output: valid JSON, correct fields, values that match a reference.
- Reference comparison for extraction and classification, scored automatically.
- Model-graded rubrics for open-ended answers, with a separate model scoring against written criteria. Spot-check those grades by hand; graders have biases too.
- Human review for a sample of cases, especially anything customer-facing or high-stakes.
Record cost per request and latency alongside quality. A model that is slightly better but three times slower may be the wrong choice for an interactive feature and the right one for an overnight batch job.
Prompts are part of the comparison
Different models respond differently to the same prompt. A prompt written and polished for one model can make another look worse than it is. Give each serious candidate a fair round of prompt adjustment against the development portion of the set, then compare them on the held-out portion. Otherwise you are measuring your prompt, not the model.
Also run each case more than once. Model output varies between runs, even at low temperature, and a model that is right four times out of five on a case behaves very differently in production from one that is right every time.
Compare offline, then test online
Offline evaluation narrows the field to two or three candidates. It cannot tell you how real users respond. For that, run an online comparison: route a share of live traffic to each candidate and measure what matters for the product, such as task completion, edits users make to generated text, thumbs up or down, escalations to a human, or conversion.
Keep everything else constant. If you change the prompt and the model at the same time, you will not know which one moved the metric. Log the model version and prompt version with every request, so results can be sliced cleanly afterwards.
Decide in advance what result would make you switch. For example: the challenger must match quality within a small margin on every category and reduce cost or latency meaningfully. Writing the rule down before the test prevents the team from reading whatever it hoped for into the numbers afterwards.
Design for switching
Whatever you choose today will not be your choice in a year. Put a thin abstraction between your application and the model provider, keep prompts in versioned configuration rather than scattered through code, and rerun the evaluation set whenever a provider updates a model. That turns a model change from a risky migration into a routine pull request.
For retrieval-based systems, evaluate retrieval and generation separately; many quality problems blamed on the model are actually search problems. We cover that in our RAG development guide.
Watch the operational details
- Context window and long inputs: test with your longest realistic documents, not just typical ones.
- Structured output support: some models follow a JSON schema far more reliably than others.
- Data terms and region: confirm where data is processed and whether it may be retained.
- Rate limits and availability: check quotas against your expected peak traffic.
- Deprecation policy: know how much notice the provider gives before retiring a model version.
What a good outcome looks like
At the end of a model selection exercise you should have a written comparison with quality scores by category, cost and latency per request, a recommended default, a fallback provider and an evaluation set that keeps running in CI. That is worth more than any leaderboard position.
Need help choosing?
We run model evaluations as part of custom AI development projects, and we are happy to help you set up your own. Contact us to talk about your use case.
