Anthropic, OpenAI, Google and xAI each release new models several times a year, and every release comes with benchmarks that claim the lead. If you are adding AI to a product or a process, it is tempting to ask which one is "the best".
The more useful question is: which one is best for this particular job, at this volume, with this data? The answer is often different for different parts of the same project. Here is how we approach the choice.
Start from the task, not the brand
Most business uses of AI fall into a few kinds of work, and they reward different strengths:
- Sorting and extracting. Classifying emails, pulling fields out of invoices, tagging products. These are high-volume and fairly simple. Small, fast and inexpensive models usually do them well.
- Answering from your content. Customer and staff assistants, knowledge base search. Here what matters most is following instructions, staying faithful to the sources and admitting when the answer is not there.
- Reasoning and writing. Drafting proposals, comparing contracts, planning multi-step work. Larger models earn their higher price here.
- Acting through tools. Agents that check availability, update a CRM or run a workflow. Reliability with tools and structured output matters more than eloquence.
What to compare
- Quality on your own examples. Public benchmarks are a starting point, not an answer. A model that tops a leaderboard can still struggle with your product names or your customers' Greek.
- Language. If your customers write in Greek, test in Greek. Quality varies between models more in smaller languages than in English.
- Speed. A chat assistant needs to start answering within a second or two. A nightly batch job does not.
- Cost at your volume. Prices are per amount of text processed, and they differ widely between model sizes. Multiply by your real monthly volume before deciding.
- How much it can read at once. Some tasks need long documents in a single request; others only need a paragraph.
- Data handling. Check the provider's terms for business use: whether your data is used for training, how long it is kept, and where it can be processed. For some clients, EU processing or a self-hosted model is a requirement.
Test with a small set of real cases
Before choosing, we build a test set: 30 to 100 real examples from the client's own work, each with what a good answer looks like. We run the candidate models through it and compare quality, speed and cost side by side. It takes days, not weeks, and it replaces opinions with evidence.
The same test set keeps paying off later. When a provider releases a new model, we run it again and know within an hour whether switching is worth it.
Why we stay provider-agnostic
Models leapfrog each other. The one that wins today may be second in six months, and prices tend to fall. So we build the AI parts of a project in a way that lets us change the model without rewriting the product, and we often use more than one: a small model to route and sort, a larger one for the hard cases.
This also protects you. If a provider changes its terms, raises prices or has an outage, you have somewhere else to go.
What about open-source models?
Open-weight models can run on infrastructure you control, which matters when data must not leave your environment. The trade-off is that you take on the hosting and the upkeep, and the largest commercial models still lead on the hardest tasks. For many businesses, a commercial API with the right data settings is the practical choice; for some, a self-hosted model is the only acceptable one.
In short
- Define the task and what a good result looks like.
- Shortlist two or three models that fit the task, the language and your data requirements.
- Test them on your own examples, and compare quality, speed and cost.
- Build so you can switch, and re-test when new models arrive.
We work with models from Anthropic, OpenAI, xAI and Google, as well as open-source models, and choose per task. See our AI services, or tell us what you want to automate.