Open-Weight LLMs: When Self-Hosting Makes Sense
Open-weight models have become good enough for many production tasks. Whether you should run one yourself depends on data, volume and your team.
Two years ago the choice was simple: if you wanted a capable language model, you called a commercial API. Today, open-weight models that you can download and run yourself are good enough for a large share of production tasks, from classification and extraction to summarisation and retrieval-augmented answers.
That opens a real decision for product teams. Running your own model brings privacy and control, but it also brings infrastructure, monitoring and upgrades that an API provider would otherwise handle. Here is how we think through the trade-off with clients.
Reasons to self-host
- Data residency and confidentiality. Sensitive data never leaves infrastructure you control, which simplifies conversations with security teams and regulators, particularly in healthcare, finance and the public sector.
- Predictable cost at volume. For steady, high-volume workloads, paying for compute can work out cheaper than paying per token. For spiky or low-volume use, it usually does not.
- Control over versions. The model only changes when you decide to change it, so behaviour stays stable between your own evaluations.
- Customisation. Open weights can be fine-tuned or adapted for narrow tasks in ways a hosted API may not allow.
- Offline or edge use. Some products need inference where there is no reliable connection.
There is also a softer benefit: independence from a single vendor's roadmap. Hosted models are retired, repriced and changed on the provider's schedule. An open model you run yourself stays available for as long as you need it, which matters for products with long support commitments or regulated validation processes.
Reasons not to
The strongest hosted models are still ahead on the hardest reasoning tasks. If your feature needs that level of capability, a smaller open model will disappoint. Self-hosting also means you now operate GPU infrastructure: capacity planning, autoscaling, cold starts, driver and runtime updates, security patching and on-call. For a small team with modest volume, that operational load often outweighs the benefits.
Be realistic about the total cost too. A comparison that sets per-token prices against the hourly price of a GPU leaves out engineering time, idle capacity outside peak hours, redundancy for failover and the cost of keeping up with new model releases. Model the whole picture across a year before deciding.
A hybrid is often the answer
Most of the systems we build do not choose one side. They route each task to the most appropriate model:
- A small self-hosted model handles high-volume, low-complexity work such as tagging, routing and embedding.
- Sensitive steps, such as processing personal data, run on infrastructure you control.
- Hard, low-volume reasoning tasks go to a hosted frontier model, with personal data removed first where possible.
- A common interface sits in front of all of them, so switching a task between models is a configuration change.
Our Sumus work is an example of mixing providers: the AI product-discovery engine we built uses both Ollama and OpenAI. On Dorfleben, a community platform for German towns, AI-powered search and recommendations draw on Weaviate, SageMaker and Ollama.
The routing logic itself should be simple and visible. A configuration file that says which task goes to which model, with a fallback for each, is easier to reason about than clever dynamic routing. It also makes cost reports straightforward, because each task's spend can be attributed to a specific model.
What running a model really involves
Downloading a model and running it on a laptop takes minutes. Running it reliably for users takes more:
- Serving with a runtime suited to production traffic, with batching and streaming.
- Hardware sizing based on model size, quantisation, context length and concurrent users.
- Autoscaling and queuing so traffic spikes degrade gracefully instead of timing out.
- Monitoring of latency, throughput, error rates and output quality, not just whether the server is up.
- An evaluation set rerun before every model upgrade, as with any AI feature.
- Security: network isolation, access control and logging that respects data protection rules.
Smaller models, sharper tasks
One reason self-hosting has become practical is that many production tasks do not need a very large model. A smaller model, sometimes fine-tuned on a few thousand examples from your own domain, can match a much larger general model on a narrow task such as classifying support tickets or extracting fields from invoices. It is also faster and cheaper to serve. The evaluation set decides whether that holds for your case; assumptions do not.
Embedding models are an especially good candidate. They are small, run efficiently on modest hardware, and are called very often in search and retrieval systems, so moving them in-house can reduce both cost and data exposure.
Plan the upgrade path from the start. New open models appear frequently, and moving to a better one should be a matter of running the evaluation set, comparing results and switching configuration, not a re-architecture. Keep the serving layer generic enough that a different model family can be dropped in, and keep the previous model available for a quick rollback.
Licences matter
Open-weight does not always mean open-source in the traditional sense. Model licences vary: some are permissive, others restrict commercial use, user numbers or particular applications. Read the licence for the exact model version you plan to deploy, and record it alongside the model in your inventory.
Keep the licence check repeatable. When you upgrade to a newer version of the same model family, the terms may have changed, so treat a licence review as part of every model upgrade rather than a one-off task.
A simple way to decide
Start with a hosted API to prove the feature is worth building. Build your evaluation set while you do. Once you know the task, the volume and the quality bar, test one or two open models against that set. If they meet the bar and the privacy or cost case is real, move that task in-house. If not, you have lost very little. We go deeper into the build-or-buy side of this in custom AI development vs off-the-shelf tools.
Talk to us
We design AI systems that use hosted and self-hosted models where each makes sense. See our custom AI development service or contact us to discuss your requirements.
