Sooner or later, every team running an LLM feature in production hears the question: should we keep calling a provider's API, or run an open-weight model on our own infrastructure? It usually follows a customer's security questionnaire, a growing usage bill, or unease about depending on one vendor. Each is a fair reason to look; none settles the answer on its own.
This guide is the set of questions we would work through with a CTO before either path is chosen. It deliberately leaves out prices. They change too often to be useful here, and the decision rarely turns on a list price. It turns on data, quality, load and the people you have to run it.
What does self-hosted actually mean here?
The term covers three different setups, and mixing them up leads to bad comparisons.
- Hosted API: you send prompts to a model provider such as OpenAI, Anthropic or Google and pay per use. The provider runs the model and the hardware.
- The same models through your cloud: several providers offer their models inside the major cloud platforms, so traffic stays within your existing cloud account, region and contracts. The provider still controls the model.
- Self-hosted open weights: you download an open-weight model and run it on GPUs you rent or own, behind a serving engine. vLLM, for example, is an open-source engine whose HTTP server is compatible with OpenAI's Completions and Chat Completions APIs, so application code can often move between the two with small changes.
Many teams who say they need to self-host actually need the middle option.
Should I self-host an LLM or use a provider's API?
Start with the API unless something rules it out. Self-host when contracts or regulators forbid any third-party processing, when an open-weight model you can serve meets your quality bar on your own evaluation set, when traffic is steady enough to keep GPUs busy, and when you have engineers to run, patch and be on call for the serving stack. Check providers' zero-retention and residency terms first, read the open model's licence, and consider a hybrid that routes sensitive or simple requests to your own model.
Is data control a reason to self-host?
It is usually the first reason given, so check what the providers' current terms say before assuming the API is ruled out. As of November 2026:
- OpenAI's documentation says data sent to its API is not used to train its models unless you opt in, that abuse monitoring logs are kept for up to 30 days by default, and that zero data retention is available to approved customers on eligible endpoints. It also offers regional data residency, with storage and processing in some regions and storage only in others.
- Anthropic's documentation describes a zero data retention arrangement for eligible API features and states that retained data is never used for model training without your express permission. It also offers HIPAA-ready API access under a signed business associate agreement.
- Google Cloud's documentation for its generative models states that customer data is not used to train or fine-tune models without permission, and lists the settings and features, including some caching and grounding options, that you must leave off or avoid to reach zero data retention.
The detail matters more than the headline. Zero retention usually needs approval or a contract change, and covers some endpoints and features but not others (stored conversations, file uploads or batch jobs may sit outside it). Read the eligibility tables and get the terms into your contract.
Self-hosting genuinely wins when your rules say data must never leave infrastructure you control, when you serve customers who will not accept any third-party processor, or when you must run somewhere no provider covers. If you handle health data, our HIPAA checklist for LLM features lists the questions to put to any provider first. Self-hosting also moves the data problem rather than removing it: your own logs and traces now hold the sensitive text, so what you log from an LLM app needs the same care on either path.
Will an open model be good enough for your task?
This is the question teams skip, and it most often ends a self-hosting plan. Public benchmarks do not tell you how a model performs on your documents, your users' questions and your output format. Only your own evaluation set does.
- Build a test set from real or realistic inputs, with expected answers or scoring rules, before you compare anything. Our eval-first approach describes how.
- Run the hosted model you use today alongside two or three open-weight candidates of a size you could realistically serve.
- Score each task segment separately, not only the overall average. An open model can match on extraction and fall behind on long multi-step reasoning, or the other way round.
- Test the whole pipeline: tool calling, structured output, long context and the languages your users write in.
- Decide in advance what gap you would accept, and what it would take to close it.
If the gap is real, fine-tuning a smaller open model on your task can sometimes close it, but that adds a training pipeline you must maintain. Our comparison of RAG and fine-tuning covers when that is worth it.
How do volume and latency change the picture?
Your usage pattern shapes the economics more than any rate card does.
- Steady, high volume makes owned or reserved capacity easier to justify, because GPUs that stay busy all day are well used.
- Spiky or low volume favours an API, because you pay for what you use and someone else absorbs idle capacity and peaks.
- Latency can go either way: a nearby self-hosted model avoids network hops and shared queues, but an undersized cluster will be slower than a well-run API. Measure time to first token and total response time under your real concurrency.
- API rate limits and quotas are a capacity constraint too. Check them against your peak, not your average.
Self-hosted cost drivers are GPU capacity (idle time included), redundancy, monitoring and engineer time. API cost drivers are token volume, long prompts and retrieved context, retries, and agent loops that call the model many times per request. Model both against your own traffic.
Do you have the team to run GPUs and serving?
Running a model in production is an operations job, not a deployment step. Someone has to own all of this:
- Capacity planning and autoscaling, including what happens when GPUs are not available in your region.
- Serving configuration: batching, quantisation choices and context limits, with a quality check after each change.
- Uptime: health checks, failover, on-call and incident response for a component that used to be someone else's problem.
- Security: patching the serving stack, isolating the model network, and controlling who can change weights or configuration.
- Licence obligations for every model you run (see below).
If that would land on one engineer who also ships features, the honest answer is usually the API or a managed cloud option for now. Growing the skill in-house versus bringing in a partner is covered in our guide to in-house versus outsourced SI teams. Much of MLOps and SI infrastructure work is exactly this unglamorous layer.
What do open-model licences require?
Open weights does not mean no conditions. Licences differ between model families, and sometimes between models in the same family.
- Some use standard open-source licences. Mistral, for example, released its Mistral 3 family (Mistral Large 3 and the Ministral 3 models) under Apache 2.0.
- Others use custom community licences. The Llama 4 Community License requires users to follow Meta's acceptable use policy, requires redistributors to display "Built with Llama", and requires companies with more than 700 million monthly active users to request a separate licence from Meta.
- Check the licence of the exact model version you deploy, and have counsel read it, before it reaches production.
What happens when models change?
Providers retire and replace hosted models on their own schedule, so an API user must plan for forced migrations. Self-hosting gives you control over timing: the weights you run today will still be there next year. The flip side is that improvements do not arrive by themselves: each upgrade means evaluating, re-tuning prompts, load testing and rolling out. On either path, the same evaluation suite should gate every model change.
Is a hybrid setup the practical answer?
For many teams it is. Common patterns:
- Route by sensitivity: requests containing regulated data go to a self-hosted or in-region model, everything else to a hosted API.
- Route by difficulty: a smaller self-hosted model handles high-volume simple work such as classification or extraction, and a larger hosted model takes the hard cases.
- Fallback: a self-hosted model keeps a reduced service running if the provider has an outage, or the reverse.
- One interface: a thin gateway between your application and every model, so switching providers or adding a self-hosted model is a configuration change, not a rewrite.
Hybrid adds moving parts. It needs an evaluation per route and clear logging of which model answered each request.
A short decision checklist
- Do our contracts, regulators or customers forbid any third-party processing? If so, self-host or use an in-region managed option that satisfies them.
- Do the provider's zero-retention and residency terms, in writing, cover the endpoints and features we use?
- Does an open model we can serve meet our quality bar on our own evaluation set?
- Is our traffic steady enough to keep GPUs busy?
- Who will run, patch and be on call for the serving stack?
- Does the licence of the exact model allow our use?
- How will we evaluate and roll out the next model change?
Where Octalcode fits
Octalcode is a studio in Lahore, with a US presence in Austin, that builds LLM applications, SI agents and RAG systems and the infrastructure beneath them. On this question we would rather help you run the comparison on your own data and read the terms than argue for either option in advance.
If you are weighing the move, our MLOps and SI infrastructure work can start with that comparison: your evaluation set, your traffic shape and your contracts, ending in a written recommendation you can act on with or without us.