Should You Self-Host Inference?

The honest economics of running your own models, and the breakeven most teams get wrong.

Self-hosting inference pays off in two cases: you push past roughly two million tokens a day, or your data legally cannot leave your network. Below that line a hosted API is cheaper, and it spares your team from running GPUs at all. For most companies the winning setup sits in between, a hybrid that routes each request by how sensitive and how heavy it is.

🧭 Part 10 of the ⚡ Hardware & Inference course

TL;DR

• API is the default, and under about a million tokens a day it is also the cheapest. No GPUs to babysit, and the smartest models are one call away.

• Self-hosting wins on volume and control. You buy it for data sovereignty and scale, and you pay for it in engineering time.

• The crossover sits near two million tokens a day, with one to two million a genuine toss-up. Below one million, the API wins on cost outright. Above ten million a day, owned hardware pays back in six to twelve months.

• The MLOps hire costs more than the GPU. A single MLOps engineer runs roughly 160,000 dollars a year in salary alone, next to a few thousand dollars for the hardware underneath them. Someone has to patch CUDA errors at 2am and watch latency drift, and that salary is the line item teams forget.

• Most teams land on a hybrid, and it runs 40 to 70 percent cheaper than going all-API. Sensitive and high-volume work stays local, hard reasoning goes to a frontier model over the wire.

The Three Ways to Run a Model

You shipped a feature on a hosted API. It works. Then finance forwards you the bill. It tripled in a quarter, because API spend scales with every new user you onboard. In the same week your security lead points out that customer records ride along inside every prompt, straight to OpenAI’s servers. Someone in standup finally says it out loud: should we just run our own?

It is the right question. The reflex answer, that owning hardware must be cheaper, is usually wrong.

Inference is the work of serving a trained model. Each prompt makes a GPU matrix multiply your text against the model’s weights to produce the reply one token at a time. It is the forever cost of AI, and we pulled it apart in Why is Inference Slow and Expensive?. Today we are not comparing models or serving engines. We are comparing where the model actually runs, because that one choice sets your bill, your compliance posture, and how much of your team’s week goes to infrastructure.

There are three places to run a model:

• A hosted API. Someone else owns the GPUs. You send text, you get text, you pay per token.

• A managed deployment. You rent GPUs from a provider that also runs the serving stack for you, but the model weights and your data stay inside your own cloud tenancy. An API endpoint without owning hardware or hiring ops.

• Full self-hosting. Your weights, your GPUs, your serving engine. Nothing leaves, and you own every layer and every failure.

Three forces decide between them:

• Volume: how many tokens a day you push.

• Sovereignty: whether your data can legally leave your network.

• People: whether you have engineers to keep GPUs alive.

And a fourth caps the whole thing: model access. The strongest models are closed-weight, so their makers never release the trained weights you would load onto your own GPUs. Which means the best model you can ever own is the best open-weight one, and any workload that needs a stronger model has to call a hosted API, no matter what the other three forces say. Weigh those four and the answer falls out. Let’s grade each option against them, simplest first.

Hosted API

In one sentence: someone else runs the GPUs; you send text, you get text, and you pay per token with nothing to install.

👍 The good:

1. There is no infrastructure to own, so a single engineer ships in an afternoon. You get frontier models, the GPT-5 and Claude Opus and Gemini tier, that no consumer hardware can run.

2. You pay only for what you use, with no GPU sitting idle between requests.

3. The energy cost per query is small too: independent researchers put a typical GPT-4o (OpenAI’s flagship model) query at about 0.3 watt-hours, in the same range as a web search. The heavy spend in AI goes into training the model once, and serving it afterward is cheap¹.

👎 The bad:

1. Every prompt leaves your network, and 44 percent of companies name data privacy as their top barrier to adopting AI for exactly this reason.

2. Cost climbs with usage, you also inherit rate limits and whatever pricing change the vendor ships next².

🎯 Best for: spiky or unpredictable traffic, low-to-moderate volume, and anything that needs the smartest available model.

⚠️ The ceiling: migrate off at either of two lines:

• Volume turns steady and large. Past roughly two million steady tokens a day, the per-token markup you pay the provider outgrows the GPUs and the salary you were avoiding, and renting stops being the cheap option.

• Compliance hardens. The moment records cannot leave your network, every prompt to a hosted API becomes a violation.

📡 Practitioner signal: plenty of self-hosters still run their real work through a hosted model³. One engineer on r/selfhosted keeps a local stack for play and routes anything serious to a hosted frontier model, because the token cost is trivial next to the electricity his own hardware would burn.

Managed Deployment

In one sentence: a provider runs the GPUs and the serving stack inside your own cloud tenancy, so your data stays home and you never hire an inference team.

👍 The good:

1. The data stays inside your own tenancy, which clears most compliance reviews.

2. You skip hiring MLOps, and most providers expose an OpenAI-compatible endpoint: the same request format your code already sends, so switching is a one-line base-URL change.

3. Fine-tuning usually comes built in: you upload your domain data and the provider runs the training, so you get a model specialized on your business without building a training pipeline.

4. You pay a subscription instead of dropping five figures on hardware up front.

👎 The bad:

1. You are still renting, by GPU-hour or by subscription. A dedicated H100 through a managed provider runs about 4 dollars an hour on demand, or as low as 3 with a reserved commitment, before the serving software layered on top⁴.

2. You get less control than running your own hardware, and you take on provider lock-in.

3. You can run only open-weight models. The frontier labs sell tokens and keep the weights, so there is nothing for a provider to host.

🎯 Best for: regulated teams, healthcare, finance, legal, that need data control without standing up an inference team.

⚠️ The ceiling: migrate to full self-hosting when the GPUs stop idling. At 4 dollars an hour, a rented H100 running around the clock is about 35,000 dollars a year, roughly the card’s own purchase price. Steady full utilization means you re-buy the hardware every year and never own it.

Full Self-Hosting

In one sentence: your weights on your GPUs behind your firewall, where nothing leaves and you own every layer, every upgrade, and every outage.

👍 The good:

1. Data never leaves the building, and no provider stands inside your walls either.

2. The per-token meter is gone entirely: once you have paid off the hardware, heavy and repetitive work costs almost nothing to run.

3. You can fine-tune whenever you want on hardware you already own, with no per-run bill.

4. One server can serve several applications at once: a helpdesk bot, a document classifier, a coding assistant. That spreads the fixed cost across all teams.

👎 The bad: This is where the hidden costs live.

1. The model has to fit in GPU memory. The rule of thumb is about half a gigabyte of VRAM per billion parameters at 4-bit quantization: a 70B model needs 35 to 40 gigabytes just to load, more than a single consumer card holds. A model that does not fit spills to CPU, where generation runs 10 to 100 times slower⁵.

2. Power adds up. A high-end card pulls 450 to 575 watts under load, roughly 50 to 65 dollars a month each, and that draw runs whenever the card is on, whether or not anyone is querying it.

3. Serving a team is its own wall. Raw generation calls and lightweight local tools top out at a handful of users. You need a real engine like vLLM to batch requests for a whole team, which we covered in vLLM vs Ollama vs SGLang vs TensorRT-LLM. Production benchmarks put its throughput at roughly 19 times a naive serving setup, holding P99 (the slowest 1 percent of responses) latency under 100 milliseconds at 128 concurrent requests.

4. The people cost more than the hardware. On top of it all sits the up-front spend, a pair of RTX 3090s or an H100-class card, plus the MLOps engineer to keep it alive. Glassdoor’s national average puts that engineer at about 160,000 dollars a year, several times the price of the cards they babysit⁶.

🎯 Best for: steady high volume past two million tokens a day, strict data control, or a standing need to fine-tune on private data.

⚠️ The ceiling: it stops making sense at two points:

• Too little volume. Below a million tokens a day, a hosted API beats it on raw cost: the GPU sits idle 22 hours a day, and you paid five figures for a machine that mostly waits.

• Too hard a task. The models strong enough for frontier reasoning are the ones you cannot own, so that work has to go out to an API anyway.

📡 Practitioner signal: on r/selfhosted, the running joke is that you will not save money doing this, you do it for the enjoyment.

The Decision

The questions come in veto order. Sovereignty comes first, because a hard compliance rule overrides every cost argument: if records cannot leave your network, the API is off the table no matter how cheap. People comes next, and it just picks which version of home: no ops team means a managed deployment, an ops team means full self-hosting. Only then you reach the volume question: past two million tokens a day, owned hardware pays back; under that, the API stays cheaper. And model access caps it: when a slice of the work needs frontier reasoning, that slice goes to an API even if everything else stays home. That last branch is the hybrid, and it is where most teams end up.

The Tradeoffs Side by Side

The breakeven also assumes you are escaping frontier prices. Run 10 million tokens a day through gpt-5.4-nano, the cheapest tier OpenAI sells, and the API bill lands in the low hundreds of dollars a month at list prices⁷. A single always-on rented H100 at the going four-dollar hourly rate runs about 2,900 dollars a month, before anyone is paid to keep it alive. So the two-million-token crossover holds only when the workload needs frontier-tier reasoning; hand the job to a lightweight model and self-hosting never catches up.

Shared infrastructure is the other big lever. Three internal apps at 700,000 tokens a day each add up to 2.1 million, which clears the breakeven even though no single app would justify a GPU on its own. The fixed cost is the same; spreading it across workloads is what makes self-hosting pencil out.

The Hybrid Architecture

Production setups usually land on a hybrid. Teams that split their traffic report 40 to 70 percent savings against an all-API stack, sometimes more. The split is simple to reason about. Local models handle the simple, high-volume, and sensitive work: classification, data extraction, processing documents full of personal data. The frontier API takes only the rare task where reasoning quality is business-critical and the volume is low.

Now the honest take. The hardware case keeps getting stronger and open models keep closing the gap on frontier reasoning. But the small-scale dream of self-hosting to dodge a 20-dollar subscription is a fantasy: the card alone costs more than years of that subscription would, before you have paid a cent of electricity. The hybrid pitch earns the same scrutiny. A router is one more system to build, test, and keep alive. A misclassified request can also send sensitive data straight to the external API you built the local lane to avoid. And most self-hosting failures are self-inflicted: a model that does not fit VRAM, a serving engine that handles four users when you have forty, an MLOps gap nobody budgeted for.

If the math points you toward self-hosting, SIE is the shortest path to trying it. Superlinked’s open-source inference engine just shipped an Apple Silicon runtime, so the whole stack, embeddings, rerankers, OCR, and open LLMs, now runs on the Mac you already own. When the prototype earns production, the same engine scales out on GPUs, with Helm on your cluster or Terraform on AWS, GCP, and Azure. Apache 2.0, 112 models, and prompts never leave your cloud.

👾 Read the quickstart ⭐ Star it on GitHub

The One Thing to Remember

For most teams, self-hosting is buying control, and control has a cost. You keep the data inside your walls; in exchange you take on the idle GPUs and the people who keep them alive. Decide on volume and sovereignty, then let the token math make the call.

---

💬 Self-hosting in production, or tried it and bailed? Tell us what the bill and the ops actually looked like.

---

Where to Next?

• 🔗 Go simpler: Why Does AI Need a GPU?, the parallel-arithmetic reason inference needs this hardware at all.

• 📖 Go deeper: The Open-Source Agent Toolkit in 2026, the seven-layer open-source stack that runs on hardware you own.

• 🔀 Related: What Does NVIDIA Actually Do?, the business you are stepping into the moment you own the cards.

🔜 Up next: What is an Inference Provider? the rent-don’t-own layer that sits between a hosted API and your own GPUs.

FAQ

Is self-hosting cheaper than a hosted API?
Only at scale. Below one million tokens a day, the API is cheaper because you avoid idle hardware and an ops salary. Between one and two million, it is close enough to call either way. Past two million, self-hosting starts to win, and past ten million a day the hardware usually earns itself back inside a year. The cost case is about volume, so run the token math before you buy a card.

What hardware do I need to run a useful model?
Enough VRAM to fit the model. The rule of thumb is about half a gigabyte per billion parameters at 4-bit quantization, so a 13B model lives comfortably on a 24GB card. Miss the fit and inference spills to CPU and crawls 10 to 100 times slower. A well-quantized smaller model on a single card beats an oversized one that does not fit.

Can I self-host GPT-5 or Claude?
No. The frontier models from OpenAI, Anthropic, and Google stay closed and API-only. Self-hosting means open-weight models like the Qwen, Llama, and GLM families. They are strong and close most of the gap for everyday work, which is why the hybrid pattern keeps frontier reasoning on an API and runs everything else locally.

How many users can one GPU serve at once?
That depends on the serving engine as much as the card. A naive setup serves a few people before requests start queuing. A production engine like vLLM batches requests and shares GPU memory across them, lifting the same card to many concurrent users with stable latency. Pick the engine before you blame the GPU.

Does self-hosting actually keep data private?
It can, if you isolate it. Keep the inference server on a private network behind an internal gateway, add real access control, and the data never leaves. The common leak is the knowledge base: treat a shared vector index as one pool and one user’s query can surface another’s confidential documents. Filter retrieval by the requester’s permissions.

1. How much energy does ChatGPT use?, Epoch AI (February 2025)

2. Self-Hosted LLM: Complete Guide, Costs, Architecture & Lessons Learned, Alpacked (June 2026)

3. Self-Hosting AI Models: Lessons Learned, r/selfhosted

4. Pricing, Together AI (current as of July 2026)

5. Self-Hosted LLM: Complete Guide, Costs, Architecture & Lessons Learned, Alpacked (June 2026)

6. MLOps Engineer: Average Salary & Pay Trends, Glassdoor (2026)

7. Pricing, OpenAI (current as of July 2026)