What is Inference?

Why reading is fast and writing is slow

Inference is running a trained model to get an answer out of it, as opposed to training, which is teaching it in the first place. For an LLM it happens in two phases. The first reads your whole prompt in one go. The second writes the answer one token at a time. Almost every cost and latency question traces back to that difference.

🧭 Part 13 of the ⚡ Hardware & Inference course

TL;DR

You read a page in seconds and write one in minutes. Inference splits the same way, for the same reason.

• Reading is parallel: your eyes take in a whole line at once, and the words in it do not queue up one at a time. The model reads your entire prompt the same way, in one trip through the model.

• Writing is serial: you cannot choose word five until word four exists. The model has the same constraint, so it takes one full trip through the model for every single token it writes.

• The margin notes: you do not reread the whole page before each new sentence, you glance at the notes you made while reading it. The KV cache is those notes, and it is why token 500 does not cost 500 times token one.

• The margin fills up: the notes grow by one entry per token, read and written alike, and they sit in GPU memory next to the model. Running a model for many users at once is mostly a fight over that space: when the margin is full, new requests wait.

• The catch: reading keeps the chip busy doing math. Writing keeps it busy hauling the model out of memory, with the math part idle. A fix for one does nothing for the other, and almost every optimization you read about is a fix for exactly one of them.

Before Inference, There Was Training

Training and inference are the same model doing two different jobs. Training changes the weights, the billions of numbers inside the model, by feeding it examples and nudging each number toward less wrong, over and over for months. Inference only reads them.

For a classifier, that single read is the whole story: one input, one trip through the weights, one label. Inference for those models is so cheap that teams run it on CPUs and never think about it again.

An LLM breaks that pattern because it cannot produce a whole answer in one trip. It produces one token, then uses that token to produce the next. Why Does AI Need a GPU? covers why a single trip wants a GPU at all. What follows is why every token needs a trip of its own.

Why Ten Thousand Tokens In Is Cheaper Than Sixty Out

You paste a 10,000-token document into the chat and ask for a three-sentence summary. For a moment, nothing. Then the answer streams in, a word at a time, like someone typing at the other end.

You have watched the two phases of inference a hundred times. You just never had names for them.

The pause is the model reading. None of the 10,000 tokens is waiting on any other, so the GPU pushes the lot through every layer of the model in one trip.

The streaming is the model writing. It cannot pick the second word until the first exists. A sentence runs about twenty tokens, so three sentences is about sixty, and that is sixty more trips through the model, each one waiting for the last.

Ten thousand tokens in on one trip. Sixty out on sixty. That gap is why every provider charges more for output than for input.

How Inference Actually Works

That gap comes from three moving parts.

Prefill is the reading half, the pause you watched before the first word. The work inside it is multiplying big grids of numbers at once, which is what a GPU is fastest at, so what limits prefill is how much arithmetic the chip can do per second. Engineers call that compute-bound. Prefill produces the first output token and fills the cache.

The cache is what makes everything after the first token cheap. Every token the model reads or writes leaves behind two short lists of numbers: a key, the label later tokens use to find it, and a value, what they take from it once found. K and V are those two words, and the KV cache is where the model keeps them. Neither ever changes once computed, so rebuilding them for every new token would repeat the same arithmetic thousands of times. Token 500 looks up its 499 predecessors instead. The cache also explains prefill’s name: it fills the cache in advance, before any writing starts.

Decode is the writing half. The name says what it does: the model’s answer comes out as raw numbers, and decode turns those back into tokens you can read. It runs one token at a time. Each trip pulls the model’s weights, all billions of them, out of GPU memory into the math units. Then it does a little arithmetic against one new token, reads the cache, and emits the next token. The arithmetic is trivial. Moving the weights is not, and that move happens on every single token. Engineers call that memory-bound: the limit is the memory bus, the link carrying data from GPU memory into the math units, and those units spend most of their time waiting on it.

Each phase idles the resource the other one needs. Prefill leaves the memory bus with slack. Decode leaves the math units waiting. Why is Inference Slow and Expensive? takes the economics of it further.

Your request never runs alone. The server pushes it through the model alongside dozens of others, and one haul of the weights out of memory then pays for everyone’s next token at once. Sharing the haul is the only reason decode is affordable at all.

But every one of those requests keeps a cache of its own, in the same GPU memory the weights sit in, and each cache grows by an entry per token. A conversation 100,000 tokens long holds a cache a thousand times the size of a 100-token question’s. So the memory fills. New requests then wait outside, and the ones already running keep growing. When they need more room than remains, the server picks one, deletes its cache, and makes it start over when its turn comes back.

Who’s Actually Building With This

vLLM, the open-source program that runs models on a server, is where the memory fight got solved. Servers used to reserve each new request enough memory for the longest that conversation could ever get. A one-line question got the same space as a 100,000-token conversation, in case it turned into one. Almost none of them did. So 60 to 80 percent of the GPU’s memory sat empty while new requests queued outside for space¹.

vLLM borrowed the fix from operating systems, the same trick that lets your laptop keep more programs open than its memory should allow. A request now reserves nothing up front. It takes memory in small fixed-size chunks called pages, grabs another whenever it fills the last one, and hands them all back the moment it ends. Empty space dropped from 70 percent to under 4, and the same GPU began serving two to four times as many tokens per second².

Two to four times as many tokens per second is the number to remember: no new hardware, no smaller model, no quality tradeoff. The whole gain came from not wasting memory they already owned. vLLM vs Ollama vs SGLang vs TensorRT-LLM compares the engines that grew out of this work.

Two different requests can also point at the same pages. Most apps send the same long instructions ahead of every user message, the system prompt. When two requests use the same system prompt, they point at the same pages for it instead of each keeping a copy, and the prefill arithmetic for those tokens runs once. That is prefix caching, and it is why an API charges less for a prefix it has seen before.

What Can Go Wrong (and What’s Overhyped)

Four things account for most of the confusion.

1. Buying arithmetic for a memory problem. Generation is slow, so you order a chip that does more arithmetic per second. It arrives and nothing gets faster. Decode’s slow part is hauling weights out of memory; the arithmetic done on them is trivial. So the extra power sits idle, and what you needed was a wider path out of memory. The exception is scale: batch enough requests together and one haul finally feeds enough arithmetic to keep a faster chip busy.

2. Benchmarking one request at a time. One request on its own leaves most of the GPU idle while it writes, so the tokens per second you measure that way is a number your production traffic will never see. Measure the thing that actually limits you: how many requests the GPU holds at once before their caches fill its memory.

3. Confusing the length you allow with the length you get. Raising max_model_len, the longest request your server will accept, costs nothing by itself: a paged cache takes memory only as tokens actually arrive. What costs you is traffic that shows up long. Every request holds a cache the size of its real length, so the day your average conversation doubles, you serve half as many people at once on the same GPU.

4. Optimizing the phase that is not your bill. Chat sends short prompts and writes long replies, so chat is almost all decode. Classification sends long documents and answers in one word, so it is almost all prefill. Buy the fix for the wrong one and your bill does not move.

The loudest claim right now is that quantization answers all of it. Storing each weight in fewer bits does help, and it helps decode most, since smaller weights mean fewer bytes to haul on every token. What is Quantization? covers the mechanics. What it cannot do is change the two facts this whole issue rests on: still one trip per token, still a cache that grows with every one. Squeeze the weights far enough and the answers get worse, which is a high price for speed that batching hands you for free.

🏗️ Engineering Lesson: before you swap a model or buy a GPU, find out whether reading or writing is your slow half. Reading runs out of arithmetic. Writing runs out of room on the path from memory. Aim your fix at the wrong half and nothing changes, except the invoice.

The One Thing to Remember

Training is a thing you pay for once. Inference is a thing you pay for on every request, forever. The bill is the symptom; the cause is one sentence. A model cannot know its fifth word until it has committed to its fourth. Everything expensive about serving an LLM follows from that one constraint, and no amount of hardware removes it.

💬 Which phase is your bottleneck, and how did you find out?
Tell me in the comments.

Where to Next?

• 📖 Go deeper: Should You Self-Host Inference?, what running this yourself actually costs.

• 🔀 Related: What is a GPU?, the hardware both phases run on.

• 🔗 Prerequisite: What Does NVIDIA Actually Do?, who supplies the chips and the software under them.

🔜 Friday: How Netflix Serves LLMs In-House. Four decisions behind running the full stack yourself, and the production gaps the design phase missed.

FAQ

What is inference in machine learning?
Inference is running a trained model to get a prediction, as opposed to training, which produced the model. Training changes the model’s weights and inference only reads them. For a classifier it is one trip through the weights. For an LLM it is one trip for the whole prompt, then one more for every token of the answer.

What is the difference between prefill and decode?
Prefill reads your entire prompt in one trip and produces the first output token. The chip’s arithmetic sets its pace. Decode then writes the rest one token at a time, each trip pulling the full weights out of memory, and it is limited by memory bandwidth. Speeding up one does nothing for the other.

What is a KV cache?
For every token it processes, the model computes a key and a value, two short lists of numbers. The KV cache stores them so later tokens look them up instead of recomputing them, which is why generating token 500 does not redo the work of the previous 499. It grows by one entry per token and sits in GPU memory, which makes it the main limit on how many requests a server holds at once.

Why do output tokens cost more than input tokens?
Your whole prompt goes through the model in one trip. Every output token needs a trip of its own. The raw gap is enormous, but a server pulls the weights once for a whole batch of users, so the cost of that pull gets split many ways. In practice output bills about three to eight times input.

Does a faster GPU speed up generation?
Only if arithmetic is your limit, and during decode it usually is not. Decode spends its time pulling weights from memory into the math units, so memory bandwidth sets the pace. Large batches change that, because one pull then feeds many requests. Measure which phase is slow, and at what batch size, before choosing hardware.

1. Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv (September 2023)

2. vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention, vLLM (June 2023)