How it works
Prefill reads the prompt and builds the key-value cache — the intermediate state the model reuses for every later token. It parallelises well across the prompt, so it saturates the accelerator's compute units. Decode then produces one token per forward pass, each pass re-reading the model weights and the growing KV cache from high-bandwidth memory. Very little arithmetic happens per byte moved, so decode is limited by memory bandwidth, not by raw FLOPS.
Serving systems recover efficiency by batching many requests through the same weight reads, by paging the KV cache to avoid fragmentation, and by speculative decoding, where a small draft model proposes tokens that the large model verifies in one pass. All three raise tokens per second per accelerator without changing the model.
Example
A summarisation call with a 20,000-token document and a 300-token summary spends most of its money on input and most of its wall-clock latency on output. A chat agent that emits 2,000 tokens of reasoning to answer a one-line question inverts that. Two features with similar prompt sizes can differ several-fold in unit cost purely through output length — which is why prompt-stated length limits are a cost control, not a style preference.
Why it matters
Unit economics decide which AI features survive contact with production. A team that measures only requests per month cannot explain its bill; a team that measures input tokens, output tokens, cache hits and retries can usually cut spend substantially without changing models. Reasoning-heavy models complicate this further, because they bill for tokens the user never sees.