Three years ago, running a capable language model in production was a line item that required executive sign-off. Today, the same quality of generation costs less than the logging infrastructure around it. Open-weight models, aggressive pricing from API providers, and serving optimizations — quantization, speculative decoding, batching — have driven the marginal cost of a token toward zero.

And yet no enterprise AI budget has shrunk. Understanding why requires separating the price of a token from the cost of a system.

Why it matters

Cost curves determine what gets built. When inference was expensive, AI features were rationed: a summary here, a draft there. At current prices, always-on assistance, per-document analysis and multi-step agent workflows are economically trivial at the model layer — which is precisely why the bottleneck moved to everything around the model.

For buyers, the shift changes what to negotiate. API price cuts make headlines, but the contracts that matter now cover context windows, retrieval infrastructure, evaluation tooling and the engineering time to make outputs trustworthy.

How it works

Three forces compounded. First, model efficiency: distillation and better training data let small models match 2023's flagships on most business tasks. Second, serving efficiency: techniques like continuous batching and KV-cache reuse multiplied the throughput of the same GPU. Third, competition: open-weight releases gave every provider a pricing ceiling, and inference-focused chips gave them a cost floor to chase.

Meanwhile, usage patterns grew more expensive in shape. Agentic workflows make dozens of model calls per user request. Long-context applications push hundreds of thousands of tokens through per session. Retrieval pipelines add embedding, reranking and storage costs. The token got cheaper; the conversation got longer.

Evidence

Public API price histories show the trajectory plainly: flagship-model input prices that were tens of dollars per million tokens in 2023 are now measured in cents for capable tiers, and open-weight models can be self-hosted at costs dominated by hardware amortization rather than licensing. Cloud providers' earnings calls, meanwhile, describe AI revenue growing faster than any price decline — the signature of demand elasticity, not a shrinking market.

Enterprise surveys tell the same story from the other side: the top reported cost drivers in production AI systems are data pipelines, evaluation and monitoring, and integration engineering — not model API bills.

The competing read

One camp argues the collapse in token prices is deflationary and will eventually show up in budgets: as models commoditize, spending concentrates in a few winners and total industry revenue disappoints the infrastructure build-out. The other camp points to every prior computing cost collapse — storage, bandwidth, compute — where cheaper units produced larger markets, and argues AI is following the same curve with a longer fuse.

Both can be right at different layers: commodity generation may be a low-margin utility while the systems built on it capture the value.

What happens next

Watch the mix shift from generation to verification. As output becomes nearly free, the scarce resource becomes confidence in output — evaluation, grounding, audit trails. The vendors who price that layer, rather than the token, are betting on where the durable margin lives.