Inference Costs Keep Falling ~10x a Year — What That Actually Changes

The efficiency trend behind this year's model releases is easy to state and easy to undersell: getting a fixed level of capability now costs roughly a tenth of what it cost a year ago, and that pace has held for multiple years running. It shows up in list prices -- Google's Gemini 3.8 Flash, released September 2 as an agentic/coding/reasoning-optimized model, is priced at $0.75 / $3.75 per million input/output tokens, a fraction of what comparable frontier capability cost when this class of model first appeared.

The direct effect is obvious: things that were too expensive to run per-request now aren't. The less obvious effect is architectural. A lot of 2024-2025-era system design in this space treated model calls as the scarce, expensive resource to be minimized at all costs -- aggressive caching, one big call instead of three small ones, avoiding a verification pass because it doubles spend. Some of that reasoning stops applying, quietly, as the cost floor keeps dropping. A three-step plan-execute-verify loop that was a hard sell on cost grounds eighteen months ago can be a rounding error today.

That doesn't mean cost stops mattering -- it means the place it bites moves. Infrastructure investment is chasing the same curve from the supply side: Positron raised $875M specifically to challenge Nvidia on inference economics with a chip design built around commodity memory rather than the scarce, expensive HBM that constrains current accelerator supply. If that bet pays off, it's another turn of the same crank -- cheaper inference at the hardware layer, not just the model layer.

The practical implication for anyone budgeting a product: a cost-per-request number from even a year ago is not a safe planning input today. Rebudget against current pricing before deciding a feature isn't affordable, and revisit that decision on a cadence that assumes the floor keeps moving, because on this year's evidence it will.