KM4BI KM4BI

I Burned 500 Million Tokens Last Week. Do You Know Yours?

AI buying is becoming capacity buying: tokens, HBM, packaging, power, routing and allocation terms now matter as much as software features.

Nate B Jones frames modern AI as an industrial capacity business rather than a pure software category. Every generated answer depends on a physical chain — chips, HBM, packaging, optics, power, cooling, data centers and operations — that turns demand into served tokens.

Microsoft is the clearest example. Even with planned capex of $190 billion, the company still expects to be capacity constrained. The bottleneck is not simply the GPU; it is the surrounding system that makes accelerators usable at scale, especially memory, packaging, power and data-center delivery.

That shift changes AI procurement. Buyers should ask whether a vendor offers reserved capacity or best-effort access, what allocation tier applies, what happens during upstream shortages, and how dependent the product is on a specific hyperscaler. Cloud providers can be suppliers and competitors for the same compute pool, because they must allocate capacity across their own products and external customers.

Demand forecasting also needs a new unit. Seats and licenses are not enough; teams need to estimate tokens per workflow, context length, model calls, agent loops, concurrency, retries and latency tiers. The video’s example of nearly 500 million tokens used in one week shows how advanced agentic workflows can overwhelm traditional budgeting assumptions.

Serving costs are improving through distillation, caching, batching, quantization, routing and software optimization. But cheaper tokens can trigger even more usage, especially with longer contexts and more autonomous agents. The practical question is not just whether AI is expensive, but where the capacity chain could break and what contractual protection exists if it does.

For the next investment review, the useful checklist is: how much AI spend is truly reserved capacity, what fallback plan exists during provider constraints, how will routing to cheaper models be measured without hurting users, and where hidden human supervision is masking product failure.

Source