The Compute Arbitrage

Every discussion of AI infrastructure begins with training. The cluster is the unit of ambition: how many accelerators, how much interconnect, how many megawatts behind the fence. That framing made sense while capability was the scarce input. It stops making sense the moment a model enters production, because production is not a single event on a single site. It is millions of small requests, arriving at inconvenient hours, from places the model was never built near.

The economics of that shift are unglamorous and decisive. A training run is a capital project with a start and an end. Inference is a utility bill that never stops arriving. Once inference dominates a provider's cost base, the questions that matter stop being about peak performance and start being about utilisation, latency, and the price of a kilowatt-hour at three in the morning, local time.

The invoice nobody publishes

The first difficulty is measurement. Providers disclose capital expenditure readily — it is a legible signal of ambition — and disclose inference margin reluctantly, if at all. Public price lists are the closest thing to a market price, but a price list is not a cost structure. It says what a token costs a customer, not what it costs the operator to serve the next one.

What can be observed is the direction of travel. Across the major providers, list prices for a fixed capability level have fallen consistently for three years, while the volume of tokens served has risen faster still. The product of the two — revenue — has grown. The ratio between them — margin per token — has compressed. This is not a contradiction. It is what a scale business looks like when the fixed cost is enormous and the marginal cost is small but not zero.

The second difficulty is that the marginal cost is physical. Serving a request occupies memory, burns power, and — critically — holds a slot that another request could have used. An idle accelerator is a wasted asset; a saturated one is a queue. Operators therefore price two things at once: the token, and the wait. Most public pricing discussions collapse the second into the first, which is why capacity crunches appear as sudden quality degradations rather than as price signals.¹

Latency is a location decision

Training tolerates distance. A batch job can wait for a queue, a scheduler, a cheaper window of power. Inference cannot. A user who waits four seconds for the first token has already formed an opinion about the product, and no amount of downstream quality recovers it. The technical consequence is a hard floor set by the speed of light in glass — roughly two hundred kilometres per millisecond, one way, before switching and congestion take their share.

That floor is a geographic constraint disguised as an engineering one. If a first token must arrive in under a second, the practical radius for capacity is measured in hundreds of kilometres from demand, not thousands. And demand is concentrated in exactly the places where land is expensive, power is fully subscribed, and neighbours are organised.

The constraint is no longer how fast you can build a cluster. It is how long you can wait for a substation. From the report

Hence the awkward shape of the current build-out: capital flowing simultaneously toward dense urban edges, where latency is cheapest and everything else is hardest, and toward remote sites with surplus power, where the arithmetic reverses. Neither is a mistake. They are two different products — real-time interaction and asynchronous workload — that the industry has been describing with a single word.

The grid sets the timetable

Accelerators arrive in quarters. Substations arrive in years. This mismatch is now the dominant fact of the sector, and it explains behaviour that otherwise looks irrational: operators signing power agreements years before they have a customer, buying land they cannot yet energise, and paying premiums for sites that already have an interconnection queue position rather than a promising one.

Interconnection is also where the politics enters. A queue position is a public asset allocated by a process that was designed for a different kind of demand. When a single facility can ask for the load of a mid-sized city, the process becomes the de facto industrial policy, whether or not anyone intended it that way.

What we watch next

Three observable developments would tell us the market has internalised this. First, disclosure: the provider that publishes steady-state utilisation alongside revenue will force the rest to do the same, and the conversation about margins will become tractable. Second, contracting: reserved-capacity products priced against a published latency target, rather than a nominal token rate, would make quality a tradable attribute. Third, routing: the quiet margin in serving a request on the cheapest adequate hardware — smaller models, older chips, different regions — is becoming a product category of its own.

None of these require a breakthrough. They require operators to treat infrastructure as an operating problem rather than a construction one. The companies that do will look, in retrospect, less like laboratories and more like utilities with unusually good software — which is a less exciting story than the one currently being told, and a considerably more durable one.