The GPU Hour Is the Wrong Unit of Compute
A GPU hour is a clear unit of resource occupancy. It can support billing, capacity reservations, and straightforward comparisons between equivalent configurations. The inherited mistake is to treat it as a standardized unit of productive computation. Devices differ, but even an identical accelerator can deliver different useful output depending on memory capacity, network topology, software, contention, and the workload's latency requirements. Time spent holding the resource does not specify the service obtained from it or the constraints under which that service was delivered.
Consider two illustrative offerings running the same model at the same quality and request mix. One costs two dollars per hour and sustains one hundred requests per second within the required latency limits; the other costs three dollars and sustains two hundred. Their compute rental costs per million compliant requests are approximately $5.56 and $4.17. The more expensive hour buys cheaper work. These figures exclude other costs deliberately. A procurement comparison must add networking, storage, operational overhead, and any idle reservation capacity before treating either result as a delivered service price.
Let C be total cost over an observation interval and W the comparable useful work completed within its contract. The economically meaningful ratio is C/W, with the contract specifying quality, request distribution, deadlines, and availability. That does not create one universal replacement unit. Training runs, interactive inference, and batch inference produce different outputs and tolerate different interruptions. Even tokens are heterogeneous: prompt processing and token generation consume different resources, and context length changes the cost of generation. Normalization must preserve these distinctions rather than hiding them inside a single attractive throughput number.
Use resource units for reservations and workload defined service units for acceptance. A contract can specify accelerator type, topology, memory, software environment, contention policy, and measured capacity on an agreed workload suite. Where workload access is restricted, publish the normalization procedure and its limits rather than pretending the machine label is sufficient. Keep benchmark configuration and delivered configuration traceable. Changes in batching, numerical format, or request selection can raise reported throughput while changing the product the buyer thought was being priced. Repeatability and representativeness are separate requirements.
The diagnostic is a reconciliation from billed hours to completed contractual work: powered availability, allocated time, useful execution, retries, queueing, and deadline compliance. This exposes whether apparent cheapness comes from genuine efficiency or from leaving expensive prerequisites outside the quote. GPU hours remain useful inventory; they become misleading when promoted into interchangeable outcomes. If your procurement or price index compares hours without a reproducible service contract, is it measuring the price of computation, or merely the price of access to a nameplate?