Throughput

The number of units completed per time interval under a declared workload. In AI systems, the unit may be requests per second, generated tokens per second, or agent tasks completed per hour. A value is comparable only when it reports concurrency, input and output sizes, hardware, batching policy, and error rate.

Throughput measures system capacity, while latency measures the time experienced by one unit. Raising concurrency may increase throughput until saturation while worsening p95 latency, queues, timeouts, and cost. Useful tests report the load curve, backpressure, and quality constraints instead of publishing only the highest observed number.