LLM inference is partly a scheduling and memory problem disguised as a model call.
A request begins with prefill, where the prompt is processed, and continues with autoregressive decoding, where output tokens are generated sequentially. Real traffic mixes different prompt lengths, generation lengths and arrival times, so serving is mostly about keeping useful work flowing without destroying latency.
The KV cache is central to that tradeoff. It avoids recomputing attention state for previous tokens during every decode step, but cached tokens consume accelerator memory. Longer contexts and more concurrent sequences therefore compete directly for the memory needed to keep requests in flight.
This is why inference engines care about continuous batching, paged KV-cache management, prefix reuse, scheduling and optimized attention kernels. The objective is not simply to build the largest possible batch. It is to balance throughput, memory pressure and latency under the traffic pattern you actually have.
GPU utilization is also easy to misread. A device can report very high utilization because kernels are continuously executing while still delivering disappointing useful throughput. Kernel efficiency, batch shapes, synchronization, memory movement and other bottlenecks still matter.
So I care more about time to first token, inter-token latency, latency percentiles, tokens per second, concurrency, memory pressure and cost per completed request.
A busy GPU is not the objective. Useful work per unit of hardware is.