🚦LLM Inference Serving

The workload: why LLM inference ≠ normal request serving

Classic request serving assumes three things: requests are roughly uniform in cost, workers are stateless, and the cost of a request is known when it arrives. An LLM chat request breaks all three. A transformer generates text autoregressively — one token per forward pass, each pass conditioned on every token that came before. To avoid recomputing attention over the whole history at every step, the engine caches each token's attention keys and values (the KV cache). That turns a request into a session: per-request state that grows linearly with every emitted token and must stay resident in GPU memory until the request finishes. And because generation stops whenever the model emits an end-of-sequence token, output length — which dominates cost — is unknown at admission time. Two requests that look identical at the load balancer can differ in cost by two orders of magnitude.

Serving one request therefore has two phases with opposite performance characters. Prefill processes all N prompt tokens in a single forward pass: large matrix multiplies, high parallelism, bounded by the GPU's compute throughput. It produces the first output token, so it determines time-to-first-token. Decode then emits one token per forward pass — and every pass must stream all model weights and the request's whole KV cache from HBM into the compute units to produce a single token's worth of math. Decode is bounded by memory bandwidth, not compute, and it determines the gap between tokens.

The cleanest way to see why this matters is arithmetic intensity — useful FLOPs per byte moved from memory. In BF16, a forward pass does roughly 2 FLOPs per parameter per token and each parameter is 2 bytes. Batch-1 decode therefore sits at about 1 FLOP/byte: the GPU streams 16 GB of weights out of HBM to do what is, by its standards, almost no math. Prefill over a 2,000-token prompt reuses each weight read 2,000 times — roughly three orders of magnitude higher intensity. On the roofline model, an H100's ridge point sits near 989 TFLOPS ÷ 3.35 TB/s ≈ 295 FLOPs/byte. Decode lives far to the left of the ridge, pinned to the memory roof; prefill lives on the compute roof.

Three consequences fall straight out of this picture, and the rest of the doc is built on them. First, batching decode is nearly free: a step that decodes 64 requests streams the weights once and does 64 tokens of math in roughly the time one token took, because the memory traffic — not the math — was the bottleneck. (Each request still reads its own KV cache, so it isn't perfectly free, and the KV share of the traffic grows with context length.) Second, bandwidth is the spec to read first when you size hardware for decode-heavy workloads; TFLOPS govern prefill. Third, prefill and decode want different optimizations, which is why schedulers mix them carefully on one GPU (§ toolkit) and why large deployments physically separate them (§ cluster).

One more distinction shapes everything downstream: online vs. offline serving. An interactive chat endpoint must hit latency targets on every request, so it runs below saturation on purpose. A batch job — generate embeddings, score a corpus, distill training data — has no latency target at all; it should run the GPU as close to saturation as memory allows. The same engine serves both, tuned to opposite ends of the same tradeoff curve, which is the subject of the next section.

Metrics & SLOs: the numbers that define "fast"

Because responses stream, "latency" is not one number. Four metrics describe a streamed response, and — this is the part interviewers probe — each is moved by a different mechanism, so "make it faster" is an underspecified requirement until you say which one.

• TTFT (time to first token) = queueing delay + prefill time. Moved by: admission control, prefill compute, prefix caching, shorter prompts.
• ITL (inter-token latency) = the gap between consecutive streamed tokens; one decode step of the batch the request is riding in. Moved by: decode batch size, interference from other requests' prefills, model size vs. bandwidth.
• TPOT (time per output token) = mean ITL over the response = (E2E − TTFT) ÷ output tokens. The smoothed version of ITL; SLOs are usually set on TTFT + TPOT (or a percentile of ITL).
• E2E latency = TTFT + TPOT × output length. For long answers it is dominated by output length — which the server doesn't control. This is why per-token metrics, not E2E, are what you put SLOs on.

The second idea is the latency–throughput frontier. Admitting more concurrent requests raises tokens/second — decode batching is nearly free, per the roofline — but every request added to a decode batch adds its KV reads to each step, and once the batch saturates compute or bandwidth, steps simply take longer. ITL climbs slowly at first, then steeply near saturation. There is no setting that maximizes both throughput and latency; you choose a point on the frontier and defend it with admission control.

That choice has a name. SLO-aware throughput — goodput — is the framing the DistServe paper built its title on: the maximum request rate that can be served while both the TTFT and TPOT targets hold. A saturated system can post impressive tokens/second while nearly every request blows its per-token SLO — high throughput, near-zero goodput. Goodput per GPU is the number capacity plans and autoscalers should optimize, and the gap between it and raw throughput is where most "we added load and everything got slow" incidents live.

Finally, think in percentiles, not means. P99 TTFT is usually dominated by queueing and by prefill interference — a long prompt landing in front of yours, or your decode steps being repeatedly preempted by other requests' prefills — rather than by model speed. The two standard fixes, chunked prefill on a single GPU and prefill/decode disaggregation across GPUs, are exactly the techniques of the next sections. When ITL is fine at P50 and ugly at P99 under load, it is almost always scheduling interference, not the model.

Capacity math: the KV cache is the budget

The capacity question for an LLM server is not "how many QPS per core" but "how many tokens of KV cache fit next to the weights, and how fast can I stream memory." Both halves are first-principles arithmetic, and being able to do it from scratch is the single highest-leverage skill for this topic — in interviews and in capacity reviews.

Each token of context stores one key vector and one value vector per layer. With grouped-query attention, the cached width is the KV heads, not the query heads:

// bytes of KV cache per token of context
kv_per_token = 2 × n_layers × n_kv_heads × head_dim × dtype_bytes
//             ↑ K and V

// Llama-3.1-8B, BF16:  2 × 32 × 8 × 128 × 2 = 131,072 B = 128 KiB/token
// Llama-3.1-70B, BF16: 2 × 80 × 8 × 128 × 2 = 327,680 B = 320 KiB/token

Now the budget. An engine reserves a fraction of VRAM (vLLM's gpu_memory_utilization defaults to 0.9), loads the weights, keeps headroom for activations and CUDA-graph capture, and hands everything left to the KV cache. On one H100 serving Llama-3.1-8B in BF16: 72 GB usable − 16 GB weights ≈ 56 GB of KV budget → ~427K concurrent tokens → ~52 concurrent requests at 8K average context. The 70B model in BF16 is 141 GB of weights — it doesn't fit on the GPU at all; you shard it across 4 GPUs with tensor parallelism or quantize it.

Because KV size scales with n_kv_heads, attention architecture is a serving decision made at pretraining time. Classic multi-head attention (Llama-2-7B: 32 KV heads) costs 512 KiB/token — 4× the budget of Llama-3.1's 8-head GQA for a similar-size model. MQA is the extreme (1 KV head). MLA compresses further: DeepSeek-V3 caches one 576-dim latent per layer — 61 × 576 × 2 B ≈ 69 KiB/token for a 671B-parameter model, about half of Llama-3.1-8B's. That is why a model 80× larger can be cheaper per token of context than a small one.

The decode speed ceiling

Bandwidth also bounds speed, not just capacity. A single-stream decode step must move at least the weights through HBM, so tokens/second ≤ bandwidth ÷ bytes-per-step: for the 8B model in BF16 on an H100, 3,350 GB/s ÷ 16 GB ≈ ~209 tokens/s, best case, before any KV reads, kernel inefficiency, or sampling overhead. No software makes batch-1 decode beat this number on this hardware — only smaller weights (quantization), fewer weight-reads per token (speculative decoding), or faster memory. Everything a GPU has left after hitting the ceiling is what batching converts into throughput.

Three readings of the calculator worth internalizing: FP8 KV cache doubles concurrent tokens for free capacity; INT4 weights almost double the single-stream decode ceiling because decode streams half the bytes; and at long contexts the batch column collapses fast — at 128K context, that same H100 holds three concurrent requests, which is why long-context serving is its own capacity-planning problem.

The single-GPU toolkit

Four techniques define modern single-GPU serving, and all three major engines ship all four. The deep mechanics — block tables, copy-on-write, scheduler internals — are covered in the vLLM doc; this section is what each technique is, why it exists, and where the engines genuinely differ.

Paged KV cache

Early engines allocated each request's KV cache contiguously at its maximum possible length, and the vLLM paper measured the result: only 20–38% of KV memory held actual token state — the rest was reserved-but-unused or fragmentation. PagedAttention (SOSP 2023) applied the OS-paging move: fixed-size KV blocks (16 tokens in vLLM), allocated on demand, addressed through a per-request block table. Near-zero waste, and with the freed memory feeding bigger batches, 2–4× throughput at the same latency. It also makes KV shareable — the foundation for everything in the prefix-reuse section. Today paged KV is table stakes in all three engines (TensorRT-LLM calls the unified scheduling+paging combination "paged attention with in-flight batching"). Mechanics: vLLM doc § PagedAttention.

Continuous batching

Static batching forms a batch of N requests, runs them to completion together, and admits the next batch. Because output lengths vary wildly, finished requests' slots sit idle while the longest one drags on, and arriving requests wait for the whole batch. Orca (OSDI 2022) introduced iteration-level scheduling: re-form the batch at every decode step, so a finished request's slot is refilled on the next step, and a new request's prefill can ride along with everyone else's decodes. Orca measured 36.9× throughput over NVIDIA's static-batched FasterTransformer at the same latency — the single biggest win in the serving literature, which is why every engine adopted it.

Two refinements matter operationally. Chunked prefill (Sarathi-Serve, OSDI 2024) answers "what happens when a 20K-token prompt arrives while 40 requests are mid-decode?" Without it, that prefill monopolizes whole steps and every streaming request stutters — the classic P99-ITL spike. Chunking splits the prefill into pieces sized to a per-step token budget and rides each piece along with the ongoing decodes: slightly slower TTFT for the long prompt, stall-free ITL for everyone else. It is on by default in vLLM and standard practice everywhere. Preemption answers "what happens when the KV budget itself runs out?" — decode steps grow every request's cache, so over-admission is discovered mid-flight. Someone gets evicted: their blocks are dropped and recomputed later, or offloaded to CPU. Engines do this transparently, but recompute burns prefill capacity, so a system that preempts often has an admission-control problem, not a scheduling problem.

Speculative decoding

Decode wastes bandwidth on math the GPU could do for free — so spend the slack buying more tokens per weight-read. A cheap drafter proposes γ tokens sequentially; the target model verifies all of them in one forward pass (prefill-style parallelism over the proposals); accepted tokens are kept, and the first rejection is replaced by a token sampled from the target's own distribution. The rejection-sampling scheme (Leviathan et al., Chen et al.) guarantees the output distribution is exactly the target model's — speculation is lossless, a pure systems trick.

The drafter has evolved: separate small models gave way to auxiliary heads on the target itself — Medusa's parallel heads, then EAGLE's feature-level autoregressive head, which is what all three engines now center on (EAGLE-3 reports up to 6.5× at batch 1; current docs list EAGLE-3 plus draft-model, n-gram lookup, and DeepSeek-style MTP heads as the menu in vLLM, TensorRT-LLM, and SGLang alike; Medusa has quietly dropped out of all three). The economics, though, are the interview-grade part: verification is only cheap while decode has bandwidth slack. As batch size grows and decode approaches the compute roof, the "free" parallel verify starts competing with other requests' work — speculation boosts low-batch latency and can cost high-batch throughput, so engines tune or disable it under load (γ shrinks, or speculation turns off past a batch-size threshold).

Prefix & KV-cache reuse

If two requests share a prefix — same system prompt, same few-shot examples, same conversation history — the KV for that prefix is byte-identical, so prefill it once and share the blocks. The savings are pure TTFT and prefill compute (decode work is untouched), and they're enormous in the workloads that dominate production: multi-turn chat re-sends the whole conversation every turn, and agentic loops re-send a growing transcript dozens of times. The engines agree on the idea and differ in the index structure:

vLLM's automatic prefix caching hashes each full block against its parent chain — O(1) lookups, block-granular matches, on by default (details). SGLang made the radix tree the organizing principle of the whole runtime: RadixAttention matches at token granularity, handles branching workloads naturally (one system prompt forking into N parallel agent calls shares everything up to the fork), and the scheduler orders requests to maximize tree hits — the SGLang paper's up-to-6.4× throughput numbers come from structured programs with heavy sharing. The flip side defines its limits: one-shot unique prompts hit nothing, and a cache that big is worth routing for — which is how the cluster layer (§ cluster) gets pulled into cache design. Reuse also composes with storage: prefixes evicted from GPU can be parked in CPU RAM or SSD and restored cheaper than recompute (LMCache, Dynamo's KV manager — same section).

Making one GPU fast

The toolkit above decides what work runs each step; this section is about making the step itself fast. Three levers: move fewer bytes (quantization), move them better (kernels), and stop stalling between kernels (graphs and schedulers).

Quantization is three separate decisions

"Quantize the model" conflates three independent choices — weights, activations, and KV cache — with different physics:

• Weight-only INT4 (GPTQ, AWQ): weights shrink 4×, so decode — which streams weights — gets dramatically faster at small batch, and the model fits smaller GPUs. The math still runs in FP16 after dequantization, so prefill barely improves. Best for latency-sensitive, low-concurrency serving.
• FP8 weights + activations (W8A8): Hopper's tensor cores execute FP8 matmuls at ~2× BF16 rate, so this is the one that helps prefill and decode. The production default on H100-class fleets.
• FP8 KV cache: halves the cost per context token → double the concurrent tokens in the same budget. (TensorRT-LLM adds NVFP4 KV on Blackwell.) Pure capacity, modest accuracy risk, calibrate and evaluate.

Tooling follows ecosystem lines: NVIDIA's Model Optimizer produces pre-quantized checkpoints TensorRT-LLM consumes (and vLLM/SGLang can run); the vLLM-adjacent llm-compressor covers FP8/INT8/INT4 in the compressed-tensors format all three engines read. Hugging Face hosts pre-quantized variants of every major model, so in practice you select rather than quantize.

Kernels: attention is special

Attention's naive implementation materializes an N×N score matrix in HBM — at 32K context that's the whole memory budget gone. FlashAttention made attention IO-aware: tile the computation through on-chip SRAM, never write the score matrix, recompute what's cheaper to recompute than to store. Exact attention, an order of magnitude less memory traffic. Serving adds a twist — the KV cache is paged, scattered across non-contiguous blocks, with ragged batch shapes — which is what FlashInfer (MLSys 2025 best paper) specializes in: block-sparse KV layouts and JIT-compiled kernel variants per attention configuration. Engines treat attention backends as pluggable and pick per hardware: SGLang auto-selects FlashAttention-3 on Hopper and TensorRT-LLM's MHA kernels on Blackwell, with FlashInfer elsewhere; vLLM similarly ships several. You should know why backends exist (paged + ragged + new GPUs each need specialized kernels), not the matrix of which is default where — it changes quarterly.

Graphs and schedulers: the CPU is the silent bottleneck

A decode step is thousands of small kernel launches; at 5–10 µs of CPU launch overhead each, a Python-driven engine can leave the GPU idle between kernels. Two standard fixes. CUDA graphs record the whole step's launch sequence once and replay it as a single unit — vLLM captures full-step graphs for decode by default (its V1 engine pairs them with torch.compile). Overlap scheduling hides the rest of the CPU work: TensorRT-LLM's scheduler (on by default) and SGLang's "zero-overhead" scheduler prepare step n+1 — batch formation, sampling bookkeeping, stop-criteria checks — while the GPU executes step n, so the GPU never waits for Python. These two are why "it's written in Python" stopped being a serving-performance argument.

Compile ahead-of-time, or JIT?

The historical architectural split: classic TensorRT-LLM compiled each model into a serialized TensorRT engine — peak fused kernels, at the cost of a minutes-to-hours build per model × GPU × configuration, and rebuilds on any change. vLLM and SGLang load checkpoints directly into a PyTorch runtime and recover kernel performance with the techniques above. The market voted: TensorRT-LLM 1.0 (September 2025) made its PyTorch-based runtime the default and filed the TensorRT-engine flow under "legacy" — its release notes call the PyTorch architecture "the default experience." The convergent stack everywhere is now: PyTorch runtime + hand-tuned attention/GEMM kernel libraries + torch.compile-style fusion + CUDA graphs.

Serving features that shape the design

Two product-level features matter architecturally — they reach into the scheduler and the cache, which is why they belong in a systems doc and an interview answer.

Structured output

JSON mode and schema-constrained generation work by masking: at every decode step, the engine compiles the grammar's current state into a bitmask over the vocabulary and zeroes out the logits of every token that would violate it. Done naively over a 100K+ token vocabulary, that walk costs milliseconds per token — a tax on every step. xgrammar fixed the economics: most grammar states' valid-token sets don't depend on runtime context, so precompile them, and overlap the rest with the GPU's forward pass — near-zero overhead. It is now the default backend in SGLang, the primary one in vLLM (alongside llguidance-family options), and one of TensorRT-LLM's two guided-decoding backends. SGLang adds jump-forward decoding: when the grammar forces a run of tokens ({"name": "), emit them directly with zero model passes.

The systems insight to volunteer: constrained decoding changes the workload. Masking perturbs the output distribution your speculative drafter was trained to predict, so acceptance rates drop and speculation gains shrink; and jump-forwards make per-step timing irregular. "We turned on JSON mode and our ITL profile changed" is a real production story.

Multi-LoRA serving

Per-customer fine-tunes don't scale as full model replicas — but a LoRA adapter is megabytes, so one base model plus hundreds of resident adapters can serve hundreds of "different models" from one GPU. The enabling kernels (the Punica/S-LoRA lineage) batch requests across different adapters into one step: the dense base-model math stays batched, and the small per-adapter matrices are gathered per request. All three engines ship this, with dynamic load/unload. The two design hooks: the router must know adapter residency (an adapter miss costs a load), and the adapter ID must be part of the prefix-cache key — vLLM folds the LoRA ID into its block hashes — or you serve one tenant's cache to another.

Scaling beyond one GPU: the cluster layer

When the model outgrows one GPU, you shard the model: tensor parallelism inside a node (cuts per-GPU memory and per-GPU bytes-streamed, so decode gets faster), pipeline parallelism across nodes (adds latency, scales capacity), expert parallelism for MoE models. That story is told in the vLLM doc. The staff material is what happens when you scale the fleet — because every property from earlier sections (two-phase requests, giant per-request state, cache hit rates worth 10× TTFT) breaks the assumptions of standard load balancing and autoscaling.

Disaggregated prefill/decode

Chunked prefill manages prefill/decode interference; DistServe asked why the two phases share GPUs at all. Colocated, they fight: prefills spike decode ITL, decodes steal prefill compute, and — the deeper point — each phase wants different resourcing. Prefill is compute-bound, gains from aggressive tensor parallelism, and is sized against TTFT; decode is bandwidth-bound, wants maximum batch, and is sized against ITL/TPOT. Splitting them into separate pools lets you optimize and scale each against its own SLO; DistServe measured up to 7.4× more requests (or 12.6× tighter SLOs) at the same goodput target. The cost: the prompt's KV cache now has to move from a prefill GPU to a decode GPU. Note what disaggregation buys — vLLM's docs say it plainly: it does not improve throughput; it buys SLO control (predictable ITL) and independent scaling. Moonshot's Mooncake (the serving platform behind Kimi, FAST 2025 best paper) is the canonical production validation: a KV-cache-centric disaggregated architecture handling ~75% more requests under real SLOs.

KV transfer is the new plumbing. A 2K-token prompt on the 8B model is ~270 MB of KV; a 32K-token agent context on a 70B model is ~10 GB — per request. Moving that fast enough to hide inside the TTFT budget requires NVLink- or RDMA-class paths and zero-copy engineering, which is why it grew dedicated infrastructure: NIXL abstracts the transport (GPU↔GPU, CPU, storage backends) for Dynamo and vLLM's connector; SGLang ships Mooncake's transfer engine as its default and NIXL as an alternative; vLLM exposes all of this behind an experimental KV-connector API. The same machinery enables KV tiering — LMCache and Dynamo's KV Block Manager park cold prefixes in CPU RAM, SSD, or remote stores and restore them cheaper than recomputing, turning the prefix cache from a per-GPU optimization into a fleet-wide, even cross-session, one.

Cache-aware routing

Now route requests into that fleet. Round-robin — correct for stateless services — is wrong twice here: it ignores that request costs vary by 100×, and it actively destroys prefix-cache hit rates by scattering a tenant's identical-prefix requests across replicas, each of which prefills the same system prompt from scratch. The fix is a stateful router that scores replicas on prefix affinity and load. SGLang's Model Gateway maintains an approximate copy of each worker's radix tree and routes to the longest match unless load skew exceeds a threshold (its v0.4 measurements: cache hit rate 20%→75%, ~2× throughput). In Kubernetes-land, llm-d — a CNCF sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA — packages the same idea as an "endpoint picker" plugged into the Gateway API: score pods on KV-affinity, queue depth, and load, then pick. NVIDIA's Dynamo ships KV-aware routing plus the disaggregation orchestration as an engine-agnostic layer over vLLM, SGLang, and TensorRT-LLM. The tension to name in an interview: affinity routing concentrates load exactly where the cache is hottest, so it needs an explicit balance valve against hot-spotting a replica.

Autoscaling on the right signal

GPU utilization is the wrong autoscaling signal for LLM serving — a decode-bound GPU reads ~100% "utilized" while leaving most of its compute idle, so the classic CPU-style policy never fires or always fires. The signals that track real headroom are the ones engines export natively: queue depth (vLLM's vllm:num_requests_waiting), KV-cache utilization (vllm:kv_cache_usage_perc), and SLO attainment — goodput per replica. Dynamo bakes this in as an SLA-driven "Planner" that profiles workloads and right-sizes prefill and decode pools separately, which is the shape to sketch in a design interview: two pools, two SLOs, two scaling loops. And remember scale-out is slow — pulling tens of GB of weights and warming CUDA graphs takes minutes, so production fleets keep warm pools and scale ahead of demand, not in response to it.

vLLM vs TensorRT-LLM vs SGLang

By 2026 the feature matrices have largely converged — every row of the table below was a differentiator once and is now near-universal. So the comparison that matters is about design center: what each engine optimizes for when tradeoffs bite, and what ecosystem you are buying into. (For the spec-sheet view — KV layout, codegen, licenses, plus TGI and llama.cpp — see the vLLM doc's table.)

Read as philosophies: vLLM optimizes for reach — neutral governance, every model and accelerator, the choice that's never wrong and rarely maximal. TensorRT-LLM optimizes for the hardware — NVIDIA builds it next to the silicon and ships Blackwell-format quantization first; the price is single-vendor lock at both layers. SGLang optimizes for the cache — it bet earliest that production traffic is dominated by shared prefixes and structured output, and built the engine outward from that bet.

The closing argument for an interview: convergent evolution is the strongest fact about this space. Paged KV, continuous batching, chunked prefill, EAGLE-3, xgrammar, FP8, P/D disaggregation — each appeared in one engine and was in all three within roughly a year. Radix-tree cache indexing is in two of three and counting. So don't choose on a feature checkbox that will be obsolete in two releases; choose on operational fit — hardware commitments, Kubernetes story, governance, and which community your team can debug alongside at 2 a.m.

Interview crib

Canonical questions for this topic, with the compressed answers. Each one is unpacked in a section above.

Why does batching help decode enormously but barely help prefill?

Decode is memory-bandwidth-bound: a step's cost is dominated by streaming the weights, which a batch shares — 64 requests get 64 tokens for roughly one token's memory traffic. Prefill is already compute-saturated by one request's matmuls, so batching adds little. (§ workload)

TTFT is fine, but P99 ITL spikes when traffic rises. What's happening?

Arriving requests' prefills are hijacking whole decode steps, so streaming requests stall whenever a long prompt lands. Fixes, in order of escalation: chunked prefill (cap prefill tokens per step), then disaggregate prefill and decode into separate pools. (§ toolkit, § cluster)

How much KV memory does a 70B GQA model need per token — derive it.

2 (K and V) × 80 layers × 8 KV heads × 128 head_dim × 2 bytes (BF16) = 320 KiB/token. The lever is KV heads: same math at 32 MHA heads would be 1.25 MiB. (§ capacity)

What bounds single-stream decode speed, and at what number?

HBM bandwidth ÷ bytes-per-step. Every decode step must re-read all weights: 8B in BF16 on an H100 → 3,350 ÷ 16 ≈ 209 tok/s, optimistically, before KV reads. Quantization shrinks the numerator's bytes; speculation amortizes reads over multiple tokens; nothing else moves it. (§ capacity)

When does speculative decoding hurt?

When decode has no bandwidth slack to spend: large batches (verify passes now compete with real work), low acceptance rate (draft overhead with no payoff), and constrained decoding (masking shifts the distribution the drafter learned, tanking α). It's a low-batch latency tool, not a throughput tool. (§ toolkit)

Why physically separate prefill and decode instead of just scheduling smarter?

Interference is only half the reason — the phases also want different parallelism and different scaling signals (TTFT vs TPOT). Separate pools optimize each independently; the price is shipping the prompt's KV cache between pools fast enough to hide in the TTFT budget. It buys SLO control, not throughput. (§ cluster)

Why does round-robin load balancing underperform for LLM fleets?

Request costs vary ~100× (unknown output lengths), and round-robin scatters identical prefixes across replicas, destroying prefix-cache hit rates — every replica re-prefills the same system prompt. Cache-aware routers score replicas on prefix affinity plus load. (§ cluster)

Why not autoscale on GPU utilization?

A bandwidth-bound decode GPU reports ~100% utilization while idling most of its compute — the metric carries no headroom signal. Scale on queue depth, KV-cache utilization, and SLO attainment (goodput), which engines export directly. (§ cluster)

INT4 weights vs FP8 W8A8 — when each?

INT4 weight-only quarters the bytes decode streams → best single-stream latency and smallest footprint, but math stays FP16, so prefill barely moves. FP8 W8A8 runs on Hopper's FP8 tensor cores → ~2× the matmul rate, helping prefill and decode — the throughput default on H100s. (§ fast)

A prefix-cache hit improves which metric — TTFT, ITL, or both?

TTFT only. The hit skips prefill compute for the shared prefix; decode still does identical work per token. (Corollary: prefix caching does nothing for decode-bound throughput — its wins are TTFT and prefill capacity.) (§ toolkit)

How can maximizing throughput ruin goodput?

Past the saturation knee, added load still raises tokens/s while pushing every in-flight request's ITL over the SLO — tokens flow, but zero requests count. Capacity is the throughput at the SLO intersection, not the curve's maximum. (§ metrics)

Further reading

Primary sources only — the papers that named the ideas and the engines' own documentation.

• Foundations: Orca (OSDI '22 — continuous batching) · PagedAttention/vLLM (SOSP '23) · Sarathi-Serve (OSDI '24 — chunked prefill) · SGLang/RadixAttention (NeurIPS '24) · GQA · DeepSeek-V2 (MLA).
• Speculative decoding: Leviathan et al. · Chen et al. · Medusa · EAGLE-3.
• Kernels & quantization: FlashAttention · FlashInfer (MLSys '25 best paper) · GPTQ · AWQ (MLSys '24 best paper) · XGrammar · S-LoRA.
• The cluster layer: DistServe (OSDI '24 — goodput, P/D disaggregation) · Mooncake (FAST '25 best paper) · NVIDIA Dynamo + NIXL · llm-d · LMCache.
• Engine docs: docs.vllm.ai · TensorRT-LLM docs · docs.sglang.io — defaults and feature claims in this doc were checked against these.
• This site: the vLLM engineering doc — block tables, copy-on-write, the scheduler, distributed inference — for everything this doc compressed.