Services

Resources

Company

vLLM

vLLM runs LLM inference as a server. You point it at a model, for example with vllm serve meta-llama/Llama-3.1-8B-Instruct, and it loads the weights onto the GPU and serves an OpenAI-compatible API on port 8000. Most of what makes it fast comes from two tricks, PagedAttention and continuous batching, and both exist to squeeze more concurrent requests onto one GPU.

How it works

Every request keeps a KV cache: the attention keys and values for each token it has seen, so the model does not recompute them on every step. Older servers reserved one contiguous slab per request, sized for the longest possible output, and the vLLM paper measured 60 to 80 percent of KV cache memory wasted that way.

PagedAttention borrows paging from operating systems: the cache is split into fixed blocks of 16 tokens, a block table maps each request to its blocks, and a block is handed out only when the request needs it. Requests that share a prompt prefix can share blocks too.

Continuous batching schedules at the level of single decode steps. As soon as one request finishes, its blocks are freed and a waiting request joins the running batch on the next step. The GPU does not sit idle waiting for the longest answer in a batch to end.

For example, Llama 3.1 8B has 32 layers and 8 KV heads of dimension 128. In 16-bit precision each token needs 2 (key and value) x 32 x 8 x 128 x 2 bytes = 128 KiB of KV cache. Say 32 GiB of GPU memory is left for the cache after the weights load: that holds 262,144 tokens.

Reserve 8,192 tokens per request up front and only 32 requests fit. If requests really use about 1,024 tokens each and get blocks as they grow, 256 fit, eight times as many, which is where vLLM gets its throughput.

Why it matters in production

Once vLLM is running, most tuning is memory flags. --gpu-memory-utilization (0.9 by default) sets how much of the GPU vLLM may use, and --max-model-len sets the longest context it will accept. When the KV cache fills, vLLM preempts requests and recomputes them later, which shows up as long time to first token. Each vLLM server runs one model and has no idea of users or per-user keys, so teams put an AI gateway or a platform such as GPUStack in front of it.

See also

On the One2N blog

References

  1. vLLM documentation

  2. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023)

Memory grid for vLLM. 32 GiB of KV cache at 128 KiB per token holds 262,144 tokens, drawn as 256 cells of 1,024 tokens. Reserving 8,192 tokens per request fits 32 requests, one column each, and in each column only one cell is used while the other seven, in red, are reserved and never used. Handing out blocks on demand fits 256 requests in the same memory, 8x as many.