LLM inference

LLM inference is running a trained large language model (LLM) on new input to produce output. Training sets the model's weights once. Inference uses those fixed weights on every request to turn a prompt into a stream of tokens. Model serving is the infrastructure that runs inference reliably for many users at once.

How it works

Inference has two phases. Prefill reads the whole prompt in parallel and fills the KV cache. Decode then generates one token at a time, and each step reads all the model weights from GPU memory. A 7-billion-parameter model in 16-bit precision has 14 GB of weights. On a GPU with about 2 TB/s of memory bandwidth, one request can decode at most about 140 tokens per second. Batching many requests shares each weight read, which is why serving engines such as vLLM batch continuously.

Why it matters in production

Prefill time sets time to first token, and decode speed sets how fast the answer streams. GPU memory decides how many requests fit in a batch, so it decides cost per token. Teams that self-host tune batch size, context length and quantization against a latency target.

See also

• KV cache

• Time to first token

• Latency and throughput

• LLMOps

On the One2N blog

• ABC of LLMOps: what does it take to run self-hosted LLMs

• Token economics, paisa, prompts & production

• Understanding MLOps from a software engineers' perspective

Further reading

• Efficient memory management for LLM serving with PagedAttention (Kwon et al., 2023)

• vLLM documentation