LLM observability

LLM observability is the practice of monitoring the calls an application makes to a large language model (LLM). It records which model was requested and which one served the answer, input and output token counts, call duration, time to first token and errors. It is the model-call layer of the wider AI observability practice.

LLM observability covers the model-call layer only. For the whole stack, see AI observability. For the agent loop, see agent observability.

How it works

An auto-instrumentation library wraps the model SDK at startup and emits OpenTelemetry spans and metrics with gen_ai.* attributes, such as gen_ai.usage.input_tokens and gen_ai.client.token.usage. Divide the token rate by the request rate to get tokens per request. In one run of our workshop, swapping the chat model raised input tokens per request from about 4,000 to about 6,500 with the same prompt, because the tokenizer changed.

Why it matters in production

Model calls are often the slowest and most expensive step in a request, and their cost varies with context size and model. Separating provider latency from your own code tells you whose problem a slow request is. Token trends catch prompt growth before the invoice does.