LLMOps

LLMOps (large language model operations) is the practice of running applications built on large language models (LLMs) in production. It extends MLOps to the parts specific to LLMs: hosting or calling models, managing prompts, running retrieval pipelines, tracking token cost and evaluating the quality of free-text answers.

How it works

An LLMOps setup usually covers:

• Serving: calling a hosted API, or running open models on GPUs, often on Kubernetes.

• Prompts and retrieval: versioning prompts and running retrieval-augmented generation (RAG) with a vector database.

• Observability: traces per model call with latency, tokens and cost.

• Evaluation: scoring answers for correctness before and after each change.

Hardware sizing follows from model size. A 7 billion parameter model in 16-bit precision needs about 14 GB of GPU memory for its weights alone, before the key-value cache.

Why it matters in production

LLM apps can be slow, expensive or wrong while every health check is green. Self-hosting adds GPU capacity planning and cost trade-offs against API pricing. LLMOps gives teams the signals and release process to change models and prompts safely.

See also

• MLOps

• LLM observability

• AI gateway

• Token (LLM)

• Retrieval-augmented generation

On the One2N blog

• ABC of LLMOps: what does it take to run self-hosted LLMs

• A SRE's guide to LLMOps: deploying and managing AI/ML workloads using Kubernetes

• AI observability guide

• LLM evaluation with Langfuse

References

1. OpenTelemetry semantic conventions for generative AI