LLMOps

LLMOps (large language model operations) is the practice of running applications built on large language models (LLMs) in production. It extends MLOps to the parts specific to LLMs: hosting or calling models, managing prompts, running retrieval pipelines, tracking token cost and evaluating the quality of free-text answers.

How it works

An LLMOps setup usually covers:

Serving: calling a hosted API, or running open models on GPUs, often on Kubernetes.
Prompts and retrieval: versioning prompts and running retrieval-augmented generation (RAG) with a vector database.
Observability: traces per model call with latency, tokens and cost.
Evaluation: scoring answers for correctness before and after each change.

Hardware sizing follows from model size. A 7 billion parameter model in 16-bit precision needs about 14 GB of GPU memory for its weights alone, before the key-value cache.

Why it matters in production

LLM apps can be slow, expensive or wrong while every health check is green. Self-hosting adds GPU capacity planning and cost trade-offs against API pricing. LLMOps gives teams the signals and release process to change models and prompts safely.