RAG (retrieval-augmented generation)

Retrieval-augmented generation (RAG) is a technique where an application first retrieves relevant passages from its own data, then passes them to a large language model (LLM) together with the user's question. The model answers from that supplied context. RAG lets an LLM use private or recent information without retraining the model.

How it works

At ingest time, documents are split into chunks, each chunk is turned into an embedding, and the embeddings are stored in a vector database. At query time the pipeline has three steps:

1. Embed the user's question.

2. Search the vector store for the closest chunks.

3. Send the question plus those chunks to the LLM to generate the answer.

For example, retrieving the top 4 chunks of 500 tokens each adds about 2,000 tokens of context to every request.

Why it matters in production

A RAG answer can be wrong because retrieval found the wrong chunks, even when the model works correctly. Each step adds latency and token cost. Teams need traces for each step and a measure of retrieval quality, such as similarity scores, to tell a retrieval problem from a generation problem.

See also

• Embeddings

• Vector database

• LLM observability

• Hallucination

• LLM-as-a-judge

On the One2N blog

• Exploring retrieval augmented generation (RAG): enhancing LLMs with external knowledge

• RAG observability with OpenTelemetry

• ABC of LLMOps: what does it take to run self-hosted LLMs

• Observing AI apps: instrument, break, compare

References

1. Retrieval-augmented generation for knowledge-intensive NLP tasks (Lewis et al., 2020)