← Community

Design A Rag Pipeline

Build retrieval that actually finds the right passage — chunk on structure, combine keyword and vector search, rerank, and measure retrieval separately from generation. Use when building RAG, document Q&A, or semantic search.

---
name: design-a-rag-pipeline
category: ai-agents
description: >-
  Build retrieval that actually finds the right passage — chunk on structure,
  combine keyword and vector search, rerank, and measure retrieval separately
  from generation. Use when building RAG, document Q&A, or semantic search.
---

# Design a RAG Pipeline

Most disappointing RAG systems are not generation problems. The model answers
badly because retrieval handed it the wrong three paragraphs, and no amount of
prompt work fixes a context that does not contain the answer.

So build and measure retrieval **first, on its own**.

## Chunk on structure, not on character count

Splitting every 1,000 characters cuts sentences in half, separates a heading
from the text it introduces, and splits tables down the middle.

Better, in order of preference:

1. **Semantic boundaries.** Markdown headings, HTML sections, function
   definitions, slide boundaries. The document already tells you where the seams
   are.
2. **Paragraphs**, merged up to a size limit.
3. **Fixed size with overlap**, only as a fallback.

Two things that consistently help:

- **Keep a parent pointer.** Retrieve the small chunk for precision, then pass
  the surrounding section for context. Small chunks match better; large chunks
  answer better.
- **Prepend the breadcrumb.** A chunk beginning "Billing → Refunds → Partial
  refunds" retrieves far better than one beginning "You can also do this within
  30 days", because the embedding otherwise has no idea what "this" refers to.

Shared publicly by the author and open for any use. See the Terms.