How do you choose a chunk size for RAG?
RAG · Intermediate 2
Short answer
Start from the documents and the questions: chunks should be big enough to make sense alone and small enough to be about one thing. Try a few sizes, often a few hundred tokens with some overlap, split on natural boundaries such as headings, and pick the one with the best retrieval recall on a test set.
Deeper answer
Small chunks give precise matches but lose context; large chunks keep context but dilute the embedding and waste the context window. Structure matters more than a magic number: splitting by section or paragraph keeps ideas whole, and overlap reduces the chance of cutting a key sentence in half.
Useful tricks: store the document title and section heading with each chunk, add a short generated context to each chunk, or retrieve small chunks but pass the surrounding larger section to the model. Decide with measurements on real questions, not by default settings.
What a strong answer covers
- Starts from the documents and questions
- Chunks make sense alone and are about one thing
- Uses overlap and natural boundaries such as headings
- Tests a few sizes on a test set and measures recall
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- What would you change for tables or code?
- How does overlap help, and what does it cost?
Common mistakes
- Quoting one universal chunk size
- Choosing chunk size without measuring retrieval
Learn it
- Your RAG app gives wrong answers. How do you find the cause?
- Design a RAG system that lets a law firm search and ask questions about 50,000 contracts.
- How would you detect hallucinations in a production RAG application?
- What are embeddings, and how are they used in search?
- How do you make a RAG system cite sources and stay grounded?
- What are hybrid search and reranking, and when do they help?
- How do you keep a RAG index up to date as documents change?
- How do metadata filters improve retrieval?
- How would you build RAG over documents full of charts and images?
- Your RAG system must handle scanned PDFs, tables and slides. How do you ingest them?
- How do you make sure users only get answers from documents they are allowed to see?
- What are query rewriting and HyDE, and when do they help?
- Your HR assistant keeps quoting last year's leave policy. How do you fix it?
- Tell me about a RAG system you built and how you measured whether it worked.
- Can you do retrieval for long documents without a vector database?
- What is RAG, and why would you use it?
- When would you use graph RAG instead of standard vector RAG?