Skip to main content

How do you choose a chunk size for RAG?

RAG · Intermediate 2

Short answer

Start from the documents and the questions: chunks should be big enough to make sense alone and small enough to be about one thing. Try a few sizes, often a few hundred tokens with some overlap, split on natural boundaries such as headings, and pick the one with the best retrieval recall on a test set.

Deeper answer

Small chunks give precise matches but lose context; large chunks keep context but dilute the embedding and waste the context window. Structure matters more than a magic number: splitting by section or paragraph keeps ideas whole, and overlap reduces the chance of cutting a key sentence in half.

Useful tricks: store the document title and section heading with each chunk, add a short generated context to each chunk, or retrieve small chunks but pass the surrounding larger section to the model. Decide with measurements on real questions, not by default settings.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Chunk documents for retrieval

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.