How would you build RAG over documents full of charts and images?
RAG · Pro 1
Short answer
Two main options: convert visuals to text, by having a vision-language model describe each chart and image and indexing those descriptions with the text; or embed page images directly with a multimodal retriever and pass the retrieved page images to a vision-capable model. Keep links to the original page so answers can cite it.
Deeper answer
Describing images is simple and works with existing text pipelines, but descriptions can miss details. Page-image retrieval, in the style of ColPali, keeps layout and visual detail, at higher storage and compute cost.
Evaluate with questions whose answers are only in charts or diagrams, because text-only tests will hide weaknesses.
What a strong answer covers
- Option one: describe visuals as text
- Option two: embed page images directly
- Vision-capable model to answer
- Keeps links to source pages
- Tests with chart-only questions
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- What is lost when you describe a chart in text?
- How would you test chart-based answers?
Common mistakes
- Testing only with text-answerable questions
- Dropping images during parsing without noticing
Learn it
- How do you choose a chunk size for RAG?
- Your RAG app gives wrong answers. How do you find the cause?
- Design a RAG system that lets a law firm search and ask questions about 50,000 contracts.
- How would you detect hallucinations in a production RAG application?
- What are embeddings, and how are they used in search?
- How do you make a RAG system cite sources and stay grounded?
- What are hybrid search and reranking, and when do they help?
- How do you keep a RAG index up to date as documents change?
- How do metadata filters improve retrieval?
- Your RAG system must handle scanned PDFs, tables and slides. How do you ingest them?
- How do you make sure users only get answers from documents they are allowed to see?
- What are query rewriting and HyDE, and when do they help?
- Your HR assistant keeps quoting last year's leave policy. How do you fix it?
- Tell me about a RAG system you built and how you measured whether it worked.
- Can you do retrieval for long documents without a vector database?
- What is RAG, and why would you use it?
- When would you use graph RAG instead of standard vector RAG?