System design answer
A customer support assistant for an Indian bank
RAG over approved policy documents, read-only account tools behind login, strict guardrails, and a clean hand-off to people.
The question“Design an AI assistant that answers customer questions for a bank, in English and Hindi, on the website and WhatsApp.”
1Clarify0-5 min
Ask before you draw. These answers change the design.
- Which questions may it answer, and which must always go to a person (fraud, disputes, complaints)?
- Which languages and channels, and how many conversations a day?
- May it see account data, and how is the customer verified?
- What does success mean: resolution rate, correct hand-offs, satisfaction, cost?
2Requirements and numbers5-8 min
It must
- Answer product, fee and process questions from current policy, with links
- Show balance and recent transactions after verification
- Hand over to an agent with a summary
- Work in English and Hindi on web and WhatsApp
And it must be
- First words in under 2 s; full answer under 8 s
- No investment advice; no data from another customer, ever
- Every answer and action logged for audit
- Data stays in an Indian region
Numbers to say out loud
- Volume
- 50,000 conversations a day, peak 15 requests a second
- Tokens per turn
- About 3,000 in (prompt, history, 5 passages) and 300 out
- Corpus
- About 2,000 policy pages, updated weekly
3High-level design8-12 min
- Customer → Web chat or WhatsApp
- Web chat or WhatsApp → API
- API → Input guardrails
- Input guardrails → Router
- Router → policy question → Hybrid search + rerank
- Hybrid search + rerank → Policy index
- Router → account question, verified → Read-only account tools
- Router → fraud, dispute, unsure → Hand-off to an agent
- Hybrid search + rerank → Model via gateway
- Read-only account tools → Model via gateway
- Model via gateway → Output guardrails
- Output guardrails → Answer with citations
4Deep dives12-26 min
Retrieval
Chunk policies by heading with the title path, keep version and effective date as metadata, filter out superseded versions, and use hybrid search (codes like "NEFT" need keywords) with a reranker. Measure recall at 5 on 300 real questions in both languages.
Account tools and safety
Account tools are read-only and take the customer id from the verified session, never from the model. Permission checks live in code. Output checks block advice and any account number that is not the customer's own.
Languages
Keep one English index and translate the question for search, answering in the customer's language; build a separate eval set per language because quality differs.
5Failures26-30 min
| What can go wrong | What stops it |
|---|---|
| Hallucinated policy | Answer only from cited passages; refuse with a link to a person when nothing matches |
| Prompt injection in a message | Tools are read-only and scoped to the session; untrusted text in marked sections |
| Model provider outage | Gateway fallback to a second region, then a scripted "we will call you back" flow |
| Stale policy | Weekly re-index with effective dates; old versions filtered out |
6Evals and monitoring30-33 min
- Golden set of 300 questions per language with required facts and correct hand-off
- Recall at 5 for retrieval, faithfulness by a calibrated judge
- Online: hand-off rate, thumbs-down reasons, 50 traces reviewed a week
7Trade-offs33-35 min
- RAG over fine-tuning: policies change weekly and answers need citations
- More refusals early in exchange for zero invented policy
- A small model for routing and a large one for answers, to control cost
Follow-ups to expect
- How would you handle a question in Tamil?
- What would you never let the bot do, and how is that enforced?
- What do you watch in the first week after launch?