System design answer
A voice agent that books appointments
Speech to text, an LLM with booking tools, text to speech, tight latency, and hand-off to the front desk.
The question“Design a phone voice agent for a clinic chain that books, moves and cancels appointments in English, Hindi and Tamil.”
1Clarify0-5 min
Ask before you draw. These answers change the design.
- Which languages and accents, and how noisy are calls?
- Which booking system, and what can be changed by phone?
- How is the caller identified?
- What must go to a person (emergencies, complaints)?
2Requirements and numbers5-8 min
It must
- Understand the request and check slots
- Book, move or cancel and confirm by SMS
- Hand over to staff with a summary
And it must be
- Reply starts within about 1 s of the caller stopping
- Never gives medical advice; emergencies go to a person at once
- Calls recorded only with consent
Numbers to say out loud
- Volume
- 3,000 calls a day, peak 40 at once
- Call length
- About 2 minutes
- Latency budget
- Speech-to-text 300 ms, model 500 ms, voice 200 ms
3High-level design8-12 min
- Caller → Phone line
- Phone line → Speech to text
- Speech to text → LLM + prompt
- LLM + prompt → tool call → Booking tools (find slots, book, cancel)
- Booking tools (find slots, book, cancel) → result → LLM + prompt
- LLM + prompt → Text to speech
- Text to speech → Caller
- LLM + prompt → emergency or unsure → Transfer to the front desk
- Booking tools (find slots, book, cancel) → SMS confirmation
4Deep dives12-26 min
Latency
Stream every stage: partial transcripts, streamed tokens, and voice that starts on the first sentence. Use a small fast model for the conversation and keep the prompt short and cached.
Safety
Emergency words trigger an immediate transfer. The agent confirms date, time and doctor back to the caller before booking, and sends an SMS confirmation.
5Failures26-30 min
| What can go wrong | What stops it |
|---|---|
| Misheard names or dates | Read back and confirm before booking |
| Medical question | Refuse and offer a call-back from staff |
| Booking system down | Take details and promise a call-back; log for staff |
6Evals and monitoring30-33 min
- 200 recorded test calls across languages and accents
- Booking accuracy, transfer accuracy, time to first word
7Trade-offs33-35 min
- A smaller model is faster but makes more mistakes; confirm before acting
- Full automation vs transfer: start with booking only
Follow-ups to expect
- How would you handle a caller switching languages mid-call?
- How do you stop double bookings?