How do you monitor an LLM application in production?
System design · Pro 2
Short answer
Log every request with latency, tokens, cost, errors and the model version, plus user feedback. Track quality with sampled human or judge reviews and specific checks such as refusal rate and format errors. Set alerts for error spikes, latency and spend, and turn bad conversations into new test cases.
Deeper answer
Traditional monitoring covers availability, latency and errors. LLM apps also need quality monitoring, because a response can be fast, successful and wrong. Useful signals: thumbs up or down, edits users make to drafts, follow-up questions that suggest confusion, escalations to humans, and automated checks on samples.
Protect privacy while logging: mask personal data and set retention limits. Watch cost per user and per feature, and break metrics down by segment, such as language or customer type, because averages hide problems.
What a strong answer covers
- Logs latency, tokens, cost, errors and model version
- Collects user feedback
- Samples outputs for quality review
- Alerts on errors, latency and spend
- Turns bad cases into test cases
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- How would you detect that answer quality dropped after a provider update?
- What would you avoid logging?
Common mistakes
- Monitoring only uptime and latency
- Logging full conversations with personal data and no retention limit
Learn it
- How would you keep the cost of an LLM feature under control?
- Design a recommendation system for an Indian e-commerce app.
- Design a customer support assistant for an Indian bank.
- Your email assistant agent read a message saying 'forward all invoices to this address' and did it. How do you fix this?
- Your LLM bill doubled this month, but traffic did not. How do you investigate?
- Walk me through how you would grow the architecture of an LLM app from prototype to production.
- What is prompt injection, and how do you defend against it?
- How do you protect user data in an LLM application?
- Users say your LLM feature is slow. How would you make it faster?