Skip to main content

Interview prep

Top 25 MLOps Engineer interview questions

18 of these 25 are scenario, use-case or real-life questions, the kind that decide an interview. Each has a short answer to say out loud, the explanation behind it, and an example you can tell.
Practice out loud
  1. 1ConceptIntermediate 1

    Why use Docker for an AI application?

    Short answer

    Docker packages the app with its exact Python version, libraries and system dependencies into an image, so it runs the same on a laptop, in CI and in production. That ends 'works on my machine' problems, makes deployments repeatable and lets you run several services together with Docker Compose, such as an API, a vector database and a worker.

    The explanation

    Good practice: small base images, pinned dependency versions, a .dockerignore, no secrets baked into images, and a non-root user. Multi-stage builds keep images small.

    For GPU work, images based on the CUDA runtime plus the NVIDIA container toolkit let containers use GPUs.

    An example you can use in the interview

    Our RAG service needed Python 3.12, a specific PyTorch version and a system library for PDF parsing. On a new teammate's laptop it failed for half a day. With a Dockerfile, everyone runs the same image, CI tests the same image, and production deploys it. Docker Compose starts the API, a Postgres database with pgvector and a background worker with one command.

    They may follow up with

    • What should never go into a Docker image?
    • What is Docker Compose for?

    Avoid

    • Baking secrets into images
    • Huge images with build tools left in

    Practice this oneLesson: Docker and Compose

  2. 2ConceptIntermediate 3

    Explain Kubernetes pods, deployments and services, and why you would use Kubernetes for AI apps.

    Short answer

    A pod runs one or more containers together. A deployment keeps a set number of identical pods running and handles rolling updates. A service gives those pods a stable network address and load-balances between them. Kubernetes helps AI apps scale replicas with load, schedule GPU workloads, restart failed containers and roll out new versions safely.

    The explanation

    Helm packages Kubernetes manifests as charts with configurable values, so you can deploy the same app to staging and production with different settings.

    Kubernetes is powerful but adds operational weight. For a small team with one service, a managed platform or serverless containers may be simpler; say when you would not use it.

    An example you can use in the interview

    For a document Q&A service I would describe it like this: the API container runs in pods; a deployment keeps, say, four replicas running and rolls out a new version one pod at a time; a service gives them one address and spreads requests. The embedding workers run as a separate deployment on GPU nodes, and an autoscaler adds pods when the queue grows.

    They may follow up with

    • What does a readiness probe do?
    • When is Kubernetes overkill?

    Avoid

    • Mixing up pods and containers
    • Recommending Kubernetes for everything

    Practice this oneLesson: Kubernetes and Helm basics

  3. 3ConceptIntermediate 3

    Why does experiment tracking matter, and what do you record?

    Short answer

    Without tracking you cannot reproduce or compare results, and you forget which run produced the model in production. Record the code version, data version, parameters, metrics, the environment and the model artefact for every run, in a tool such as MLflow or Weights and Biases.

    The explanation

    Tracking turns 'I think the second run was better' into a table you can sort. It also answers audit questions later: which data trained this model, and who approved it.

    Good habits: log automatically from the training script, tag runs with the question they answer, keep the evaluation set fixed so scores are comparable, and register the chosen model in a model registry with its run linked. For LLM work the same applies to prompts and model versions.

    An example you can use in the interview

    Three months after shipping a churn model, a manager asked why the current version did better than the January one. Because every run in MLflow had the git commit, data snapshot ID, hyperparameters and metrics, I could show that the gain came from adding two features, not from tuning, and rebuild the January model exactly to compare.

    They may follow up with

    • How would you make a run fully reproducible?
    • What is a model registry for?

    Avoid

    • Tracking metrics but not data or code versions
    • Comparing runs evaluated on different data

    Practice this oneLesson: Experiment tracking and model registry

  4. 4ConceptPro 1

    What does an LLM gateway do, and when do you need one?

    Short answer

    A gateway sits between your apps and model providers and gives one API for many models. It handles routing, fallbacks when a provider fails, retries, rate limits and budgets per team, caching, and central logging of cost and usage, and it keeps provider keys in one place. You need one once several teams or models are in play.

    The explanation

    Tools such as LiteLLM or managed gateways provide this. Routing can also be smart: send easy requests to a cheap model and hard ones to a strong one, as RouteLLM research explored.

    The gateway is also where you enforce policy: which models are approved, data rules, and spending caps. Keep it highly available, because it becomes a single point of failure.

    An example you can use in the interview

    At a company where five teams each called three model providers with their own keys, nobody knew total spend, and an outage at one provider broke two products. A gateway such as LiteLLM put one API in front of all models, with per-team budgets, automatic fallback to a second provider, central logs of cost and usage, and keys stored in one place.

    They may follow up with

    • How would you route between a cheap and an expensive model?
    • What new risk does a gateway introduce?

    Avoid

    • Hard-coding one provider in every app
    • Making the gateway a single point of failure

    Practice this oneLesson: LLM gateways

  5. 5ConceptPro 1

    Why use vLLM to serve an open-weight model instead of a simple Hugging Face script?

    Short answer

    vLLM is built for high-throughput inference. PagedAttention manages the key-value cache in small blocks, so far less GPU memory is wasted and more requests fit at once, and continuous batching adds new requests to a running batch instead of waiting. It also offers an OpenAI-compatible server, quantisation support and tensor parallelism across GPUs.

    The explanation

    A naive generate loop serves one request at a time and wastes memory on padding, so throughput collapses under load. vLLM keeps the GPU busy.

    Before self-hosting, compare total cost with a hosted API: GPUs cost money even when idle, and you take on operations. Self-hosting makes sense for data residency, steady high volume, or customised models.

    An example you can use in the interview

    We first served an open model with a simple Hugging Face generate loop. It handled one request at a time, so under load users queued for a long time and much of the GPU sat idle. vLLM, with continuous batching and PagedAttention, served many requests at once on the same GPU, and its OpenAI-compatible API meant our application code barely changed.

    They may follow up with

    • When is a hosted API cheaper than self-hosting?
    • What is the KV cache?

    Avoid

    • Assuming self-hosting is always cheaper
    • Ignoring GPU memory planning

    Practice this oneLesson: Open-weight models with vLLM

  6. 6ConceptPro 1

    What would you capture in traces for an LLM application?

    Short answer

    One trace per user request, with a span for each step: retrieval (query, document IDs, scores), each model call (model and version, prompt template version, input and output tokens, latency, cost, parameters), each tool call with arguments and results, and errors. Link user feedback to the trace. Mask personal data and sample if volume is high.

    The explanation

    Tools such as Langfuse or OpenTelemetry-based platforms make this easy. The payoff is debugging: when a user reports a bad answer, you can see exactly what was retrieved and what the model saw.

    Traces also feed evals: pick bad or interesting production traces and add them to your test set.

    An example you can use in the interview

    For a single question to a support assistant, the trace shows: the retrieval span with the rewritten query and the IDs and scores of five chunks; the model span with model version, prompt version 12, 3,200 input and 280 output tokens, 2.1 seconds and ₹0.40; a tool span calling get_order with its result; and the user's thumbs-down linked to it. Phone numbers are masked before the trace is stored.

    They may follow up with

    • How would you link a thumbs-down to the right trace?
    • What personal data would you mask?

    Avoid

    • Logging only the final answer
    • Storing full personal data in traces forever

    Practice this oneLesson: Tracing LLM apps

  7. 7ConceptPro 1

    How would you evaluate an LLM application?

    Short answer

    Build a test set of realistic inputs with what a good answer needs. Check what you can with code (format, required facts, forbidden content), use human or model review for subjective quality, and track scores for every change. In production, add user feedback and monitoring, and turn failures into new test cases.

    The explanation

    A layered approach, as Hamel Husain describes: level 1 is fast unit-test style assertions run on every change; level 2 is human and model-based review of samples, with the judge checked against human labels; level 3 is A/B testing with real users for product-level impact.

    Start by looking at real outputs and writing down how they fail, then design checks for those failures. Keep the eval set versioned, add cases from production bugs, and report results by category, so a gain in one area does not hide a loss in another.

    An example you can use in the interview

    For an app that writes product descriptions for an online store, I would build 100 real products with what a good description must include: the material, size and price, and no claims we cannot back. Code checks length, format and banned words. A judge model, checked against human ratings, scores readability. Every prompt change runs this set, and descriptions the store team rejects become new test cases.

    They may follow up with

    • Where do your first test cases come from?
    • How do you evaluate subjective quality such as tone?

    Avoid

    • Relying on vibe checks of a few examples
    • Using only generic benchmarks instead of task-specific tests

    Practice this oneLesson: Test sets and evals

  8. 8Use casePro 1

    When would you serve predictions in batch, and when online?

    Short answer

    Batch scoring runs on a schedule over many records and stores the results, which is cheap and simple when predictions can be a few hours old, such as nightly churn scores. Online serving computes a prediction per request behind an API, needed when the answer depends on fresh input, such as fraud checks at payment time.

    The explanation

    Ask how fresh the prediction must be and how many entities there are. Recommendations for millions of users can be precomputed nightly; a search ranking that depends on the query cannot.

    Online serving needs a model server, features available at request time (often a feature store to keep training and serving features consistent), a latency budget, autoscaling and fallbacks. A common hybrid precomputes heavy features in batch and does a light model call online.

    An example you can use in the interview

    At a lending app, we scored every customer's risk of missing an EMI each night in batch, because collections teams plan calls in the morning and a day-old score is fine. But the fraud check on a new loan application ran online in under 200 milliseconds, because it depended on the device and details submitted at that moment.

    They may follow up with

    • What is training-serving skew, and how do you avoid it?
    • How would you add a fallback if the model server is down?

    Avoid

    • Choosing online serving when batch would do
    • Computing features differently in training and serving

    Practice this oneLesson: Serving a model

  9. 9ScenarioPro 1

    How do you detect and handle model drift in production?

    Short answer

    Monitor inputs and outputs, not just uptime. Data drift is when input distributions change; concept drift is when the relationship between inputs and the target changes. Track feature statistics, prediction distributions and, when labels arrive, real performance. Alert on meaningful shifts, investigate, and retrain or roll back with a tested pipeline.

    The explanation

    Labels often arrive late: you learn whether a loan defaulted months later. So use proxies in the meantime: compare live feature distributions with training data using measures such as population stability index or a statistical test, watch the share of each predicted class, and track business metrics.

    Not every shift needs retraining; a festival season can shift spending without breaking the model. Set thresholds with care to avoid alert fatigue, keep a reference dataset per model version, and make retraining a routine, versioned pipeline with evaluation gates rather than a manual scramble.

    An example you can use in the interview

    Our delivery-time model's errors grew during the monsoon. Feature monitoring showed the share of orders in rainy hours had jumped, which is data drift, and late-delivery labels showed the model underestimated rain delays more than before, which looked like concept drift. We alerted on the error trend, retrained with the last two monsoon seasons weighted more, tested on held-out recent weeks, and deployed with the old model ready for rollback.

    They may follow up with

    • What would you monitor when labels arrive months later?
    • When is retraining the wrong fix?

    Avoid

    • Monitoring only latency and errors
    • Retraining automatically without an evaluation gate

    Practice this oneLesson: Monitoring and drift

  10. 10ScenarioPro 1

    Your model scored well offline but performed badly after launch. What could explain it?

    Short answer

    Common causes: data leakage made offline scores too optimistic; training data does not match live data (skew); features are computed differently in production; the population changed (drift); or the offline metric does not reflect the business goal. I would compare feature distributions offline versus online, check for leakage, and replay live inputs through the training pipeline.

    The explanation

    A gap between offline and online results usually means the evaluation did not reflect reality. Leakage, such as a feature that is only known after the event, inflates offline scores. Training-serving skew happens when production code computes a feature differently, uses stale values or fills missing data another way.

    To diagnose, log production inputs and predictions, compare their distributions to training data, and recompute some live predictions offline to see if they match. Also check the metric: a model that improves AUC may not improve the decision the business makes. Fix the cause, then add monitoring so the gap is caught early.

    An example you can use in the interview

    A churn model I worked on had excellent validation scores but did not help the retention team at all. Comparing features, I found 'days since last complaint' was computed from a table that, in training, included complaints logged after the customer had already decided to leave, which was leakage. In production that information did not exist yet. Rebuilding the training set with point-in-time features dropped the offline score, but the new model's online results matched it, and the team could trust it.

    They may follow up with

    • How would you detect training-serving skew?
    • How would you set up an evaluation that predicts online results better?

    Avoid

    • Retraining immediately without diagnosing
    • Assuming it is drift without checking for leakage

    Practice this oneLesson: Monitoring and drift

  11. 11Use caseIntermediate 3

    How would you serve an LLM feature as an API for a web app?

    Short answer

    Put a backend service, for example FastAPI, between the browser and the model provider so keys stay secret and you can add auth, rate limits and logging. Stream tokens to the client with server-sent events so users see text at once, set timeouts and retries, validate inputs and outputs, and run it with multiple workers behind a load balancer.

    The explanation

    Calls to model APIs are slow and I/O-bound, so async handlers or enough workers matter. Handle client disconnects so you stop paying for tokens nobody reads. Return clear errors when the provider is down, and consider a fallback model.

    Log request IDs, latency, token counts and cost per request from day one; they are what you will need when something goes wrong.

    An example you can use in the interview

    For a legal-research web app, a FastAPI service sat between the browser and the model API. It checked the user's session, applied a rate limit per user, logged each request, and streamed tokens to the browser with server-sent events, so text appeared at once. It had a 60-second timeout with one retry on connection errors, validated that responses had citations, and ran as four workers behind a load balancer.

    They may follow up with

    • Why not call the model API directly from the browser?
    • How do you handle the provider being down?

    Avoid

    • Exposing API keys in the frontend
    • No timeouts on model calls

    Practice this oneLesson: Serving an AI app

  12. 12Use caseIntermediate 3

    How do you manage API keys and secrets across environments?

    Short answer

    Never commit secrets to code. Keep them in a secret manager or the platform's encrypted environment variables, with different keys for development, staging and production, scoped to least privilege. Inject them at runtime, rotate them regularly and after any leak, restrict who can read them, and scan repositories for accidental commits.

    The explanation

    If a key does leak, revoke it immediately: deleting the commit is not enough because history and forks keep it. Provider spending limits reduce the damage of a leaked model API key.

    Infrastructure as code, such as Terraform, should reference secrets, not contain them, and its state files can hold sensitive values, so store state securely.

    An example you can use in the interview

    On one team a developer accidentally pushed an API key to a public GitHub repo. We revoked it within minutes, then moved every secret to AWS Secrets Manager with separate keys for dev, staging and production, gave the app's role read access to only its own secrets, added a secret scanner to CI, and set a 90-day rotation reminder. Local development used a .env file that was in .gitignore.

    They may follow up with

    • What do you do in the first hour after a key leaks?
    • Why separate keys per environment?

    Avoid

    • Keys in .env files committed to Git
    • Sharing one production key across all developers

    Practice this oneLesson: Environments, secrets and Terraform

  13. 13Use casePro 1

    How would you set up CI/CD for an LLM application?

    Short answer

    On every pull request run the normal checks (lint, types, unit tests) plus an eval suite: a regression set of prompts with code checks and judge-based scores, compared with the main branch. Block merges that fall below thresholds. Version prompts and model settings in the repository, deploy to staging first, then roll out gradually with monitoring and a quick rollback.

    The explanation

    Evals cost money and time, so run a fast smoke set on every pull request and a fuller set nightly or before release. Cache model responses where inputs are unchanged.

    Treat a model version upgrade like a code change: it goes through the same pipeline. Secrets come from the CI system's secret store, never the repository.

    An example you can use in the interview

    In our GitHub Actions pipeline, every pull request ran lint and unit tests, then an eval job over 150 saved support questions with code checks for format and banned phrases, plus an LLM judge for accuracy. The job compared scores with main and failed if accuracy fell more than two points. Merges deployed to staging, then to 10% of users with monitoring, then everyone.

    They may follow up with

    • How would you keep eval costs reasonable in CI?
    • How would you roll back a bad prompt change?

    Avoid

    • Changing prompts outside version control
    • No evals in the pipeline

    Practice this oneLesson: CI/CD with GitHub Actions

  14. 14ScenarioPro 1

    How do you version prompts and models so you can roll back safely?

    Short answer

    Treat prompts, model names and parameters as versioned configuration in the repository or a prompt registry, with each production request logging which versions it used. Pin exact model versions rather than aliases that change underneath you. Release changes behind a flag or as a canary, compare metrics with the previous version, and keep the old version ready to switch back.

    The explanation

    Providers update and retire models, so an alias such as 'latest' can change behaviour without any change on your side. Pinning plus a scheduled upgrade, run through your evals, keeps changes deliberate.

    Registries such as MLflow's prompt registry or a simple versioned file work; what matters is that any answer in production can be traced to the exact prompt and model that produced it.

    An example you can use in the interview

    We pinned the exact dated model version in config, kept prompts in the repo with a version number, and logged both on every request. When we tried a new model behind a flag for 5% of traffic, complaints about formatting rose, visible within a day in our dashboard split by version. We switched the flag off in a minute, fixed the prompt for the new model, and re-ran the canary.

    They may follow up with

    • Why pin model versions?
    • What would you log to trace an answer back to its prompt?

    Avoid

    • Editing prompts directly in production
    • Using moving model aliases

    Practice this oneLesson: Versioning prompts and models

  15. 15Use casePro 1

    How would you build a test set for an LLM feature when you have no labelled data?

    Short answer

    Start from real or realistic inputs: logs, support tickets, or examples written with domain experts. Cover common cases, edge cases and known failure types. For each, write what a good answer must include or avoid. Begin small, around 30 to 50 cases, review them with people who know the domain, and grow the set from production failures.

    The explanation

    Synthetic examples generated by a model can fill gaps, but review them, because they tend to be easier and more uniform than real inputs. Balance the set across categories that matter to the business, and include adversarial inputs such as prompt injection attempts and off-topic questions.

    Write criteria, not just one reference answer, since many good answers are possible. Version the set, and keep a held-out part you do not tune against, so you can confirm improvements are real.

    An example you can use in the interview

    When we started an AI assistant for a telecom's support team with no labelled data, I took 200 recent tickets, worked with two senior agents to pick 50 that covered common and tricky cases, and for each wrote what a correct reply must say and must not say. Every week we added the cases where the assistant went wrong in production. Within two months the set had about 150 cases.

    They may follow up with

    • How do you stop your test set from becoming too easy?
    • How large should it be at first?

    Avoid

    • Using only synthetic, model-written examples
    • Writing one exact reference answer and checking for an exact match

    Practice this oneLesson: Test sets and evals

  16. 16ScenarioPro 1

    How do you stop a prompt or model change from breaking things that used to work?

    Short answer

    Keep a regression test set of past inputs, including every bug you have fixed, with checks for each. Run it automatically on every change, compare scores with the previous version, and block or review changes that make any category worse. Do the same before switching model versions.

    The explanation

    Prompts are code with unpredictable side effects: fixing one case can quietly break five others. Automating the eval set in CI turns this into a visible number. Fast deterministic checks run on every push; slower judge-based or human checks run before releases.

    Pin model versions in production so behaviour does not shift under you, and treat a model upgrade as a change that must pass the same tests. Watch per-category results, not only the average.

    An example you can use in the interview

    After a teammate improved our summarisation prompt for long documents, short documents started getting padded summaries. Our regression set had only long cases, so it missed this. We added a 'short document' category with 20 cases, ran the set on every pull request, and set a rule that no category may drop more than three points without a review. The next prompt change that hurt short summaries was caught before merge.

    They may follow up with

    • What would you do when a new model is better on average but worse in one category?
    • Which checks run on every push, and which only before release?

    Avoid

    • Testing only the case you were fixing
    • Letting the provider's model updates reach production untested

    Practice this oneLesson: Regression tests for prompts

  17. 17Use casePro 2

    How do you monitor an LLM application in production?

    Short answer

    Log every request with latency, tokens, cost, errors and the model version, plus user feedback. Track quality with sampled human or judge reviews and specific checks such as refusal rate and format errors. Set alerts for error spikes, latency and spend, and turn bad conversations into new test cases.

    The explanation

    Traditional monitoring covers availability, latency and errors. LLM apps also need quality monitoring, because a response can be fast, successful and wrong. Useful signals: thumbs up or down, edits users make to drafts, follow-up questions that suggest confusion, escalations to humans, and automated checks on samples.

    Protect privacy while logging: mask personal data and set retention limits. Watch cost per user and per feature, and break metrics down by segment, such as language or customer type, because averages hide problems.

    An example you can use in the interview

    For a tutoring chatbot, every request logged model version, tokens, latency, cost and whether the student clicked thumbs-up or down. Each day a judge model and one teacher reviewed 50 sampled conversations for correctness. We alerted on error rate above 2%, p95 latency above 6 seconds and daily spend above budget. Thumbs-down conversations with a teacher's correction went into the test set every week.

    They may follow up with

    • How would you detect that answer quality dropped after a provider update?
    • What would you avoid logging?

    Avoid

    • Monitoring only uptime and latency
    • Logging full conversations with personal data and no retention limit

    Practice this oneLesson: Deploy and monitor an LLM app

  18. 18Use casePro 2

    How would you load test an LLM feature and set a latency budget?

    Short answer

    Start from the user experience: for example, first token within one second and a full answer within six seconds at the 95th percentile. Break that budget into parts (retrieval, model, tools). Then simulate realistic traffic with a tool such as Locust or k6, ramping up concurrency, and measure time to first token, latency percentiles, errors and rate limits.

    The explanation

    Use realistic prompts and output lengths, because latency scales with tokens. Test against provider rate limits and your own quotas, and watch for queueing in your service before the model is even called.

    Report percentiles, not averages: a 2-second average can hide 10-second tails that users feel. Re-run the test after model, prompt or infrastructure changes.

    An example you can use in the interview

    For a chat feature in an insurance app, we set the budget at first token under 1.5 seconds and full answer under 8 seconds at the 95th percentile, split into 300 ms for retrieval, under a second to first token, and the rest for generation. Using k6 we ramped from 10 to 200 concurrent users. At 120 users, the provider's rate limit kicked in and the 95th percentile doubled, so we requested a higher limit and added a queue with a friendly wait message.

    They may follow up with

    • Why do percentiles matter more than averages?
    • What happens when you hit the provider's rate limit?

    Avoid

    • Testing with tiny prompts
    • Reporting averages only

    Practice this oneLesson: Latency budgets and load testing

  19. 19ScenarioPro 2

    Users say your LLM feature is slow. How would you make it faster?

    Short answer

    Measure first: find where time goes, such as retrieval, the model call or tools. Then stream the answer so text appears at once, cut input and output length, cache repeated prompt prefixes and common answers, use a smaller model for simple steps, and run independent steps in parallel.

    The explanation

    Perceived speed matters as much as total time: streaming and showing progress make an app feel fast. Output tokens are generated one by one, so asking for shorter answers often helps most. Long retrieved context adds input time and cost; rerank and send fewer, better chunks.

    Architecture changes help too: precompute what you can, run retrieval and other lookups in parallel, avoid chains of sequential model calls where one call would do, and choose providers or regions close to users. Measure the effect of each change on both latency and quality.

    An example you can use in the interview

    Users of our policy assistant waited about 9 seconds for an answer. Tracing showed 1 second for retrieval, 1.5 seconds for a query-rewriting call and 6 seconds generating long answers. We streamed the answer, ran rewriting with a small, fast model, asked for shorter answers with a 'show more' option, and cached the system prompt. The first words now appeared in about 1.5 seconds, and complaints stopped.

    They may follow up with

    • Which usually matters more for latency: input or output length?
    • How would you speed up a pipeline of three sequential model calls?

    Avoid

    • Optimising without measuring where the time goes
    • Cutting quality without checking the eval set

    Practice this oneLesson: Caching and speed

  20. 20Use casePro 2

    How would you keep the cost of an LLM feature under control?

    Short answer

    Measure cost per request and per user, then cut it: use the cheapest model that passes your evals, shorten prompts and outputs, cache repeated prefixes and answers, batch work that can wait, and stop runaway usage with per-user limits and a provider spending cap. Recheck when traffic or prices change.

    The explanation

    Cost is tokens times price times volume, so attack each factor. Routing simple requests to a small model often gives the biggest saving. Trimming retrieved context and conversation history cuts input tokens. Batch APIs are cheaper for offline jobs.

    Guardrails stop surprises: rate limits per user, maximum steps for agents, a hard monthly cap, and alerts when daily spend jumps. For a product in India, cost per active user has to fit low price points, so model the unit economics before launch.

    An example you can use in the interview

    For a document Q&A feature costing about ₹4 per question, I logged cost by step and found the long system prompt and ten retrieved chunks were most of it. Caching the fixed prompt prefix, cutting to five reranked chunks, and moving simple greeting messages to a smaller model brought it to about ₹1.20 with no drop on our eval set. We also added a per-user daily cap.

    They may follow up with

    • How would you calculate cost per active user?
    • What would you do if one user generated a large share of the cost?

    Avoid

    • Only switching to a cheaper model without checking quality
    • Having no hard spending limit

    Practice this oneLesson: Cost control

  21. 21ScenarioPro 2

    Your LLM bill doubled this month, but traffic did not. How do you investigate?

    Short answer

    Break the cost down by feature, model, and input versus output tokens, day by day, to find when and where it changed. Typical causes: a prompt or context that grew, retries or agent loops, a switch to a more expensive model, caching that stopped working, or one customer's heavy use. Fix the cause, then add per-feature budgets and alerts.

    The explanation

    Cost is tokens times price, so a jump with flat traffic means more tokens per request or a higher price per token. Good usage logging, with the feature, model, tokens and cache hits on every call, turns this from guesswork into a quick query.

    Look at the date the change started and compare with deployments. A new retrieval setting that sends twenty chunks instead of five, a longer system prompt that broke prompt caching, or a retry bug can each multiply cost. Afterwards, set budgets per feature and alert on cost per request, not just the monthly total.

    An example you can use in the interview

    Our monthly bill jumped from about ₹1.6 lakh to ₹3.4 lakh while requests stayed flat. Our usage table logged feature, model and tokens per call, so I grouped by day and saw input tokens per request triple on the 9th. That day we had raised the retriever's top-k from 5 to 15 to fix a recall issue, and a new date line at the top of the system prompt had stopped prompt caching from matching. We moved the date to the end, reranked down to six chunks, and added an alert on cost per request.

    They may follow up with

    • What would you log on every model call?
    • How would you catch this earlier next time?

    Avoid

    • Switching to a cheaper model before finding the cause
    • Only looking at the monthly total

    Practice this oneLesson: Cost control

  22. 22ScenarioPro 1

    Your model provider has an outage during peak hours. What should already be in place?

    Short answer

    A gateway or client layer with timeouts and retries, a fallback to a second model or provider that has been tested on our eval set, graceful degradation such as cached answers or a clear 'try again' message, monitoring that alerts on error rates, and a runbook saying who decides to switch and how we tell users.

    The explanation

    Outages are a matter of when, not if. The response should be designed in advance, because choosing a backup model in the middle of an incident leads to surprises in quality or format.

    A gateway makes switching a configuration change rather than a code change. The fallback model must be evaluated on the same tests so you know its quality, and prompts may need small adjustments. For some features, degrading gracefully is better than switching, for example showing search results without an AI summary. After the incident, review what happened and update the runbook.

    An example you can use in the interview

    We ran a support assistant through LiteLLM with our main provider and a second one as backup. Every week our eval suite also ran against the backup, so we knew it scored a little lower but acceptably. When the main provider had an hour-long incident at 7 p.m., the gateway's error-rate alert fired, the on-call engineer followed the runbook, and traffic moved in a few minutes. The ticket-summary feature, which was less important, simply showed 'summary unavailable' instead.

    They may follow up with

    • How would you test the fallback before you need it?
    • Which features would you degrade rather than switch?

    Avoid

    • Having a fallback model that was never tested
    • No alert until users complain

    Practice this oneLesson: LLM gateways

  23. 23Use caseIntermediate 3

    What responsible AI checks would you run before launching an AI feature?

    Short answer

    Identify who could be harmed and how; test quality and fairness across user groups and languages; red-team for misuse, prompt injection and harmful output; check privacy and data handling against the DPDP Act and your policies; make sure users know they are dealing with AI; set up monitoring, an incident process and an owner; and document decisions.

    The explanation

    Frameworks help structure this: the NIST AI Risk Management Framework's govern, map, measure and manage functions, and the EU AI Act's risk tiers if you serve EU users. Scale effort to risk: a writing helper needs less than a credit decision.

    A launch checklist with named sign-offs turns principles into practice, and a post-launch review after a few weeks catches what testing missed.

    An example you can use in the interview

    Before launching an AI loan-eligibility explainer, we listed who could be harmed: applicants misled about eligibility. We tested answers in English, Hindi and Tamil, checked results across income groups, red-teamed for prompts asking it to promise approval, confirmed only needed data was used under our privacy policy, labelled it as AI, named a product owner, and set up a kill switch and an incident channel.

    They may follow up with

    • How would you scale these checks for a low-risk feature?
    • Who should sign off?

    Avoid

    • Treating responsible AI as a one-time checkbox
    • No owner after launch

    Practice this oneLesson: AI risk frameworks

  24. 24ScenarioPro 2

    Your AI assistant gave harmful advice to a customer. What do you do?

    Short answer

    Contain first: disable or restrict the feature or the triggering path, using a kill switch if one exists. Assess impact from logs (who was affected, how often), inform stakeholders and, where required, affected users and regulators. Find the root cause, fix it, add the case to evals and guardrails, and write a blameless post-mortem with follow-up actions and owners.

    The explanation

    Preparation makes this manageable: a documented incident process, on-call owners, logs that let you search conversations, and switches to fall back to a safer mode. Severity levels help decide how fast to act.

    AI incidents often have several causes, such as retrieval of an outdated document plus missing guardrails, so look past the first explanation. Public databases such as the AI Incident Database show common patterns.

    An example you can use in the interview

    If a health app's assistant told a user to stop a prescribed medicine, I would first switch off its medication answers with the feature flag, so it says 'please ask your doctor'. Then search the logs for similar answers and count affected users, inform our clinical lead and legal team, and contact affected users. The root cause might be a retrieved forum post; we would remove that source, add a guardrail and test cases, and write a blameless review.

    They may follow up with

    • What would your kill switch actually do?
    • Who needs to be told, and when?

    Avoid

    • Quietly patching without assessing impact
    • Blaming the model and moving on

    Practice this oneLesson: Incident response for AI

  25. 25Real lifePro 1

    Tell me about a production incident you handled.

    Short answer

    Tell it in order: how you found out, how you assessed impact, what you did first to limit harm, how you found the root cause, the fix, how you communicated, and what you changed afterwards. Emphasise calm steps and the follow-up, such as a new alert, test or runbook entry. A blameless tone shows maturity.

    The explanation

    Interviewers use this to judge how you behave under pressure. The best answers show that you contained the problem before chasing the cause, kept the right people informed, and turned the incident into a lasting improvement.

    For AI systems, include anything model-specific: a quality drop after a model update, harmful output, a cost spike or a prompt-injection attempt. Give times and numbers where you can, and say honestly what you would do better. If you have only handled a small incident, a small one told well is fine.

    An example you can use in the interview

    One Friday our document assistant started answering some questions in the wrong language. A support lead flagged it at 4 p.m. Logs showed it began after that morning's prompt change. I rolled back to the previous prompt version within twenty minutes and posted an update in the support channel. The cause was an instruction we had moved below the retrieved text. On Monday we added twenty multilingual cases to the regression suite, and prompt changes now need that suite to pass before release.

    They may follow up with

    • What did the post-incident review change?
    • How did you communicate with users?

    Avoid

    • Blaming a colleague
    • Skipping the follow-up and prevention

    Practice this oneLesson: Incident response for AI

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.