Skip to main content

How would you load test an LLM feature and set a latency budget?

Deployment and LLMOps · Pro 2

Short answer

Start from the user experience: for example, first token within one second and a full answer within six seconds at the 95th percentile. Break that budget into parts (retrieval, model, tools). Then simulate realistic traffic with a tool such as Locust or k6, ramping up concurrency, and measure time to first token, latency percentiles, errors and rate limits.

Deeper answer

Use realistic prompts and output lengths, because latency scales with tokens. Test against provider rate limits and your own quotas, and watch for queueing in your service before the model is even called.

Report percentiles, not averages: a 2-second average can hide 10-second tails that users feel. Re-run the test after model, prompt or infrastructure changes.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Latency budgets and load testing

Practice Deployment and LLMOps questions

More on this topic

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.