Skip to main content

What would you capture in traces for an LLM application?

Deployment and LLMOps · Pro 1

Short answer

One trace per user request, with a span for each step: retrieval (query, document IDs, scores), each model call (model and version, prompt template version, input and output tokens, latency, cost, parameters), each tool call with arguments and results, and errors. Link user feedback to the trace. Mask personal data and sample if volume is high.

Deeper answer

Tools such as Langfuse or OpenTelemetry-based platforms make this easy. The payoff is debugging: when a user reports a bad answer, you can see exactly what was retrieved and what the model saw.

Traces also feed evals: pick bad or interesting production traces and add them to your test set.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Tracing LLM apps

Practice Deployment and LLMOps questions

More on this topic

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.