Skip to main content

When would you serve predictions in batch, and when online?

Deployment and LLMOps · Pro 1

Short answer

Batch scoring runs on a schedule over many records and stores the results, which is cheap and simple when predictions can be a few hours old, such as nightly churn scores. Online serving computes a prediction per request behind an API, needed when the answer depends on fresh input, such as fraud checks at payment time.

Deeper answer

Ask how fresh the prediction must be and how many entities there are. Recommendations for millions of users can be precomputed nightly; a search ranking that depends on the query cannot.

Online serving needs a model server, features available at request time (often a feature store to keep training and serving features consistent), a latency budget, autoscaling and fallbacks. A common hybrid precomputes heavy features in batch and does a light model call online.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Serving a model

Practice Deployment and LLMOps questions

More on this topic

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.