Why use Docker for an AI application?
Deployment and LLMOps · Intermediate 1
Short answer
Docker packages the app with its exact Python version, libraries and system dependencies into an image, so it runs the same on a laptop, in CI and in production. That ends 'works on my machine' problems, makes deployments repeatable and lets you run several services together with Docker Compose, such as an API, a vector database and a worker.
Deeper answer
Good practice: small base images, pinned dependency versions, a .dockerignore, no secrets baked into images, and a non-root user. Multi-stage builds keep images small.
For GPU work, images based on the CUDA runtime plus the NVIDIA container toolkit let containers use GPUs.
What a strong answer covers
- Packages code with exact dependencies
- Runs the same everywhere
- Repeatable deployments
- Compose runs several services together
- Good practice: no secrets, small images
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- What should never go into a Docker image?
- What is Docker Compose for?
Common mistakes
- Baking secrets into images
- Huge images with build tools left in
Learn it
Practice Deployment and LLMOps questionsMore on this topic
- When would you serve predictions in batch, and when online?
- How would you set up CI/CD for an LLM application?
- What is the difference between data parallelism and model parallelism?
- Why does experiment tracking matter, and what do you record?
- Your model scored well offline but performed badly after launch. What could explain it?
- Explain Kubernetes pods, deployments and services, and why you would use Kubernetes for AI apps.
- What is the KV cache, and why can it cause out-of-memory errors when serving an LLM?
- How would you load test an LLM feature and set a latency budget?
- Your model provider has an outage during peak hours. What should already be in place?
- How do you manage API keys and secrets across environments?
- How do you detect and handle model drift in production?
- A bank wants your AI system but says no data may leave its network. What are the options?
- Why does an LLM reply have a wait before the first word and then a different speed after it?
- What is quantisation, and what are the trade-offs?
- When would you use a semantic cache for an LLM app, and what can go wrong?
- How would you serve an LLM feature as an API for a web app?
- How would you serve a model that is too big for one GPU, and how would you scale it to more users?
- How would you add single sign-on to an enterprise AI assistant?
- How do you version prompts and models so you can roll back safely?
- What would you capture in traces for an LLM application?
- What does an LLM gateway do, and when do you need one?
- Why use vLLM to serve an open-weight model instead of a simple Hugging Face script?