A bank wants your AI system but says no data may leave its network. What are the options?
Deployment and LLMOps · Pro 2
Short answer
Options include a private deployment of a hosted model in the bank's cloud tenancy and region, such as Azure OpenAI or Bedrock in its own account, if policy allows; or self-hosting open-weight models with vLLM on the bank's own GPUs, on-premise or even air-gapped. Each changes cost, model quality and who operates it.
Deeper answer
Clarify what 'leave the network' means: their cloud account may count as inside. Regulators such as the RBI have outsourcing and data localisation expectations, so involve the bank's risk team early.
Air-gapped deployments need offline model updates, internal monitoring and a support model, which take real planning. Benchmark the open-weight model on the bank's tasks before promising parity with frontier models.
What a strong answer covers
- Clarifies what the rule requires
- Private cloud deployment in their tenancy
- Self-hosted open-weight models on their GPUs
- Air-gapped option and its operations
- Compares cost, quality and operations
Answer it out loud in the mock interview about 2 min
Follow-ups to expect
- What changes when the deployment is air-gapped?
- How would you compare open-weight and hosted model quality for this client?
Common mistakes
- Assuming open-weight models match frontier quality
- Not clarifying what the rule actually requires
Learn it
- When would you serve predictions in batch, and when online?
- How would you set up CI/CD for an LLM application?
- What is the difference between data parallelism and model parallelism?
- Why does experiment tracking matter, and what do you record?
- Your model scored well offline but performed badly after launch. What could explain it?
- Explain Kubernetes pods, deployments and services, and why you would use Kubernetes for AI apps.
- What is the KV cache, and why can it cause out-of-memory errors when serving an LLM?
- How would you load test an LLM feature and set a latency budget?
- Your model provider has an outage during peak hours. What should already be in place?
- How do you manage API keys and secrets across environments?
- How do you detect and handle model drift in production?
- Why does an LLM reply have a wait before the first word and then a different speed after it?
- What is quantisation, and what are the trade-offs?
- When would you use a semantic cache for an LLM app, and what can go wrong?
- How would you serve an LLM feature as an API for a web app?
- How would you serve a model that is too big for one GPU, and how would you scale it to more users?
- How would you add single sign-on to an enterprise AI assistant?
- How do you version prompts and models so you can roll back safely?
- What would you capture in traces for an LLM application?
- Why use Docker for an AI application?
- What does an LLM gateway do, and when do you need one?
- Why use vLLM to serve an open-weight model instead of a simple Hugging Face script?