What is data poisoning, and how would you protect a model you train or fine-tune from it?
AI security · Pro 1
Short answer
Data poisoning is when an attacker slips harmful examples into training or fine-tuning data so the model learns wrong behaviour, sometimes a hidden backdoor that only appears with a trigger. Defences: know where data comes from, restrict who can add it, version and review datasets, filter outliers and duplicates, test for triggers, and compare behaviour against a clean baseline.
Deeper answer
Poisoning can aim to lower quality in general, or to plant a backdoor: the model behaves normally until it sees a trigger phrase or pattern. Sources of risk include scraped web data, user feedback loops, public datasets and contractors labelling data. RAG knowledge bases can be poisoned in a similar way by adding documents.
Controls: data provenance and an approved-sources list; access control and signed, versioned datasets; review of new data, especially user-submitted examples; statistical checks for odd clusters, label flips and near-duplicates; holding out a trusted evaluation set the pipeline cannot change; red-team tests with candidate triggers; and the ability to roll back to a known-good model. No single filter catches everything, so provenance matters most.
What a strong answer covers
- Defines poisoning: harmful training or fine-tuning data
- Explains backdoors and triggers
- Provenance and access control for data
- Data checks and a trusted evaluation set
- Rollback to a known-good model
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- How is a backdoor different from general poisoning?
- How could a feedback loop from users poison a model?
Common mistakes
- Assuming public datasets are clean because they are popular
- Evaluating only on data that came through the same pipeline
Learn it
Data poisoning, backdoors and adversarial examples
- Attackers now use LLMs to write phishing emails. What changes for defenders, and what still works?
- What is attack success rate in AI red teaming, and what are its limits as a metric?
- Threat model an LLM chatbot for a retail bank that can answer questions and show a customer's recent transactions.
- What is the CIA triad, and how would you use it to judge a security problem?
- As a security engineer, how would you protect a company's finance team from deepfake voice or video calls from the 'CEO'?
- You find code that builds a SQL query by joining strings with user input. What is the risk and how do you fix it?
- What is the difference between hashing and encryption, and which do you use for passwords?
- An employee's laptop shows signs of malware at 11 pm. Walk me through your incident response.
- Explain least privilege. How would you apply it to a new team's cloud and app access?
- Your SOC wants an LLM copilot to summarise alerts and suggest actions. What are the risks and how do you manage them?
- What is tool poisoning in MCP or agent tools, and how do you defend an agent against it?
- What is MITRE ATT&CK, and how does a detection team actually use it?
- A vendor says their ML intrusion detector is 99% accurate. Why might it still drown your SOC in false alarms?
- What is a model extraction attack, and how would you defend a model you serve through an API?
- Walk me through the OWASP Top 10 for LLM applications. Which risks would you check first in a RAG chatbot?
- How would you defend a company against phishing? Where do SPF, DKIM and DMARC fit in?
- A teammate wants to load a model file they downloaded from a public hub. What supply chain risks do you check?
- You are asked to red-team your company's customer support chatbot before launch. How do you go about it?
- Tell me about a security incident or near miss you handled. What did you do and what changed afterwards?
- Staff are pasting company data into public AI tools without approval. How would you handle shadow AI?
- Your SOC gets thousands of SIEM alerts a day and analysts are ignoring them. How do you fix alert fatigue?