What does a good data strategy for an AI product look like?
AI product management · Intermediate 3
Short answer
Know what data you need for evals, retrieval and possibly fine-tuning; how you collect it with consent; how you label and keep quality high; who can access it; and how product usage creates new data, such as corrections and feedback, that improves the product over time. Treat a well-maintained eval set as one of your most valuable assets.
Deeper answer
Many teams think 'data strategy' means training a model, but for most AI products the data that matters is: documents for retrieval, examples for evals, and logs of real usage. Design feedback capture into the product from day one: thumbs, edits and escalations are labels.
Respect limits: consent and purpose under the DPDP Act, customer contracts, and retention rules. A data flywheel only works if users trust you with their data.
What a strong answer covers
- Data for evals, retrieval and fine-tuning
- Collection with consent and purpose
- Labelling and quality
- Feedback loop from product usage
- Access control and retention
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- What product signals can act as labels?
- How would you keep the eval set from going stale?
Common mistakes
- Equating data strategy with training a model
- Collecting data without consent or purpose
Learn it
- Should we build our own AI feature or buy a vendor product?
- Your AI feature launched, usage is high, but customer satisfaction dropped. What do you do?
- How would you measure the success of an AI feature?
- How do you design a product experience when the AI is sometimes wrong?
- You launched an AI tool internally and nobody uses it. What do you do?
- How do you decide whether a problem is a good fit for AI?
- As a PM, how do you work with engineers on evals?
- How would you price an AI feature whose cost grows with usage?
- When should a product keep a human in the loop?