Skip to main content

What is benchmark contamination, and how can you detect or avoid it?

Evals · Pro 1

Short answer

Contamination is when a model was trained on a benchmark's questions, so its score shows memory rather than skill. Labs detect it by checking for long word sequences shared between test and training data, or by seeing if the model finds the test text unusually easy to predict. The safest guard is your own private, unseen test set.

Deeper answer

Public benchmarks are posted online, so their questions end up in web crawls and in training data, often by accident. The two common checks are n-gram overlap, which needs access to the training data, and perplexity, where very low perplexity on test questions suggests the model has seen them.

As a user of models you rarely see training data, so treat scores on old, popular benchmarks with care, prefer fresh benchmarks, and keep a private test set from your own data that never goes online. The same problem happens inside your own project if test examples leak into fine-tuning data.

What a strong answer covers

Answer it out loud in the mock interview about 1 min

Follow-ups to expect

Common mistakes

Learn it

Choosing a model: benchmarks, leaderboards and your own tests

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.