Your model is 99% accurate. That is usually the first thing to be suspicious about

The first machine-learning project most students build reports an accuracy above 95 per cent, and the natural reaction is satisfaction. The correct reaction is suspicion. Not because good models are impossible, but because in practice a very high score on your own test set is more often a symptom of a broken evaluation than evidence of a working model — and the ways it breaks are few enough, and specific enough, to check one by one.

The number that means nothing

Start with the metric itself. Accuracy is the fraction of predictions that were correct, and it is close to useless whenever the classes are unbalanced.

Build a screening model for a disease that affects one person in a thousand. A model that outputs "healthy" for every single input, with no learning whatsoever, is 99.9 per cent accurate. It has also never detected a case, which was the entire point. Any dataset where the interesting class is rare — fraud, equipment failure, rare disease, defect detection — makes accuracy a measure of the base rate rather than of the model.

The arithmetic gets worse when you make the model good. Suppose it correctly flags 99 per cent of true cases and correctly clears 99 per cent of healthy people, which is excellent by any standard. Run it on a hundred thousand people where one in a thousand has the disease: it catches about 99 of the 100 real cases and also flags about 999 healthy people. Of everyone the model marked positive, fewer than one in ten actually has the condition. Nothing is wrong with the model. That is simply what a rare condition does to a test, and it is why deployment decisions need precision, recall and the full confusion matrix rather than a single headline number.

Leakage: when the answer was in the training data

The second failure is the one that produces those suspiciously beautiful scores. Data leakage means information from the test set, or from the future, or from the answer itself, reached the model during training. The model did not generalise. It looked something up.

It arrives in several forms, and only the first is obvious.

Duplicate records across the split. If the same image, augmented copy, or near-identical row appears in both training and test sets, the test is partly a memory exam.

Splitting on the wrong unit. This is the subtle one and it is everywhere in health data. Take ten thousand chest X-rays from two thousand patients and split them randomly, and the same patient's scans land on both sides. The model can recognise the patient rather than the disease, and the test set silently stops being unseen. The rule is to split by the unit you want to generalise across — patient, site, machine, subject — not by row.

Temporal leakage. Any model whose job is prediction must be trained only on information that existed before the moment being predicted. Shuffling a time series randomly lets the model see the future, and financial and demand-forecasting models fail this constantly.

Target leakage. A feature quietly encodes the outcome — a "treatment_started" column in a model predicting diagnosis, or an identifier assigned only after the event. The model achieves near-perfect scores because the label is sitting in the inputs under a different name.

Preprocessing before the split. Normalising, imputing missing values or selecting features using statistics computed over the entire dataset leaks test-set information into training. Every such step belongs inside the cross-validation loop, not before it.

This is not a beginners' problem. A 2023 review of machine-learning-based science found leakage affecting hundreds of published papers across seventeen fields, with reported results that did not survive correction.

A model cannot tell you what it learned. It can only tell you how well it scored — and a score is only meaningful if the test genuinely contained nothing the model had already been shown.

Shortcut learning: solving an easier problem

Even with a clean split, a model will happily learn something other than what you intended, provided it correlates with the label in your data. Neural networks are relentless optimisers with no notion of relevance, and they will always take the cheapest available route to a low loss.

Medical imaging has produced the clearest documented examples. Models trained to detect pneumonia from chest radiographs learned to identify which hospital and which department a scan came from — sicker patients are imaged with portable machines, portable images look different, and the metadata burned into the picture was more predictive than the lungs. Performance was excellent in-house and dropped sharply at other hospitals. Dermatology models learned that a ruler in the frame indicated a lesion a clinician had already thought worth measuring. During the pandemic, systematic reviews of hundreds of covid chest-imaging models found almost none fit for clinical use, with some trained using paediatric scans as the healthy control group — a model that had learned, in effect, to tell children from adults.

None of these systems malfunctioned. Each solved the problem the dataset actually posed, which was easier than the problem the researchers thought they had posed. Shortcut learning is why a held-out test set from the same source is weak evidence, and why external validation on data from a different site, machine or population is the test that counts.

Shift: the world moves after you deploy

The last failure needs no mistake at all. A model is a frozen summary of the data it was trained on, and the data-generating process does not hold still.

Covariate shift means the inputs change — a new imaging device, a different camera, a customer base that grew into a new region. Label shift means the outcome's frequency changes, which is exactly what a disease outbreak, a fraud campaign or a seasonal swing does. Concept drift means the relationship itself changes: the features that predicted default before an interest-rate change stop predicting it after.

There is also a feedback loop unique to deployed models. If a model influences the decisions that generate future data, it corrupts its own training distribution — a fraud model that blocks certain transactions never learns what those would have done, and a maintenance model that triggers early servicing never sees the failures it prevented. This is why monitoring inputs and outcomes after deployment is not operational housekeeping; it is the only way anyone finds out the model has quietly stopped working.

What a real evaluation looks like

The discipline that separates a genuine result from a reported one is not complicated, only unglamorous.

Split by the unit you intend to generalise across, and do it before any preprocessing. Hold out a test set you touch exactly once — every peek at it turns it into a validation set, and a model tuned across fifty experiments against the same test data is overfitted to it whether or not you fitted weights on it. Report metrics appropriate to the base rate, with the confusion matrix and, for anything that produces probabilities, a calibration check: a model saying "80 per cent" should be right about 80 per cent of those times. Compare against a baseline dumb enough to be embarrassing — the majority class, last week's value, a simple regression — because a deep network that cannot beat those has told you something important. Then validate externally, and if the decision matters, prospectively, on data collected after the model was frozen.

Why it matters for students and researchers

Every one of these failures is a claim about generalisation, and generalisation is the only thing a model is ever for. That makes evaluation the core scientific skill in the field, not the paperwork after the interesting part.

It is also where machine learning meets ordinary research method, and where the field embarrassed itself for a decade before catching up. The reproducibility problems in ML-based science come almost entirely from evaluation, not from architecture — which is good news, because it means the fix is available to anyone willing to be rigorous rather than to whoever has the largest cluster. For a student, this is the highest-leverage thing to get right: a modest model with an honest evaluation is a result, and a state-of-the-art model with a leaked test set is nothing at all, however good the number looks.

Frequently asked questions

Why is high accuracy a bad sign in machine learning?

Because it is usually explained by something other than a good model — an imbalanced dataset where predicting the majority class is nearly always right, or leakage that let information from the test set into training. Accuracy above about 95 per cent on a hard real-world problem is worth investigating before it is celebrated.

What is data leakage?

It is any situation where information the model should not have had reaches it during training: duplicated rows across the split, the same patient in both sets, future information in a time series, a feature that encodes the answer, or preprocessing computed over the whole dataset before splitting.

What is shortcut learning?

It is when a model achieves the right answer using an unintended correlation — hospital markings on an X-ray, a ruler in a skin photograph, background colour in an animal dataset. The model performs well on similar data and fails elsewhere, because it never learned the intended signal.

Why do models get worse after deployment?

Because the data changes. New equipment, new populations, seasonal effects and changing relationships all shift the distribution away from what the model was trained on, and a deployed model can also distort the very data it later learns from.

How should a model be evaluated properly?

Split by the unit you want to generalise across before preprocessing, use a test set only once, report precision and recall with the confusion matrix rather than accuracy alone, compare against a trivial baseline, and validate on data from a different source or a later time period.