Data Leakage in Machine Learning: Five Python Experiments That Fool Model Evaluation

Category: AI & Machine Learning · Level: intermediate · Data: reproducible synthetic experiments · Tools: scikit-learn, NumPy

A classifier can report 98% accuracy and still be worthless. Sometimes the model is weak; sometimes the data has been contaminated by information the model will never receive in production. This is data leakage, one of the most damaging—and common—errors in applied machine learning.

We’ll reproduce five distinct failure modes in Python: supervised feature selection before the split, a future target proxy, repeated customers across folds, shuffled validation across a regime change, and preprocessing outside a cross-validation pipeline. We compare each mistake with a defensible alternative. The key result is not a magic performance number: it is learning to define what is known at prediction time.

1. The timeline is the specification

Imagine predicting at Monday 09:00 whether a customer will churn within 30 days. A feature generated Tuesday using support tickets from the following month is not a legitimate input, even if it exists in the historical database today. Eligibility is determined by the prediction timestamp, not by when you export the training CSV.

Write the prediction contract explicitly: the entity, timestamp, available raw fields, label definition, label maturation delay, and deployment population. Without those rules, a train/test split can be syntactically correct but scientifically invalid.

2. Leakage #1: selecting features on the entire dataset

Suppose we generate 2,500 independent Gaussian features and 240 completely random binary labels. There is no true predictive relationship. But if we select the “best” features using all 240 labels before holding out the test set, we choose features partly because they happen to correlate with test labels by chance.

# Wrong: the selection algorithm has seen test-set labels.
selector = SelectKBest(f_classif, k=15).fit(X, y)
X_train_sel = selector.transform(X_train)
X_test_sel = selector.transform(X_test)

# Right: fit feature selection on training folds only.
model = make_pipeline(
    SelectKBest(f_classif, k=15),
    StandardScaler(),
    LogisticRegression(max_iter=1000),
)
model.fit(X_train, y_train)

The bundled script generates the data deterministically and prints both holdout scores. In one verified run of this synthetic example, the leaked score was 0.726 while the proper pipeline scored 0.536—approximately the chance-level result we should expect with random labels. Different seeds will change the realized numbers; the validity principle is invariant.

3. Leakage #2: the outcome hidden in a feature

A business system may include a “cancellation processed” event recorded after customer churn. It correlates strongly with the label because it is effectively the label. Training on that field teaches the model to recognize the aftermath instead of forecast the future.

Our synthetic experiment creates future_label, a noisy copy of the target with 95% agreement. The defective feature set produces an extremely high holdout AUC even though the legitimate features are much weaker. The holdout split does not rescue you here: both training and testing data contain the impossible future feature.

Fix the problem by documenting each feature’s availability time, not merely its database creation time. Introduce point-in-time feature joins, include event timestamp and ingestion timestamp, and verify the join using a sample of raw events. Ask: “Could this value have been queried at the moment the prediction was made?”

4. Leakage #3: repeated people or devices in different folds

Suppose every customer generates five rows and the target remains nearly constant within that customer. A random row-wise split puts some rows from the same customer into both train and test. The model may learn customer identity rather than predicting behavior for new customers.

from sklearn.model_selection import GroupKFold

cv = GroupKFold(n_splits=5)
scores = cross_val_score(model, X, y, groups=customer_ids, cv=cv)

In the executable example, an explicitly encoded customer-ID feature allows a decision tree to memorize labels. A random-row CV achieved 1.000 accuracy on the authored synthetic data; customer-held-out CV achieved 0.575. Neither figure should be generalized to production. The lesson is that the split must match deployment: new customers, new hospitals, new machines or new households often require group-disjoint validation.

5. Leakage #4: randomly shuffling a time-dependent world

Time-series data is not IID by default. If you train on next month’s prices, promotions or fraud behavior and evaluate on last week’s events, you have given the model future conditions that a real-time deployment would not have known. Random K-fold can also hide changes in relationships across market regimes.

The companion script constructs two periods in which a simple feature-label relationship reverses, then compares random K-fold with TimeSeriesSplit. The two estimates can differ substantially; a time split is more aligned with sequential deployment. But note that TimeSeriesSplit alone is not enough if labels overlap test boundaries or delayed outcomes leak into training. For financial problems, consider an embargo or purged walk-forward split when observation windows overlap.

6. Leakage #5: preprocessing fitted before cross-validation

Scaling, imputation, principal component analysis, encoding and feature selection all learn parameters. If they learn those parameters from validation folds, the “validation” data influenced the model-building process.

# Wrong even if it occasionally produces a similar score:
X_scaled = StandardScaler().fit_transform(X)
cross_val_score(LogisticRegression(), X_scaled, y, cv=5)

# Correct: each training fold learns its own scaling parameters.
pipe = make_pipeline(StandardScaler(), LogisticRegression())
cross_val_score(pipe, X, y, cv=5)

Our fifth demo deliberately shows a subtle result: in this particular data draw the leaked and proper scores were both 0.971. That does not make the first method valid. Leakage is an experimental-design defect, not a guarantee that every test score will be measurably inflated. Different transformations and dataset shifts can amplify it.

Five synthetic leakage experiments comparing contaminated and defensible validation.

7. What an honest validation plan looks like

Start by freezing the intended deployment scenario. Then split entities and time before any supervised preprocessing. Build a single scikit-learn pipeline that owns all fitted transformations. Tune hyperparameters only inside the development data; reserve the final holdout for one honest evaluation. For small datasets, nested cross-validation can estimate the performance of the whole model-selection procedure rather than the best of many tried configurations.

Measure more than accuracy. In an imbalanced fraud dataset, precision, recall and PR-AUC may matter more. Calibrate predicted probabilities when decisions depend on thresholds. Report confidence intervals or variability across independent temporal windows, and retain subgroup breakdowns when relevant to the application.

8. How to audit your own project

For each feature, document its provenance and availability timestamp. For each transform, identify which rows were passed to fit. For each hyperparameter, identify the dataset that informed its choice. For each duplicate or related record, identify the grouping policy. Review the final test set to ensure it was not used repeatedly while developing the model. A surprising performance collapse after “fixing leakage” is not bad news: it is a more realistic estimate of how the system will behave.

Download and run leakage_lab.py from the accompanying research bundle:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python leakage_lab.py

Everything is synthetic, deterministic and offline. No claims about real customers or deployment accuracy are made. The five printouts are starting points for reasoning, not benchmark leaderboards.

Takeaway: you cannot trust model evaluation until you can justify the information boundary. Extend this lesson to financial backtests in Mean Reversion Trading in Python and to LLM Application Evaluation. Assess your own foundations through the Algonique AI Skill Assessment.

Frequently asked questions

Does a random train/test split prevent data leakage? No. A post-outcome feature, a group identifier or future-dated observation can be present in both subsets.

Is it always wrong to standardize before splitting? It compromises a holdout or CV evaluation when the scaler learns from those held-out rows. Always fit the scaler within the training partition.

What is the difference between overfitting and leakage? Overfitting is learning idiosyncrasies of the training data; leakage is allowing impermissible information into model fitting or evaluation. They can occur together.

Should I use TimeSeriesSplit for all event data? Not automatically. Choose splits that reproduce how predictions will be deployed, including group structure, label maturation and temporal dependencies.