Write a prediction-time contract
Leakage occurs when model development uses information that would not be available in the intended prediction setting. For a subscription cancellation model, specify the prediction timestamp, observation window, and target window. For example, use account activity available on the first day of a month to predict cancellation during the following month. That contract determines which columns are legitimate.
Build a feature inventory with a source, event timestamp, availability timestamp, and reason for inclusion. A payment dispute might have occurred before prediction but been recorded later. Event time alone therefore does not prove availability. Ask an operational colleague to verify when each field actually appears in the system.
Split entities and time before transformations
Start with raw records and assign training, validation, and test membership before learning transformations. If several records belong to the same customer, device, patient, or document family, consider grouping them. Otherwise a model may recognize an entity rather than generalize to a new one.
For forecasting-like tasks, keep later periods out of training. When target windows overlap, inspect whether adjacent examples share future outcomes; a gap between periods may be needed. There is no universally correct split. Document whether your test estimates future performance for existing customers, performance on new customers, or something else. Different deployment questions can legitimately require different evaluation datasets.
Keep learned preprocessing inside the training boundary
Imputation medians, scaling parameters, selected features, text vocabularies, and dimensionality reductions must be learned without looking at held-out examples. Fitting these transformations on the full dataset lets the evaluation distribution influence model construction, even if labels are not directly included. Scikit-learn recommends splitting first and using pipelines to reduce this risk. [1](https://scikit-learn.org/stable/common_pitfalls.html)
During cross-validation, the same principle applies inside every fold. Place preprocessing and the estimator in the object being cross-validated. Fitting a scaler once before cross-validation still leaks information between folds. Target encoding needs additional care: an example's encoding must not casually include its own target or targets from its validation fold.
Audit joins and aggregates with timestamps
Feature engineering often hides leakage in perfectly valid SQL. A lifetime customer-revenue aggregate may include purchases after the prediction date. Joining a current customer snapshot to historical training rows may import account status that only became known months later. The query runs successfully while answering the wrong historical question.
Create a small audit table showing one entity, its prediction timestamp, eligible source records, and resulting aggregate. Manually calculate features for several rows. Include a deliberately future-dated event and verify that it is excluded. Prefer point-in-time joins with explicit availability conditions, and retain source timestamps so future reviewers can reconstruct why a feature value was permissible.
Treat test-set curiosity as a leakage route
A final test set stops being independent when its results guide repeated model, feature, or threshold choices. Keep an experiment log and use validation data for those choices. Open the final test only after the selection process is complete. If the result prompts substantial redesign, acknowledge that the test has become development evidence and plan a new evaluation.
A suspiciously strong score deserves investigation, not celebration. Compare against a feature-ignoring baseline, inspect the most influential fields, and remove near-identifiers experimentally. DummyClassifier offers a straightforward reference that cannot learn predictive relationships from input features. [2](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html)
Make leakage checks executable
Turn the prediction contract into checks: no source availability timestamp after prediction, no customer overlap where groups must be separate, and no prohibited outcome fields in the feature schema. Test duplicate detection using normalized text or source identifiers where appropriate. Add a fixture that should fail each check so you know the checks actually run.
Publish an audit summary alongside model metrics. List unresolved risks, including historical backfills or undocumented field updates. For the cancellation example, ask a reviewer to reconstruct one prediction using only records available that morning. If this cannot be done, the project is not yet ready for a trustworthy offline evaluation, regardless of its reported accuracy.
Sources & further reading
Last editorial review: 27 October 2026. Examples are educational, not live employer data or guaranteed outcomes.