A leak that scores perfect calibration: measuring same-day leakage in delay prediction
Computing "how bad is today at this airport" over the whole day rather than the hours that had elapsed inflates PR-AUC by 21% and drives expected calibration error to 0.00008, which any results table prints as 0.000. Good calibration is not evidence of a sound pipeline.
Published flight-delay models are routinely evaluated on a random split, with historical features computed over the whole dataset. Both are leaks. While building TransportPredictor I measured what the second one is worth, and found that its clearest symptom is a metric usually read as reassurance.
Setup, and why the flights are American
The scarce thing in European aviation data is not volume, it is the target. No source publishes per-flight actual departure times for European airports on terms this project accepts, which means no account, no API key, no quota and no retention cap. The US Department of Transportation publishes exactly that, in the public domain, for every domestic flight.
So the method is built where the ground truth exists at scale. Method transfers between regions. A fitted model does not, and nothing in this piece describes Frankfurt or any European airport. Leakage is a property of how a feature is computed against a clock, not of a country, which is why this particular question can be answered on the corpus that can answer it.
5,834,764 US domestic flights departing 15 large hubs, January 2023 to December 2024. The target is a departure 15 minutes late or more; the test base rate is 18.7%. The prediction is made two hours before the scheduled departure, so a feature is admissible only if it was observable at that instant. The split is chronological with no overlap: the last two months are held out.
Two versions of one feature
The strongest single predictor is the delay rate observed at the departure airport so far that day. There are two ways to compute it:
- as-of: only flights that had actually departed by the prediction instant;
- whole day: a group-by over the calendar day, including the hours that had not happened yet.
The second is not a strawman. It is what a group-by gives you by default.
| Encoding | PR-AUC | Brier skill | Calibration error |
|---|---|---|---|
| as-of | 0.287 | +0.015 | 0.050 |
| whole day | 0.348 | +0.072 | 0.00008 |
The leak is worth +21.3% PR-AUC and nearly quintuples Brier skill.
The number that matters is the last one
The leaky feature’s expected calibration error is 0.00008. Printed to the three decimals a results table normally carries, that reads 0.000, and six of its nine populated reliability bins have a gap of exactly zero. The honest encoding scores 0.050, about six hundred and fifty times worse.
A model that has seen the answer is calibrated on the data it saw, because within each bin its prediction and the outcome are the same quantity. Calibration is normally reported as evidence that a pipeline is sound. Here it is the clearest available evidence that it is not.
Why a coarse check misses it
The same test on a route-level delay rate moves PR-AUC from 0.219 to 0.223, a gain of 1.7%. Route delay rates are stable over months, so seeing the future barely improves the estimate. Anyone who tests for leakage only on a coarse, slow-moving grouping will conclude their pipeline is clean.
What this does not claim
Nothing here says the as-of model is good. At 0.287 against a base rate of 0.187 it is a modest signal, which is what two hours of advance notice buys. The claim is only about the size of the gap between the two encodings, and about where it hides.
The rest of the work
This measurement came out of a larger project, and the rest of it is published as a dashboard. It keeps the same boundary and labels it on every page: Frankfurt and 31 other European airports are covered by EUROCONTROL’s published daily series, while the per-flight model is the American one described above. So the dashboard carries Frankfurt’s daily punctuality since 2022, those 32 airports ranked before and after the network-wide day effect is removed, and a model card that includes a numbered list of what the model does not tell you, starting with the fact that it describes no European airport.
Everything behind it is open bulk data. No account, no API key, no quota, and every figure traceable to the commit that produced it.