Ahmed Maaloul · Research

A leak that scores perfect calibration: measuring same-day leakage in delay prediction

Sun, 13 Sep 2026 00:00:00 GMT

Published flight-delay models are routinely evaluated on a random split, with historical features computed over the whole dataset. Both are leaks. While building TransportPredictor I measured what the second one is worth, and found that its clearest symptom is a metric usually read as reassurance.

Setup, and why the flights are American

The scarce thing in European aviation data is not volume, it is the target. No source publishes per-flight actual departure times for European airports on terms this project accepts, which means no account, no API key, no quota and no retention cap. The US Department of Transportation publishes exactly that, in the public domain, for every domestic flight.

So the method is built where the ground truth exists at scale. Method transfers between regions. A fitted model does not, and nothing in this piece describes Frankfurt or any European airport. Leakage is a property of how a feature is computed against a clock, not of a country, which is why this particular question can be answered on the corpus that can answer it.

5,834,764 US domestic flights departing 15 large hubs, January 2023 to December 2024. The target is a departure 15 minutes late or more; the test base rate is 18.7%. The prediction is made two hours before the scheduled departure, so a feature is admissible only if it was observable at that instant. The split is chronological with no overlap: the last two months are held out.

Two versions of one feature

The strongest single predictor is the delay rate observed at the departure airport so far that day. There are two ways to compute it:

as-of: only flights that had actually departed by the prediction instant;
whole day: a group-by over the calendar day, including the hours that had not happened yet.

The second is not a strawman. It is what a group-by gives you by default.

Encoding	PR-AUC	Brier skill	Calibration error
as-of	0.287	+0.015	0.050
whole day	0.348	+0.072	0.00008

The leak is worth +21.3% PR-AUC and nearly quintuples Brier skill.

The number that matters is the last one

The leaky feature’s expected calibration error is 0.00008. Printed to the three decimals a results table normally carries, that reads 0.000, and six of its nine populated reliability bins have a gap of exactly zero. The honest encoding scores 0.050, about six hundred and fifty times worse.

A model that has seen the answer is calibrated on the data it saw, because within each bin its prediction and the outcome are the same quantity. Calibration is normally reported as evidence that a pipeline is sound. Here it is the clearest available evidence that it is not.

Why a coarse check misses it

The same test on a route-level delay rate moves PR-AUC from 0.219 to 0.223, a gain of 1.7%. Route delay rates are stable over months, so seeing the future barely improves the estimate. Anyone who tests for leakage only on a coarse, slow-moving grouping will conclude their pipeline is clean.

What this does not claim

Nothing here says the as-of model is good. At 0.287 against a base rate of 0.187 it is a modest signal, which is what two hours of advance notice buys. The claim is only about the size of the gap between the two encodings, and about where it hides.

The rest of the work

This measurement came out of a larger project, and the rest of it is published as a dashboard. It keeps the same boundary and labels it on every page: Frankfurt and 31 other European airports are covered by EUROCONTROL’s published daily series, while the per-flight model is the American one described above. So the dashboard carries Frankfurt’s daily punctuality since 2022, those 32 airports ranked before and after the network-wide day effect is removed, and a model card that includes a numbered list of what the model does not tell you, starting with the fact that it describes no European airport.

Everything behind it is open bulk data. No account, no API key, no quota, and every figure traceable to the commit that produced it.

→ Open the TransportPredictor dashboard

A containment metric with no lower bound: auditing retrieval scoring in GraphRAG

Thu, 10 Sep 2026 00:00:00 GMT

Retrieval-augmented systems built on knowledge graphs are frequently evaluated with a containment rule: a gold passage counts as retrieved once the context contains the surface form of an entity extracted from it. While building Synapse, I found that this rule has a property nobody wants in a metric: it has no lower bound.

The null-control result

The cleanest way to show it is a retriever that cannot possibly be doing its job. On a 14-question internal multi-hop benchmark, I scored a null retriever that returns nothing but the graph’s alphabetical entity vocabulary. No descriptions, no edges, no prose, and no ability to answer any question.

Under containment scoring, it gets 100% fact recall. Under a prose-grounded rule, which only credits a fact when the context carries actual evidence for it, the same retriever gets 0%.

It replicates, and it is not one system’s bug

The same mechanism shows up on HotpotQA (n = 50). Its Both-gold rule credits a question only when both of its gold paragraphs are retrieved; a graph-only retriever scores 100% on it under containment and 0% under prose-grounding, with 93% of its credited hits coming from entity-name blocks rather than retrieved evidence.

To rule out an implementation quirk, I ran the same null control against a second, independently authored GraphRAG system, LightRAG. Same picture: a zero-evidence retriever scores 100% under containment and 0% under prose-grounding. The inflation is a property of the metric, not of any one codebase.

What survives honest scoring

Under prose-grounded scoring the knowledge graph’s advantage is real but budget-neutral: a plain passage baseline matches it while reading 45% of the context. That is a more modest claim than most GraphRAG evaluations make, and I think it is the accurate one for setups like this.

The fix

The paper proposes channel-gated scoring: credit an entity only when it arrives through a channel that can carry evidence, which costs prose-carrying systems nothing and removes the free floor entirely. The null-control protocol is released with the code, so you can run the same audit against your own retrieval eval before trusting its numbers.

If you evaluate a GraphRAG system with recall-style metrics, running a null control is the cheapest sanity check available: if a retriever that returns only entity names can score, the metric is measuring vocabulary, not retrieval.

Measuring GraphRAG retrieval in Synapse

Thu, 10 Sep 2026 00:00:00 GMT

Most RAG projects ship without measuring retrieval at all. Synapse ships with a small harness, backend/eval/, that turns “it feels grounded” into numbers you can track across changes. This note documents what it measures and how to read the results.

Setup

The harness seeds a fixture graph into Neo4j (15 entities, 10 relationships), runs the production retrieval path (retrieve_subgraph) for each query, and scores the entities it returns. Embeddings are computed locally with fastembed, so the whole thing runs offline:

docker compose up -d neo4j
make eval        # writes backend/eval/results.md

The 8 queries are deliberately paraphrased so they share no lexical overlap with the entity names. “What technology stores data as connected nodes and edges?” has to reach Neo4j through embedding similarity; keyword search would miss most of the set. That is the property being tested: hybrid retrieval (vector and full-text seeds, then 1-hop graph expansion) should survive paraphrase.

Results

Metric	Score
Hit@1	88%
Recall@8	100%
Precision@8	17%
MRR	0.917

Hit@1 and MRR are the quality signal: for 7 of 8 queries the top-ranked entity is a correct one, and every query has all of its expected entities somewhere in the top 8. MRR is the mean reciprocal rank, the average of one divided by the position of the first correct entity, so 0.917 says the first hit is almost always the top one.

Precision@8 looks alarming until you notice it is low by construction. Most queries have only one or two relevant entities, and the retriever returns 8 candidates because the UI highlights a neighborhood, not a single node. Returning fewer candidates would raise precision and make the product worse.

What this does not claim

This is a fixture-sized eval: 8 queries over a 15-entity graph. It catches regressions in the retrieval path and it validates that paraphrase survives the trip through embeddings, which is what I need from CI. It says nothing about retrieval quality on large graphs, noisy extractions, or adversarial queries.

It also inherits a subtler limitation of entity-level scoring itself, which turned into its own investigation: a scoring rule that credits entity names can be gamed by a retriever that returns nothing else. That result, a null-control audit of containment scoring with a replication on HotpotQA and LightRAG, is written up separately in the paper.