Research
Publications and notes, from the systems I build.
Research and engineering notes on retrieval, evaluation, and the systems that keep AI accurate in production.
Subscribe via RSS- Benchmark 4 min read A leak that scores perfect calibration: measuring same-day leakage in delay prediction Computing "how bad is today at this airport" over the whole day rather than the hours that had elapsed inflates PR-AUC by 21% and drives expected calibration error to 0.00008, which any results table prints as 0.000. Good calibration is not evidence of a sound pipeline.
- Research 2 min read A containment metric with no lower bound: auditing retrieval scoring in GraphRAG A null retriever that returns only entity names scores 100% fact recall under containment scoring. Why that metric has no floor, and a channel-gated fix that costs honest systems nothing.
- Benchmark 2 min read Measuring GraphRAG retrieval in Synapse Eight paraphrased queries against a seeded Neo4j fixture: Hit@1 88%, MRR 0.92. What Synapse's eval harness measures, and what it deliberately does not claim.