A containment metric with no lower bound: auditing retrieval scoring in GraphRAG
A null retriever that returns only entity names scores 100% fact recall under containment scoring. Why that metric has no floor, and a channel-gated fix that costs honest systems nothing.
Retrieval-augmented systems built on knowledge graphs are frequently evaluated with a containment rule: a gold passage counts as retrieved once the context contains the surface form of an entity extracted from it. While building Synapse, I found that this rule has a property nobody wants in a metric: it has no lower bound.
The null-control result
The cleanest way to show it is a retriever that cannot possibly be doing its job. On a 14-question internal multi-hop benchmark, I scored a null retriever that returns nothing but the graph’s alphabetical entity vocabulary. No descriptions, no edges, no prose, and no ability to answer any question.
Under containment scoring, it gets 100% fact recall. Under a prose-grounded rule, which only credits a fact when the context carries actual evidence for it, the same retriever gets 0%.
It replicates, and it is not one system’s bug
The same mechanism shows up on HotpotQA (n = 50). Its Both-gold rule credits a question only when both of its gold paragraphs are retrieved; a graph-only retriever scores 100% on it under containment and 0% under prose-grounding, with 93% of its credited hits coming from entity-name blocks rather than retrieved evidence.
To rule out an implementation quirk, I ran the same null control against a second, independently authored GraphRAG system, LightRAG. Same picture: a zero-evidence retriever scores 100% under containment and 0% under prose-grounding. The inflation is a property of the metric, not of any one codebase.
What survives honest scoring
Under prose-grounded scoring the knowledge graph’s advantage is real but budget-neutral: a plain passage baseline matches it while reading 45% of the context. That is a more modest claim than most GraphRAG evaluations make, and I think it is the accurate one for setups like this.
The fix
The paper proposes channel-gated scoring: credit an entity only when it arrives through a channel that can carry evidence, which costs prose-carrying systems nothing and removes the free floor entirely. The null-control protocol is released with the code, so you can run the same audit against your own retrieval eval before trusting its numbers.
If you evaluate a GraphRAG system with recall-style metrics, running a null control is the cheapest sanity check available: if a retriever that returns only entity names can score, the metric is measuring vocabulary, not retrieval.