Measuring GraphRAG retrieval in Synapse
Eight paraphrased queries against a seeded Neo4j fixture: Hit@1 88%, MRR 0.92. What Synapse's eval harness measures, and what it deliberately does not claim.
Most RAG projects ship without measuring retrieval at all. Synapse ships with a
small harness, backend/eval/, that turns “it feels grounded” into numbers you
can track across changes. This note documents what it measures and how to read
the results.
Setup
The harness seeds a fixture graph into Neo4j (15 entities, 10 relationships),
runs the production retrieval path (retrieve_subgraph) for each query, and
scores the entities it returns. Embeddings are computed locally with
fastembed, so the whole thing runs offline:
docker compose up -d neo4j
make eval # writes backend/eval/results.mdThe 8 queries are deliberately paraphrased so they share no lexical overlap with the entity names. “What technology stores data as connected nodes and edges?” has to reach Neo4j through embedding similarity; keyword search would miss most of the set. That is the property being tested: hybrid retrieval (vector and full-text seeds, then 1-hop graph expansion) should survive paraphrase.
Results
| Metric | Score |
|---|---|
| Hit@1 | 88% |
| Recall@8 | 100% |
| Precision@8 | 17% |
| MRR | 0.917 |
Hit@1 and MRR are the quality signal: for 7 of 8 queries the top-ranked entity is a correct one, and every query has all of its expected entities somewhere in the top 8. MRR is the mean reciprocal rank, the average of one divided by the position of the first correct entity, so 0.917 says the first hit is almost always the top one.
Precision@8 looks alarming until you notice it is low by construction. Most queries have only one or two relevant entities, and the retriever returns 8 candidates because the UI highlights a neighborhood, not a single node. Returning fewer candidates would raise precision and make the product worse.
What this does not claim
This is a fixture-sized eval: 8 queries over a 15-entity graph. It catches regressions in the retrieval path and it validates that paraphrase survives the trip through embeddings, which is what I need from CI. It says nothing about retrieval quality on large graphs, noisy extractions, or adversarial queries.
It also inherits a subtler limitation of entity-level scoring itself, which turned into its own investigation: a scoring rule that credits entity names can be gamed by a retriever that returns nothing else. That result, a null-control audit of containment scoring with a replication on HotpotQA and LightRAG, is written up separately in the paper.