Skip to content
All projects
Shipped

Groundtruth

A retrieval-evaluation harness that measures whether a change to a RAG pipeline made it better or worse — with statistics, not eyeballing a handful of queries.

On this page
Type
project

Built with

The problem

A retrieval system's maintainer changes the chunk size, the embedding model, dense vs. hybrid search, or adds a reranker every few weeks. "Did that help?" is usually answered by trying a handful of queries by eye — which cannot show a regression that only affects one query category, and cannot tell a real improvement from noise.

Why build it

Retrieval failures and generation failures are different problems with different fixes, so this project never blends them into one score. Aggregate numbers hide which query category a change helped or hurt. And with roughly 100 queries, a higher point estimate is not automatically a real difference — so results are reported as confidence intervals, not single numbers.

Architecture

Two pipelines, deliberately never merged:

Retrieval eval:   corpus → chunking → retriever (dense / BM25+RRF / reranker)
                     → top-K passages → compare against golden dataset
                     → Recall@K, MRR, nDCG → bootstrap CIs → error taxonomy
                     → CI regression gate (Recall@10)

Generation eval:  retrieved context → generator → answer with citations
                     → faithfulness / answer relevance / context relevance

A small FastAPI RAG demo (web UI, no build step) shows both blocks side by side for any query, so it's visible which stage caused a bad answer — retrieval returning the wrong passage, or generation misusing a passage it was correctly given.

Data model

Two datasets with different jobs. A synthetic corpus (158 fictional legal-style documents, 100 queries, labels correct by construction) is the deterministic set the CI gate actually uses — seeded generation, programmatic review, versioned, never silently edited. A tiny real-world benchmark (12 public-domain legal excerpts, 19 hand-written queries) exists only as a generalization smoke-check, explicitly not a benchmark: "don't read synthetic absolute metrics as real legal-retrieval performance" is stated directly in the docs, not left implicit.

Testing

CI (.github/workflows/evaluation.yml) runs on every push/PR to main: unit tests → integration tests → evaluation tests including the regression gate → a full evaluation run → uploads the comparison report and dashboard. It makes zero LLM calls — the gate has to be reproducible without an API key or a rate limit in the loop.

A deliberately broken retrieval config (3-document candidate pool, 40-character chunks) exists specifically to prove the gate actually fails: EVAL_CONFIG=broken pytest reliably produces a Recall@10 drop of −0.97 against a threshold of 0.01, and the gate correctly fails the build.

Decisions

Paired bootstrap over point estimates. Since every configuration is scored on the same queries, src/evaluation/statistics.py resamples queries with replacement 10,000 times, recomputes the mean per-query difference each time, and takes the 95% interval from the percentiles. A difference only counts if that interval excludes zero — otherwise it's reported as "inconclusive," meaning the data can't separate the configs, not that they're equal. In the actual results below, most Recall@10 differences among real configs come back inconclusive; only the reranker's ranking gains and the deliberately broken config's collapse are clearly real.

The regression gate refuses to compare across dataset versions, and raises on a missing baseline or an invalid threshold rather than silently passing. Re-approving the baseline to turn a red build green is explicitly against the project's own rules.

Failure labels are heuristics, and the project says so. classify_failure applies rules in a fixed order over computed signals (term overlap, BM25 rank, candidate-pool size) and falls back to unknown as a genuine admission, not a default to be embarrassed about.

Results

On the synthetic set (100 queries), baseline Recall@10 is 0.985. Of four real configuration changes tested, only two held up under the paired bootstrap: the reranker improved MRR and nDCG with a confidence interval excluding zero, and hybrid search's nDCG gain did too — the rest, including most Recall@10 deltas, came back statistically inconclusive at this sample size. The deliberately broken config collapsed to Recall@10 0.015, a clearly real regression (−0.97, CI nowhere near zero).

Baseline error analysis: of 100 queries, 30 were imperfectly retrieved — 40% semantic mismatch, 27% chunk-boundary issues, 20% ranking failures. Switching to 100-character chunks shifted 29 of its 40 imperfect queries specifically into the chunk-boundary category, a visible, attributable effect rather than just a different aggregate number.

Lessons

"Better" needs a confidence interval, not just a bigger number. Several configuration changes that looked like improvements on paper came back "inconclusive" once bootstrapped — the dataset simply wasn't large enough to distinguish them from noise. Reporting that honestly is more useful than reporting a point estimate that implies more certainty than the data supports.

Keeping retrieval and generation evaluation as separate pipelines, with no combined score, was worth the extra code. A single blended metric would have hidden which stage of two is actually responsible when a change makes the end answer worse.