Citebench

A citation-grounded retrieval bench that scores whether retrieval actually finds the evidence (Recall@k, MRR, nDCG) and whether the system abstains instead of inventing an answer — measured over a frozen ~1.5k-doc public corpus.

GitHub

What it is

Citebench is a citation-grounded retrieval bench. It measures two things objectively: whether retrieval actually finds the evidence, and whether the system refuses when it cannot. Questions are asked against a frozen public corpus (~1,500 Kubernetes, Azure, PostgreSQL, FastAPI and xv6 docs) and every answer is scored for grounding. If the corpus cannot support a question, the system must abstain — not invent a password, a payroll row, or a citation.

How it works

A pinned, manifest-based corpus is chunked and embedded locally into Postgres 16 + pgvector. Retrieval fuses dense and full-text results with reciprocal rank fusion, reranks with FlashRank, and either generates an answer or abstains. Every ask emits a run trace, latency metrics, and a scored eval row, so the retrieval quality is visible rather than anecdotal.

pinned public docs (corpus/MANIFEST.md)
  → ingest → Postgres 16 + pgvector (512/64 chunks)
  → retrieve → RRF (k=60) → FlashRank top-50 → top-8
  → generate | abstain (Groq, LangGraph)
  → traces · /metrics (p50/p95) · evals (RAGAS + redteam)

What it measures

Retrieval: Recall@k, MRR and nDCG against hand-labeled questions, so a plausible-sounding answer with no evidence is caught.
Calibration: an answer/abstain matrix that checks the system refuses when the corpus does not support the question.
Gates: a RAGAS faithfulness threshold plus redteam cases run in CI.

Stack

FastAPI + LangGraph on the API, pgvector on Postgres 16 for storage, TanStack Start + shadcn/ui on the web, RAGAS for evaluation, and Groq for generation. Embeddings run locally via fastembed/ONNX, so ingest and retrieve work without a hosted vector service.

Links

Source Docs