A wrong answer is an observation, not a diagnosis. The source may be missing from the index, the retriever may rank it too low, or the model may ignore evidence that was correctly supplied. One end-to-end accuracy number cannot tell those cases apart.
A useful evaluation preserves the boundary between the evidence the system found and the answer it produced.
What the research separates
The Ragas paper was first submitted on September 26, 2023. It proposes automated evaluation across dimensions including faithfulness, answer relevance, and context relevance. Its contribution is a way to inspect different aspects of a retrieval-augmented pipeline rather than relying on a single response score. Automated judges still need validation against the application's own review criteria. Read the Ragas paper.
For an engineering evaluation, start with human-reviewed questions and evidence labels. Judge-based metrics can supplement that set; they should not silently become the definition of correctness.
Keep a reviewable record
For each question, store the expected evidence IDs, a concise answer rubric, the retrieved IDs in order, the exact context sent to the model, and the generated answer. Record the corpus version and retrieval settings alongside them.
Avoid grading only the final text. A plausible answer drawn from the model's prior knowledge may be correct today but fail the requirement to use the organization's current documents.
Recall is not precision
At a chosen cutoff K, recall measures the fraction of labeled relevant items retrieved. Precision measures the fraction of retrieved items that are relevant. Decide whether an item means a document, a passage, or a source section before calculating either metric.
For a clearly hypothetical example, suppose a question has four relevant passages and the top five results contain three of them. Recall@5 is 3 / 4 = 0.75; Precision@5 is 3 / 5 = 0.60. These are arithmetic examples, not results from a deployed system.
def recall_at_k(retrieved_ids, relevant_ids, k): if k < 1: raise ValueError("k must be positive") relevant = set(relevant_ids) if not relevant: return None # Evaluate unanswerable questions separately. retrieved = set(list(dict.fromkeys(retrieved_ids))[:k]) return len(retrieved & relevant) / len(relevant)
Deduplication is explicit here. Otherwise repeated chunks from the same labeled source can create a misleading impression of evidence coverage.
Diagnose before changing the prompt
| Observation | Next investigation |
|---|---|
| Expected source is absent from the index | Ingestion, parsing, permissions, freshness |
| Source exists but is not retrieved | Query formulation, retrieval, filters |
| Evidence is retrieved but omitted from context | Packing, truncation, deduplication |
| Evidence reaches the model but answer is unsupported | Instructions, reasoning, answer validation |
Track questions the system should decline, too. A retriever will usually return something even when nothing is sufficient. Separating answerable and unanswerable cases makes false confidence visible.
A useful release gate
Compare changes against a fixed baseline and review regressions by category. Keep a held-out set for final checks; repeatedly tuning against the same questions makes the test less informative. Report the sample size, label policy, failures, and latency alongside any headline score.