Measuring retrieval with recall@k

Score the retriever on its own

Recall at k asks whether the passage containing the answer appears in the top k results. If it does not, no amount of prompt engineering downstream can produce a correct answer.

Building the labelled set is the real work: real queries paired with the passages that genuinely answer them. A few dozen carefully labelled queries are enough to make pipeline changes measurable.

Keep this metric separate from end-to-end answer quality. Combining them means a retrieval regression can be masked by a generation improvement, and you will not see it until users do.