MRR
MRR, or mean reciprocal rank, is a search metric that averages 1 divided by the rank of the first correct result across a set of queries. A query whose first correct result is at position 1 scores 1, at position 2 scores 0.5, at position 4 scores 0.25, and a query with no correct result scores 0.
It is used when each query has one correct answer or one target item, as in question answering and known-item search, and it was the main measure of the TREC-8 Question Answering track in 1999 (Voorhees). It is often reported with a cutoff, such as MRR@10, which ignores correct answers below position 10.
Example
Three queries have their first correct result at positions 1 and 3 and nowhere in the list. Their reciprocal ranks are 1, 0.333 and 0, so MRR = (1 + 0.333 + 0) / 3 = 0.444.
How it works in NEEDLE
NEEDLE reports MRR alongside recall for its known-answer verticals: MRR@5 for Finance and Legal and MRR@10 for Scholar, where a hit means the gold answer, paper identifier or citation appears in a result. See the live NEEDLE results.
FAQ
What is a good MRR?
An MRR of 1 means the first correct result was always at position 1. There is no universal threshold below that: MRR depends on how hard the queries are, so values are comparable only between systems tested on the same queries.
What is the difference between MRR and recall@k?
Recall@k only asks whether a correct result is in the top k, and MRR also accounts for its position. Two engines with the same recall@10 can have very different MRR if one puts the answer first and the other puts it ninth.
Related terms
Written by Keenable.
Published
