LLM-as-a-judge
LLM-as-a-judge is the use of a large language model to grade search results or model answers in place of human raters. The judge receives the task, the item to grade and a rubric, and returns a score or a preference. The approach makes large evaluations cheap and repeatable, but its scores need checks for bias, such as favoring longer answers or a familiar source.
It was popularized by Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (NeurIPS 2023), who found that strong models such as GPT-4 agreed with human preferences more than 80% of the time, about as often as humans agreed with each other. Search benchmarks, including NEEDLE, use the same approach to rate the relevance of each result.
Example
An LLM judge is shown the query “ECB rate decision”, a result title, its URL, its publication date and its snippet, and rates the result from 0 to 4. A snippet about last year’s decision gets a low grade even though it is on topic, because the rubric asks for the current decision.
How it works in NEEDLE
NEEDLE uses LLM-as-a-judge for News and AgenticRare: one call per result on a 0–4 Needs Met scale, at temperature 0. The judge sees the query, title, URL, published date and snippet but not the engine name, and the rubric caps ratings for a wrong entity, stale coverage of a dated event and ignored search operators. See the live NEEDLE results.
FAQ
Can an LLM judge replace human raters?
An LLM judge can replace human raters in large, repeated evaluations. Teams usually check it against a sample of human ratings first, because judges can favor long answers, the first option shown or text in their own style.
Why hide the engine name from an LLM judge?
Without the engine name, the judge cannot rate a result by the brand’s reputation and has to grade what the result itself contains.
Related terms
Written by Keenable.
Published
