Ultimate (ceiling)
Ultimate is a synthetic engine in the NEEDLE benchmark that pools the results of every tested engine for a query, removes duplicates and ranks them by their judged relevance. Its score is the best result the whole field could reach together, a ceiling for comparison. The gap between an engine and Ultimate shows how much better results existed among the other engines.
Ultimate appears as the italic row in each NEEDLE leaderboard. A pooled ceiling separates two failures: an engine that ranked poorly while better results existed elsewhere, and a query where no engine found good results at all.
Example
On a rare-entity query, engine A returns two relevant pages and engine B returns one different relevant page. Ultimate holds all three, so it scores higher than either engine, and both A and B show a gap to the ceiling.
How it works in NEEDLE
In News and AgenticRare, NEEDLE normalizes nDCG@5 against Ultimate’s ordering, so Ultimate scores exactly 1 and every engine’s score is a share of it. The standings radar plots each engine as a share of Ultimate as well. See the live NEEDLE results.
FAQ
Is Ultimate a real search engine?
No. Ultimate is a synthetic reference built after each run from the other engines’ results, so it cannot be queried.
Why compare engines to Ultimate instead of to each other?
The share of Ultimate measures how much of the achievable quality an engine delivers on the same queries. A low Ultimate score shows that the whole field struggled on those queries.
Related terms
Written by Keenable.
Published
