NEEDLE
NEEDLE is Keenable’s open-source benchmark of web search APIs for AI agents. The name stands for News, Everyday, Expert, Deep-tail and Legal Evaluation. It sends the same queries to every engine, scores only the rankings, titles and snippets each one returns, and builds new query sets from public sources every hour for News and every day for the other verticals, so there is no fixed test set to overfit.
Keenable published NEEDLE in August 2026, with code in the public keenableai/needle repository, runs in plain GitHub Actions and a live dashboard. Its five verticals, News, Finance, Scholar, AgenticRare and Legal, match the query types Keenable sees in its own search traffic from AI agents.
Key facts
Verticals: NEEDLE has five verticals, News, Finance, Scholar, AgenticRare and Legal, each with its own query source and scoring.
Schedule: NEEDLE runs News every hour and the other verticals once a day, at fixed times in UTC.
Scoring: News and AgenticRare use nDCG@5 with an LLM judge, Finance uses answer-recall@5 and MRR@5, Scholar uses recall@10 and MRR@10 and Legal uses recall@5 and MRR@5, both by identifier match.
Ceiling: Every NEEDLE leaderboard includes Ultimate, a synthetic engine that pools all engines’ results and shows the best score the field could reach.
Engines: As of October 2026, NEEDLE compares more than 20 search APIs, including Google and Bing through SERP providers.
How it works in NEEDLE
Every run publishes its scores, the overlap between engines and the per-query results, and the full artifacts are archived in the keenable-ai/needle-results dataset on Hugging Face. See the live NEEDLE results.
FAQ
Is NEEDLE open source?
Yes. The NEEDLE code is in the public keenableai/needle repository on GitHub, every scheduled run executes in GitHub Actions, and run results are archived in a public Hugging Face dataset.
Which search APIs does NEEDLE compare?
NEEDLE compares Keenable with other web search APIs, including Exa, Brave, Tavily, Parallel and Perplexity, and with Google and Bing through SERP providers. The full list is on the NEEDLE dashboard.
Related terms
Written by Keenable.
Published
