Near-duplicate removal
Near-duplicate removal is dropping pages that are almost identical to a page already kept, such as the same article on several mirror sites or the same product page with a different tracking parameter. A search index removes near-duplicates to save space and crawl budget, and a result list removes them so that each slot shows a different page.
Exact duplicates can be found by hashing a page, but near-duplicates need similarity sketches. The classic methods are Broder’s shingling and MinHash (1997) and Charikar’s SimHash (2002), and Manku, Jain and Das Sarma (WWW 2007) showed that 64-bit SimHash fingerprints work for near-duplicate detection on a crawl of 8 billion pages.
Example
A wire-service story appears on 40 news sites with different headers and ads. After content extraction the article text is almost the same everywhere, so the index keeps one copy and records the others as its duplicates.
How it works in NEEDLE
NEEDLE applies a redundancy penalty in News and AgenticRare: when an engine returns the same normalized URL twice, the repeated result’s rating drops to 0 before nDCG@5 is computed. The pooled ideal ranking is already deduplicated. See the live NEEDLE results.
FAQ
What is the difference between duplicate and near-duplicate pages?
Duplicate pages are identical byte for byte and can be found with a simple hash. Near-duplicate pages differ in small parts, such as ads, dates or navigation, so detecting them needs a similarity measure.
Why remove near-duplicates from search results?
Near-duplicate removal gives each result slot a different page, so an agent that reads the top five results gets five sources.
Related terms
Written by Keenable.
Published
