Crawl freshness
Crawl freshness is how recently a search engine re-fetched the pages in its index. A page crawled months ago may still be in the index with an old price, an old headline or a dead link, so crawl freshness limits how current any answer built on that index can be. Engines manage it by revisiting pages that change often, such as news front pages, more often than pages that rarely change.
Freshness became a formal metric in web crawling research when Cho and Garcia-Molina (SIGMOD 2000) defined the freshness of a stored page as whether it still matches the live page, and its age as how long it has been out of date.
Example
A product page changes its price every week, but the crawler revisits it once a month. After each visit the stored copy is current for at most a week and out of date for about three, so a search for the price returns an old figure roughly three times out of four.
FAQ
How often do search engines re-crawl pages?
There is no single interval. Search engines set crawl frequency per page, revisiting pages that change often, such as news front pages, within minutes or hours and stable pages over weeks or months.
What is the difference between crawl freshness and data freshness?
Crawl freshness describes the index: how recently its stored copies were re-fetched. Data freshness describes the results a query gets back, which also depends on how fast new pages are discovered and indexed.
Related terms
Written by Keenable.
Published
