Web index
A web index is a searchable copy of web pages that a search engine builds by crawling and then queries instead of the live web. For each page it stores the text, metadata such as titles, dates and links, and data structures, such as inverted indexes or vectors, that find matching pages in milliseconds. The size, freshness and quality of a web index limit what any search on it can return.
Only a few organizations run an independent web index, and many search APIs query someone else’s. Google says its Search index covers hundreds of billions of webpages and is well over 100,000,000 gigabytes in size.
Example
When an agent searches for “EU AI Act obligations for general-purpose models”, the engine does not visit any website. It looks the query up in its web index, ranks the stored pages and returns titles, URLs and snippets from its stored copies, which may be hours or months old.
FAQ
How big is a web index?
A large web index holds tens to hundreds of billions of documents. Google says its index covers hundreds of billions of webpages, and Keenable says its web index holds more than 100 billion documents (August 2026).
Does a web index store the whole page?
Most web indexes store the extracted text and metadata of each page plus the structures needed to find it, not a full rendered copy with images and scripts.
Related terms
Written by Keenable.
Published
