Content extraction
Content extraction is the step that turns a web page’s HTML into its main text by dropping navigation menus, ads, cookie banners, footers and other boilerplate. Search engines and AI agents use the extracted text for indexing, snippets and answers, so extraction errors show up later as missing facts or as noise in the text a language model reads.
Content extraction is also called boilerplate removal. It became a research topic in the 2000s: the CleanEval shared task (2007) compared extractors on real pages, and open-source tools such as boilerpipe (Kohlschütter et al., 2010) and Trafilatura (Barbaresi, 2021) are still widely used. Search APIs that return page text run this step before they return it.
Example
A news article page can weigh 180 KB of HTML while the story itself is about 6 KB of text. Extraction keeps the headline, byline, date and body, and drops the dozens of links to other stories, the newsletter form and the comments widget.
FAQ
Is content extraction the same as web scraping?
No. Web scraping downloads pages or specific fields from them. Content extraction decides which part of a downloaded page is the main text, and a scraper often runs it after fetching a page.
Why does content extraction matter for AI agents?
Content extraction decides what text the language model reads. If it keeps menus and ads, they take up the model’s context window and unrelated text can end up in the answer.
Related terms
Written by Keenable.
Published
