Mathematical Model and Algorithm for Accurate Main Content Extraction From News Websites

Abstract

Irrelevant elements like ads, menus, and footers in web pages hinder data extraction and reduce the performance of Retrieval-Augmented Generation (RAG) systems in Large Language Models (LLMs). This paper tackles the challenge of accurately identifying and extracting the main content from web pages to enhance the efficiency of these systems. We present a novel mathematical model and algorithm that leverages the Document Object Model (DOM) structure, effectively isolating relevant content with high accuracy. Our approach is language-neutral and performs well across diverse languages, including those with complex tokenization, such as Arabic. To validate the model, we created a dataset from 500 websites, allowing for comprehensive evaluation and benchmarking. The algorithm’s practical application demonstrates a reduction in token usage for LLM tasks, contributing to cost-effectiveness. This work introduces a robust, open-source tool for the academic and commercial communities, fostering further innovation in web content extraction and information retrieval.

More by Manuel Mazzara

• Anchored Rank-Adversarial Fine-Tuning Pretrained Visual Place Recognition ModelsarXiv · 2026
• See the Future, Classify the Present: Stochastic Latent World Modeling for Hyperspectral ImagesarXiv · 2026
• Autoresearch with Coding Agents: Generalizers and Metric-Maximizers on Quran Recitation DataLecture notes in computer science · 2026 · Citations: 1
• What Counts as a Mistake? Annotating Recitation Events in Quran Memorization TranscriptsarXiv · 2026