ENTROPY: A New Measure to Gauge Search Engine Optimisation

Abstract

Evaluating the diversity and robustness of search engine results remains a critical challenge, as traditional SEO metrics fail to capture semantic variation and redundancy. This study introduces a unified entropy-distance metric that integrates Shannon entropy (lexical diversity) with cosine distance (semantic dispersion). A large-scale dataset of$\text{6. 6 M}$English Wikipedia articles was clustered into 15 topical groups, from which$\text{1, 6 5 0}$paraphrased queries were generated using multiple linguistic transformation strategies. Search results were retrieved from Google (top-10), Bing (top-50), and DuckDuckGo (top-50), yielding over 170k snippets. Empirical evaluation shows that DuckDuckGo achieved the highest diversity (entropy = 8.26; log score$=2.82$), Google produced narrower but semantically coherent outputs (entropy$=6.21$; log score$=2.73$), while Bing maintained moderate performance (entropy$=7.70$;$\log$score = 2.68). Paraphrase types such as translation and temporal framing significantly increased semantic variability across engines ($\mathrm{p} <10^{-290}$). The entropy-distance metric offers a scalable, unsupervised, and language-agnostic diagnostic for benchmarking retrieval robustness, paraphrase sensitivity, and diversity in search engine evaluation.

More by Iskander Akhmetov

• A Lightweight Extractive Compression Pipeline for Long-Context Retrieval-Augmented GenerationComputación y Sistemas · 2026
• Character N-Gram Features for Kazakh POS Tagging Under Strict OOV EvaluationInternational Conference on Computer Science and Engineering · 2026
• KazakhStopEval: Stop-Word Inventories, Validation Benchmarks, and NLP Evaluation Artifacts for Kazakh Library OCR CollectionsMendeley Data · 2026
• Hierarchical Multi-Agent Extractive Framework for Long-Document Mind Map GenerationComputación y Sistemas · 2026