Dictionary extraction based on statistical data

Abstract

Automatic text summarization is an actual problem when working with a large amount of information. Most of the algorithms that work on the basis of statistical data build a summary text content by counting the similarity of text units and units importance. Text unit could be a word, sentence or paragraph, in our case unit is a sentence. Similarity is considered the presence of key-words in the sentences. Key-words are words that indicate the topic of the text. In this research work we will describe an automatic extraction of key-words dictionary, where key-words are N-grams with N from 1 to 5. Two algorithms were implemented: getting of words that occur only in one of two different corpora and getting of words with high importance. Importance of N- gram denotes its belonging to the topic of the text. Used text languages are Russian and Kazakh. The algorithms show important results, both of them make sense in constructing of full key-words dictionary.

More by Sanzhar Aubakirov

• From Raw GPS to GTFS: A Real-World Open Dataset for Bus Travel Time PredictionData · 2025 · Citations: 12
• Vehicle Routing Optimization: Urban Logistics Real-case ApplicationIEEE International Conference on Smart Information Systems and Technologies · 2025 · Citations: 4
• USING MLOPS FOR DEPLOYMENT OF OPINION MINING MODEL AS A SERVICE FOR SMART CITY APPLICATIONSScientific Journal of Astana IT University · 2025 · Citations: 2
• From What-If Scenarios to Event Associations: A Novel Approach to Social Media Event AnalysisInternational Conference on Data Science, Technology and Applications · 2025