Article · Bulletin of the National Engineering Academy of the Republic of Kazakhstan · June 30, 2023
Development of an algorithm for obtaining data from thematic internet resources
Сакен Мамбетов, Ye.Ye. Begimbayeva, A. Khikmetov, Serik Joldasbayev
Abstract
With the rapid development of the Internet, users are actively sharing their personal data and other information on many social networks. Information on the Internet should be analyzed to make sure that it is reliable and does not pose a threat to the public. Based on this, there is a need to collect, monitor and analyze this information. Data collection is a complex task, depending on the structure of each web page. Since not all resources allow you to collect information, you have to use many methods. The proposed article shows effective ways of using syntactic analysis to obtain information. The method of semantic analysis (parsing) of the contents of web pages is explained using a program written in Python based on the BeatifulSoup library. In addition, the focus is on methods of collecting information through other APIs, using tools to emulate user behavior in the browser. An algorithm for extracting information from thematic Internet resources using the BeatifulSoup + Requests library is presented. As a result, information was obtained from Englishand Russian-speaking hacker and carding forums.
More by Serik Joldasbayev
- COMPARISON OF YOLOV8N AND YOLOV12N MODELS FOR SCALABLE REAL TIME VIDEO MONITORING OF CONSTRUCTION SITESScientific Journal of Astana IT University · 2026
- Cyber Threat Risk Assessment from Hacker Forum Emotion and Activity SignalsInternational Conference on Cybersecurity, Digital Forensics, and AI Applications · 2026
- Design and Empirical Evaluation of a Four-Layer AI Agent Architecture for Automated Web Application Security TestingEngineering Technology & Applied Science Research · 2026 · Citations: 2
- AUTOMATION OF THE SYSTEM FOR INDUSTRIAL PRACTICE AND INTERNSHIPS FOR STUDENTS IN ORGANIZATIONS OUTSIDE OF THE UNIVERSITYNEWS OF THE NATIONAL ACADEMY OF SCIENCES OF THE REPUBLIC OF KAZAKHSTAN · 2025 · Citations: 1
