Skip to content
DSML.KZ
Publications

Article · DOAJ · November 1, 2017

News Classification using Apache Lucene

Sanzhar Aubakirov, Darkhan Akhmed-Zaki, Paulo Trigo

Sanzhar Aubakirov
Read on doaj.org

Abstract

In this paper, we describe the binary classification of textual messages using Apache Lucene. We first describe the problem of “news classification” and the method to gather the dataset samples used for models’ training and testing. We focus on creating and training classifiers based on different indexes and models’ evaluation metrics. We choose three main attributes that affects Apache Lucene indexes, and as a consequence affects accuracy of classifiers. One of the attributes are words on which it is built. We use Ngram concept instead of words, where different N varies from one to five. Second attribute is text preprocessing method, such as stemming and stop words filtering. Third attribute is classification method. In this research we choose k-nearest neighbors and naive Bayes classification methods. Variation of this attributes produces various index types. As a result, we provide analysis of the correlation between index type and the accuracy of classifiers.

Text and Document Classification TechnologiesTopic ModelingData Mining and Machine Learning Applicationsnews classificationApache Lucenetext indexing

More by Sanzhar Aubakirov