Article · LA Referencia · January 1, 2002
A common solution for tokenization and part-of-speech tagging: one-pass Viterbi algorithm vs. Iterative approaches
Jorge Graña Gil, Miguel Á. Alonso, Manuel Vilares Ferro
Abstract
Current taggers assume that input texts are already tokenized, i.e. correctly segmented in \\emph{tokens} or high level information units that identify each individual component of the texts. This working hypothesis is unrealistic, due to the heterogeneous nature of the application texts and their sources. The greatest troubles arise when this segmentation is ambiguous. The choice of the correct segmentation alternative depends on the context, which is precisely what taggers study. In this work, we develop a tagger able not only to decide the tag to be assigned to every token, but also to decide whether some of them form or not the same term, according to different segmentation alternatives. For this task, we design an extension of the Viterbi algorithm able to evaluate streams of tokens of different lengths over the same structure. We also compare its time and space complexities with those of the classic and iterative versions of the algorithm.
More by Miguel Alonso
- Public Sentiment Analysis and Topic Modeling Regarding COVID-19’s Three Waves of Total Lockdown: A Case Study on Movement Control Order in MalaysiaKSII Transactions on Internet and Information Systems · 2022 · Citations: 11
- Construction and Evaluation of Sentiment Datasets for Low-Resource Languages: The Case of UzbekLecture notes in computer science · 2022 · Citations: 42
- Sentiment Analysis for Fake News DetectionElectronics · 2021 · Citations: 270
- On the Use of Parsing for Named Entity RecognitionApplied Sciences · 2021 · Citations: 24
