Dataset of stopwords extracted from Uzbek texts

Data in Brief

Jernej Vičič¹  Shukurla Bekchanov²  Khabibulla Madatov² 

[1] Research Centre of the Slovenian Academy of Sciences and Arts, The Fran Ramovš Institute, Novi trg 2, 1000 Ljubljana, Slovenija;Urgench state university, 14, Kh. Alimdjan str, Urgench city, 220100, Uzbekistan;

关键词: Stop words;    Machine Learning;    Unigram;    Bigram;    Collocation;   

DOI  :  

来源: DOAJ

【 摘 要 】

Filtering stop words is an important task when processing text queries to search for information in large data sets. It enables a reduction of the search space without losing the semantic meaning. The stop words, which have only grammatical roles and not contributing to information content still add up to the complexity of the query. Existing mathematical models that are used to tackle this problem are not suitable for all families of natural languages [1]. For example, they do not cover families of languages to which Uzbek can be included. In the present work, the collocation method of this problem is o ered for families of languages that include the Uzbek language as well. This method concerns the so-called agglutinative languages, in which the task of recognizing stop words is much more difficult, since the stop words are “masked” in the text. In this work the unigram, the bigram and the collocation methods are applied to the “School corpus” that corresponds to the type of languages being studied.

【 授权许可】

Unknown