In this thesis, we present a new clustering algorithm we callSignificance Feature Clustering, which is designed to clustertext documents. Its central premise is the mapping of rawfrequency count vectors to discrete-valued significance vectorswhich contain values of -1, 0, or 1. These values representwhether a word is significantly negative,neutral, orsignificantly positive, respectively. Initially, standardtf-idf vectors are computed from raw frequency vectors, then thesetf-idf vectors are transformed to significance vectors using aparameter alpha, where alpha controls the mapping -1, 0, or1 for each vector entry. SFC clusters agglomeratively, with eachdocument;;s significance vector representing a cluster of size onecontaining just the document, and iteratively merges the twoclusters that exhibit the most similar average using cosinesimilarity. We show that by using a good alpha value, thesignificance vectors produced by SFC provide an accurateindication of which words are significant to which documents, aswell as the type of significance, and therefore correspondinglyyield a good clustering in terms of a well-known definition ofclustering quality. We further demonstrate that a user need notmanually select an alpha as we develop a new definition ofclustering quality that is highly correlated with text clusteringquality. Our metric extends the family of metrics known as internal similarity, so that it can be applied to a tree ofclusters rather than a set, but it also factors in an aspect ofrecall that was absent from previous internal similarity metrics.Using this new definition of internal similarity, which we callmaximum tree internal similarity, we show that a close tooptimal text clustering may be picked from any number ofclusterings created by different alpha;;s. The automaticallyselected clusterings have qualities that are close to that of awell-known and powerful hierarchical clustering algorithm.
PDF
download
878987797
028-85220240
