Strong Ties vs. Weak Ties: Studying the Clustering Paradox for Decentralized Search

7th Workshop on Large-Scale Distributed Systems for Information Retrieval

Weimao Ke ; Javed Mostafa

Others  :  http://CEUR-WS.org/Vol-480/paper6.pdf
PID  :  11485

来源: CEUR

【 摘 要 】

We studied decentralized search in information networks and focused on the impact of network clustering on the findability of relevant information sources. We developed a multiagent system to simulate peer-to-peer networks, in which peers worked with one another to forward queries to targets containing relevant information, and evaluated the effectiveness, efficiency, and scalability of the decentralized search. Experiments on a network of 181 peers showed that the RefNet method based on topical similarity cues outper- formed random walks and was able to reach relevant peers through short search paths. When the network was extended to a larger community of 5890 peers, however, the advantage of the RefNet model was constrained due to noise of many topically irrelevant connections or weak ties. By applying topical clustering and a clustering exponent α to guide network rewiring, we studied the role of strong ties vs. weak ties, particularly their influence on distributed search. Interestingly, an inflection point was discovered for α, below which performance suffered from many remote connections that disoriented searches and above which performance degraded due to lack of weak ties that could move queries quickly from one segment to another. The inflection threshold for the 5890-peer network was α ≈ 3.5. Further experiments on larger networks of up to 4 million peers demonstrated that clustering optimization is crucial for decentralized search. Although overclustering only moderately degraded search performance on small networks, it led to dramatic loss in search efficiency for large networks. We explain the implication on scalability of distributed systems that rely on clustering for search.