Study and optimization on system architectures of Larbin

网络与信息安全学报

Xuan WANG,Li LI¹  Yi-xia HUO,Yun-fei CI,Guo-zhen SHI² 

[1] School of Information Security,Beijing Electronic Science and Technology Institute,Beijing 100070,China ； School of Computer,Xidian University,Xi'an 710000,China;School of Information Security,Beijing Electronic Science and Technology Institute,Beijing 100070,China;

关键词: search engine;    web crawler;    larbin;    open source;    optimization;   

DOI  :  10.11959/j.issn.2096-109x.2016.00076

来源: DOAJ

【 摘 要 】

Web crawler is an important part of the search engine,its performance will directly affect the accuracy and timeliness of the search engine.Larbin is an efficient and simple open source crawler with relatively perfect in functions.Several typical open-source crawler were firstly introduced and a multi-dimensional comparison was made among them.Then,the system architecture and working mechanism of Larbin were given in detail.Its short-comings in the program structure and process were pointed out,and improved programs were proposed.Experimen-tal results show that improved program is better in speed and performance.

【 授权许可】

Unknown