Crawling and Analysis of Data Based on Social Networking on Stock Comments

6th Annual 2018 International Conference on Geo-Spatial Knowledge and Intelligence

Gao, Gang^1 ; Liu, Yupeng^1 ; Bai, Gang^1

Department of Software Engineering, Harbin University of Science and Technology, China^1

关键词: Analysis of data;    Baseline systems;    Character patterns;    Crawler program;    Investment strategy;    Regular expressions;    Research object;    Social network informations;   

Others  :  https://iopscience.iop.org/article/10.1088/1755-1315/234/1/012093/pdf
DOI  :  10.1088/1755-1315/234/1/012093

来源: IOP

【 摘 要 】

In this paper, stock data on social networking platforms for the research object, a reptile related technologies. The main purpose of this paper is to design web crawler programs for web sites to crawl stock information and comments. Analysis of crawling information will be used to help investor to do their own investment strategies. The details and application of Python crawler are described in detail. And we have compared with the baseline system to exhibit our crawler's advantages. The work as follows:1. Analysis of the existence of the current social network information during the crawling problem, which leads to the need to achieve the design goal of reptiles; 2. In order to crawl enormous data from different social networks, we constructed several different social networks information suitable for crawling web crawler by using different methods.; 3. Using regular expression to quickly parse large amounts of text to find specific character patterns; extract, edit, replace, or delete text substrings. 4. Climb take the information including stock opening price, stock closing price, turnover and content comment; 5. Putting the information into the document for data storage.