The Topic-Specific Web Crawler of Oriented e-commerce website Design and Implementation
Abstract
Web Crawler is a procedure of automatically downloading website pages, it downloads website pages from the World Wide Web for search engine, and works as an important component of search engine. Traditional Web Crawler starts from one or several of the initial URL of a website, and get some new URLs from the website pages, in the process of continuously downloading website html pages, it finds some new URLs and determine which URLs will be added into a queue, it works until the URL Queue is empty.
The Web Crawler, which is designed by me, is to collect information on the e-commerce websites, so that users can find as much information as they concerned.
The Web Crawler which downloads e-commerce websites, has a very complicated workflow, and needs doing an analysis for the website and filter links which are unrelated to e-commerce website, then keeps the useful links and places them into the URL queue. Then, under certain searching strategy, it would choose the next URL from the queue to download the website page, and repeat this process until the URL queue is empty. In addition, all the pages are stored on the local driver.
Based on the analysis of the principle of the Web Crawler, and the multithreading technology, this article designs this Web Crawler procedure.