Proceedings of the 2018 8th International Conference on Mechatronics, Computer and Education Informationization (MCEI 2018)

Analysis and Design of Improved Intelligent Search Strategy for Web Crawler

Authors
Hongsheng Xu, Bin Zhao, Ganglong Fan
Corresponding Author
Hongsheng Xu
Available Online June 2018.
DOI
10.2991/mcei-18.2018.41How to use a DOI?
Keywords
Intelligent search; Network crawler; Spider; URL; Link filtering
Abstract

This paper mainly studies the design and implementation of the search engine's searcher Spider program, and introduces the concept and technical essentials of Spider program in detail. Network crawler is a web crawler program which can run in the background with configuration file as the initial URL crawling down with the width first algorithm and saving the target URL. The paper presents analysis and design of improved intelligent search strategy for Web crawler. Based on multi-thread web crawler, the client can access the server through socket, and the client sends its own set request to the server.

Copyright
© 2018, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

Volume Title
Proceedings of the 2018 8th International Conference on Mechatronics, Computer and Education Informationization (MCEI 2018)
Series
Advances in Computer Science Research
Publication Date
June 2018
ISBN
10.2991/mcei-18.2018.41
ISSN
2352-538X
DOI
10.2991/mcei-18.2018.41How to use a DOI?
Copyright
© 2018, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

TY  - CONF
AU  - Hongsheng Xu
AU  - Bin Zhao
AU  - Ganglong Fan
PY  - 2018/06
DA  - 2018/06
TI  - Analysis and Design of Improved Intelligent Search Strategy for Web Crawler
BT  - Proceedings of the 2018 8th International Conference on Mechatronics, Computer and Education Informationization (MCEI 2018)
PB  - Atlantis Press
SP  - 214
EP  - 218
SN  - 2352-538X
UR  - https://doi.org/10.2991/mcei-18.2018.41
DO  - 10.2991/mcei-18.2018.41
ID  - Xu2018/06
ER  -