Proceedings of the 2015 International Conference on Automation, Mechanical Control and Computational Engineering

Analysis of CDN Website Logs Based on Hadoop

Authors
Qing Song
Corresponding Author
Qing Song
Available Online April 2015.
DOI
https://doi.org/10.2991/amcce-15.2015.110How to use a DOI?
Keywords
log analysis mass data Hadoop MapReduce
Abstract
This paper designs a framework of CDN website log system based on Hadoop and a set of algorithm on the basis of user action mode excavation to analyze and process logs from searching engines. The monitor and regulation of colonies can be realized in platform monitoring modules. Under the guideline of data excavation process, this paper adopts Hadoop, an analysis tool for mass data as the experiment platform. The MapReduce reflection/excavation programming model is used. Simple and applicable HIVE from SQL and Hbase mass data pool are used to process mass logs. The writer conducts a detailed analysis on user searching action from such perspectives as topics, hits, URL order and conversational analysis to optimize platform performance and compare the system before and after the optimization. Experiment data is shown in this paper to explain that the log platform here is quite stable and efficient.
Open Access
This is an open access article distributed under the CC BY-NC license.

Download article (PDF)

Proceedings
2015 International Conference on Automation, Mechanical Control and Computational Engineering
Part of series
Advances in Intelligent Systems Research
Publication Date
April 2015
ISBN
978-94-62520-64-6
ISSN
1951-6851
DOI
https://doi.org/10.2991/amcce-15.2015.110How to use a DOI?
Open Access
This is an open access article distributed under the CC BY-NC license.

Cite this article

TY  - CONF
AU  - Qing Song
PY  - 2015/04
DA  - 2015/04
TI  - Analysis of CDN Website Logs Based on Hadoop
BT  - 2015 International Conference on Automation, Mechanical Control and Computational Engineering
PB  - Atlantis Press
SP  - 599
EP  - 604
SN  - 1951-6851
UR  - https://doi.org/10.2991/amcce-15.2015.110
DO  - https://doi.org/10.2991/amcce-15.2015.110
ID  - Song2015/04
ER  -