五、基于hadoop的nginx访问日志分析--userAgent和spider
useragent:
代码(不包含蜘蛛):
# cat top_10_useragent.py
#!/usr/bin/env python
# coding=utf-8 from mrjob.job import MRJob
from mrjob.step import MRStep
from nginx_accesslog_parser import NginxLineParser import heapq class UserAgent(MRJob): nginx_line_parser = NginxLineParser() def mapper(self, _, line): self.nginx_line_parser.parse(line)
field_item = self.nginx_line_parser.http_user_agent
if field_item is not None:
yield field_item, 1 def reducer_sum(self, key, values): yield None, (sum(values), key) def reducer_top100(self, _, values):
for count, path in heapq.nlargest(10, values):
yield count, path
# for count, path in sorted(values, reverse=True)[:10]:
# yield count, path def steps(self):
return (
MRStep(mapper=self.mapper,
reducer=self.reducer_sum
),
MRStep(reducer=self.reducer_top100)
) def main():
UserAgent.run() if __name__ == '__main__':
main()
结果:
# python3 top_10_useragent.py access_all.log-20161227
No configs found; falling back on auto-configuration
Creating temp directory /tmp/top_10_useragent.root.20161228.090725.308144
Running step 1 of 2...
Running step 2 of 2...
Streaming final output from /tmp/top_10_useragent.root.20161228.090725.308144/output...
85262 "IE"
79611 "Chrome"
48560 "Other"
10662 "Firefox"
7927 "Mobile Safari UI/WKWebView"
7182 "Sogou Explorer"
6681 "QQ Browser"
1988 "Mobile Safari"
1781 "Maxthon"
1404 "Edge"
Removing temp directory /tmp/top_10_useragent.root.20161228.090725.308144...
蜘蛛:
#!/usr/bin/env python
# coding=utf-8 from mrjob.job import MRJob
from mrjob.step import MRStep
from nginx_accesslog_parser import NginxLineParser import heapq class Spider(MRJob): nginx_line_parser = NginxLineParser() def mapper(self, _, line): self.nginx_line_parser.parse(line)
field_item = self.nginx_line_parser.user_agent_type
if field_item is not None:
yield field_item, 1 def reducer_sum(self, key, values): yield None, (sum(values), key) def reducer_top100(self, _, values):
for count, path in heapq.nlargest(10, values):
yield count, path
# for count, path in sorted(values, reverse=True)[:10]:
# yield count, path def steps(self):
return (
MRStep(mapper=self.mapper,
reducer=self.reducer_sum
),
MRStep(reducer=self.reducer_top100)
) def main():
Spider.run() if __name__ == '__main__':
main()
执行结果:
# python3 top_10_spider.py access_all.log-20161227
No configs found; falling back on auto-configuration
Creating temp directory /tmp/top_10_spider.root.20161228.091326.295972
Running step 1 of 2...
Running step 2 of 2...
Streaming final output from /tmp/top_10_spider.root.20161228.091326.295972/output...
33542 "magpie-crawler"
25880 "Other"
16578 "Sogou web spider"
6383 "bingbot"
3688 "Baiduspider"
1487 "Yahoo! Slurp"
1096 "JikeSpider"
731 "YisouSpider"
648 "Baiduspider-image"
470 "Googlebot"
Removing temp directory /tmp/top_10_spider.root.20161228.091326.295972...
五、基于hadoop的nginx访问日志分析--userAgent和spider的更多相关文章
- 一、基于hadoop的nginx访问日志分析---解析日志篇
前一阵子,搭建了ELK日志分析平台,用着挺爽的,再也不用给开发拉各种日志,节省了很多时间. 这篇博文是介绍用python代码实现日志分析的,用MRJob实现hadoop上的mapreduce,可以直接 ...
- 四、基于hadoop的nginx访问日志分析---top 10 request
代码: # cat top_10_request.py #!/usr/bin/env python # coding=utf-8 from mrjob.job import MRJob from mr ...
- 二、基于hadoop的nginx访问日志分析---计算日pv
代码: # pv_day.py#!/usr/bin/env python # coding=utf-8 from mrjob.job import MRJob from nginx_accesslog ...
- 三、基于hadoop的nginx访问日志分析--计算时刻pv
代码: # cat pv_hour.py #!/usr/bin/env python # coding=utf-8 from mrjob.job import MRJob from nginx_acc ...
- nginx访问日志分析,筛选时间大于1秒的请求
处理nginx访问日志,筛选时间大于1秒的请求 #!/usr/bin/env python ''' 处理访问日志,筛选时间大于1秒的请求 ''' with open('test.log','a+' ...
- Nginx 访问日志分析
0:Nginx日志格式配置 # vim nginx.conf ## # Logging Settings ## log_format access '$remote_addr - $remote_us ...
- Nginx访问日志分析
nginx默认的日志格式 log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$sta ...
- 13 Nginx访问日志分析
#!/bin/bash export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin # Nginx 日志格式: # ...
- 采集并分析Nginx访问日志
日志服务支持通过数据接入向导配置采集Nginx日志,并自动创建索引和Nginx日志仪表盘,帮助您快速采集并分析Nginx日志. 许多个人站长选取了Nginx作为服务器搭建网站,在对网站访问情况进行分析 ...
随机推荐
- Oracle分区索引
索引与表类似,也可以分区: 分区索引分为两类: Locally partitioned index(局部分区索引) Globally partitioned index(全局分区索引) 下面就来详细解 ...
- Python来做应用题及思路
Python来做应用题及思路 最近找工作头疼没事就开始琢磨python解应用题应该可以,顺便还可以整理下思路当然下面的解法只是个人理解,也欢迎大佬们给意见或者指点更好的解决办法等于优化代码了嘛,也欢迎 ...
- IE8.0.6001这个版本执行JS报错
现场: OS winserver 2008 IE8.0.6001.18702 JS报错,不能登陆 OS XP IE8.0.6001.18702CO JS报错,不能登陆 公司测试: ...
- IIS将错误信息发送到浏览器
本文版权归博客园和dige1993所有,访问作者博客:http://www.cnblogs.com/dige1993 最近又开始玩ASP了,调试的时候出现错误不清楚详细错误信息特别不方便,记得以前可以 ...
- [KMP]【学习笔记】
Oulipo Time Limit: 1000MS Memory Limit: 65536K Total Submissions: 36916 Accepted: 14904 Descript ...
- python_接口开发
一.GET和POST请求from flask import Flask,jsonifydata = { 'name':'Jerry', 'sex':'男', 'age':'18'}app = Flas ...
- 转 jQuery 中bind(),live(),delegate(),on() 区别
当我们试图绑定一些事件到DOM元素上的时候,我相信上面这4个方法是最常用的.而它们之间到底有什么不同呢?在什么场合下用什么方法是最有效的呢? 准备知识: 当我们在开始的时候,有些知识是必须具备的: D ...
- dubbox 增加google-gprc/protobuf支持
好久没写东西了,今年实在太忙,基本都在搞业务开发,晚上来补一篇,作为今年的收官博客.google-rpc 正式发布以来,受到了不少人的关注,这么知名的rpc框架,不集成到dubbox中有点说不过去. ...
- [LeetCode] Binary Tree Level Order Traversal II 二叉树层序遍历之二
Given a binary tree, return the bottom-up level order traversal of its nodes' values. (ie, from left ...
- web 前端常用组件【07】弹出层 Layer
web 项目中总是需要弹出框,来让用户进行下一步的操作. 大到弹出另外一个页面,小到弹出提示.确认等. 经手几个项目,还是感觉 Layer 用起来比较的轻松,你能想到的 Layer 都能帮你做到. 感 ...