统计apachelog各访问状态个数(使用MapReduce)
统计日志文件中各访问状态的个数.
1.将日志数据上传到hdfs
路径 /mapreduce/data/apachelog/in 中
内容如下
::::::: - - [/Feb/::: +] "GET / HTTP/1.1"
::::::: - - [/Feb/::: +] "GET /tomcat.css HTTP/1.1"
::::::: - - [/Feb/::: +] "GET /tomcat.png HTTP/1.1"
::::::: - - [/Feb/::: +] "GET /bg-nav.png HTTP/1.1"
::::::: - - [/Feb/::: +] "GET /asf-logo.png HTTP/1.1"
::::::: - - [/Feb/::: +] "GET /bg-upper.png HTTP/1.1"
::::::: - - [/Feb/::: +] "GET /bg-button.png HTTP/1.1"
::::::: - - [/Feb/::: +] "GET /bg-middle.png HTTP/1.1"
127.0.0.1 - - [/Feb/::: +] "GET / HTTP/1.1"
127.0.0.1 - - [/Feb/::: +] "GET / HTTP/1.1"
::::::: - - [/Feb/::: +] "GET / HTTP/1.1"
::::::: - - [/Feb/::: +] "GET / HTTP/1.1"
127.0.0.1 - - [/Feb/::: +] "GET / HTTP/1.1"
127.0.0.1 - - [/Feb/::: +] "GET / HTTP/1.1"
::::::: - - [/Feb/::: +] "GET / HTTP/1.1"
::::::: - - [/Feb/::: +] "GET / HTTP/1.1"
127.0.0.1 - - [/Feb/::: +] "GET / HTTP/1.1"
127.0.0.1 - - [/Feb/::: +] "GET / HTTP/1.1"
::::::: - - [/Feb/::: +] "GET /sentiment_ms/login HTTP/1.1"
::::::: - - [/Feb/::: +] "GET / HTTP/1.1"
::::::: - - [/Feb/::: +] "GET / HTTP/1.1"
2.代码
package com.zhen.apachelog; import java.io.IOException; import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser; public class ApacheLog { public static class apacheMapper extends Mapper<Object, Text, Text, IntWritable>{ @Override
protected void map(Object key, Text value, Mapper<Object, Text, Text, IntWritable>.Context context)
throws IOException, InterruptedException {
String valueStr = value.toString();
String[] strings = valueStr.split("\" ");
String status = strings[].split(" ")[];
context.write(new Text(status), new IntWritable());
} } public static class apacheReduce extends Reducer<Text, IntWritable, Text, IntWritable>{ @Override
protected void reduce(Text key, Iterable<IntWritable> value,
Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException {
int count = ;
for (IntWritable intWritable : value) {
count+=intWritable.get();
}
context.write(key, new IntWritable(count));
} } public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException { Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf,args).getRemainingArgs(); Job job = new Job(conf,"ApacheLog");
job.setJarByClass(ApacheLog.class); job.setMapperClass(apacheMapper.class);
job.setReducerClass(apacheReduce.class); job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPaths(job, args[]);
FileOutputFormat.setOutputPath(job, new Path(args[])); System.exit(job.waitForCompletion(true)?:);
} }
3.将代码生成jar包
4.调用
EFdeMacBook-Pro:hadoop-2.8.0 FengZhen$ hadoop jar /Users/FengZhen/Desktop/ApacheLog.jar com.zhen.apachelog.ApacheLog /mapreduce/data/apachelog/in /mapreduce/data/apachelog/out
17/09/13 15:32:22 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
17/09/13 15:32:23 INFO input.FileInputFormat: Total input files to process : 1
17/09/13 15:32:23 INFO mapreduce.JobSubmitter: number of splits:1
17/09/13 15:32:23 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1505268150495_0017
17/09/13 15:32:23 INFO impl.YarnClientImpl: Submitted application application_1505268150495_0017
17/09/13 15:32:23 INFO mapreduce.Job: The url to track the job: http://192.168.1.64:8088/proxy/application_1505268150495_0017/
17/09/13 15:32:23 INFO mapreduce.Job: Running job: job_1505268150495_0017
17/09/13 15:32:32 INFO mapreduce.Job: Job job_1505268150495_0017 running in uber mode : false
17/09/13 15:32:32 INFO mapreduce.Job: map 0% reduce 0%
17/09/13 15:32:37 INFO mapreduce.Job: map 100% reduce 0%
17/09/13 15:32:43 INFO mapreduce.Job: map 100% reduce 100%
17/09/13 15:32:43 INFO mapreduce.Job: Job job_1505268150495_0017 completed successfully
17/09/13 15:32:43 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=216
FILE: Number of bytes written=272795
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1776
HDFS: Number of bytes written=13
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3160
Total time spent by all reduces in occupied slots (ms)=3167
Total time spent by all map tasks (ms)=3160
Total time spent by all reduce tasks (ms)=3167
Total vcore-milliseconds taken by all map tasks=3160
Total vcore-milliseconds taken by all reduce tasks=3167
Total megabyte-milliseconds taken by all map tasks=3235840
Total megabyte-milliseconds taken by all reduce tasks=3243008
Map-Reduce Framework
Map input records=21
Map output records=21
Map output bytes=168
Map output materialized bytes=216
Input split bytes=150
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=216
Reduce input records=21
Reduce output records=2
Spilled Records=42
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=54
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=358612992
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1626
File Output Format Counters
Bytes Written=13
5.查看结果
EFdeMacBook-Pro:lib FengZhen$ hadoop fs -ls /mapreduce/data/apachelog/out
Found 2 items
-rw-r--r-- 1 FengZhen supergroup 0 2017-09-13 15:32 /mapreduce/data/apachelog/out/_SUCCESS
-rw-r--r-- 1 FengZhen supergroup 13 2017-09-13 15:32 /mapreduce/data/apachelog/out/part-r-00000
EFdeMacBook-Pro:lib FengZhen$ hadoop fs -text /mapreduce/data/apachelog/out/part-r-00000
200 8
404 13
统计apachelog各访问状态个数(使用MapReduce)的更多相关文章
- 部署Nginx网站服务实现访问状态统计以及访问控制功能
原文:https://blog.51cto.com/11134648/2130987 Nginx专为性能优化而开发,最知名的优点是它的稳定性和低系统资源消耗,以及对HTTP并发连接的高处理能力,单个物 ...
- shell+curl监控网站页面(域名访问状态),并利用sedemail发送邮件
应领导要求,对公司几个主要站点的域名访问情况进行监控.下面分享一个监控脚本,并利用sendemail进行邮件发送. 监控脚本如下:下面是写了一个多线程的网站状态检测脚本,直接从文件中读出站点地址,然后 ...
- Javascript 统计复选框选中个数
var checked = document.getElementsByName("checked_c[]"); var checked_counts = 0; for(var i ...
- 学习笔记_过滤器应用_1(分ip统计网站的访问次数)
分ip统计网站的访问次数 ip count 192.168.1.111 2 192.168.1.112 59 统计工作需要在所有资源之前都执行,那么就可以放到Filter中了. 我们这个过滤器不打算做 ...
- Linux 统计文件夹下文件个数
查看统计当前目录下文件的个数,包括子目录里的. ls -lR| grep "^-" | wc -l Linux下查看某个目录下的文件.或文件夹个数用到3个命令:ls列目录.用gre ...
- 学c语言做练习之统计文件中字符的个数
统计文件中字符的个数(采用命令行参数) #include<stdio.h> #include<stdlib.h> int main(int argc, char *argv[] ...
- 题目--统计一行文本的单词个数(PTA预习题)
PTA预习题——统计一行文本的单词个数 7-1 统计一行文本的单词个数 (15 分) 本题目要求编写程序统计一行字符中单词的个数.所谓“单词”是指连续不含空格的字符串,各单词之间用空格分隔,空格数可以 ...
- shell+curl监控网站页面(域名访问状态),并利用sendemail发送邮件
应领导要求,对公司几个主要站点的域名访问情况进行监控.下面分享一个监控脚本,并利用sendemail进行邮件发送. 监控脚本如下:下面是写了一个多线程的网站状态检测脚本,直接从文件中读出站点地址,然后 ...
- Linux上统计文件夹下文件个数以及目录个数
对于linux终端用户而言,统计文件夹下文件的多少是经常要做的操作,于我而言,我会经常在谷歌搜索一个命令,“如何在linux统计文件夹的个数”,然后点击自己想要的答案,但是有时候不知道统计文件夹命令运 ...
随机推荐
- SpringSecurity学习一----------最简单的权限控制系统
© 版权声明:本文为博主原创文章,转载请注明出处 1.项目结构 2.pom.xml <project xmlns="http://maven.apache.org/POM/4.0.0& ...
- 转:Exception loading sessions from persistent storage
直用tomcat一段时间都正常无事,最近一次启动tomcat就发生以下异常: 严重: IOException while loading persisted sessions: java.io.EOF ...
- PLSQL中显示Cursor、隐示Cursor、动态Ref Cursor差别
一.显式cursor 显式是相对与隐式cursor而言的,就是有一个明白的声明的cursor.显式游标的声明类似例如以下(具体的语法參加plsql ref doc ): cursor cursor_n ...
- Unicode utf8等编码类型的原理
1.ASCII码 我们知道,在计算机内部,所有的信息最终都表示为一个二进制的字符串.每一个二进制位(bit)有0和1两种状态,因此八个二进制位就可以组合出256种状态,这被称为一个字节(byte). ...
- Visual Assist X安装路径
C:\Users\系统用户名\AppData\Local\Microsoft\VisualStudio\VS版本号\Extensions\VAX插件目录\
- Java编程之路相关书籍(三个维度)
一.关于Java的技术学习.能够依照以下分三个维度进行学习 : (1)向下发展,也就是底层的方向 建议看<深入Java虚拟机>.<Java虚拟机规范>.<Thinking ...
- 篇三、开发前知识补充:Android的长度单位和屏幕分辨率,这个也是转载~~
这篇文章有点早,不过很实用.单位的实用看最后的红色标注的部分. 屏幕分辨率基础 1.术语和概念 术语 说明 备注 Screen size(屏幕尺寸) 指的是手机实际的物理尺寸,比如常用的2.8英寸,3 ...
- centos7.2 开发 部署 .net core
1.centos7.2 安装 nginx官方文档:http://nginx.org/en/linux_packages.html#mainline 创建nginx.repo 文件 Pre-Built ...
- erlang 求N以内的质数
素数,又称质数,在一个大于1的自然数中,除了1和此整数自身之外,不能被其他自然数整除的数. 比1大但不是素数的数称为合数. 1和0既不是素数,也不是合数. 算术基本定理证明每个大于1的正整数都可以写成 ...
- ActivityLifecycleCallbacks 如何控制activity的生命周期
Android开发 - ActivityLifecycleCallbacks使用方法初探 初识 ActivityLifecycleCallbacks 利用ActivityLifecycleCallba ...