Hadoop2.4.1入门实例:MaxTemperature
注意:以下内容在2.x版本与1.x版本同样适用,已在2.4.1与1.2.0进行测试。
一、前期准备
1、创建伪分布Hadoop环境,请参考官方文档。或者http://blog.csdn.net/jediael_lu/article/details/38637277
2、准备数据文件如下sample.txt:
123456798676231190101234567986762311901012345679867623119010123456798676231190101234561+00121534567890356
123456798676231190101234567986762311901012345679867623119010123456798676231190101234562+01122934567890456
123456798676231190201234567986762311901012345679867623119010123456798676231190101234562+02120234567893456
123456798676231190401234567986762311901012345679867623119010123456798676231190101234561+00321234567803456
123456798676231190101234567986762311902012345679867623119010123456798676231190101234561+00429234567903456
123456798676231190501234567986762311902012345679867623119010123456798676231190101234561+01021134568903456
123456798676231190201234567986762311902012345679867623119010123456798676231190101234561+01124234578903456
123456798676231190301234567986762311905012345679867623119010123456798676231190101234561+04121234678903456
123456798676231190301234567986762311905012345679867623119010123456798676231190101234561+00821235678903456
二、编写代码
1、创建Map
package org.jediael.hadoopDemo.maxtemperature; import java.io.IOException; import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper; public class MaxTemperatureMapper extends
Mapper<LongWritable, Text, Text, IntWritable> {
private static final int MISSING = 9999; @Override
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
String year = line.substring(15, 19);
int airTemperature;
if (line.charAt(87) == '+') { // parseInt doesn't like leading plus
// signs
airTemperature = Integer.parseInt(line.substring(88, 92));
} else {
airTemperature = Integer.parseInt(line.substring(87, 92));
}
String quality = line.substring(92, 93);
if (airTemperature != MISSING && quality.matches("[01459]")) {
context.write(new Text(year), new IntWritable(airTemperature));
}
}
}
2、创建Reduce
package org.jediael.hadoopDemo.maxtemperature; import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer; public class MaxTemperatureReducer extends
Reducer<Text, IntWritable, Text, IntWritable> {
@Override
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int maxValue = Integer.MIN_VALUE;
for (IntWritable value : values) {
maxValue = Math.max(maxValue, value.get());
}
context.write(key, new IntWritable(maxValue));
}
}
3、创建main方法
package org.jediael.hadoopDemo.maxtemperature; import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class MaxTemperature {
public static void main(String[] args) throws Exception {
if (args.length != 2) {
System.err
.println("Usage: MaxTemperature <input path> <output path>");
System.exit(-1);
}
Job job = new Job();
job.setJarByClass(MaxTemperature.class);
job.setJobName("Max temperature");
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(MaxTemperatureMapper.class);
job.setReducerClass(MaxTemperatureReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
4、导出成MaxTemp.jar,并上传至运行程序的服务器。
三、运行程序
1、创建input目录并将sample.txt复制到input目录
hadoop fs -put sample.txt /
2、运行程序
export HADOOP_CLASSPATH=MaxTemp.jar
hadoop org.jediael.hadoopDemo.maxtemperature.MaxTemperature /sample.txt output10
注意输出目录不能已经存在,否则会创建失败。
3、查看结果
(1)查看结果
[jediael@jediael44 code]$ hadoop fs -cat output10/*
14/07/09 14:51:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
1901 42
1902 212
1903 412
1904 32
1905 102
(2)运行时输出
[jediael@jediael44 code]$ hadoop org.jediael.hadoopDemo.maxtemperature.MaxTemperature /sample.txt output10
14/07/09 14:50:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/09 14:50:41 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/07/09 14:50:42 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
14/07/09 14:50:43 INFO input.FileInputFormat: Total input paths to process : 1
14/07/09 14:50:43 INFO mapreduce.JobSubmitter: number of splits:1
14/07/09 14:50:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1404888618764_0001
14/07/09 14:50:44 INFO impl.YarnClientImpl: Submitted application application_1404888618764_0001
14/07/09 14:50:44 INFO mapreduce.Job: The url to track the job: http://jediael44:8088/proxy/application_1404888618764_0001/
14/07/09 14:50:44 INFO mapreduce.Job: Running job: job_1404888618764_0001
14/07/09 14:50:57 INFO mapreduce.Job: Job job_1404888618764_0001 running in uber mode : false
14/07/09 14:50:57 INFO mapreduce.Job: map 0% reduce 0%
14/07/09 14:51:05 INFO mapreduce.Job: map 100% reduce 0%
14/07/09 14:51:15 INFO mapreduce.Job: map 100% reduce 100%
14/07/09 14:51:15 INFO mapreduce.Job: Job job_1404888618764_0001 completed successfully
14/07/09 14:51:16 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=94
FILE: Number of bytes written=185387
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1051
HDFS: Number of bytes written=43
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=5812
Total time spent by all reduces in occupied slots (ms)=7023
Total time spent by all map tasks (ms)=5812
Total time spent by all reduce tasks (ms)=7023
Total vcore-seconds taken by all map tasks=5812
Total vcore-seconds taken by all reduce tasks=7023
Total megabyte-seconds taken by all map tasks=5951488
Total megabyte-seconds taken by all reduce tasks=7191552
Map-Reduce Framework
Map input records=9
Map output records=8
Map output bytes=72
Map output materialized bytes=94
Input split bytes=97
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=94
Reduce input records=8
Reduce output records=5
Spilled Records=16
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=154
CPU time spent (ms)=1450
Physical memory (bytes) snapshot=303112192
Virtual memory (bytes) snapshot=1685733376
Total committed heap usage (bytes)=136515584
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=954
File Output Format Counters
Bytes Written=43
Hadoop2.4.1入门实例:MaxTemperature的更多相关文章
- React 入门实例教程(转载)
本人转载自: React 入门实例教程
- struts入门实例
入门实例 1 .下载struts-2.3.16.3-all .不摆了.看哈就会下载了. 2 . 解压 后 找到 apps 文件夹. 3. 打开后将 struts2-blank.war ...
- Vue.js2.0从入门到放弃---入门实例
最近,vue.js越来越火.在这样的大浪潮下,我也开始进入vue的学习行列中,在网上也搜了很多教程,按着教程来做,也总会出现这样那样的问题(坑啊,由于网上那些教程都是Vue.js 1.x版本的,现在用 ...
- wxPython中文教程入门实例
这篇文章主要为大家分享下python编程中有关wxPython的中文教程,分享一些wxPython入门实例,有需要的朋友参考下 wxPython中文教程入门实例 wx.Window 是一个基类 ...
- Omnet++ 4.0 入门实例教程
http://blog.sina.com.cn/s/blog_8a2bb17d01018npf.html 在网上找到的一个讲解omnet++的实例, 是4.0下面实现的. 我在4.2上试了试,可以用. ...
- Spring中IoC的入门实例
Spring中IoC的入门实例 Spring的模块化是很强的,各个功能模块都是独立的,我们可以选择的使用.这一章先从Spring的IoC开始.所谓IoC就是一个用XML来定义生成对象的模式,我们看看如 ...
- Node.js入门实例程序
在使用Node.js创建实际“Hello, World!”应用程序之前,让我们看看Node.js的应用程序的部分.Node.js应用程序由以下三个重要组成部分: 导入需要模块: 我们使用require ...
- Java AIO 入门实例(转)
Java7 AIO入门实例,首先是服务端实现: 服务端代码 SimpleServer: public class SimpleServer { public SimpleServer(int port ...
- Akka入门实例
Akka入门实例 Akka 是一个用 Scala 编写的库,用于简化编写容错的.高可伸缩性的 Java 和 Scala 的 Actor 模型应用. Actor模型并非什么新鲜事物,它由Carl Hew ...
随机推荐
- linux 下配置mysql区分大小写(不区分可能出现找不到表的情况)怎么样使用yum来安装mysql
Linux 默认情况下,数据库是区分大小写的:因此,要将mysql设置成不区分大小写 在my.cof 设置 lower_case_table_names=1(1忽略大小写,0区分大小写) 检查方式:在 ...
- Ubuntu 修改 Apache2 用户组 方法
检查/etc/apache2/envvars文件,发现其中需要使用/etc/apache2/envvars文件中的以下几个环境变量 export APACHE_RUN_USER=www-data ex ...
- 微信小程序开发工具 常用快捷键
格式调整 Ctrl+S:保存文件 Ctrl+[, Ctrl+]:代码行缩进 Ctrl+Shift+[, Ctrl+Shift+]:折叠打开代码块 Ctrl+C Ctrl+V:复制粘贴,如果没有选中任何 ...
- Codeforces 474D Flowers
http://codeforces.com/problemset/problem/474/D 思路:F[i]=F[i-1]+(i>=K)F[i-k] #include<cstdio> ...
- awk详解
一.简介 强大的文本分析工具,基于指定规则浏览和抽取信息.简单来说awk就是把文件逐行的读入,以空格为默认分隔符将每行切片,切开的部分再进行各种分析处理.awk有3个不同版本: awk.nawk和ga ...
- 编译型/解释型语言,什么时候用shell
编译型语言 很多传统的程序设计语言,例如Fortran.Ada.Pascal.C.C++和Java,都是编译型语言.这类语言需要预先将我们写好的源代码(source code)转换成目标代码(obje ...
- 主题简介 ASP .NET
由控件的外观.样式组成的集合,由一个文件组构成,存放在App_Themes文件夹下. 主题包括:皮肤文件(.Skin).CSS文件(.CSS).图片.其它资源等. 主题的作用:统一设置Web页面的外观 ...
- PC--CSS优化
优化你的CSS 所谓高效的CSS就是让浏览器在查找style匹配的元素的时候尽量进行少的查找,下面列出一些我们常见的写CSS犯一些低效错误: 1.不要在ID选择器前使用标签名一般写法:DIV#divB ...
- Atitit.软件guibuttonand面板---os区-----linux windows搜索文件 目录
Atitit.软件guibuttonand面板---os区-----搜索文件 1. Find 1 2. 寻找文件夹 1 3. 2. Locate// everything 1 4. 3. Wherei ...
- Android开发之去掉标题栏的三种方法,推荐第三种
Android:去掉标题栏的三种方法和全屏的三种方法 第一种:一般入门的时候常常使用的一种方法 onCreate函数中增加下面代码: requestWindowFeature(Window.FEATU ...