hadoop学习记录1 初始hadoop
起因
因为工作需要用到,所以需要学习hadoop,所以记录这篇文章,主要分享自己快速搭建hadoop环境与运行一个demo
搭建环境
网上搭建hadoop环境的例子我看蛮多的.但是我看都比较复杂,要求安装java,hadoop,然后各种设置..很多参数变量都不明白是啥意思...我的目标很简单,首先应该是用最简单的方法搭建好一个环境.各种变量呀参数呀这些我觉得一开始对我都不太重要..我只要能跑起来1个自己的简单demo就行.而且现实中基本上环境也不会让我来维护..所以对我来说简单就行.
刚好最近我一直在看docker..所以我就打算用docker来搭建这个环境.算是同时学习hadoop和docker吧.
首先安装docker....很简单...这里就不介绍了.官方有一键安装脚本...
docker hub中有1个官方的hadoop的例子.
https://hub.docker.com/r/sequenceiq/hadoop-docker/
我稍微修改了一下命令:
额外挂载了1个目录,因为我要上传我自己写的demo jar到docker里去用hadoop运行.
另外把这个container取名字为hadoop2,因为我跑了很多容器,取名字便于区分,而且后面可能要用多个hadoop容器来制作集群.
docker run -it -v /dockerVolumes/hadoop2:/dockerVolume --name hadoop2 sequenceiq/hadoop-docker:2.7. /etc/bootstrap.sh -bash
运行好这个命令,这个容器就已经运行起来了.我们可以跑一下官方的example.
cd $HADOOP_PREFIX
# run the mapreduce
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7..jar grep input output 'dfs[a-z.]+' # check the output
bin/hdfs dfs -cat output/*
输出内容:
bash-4.1# clear
bash-4.1# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0.jar grep input output 'dfs[a-z.]+'
18/06/11 07:35:38 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/06/11 07:35:39 INFO input.FileInputFormat: Total input paths to process : 31
18/06/11 07:35:39 INFO mapreduce.JobSubmitter: number of splits:31
18/06/11 07:35:40 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1528635021541_0007
18/06/11 07:35:40 INFO impl.YarnClientImpl: Submitted application application_1528635021541_0007
18/06/11 07:35:40 INFO mapreduce.Job: The url to track the job: http://e1bed6899d06:8088/proxy/application_1528635021541_0007/
18/06/11 07:35:40 INFO mapreduce.Job: Running job: job_1528635021541_0007
18/06/11 07:35:45 INFO mapreduce.Job: Job job_1528635021541_0007 running in uber mode : false
18/06/11 07:35:45 INFO mapreduce.Job: map 0% reduce 0%
18/06/11 07:36:02 INFO mapreduce.Job: map 10% reduce 0%
18/06/11 07:36:03 INFO mapreduce.Job: map 19% reduce 0%
18/06/11 07:36:19 INFO mapreduce.Job: map 35% reduce 0%
18/06/11 07:36:20 INFO mapreduce.Job: map 39% reduce 0%
18/06/11 07:36:33 INFO mapreduce.Job: map 42% reduce 0%
18/06/11 07:36:35 INFO mapreduce.Job: map 55% reduce 0%
18/06/11 07:36:36 INFO mapreduce.Job: map 55% reduce 15%
18/06/11 07:36:39 INFO mapreduce.Job: map 55% reduce 18%
18/06/11 07:36:45 INFO mapreduce.Job: map 58% reduce 18%
18/06/11 07:36:46 INFO mapreduce.Job: map 61% reduce 18%
18/06/11 07:36:47 INFO mapreduce.Job: map 65% reduce 18%
18/06/11 07:36:48 INFO mapreduce.Job: map 65% reduce 22%
18/06/11 07:36:49 INFO mapreduce.Job: map 71% reduce 22%
18/06/11 07:36:51 INFO mapreduce.Job: map 71% reduce 24%
18/06/11 07:36:57 INFO mapreduce.Job: map 74% reduce 24%
18/06/11 07:36:59 INFO mapreduce.Job: map 77% reduce 24%
18/06/11 07:37:00 INFO mapreduce.Job: map 77% reduce 26%
18/06/11 07:37:01 INFO mapreduce.Job: map 84% reduce 26%
18/06/11 07:37:03 INFO mapreduce.Job: map 87% reduce 28%
18/06/11 07:37:06 INFO mapreduce.Job: map 87% reduce 29%
18/06/11 07:37:08 INFO mapreduce.Job: map 90% reduce 29%
18/06/11 07:37:09 INFO mapreduce.Job: map 94% reduce 29%
18/06/11 07:37:11 INFO mapreduce.Job: map 100% reduce 29%
18/06/11 07:37:12 INFO mapreduce.Job: map 100% reduce 100%
18/06/11 07:37:12 INFO mapreduce.Job: Job job_1528635021541_0007 completed successfully
18/06/11 07:37:12 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=345
FILE: Number of bytes written=3697476
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=80529
HDFS: Number of bytes written=437
HDFS: Number of read operations=96
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=31
Launched reduce tasks=1
Data-local map tasks=31
Total time spent by all maps in occupied slots (ms)=400881
Total time spent by all reduces in occupied slots (ms)=52340
Total time spent by all map tasks (ms)=400881
Total time spent by all reduce tasks (ms)=52340
Total vcore-seconds taken by all map tasks=400881
Total vcore-seconds taken by all reduce tasks=52340
Total megabyte-seconds taken by all map tasks=410502144
Total megabyte-seconds taken by all reduce tasks=53596160
Map-Reduce Framework
Map input records=2060
Map output records=24
Map output bytes=590
Map output materialized bytes=525
Input split bytes=3812
Combine input records=24
Combine output records=13
Reduce input groups=11
Reduce shuffle bytes=525
Reduce input records=13
Reduce output records=11
Spilled Records=26
Shuffled Maps =31
Failed Shuffles=0
Merged Map outputs=31
GC time elapsed (ms)=2299
CPU time spent (ms)=11090
Physical memory (bytes) snapshot=8178929664
Virtual memory (bytes) snapshot=21830377472
Total committed heap usage (bytes)=6461849600
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=76717
File Output Format Counters
Bytes Written=437
18/06/11 07:37:12 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/06/11 07:37:12 INFO input.FileInputFormat: Total input paths to process : 1
18/06/11 07:37:12 INFO mapreduce.JobSubmitter: number of splits:1
18/06/11 07:37:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1528635021541_0008
18/06/11 07:37:12 INFO impl.YarnClientImpl: Submitted application application_1528635021541_0008
18/06/11 07:37:12 INFO mapreduce.Job: The url to track the job: http://e1bed6899d06:8088/proxy/application_1528635021541_0008/
18/06/11 07:37:12 INFO mapreduce.Job: Running job: job_1528635021541_0008
18/06/11 07:37:24 INFO mapreduce.Job: Job job_1528635021541_0008 running in uber mode : false
18/06/11 07:37:24 INFO mapreduce.Job: map 0% reduce 0%
18/06/11 07:37:29 INFO mapreduce.Job: map 100% reduce 0%
18/06/11 07:37:35 INFO mapreduce.Job: map 100% reduce 100%
18/06/11 07:37:35 INFO mapreduce.Job: Job job_1528635021541_0008 completed successfully
18/06/11 07:37:35 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=291
FILE: Number of bytes written=230541
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=569
HDFS: Number of bytes written=197
HDFS: Number of read operations=7
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3210
Total time spent by all reduces in occupied slots (ms)=3248
Total time spent by all map tasks (ms)=3210
Total time spent by all reduce tasks (ms)=3248
Total vcore-seconds taken by all map tasks=3210
Total vcore-seconds taken by all reduce tasks=3248
Total megabyte-seconds taken by all map tasks=3287040
Total megabyte-seconds taken by all reduce tasks=3325952
Map-Reduce Framework
Map input records=11
Map output records=11
Map output bytes=263
Map output materialized bytes=291
Input split bytes=132
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=291
Reduce input records=11
Reduce output records=11
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=55
CPU time spent (ms)=1090
Physical memory (bytes) snapshot=415494144
Virtual memory (bytes) snapshot=1373601792
Total committed heap usage (bytes)=354942976
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=437
File Output Format Counters
Bytes Written=197
可以看到利用了docker...安装hadoop就1行命令....就能成功运行官方example了.超级简单
运行自己写的demo
我自己尝试写了个demo.就是读取一个txt里的文字,然后统计它的字符数量
1.首先我往hdfs里创建1个txt:
hdfs的命令可以参考 https://blog.csdn.net/zhaojw_420/article/details/53161624
hdfs dfs -put in.txt /myinput/in.txt
2.写自己的mapper和reducer
代码参考 https://gitee.com/abcwt112/hadoopDemo
参考里面的MyFirstMapper和MyFirstReducer和MyFirstStarter
package demo; import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.Reducer; import java.io.IOException;
import java.util.Iterator; public class MyFirstReducer extends Reducer<IntWritable, IntWritable, IntWritable, IntWritable> {
@Override
protected void reduce(IntWritable key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int total = 0;
for (IntWritable value : values) {
total += value.get();
}
context.write(new IntWritable(1), new IntWritable(total));
} }
package demo; import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper; import java.io.IOException; public class MyFirstMapper extends Mapper<LongWritable, Text, IntWritable, IntWritable> {
@Override
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String line = value.toString();
context.write(new IntWritable(0), new IntWritable(line.length()));
}
}
package demo; import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import java.io.FileInputStream;
import java.io.IOException; public class MyFirstStarter {
public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
Job job = new Job();
job.setJarByClass(MyFirstStarter.class);
job.setJobName("============ My First Job =============="); FileInputFormat.addInputPath(job, new Path("/myinput/in.txt"));
FileOutputFormat.setOutputPath(job, new Path("/myout")); job.setMapperClass(MyFirstMapper.class);
job.setReducerClass(MyFirstReducer.class); job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(IntWritable.class); System.exit(job.waitForCompletion(true) ? 0: 1);
}
}
运行mvn package以后打成jar包丢掉linux的/dockerVolumes/hadoop2目录就可以了.因为在docker里挂载了目录,所以会自动丢到hadoop2这个容器里.
另外提一句...我mvn package打出来的jar里的MF文件没有指定main方法...导致各种找不到入口....在同事的帮助下了解到可以通过maven配置来解决:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>${mainClass}</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build> <properties>
<mainClass>demo.MyFirstStarter</mainClass>
</properties>
另外docker安装的hadoop里的jdk是1.7我的环境是1.8..所以我再pom里还额外指定了用1.7去编码..
3.在hadoop2这个容器里运行我自己写的demo.
在$HADOOP_PREFIX目录下运行bin/hadoop jar /dockerVolume/hadoopDemo-1.0-SNAPSHOT.jar
bash-4.1# bin/hadoop jar /dockerVolume/hadoopDemo-1.0-SNAPSHOT.jar
18/06/11 07:54:11 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/06/11 07:54:12 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
18/06/11 07:54:13 INFO input.FileInputFormat: Total input paths to process : 1
18/06/11 07:54:13 INFO mapreduce.JobSubmitter: number of splits:1
18/06/11 07:54:13 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1528635021541_0009
18/06/11 07:54:13 INFO impl.YarnClientImpl: Submitted application application_1528635021541_0009
18/06/11 07:54:13 INFO mapreduce.Job: The url to track the job: http://e1bed6899d06:8088/proxy/application_1528635021541_0009/
18/06/11 07:54:13 INFO mapreduce.Job: Running job: job_1528635021541_0009
18/06/11 07:54:20 INFO mapreduce.Job: Job job_1528635021541_0009 running in uber mode : false
18/06/11 07:54:20 INFO mapreduce.Job: map 0% reduce 0%
18/06/11 07:54:25 INFO mapreduce.Job: map 100% reduce 0%
18/06/11 07:54:31 INFO mapreduce.Job: map 100% reduce 100%
18/06/11 07:54:31 INFO mapreduce.Job: Job job_1528635021541_0009 completed successfully
18/06/11 07:54:31 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=1606
FILE: Number of bytes written=232725
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=6940
HDFS: Number of bytes written=7
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3059
Total time spent by all reduces in occupied slots (ms)=3265
Total time spent by all map tasks (ms)=3059
Total time spent by all reduce tasks (ms)=3265
Total vcore-seconds taken by all map tasks=3059
Total vcore-seconds taken by all reduce tasks=3265
Total megabyte-seconds taken by all map tasks=3132416
Total megabyte-seconds taken by all reduce tasks=3343360
Map-Reduce Framework
Map input records=160
Map output records=160
Map output bytes=1280
Map output materialized bytes=1606
Input split bytes=104
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=1606
Reduce input records=160
Reduce output records=1
Spilled Records=320
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=43
CPU time spent (ms)=1140
Physical memory (bytes) snapshot=434499584
Virtual memory (bytes) snapshot=1367728128
Total committed heap usage (bytes)=354942976
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=6836
File Output Format Counters
Bytes Written=7
运行成功!
查看输出结果
bash-4.1# bin/hdfs dfs -ls /myout
Found 2 items
-rw-r--r-- 1 root supergroup 0 2018-06-11 07:54 /myout/_SUCCESS
-rw-r--r-- 1 root supergroup 7 2018-06-11 07:54 /myout/part-r-00000
bash-4.1# bin/hdfs dfs -cat /myout/part-r-00000
1 6676
bash-4.1#
总共6676个字符..
6836 - 160个换行符 = 6676
成功运行自己写的demo!
hadoop学习记录1 初始hadoop的更多相关文章
- Hadoop学习总结之五：Hadoop的运行痕迹
Hadoop学习总结之五:Hadoop的运行痕迹 Hadoop 学习总结之一:HDFS简介 Hadoop学习总结之二:HDFS读写过程解析 Hadoop学习总结之三:Map-Reduce入门 Ha ...
- Hadoop学习记录
http://blog.csdn.net/m_star_jy_sy/article/details/26476907配置windows里eclipse连接hadoop集群 hadoop常见命令 启动H ...
- Hadoop学习记录（1）|伪分布安装
本文转载自向着梦想奋斗博客 Hadoop是什么? 适合大数据的分布式存储于计算平台 不适用小规模数据 作者:Doug Cutting 受Google三篇论文的启发 Hadoop核心项目 HDFS(Ha ...
- Hadoop学习记录（5）|集群搭建|节点动态添加删除
集群概念 计算机集群是一种计算机系统,通过一组松散继承的计算机软件或硬件连接连接起来高度紧密地协作完成计算工作. 集群系统中的单个计算机通常称为节点,通过局域网连接. 集群特点: 1.效率高,通过多态 ...
- Hadoop学习之Ubuntu12.04 Hadoop 环境搭建笔记
SSH无密码配置 Hadoop在Ubuntu12.04上搭建环境 报错及问题 SSH无密码配置 参考:Linux(Centos)配置OpenSSH无密码登陆 注意问题: Hadoop集成环境三台机器都 ...
- Hadoop学习1—浅谈hadoop
大数据这个词越来越热,本人一直想学习一下,正巧最近有时间了解一下.先从hadoop入手,在此记录学习中的点滴. 什么是hadoop? What Is Apache Hadoop? The Apache ...
- hadoop学习笔记--找到执行hadoop的入口
参与个hadoop项目,之前没搞过,赶紧学习: 照葫芦画瓢,得到代码是hdfs2local.sh脚本和LiaoNingFilter.jar包,迫不及待用jd-gui打开jar包,搜索到main(在MA ...
- hadoop学习通过虚拟机安装hadoop完全分布式集群
要想深入的学习hadoop数据分析技术,首要的任务是必须要将hadoop集群环境搭建起来,可以将hadoop简化地想象成一个小软件,通过在各个物理节点上安装这个小软件,然后将其运行起来,就是一个had ...
- hadoop学习记录（三）HBase基本概念
这一次开始学习HBase数据库. 我用的是VMWare + ubuntu16.04 +Hbase1.1.5 +hadoop2.6.0的组合. 经过亲自安装验证,版本间没有发生冲突,可以进行学习和开发. ...
随机推荐
- 批量归一化batch_normalization
为了解决在深度神经网络训练初期降低梯度消失/爆炸问题,Sergey loffe和Christian Szegedy提出了使用批量归一化的技术的方案,该技术包括在每一层激活函数之前在模型里加一个操作,简 ...
- 用eclipse打包可执行的jar（含第三方jar包）
在eclipse中的解决方式如下: 在工程目录下(与src同层)建立lib目录,将第三方Jar包放到这个目录里(copy,paste即可)[如果直接引用本地的jar,一旦换电脑就呵呵了...] 右击工 ...
- 《DSP using MATLAB》示例Example 8.22
代码: %% ------------------------------------------------------------------------ %% Output Info about ...
- 基于C#的UDP协议的同步实现
一.摘要 总结基于C#的UDP协议的同步通信. 二.实验平台 Visual Studio 2010 三.实验原理 UDP传输协议同TCP传输协议的区别可查阅相关文档,此处不再赘述. 四.实例 4.1 ...
- Google Chrome 调试JS简单教程[更新]
题外话,刚开始我写这篇内容只是将自己了解的一些知识放上来,不巧的是我分析了我的来访日志,很多朋友都有这个需求,为了大家没有白来,我决定充实下这篇文章.最近更新时间2014-02-14 chrome版本 ...
- adobe reader DC 字体设置
adobe reader DC 字体设置 一直使用adobe reader阅读pdf文档,系统提醒我升级一个reader助手, 升级之后: 感觉字体颜色变浅,笔画也变细了,整体有些模糊不清. goog ...
- poj 2187 Beauty Contest——旋转卡壳
题目:http://poj.org/problem?id=2187 学习材料:https://blog.csdn.net/wang_heng199/article/details/74477738 h ...
- dwr的A request has been denied as a potential CSRF attack.错误
虽然DWR是个很早就出现的Ajax框架,但一直都没去使用过,今天正好没事就看了一下并参照文档照做了个demo, 在其中碰到一个问题: 后台打印出错误信息:“严重: A request has been ...
- cowboy页面重定向的例子
创建工程 rebar-creator create-app testCowboy testCowboy_app.erl -module(testCowboy_app). -behaviour(appl ...
- Markdown初步使用
一.兼容 HTML Markdown 的理念是,能让文档更容易读.写和随意改.HTML 是一种发布的格式,Markdown 是一种书写的格式.就这样,Markdown 的格式语法只涵盖纯文本可以涵盖的 ...