Hadoop入门学习笔记-第三天（Yarn高可用集群配置及计算案例）

什么是mapreduce

首先让我们来重温一下 hadoop 的四大组件：
HDFS：分布式存储系统
MapReduce：分布式计算系统
YARN： hadoop 的资源调度系统
Common： 以上三大组件的底层支撑组件，主要提供基础工具包和 RPC 框架等

Mapreduce 是一个分布式运算程序的编程框架，是用户开发“基于 hadoop 的数据分析 应用”的核心框架
Mapreduce 核心功能是将用户编写的业务逻辑代码和自带默认组件整合成一个完整的 分布式运算程序，并发运行在一个 hadoop 集群上

一、yarn单机版配置
1.修改mapred-site.xml 文件指定调度方式为yarn
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
2.修改yarn-site.xml 配置yarn服务器
<property>
<name>yarn.resourcemanager.hostname</name>
<value>nameNode10.com</value>
</property>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
3.在指定机器上面启动./start-yarn.sh

二、yarn ha高可用 集群配置
1.修改yarn-site.xml 配置yarn集群
<!-- 超时的周期 -->
<property>
<name>yarn.resourcemanager.connect.retry-interval.ms</name>
<value>2000</value>
</property>
<!-- 打开高可用 -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<!-- 启动故障自动恢复 -->
<property>
<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
<value>true</value>
</property>

<property>
<name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
<value>true</value>
</property>
<!-- 给yarn cluster 取个名字yarn-rm-cluster -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yarn-rm-cluster</value>
</property>
<!-- 给ResourceManager 取个名字 rm1,rm2 -->
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!-- 配置ResourceManager rm1 hostname -->
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>nameNode10.com</value>
</property>
<!-- 配置ResourceManager rm2 hostname -->
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>dateNode30.com</value>
</property>
<!-- 启用resourcemanager 自动恢复 -->
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<!-- 配置Zookeeper地址 -->
<property>
<name>yarn.resourcemanager.zk.state-store.address</name>
<value>nameNode10.com:2181,dataNode20.com:2181,nameNode40.com:2181</value>
</property>
<!-- 配置Zookeeper地址 -->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>nameNode10.com:2181,dataNode20.com:2181,nameNode40.com:2181</value>
</property>

<!-- rm1端口号 -->
<property>
<name>yarn.resourcemanager.address.rm1</name>
<value>nameNode10.com:8032</value>
</property>
<!-- rm1调度器的端口号 -->
<property>
<name>yarn.resourcemanager.scheduler.address.rm1</name>
<value>nameNode10.com:8034</value>
</property>
<!-- rm1 webapp端口号 -->
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>nameNode10.com:8088</value>
</property>

<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>nodeNode30.com:8032</value>
</property>
<!-- rm2调度器的端口号 -->
<property>
<name>yarn.resourcemanager.scheduler.address.rm2</name>
<value>nodeNode30.com:8034</value>
</property>
<!-- rm2 webapp端口号 -->
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>nodeNode30.com:8088</value>
</property>

<!-- 执行MapReduce需要配置的shuffle过程 低版本，配置成mapreduces.shuffle -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

2.输入hadoop classpath 会出现以下路径(这步不配置，执行自带案例demo时会报错)
/software/hadoop-3.2.0/etc/hadoop:/software/hadoop-3.2.0/share/hadoop/common/lib/*:/software/hadoop-3.2.0/share/hadoop/common/*:/software/hadoop-3.2.0/share/hadoop/hdfs:/software/hadoop-3.2.0/share/hadoop/hdfs/lib/*:/software/hadoop-3.2.0/share/hadoop/hdfs/*:/software/hadoop-3.2.0/share/hadoop/mapreduce/lib/*:/software/hadoop-3.2.0/share/hadoop/mapreduce/*:/software/hadoop-3.2.0/share/hadoop/yarn:/software/hadoop-3.2.0/share/hadoop/yarn/lib/*:/software/hadoop-3.2.0/share/hadoop/yarn/*
将上面路径拷贝到 yarn-site.xml文件中
<property>
<name>yarn.application.classpath</name>
<value>/software/hadoop-3.2.0/etc/hadoop:/software/hadoop-3.2.0/share/hadoop/common/lib/*:/software/hadoop-3.2.0/share/hadoop/common/*:/software/hadoop-3.2.0/share/hadoop/hdfs:/software/hadoop-3.2.0/share/hadoop/hdfs/lib/*:/software/hadoop-3.2.0/share/hadoop/hdfs/*:/software/hadoop-3.2.0/share/hadoop/mapreduce/lib/*:/software/hadoop-3.2.0/share/hadoop/mapreduce/*:/software/hadoop-3.2.0/share/hadoop/yarn:/software/hadoop-3.2.0/share/hadoop/yarn/lib/*:/software/hadoop-3.2.0/share/hadoop/yarn/*</value>
</property>

3.单步骤启动
yarn-daemon.sh start resourcemanager
yarn-daemon.sh start nodemanager
4.集成启动 yarn-start.sh

5.访问控制台查看(访问备用节点会自动跳转到主节点控制台)
http://datanode30.com:8088/cluster/
http://namenode10.com:8088/cluster/
主备状态查看
yarn rmadmin -getServiceState rm1

6.执行例子
1.计算圆周率 /software/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar pi 10 10
2.统计文中每个单词个数
hadoop fs -mkdir /input
hadoop fs -put -f ./file/file1.txt /input
hadoop jar /software/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount /input /output
hadoop fs -ls /output
hadoop fs -cat /output/part-r-00000