Hadoop 2.6.0分布式部署參考手冊
Hadoop 2.6.0分布式部署參考手冊
关于本參考手冊的word文档。能够到例如以下地址下载：http://download.csdn.net/detail/u012875880/8291493
1.环境说明
1.1安装环境说明
本列中。操作系统为Centos 7.0。JDK版本号为Oracle HotSpot 1.7，Hadoop版本号为Apache Hadoop 2.6.0。操作用户为hadoop。
2.2 Hadoop集群环境说明：
集群各节点信息參考例如以下：
|
主机名 |
IP地址 |
角色 |
|
ResourceManager |
172.15.0.2 |
ResourceManager & MR JobHistory Server |
|
NameNode |
172.15.0.3 |
NameNode |
|
SecondaryNameNode |
172.15.0.4 |
SecondaryNameNode |
|
DataNode01 |
172.15.0.5 |
DataNode & NodeManager |
|
DataNode02 |
172.15.0.6 |
DataNode & NodeManager |
|
DataNode03 |
172.15.0.7 |
DataNode & NodeManager |
|
DataNode04 |
172.15.0.8 |
DataNode & NodeManager |
|
DataNode05 |
172.15.0.9 |
DataNode & NodeManager |
注：上述表中用”&”连接多个角色，如主机”ResourceManager”有两个角色。分别为ResourceManager和MR JobHistory Server。
2.基础环境安装及配置
2.1 加入hadoop用户
useradd hadoop
用户“hadoop”即为Hadoop集群的安装和使用用户。
2.2 JDK 1.7安装
Centos 7自带的JDK版本号为 OpenJDK 1.7，本例中须要将其更换为Oracle HotSpot 1.7版。本例中採用解压二进制包方式安装。安装文件夹为/opt/。
① 查看当前JDK rpm包
rpm -qa | grep jdk
java-1.7.0-openjdk-1.7.0.51-2.4.5.5.el7.x86_64
java-1.7.0-openjdk-headless-1.7.0.51-2.4.5.5.el7.x86_64
② 删除自带JDK
rpm -e --nodeps java-1.7.0-openjdk-1.7.0.51-2.4.5.5.el7.x86_64
rpm -e --nodeps java-1.7.0-openjdk-headless-1.7.0.51-2.4.5.5.el7.x86_64
③ 安装指定JDK
进入安装包所在文件夹并解压
④ 配置环境变量
编辑~/.bashrc或者/etc/profile。加入例如以下内容：
#JAVA
export JAVA_HOME=/opt/jdk1.7
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=$JAVA_HOME/lib
export CLASSPATH=$CLASSPATH:$JAVA_HOME/jre/lib
2.3 SSH无password登陆配置
① 须要设置如上表格所看到的8台主机间的SSH无password登陆。
② 进入hadoop用户的根文件夹下并通过命令ssh-keygen -t rsa 生成秘钥对
③ 创建公钥认证文件authorized_keys并将生成的~/.ssh文件夹下的id_rsa.pub文件 的内容输出至该文件：
more id_rsa.pub > auhorized_keys
④ 分别改变~/.ssh文件夹和authorized_keys文件的权限：
chmod 700 ~/.ssh;chmod 600 ~/.ssh/authorized_keys
⑤ 每一个节点主机都反复以上步骤，并将各自的~/.ssh/id_rsa.pub文件的公钥拷贝至其 他主机。
对于以上操作，也能够通过一句命令搞定：
rm -rf ~/.ssh;ssh-keygen -t rsa;chmod 700 ~/.ssh;more ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys;chmod 600 ~/.ssh/authorized_keys;
注：在centos 6中能够用dsa方式：ssh-keygen -t dsa命令来设置无password登陆，在centos 7中仅仅能用rsa方式。否则仅仅能ssh无password登陆本机。无能登陆它机。
2.4 改动hosts映射文件
分别编辑各节点上的/etc/hosts文件，加入例如以下内容：
172.15.0.2 ResourceManager
172.15.0.3 NameNode
172.15.0.4 SecondaryNameNode
172.15.0.5 DataNode01
172.15.0.6 DataNode02
172.15.0.7 DataNode03
172.15.0.8 DataNode04
172.15.0.9 DataNode05
172.15.0.5 NodeManager01
172.15.0.6 NodeManager02
172.15.0.7 NodeManager03
172.15.0.8 NodeManager04
172.15.0.9 NodeManager05
3.Hadoop安装及配置
3.1 通用部分安装及配置
下面操作内容为通用操作部分，及在每一个节点上的内容一样。
分别在每一个节点上反复例如以下操作：
① 将hadoop安装包（hadoop-2.6.0.tar）拷贝至/opt文件夹下，并解压：
tar -xvf hadoop-2.6.0.tar
解压后的hadoop-2.6.0文件夹(/opt/hadoop-2.6.0)即为hadoop的安装根文件夹。
② 更改hadoop安装文件夹hadoop-2.6.0的全部者为hadoop用户：
chown -R hadoop.hadoop /opt/hadoop-2.6.0
③ 加入环境变量：
#hadoop
export HADOOP_HOME=/opt/hadoop-2.6.0
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
3.2 各节点配置
分别将例如以下配置文件解压并分发至每一个节点的Hadoop“$HADOOP_HOME/etc/hadoop”文件夹中。如提示是否覆盖文件。确认就可以。
Hadoop配置文件：http://download.csdn.net/detail/u012875880/8291517
注：关于各节点的配置參数设置，请參考后面的“附录1”或“附录2”
4.格式化/启动集群
4.1 格式化集群HDFS文件系统
安装完成后。需登陆NameNode节点或任一DataNode节点运行hdfs namenode -format格式化集群HDFS文件系统；
注：假设非第一次格式化HDFS文件系统。则须要在进行格式化操作前分别将NameNode的dfs.namenode.name.dir和各个DataNode节点的dfs.datanode.data.dir文件夹(在本例中为/home/hadoop/hadoopdata)下的全部内容清空。
4.2启动Hadoop集群
分别登陆例如以下主机并运行对应命令：
① 登陆ResourceManger运行start-yarn.sh命令启动集群资源管理系统yarn
② 登陆NameNode运行start-dfs.sh命令启动集群HDFS文件系统
③ 分别登陆SecondaryNameNode、DataNode01、DataNode02、DataNode03、DataNode04 节点执行jps命令。查看每一个节点是否有例如以下Java进程执行：
ResourceManger节点执行的进程：ResouceNamager
NameNode节点执行的进程：NameNode
SecondaryNameNode节点执行的进程：SecondaryNameNode
各个DataNode节点执行的进程：DataNode & NodeManager
假设以上操作正常则说明Hadoop集群已经正常启动。
附录1 关键配置内容參考
1 core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://NameNode:9000</value>
<description>NameNode URI</description>
</property>
</configuration>
- 属性”fs.defaultFS“表示NameNode节点地址，由”hdfs://主机名(或ip)：port号”组成。
2 hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///home/jack/hadoopdata/hdfs/datanode</value>
</property
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>SecondaryNameNode:50090</value>
</property>
</configuration>
- 属性“dfs.namenode.name.dir”表示NameNode存储命名空间和操作日志相关的元数据信息的本地文件系统文件夹。该项默认本地路径为”/tmp/hadoop-{username}/dfs/name”；
- 属性”dfs.datanode.data.dir“表示DataNode节点存储HDFS文件的本地文件系统文件夹。由”file://本地文件夹”组成,该项默认本地路径为”/tmp/hadoop-{username}/dfs/data”。
- 属性“dfs.namenode.secondary.http-address”表示SecondNameNode主机及port号（假设无需额外指定SecondNameNode角色，能够不进行此项配置）；
3 mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>Execution framework set to Hadoop YARN.</description>
</property>
</configuration>
- 属性”mapreduce.framework.name“表示执行mapreduce任务所使用的执行框架，默觉得local，须要将其改为”yarn”
4 yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>ResourceManager</value>
<description>ResourceManager host</description>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>Shuffle service that needs to be set for Map Reduce applications.</description>
</property>
</configuration>
- 属性”yarn.resourcemanager.hostname”用来指定ResourceManager主机地址；
- 属性”yarn.nodemanager.aux-service“表示MR applicatons所使用的shuffle工具类
5 hadoop-env.sh
JAVA_HOME表示当前的Java安装文件夹
export JAVA_HOME=/opt/jdk-1.7
6 slaves
集群中的master节点(NameNode、ResourceManager)须要配置其所拥有的slaver节点，当中：
NameNode节点的slaves内容为：
DataNode01
DataNode02
DataNode03
DataNode04
DataNode05
ResourceManager节点的slaves内容为：
NodeManager01
NodeManager02
NodeManager03
NodeManager04
NodeManager05
附录2 具体配置内容參考
注：下面的红色字体部分的配置參数为必须配置的部分，其它配置皆为默认配置。
1 core-site.xml
<configuration>
<!--Configurations for NameNode(SecondaryNameNode)、DataNode、NodeManager:-->
<property>
<name>fs.defaultFS</name>
<value>hdfs://NameNode:9000</value>
<description>NameNode URI</description>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
<description>Size of read/write buffer used in SequenceFiles,The default value is 131072</description>
</property>
</configuration>
l 属性”fs.defaultFS“表示NameNode节点地址，由”hdfs://主机名(或ip)：port号”组成。
2 hdfs-site.xml
<configuration>
<!--Configurations for NameNode:-->
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>SecondaryNameNode:50090</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property
<property>
<name>dfs.blocksize</name>
<value>268435456</value>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>100</value>
</property>
<!--Configurations for DataNode:-->
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>
</configuration>
- l 属性“dfs.namenode.name.dir”表示NameNode存储命名空间和操作日志相关的元数据信息的本地文件系统文件夹。该项默认本地路径为”/tmp/hadoop-{username}/dfs/name”。
- l 属性”dfs.datanode.data.dir“表示DataNode节点存储HDFS文件的本地文件系统文件夹，由”file://本地文件夹”组成,该项默认本地路径为”/tmp/hadoop-{username}/dfs/data”。
- l 属性“dfs.namenode.secondary.http-address”表示SecondNameNode主机及port号（假设无需额外指定SecondNameNode角色，能够不进行此项配置）；
3 mapred-site.xml
<configuration>
<!--Configurations for MapReduce Applications:-->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>Execution framework set to Hadoop YARN.</description>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>1024</value>
<description>Larger resource limit for maps.</description>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>Xmx1024M</value>
<description>Larger heap-size for child jvms of maps.</description>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>1024</value>
<description>Larger resource limit for reduces.</description>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>Xmx2560M</value>
<description></description>
</property>
<property>
<name>mapreduce.task.io.sort.mb</name>
<value>512</value>
<description></description>
</property>
<property>
<name>mapreduce.task.io.sort.factor</name>
<value>10</value>
<description>More streams merged at once while sorting files.</description>
</property>
<property>
<name>mapreduce.reduce.shuffle.parallelcopies</name>
<value>5</value>
<description>Higher number of parallel copies run by reduces to fetch outputs from very large number of maps.</description>
</property>
<!--Configurations for MapReduce JobHistory Server:-->
<property>
<name>mapreduce.jobhistory.address</name>
<value>ResourceManager:10020</value>
<description>MapReduce JobHistory Server host:port Default port is 10020</description>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>ResourceManager:19888</value>
<description>MapReduce JobHistory Server Web UI host:port Default port is 19888</description>
</property>
<property>
<name>mapreduce.jobhistory.intermediate-done-dir</name>
<value>/mr-history/tmp</value>
<description>Directory where history files are written by MapReduce jobs. Defalut is "/mr-history/tmp"</description>
</property>
<property>
<name>mapreduce.jobhistory.done-dir</name>
<value>/mr-history/done</value>
<description>Directory where history files are managed by the MR JobHistory Server.Default value is "/mr-history/done"</description>
</property>
</configuration>
- l 属性”mapreduce.framework.name“表示执行mapreduce任务所使用的执行框架。默觉得local。须要将其改为”yarn”
4 yarn-site.xml
<configuration>
<!--Configurations for ResourceManager and NodeManager:-->
<property>
<name>yarn.acl.enable</name>
<value>false</value>
<description>Enable ACLs?
Defaults to false. The value of the optional is "true" or "false"</description>
</property>
<property>
<name>yarn.admin.acl</name>
<value>*</value>
<description>ACL to set admins on the cluster. ACLs are of for comma-separated-usersspacecomma-separated-groups. Defaults to special value of * which means anyone. Special value of just space means no one has access</description>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>false</value>
<description>Configuration to enable or disable log aggregation</description>
</property>
<!--Congrations for ResourceManager:-->
<property>
<name>yarn.resourcemanager.address</name>
<value>ResourceManager:8032</value>
<description>ResourceManager host:port for clients to submit jobs.NOTES:host:port If set, overrides the hostname set in yarn.resourcemanager.hostname.</description>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>ResourceManager:8030</value>
<description>ResourceManager host:port for ApplicationMasters to talk to Scheduler to obtain resources.NOTES:host:port If set, overrides the hostname set in yarn.resourcemanager.hostname</description>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>ResourceManager:8031</value>
<description>ResourceManager host:port for NodeManagers.NOTES:host:port If set, overrides the hostname set in yarn.resourcemanager.hostname</description>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>ResourceManager:8033</value>
<description>ResourceManager host:port for administrative commands.NOTES:host:port If set, overrides the hostname set in yarn.resourcemanager.hostname.</description>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>ResourceManager:8088</value>
<description>ResourceManager web-ui host:port. NOTES:host:port If set, overrides the hostname set in yarn.resourcemanager.hostname</description>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>ResourceManager</value>
<description>ResourceManager host</description>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
<description>ResourceManager Scheduler class CapacityScheduler (recommended), FairScheduler (also recommended), or FifoScheduler.The default value is "org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler".
</description>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
<description>Minimum limit of memory to allocate to each container request at the Resource Manager.NOTES:In MBs</description>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>8192</value>
<description>Maximum limit of memory to allocate to each container request at the Resource Manager.NOTES:In MBs</description>
</property>
<!--Congrations for History Server:-->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>-1</value>
<description>How long to keep aggregation logs before deleting them. -1 disables. Be careful, set this too small and you will spam the name node.</description>
</property>
<property>
<name>yarn.log-aggregation.retain-check-interval-seconds</name>
<value>-1</value>
<description>Time between checks for aggregated log retention. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful, set this too small and you will spam the name node.</description>
</property>
<!--Configurations for Configurations for NodeManager:-->
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>8192</value>
<description>Resource i.e. available physical memory, in MB, for given NodeManager.
The default value is 8192.
NOTES:Defines total available resources on the NodeManager to be made available to running containers
</description>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>2.1</value>
<description>Maximum ratio by which virtual memory usage of tasks may exceed physical memory.
The default value is 2.1
NOTES:The virtual memory usage of each task may exceed its physical memory limit by this ratio. The total amount of virtual memory used by tasks on the NodeManager may exceed its physical memory usage by this ratio.
</description>
</property>
<property>
<name>yarn.nodemanager.local-dir</name>
<value>${hadoop.tmp.dir}/nm-local-dir</value>
<description>Comma-separated list of paths on the local filesystem where intermediate data is written.
The default value is "${hadoop.tmp.dir}/nm-local-dir"
NOTES:Multiple paths help spread disk i/o.
</description>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>${yarn.log.dir}/userlogs</value>
<description>Comma-separated list of paths on the local filesystem where logs are written
The default value is "${yarn.log.dir}/userlogs"
NOTES:Multiple paths help spread disk i/o.
</description>
</property>
<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>10800</value>
<description>Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled.
The default value is "10800"
</description>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/logs</value>
<description>HDFS directory where the application logs are moved on application completion. Need to set appropriate permissions. Only applicable if log-aggregation is enabled.
The default value is "/logs" or "/tmp/logs"
</description>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir-suffix</name>
<value>logs</value>
<description>Suffix appended to the remote log dir. Logs will be aggregated to ${yarn.nodemanager.remote-app-log-dir}/${user}/${thisParam} Only applicable if log-aggregation is enabled.</description>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>Shuffle service that needs to be set for Map Reduce applications.</description>
</property>
</configuration>
- 属性”yarn.resourcemanager.hostname”用来指定ResourceManager主机地址；
- l 属性”yarn.nodemanager.aux-service“表示MR applicatons所使用的shuffle工具类
5 hadoop-env.sh
JAVA_HOME表示当前的Java安装文件夹
export JAVA_HOME=/opt/jdk-1.7
6 slaves
集群中的master节点(NameNode、ResourceManager)须要配置其所拥有的slaver节点，当中：
NameNode节点的slaves内容为：
DataNode01
DataNode02
DataNode03
DataNode04
DataNode05
ResourceManager节点的slaves内容为：
NodeManager01
NodeManager02
NodeManager03
NodeManager04
NodeManager05
附录3 具体參数配置參考
Configuring the Hadoop Daemons in Non-Secure Mode
This section deals with important parameters to be specified in the given configuration files:
· conf/core-site.xml
|
Parameter |
Value |
Notes |
|
fs.defaultFS |
NameNode URI |
hdfs://host:port/ |
|
io.file.buffer.size |
131072 |
Size of read/write buffer used in SequenceFiles. |
· conf/hdfs-site.xml
o Configurations for NameNode:
|
Parameter |
Value |
Notes |
|
dfs.namenode.name.dir |
Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently. |
If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. |
|
dfs.namenode.hosts /dfs.namenode.hosts.exclude |
List of permitted/excluded DataNodes. |
If necessary, use these files to control the list of allowable datanodes. |
|
dfs.blocksize |
268435456 |
HDFS blocksize of 256MB for large file-systems. |
|
dfs.namenode.handler.count |
100 |
More NameNode server threads to handle RPCs from large number of DataNodes. |
o Configurations for DataNode:
|
Parameter |
Value |
Notes |
|
dfs.datanode.data.dir |
Comma separated list of paths on the local filesystem of a DataNode where it should store its blocks. |
If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. |
· conf/yarn-site.xml
o Configurations for ResourceManager and NodeManager:
|
Parameter |
Value |
Notes |
|
yarn.acl.enable |
true / false |
Enable ACLs? Defaults to false. |
|
yarn.admin.acl |
Admin ACL |
ACL to set admins on the cluster. ACLs are of for comma-separated-usersspacecomma-separated-groups. Defaults to special value of * which means anyone. Special value of just space means no one has access. |
|
yarn.log-aggregation-enable |
false |
Configuration to enable or disable log aggregation |
o Configurations for ResourceManager:
|
Parameter |
Value |
Notes |
|
yarn.resourcemanager.address |
ResourceManager host:port for clients to submit jobs. |
host:port |
|
yarn.resourcemanager.scheduler.address |
ResourceManager host:port for ApplicationMasters to talk to Scheduler to obtain resources. |
host:port |
|
yarn.resourcemanager.resource-tracker.address |
ResourceManager host:port for NodeManagers. |
host:port |
|
yarn.resourcemanager.admin.address |
ResourceManager host:port for administrative commands. |
host:port |
|
yarn.resourcemanager.webapp.address |
ResourceManager web-ui host:port. |
host:port |
|
yarn.resourcemanager.hostname |
ResourceManager host. |
host |
|
yarn.resourcemanager.scheduler.class |
ResourceManager Scheduler class. |
CapacityScheduler (recommended), FairScheduler (also recommended), or FifoScheduler |
|
yarn.scheduler.minimum-allocation-mb |
Minimum limit of memory to allocate to each container request at theResource Manager. |
In MBs |
|
yarn.scheduler.maximum-allocation-mb |
Maximum limit of memory to allocate to each container request at theResource Manager. |
In MBs |
|
yarn.resourcemanager.nodes.include-path /yarn.resourcemanager.nodes.exclude-path |
List of permitted/excluded NodeManagers. |
If necessary, use these files to control the list of allowable NodeManagers. |
o Configurations for NodeManager:
|
Parameter |
Value |
Notes |
|
yarn.nodemanager.resource.memory-mb |
Resource i.e. available physical memory, in MB, for givenNodeManager |
Defines total available resources on the NodeManager to be made available to running containers |
|
yarn.nodemanager.vmem-pmem-ratio |
Maximum ratio by which virtual memory usage of tasks may exceed physical memory |
The virtual memory usage of each task may exceed its physical memory limit by this ratio. The total amount of virtual memory used by tasks on the NodeManager may exceed its physical memory usage by this ratio. |
|
yarn.nodemanager.local-dirs |
Comma-separated list of paths on the local filesystem where intermediate data is written. |
Multiple paths help spread disk i/o. |
|
yarn.nodemanager.log-dirs |
Comma-separated list of paths on the local filesystem where logs are written. |
Multiple paths help spread disk i/o. |
|
yarn.nodemanager.log.retain-seconds |
10800 |
Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled. |
|
yarn.nodemanager.remote-app-log-dir |
/logs |
HDFS directory where the application logs are moved on application completion. Need to set appropriate permissions. Only applicable if log-aggregation is enabled. |
|
yarn.nodemanager.remote-app-log-dir-suffix |
logs |
Suffix appended to the remote log dir. Logs will be aggregated to ${yarn.nodemanager.remote-app-log-dir}/${user}/${thisParam} Only applicable if log-aggregation is enabled. |
|
yarn.nodemanager.aux-services |
mapreduce_shuffle |
Shuffle service that needs to be set for Map Reduce applications. |
o Configurations for History Server (Needs to be moved elsewhere):
|
Parameter |
Value |
Notes |
|
yarn.log-aggregation.retain-seconds |
-1 |
How long to keep aggregation logs before deleting them. -1 disables. Be careful, set this too small and you will spam the name node. |
|
yarn.log-aggregation.retain-check-interval-seconds |
-1 |
Time between checks for aggregated log retention. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful, set this too small and you will spam the name node. |
· conf/mapred-site.xml
o Configurations for MapReduce Applications:
|
Parameter |
Value |
Notes |
|
mapreduce.framework.name |
yarn |
Execution framework set to Hadoop YARN. |
|
mapreduce.map.memory.mb |
1536 |
Larger resource limit for maps. |
|
mapreduce.map.java.opts |
-Xmx1024M |
Larger heap-size for child jvms of maps. |
|
mapreduce.reduce.memory.mb |
3072 |
Larger resource limit for reduces. |
|
mapreduce.reduce.java.opts |
-Xmx2560M |
Larger heap-size for child jvms of reduces. |
|
mapreduce.task.io.sort.mb |
512 |
Higher memory-limit while sorting data for efficiency. |
|
mapreduce.task.io.sort.factor |
100 |
More streams merged at once while sorting files. |
|
mapreduce.reduce.shuffle.parallelcopies |
50 |
Higher number of parallel copies run by reduces to fetch outputs from very large number of maps. |
o Configurations for MapReduce JobHistory Server:
|
Parameter |
Value |
Notes |
|
mapreduce.jobhistory.address |
MapReduce JobHistory Server host:port |
Default port is 10020. |
|
mapreduce.jobhistory.webapp.address |
MapReduce JobHistory Server Web UI host:port |
Default port is 19888. |
|
mapreduce.jobhistory.intermediate-done-dir |
/mr-history/tmp |
Directory where history files are written by MapReduce jobs. |
Hadoop 2.6.0分布式部署參考手冊的更多相关文章
- ANTLR4权威參考手冊（一）
写在前面的话: 此文档是对伟大的Terence Parr的著作<the definitive antlr4 reference>的翻译本.致敬!欢迎转载,请注明原地址,请尊重劳动成果.翻译 ...
- 6. GC 调优(工具篇) - GC參考手冊
进行GC性能调优时, 须要明白了解, 当前的GC行为对系统和用户有多大的影响. 有多种监控GC的工具和方法, 本章将逐一介绍经常使用的工具. 您应该已经阅读了前面的章节: 垃圾收集简单介绍 - GC參 ...
- HTML5 界面元素 Canvas 參考手冊
HTML5 界面元素 Canvas 參考手冊 太阳火神的漂亮人生 (http://blog.csdn.net/opengl_es) 本文遵循"署名-非商业用途-保持一致"创作公用协 ...
- 刚開始学习的人非常有用之chm结尾的參考手冊打开后无法正常显示
从网上下载了struts2的參考手冊.chm(本文适用全部已.chm结尾的文件)不能正常打开使用. 如图: watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQv/ ...
- bash參考手冊之五（shell变量）续三
LINENO 当前在运行的脚本或者shell函数的行号. LINES 命令select用来确定打印选择列表的列宽.收到SIGWINCH后,自己主动设置. MACHTYPE 是一个字符串,描写叙述了正在 ...
- bash參考手冊之六（Bash特性）
6 Bash 特性 这部分描写叙述Bash独有的特性. * 调用Bash : Bash能够接受的命令行选项. * Bash启动文件 : Bash何时及怎样运行脚本. * 交互Shell : 什么 ...
- MySQL中文參考手冊
非常好的中文手冊: 链接:http://www.sdau.edu.cn/support/mysq_doc/manual_toc.html
- Hadoop生态圈-zookeeper完全分布式部署
Hadoop生态圈-zookeeper完全分布式部署 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 本篇博客部署是建立在Hadoop高可用基础之上的,关于Hadoop高可用部署请参 ...
- Hadoop生态圈-phoenix完全分布式部署以及常用命令介绍
Hadoop生态圈-phoenix完全分布式部署 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. phoenix只是一个插件,我们可以用hive给hbase套上一个JDBC壳,但是你 ...
随机推荐
- java该HashTable,HashMap和HashSet
同一时候我们也对HashSet和HashMap的核心方法hashcode进行了具体解释,见<探索equals()和hashCode()方法>. 万事俱备,那么以下我们就对基于hash算法的 ...
- Swing界面刷新问题(转)
在Java Swing编程中,往往会遇到需要动态刷新界面的时候,例如动态刷新JLabel的文本,JTextField里的文本等等.但是往往却没有达到我们预期的效果,我相信很多朋友都遇到过本文将要说的这 ...
- WPF对于xml的简单操作（下下）插入节点并排序
正如T所说,下下,这个方法不堪入目,￣□￣|| 贴上再说 //先搞个struct声明 private struct datastruct { public string x; public strin ...
- ExcelHelper Excel，Export，Import
using System; using System.Collections.Generic; using System.Data; using System.Data.Odbc; using Sys ...
- C++ 操作 MySQL
使用VS2008如IDE, 准备操作: 1. 项目属性 C++ 其他包括文件夹 为 mysql 安装文件夹的include 如:"C:\Program Files (x86)\MySQL\ ...
- [TroubleShooting]'trn\bak' is incorrectly formed. SQL Server cannot process this media family.
SQL Server online consultants came across an interesting scenario where one of our client was una ...
- mysql安装前的系统准备工作（转）
一.系统环境总结:
- SQL在declare声明变量
在sql添加的声明变量. declare @local_variable data_type 你需要指定一个变量声明的类型, 能够使用set和select对变量进行赋值, 在sql语句中就能够使用@l ...
- Windows Server 2012 R2在桌面上显示计算机/网络图标
原文 Windows Server 2012 R2在桌面上显示计算机/网络图标 从Windows2012开始,微软取消了服务器桌面个性化选项,如何重新调出配置界面,可以使用微软命令调出.具体方法如下: ...
- 安卓反汇编工具arm-eabi-objdump
安卓反汇编工具 在Arm平台系统自带的反编译工具在android/prebuild/linux-/toolchail/arm-abil-/bin目录下的arm_eabi-objdump进行反汇编 ar ...