Hadoop Yarn配置项 yarn.nodemanager.resource.local-dirs探讨
1. What is the recommended value for "yarn.nodemanager.resource.local-dirs"?
We only have one value (directory) configured for the above property, which has a size of 200GB.
Our hive jobs' map/reduce fill this folder up, and yarn places this node in the blocklist. Moving to tez engine and/or increasing the quota size may fix this, but we'd like to know the recommended value.
个解答,截止Sourygna Luangsay · 2015年10月28日 08:04
If you use the same partitions for yarn intermediate data than for the HDFS blocks, then you might also consider setting the fs.datanode.du.reserved property, which reserves some space on those partitions for non-hdfs use (such as intermediate yarn data).
One base recommendation I saw on my first Hadoop training long time ago was to dedicate 25% of the "data disks" for that kind of intermediate data.
I guess the optimal answer should consider the maximum amount of intermediate data you can get at the same time (when launching a job,
do you use all the data of HDFS as input data?) and dedicate the space for yarn.nodemanager.resource.local-dirs accordingly.
I would also recommend turning on the property mapreduce.map.output.compress in order to reduce the size of the intermediate data.
个解答,截止Jean-Philippe Player · 2015年10月27日 20:58
You would assign one folder to each of the datanode disks, closely mapping dfs.datanode.data.dir. On a 12 disk system you would have 12 yarn local-dir locations.
2.Though Dataflow can be used with an out of the box Hadoop installation , there are a couple of configuration properties which may improve DataFlow/Hadoop performance
Using the O/S file system (i.e. /tmp or /var/) can be problematic especially if any applications log a lot of information or require large local files. So we have two properties to overcome this bottleneck.
The first is yarn.nodemanager.local-dirs. This setting specifies the directories to use as base directories for the containers run within YARN.
For each application and container created in YARN, a set of directories will be created underneath these local directories. These are then cleaned up when the application completes.
Here’s the setting from the yarn-site.xml file on one of our clusters. Note we have eight data disks per node on these clusters and create a directory for YARN on each data filesystem.
- <property>
- <name>yarn.nodemanager.local-dirs</name>
- <value>
- /hadoop/hdfs/data1/hadoop/yarn/local,/hadoop/hdfs/data2/hadoop/yarn/local,/hadoop/hdfs/data3/hadoop/yarn/local,/hadoop/hdfs/data4/hadoop/yarn/local,/hadoop/hdfs/data5/hadoop/yarn/local,/hadoop/hdfs/data6/hadoop/yarn/local,/hadoop/hdfs/data7/hadoop/yarn/local,/hadoop/hdfs/data8/hadoop/yarn/local
- </value>
- <source>yarn-site.xml</source>
- </property>
The second is yarn.nodemanager.log-dirs. Much like the local-dirs property, this setting specifies where container log files should go on the local disk. YARN spreads the load around if you specify multiple directories.
And here’s a sample setting:
- <property>
- <name>yarn.nodemanager.log-dirs</name>
- <value>
- /hadoop/hdfs/data1/hadoop/yarn/log,/hadoop/hdfs/data2/hadoop/yarn/log,/hadoop/hdfs/data3/hadoop/yarn/log,/hadoop/hdfs/data4/hadoop/yarn/log,/hadoop/hdfs/data5/hadoop/yarn/log,/hadoop/hdfs/data6/hadoop/yarn/log,/hadoop/hdfs/data7/hadoop/yarn/log,/hadoop/hdfs/data8/hadoop/yarn/log
- </value>
- <source>yarn-site.xml</source>
- </property>
Another YARN property you want to validate is the yarn.nodemanager.resource.memory-mb. This setting specifies the amount of memory YARN is allowed to allocate per worker node.
YARN will only allocate this much memory in total to containers. So it’s important to set this to some value less than the physical memory per worker node.
HDP appears to automatically pick 75% of the physical memory for this setting as our machines have 16GB of RAM each.
Here’s an example:
- <property>
- <name>yarn.nodemanager.resource.memory-mb</name>
- <value></value>
- <source>yarn-site.xml</source>
- </property>
当然也可以考虑使用nfs挂载,相关资料如下
3.How can I change yarn.nodemanager.local-dirs to point to file:/// (high performance NFS mount point)
Hi,I'm trying to change the "yarn.nodemanager.local-dirs" to point to "file:///fast_nfs/yarn/local". This is indeed a high-performance NFS mount-point that all the nodes in my cluster have.
When I try to change it in Ambari I can't and the message "Must be a slash or drive at the start, and must not contain white spaces" is displayed.
If I manually change the /etc/hadoop/conf/yarn-site.xml in all the nodes, after restarting YARN the "file:///" is removed from that option.
I want to have all the shuffle happening in my high-performance NFS array instead of in HDFS.
How can I change this behaviour in HDP?
The culprit is "file:/// "you should get a was to create a mount point /fast_nfs/yarn/local, hence the message "Must be a slash or drive ........" like te list below
- /hadoop/yarn/local,/opt/hadoop/yarn/local,/usr/hadoop/yarn/local,/var/hadoop/yarn/local
Hope that helps
4. How to set yarn.nodemanager.local-dirs on M3 cluster to write to mapr fs
We are running a four node M3 cluster with one node running NFS. We are getting the the following error.
1/1 local-dirs are bad: /mapr/clustername/tmp/host_name on the nodes that does not have NFS running.
What is the best way to set this property in the yarn-site.xml to allow all nodes to use mapr fs /tmp as the default location and not the local file system /tmp
I believe the property "yarn.nodemanager.local-dirs" is meant to be a location on the local file system. It cannot be a location of the distributed file system (HDFS or MapR FS).
This property determines the location where the node manager maintains intermediate data (for example during the shuffle phase).
You can the find gory details here: http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
The default location as you mentioned is /tmp. If you want to improve performance, you could provide multiple directories on separate disks for better I/O throughput.
But, you should ascertain that this is a indeed bottleneck and if a separate disk is warranted for this purpose (or you are better of using it as a MapR data disk).
One other thing, the NFS mounted location (/mapr/clustername/tmp/host_name) is not a part of the distributed FS.
MapR makes it seamless to work between its distributed file system and the POSIX file system. But the files of the POSIX system are not stored in any containers/chunks/blocks, etc.
Since the path you specified is really a local directory on the node running NFS, you don't get an error message on that node . But on the other nodes, the system can't find a local directory by that name and hence it is complaining.
Hadoop Yarn配置项 yarn.nodemanager.resource.local-dirs探讨的更多相关文章
- hadoop集群配置方法---mapreduce应用:xml解析+wordcount详解---yarn配置项解析
注:以下链接均为近期hadoop集群搭建及mapreduce应用开发查找到的资料.使用hadoop2.6.0,其中hadoop集群配置过程下面的文章都有部分参考. hadoop集群配置方法: ---- ...
- Hadoop学习之YARN框架
转自:http://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop-yarn/,非常感谢分享! 对于业界的大数据存储及分布式处理系统来说,H ...
- Hadoop生态系统之Yarn
Apache YARN(Yet Another Resource Negotiator) 是Hadoop的集群资源管理系统.YARN被引入Hadoop2最初是为了改善MapReduce的实现,但它具有 ...
- hadoop备战:yarn框架的搭建(mapreduce2)
昨天没有写好了没有更新,今天一起更新,yarn框架也是刚搭建好的. 我这里把hadoop放在了我的个人用户hadoop下了,你也能够尝试把它放在/usr/local,考虑的问题就相对多点. 主要的软硬 ...
- hadoop备战:yarn框架的简单介绍(mapreduce2)
新 Hadoop Yarn 框架原理及运作机制 重构根本的思想是将 JobTracker 两个基本的功能分离成单独的组件,这两个功能是资源管理和任务调度 / 监控.新的资源管理器全局管理全部应用程序计 ...
- Hadoop核心组件之YARN
YARN概述 Yet Another Resource Negotiator:另外资源的协调者 通用的资源管理系统 为上层应用提供统一的资源管理和调度 操作系统级别的调度框架,可以让各种计算框架运行在 ...
- Hadoop学习笔记—Yarn
目录 一些基本知识 ResourceManager 的恢复 Resource Manager的HA YARN Node Labels YARN Node Attributes Web Applicat ...
- Hadoop 2.2 YARN分布式集群搭建配置流程
搭建环境准备:JDK1.6,SSH免密码通信 系统:CentOS 6.3 集群配置:NameNode和ResourceManager在一台服务器上,三个数据节点 搭建用户:YARN Hadoop2.2 ...
- Hadoop数据操作系统YARN全解析
“ Hadoop 2.0引入YARN,大大提高了集群的资源利用率并降低了集群管理成本.其在异构集群中是怎样应用的?Hulu又有哪些成功实践可以分享? 为了能够对集群中的资源进行统一管理和调度,Hado ...
随机推荐
- mysql 架构篇系列 4 复制架构一主一从搭建(半同步复制)
一.概述 在mysql 5.5之前,mysql 的复制是异步操作,主库和从库的数据之间存在一定的延时,这样存在一个隐患:当主库上写入一个事务并提交成功,而从库尚未得到主库推送的Binlog日志时,主库 ...
- 面试 Linux 运维工作至少需要知道哪些知识?
前言 我们已经发过不少 Linux 面试题,但是单独的面试题总感觉会过于零碎,没有体系化内容给人的帮助大. 知乎上有这样一个问题:一个新手面试 Linux 运维工作至少需要知道哪些知识?其中有一个答案 ...
- leetcode — binary-tree-level-order-traversal
import org.lep.leetcode.binarytreeinordertraversal.BinaryTreeInOrderTraversal; import java.util.Arra ...
- scrapy爬虫学习系列五:图片的抓取和下载
系列文章列表: scrapy爬虫学习系列一:scrapy爬虫环境的准备: http://www.cnblogs.com/zhaojiedi1992/p/zhaojiedi_python_00 ...
- 用户身份切换之初窥企业远程用户没root还有root权限
一直很困扰我,既然企业不让用root不能登录,那怎么操作文件呢? 原来...... su - 用来切换初始变量 $PATH $HOME等 sudo 用的时候会su到root需要root的密码,这 ...
- VS2015安装水晶报表
最近在做一个打印功能,本来是不想用水晶报表的.想直接用微软原生的报表rdlc完成,但是整了一个上午老是打印乱码,且网上找资料找也找不出. 无奈放弃,然后就想到用水晶报表了,以前用过水晶报表,不过一直都 ...
- GAN模型生成手写字
概述:在前期的文章中,我们用TensorFlow完成了对手写数字的识别,得到了94.09%的识别准确度,效果还算不错.在这篇文章中,笔者将带领大家用GAN模型,生成我们想要的手写数字. GAN简介 对 ...
- .Net语言 APP开发平台——Smobiler学习日志:如何设置页面的title
1.修改Mobile Form的TitleText的属性 输入需要显示标题,如图1: 2.修改Mobile Form的TitleStyle属性 其中包括Image属性(窗体图标).BackColor属 ...
- 【Parallel】.Net 并行执行程序的使用心得
一.摘要 官方介绍:提供对并行循环和区域的支持. 命名空间:using System.Threading.Tasks 三个静态方法:Parallel.Invoke,Parallel.For,Paral ...
- C# 操作Word 中的OLE——插入、编辑、读取 OLE
概述 OLE,Object Linking and Embedding,即对象连接与嵌入.我们在设计程序时,OLE可以用来创建复合文档,把文字.声音.图像.表格.应用程序等类型的信息组合在一起,在Wo ...