Hadoop Yarn配置项 yarn.nodemanager.resource.local-dirs探讨
1. What is the recommended value for "yarn.nodemanager.resource.local-dirs"?
We only have one value (directory) configured for the above property, which has a size of 200GB.
Our hive jobs' map/reduce fill this folder up, and yarn places this node in the blocklist. Moving to tez engine and/or increasing the quota size may fix this, but we'd like to know the recommended value.
个解答,截止Sourygna Luangsay · 2015年10月28日 08:04
If you use the same partitions for yarn intermediate data than for the HDFS blocks, then you might also consider setting the fs.datanode.du.reserved property, which reserves some space on those partitions for non-hdfs use (such as intermediate yarn data).
One base recommendation I saw on my first Hadoop training long time ago was to dedicate 25% of the "data disks" for that kind of intermediate data.
I guess the optimal answer should consider the maximum amount of intermediate data you can get at the same time (when launching a job,
do you use all the data of HDFS as input data?) and dedicate the space for yarn.nodemanager.resource.local-dirs accordingly.
I would also recommend turning on the property mapreduce.map.output.compress in order to reduce the size of the intermediate data.
个解答,截止Jean-Philippe Player · 2015年10月27日 20:58
You would assign one folder to each of the datanode disks, closely mapping dfs.datanode.data.dir. On a 12 disk system you would have 12 yarn local-dir locations.
2.Though Dataflow can be used with an out of the box Hadoop installation , there are a couple of configuration properties which may improve DataFlow/Hadoop performance
Using the O/S file system (i.e. /tmp or /var/) can be problematic especially if any applications log a lot of information or require large local files. So we have two properties to overcome this bottleneck.
The first is yarn.nodemanager.local-dirs. This setting specifies the directories to use as base directories for the containers run within YARN.
For each application and container created in YARN, a set of directories will be created underneath these local directories. These are then cleaned up when the application completes.
Here’s the setting from the yarn-site.xml file on one of our clusters. Note we have eight data disks per node on these clusters and create a directory for YARN on each data filesystem.
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>
/hadoop/hdfs/data1/hadoop/yarn/local,/hadoop/hdfs/data2/hadoop/yarn/local,/hadoop/hdfs/data3/hadoop/yarn/local,/hadoop/hdfs/data4/hadoop/yarn/local,/hadoop/hdfs/data5/hadoop/yarn/local,/hadoop/hdfs/data6/hadoop/yarn/local,/hadoop/hdfs/data7/hadoop/yarn/local,/hadoop/hdfs/data8/hadoop/yarn/local
</value>
<source>yarn-site.xml</source>
</property>
The second is yarn.nodemanager.log-dirs. Much like the local-dirs property, this setting specifies where container log files should go on the local disk. YARN spreads the load around if you specify multiple directories.
And here’s a sample setting:
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>
/hadoop/hdfs/data1/hadoop/yarn/log,/hadoop/hdfs/data2/hadoop/yarn/log,/hadoop/hdfs/data3/hadoop/yarn/log,/hadoop/hdfs/data4/hadoop/yarn/log,/hadoop/hdfs/data5/hadoop/yarn/log,/hadoop/hdfs/data6/hadoop/yarn/log,/hadoop/hdfs/data7/hadoop/yarn/log,/hadoop/hdfs/data8/hadoop/yarn/log
</value>
<source>yarn-site.xml</source>
</property>
Another YARN property you want to validate is the yarn.nodemanager.resource.memory-mb. This setting specifies the amount of memory YARN is allowed to allocate per worker node.
YARN will only allocate this much memory in total to containers. So it’s important to set this to some value less than the physical memory per worker node.
HDP appears to automatically pick 75% of the physical memory for this setting as our machines have 16GB of RAM each.
Here’s an example:
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value></value>
<source>yarn-site.xml</source>
</property>
当然也可以考虑使用nfs挂载,相关资料如下
3.How can I change yarn.nodemanager.local-dirs to point to file:/// (high performance NFS mount point)
Hi,I'm trying to change the "yarn.nodemanager.local-dirs" to point to "file:///fast_nfs/yarn/local". This is indeed a high-performance NFS mount-point that all the nodes in my cluster have.
When I try to change it in Ambari I can't and the message "Must be a slash or drive at the start, and must not contain white spaces" is displayed.
If I manually change the /etc/hadoop/conf/yarn-site.xml in all the nodes, after restarting YARN the "file:///" is removed from that option.
I want to have all the shuffle happening in my high-performance NFS array instead of in HDFS.
How can I change this behaviour in HDP?
The culprit is "file:/// "you should get a was to create a mount point /fast_nfs/yarn/local, hence the message "Must be a slash or drive ........" like te list below
/hadoop/yarn/local,/opt/hadoop/yarn/local,/usr/hadoop/yarn/local,/var/hadoop/yarn/local
Hope that helps
4. How to set yarn.nodemanager.local-dirs on M3 cluster to write to mapr fs
We are running a four node M3 cluster with one node running NFS. We are getting the the following error.
1/1 local-dirs are bad: /mapr/clustername/tmp/host_name on the nodes that does not have NFS running.
What is the best way to set this property in the yarn-site.xml to allow all nodes to use mapr fs /tmp as the default location and not the local file system /tmp
I believe the property "yarn.nodemanager.local-dirs" is meant to be a location on the local file system. It cannot be a location of the distributed file system (HDFS or MapR FS).
This property determines the location where the node manager maintains intermediate data (for example during the shuffle phase).
You can the find gory details here: http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
The default location as you mentioned is /tmp. If you want to improve performance, you could provide multiple directories on separate disks for better I/O throughput.
But, you should ascertain that this is a indeed bottleneck and if a separate disk is warranted for this purpose (or you are better of using it as a MapR data disk).
One other thing, the NFS mounted location (/mapr/clustername/tmp/host_name) is not a part of the distributed FS.
MapR makes it seamless to work between its distributed file system and the POSIX file system. But the files of the POSIX system are not stored in any containers/chunks/blocks, etc.
Since the path you specified is really a local directory on the node running NFS, you don't get an error message on that node . But on the other nodes, the system can't find a local directory by that name and hence it is complaining.
Hadoop Yarn配置项 yarn.nodemanager.resource.local-dirs探讨的更多相关文章
- hadoop集群配置方法---mapreduce应用:xml解析+wordcount详解---yarn配置项解析
注:以下链接均为近期hadoop集群搭建及mapreduce应用开发查找到的资料.使用hadoop2.6.0,其中hadoop集群配置过程下面的文章都有部分参考. hadoop集群配置方法: ---- ...
- Hadoop学习之YARN框架
转自:http://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop-yarn/,非常感谢分享! 对于业界的大数据存储及分布式处理系统来说,H ...
- Hadoop生态系统之Yarn
Apache YARN(Yet Another Resource Negotiator) 是Hadoop的集群资源管理系统.YARN被引入Hadoop2最初是为了改善MapReduce的实现,但它具有 ...
- hadoop备战:yarn框架的搭建(mapreduce2)
昨天没有写好了没有更新,今天一起更新,yarn框架也是刚搭建好的. 我这里把hadoop放在了我的个人用户hadoop下了,你也能够尝试把它放在/usr/local,考虑的问题就相对多点. 主要的软硬 ...
- hadoop备战:yarn框架的简单介绍(mapreduce2)
新 Hadoop Yarn 框架原理及运作机制 重构根本的思想是将 JobTracker 两个基本的功能分离成单独的组件,这两个功能是资源管理和任务调度 / 监控.新的资源管理器全局管理全部应用程序计 ...
- Hadoop核心组件之YARN
YARN概述 Yet Another Resource Negotiator:另外资源的协调者 通用的资源管理系统 为上层应用提供统一的资源管理和调度 操作系统级别的调度框架,可以让各种计算框架运行在 ...
- Hadoop学习笔记—Yarn
目录 一些基本知识 ResourceManager 的恢复 Resource Manager的HA YARN Node Labels YARN Node Attributes Web Applicat ...
- Hadoop 2.2 YARN分布式集群搭建配置流程
搭建环境准备:JDK1.6,SSH免密码通信 系统:CentOS 6.3 集群配置:NameNode和ResourceManager在一台服务器上,三个数据节点 搭建用户:YARN Hadoop2.2 ...
- Hadoop数据操作系统YARN全解析
“ Hadoop 2.0引入YARN,大大提高了集群的资源利用率并降低了集群管理成本.其在异构集群中是怎样应用的?Hulu又有哪些成功实践可以分享? 为了能够对集群中的资源进行统一管理和调度,Hado ...
随机推荐
- C++版- Leetcode 3. Longest Substring Without Repeating Characters解题报告
Leetcode 3. Longest Substring Without Repeating Characters 提交网址: https://leetcode.com/problems/longe ...
- SpringBoot入门教程(七)整合themeleaf+bootstrap
Thymeleaf是用于Web和独立环境的现代服务器端Java模板引擎.Thymeleaf的主要目标是将优雅的自然模板带到您的开发工作流程中—HTML能够在浏览器中正确显示,并且可以作为静态原型,从而 ...
- Django里URL配置中name参数的作用
先看一下URL配置,新增了如下内容: 修改视图views.py 增加一个页面 现在要求是在name页面里面增加一个链接,显示详细信息.修改name页面 这个功能实现了,可是之前添加的那个name参数是 ...
- centos7下误执行chmod -R 777 /后的权限修复方法
今天由于权限问题zz一般把/usr/bin和/usr/lib两个目录用chmod -R 777执行了一遍,结果各种问题出现,su root就报su:鉴定故障的错误.然后上网找教程很多都要求在root权 ...
- Asp.net Core 使用Jenkins + Dockor 实现持续集成、自动化部署(一):Jenkins安装
2019/1/31更新,经过我一段时间的使用 建议大家的jenkins还是不要使用docker方式安装 建议大家的jenkins还是不要使用docker方式安装 建议大家的jenkins还是不要使用d ...
- leetcode — recover-binary-search-tree
import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; import java.util. ...
- RDIFramework.NET ━ .NET快速信息化系统开发框架 V3.2-模块管理按子系统进行分类管理
在RDIFramework.NET以往的框架中,模块管理界面展示了整个框架所管理的所有模块,如果系统过多,达几十个甚至上百个子系统时,管理起来就非常的麻烦,不光加载效率会很低,页面展示也会很不友好.框 ...
- 我和Session的不解之“缘”(故事型技术长文)
本文讲述了一路走来对Session的认知.文章有点长,不过是故事型的,应该不枯燥.相信读完也一定会有所收获. (一) “当你登陆系统后,服务器会创建一个Session,保存你的登陆信息,下次再访问时就 ...
- Docker最全教程——从理论到实战(五)
往期内容链接 Docker最全教程——从理论到实战(一) Docker最全教程——从理论到实战(二) Docker最全教程——从理论到实战(三) Docker最全教程——从理论到实战(四) 本篇教程持 ...
- vue2.5.2版本 :MAC设置应用在127.0.0.1:80端口访问; 并将127.0.0.1指向www.yours.com ;问题“ Invalid Host header”
0.设置自己的host文件,将127.0.0.1指向自己想要访问的域名 127.0.0.1 www.yours.com 1.MAC设置应用在127.0.0.1:80端口访问: config/index ...