以下基于 hadoop版本 hadoop-2.8.4

给各个节点打标签

yarn rmadmin -addToClusterNodeLabels fastcpu,normal

        # 是否独占默认是true，不独占其他队列就可以normal标签资源

        yarn rmadmin -addToClusterNodeLabels "fastcpu,normal(exclusive=false)"

yarn rmadmin -replaceLabelsOnNode "container-192-168-200-140,fastcpu"

# container-192-168-200-141不打标签,也可以不执行此命令

yarn rmadmin -replaceLabelsOnNode "container-192-168-200-141, "

yarn rmadmin -replaceLabelsOnNode "container-192-168-200-142,normal"
1.spark任务可以通过spark-submit方式指定参数spark.yarn.am.nodeLabelExpression和spark.yarn.executor.nodeLabelExpression来实现，使任务提交到特定的分区。
./bin/spark-submit --class org.apache.spark.examples.SparkPi

  --master yarn --deploy-mode cluster --driver-memory 3g --executor-memory 2g

  --conf spark.yarn.am.nodeLabelExpression=fastcpu

  --conf spark.yarn.executor.nodeLabelExpression=fastcpu  jars/spark-examples.jar 10
2.hadoop distribute shell / hadoop job 也可以。
hadoop jar ./share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.8.4.jar -shell_command "sleep 100000" -jar ./share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.8.4.jar  -num_containers 5 -queue etl-ywpt -node_label_expression fastcpu
mr example wordcount:
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.4.jar wordcount -D mapreduce.job.node-label-expression="fastcpu" -D mapreduce.job.queuename="etl-ywpt" /hadoop-hadoop-namenode-container-192-168-200-140.log /output
3.另外还可以通过yarn代码的方式提交；hive可以指定如下参数 提交指定队列及label。
set mapreduce.job.node-label-expression=fastcpu;

set mapreduce.job.queuename=etl-bi;

yarn web ui

yarn 8088调度页面

node 的 label 页面

容量调度才能使用label便签，公平调度暂时不支持标签。
配置文件 capacity-scheduler.xml
<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<configuration>

   <property>

    <name>yarn.scheduler.capacity.maximum-applications</name>

    <value>10000</value>

    <description>

      Maximum number of applications that can be pending and running.

    </description>

  </property>

  <property>

     <name>yarn.scheduler.capacity.root.queues</name>

     <value>default,etl-ywpt,etl-bi,etl-ly,etl-test,hue,bi</value>

  </property>

 <!--

  <property>

   <name>yarn.scheduler.capacity.root.acl_submit_applications</name>

   <value> </value>

  </property>

 <property>

 <name>yarn.scheduler.capacity.root.acl_administer_queue</name>

 <value>hadoop </value>

</property>

-->

 <property>

    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>

    <value>0.2</value>

    <description>

      Maximum percent of resources in the cluster which can be used to run

      application masters i.e. controls number of concurrent running

      applications.

    </description>

  </property>

 <property>

  <name>yarn.scheduler.capacity.resource-calculator</name>

  <value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>

 </property>

 <!--

  <property>

    <name>yarn.scheduler.capacity.resource-calculator</name>

    <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>

    <description>

      The ResourceCalculator implementation to be used to compare

      Resources in the scheduler.

      The default i.e. DefaultResourceCalculator only uses Memory while

      DominantResourceCalculator uses dominant-resource to compare

      multi-dimensional resources such as Memory, CPU etc.

    </description>

  </property>

  -->

<!-- label两种模式 normal/fastcpu -->

<!-- root队列可提交的标签 -->

<property>

   <name>yarn.scheduler.capacity.root.capacity</name>

   <value>100</value>

</property>

<property>

      <name>yarn.scheduler.capacity.root.maximum-capacity</name>

      <value>100</value>

</property>

<property>

  <name>yarn.scheduler.capacity.root.accessible-node-labels</name>

  <value>*</value>

</property>

<property>

    <name>yarn.scheduler.capacity.root.accessible-node-labels.normal.capacity</name>

    <value>100</value>

    <description>root队列对normal标签节点可用的百分比</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.accessible-node-labels.fastcpu.capacity</name>

    <value>100</value>

    <description>root队列对fastcpu标签节点可用的百分比</description>

</property>

<!-- default队列 -->

 <property>

    <name>yarn.scheduler.capacity.root.default.accessible-node-labels</name>

    <value>normal</value>

    <description>default队列应用可用的节点标签</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.default.default-node-label-expression</name>

    <value>normal</value>

    <description>default队列应用默认节点标签</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.default.accessible-node-labels.normal.capacity</name>

    <value>10</value>

    <description>default队列对normal标签节点可用的百分比</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.default.accessible-node-labels.normal.maximum-capacity</name>

    <value>50</value>

</property>

 <property>

    <name>yarn.scheduler.capacity.root.default.capacity</name>

    <value>10</value>

    <description>Default queue target capacity.</description>

  </property>

  <property>

    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>

    <value>20</value>

    <description>

      The maximum capacity of the default queue.

    </description>

  </property>

<!-- etl-ywpt队列 -->

<property>

    <name>yarn.scheduler.capacity.root.etl-ywpt.accessible-node-labels</name>

    <value>*</value>

    <description>etl-ywpt队列应用可用的节点标签</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.etl-ywpt.default-node-label-expression</name>

    <value>normal</value>

    <description>etl-ywpt队列应用默认节点标签</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.etl-ywpt.accessible-node-labels.normal.capacity</name>

    <value>10</value>

    <description>etl-ywpt队列对fastcpu标签节点可用的百分比</description>

</property>

<property>

      <name>yarn.scheduler.capacity.root.etl-ywpt.accessible-node-labels.normal.maximum-capacity</name>

      <value>20</value>

</property>

<property>

    <name>yarn.scheduler.capacity.root.etl-ywpt.accessible-node-labels.fastcpu.capacity</name>

    <value>50</value>

    <description>etl-ywpt队列对fastcpu标签节点可用的百分比</description>

</property>

<property>

      <name>yarn.scheduler.capacity.root.etl-ywpt.accessible-node-labels.fastcpu.maximum-capacity</name>

      <value>60</value>

</property>

<property>

  <name>yarn.scheduler.capacity.root.etl-ywpt.capacity</name>

  <value>5</value>

</property>

<property>

    <name>yarn.scheduler.capacity.root.etl-ywpt.maximum-capacity</name>

    <value>30</value>

</property>

<!-- etl-bi队列 -->

<property>

    <name>yarn.scheduler.capacity.root.etl-bi.accessible-node-labels</name>

    <value>fastcpu</value>

    <description>etl-bi队列应用可用的节点标签</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.etl-bi.default-node-label-expression</name>

    <value>fastcpu</value>

    <description>etl-bi队列应用默认节点标签</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.etl-bi.accessible-node-labels.fastcpu.capacity</name>

    <value>50</value>

    <description>etl-bi队列对fastcpu标签节点可用的百分比</description>

</property>

<property>

      <name>yarn.scheduler.capacity.root.etl-bi.accessible-node-labels.fastcpu.maximum-capacity</name>

      <value>100</value>

</property>

<property>

  <name>yarn.scheduler.capacity.root.etl-bi.capacity</name>

  <value>3</value>

</property>

<property>

    <name>yarn.scheduler.capacity.root.etl-bi.maximum-capacity</name>

    <value>20</value>

</property>

<!-- etl-ly队列 -->

<property>

    <name>yarn.scheduler.capacity.root.etl-ly.accessible-node-labels</name>

    <value>normal</value>

    <description>etl-ly队列应用可用的节点标签</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.etl-ly.default-node-label-expression</name>

    <value>normal</value>

    <description>etl-ly队列应用默认节点标签</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.etl-ly.accessible-node-labels.normal.capacity</name>

    <value>20</value>

    <description>default队列对normal标签节点可用的百分比</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.etl-ly.accessible-node-labels.fastcpu.maximum-capacity</name>

    <value>100</value>

</property>

<property>

  <name>yarn.scheduler.capacity.root.etl-ly.capacity</name>

  <value>52</value>

</property>

<property>

    <name>yarn.scheduler.capacity.root.etl-ly.maximum-capacity</name>

    <value>55</value>

</property>

<!-- etl-test队列 -->

<property>

    <name>yarn.scheduler.capacity.root.etl-test.accessible-node-labels</name>

    <value>normal</value>

    <description>etl-test队列应用可用的节点标签</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.etl-test.default-node-label-expression</name>

    <value>normal</value>

    <description>etl-test队列应用默认节点标签</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.etl-test.accessible-node-labels.normal.capacity</name>

    <value>10</value>

    <description>etl-test队列对normal标签节点可用的百分比</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.etl-test.accessible-node-labels.normal.maximum-capacity</name>

    <value>100</value>

</property>

<property>

  <name>yarn.scheduler.capacity.root.etl-test.capacity</name>

  <value>10</value>

</property>

<property>

  <name>yarn.scheduler.capacity.root.etl-test.maximum-capacity</name>

  <value>50</value>

</property> 

<!-- hue队列  -->

<property>

    <name>yarn.scheduler.capacity.root.hue.accessible-node-labels</name>

    <value>normal</value>

    <description>hue队列应用可用的节点标签</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.hue.default-node-label-expression</name>

    <value>normal</value>

    <description>hue队列应用默认节点标签</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.hue.accessible-node-labels.normal.capacity</name>

    <value>10</value>

    <description>hue队列对normal标签节点可用的百分比</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.hue.accessible-node-labels.normal.maximum-capacity</name>

    <value>100</value>

</property>

<property>

  <name>yarn.scheduler.capacity.root.hue.capacity</name>

  <value>10</value>

</property>

<property>

  <name>yarn.scheduler.capacity.root.hue.maximum-capacity</name>

  <value>50</value>

</property> 

 <!-- bi队列 -->

 <property>

    <name>yarn.scheduler.capacity.root.bi.accessible-node-labels</name>

    <value>normal</value>

    <description>bi队列应用可用的节点标签</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.bi.default-node-label-expression</name>

    <value>normal</value>

    <description>bi队列应用默认节点标签</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.bi.accessible-node-labels.normal.capacity</name>

    <value>40</value>

    <description>bi队列对fastcpu标签节点可用的百分比</description>

</property>

<property>

    <name>yarn.scheduler.capacity.root.bi.accessible-node-labels.normal.maximum-capacity</name>

    <value>100</value>

</property>

<property>

  <name>yarn.scheduler.capacity.root.bi.capacity</name>

  <value>10</value>

</property>

<property>

  <name>yarn.scheduler.capacity.root.bi.maximum-capacity</name>

  <value>20</value>

</property> 

</configuration>

YARN label 特性 & 指定队列及label提交任务的更多相关文章

  1. django-celery 创建多个broker队列 异步执行任务时指定队列

    一.这里不再详细述说 django 框架中如何使用celery, 重点放在如何实现创建多个队列, 并指定队列存放异步任务 笔者使用   django-celery==3.2.2 模块, 配置项及配置参 ...

  2. 洛谷 P1160 队列安排 Label：链表 数据结构

    题目描述 一个学校里老师要将班上N个同学排成一列,同学被编号为1-N,他采取如下的方法: 1.先将1号同学安排进队列,这时队列中只有他一个人: 2.2-N号同学依次入列,编号为i的同学入列方式为:老师 ...

  3. 转： 【Java并发编程】之二十一：并发新特性—阻塞队列和阻塞栈（含代码）

    转载请注明出处:http://blog.csdn.net/ns_code/article/details/17511147 阻塞队列 阻塞队列是Java5并发新特性中的内容,阻塞队列的接口是Java. ...

  4. db2 v9.7 新特性cur_commit 能够实现未提交读新特性cur_commit 能够实现未提交读

    db2 get db cfg|find "CUR_COMMIT" 当前已落实                                   (CUR_COMMIT) = ON ...

  5. 【Java并发编程】：并发新特性—塞队列和阻塞栈

    阻塞队列 阻塞队列是Java5并发新特性中的内容,阻塞队列的接口是Java.util.concurrent.BlockingQueue,它有多个实现类:ArrayBlockingQueue.Delay ...

  6. 使用FROM确认按钮(键盘13号键)提交特性并使用ajax.POST提交.

    如果又想使用FROM确认按钮(键盘13号键)提交特性  还能继续用AJAX.POST提交.就需要使用return false 来阻止FROM默认提交 代码如下: HTML页面 这里最关键就是用了ret ...

  7. label 标签的用法,点label选中单选、复选框或文本框

    <label>拥有的权限</label> <label class="checkbox" id="privilege_id" st ...

  8. Java高级特性—消息队列

    1)什么是jms JMS即Java消息服务(Java Message Service)应用程序接口是一个Java平台中关于面向消息中间件(MOM)的API. 它便于消息系统中的Java应用程序进行消息 ...

  9. Qt添加窗口背景图片、Label图片显示、、Label文字显示

    一.添加窗口背景图片 重写MainWindow绘制事件 void MainWindow::paintEvent(QPaintEvent *event) { QPainter painter(this) ...

随机推荐

  1. spring boot编译项目打jar包

    <build> <plugins> <!--jar包打包--> <plugin> <groupId>org.springframework. ...

  2. Python使用plotly绘制数据图表的方法

    转载:http://www.jb51.net/article/118936.htm 本篇文章主要介绍了Python使用plotly绘制数据图表的方法,实例分析了plotly绘制的技巧. 导语:使用 p ...

  3. 《Java并发编程实战》笔记-状态依赖方法的标准形式

    void stateDependentMethod() throws InterruptedException { //必须通过一个锁来保护条件谓词 synchronized(lock) { whil ...

  4. flume http source示例讲解

    一.介绍 flume自带的Http Source可以通过Http Post接收事件. 场景:对于有些应用程序环境,它可能不能部署Flume SDK及其依赖项,或客户端代码倾向于通过HTTP而不是Flu ...

  5. mysql查询优化之三：查询优化器提示（hint）

    目录: <MySQL中的两种临时表>--强制使用临时表 SQL_BUFFER_RESULT <MySQL 多表关联更新及删除> <mysql查询优化之三:查询优化器提示( ...

  6. go语言学习--channel的关闭

    在使用Go channel的时候,一个适用的原则是不要从接收端关闭channel,也不要在多个并发发送端中关闭channel.换句话说,如果sender(发送者)只是唯一的sender或者是chann ...

  7. tf.nn.embedding_lookup

    tf.nn.embedding_lookup(params, ids, partition_strategy=’mod’, name=None, validate_indices=True, max_ ...

  8. python的json模块介绍

    转载:https://blog.csdn.net/xsj_blog/article/details/51921664 对于数据传递方面,XML是一种选择,还有一种选择是JSON,它是一种轻量级的数据交 ...

  9. SAS数据集推送到sql server 数据库 实现代码段

    libname fdsas ODBC datasrc=fdsas user=fdsas password=fdsas123 preserve_tab_names=yes connection=shar ...

  10. vue-生存周期

    beforeCreate   实例初始化之后 created           实例创建之后 beforeMount   实例挂载前    文本节点 mounted        渲染实例 防止花括 ...