Overview

YARN provides API not for application developers but for the great developers working on new computing engines. YARN make it easy and unified for resource management for the computing engines. It fills the gap between mputation and storage. NoSQL database like HBase use slider apdaters to YARN.

With YARN 

Withou YARN

Entities in YARN

The base of Distribution is HDFS and YARN. HDFS for managing storage. YARN for managing computing.
Client: who submits the job: connects to MR or HDFS framework.
YARN Resource Manager: allocate computing resource required by the job.
Scheduleer:job scheduling,locate the resources.
Application Manager:performan any monitoring or tracking of application/job status.
YARN Node Manager: on all slave nodes. launch / manager containers.
MR Application Master:carry out execution of the job associated with it. different between computing engines. It coordinates the tasks running and monitors the progress and aggregates it and since reports to its client . It is spawn under node manager on the instruction by RM. spawn for every job and end with the job done.
YARN Child: manages the run of the map and reduce tasks,send updates / progress to application master.
HDFS:i/o

The process of job run in YARN

Job submission:Your program triggers the job client and the job client contacts the RM for the new job id. copy the job resource to HDFS with high replica and then submit the job.

Job Initialization: Then RM (the scheduler)picks up the job from the job queue(FIFO,capacity,fair) and contacts NM,sponsor new container (Linux kernel feature, a abstruct of resource like cpu,mem,disk,network bandwidth. doker uses it too) and launches AM for the job.
Job Assigement: AM creates new objects , it retrives the input splits from HDFS and crete one task per input split. AM then decides if the job is samll or not. If it is small job , run its jvm on a single node. If not,contacts RM locate computing resources.
Job Execution:RM considers data locality while assigning the resources(Scheduler at this time knows where the splits are located.It gathers this info from the heartbeats of NM. Based on it, it consider data locality when allocating resources. try as best, then consider the rack local nodes ,if still fails, it will pick random from available noedes).AM then communicates node managers which launches the yarn child(a java program the main class is YarnChild,seperate JVM from long running system demons from the suer code). yarn child retrieves the code and other resource from HDFS and then run the tasks(mr).Yan child sends the progress to AM(every 3 seconds) which aggregrates(each yarn client's information) the report and sends the report to the client.
Job Exmpletion:On job completion , yarn child and AM terminates themseves for the next job.

 

MR execution in YARN的更多相关文章

  1. Yarn源码分析之MRAppMaster上MapReduce作业处理总流程(一)

    我们知道,如果想要在Yarn上运行MapReduce作业,仅需实现一个ApplicationMaster组件即可,而MRAppMaster正是MapReduce在Yarn上ApplicationMas ...

  2. hadoop多机安装HA+YARN

    HA 相比于Hadoop1.0,Hadoop 2.0中的HDFS增加了两个重大特性,HA(热备)和Federation(联邦).HA即为High Availability,用于解决NameNode单点 ...

  3. hadoop多机安装YARN

    hadoop伪分布安装称为测试环境安装,多机分布称为生成环境安装.以下安装没有进行HA(热备)和Federation(联邦).除非是性能需要,否则没必要安装Federation,HA可以一试,涉及到Z ...

  4. Hadoop2.4.1 64-Bit QJM HA and YARN HA + Zookeeper-3.4.6 + Hbase-0.98.8-hadoop2-bin HA Install

    Hadoop2.4.1 64-Bit QJM HA and YARN HA Install + Zookeeper-3.4.6 + Hbase-0.98.8-hadoop2-bin HA(Hadoop ...

  5. Hadoop 5、HDFS HA 和 YARN

    Hadoop 2.0 产生的背景Hadoop 1.0 中HDFS和MapReduce存在高可用和扩展方面的问题 HDFS存在的问题 NameNode单点故障,难以用于在线场景 NameNode压力过大 ...

  6. YARN的基础配置

    基于HADOOP3.0+Centos7.0的yarn基础配置: 执行步骤:(1)配置集群yarn (2)启动.测试集群(3)在yarn上执行wordcount案例 一.配置yarn集群 1.配置yar ...

  7. YARN的三种调度器的使用

    YRAN提供了三种调度策略 一.FIFO-先进先出调度器 YRAN默认情况下使用的是该调度器,即所有的应用程序都是按照提交的顺序来执行的,这些应用程序都放在一个队列中,只有在前面的一个任务执行完成之后 ...

  8. Hadoop YARN上运行MapReduce程序

    (1)配置集群 (a)配置hadoop-2.7.2/etc/hadoop/yarn-env.sh 配置一下JAVA_HOME export JAVA_HOME=/home/hadoop/bigdata ...

  9. hadoop3.1集成yarn ha

    1.角色分配

随机推荐

  1. mysql事务隔离

    一.事务的特性 原子性.一致性.隔离性.持久性 二.事务的隔离级别 1.未提交读 (Read Uncommitpeatableted) 臭名昭著的脏读 ,事务A读到事务B未提交的数据 2.提交读RC( ...

  2. android软件开发之获取本地音乐属性

    歌曲的名称 :MediaStore.Audio.Media.TITLString tilte = cursor.getString(cursor.getColumnIndexOrThrow(Media ...

  3. LeetCode 中级 - 救生艇(105)

    第 i 个人的体重为 people[i],每艘船可以承载的最大重量为 limit. 每艘船最多可同时载两人,但条件是这些人的重量之和最多为 limit. 返回载到每一个人所需的最小船数.(保证每个人都 ...

  4. 第四模块MySQL50题作业,以及由作业引申出来的一些高端玩法

    一.表关系 先参照如下表结构创建7张表格,并创建相关约束                 班级表:class       学生表:student       cid caption grade_id ...

  5. IoC和AOP扩展

    一.构造注入 二.使用p命名空间注入属性值 三.注入不同数据类型 <?xml version="1.0" encoding="UTF-8"?> &l ...

  6. ACM 2003~2005

    ACM 2003 求实数的绝对值 import java.util.Scanner; public class Lengxc { public static void main(String[] ar ...

  7. redis安装make失败,make[1]: *** [adlist.o] Error 127....

    解压后 执行make后报错: cd src && make all make[1]: Entering directory `/home/liuchaofan/redis-3.0.7/ ...

  8. Python-知识点小计

    1.python赋值.浅拷贝.深拷贝区别:https://www.cnblogs.com/xueli/p/4952063.html: 2.python的hasattr(),getattr(),sett ...

  9. wdcp v3 pureftpd 无法登录问题解决

    wdcp v3 新建站点和ftp账号 单位无法登录ftp 在端口中也确实可以看到有进行在登录状态 错误原因: 防火墙端口没有开启该端口范围  20000-30000 这时候发现 改端口为20078  ...

  10. 微信小程序 唯一标识 加减

    var nums = 'goods_list[' + e.currentTarget.dataset.indexs+'].goods_num' //console.log(nuns) var num ...