spark提交至yarn的的动态资源分配

1、为什么开启动态资源分配

⽤户提交Spark应⽤到Yarn上时，可以通过spark-submit的num-executors参数显示地指定executor 个数，随后，ApplicationMaster会为这些executor申请资源，每个executor作为⼀个Container在 Yarn上运⾏。Spark调度器会把Task按照合适的策略分配到executor上执⾏。所有任务执⾏完后， executor被杀死，应⽤结束。在job运⾏的过程中，⽆论executor是否领取到任务，都会⼀直占有着 资源不释放。很显然，这在任务量⼩且显示指定⼤量executor的情况下会很容易造成资源浪费

2.yarn-site.xml加入配置，并重启yarn服务

spark版本:2.2.1,hadoop版本:cdh5.14.2-2.6.0，不是clouder集成的cdh是手动单独搭建的

vim etc/hadoop/yarn-site.xml

<property>
 　　<name>yarn.nodemanager.aux-services</name>
 　　<value>spark_shuffle,mapreduce_shuffle</value>
 </property> <property>
 　　<name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
 　　<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>

重启yarn时的需要注意的异常：nodemanager没有正常启动，yarn的8080页面的core与memory都为空

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2349)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2373)
    ... 10 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2255)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2347)
    ... 11 more
2020-02-17 19:54:59,185 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager metrics system...
2020-02-17 19:54:59,185 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system stopped.
2020-02-17 19:54:59,185 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system shutdown complete.
2020-02-17 19:54:59,185 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2381)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:121)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:236)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:318)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:562)
    at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2349)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2373)
    ... 10 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2255)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2347)
    ... 11 more
2020-02-17 19:54:59,189 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at bigdata.server1/192.168.121.12
************************************************************/

原因是缺少了：sparkShuffle的jar包

mv  spark/yarn/spark-2.11-2.2.1-shuffle_.jar /opt/modules/hadoop-2.6.0-cdh5.14.2/share/hadoop/yarn/

nodemanager依然启动不了，查询nodemanger.log日志、继续报错：

java.lang.NoSuchMethodError: org.spark_project.com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z

添加了jackson的包没啥用，网上有一样的报错方式：https://www.oschina.net/question/3721355_2269200

结果：未解决

3.spark的动态资源分配开启

可以在spark-defaults.conf中添加了如下配置：

spark.shuffle.service.enabled true //启⽤External shuffle Service服务
spark.shuffle.service.port 7337 //Shuffle Service服务端⼝，必须和yarn-site中的⼀致
spark.dynamicAllocation.enabled true //开启动态资源分配
spark.dynamicAllocation.minExecutors 1 //每个Application最⼩分配的executor数
spark.dynamicAllocation.maxExecutors 30 //每个Application最⼤并发分配的executor数
spark.dynamicAllocation.schedulerBacklogTimeout 1s
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout 5s

也可以在代码或者脚本中添加sparkconf

4.hadoop版本cdh5.14.2-2.6.0与spark 2.2.1单独搭建的会报错

5.使用clouderCDH 5.14.0 的版本测试

1.在yarn-site.xml添加上面的配置

1

2.普通提交,spark2版本进行shell提交，观察yarn

spark2-shell --master yarn-client \
 --executor-memory 2G \
 --num-executors 10

可以看到10个executor(driver占一核)没有任务也是申请到资源，占着不用，造成了资源浪费

3.使用spark的动态资源分配提交

spark2-shell —master yarn —eploy-mode client \
//指定队列
—queue "test" \
//日志配置
—conf spark.driver.extraJava0ptions=-Dlog4j.configuration=log4j-yarn.properties \
—conf spark.executor.extraJava0ptions=-Dlog4j.configuration=log4j-yarn.properties \
—conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
//推测执行等待时间
—conf spark.locality.wait=10 \
//最大失败重试次数
—conf spark.task.maxFailures=8 \
—conf spark.ui.killEnabled=false \
—conf spark.logConf=true \
//非堆内存配置
—conf spa rk.yarn.d river.memoryOverhead=512 \
—conf spark.yarn.executor.memoryOverhead=1024 \
—conf spark.yarn.maxAppAttempts=4 \
—conf spark.yarn.am.attemptFailuresValidityInterval=lh \
—conf spark.yarn.executor.failuresValidityInterval=lh \
//动态资源开启
—conf spark.dynamicAllocation.enabled=true \
//最大最小申请的Executors数
—conf spark.dynamicAllocation.minExecutors=l \
—conf spark.dynamicAllocation.maxExecutors=30 \
—conf spark.dynamicAllocation.executorldleTimeout=3s \
—conf spark.shuffle.service.enabled=true

 可以看到申请的只有1个executor(driver端的),暂时没有提交任务，最小申请为1个，

sc.textFile("file:///etc/hosts").flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_ + _).count()

　 提交一个wrodcount程序跑一下，发现使用2个executor(1个driver），说明这种数量级的数据就2个就可以满足了，不需要开启更多的资源去空转，占用　

 4.动态资源的好处

1.多个部门去使用集群资源，有运行的任务时候申请资源，没有时将资源回收给yarn，供其他人使用

2.防止小数据申请大资源，造成资源浪费，executor空转

3.在进行流式处理时不建议开启，流式处理的数据量在不同时段是不同的，需要最大利用资源，从而提高消费速度，以免造成数据堆积，流式处理时如果一直去判断数据量的大小进行动态申请时，创建与销毁资源也需要时间，从而让流式处理造成了延迟