某日 收到告警 线上集群rm切换 观察resourcemanager 日志报错如下

这行不明显 再看看其他日志报错

在 app attempt_removed 时候发生了空指针错误

    break;
case APP_ATTEMPT_REMOVED:
if (!(event instanceof AppAttemptRemovedSchedulerEvent)) {
throw new RuntimeException("Unexpected event type: " + event);
}
AppAttemptRemovedSchedulerEvent appAttemptRemovedEvent =
(AppAttemptRemovedSchedulerEvent) event;
removeApplicationAttempt(
appAttemptRemovedEvent.getApplicationAttemptID(),
appAttemptRemovedEvent.getFinalAttemptState(),
appAttemptRemovedEvent.getKeepContainersAcrossAppAttempts());
break;
case CONTAINER_EXPIRED:

定位到代码问题在这里 新增标记内容

private void removeApplicationAttempt(
ApplicationAttemptId applicationAttemptId,
RMAppAttemptState rmAppAttemptFinalState, boolean keepContainers) {
LOG.info("Application " + applicationAttemptId + " is done." +
" finalState=" + rmAppAttemptFinalState);
try {
writeLock.lock();
SchedulerApplication<FSAppAttempt> application =
applications.get(applicationAttemptId.getApplicationId());
FSAppAttempt attempt = getSchedulerApp(applicationAttemptId); if (attempt == null || application == null) {
LOG.info("Unknown application " + applicationAttemptId + " has completed!");
return;
}
//已经停止了就不用再次停止了 新增
// Check if the attempt is already stopped and don't stop it twice.
if (attempt.isStopped()) {
LOG.info("Application " + applicationAttemptId + " has already been "
+ "stopped!");
return;
} // Release all the running containers
for (RMContainer rmContainer : attempt.getLiveContainers()) {
if (keepContainers
&& rmContainer.getState().equals(RMContainerState.RUNNING)) {
// do not kill the running container in the case of work-preserving AM
// restart.
LOG.info("Skip killing " + rmContainer.getContainerId());
continue;
}
super.completedContainer(rmContainer,
SchedulerUtils.createAbnormalContainerStatus(
rmContainer.getContainerId(),
SchedulerUtils.COMPLETED_APPLICATION),
RMContainerEventType.KILL);
}

增加如下代码

 @Override
public String moveApplication(ApplicationId appId,
String queueName) throws YarnException {
try {
writeLock.lock();
SchedulerApplication<FSAppAttempt> app = applications.get(appId);
if (app == null) {
throw new YarnException("App to be moved " + appId + " not found.");
}
FSAppAttempt attempt = (FSAppAttempt) app.getCurrentAppAttempt();
// To serialize with FairScheduler#allocate, synchronize on app attempt
try {
attempt.getWriteLock().lock();
FSLeafQueue oldQueue = (FSLeafQueue) app.getQueue();
// Check if the attempt is already stopped: don't move stopped app
// attempt. The attempt has already been removed from all queues.
if (attempt.isStopped()) {
LOG.info("Application " + appId + " is stopped and can't be moved!"throw new YarnException("Application " ++ " is stopped and can't be moved!");
}
String destQueueName = handleMoveToPlanQueue(queueName);
FSLeafQueue targetQueue = queueMgr.getLeafQueue(destQueueName, false);
if (targetQueue == null) {
throw new YarnException("Target queue " + queueName
+ " not found or is not a leaf queue.");
}
if (targetQueue == oldQueue) {
return oldQueue.getQueueName();
}
private void executeMove(SchedulerApplication<FSAppAttempt> app,
FSAppAttempt attempt, FSLeafQueue oldQueue, FSLeafQueue newQueue) {
// Check current runs state. Do not remove the attempt from the queue until
// after the check has been performed otherwise it could remove the app
// from a queue without moving it to a new queue.
boolean wasRunnable = oldQueue.isRunnableApp(attempt);
// if app was not runnable before, it may be runnable now
boolean nowRunnable = maxRunningEnforcer.canAppBeRunnable(newQueue,
attempt.getUser());
if (wasRunnable && !nowRunnable) {
throw new IllegalStateException("Should have already verified that app "
+ attempt.getApplicationId() + " would be runnable in new queue");
} // Now it is safe to remove from the queue.
oldQueue.removeApp(attempt); if (wasRunnable) {
maxRunningEnforcer.untrackRunnableApp(attempt);
} else if (nowRunnable) {
// App has changed from non-runnable to runnable
maxRunningEnforcer.untrackNonRunnableApp(attempt);
} attempt.move(newQueue); // This updates all the metrics
app.setQueue(newQueue);
newQueue.addApp(attempt, nowRunnable); if (nowRunnable) {
maxRunningEnforcer.trackRunnableApp(attempt);
}
if (wasRunnable) {
maxRunningEnforcer.updateRunnabilityOnAppRemoval(attempt, oldQueue);
}
}

问题解决

参考 https://issues.apache.org/jira/secure/attachment/12841441/YARN-5136.2.patch

记录一次 hadoop yarn resourceManager无故切换的故障的更多相关文章

  1. 记录一次线上yarn RM频繁切换的故障

    周末一大早被报警惊醒,rm频繁切换 急急忙忙排查 看到两处错误日志 错误信息1 ervation <memory:0, vCores:0> 2019-12-21 11:51:57,781 ...

  2. Hadoop记录-yarn ResourceManager Active频繁易主问题排查(转载)

    一.故障现象 两个节点的ResourceManger频繁在active和standby角色中切换.不断有active易主的告警发出 许多任务的状态没能成功更新,导致一些任务状态卡在NEW_SAVING ...

  3. Hadoop官方文档翻译—— YARN ResourceManager High Availability 2.7.3

    ResourceManager High Availability (RM高可用) Introduction(简介) Architecture(架构) RM Failover(RM 故障切换) Rec ...

  4. Spark&Hive:如何使用scala开发spark访问hive作业,如何使用yarn resourcemanager。

    背景: 接到任务,需要在一个一天数据量在460亿条记录的hive表中,筛选出某些host为特定的值时才解析该条记录的http_content中的经纬度: 解析规则譬如: 需要解析host: api.m ...

  5. Hadoop yarn配置参数

    参照site:http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml 我们在配置yar ...

  6. Hadoop Yarn 安装

    环境:Linux, 8G 内存.60G 硬盘 , Hadoop 2.2.0 为了构建基于Yarn体系的Spark集群.先要安装Hadoop集群,为了以后查阅方便记录了我本次安装的详细步骤. 事前准备 ...

  7. 通过tarball形式安装HBASE Cluster(CDH5.0.2)——配置分布式集群中的YARN ResourceManager 的HA

    <?xml version="1.0"?> <!-- Licensed under the Apache License, Version 2.0 (the &q ...

  8. hadoop yarn HA集群搭建

    可先完成hadoop namenode HA的搭建:http://www.cnblogs.com/kisf/p/7458519.html 搭建yarnde HA只需要在namenode HA配置基础上 ...

  9. hadoop+yarn+hbase+storm+kafka+spark+zookeeper)高可用集群详细配置

    配置 hadoop+yarn+hbase+storm+kafka+spark+zookeeper 高可用集群,同时安装相关组建:JDK,MySQL,Hive,Flume 文章目录 环境介绍 节点介绍 ...

随机推荐

  1. Gradle 如何配置将编译的 JAR 发布到 Archiva 中

    有时候我们希望将我们的jar 开发包发布到 Archiva 中. 如何配置 Gradle 的编译脚本呢? 首先你需要启用 Gradle 的 Maven-publish 插件. plugins { id ...

  2. Windows Storage 驱动开发 葵花宝典 - 翻译

    Roadmap for Developing Windows Storage Drivers Last Updated: 4/20/2017   To create a storage driver, ...

  3. nodejs中http服务器,如何使用GET,POST请求发送数据、npm、以及一些插件的介绍

    浏览器给服务器传递参数,最常用的是地址栏传参(get),以及表单提交(post) 先说get传参,就是在url后跟上?key=value&key2=value2...... 但是按照前几篇的h ...

  4. 微信小程序之简单记账本开发记录(三)

    昨天已经编辑了主界面,在wxml文件中设置好跳转链接之后,就可以进行下一步的开发了 在pages中建立一个新的页面文件夹作为之后的支出页面 编辑后台,今天先搭建大致界面

  5. "数字经济"云安全共测大赛Web-Writeup

    gameapp 这题首先反编译apk,简单看了看代码,主要是有startgame和score两个api,然后用模拟器(手机登不上)安装apk抓了下包,数据经过了rsa加密,所以首先用python实现r ...

  6. Resend a Request by fiddler

    Resend a Request You can resend a request directly from the Sessions List, or save requests to resen ...

  7. Maven-Profile 环境隔离

    作用 快速切换不同的配置环境,比如开发时是连接的本地数据库,发布线上时是另外的数据库,每次编译打包时都要修该配置文件比较麻烦,这时就可以使用环境隔离了. 配置 本地(Local), 开发(Dev), ...

  8. uwsgi+nginx部署django项目

    1. 概念解析(wsgi协议,uwsgi协议,uWSGI) 参考:https://www.cnblogs.com/wspblog/p/8575101.html 1.1 现实世界的web请求: 1.2  ...

  9. Mac 平台安装MySQL

    Mac 平台安装MySQL   一.下载MySQL MySQL官网上https://dev.mysql.com/downloads/mysql/,下载Community Server版 出现如下界面, ...

  10. linux下如何使rtc设备注册为指定的设备文件/dev/rtc1?

    答: 通过设备树中的aliases节点来指定即可; 如某rtc设备的节点名为rtc@68,那么想让系统为该设备生成指定的设备文件/dev/rtc1,那么就在设备树的根节点中增加aliases节点,示例 ...