记录一次 hadoop yarn resourceManager无故切换的故障
某日 收到告警 线上集群rm切换 观察resourcemanager 日志报错如下
这行不明显 再看看其他日志报错
在 app attempt_removed 时候发生了空指针错误
break;
case APP_ATTEMPT_REMOVED:
if (!(event instanceof AppAttemptRemovedSchedulerEvent)) {
throw new RuntimeException("Unexpected event type: " + event);
}
AppAttemptRemovedSchedulerEvent appAttemptRemovedEvent =
(AppAttemptRemovedSchedulerEvent) event;
removeApplicationAttempt(
appAttemptRemovedEvent.getApplicationAttemptID(),
appAttemptRemovedEvent.getFinalAttemptState(),
appAttemptRemovedEvent.getKeepContainersAcrossAppAttempts());
break;
case CONTAINER_EXPIRED:
定位到代码问题在这里 新增标记内容
private void removeApplicationAttempt(
ApplicationAttemptId applicationAttemptId,
RMAppAttemptState rmAppAttemptFinalState, boolean keepContainers) {
LOG.info("Application " + applicationAttemptId + " is done." +
" finalState=" + rmAppAttemptFinalState);
try {
writeLock.lock();
SchedulerApplication<FSAppAttempt> application =
applications.get(applicationAttemptId.getApplicationId());
FSAppAttempt attempt = getSchedulerApp(applicationAttemptId); if (attempt == null || application == null) {
LOG.info("Unknown application " + applicationAttemptId + " has completed!");
return;
}
//已经停止了就不用再次停止了 新增
// Check if the attempt is already stopped and don't stop it twice.
if (attempt.isStopped()) {
LOG.info("Application " + applicationAttemptId + " has already been "
+ "stopped!");
return;
} // Release all the running containers
for (RMContainer rmContainer : attempt.getLiveContainers()) {
if (keepContainers
&& rmContainer.getState().equals(RMContainerState.RUNNING)) {
// do not kill the running container in the case of work-preserving AM
// restart.
LOG.info("Skip killing " + rmContainer.getContainerId());
continue;
}
super.completedContainer(rmContainer,
SchedulerUtils.createAbnormalContainerStatus(
rmContainer.getContainerId(),
SchedulerUtils.COMPLETED_APPLICATION),
RMContainerEventType.KILL);
}
增加如下代码
@Override
public String moveApplication(ApplicationId appId,
String queueName) throws YarnException {
try {
writeLock.lock();
SchedulerApplication<FSAppAttempt> app = applications.get(appId);
if (app == null) {
throw new YarnException("App to be moved " + appId + " not found.");
}
FSAppAttempt attempt = (FSAppAttempt) app.getCurrentAppAttempt();
// To serialize with FairScheduler#allocate, synchronize on app attempt
try {
attempt.getWriteLock().lock();
FSLeafQueue oldQueue = (FSLeafQueue) app.getQueue();
// Check if the attempt is already stopped: don't move stopped app
// attempt. The attempt has already been removed from all queues.
if (attempt.isStopped()) {
LOG.info("Application " + appId + " is stopped and can't be moved!"throw new YarnException("Application " ++ " is stopped and can't be moved!");
}
String destQueueName = handleMoveToPlanQueue(queueName);
FSLeafQueue targetQueue = queueMgr.getLeafQueue(destQueueName, false);
if (targetQueue == null) {
throw new YarnException("Target queue " + queueName
+ " not found or is not a leaf queue.");
}
if (targetQueue == oldQueue) {
return oldQueue.getQueueName();
}
private void executeMove(SchedulerApplication<FSAppAttempt> app,
FSAppAttempt attempt, FSLeafQueue oldQueue, FSLeafQueue newQueue) {
// Check current runs state. Do not remove the attempt from the queue until
// after the check has been performed otherwise it could remove the app
// from a queue without moving it to a new queue.
boolean wasRunnable = oldQueue.isRunnableApp(attempt);
// if app was not runnable before, it may be runnable now
boolean nowRunnable = maxRunningEnforcer.canAppBeRunnable(newQueue,
attempt.getUser());
if (wasRunnable && !nowRunnable) {
throw new IllegalStateException("Should have already verified that app "
+ attempt.getApplicationId() + " would be runnable in new queue");
} // Now it is safe to remove from the queue.
oldQueue.removeApp(attempt); if (wasRunnable) {
maxRunningEnforcer.untrackRunnableApp(attempt);
} else if (nowRunnable) {
// App has changed from non-runnable to runnable
maxRunningEnforcer.untrackNonRunnableApp(attempt);
} attempt.move(newQueue); // This updates all the metrics
app.setQueue(newQueue);
newQueue.addApp(attempt, nowRunnable); if (nowRunnable) {
maxRunningEnforcer.trackRunnableApp(attempt);
}
if (wasRunnable) {
maxRunningEnforcer.updateRunnabilityOnAppRemoval(attempt, oldQueue);
}
}
问题解决
参考 https://issues.apache.org/jira/secure/attachment/12841441/YARN-5136.2.patch
记录一次 hadoop yarn resourceManager无故切换的故障的更多相关文章
- 记录一次线上yarn RM频繁切换的故障
周末一大早被报警惊醒,rm频繁切换 急急忙忙排查 看到两处错误日志 错误信息1 ervation <memory:0, vCores:0> 2019-12-21 11:51:57,781 ...
- Hadoop记录-yarn ResourceManager Active频繁易主问题排查(转载)
一.故障现象 两个节点的ResourceManger频繁在active和standby角色中切换.不断有active易主的告警发出 许多任务的状态没能成功更新,导致一些任务状态卡在NEW_SAVING ...
- Hadoop官方文档翻译—— YARN ResourceManager High Availability 2.7.3
ResourceManager High Availability (RM高可用) Introduction(简介) Architecture(架构) RM Failover(RM 故障切换) Rec ...
- Spark&Hive:如何使用scala开发spark访问hive作业,如何使用yarn resourcemanager。
背景: 接到任务,需要在一个一天数据量在460亿条记录的hive表中,筛选出某些host为特定的值时才解析该条记录的http_content中的经纬度: 解析规则譬如: 需要解析host: api.m ...
- Hadoop yarn配置参数
参照site:http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml 我们在配置yar ...
- Hadoop Yarn 安装
环境:Linux, 8G 内存.60G 硬盘 , Hadoop 2.2.0 为了构建基于Yarn体系的Spark集群.先要安装Hadoop集群,为了以后查阅方便记录了我本次安装的详细步骤. 事前准备 ...
- 通过tarball形式安装HBASE Cluster(CDH5.0.2)——配置分布式集群中的YARN ResourceManager 的HA
<?xml version="1.0"?> <!-- Licensed under the Apache License, Version 2.0 (the &q ...
- hadoop yarn HA集群搭建
可先完成hadoop namenode HA的搭建:http://www.cnblogs.com/kisf/p/7458519.html 搭建yarnde HA只需要在namenode HA配置基础上 ...
- hadoop+yarn+hbase+storm+kafka+spark+zookeeper)高可用集群详细配置
配置 hadoop+yarn+hbase+storm+kafka+spark+zookeeper 高可用集群,同时安装相关组建:JDK,MySQL,Hive,Flume 文章目录 环境介绍 节点介绍 ...
随机推荐
- Gradle 如何配置将编译的 JAR 发布到 Archiva 中
有时候我们希望将我们的jar 开发包发布到 Archiva 中. 如何配置 Gradle 的编译脚本呢? 首先你需要启用 Gradle 的 Maven-publish 插件. plugins { id ...
- Windows Storage 驱动开发 葵花宝典 - 翻译
Roadmap for Developing Windows Storage Drivers Last Updated: 4/20/2017 To create a storage driver, ...
- nodejs中http服务器,如何使用GET,POST请求发送数据、npm、以及一些插件的介绍
浏览器给服务器传递参数,最常用的是地址栏传参(get),以及表单提交(post) 先说get传参,就是在url后跟上?key=value&key2=value2...... 但是按照前几篇的h ...
- 微信小程序之简单记账本开发记录(三)
昨天已经编辑了主界面,在wxml文件中设置好跳转链接之后,就可以进行下一步的开发了 在pages中建立一个新的页面文件夹作为之后的支出页面 编辑后台,今天先搭建大致界面
- "数字经济"云安全共测大赛Web-Writeup
gameapp 这题首先反编译apk,简单看了看代码,主要是有startgame和score两个api,然后用模拟器(手机登不上)安装apk抓了下包,数据经过了rsa加密,所以首先用python实现r ...
- Resend a Request by fiddler
Resend a Request You can resend a request directly from the Sessions List, or save requests to resen ...
- Maven-Profile 环境隔离
作用 快速切换不同的配置环境,比如开发时是连接的本地数据库,发布线上时是另外的数据库,每次编译打包时都要修该配置文件比较麻烦,这时就可以使用环境隔离了. 配置 本地(Local), 开发(Dev), ...
- uwsgi+nginx部署django项目
1. 概念解析(wsgi协议,uwsgi协议,uWSGI) 参考:https://www.cnblogs.com/wspblog/p/8575101.html 1.1 现实世界的web请求: 1.2 ...
- Mac 平台安装MySQL
Mac 平台安装MySQL 一.下载MySQL MySQL官网上https://dev.mysql.com/downloads/mysql/,下载Community Server版 出现如下界面, ...
- linux下如何使rtc设备注册为指定的设备文件/dev/rtc1?
答: 通过设备树中的aliases节点来指定即可; 如某rtc设备的节点名为rtc@68,那么想让系统为该设备生成指定的设备文件/dev/rtc1,那么就在设备树的根节点中增加aliases节点,示例 ...