记录一次 hadoop yarn resourceManager无故切换的故障
某日 收到告警 线上集群rm切换 观察resourcemanager 日志报错如下
这行不明显 再看看其他日志报错
在 app attempt_removed 时候发生了空指针错误
break;
case APP_ATTEMPT_REMOVED:
if (!(event instanceof AppAttemptRemovedSchedulerEvent)) {
throw new RuntimeException("Unexpected event type: " + event);
}
AppAttemptRemovedSchedulerEvent appAttemptRemovedEvent =
(AppAttemptRemovedSchedulerEvent) event;
removeApplicationAttempt(
appAttemptRemovedEvent.getApplicationAttemptID(),
appAttemptRemovedEvent.getFinalAttemptState(),
appAttemptRemovedEvent.getKeepContainersAcrossAppAttempts());
break;
case CONTAINER_EXPIRED:
定位到代码问题在这里 新增标记内容
private void removeApplicationAttempt(
ApplicationAttemptId applicationAttemptId,
RMAppAttemptState rmAppAttemptFinalState, boolean keepContainers) {
LOG.info("Application " + applicationAttemptId + " is done." +
" finalState=" + rmAppAttemptFinalState);
try {
writeLock.lock();
SchedulerApplication<FSAppAttempt> application =
applications.get(applicationAttemptId.getApplicationId());
FSAppAttempt attempt = getSchedulerApp(applicationAttemptId); if (attempt == null || application == null) {
LOG.info("Unknown application " + applicationAttemptId + " has completed!");
return;
}
//已经停止了就不用再次停止了 新增
// Check if the attempt is already stopped and don't stop it twice.
if (attempt.isStopped()) {
LOG.info("Application " + applicationAttemptId + " has already been "
+ "stopped!");
return;
} // Release all the running containers
for (RMContainer rmContainer : attempt.getLiveContainers()) {
if (keepContainers
&& rmContainer.getState().equals(RMContainerState.RUNNING)) {
// do not kill the running container in the case of work-preserving AM
// restart.
LOG.info("Skip killing " + rmContainer.getContainerId());
continue;
}
super.completedContainer(rmContainer,
SchedulerUtils.createAbnormalContainerStatus(
rmContainer.getContainerId(),
SchedulerUtils.COMPLETED_APPLICATION),
RMContainerEventType.KILL);
}
增加如下代码
@Override
public String moveApplication(ApplicationId appId,
String queueName) throws YarnException {
try {
writeLock.lock();
SchedulerApplication<FSAppAttempt> app = applications.get(appId);
if (app == null) {
throw new YarnException("App to be moved " + appId + " not found.");
}
FSAppAttempt attempt = (FSAppAttempt) app.getCurrentAppAttempt();
// To serialize with FairScheduler#allocate, synchronize on app attempt
try {
attempt.getWriteLock().lock();
FSLeafQueue oldQueue = (FSLeafQueue) app.getQueue();
// Check if the attempt is already stopped: don't move stopped app
// attempt. The attempt has already been removed from all queues.
if (attempt.isStopped()) {
LOG.info("Application " + appId + " is stopped and can't be moved!"throw new YarnException("Application " ++ " is stopped and can't be moved!");
}
String destQueueName = handleMoveToPlanQueue(queueName);
FSLeafQueue targetQueue = queueMgr.getLeafQueue(destQueueName, false);
if (targetQueue == null) {
throw new YarnException("Target queue " + queueName
+ " not found or is not a leaf queue.");
}
if (targetQueue == oldQueue) {
return oldQueue.getQueueName();
}
private void executeMove(SchedulerApplication<FSAppAttempt> app,
FSAppAttempt attempt, FSLeafQueue oldQueue, FSLeafQueue newQueue) {
// Check current runs state. Do not remove the attempt from the queue until
// after the check has been performed otherwise it could remove the app
// from a queue without moving it to a new queue.
boolean wasRunnable = oldQueue.isRunnableApp(attempt);
// if app was not runnable before, it may be runnable now
boolean nowRunnable = maxRunningEnforcer.canAppBeRunnable(newQueue,
attempt.getUser());
if (wasRunnable && !nowRunnable) {
throw new IllegalStateException("Should have already verified that app "
+ attempt.getApplicationId() + " would be runnable in new queue");
} // Now it is safe to remove from the queue.
oldQueue.removeApp(attempt); if (wasRunnable) {
maxRunningEnforcer.untrackRunnableApp(attempt);
} else if (nowRunnable) {
// App has changed from non-runnable to runnable
maxRunningEnforcer.untrackNonRunnableApp(attempt);
} attempt.move(newQueue); // This updates all the metrics
app.setQueue(newQueue);
newQueue.addApp(attempt, nowRunnable); if (nowRunnable) {
maxRunningEnforcer.trackRunnableApp(attempt);
}
if (wasRunnable) {
maxRunningEnforcer.updateRunnabilityOnAppRemoval(attempt, oldQueue);
}
}
问题解决
参考 https://issues.apache.org/jira/secure/attachment/12841441/YARN-5136.2.patch
记录一次 hadoop yarn resourceManager无故切换的故障的更多相关文章
- 记录一次线上yarn RM频繁切换的故障
周末一大早被报警惊醒,rm频繁切换 急急忙忙排查 看到两处错误日志 错误信息1 ervation <memory:0, vCores:0> 2019-12-21 11:51:57,781 ...
- Hadoop记录-yarn ResourceManager Active频繁易主问题排查(转载)
一.故障现象 两个节点的ResourceManger频繁在active和standby角色中切换.不断有active易主的告警发出 许多任务的状态没能成功更新,导致一些任务状态卡在NEW_SAVING ...
- Hadoop官方文档翻译—— YARN ResourceManager High Availability 2.7.3
ResourceManager High Availability (RM高可用) Introduction(简介) Architecture(架构) RM Failover(RM 故障切换) Rec ...
- Spark&Hive:如何使用scala开发spark访问hive作业,如何使用yarn resourcemanager。
背景: 接到任务,需要在一个一天数据量在460亿条记录的hive表中,筛选出某些host为特定的值时才解析该条记录的http_content中的经纬度: 解析规则譬如: 需要解析host: api.m ...
- Hadoop yarn配置参数
参照site:http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml 我们在配置yar ...
- Hadoop Yarn 安装
环境:Linux, 8G 内存.60G 硬盘 , Hadoop 2.2.0 为了构建基于Yarn体系的Spark集群.先要安装Hadoop集群,为了以后查阅方便记录了我本次安装的详细步骤. 事前准备 ...
- 通过tarball形式安装HBASE Cluster(CDH5.0.2)——配置分布式集群中的YARN ResourceManager 的HA
<?xml version="1.0"?> <!-- Licensed under the Apache License, Version 2.0 (the &q ...
- hadoop yarn HA集群搭建
可先完成hadoop namenode HA的搭建:http://www.cnblogs.com/kisf/p/7458519.html 搭建yarnde HA只需要在namenode HA配置基础上 ...
- hadoop+yarn+hbase+storm+kafka+spark+zookeeper)高可用集群详细配置
配置 hadoop+yarn+hbase+storm+kafka+spark+zookeeper 高可用集群,同时安装相关组建:JDK,MySQL,Hive,Flume 文章目录 环境介绍 节点介绍 ...
随机推荐
- 二进制学习——Blob,ArrayBuffer、File、FileReader和FormData的区别
前言: Blob.ArrayBuffer.File.fileReader.formData这些名词总是经常看到,知道一点又好像不知道,像是同一个东西好像又不是,总是模模糊糊,最近终于下决心要弄清楚. ...
- 【概率论】4-5:均值和中值(The Mean and the Median)
title: [概率论]4-5:均值和中值(The Mean and the Median) categories: - Mathematic - Probability keywords: - Me ...
- postgresql 一些操作
postgresql 对sql语句敏感的. 所以尽量标准化输入 #############查看版本信息 ############ 1.查看客户端版本 psql --version 1 2.查看服务器端 ...
- JS对象-不可扩展对象、密封对象、冻结对象
不可扩展对象 Object.preventExtensions() 仅阻止添加自身的属性.但属性仍然可以添加到对象原型. 可以用 Object.isExtensible(obj) 来判断对象是否可扩展 ...
- CISCO实验记录七:OSPF
一.要求 1.使用OSPF创建路由表 2.查看OSPF邻居 二.实现 1.使用OSPF创建路由表 #router ospf 1 #network 192.168.1.0 0.0.0.255 area ...
- memcached出现:Fatal error: Call to undefined method Memcached::connect()
今天安装了memcached的服务端和客户端 装好试了一下 $mem = new Memcached; $mem -> connect("127.0.0.1",11211) ...
- pip 安装报错
pip3 install uwsgi 报错 Command in /tmp/pip-build-5m77h_mm/uwsgi/ yum -y install python36-devel 解决
- uwsgi+nginx部署django项目
1. 概念解析(wsgi协议,uwsgi协议,uWSGI) 参考:https://www.cnblogs.com/wspblog/p/8575101.html 1.1 现实世界的web请求: 1.2 ...
- Dart 语法中文在线学习网址收藏
为了学习flutter UI框架,必须先学好dart语言,故收藏了有关 Dart 语法中文在线学习网址 http://dart.goodev.org/guides/language/language- ...
- MySQL 临时表和复制表
MySQL 临时表在我们需要保存一些临时数据时是非常有用的.临时表只在当前连接可见,当关闭连接时,Mysql会自动删除表并释放所有空间. 临时表在MySQL 3.23版本中添加,如果你的MySQL版本 ...