hadoop job -kill 调用的是CLI.java里面的job.killJob(); 这里会分几种情况,如果是能查询到状态是RUNNING的话,是直接向AppMaster发送kill请求的。
YARNRunner.java

  @Override
public void killJob(JobID arg0) throws IOException, InterruptedException {
/* check if the status is not running, if not send kill to RM */
JobStatus status = clientCache.getClient(arg0).getJobStatus(arg0);
ApplicationId appId = TypeConverter.toYarn(arg0).getAppId();
 
// get status from RM and return
if (status == null) {
killUnFinishedApplication(appId);
return;
}
 
if (status.getState() != JobStatus.State.RUNNING) {
killApplication(appId);
return;
}
 
try {
/* send a kill to the AM */
clientCache.getClient(arg0).killJob(arg0);
long currentTimeMillis = System.currentTimeMillis();
long timeKillIssued = currentTimeMillis;
while ((currentTimeMillis < timeKillIssued + 10000L)
&& !isJobInTerminalState(status)) {
try {
Thread.sleep(1000L);
} catch (InterruptedException ie) {
/** interrupted, just break */
break;
}
currentTimeMillis = System.currentTimeMillis();
status = clientCache.getClient(arg0).getJobStatus(arg0);
if (status == null) {
killUnFinishedApplication(appId);
return;
}
}
} catch(IOException io) {
LOG.debug("Error when checking for application status", io);
}
if (status != null && !isJobInTerminalState(status)) {
killApplication(appId);
}
}

MRClientService.java

    @SuppressWarnings("unchecked")
@Override
public KillJobResponse killJob(KillJobRequest request)
throws IOException {
JobId jobId = request.getJobId();
UserGroupInformation callerUGI = UserGroupInformation.getCurrentUser();
String message = "Kill job " + jobId + " received from " + callerUGI
+ " at " + Server.getRemoteAddress();
LOG.info(message);
verifyAndGetJob(jobId, JobACL.MODIFY_JOB);
appContext.getEventHandler().handle(
new JobDiagnosticsUpdateEvent(jobId, message));
appContext.getEventHandler().handle(
new JobEvent(jobId, JobEventType.JOB_KILL));
KillJobResponse response =
recordFactory.newRecordInstance(KillJobResponse.class);
return response;
}

yarn application -kill 使用的是ApplicationCLI.java,是向RM发送kill请求

/**
* Kills the application with the application id as appId
*
* @param applicationId
* @throws YarnException
* @throws IOException
*/
private void killApplication(String applicationId) throws YarnException,
IOException {
ApplicationId appId = ConverterUtils.toApplicationId(applicationId);
ApplicationReport appReport = null;
try {
appReport = client.getApplicationReport(appId);
} catch (ApplicationNotFoundException e) {
sysout.println("Application with id '" + applicationId +
"' doesn't exist in RM.");
throw e;
}
 
if (appReport.getYarnApplicationState() == YarnApplicationState.FINISHED
|| appReport.getYarnApplicationState() == YarnApplicationState.KILLED
|| appReport.getYarnApplicationState() == YarnApplicationState.FAILED) {
sysout.println("Application " + applicationId + " has already finished ");
} else {
sysout.println("Killing application " + applicationId);
client.killApplication(appId);
}
}

YarnClientImpl.java

  @Override
public void killApplication(ApplicationId applicationId)
throws YarnException, IOException {
KillApplicationRequest request =
Records.newRecord(KillApplicationRequest.class);
request.setApplicationId(applicationId);
 
try {
int pollCount = 0;
long startTime = System.currentTimeMillis();
 
while (true) {
KillApplicationResponse response =
rmClient.forceKillApplication(request);
if (response.getIsKillCompleted()) {
LOG.info("Killed application " + applicationId);
break;
}
 
long elapsedMillis = System.currentTimeMillis() - startTime;
if (enforceAsyncAPITimeout() &&
elapsedMillis >= this.asyncApiPollTimeoutMillis) {
throw new YarnException("Timed out while waiting for application " +
applicationId + " to be killed.");
}
 
if (++pollCount % 10 == 0) {
LOG.info("Waiting for application " + applicationId + " to be killed.");
}
Thread.sleep(asyncApiPollIntervalMillis);
}
} catch (InterruptedException e) {
LOG.error("Interrupted while waiting for application " + applicationId
+ " to be killed.");
}
}

hadoop job -kill 和 yarn application -kill 区别的更多相关文章

  1. hadoop job -kill 与 yarn application -kii(作业卡了或作业重复提交或MapReduce任务运行到running job卡住)

    问题详情  解决办法 [hadoop@master ~]$ hadoop job -kill job_1493782088693_0001 DEPRECATED: Use of this script ...

  2. yarn application -kill application_id yarn kill 超时任务脚本

    需求:kill 掉yarn上超时的任务,实现不同队列不同超时时间的kill机制,并带有任务名的白名单功能 此为python脚本,可配置crontab使用 # _*_ coding=utf-8 _*_ ...

  3. yarn application命令介绍

    yarn application 1.-list     列出所有 application 信息    示例:yarn  application -list 2.-appStates <Stat ...

  4. Linux kill、kill-15、kill-9区别

    进程状态转换图 kill和kill -9,两个命令在linux中都有杀死进程的效果,然而两命令的执行过程却大有不同,在程序中如果用错了,可能会造成莫名其妙的现象. 执行kill(不加 -* 默认kil ...

  5. Hadoop 新 MapReduce 框架 Yarn 详解【转】

    [转自:http://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop-yarn/] 简介: 本文介绍了 Hadoop 自 0.23.0 版本 ...

  6. spark-shell启动报错:Yarn application has already ended! It might have been killed or unable to launch application master

    spark-shell不支持yarn cluster,以yarn client方式启动 spark-shell --master=yarn --deploy-mode=client 启动日志,错误信息 ...

  7. yarn application ID 增长达到10000后

    Job, Task, and Task Attempt IDs In Hadoop 2, MapReduce job IDs are generated from YARN application I ...

  8. spark利用yarn提交任务报:YARN application has exited unexpectedly with state UNDEFINED

    spark用yarn提交任务会报ERROR cluster.YarnClientSchedulerBackend: YARN application has exited unexpectedly w ...

  9. 【深入浅出 Yarn 架构与实现】3-1 Yarn Application 流程与编写方法

    本篇学习 Yarn Application 编写方法,将带你更清楚的了解一个任务是如何提交到 Yarn ,在运行中的交互和任务停止的过程.通过了解整个任务的运行流程,帮你更好的理解 Yarn 运作方式 ...

随机推荐

  1. [原创]SOUI GDI+渲染引擎下的字体特效,抛砖引玉

    由于SOUI是一种双渲染引擎的DUI库,默认在SKIA渲染引擎下是支持特效字体的,具体请参考DEMO中的源码. 但是使用GDI+渲染时是没有这些特效的,着实比较苦恼,此代抛砖引玉,细节实现 请自己再去 ...

  2. 【#】Spring3 MVC (三)---请求处理方法 参数及返回值总结

    博客分类:  spring MVCJSPServletCC++ @RequestMapping("/xxxx") public String  aaa(){ } 在处理用户请求的方 ...

  3. [operator]ELK6 index pattern的问题

    完成了EL/FK的搭建之后,在kibana的主页只能看到默认的索引? 其实这个索引名字的设置是在logstash-smaple.conf(elk6.4)里的设置,比如我这样设置 input { bea ...

  4. webstorm最新破解方法

    方法来自 Rover12421 大神. 1.从官网下载WebStorm2016.1安装. 2.下载 破解补丁 并解压,记住路径 3.编辑WebStorm安装目录下 bin 文件夹中的 WebStorm ...

  5. charCodeAt方法以及Unicode中文汉字编码范围

    js的charCodeAt() 方法可返回指定位置的字符的 Unicode 编码.这个返回值是 0 - 65535 之间的整数. 在字符串 "Hello world!" 中,我们将 ...

  6. javascript总结21:javascript-JSON与遍历

    1 什么是JSON JavaScript Object Notation(JavaScript对象表示形式) JavaScript的子集 JSON和对象字面量的区别 JSON的属性必须用双引号引号引起 ...

  7. 编写高质量代码改善C#程序的157个建议——建议125:避免用FCL的类型名称命名自己的类型

    建议125:避免用FCL的类型名称命名自己的类型 试想过自己写一个Socket类型吗?如果没有,我们来尝试一下: public class Socket { //省略 } 把以上代码同某些其他工具类封 ...

  8. 多线程的那点儿事(之windows锁)

    在windows系统中,系统本身为我们提供了很多锁.通过这些锁的使用,一方面可以加强我们对锁的认识,另外一方面可以提高代码的性能和健壮性.常用的锁以下四种:临界区,互斥量,信号量,event. (1) ...

  9. 搭建python+PyQt+Eric平台

    搭建python+PyQt+Eric平台 预备安装程序: 2.1.下载Python3.2 官方网站:http://www.python.org/ 下载地址:http://www.python.org/ ...

  10. 简单介绍Java的静态分派和动态分派

    最近复习JVM的知识,对于静态分派和动态分派的理解有点混乱,于是自己尝试写写代码,在分析中巩固知识. 有如下一段代码,请问每一段分别输出什么? package com.khlin.my.test; c ...