Spark2.3（四十）：如何使用java通过yarn api调度spark app，并根据appId监控任务，关闭任务，获取任务日志
背景：
调研过OOZIE和AZKABA,这种都是只是使用spark-submit.sh来提交任务，任务提交上去之后获取不到ApplicationId，更无法跟踪spark application的任务状态，无法kill application，更无法获取application的日志信息。因此，为了实现一个spark的调度平台所以有了以下调研及测试结论。
调研目前流行的SPARK任务调度：Oozie和Azkaban。
但是这两个平台不能满足以下功能（这些功能是希望有的）：
1) 无法满足即安全（使用shell提交任务，操作用户权限控制）又可以Spark状态监控（跟踪SPARK application的任务状态）；
2) 无法满足监控集群运行状态；
3) 无法满足对每个任务设置监控策略。比如：任务假死状态判定。
一个合格的spark调度平台要具有的基本功能：可以submit，kill，监控，获取日志，跟踪历史记录。
本篇文章主要讲解如何使用YarnClient API实现，借助于YarnClient来实现监控任务，杀死任务，获取日志，使用org.apache.spark.deploy.yarn.Client提交spark任务并返回spark任务的applicationId。
备注：之前研究过使用SparkLauncher类进行调度，该方案也是一种不错的方案，如果读者你喜欢也可以尝试使用SparkLauncher，它一样可以提交后返回spark任务的applicationid（提交后无状态，需要等待applicaitonId不为空为止）。
环境配置：
1）由于我们是使用java 代码（需要发布到web项目中，而不是shell调用[不可以再shell中设置环境变量]）去调用，因此我们需要centos系统环境变量中包含以下变量：
SPARK_KAFKA_VERSION
HADOOP_HOME
HADOOP_COMMON_HOME
SPARK_HOME SPARK_CONF_DIR
HADOOP_CONF_DIR
YARN_CONF_DIR SPARK_DIST_CLASSPATH
SPARK_EXTRA_LIB_PATH
LD_LIBRARY_PATH
如果你对spark-env.sh文件比较熟悉的话，你会发现上边这些变量来自于该文件，那么，我们嗯只需要把spark-env.sh引入到/ect/profile就可以。
spark-env.sh
bash-4.1$ more /home1/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/conf/spark-env.sh
#!/usr/bin/env bash
##
# Generated by Cloudera Manager and should not be modified directly
## SELF="$(cd $(dirname $BASH_SOURCE) && pwd)"
if [ -z "$SPARK_CONF_DIR" ]; then
export SPARK_CONF_DIR="$SELF"
fi export SPARK_HOME=/home1/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2 SPARK_PYTHON_PATH=""
if [ -n "$SPARK_PYTHON_PATH" ]; then
export PYTHONPATH="$PYTHONPATH:$SPARK_PYTHON_PATH"
fi export HADOOP_HOME=/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/hadoop
export HADOOP_COMMON_HOME="$HADOOP_HOME" if [ -n "$HADOOP_HOME" ]; then
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native
fi SPARK_EXTRA_LIB_PATH="/home1/opt/cloudera/parcels/GPLEXTRAS-5.13.0-1.cdh5.13.0.p0.29/lib/hadoop/lib/native"
if [ -n "$SPARK_EXTRA_LIB_PATH" ]; then
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SPARK_EXTRA_LIB_PATH
fi export LD_LIBRARY_PATH HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-$SPARK_CONF_DIR/yarn-conf}
export HADOOP_CONF_DIR PYLIB="$SPARK_HOME/python/lib"
if [ -f "$PYLIB/pyspark.zip" ]; then
PYSPARK_ARCHIVES_PATH=
for lib in "$PYLIB"/*.zip; do
if [ -n "$PYSPARK_ARCHIVES_PATH" ]; then
PYSPARK_ARCHIVES_PATH="$PYSPARK_ARCHIVES_PATH,local:$lib"
else
PYSPARK_ARCHIVES_PATH="local:$lib"
fi
done
export PYSPARK_ARCHIVES_PATH
fi # Spark uses `set -a` to export all variables created or modified in this
# script as env vars. We use a temporary variables to avoid env var name
# collisions.
# If PYSPARK_PYTHON is unset, set to CDH_PYTHON
TMP_PYSPARK_PYTHON=${PYSPARK_PYTHON:-''}
# If PYSPARK_DRIVER_PYTHON is unset, set to CDH_PYTHON
TMP_PYSPARK_DRIVER_PYTHON=${PYSPARK_DRIVER_PYTHON:-} if [ -n "$TMP_PYSPARK_PYTHON" ] && [ -n "$TMP_PYSPARK_DRIVER_PYTHON" ]; then
export PYSPARK_PYTHON="$TMP_PYSPARK_PYTHON"
export PYSPARK_DRIVER_PYTHON="$TMP_PYSPARK_DRIVER_PYTHON"
fi # Add the Kafka jars configured by the user to the classpath.
SPARK_DIST_CLASSPATH=
SPARK_KAFKA_VERSION=${SPARK_KAFKA_VERSION:-'0.10'}
case "$SPARK_KAFKA_VERSION" in
0.9)
SPARK_DIST_CLASSPATH="$SPARK_HOME/kafka-0.9/*"
;;
0.10)
SPARK_DIST_CLASSPATH="$SPARK_HOME/kafka-0.10/*"
;;
None)
;;
*)
echo "Invalid Kafka version: $SPARK_KAFKA_VERSION"
exit 1
;;
esac export SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:$(paste -sd: "$SELF/classpath.txt")"
接下来在/ect/profile文件最后一样追加
source /home1/opt/cloudera/parcels/SPARK2-2.3..cloudera3-.cdh5.13.3.p0./lib/spark2/conf/spark-env.sh
，保存，然后source /etc/profile使其生效。
2）需要修改yarn上传资源文件存储位置，否则会出现错误找不到资源文件（文件之所以找不到，是因为那些资源文件spark_lib.zip,spark_conf.zip,*.jar被上传到本地的/curent_user[root、zhangsan、lisi]/.sparkStaging/{appId}/*.jar下，在其他executor|container上找不到），必须修改yarn资源文件上传到hdfs目录下：
第一步：提交任务代码中设置SparkConf变量：
sparkConf.set("spark.yarn.stagingDir", "hdfs://vm192.168.0.141.com.cn:8020/user/");
第二步：手动创建hdfs目录 /user/.sparkStaging，给分配权限：
bash-4.1$ sudo -uhdfs hadoop fs -mkdir /user/.sparkStaging
bash-4.1$ sudo -uhdfs hadoop fs -chown zhangsan:zhangsan /user/.sparkStaging
第三步：导入pom.xml依赖包
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<es.version>6.4.2</es.version>
<spark.version>2.3.0</spark.version>
<scala.version>2.11</scala.version>
</properties> <dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-yarn-client -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-client</artifactId>
<version>2.6.5</version>
</dependency> <!--Spark -->
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-yarn -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-yarn_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency> <dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql-kafka-0-10_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency> <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-launcher -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-launcher_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency> <dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.10.0.1</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.6</version>
</dependency>
<dependency>
<groupId>com.twitter</groupId>
<artifactId>bijection-avro_${scala.version}</artifactId>
<version>0.9.5</version>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-avro_${scala.version}</artifactId>
<version>3.2.0</version>
<type>jar</type>
</dependency> <dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-spark-20_${scala.version}</artifactId>
<version>${es.version}</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>${es.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.alibaba/fastjson -->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.54</version>
</dependency> <dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
spark提交任务：
参数类YarnSubmitConditions：
import java.util.List;
import java.util.Map; public class YarnSubmitConditions {
private List<String> otherArgs;
private String applicationJar;
private String mainClass;
private String appName;
private String[] additionalJars;
private String sparkYarnJars;
public String[] files;
public String yarnResourcemanagerAddress;
public String sparkFsDefaultFS;
private String driverMemory;
private String numExecutors;
private String executorMemory;
private String executorCores;
private String sparkHome;
private String deployMode;
private String master;
public Map<String, String> sparkProperties; public List<String> getOtherArgs() {
return otherArgs;
} public void setOtherArgs(List<String> otherArgs) {
this.otherArgs = otherArgs;
} public String getApplicationJar() {
return applicationJar;
} public void setApplicationJar(String applicationJar) {
this.applicationJar = applicationJar;
} public String getMainClass() {
return mainClass;
} public void setMainClass(String mainClass) {
this.mainClass = mainClass;
} public String getAppName() {
return appName;
} public void setAppName(String appName) {
this.appName = appName;
} public String[] getAdditionalJars() {
return additionalJars;
} public void setAdditionalJars(String[] additionalJars) {
this.additionalJars = additionalJars;
} public String getSparkYarnJars() {
return sparkYarnJars;
} public void setSparkYarnJars(String sparkYarnJars) {
this.sparkYarnJars = sparkYarnJars;
} public String[] getFiles() {
return files;
} public void setFiles(String[] files) {
this.files = files;
} public String getYarnResourcemanagerAddress() {
return yarnResourcemanagerAddress;
} public void setYarnResourcemanagerAddress(String yarnResourcemanagerAddress) {
this.yarnResourcemanagerAddress = yarnResourcemanagerAddress;
} public Map<String, String> getSparkProperties() {
return sparkProperties;
} public void setSparkProperties(Map<String, String> sparkProperties) {
this.sparkProperties = sparkProperties;
} public String getSparkFsDefaultFS() {
return sparkFsDefaultFS;
} public void setSparkFsDefaultFS(String sparkFsDefaultFS) {
this.sparkFsDefaultFS = sparkFsDefaultFS;
} public String getDriverMemory() {
return driverMemory;
} public void setDriverMemory(String driverMemory) {
this.driverMemory = driverMemory;
} public String getNumExecutors() {
return numExecutors;
} public void setNumExecutors(String numExecutors) {
this.numExecutors = numExecutors;
} public String getExecutorMemory() {
return executorMemory;
} public void setExecutorMemory(String executorMemory) {
this.executorMemory = executorMemory;
} public String getExecutorCores() {
return executorCores;
} public void setExecutorCores(String executorCores) {
this.executorCores = executorCores;
} public String getSparkHome() {
return sparkHome;
} public void setSparkHome(String sparkHome) {
this.sparkHome = sparkHome;
} public String getDeployMode() {
return deployMode;
} public void setDeployMode(String deployMode) {
this.deployMode = deployMode;
} public String getMaster() {
return master;
} public void setMaster(String master) {
this.master = master;
}
}
提交函数：
/**
* 提交任务到yarn集群
*
* @param conditions
* yarn集群，spark，hdfs具体信息，参数等
* @return appid
*/
public static String submitSpark(YarnSubmitConditions conditions) {
logger.info("初始化spark on yarn参数");
// 初始化yarn客户端
logger.info("初始化spark on yarn客户端"); List<String> args = Lists.newArrayList(//
"--jar", conditions.getApplicationJar(),//
"--class", conditions.getMainClass()//
);
if (conditions.getOtherArgs() != null && conditions.getOtherArgs().size() > 0) {
for (String s : conditions.getOtherArgs()) {
args.add("--arg");
args.add(org.apache.commons.lang.StringUtils.join(new String[] { s }, ","));
}
} // identify that you will be using Spark as YARN mode
System.setProperty("SPARK_YARN_MODE", "true"); System.out.println("SPARK_YARN_MODE:" + System.getenv("SPARK_YARN_MODE"));
System.out.println("SPARK_CONF_DIR:" + System.getenv("SPARK_CONF_DIR"));
System.out.println("HADOOP_CONF_DIR:" + System.getenv("HADOOP_CONF_DIR"));
System.out.println("YARN_CONF_DIR:" + System.getenv("YARN_CONF_DIR"));
System.out.println("SPARK_KAFKA_VERSION:" + System.getenv("SPARK_KAFKA_VERSION"));
System.out.println("HADOOP_HOME:" + System.getenv("HADOOP_HOME"));
System.out.println("HADOOP_COMMON_HOME:" + System.getenv("HADOOP_COMMON_HOME"));
System.out.println("SPARK_HOME:" + System.getenv("SPARK_HOME"));
System.out.println("SPARK_DIST_CLASSPATH:" + System.getenv("SPARK_DIST_CLASSPATH"));
System.out.println("SPARK_EXTRA_LIB_PATH:" + System.getenv("SPARK_EXTRA_LIB_PATH"));
System.out.println("LD_LIBRARY_PATH:" + System.getenv("LD_LIBRARY_PATH")); SparkConf sparkConf = new SparkConf(); sparkConf.setSparkHome(conditions.getSparkHome());
sparkConf.setMaster(conditions.getMaster());
sparkConf.set("spark.submit.deployMode", conditions.getDeployMode());
sparkConf.setAppName(conditions.getAppName()); // --driver-memory
sparkConf.set("spark.driver.memory", conditions.getDriverMemory());
// --executor-memory
sparkConf.set("spark.executor.memory", conditions.getExecutorMemory());
// --executor-cores
sparkConf.set("spark.executor.cores", conditions.getExecutorCores());
// --num-executors
sparkConf.set("spark.executor.instance", conditions.getNumExecutors());
// The folder '.sparkStaging' will be created auto.
// System.out.println("SPARK_YARN_STAGING_DIR:"+System.getenv("SPARK_YARN_STAGING_DIR"))
sparkConf.set("spark.yarn.stagingDir", "hdfs://vm192.168.0.141.com.cn:8020/user/");
// sparkConf.set("spark.jars",);
// sparkConf.set("spark.yarn.jars", conditions.getSparkYarnJars());
if (conditions.getAdditionalJars() != null && conditions.getAdditionalJars().length > 0) {
sparkConf.set("spark.repl.local.jars", org.apache.commons.lang.StringUtils.join(conditions.getAdditionalJars(), ","));
sparkConf.set("spark.yarn.dist.jars", org.apache.commons.lang.StringUtils.join(conditions.getAdditionalJars(), ","));
} // "--files","hdfs://node1:8020/user/root/yarn-site.xml",
if (conditions.getFiles() != null && conditions.getFiles().length > 0) {
sparkConf.set("spark.files", org.apache.commons.lang.StringUtils.join(conditions.getFiles(), ","));
} for (Map.Entry<String, String> e : conditions.getSparkProperties().entrySet()) {
sparkConf.set(e.getKey().toString(), e.getValue().toString());
} // mapred-site.xml
// 指定使用yarn框架
sparkConf.set("mapreduce.framework.name", "yarn");
// 指定historyserver
sparkConf.set("mapreduce.jobhistory.address", "vm192.168.0.141.com.cn:10020"); // yarn-site.xml
// 添加这个参数，不然spark会一直请求0.0.0.0:8030,一直重试
sparkConf.set("yarn.resourcemanager.hostname", conditions.getYarnResourcemanagerAddress().split(":")[0]);
// 指定资源分配器
sparkConf.set("yarn.resourcemanager.scheduler.address", "vm192.168.0.141.com.cn:8030");
// 设置为true，不删除缓存的jar包，因为现在提交yarn任务是使用的代码配置，没有配置文件，删除缓存的jar包有问题，
sparkConf.set("spark.yarn.preserve.staging.files", "false"); // spark2.2
// 初始化 yarn的配置
// Configuration cf = new Configuration();
// String cross_platform = "false";
// String os = System.getProperty("os.name");
// if (os.contains("Windows")) {
// cross_platform = "true";
// }
// 配置使用跨平台提交任务
// cf.set("mapreduce.app-submission.cross-platform", cross_platform);
// 设置yarn资源，不然会使用localhost:8032
// cf.set("yarn.resourcemanager.address",
// conditions.getYarnResourcemanagerAddress());
// 设置namenode的地址，不然jar包会分发，非常恶心
// cf.set("fs.defaultFS", conditions.getSparkFsDefaultFS()); // spark2.2
// Client client = new Client(cArgs, cf, sparkConf);
// spark2.3
ClientArguments cArgs = new ClientArguments(args.toArray(new String[args.size()]));
org.apache.spark.deploy.yarn.Client client = new Client(cArgs, sparkConf); logger.info("提交任务，任务名称：" + conditions.getAppName()); try {
ApplicationId appId = client.submitApplication();
return appId.toString();
} catch (Exception e) {
logger.error("提交spark任务失败", e);
return null;
} finally {
if (client != null) {
client.stop();
}
}
}
测试函数
private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(TestSubmit.class);
public static void main(String[] args) {
YarnSubmitConditions conditions = new YarnSubmitConditions();
conditions.setAppName("test yarn submit app");
conditions.setMaster("yarn");
conditions.setSparkHome("/home1/opt/cloudera/parcels/SPARK2/lib/spark2/");
conditions.setDeployMode("cluster");
conditions.setDriverMemory("3g");
conditions.setExecutorMemory("3g");
conditions.setExecutorCores("1");
conditions.setNumExecutors("5");
// /etc/hadoop/conf.cloudera.yarn/core-site.xml
conditions.setYarnResourcemanagerAddress("vm192.168.0.141.com.cn:8032");
// /etc/hadoop/conf.cloudera.yarn/yarn-site.xml
conditions.setSparkFsDefaultFS("hdfs://vm192.168.0.141.com.cn:8020");
conditions.setFiles(new String[] { "/etc/hadoop/conf.cloudera.yarn/hdfs-site.xml",//
"/etc/hadoop/conf.cloudera.yarn/mapred-site.xml",//
"/etc/hadoop/conf.cloudera.yarn/yarn-site.xml",//
});
conditions.setApplicationJar("/home1/zhangsan/mrs-streaming-driver.jar");
conditions.setMainClass("com.boco.mrs.streaming.Main");
conditions.setOtherArgs(Arrays.asList("RSRP", "TestBroadcastDriver"));
List<String> sparkJars = getSparkJars("/home1/zhangsan/sparkjars/");
conditions.setAdditionalJars(sparkJars.toArray(new String[sparkJars.size()]));
Map<String, String> propertiesMap = null;
try {
propertiesMap = getSparkProperties("/home1/zhangsan/conf/spark-properties-mrs.conf");
} catch (IOException e) {
e.printStackTrace();
}
conditions.setSparkProperties(propertiesMap);
String appId = submitSpark(conditions);
System.out.println("application id is " + appId);
System.out.println("Complete ....");
}
/**
* 加载sparkjars下的jar文件
* */
private static List<String> getSparkJars(String dir) {
List<String> items = new ArrayList<String>();
File file = new File(dir);
for (File item : file.listFiles()) {
items.add(item.getPath());
}
return items;
}
/**
* 加载spark-properties.conf配置文件
* */
private static Map<String, String> getSparkProperties(String filePath) throws IOException {
Map<String, String> propertiesMap = new HashMap<String, String>();
BufferedReader reader = new BufferedReader(new FileReader(filePath));
String line = null;
while ((line = reader.readLine()) != null) {
if (line.trim().length() > 0 && !line.startsWith("#") && line.indexOf("=") != -1) {
String[] fields = line.split("=");
propertiesMap.put(fields[0], fields[1]);
}
}
reader.close();
return propertiesMap;
}
测试函数执行脚本：
bash-4.1$ more test.sh
#/bin/sh
#LANG=zh_CN.utf8
#export LANG
export SPARK_KAFKA_VERSION=0.10
export LANG=zh_CN.UTF- java -cp ./sparkjars/*:./mrs-streaming-driver.jar com.dx.mrs.streaming.batchmodule.TestSubmit
执行日志：
bash-4.1$ ./test.sh
log4j:WARN No appenders could be found for logger (com.dx.mrs.streaming.batchmodule.TestSubmit).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
SPARK_YARN_MODE:null
SPARK_CONF_DIR:/home1/opt/cloudera/parcels/SPARK2-2.3..cloudera3-.cdh5.13.3.p0./lib/spark2/conf
HADOOP_CONF_DIR:/home1/opt/cloudera/parcels/SPARK2-2.3..cloudera3-.cdh5.13.3.p0./lib/spark2/conf/yarn-conf
YARN_CONF_DIR:null
SPARK_KAFKA_VERSION:0.10
HADOOP_HOME:/home1/opt/cloudera/parcels/CDH-5.13.-.cdh5.13.0.p0./lib/hadoop
HADOOP_COMMON_HOME:/home1/opt/cloudera/parcels/CDH-5.13.-.cdh5.13.0.p0./lib/hadoop
SPARK_HOME:/home1/opt/cloudera/parcels/SPARK2-2.3..cloudera3-.cdh5.13.3.p0./lib/spark2
SPARK_DIST_CLASSPATH:/home1/opt/cloudera/parcels/SPARK2-2.3..cloudera3-.cdh5.13.3.p0./lib/spark2/kafka-0.10/*:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/activation-1.1.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/aopalliance-1.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/apacheds-i18n-2.0.0-M15.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/apacheds-kerberos-codec-2.0.0-M15.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/api-asn1-api-1.0.0-M20.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/api-util-1.0.0-M20.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/asm-3.2.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/avro-1.7.6-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/aws-java-sdk-bundle-1.11.134.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/azure-data-lake-store-sdk-2.2.3.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/commons-beanutils-1.9.2.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/commons-beanutils-core-1.8.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/commons-codec-1.4.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/commons-configuration-1.6.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/commons-daemon-1.0.13.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/commons-digester-1.8.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/commons-el-1.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/commons-math3-3.1.1.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/commons-net-3.1.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/curator-client-2.7.1.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/curator-framework-2.7.1.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/curator-recipes-2.7.1.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/guava-11.0.2.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/guice-3.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-annotations-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-ant-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-archive-logs-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-archives-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-auth-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-aws-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-azure-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-azure-datalake-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-common-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-datajoin-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-distcp-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-extras-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-gridmix-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-hdfs-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-hdfs-nfs-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-mapreduce-client-app-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-mapreduce-client-common-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-mapreduce-client-core-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-mapreduce-client-hs-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-mapreduce-client-hs-plugins-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-mapreduce-client-nativetask-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-mapreduce-examples-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-nfs-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-openstack-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-rumen-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-sls-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-streaming-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-yarn-api-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-yarn-applications-distributedshell-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-yarn-client-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-yarn-common-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-yarn-registry-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-yarn-server-common-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-yarn-server-nodemanager-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hadoop-yarn-server-web-proxy-2.6.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hamcrest-core-1.3.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/htrace-core4-4.0.1-incubating.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/httpclient-4.2.5.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/httpcore-4.2.5.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/hue-plugins-3.9.0-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/jackson-annotations-2.2.3.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/jackson-core-2.2.3.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/jackson-core-asl-1.8.8.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/jackson-databind-2.2.3.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/jackson-mapper-asl-1.8.8.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/jasper-compiler-5.5.23.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/jasper-runtime-5.5.23.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/java-xmlbuilder-0.4.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/javax.inject-1.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/jaxb-api-2.2.2.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/jaxb-impl-2.2.3-1.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/jets3t-0.9.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/jettison-1.1.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/jline-2.11.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/jsch-0.1.42.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/jsr305-3.0.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/leveldbjni-all-1.8.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/log4j-1.2.17.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/metrics-core-3.0.2.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/microsoft-windowsazure-storage-sdk-0.6.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/mockito-all-1.8.5.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/netty-3.10.5.Final.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/okhttp-2.4.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/okio-1.4.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/paranamer-2.3.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/protobuf-java-2.5.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/slf4j-api-1.7.5.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/slf4j-log4j12-1.7.5.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/snappy-java-1.0.4.1.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/spark-1.6.0-cdh5.13.0-yarn-shuffle.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/stax-api-1.0-2.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/xercesImpl-2.9.1.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/xml-apis-1.3.04.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/xmlenc-0.52.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/jars/zookeeper-3.4.5-cdh5.13.0.jar:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/hadoop/LICENSE.txt:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/hadoop/NOTICE.txt:/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/hadoop/lib/jsp-api-2.1.jar:/home1/opt/cloudera/parcels/GPLEXTRAS-5.13.0-1.cdh5.13.0.p0.29/lib/hadoop/lib/COPYING.hadoop-lzo:/home1/opt/cloudera/parcels/GPLEXTRAS-5.13.0-1.cdh5.13.0.p0.29/lib/hadoop/lib/hadoop-lzo-0.4.15-cdh5.13.0.jar
SPARK_EXTRA_LIB_PATH:null
LD_LIBRARY_PATH::/home1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/hadoop/lib/native:/home1/opt/cloudera/parcels/GPLEXTRAS-5.13.0-1.cdh5.13.0.p0.29/lib/hadoop/lib/native
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/01/10 22:30:26 WARN SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.executor.memoryOverhead' instead.
19/01/10 22:30:27 INFO TestSubmit: 提交任务，任务名称：test yarn submit app
19/01/10 22:30:27 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/01/10 22:30:29 INFO Client: Requesting a new application from cluster with 6 NodeManagers
19/01/10 22:30:29 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (30282 MB per container)
19/01/10 22:30:29 INFO Client: Will allocate AM container, with 3456 MB memory including 384 MB overhead
19/01/10 22:30:29 INFO Client: Setting up container launch context for our AM
19/01/10 22:30:29 INFO Client: Setting up the launch environment for our AM container
19/01/10 22:30:29 INFO Client: Preparing resources for our AM container
19/01/10 22:30:34 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
19/01/10 22:30:36 INFO Client: Uploading resource file:/tmp/spark-03699598-b859-4a74-a65f-bc63e9fae733/__spark_libs__4116956896087694051.zip -> hdfs://vm192.168.0.141.com.cn:8020/user/.sparkStaging/application_1543820999543_0236/__spark_libs__4116956896087694051.zip
19/01/10 22:30:43 INFO Client: Uploading resource file:/home1/zhangsan/mrs-streaming-driver.jar -> hdfs://vm192.168.0.141.com.cn:8020/user/.sparkStaging/application_1543820999543_0236/mrs-streaming-driver.jar
19/01/10 22:31:33 INFO Client: Uploading resource file:/home1/zhangsan/sparkjars/elasticsearch-cli-6.4.2.jar -> hdfs://vm192.168.0.141.com.cn:8020/user/.sparkStaging/application_1543820999543_0236/elasticsearch-cli-6.4.2.jar
19/01/10 22:31:33 INFO Client: Uploading resource file:/home1/zhangsan/sparkjars/elasticsearch-6.4.2.jar -> hdfs://vm192.168.0.141.com.cn:8020/user/.sparkStaging/application_1543820999543_0236/elasticsearch-6.4.2.jar
......
19/01/10 22:31:33 INFO Client: Uploading resource file:/tmp/spark-03699598-b859-4a74-a65f-bc63e9fae733/__spark_conf__339930271770719398.zip -> hdfs://vm192.168.0.141.com.cn:8020/user/.sparkStaging/application_1543820999543_0236/__spark_conf__.zip
19/01/10 22:31:34 INFO SecurityManager: Changing view acls to: zhangsan
19/01/10 22:31:34 INFO SecurityManager: Changing modify acls to: zhangsan
19/01/10 22:31:34 INFO SecurityManager: Changing view acls groups to:
19/01/10 22:31:34 INFO SecurityManager: Changing modify acls groups to:
19/01/10 22:31:34 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(zhangsan); groups with view permissions: Set(); users with modify permissions: Set(zhangsan); groups with modify permissions: Set()
19/01/10 22:31:34 INFO Client: Submitting application application_1543820999543_0236 to ResourceManager
19/01/10 22:31:34 INFO YarnClientImpl: Submitted application application_1543820999543_0236
application id is application_1543820999543_0236
Complete ....
19/01/10 22:31:34 INFO ShutdownHookManager: Shutdown hook called
19/01/10 22:31:34 INFO ShutdownHookManager: Deleting directory /tmp/spark-03699598-b859-4a74-a65f-bc63e9fae733
bash-4.1$
目前调试通之后，测试通过yarn的cluster方式，client模式下任务提交到yarn上去无响应。
spark任务状态：
任务状态封装类
public class SparkTaskState{
private String appId;
private String state;
private float progress;
private String finalStatus;
public String getAppId() {
return appId;
}
public void setAppId(String appId) {
this.appId = appId;
}
public String getState() {
return state;
}
public void setState(String state) {
this.state = state;
}
public float getProgress() {
return progress;
}
public void setProgress(float progress) {
this.progress = progress;
}
public String getFinalStatus() {
return finalStatus;
}
public void setFinalStatus(String finalStatus) {
this.finalStatus = finalStatus;
}
}
/**
* 获取spark任务状态
*
* @param yarnResourcemanagerAddress
* yarn资源管理器地址， 例如：master:8032，查看yarn集群获取具体地址
* @param appIdStr
* 需要取消的任务id
*/
public static SparkTaskState getStatus(String yarnResourcemanagerAddress, String appIdStr) {
logger.info("获取任务状态启动，任务id：" + appIdStr); // 初始化 yarn的配置
Configuration cf = new Configuration(); boolean cross_platform = false;
String os = System.getProperty("os.name");
if (os.contains("Windows")) {
cross_platform = true;
}
cf.setBoolean("mapreduce.app-submission.cross-platform", cross_platform);// 配置使用跨平台提交任务 // 设置yarn资源，不然会使用localhost:8032
cf.set("yarn.resourcemanager.address", yarnResourcemanagerAddress); logger.info("获取任务状态，任务id:" + appIdStr);
SparkTaskState taskState = new SparkTaskState(); // 设置任务id
taskState.setAppId(appIdStr); YarnClient yarnClient = YarnClient.createYarnClient();
// 初始化yarn的客户端
yarnClient.init(cf);
// yarn客户端启动
yarnClient.start(); ApplicationReport report = null;
try {
report = yarnClient.getApplicationReport(getAppId(appIdStr));
} catch (Exception e) {
logger.error("获取spark任务状态失败");
} if (report != null) {
YarnApplicationState state = report.getYarnApplicationState();
taskState.setState(state.name()); // 任务执行进度
float progress = report.getProgress();
taskState.setProgress(progress); // 最终状态
FinalApplicationStatus status = report.getFinalApplicationStatus();
taskState.setFinalStatus(status.name());
} else {
taskState.setState("failed");
taskState.setProgress(0.0f);
taskState.setFinalStatus("failed");
} // 关闭yarn客户端
yarnClient.stop(); logger.info("获取任务状态结束，任务状态：" + JSON.toJSONString(taskState)); return taskState;
} private static ApplicationId getAppId(String appIdStr) {
return ConverterUtils.toApplicationId(appIdStr);
}
spark日志跟踪：
请参考《https://www.cnblogs.com/lyy-blog/p/9635601.html》
spark关闭任务：
/**
* 停止spark任务
*
* @param yarnResourcemanagerAddress
* yarn资源管理器地址， 例如：master:8032，查看yarn集群获取具体地址
* @param appIdStr
* 需要取消的任务id
*/
public static void killJob(String yarnResourcemanagerAddress, String appIdStr) {
logger.info("取消spark任务,任务id：" + appIdStr); // 初始化 yarn的配置
Configuration cf = new Configuration(); boolean cross_platform = false;
String os = System.getProperty("os.name");
if (os.contains("Windows")) {
cross_platform = true;
}
// 配置使用跨平台提交任务
cf.setBoolean("mapreduce.app-submission.cross-platform", cross_platform);
// 设置yarn资源，不然会使用localhost:8032
cf.set("yarn.resourcemanager.address", yarnResourcemanagerAddress); // 创建yarn的客户端，此类中有杀死任务的方法
YarnClient yarnClient = YarnClient.createYarnClient(); // 初始化yarn的客户端
yarnClient.init(cf); // yarn客户端启动
yarnClient.start(); try {
// 根据应用id，杀死应用
yarnClient.killApplication(getAppId(appIdStr));
} catch (Exception e) {
logger.error("取消spark任务失败", e);
} // 关闭yarn客户端
yarnClient.stop();
}
参考文章：https://blog.csdn.net/weixin_36647532/article/details/80766350
Spark2.3（四十）：如何使用java通过yarn api调度spark app，并根据appId监控任务，关闭任务，获取任务日志的更多相关文章
- Spark（四十四）：使用Java调用spark-submit.sh(支持 --deploy-mode client和cluster两种方式)并获取applicationId
之前也介绍过使用yarn api来submit spark任务,通过提交接口返回applicationId的用法,具体参考<Spark2.3(四十):如何使用java通过yarn api调度sp ...
- 《手把手教你》系列技巧篇（四十五）-java+ selenium自动化测试-web页面定位toast-上篇（详解教程）
1.简介 在使用appium写app自动化的时候介绍toast的相关元素的定位,在Web UI测试过程中,也经常遇到一些toast,那么这个toast我们这边如何进行测试呢?今天宏哥就分两篇介绍一下. ...
- 《手把手教你》系列技巧篇（四十六）-java+ selenium自动化测试-web页面定位toast-下篇（详解教程）
1.简介 终于经过宏哥的不懈努力,偶然发现了一个toast的web页面,所以直接就用这个页面来夯实一下,上一篇学过的知识-处理toast元素. 2.安居客 事先声明啊,宏哥没有收他们的广告费啊,纯粹是 ...
- 《手把手教你》系列技巧篇（四十八）-java+ selenium自动化测试-判断元素是否可操作（详解教程）
1.简介 webdriver有三种判断元素状态的方法,分别是isEnabled,isSelected 和 isDisplayed,其中isSelected在前面的内容中已经简单的介绍了,isSelec ...
- 《手把手教你》系列技巧篇（四十二）-java+ selenium自动化测试 - 处理iframe -下篇（详解教程）
1.简介 经过宏哥长时间的查找,终于找到了一个含有iframe的网页.所以今天这一篇的主要内容就是用这个网页的iframe,宏哥给小伙伴或者童鞋们演示一下,在处理过程中遇到的问题以及宏哥是如何解决的. ...
- 聊聊高并发（四十）解析java.util.concurrent各个组件（十六） ThreadPoolExecutor源代码分析
ThreadPoolExecutor是Executor运行框架最重要的一个实现类.提供了线程池管理和任务管理是两个最主要的能力.这篇通过分析ThreadPoolExecutor的源代码来看看怎样设计和 ...
- J2EE进阶(十四)超详细的Java后台开发面试题之Spring IOC与AOP
J2EE进阶(十四)超详细的Java后台开发面试题之Spring IOC与AOP 前言 搜狐畅游笔试题中有一道问答题涉及到回答谈谈对Spring IOC与AOP的理解.特将相关内容进行整理. ...
- JAVA之旅（三十四）——自定义服务端，URLConnection，正则表达式特点，匹配，切割，替换，获取，网页爬虫
JAVA之旅(三十四)--自定义服务端,URLConnection,正则表达式特点,匹配,切割,替换,获取,网页爬虫 我们接着来说网络编程,TCP 一.自定义服务端 我们直接写一个服务端,让本机去连接 ...
- Java进阶(四十)Java类、变量、方法修饰符讲解
Java进阶(四十)Java类.变量.方法修饰符讲解 Java类修饰符 abstract: 将一个类声明为抽象类,没有实现的方法,需要子类提供方法实现. final: 将一个类生命为最终(即非继承类) ...
随机推荐
- 51Nod1863 Travel 主席树 最短路 Dijkstra 哈希
原文链接https://www.cnblogs.com/zhouzhendong/p/51Nod1863.html 题目传送门 - 51Nod1863 题意 有 n 个城市,有 m 条双向路径连通它们 ...
- 简单的线程Thread使用
static void Main(string[] args) { for (int i = 0; i < 5; i++) { aa a = new aa(); a.age = i; Threa ...
- 关于ACtion类利用file类型取得上传文件的内容，名字和内容类型
上面的Action提供了两个属性:uploadFileName和uploadContentType,这个两个属性分别用于封转上传文件的文件名.上传文件的文件类型.Action类直接通过File类型属性 ...
- idea工具maven生命周期clean,install,package等区别
idea工具maven projects里面有9种生命周期,生命周期是包含在一个项目构建中的一系列有序的阶段. 一.最常用的两种打包方法: 1.clean,package(如果报错,很可能就是jar依 ...
- Zookeeper的概述、安装部署及选举机制
一.Zookeeper概述 1.Zookeeper是Hadoop生态的管理者,它致力于开发和维护开源服务器,实现高度可靠的分布式协调. 2.Zookeeper的两大功能: (1)存储数据 (2)监听 ...
- Django之视图函数总结
Django之视图函数总结 HttpRequest与HttpResponse http请求中产生两个核心对象: HttpRequest对象:用户请求相关的所有信息(对象) HttpResponse对象 ...
- JVM之对象分配：栈上分配 & TLAB分配
1. Java对象分配流程 2. 栈上分配 2.1 本质:Java虚拟机提供的一项优化技术 2.2 基本思想: 将线程私有的对象打散分配在栈上 2.3 优点: 2.3.1 可以在函数调用结束后自行销毁 ...
- java多线程之守护线程（Daemon）
https://blog.csdn.net/u010739551/article/details/51065923/
- mysql查询根据时间排序
表数据: mysql查询根据时间排序,如果有相同时间则只查询出来一个 所以需要再判断,如果时间相同,则根据id进行降序排序
- 一个不错的Node.js进阶学习引导
https://cnodejs.org/topic/58ad76db7872ea0864fedfcc