搭建Spark环境后,调测Spark样例时,出现下面的错误:
WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

[hadoop@gpmaster bin]$ ./run-example org.apache.spark.examples.SparkPi
15/10/01 08:59:33 INFO spark.SparkContext: Running Spark version 1.5.0
.......................
15/10/01 08:59:35 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.1.128:17514]
15/10/01 08:59:35 INFO util.Utils: Successfully started service 'sparkDriver' on port 17514.
.......................
15/10/01 08:59:36 INFO ui.SparkUI: Started SparkUI at http://192.168.1.128:4040
15/10/01 08:59:37 INFO spark.SparkContext: Added JAR file:/home/hadoop/spark/lib/spark-examples-1.5.0-hadoop2.6.0.jar at http://192.168.1.128:36471/jars/spark-examples-1.5.0-hadoop2.6.0.jar with timestamp 1443661177865
15/10/01 08:59:37 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
15/10/01 08:59:38 INFO client.AppClient$ClientEndpoint: Connecting to master spark://192.168.1.128:7077...
15/10/01 08:59:38 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20151001085938-0000
.................................
15/10/01 08:59:40 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
15/10/01 08:59:55 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:00:10 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:00:25 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:00:40 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:00:55 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:01:10 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:01:25 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:01:40 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:01:55 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:02:10 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
15/10/01 09:02:25 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

从警告信息大致可以知道:
初始化job时没有获取到任何资源;提示检查集群,确保workers可以被注册并有足够的内存资源。

可能的原因有几点,可以逐个排查:
1. 主机主机名和ip是否配置正确
先查看/etc/hosts文件配置是否正确

同时可以通过spark-shell查看SparkContext获取的上下文信息, 如下操作:
[hadoop@gpmaster bin]$ ./spark-shell
........
scala> sc.getConf.getAll.foreach(println)
(spark.fileserver.uri,http://192.168.1.128:34634)
(spark.app.name,Spark shell)
(spark.driver.port,25392)
(spark.app.id,app-20151001090322-0001)
(spark.repl.class.uri,http://192.168.1.128:24988)
(spark.externalBlockStore.folderName,spark-1254a794-fbfa-4b4c-9757-b5a94dc26ffc)
(spark.jars,)
(spark.executor.id,driver)
(spark.submit.deployMode,client)
(spark.driver.host,192.168.1.128)
(spark.master,spark://192.168.1.128:7077)

scala> sc.getConf.toDebugString
res8: String = 
spark.app.id=app-20151001090322-0001
spark.app.name=Spark shell
spark.driver.host=192.168.1.128
spark.driver.port=25392
spark.executor.id=driver
spark.externalBlockStore.folderName=spark-1254a794-fbfa-4b4c-9757-b5a94dc26ffc
spark.fileserver.uri=http://192.168.1.128:34634
spark.jars=
spark.master=spark://192.168.1.128:7077
spark.repl.class.uri=http://192.168.1.128:24988
spark.submit.deployMode=client

2. 内存不足
我的环境就是因为内存的原因。
我集群环境中,spark-env.sh 文件配置如下:
export JAVA_HOME=/usr/java/jdk1.7.0_60
export SCALA_HOME=/usr/local/scala
export SPARK_MASTER_IP=192.168.1.128
export SPARK_WORKER_MEMORY=100m
export HADOOP_CONF_DIR=/home/hadoop/hadoop-2.6.0/etc/hadoop
export MASTER=spark://192.168.1.128:7077

因为我的集群环境,每个节点只剩下500MB内存了,由于我没有配置SPARK_EXECUTOR_MEMORY参数,默认会使用1G内存,所以会出现内存不足,从而出现上面日志报的警告信息。

所以解决办法是添加如下参数:

export SPARK_EXECUTOR_MEMORY=512m

3.端口号被占用,之前的程序已运行。

Spark执行样例报警告:WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources的更多相关文章

  1. 18/03/18 04:53:44 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

    1:遇到这个问题是在启动bin/spark-shell以后,然后呢,执行spark实现wordcount的例子的时候出现错误了,如: scala> sc.textFile()).reduceBy ...

  2. Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

    Spark submit任务到Spark集群时,会出现如下异常: Exception 1:Initial job has not accepted any resources; check your ...

  3. azure iothub create-device-identity样例报错: unable to find valid certification path ,及iothub-explorer Error: CERT_UNTRUSTED

    https://docs.microsoft.com/zh-cn/azure/iot-hub/iot-hub-java-java-getstarted 在IDEA中执行上述的代码,会出现下面的报错信息 ...

  4. Pytest执行用例报Hint: make sure your test modules/packages have valid Python names.

    近日,使用Pytest+Appium 实现APP端UI自动化,遇到Pytest收集用例失败的情况. 报错信息如下: test_room.py:None (test_room.py) ImportErr ...

  5. Eureka 的 Application Service client的注冊以及执行演示样例

            Eureka 服务器架起来了(关于架设步骤參考博客<Linux 下 Eureka 服务器的部署>),如今怎样把我们要负载均衡的服务器(也就是从 Application Cl ...

  6. spark mllib lda 中文分词、主题聚合基本样例

    github https://github.com/cclient/spark-lda-example spark mllib lda example 官方示例较为精简 在官方lda示例的基础上,给合 ...

  7. Android OpenCV样例调试+报错处理

    1.OpenCV样例调试:<OpenCV Sample - image-manipulations>       blog+报错:E/CAMERA_ACTIVITY(17665): Cam ...

  8. Eureka 的 Application Client client的执行演示样例

            上篇以一个 demo 演示样例介绍了 Eureka 的 Application Service 客户端角色.今天我们继续了解 Eureka 的 Application Client 客 ...

  9. 在Ubuntu下构建Bullet以及执行Bullet的样例程序

    在Ubuntu下构建Bullet以及执行Bullet的样例程序 1.找到Bullet的下载页,地址是:https://code.google.com/p/bullet/downloads/list 2 ...

随机推荐

  1. js 捕获型事件

    true 为捕获型事件 false 为冒泡型事件

  2. linux中的&&和&,|和||

    在linux中,&和&&,|和||介绍如下: &  表示任务在后台执行,如要在后台运行redis-server,则有  python a.py & && ...

  3. mui app在线更新

    一参考资料 二代码 HTML代码 CSS代码 JS代码 接口代码 一.参考资料 http://ask.dcloud.net.cn/article/182 二.代码 1. HTML代码 <div ...

  4. Java 基于javaMail的邮件发送(支持附件)

    基于JavaMail的Java邮件发送Author xiuhong.chen@hand-china.com Desc 简单邮件发送 Date 2017/12/8 项目中需要根据物料资质的状况实时给用户 ...

  5. Hive 表结构操作

    添加列  add columns alter table table_name add columns (id int comment '主键ID' ) ; 默认在表所有字段之后,分区字段之前. 替换 ...

  6. python脚本10_打印斐波那契数列的第101项

    #打印斐波那契数列的第101项 a = 1 b = 1 for count in range(99): a,b = b,a+b else: print(b) 方法2: #打印斐波那契数列的第101项 ...

  7. 值类型的TryParse

    值类型(Struct(如:DateTime).基本类型(如:double).枚举类型)的TryParse方法,通常可使用该方法将“字符串”转换为当前类型,并out出.比如:日期格式的字符串   转换为 ...

  8. 使用python编写微信跳一跳的自动脚本

    实现思路: 调用adb命令,截图 寻找小小人的底部中心点role(从下到上扫描,直到找到小小人相同像素的点,至于小小人像素点rgb是什么,可以使用photoshop查看) 寻找棋盘最高点top,然后寻 ...

  9. 二十三 Python分布式爬虫打造搜索引擎Scrapy精讲—craw母版l创建自动爬虫文件—以及 scrapy item loader机制

    用命令创建自动爬虫文件 创建爬虫文件是根据scrapy的母版来创建爬虫文件的 scrapy genspider -l  查看scrapy创建爬虫文件可用的母版 Available templates: ...

  10. python:input()和raw_input()

    1.input() 接受各种合法类型的输入,比如输入字符串,则需要使用双引号,否则报错. input()会自动判断类型,比如输入的是 1.1,则返回的类型是float. 示例: 2.raw_input ...