【异常】Reason: Executor heartbeat timed out after 140927 ms

1 详细异常

ERROR scheduler.JobScheduler: Error running job streaming job  ms.
org.apache.spark.SparkException: Job aborted due to stage failure: Task  in stage 0.0 failed  times,
 most recent failure: Lost task 0.3 in stage 0.0 (TID , , executor ): ExecutorLostFailure (executor  exited caused by one of the running tasks) Reason: Executor heartbeat timed out after  ms
Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$failJobAndIndependentStages(DAGScheduler.scala:)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$.apply(DAGScheduler.scala:)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$.apply(DAGScheduler.scala:)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$.apply(DAGScheduler.scala:)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$.apply(DAGScheduler.scala:)
    at scala.Option.foreach(Option.scala:)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:)
    at org.apache.spark.util.EventLoop$$anon$.run(EventLoop.scala:)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$.apply(RDD.scala:)
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$.apply(RDD.scala:)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:)
    at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:)
    at com.wm.bigdata.phoenix.etl.WmPhoniexEtlToHbase$$anonfun$main$.apply(WmPhoniexEtlToHbase.scala:)
    at com.wm.bigdata.phoenix.etl.WmPhoniexEtlToHbase$$anonfun$main$.apply(WmPhoniexEtlToHbase.scala:)
    at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$$$anonfun$apply$mcV$sp$.apply(DStream.scala:)
    at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$$$anonfun$apply$mcV$sp$.apply(DStream.scala:)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$$$anonfun$apply$mcV$sp$.apply$mcV$sp(ForEachDStream.scala:)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$$$anonfun$apply$mcV$sp$.apply(ForEachDStream.scala:)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$$$anonfun$apply$mcV$sp$.apply(ForEachDStream.scala:)
    at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$.apply$mcV$sp(ForEachDStream.scala:)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$.apply(ForEachDStream.scala:)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$.apply(ForEachDStream.scala:)
    at scala.util.Try$.apply(Try.scala:)
    at org.apache.spark.streaming.scheduler.Job.run(Job.scala:)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$.apply$mcV$sp(JobScheduler.scala:)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$.apply(JobScheduler.scala:)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$.apply(JobScheduler.scala:)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
    at java.lang.Thread.run(Thread.java:)

2 查询Stack Overflow里面问答

https://stackoverflow.com/questions/54036028/spark-executor-heartbeat-timed-out-after-x-ms

 

3 解决

提交spark submit任务的时候，加大超时时间设置

--conf spark.network.timeout  --conf spark.executor.heartbeatInterval=   --conf spark.driver.maxResultSize=4g