spark报错处理

Spark报错处理

1、问题：org.apache.spark.SparkException: Exception thrown in awaitResult

分析：出现这个情况的原因是spark启动的时候设置的是hostname启动的，导致访问的时候DNS不能解析主机名导致。

问题解决：

第一种方法：确保URL是spark://服务器ip:7077，而不是spark://hostname:7077；启动的时候指定-h  ip地址

第二种方法：修改主机的host文件添加主机的解析记录（推荐这种方式）

Ip     主机名

第三种方法：hive.metastore.try.direct.sql: false         (in hive-site.xml)

2、spark2.x版本使用hive，即copy一份hive-site.xml文件到spark2.x的conf目录下。

使用spark的bin目录下的spark-sql进入终端时总提示一个warning：

Thu Jun 15 12:56:05 CST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

解决方法：

修改hive-site.xml文件下的mysql连接的url，设置useSSL=false。由于hive-site.xml文件采用的是xml格式，所以不支持直接使用&连接，需要使用&进行连接。

<value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true&amp;useSSL=false</value>

 

重启spark即可，

#../sbin/stop-all.sh

#../sbin/start-all.sh

 

 

3、 问题：

Spark运行了一段时间，数据量上来以后，出现了一个这样的报错：

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

   at java.lang.Thread.run(Thread.java:745)

17/10/26 20:29:00 ERROR Executor: Exception in task 39.1 in stage 8.0 (TID 1122)

java.io.FileNotFoundException: /tmp/spark-2de5fa03-a7cb-47a2-9540-403de85d0371/executor-eebecccb-4cdb-4b85-80a3-73c4baa4c7bd/blockmgr-fc644c14-23e8-401c-aee8-00bc108bf607/2b/temp_shuffle_75eb7338-be41-41b4-bed4-5dcb0c1d0fdf (No space left on device)

   at java.io.FileOutputStream.open0(Native Method)

   at java.io.FileOutputStream.open(FileOutputStream.java:270)

   at java.io.FileOutputStream.<init>(FileOutputStream.java:213)

   at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:102)

   at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)

   at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)

   at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151)

   at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)

   at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)

   at org.apache.spark.scheduler.Task.run(Task.scala:108)

   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)

   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

 

从日志报错来看说是没有空间了，spark默认是把临时文件存放到/tmp目录下。需要修改啊！！！放到一个大存储的地方：

 

解决方法：

修改spark-env.sh

export SPARK_DRIVER_MEMORY=5g

export SPARK_LOCAL_DIRS=/data/sparktmp

 

不要添加到spark-defaault.conf里面去，因为spark从1.0版本已经放弃了spark.local.dir参数。

 

源码分析：

（1） DiskBlockManager类中的下面的方法

通过日志我们最终定位这块出现的错误

/**

* Create local directories for storing block data. These directories are

* located inside configured local directories and won't

* be deleted on JVM exit when using the external shuffle service.

*/

private def createLocalDirs(conf: SparkConf): Array[File] = {

Utils.getConfiguredLocalDirs(conf).flatMap { rootDir =>

try {

val localDir = Utils.createDirectory(rootDir, "blockmgr")

logInfo(s"Created local directory at $localDir")

Some(localDir)

} catch {

case e: IOException =>

logError(s"Failed to create local dir in $rootDir. Ignoring this directory.", e)

None

}

}

}

（2） SparkConf.scala 类中的方法

这个方法告诉我们在spark-defaults.conf 中配置spark.local.dir参数在spark1.0 版本后已经过时。

/** Checks for illegal or deprecated config settings. Throws an exception for the former. Not

* idempotent - may mutate this conf object to convert deprecated settings to supported ones. */

private[spark] def validateSettings() {

if (contains("spark.local.dir")) {

val msg = "In Spark 1.0 and later spark.local.dir will be overridden by the value set by " +

"the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN)."

logWarning(msg)

}

val executorOptsKey = "spark.executor.extraJavaOptions"

val executorClasspathKey = "spark.executor.extr

。。。。

}

（3）Utils.scala 类中的方法

通过分析下面的代码，我们发现不在spark-env.sh 下配置SPARK_LOCAL_DIRS的情况下，

通过该conf.get("spark.local.dir", System.getProperty("java.io.tmpdir")).split(",")设置spark.local.dir，然后或根据路径创建，导致上述错误。

故我们直接在spark-env.sh 中设置SPARK_LOCAL_DIRS 即可解决。

然后我们直接在spark-env.sh 中配置：

export SPARK_LOCAL_DIRS=/home/hadoop/data/sparktmp

/**

* Return the configured local directories where Spark can write files. This

* method does not create any directories on its own, it only encapsulates the

* logic of locating the local directories according to deployment mode.

*/

def getConfiguredLocalDirs(conf: SparkConf): Array[String] = {

val shuffleServiceEnabled = conf.getBoolean("spark.shuffle.service.enabled", false)

if (isRunningInYarnContainer(conf)) {

// If we are in yarn mode, systems can have different disk layouts so we must set it

// to what Yarn on this system said was available. Note this assumes that Yarn has

// created the directories already, and that they are secured so that only the

// user has access to them.

getYarnLocalDirs(conf).split(",")

} else if (conf.getenv("SPARK_EXECUTOR_DIRS") != null) {

conf.getenv("SPARK_EXECUTOR_DIRS").split(File.pathSeparator)

} else if (conf.getenv("SPARK_LOCAL_DIRS") != null) {

conf.getenv("SPARK_LOCAL_DIRS").split(",")

} else if (conf.getenv("MESOS_DIRECTORY") != null && !shuffleServiceEnabled) {

// Mesos already creates a directory per Mesos task. Spark should use that directory

// instead so all temporary files are automatically cleaned up when the Mesos task ends.

// Note that we don't want this if the shuffle service is enabled because we want to

// continue to serve shuffle files after the executors that wrote them have already exited.

Array(conf.getenv("MESOS_DIRECTORY"))

} else {

if (conf.getenv("MESOS_DIRECTORY") != null && shuffleServiceEnabled) {

logInfo("MESOS_DIRECTORY available but not using provided Mesos sandbox because " +

"spark.shuffle.service.enabled is enabled.")

}

// In non-Yarn mode (or for the driver in yarn-client mode), we cannot trust the user

// configuration to point to a secure directory. So create a subdirectory with restricted

// permissions under each listed directory.

conf.get("spark.local.dir", System.getProperty("java.io.tmpdir")).split(",")

}

}

3、Join condition is missing or trivial.Use the CROSS JOIN syntax to allow cartesian products between these relations.;

解决方法：

spark.sql.crossjoin.enabled: true

4、Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method "eval(Lorg/apache/spark/sql/catalyst/InternalRow;)Z" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate" grows beyond 64 KB

解决方法：

spark.sql.codegen.wholeStage : false

5、java.lang.OutOfMemoryError: Java heap space

解决方法：

spark.driver.memory : 10g   <to a higher-value>

spark.sql.ui.retainedExecutions: 5   <to some lower-value>