错误一：

// :: ERROR Executor: Exception in task 0.0 in stage 0.0 (TID )

java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V

    at Person.<init>(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame$.$anonfun$main$(RDD_To_DataFrame.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)

    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:)

    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$$$anon$.hasNext(WholeStageCodegenExec.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:)

    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:)

    at org.apache.spark.rdd.RDD.iterator(RDD.scala:)

    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:)

    at org.apache.spark.scheduler.Task.run(Task.scala:)

    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

// :: ERROR TaskSetManager: Task  in stage 0.0 failed  times; aborting job

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task  in stage 0.0 failed  times, most recent failure: Lost task 0.0 in stage 0.0 (TID , localhost, executor driver): java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V

    at Person.<init>(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame$.$anonfun$main$(RDD_To_DataFrame.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)

    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:)

    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$$$anon$.hasNext(WholeStageCodegenExec.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:)

    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:)

    at org.apache.spark.rdd.RDD.iterator(RDD.scala:)

    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:)

    at org.apache.spark.scheduler.Task.run(Task.scala:)

    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

Driver stacktrace:

    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$.apply(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$.apply(DAGScheduler.scala:)

    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:)

    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:)

    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$.apply(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$.apply(DAGScheduler.scala:)

    at scala.Option.foreach(Option.scala:)

    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:)

    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:)

    at org.apache.spark.util.EventLoop$$anon$.run(EventLoop.scala:)

    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:)

    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)

    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)

    at org.apache.spark.SparkContext.runJob(SparkContext.scala:)

    at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:)

    at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:)

    at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:)

    at org.apache.spark.sql.Dataset$$anonfun$head$.apply(Dataset.scala:)

    at org.apache.spark.sql.Dataset$$anonfun$head$.apply(Dataset.scala:)

    at org.apache.spark.sql.Dataset$$anonfun$.apply(Dataset.scala:)

    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:)

    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:)

    at org.apache.spark.sql.Dataset.head(Dataset.scala:)

    at org.apache.spark.sql.Dataset.take(Dataset.scala:)

    at org.apache.spark.sql.Dataset.showString(Dataset.scala:)

    at org.apache.spark.sql.Dataset.show(Dataset.scala:)

    at org.apache.spark.sql.Dataset.show(Dataset.scala:)

    at org.apache.spark.sql.Dataset.show(Dataset.scala:)

    at RDD_To_DataFrame$.main(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame.main(RDD_To_DataFrame.scala)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)

    at java.lang.reflect.Method.invoke(Method.java:)

    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:)

Caused by: java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V

    at Person.<init>(RDD_To_DataFrame.scala:)

    at RDD_To_DataFrame$.$anonfun$main$(RDD_To_DataFrame.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at scala.collection.Iterator$$anon$.next(Iterator.scala:)

    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)

    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:)

    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$$$anon$.hasNext(WholeStageCodegenExec.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.sql.execution.SparkPlan$$anonfun$.apply(SparkPlan.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$$$anonfun$apply$.apply(RDD.scala:)

    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:)

    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:)

    at org.apache.spark.rdd.RDD.iterator(RDD.scala:)

    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:)

    at org.apache.spark.scheduler.Task.run(Task.scala:)

    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

错误处理：将IDEA中的Scala 改为2.10.4版本

这个问题主要出现在 Spark程序使用 case class 类时

错误二：

Error:(, ) No TypeTag available for (Array[String],)

        val documentDF= spark.createDataFrame(Seq(

错误处理：将IDEA中的Scala 改为2.12.3版本

这个问题主要出现在 Spark程序使用 Seq时：

比如：

val df= spark.createDataFrame(Seq(

         (,Array("soyo","spark","soyo2","soyo","")),

         (,Array("soyo","hadoop","soyo","hadoop","xiaozhou","soyo2","spark","","")),

         (,Array("soyo","spark","soyo2","hadoop","soyo3","")),

         (,Array("soyo","spark","soyo20","hadoop","soyo2","","")),

         (,Array("soyo","","spark","","spark","spark",""))

       )).toDF("id","words")

IDEA Spark程序报错处理的更多相关文章

  1. 解决spark程序报错：Caused by: java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]

    报错信息: 09-05-2017 09:58:44 CST xxxx_job_1494294485570174 INFO - at org.apache.spark.sql.catalyst.erro ...

  2. eclispe集成Scalas环境后，导入外部Spark包报错：object apache is not a member of package org

    在Eclipse中集成scala环境后,发现导入的Spark包报错,提示是:object apache is not a member of package org,网上说了一大推,其实问题很简单: ...

  3. 运行编译后的程序报错 error while loading shared libraries: lib*.so: cannot open shared object file: No such file or directory

    运行编译后的程序报错  error while loading shared libraries: lib*.so: cannot open shared object file: No such f ...

  4. Window7中Eclipse运行MapReduce程序报错的问题

    按照文档:http://www.micmiu.com/bigdata/hadoop/hadoop2x-eclipse-mapreduce-demo/安装配置好Eclipse后,运行WordCount程 ...

  5. eclipse运行hadoop程序报错:Connection refused: no further information

    eclipse运行hadoop程序报错:Connection refused: no further information log4j:WARN No appenders could be foun ...

  6. WinDbg抓取程序报错dump文件的方法

    程序崩溃的两种主要现象: a. 程序在运行中的时候,突然弹出错误窗口,然后点错误窗口的确定时,程序直接关闭 例如: “应用程序错误” “C++错误之类的窗口” “程序无响应” “假死”等 此种崩溃特点 ...

  7. 记录微信小程序报错 Unexpected end of JSON input;at pages/flow/checkout page getOrderData function

    微信小程序报错 Unexpected end of JSON input;at pages/flow/checkout page getOrderData function 这个报错是在将数组对象通过 ...

  8. 小程序-报错 xxx is not defined （已解决）

    小程序-报错 xxx is not defined (已解决) 问题情境: 这样一段代码,微信的小程序报错 is not defined 我 wxml 想这样调用 //wxml 代码 <view ...

  9. debug运行java程序报错

    debug运行java程序报错 ERROR: transport error 202: connect failed: Connection timed out ERROR: JDWP Transpo ...

随机推荐

  1. du 命令计算隐藏文件夹或文件

    du -sh * .[^.]*

  2. 3D NAND闪存是个啥？让国内如此疯狂

    Repost: https://news.mydrivers.com/1/477/477251.htm 上个月底武汉新芯科技主导的国家级存储器产业基地正式动工,在大基金的支持下该项目将投资240亿美元 ...

  3. case when里的like功能 ////// 截取(substr)

    case when里的like功能 假如要用到case when又要用到like这样的功能,即如果字符串包含‘语文’就怎么怎么样,包含‘数学’就怎么怎么样,包含‘英语’就怎么怎么样,like是用于wh ...

  4. <MySQL>入门二 增删改 DML

    -- DML语言 /* 数据操作的语言 插入:insert 修改:update 删除:delete */ 1.插入 -- 插入语句 /* 语法:insert into 表名(列名...) values ...

  5. Oracle创建用户、角色、授权、建表空间

    oracle数据库的权限系统分为系统权限与对象权限.系统权限( database system privilege )可以让用户执行特定的命令集.例如,create table权限允许用户创建表,gr ...

  6. Linux学习笔记(六) 进程管理

    1.进程基础 当输入一个命令时,shell 会同时启动一个进程,这种任务与进程分离的方式是 Linux 系统上重要的概念 每个执行的任务都称为进程,在每个进程启动时,系统都会给它指定一个唯一的 ID, ...

  7. 深度完整的了解MySQL锁

    今天就讲讲MySQL的锁 主讲:Mysql的悲观锁 和 乐观锁官方:If you query data and then insert or update related data within th ...

  8. BC in fluent

    Boundary conditions in Fluent Table of Contents 1. Boundary Conditions (BC) 1.1. Turbulence Paramete ...

  9. 【Codeforces 923A】Primal Sport

    [链接] 我是链接,点我呀:) [题意] 题意 [题解] 考虑怎么得到数字x2=N,假设是质数p的倍数 那么x1肯定在x2-p+1~x2这个范围内才行 因为p的倍数要刚好大于等于x1, 所以x1肯定是 ...

  10. 【codeforces 709D】Recover the String

    [题目链接]:http://codeforces.com/problemset/problem/709/D [题意] 给你一个序列; 给出01子列和10子列和00子列以及11子列的个数; 然后让你输出 ...