spark-streaming问题集锦

报错：

// :: ERROR Utils: Exception encountered
py4j.Py4JException: Cannot obtain a new communication channel
    at py4j.CallbackClient.sendCommand(CallbackClient.java:)
    at py4j.CallbackClient.sendCommand(CallbackClient.java:)
    at py4j.reflection.PythonProxyHandler.invoke(PythonProxyHandler.java:)
    at com.sun.proxy.$Proxy8.dumps(Unknown Source)
    at org.apache.spark.streaming.api.python.PythonTransformFunctionSerializer$.serialize(PythonDStream.scala:)
    at org.apache.spark.streaming.api.python.TransformFunction$$anonfun$writeObject$.apply$mcV$sp(PythonDStream.scala:)
    at org.apache.spark.streaming.api.python.TransformFunction$$anonfun$writeObject$.apply(PythonDStream.scala:)
    at org.apache.spark.streaming.api.python.TransformFunction$$anonfun$writeObject$.apply(PythonDStream.scala:)
    at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:)
    at org.apache.spark.streaming.api.python.TransformFunction.writeObject(PythonDStream.scala:)
    at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
    at java.lang.reflect.Method.invoke(Method.java:)
    at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:)

因为代码中设置了local directory：

 ssc.checkpoint("checkpoint")

需要设置成HDFS directory。

【参考】：http://45.55.83.242:4000/blog/2016/02/22/note-Some-Spark-issues-and-solutions/