1、完全分布式  ./bin/run-example streaming.NetworkWordCount localhost 9999无法正常运行：

  1 [hadoop@slaver1 spark-1.5.1-bin-hadoop2.4]$ ./bin/run-example streaming.NetworkWordCount slaver1 9999

  2 18/04/23 04:11:20 INFO SparkContext: Running Spark version 1.5.1

  3 18/04/23 04:11:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

  4 18/04/23 04:11:21 WARN SparkConf:

  5 SPARK_WORKER_INSTANCES was detected (set to '1').

  6 This is deprecated in Spark 1.0+.

  7

  8 Please instead use:

  9  - ./spark-submit with --num-executors to specify the number of executors

 10  - Or set SPARK_EXECUTOR_INSTANCES

 11  - spark.executor.instances to configure the number of instances in the spark config.

 12

 13 18/04/23 04:11:21 INFO SecurityManager: Changing view acls to: hadoop

 14 18/04/23 04:11:21 INFO SecurityManager: Changing modify acls to: hadoop

 15 18/04/23 04:11:21 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)

 16 18/04/23 04:11:22 INFO Slf4jLogger: Slf4jLogger started

 17 18/04/23 04:11:22 INFO Remoting: Starting remoting

 18 18/04/23 04:11:23 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.19.131:48823]

 19 18/04/23 04:11:23 INFO Utils: Successfully started service 'sparkDriver' on port 48823.

 20 18/04/23 04:11:23 INFO SparkEnv: Registering MapOutputTracker

 21 18/04/23 04:11:23 INFO SparkEnv: Registering BlockManagerMaster

 22 18/04/23 04:11:23 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-a48ee060-c8e8-4089-92b5-180cd8081890

 23 18/04/23 04:11:23 INFO MemoryStore: MemoryStore started with capacity 534.5 MB

 24 18/04/23 04:11:23 INFO HttpFileServer: HTTP File server directory is /tmp/spark-b6aec1ea-9a70-4814-b1f8-e752c27b9cee/httpd-fed4eaa6-5aec-4656-8d38-984997030d43

 25 18/04/23 04:11:23 INFO HttpServer: Starting HTTP Server

 26 18/04/23 04:11:23 INFO Utils: Successfully started service 'HTTP file server' on port 55775.

 27 18/04/23 04:11:23 INFO SparkEnv: Registering OutputCommitCoordinator

 28 18/04/23 04:11:24 INFO Utils: Successfully started service 'SparkUI' on port 4040.

 29 18/04/23 04:11:24 INFO SparkUI: Started SparkUI at http://192.168.19.131:4040

 30 18/04/23 04:11:24 INFO SparkContext: Added JAR file:/home/hadoop/soft/spark-1.5.1-bin-hadoop2.4/lib/spark-examples-1.5.1-hadoop2.4.0.jar at http://192.168.19.131:55775/jars/spark-examples-1.5.1-hadoop2.4.0.jar with timestamp 1524471084606

 31 18/04/23 04:11:24 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.

 32 18/04/23 04:11:24 INFO Executor: Starting executor ID driver on host localhost

 33 18/04/23 04:11:25 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 36228.

 34 18/04/23 04:11:25 INFO NettyBlockTransferService: Server created on 36228

 35 18/04/23 04:11:25 INFO BlockManagerMaster: Trying to register BlockManager

 36 18/04/23 04:11:25 INFO BlockManagerMasterEndpoint: Registering block manager localhost:36228 with 534.5 MB RAM, BlockManagerId(driver, localhost, 36228)

 37 18/04/23 04:11:25 INFO BlockManagerMaster: Registered BlockManager

 38 18/04/23 04:11:26 INFO EventLoggingListener: Logging events to hdfs://slaver1:9000/spark/history/local-1524471084719.snappy

 39 18/04/23 04:11:28 INFO ReceiverTracker: Starting 1 receivers

 40 18/04/23 04:11:28 INFO ReceiverTracker: ReceiverTracker started

 41 18/04/23 04:11:28 INFO ForEachDStream: metadataCleanupDelay = -1

 42 18/04/23 04:11:28 INFO ShuffledDStream: metadataCleanupDelay = -1

 43 18/04/23 04:11:28 INFO MappedDStream: metadataCleanupDelay = -1

 44 18/04/23 04:11:28 INFO FlatMappedDStream: metadataCleanupDelay = -1

 45 18/04/23 04:11:28 INFO SocketInputDStream: metadataCleanupDelay = -1

 46 18/04/23 04:11:28 INFO SocketInputDStream: Slide time = 1000 ms

 47 18/04/23 04:11:28 INFO SocketInputDStream: Storage level = StorageLevel(false, false, false, false, 1)

 48 18/04/23 04:11:28 INFO SocketInputDStream: Checkpoint interval = null

 49 18/04/23 04:11:28 INFO SocketInputDStream: Remember duration = 1000 ms

 50 18/04/23 04:11:28 INFO SocketInputDStream: Initialized and validated org.apache.spark.streaming.dstream.SocketInputDStream@29e21cd6

 51 18/04/23 04:11:28 INFO FlatMappedDStream: Slide time = 1000 ms

 52 18/04/23 04:11:28 INFO FlatMappedDStream: Storage level = StorageLevel(false, false, false, false, 1)

 53 18/04/23 04:11:28 INFO FlatMappedDStream: Checkpoint interval = null

 54 18/04/23 04:11:28 INFO FlatMappedDStream: Remember duration = 1000 ms

 55 18/04/23 04:11:28 INFO FlatMappedDStream: Initialized and validated org.apache.spark.streaming.dstream.FlatMappedDStream@3bd33b15

 56 18/04/23 04:11:28 INFO MappedDStream: Slide time = 1000 ms

 57 18/04/23 04:11:28 INFO MappedDStream: Storage level = StorageLevel(false, false, false, false, 1)

 58 18/04/23 04:11:28 INFO MappedDStream: Checkpoint interval = null

 59 18/04/23 04:11:28 INFO MappedDStream: Remember duration = 1000 ms

 60 18/04/23 04:11:28 INFO MappedDStream: Initialized and validated org.apache.spark.streaming.dstream.MappedDStream@28cbfe62

 61 18/04/23 04:11:28 INFO ShuffledDStream: Slide time = 1000 ms

 62 18/04/23 04:11:28 INFO ShuffledDStream: Storage level = StorageLevel(false, false, false, false, 1)

 63 18/04/23 04:11:28 INFO ShuffledDStream: Checkpoint interval = null

 64 18/04/23 04:11:28 INFO ShuffledDStream: Remember duration = 1000 ms

 65 18/04/23 04:11:28 INFO ShuffledDStream: Initialized and validated org.apache.spark.streaming.dstream.ShuffledDStream@68a9e8da

 66 18/04/23 04:11:28 INFO ForEachDStream: Slide time = 1000 ms

 67 18/04/23 04:11:28 INFO ForEachDStream: Storage level = StorageLevel(false, false, false, false, 1)

 68 18/04/23 04:11:28 INFO ForEachDStream: Checkpoint interval = null

 69 18/04/23 04:11:28 INFO ForEachDStream: Remember duration = 1000 ms

 70 18/04/23 04:11:28 INFO ForEachDStream: Initialized and validated org.apache.spark.streaming.dstream.ForEachDStream@6af675e4

 71 18/04/23 04:11:28 INFO RecurringTimer: Started timer for JobGenerator at time 1524471089000

 72 18/04/23 04:11:28 INFO JobGenerator: Started JobGenerator at 1524471089000 ms

 73 18/04/23 04:11:28 INFO JobScheduler: Started JobScheduler

 74 18/04/23 04:11:28 INFO StreamingContext: StreamingContext started

 75 18/04/23 04:11:28 INFO ReceiverTracker: Receiver 0 started

 76 18/04/23 04:11:29 INFO DAGScheduler: Got job 0 (start at NetworkWordCount.scala:57) with 1 output partitions

 77 18/04/23 04:11:29 INFO DAGScheduler: Final stage: ResultStage 0(start at NetworkWordCount.scala:57)

 78 18/04/23 04:11:29 INFO DAGScheduler: Parents of final stage: List()

 79 18/04/23 04:11:29 INFO DAGScheduler: Missing parents: List()

 80 18/04/23 04:11:29 INFO DAGScheduler: Submitting ResultStage 0 (Receiver 0 ParallelCollectionRDD[0] at makeRDD at ReceiverTracker.scala:554), which has no missing parents

 81 18/04/23 04:11:29 INFO JobScheduler: Added jobs for time 1524471089000 ms

 82 18/04/23 04:11:29 INFO JobScheduler: Starting job streaming job 1524471089000 ms.0 from job set of time 1524471089000 ms

 83 18/04/23 04:11:29 INFO SparkContext: Starting job: print at NetworkWordCount.scala:56

 84 18/04/23 04:11:29 INFO MemoryStore: ensureFreeSpace(55336) called with curMem=0, maxMem=560497950

 85 18/04/23 04:11:29 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 54.0 KB, free 534.5 MB)

 86 18/04/23 04:11:29 INFO MemoryStore: ensureFreeSpace(18523) called with curMem=55336, maxMem=560497950

 87 18/04/23 04:11:29 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 18.1 KB, free 534.5 MB)

 88 18/04/23 04:11:29 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:36228 (size: 18.1 KB, free: 534.5 MB)

 89 18/04/23 04:11:29 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:861

 90 18/04/23 04:11:29 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (Receiver 0 ParallelCollectionRDD[0] at makeRDD at ReceiverTracker.scala:554)

 91 18/04/23 04:11:29 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks

 92 18/04/23 04:11:30 INFO JobScheduler: Added jobs for time 1524471090000 ms

 93 18/04/23 04:11:30 INFO DAGScheduler: Registering RDD 3 (map at NetworkWordCount.scala:55)

 94 18/04/23 04:11:30 INFO DAGScheduler: Got job 1 (print at NetworkWordCount.scala:56) with 1 output partitions

 95 18/04/23 04:11:30 INFO DAGScheduler: Final stage: ResultStage 2(print at NetworkWordCount.scala:56)

 96 18/04/23 04:11:30 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1)

 97 18/04/23 04:11:30 INFO DAGScheduler: Missing parents: List()

 98 18/04/23 04:11:30 INFO DAGScheduler: Submitting ResultStage 2 (ShuffledRDD[4] at reduceByKey at NetworkWordCount.scala:55), which has no missing parents

 99 18/04/23 04:11:30 INFO MemoryStore: ensureFreeSpace(2360) called with curMem=73859, maxMem=560497950

100 18/04/23 04:11:30 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.3 KB, free 534.5 MB)

101 18/04/23 04:11:30 INFO MemoryStore: ensureFreeSpace(1440) called with curMem=76219, maxMem=560497950

102 18/04/23 04:11:30 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1440.0 B, free 534.5 MB)

103 18/04/23 04:11:30 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:36228 (size: 1440.0 B, free: 534.5 MB)

104 18/04/23 04:11:30 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:861

105 18/04/23 04:11:30 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (ShuffledRDD[4] at reduceByKey at NetworkWordCount.scala:55)

106 18/04/23 04:11:30 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks

107 18/04/23 04:11:30 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, ANY, 2721 bytes)

108 18/04/23 04:11:30 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)

109 18/04/23 04:11:30 INFO Executor: Fetching http://192.168.19.131:55775/jars/spark-examples-1.5.1-hadoop2.4.0.jar with timestamp 1524471084606

110 18/04/23 04:11:30 INFO Utils: Fetching http://192.168.19.131:55775/jars/spark-examples-1.5.1-hadoop2.4.0.jar to /tmp/spark-b6aec1ea-9a70-4814-b1f8-e752c27b9cee/userFiles-63aba9cd-16f7-4843-81b6-ec48b9b6df67/fetchFileTemp3110460484590096544.tmp

111 18/04/23 04:11:31 INFO JobScheduler: Added jobs for time 1524471091000 ms

112 18/04/23 04:11:31 INFO Executor: Adding file:/tmp/spark-b6aec1ea-9a70-4814-b1f8-e752c27b9cee/userFiles-63aba9cd-16f7-4843-81b6-ec48b9b6df67/spark-examples-1.5.1-hadoop2.4.0.jar to class loader

113 18/04/23 04:11:31 INFO RecurringTimer: Started timer for BlockGenerator at time 1524471091600

114 18/04/23 04:11:31 INFO BlockGenerator: Started BlockGenerator

115 18/04/23 04:11:31 INFO BlockGenerator: Started block pushing thread

116 18/04/23 04:11:31 INFO ReceiverTracker: Registered receiver for stream 0 from 192.168.19.131:48823

117 18/04/23 04:11:31 INFO ReceiverSupervisorImpl: Starting receiver

118 18/04/23 04:11:31 INFO ReceiverSupervisorImpl: Called receiver onStart

119 18/04/23 04:11:31 INFO ReceiverSupervisorImpl: Waiting for receiver to be stopped

120 18/04/23 04:11:31 INFO SocketReceiver: Connecting to slaver1:9999

121 18/04/23 04:11:31 INFO SocketReceiver: Connected to slaver1:9999

122 18/04/23 04:11:32 INFO JobScheduler: Added jobs for time 1524471092000 ms

123 18/04/23 04:11:33 INFO JobScheduler: Added jobs for time 1524471093000 ms

124 18/04/23 04:11:34 INFO JobScheduler: Added jobs for time 1524471094000 ms

125 18/04/23 04:11:35 INFO JobScheduler: Added jobs for time 1524471095000 ms

126 18/04/23 04:11:36 INFO JobScheduler: Added jobs for time 1524471096000 ms

127 18/04/23 04:11:37 INFO JobScheduler: Added jobs for time 1524471097000 ms

128 18/04/23 04:11:38 INFO JobScheduler: Added jobs for time 1524471098000 ms

129 18/04/23 04:11:39 INFO JobScheduler: Added jobs for time 1524471099000 ms

130 18/04/23 04:11:40 INFO JobScheduler: Added jobs for time 1524471100000 ms

131 18/04/23 04:11:41 INFO JobScheduler: Added jobs for time 1524471101000 ms

132 18/04/23 04:11:42 INFO JobScheduler: Added jobs for time 1524471102000 ms

133 18/04/23 04:11:43 INFO JobScheduler: Added jobs for time 1524471103000 ms

134 18/04/23 04:11:44 INFO JobScheduler: Added jobs for time 1524471104000 ms

135 18/04/23 04:11:45 INFO JobScheduler: Added jobs for time 1524471105000 ms

136 18/04/23 04:11:46 INFO JobScheduler: Added jobs for time 1524471106000 ms

137 18/04/23 04:11:47 INFO JobScheduler: Added jobs for time 1524471107000 ms

138 ^A18/04/23 04:11:48 INFO JobScheduler: Added jobs for time 1524471108000 ms

139 18/04/23 04:11:49 INFO JobScheduler: Added jobs for time 1524471109000 ms

140 18/04/23 04:11:50 INFO JobScheduler: Added jobs for time 1524471110000 ms

141 18/04/23 04:11:51 INFO JobScheduler: Added jobs for time 1524471111000 ms

142 18/04/23 04:11:52 INFO JobScheduler: Added jobs for time 1524471112000 ms

143 18/04/23 04:11:53 INFO JobScheduler: Added jobs for time 1524471113000 ms

2、启动过程如上所示，下面就是问题，当在nc -lk 9999命令窗口，输入例如hello world hello world hadoop world spark world flume world hello world的字符的时候，另一个窗口执行如下命令[hadoop@slaver1 spark-1.5.1-bin-hadoop2.4]$ ./bin/run-example streaming.NetworkWordCount 192.168.19.131 9999  ，以后出现这样的错误：

18/04/23 04:12:25 INFO MemoryStore: ensureFreeSpace(79) called with curMem=77659, maxMem=560497950

18/04/23 04:12:25 INFO MemoryStore: Block input-0-1524471145600 stored as bytes in memory (estimated size 79.0 B, free 534.5 MB)

18/04/23 04:12:25 INFO BlockManagerInfo: Added input-0-1524471145600 in memory on localhost:36228 (size: 79.0 B, free: 534.5 MB)

18/04/23 04:12:25 INFO BlockGenerator: Pushed block input-0-

3、解决方法，将你的虚拟机核数修改位多核，再次执行，可以看到统计的数量：修改为2核执行速度已经很快了，开始我的内存是1G，后来添加到2G还是解决不了问题，把核数修改为2核，解决问题：

再次执行结果如下所示：

INFO JobScheduler: Added jobs for time 1524468752000 ms/INFO MemoryStore: Block input-0-1524469143000 stored as bytes in memory/完全分布式 ./bin/run-example streaming.NetworkWordCount localhost 9999无法正常运行的更多相关文章

  1. Spark小课堂Week6 启动日志详解

    Spark小课堂Week6 启动日志详解 作为分布式系统,Spark程序是非常难以使用传统方法来进行调试的,所以我们主要的武器是日志,今天会对启动日志进行一下详解. 日志详解 今天主要遍历下Strea ...

  2. Spark Streaming揭秘 Day28 在集成开发环境中详解Spark Streaming的运行日志内幕

    Spark Streaming揭秘 Day28 在集成开发环境中详解Spark Streaming的运行日志内幕 今天会逐行解析一下SparkStreaming运行的日志,运行的是WordCountO ...

  3. Spark Streaming 002 统计单词的例子

    1.准备 事先在hdfs上创建两个目录: 保存上传数据的目录:hdfs://alamps:9000/library/SparkStreaming/data checkpoint的目录:hdfs://a ...

  4. <Spark><Spark Streaming><作业分析><JobHistory>

    Intro 这篇是对一个Spark (Streaming)作业的log进行分析.用来加深对Spark application运行过程,优化空间的各种理解. Here to Start 从我这个初学者写 ...

  5. SparkStreaming实时日志分析--实时热搜词

    Overview 整个项目的整体架构如下: 关于SparkStreaming的部分: Flume传数据到SparkStreaming:为了简单使用的是push-based的方式.这种方式可能会丢失数据 ...

  6. <译>Spark Sreaming 编程指南

    Spark Streaming 编程指南 Overview A Quick Example Basic Concepts Linking Initializing StreamingContext D ...

  7. Apache Spark 2.2.0 中文文档 - Spark Streaming 编程指南 | ApacheCN

    Spark Streaming 编程指南 概述 一个入门示例 基础概念 依赖 初始化 StreamingContext Discretized Streams (DStreams)(离散化流) Inp ...

  8. spark stream初探

    spark带了一个NetworkWordCount测试程序,用以统计来自某TCP连接的单词输入: /usr/local/spark/bin/run-example streaming.NetworkW ...

  9. Spark Streaming编程指南

    Overview A Quick Example Basic Concepts Linking Initializing StreamingContext Discretized Streams (D ...

随机推荐

  1. Linux嗅探ettercap

    场景 拿到一台C段的Linux服务器,对目标主机进行嗅探 ettercap安装 操作环境 Centos 6 $ sudo yum install -y libtool-ltdl ncurses-dev ...

  2. C/C++经典面试题一

    1.变量的声明和定义有什么区别? 常量:在程序执行过程中,不会发生改变的量,不能被改变的量 变量:在程序执行过程中,可以被改变的量 定义变量的方式:数据类型 变量名 = 常量: int num = 1 ...

  3. Ubuntu的内核转储工具【转】

    转自:http://www.cnblogs.com/wwang/archive/2010/11/19/1881304.html 在我的上一篇博文<Linux内核的Oops>的最后介绍到一个 ...

  4. C#基础巩固之基础类型

    注:以下笔记全摘录自CLR via C# 3 1.所有类型都从System.Object派生:”运行时“要求每个类型最终都从System.Object派生. 2.System.Object提供了四个公 ...

  5. U3D虚拟摇杆制作

    来自https://www.cnblogs.com/jiuxuan/p/7453762.html 1.创建两个Image,修改第一个Image名称为 Background,把第二个Image放入 Ba ...

  6. [JLOI2011]飞行路线 不同的算法，不同的悲伤

    题目 :BZOJ2763 洛谷P4568 [JLOI2011]飞行路线 一道最短路的题目,想想写个题解也不错(好久没写题解了_(:з」∠)_) 然后这道题中心思路是dijikstra处理最短路,所以没 ...

  7. K-query SPOJ - KQUERY 离线　线段树/树状数组　区间大于Ｋ的个数

    题意: 给一个数列,一些询问,问你区间$[l.r]$大于$K$的个数 题解: 又一个"人尽皆知傻逼题"? 我们用一个01序列表示当前询问时,该位置的数字是否对答案有贡献, 显然,对 ...

  8. Freeswitch 入门

    让我们从最初的运行开始,一步一步进入 FreeSWITCH 的神秘世界. 命令行参数 一般来说,FreeSWITCH 不需要任何命令行参数就可以启动,但在某些情况下,你需要以一些特殊的参数启动.在此, ...

  9. 一种简单的生产环境部署Node.js程序方法

    最近在部署Node.js程序时,写了段简单的脚本,发觉还挺简单的,忍不住想与大家分享. 配置文件 首先,本地测试环境和生产环境的数据库连接这些配置信息是不一样的,需要将其分开为两个文件存储 到conf ...

  10. 7)django-示例(cbv)

    CBV(class base view)一个url根据method方式调用相应的方法.method常用有get,post 如果是GET请求,Home类会调用get方法,如果是POST提交数据,则类会调 ...