Spark小课堂Week6 启动日志详解

作为分布式系统，Spark程序是非常难以使用传统方法来进行调试的，所以我们主要的武器是日志，今天会对启动日志进行一下详解。

日志详解

今天主要遍历下Streaming的启动日志。

  • 授权等操作
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

16/07/19 15:06:04 INFO SparkContext: Running Spark version 1.4.1

16/07/19 15:06:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/07/19 15:06:10 INFO SecurityManager: Changing view acls to:

16/07/19 15:06:10 INFO SecurityManager: Changing modify acls to:

16/07/19 15:06:10 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(); users with modify permissions: Set()

16/07/19 15:06:11 INFO Slf4jLogger: Slf4jLogger started
  • 有中心式架构，元数据服务叫Driver，这里是启动了一个akka服务，会提供rpc调用。
16/07/19 15:06:11 INFO Remoting: Starting remoting

16/07/19 15:06:11 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@84.232.207.171:46561]

16/07/19 15:06:11 INFO Utils: Successfully started service 'sparkDriver' on port 46561.
  • SparkEnv类似一个地址簿，会管理所有Spark自身的服务
  • 启动数据管理服务，这里是初始分配存储，包括分配磁盘和内存两部分
16/07/19 15:06:11 INFO SparkEnv: Registering MapOutputTracker

16/07/19 15:06:11 INFO SparkEnv: Registering BlockManagerMaster

16/07/19 15:06:11 INFO DiskBlockManager: Created local directory at C:\Users\kfzx-zhongw\AppData\Local\Temp\spark-45392ea0-70f1-4562-b251-22521756a1d3\blockmgr-b6eb9263-5a04-4cc9-83b9-fcbe153808b8

16/07/19 15:06:11 INFO MemoryStore: MemoryStore started with capacity 133.6 MB
  • 这里的file server主要是管理jar包等基础文件的
16/07/19 15:06:11 INFO HttpFileServer: HTTP File server directory is C:\Users\kfzx-zhongw\AppData\Local\Temp\spark-45392ea0-70f1-4562-b251-22521756a1d3\httpd-69699a3d-70f7-4a99-8ae8-9527ca666cdb

16/07/19 15:06:11 INFO HttpServer: Starting HTTP Server

16/07/19 15:06:11 INFO Utils: Successfully started service 'HTTP file server' on port 46562.

16/07/19 15:06:11 INFO SparkEnv: Registering OutputCommitCoordinator
  • 启动UI服务，底层是一个Jetty
16/07/19 15:06:11 INFO Utils: Successfully started service 'SparkUI' on port 4040.

16/07/19 15:06:11 INFO SparkUI: Started SparkUI at http://84.232.207.171:4040
  • 这里启动数据传输服务，是一个netty服务
16/07/19 15:06:11 INFO Executor: Starting executor ID driver on host localhost

16/07/19 15:06:12 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46581.

16/07/19 15:06:12 INFO NettyBlockTransferService: Server created on 46581

16/07/19 15:06:12 INFO BlockManagerMaster: Trying to register BlockManager

16/07/19 15:06:12 INFO BlockManagerMasterEndpoint: Registering block manager localhost:46581 with 133.6 MB RAM, BlockManagerId(driver, localhost, 46581)

16/07/19 15:06:12 INFO BlockManagerMaster: Registered BlockManager

  • 至此，Spark的核心部分启动完成，下面是SparkStreaming的启动。

  • 业务逻辑对象初始化，这里是从后往前调用，从前往后运行。

16/07/19 15:06:12 INFO ReceiverTracker: ReceiverTracker started

16/07/19 15:06:12 INFO ForEachDStream: metadataCleanupDelay = -1

16/07/19 15:06:12 INFO ShuffledDStream: metadataCleanupDelay = -1

16/07/19 15:06:12 INFO MappedDStream: metadataCleanupDelay = -1

16/07/19 15:06:12 INFO FlatMappedDStream: metadataCleanupDelay = -1

16/07/19 15:06:12 INFO SocketInputDStream: metadataCleanupDelay = -1

16/07/19 15:06:12 INFO SocketInputDStream: Slide time = 10000 ms

16/07/19 15:06:12 INFO SocketInputDStream: Storage level = StorageLevel(false, false, false, false, 1)

16/07/19 15:06:12 INFO SocketInputDStream: Checkpoint interval = null

16/07/19 15:06:12 INFO SocketInputDStream: Remember duration = 10000 ms

16/07/19 15:06:12 INFO ReceiverTracker: Starting 1 receivers

16/07/19 15:06:12 INFO SocketInputDStream: Initialized and validated org.apache.spark.streaming.dstream.SocketInputDStream@819867

16/07/19 15:06:12 INFO FlatMappedDStream: Slide time = 10000 ms

16/07/19 15:06:12 INFO FlatMappedDStream: Storage level = StorageLevel(false, false, false, false, 1)

16/07/19 15:06:12 INFO FlatMappedDStream: Checkpoint interval = null

16/07/19 15:06:12 INFO FlatMappedDStream: Remember duration = 10000 ms

16/07/19 15:06:12 INFO FlatMappedDStream: Initialized and validated org.apache.spark.streaming.dstream.FlatMappedDStream@18102ee

16/07/19 15:06:12 INFO MappedDStream: Slide time = 10000 ms

16/07/19 15:06:12 INFO MappedDStream: Storage level = StorageLevel(false, false, false, false, 1)

16/07/19 15:06:12 INFO MappedDStream: Checkpoint interval = null

16/07/19 15:06:12 INFO MappedDStream: Remember duration = 10000 ms

16/07/19 15:06:12 INFO MappedDStream: Initialized and validated org.apache.spark.streaming.dstream.MappedDStream@cd9a2c

16/07/19 15:06:12 INFO ShuffledDStream: Slide time = 10000 ms

16/07/19 15:06:12 INFO ShuffledDStream: Storage level = StorageLevel(false, false, false, false, 1)

16/07/19 15:06:12 INFO ShuffledDStream: Checkpoint interval = null

16/07/19 15:06:12 INFO ShuffledDStream: Remember duration = 10000 ms

16/07/19 15:06:12 INFO ShuffledDStream: Initialized and validated org.apache.spark.streaming.dstream.ShuffledDStream@11d1ffb

16/07/19 15:06:12 INFO ForEachDStream: Slide time = 10000 ms

16/07/19 15:06:12 INFO ForEachDStream: Storage level = StorageLevel(false, false, false, false, 1)

16/07/19 15:06:12 INFO ForEachDStream: Checkpoint interval = null

16/07/19 15:06:12 INFO ForEachDStream: Remember duration = 10000 ms

16/07/19 15:06:12 INFO ForEachDStream: Initialized and validated org.apache.spark.streaming.dstream.ForEachDStream@167e3df
  • Streaming中最重要的组件，四个组件是从外置内套着的。
16/07/19 15:06:12 INFO RecurringTimer: Started timer for JobGenerator at time 1468911980000

16/07/19 15:06:12 INFO JobGenerator: Started JobGenerator at 1468911980000 ms

16/07/19 15:06:12 INFO JobScheduler: Started JobScheduler

16/07/19 15:06:12 INFO StreamingContext: StreamingContext started
  • 启动Job0，这个是Streaming的socket数据接收服务
16/07/19 15:06:12 INFO SparkContext: Starting job: start at JavaOnlineWordCount.java:32

16/07/19 15:06:12 INFO DAGScheduler: Got job 0 (start at JavaOnlineWordCount.java:32) with 1 output partitions (allowLocal=false)

16/07/19 15:06:12 INFO DAGScheduler: Final stage: ResultStage 0(start at JavaOnlineWordCount.java:32)

16/07/19 15:06:12 INFO DAGScheduler: Parents of final stage: List()

16/07/19 15:06:12 INFO DAGScheduler: Missing parents: List()

16/07/19 15:06:12 INFO DAGScheduler: Submitting ResultStage 0 (ParallelCollectionRDD[0] at start at JavaOnlineWordCount.java:32), which has no missing parents

16/07/19 15:06:13 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes

16/07/19 15:06:13 INFO MemoryStore: ensureFreeSpace(46352) called with curMem=0, maxMem=140142182

16/07/19 15:06:13 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 45.3 KB, free 133.6 MB)

16/07/19 15:06:13 INFO MemoryStore: ensureFreeSpace(15070) called with curMem=46352, maxMem=140142182

16/07/19 15:06:13 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 14.7 KB, free 133.6 MB)

16/07/19 15:06:13 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:46581 (size: 14.7 KB, free: 133.6 MB)

16/07/19 15:06:13 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:874

16/07/19 15:06:13 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (ParallelCollectionRDD[0] at start at JavaOnlineWordCount.java:32)

16/07/19 15:06:13 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks

16/07/19 15:06:13 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, PROCESS_LOCAL, 2000 bytes)

16/07/19 15:06:13 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)

16/07/19 15:06:13 INFO RecurringTimer: Started timer for BlockGenerator at time 1468911973400

16/07/19 15:06:13 INFO BlockGenerator: Started block pushing thread

16/07/19 15:06:13 INFO BlockGenerator: Started BlockGenerator

16/07/19 15:06:13 INFO ReceiverSupervisorImpl: Starting receiver

16/07/19 15:06:13 INFO ReceiverSupervisorImpl: Called receiver onStart

16/07/19 15:06:13 INFO SocketReceiver: Connecting to localhost:9999

16/07/19 15:06:13 INFO SocketReceiver: Connected to localhost:9999

16/07/19 15:06:13 INFO ReceiverTracker: Registered receiver for stream 0 from 84.232.207.171:46561
  • 获取外部输入数据，这里会调用memoryStore
16/07/19 15:06:19 INFO MemoryStore: ensureFreeSpace(15) called with curMem=61422, maxMem=140142182

16/07/19 15:06:19 INFO MemoryStore: Block input-0-1468911978800 stored as bytes in memory (estimated size 15.0 B, free 133.6 MB)

16/07/19 15:06:19 INFO BlockManagerInfo: Added input-0-1468911978800 in memory on localhost:46581 (size: 15.0 B, free: 133.6 MB)

16/07/19 15:06:19 WARN BlockManager: Block input-0-1468911978800 replicated to only 0 peer(s) instead of 1 peers

16/07/19 15:06:19 INFO BlockGenerator: Pushed block input-0-1468911978800
  • 执行业务逻辑计算，这里是核心！！！
16/07/19 15:06:20 INFO JobScheduler: Added jobs for time 1468911980000 ms

16/07/19 15:06:20 INFO JobScheduler: Starting job streaming job 1468911980000 ms.0 from job set of time 1468911980000 ms

16/07/19 15:06:20 INFO SparkContext: Starting job: print at JavaOnlineWordCount.java:30

16/07/19 15:06:20 INFO DAGScheduler: Registering RDD 3 (mapToPair at JavaOnlineWordCount.java:28)

16/07/19 15:06:20 INFO DAGScheduler: Got job 1 (print at JavaOnlineWordCount.java:30) with 1 output partitions (allowLocal=true)

16/07/19 15:06:20 INFO DAGScheduler: Final stage: ResultStage 2(print at JavaOnlineWordCount.java:30)

16/07/19 15:06:20 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1)

16/07/19 15:06:20 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 1)

16/07/19 15:06:20 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[3] at mapToPair at JavaOnlineWordCount.java:28), which has no missing parents

16/07/19 15:06:20 INFO MemoryStore: ensureFreeSpace(4016) called with curMem=61437, maxMem=140142182

16/07/19 15:06:20 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.9 KB, free 133.6 MB)

16/07/19 15:06:20 INFO MemoryStore: ensureFreeSpace(2297) called with curMem=65453, maxMem=140142182

16/07/19 15:06:20 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.2 KB, free 133.6 MB)

16/07/19 15:06:20 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:46581 (size: 2.2 KB, free: 133.6 MB)

16/07/19 15:06:20 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:874

16/07/19 15:06:20 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[3] at mapToPair at JavaOnlineWordCount.java:28)

16/07/19 15:06:20 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks

16/07/19 15:06:20 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, NODE_LOCAL, 1277 bytes)

16/07/19 15:06:20 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)

16/07/19 15:06:20 INFO BlockManager: Found block input-0-1468911978800 locally

16/07/19 15:06:20 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 894 bytes result sent to driver

16/07/19 15:06:20 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 80 ms on localhost (1/1)

16/07/19 15:06:20 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool

16/07/19 15:06:20 INFO DAGScheduler: ShuffleMapStage 1 (mapToPair at JavaOnlineWordCount.java:28) finished in 0.084 s

16/07/19 15:06:20 INFO DAGScheduler: looking for newly runnable stages

16/07/19 15:06:20 INFO DAGScheduler: running: Set(ResultStage 0)

16/07/19 15:06:20 INFO DAGScheduler: waiting: Set(ResultStage 2)

16/07/19 15:06:20 INFO DAGScheduler: failed: Set()

16/07/19 15:06:20 INFO DAGScheduler: Missing parents for ResultStage 2: List()

16/07/19 15:06:20 INFO DAGScheduler: Submitting ResultStage 2 (ShuffledRDD[4] at reduceByKey at JavaOnlineWordCount.java:29), which is now runnable

16/07/19 15:06:20 INFO MemoryStore: ensureFreeSpace(3056) called with curMem=67750, maxMem=140142182

16/07/19 15:06:20 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 3.0 KB, free 133.6 MB)

16/07/19 15:06:20 INFO MemoryStore: ensureFreeSpace(1825) called with curMem=70806, maxMem=140142182

16/07/19 15:06:20 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1825.0 B, free 133.6 MB)

16/07/19 15:06:20 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:46581 (size: 1825.0 B, free: 133.6 MB)

16/07/19 15:06:20 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:874

16/07/19 15:06:20 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (ShuffledRDD[4] at reduceByKey at JavaOnlineWordCount.java:29)

16/07/19 15:06:20 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks

16/07/19 15:06:20 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, PROCESS_LOCAL, 1165 bytes)

16/07/19 15:06:20 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)

16/07/19 15:06:20 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks

16/07/19 15:06:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 5 ms

16/07/19 15:06:20 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 882 bytes result sent to driver

16/07/19 15:06:20 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 33 ms on localhost (1/1)

16/07/19 15:06:20 INFO DAGScheduler: ResultStage 2 (print at JavaOnlineWordCount.java:30) finished in 0.033 s

16/07/19 15:06:20 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool

16/07/19 15:06:20 INFO DAGScheduler: Job 1 finished: print at JavaOnlineWordCount.java:30, took 0.184378 s

...
  • 结果输出：
-------------------------------------------

Time: 1468911980000 ms

-------------------------------------------

(123,1)

(1231,1)
  • 执行清理操作
16/07/19 15:06:20 INFO JobScheduler: Finished job streaming job 1468911980000 ms.0 from job set of time 1468911980000 ms

16/07/19 15:06:20 INFO JobScheduler: Total delay: 0.440 s for time 1468911980000 ms (execution: 0.359 s)

16/07/19 15:06:20 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer()

16/07/19 15:06:20 INFO InputInfoTracker: remove old batch metadata:

关于

小课堂是在公司进行内部交流的一系列主题，偏基础，但是比较零散，持续更新中。

Spark小课堂Week6 启动日志详解的更多相关文章

  1. Linux启动过程详解（inittab、rc.sysinit、rcX.d、rc.local）

    启动第一步--加载BIOS 当你打开计算机电源,计算机会首先加载BIOS信息,BIOS信息是如此的重要,以至于计算机必须在最开始就找到它.这是因为BIOS中包含了CPU的相关信息.设备启动顺序信息.硬 ...

  2. Linux启动过程详解

    Linux启动过程详解 附上两张图,加深记忆 图1: 图2: 第一张图比较简洁明了,下面对第一张图的步骤进行详解: 加载BIOS 当你打开计算机电源,计算机会首先加载BIOS信息,BIOS信息是如此的 ...

  3. linux 开机启动过程详解

    Linux开机执行内核后会启动init进程,该进程根据runlevel(如x)执行/etc/rcx.d/下的程序,其下的程序是符号链接,真正的程序放在/etc/init.d/下.开机启动的程序(服务等 ...

  4. Linux开机启动程序详解

    Linux开机启动程序详解我们假设大家已经熟悉其它操作系统的引导过程,了解硬件的自检引导步骤,就只从Linux操作系统的引导加载程序(对个人电脑而言通常是LILO)开始,介绍Linux开机引导的步骤. ...

  5. Linux开机启动程序详解[转]

    Linux开机启动程序详解 我们假设大家已经熟悉其它操作系统的引导过程,了解硬件的自检引导步骤,就只从Linux操作系统的引导加载程序(对个人电脑而言通常是LILO)开始,介绍Linux开机引导的步骤 ...

  6. linux系统设置服务开机启动3种方法，Linux开机启动程序详解

    linux系统设置服务开机启动 方法1:.利用ntsysv伪图形进行设置,利用root登陆 终端命令下输入ntsysv 回车:如下图     方法2:利用命令行chkconfig命令进行设置 简要说明 ...

  7. Spark小课堂Week2 Hello Streaming

    Spark小课堂Week2 Hello Streaming 我们是怎么进行数据处理的? 批量方式处理 目前最常采用的是批量方式处理,指非工作时间运行,定时或者事件触发.这种方式的好处是逻辑简单,不影响 ...

  8. Spark小课堂Week1 Hello Spark

    Spark小课堂Week1 Hello Spark 看到Spark这个词,你的第一印象是什么? 这是一朵"火花",官方的定义是Spark是一个高速的.通用的.分布式计算系统!!! ...

  9. Linux启动流程详解【转载】

    在BIOS阶段,计算机的行为基本上被写死了,可以做的事情并不多:一般就是通电.BIOS.主引导记录.操作系统这四步.所以我们一般认为加载内核是linux启动流程的第一步. 第一步.加载内核 操作系统接 ...

随机推荐

  1. A Brief Introduction to the Design of UBIFS

    http://pan.baidu.com/s/1dDy0jip 译文:http://blog.csdn.net/kickxxx/article/details/6573396 项目闲暇,想了解下UBI ...

  2. QTP 中对象操作

    1.创建对象(将浏览器设置为一个对象) set oIE=createobject("internetexplorer.application") 2.将浏览器设置为可见的 oIE. ...

  3. HBase-配置说明

    转载自:http://www.aboutyun.com/thread-7914-1-1.html hbase.rootdir这个目录是region  server的共享目录,用来持久化Hbase.UR ...

  4. df 和 du 命令详解

    df命令详细用法 a:显示全部的档案系统和各分割区的磁盘使用情形 i:显示i -nodes的使用量 k:大小用k来表示 (默认值) t:显示某一个档案系统的所有分割区磁盘使用量 x:显示不是某一个档案 ...

  5. c语言实现：4和7幸运数字的题

    #include <stdio.h> #include <math.h> #include <vector> using namespace std; int ma ...

  6. WampServer下修改和重置MySQL密码(转)

    转自:www.2cto.com/database/201504/387589.html WampServer安装后密码是空的, 修改一般有两种方式: 一是通过phpMyAdmin直接修改: 二是使用W ...

  7. LeetCode 264

    Ugly Number II Write a program to find the n-th ugly number. Ugly numbers are positive numbers whose ...

  8. 自动化中的PageObject思想

    谈着这个话题很大,因为我自己在自动化分层的方面做的就不是很好,最近在做java+selenium中,如果不分层去管理,代码就显得很乱,如下: 下面代码主要是做了一个登录,登录后在修改自己的一些个人信息 ...

  9. poj3393[模拟题]

    Lucky and Good Months by Gregorian Calendar Time Limit: 1000MS   Memory Limit: 65536K Total Submissi ...

  10. 自己改写的asp.net MVC EF Respoistory 仓储模式

    之前改写网上收集的Respoistory 模式感觉很多地方都是不可取的,这里经过这段时间的充电重新改写一版,当然注释已加,还有不懂的,可以留言我. 首先还是拿出IRespoistory的接口层代码 u ...