SparkBench安装以及快速开始

欢迎关注我的GitHub~

本指南假定已经可以正常使用Spark 2.x，并且可以访问安装它的系统。

系统环境

CentOS 7.7.1908

Ambari-Spark 2.3.2

spark-bench_2.3.0_0.4.0-RELEASE

安装步骤

下载安装包

从下载地址获取最新版本的安装包。

解压缩安装包

接着在系统上解压缩安装包：

[root@master opt]# pwd

/opt

[root@master opt]# ll soft/

total 273792

drwxr-xr-x. 2 root root       164 Nov  5 14:51 ambari2.7.3.0+hdp3.1.0.0

drwxr-xr-x  4 root root        36 Nov  6 10:57 etc

-rw-r--r--. 1 root root 194151339 Nov  4 10:37 jdk-8u231-linux-x64.tar.gz

-rw-r--r--  1 root root     90472 Nov  4 15:25 libtirpc-0.2.4-0.16.el7.i686.rpm

-rw-r--r--  1 root root     93252 Nov  4 15:25 libtirpc-devel-0.2.4-0.16.el7.i686.rpm

-rw-r--r--. 1 root root     26024 Nov  4 14:19 mysql80-community-release-el7-3.noarch.rpm

-rw-r--r--. 1 root root    846263 Nov  4 14:27 mysql-connector-java-5.1.24.jar

-rw-r--r--  1 root root  85142775 Nov 13 10:50 spark-bench_2.3.0_0.4.0-RELEASE.tgz

[root@master opt]# tar -zxf soft/spark-bench_2.3.0_0.4.0-RELEASE.tgz -C /opt/

[root@master opt]# ll

total 4

drwxr-xr-x. 2 root root   42 Nov  5 14:29 os

drwxr-xr-x. 4 root root 4096 Nov 13 11:13 soft

drwxr-xr-x  5 2000 2000   61 Mar 23  2018 spark-bench_2.3.0_0.4.0-RELEASE

设置环境变量

有两种方法来设置运行示例所需的Spark的home和master变量。

设置bash环境变量

第一种方式是设置bash环境变量。此处仅作为讲解，不使用此方法进行设置环境变量。

进入安装目录的bin文件夹下，有一个spark-bench-env.sh.template文件，然后修改该文件，设置两个环境变量：

1.SPARK_HOME：这个是Spark集群或者单机Spark的完整安装路径。

2.SPARK_MASTER_HOST：这个变量和spark-submit脚本中--master的输入内容相同。如果你是在独立模式下运行的，可能是local[2]，Yarn集群上的yarn，IP地址或者端口号。

这种方式可以在bash配置文件中设置这些环境变量，也可以通过取消注释spark-bench-env.sh.template中的行并将其填充到位来进行设置，注意设置后需要将spark-bench-env.sh.template改为spark-bench-env.sh。

修改后的spark-bench-env.sh.template文件内容如下：

#!/bin/bash

# ############################################################### #

# PLEASE SET THE FOLLOWING VARIABLES TO REFLECT YOUR ENVIRONMENT  #

# ############################################################### #

# set this to the directory where Spark is installed in your environment, for example: /opt/spark-spark-2.1.0-bin-hadoop2.6

export SPARK_HOME=/usr/hdp/3.1.0.0-78/spark2/

# set this to the master for your environment, such as local[2], yarn, 10.29.0.3, etc.

export SPARK_MASTER_HOST=local[*]

修改示例配置文件以包含环境信息[推荐方式]

例如，在minimal-example.conf中，原本内容如下所示：

[root@master examples]# pwd

/opt/spark-bench_2.3.0_0.4.0-RELEASE/examples

[root@master examples]# cat minimal-example.conf

spark-bench = {

  spark-submit-config = [{

    workload-suites = [

      {

        descr = "One run of SparkPi and that's it!"

        benchmark-output = "console"

        workloads = [

          {

            name = "sparkpi"

            slices = 10

          }

        ]

      }

    ]

  }]

}

添加spark-home和master关键字之后，就成了：

spark-bench = {

  export SPARK_HOME=/usr/hdp/current/spark2-client

  // export SPARK_MASTER_HOST=yarn  // 现在已经改成了spark-args的形式了

  spark-submit-config = [{

    spark-args = {

      master = "local[*]" // or whatever the correct master is for your environment

    }

    workload-suites = [

      {

        descr = "One run of SparkPi and that's it!"

        benchmark-output = "console"

        workloads = [

          {

            name = "sparkpi"

            slices = 10

          }

        ]

      }

    ]

  }]

}

备注：官方文档推荐使用第二种方式，但是在新版本中，个人认为可以将两种方式结合起来，比如在spark-bench-env.sh.template中设置SPARK_HOME，然后在minimal-example.conf示例文件中只添加SPARK_MASTER_HOST，可能会更加方便一点。

示例运行

基于local[*]运行minimal-example.conf示例

点击查看代码

[root@master spark-bench_2.3.0_0.4.0-RELEASE]# ./bin/spark-bench.sh ./examples/minimal-example.conf

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

19/11/12 22:54:58 INFO CLIKickoff$: args received: {"spark-bench":{"spark-submit-config":[{"spark-args":{"master":"local[*]"},"workload-suites":[{"benchmark-output":"console","descr":"One run of SparkPi and that's it!","workloads":[{"name":"sparkpi","slices":10}]}]}]}}

19/11/12 22:54:58 INFO SparkContext: Running Spark version 2.3.2.3.1.0.0-78

19/11/12 22:54:58 INFO SparkContext: Submitted application: com.ibm.sparktc.sparkbench.cli.CLIKickoff

19/11/12 22:54:58 INFO SecurityManager: Changing view acls to: root

19/11/12 22:54:58 INFO SecurityManager: Changing modify acls to: root

19/11/12 22:54:58 INFO SecurityManager: Changing view acls groups to:

19/11/12 22:54:58 INFO SecurityManager: Changing modify acls groups to:

19/11/12 22:54:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()

19/11/12 22:54:58 INFO Utils: Successfully started service 'sparkDriver' on port 44042.

19/11/12 22:54:58 INFO SparkEnv: Registering MapOutputTracker

19/11/12 22:54:58 INFO SparkEnv: Registering BlockManagerMaster

19/11/12 22:54:58 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information

19/11/12 22:54:58 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up

19/11/12 22:54:58 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-49b14697-c68a-41e2-9d75-6bcb853f9082

19/11/12 22:54:58 INFO MemoryStore: MemoryStore started with capacity 413.9 MB

19/11/12 22:54:59 INFO SparkEnv: Registering OutputCommitCoordinator

19/11/12 22:54:59 INFO log: Logging initialized @3319ms

19/11/12 22:54:59 INFO Server: jetty-9.3.z-SNAPSHOT, build timestamp: 2018-06-05T13:11:56-04:00, git hash: 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827

19/11/12 22:54:59 INFO Server: Started @3427ms

19/11/12 22:54:59 INFO AbstractConnector: Started ServerConnector@72bca894{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}

19/11/12 22:54:59 INFO Utils: Successfully started service 'SparkUI' on port 4040.

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@320e400{/jobs,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7bdf6bb7{/jobs/json,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1bc53649{/jobs/job,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@475b7792{/jobs/job/json,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@751e664e{/stages,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@160c3ec1{/stages/json,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@182b435b{/stages/stage,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7577b641{/stages/stage/json,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3704122f{/stages/pool,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3153ddfc{/stages/pool/json,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@60afd40d{/storage,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@28a2a3e7{/storage/json,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3f2049b6{/storage/rdd,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@10b3df93{/storage/rdd/json,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@ea27e34{/environment,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@33a2499c{/environment/json,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@e72dba7{/executors,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@33c2bd{/executors/json,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1dfd5f51{/executors/threadDump,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3c321bdb{/executors/threadDump/json,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@24855019{/static,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5a2f016d{/,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1a38ba58{/api,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1deb2c43{/jobs/job/kill,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3bb9efbc{/stages/stage/kill,null,AVAILABLE,@Spark}

19/11/12 22:54:59 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://master.org.cn:4040

19/11/12 22:54:59 INFO SparkContext: Added JAR file:/opt/spark-bench_2.3.0_0.4.0-RELEASE/lib/spark-bench-2.3.0_0.4.0-RELEASE.jar at spark://master.org.cn:44042/jars/spark-bench-2.3.0_0.4.0-RELEASE.jar with timestamp 1573617299281

19/11/12 22:54:59 INFO Executor: Starting executor ID driver on host localhost

19/11/12 22:54:59 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46637.

19/11/12 22:54:59 INFO NettyBlockTransferService: Server created on master.org.cn:46637

19/11/12 22:54:59 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy

19/11/12 22:54:59 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, master.org.cn, 46637, None)

19/11/12 22:54:59 INFO BlockManagerMasterEndpoint: Registering block manager master.org.cn:46637 with 413.9 MB RAM, BlockManagerId(driver, master.org.cn, 46637, None)

19/11/12 22:54:59 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, master.org.cn, 46637, None)

19/11/12 22:54:59 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, master.org.cn, 46637, None)

19/11/12 22:54:59 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3c89bb12{/metrics/json,null,AVAILABLE,@Spark}

19/11/12 22:55:01 INFO EventLoggingListener: Logging events to hdfs:/spark2-history/local-1573617299355

19/11/12 22:55:02 INFO SparkContext: Starting job: reduce at SparkPi.scala:58

19/11/12 22:55:02 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:58) with 10 output partitions

19/11/12 22:55:02 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:58)

19/11/12 22:55:02 INFO DAGScheduler: Parents of final stage: List()

19/11/12 22:55:02 INFO DAGScheduler: Missing parents: List()

19/11/12 22:55:02 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:54), which has no missing parents

19/11/12 22:55:02 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1960.0 B, free 413.9 MB)

19/11/12 22:55:02 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1271.0 B, free 413.9 MB)

19/11/12 22:55:02 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on master.org.cn:46637 (size: 1271.0 B, free: 413.9 MB)

19/11/12 22:55:02 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1039

19/11/12 22:55:02 INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:54) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))

19/11/12 22:55:02 INFO TaskSchedulerImpl: Adding task set 0.0 with 10 tasks

19/11/12 22:55:02 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7853 bytes)

19/11/12 22:55:02 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)

19/11/12 22:55:02 INFO Executor: Fetching spark://master.org.cn:44042/jars/spark-bench-2.3.0_0.4.0-RELEASE.jar with timestamp 1573617299281

19/11/12 22:55:02 INFO TransportClientFactory: Successfully created connection to master.org.cn/192.168.0.219:44042 after 38 ms (0 ms spent in bootstraps)

19/11/12 22:55:02 INFO Utils: Fetching spark://master.org.cn:44042/jars/spark-bench-2.3.0_0.4.0-RELEASE.jar to /tmp/spark-4460cfe3-6a98-45b2-bf86-01387519d92d/userFiles-a94f5012-b587-4564-80ea-ed23b6f3507b/fetchFileTemp7487710686530246224.tmp

19/11/12 22:55:03 INFO Executor: Adding file:/tmp/spark-4460cfe3-6a98-45b2-bf86-01387519d92d/userFiles-a94f5012-b587-4564-80ea-ed23b6f3507b/spark-bench-2.3.0_0.4.0-RELEASE.jar to class loader

19/11/12 22:55:03 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 867 bytes result sent to driver

19/11/12 22:55:03 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 7853 bytes)

19/11/12 22:55:03 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)

19/11/12 22:55:03 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 824 bytes result sent to driver

19/11/12 22:55:03 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, executor driver, partition 2, PROCESS_LOCAL, 7853 bytes)

19/11/12 22:55:03 INFO Executor: Running task 2.0 in stage 0.0 (TID 2)

19/11/12 22:55:04 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 867 bytes result sent to driver

19/11/12 22:55:04 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1190 ms on localhost (executor driver) (1/10)

19/11/12 22:55:04 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 121 ms on localhost (executor driver) (2/10)

19/11/12 22:55:04 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, executor driver, partition 3, PROCESS_LOCAL, 7853 bytes)

19/11/12 22:55:04 INFO Executor: Running task 3.0 in stage 0.0 (TID 3)

19/11/12 22:55:04 INFO Executor: Finished task 3.0 in stage 0.0 (TID 3). 824 bytes result sent to driver

19/11/12 22:55:04 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, localhost, executor driver, partition 4, PROCESS_LOCAL, 7853 bytes)

19/11/12 22:55:04 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 93 ms on localhost (executor driver) (3/10)

19/11/12 22:55:04 INFO Executor: Running task 4.0 in stage 0.0 (TID 4)

19/11/12 22:55:04 INFO Executor: Finished task 4.0 in stage 0.0 (TID 4). 824 bytes result sent to driver

19/11/12 22:55:04 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, localhost, executor driver, partition 5, PROCESS_LOCAL, 7853 bytes)

19/11/12 22:55:04 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 13 ms on localhost (executor driver) (4/10)

19/11/12 22:55:04 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 39 ms on localhost (executor driver) (5/10)

19/11/12 22:55:04 INFO Executor: Running task 5.0 in stage 0.0 (TID 5)

19/11/12 22:55:04 INFO Executor: Finished task 5.0 in stage 0.0 (TID 5). 867 bytes result sent to driver

19/11/12 22:55:04 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, localhost, executor driver, partition 6, PROCESS_LOCAL, 7853 bytes)

19/11/12 22:55:04 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 40 ms on localhost (executor driver) (6/10)

19/11/12 22:55:04 INFO Executor: Running task 6.0 in stage 0.0 (TID 6)

19/11/12 22:55:04 INFO Executor: Finished task 6.0 in stage 0.0 (TID 6). 824 bytes result sent to driver

19/11/12 22:55:04 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, localhost, executor driver, partition 7, PROCESS_LOCAL, 7853 bytes)

19/11/12 22:55:04 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 10 ms on localhost (executor driver) (7/10)

19/11/12 22:55:04 INFO Executor: Running task 7.0 in stage 0.0 (TID 7)

19/11/12 22:55:04 INFO Executor: Finished task 7.0 in stage 0.0 (TID 7). 824 bytes result sent to driver

19/11/12 22:55:04 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, localhost, executor driver, partition 8, PROCESS_LOCAL, 7853 bytes)

19/11/12 22:55:04 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 26 ms on localhost (executor driver) (8/10)

19/11/12 22:55:04 INFO Executor: Running task 8.0 in stage 0.0 (TID 8)

19/11/12 22:55:04 INFO Executor: Finished task 8.0 in stage 0.0 (TID 8). 824 bytes result sent to driver

19/11/12 22:55:04 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, localhost, executor driver, partition 9, PROCESS_LOCAL, 7853 bytes)

19/11/12 22:55:04 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 20 ms on localhost (executor driver) (9/10)

19/11/12 22:55:04 INFO Executor: Running task 9.0 in stage 0.0 (TID 9)

19/11/12 22:55:04 INFO Executor: Finished task 9.0 in stage 0.0 (TID 9). 781 bytes result sent to driver

19/11/12 22:55:04 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 62 ms on localhost (executor driver) (10/10)

19/11/12 22:55:04 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:58) finished in 1.724 s

19/11/12 22:55:04 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool

19/11/12 22:55:04 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:58, took 1.895388 s

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 15

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 16

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 14

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 7

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 3

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 4

19/11/12 22:55:05 INFO BlockManagerInfo: Removed broadcast_0_piece0 on master.org.cn:46637 in memory (size: 1271.0 B, free: 413.9 MB)

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 17

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 9

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 24

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 18

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 11

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 21

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 5

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 19

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 12

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 22

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 8

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 0

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 23

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 20

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 6

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 13

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 10

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 1

19/11/12 22:55:05 INFO ContextCleaner: Cleaned accumulator 2

19/11/12 22:55:06 INFO SharedState: loading hive config file: file:/etc/spark2/3.1.0.0-78/0/hive-site.xml

19/11/12 22:55:06 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('/apps/spark/warehouse').

19/11/12 22:55:06 INFO SharedState: Warehouse path is '/apps/spark/warehouse'.

19/11/12 22:55:06 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@17884d{/SQL,null,AVAILABLE,@Spark}

19/11/12 22:55:06 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@68e24e7{/SQL/json,null,AVAILABLE,@Spark}

19/11/12 22:55:06 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@211a9647{/SQL/execution,null,AVAILABLE,@Spark}

19/11/12 22:55:06 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@340c5fb6{/SQL/execution/json,null,AVAILABLE,@Spark}

19/11/12 22:55:06 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@c262f2f{/static/sql,null,AVAILABLE,@Spark}

19/11/12 22:55:07 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint

19/11/12 22:55:08 INFO SuiteKickoff$: One run of SparkPi and that's it!

19/11/12 22:55:10 WARN Utils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.

19/11/12 22:55:12 INFO CodeGenerator: Code generated in 724.650089 ms

19/11/12 22:55:12 INFO CodeGenerator: Code generated in 119.823016 ms

19/11/12 22:55:13 INFO CodeGenerator: Code generated in 47.312121 ms

19/11/12 22:55:14 INFO SparkContext: Starting job: show at SparkFuncs.scala:108

19/11/12 22:55:14 INFO DAGScheduler: Got job 1 (show at SparkFuncs.scala:108) with 1 output partitions

19/11/12 22:55:14 INFO DAGScheduler: Final stage: ResultStage 1 (show at SparkFuncs.scala:108)

19/11/12 22:55:14 INFO DAGScheduler: Parents of final stage: List()

19/11/12 22:55:14 INFO DAGScheduler: Missing parents: List()

19/11/12 22:55:14 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[8] at show at SparkFuncs.scala:108), which has no missing parents

19/11/12 22:55:14 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 12.2 KB, free 413.9 MB)

19/11/12 22:55:14 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 4.6 KB, free 413.9 MB)

19/11/12 22:55:14 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on master.org.cn:46637 (size: 4.6 KB, free: 413.9 MB)

19/11/12 22:55:14 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1039

19/11/12 22:55:14 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[8] at show at SparkFuncs.scala:108) (first 15 tasks are for partitions Vector(0))

19/11/12 22:55:14 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks

19/11/12 22:55:14 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 10, localhost, executor driver, partition 0, PROCESS_LOCAL, 8302 bytes)

19/11/12 22:55:14 INFO Executor: Running task 0.0 in stage 1.0 (TID 10)

19/11/12 22:55:14 INFO Executor: Finished task 0.0 in stage 1.0 (TID 10). 2006 bytes result sent to driver

19/11/12 22:55:14 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 10) in 38 ms on localhost (executor driver) (1/1)

19/11/12 22:55:14 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool

19/11/12 22:55:14 INFO DAGScheduler: ResultStage 1 (show at SparkFuncs.scala:108) finished in 0.119 s

19/11/12 22:55:14 INFO DAGScheduler: Job 1 finished: show at SparkFuncs.scala:108, took 0.125345 s

+-------+-------------+-------------+------------------+-----+------+--------+------+---+-----------------------+-----------------------------+----------------------------------+-------------------------------+-----------------+-----------------------------+------------------------------------+----------------------+-----------------+-----------------------------+----------------+--------------------+----------------------------+---------------------------------------+--------------------------------+--------------------+-----------------------------+--------------------------------+--------------------------------------------+--------------------------------+------------------------------+-----------------+----------------------------------+-----------------------+------------+---------------------------------+-------------------------------+----------------------+-------------------------------+-------------------------+--------------------+---------------------+-----------------------------------+------------------------+------------------+------------------------+-------------------+--------------------------------+--------------------+

|   name|    timestamp|total_runtime|    pi_approximate|input|output|saveMode|slices|run|spark.sql.warehouse.dir|spark.history.kerberos.keytab|spark.io.compression.lz4.blockSize|spark.executor.extraJavaOptions|spark.driver.host|spark.history.fs.logDirectory|spark.sql.autoBroadcastJoinThreshold|spark.eventLog.enabled|spark.driver.port|spark.driver.extraLibraryPath|spark.yarn.queue|          spark.jars|spark.sql.orc.filterPushdown|spark.shuffle.unsafe.file.output.buffer|spark.yarn.historyServer.address|      spark.app.name|spark.sql.hive.metastore.jars|spark.history.kerberos.principal|spark.unsafe.sorter.spill.reader.buffer.size|spark.history.fs.cleaner.enabled|spark.shuffle.io.serverThreads|spark.executor.id|spark.sql.hive.convertMetastoreOrc|spark.submit.deployMode|spark.master|spark.history.fs.cleaner.interval|spark.history.fs.cleaner.maxAge|spark.history.provider|spark.executor.extraLibraryPath|spark.shuffle.file.buffer|  spark.eventLog.dir|spark.history.ui.port|spark.sql.statistics.fallBackToHdfs|spark.shuffle.io.backLog|spark.sql.orc.impl|spark.history.store.path|       spark.app.id|spark.sql.hive.metastore.version|         description|

+-------+-------------+-------------+------------------+-----+------+--------+------+---+-----------------------+-----------------------------+----------------------------------+-------------------------------+-----------------+-----------------------------+------------------------------------+----------------------+-----------------+-----------------------------+----------------+--------------------+----------------------------+---------------------------------------+--------------------------------+--------------------+-----------------------------+--------------------------------+--------------------------------------------+--------------------------------+------------------------------+-----------------+----------------------------------+-----------------------+------------+---------------------------------+-------------------------------+----------------------+-------------------------------+-------------------------+--------------------+---------------------+-----------------------------------+------------------------+------------------+------------------------+-------------------+--------------------------------+--------------------+

|sparkpi|1573617302076|   2168769711|3.1438591438591437|     |      |   error|    10|  0|   /apps/spark/wareh...|                         none|                             128kb|                   -XX:+UseNUMA|    master.org.cn|         hdfs:///spark2-hi...|                            26214400|                  true|            44042|         /usr/hdp/current/...|         default|file:/opt/spark-b...|                        true|                                     5m|             master.org.cn:18081|com.ibm.sparktc.s...|         /usr/hdp/current/...|                            none|                                          1m|                            true|                           128|           driver|                              true|                 client|    local[*]|                               7d|                            90d|  org.apache.spark....|           /usr/hdp/current/...|                       1m|hdfs:///spark2-hi...|                18081|                               true|                    8192|            native|    /var/lib/spark2/s...|local-1573617299355|                             3.0|One run of SparkP...|

+-------+-------------+-------------+------------------+-----+------+--------+------+---+-----------------------+-----------------------------+----------------------------------+-------------------------------+-----------------+-----------------------------+------------------------------------+----------------------+-----------------+-----------------------------+----------------+--------------------+----------------------------+---------------------------------------+--------------------------------+--------------------+-----------------------------+--------------------------------+--------------------------------------------+--------------------------------+------------------------------+-----------------+----------------------------------+-----------------------+------------+---------------------------------+-------------------------------+----------------------+-------------------------------+-------------------------+--------------------+---------------------+-----------------------------------+------------------------+------------------+------------------------+-------------------+--------------------------------+--------------------+

19/11/12 22:55:15 INFO SparkContext: Invoking stop() from shutdown hook

19/11/12 22:55:15 INFO AbstractConnector: Stopped Spark@72bca894{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}

19/11/12 22:55:15 INFO SparkUI: Stopped Spark web UI at http://master.org.cn:4040

19/11/12 22:55:15 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

19/11/12 22:55:15 INFO MemoryStore: MemoryStore cleared

19/11/12 22:55:15 INFO BlockManager: BlockManager stopped

19/11/12 22:55:15 INFO BlockManagerMaster: BlockManagerMaster stopped

19/11/12 22:55:15 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

19/11/12 22:55:15 INFO SparkContext: Successfully stopped SparkContext

19/11/12 22:55:15 INFO ShutdownHookManager: Shutdown hook called

19/11/12 22:55:15 INFO ShutdownHookManager: Deleting directory /tmp/spark-4460cfe3-6a98-45b2-bf86-01387519d92d

19/11/12 22:55:15 INFO ShutdownHookManager: Deleting directory /tmp/spark-bf2afab1-014b-4a2e-a43b-565cd57f4a69

基于yarn运行minimal-example.conf示例

基于yarn运行minimal-example.conf示例，需要使用hdfs用户来运行，否则报错没有写入hdfs文件系统的权限：