Caused by: java.net.ConnectException: Connection refused: master/192.168.3.129:7077
1：启动Spark Shell，spark-shell是Spark自带的交互式Shell程序，方便用户进行交互式编程，用户可以在该命令行下用scala编写spark程序。
启动Spark Shell，出现的错误如下所示：
[root@master spark-1.6.-bin-hadoop2.]# bin/spark-shell --master spark://master:7077 --executor-memory 512M --total-executor-cores 2
// :: WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
// :: INFO SecurityManager: Changing view acls to: root
// :: INFO SecurityManager: Changing modify acls to: root
// :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
// :: INFO HttpServer: Starting HTTP Server
// :: INFO Utils: Successfully started service 'HTTP class server' on port .
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.6.
/_/ Using Scala version 2.10. (Java HotSpot(TM) Client VM, Java 1.7.0_65)
Type in expressions to have them evaluated.
Type :help for more information.
// :: INFO SparkContext: Running Spark version 1.6.
// :: INFO SecurityManager: Changing view acls to: root
// :: INFO SecurityManager: Changing modify acls to: root
// :: INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
// :: INFO Utils: Successfully started service 'sparkDriver' on port .
// :: INFO Slf4jLogger: Slf4jLogger started
// :: INFO Remoting: Starting remoting
// :: INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.3.129:43806]
// :: INFO Utils: Successfully started service 'sparkDriverActorSystem' on port .
// :: INFO SparkEnv: Registering MapOutputTracker
// :: INFO SparkEnv: Registering BlockManagerMaster
// :: INFO DiskBlockManager: Created local directory at /tmp/blockmgr-7face114-24b5-4f0e-adb6-8a104e387c78
// :: INFO MemoryStore: MemoryStore started with capacity 517.4 MB
// :: INFO SparkEnv: Registering OutputCommitCoordinator
// :: INFO Utils: Successfully started service 'SparkUI' on port .
// :: INFO SparkUI: Started SparkUI at http://192.168.3.129:4040
// :: INFO AppClient$ClientEndpoint: Connecting to master spark://master:7077...
// :: WARN AppClient$ClientEndpoint: Failed to connect to master master:
java.io.IOException: Failed to connect to master/192.168.3.129:
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:)
at org.apache.spark.rpc.netty.Outbox$$anon$.call(Outbox.scala:)
at org.apache.spark.rpc.netty.Outbox$$anon$.call(Outbox.scala:)
at java.util.concurrent.FutureTask.run(FutureTask.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: java.net.ConnectException: Connection refused: master/192.168.3.129:
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:)
at io.netty.util.concurrent.SingleThreadEventExecutor$.run(SingleThreadEventExecutor.java:)
... more
// :: INFO AppClient$ClientEndpoint: Connecting to master spark://master:7077...
// :: WARN AppClient$ClientEndpoint: Failed to connect to master master:
java.io.IOException: Failed to connect to master/192.168.3.129:
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:)
at org.apache.spark.rpc.netty.Outbox$$anon$.call(Outbox.scala:)
at org.apache.spark.rpc.netty.Outbox$$anon$.call(Outbox.scala:)
at java.util.concurrent.FutureTask.run(FutureTask.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: java.net.ConnectException: Connection refused: master/192.168.3.129:
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:)
at io.netty.util.concurrent.SingleThreadEventExecutor$.run(SingleThreadEventExecutor.java:)
... more
// :: INFO AppClient$ClientEndpoint: Connecting to master spark://master:7077...
// :: WARN AppClient$ClientEndpoint: Failed to connect to master master:
java.io.IOException: Failed to connect to master/192.168.3.129:
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:)
at org.apache.spark.rpc.netty.Outbox$$anon$.call(Outbox.scala:)
at org.apache.spark.rpc.netty.Outbox$$anon$.call(Outbox.scala:)
at java.util.concurrent.FutureTask.run(FutureTask.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: java.net.ConnectException: Connection refused: master/192.168.3.129:
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:)
at io.netty.util.concurrent.SingleThreadEventExecutor$.run(SingleThreadEventExecutor.java:)
... more
// :: INFO AppClient$ClientEndpoint: Connecting to master spark://master:7077...
// :: WARN AppClient$ClientEndpoint: Failed to connect to master master:
java.io.IOException: Failed to connect to master/192.168.3.129:
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:)
at org.apache.spark.rpc.netty.Outbox$$anon$.call(Outbox.scala:)
at org.apache.spark.rpc.netty.Outbox$$anon$.call(Outbox.scala:)
at java.util.concurrent.FutureTask.run(FutureTask.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: java.net.ConnectException: Connection refused: master/192.168.3.129:
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:)
at io.netty.util.concurrent.SingleThreadEventExecutor$.run(SingleThreadEventExecutor.java:)
... more
// :: INFO AppClient$ClientEndpoint: Connecting to master spark://master:7077...
// :: ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
// :: INFO AppClient$ClientEndpoint: Connecting to master spark://master:7077...
// :: WARN SparkDeploySchedulerBackend: Application ID is not initialized yet.
// :: WARN AppClient$ClientEndpoint: Failed to connect to master master:
java.io.IOException: Failed to connect to master/192.168.3.129:
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:)
at org.apache.spark.rpc.netty.Outbox$$anon$.call(Outbox.scala:)
at org.apache.spark.rpc.netty.Outbox$$anon$.call(Outbox.scala:)
at java.util.concurrent.FutureTask.run(FutureTask.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: java.net.ConnectException: Connection refused: master/192.168.3.129:
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:)
at io.netty.util.concurrent.SingleThreadEventExecutor$.run(SingleThreadEventExecutor.java:)
... more
// :: INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port .
// :: INFO NettyBlockTransferService: Server created on
// :: INFO BlockManagerMaster: Trying to register BlockManager
// :: INFO BlockManagerMasterEndpoint: Registering block manager 192.168.3.129: with 517.4 MB RAM, BlockManagerId(driver, 192.168.3.129, )
// :: INFO BlockManagerMaster: Registered BlockManager
// :: INFO SparkUI: Stopped Spark web UI at http://192.168.3.129:4040
// :: INFO SparkDeploySchedulerBackend: Shutting down all executors
// :: INFO SparkDeploySchedulerBackend: Asking each executor to shut down
// :: WARN AppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master
// :: ERROR MapOutputTrackerMaster: Error communicating with MapOutputTracker
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:)
at scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:)
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:)
at scala.concurrent.Await$$anonfun$result$.apply(package.scala:)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:)
at scala.concurrent.Await$.result(package.scala:)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:)
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:)
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:)
at org.apache.spark.MapOutputTracker.askTracker(MapOutputTracker.scala:)
at org.apache.spark.MapOutputTracker.sendTracker(MapOutputTracker.scala:)
at org.apache.spark.MapOutputTrackerMaster.stop(MapOutputTracker.scala:)
at org.apache.spark.SparkEnv.stop(SparkEnv.scala:)
at org.apache.spark.SparkContext$$anonfun$stop$.apply$mcV$sp(SparkContext.scala:)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:)
at org.apache.spark.SparkContext.stop(SparkContext.scala:)
at org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend.dead(SparkDeploySchedulerBackend.scala:)
at org.apache.spark.deploy.client.AppClient$ClientEndpoint.markDead(AppClient.scala:)
at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$$$anonfun$run$.apply$mcV$sp(AppClient.scala:)
at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:)
at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$.run(AppClient.scala:)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$(ScheduledThreadPoolExecutor.java:)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at java.lang.Thread.run(Thread.java:)
// :: ERROR Utils: Uncaught exception in thread appclient-registration-retry-thread
org.apache.spark.SparkException: Error communicating with MapOutputTracker
at org.apache.spark.MapOutputTracker.askTracker(MapOutputTracker.scala:)
at org.apache.spark.MapOutputTracker.sendTracker(MapOutputTracker.scala:)
at org.apache.spark.MapOutputTrackerMaster.stop(MapOutputTracker.scala:)
at org.apache.spark.SparkEnv.stop(SparkEnv.scala:)
at org.apache.spark.SparkContext$$anonfun$stop$.apply$mcV$sp(SparkContext.scala:)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:)
at org.apache.spark.SparkContext.stop(SparkContext.scala:)
at org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend.dead(SparkDeploySchedulerBackend.scala:)
at org.apache.spark.deploy.client.AppClient$ClientEndpoint.markDead(AppClient.scala:)
at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$$$anonfun$run$.apply$mcV$sp(AppClient.scala:)
at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:)
at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$.run(AppClient.scala:)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$(ScheduledThreadPoolExecutor.java:)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:)
at scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:)
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:)
at scala.concurrent.Await$$anonfun$result$.apply(package.scala:)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:)
at scala.concurrent.Await$.result(package.scala:)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:)
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:)
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:)
at org.apache.spark.MapOutputTracker.askTracker(MapOutputTracker.scala:)
... more
// :: INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
// :: INFO SparkContext: Successfully stopped SparkContext
// :: ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[appclient-registration-retry-thread,,main]
org.apache.spark.SparkException: Exiting due to error from cluster scheduler: All masters are unresponsive! Giving up.
at org.apache.spark.scheduler.TaskSchedulerImpl.error(TaskSchedulerImpl.scala:)
at org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend.dead(SparkDeploySchedulerBackend.scala:)
at org.apache.spark.deploy.client.AppClient$ClientEndpoint.markDead(AppClient.scala:)
at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$$$anonfun$run$.apply$mcV$sp(AppClient.scala:)
at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:)
at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$.run(AppClient.scala:)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$(ScheduledThreadPoolExecutor.java:)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at java.lang.Thread.run(Thread.java:)
// :: INFO DiskBlockManager: Shutdown hook called
// :: INFO ShutdownHookManager: Shutdown hook called
// :: INFO ShutdownHookManager: Deleting directory /tmp/spark-bf09944d---89c6-e8b415c9c315/userFiles-12e582c1--490f-a8a2-64264d764463
// :: INFO ShutdownHookManager: Deleting directory /tmp/spark-7c648867-90b8-4d3c-af09-b1f3d16d1b30
// :: INFO ShutdownHookManager: Deleting directory /tmp/spark-bf09944d---89c6-e8b415c9c315
2：解决方法，是你必须先启动你的Spark集群，这样再启动Spark Shell即可：
在master节点，启动你的spark集群，启动方式如下所示：
[root@master spark-1.6.1-bin-hadoop2.6]# sbin/start-all.sh
然后再启动你的Spark Shell即可，解决上面的错误：
[root@master spark-1.6.1-bin-hadoop2.6]# bin/spark-shell --master spark://master:7077 --executor-memory 512M --total-executor-cores 2
启动的内容，注意一些重点内容：
第33行，第47行，第119行。
[root@master spark-1.6.1-bin-hadoop2.6]# bin/spark-shell --master spark://master:7077 --executor-memory 512M --total-executor-cores 2
18/02/22 01:51:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/02/22 01:51:00 INFO SecurityManager: Changing view acls to: root
18/02/22 01:51:00 INFO SecurityManager: Changing modify acls to: root
18/02/22 01:51:00 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
18/02/22 01:51:00 INFO HttpServer: Starting HTTP Server
18/02/22 01:51:00 INFO Utils: Successfully started service 'HTTP class server' on port 58729.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.6.1
/_/ Using Scala version 2.10.5 (Java HotSpot(TM) Client VM, Java 1.7.0_65)
Type in expressions to have them evaluated.
Type :help for more information.
18/02/22 01:51:06 INFO SparkContext: Running Spark version 1.6.1
18/02/22 01:51:06 INFO SecurityManager: Changing view acls to: root
18/02/22 01:51:06 INFO SecurityManager: Changing modify acls to: root
18/02/22 01:51:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
18/02/22 01:51:06 INFO Utils: Successfully started service 'sparkDriver' on port 45298.
18/02/22 01:51:06 INFO Slf4jLogger: Slf4jLogger started
18/02/22 01:51:06 INFO Remoting: Starting remoting
18/02/22 01:51:07 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.3.129:36868]
18/02/22 01:51:07 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 36868.
18/02/22 01:51:07 INFO SparkEnv: Registering MapOutputTracker
18/02/22 01:51:07 INFO SparkEnv: Registering BlockManagerMaster
18/02/22 01:51:07 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-3b5e312e-7f6b-491d-8539-4a5f38d3839a
18/02/22 01:51:07 INFO MemoryStore: MemoryStore started with capacity 517.4 MB
18/02/22 01:51:07 INFO SparkEnv: Registering OutputCommitCoordinator
18/02/22 01:51:07 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/02/22 01:51:07 INFO SparkUI: Started SparkUI at http://192.168.3.129:4040
18/02/22 01:51:07 INFO AppClient$ClientEndpoint: Connecting to master spark://master:7077...
18/02/22 01:51:08 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20180222015108-0000
18/02/22 01:51:08 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46282.
18/02/22 01:51:08 INFO NettyBlockTransferService: Server created on 46282
18/02/22 01:51:08 INFO BlockManagerMaster: Trying to register BlockManager
18/02/22 01:51:08 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.3.129:46282 with 517.4 MB RAM, BlockManagerId(driver, 192.168.3.129, 46282)
18/02/22 01:51:08 INFO BlockManagerMaster: Registered BlockManager
18/02/22 01:51:08 INFO AppClient$ClientEndpoint: Executor added: app-20180222015108-0000/0 on worker-20180222174932-192.168.3.130-39616 (192.168.3.130:39616) with 1 cores
18/02/22 01:51:08 INFO SparkDeploySchedulerBackend: Granted executor ID app-20180222015108-0000/0 on hostPort 192.168.3.130:39616 with 1 cores, 512.0 MB RAM
18/02/22 01:51:08 INFO AppClient$ClientEndpoint: Executor added: app-20180222015108-0000/1 on worker-20180222174932-192.168.3.131-58163 (192.168.3.131:58163) with 1 cores
18/02/22 01:51:08 INFO SparkDeploySchedulerBackend: Granted executor ID app-20180222015108-0000/1 on hostPort 192.168.3.131:58163 with 1 cores, 512.0 MB RAM
18/02/22 01:51:09 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
18/02/22 01:51:09 INFO SparkILoop: Created spark context..
Spark context available as sc.
18/02/22 01:51:10 INFO AppClient$ClientEndpoint: Executor updated: app-20180222015108-0000/1 is now RUNNING
18/02/22 01:51:10 INFO AppClient$ClientEndpoint: Executor updated: app-20180222015108-0000/0 is now RUNNING
18/02/22 01:51:16 INFO HiveContext: Initializing execution hive, version 1.2.1
18/02/22 01:51:17 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
18/02/22 01:51:17 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
18/02/22 01:51:21 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
18/02/22 01:51:22 INFO ObjectStore: ObjectStore, initialize called
18/02/22 01:51:26 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
18/02/22 01:51:26 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
18/02/22 01:51:26 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
18/02/22 01:51:30 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
18/02/22 01:51:35 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
18/02/22 01:51:38 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/02/22 01:51:38 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/02/22 01:51:39 INFO SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slaver2:55056) with ID 1
18/02/22 01:51:39 INFO BlockManagerMasterEndpoint: Registering block manager slaver2:57607 with 146.2 MB RAM, BlockManagerId(1, slaver2, 57607)
18/02/22 01:51:39 INFO SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slaver1:47165) with ID 0
18/02/22 01:51:40 INFO BlockManagerMasterEndpoint: Registering block manager slaver1:38278 with 146.2 MB RAM, BlockManagerId(0, slaver1, 38278)
18/02/22 01:51:40 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/02/22 01:51:40 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/02/22 01:51:40 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
18/02/22 01:51:40 INFO ObjectStore: Initialized ObjectStore
18/02/22 01:51:41 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
18/02/22 01:51:42 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
Java HotSpot(TM) Client VM warning: You have loaded library /tmp/libnetty-transport-native-epoll870809507217922299.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
18/02/22 01:51:44 INFO HiveMetaStore: Added admin role in metastore
18/02/22 01:51:44 INFO HiveMetaStore: Added public role in metastore
18/02/22 01:51:44 INFO HiveMetaStore: No user is added in admin role, since config is empty
18/02/22 01:51:45 INFO HiveMetaStore: 0: get_all_databases
18/02/22 01:51:45 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_all_databases
18/02/22 01:51:45 INFO HiveMetaStore: 0: get_functions: db=default pat=*
18/02/22 01:51:45 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_functions: db=default pat=*
18/02/22 01:51:45 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
18/02/22 01:51:46 INFO SessionState: Created HDFS directory: /tmp/hive/root
18/02/22 01:51:46 INFO SessionState: Created local directory: /tmp/root
18/02/22 01:51:46 INFO SessionState: Created local directory: /tmp/afacd186-3b65-4cf9-a9b3-dad36055ed80_resources
18/02/22 01:51:46 INFO SessionState: Created HDFS directory: /tmp/hive/root/afacd186-3b65-4cf9-a9b3-dad36055ed80
18/02/22 01:51:46 INFO SessionState: Created local directory: /tmp/root/afacd186-3b65-4cf9-a9b3-dad36055ed80
18/02/22 01:51:46 INFO SessionState: Created HDFS directory: /tmp/hive/root/afacd186-3b65-4cf9-a9b3-dad36055ed80/_tmp_space.db
18/02/22 01:51:46 INFO HiveContext: default warehouse location is /user/hive/warehouse
18/02/22 01:51:46 INFO HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
18/02/22 01:51:46 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
18/02/22 01:51:46 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
18/02/22 01:51:47 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
18/02/22 01:51:47 INFO ObjectStore: ObjectStore, initialize called
18/02/22 01:51:47 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
18/02/22 01:51:47 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
18/02/22 01:51:47 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
18/02/22 01:51:48 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
18/02/22 01:51:49 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
18/02/22 01:51:51 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/02/22 01:51:51 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/02/22 01:51:51 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/02/22 01:51:51 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/02/22 01:51:51 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
18/02/22 01:51:51 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
18/02/22 01:51:51 INFO ObjectStore: Initialized ObjectStore
18/02/22 01:51:51 INFO HiveMetaStore: Added admin role in metastore
18/02/22 01:51:51 INFO HiveMetaStore: Added public role in metastore
18/02/22 01:51:51 INFO HiveMetaStore: No user is added in admin role, since config is empty
18/02/22 01:51:52 INFO HiveMetaStore: 0: get_all_databases
18/02/22 01:51:52 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_all_databases
18/02/22 01:51:52 INFO HiveMetaStore: 0: get_functions: db=default pat=*
18/02/22 01:51:52 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_functions: db=default pat=*
18/02/22 01:51:52 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
18/02/22 01:51:52 INFO SessionState: Created local directory: /tmp/e209230b-e230-4688-9b83-b04d182b952d_resources
18/02/22 01:51:52 INFO SessionState: Created HDFS directory: /tmp/hive/root/e209230b-e230-4688-9b83-b04d182b952d
18/02/22 01:51:52 INFO SessionState: Created local directory: /tmp/root/e209230b-e230-4688-9b83-b04d182b952d
18/02/22 01:51:52 INFO SessionState: Created HDFS directory: /tmp/hive/root/e209230b-e230-4688-9b83-b04d182b952d/_tmp_space.db
18/02/22 01:51:52 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext. scala>
Caused by: java.net.ConnectException: Connection refused: master/192.168.3.129:7077的更多相关文章
- Caused by: java.net.ConnectException: Call From master/192.168.199.130 to master:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.
1:安装好hive,准备启动的时候出现下面的错误(由于hive是基于Hadoop的,所以必须先将你的集群启动起来,我就是没有启动集群,直接启动hive导致的错误): [root@master bin] ...
- error: not found: value sqlContext/import sqlContext.implicits._/error: not found: value sqlContext /import sqlContext.sql/Caused by: java.net.ConnectException: Connection refused
1.今天启动启动spark的spark-shell命令的时候报下面的错误,百度了很多,也没解决问题,最后想着是不是没有启动hadoop集群的问题 ,可是之前启动spark-shell命令是不用启动ha ...
- Caused by: java.net.ConnectException: Connection refused/Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
1.使用sqoop技术将mysql的数据导入到Hive出现的错误如下所示: 第一次使用命令如下所示: [hadoop@slaver1 sqoop--cdh5.3.6]$ bin/sqoop impor ...
- springboot+mongodb报错Caused by: java.net.ConnectException: Connection refused (Connection refused)
com.mongodb.MongoSocketOpenException: Exception opening socket at com.mongodb.connection.SocketStrea ...
- 访问Storm ui界面，出现org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused的问题解决（图文详解）
不多说,直接上干货! 前期博客 apache-storm-0.9.6.tar.gz的集群搭建(3节点)(图文详解) 问题详情 org.apache.thrift7.transport.TTranspo ...
- 异常-CDH的service无法启动并抛出异常-org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused (Connection refused)
1 详细异常 org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused (Connectio ...
- exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
1.虽然,不是大错,还说要贴一下,由于我运行run-example streaming.NetworkWordCount localhost 9999的测试案例,出现的错误,第一感觉就是Spark没有 ...
- java.net.ConnectException: Call From slaver1/192.168.19.128 to slaver1:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org
1:练习spark的时候,操作大概如我读取hdfs上面的文件,然后spark懒加载以后,我读取详细信息出现如下所示的错误,错误虽然不大,我感觉有必要记录一下,因为错误的起因是对命令的不熟悉造成的,错误 ...
- 启动storm之后浏览器访问报错,org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
原因是zookeeper没有启动 Internal Server Error org.apache.thrift7.transport.TTransportException: java.net.Co ...
随机推荐
- 后台拼接json字符串，传到前台时注意特殊符号处理
1.后台拼接常用的形式: sb.AppendFormat("\"字段名\":\"{0}\"", i + 1); 这个要注意字符串截断 ...
- boost::asio实现一个echo服务器
以前使用ACE实现Server框架,但是觉得太笨重,决定采用boost.asio来写服务器程序: 1.服务器构建在linux上面:当然也可以在windows下运行 2.io部分采用非阻塞模式.业务逻辑 ...
- linux系统的三种网络连接模式
VMWare提供了三种工作模式,它们是bridged(桥接模式).NAT(网络地址转换模式)和host-only(主机模式).要想在网络管理和维护中合理应用它们,你就应该先了解一下这三种工作模式. 1 ...
- redhat yum
背景 这个redhat可让我好一顿折腾,对于这个yum的安装,如果成功则罢,不成功可有的收拾的.还是centos比较好啊. 转载 1 下载下面四个软件包(可以从163下载,版本号自己搜索) wget ...
- freeswitch反注册记录
应用情景: 使用阿里服务器,落地使用本地的模拟线路(O口网关). 1.FreeSWITCH 服务器开一个账号,比如 5000 internal , O口 SIP设置页面按照网关注册 5000 的账号信 ...
- ubuntu 问题
1.打开ubuntu之后的开启页面出现:所选模式均不匹配可能的模式:为 CRTC 63 尝试模式CRTC 63:尝试 800x600@60Hz 模式输出在 1366x768@60Hz (通过 0)CR ...
- PHP字符串比较,看起来值完全一样,但是就是不相等的解决方案(‌)
1 前言 字符串比较,看起来完全一样,然后用strcmp比较,永远不相等,用var_dump查看才知道,其中一个字符多了看不见的特殊符号,而我长度是3. 2 样例 当你选中它,显示出来的就是人眼所见 ...
- spring中BeanFactory和FactoryBean的区别
共同点: 都是接口 区别: BeanFactory 以Factory结尾,表示它是一个工厂类,用于管理Bean的一个工厂 在Spring中,所有的Bean都是由BeanFactory(也就是IOC容器 ...
- Solidity(address的四个方法)
address的四个方法send,call,callcode,delegatecall 例子:发送以太币的send方法//下面是send方法,涉及到以太币的情况可能用到payable,senddemo ...
- mysql5.7设置简单密码报错ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
注:本文来源于< mysql5.7设置简单密码报错ERROR 1819 (HY000): Your password does not satisfy the current policy r ...