mvn 引入自定义jar 解决 mongo-spark 报错
[root@hadoop1 bin]# ./spark-submit --class myprojectpackaging.App /usr/local/hadoop/spark-2.2.-bin-hadoop2./mycode/myprojectname/target/myprojectname-1.0-SNAPSHOT.jar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop/spark-2.2.-bin-hadoop2./jars/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/hadoop-2.6./share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Hello World!
// :: INFO spark.SparkContext: Running Spark version 2.2.
// :: WARN util.Utils: Your hostname, hadoop1 resolves to a loopback address: 127.0.0.1; using 192.168.2.51 instead (on interface eno1)
// :: WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
// :: INFO spark.SparkContext: Submitted application: MongoSparkConnectorIntro
// :: INFO spark.SecurityManager: Changing view acls to: root
// :: INFO spark.SecurityManager: Changing modify acls to: root
// :: INFO spark.SecurityManager: Changing view acls groups to:
// :: INFO spark.SecurityManager: Changing modify acls groups to:
// :: INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
// :: INFO util.Utils: Successfully started service 'sparkDriver' on port .
// :: INFO spark.SparkEnv: Registering MapOutputTracker
// :: INFO spark.SparkEnv: Registering BlockManagerMaster
// :: INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
// :: INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
// :: INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-a5531bf0-ae40-44be-9a91-a2aca6c81267
// :: INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
// :: INFO spark.SparkEnv: Registering OutputCommitCoordinator
// :: INFO util.log: Logging initialized @1291ms
// :: INFO server.Server: jetty-9.3.z-SNAPSHOT
// :: INFO server.Server: Started @1353ms
// :: INFO server.AbstractConnector: Started ServerConnector@6548bb7d{HTTP/1.1,[http/1.1]}{0.0.0.0:}
// :: INFO util.Utils: Successfully started service 'SparkUI' on port .
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7668d560{/jobs,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@438bad7c{/jobs/json,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4fdf8f12{/jobs/job,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6979efad{/jobs/job/json,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4a67318f{/stages,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@17f9344b{/stages/json,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@54e81b21{/stages/stage,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@50cf5a23{/stages/stage/json,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@273c947f{/stages/pool,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1af1347d{/stages/pool/json,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@20765ed5{/storage,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2899a8db{/storage/json,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@c1a4620{/storage/rdd,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@130a0f66{/storage/rdd/json,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@12365c88{/environment,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2237bada{/environment/json,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5710768a{/executors,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6e0d4a8{/executors/json,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@{/executors/threadDump,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5bf61e67{/executors/threadDump/json,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@b273a59{/static,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@30865a90{/,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@777c9dc9{/api,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@22175d4f{/jobs/job/kill,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3b809711{/stages/stage/kill,null,AVAILABLE,@Spark}
// :: INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.2.51:4040
// :: INFO spark.SparkContext: Added JAR file:/usr/local/hadoop/spark-2.2.-bin-hadoop2./mycode/myprojectname/target/myprojectname-1.0-SNAPSHOT.jar at spark://192.168.2.51:56112/jars/myprojectname-1.0-SNAPSHOT.jar with timestamp 1512096533656
// :: INFO executor.Executor: Starting executor ID driver on host localhost
// :: INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port .
// :: INFO netty.NettyBlockTransferService: Server created on 192.168.2.51:
// :: INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
// :: INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.2.51, , None)
// :: INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.2.51: with 366.3 MB RAM, BlockManagerId(driver, 192.168.2.51, , None)
// :: INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.2.51, , None)
// :: INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.2.51, , None)
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@44c5a16f{/metrics/json,null,AVAILABLE,@Spark}
// :: INFO internal.SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/usr/local/hadoop/spark-2.2.0-bin-hadoop2.7/bin/spark-warehouse/').
// :: INFO internal.SharedState: Warehouse path is 'file:/usr/local/hadoop/spark-2.2.0-bin-hadoop2.7/bin/spark-warehouse/'.
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@70e3f36f{/SQL,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@23e44287{/SQL/json,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@61f3fbb8{/SQL/execution,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@432034a{/SQL/execution/json,null,AVAILABLE,@Spark}
// :: INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@173373b4{/static/sql,null,AVAILABLE,@Spark}
// :: INFO state.StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
Exception in thread "main" java.lang.NoClassDefFoundError: com/mongodb/spark/MongoSpark
at myprojectpackaging.App.main(App.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$(SparkSubmit.scala:)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.mongodb.spark.MongoSpark
at java.net.URLClassLoader.findClass(URLClassLoader.java:)
at java.lang.ClassLoader.loadClass(ClassLoader.java:)
at java.lang.ClassLoader.loadClass(ClassLoader.java:)
... more
// :: INFO spark.SparkContext: Invoking stop() from shutdown hook
// :: INFO server.AbstractConnector: Stopped Spark@6548bb7d{HTTP/1.1,[http/1.1]}{0.0.0.0:}
// :: INFO ui.SparkUI: Stopped Spark web UI at http://192.168.2.51:4040
// :: INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
// :: INFO memory.MemoryStore: MemoryStore cleared
// :: INFO storage.BlockManager: BlockManager stopped
// :: INFO storage.BlockManagerMaster: BlockManagerMaster stopped
// :: INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
// :: INFO spark.SparkContext: Successfully stopped SparkContext
// :: INFO util.ShutdownHookManager: Shutdown hook called
// :: INFO util.ShutdownHookManager: Deleting directory /tmp/spark-94c896a5-3f94-48ce-92ff-2e5b94204d0b
mvn 引入自定义jar 解决 mongo-spark 报错的更多相关文章
- larave5.6 引入自定义函数库时,报错不能重复定义
方法一:使用function_exists判断 方法二:使用命名空间 namespace test; function test(){ echo 'test/test'; } namespace te ...
- idea lib下有jar包但是仍然报错 找不到类
现象: idea lib下有jar包但是仍然报错 找不到类 但是有个奇怪现象 同样的配置下项目在eclipse中可以正常编译 启动. package com.puhui.car.aspect; imp ...
- maven仓库中有jar包pom还报错
maven仓库中有jar包pom还报错 就报错,咋啦? 这个包来源不明,自己拷贝进来的吧?你当我mvn是傻子?我要去网上验证一下: 我自己有个_remote.respositories文件,如果自己用 ...
- 解决:MySQL 报错:1045 - Access denied for user 'root'@'localhost'(using password YES)
一.前言 今年疯狂迷上了开源,只要看到好的开源项目,就会不顾一切一股脑扎进去研究,五一期间发现一个很好的关于众筹的开源项目,但不巧,这个项目竟然是 PHP 写的,没学过 PHP,自然对这个开源项目毫无 ...
- IDEA导出jar包后运行报错 找不到或无法加载主类
开发工具:IDEA16 运行环境:ubuntu 问题:根据网上的Idea导出jar包的方法,将我的项目导出jar包后运行报错:找不到或无法加载主类. 为了找到这个原因,我重新搭建了一个测试例子,在 ...
- spark报错处理
Spark报错处理 1.问题:org.apache.spark.SparkException: Exception thrown in awaitResult 分析:出现这个情况的原因是spark启动 ...
- 解决kylin查询报错:org.apache.kylin.rest.exception.InternalErrorException
报错信息: -- ::, ERROR [Query 12e9c054-760c---b1f06724c9b6-] service.QueryService: : Exception when exec ...
- homestead虚拟机,通过npm下载依赖包和解决运行gulp报错问题 yarn出错问题
homestead虚拟机,通过npm下载依赖包和解决运行gulp报错问题 yarn出错问题 1. 在虚拟器运行 npm 下载依赖组件时报错: npm ERR! EPROTO: protocol err ...
- spark报错:invalid token
启动spark报错,启动container失败,去看yarn的日志,显示invalid token, 经过排查是hadoop子节点的配置和主节点的配置不一致导致的,同步之后,问题解决.
随机推荐
- 九度oj 题目1086:最小花费
题目描述: 在某条线路上有N个火车站,有三种距离的路程,L1,L2,L3,对应的价格为C1,C2,C3.其对应关系如下: 距离s 票价 0<S<=L1 ...
- BZOJ 1087 [SCOI2005]互不侵犯King ——状压DP
[题目分析] 沉迷水题,吃枣药丸. [代码] #include <cstdio> #include <cstring> #include <iostream> #i ...
- VMware虚拟机 NAT模式 配置静态ip
前言:Ubuntu 16.04 VMware虚拟机 NAT模式 配置静态ip,这个问题困扰我好长时间,桥接的静态ip我会了,然而用NAT 的方式配置集群会更好.(NAT 方式客户机之间的通讯不经过路由 ...
- VMware 如何通过现有虚拟机克隆新的虚拟机 (图文)
本文做的是克隆主机,并重命名主机名,做好主机名与IP的对应关系,并可以通过主机名访问对方的主机. 首先说一下克隆虚拟机的作用 克隆虚拟机的作用 因工作需要,需要用到多个虚拟机环境时,再新建几个比较麻烦 ...
- poj2513字典树+欧拉图判断+并查集断连通
题意:俩头带有颜色的木棒,要求按颜色同的首尾相连,可能否? 思路:棒子本身是一条边,以俩端为顶点(同颜色共点),即求是否有无向图欧拉路(每条棒子只有一根, 边只能用一次,用一次边即选一次棒子). 先判 ...
- Strom运行监控
https://blog.csdn.net/wenxuechaozhe/article/details/80548709
- Nginx反向代理新篇-使用location对多个URL做反向代理
1.原理 Nginx解析location/后面的字符串,配置不同的字符串匹配不同的URL进行反向代理. 2.nginx.conf配置文件 worker_processes 1; events { wo ...
- codevs——1507 酒厂选址
1507 酒厂选址 时间限制: 1 s 空间限制: 128000 KB 题目等级 : 黄金 Gold 题解 查看运行结果 题目描述 Description Abstinence(戒酒) ...
- codevs——1958 刺激
1958 刺激 时间限制: 1 s 空间限制: 128000 KB 题目等级 : 黄金 Gold 题解 题目描述 Description saffah的一个朋友S酷爱滑雪,并且追求刺 ...
- Java 一个?格式的解决
用Java 出现了这样的一个问题?好几天都没解决掉 然后最近一直找资料 截个图: 本来格式中时没有这个?号的,代码里面用GBK和utf-8都不能解决. 即使我加了 Str.trim(Str)去除 字符 ...