(待整理)flume操作----------hivelogsToHDFS案例----------运行时,发生NoClassDefFoundError错误
1.
2.错误日志
命令为 bin/flume-ng agent --name a2 --conf conf/ --conf-file job/file-hdfs.conf Info: Sourcing environment configuration script /opt/modules/flume/conf/flume-env.sh
Info: Including Hive libraries found via () for Hive access
+ exec /opt/modules/jdk1.8.0_121/bin/java -Xmx20m -cp '/opt/modules/flume/conf:/opt/modules/flume/lib/*:/lib/*' -Djava.library.path= org.apache.flume.node.Application --name a2 --conf-file job/file-hdfs.conf
Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoClassDefFoundError: org/apache/htrace/SamplerBuilder
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:635)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
3.情况好转
把如图的两个jar放入flume下的lib目录
重新运行flume,没有报错,但是没有动静,如图
同时启动hive,在hdfs并没有产生/flume/%Y%m%d/%H目录
问题待解决!!!
4.进一步实验
把那两个jar移除,同时把conf中sink指定的02号机namenode关闭掉,再启动01号机上的flume,没有发生错误但是在hdfs上任然没有flume目录
猜想原因:能够不报错,可能是因为JVM记录着原来的变量??????
问题待解决!!!
案列3,发生同样的情况,HDFS上没有flume文件夹
在命令中加入了输出日志
bin/flume-ng agent --conf conf/ --name a3 --conf-file job/dir-hdfs.conf -Dflume.root.logger=INFO,console
发现错误日志
-- ::, (conf-file-poller-) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:)] Starting Source r3
-- ::, (lifecycleSupervisor--) [INFO - org.apache.flume.source.SpoolDirectorySource.start(SpoolDirectorySource.java:)] SpoolDirectorySource source starting with directory: /opt/module/flume/upload
-- ::, (lifecycleSupervisor--) [ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:)] Unable to start EventDrivenSourceRunner: { source:Spool Directory source r3: { spoolDir: /opt/module/flume/upload } } - Exception follows.
java.lang.IllegalStateException: Directory does not exist: /opt/module/flume/upload
at com.google.common.base.Preconditions.checkState(Preconditions.java:)
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.<init>(ReliableSpoolingFileEventReader.java:)
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.<init>(ReliableSpoolingFileEventReader.java:)
上述日志中错误原因是:
conf中少了s
改正之后重新运行flume:
同时上传NOTICE文件到upload中,此时upload文件中
但是flume打印出来的日志提示:
[ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:447)] process failed
java.lang.NoClassDefFoundError: org/apache/htrace/SamplerBuilder
-- ::, (lifecycleSupervisor--) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:)] Component type: SOURCE, name: r3 started
-- ::, (pool--thread-) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:)] Last read took us just up to a file boundary. Rolling to the next file, if there is one.
-- ::, (pool--thread-) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.rollCurrentFile(ReliableSpoolingFileEventReader.java:)] Preparing to move file /opt/modules/flume/upload/NOTICE to /opt/modules/flume/upload/NOTICE.COMPLETED
-- ::, (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.HDFSDataStream.configure(HDFSDataStream.java:)] Serializer = TEXT, UseRawLocalFileSystem = false
-- ::, (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:)] Creating hdfs://hadoop-senior02.itguigu.com:9000/flume/upload/20190123/07/upload-.1548198334086.tmp
-- ::, (hdfs-k3-call-runner-) [WARN - org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-- ::, (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:)] process failed
java.lang.NoClassDefFoundError: org/apache/htrace/SamplerBuilder
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:)
经查询:flime/lib中缺少htrace-core-3.1.0-incubating.jar包,mvn工程的话,通过mvn install安装(参考http://blog.51cto.com/enetq/1827028)。我直接找到此jar包手动拷贝进lib/xia
上面问题解决了,继续:cp NOTICE upload/,但是flume报错,日志如下:
java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
Info: Sourcing environment configuration script /opt/modules/flume/conf/flume-env.sh
Info: Including Hive libraries found via () for Hive access
+ exec /opt/modules/jdk1..0_121/bin/java -Xmx20m -cp '/opt/modules/flume/conf:/opt/modules/flume/lib/*:/lib/*' -Djava.library.path= org.apache.flume.node.Application -n a3 -f job/dir-hdfs.conf
Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
at org.apache.hadoop.ipc.Server.<clinit>(Server.java:)
at org.apache.hadoop.ipc.ProtobufRpcEngine.<clinit>(ProtobufRpcEngine.java:)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:)
解决:把commons-io-2.4.jar放进flume/lib/目录下
再重新过程,出现HDFS IO error,见日志:
-- ::, (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:)] HDFS IO error
java.net.ConnectException: Call From hadoop-senior01/192.168.10.20 to hadoop-senior02: failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:)
(插曲) 因为file-hdfs.conf,之前也出现了问题,现在配置基本改好了。运行此配置出现,如日志所示问题:
xecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at java.lang.Thread.run(Thread.java:)
-- ::, (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:)] Creating hdfs://hadoop-senior01/flume/20190123/15/logs-.1548230362663.tmp
-- ::, (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:)] HDFS IO error
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category WRITE is not supported in state standby
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:)
打不开HA中的standby节点中的目录,改成active namenode之后,flume运行过程成功!
继续,dir-file.conf还是出问题,经对比file-file.conf(成功),dir-file.conf中指定了9000端口,去掉,成功!!!
a2.sinks.k2.hdfs.path = hdfs://hadoop-senior02/flume/%Y%m%d/
%H
有关参考:https://blog.csdn.net/dai451954706/article/details/50449436
https://blog.csdn.net/woloqun/article/details/81350323
(待整理)flume操作----------hivelogsToHDFS案例----------运行时,发生NoClassDefFoundError错误的更多相关文章
- cocos2d-x 运行时xcode提示错误:"vtable for XXX", referenced from 问题已解决;
vtable/引用和虚函数相关,今天在添加一个层的时候报了这个错误,很低级的错误,忘了实现虚函数了(谨记!!) 若如果实现了虚函数还依然如此的话,可能是创建的时候忘了钩上 -desktop 选项了,把 ...
- (已解决)cocos2d-x 运行时xcode提示错误:"vtable for XXX", referenced from;
vtable/引用和虚函数相关,今天在添加一个层的时候报了这个错误,很低级的错误,忘了实现虚函数了(谨记!!) 若如果实现了虚函数还依然如此的话,可能是创建的时候忘了钩上 -desktop 选项了,把 ...
- Flume实战案例运维篇
Flume实战案例运维篇 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.Flume概述 1>.什么是Flume Flume是一个分布式.可靠.高可用的海量日志聚合系统,支 ...
- Java运行时,各种类型存储介绍
Java的内存分配 Java程序运行时的内存结构分成:方法区.栈内存.堆内存.本地方法栈几种. 方法区 存放装载的类数据信息,包括:基本信息:每个类的全限定名.每个类的直接超类的全限定 ...
- .net core跨平台应用研究-ubuntu core下配置.net core运行时
引言 年初研究了一阵子.net core跨平台应用,先后发表了几篇应用研究的文章.因工作原因,忙于项目上线,有一阵子没来博客园写文章了.最近项目基本收尾,抽空翻了下自己的博客,廖廖几篇文章,真让人汗颜 ...
- 乘风破浪,.Net Core遇见Dapr,为云原生而生的分布式应用运行时
Dapr是一个由微软主导的云原生开源项目,国内云计算巨头阿里云也积极参与其中,2019年10月首次发布,到今年2月正式发布V1.0版本.在不到一年半的时间内,github star数达到了1.2万,超 ...
- runtime运行时
/** * Describes the instance variables declared by a class. * * @param cls The class to inspect. * @ ...
- Objective C运行时(runtime)
#import <objc/runtime.h> void setBeingRemoved(id __self, SEL _cmd) { NSLog(@"------------ ...
- iOS运行时Runtime浅析
运行时是iOS中一个很重要的概念,iOS运行过程中都会被转化为runtime的C代码执行.例如[target doSomething];会被转化成objc)msgSend(target,@select ...
随机推荐
- 浅谈App原生开发、混合开发及HTML5开发的优劣
App混合开发(英文名:Hybrid App),是指在开发一款App产品的时候为了提高效率.节省成本即利用了原生的开发技术还应用了HTML5开发技术,是原生和HTML5技术的混合应用.目前App的开发 ...
- P vs NP
Complexity Class Computational problem Decision Problems Model of computation Time-complexity classe ...
- Python初学者第六天 列表操作练习
6day 列表练习: 1.创建一个空列表,命名为names,往里面添加old_drilver,rain,jack,shanshan,peiqi,black_girl: names = [] names ...
- LINUX系统文件类型分类
使用LINUX过程中,时常会对文件进行操作.但是linux中文件究竟有哪些分类呢?今天让我们大家来总结一下吧! 概述: linux文件可以分为五种类型,分别为:普通文件.目录.字符设备或块设备.套接口 ...
- python 生成器&迭代器
列表生成式 要生成[1x1, 2x2, 3x3, ..., 10x10]>>> [x * x for x in range(1, 11)]for循环后面还可以加上if判断>&g ...
- CentOS7 Firewall超详细使用方法
CentOs7改变的最大处就是防火墙了,下面列用了常用的防火墙规则,端口转发和伪装 一.Firewalld基础规则 --get-default-zone 打印已设置为默认区域的当前区域,默认情况下默认 ...
- 魅族首页导航效果(不兼容IE)
<!DOCTYPE html> <html> <head lang="en"> <meta charset="UTF-8&quo ...
- js秒换成天时分
js秒换成天时分 function timeStamp( second_time ){ var time = parseInt(second_time) + "秒"; if( pa ...
- 谈谈Ajax(一)
解决办法是:比如我是使用SSM框架(Spring+SpringMVC+MyBatis/MyBatis Plus)开发web应用,通常Controller的参数列表与ajax的data保持一致即可,少一 ...
- WinCE下SQLCE数据库开发(VS,VB.net,VC++)
WinCE下SQLCE数据库开发(VS,VB.net,VC++) WinCE下SQLCE数据库开发 微软的SQL Server数据库由于其功能强大.方便使用,因此在很多行业都被广泛应用.基于智能设 ...