hbase读取快照数据-lzo压缩遇到的问题
1.读取hbase快照数据时报UnsatisfiedLinkError: no gplcompression in java.library.path错：
2019-09-04 17:36:07,441 ERROR [testJobName,1,ReaderTask_HbaseReader_0_1] GPLNativeCodeLoader:35 | Could not load native gpl library
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:31)
at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:60)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:127)
at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:116)
at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:328)
at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97)
at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452)
at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60)
at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209)
at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98)
at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73)
at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-09-04 17:36:07,460 ERROR [testJobName,1,ReaderTask_HbaseReader_0_1] LzoCodec:70 | Cannot load native-lzo without native-hadoop
在使用java 命令执行时可加上 -Djava.library.path=/opt/hadoop/lib/native，就没有此报错
java -Djava.library.path=/opt/hadoop/lib/native -cp lib/*:conf/*:stream-0.0.1.jar com.hbasereader.HbaseReader
2.解决完上述异常后，又出现native-lzo library not available异常
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available
at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:102)
at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452)
at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60)
at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209)
at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98)
at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73)
at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: native-lzo library not available
at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:330)
at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97)
... 21 more at com.suning.data.integration.engine.stream.core.exception.DiException.asDiException(DiException.java:51)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:131)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73)
at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available
at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:102)
at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452)
at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60)
at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209)
at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98)
at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152)
at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95)
... 12 more
Caused by: java.lang.RuntimeException: native-lzo library not available
at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:330)
at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97)
... 21 more
错误是从这报出来的：
经过一周的排查终于找到原因了，真是坑爹，因为引用了hadoop-gpl-compression-xxx.jar，而hadoop-gpl-compression是比较老的版本，
将hadoop-gpl-compression注释掉，使用hadoop-lzo的jar包，问题解决
目前在使用Twitter维护的版本，在GitHub上，下载源码 git clone https://github.com/twitter/hadoop-lzo.git ，使用maven编译打包后，将hadoop-lzo-xxx.jar加入到项目中，完美解决上述native-lzo library not available 问题。
至于如何使用maven编译hadoop-lzo，可以参照我的一篇博客： Hadoop3.1.2 + Hbase2.2.0 设置lzo压缩算法
注意：hadoop-gpl-compression已经过时，请使用hadoop-lzo
hadoop-gpl-compression已经过时，请使用hadoop-lzo
hadoop-gpl-compression已经过时，请使用hadoop-lzo
重要的事情说三遍！！！
hbase读取快照数据-lzo压缩遇到的问题的更多相关文章
- 开启hadoop和Hbase集群的lzo压缩功能（转）
原文链接:开启hadoop和Hbase集群的lzo压缩功能 问题导读: 1.如何启动hadoop.hbase集群的压缩功能? 2.lzo的作用是什么? 3.hadoop配置文件需要做哪些修改? 首先我 ...
- 使用TableSnapshotInputFormat读取Hbase快照数据
根据快照名称读取hbase快照中的数据,在网上查了好多资料,很少有资料能够给出清晰的方案,根据自己的摸索终于实现,现将代码贴出,希望能给大家有所帮助: public void read(org.apa ...
- hbase开放lzo压缩
hbase仅仅支持对gzip的压缩,对lzo压缩支持不好. 在io成为系统瓶颈的情况下,一般开启lzo压缩会提高系统的吞吐量. 但这须要參考详细的应用场景,即是否值得进行压缩.压缩率是否足够等等. ...
- 从hbase读取数据优化策略和实验对照结果
起因:工作须要.我须要每5分钟从hbase中.导出一部分数据,然后导入到ES中.可是在開始阶段编写的python脚本,我发现从hbase读取数据的速度较慢,耗费大量的时间.影响整个导数过程,恐怕无法在 ...
- Spark读取Hbase中的数据
大家可能都知道很熟悉Spark的两种常见的数据读取方式(存放到RDD中):(1).调用parallelize函数直接从集合中获取数据,并存入RDD中:Java版本如下: JavaRDD<Inte ...
- Hive直接读取Hbase及MySQL数据
0.概述 Hive对外提供了StorageHandler接口,提供了访问各种存储组件中的数据的能力.Hbase提供了HbaseStorageHandler,使得hive可以通过建立外部映射表访问hba ...
- HBase 高性能获取数据(多线程批量式解决办法) + MySQL和HBase性能测试比较
摘要: 在前篇博客里已经讲述了通过一个自定义 HBase Filter来获取数据的办法,在末尾指出此办法的性能是不能满足应用要求的,很显然对于如此成熟的HBase来说,高性能获取数据应该不是问题. ...
- hbase性能调优之压缩测试
文章概述: 1.顺序写 2.顺序读 3.随机写 4.随机读 5.SCAN数据 0 性能测试工具 hbase org.apache.hadoop.hbase.PerformanceEvaluation ...
- MR案例：MR和Hive中使用Lzo压缩
在MapReduce中使用lzo压缩 1).首先将数据文件在本地使用lzop命令压缩.具体配置过详见配置hadoop集群的lzo压缩 //压缩lzop,解压缩lzop -d [root@ncst wo ...
随机推荐
- ThreadLocal线程隔离
package com.cookie.test; import java.util.concurrent.atomic.AtomicInteger; /** * author : cxq * Date ...
- SpringBoot:如何优雅地处理全局异常？
之前用springboot的时候,只知道捕获异常使用try{}catch,一个接口一个try{}catch,这也是大多数开发人员异常处理的常用方式,虽然屡试不爽,但会造成一个问题,就是一个Contro ...
- Python 环境管理
Python 版本管理器:pyenv zsh 配置 # 安装 curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv ...
- JavaWeb——使用会话维持状态2
在这次的例子里面,将完成一类似购物车的功能,在客户访问网站的时候,会选中自己将要购买的商品,而购物车将始终维持着商品的状态,会话将联系起选择第一个商品(第一个请求),选择其他商品(其他请求)以及付款等 ...
- ServerResponse（服务器统一响应数据格式）
ServerResponse(服务器统一响应数据格式) 前言: 其实严格来说,ServerResponse应该归类到common包中.但是我实在太喜欢这玩意儿了.而且用得也非常频繁,所以忍不住推荐一下 ...
- 类spring ioc 泛型保留
类spring ioc 泛型保留 什么是泛型擦除 Java并不会传递泛型类,举个直观的栗子: @Component public class BaseProvider<T> { publi ...
- keras+ ctpn 原理流程图
- HillCrest Sensor HAL
1. 抽象定义 Google为Sensor提供了统一的HAL接口,不同的硬件厂商需要根据该接口来实现并完成具体的硬件抽象层,Android中Sensor的HAL接口定义在:hardware/libha ...
- UVA 10395 素数筛
Twin Primes Twin primes are pairs of primes of the form (p; p + 2). The term \twin prime" was c ...
- Ribbon实现客户端负载均衡
什么是Ribbon? 客户端负载均衡组件. 前期准备: 搭建一个Eureka集群和一个注册服务 https://www.cnblogs.com/noneplus/p/11374883.html 创建服 ...