Write Custom Java to Create LZO Files
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LZO
- Created by Lefty Leverenz, last modified on Sep 19, 2017
LZO Compression
General LZO Concepts
LZO is a lossless data compression library that favors speed over compression ratio. See http://www.oberhumer.com/opensource/lzo and http://www.lzop.org for general information about LZO and see Compressed Data Storage for information about compression in Hive.
Imagine a simple data file that has three columns
- id
- first name
- last name
Let's populate a data file containing 4 records:
19630001 john lennon
19630002 paul mccartney
19630003 george harrison
19630004 ringo starr
Let's call the data file /path/to/dir/names.txt
.
In order to make it into an LZO file, we can use the lzop utility and it will create a names.txt.lzo
file.
Now copy the file names.txt.lzo
to HDFS.
Prerequisites
Lzo/Lzop Installations
lzo
and lzop
need to be installed on every node in the Hadoop cluster. The details of these installations are beyond the scope of this document.
core-site.xml
Add the following to your core-site.xml
:
com.hadoop.compression.lzo.LzoCodec
com.hadoop.compression.lzo.LzopCodec
For example:
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,
com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>
com.hadoop.compression.lzo.LzoCodec</value>
</property>
Next we run the command to create an LZO index file:
hadoop jar /path/to/jar/hadoop-lzo-cdh4-0.4.15-gplextras.jar com.hadoop.compression.lzo.LzoIndexer /path/to/HDFS/dir/containing/lzo/files
This creates names.txt.lzo
on HDFS.
Table Definition
The following hive -e
command creates an LZO-compressed external table:
hive -e "CREATE EXTERNAL TABLE IF NOT EXISTS hive_table_name (column_1 datatype_1......column_N datatype_N)
PARTITIONED BY (partition_col_1 datatype_1 ....col_P datatype_P)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS INPUTFORMAT \"com.hadoop.mapred.DeprecatedLzoTextInputFormat\"
OUTPUTFORMAT \"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\";
Note: The double quotes have to be escaped so that the 'hive -e
' command works correctly.
See CREATE TABLE and Hive CLI for information about command syntax.
Hive Queries
Option 1: Directly Create LZO Files
- Directly create LZO files as the output of the Hive query.
- Use
lzop
command utility or your custom Java to generate.lzo.index
for the.lzo
files.
Hive Query Parameters
SET mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzoCodec
SET hive.exec.compress.output=true
SET mapreduce.output.fileoutputformat.compress=true
For example:
hive -e "SET mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzoCodec; SET hive.exec.compress.output=true;SET mapreduce.output.fileoutputformat.compress=true; <query-string>"
Note: If the data sets are large or number of output files are large , then this option does not work.
Option 2: Write Custom Java to Create LZO Files
- Create text files as the output of the Hive query.
- Write custom Java code to
- convert Hive query generated text files to
.lzo
files - generate
.lzo.index
files for the.lzo
files generated above
- convert Hive query generated text files to
Hive Query Parameters
Prefix the query string with these parameters:
SET hive.exec.compress.output=false
SET mapreduce.output.fileoutputformat.compress=false
For example:
hive -e "SET hive.exec.compress.output=false;SET mapreduce.output.fileoutputformat.compress=false;<query-string>"
Write Custom Java to Create LZO Files的更多相关文章
- How to create PDF files in a Python/Django application using ReportLab
https://assist-software.net/blog/how-create-pdf-files-python-django-application-using-reportlab CONT ...
- 转载Java NIO中的Files类的使用
Java NIO中的Files类(java.nio.file.Files)提供了多种操作文件系统中文件的方法. Files.exists() Files.exits()方法用来检查给定的Path在文件 ...
- Java使用JSP Tag Files & JSP EL Functions打造你自己的页面模板
1. 简单说明:在JSP 2.0后, 你不再需要大刀阔斧地定义一堆TagSupport或BodyTagSupport, 使用JSP Tag Files技术可以实现功能强大的页面模板技术. 在这里抛砖引 ...
- Java NIO.2 使用Files类遍历文件夹
在以前的Java版本中,如果要遍历某个文件夹下所有的子文件.子文件夹,需要我们自己写递归,很麻烦. 在Java7以后,我们可以NIO.2中的Files工具类来遍历某个文件夹(会自动递归). 大致用法: ...
- 使用Java内存映射(Memory-Mapped Files)处理大文件
>>NIO中的内存映射 (1)什么是内存映射文件内存映射文件,是由一个文件到一块内存的映射,可以理解为将一个文件映射到进程地址,然后可以通过操作内存来访问文件数据.说白了就是使用虚拟内存将 ...
- Using Custom Java code in ODI
在ODI中调用jar包java方法的过程如下: 1.编写Java代码如下 代码写hello world字符串到一个文件. package odi; import java.io.File; impor ...
- Create XML Files Out Of SQL Server With SSIS And FOR XML Syntax
So you want to spit out some XML from SQL Server into a file, how can you do that? There are a coupl ...
- [Tools] Batch Create Markdown Files from a Template with Node.js and Mustache
Creating Markdown files from a template is a straightforward process with Node.js and Mustache. You ...
- Cognos权限Custom Java Provider表结构实例
select * from org_user;USER_ID USER_CODE USER_NAME FULL_NAME EMAIL PWD2 889 zhangsan 张三 123@126.com ...
随机推荐
- 三读bootmem【转】
转自:http://blog.csdn.net/lights_joy/article/details/2704788 版权声明:本文为博主原创文章,未经博主允许不得转载. 目录(?)[-] 11 ...
- Android系统默认语言改为中文
第一种方法: 修改 build/tools/buildinfo.sh echo "ro.product.locale.language=zh"echo "ro.produ ...
- Android调起地图导航
想要使用导航功能可以使用各个地图的开放平台集成导航模块,如果不想集成也可以调起相关app导航 调起其他app首先得使用到该app包名,先贴出来 public final static Strin ...
- est6 -- Object.is()、Object.assign()、Object.defineProperty()、Symbol、Proxy
Object.is()用来比较两个值是否严格相等.它与严格比较运算符(===)的行为基本一致,不同之处只有两个:一是+0不等于-0,二是NaN等于自身. + === - //true NaN === ...
- linux sed 替换(整行替换,部分替换)、删除delete、新增add、选取
sed命令行格式为: sed [-nefri] ‘command’ 输入文本 常用选项: -n∶使用安静(silent)模式.在一般 sed 的用法中,所有来自 STDI ...
- POJ 3171 区间覆盖最小值&&线段树优化dp
Cleaning Shifts Time Limit: 1000MS Memory Limit: 65536K Total Submissions: 4715 Accepted: 1590 D ...
- Codeforces 620F Xors on Segments(暴力+DP)
题目链接 Xors on Segments 预处理出$x[i]$ $=$ $1$ $xor$ $2$ $xor$ $3$ $xor$ $……$ $xor$ $i$ 话说这题$O(n^{2})$居然能过 ...
- MySQL通用编程
第一阶段:基础入门 第一章:关系模型 第二章:基本查询 第三章:复杂查询 第四章:权限控制 第五章:查询优化 第二阶段:模型设计 第六章:设计选择 第七章:函数依赖 第八章:分解算法 第九章:设计过程 ...
- 如何通过ShareSDK的 Unity3D快速接入Android/iOS分享与授权
Unity3D是由Unity Technologies开发的一个让玩家轻松创建诸如三维视频游戏.建筑可视化.实时三维动画等类型互动内容的多平台的综合型游戏开发工具,是一个全面整合的专业游戏引擎:在游戏 ...
- Linux 在VMware中搭建CentOS6.5虚拟机
原文:http://www.cnblogs.com/PurpleDream/p/4263465.html Linux 在VMware中搭建CentOS6.5虚拟机 前言: 本文主要是我在大家 ...