Hive集成HBase;安装pig

Hive集成HBase

配置

将hive的lib/中的HBase.jar包用实际安装的Hbase的jar包替换掉

cd /opt/hive/lib/

ls hbase-0.94.2* 

rm -rf hbase-0.92*

cp /opt/hbase/hbase-0.94.2* 

将Hive的lib/中的zookeeper.jar包用HBase中lib/中的替换掉

步骤同上

 

在hive-site.xml中添加：

<property>

        <name>hive.aux.jars.path</name>

        <value>file:///opt/hive/lib/hive-hbase-handler-0.9.0.jar,file:///opt/hive/lib/hbase-0.94.2.jar,file:///opt/hive/lib/zookeeper-3.4.3.jar</value>

</property>

 

运行

cd /opt/hive/bin

./hive -hiveconf hbase.master=master:60000

 

 

流程如下：

先启动hbase，才能在hive里创建表；

在hive创建表后，在hbase你添加数据；

===========启动hbase，并在其中添加数据==============

[hadoop@localhost bin]$ cd /opt/hbase/bin 
[hadoop@localhost bin]$ ./start-hbase.sh

[hadoop@localhost bin]$ ./hbase shell

在hbase中添加数据

hbase(main):004:0> put 'htest','1','f:values','test'

hbase(main):005:0> scan 'htest'

 

===============启动hive，并创建表格===========

cd /opt/hive/bin

./hive -hiveconf hbase.master=master:60000

hive> create table htest(key int,value string) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ('hbase.columns.mapping'=':key,f:value') tblproperties('hbase.table.name'='htest');

hive> show tables; 

hive> select * from htest;

 

 

 

安装Pig

解压并安装

tar -zxvf pig-0.10.0.tar.gz /opt/

mv pig-0.10.0/ pig

chown -R hadoop:hadoop pig

 

配置

因为pig/conf里没有xxx-en.vsh文件，所以在pig/bin理修改pig

cd /opt/pig/bin

vi pig

添加下面内容：

export JAVA_HOME=/usr/program/jdk1.6.0_13/

export PIG_INSTALL=/opt/pig

export HADOOP_INSTALL=/home/hadoop/hadoop-env/hadoop-1.0.1/

export PATH=$PIG_INSTALL/bin:HADOOP_INSTALL/bin:$PATH

export PIG_CLASSPATH=$HADOOP_INSTALL/conf

 

执行

先启动hadoop，再启动hivve

cd /opt/hive/bin

./pig

 

 

=======上传数据到hadoop hdfs=========================

 

hadoop fs -copyFromLocal /opt/data/test.txt /opt/data/test.txt 将电脑数据上传到hadoop fs中

hadoop fs -ls /opt/data/test.txt 

hadoop fs -cat /opt/data/test.txt

 

=========pig中显示数据========================

grunt>A = LOAD '/opt/data/test.txt' USING PigStorage('#') AS (id,name);

grunt> B = FOREACH A GENERATE name;

grunt> STORE B INTO 'opt/data/dist.txt' USING PigStorage('\t'); 

grunt> dump A;

 

 

 

Pig Latin常用命令

LOAD....USING PigStorage('')......AS......;

FOREACH......GENERATE......;

FILTER......BY......;

DUMP;

STORE......INTO;

GROUP ......BY;

 

[hadoop@localhost bin]$ hadoop fs -ls /user/hive/warehouse/my 

查看hive中的数据仓库