2.11-2.12 HBase的数据迁移常见方式

一、importtsv

把hdfs中数据抽取到HBase表中；

1、准备数据

##student.tsv
[root@hadoop-senior datas]# cat student.tsv
10001    zhangsan    35    male    beijing    0109876543
10002    lisi    32    male    shanghia    0109876563
10003    zhaoliu    35    female    hangzhou    01098346543
10004    qianqi    35    male    shenzhen    01098732543

##
[root@hadoop-senior hadoop-2.5.0]# bin/hdfs dfs -mkdir -p /user/root/hbase/importtsv
[root@hadoop-senior hadoop-2.5.0]# bin/hdfs dfs -put /opt/datas/student.tsv /user/root/hbase/importtsv

##创建HBase表
hbase(main):005:0> create 'student', 'info'
0 row(s) in 0.1530 seconds

=> Hbase::Table - student

2、执行

##执行，下列命令全部执行
export HBASE_HOME=/opt/modules/hbase-0.98.6-hadoop2
export HADOOP_HOME=/opt/modules/hadoop-2.5.0
HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase mapredcp`:${HBASE_HOME}/conf \
        ${HADOOP_HOME}/bin/yarn jar \
${HBASE_HOME}/lib/hbase-server-0.98.6-hadoop2.jar importtsv \
-Dimporttsv.columns=HBASE_ROW_KEY,\
info:name,info:age,info:sex,info:address,info:phone \
student \
hdfs://hadoop-senior.ibeifeng.com:8020/user/root/hbase/importtsv

##查看结果
hbase(main):006:0> scan 'student'
ROW                                          COLUMN+CELL
 10001                                       column=info:address, timestamp=1558594471571, value=beijing
 10001                                       column=info:age, timestamp=1558594471571, value=35
 10001                                       column=info:name, timestamp=1558594471571, value=zhangsan
 10001                                       column=info:phone, timestamp=1558594471571, value=0109876543
 10001                                       column=info:sex, timestamp=1558594471571, value=male
 10002                                       column=info:address, timestamp=1558594471571, value=shanghia
 10002                                       column=info:age, timestamp=1558594471571, value=32
 10002                                       column=info:name, timestamp=1558594471571, value=lisi
 10002                                       column=info:phone, timestamp=1558594471571, value=0109876563
 10002                                       column=info:sex, timestamp=1558594471571, value=male
 10003                                       column=info:address, timestamp=1558594471571, value=hangzhou
 10003                                       column=info:age, timestamp=1558594471571, value=35
 10003                                       column=info:name, timestamp=1558594471571, value=zhaoliu
 10003                                       column=info:phone, timestamp=1558594471571, value=01098346543
 10003                                       column=info:sex, timestamp=1558594471571, value=female
 10004                                       column=info:address, timestamp=1558594471571, value=shenzhen
 10004                                       column=info:age, timestamp=1558594471571, value=35
 10004                                       column=info:name, timestamp=1558594471571, value=qianqi
 10004                                       column=info:phone, timestamp=1558594471571, value=01098732543
 10004                                       column=info:sex, timestamp=1558594471571, value=male

二、bulk load

1、bulk load

HBase支持bulk load的入库方式，它是利用hbase的数据信息按照特定格式存储在hdfs内这一原理，直接在HDFS中生成持久化的HFile数据格式文件，
然后上传至合适位置，即完成巨量数据快速入库的办法。配合mapreduce完成，高效便捷，而且不占用region资源，增添负载，在大数据量写入时能
极大的提高写入效率，并降低对HBase节点的写入压力。

通过使用先生成HFile，然后再BulkLoad到Hbase的方式来替代之前直接调用HTableOutputFormat的方法有如下的好处：
（1）消除了对HBase集群的插入压力
（2）提高了Job的运行速度，降低了Job的执行时间

2、生成HFile

##建表
hbase(main):007:0> create 'student2', 'info'
0 row(s) in 0.1320 seconds
=> Hbase::Table - student2

##生成Hfile
export HBASE_HOME=/opt/modules/hbase-0.98.6-hadoop2
export HADOOP_HOME=/opt/modules/hadoop-2.5.0
HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase mapredcp`:${HBASE_HOME}/conf \
        ${HADOOP_HOME}/bin/yarn jar \
${HBASE_HOME}/lib/hbase-server-0.98.6-hadoop2.jar importtsv \
-Dimporttsv.columns=HBASE_ROW_KEY,\
info:name,info:age,info:sex,info:address,info:phone \
-Dimporttsv.bulk.output=hdfs://hadoop-senior.ibeifeng.com:8020/user/root/hbase/hfileoutput \
student2 \
hdfs://hadoop-senior.ibeifeng.com:8020/user/root/hbase/importtsv

##查看
[root@hadoop-senior hadoop-2.5.0]# bin/hdfs dfs -ls /user/root/hbase/hfileoutput/info
Found 1 items
-rw-r--r--   1 root supergroup       1888 2019-05-24 13:31 /user/root/hbase/hfileoutput/info/8c28c6c654bc4fe2aa2c32ef54480771

2、将数据导入进表student2

##导数据
export HBASE_HOME=/opt/modules/hbase-0.98.6-hadoop2
export HADOOP_HOME=/opt/modules/hadoop-2.5.0
HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase mapredcp`:${HBASE_HOME}/conf \
        ${HADOOP_HOME}/bin/yarn jar \
${HBASE_HOME}/lib/hbase-server-0.98.6-hadoop2.jar \
completebulkload \
hdfs://hadoop-senior.ibeifeng.com:8020/user/root/hbase/hfileoutput \
student2

##scan student2
hbase(main):008:0> scan 'student2'
ROW                                          COLUMN+CELL
 10001                                       column=info:address, timestamp=1558675878109, value=beijing
 10001                                       column=info:age, timestamp=1558675878109, value=35
 10001                                       column=info:name, timestamp=1558675878109, value=zhangsan
 10001                                       column=info:phone, timestamp=1558675878109, value=0109876543
 10001                                       column=info:sex, timestamp=1558675878109, value=male
 10002                                       column=info:address, timestamp=1558675878109, value=shanghia
 10002                                       column=info:age, timestamp=1558675878109, value=32
 10002                                       column=info:name, timestamp=1558675878109, value=lisi
 10002                                       column=info:phone, timestamp=1558675878109, value=0109876563
 10002                                       column=info:sex, timestamp=1558675878109, value=male
 10003                                       column=info:address, timestamp=1558675878109, value=hangzhou
 10003                                       column=info:age, timestamp=1558675878109, value=35
 10003                                       column=info:name, timestamp=1558675878109, value=zhaoliu
 10003                                       column=info:phone, timestamp=1558675878109, value=01098346543
 10003                                       column=info:sex, timestamp=1558675878109, value=female
 10004                                       column=info:address, timestamp=1558675878109, value=shenzhen
 10004                                       column=info:age, timestamp=1558675878109, value=35
 10004                                       column=info:name, timestamp=1558675878109, value=qianqi
 10004                                       column=info:phone, timestamp=1558675878109, value=01098732543
 10004                                       column=info:sex, timestamp=1558675878109, value=male
4 row(s) in 0.0420 seconds

3、在MapReduce中生成HFile文件