使用sqoop将MySQL数据库中的数据导入Hbase
前提:安装好 sqoop、hbase。
下载jbdc驱动:mysql-connector-java-5.1.10.jar
将 mysql-connector-java-5.1.10.jar 拷贝到 /usr/lib/sqoop/lib/ 下
MySQL导入HBase命令:
sqoop import --connect jdbc:mysql://10.10.97.116:3306/rsearch --table researchers --hbase-table A --column-family person --hbase-row-key id --hbase-create-table --username
'root' -P
说明:
--connect jdbc:mysql://10.10.97.116:3306/rsearch 表示远程或者本地 Mysql 服务的URI,3306是Mysql默认监听port,rsearch是数据库,若是其它数据库,如Oracle,仅仅需改动URI就可以。
--table researchers 表示导出rsearch数据库的researchers表。
--hbase-table A 表示在HBase中建立表A。
--column-family person 表示在表A中建立列族person。
--hbase-row-key id 表示表A的row-key是researchers表的id字段。
--hbase-create-table 表示在HBase中建立表。
--username 'root' 表示使用用户root连接Mysql。
注意:
HBase的全部节点必须可以訪问MySQL数据库。不然会出现例如以下错误:
java.sql.SQLException: null, message from server: "Host '10.10.104.3' is not allowed to connect to this MySQL server"
- 在MySQL数据库server节点上运行下面命令同意远程机器使用对应用户訪问本地数据库server:
- [root@gc01vm6 htdocs] # /opt/lampp/bin/mysql
- mysql> use mysql;
- Database changed
- mysql> GRANT ALL PRIVILEGES ON rsearch.* TO 'root'@'10.10.104.3' IDENTIFIED BY '' WITH GRANT OPTION;
- mysql> GRANT ALL PRIVILEGES ON rsearch.* TO 'root'@'10.10.104.5' IDENTIFIED BY '' WITH GRANT OPTION;
- mysql> GRANT ALL PRIVILEGES ON rsearch.* TO 'root'@'10.10.104.2' IDENTIFIED BY '' WITH GRANT OPTION;
这里10.10.104.2,10.10.104.3,10.10.104.5 是HBase节点。
-------------------------------------------------------------------------------------------------
MySQL导入HBase的日志:
[root@gd02 hadoop]# sqoop import --connect
jdbc:mysql://10.10.97.116:3306/rsearch --table researchers --hbase-table A --column-family person --hbase-row-key id --hbase-create-table --username 'root' -P
Enter password:
11/06/29 19:08:00 INFO tool.CodeGenTool: Beginning code generation
11/06/29 19:08:00 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `researchers` AS t LIMIT 1
11/06/29 19:08:00 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `researchers` AS t LIMIT 1
11/06/29 19:08:00 INFO orm.CompilationManager: HADOOP_HOME is /usr/lib/hadoop
11/06/29 19:08:00 INFO orm.CompilationManager: Found hadoop core jar at: /usr/lib/hadoop/hadoop-core.jar
Note: /tmp/sqoop-root/compile/d4dd4cb4e1e325fce31ca72c00a5589c/researchers.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
11/06/29 19:08:02 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/d4dd4cb4e1e325fce31ca72c00a5589c/researchers.jar
11/06/29 19:08:02 WARN manager.MySQLManager: It looks like you are importing from mysql.
11/06/29 19:08:02 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
11/06/29 19:08:02 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
11/06/29 19:08:02 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
11/06/29 19:08:02 INFO mapreduce.ImportJobBase: Beginning import of researchers
11/06/29 19:08:02 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `researchers` AS t LIMIT 1
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-cdh3u0--1, built on 03/26/2011 00:21 GMT
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:host.name=gd02
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_13
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.6.0_13/jre
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/usr/lib/hadoop/conf:/usr/java/jdk1.6.0_13/lib/tools.jar:/usr/lib/hadoop:/usr/lib/hadoop/hadoop-core-0.20.2-cdh3u0.jar:/usr/lib/hadoop/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop/lib/commons-net-1.4.1.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/hadoop-fairscheduler-0.20.2-cdh3u0.jar:/usr/lib/hadoop/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.jar:/usr/lib/hadoop/lib/jetty-servlet-tester-6.1.26.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/junit-4.5.jar:/usr/lib/hadoop/lib/kfs-0.2.2.jar:/usr/lib/hadoop/lib/log4j-1.2.15.jar:/usr/lib/hadoop/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop/lib/oro-2.0.8.jar:/usr/lib/hadoop/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/sqoop/conf:/usr/lib/hbase/conf::/usr/lib/sqoop/lib/ant-contrib-1.0b3.jar:/usr/lib/sqoop/lib/ant-eclipse-1.0-jvm1.2.jar:/usr/lib/sqoop/lib/commons-io-1.4.jar:/usr/lib/sqoop/lib/hadoop-mrunit-0.20.2-CDH3b2-SNAPSHOT.jar:/usr/lib/sqoop/lib/ivy-2.0.0-rc2.jar:/usr/lib/sqoop/lib/mysql-connector-java-5.1.10.jar:/usr/lib/hbase/hbase-0.90.1-cdh3u0.jar:/usr/lib/hbase/hbase-0.90.1-cdh3u0-tests.jar:/usr/lib/hbase/lib/activation-1.1.jar:/usr/lib/hbase/lib/asm-3.1.jar:/usr/lib/hbase/lib/avro-1.3.3.jar:/usr/lib/hbase/lib/commons-cli-1.2.jar:/usr/lib/hbase/lib/commons-codec-1.4.jar:/usr/lib/hbase/lib/commons-el-1.0.jar:/usr/lib/hbase/lib/commons-httpclient-3.1.jar:/usr/lib/hbase/lib/commons-lang-2.5.jar:/usr/lib/hbase/lib/commons-logging-1.1.1.jar:/usr/lib/hbase/lib/commons-net-1.4.1.jar:/usr/lib/hbase/lib/core-3.1.1.jar:/usr/lib/hbase/lib/guava-r06.jar:/usr/lib/hbase/lib/hadoop-core.jar:/usr/lib/hbase/lib/hbase-0.90.1-cdh3u0.jar:/usr/lib/hbase/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hbase/lib/jackson-jaxrs-1.5.5.jar:/usr/lib/hbase/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hbase/lib/jackson-xc-1.5.5.jar:/usr/lib/hbase/lib/jasper-compiler-5.5.23.jar:/usr/lib/hbase/lib/jasper-runtime-5.5.23.jar:/usr/lib/hbase/lib/jaxb-api-2.1.jar:/usr/lib/hbase/lib/jaxb-impl-2.1.12.jar:/usr/lib/hbase/lib/jersey-core-1.4.jar:/usr/lib/hbase/lib/jersey-json-1.4.jar:/usr/lib/hbase/lib/jersey-server-1.4.jar:/usr/lib/hbase/lib/jettison-1.1.jar:/usr/lib/hbase/lib/jetty-6.1.26.jar:/usr/lib/hbase/lib/jetty-util-6.1.26.jar:/usr/lib/hbase/lib/jruby-complete-1.0.3.jar:/usr/lib/hbase/lib/jsp-2.1-6.1.14.jar:/usr/lib/hbase/lib/jsp-api-2.1-6.1.14.jar:/usr/lib/hbase/lib/jsp-api-2.1.jar:/usr/lib/hbase/lib/jsr311-api-1.1.1.jar:/usr/lib/hbase/lib/log4j-1.2.16.jar:/usr/lib/hbase/lib/protobuf-java-2.3.0.jar:/usr/lib/hbase/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hbase/lib/servlet-api-2.5.jar:/usr/lib/hbase/lib/slf4j-api-1.5.8.jar:/usr/lib/hbase/lib/slf4j-log4j12-1.5.8.jar:/usr/lib/hbase/lib/stax-api-1.0.1.jar:/usr/lib/hbase/lib/thrift-0.2.0.jar:/usr/lib/hbase/lib/xmlenc-0.52.jar:/usr/lib/hbase/lib/zookeeper.jar:/usr/lib/zookeeper/zookeeper-3.3.3-cdh3u0.jar:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/sqoop/sqoop-1.2.0-cdh3u0.jar:/usr/lib/sqoop/sqoop-test-1.2.0-cdh3u0.jar:
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/jdk1.6.0_13/jre/lib/amd64/server:/usr/java/jdk1.6.0_13/jre/lib/amd64:/usr/java/jdk1.6.0_13/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.18-164.el5
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:user.name=root
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=gd05:2181,gd03:2181,gd02:2181 sessionTimeout=180000 watcher=hconnection
11/06/29 19:08:02 INFO zookeeper.ClientCnxn: Opening socket connection to server gd03/10.10.104.3:2181
11/06/29 19:08:02 INFO zookeeper.ClientCnxn: Socket connection established to gd03/10.10.104.3:2181, initiating session
11/06/29 19:08:02 INFO zookeeper.ClientCnxn: Session establishment complete on server gd03/10.10.104.3:2181, sessionid = 0x130b2e901cd0012, negotiated timeout = 180000
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=gd05:2181,gd03:2181,gd02:2181 sessionTimeout=180000 watcher=hconnection
11/06/29 19:08:02 INFO zookeeper.ClientCnxn: Opening socket connection to server gd03/10.10.104.3:2181
11/06/29 19:08:02 INFO zookeeper.ClientCnxn: Socket connection established to gd03/10.10.104.3:2181, initiating session
11/06/29 19:08:02 INFO zookeeper.ClientCnxn: Session establishment complete on server gd03/10.10.104.3:2181, sessionid = 0x130b2e901cd0013, negotiated timeout = 180000
11/06/29 19:08:02 INFO client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x130b2e901cd0013
11/06/29 19:08:02 INFO zookeeper.ZooKeeper: Session: 0x130b2e901cd0013 closed
11/06/29 19:08:02 INFO zookeeper.ClientCnxn: EventThread shut down
11/06/29 19:08:02 INFO mapreduce.HBaseImportJob: Creating missing column family person
11/06/29 19:08:02 INFO client.HBaseAdmin: Started disable of A
11/06/29 19:08:03 INFO client.HBaseAdmin: Disabled A
11/06/29 19:08:03 INFO client.HBaseAdmin: Started enable of A
11/06/29 19:08:06 INFO client.HBaseAdmin: Enabled table A
11/06/29 19:08:07 INFO mapred.JobClient: Running job: job_201106212352_0010
11/06/29 19:08:08 INFO mapred.JobClient: map 0% reduce 0%
11/06/29 19:08:19 INFO mapred.JobClient: map 40% reduce 0%
11/06/29 19:08:20 INFO mapred.JobClient: map 80% reduce 0%
11/06/29 19:08:34 INFO mapred.JobClient: map 100% reduce 0%
11/06/29 19:08:34 INFO mapred.JobClient: Job complete: job_201106212352_0010
11/06/29 19:08:34 INFO mapred.JobClient: Counters: 11
11/06/29 19:08:34 INFO mapred.JobClient: Job Counters
11/06/29 19:08:34 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=82848
11/06/29 19:08:34 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
11/06/29 19:08:34 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
11/06/29 19:08:34 INFO mapred.JobClient: Launched map tasks=5
11/06/29 19:08:34 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
11/06/29 19:08:34 INFO mapred.JobClient: FileSystemCounters
11/06/29 19:08:34 INFO mapred.JobClient: HDFS_BYTES_READ=527
11/06/29 19:08:34 INFO mapred.JobClient: FILE_BYTES_WRITTEN=310685
11/06/29 19:08:34 INFO mapred.JobClient: Map-Reduce Framework
11/06/29 19:08:34 INFO mapred.JobClient: Map input records=81868
11/06/29 19:08:34 INFO mapred.JobClient: Spilled Records=0
11/06/29 19:08:34 INFO mapred.JobClient: Map output records=81868
11/06/29 19:08:34 INFO mapred.JobClient: SPLIT_RAW_BYTES=527
11/06/29 19:08:34 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 28.108 seconds (0 bytes/sec)
11/06/29 19:08:34 INFO mapreduce.ImportJobBase: Retrieved 81868 records.
參考资料:
利用sqoop将mysql数据同步到hive手记
http://www.54chen.com/java-ee/sqoop-mysql-to-hive.html
利用Sqoop将数据从数据库导入到HDFS
http://www.cnblogs.com/gpcuster/archive/2011/03/01/1968027.html
Sqoop
http://www.duyifan.com/
MySQL向Hive/HBase的迁移工具
http://www.javabloger.com/article/hadoop-hive-mysql-sqoop.html
官方手冊
http://archive.cloudera.com/cdh/3/sqoop/SqoopUserGuide.html
使用sqoop将MySQL数据库中的数据导入Hbase的更多相关文章
- python制作简单excel统计报表3之将mysql数据库中的数据导入excel模板并生成统计图
python制作简单excel统计报表3之将mysql数据库中的数据导入excel模板并生成统计图 # coding=utf-8 from openpyxl import load_workbook ...
- mysql数据库中的数据导入与导出
需求: 在本地电脑上的mysql中创建了一个数据库,并且在该数据库中创建了很多表,表中数据比较多: 现在想换一台电脑,但是又不想重新建数据库.建表.造数据. 解决方案: 利用mysql提供的命令,在本 ...
- Sqoop(三)将关系型数据库中的数据导入到HDFS(包括hive,hbase中)
一.说明: 将关系型数据库中的数据导入到 HDFS(包括 Hive, HBase) 中,如果导入的是 Hive,那么当 Hive 中没有对应表时,则自动创建. 二.操作 1.创建一张跟mysql中的i ...
- mysql数据库中插入数据INSERT INTO SET的优势
往mysql数据库中插入数据.以前常用 INSERT INTO 表名 (列名1,列名2…) VALUES(列值1,列值2); 如果在PHP程序中,就会写成如下示例(往商品库里增加商品) $sql = ...
- JDBC Java 程序从 MySQL 数据库中读取数据,并备份到 xml 文档中
MySQL 版本:Server version: 5.7.17-log MySQL Community Server (GPL) 相关内容:JDBC Java 程序从 MySQL 数据库中读取数据,并 ...
- 使用grabit分析mysql数据库中的数据血缘关系
使用grabit分析mysql数据库中的数据血缘关系 Grabit 是一个辅助工具,用于从数据库.GitHub 等修订系统.bitbucket 和文件系统等各种来源收集 SQL 脚本和存储过程,然后将 ...
- 如何将数据库中的数据导入到Solr中
要使用solr实现网站中商品搜索,需要将mysql数据库中数据在solr中创建索引. 1.需要在solr的schema.xml文件定义要存储的商品Field. 商品表中的字段为: 配置内容是: < ...
- phpexcel的写操作将数据库中的数据导入到excel中
这个版本据说是可以支持excel2007,但是我使用2007编辑的xlsx是无法获得该库的支持.于是乎我就将它转化为2003.感觉支持地很好. 下面介绍一下具体的使用: require_once('. ...
- 详解大数据采集引擎之Sqoop&采集oracle数据库中的数据
一.Sqoop的简介: Sqoop是一个数据采集引擎/数据交换引擎,采集关系型数据库(RDBMS)中的数据,主要用于在RDBMS与HDFS/Hive/HBase之间进行数据传递,可以通过sqoop i ...
随机推荐
- python 什么叫迭代
# -*- coding: cp936 -*- #xiaodeng #python 27 #什么叫迭代 #如果给定一个list/tuple,可以通过for循环遍历出来,这种遍历我们称之为迭代 #案例3 ...
- jenkins关闭和重启
我们用jar -jar jenkins.war来启动jenkins服务器,那么我们如何关闭或者重启jenkins服务器呢?经过搜索找到了相应的方法. 关闭jenkins服务 只需要在访问jenkins ...
- 【laravel5.* + 钉钉实现WEB第三方登录】 使用redis 作为持久化存储
1.去钉钉开发者平台>自助者工具,创建扫码登录授权应用,填写名称.描述.授权页面logo地址(这个图片最后会出现在用户扫码设备中,建议使用压缩图片减少用户加载时间).回调域名(一般都是写一个子域 ...
- 【laravel5.*】添加ide_helper.php 助手
1.参照文档:https://github.com/barryvdh/laravel-ide-helper#automatic-phpdoc-generation-for-laravel-facade ...
- Mybatis 如何自动生成bean dao xml 配置文件 generatorconfig.xml (mysql)
1/自动生成的jar包:mybatis-generator-core-1.3.2.jar 2/generatorconfig.xml文件如: <?xml version="1.0& ...
- HDUOJ----Coin Change
Coin Change Time Limit: 1000/1000 MS (Java/Others) Memory Limit: 32768/32768 K (Java/Others)Total ...
- libgdx游戏引擎教程
第一讲:libgdx游戏引擎教程(一)性能优良的游戏引擎—libgdx http://www.apkbus.com/android-57355-1-1.html 第二讲: libgdx游戏引擎教程(二 ...
- 【LeetCode】234. Palindrome Linked List (2 solutions)
Palindrome Linked List Given a singly linked list, determine if it is a palindrome. Follow up:Could ...
- Linux命令之lsb_release - 查看当前系统的发行版信息
用途说明 lsb_release命令用来查看当前系统的发行版信息(prints certain LSB (Linux Standard Base) and Distribution informati ...
- mysqlsla快速入门
小强软件测试,因为不是天生丽质,所以必须天生励志. 性能.python自动化班长期招生,咨询QQ:2083503238 官网:http://xqtesting.sxl.cn QQ群:229390571 ...