sqoop-1.4.6安装与使用
一、安装
export HADOOP_COMMON_HOME=/usr/local/hadoop-2.6.3
export HADOOP_MAPRED_HOME=/usr/local/hadoop-2.6.3
export HBASE_HOME=/usr/local/hbase-1.1.3
export HIVE_HOME=/usr/local/hive-2.0.0
#export ZOOCFGDIR=
Argument Description
--connect <jdbc-uri> Specify JDBC connect string
--connection-manager <class-name> Specify connection manager class to use
--driver <class-name> Manually specify JDBC driver class to use
--hadoop-mapred-home <dir> Override $HADOOP_MAPRED_HOME
--help Print usage instructions
--password-file Set path for a file containing the authentication password
-P Read password from console
--password <password> Set authentication password
--username <username> Set authentication username
--verbose Print more information while working
--connection-param-file <filename> Optional properties file that provides connection parameters
--relaxed-isolation Set connection transaction isolation to read uncommitted for the mappers.
$ sqoop import --connect jdbc:mysql://database.example.com/employees
$ sqoop import --connect jdbc:mysql://database.example.com/employees \
--username venkatesh --password-file ${user.home}/.password
--append |
Append data to an existing dataset in HDFS |
--as-avrodatafile |
Imports data to Avro Data Files |
--as-sequencefile |
Imports data to SequenceFiles |
--as-textfile |
Imports data as plain text (default) |
--as-parquetfile |
Imports data to Parquet Files |
--boundary-query <statement> |
Boundary query to use for creating splits |
--columns <col,col,col…> |
Columns to import from table --columns "name,employee_id,jobtitle" |
--delete-target-dir |
Delete the import target directory if it exists |
--direct |
Use direct connector if exists for the database |
--fetch-size <n> |
Number of entries to read from database at once. |
--inline-lob-limit <n> |
Set the maximum size for an inline LOB |
-m,--num-mappers <n> |
Use n map tasks to import in parallel |
-e,--query <statement> |
Import the results of statement . select min(<split-by>), max(<split-by>) from <table name> |
--split-by <column-name> |
Column of the table used to split work units. Cannot be used with --autoreset-to-one-mapper option.以某个字段平衡负载 |
--autoreset-to-one-mapper |
Import should use one mapper if a table has no primary key and no split-by column is provided. Cannot be used with--split-by <col> option. |
--table <table-name> |
Table to read |
--target-dir <dir> |
HDFS destination dir |
--warehouse-dir <dir> |
HDFS parent for table destination |
--where <where clause> |
WHERE clause to use during import |
-z,--compress |
Enable compression |
--compression-codec <c> |
Use Hadoop codec (default gzip) |
--null-string <null-string> |
The string to be written for a null value for string columns |
--null-non-string <null-string> |
The string to be written for a null value for non-string columns |
bin/sqoop list-databases --connect jdbc:mysql://yangxw:3306/mysql --username root --password root
bin/sqoop import --connect jdbc:mysql://yangxw:3306/classicmodels --username root --password root --table customers --target-dir /mysql_hadoop
$ sqoop import \
--query 'SELECT a.*, b.* FROM a JOIN b on (a.id == b.id) WHERE $CONDITIONS' \
--split-by a.id --target-dir /user/foo/joinresults
$ sqoop import ... --map-column-java id=String,value=Integer
--hive-home <dir> Override $HIVE_HOME
--hive-import Import tables into Hive (Uses Hive’s default delimiters if none are set.)
--hive-overwrite Overwrite existing data in the Hive table.
--create-hive-table If set, then the job will fail if the target hive
table exits. By default this property is false.--hive-table <table-name> Sets the table name to use when importing to Hive.
--hive-drop-import-delims Drops \n, \r, and \01 from string fields when importing to Hive.
--hive-delims-replacement Replace \n, \r, and \01 from string fields with user defined string when importing to Hive.
--hive-partition-key Name of a hive field to partition are sharded on
--hive-partition-value <v> String-value that serves as partition key for this imported into hive in this job.
--map-column-hive <map> Override default mapping from SQL type to Hive type for configured columns.
--column-family <family> |
Sets the target column family for the import |
--hbase-create-table |
If specified, create missing HBase tables |
--hbase-row-key <col> |
Specifies which input column to use as the row key |
In case, if input table contains composite(复合) | |
key, then <col> must be in the form of a | |
comma-separated(逗号分隔) list of composite key | |
attributes | |
--hbase-table <table-name> |
Specifies an HBase table to use as the target instead of HDFS |
--hbase-bulkload |
Enables bulk loading 指导入 |
- bin/sqoop import --connect jdbc:mysql://yangxw:3306/classicmodels --username root --password root --table orders --target-dir /mysql_hadoop/orders4 --hbase-table orders --column-family orders --hbase-create-table
- 报以下错误:无法创建hbase的表:
16/03/24 18:30:23 INFO mapreduce.HBaseImportJob: Creating missing HBase table orders
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.HTableDescriptor.addFamily(Lorg/apache/hadoop/hbase/HColumnDescriptor;)V
at org.apache.sqoop.mapreduce.HBaseImportJob.jobSetup(HBaseImportJob.java:222)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:264)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673)
at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
hbase(main):002:0> create 'orders','CF1'
0 row(s) in 1.6730 seconds
=> Hbase::Table - orders
bin/sqoop import --connect jdbc:mysql://yangxw:3306/classicmodels --username root --password root --table orders --target-dir /mysql_hadoop/orders5 --hbase-table orders --column-family CF1
sqoop-1.4.6安装与使用的更多相关文章
- Sqoop 1.99.4 安装
1.安装准备工作:已经装好的 hadoop 环境是 hadoop-2.5.1 64位下载的sqoop安装包(注意是hadoop200)http://www.us.apache.org/dist/sqo ...
- [sqoop1.99.7] sqoop入门-下载、安装、运行和常用命令
一.简介 Apache Sqoop is a tool designed for efficiently transferring data betweeen structured, semi-str ...
- HIVE之 Sqoop 1.4.6 安装、hive与oracle表互导
1. sqoop数据迁移 1.1 概述 sqoop是apache旗下一款“Hadoop和关系数据库服务器之间传送数据”的工具. 导入数据:MySQL,Oracle导入数据到Hadoop的HDFS.HI ...
- Sqoop之 Sqoop 1.4.6 安装
1. sqoop数据迁移 1.1 概述 sqoop是apache旗下一款“Hadoop和关系数据库服务器之间传送数据”的工具. 导入数据:MySQL,Oracle导入数据到Hadoop的HDFS.HI ...
- Sqoop 1.99.6 安装和使用
安装 1.安装准备工作: 下载的sqoop安装包 http://mirrors.hust.edu.cn/apache/sqoop/1.99.6/sqoop-1.99.6.tar.gz ...
- sqoop 1.99.7 安装及配置
一 下载sqoop 1.99.7 http://mirror.bit.edu.cn/apache/sqoop/1.99.7/ 二 解压安装文件 三 配置Sqoop 环境变量 最后把mysql的驱动j ...
- Sqoop 1.4.6 安装配置
配置环境变量 # SQOOP SQOOP_HOME=/home/hadoop/development/src/sqoop-1.4.6-cdh5.6.0 PATH=$PATH:$SQOOP_HOME/b ...
- Sqoop(一)安装及基本使用
Sqoop: 1.sqoop从数据库中导入数据到HDFS 2.SQOOP从数据库导入数据到hive 3.sqoop从hive中将数据导出到数据库 sqoop底层还是执行的m ...
- sqoop 1.4.7 安装配置/连接测试
环境: hadoop2.7.7 mysql 8 zk 3.4.10 hive 3 1.上传并解压tar包后进入conf目录 拷贝sqoop-env-template.sh并重命名为sqoop-env. ...
- 安装sqoop
安装sqoop 1.默认已经安装好java+hadoop 2.下载对应hadoop版本的sqoop版本 3.解压安装包 tar zxvf sqoop-1.4.6.bin__hadoop-2.0.4-a ...
随机推荐
- Virtualization state: Optimized (version 7.4 installed)
[Virtualization state: Optimized (version 7.4 installed)] [root@localhost ~]# cd /mnt/ [root@localho ...
- Python支付接口汇总大全(包含微信、支付宝等)
微信接口 wzhifuSDK- 由微信支付SDK 官方PHP Demo移植而来,v3.37下载地址 weixin_pay- 是一个简单的微信支付的接口 weixin_pay- 微信支付接口(V3.3. ...
- Flask之蓝图的使用
蓝图,听起来就是一个很宏伟的东西 在Flask中的蓝图 blueprint 也是非常宏伟的 它的作用就是将 功能 与 主服务 分开怎么理解呢? 比如说,你有一个客户管理系统,最开始的时候,只有一个查看 ...
- 壹度DIY_微信小程序组件_小程序插件开发
开源免费插件,diy特有的页面机制,搭配30+自定义组件,让你的站点每一个页面都可以完全自定义,可无缝对接任意小程序,如有疑问加入qq壹度小程序交流群:302866773:或wx:liu2417301 ...
- hive优化-数据倾斜优化
数据倾斜解决方法,通常从以下几个方面进行考量: 业务上丢弃 • 不参与关联:在on条件上直接过滤 • 随机数打散:比如 null.空格.0等“Other”性质的特殊值 倾斜键记录单独处理 • ...
- requests+mongodb爬取今日头条,多进程
import json import os from urllib.parse import urlencode import pymongo import requests from bs4 imp ...
- python 爬虫(爬取网页的img并下载)
from urllib.request import urlopen # 引用第三方库 import requests #引用requests/用于访问网站(没安装需要安装) from pyquery ...
- mysql库地址
https://dev.mysql.com/downloads/connector/
- vue.js使用axios
使用axios的两种调用方式 1.安装axios $ cnpm install axios 2.在vue入口文件main.js中引入(推荐全局引入),或是在当前页面中引入(局部) import axi ...
- MapWinGIS介绍
MapWinGIS是一个不错的开源组件GIS项目,基于微软的COM思想编写,团队的开发人员主要来自爱荷华大学,而且开发团队中还有一位中国人.它的功能类似于AO(当然没有AO那么强大),其核心是一个名字 ...