how to use Sqoop to import/ export data
Sqoop is a tool designed for efficiently transferring data between RDBMS and HDFS, we can import data from mysql, oracle, and other data bases into HDFS very easily; meanwhile we can dump data into data base from HDFS. For detailed documentation, please refer to sqoop documentation.
Before using Sqoop, please follow steps to setup it correctly.
Sqoop - Import
the following command is used for import
sqoop import (generic-args) (import-args)
given a table named stock_info, and the schema is:
Case 1: we can use below command to import stock_info data to hadoop hdfs file system:
sqoop import --connect jdbc:mysql://host:port/dbname --username loginuser --password loginuser --table stock_info --m 1
and the result looks like:
we can verify result in hdfs by running command
hadoop fs -cat /emp/part-m-*
Case 2: sepcify the target directory in hdfs by running the following import command
sqoop import --connect jdbc:mysql://host:port/dbname --username loginuser --password loginuser --table stock_info --m 1 --target-dir /temp
then we can verify result by executing the same command as above
Case 3: imcremental import by specifying --incremental, --check-column and --append arguments. Note we should change 'last_chg_date' when applying other tables.
sqoop import --connect jdbc:mysql://host:port/dbname --username loginuser --password loginuser --table stock_info --m 1 --target-dir /temp --incremental lastmodified --check-column last_chg_date --append
Case 4: specify target file format as parquet format by adding argument '--as-parquetfile'
sqoop import --connect jdbc:mysql://host:port/dbname --username loginuser --password loginuser --table stock_info --m 1 --target-dir /temp --incremental lastmodified --check-column last_chg_date --append --as-parquetfile
Case 5: import all tables
sqoop import-all-tables --connect jdbc:mysql://host:port/dbname --username loginuser --password loginuser
Sqoop - Export
export means to dump data from hdfs to mysql, oracle or other data bases, command syntax is like
sqoop export (generic-args) (export-args)
given there are many parquet files under stock_info folder which is imported by sqoop import command incrementally
then we want to dump data back into mysql data base, using the following command
sqoop export --connent jdbc:mysql://host:port/dbname --username loginuser --password loginuser --table stock_info --export-dir /user/hlli/stock_info
finally verify data in mysql command line
select * from stock_info;
Incremental importing data
by using linux timer 'crontab' to schedule a job to execute importing periodically.
cd /var/spool/cron
touch hlli (please change hlli to your user name here)
vi hlli
*/ * * * * /usr/lib/sqoop/bin/sqoop import --connect jdbc:mysql://host:port/dbname --username loginuser --password loginuser --table stock_info --m 1 --target-dir /temp --incremental lastmodified --check-column last_chg_date --append --as-parquetfile
if it works, you will receive email in '/var/spool/mail/hlli'; meanwhile we can verify data by running command
hadoop fs -ls /
Commonly used Sqoop commands
sqoop help import
sqoop help export
sqoop help job
sqoop help codegen
sqoop help eval
sqoop help list-tables
sqoop help list-databases
sqoop help import-all-tables
References:
- http://sqoop.apache.org/
- http://man.linuxde.net/crontab
how to use Sqoop to import/ export data的更多相关文章
- 1.3 Quick Start中 Step 7: Use Kafka Connect to import/export data官网剖析(博主推荐)
不多说,直接上干货! 一切来源于官网 http://kafka.apache.org/documentation/ Step 7: Use Kafka Connect to import/export ...
- hadoop kafka import/export data (8)
reference: http://kafka.apache.org/quickstart need to solve issue ISSUE 1: [2019-01-29 15:59:39,272] ...
- Data import/export of Netezza using external table
Introduction External table is a special table in Netezza system, which could be used to import/exp ...
- sqoop import/export使用经验
一.先创建一个小表(test_01)进行测试(主节点IP:169.254.109.130/oracle服务器IP:169.254.109.100) 1.测试连接oracle; sqoop list-t ...
- Sqoop import export参数
通用参数 import export 通用通用参数选项 含义说明–connect 指定JDBC连接字符串–connection-manager 指定要使用的连接管理器类–dri ...
- [Hive - LanguageManual] Import/Export
LanguageManual ImportExport Skip to end of metadata Added by Carl Steinbach, last edited by Le ...
- 前端 高级 (二十五)vue2.0项目实战一 配置简要说明、代码简要说明、Import/Export、轮播和列表例子
一.启动服务自动打开浏览器运行 二.配置简要说明 1.node_modules 安装好的依赖文件,中间件等,所在位置 2.package.jason 配置当前项目要安装的中间件和依赖文件 { &quo ...
- 探讨ES6的import export default 和CommonJS的require module.exports
今天来扒一扒在node和ES6中的module,主要是为了区分node和ES6中的不同意义,避免概念上的混淆,同时也分享一下,自己在这个坑里获得的心得. 在ES6之前 模块的概念是在ES6发布之前就出 ...
- ES6中的export,import ,export default
ES6模块主要有两个功能:export和importexport用于对外输出本模块(一个文件可以理解为一个模块)变量的接口import用于在一个模块中加载另一个含有export接口的模块.也就是说使用 ...
随机推荐
- 最近做手机端,GPS,微信QQ分享总结的问题
Android端 百度地图: 1.libs包中armeabi下liblocSDK4d.so文件丢失,导致百度定位失效. 微信分享: 1.分享App,app的内容(图片加描述)不能超过32kb ,不然无 ...
- duilib入门简明教程 -- VS环境配置(2)
会SVN和配置VS环境的请跳过此篇~ 既然是入门教程,那当然得基础点,因为搜索duilib相关资料时,发现有些小伙伴到处都是编译错误,以及路径配置错误等等,还有人不知道SVN ...
- windowsPhone一些不常见控件
1.InkPresenter:可以产生手写效果的控件. http://www.cnblogs.com/randylee/archive/2010/08/10/1791222.html 2.Thumb: ...
- 平台播放声音(ext.js)
首先把需要的两个js文件放在public/core路径下 (文件已经上传到博客了) 音频文件放在文件一级目录 代码:JxCustom.loadAudio("wav/NG.wav") ...
- 如何较为方便的在GMap.Net中实现车辆运行轨迹
一.简单的思路 要实现车辆运行轨迹,我们可能需要一个定时触发的机制用来更新Marker的位置,除了位置移动,我们可能还需要动态改变车辆的方向,如下图: 首先,位置移动是最简单的,关键是方向的动态改变如 ...
- mvc - view传值到js
http://www.cnblogs.com/akwwl/p/5238975.html
- angular HTML属性绑定
- 学习使用MS SQL Server游标(CURSOR)
说实的,使用MS SQL Server这样久,游标一直没有使用过.以前实现相似的功能,都是使用WHILE循环加临时表来实现.刚才有参考网上示例练习写了一下.了解到游标概念与语法. 下面代码示例中,先是 ...
- 第五篇 Python内置函数
内置函数 abs() delattr() hash() memoryview() set() all() dict() help() min() setattr() any() dir() ...
- loj #2051. 「HNOI2016」序列
#2051. 「HNOI2016」序列 题目描述 给定长度为 n nn 的序列:a1,a2,⋯,an a_1, a_2, \cdots , a_na1,a2,⋯,an,记为 a[1: ...