使用 sqoop 将mysql数据导入到hdfs(import)
Sqoop 将mysql 数据导入到hdfs(import)
1.创建mysql表
CREATE TABLE `sqoop_test` (
`id` int() DEFAULT NULL,
`name` varchar() DEFAULT NULL,
`age` int() DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1
插入数据
2.hive 建表
hive> create external table sqoop_test(id int,name string,age int)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS TEXTFILE
> location '/user/hive/external/sqoop_test';
OK
Time taken: 0.145 seconds
3.使用sqoop将mysql数据导入到hdfs
sqoop import --connect jdbc:mysql://localhost:3306/sqooptest --username root --password 123qwe --table sqoop_test --columns id,name,age --fields-terminated-by , --delete-target-dir --target-dir /user/hive/external/sqoop_test/ -m 1
--delete-target-dir:如果目标目录存在则删除。
EFdeMacBook-Pro:bin FengZhen$ sqoop import --connect jdbc:mysql://localhost:3306/sqooptest --username root --password 123qwe --table sqoop_test --columns id,name,age --fields-terminated-by , --delete-target-dir --target-dir /user/hive/external/sqoop_test/ -m 1
Warning: /Users/FengZhen/Desktop/Hadoop/sqoop-1.4..bin__hadoop-2.0.-alpha/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /Users/FengZhen/Desktop/Hadoop/sqoop-1.4..bin__hadoop-2.0.-alpha/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hadoop-2.8./share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hbase-1.3./lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.
// :: WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
// :: INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
// :: INFO tool.CodeGenTool: Beginning code generation
// :: INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sqoop_test` AS t LIMIT
// :: INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sqoop_test` AS t LIMIT
// :: INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /Users/FengZhen/Desktop/Hadoop/hadoop-2.8.
// :: INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-FengZhen/compile/1a0c4154ffefb21d4af720813dd0b3fc/sqoop_test.jar
// :: WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
// :: INFO tool.ImportTool: Destination directory /user/hive/external/sqoop_test deleted.
// :: WARN manager.MySQLManager: It looks like you are importing from mysql.
// :: WARN manager.MySQLManager: This transfer can be faster! Use the --direct
// :: WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
// :: INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
// :: INFO mapreduce.ImportJobBase: Beginning import of sqoop_test
// :: INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
// :: INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
// :: INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
// :: INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:
// :: INFO db.DBInputFormat: Using read commited transaction isolation
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1505268150495_0008
// :: INFO impl.YarnClientImpl: Submitted application application_1505268150495_0008
// :: INFO mapreduce.Job: The url to track the job: http://192.168.1.64:8088/proxy/application_1505268150495_0008/
// :: INFO mapreduce.Job: Running job: job_1505268150495_0008
// :: INFO mapreduce.Job: Job job_1505268150495_0008 running in uber mode : false
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: Job job_1505268150495_0008 completed successfully
// :: INFO mapreduce.Job: Counters:
File System Counters
FILE: Number of bytes read=
FILE: Number of bytes written=
FILE: Number of read operations=
FILE: Number of large read operations=
FILE: Number of write operations=
HDFS: Number of bytes read=
HDFS: Number of bytes written=
HDFS: Number of read operations=
HDFS: Number of large read operations=
HDFS: Number of write operations=
Job Counters
Launched map tasks=
Other local map tasks=
Total time spent by all maps in occupied slots (ms)=
Total time spent by all reduces in occupied slots (ms)=
Total time spent by all map tasks (ms)=
Total vcore-milliseconds taken by all map tasks=
Total megabyte-milliseconds taken by all map tasks=
Map-Reduce Framework
Map input records=
Map output records=
Input split bytes=
Spilled Records=
Failed Shuffles=
Merged Map outputs=
GC time elapsed (ms)=
CPU time spent (ms)=
Physical memory (bytes) snapshot=
Virtual memory (bytes) snapshot=
Total committed heap usage (bytes)=
File Input Format Counters
Bytes Read=
File Output Format Counters
Bytes Written=
// :: INFO mapreduce.ImportJobBase: Transferred bytes in 18.6372 seconds (1.3951 bytes/sec)
// :: INFO mapreduce.ImportJobBase: Retrieved records.
可在hdfs看到传入的数据
EFdeMacBook-Pro:jarfile FengZhen$ hadoop fs -ls /user/hive/external/sqoop_test
Found items
-rw-r--r-- FengZhen supergroup -- : /user/hive/external/sqoop_test/_SUCCESS
-rw-r--r-- FengZhen supergroup -- : /user/hive/external/sqoop_test/part-m-
可在hive中查看数据。
hive> select * from sqoop_test;
OK
fz
dx
test
Time taken: 1.756 seconds, Fetched: row(s)
使用sqoop将hive数据导出到mysql(export)
使用 sqoop 将mysql数据导入到hdfs(import)的更多相关文章
- Sqoop将mysql数据导入hbase的血与泪
Sqoop将mysql数据导入hbase的血与泪(整整搞了大半天) 版权声明:本文为yunshuxueyuan原创文章.如需转载请标明出处: https://my.oschina.net/yunsh ...
- 使用sqoop把mysql数据导入hive
使用sqoop把mysql数据导入hive export HADOOP_COMMON_HOME=/hadoop export HADOOP_MAPRED_HOME=/hadoop cp /hive ...
- 使用 sqoop 将mysql数据导入到hive表(import)
Sqoop将mysql数据导入到hive表中 先在mysql创建表 CREATE TABLE `sqoop_test` ( `id` ) DEFAULT NULL, `name` varchar() ...
- 使用sqoop将mysql数据导入到hive中
首先准备工具环境:hadoop2.7+mysql5.7+sqoop1.4+hive3.1 准备一张数据库表: 接下来就可以操作了... 一.将MySQL数据导入到hdfs 首先我测试将zhaopin表 ...
- Sqoop1.99.7将MySQL数据导入到HDFS中
准备 本示例将实现从MySQL数据库中将数据导入到HDFS中 参考文档: http://sqoop.apache.org/docs/1.99.7/user/Sqoop5MinutesDemo.html ...
- 大数据之路week07--day07 (Sqoop 从mysql增量导入到HDFS)
我们之前导入的都是全量导入,一次性全部导入,但是实际开发并不是这样,例如web端进行用户注册,mysql就增加了一条数据,但是HDFS中的数据并没有进行更新,但是又再全部导入一次又完全没有必要. 所以 ...
- python脚本 用sqoop把mysql数据导入hive
转:https://blog.csdn.net/wulantian/article/details/53064123 用python把mysql数据库的数据导入到hive中,该过程主要是通过pytho ...
- 使用sqoop将mysql数据导入到hadoop
hadoop的安装配置这里就不讲了. Sqoop的安装也很简单. 完成sqoop的安装后,可以这样测试是否可以连接到mysql(注意:mysql的jar包要放到 SQOOP_HOME/lib 下): ...
- sqoop将mysql数据导入hbase、hive的常见异常处理
原创不易,如需转载,请注明出处https://www.cnblogs.com/baixianlong/p/10700700.html,否则将追究法律责任!!! 一.需求: 1.将以下这张表(test_ ...
随机推荐
- SpringBoot启动流程分析(五):SpringBoot自动装配原理实现
SpringBoot系列文章简介 SpringBoot源码阅读辅助篇: Spring IoC容器与应用上下文的设计与实现 SpringBoot启动流程源码分析: SpringBoot启动流程分析(一) ...
- 自定义type='file'上传文件样式
改变默认的上传文件样式: 用label作为替代 <input id="file_-1" type="file" name="file" ...
- MBA人物俞洪敏:亿万富翁的生活表
我的智商非常一般,就是比别人勤奋.我的脑袋不属于特别笨的那种,但肯定也不是顶尖聪明的类型.在北大的50个同学当中,我的智商应该属于中下水平,这说明我不是顶尖高智商. 我的勤奋一般人跟不上.我平均每天工 ...
- 【JMeter4.0学习(八)】之断言
目录 响应断言 一.响应断言 1.添加线程组 2.添加HTTP请求默认值 3.添加HTTP请求1 4.先运行“HTTP请求1”,查看结果树的“取样器结果.请求.响应数据” ①取样器结果 ②请求 ③响应 ...
- 小技巧:怎样以另外的Windows用户执行SSMS
可能会碰到这种问题.你须要在一台机器上面使用不同的Windows账户连接到SQL Server做測试.默认情况下,你须要用不同的Windows账户登录然后測试. 实际上不须要每一个windows登陆. ...
- byte[] 、Bitmap与Drawbale 三者直接的转换
经常遇到这种类似头疼的问题 byte[] .Bitmap与Drawbale 三者直接的转换 1.byte[] ->Bitmap Bitmap Bitmap = BitmapFactory.dec ...
- [转]jquery中innerWidth(),outerWidth(),outerWidth(true)和width()的区别
转自:http://www.cnblogs.com/keyi/p/5933981.html jquery中innerWidth(),outerWidth(),outerWidth(true)和wi ...
- echart 图表自定义样式
initChart: function (id) { this.charts = echarts.init(document.getElementById(id)) this.charts.setOp ...
- iOS 蓝牙开发之(CoreBlueTooth)
CoreBlueTooth 简介: 可用于第三方的蓝牙交互设备 设备必须支持蓝牙4.0 iPhone的设备必须是4S或者更新 iPad设备必须是iPad mini或者更新 iOS的系统必须是iOS 6 ...
- struts2 封装获取表单数据的方式
一.属性封装 1.在action中设置成员变量,变量名与表单中的name属性值相同 2.生成变量的set方法 实例 获取用户输入的用户名和密码 jsp页面 java代码 二.模型驱动(常用) 1.ac ...