使用sqoop将hive中的数据传到mysql中

1.新建hive表

hive> create external table sqoop_test(id int,name string,age int)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> STORED AS TEXTFILE
> location '/user/hive/external/sqoop_test';
OK
Time taken: 0.145 seconds

2.给hive表添加数据

数据如下
1,fz,13
2,test,13
3,dx,18

3.将文件上传到hdfs对应目录下

hadoop fs -put sqoop_test.txt /user/hive/external/sqoop_test/
EFdeMacBook-Pro:testfile FengZhen$ hadoop fs -ls /user/hive/external/sqoop_test/
// :: WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found items
-rw-r--r-- FengZhen supergroup -- : /user/hive/external/sqoop_test/sqoop_test.txt

上传成功
进入hive 命令行可查看到数据

hive> select * from sqoop_test;
OK
fz
test
dx
Time taken: 0.089 seconds, Fetched: row(s)

4.在mysql新建表,表结构和hive中的相同

CREATE TABLE `sqoop_test` (
`id` int() DEFAULT NULL,
`name` varchar() DEFAULT NULL,
`age` int() DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1

5.使用sqoop传输数据

sqoop export 
--connect jdbc:mysql://localhost:3306/sqooptest --username root --password 123qwe --table sqoop_test
--export-dir /user/hive/external/sqoop_test --input-fields-terminated-by ,
EFdeMacBook-Pro:bin FengZhen$ sqoop export --connect jdbc:mysql://localhost:3306/sqooptest --username root --password 123qwe --table sqoop_test --export-dir /user/hive/external/sqoop_test --input-fields-terminated-by ,
Warning: /Users/FengZhen/Desktop/Hadoop/sqoop-1.4..bin__hadoop-2.0.-alpha/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /Users/FengZhen/Desktop/Hadoop/sqoop-1.4..bin__hadoop-2.0.-alpha/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hadoop-2.8./share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hbase-1.3./lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.
// :: WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
// :: INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
// :: INFO tool.CodeGenTool: Beginning code generation
// :: INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sqoop_test` AS t LIMIT
// :: INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sqoop_test` AS t LIMIT
// :: INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /Users/FengZhen/Desktop/Hadoop/hadoop-2.8.
// :: INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-FengZhen/compile/7a078053fb0424d718e08c56fc9bab27/sqoop_test.jar
// :: INFO mapreduce.ExportJobBase: Beginning export of sqoop_test
// :: INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
// :: WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
// :: INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
// :: INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
// :: INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
// :: INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
// :: INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:
// :: INFO input.FileInputFormat: Total input files to process :
// :: INFO input.FileInputFormat: Total input files to process :
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1505268150495_0004
// :: INFO impl.YarnClientImpl: Submitted application application_1505268150495_0004
// :: INFO mapreduce.Job: The url to track the job: http://192.168.1.64:8088/proxy/application_1505268150495_0004/
// :: INFO mapreduce.Job: Running job: job_1505268150495_0004
// :: INFO mapreduce.Job: Job job_1505268150495_0004 running in uber mode : false
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: Job job_1505268150495_0004 completed successfully
// :: INFO mapreduce.Job: Counters:
File System Counters
FILE: Number of bytes read=
FILE: Number of bytes written=
FILE: Number of read operations=
FILE: Number of large read operations=
FILE: Number of write operations=
HDFS: Number of bytes read=
HDFS: Number of bytes written=
HDFS: Number of read operations=
HDFS: Number of large read operations=
HDFS: Number of write operations=
Job Counters
Launched map tasks=
Data-local map tasks=
Total time spent by all maps in occupied slots (ms)=
Total time spent by all reduces in occupied slots (ms)=
Total time spent by all map tasks (ms)=
Total vcore-milliseconds taken by all map tasks=
Total megabyte-milliseconds taken by all map tasks=
Map-Reduce Framework
Map input records=
Map output records=
Input split bytes=
Spilled Records=
Failed Shuffles=
Merged Map outputs=
GC time elapsed (ms)=
CPU time spent (ms)=
Physical memory (bytes) snapshot=
Virtual memory (bytes) snapshot=
Total committed heap usage (bytes)=
File Input Format Counters
Bytes Read=
File Output Format Counters
Bytes Written=
// :: INFO mapreduce.ExportJobBase: Transferred bytes in 26.9573 seconds (28.1185 bytes/sec)
// :: INFO mapreduce.ExportJobBase: Exported records.

传输完成,mysql已经有数据了。

使用sqoop将mysql数据导入到hdfs

使用 sqoop 将 hive 数据导出到 mysql (export)的更多相关文章

  1. 利用sqoop将hive数据导入导出数据到mysql

    一.导入导出数据库常用命令语句 1)列出mysql数据库中的所有数据库命令  #  sqoop list-databases --connect jdbc:mysql://localhost:3306 ...

  2. 从hive将数据导出到mysql(转)

    从hive将数据导出到mysql http://abloz.com 2012.7.20 author:周海汉 在上一篇文章<用sqoop进行mysql和hdfs系统间的数据互导>中,提到s ...

  3. Hive数据导出的几种方式

    在hive的日常使用中,经常需要将hive表中的数据导出来,虽然hive提供了多种导出方式,但是面对不同的数据量.不同的需求,如果随意就使用某种导出方式,可能会导致导出时间过长,导出的结果不满足需求, ...

  4. MSSQL数据导出到MYSQL

    MSSQL数据导出到MYSQL 花了一天时间把MSSQL里的数据导出到MYSQL, 好麻烦,二个数据库都是阿里云买的云服务器. 先上阿里云控制面板,备份下MSSQL数据库,下载备份下来,在本地电脑上还 ...

  5. 使用JDBC+POI把Excel中的数据导出到MySQL

    POI是Apache的一套读MS文档的API,用它还是可以比较方便的读取Office文档的.目前支持Word,Excel,PowerPoint生成的文档,还有Visio和Publisher的. htt ...

  6. 如何利用sqoop将hive数据导入导出数据到mysql

    运行环境  centos 5.6   hadoop  hive sqoop是让hadoop技术支持的clouder公司开发的一个在关系数据库和hdfs,hive之间数据导入导出的一个工具. 上海尚学堂 ...

  7. [Sqoop]将Hive数据表导出到Mysql

    业务背景 mysql表YHD_CATEG_PRIOR的结构例如以下: -- Table "YHD_CATEG_PRIOR" DDL CREATE TABLE `YHD_CATEG_ ...

  8. 用java代码调用shell脚本执行sqoop将hive表中数据导出到mysql

    1:创建shell脚本 touch sqoop_options.sh chmod 777 sqoop_options.sh 编辑文件  特地将执行map的个数设置为变量  测试 可以java代码传参数 ...

  9. Hive总结(八)Hive数据导出三种方式

    今天我们再谈谈Hive中的三种不同的数据导出方式. 依据导出的地方不一样,将这些方式分为三种: (1).导出到本地文件系统. (2).导出到HDFS中: (3).导出到Hive的还有一个表中. 为了避 ...

随机推荐

  1. jQuery 遍历 - eq() 和siblings() 方法

    eq() 方法将匹配元素集缩减值指定 index 上的一个. 通过为 index 为 2 的 div 加入适当的类.将其变为蓝色: <!DOCTYPE html> <html> ...

  2. 第3章 如何编写函数定义 3.7 if特殊表

    这部分来学习下if特殊表,之前学了defun和let,不好意思,博客中没有写但是鄙人已经看了,哈哈. 什么是if表 if条件特殊表是为了让计算机对条件加以判断,然后选择不同的执行路径的. if特殊表的 ...

  3. Teradata架构

    Teradata在整体上是按Shared Nothing 架构体系进行组织的,他的定位就是大型数据仓库系统,定位比较高,他的软硬件都是NCR自己的,其他的都不识别:所以一般的企业用不起,价格很贵.由于 ...

  4. oracle10g安装问题

    oracle10g的安装还是比较容易的,一直下一步就行了,但是今天安装的时候遇到了一个新问题,在安装的过程中提示提示一些 Configuration Assistant失败刚开始,我直接跳过去,但后面 ...

  5. [LeetCode] Remove Duplicates from Sorted Array II [27]

    题目 Follow up for "Remove Duplicates": What if duplicates are allowed at most twice? For ex ...

  6. Coursera machine learning 第二周 quiz 答案 Octave/Matlab Tutorial

    https://www.coursera.org/learn/machine-learning/exam/dbM1J/octave-matlab-tutorial Octave Tutorial 5  ...

  7. C语言基础知识【数组】

    2017年7月11日17:34:05C 数组1.C 语言支持数组数据结构,它可以存储一个固定大小的相同类型元素的顺序集合.数组是用来存储一系列数据,但它往往被认为是一系列相同类型的变量.数组的声明并不 ...

  8. Entity Framework 4.1:复杂类型

    这篇文章将讨论复杂类型. 默认情况下,EF4.1 将类映射到表,这是约定,但是有时候,我们需要模型比表的粒度更细一些. 地址是一个典型的例子,看一下下面的客户类. )] publicstring St ...

  9. 一些blog地址总结整理:

    女神 python之路-网络编程初版:https://www.cnblogs.com/Eva-J/articles/8066842.html python之路-网络编程(重点看这个,更细致):http ...

  10. iOS 字符串截取,将字符串中用括号包含的内容去除

    //去除字符串中用括号括住的位置 -(NSString *)handleStringWithString:(NSString *)str{ NSMutableString * muStr = [NSM ...