sqoop2的使用测试

查看现有link

删除link

sqoop:000> delete link -name mysql2
sqoop:000> show link
+------+----------------+---------+
| Name | Connector Name | Enabled |
+------+----------------+---------+
+------+----------------+---------+

创建mysql link

sqoop:000> create link -c generic-jdbc-connector

Creating link for connector with name generic-jdbc-connector Please fill following values to create new link object

Name: mysqllink

Database connection

Driver class: com.mysql.jdbc.Driver

Connection String: jdbc:mysql://127.0.0.1:3306/test

Username: root

Password: ****

Fetch Size: 30

Connection Properties: There are currently 0 values in the map: entry#

SQL Dialect

Identifier enclose: 此处为空格

New link was successfully created with validation status OK and name mysqllink

查看新建的链接：

创建hdfs链接

sqoop:000> create link -c hdfs-connector

Creating link for connector with name hdfs-connector Please fill following values to create new link object

Name: hdfslink

HDFS cluster

URI: localhost:9000

Conf directory: /root/hadoop/hadoop-2.7.4/etc/hadoop

Additional configs:: There are currently 0 values in the map: entry#

New link was successfully created with validation status OK and name hdfslink

--------------------------------------------------------------------------------------

在创建mysql链接时，Identifier enclose：指定SQL中标识符的定界符，也就是说，有的SQL标示符是一个引号：select * from "table_name"，这种定界符在MySQL中是会报错的。这个属性默认值就是双引号，所以不能使用回车，必须将之覆盖，可以使用空格覆盖

hadoop fs -mkdir /sqoop

在sqoop客户端设置查看job详情：

set option --name verbose --value true

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

org.apache.hadoop.ipc.RemoteException: User: root is not allowed to impersonate root

如下处理：

在core-site.xml添加如下属性，其中

hadoop.proxyuser.root.groups中的root是用户，value里面的root是group

<property>

      <name>hadoop.proxyuser.root.groups</name>

      <value>root</value>

      <description>Allow the superuser oozie to impersonate any members of the group group1 and group2</description>

 </property>

 <property>

      <name>hadoop.proxyuser.root.hosts</name>

      <value>localhost</value>

      <description>The superuser can connect only from host1 and host2 to impersonate a user</description>

  </property>

--------------------------------------------------------------------------

create job -f "mysql-link" -t "hdfs-link"

-f指定from，即是数据源位置，-t指定to，即是目的地位置。本例是从MySQL传递数据到HDFS，所以就是from mysql to HDFS。参数值就是在创建链接（link）时指定的Name。

以下是各个属性

Name：一个标示符，自己指定即可。

Schema Name：指定Database或Schema的名字，在MySQL中，Schema同Database类似，具体什么区别没有深究过，但官网描述在创建时差不多。这里指定数据库名字为db_ez即可，本例的数据库。

Table Name：本例使用的数据库表为tb_forhadoop，自己指定导出的表。多表的情况请自行查看官方文档。

SQL Statement：就是sql查询语句，文档上说需要指定一个$condition，但我一直没有创建成功，貌似是一个条件子句。

配置完以上几项，又回出现element#提示符，提示输入一些hash值，这里我没有再指定什么。直接回车过。而以下几个配置我也是直接回车，使用默认值，大概是一些与数据库相关的参数。

Partition column:

Partition column nullable:

Boundary query

Last value

后面需要配置数据目的地各项值：

Null alue：大概说的是如果有空值用什么覆盖

File format：指定在HDFS中的数据文件是什么文件格式，这里使用TEXT_FILE，即最简单的文本文件。

Compression codec：用于指定使用什么压缩算法进行导出数据文件压缩，我指定NONE，这个也可以使用自定义的压缩算法CUSTOM，用Java实现相应的接口。

Custom codec：这个就是指定的custom压缩算法，本例选择NONE，所以直接回车过去。

Output directory：指定存储在HDFS文件系统中的路径，这里最好指定一个存在的路径，或者存在但路劲下是空的，貌似这样才能成功。

Append mode：用于指定是否是在已存在导出文件的情况下将新数据追加到数据文件中。

Extractors：不清楚是什么，我取了一个1

Loaders：同上

最后再次出现element#提示符，用于输入extra mapper jars的属性，可以什么都不写。直接回车。

至此若出现successful则证明已经成功创建。

创建job mysqltohdfs，将mysql导入到hdfs:

sqoop:000> show job -n mysqltohdfs
2018-05-28 17:12:55,421 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-28 17:12:55,425 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:12:55,431 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
1 job(s) to show:
Job with name mysqltohdfs (Enabled: true, Created by root at 5/28/18 3:52 PM, Updated by root at 5/28/18 3:52 PM)
Throttling resources
    Extractors:
    Loaders:
Classpath configuration
    Extra mapper jars:
From link: mysqllink
Database source
    Schema name: test   数据库必填
    Table name: role   表名必填
    SQL statement:
    Column names:
    Partition column:
    Partition column nullable:
    Boundary query:
Incremental read
    Check column:
    Last value:
To link: hdfslink
Target configuration
    Override null value:
    Null value:
    File format: TEXT_FILE    必选
    Compression codec: NONE 必选
    Custom codec:
    Output directory: hdfs://localhost:9000/sqoop   数据输出目标路径
    Append mode:

sqoop:000> start job -n mysqltohdfs
2018-05-28 17:07:02,252 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-28 17:07:02,256 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,259 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,262 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,263 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,264 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,268 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Found primary key columns [roleid]
2018-05-28 17:07:02,268 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Using partition column: roleid
2018-05-28 17:07:02,269 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Using min/max query: SELECT MIN( roleid ), MAX( roleid ) FROM test . role
2018-05-28 17:07:02,269 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Boundaries for the job: min=1, max=650, columnType=4
2018-05-28 17:07:02,269 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Using dataSql: SELECT * FROM test . role WHERE ${CONDITIONS}
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer - Field names: roleid , name , dateid , addr , sex , level
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file mapred-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file kms-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file core-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file hdfs-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file yarn-site.xml
2018-05-28 17:07:02,270 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file httpfs-site.xml
2018-05-28 17:07:02,318 [pool-1-thread-5] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - Running on unsecured cluster, skipping delegation token generation.
2018-05-28 17:07:02,318 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsToInitializer - Using working directory: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2
2018-05-28 17:07:02,354 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:02,904 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:02,905 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:02,905 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:02,905 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:02,906 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:02,909 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:02,960 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:02,960 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:02,960 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,008 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,009 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,014 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,014 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,014 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,014 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,015 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,018 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,020 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,021 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,021 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,103 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,104 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,107 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,107 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,107 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,107 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,108 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,116 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,122 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,122 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,123 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,178 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,179 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,182 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,183 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,184 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,184 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,184 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,186 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,187 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,187 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,188 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,273 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,274 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,276 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,277 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,277 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,277 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,277 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,279 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,280 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,280 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,281 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,340 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,340 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,347 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,347 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,347 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,348 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,348 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,351 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,352 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,352 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,352 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,423 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,424 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,428 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,429 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,429 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,429 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,439 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,442 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,444 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,444 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,446 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,535 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,535 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,538 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,538 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,539 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,540 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,541 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,565 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,566 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,566 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,567 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,618 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,618 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,621 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,621 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,621 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,621 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,622 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,634 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid < ?
2018-05-28 17:07:03,635 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,635 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,636 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,685 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,686 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,688 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-28 17:07:03,689 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-28 17:07:03,689 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor
2018-05-28 17:07:03,689 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.hdfs.HdfsLoader
2018-05-28 17:07:03,689 [OutputFormatLoader-consumer] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,691 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Creating PreparedStatement with query: SELECT * FROM test . role WHERE ? <= roleid AND roleid <= ?
2018-05-28 17:07:03,731 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-28 17:07:03,731 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-28 17:07:03,731 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-28 17:07:03,771 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-28 17:07:03,772 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-28 17:07:03,775 [Thread-141] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.jdbc.GenericJdbcFromDestroyer
2018-05-28 17:07:03,776 [Thread-141] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromDestroyer - Running generic JDBC connector destroyer
2018-05-28 17:07:03,777 [Thread-141] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.hdfs.HdfsToDestroyer
2018-05-28 17:07:03,795 [Thread-141] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-28 17:07:03,798 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt of size 53
2018-05-28 17:07:03,816 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt of size 120
2018-05-28 17:07:03,838 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/60f3cc4e-3e1e-4276-9e75-4ce652a82be6.txt of size 0
2018-05-28 17:07:03,849 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt of size 50
2018-05-28 17:07:03,862 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt of size 58
2018-05-28 17:07:03,874 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/881898e2-8545-437e-b4cb-4838ac489c87.txt of size 102
2018-05-28 17:07:03,895 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/a61f0265-0379-4a8b-8e70-f367fb2479a3.txt of size 0
2018-05-28 17:07:03,908 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/ad14cb0d-29e4-486e-b342-d6bec60d2463.txt of size 0
2018-05-28 17:07:03,919 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/be952dc7-ab88-4ab5-9ec1-0c2e8d064ccf.txt of size 0
2018-05-28 17:07:03,933 [Thread-141] INFO org.apache.sqoop.connector.hdfs.HdfsToDestroyer - Committing file: hdfs://localhost:9000/sqoop/.ddb5b1a1-35f7-44ae-b719-3cf63f0677c2/e3f5125d-872e-4312-b1dd-2154be3b74f0.txt of size 0
2018-05-28 17:07:03,960 [Thread-141] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.jdbc.GenericJdbcFromDestroyer
2018-05-28 17:07:03,960 [Thread-141] INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromDestroyer - Running generic JDBC connector destroyer
2018-05-28 17:07:03,961 [Thread-141] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.hdfs.HdfsToDestroyer
2018-05-28 17:07:03,979 [Thread-141] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
Exception in thread "Thread-141" java.lang.NoSuchMethodError: org.apache.http.impl.client.DefaultHttpClient.execute(Lorg/apache/http/client/methods/HttpUriRequest;)Lorg/apache/http/client/methods/CloseableHttpResponse;
        at org.apache.hadoop.mapred.JobEndNotifier.httpNotification(JobEndNotifier.java:76)
        at org.apache.hadoop.mapred.JobEndNotifier.localRunnerNotification(JobEndNotifier.java:86)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:562)
2018-05-28 17:07:05,326 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-28 17:07:04 CST: FAILED

虽然有异常抛出，但数据成功写入hdfs,查看结果如下：

[root@host ~]# hdfs dfs -ls -R /sqoop
-rw-r--r--   1 root supergroup         53 2018-05-28 17:07 /sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt
-rw-r--r--   1 root supergroup        120 2018-05-28 17:07 /sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
-rw-r--r--   1 root supergroup          0 2018-05-28 17:07 /sqoop/60f3cc4e-3e1e-4276-9e75-4ce652a82be6.txt
-rw-r--r--   1 root supergroup         50 2018-05-28 17:07 /sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt
-rw-r--r--   1 root supergroup         58 2018-05-28 17:07 /sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt
-rw-r--r--   1 root supergroup        102 2018-05-28 17:07 /sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
-rw-r--r--   1 root supergroup          0 2018-05-28 17:07 /sqoop/a61f0265-0379-4a8b-8e70-f367fb2479a3.txt
-rw-r--r--   1 root supergroup          0 2018-05-28 17:07 /sqoop/ad14cb0d-29e4-486e-b342-d6bec60d2463.txt
-rw-r--r--   1 root supergroup          0 2018-05-28 17:07 /sqoop/be952dc7-ab88-4ab5-9ec1-0c2e8d064ccf.txt
-rw-r--r--   1 root supergroup          0 2018-05-28 17:07 /sqoop/e3f5125d-872e-4312-b1dd-2154be3b74f0.txt

[root@host ~]# hdfs dfs -cat /sqoop/*
400,NULL,'2017-11-15 14:49:56.000','anhui hefei',0,4
1,NULL,'2017-11-16 14:49:11.000','henan luohe linying',1,10
40,NULL,'2017-11-13 14:50:25.000','guangdong shenzhen',1,20
110,NULL,'2017-11-14 14:50:47.000','beijing',1,20
200,NULL,'2017-11-14 14:49:47.000','shandong qingdao',0,8
600,NULL,'2017-11-15 14:50:05.000','hunan changsha',0,91
650,NULL,'2017-11-01 17:24:34.000',NULL,1,29

查看作业的状态

sqoop:000> status job -n mysqltohdfs
2018-05-29 10:55:44,766 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-29 10:55:44,768 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 10:55:44,771 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 10:55:44,789 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
Submission details
Job Name: mysqltohdfs
Server URL: http://localhost:12000/sqoop/
Created by: root
Creation date: 2018-05-28 17:07:02 CST
Lastly updated by: root
External ID: job_local1297704653_0002
http://localhost:8080/
2018-05-28 17:07:04 CST: FAILED

hdfs导入到mysql

创建空表role1：

mysql> create table role1 select * from role limit 0; Query OK, 0 rows affected (0.31 sec) Records: 0 Duplicates: 0 Warnings: 0

mysql> select * from role1 ;

Empty set (0.00 sec)

创建job

sqoop:000> create job -f hdfslink -t mysqllink

Creating job for links with from name hdfslink and to name mysqllink 2018-05-29 11:17:23,362 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:17:23,363 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:17:23,367 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:17:23,368 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit Please fill following values to create new job object Name: hdfs2018-05-29 11:17:27,053 [UpdateThread] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit

Name: hdfstomysql

Input configuration

Input directory: hdfs://localhost:9000/sqoop

Override null value: Null value:

Incremental import

Incremental type: 0 : NONE 1 : NEW_FILES

Choose: 0

Last imported date:

Database target

Schema name: test

Table name: role1

Column names:

There are currently 0 values in the list: element#

Staging table:

Clear stage table:

Throttling resources

Extractors: Loaders:

Classpath configuration

Extra mapper jars: There are currently 0 values in the list: element# 2018-05-29 11:18:02,792 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request 2018-05-29 11:18:02,793 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:18:02,795 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit 2018-05-29 11:18:02,798 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit New job was successfully created with validation status OK and name hdfstomysql

查看job:

sqoop:000> show job
2018-05-29 11:18:06,188 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-29 11:18:06,193 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:06,196 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:06,198 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
+----+-------------+------------------------------------+------------------------------------+---------+
| Id |    Name     |           From Connector           |            To Connector            | Enabled |
+----+-------------+------------------------------------+------------------------------------+---------+
| 15 | mysqltohdfs | mysqllink (generic-jdbc-connector) | hdfslink (hdfs-connector)          | true    |
| 17 | hdfstomysql | hdfslink (hdfs-connector)          | mysqllink (generic-jdbc-connector) | true    |
+----+-------------+------------------------------------+------------------------------------+---------+

执行job
sqoop:000> start job -n hdfstomysql
2018-05-29 11:18:19,429 [pool-1-thread-5] INFO org.apache.sqoop.handler.JobRequestHandler - Got job request
2018-05-29 11:18:19,432 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,434 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,437 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,438 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,440 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,442 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file mapred-site.xml
2018-05-29 11:18:19,442 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file kms-site.xml
2018-05-29 11:18:19,442 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file core-site.xml
2018-05-29 11:18:19,443 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file hdfs-site.xml
2018-05-29 11:18:19,443 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file yarn-site.xml
2018-05-29 11:18:19,443 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsUtils - Found Hadoop configuration file httpfs-site.xml
2018-05-29 11:18:19,484 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsFromInitializer - Input directory: hdfs://localhost:9000/sqoop
2018-05-29 11:18:19,489 [pool-1-thread-5] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - Running on unsecured cluster, skipping delegation token generation.
2018-05-29 11:18:19,509 [pool-1-thread-5] INFO org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer - Using query to insert data: INSERT INTO test . role1 VALUES (?,?,?,?,?,?)
2018-05-29 11:18:19,514 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:19,666 [pool-1-thread-5] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:19,671 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt
2018-05-29 11:18:19,671 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/60f3cc4e-3e1e-4276-9e75-4ce652a82be6.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/a61f0265-0379-4a8b-8e70-f367fb2479a3.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/ad14cb0d-29e4-486e-b342-d6bec60d2463.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/be952dc7-ab88-4ab5-9ec1-0c2e8d064ccf.txt
2018-05-29 11:18:19,672 [pool-1-thread-5] INFO org.apache.sqoop.connector.hdfs.HdfsPartitioner - Will process input file: hdfs://localhost:9000/sqoop/e3f5125d-872e-4312-b1dd-2154be3b74f0.txt
2018-05-29 11:18:20,386 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,388 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,389 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,389 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,389 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt (offset=0, end=26, length=26), hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt (offset=26, end=53, length=27)}
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 0
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 26
2018-05-29 11:18:20,416 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 26
2018-05-29 11:18:20,470 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,473 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,489 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 53
2018-05-29 11:18:20,490 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/40306f6b-8dc9-41d4-b814-ac4630bcbf08.txt
2018-05-29 11:18:20,490 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 26
2018-05-29 11:18:20,490 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 53
2018-05-29 11:18:20,490 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 27
2018-05-29 11:18:20,503 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,511 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 53
2018-05-29 11:18:20,511 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 53
2018-05-29 11:18:20,512 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,512 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,512 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,565 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,565 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,589 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,590 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,590 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,590 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,590 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt (offset=0, end=39, length=39)}
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 0
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 39
2018-05-29 11:18:20,609 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 39
2018-05-29 11:18:20,622 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,624 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,627 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 120
2018-05-29 11:18:20,628 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,628 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,628 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,629 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,630 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,633 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,633 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,633 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,634 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,634 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt (offset=39, end=78, length=39)}
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 39
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 78
2018-05-29 11:18:20,658 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 39
2018-05-29 11:18:20,667 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,671 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 60
2018-05-29 11:18:20,671 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 120
2018-05-29 11:18:20,671 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,671 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,672 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,672 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,673 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,675 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,675 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,675 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,675 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,675 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt (offset=78, end=99, length=21), hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt (offset=99, end=120, length=21)}
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 78
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 99
2018-05-29 11:18:20,692 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 21
2018-05-29 11:18:20,700 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 120
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 120
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/44f1542c-3475-4154-9d42-0ad2b8f70ddd.txt
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 99
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 120
2018-05-29 11:18:20,704 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 21
2018-05-29 11:18:20,712 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,715 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 120
2018-05-29 11:18:20,715 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 120
2018-05-29 11:18:20,716 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,716 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,716 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,717 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,717 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,721 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,721 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,721 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,721 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,721 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt (offset=0, end=25, length=25), hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt (offset=25, end=50, length=25)}
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 0
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 25
2018-05-29 11:18:20,742 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 25
2018-05-29 11:18:20,750 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,751 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,753 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 50
2018-05-29 11:18:20,753 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/63777ee8-6c52-4d3c-a6c8-78bab90227f7.txt
2018-05-29 11:18:20,753 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 25
2018-05-29 11:18:20,754 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 50
2018-05-29 11:18:20,754 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 25
2018-05-29 11:18:20,759 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,762 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 50
2018-05-29 11:18:20,763 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 50
2018-05-29 11:18:20,763 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,763 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,763 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,764 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,764 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,767 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,767 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,767 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,767 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,767 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt (offset=0, end=29, length=29), hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt (offset=29, end=58, length=29)}
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 0
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 29
2018-05-29 11:18:20,787 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 29
2018-05-29 11:18:20,796 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,798 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,801 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 58
2018-05-29 11:18:20,802 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/70b3b82c-c188-4fdc-afd7-7aee82077f47.txt
2018-05-29 11:18:20,802 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 29
2018-05-29 11:18:20,802 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 58
2018-05-29 11:18:20,802 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 29
2018-05-29 11:18:20,808 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,811 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 58
2018-05-29 11:18:20,811 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 58
2018-05-29 11:18:20,811 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,812 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,812 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,813 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,813 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,815 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,816 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,816 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,816 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,817 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt (offset=0, end=39, length=39)}
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 0
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 39
2018-05-29 11:18:20,834 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 39
2018-05-29 11:18:20,843 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,844 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 0
2018-05-29 11:18:20,848 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 102
2018-05-29 11:18:20,848 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,848 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,848 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,849 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,849 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,851 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is starting
2018-05-29 11:18:20,852 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Starting progress service
2018-05-29 11:18:20,852 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class org.apache.sqoop.connector.jdbc.GenericJdbcLoader
2018-05-29 11:18:20,852 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Running extractor class org.apache.sqoop.connector.hdfs.HdfsExtractor
2018-05-29 11:18:20,852 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hadoop.security.SecurityUtils - No delegation tokens found
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Working on partition: {hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt (offset=39, end=70, length=31), hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt (offset=70, end=102, length=32)}
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 39
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 70
2018-05-29 11:18:20,872 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 31
2018-05-29 11:18:20,882 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,889 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 57
2018-05-29 11:18:20,889 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 102
2018-05-29 11:18:20,890 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting file hdfs://localhost:9000/sqoop/881898e2-8545-437e-b4cb-4838ac489c87.txt
2018-05-29 11:18:20,890 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     from offset 70
2018-05-29 11:18:20,890 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     to offset 102
2018-05-29 11:18:20,890 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor -     of length 32
2018-05-29 11:18:20,898 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting text file
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Start position: 102
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.connector.hdfs.HdfsExtractor - Extracting ended on position: 102
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Extractor has finished
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopMapper - Stopping progress service
2018-05-29 11:18:20,903 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2018-05-29 11:18:20,904 [OutputFormatLoader-consumer] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Loader has finished
2018-05-29 11:18:20,905 [LocalJobRunner Map Task Executor #0] INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
2018-05-29 11:18:20,914 [Thread-255] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.hdfs.HdfsFromDestroyer
2018-05-29 11:18:20,915 [Thread-255] INFO org.apache.sqoop.connector.hdfs.HdfsFromDestroyer - Running HDFS connector destroyer
2018-05-29 11:18:20,915 [Thread-255] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.jdbc.GenericJdbcToDestroyer
2018-05-29 11:18:20,915 [Thread-255] INFO org.apache.sqoop.connector.jdbc.GenericJdbcToDestroyer - Running generic JDBC connector destroyer
2018-05-29 11:18:20,926 [Thread-255] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.hdfs.HdfsFromDestroyer
2018-05-29 11:18:20,926 [Thread-255] INFO org.apache.sqoop.connector.hdfs.HdfsFromDestroyer - Running HDFS connector destroyer
2018-05-29 11:18:20,927 [Thread-255] INFO org.apache.sqoop.job.mr.SqoopDestroyerExecutor - Executing destroyer class class org.apache.sqoop.connector.jdbc.GenericJdbcToDestroyer
2018-05-29 11:18:20,927 [Thread-255] INFO org.apache.sqoop.connector.jdbc.GenericJdbcToDestroyer - Running generic JDBC connector destroyer
Exception in thread "Thread-255" java.lang.NoSuchMethodError: org.apache.http.impl.client.DefaultHttpClient.execute(Lorg/apache/http/client/methods/HttpUriRequest;)Lorg/apache/http/client/methods/CloseableHttpResponse;
        at org.apache.hadoop.mapred.JobEndNotifier.httpNotification(JobEndNotifier.java:76)
        at org.apache.hadoop.mapred.JobEndNotifier.localRunnerNotification(JobEndNotifier.java:86)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:562)
2018-05-29 11:18:22,031 [pool-1-thread-5] INFO org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commit
2018-05-29 11:18:21 CST: FAILED

查看mysql表中数据

mysql> select * from test.role1;
+--------+------+---------------------+---------------------+------+-------+
| roleid | name | dateid              | addr                | sex | level |
+--------+------+---------------------+---------------------+------+-------+
|    400 | NULL | 2017-11-15 14:49:56 | anhui hefei         |    0 |     4 |
|      1 | NULL | 2017-11-16 14:49:11 | henan luohe linying |    1 |    10 |
|     40 | NULL | 2017-11-13 14:50:25 | guangdong shenzhen |    1 |    20 |
|    110 | NULL | 2017-11-14 14:50:47 | beijing             |    1 |    20 |
|    200 | NULL | 2017-11-14 14:49:47 | shandong qingdao    |    0 |     8 |
|    600 | NULL | 2017-11-15 14:50:05 | hunan changsha      |    0 |    91 |
|    650 | NULL | 2017-11-01 17:24:34 | NULL                |    1 |    29 |
+--------+------+---------------------+---------------------+------+-------+
7 rows in set (0.00 sec)

sqoop2的使用测试的更多相关文章

sqoop1 与sqoop2的对比
Sqoop是一款开源的工具,主要用于在Hadoop和传统的数据库(mysql.postgresql等)进行数据的传递,可以将一个关系型数据库(例如:MySQL.Oracle.Postgres等)中的数 ...
sqoop2安装总结
sqoop2安装 1. 下载解压缩此次安装版本为1.99.6 # Decompress Sqoop distribution tarball tar -xvf sqoop-<version&g ...
Sqoop2开启Kerberos安全模式
Sqoop2开启Kerberos安全模式, 基于版本sqoop-1.99.7, 在已经安装好的sqoop2环境上配置kerberos. 1.安装规划 10.43.159.9 zdh-9 sqoop2k ...
SignalR系列续集[系列8:SignalR的性能监测与服务器的负载测试]
目录 SignalR系列目录前言也是好久没写博客了,近期确实很忙,嗯..几个项目..头要炸..今天忙里偷闲.继续我们的小系列.. 先谢谢大家的支持.. 我们来聊聊SignalR的性能监测与服务器的 ...
Apache Ignite之集群应用测试
集群发现机制在Ignite中的集群号称是无中心的,而且支持命令行启动和嵌入应用启动,所以按理说很简单.而且集群有自动发现机制感觉对于懒人开发来说太好了,抱着试一试的心态测试一下吧. 在Apache ...
测试一下StringBuffer和StringBuilder及字面常量拼接三种字符串的效率
之前一篇里写过字符串常用类的三种方式<java中的字符串相关知识整理>,只不过这个只是分析并不知道他们之间会有多大的区别,或者所谓的StringBuffer能提升多少拼接效率呢?为此写个简 ...
TechEmpower 13轮测试中的ASP.NET Core性能测试
应用性能直接影响到托管服务的成本,因此公司在开发应用时需要格外注意应用所使用的Web框架,初创公司尤其如此.此外,糟糕的应用性能也会影响到用户体验,甚至会因此受到相关搜索引擎的降级处罚.在选择框架时, ...
.NET Core系列：4 测试
2016.6.27 微软已经正式发布了.NET Core 1.0 RTM,但是工具链还是预览版,同样的大量的开源测试库也都是至少发布了Alpha测试版支持.NET Core, 这篇文章 The Sta ...
渗透测试工具BurpSuite做网站的安全测试（基础版）
渗透测试工具BurpSuite做网站的安全测试(基础版) 版权声明:本文为博主原创文章,未经博主允许不得转载. 学习网址: https://t0data.gitbooks.io/burpsuite/c ...

随机推荐

repo学习笔记
1. 遍历所有的git仓库,并在每个仓库执行-c所指定的命令(被执行的命令不限于git命令,而是任何被系统支持的命令,比如:ls . pwd .cp 等 . $ repo forall -c &quo ...
作为一名IT从业者，你在工作和学习中，遇到哪些问题
版权声明:襄阳雷哥的版权声明 https://blog.csdn.net/FansUnion/article/details/28448975 大家都是IT从业者,遇到的问题多少与类似. 假设能把这些 ...
单变量微积分笔记20——三角替换1（sin和cos）
sin和cos的常用公式基本公式: 半角公式: 微分公式: 积分公式: 三角替换示例1 根据微分公式,cosxdx = dsinx 示例2 示例3 半角公式示例1 示例2 解法1: 解法2: 综 ...
vue-cli、webpack提取第三方库-----DllPlugin、DllReferencePlugin
需要安装的插件有 extract-text-webpack-plugin assets-webpack-plugin clean-webpack-plugin npm install extract- ...
WebClient类
WebClient类提供向 URI 标识的资源发送数据和从 URI 标识的资源接收数据的公共方法. 其实就相当于创建一个请求客户端.可以获取网页和各种各样的信息,包括交互. 通过MSDN来看看WebC ...
c# ProgressBar进度条方向和美观
protected override CreateParams CreateParams { get { CreateParams cp = base.CreateParams; cp.Style | ...
谈windows中的句柄
谈windows中的句柄每当一个进程打开一个对象,系统就返回一个句柄作为凭证,由此可以想到,句柄是依赖于具体的进程的,换句话说,句柄一定属于某个进程,以后在访问这个对象时就要使用这个凭证! ...
hosts,No web site is configured at this address.
解决办法: IIS管理器->右击网站->属性->网站——>IP地址一项->选择全部未分配-> 确定关闭问题解决.
TFS 2012如何切换用户
TFS 2012如何切换用户编写人:左丘文 2018-3-8 春节假期来后,准备干活的时候,才发现TFS账户登入的是另外一个账户.现在想切换为自己的账户时,发现Vs 2012中没找到可以登出的功能, ...
【Spring学习笔记-MVC-9】SpringMVC数据格式化之日期转换@DateTimeFormat
作者:ssslinppp 1. 摘要本文主要讲解Spring mvc数据格式化的具体步骤: 并讲解前台日期格式如何转换为java对象: 在之前的文章<[Spring学习笔记-MVC ...

sqoop2的使用测试

org.apache.hadoop.ipc.RemoteException: User: root is not allowed to impersonate root

sqoop2的使用测试的更多相关文章

随机推荐

热门专题