Sqoop的安装和验证

　　Sqoop是一个用来完成Hadoop和关系型数据库中的数据相互转移的工具，它可以将关系型数据库中的数据导入到Hadoop的HDFS中，也可以将HDFS的数据导入到关系型数据库中。

　　Kafka是一个开源的分布式消息订阅系统

　　一、Sqoop的安装

　　1.http://www-eu.apache.org/dist/sqoop/1.4.7/下载sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz并解压到/home/jun下

[jun@master sqoop-1.4..bin__hadoop-2.6.]$ ls -l
total
drwxr-xr-x.  jun jun     Dec    bin
-rw-rw-r--.  jun jun    Dec    build.xml
-rw-rw-r--.  jun jun    Dec    CHANGELOG.txt
-rw-rw-r--.  jun jun     Dec    COMPILING.txt
drwxr-xr-x.  jun jun      Dec    conf
drwxr-xr-x.  jun jun      Dec    docs
drwxr-xr-x.  jun jun       Dec    ivy
-rw-rw-r--.  jun jun    Dec    ivy.xml
drwxr-xr-x.  jun jun     Dec    lib
-rw-rw-r--.  jun jun    Dec    LICENSE.txt
-rw-rw-r--.  jun jun      Dec    NOTICE.txt
-rw-rw-r--.  jun jun    Dec    pom-old.xml
-rw-rw-r--.  jun jun     Dec    README.txt
-rw-rw-r--.  jun jun  Dec    sqoop-1.4..jar
-rw-rw-r--.  jun jun     Dec    sqoop-patch-review.py
-rw-rw-r--.  jun jun   Dec    sqoop-test-1.4..jar
drwxr-xr-x.  jun jun       Dec    src
drwxr-xr-x.  jun jun      Dec    testdata

　　2.配置MySQL连接器

[jun@master sqoop-1.4..bin__hadoop-2.6.]$ cp /home/jun/Resources/mysql-connector-java-5.1./mysql-connector-java-5.1..jar   /home/jun/sqoop-1.4..bin__hadoop-2.6./lib/

　　3.配置Sqoop环境变量

　　编辑配置文件

[jun@master lib]$ cd /home/jun/sqoop-1.4..bin__hadoop-2.6./conf/
[jun@master conf]$ ls
oraoop-site-template.xml  sqoop-env-template.cmd  sqoop-env-template.sh  sqoop-site-template.xml  sqoop-site.xml
[jun@master conf]$ cp sqoop-env-template.sh sqoop-env.sh
[jun@master conf]$ gedit sqoop-env.sh

　　增加下面的配置

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/home/jun/hadoop
 
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/home/jun/hadoop
 
#set the path to where bin/hbase is available
export HBASE_HOME=/home/jun/hbase-1.2.6.1
 
#Set the path to where bin/hive is available
export HIVE_HOME=/home/jun/apache-hive-2.3.-bin
 
#Set the path for where zookeper config dir is
export ZOOCFGDIR=/usr/local/zk

　　4.配置linux环境变量

#sqoop
export SQOOP_HOME=/home/jun/sqoop-1.4..bin__hadoop-2.6.
export PATH=$PATH:$SQOOP_HOME/bin

　　5.启动Sqoop，如果出现下面的内容就说明安装成功

[jun@master ~]$ sqoop-help
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
usage: sqoop COMMAND [ARGS]
 
Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information
 
See 'sqoop help COMMAND' for information on a specific command.

　　6.测试与MySQL的连接

　　（1）列出MySQL的所有数据库

[jun@master ~]$ sqoop-list-databases --connect jdbc:mysql://localhost:3306 --username root -P
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Enter password:
// :: INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Mon Jul  :: CST  WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
information_schema
hive_db
mysql
performance_schema
sys

　　（2）列出数据库下的所有数据表

[jun@master ~]$ sqoop-list-tables --connect jdbc:mysql://localhost:3306/mysql --username root -P
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Enter password:
// :: INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Mon Jul  :: CST  WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
columns_priv
db
engine_cost
event
func
general_log
gtid_executed
help_category
help_keyword
help_relation
help_topic
innodb_index_stats
innodb_table_stats
ndb_binlog_index
plugin
proc
procs_priv
proxies_priv
server_cost
servers
slave_master_info
slave_relay_log_info
slave_worker_info
slow_log
tables_priv
time_zone
time_zone_leap_second
time_zone_name
time_zone_transition
time_zone_transition_type
user

　　（3）执行MySQL的查询语句

[jun@master ~]$ sqoop-eval --connect jdbc:mysql://localhost:3306/mysql --username root -P --query "select * from plugin"
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4..bin__hadoop-2.6./../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Enter password:
// :: INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Mon Jul  :: CST  WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
-----------------------------------------------
| name                 | dl                   |
-----------------------------------------------
| validate_password    | validate_password.so |
-----------------------------------------------