kerberos系列之hive认证配置

大数据安全系列之hive的kerberos认证配置，其它系列链接如下

https://www.cnblogs.com/bainianminguo/p/12548076.html-----------安装kerberos

https://www.cnblogs.com/bainianminguo/p/12548334.html-----------hadoop的kerberos认证

https://www.cnblogs.com/bainianminguo/p/12548175.html-----------zookeeper的kerberos认证

https://www.cnblogs.com/bainianminguo/p/12584732.html-----------hive的kerberos认证

https://www.cnblogs.com/bainianminguo/p/12584880.html-----------es的search-guard认证

https://www.cnblogs.com/bainianminguo/p/12639821.html-----------flink的kerberos认证

https://www.cnblogs.com/bainianminguo/p/12639887.html-----------spark的kerberos认证

一、安装mysql

1、卸载mariadb

[root@cluster2-host1 yum.repos.d]# rpm -qa|grep mariadb

mariadb-libs-5.5.44-2.el7.centos.x86_64

[root@cluster2-host1 yum.repos.d]# rpm -e --nodeps mariadb-libs-5.5.44-2.el7.centos.x86_64

[root@cluster2-host1 yum.repos.d]# rpm -qa|grep mariadb

2、创建mysql用户

[root@cluster2-host1 yum.repos.d]# groupadd mysql

[root@cluster2-host1 yum.repos.d]# useradd mysql -g mysql

3、下载mysql安装包

https://cdn.mysql.com//Downloads/MySQL-5.7/mysql-5.7.29-1.el7.x86_64.rpm-bundle.tar

4、解压安装包

[root@cluster2-host1 data]# tar -xvf mysql-5.7.29-1.el7.x86_64.rpm-bundle.tar -C /usr/local/mysql/

[root@cluster2-host1 mysql]# ll

total 533048

-rw-r--r--. 1 7155 31415  27768112 Dec 19 03:12 mysql-community-client-5.7.29-1.el7.x86_64.rpm

-rw-r--r--. 1 7155 31415    318972 Dec 19 03:12 mysql-community-common-5.7.29-1.el7.x86_64.rpm

-rw-r--r--. 1 7155 31415   4085448 Dec 19 03:12 mysql-community-devel-5.7.29-1.el7.x86_64.rpm

-rw-r--r--. 1 7155 31415  47521016 Dec 19 03:12 mysql-community-embedded-5.7.29-1.el7.x86_64.rpm

-rw-r--r--. 1 7155 31415  23354680 Dec 19 03:12 mysql-community-embedded-compat-5.7.29-1.el7.x86_64.rpm

-rw-r--r--. 1 7155 31415 131015588 Dec 19 03:12 mysql-community-embedded-devel-5.7.29-1.el7.x86_64.rpm

-rw-r--r--. 1 7155 31415   2596180 Dec 19 03:12 mysql-community-libs-5.7.29-1.el7.x86_64.rpm

-rw-r--r--. 1 7155 31415   1353080 Dec 19 03:12 mysql-community-libs-compat-5.7.29-1.el7.x86_64.rpm

-rw-r--r--. 1 7155 31415 183618644 Dec 19 03:12 mysql-community-server-5.7.29-1.el7.x86_64.rpm

-rw-r--r--. 1 7155 31415 124193252 Dec 19 03:12 mysql-community-test-5.7.29-1.el7.x86_64.rpm

5、rpm的方式安装mysql

[root@cluster2-host1 mysql]# rpm -ivh mysql-community-common-5.7.29-1.el7.x86_64.rpm

warning: mysql-community-common-5.7.29-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:mysql-community-common-5.7.29-1.e################################# [100%]

[root@cluster2-host1 mysql]# rpm -ivh mysql-community-libs-5.7.29-1.el7.x86_64.rpm

warning: mysql-community-libs-5.7.29-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:mysql-community-libs-5.7.29-1.el7################################# [100%]

[root@cluster2-host1 mysql]# rpm -ivh mysql-community-client-5.7.29-1.el7.x86_64.rpm

warning: mysql-community-client-5.7.29-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:mysql-community-client-5.7.29-1.e################################# [100%]

[root@cluster2-host1 mysql]# rpm -ivh mysql-community-server-5.7.29-1.el7.x86_64.rpm

warning: mysql-community-server-5.7.29-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

error: Failed dependencies:

	net-tools is needed by mysql-community-server-5.7.29-1.el7.x86_64

[root@cluster2-host1 mysql]# yum install net-tools -y

[root@cluster2-host1 mysql]# rpm -ivh mysql-community-server-5.7.29-1.el7.x86_64.rpm

warning: mysql-community-server-5.7.29-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:mysql-community-server-5.7.29-1.e################################# [100%]

6、启动mysql

[root@cluster2-host1 mysql]# service mysqld start

7、查看mysql的临时密码

[root@cluster2-host1 mysql]# grep "A temporary password" /var/log/mysqld.log

2020-03-02T07:59:38.098144Z 1 [Note] A temporary password is generated for root@localhost: ln/Ot4j-j#hQ

[root@cluster2-host1 mysql]#

8、修改mysql的临时密码

set global validate_password_policy=0;

set global validate_password.length=1;

alter user user() identified by "123456";

9、设置mysql支持远程访问

[root@cluster2-host1 conf]# mysql -u root -p

update user set host = '%' where user = 'root';

10、在其它节点确认可以远程访问mysql即可

mysql -h 10.87.18.34 -p3306 -uroot -p

二、安装hive

1、解压和重命名hive安装路径

 538  tar -zxvf apache-hive-1.2.0-bin.tar.gz -C /usr/local/

  539  cd /usr/local/

  540  ll

  541  mv apache-hive-1.2.0-bin/ hive

2、修改hive的env文件

[root@cluster2-host1 conf]# pwd

/usr/local/hive/conf

[root@cluster2-host1 conf]# cp hive-env.sh.template hive-env.sh

3、修改hive-env文件

export HIVE_HOME=/usr/local/hive

export HADOOP_HOME=/usr/local/hadoop

# Hive Configuration Directory can be controlled by:

export HIVE_CONF_DIR=/usr/local/hive/conf

export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

export PATH=${HIVE_HOME}/bin:$PATH:$HOME/bin:

4、修改hive的配置文件

vim hive-default.xml

    <property>

        <name>javax.jdo.option.ConnectionUserName</name>

        <value>root</value>

    </property>

    <property>

        <name>javax.jdo.option.ConnectionPassword</name>

        <value>123456</value>

    </property>

   <property>

        <name>javax.jdo.option.ConnectionURL</name>mysql

        <value>jdbc:mysql://10.87.18.34:3306/hive?</value>

    </property>

    <property>

        <name>javax.jdo.option.ConnectionDriverName</name>

        <value>com.mysql.jdbc.Driver</value>

    </property>

5、安装hive的mysql驱动

拷贝驱动到hive/lib目录

  474  cd mysql-connector-java-5.1.48

  475  ll

  476  cp mysql-connector-java-5.1.48-bin.jar /usr/local/hive/lib/

6、在mysql中创建hive的database

mysql> create database hive;

Query OK, 1 row affected (0.00 sec)

mysql> show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| hive               |

| mysql              |

| performance_schema |

| sys                |

+--------------------+

5 rows in set (0.00 sec)

7、拷贝mysql连接驱动到其他未按照hive的节点

[root@cluster2-host1 lib]# scp mysql-connector-java-5.1.48-bin.jar root@cluster2-host2:/usr/local/hive/lib/

mysql-connector-java-5.1.48-bin.jar                                                                                                                                                                                                         100%  983KB 983.4KB/s   00:00

[root@cluster2-host1 lib]# scp mysql-connector-java-5.1.48-bin.jar root@cluster2-host3:/usr/local/hive/lib/

mysql-connector-java-5.1.48-bin.jar

三、配置hive的kerberos配置

1、创建主体文件

kadmin.local:  addprinc hive/cluster2-host1

kadmin.local:  ktadd -norandkey -k /etc/security/keytab/hive.keytab hive/cluster2-host1

2、拷贝秘钥文件到hive的目录

 scp /etc/security/keytab/hive.keytab /usr/local/hive/conf/

3、修改hive的配置文件

 <property>

        <name>hive.server2.enable.doAs</name>

        <value>true</value>

  </property>

  <property>

        <name>hive.server2.authentication</name>

        <value>KERBEROS</value>

  </property>

  <property>

        <name>hive.server2.authentication.kerberos.principal</name>

        <value>hive/cluster2-host1@HADOOP.COM</value>

  </property>

  <property>

        <name>hive.server2.authentication.kerberos.keytab</name>

        <value>/usr/local/hive/conf/hive.keytab</value>

  </property>

  <property>

        <name>hive.server2.authentication.spnego.keytab</name>

        <value>/usr/local/hive/conf/hive.keytab</value>

  </property>

  <property>

        <name>hive.server2.authentication.spnego.principal</name>

        <value>hive/cluster2-host1@HADOOP.COM</value>

  </property>

  <property>

        <name>hive.metastore.sasl.enabled</name>

        <value>true</value>

  </property>

  <property>

        <name>hive.metastore.kerberos.keytab.file</name>

        <value>/usr/local/hive/conf/hive.keytab</value>

  </property>

  <property>

        <name>hive.metastore.kerberos.principal</name>

        <value>hive/cluster2-host1@HADOOP.COM</value>

  </property>

4、修改hadoop的core-site.xml配置文件

      <property>

        <name>hadoop.proxyuser.hive.users</name>

        <value>*</value>

      </property>

      <property>

        <name>hadoop.proxyuser.hive.hosts</name>

        <value>*</value>

      </property>

5、启动hive

[root@cluster2-host1 hive]# nohup ./bin/hive --service metastore > metastore.log 2>&1 &

[1] 5637

[root@cluster2-host1 hive]# nohup ./bin/hiveserver2 > hive.log 2>&1 &

[2] 7361

6、通过beeline的方式连接hive

[root@cluster2-host1 hive]#   ./bin/beeline -u "jdbc:hive2://cluster2-host1:10000/default;principal=hive/cluster2-host1@HADOOP.COM"

ls: cannot access /usr/local/spark/lib/spark-assembly-*.jar: No such file or directory

Connecting to jdbc:hive2://cluster2-host1:10000/default;principal=hive/cluster2-host1@HADOOP.COM

Connected to: Apache Hive (version 1.2.0)

Driver: Hive JDBC (version 1.2.0)

Transaction isolation: TRANSACTION_REPEATABLE_READ

Beeline version 1.2.0 by Apache Hive

0: jdbc:hive2://cluster2-host1:10000/default>

7、登陆进去创建hive表

create database myhive;

CREATE TABLE student(id int, name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' LINES TERMINATED BY '\n' STORED AS TEXTFILE;

查看创建的表

0: jdbc:hive2://cluster2-host1:10000/default> show tables;

+------------+--+

|  tab_name  |

+------------+--+

| student    |

| test1      |

| test2      |

| test3      |

| test4      |

| test_user  |

+------------+--+

8、检查mysql有hive的元数据信息

9、准备本地数据

2014001 小王1

2014002 小李2

2014003 小明3

2014004 阿狗4

2014005 姚明5

10、上传数据

load data local inpath '/data/hive.txt' into table test1;

11、查看hdfs上的数据

[root@cluster2-host1 data]# hdfs dfs -ls /user/hive/warehouse/test1

Found 1 items

-rwxr-xr-x   2 hdfs supergroup        112 2020-03-05 04:55 /user/hive/warehouse/test1/hive.txt