一、基础环境

Hive 组件需要基于之前已部署完毕的 Hadoop 全分布系统，在 master 节点上实现 Hive 组件安装。

1、Hadoop-全分布式配置（全部配置）
2、Hadoop-集群运行（部分配置）

# 查看之前部署的环境

[root@master ~]# su - hadoop

[hadoop@master ~]$ jps

47967 Jps

[hadoop@master hadoop]$ start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

Starting namenodes on [master]

master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.example.com.out

192.168.100.30: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.example.com.out

192.168.100.20: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.example.com.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.example.com.out

starting yarn daemons

starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.example.com.out

192.168.100.30: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.example.com.out

192.168.100.20: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.example.com.out

[hadoop@master hadoop]$ jps

44678 NameNode

45030 ResourceManager

45289 Jps

44874 SecondaryNameNode

[hadoop@master hadoop]$ exit

logout

要确保 Hadoop 系统能够正常运行

二、安装MySQL

注意：使用 root 用户登录

1、传输需要的包到/opt目录下

[root@master ~]# cd /opt/software/

[root@master software]# ls

apache-hive-2.0.0-bin.tar.gz  jdk-8u152-linux-x64.tar.gz  mysql-connector-java-5.1.46.jar

hadoop-2.7.1.tar.gz           mysql-5.7.18.zip

2、卸载自带的MariaDB

[root@master software]# rpm -qa | grep mariadb

mariadb-libs-5.5.52-1.el7.x86_64

[root@master software]# rpm -e --nodeps mariadb-libs-5.5.52-1.el7.x86_64

3、解压mysql-5.7.18.zip

[root@master software]# ls

apache-hive-2.0.0-bin.tar.gz  jdk-8u152-linux-x64.tar.gz  mysql-connector-java-5.1.46.jar

hadoop-2.7.1.tar.gz           mysql-5.7.18.zip

[root@master software]# unzip mysql-5.7.18.zip

Archive:  mysql-5.7.18.zip

   creating: mysql-5.7.18/

  inflating: mysql-5.7.18/mysql-community-client-5.7.18-1.el7.x86_64.rpm

  inflating: mysql-5.7.18/mysql-community-common-5.7.18-1.el7.x86_64.rpm

  inflating: mysql-5.7.18/mysql-community-devel-5.7.18-1.el7.x86_64.rpm

  inflating: mysql-5.7.18/mysql-community-libs-5.7.18-1.el7.x86_64.rpm

  inflating: mysql-5.7.18/mysql-community-server-5.7.18-1.el7.x86_64.rpm

4、依次安装MySQL数据库

[root@master software]# cd mysql-5.7.18/

[root@master mysql-5.7.18]# ls

mysql-community-client-5.7.18-1.el7.x86_64.rpm  mysql-community-libs-5.7.18-1.el7.x86_64.rpm

mysql-community-common-5.7.18-1.el7.x86_64.rpm  mysql-community-server-5.7.18-1.el7.x86_64.rpm

mysql-community-devel-5.7.18-1.el7.x86_64.rpm

[root@master mysql-5.7.18]# rpm -ivh mysql-community-common-5.7.18-1.el7.x86_64.rpm

warning: mysql-community-common-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:mysql-community-common-5.7.18-1.e################################# [100%]

[root@master mysql-5.7.18]# rpm -ivh mysql-community-libs-5.7.18-1.el7.x86_64.rpm

warning: mysql-community-libs-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:mysql-community-libs-5.7.18-1.el7################################# [100%]

[root@master mysql-5.7.18]# rpm -ivh mysql-community-client-5.7.18-1.el7.x86_64.rpm

warning: mysql-community-client-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:mysql-community-client-5.7.18-1.e################################# [100%]

[root@master mysql-5.7.18]# rpm -ivh mysql-community-server-5.7.18-1.el7.x86_64.rpm

warning: mysql-community-server-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:mysql-community-server-5.7.18-1.e################################# [100%]

5、修改配置

添加配置信息到/etc/my.cnf 文件 symbolic-links=0 的下方

[root@master mysql-5.7.18]# vi /etc/my.cnf

default-storage-engine=innodb

innodb_file_per_table

collation-server=utf8_general_ci

init-connect='SET NAMES utf8'

character-set-server=utf8

6、启动MySQL

[root@master mysql-5.7.18]# systemctl start mysqld

[root@master mysql-5.7.18]# systemctl status mysqld

● mysqld.service - MySQL Server

   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)

   Active: active (running) since Sat 2022-04-09 10:12:27 CST; 44s ago

     Docs: man:mysqld(8)

           http://dev.mysql.com/doc/refman/en/using-systemd.html

  Process: 45577 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)

  Process: 45499 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)

 Main PID: 45580 (mysqld)

   CGroup: /system.slice/mysqld.service

           └─45580 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

Apr 09 10:12:17 master.example.com systemd[1]: Starting MySQL Server...

Apr 09 10:12:27 master.example.com systemd[1]: Started MySQL Server.

7、查询MySQL默认密码

[root@master ~]# cat /var/log/mysqld.log | grep password

2022-04-09T02:12:24.671882Z 1 [Note] A temporary password is generated for root@localhost: /RrHz+J-t1fj

默认密码为/RrHz+J-t1fj

8、初始化MySQL

是否更改 root 用户密码n
是否使用设定的密码继续y
是否删除匿名用户y
是否拒绝 root 用户远程登录n
是否删除测试数据库y
是否重新加载授权表y

[root@master ~]# mysql_secure_installation

Securing the MySQL server deployment.

Enter password for user root: 

The existing password for the user account root has expired. Please set a new password.

New password: 

Re-enter new password:

The 'validate_password' plugin is installed on the server.

The subsequent steps will run with the existing configuration

of the plugin.

Using existing password for root.

Estimated strength of the password: 100

Change the password for root ? ((Press y|Y for Yes, any other key for No) : n

Estimated strength of the password: 100

Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No) : y

By default, a MySQL installation has an anonymous user,

allowing anyone to log into MySQL without having to have

a user account created for them. This is intended only for

testing, and to make the installation go a bit smoother.

You should remove them before moving into a production

environment.

Remove anonymous users? (Press y|Y for Yes, any other key for No) : n

 ... skipping.

Normally, root should only be allowed to connect from

'localhost'. This ensures that someone cannot guess at

the root password from the network.

Disallow root login remotely? (Press y|Y for Yes, any other key for No) : n

 ... skipping.

By default, MySQL comes with a database named 'test' that

anyone can access. This is also intended only for testing,

and should be removed before moving into a production

environment.

Remove test database and access to it? (Press y|Y for Yes, any other key for No) : n

 ... skipping.

Reloading the privilege tables will ensure that all changes

made so far will take effect immediately.

Reload privilege tables now? (Press y|Y for Yes, any other key for No) : n

 ... skipping.

All done!

9、进入数据库，添加权限

添加 root 用户从本地和远程访问 MySQL 数据库表单的授权

[root@master ~]# mysql -uroot -p

Enter password:

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 12

Server version: 5.7.18 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> grant all privileges on *.* to root@'localhost' identified by 'Wangzhigang123!';

Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> grant all privileges on *.* to root@'%' identified by 'Wangzhigang123!';

Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> flush privileges;

Query OK, 0 rows affected (0.00 sec)

mysql> select user,host from mysql.user where user='root';

+------+-----------+

| user | host      |

+------+-----------+

| root | %         |

| root | localhost |

+------+-----------+

2 rows in set (0.00 sec)

mysql> exit;

Bye

[root@master ~]#

二、配置Hive组件

1、设置Hive环境变量

# 解压hive文件

[root@master ~]# tar -zxf /opt/software/apache-hive-2.0.0-bin.tar.gz

[root@master ~]# mv apache-hive-2.0.0-bin/ /usr/local/src/hive

# 修改用户和用户组为 hadoop

[root@master ~]# chown -R hadoop:hadoop /usr/local/src/hive

# 添加配置信息

[root@master ~]# vi /etc/profile

# set hive environment

export HIVE_HOME=/usr/local/src/hive

export PATH=$PATH:$HIVE_HOME/bin

[root@master ~]# source /etc/profile

2、修改Hive组件配置文件

[root@master ~]# su - hadoop

[hadoop@master ~]$ cp /usr/local/src/hive/conf/hive-default.xml.template /usr/local/src/hive/conf/hive-site.xml

[hadoop@master ~]$ vi /usr/local/src/hive/conf/hive-site.xml

# 设置 MySQL 数据库连接

    <name>javax.jdo.option.ConnectionURL</name>

    <value>jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false</value>

    <description>JDBC connect string for a JDBC metastore</description>

# 配置 MySQL 数据库 root 的密码

    <name>javax.jdo.option.ConnectionPassword</name>

    <value>Wangzhigang123!</value>

    <description>password to use against metastore database</description>

# 验证元数据存储版本一致性,默认 false

    <name>hive.metastore.schema.verification</name>

    <value>false</value>

# 配置数据库驱动

    <name>javax.jdo.option.ConnectionDriverName</name>

    <value>com.mysql.jdbc.Driver</value>

    <description>Driver class name for a JDBC metastore</description>

# 配置数据库用户名

    <name>javax.jdo.option.ConnectionUserName</name>

    <value>root</value>

    <description>Username to use against metastore database</description>

# 配置目录信息（共4处）

    <name>hive.querylog.location</name>

    <value>/usr/local/src/hive/tmp</value>

    <description>Location of Hive run time structured log file</description>

    <name>hive.exec.local.scratchdir</name>

    <value>/usr/local/src/hive/tmp</value>

    <name>hive.downloaded.resources.dir</name>

    <value>/usr/local/src/hive/tmp/resources</value>

    <name>hive.server2.logging.operation.log.location</name>

    <value>/usr/local/src/hive/tmp/operation_logs</value>

[hadoop@master ~]$ mkdir -p /usr/local/src/hive/tmp{resources,operation_logs}

3、重启all.sh

[hadoop@master ~]$ cp /opt/software/mysql-connector-java-5.1.46.jar /usr/local/src/hive/lib/

[hadoop@master ~]$ stop-all.sh

This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh

Stopping namenodes on [master]

master: stopping namenode

192.168.100.20: stopping datanode

192.168.100.30: stopping datanode

Stopping secondary namenodes [0.0.0.0]

0.0.0.0: stopping secondarynamenode

stopping yarn daemons

stopping resourcemanager

192.168.100.20: stopping nodemanager

192.168.100.30: stopping nodemanager

no proxyserver to stop

[hadoop@master ~]$ start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

Starting namenodes on [master]

master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.example.com.out

192.168.100.20: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.example.com.out

192.168.100.30: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.example.com.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.example.com.out

starting yarn daemons

starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.example.com.out

192.168.100.30: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.example.com.out

192.168.100.20: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.example.com.out

4、初始化数据库

[hadoop@master ~]$ schematool -initSchema -dbType mysql

which: no hbase in (/usr/local/src/jdk/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/src/hive/bin:/home/hadoop/.local/bin:/home/hadoop/bin)

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Metastore connection URL:        jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&useSSL=false

Metastore Connection Driver :    com.mysql.jdbc.Driver

Metastore connection User:       root

Starting metastore schema initialization to 2.0.0

Initialization script hive-schema-2.0.0.mysql.sql

Initialization script completed

schemaTool completed

5、启动hive

[hadoop@master ~]$ hive

which: no hbase in (/usr/local/src/jdk/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/src/hive/bin:/home/hadoop/.local/bin:/home/hadoop/bin)

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/usr/local/src/hive/lib/hive-common-2.0.0.jar!/hive-log4j2.properties

Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

hive> show databases;

OK

default

Time taken: 0.241 seconds, Fetched: 1 row(s)

hive> exit;

[hadoop@master ~]$

6、验证

[hadoop@master ~]$ mysql -uroot -pWangzhigang123!

mysql: [Warning] Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 24

Server version: 5.7.18 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| hive               |

| mysql              |

| performance_schema |

| sys                |

+--------------------+

5 rows in set (0.00 sec)

mysql> exit;

Bye

[hadoop@master ~]$ jps

47057 NameNode

47411 ResourceManager

47254 SecondaryNameNode

47967 Jps

[hadoop@master ~]$

slave1节点

[root@slave1 ~]# su - hadoop

[hadoop@slave1 ~]$ jps

43621 Jps

43351 DataNode

43464 NodeManager

slave2节点

[root@slave2 ~]# su - hadoop

[hadoop@slave2 ~]$ jps

41607 Jps

41337 DataNode

41450 NodeManager

声明：未经许可，不得转载

Hadoop-Hive组件部署的更多相关文章

hadoop hive组件介绍及常用cli命令
Hive架构图 Hive产生原因 1 关系型数据库以产生多年sql成熟 2 简化开发降低成本 3 java成员可编写udf函数 Hive是什么 Hive是基于hadoop的一个数据库工具,使用Hql作 ...
Hadoop生态组件Hive，Sqoop安装及Sqoop从HDFS/hive抽取数据到关系型数据库Mysql
一般Hive依赖关系型数据库Mysql,故先安装Mysql $: yum install mysql-server mysql-client [yum安装] $: /etc/init.d/mysqld ...
Docker+Hadoop+Hive+Presto 使用Docker部署Hadoop环境和Presto
Background 一. 什么是Presto Presto通过使用分布式查询,可以快速高效的完成海量数据的查询.如果你需要处理TB或者PB级别的数据,那么你可能更希望借助于Hadoop和HDFS来完 ...
Hadoop.之.入门部署
一.课程目标 ->大数据是什么?大数据能做什么? ->什么是Hadoop?Hadoop的设计思想? ->Hadoop如何解决大数据的问题?(什么是hdfs与yarn.MapReduc ...
单机安装hadoop+hive+presto
系统环境在个人笔记本上使用virtualbox虚拟机 os:centos -7.x86-64.everything.1611 ,内核 3.10.0-514.el7.x86_64 注:同样可以使用r ...
Hive 组件安装配置
下载和解压安装文件基础环境和安装准备 Hive组件的部署规划和软件包路径如下: (1)当前环境中已安装 Hadoop全分布系统 (2)本地安装 MySQL数据库(账号 root,密码 Passwor ...
远程调试hadoop各组件
远程调试对应用程序开发十分有用.例如,为不能托管开发平台的低端机器开发程序,或在专用的机器上(比如服务不能中断的 Web 服务器)调试程序.其他情况包括:运行在内存小或 CUP 性能低的设备上的 Ja ...
Sqoop是一款开源的工具，主要用于在HADOOP(Hive)与传统的数据库(mysql、oracle...)间进行数据的传递
http://niuzhenxin.iteye.com/blog/1706203 Sqoop是一款开源的工具,主要用于在HADOOP(Hive)与传统的数据库(mysql.postgresql.. ...
Hadoop生态圈-Sqoop部署以及基本使用方法
Hadoop生态圈-Sqoop部署以及基本使用方法作者:尹正杰版权声明:原创作品,谢绝转载!否则将追究法律责任. Sqoop(发音:skup)是一款开源的工具,主要用于在Hadoop(Hive)与 ...

随机推荐

vue2版本中slot的基本使用详解
前言在vue的开发过程中,我们会经常使用到vue的slot插槽组件,vue官方文档的描述: Vue 实现了一套内容分发的 API,这套 API 的设计灵感源自 Web Components 规范草案 ...
K8S搭建自动化部署环境 Jenkins下载、安装和启动
镜像下载.域名解析.时间同步请点击阿里云开源镜像站一.jenkins 下载 jenkins下载地址:https://jenkins.io/zh/download/ 如下图,左边是稳定版本,右边是每 ...
4月18日 python学习总结异常处理、网络编程
一. 异常 1.什么是异常异常是错误发生的信号,程序一旦出错,如果程序中还没有相应的处理机制那么该错误就会产生一个异常抛出来,程序的运行也随之终止 2.一个异常分为三部分: 1.异常的追踪信息 2 ...
python 发送GET请求
# #博客地址:https://blog.csdn.net/qq_36374896 # 特点:把数据拼接到请求路径的后面传递给服务器 # # 优点:速度快 # # 缺点:承载的数据量小,不安全 imp ...
浅谈systemd原理和应用
多不说,直接上代码(可谓配置): [Unit] Description=demo app After=network-is-online.target [Service] Type=Simple Ex ...
怎样查看一个 linux 命令的概要与用法？假设你在/bin 目录中偶然看到一个你从没见过的的命令，怎样才能知道它的作用和用法呢？
使用命令 whatis 可以先出显示出这个命令的用法简要,比如,你可以使用 whatiszcat 去查看'zcat'的介绍以及使用简要. [root@localhost ~]# whatis zcat ...
说说&和&&的区别？
&和&&都可以用作逻辑与的运算符,表示逻辑与(and),当运算符两边的表达式的结果都为true时,整个运算结果才为true,否则,只要有一方为false,则结果为false.
你将如何使用 thread dump？你将如何分析 Thread dump？
新建状态(New) 用 new 语句创建的线程处于新建状态,此时它和其他 Java 对象一样,仅仅在堆区中被分配了内存. 就绪状态(Runnable) 当一个线程对象创建后,其他线程调用它的 sta ...
js技术之截取substring(int,int)
案例:把所有单词以空格为分割并将首字母转为大写 <!DOCTYPE html><html lang="en"><head> <meta c ...
C语言常用字符串函数
string.h头文件中常用的函数 C 库函数 - strcat() char *strcat(char *dest, const char *src) 把 src 所指向的字符串追加到 dest 所 ...

Hadoop-Hive组件部署