大数据学习（11）—— Hive元数据服务模式搭建
这一篇介绍Hive的安装及操作。版本是Hive3.1.2。
调整部署节点
在Hadoop篇里，我用了5台虚拟机来搭建集群，但是我的电脑只有8G内存，虚拟机启动之后卡到没法操作，把自己坑惨了。
Hive的运行是基于Hadoop集群的，为了操作的流畅性，我把Hadoop集群全部重新搭建了，一共只有3台虚拟机。
| 主机 | NN | RM | ZKFC | DN | NM | JN | ZK | MySQL | Hive服务端 | Hive客户端 |
| server01 | • | • | • | • | • | • | • | |||
| server02 | • | • | • | • | • | • | • | • | ||
| server03 | • | • | • | • | • | • |
安装mysql
一般我们使用关系型数据库来存储Hive的元数据，mysql用的比较多。
先检查一下是否有mysql的yum源，注意我们要安装的是mysql-server。下图中没有这个源，需要下载。
[root@server03 ~]# yum list|grep mysql
akonadi-mysql.x86_64 1.9.2-4.el7 base
apr-util-mysql.x86_64 1.5.2-6.el7 base
dovecot-mysql.x86_64 1:2.2.36-6.el7_8.1 updates
freeradius-mysql.x86_64 3.0.13-10.el7_6 base
libdbi-dbd-mysql.x86_64 0.8.3-16.el7 base
mysql-connector-java.noarch 1:5.1.25-3.el7 base
mysql-connector-odbc.x86_64 5.2.5-8.el7 base
pcp-pmda-mysql.x86_64 4.3.2-7.el7_8 updates
php-mysql.x86_64 5.4.16-48.el7 base
php-mysqlnd.x86_64 5.4.16-48.el7 base
qt-mysql.i686 1:4.8.7-8.el7 base
qt-mysql.x86_64 1:4.8.7-8.el7 base
qt5-qtbase-mysql.i686 5.9.7-2.el7 base
qt5-qtbase-mysql.x86_64 5.9.7-2.el7 base
redland-mysql.x86_64 1.0.16-6.el7 base
rsyslog-mysql.x86_64 8.24.0-52.el7_8.2 updates
下载mysql的repo源
[root@server03 ~]# wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
--2020-10-12 22:36:02-- http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
正在解析主机 repo.mysql.com (repo.mysql.com)... 23.45.57.22
正在连接 repo.mysql.com (repo.mysql.com)|23.45.57.22|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：6140 (6.0K) [application/x-redhat-package-manager]
正在保存至: “mysql-community-release-el7-5.noarch.rpm” 100%[======================================================================================>] 6,140 --.-K/s 用时 0.002s 2020-10-12 22:36:02 (3.80 MB/s) - 已保存 “mysql-community-release-el7-5.noarch.rpm” [6140/6140])
安装mysql-community-release-el7-5.noarch.rpm包
[root@server03 home]# rpm -ivh mysql-community-release-el7-5.noarch.rpm
准备中... ################################# [100%]
正在升级/安装...
1:mysql-community-release-el7-5 ################################# [100%]
安装mysql
[root@server03 home]# yum -y install mysql-server
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.ustc.edu.cn
* extras: mirrors.cn99.com
* updates: mirrors.njupt.edu.cn
正在解决依赖关系
--> 正在检查事务
---> 软件包 mysql-community-server.x86_64.0.5.6.49-2.el7 将被 安装
--> 正在处理依赖关系 mysql-community-common(x86-64) = 5.6.49-2.el7，它被软件包 mysql-community-server-5.6.49-2.el7.x86_64 需要
...........................................................
perl-Scalar-List-Utils.x86_64 0:1.27-248.el7 perl-Socket.x86_64 0:2.010-5.el7
perl-Storable.x86_64 0:2.45-3.el7 perl-Text-ParseWords.noarch 0:3.29-4.el7
perl-Time-HiRes.x86_64 4:1.9725-3.el7 perl-Time-Local.noarch 0:1.2300-2.el7
perl-constant.noarch 0:1.27-2.el7 perl-libs.x86_64 4:5.16.3-295.el7
perl-macros.x86_64 4:5.16.3-295.el7 perl-parent.noarch 1:0.225-244.el7
perl-podlators.noarch 0:2.5.1-3.el7 perl-threads.x86_64 0:1.87-4.el7
perl-threads-shared.x86_64 0:1.43-6.el7 替代:
mariadb-libs.x86_64 1:5.5.64-1.el7 完毕！
启动mysql
[root@server03 home]# service mysqld start
Redirecting to /bin/systemctl start mysqld.service
需要更改权限才能实现远程连接MYSQL数据库。可以通过以下方式来修改权限。
[root@server03 ~]# mysql -uroot
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 7
Server version: 5.6.49 MySQL Community Server (GPL) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A Database changed
mysql> select host,user,password from user;
+-----------+------+-------------------------------------------+
| host | user | password |
+-----------+------+-------------------------------------------+
| localhost | root | |
| server03 | root | |
| 127.0.0.1 | root | |
| ::1 | root | |
| localhost | | |
| server03 | | |
+-----------+------+-------------------------------------------+
6 rows in set (0.00 sec) mysql> grant all privileges on *.* to 'root'@'%' identified by '111111' with grant option;
Query OK, 0 rows affected (0.00 sec) mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec) mysql> select host,user,password from user;
+-----------+------+-------------------------------------------+
| host | user | password |
+-----------+------+-------------------------------------------+
| localhost | root | |
| server03 | root | |
| 127.0.0.1 | root | |
| ::1 | root | |
| localhost | | |
| server03 | | |
| % | root | *A46E551C358E21DEFC306B31BB84ADBFC2A75AAB |
+-----------+------+-------------------------------------------+
7 rows in set (0.00 sec)
至此，mysql配置完毕。
安装Hive服务端
这里采用最常用的远程元数据模式来安装Hive。server03上安装Hive服务端，server02上安装Hive客户端。
在server03上解压hive压缩包到/usr目录下，并将用户和组修改为hadoop。将conf目录下的hive-default.xml.template复制一份，命名为hive-site.xml。注意，这个名字必须是hive-site.xml，跟hdfs类似。
[hadoop@server03 conf]$ pwd
/usr/apache-hive-3.1.2/conf
[hadoop@server03 conf]$ cp hive-default.xml.template hive-site.xml
修改hive-site.xml文件中的配置如下
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/opt/hive_remote/warehouse</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://server03:3306/hive_remote?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>111111</value>
</property>
</configuration>
第一个配置项是hdfs里的路径，我们不需要创建这个目录，元数据初始化的时候会自动在hdfs创建/opt/hive_remote/warehouse
配置hive环境变量
JAVA_HOME=/usr/java/jdk-11.0.8/
ZOOKEEPER_HOME=/usr/apache-zookeeper-3.5.8/
HADOOP_HOME=/usr/hadoop-3.3.0
HIVE_HOME=/usr/apache-hive-3.1.2
PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin
在网上下载mysql驱动包mysql-connector-java-bin.jar，放在hive的lib目录下。在server03上执行元数据初始化操作。
[hadoop@server03 ~]$ schematool -dbType mysql -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/apache-hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1380)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1361)
at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:536)
at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:554)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:448)
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141)
at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5104)
at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
这里报错了。原因是hive里的guava.jar与hadoop里的guava.jar版本不一致。需要把hive里的低版本删除，再把hadoop里的高版本拷贝过来。
[hadoop@server03 ~]$ ll /usr/hadoop-3.3.0/share/hadoop/common/lib |grep guava
-rw-r--r--. 1 hadoop hadoop 2747878 7月 7 02:47 guava-27.0-jre.jar
-rw-r--r--. 1 hadoop hadoop 2199 7月 7 02:47 listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
[hadoop@server03 ~]$ ll /usr/apache-hive-3.1.2/lib/ |grep guava
-rw-r--r--. 1 hadoop hadoop 2308517 9月 27 2018 guava-19.0.jar
-rw-r--r--. 1 hadoop hadoop 971309 5月 21 2019 jersey-guava-2.25.1.jar
[hadoop@server03 ~]$ rm -f /usr/apache-hive-3.1.2/lib/guava-19.0.jar
[hadoop@server03 ~]$ cp /usr/hadoop-3.3.0/share/hadoop/common/lib/guava-27.0-jre.jar /usr/apache-hive-3.1.2/lib/
[hadoop@server03 ~]$ ll /usr/apache-hive-3.1.2/lib/ |grep guava
-rw-r--r--. 1 hadoop hadoop 2747878 10月 19 00:15 guava-27.0-jre.jar
-rw-r--r--. 1 hadoop hadoop 971309 5月 21 2019 jersey-guava-2.25.1.jar
再次执行初始化脚本
[hadoop@server03 ~]$ schematool -dbType mysql -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/apache-hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:mysql://server03:3306/hive_remote?createDatabaseIfNotExist=true
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: root
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.mysql.sql Initialization script completed
schemaTool completed
Hive服务端就搭建成功了，下面启动服务。
[hadoop@server03 ~]$ hive --service metastore
2020-10-21 08:14:44: Starting Hive Metastore Server
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/apache-hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
这是一个阻塞式窗口，再开一个窗口看看进程。为了看得更清楚，先不启动Hadoop集群。
[hadoop@server03 ~]$ jps
3910 Jps
3656 RunJar
RunJar就是Hive的服务端进程。
安装Hive客户端
在server02上安装Hive客户端。客户端做的事情与服务端差不多，只是不需要执行元数据初始化的动作。修改hive-site.xml配置如下：
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive_remote/warehouse</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://node03:9083</value>
</property>
执行hive，进入到hive的cli窗口
[hadoop@server02 ~]$ hive
which: no hbase in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/java/jdk-11.0.8/bin:/usr/apache-zookeeper-3.5.8/bin:/usr/hadoop-3.3.0/bin:/usr/hadoop-3.3.0/sbin:/usr/apache-hive-3.1.2/bin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/apache-hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 52f8c152-cd38-4293-b2b6-3efc17546299
Exception in thread "main" java.lang.ClassCastException: class jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and java.net.URLClassLoader are in module java.base of loader 'bootstrap')
at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:413)
at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:389)
at org.apache.hadoop.hive.cli.CliSessionState.<init>(CliSessionState.java:60)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
执行报错了。到网上查这个错误信息，得出Hive3.1.2只支持jdk8。之前搭建Hadoop集群用的jdk11，哦豁，这下悲剧了。
没办法，只能再装一个jdk8了。
切换Java版本
[root@server01 home]# rpm -ivh jdk-8u271-linux-x64.rpm
警告：jdk-8u271-linux-x64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID ec551f03: NOKEY
准备中... ################################# [100%]
正在升级/安装...
1:jdk1.8-2000:1.8.0_271-fcs ################################# [100%]
Unpacking JAR files...
tools.jar...
plugin.jar...
javaws.jar...
deploy.jar...
rt.jar...
jsse.jar...
charsets.jar...
localedata.jar...
安装完毕后，看看java目录，现在有两个版本的java
[root@server01 java]# cd /usr/java
[root@server01 java]# ll
总用量 0
lrwxrwxrwx. 1 root root 16 9月 23 18:21 default -> /usr/java/latest
drwxr-xr-x. 9 root root 128 9月 23 18:21 jdk-11.0.8
drwxr-xr-x. 9 root root 286 10月 23 13:04 jdk1.8.0_271-amd64
lrwxrwxrwx. 1 root root 20 9月 23 18:21 latest -> /usr/java/jdk-11.0.8
先把目录名字修改一下，太长了不方便
[root@server01 java]# mv jdk1.8.0_271-amd64/ jdk1.8.0
[root@server01 java]# ll
总用量 0
lrwxrwxrwx. 1 root root 16 9月 23 18:21 default -> /usr/java/latest
drwxr-xr-x. 9 root root 128 9月 23 18:21 jdk-11.0.8
drwxr-xr-x. 9 root root 286 10月 23 13:04 jdk1.8.0
lrwxrwxrwx. 1 root root 20 9月 23 18:21 latest -> /usr/java/jdk-11.0.8
接着修改环境变量/etc/profile和hadoop-env.sh里的配置，执行source /etc/profile让配置生效
#JAVA_HOME=/usr/java/jdk-11.0.8
JAVA_HOME=/usr/java/jdk1.8.0
再切换java版本
[root@server01 java]# alternatives --install /usr/bin/java java /usr/java/jdk1.8.0/bin/java 1400
[root@server01 java]# alternatives --config java 共有 3 个提供“java”的程序。 选项 命令
-----------------------------------------------
*+ 1 /usr/java/jdk-11.0.8/bin/java
2 /usr/java/jdk1.8.0_271-amd64/bin/java
3 /usr/java/jdk1.8.0/bin/java 按 Enter 保留当前选项[+]，或者键入选项编号：3
第2个版本是安装rpm包的时候自动生成的，我们的路径修改之后应该把它删掉
[root@server01 java]# alternatives --remove java /usr/java/jdk1.8.0_271-amd64/bin/java
[root@server01 java]# alternatives --config java 共有 2 个提供“java”的程序。 选项 命令
-----------------------------------------------
* 1 /usr/java/jdk-11.0.8/bin/java
+ 2 /usr/java/jdk1.8.0/bin/java 按 Enter 保留当前选项[+]，或者键入选项编号：2
这个时候就只有2条记录了。我们来检查一下版本切换生效没有。
[root@server01 java]# java -version
java version "1.8.0_271"
Java(TM) SE Runtime Environment (build 1.8.0_271-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode)
[root@server01 java]# which java
/usr/bin/java
[root@server01 java]# ll /usr/bin/java
lrwxrwxrwx. 1 root root 22 10月 23 13:11 /usr/bin/java -> /etc/alternatives/java
[root@server01 java]# ll /etc/alternatives/java
lrwxrwxrwx. 1 root root 27 10月 23 13:11 /etc/alternatives/java -> /usr/java/jdk1.8.0/bin/java
完美完成Java版本切换。接下来我们就能继续Hive环境搭建了。
启动服务
启动zookeeper集群和hadoop集群就不用说了，轻车熟路。
在server03上启动hive元数据服务，再次提示，这是个阻塞式窗口，不用等它退出到命令行界面
[hadoop@server03 bin]$ hive --service metastore
2020-10-23 15:48:01: Starting Hive Metastore Server
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/apache-hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
在server02上启动hive客户端，出现hive提示符就可以尽情玩耍了。
[hadoop@server02 ~]$ hive
which: no hbase in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/java/jdk1.8.0/bin:/usr/apache-zookeeper-3.5.8/bin:/usr/hadoop-3.3.0/bin:/usr/hadoop-3.3.0/sbin:/usr/apache-hive-3.1.2/bin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/apache-hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = d252045d-6dc8-41ee-9808-c324521a880f Logging initialized using configuration in jar:file:/usr/apache-hive-3.1.2/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Hive Session ID = c39b1a09-a43a-4ccb-bcf9-9608969cbd12
hive>
基本操作实战
数据库操作，简直跟mysql一模一样
hive> show databases;
OK
default
Time taken: 4.862 seconds, Fetched: 1 row(s)
hive> create database test;
OK
Time taken: 4.09 seconds
hive> show databases;
OK
default
test
Time taken: 0.099 seconds, Fetched: 2 row(s)
hive> use test;
OK
Time taken: 0.138 seconds
hive>
这里我们再试验一下Hadoop里的例子，数据稍微做一下调整，存放在/opt/hive_remote/warehouse/data1中。
气温文件，年-月-日-温度
1949-10-01-34
1949-10-01-38
1949-10-02-36
1950-01-01-32
1950-10-01-37
1951-12-01-23
1950-10-02-41
1950-10-03-27
1951-07-01-45
1951-07-02-46
1951-07-03-47
在hive里创建一张表来保存数据。
hive> create table hive1(
> year string,
> month string,
> day string,
> temperature smallint
> )
> row format delimited
> fields terminated by '-';
OK
Time taken: 3.818 seconds
把数据导入表里
hive> load data local inpath '/opt/hive_remote/warehouse/data1'into table hive1;
Loading data to table test.hive1
OK
Time taken: 15.852 seconds
hive> select * from hive1;
OK
1949 10 01 34
1949 10 01 38
1949 10 02 36
1950 01 01 32
1950 10 01 37
1951 12 01 23
1950 10 02 41
1950 10 03 27
1951 07 01 45
1951 07 02 46
1951 07 03 47
Time taken: 12.108 seconds, Fetched: 11 row(s)
好了，数据准备完毕。还是那个需求，从中找出每个月温度最高的两天所对应的气温。先建一张中间表，把一天内有多条记录的去掉，只保留最高温。
hive> create table hive1_mid(
> year string,
> month string,
> day string,
> temperature smallint
> );
OK
Time taken: 1.627 seconds
hive> insert into hive1_mid select year,month,day,max(temperature) from hive1 group by year,month,day;
Query ID = hadoop_20201101093442_c624ad0c-4b55-406d-b49e-e8aa0315e7a2
Total jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1604193604905_0002, Tracking URL = http://server01:8088/proxy/application_1604193604905_0002/
Kill Command = /usr/hadoop-3.3.0/bin/mapred job -kill job_1604193604905_0002
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2020-11-01 09:35:35,506 Stage-1 map = 0%, reduce = 0%
2020-11-01 09:36:41,001 Stage-1 map = 0%, reduce = 0%
2020-11-01 09:36:50,906 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.59 sec
2020-11-01 09:37:52,195 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.59 sec
2020-11-01 09:38:00,629 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 4.87 sec
2020-11-01 09:38:05,778 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 5.72 sec
MapReduce Total cumulative CPU time: 5 seconds 720 msec
Ended Job = job_1604193604905_0002
Loading data to table test.hive1_mid
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1604193604905_0003, Tracking URL = http://server01:8088/proxy/application_1604193604905_0003/
Kill Command = /usr/hadoop-3.3.0/bin/mapred job -kill job_1604193604905_0003
Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 1
2020-11-01 09:39:00,431 Stage-3 map = 0%, reduce = 0%
2020-11-01 09:40:08,674 Stage-3 map = 0%, reduce = 0%
2020-11-01 09:40:17,031 Stage-3 map = 100%, reduce = 0%, Cumulative CPU 1.67 sec
2020-11-01 09:40:52,565 Stage-3 map = 100%, reduce = 100%, Cumulative CPU 4.37 sec
MapReduce Total cumulative CPU time: 4 seconds 370 msec
Ended Job = job_1604193604905_0003
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 5.72 sec HDFS Read: 16922 HDFS Write: 502 SUCCESS
Stage-Stage-3: Map: 1 Reduce: 1 Cumulative CPU: 4.37 sec HDFS Read: 12824 HDFS Write: 327 SUCCESS
Total MapReduce CPU Time Spent: 10 seconds 90 msec
OK
Time taken: 400.594 seconds
hive> select * from hive1_mid;
OK
1949 10 01 38
1949 10 02 36
1950 01 01 32
1950 10 01 37
1950 10 02 41
1950 10 03 27
1951 07 01 45
1951 07 02 46
1951 07 03 47
1951 12 01 23
Time taken: 7.142 seconds, Fetched: 10 row(s)
接下来的任务就简单了，从中间表中按照年月分组，获取每组记录里温度最高的两条。
hive> select a.year,a.month,a.day,a.temperature from hive1_mid a
> left join hive1_mid b
> on a.year=b.year and a.month=b.month and a.temperature < b.temperature
> group by a.year,a.month,a.day,a.temperature
> having count(*) < 2;
Query ID = hadoop_20201102233727_a7798ec5-55f2-4f81-a081-1473cf34ebca
Total jobs = 1
Execution completed successfully
MapredLocal task succeeded
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1604330185721_0003, Tracking URL = http://server01:8088/proxy/application_1604330185721_0003/
Kill Command = /usr/hadoop-3.3.0/bin/mapred job -kill job_1604330185721_0003
Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1
2020-11-02 23:39:00,286 Stage-2 map = 0%, reduce = 0%
2020-11-02 23:39:50,116 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 2.54 sec
2020-11-02 23:40:24,842 Stage-2 map = 100%, reduce = 67%, Cumulative CPU 2.54 sec
2020-11-02 23:40:40,775 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 6.31 sec
MapReduce Total cumulative CPU time: 6 seconds 310 msec
Ended Job = job_1604330185721_0003
MapReduce Jobs Launched:
Stage-Stage-2: Map: 1 Reduce: 1 Cumulative CPU: 6.31 sec HDFS Read: 14363 HDFS Write: 295 SUCCESS
Total MapReduce CPU Time Spent: 6 seconds 310 msec
OK
1949 10 01 38
1949 10 02 36
1950 01 01 32
1950 10 01 37
1950 10 02 41
1951 07 02 46
1951 07 03 47
1951 12 01 23
Time taken: 204.409 seconds, Fetched: 8 row(s)
上面的操作步骤跟hadoop高可用环境搭建里的map/reduce代码比起来是不是简洁多了？hive帮我们把sql语句转换成了对应的map/reduce任务，而不需要我们操心，对使用者来讲体验好多了。
大数据学习（11）—— Hive元数据服务模式搭建的更多相关文章
- 大数据学习（16）—— HBase环境搭建和基本操作
部署规划 HBase全称叫Hadoop Database,它的数据存储在HDFS上.我们的实验环境依然基于上个主题Hive的配置,参考大数据学习(11)-- Hive元数据服务模式搭建. 在此基础上, ...
- 大数据学习系列之六 ----- Hadoop+Spark环境搭建
引言 在上一篇中 大数据学习系列之五 ----- Hive整合HBase图文详解 : http://www.panchengming.com/2017/12/18/pancm62/ 中使用Hive整合 ...
- 大数据学习路线之linux系统基础搭建
学习大数据是必须掌握一定Linux知识的,工欲善其事,必先利其器.在学习之前,首先需要搭建Linux系统,本节将讲解VMware Workstation的安装和CentOS 7系统的安装. 1.2.1 ...
- 大数据学习（19）—— Flume环境搭建
系统要求 Java1.8或以上 内存要足够大 硬盘足够大 Agent对源和目的要有读写权限 Flume部署 我这8G内存的电脑之前搭建Hadoop.Hive和HBase已经苟延残喘了,怀疑会卡死,硬着 ...
- 大数据学习（25）—— 用IDEA搭建Spark开发环境
IDEA是一个优秀的Java IDE工具,它同样支持其他语言.Spark是用Scala语言编写的,用Scala开发Spark是最舒畅的.当然,Spark也提供Java和Python的API. Java ...
- 大数据学习——关于hive中的各种join
准备数据 2,b 3,c 4,d 7,y 8,u 2,bb 3,cc 7,yy 9,pp 建表: create table a(id int,name string) row format delim ...
- 大数据学习笔记——Hive完整部署流程
Hive详细部署教程 此篇博客承接上篇Hadoop和Zookeeper的部署教程,将会详细地对HIve的部署做一个整理,Hive相当于是封装在HDFS和Mapreduce上的一套sql引擎,只需要安装 ...
- 大数据学习系列之二 ----- HBase环境搭建(单机)
引言 在上一篇中搭建了Hadoop的单机环境,这一篇则搭建HBase的单机环境 环境准备 1,服务器选择 阿里云服务器:入门型(按量付费) 操作系统:linux CentOS 6.8 Cpu:1核 内 ...
- [大数据学习研究] 3. hadoop分布式环境搭建
1. Java安装与环境配置 Hadoop是基于Java的,所以首先需要安装配置好java环境.从官网下载JDK,我用的是1.8版本. 在Mac下可以在终端下使用scp命令远程拷贝到虚拟机linux中 ...
随机推荐
- url参数接收的一些安全应用场景
越权漏洞,从原来的修改id越权到后面的自己加参数,减参数越权,到现在的加特殊字符.攻击手段在进步: 以php和java为例,聊聊参数接收的最大接受能力,可以插入哪些脏数据? demo1.php: &l ...
- Java中对象调用方法的顺序
Java虚拟机会预先为加载到内存中的每个类维护一个方法表(Method Table),其中列出了所有类中所有方法的签名. 现在有2个类A和B,其中,B是A的子类,和一个B类型的对象x,当调用x.f(a ...
- Redis之缓存设计
缓存能够有效地加速应用的读写速度,同时也可以降低后端负载,对日常应用的开发至关重要.但是将缓存加入应用架构后也会带来一些问题,本章将针对这些问题介绍缓存使用技巧和设计方案,包含如下内容: □ 缓存的收 ...
- zbxtable的使用
实验环境: zabbix server 172.16.1.121 mysql 172.16.1.121 访问端 172.16.1.122 54.1 zbxtable 1 说明 ZbxTable使用Go ...
- API安全综述
API安全综述 译自:An Overview on API Security. 本文概括了API防护有关的方方面面,从上层视角介绍了API防护中主要注意的点,并给出了相应的建议.本文可以作为一个API ...
- 浅读tomcat架构设计之Pipeline-Valve管道(4)
tomcat Container容器处理请求是使用Pipeline-Valve管道来处理的,后续写的tomcat内存马,和他紧密结合 Pipeline-Valve是责任链模式,责任链模式是指在一个请求 ...
- Object 中的默认方法
1.public final native Class<?> getClass() getClass方法,返回该实例的java.lang.Class类,例如 Object obj = ne ...
- Spring：Spring的各jar包依赖及作用详解
spring-core.jar(必须有的核心jar包) 这个jar 文件包含Spring 框架基本的核心工具类.Spring 其它组件要都要使用到这个包里的类,是其它组件的基本核心,当然你也可以在自己 ...
- 苦B程序员的数据验证之路
发生了什么事 在一次苦B程序员和苦C程序员的结对编程中发生的一段对话 代码是这样的: public void deleteAllExtendAclsFromContent(String content ...
- 单选按钮(radio)的取值和点击事件
笔记走一波:获取单选按钮(radio)的选中值,以及它的点击事件的实现 首先要引入Jquery <script type="text/javascript" src=&quo ...