【Hive学习之一】Hive简介

环境
　　虚拟机：VMware 10
　　Linux版本：CentOS-6.5-x86_64
　　客户端：Xshell4
　　FTP：Xftp4
　　jdk8
　　hadoop-3.1.1
　　apache-hive-3.1.1

一、简介

Hive是基于Hadoop的一个数据仓库工具，可以将结构化的数据文件映射成一张表，并提供类sql语句的查询功能；
Hive使用Hql作为查询接口，使用HDFS存储，使用mapreduce计算；Hive的本质是将Hql转化为mapreduce；让非java编程者对hdfs的数据做mapreduce操作

Hive: 数据仓库。与关系型数据库区别：
①数据库可以用在Online的应用中，Hive主要进行离线的大数据分析；

②数据库的查询语句为SQL，Hive的查询语句为HQL；

③数据库数据存储在LocalFS，Hive的数据存储在HDFS；

④Hive执行MapReduce，MySQL执行Executor；

⑤Hive没有索引；

⑥Hive延迟性高；

⑦Hive可扩展性高；

⑧Hive数据规模大；

二、架构
Hive的架构

（1）用户接口主要有三个：Hive命令行模式（CLI），最常用模式；Hive Client（如JavaApi方式），对外提供RPC服务 ； Hive Web Interface（HWI）：浏览器方式。 
（2）Hive运行时将元数据存储在数据库中，如mysql、derby。Hive中的元数据包括表的名字，表的列和分区及其属性，表的属性（是否为外部表等），表的数据所在目录等。 
（3）解释器、编译器、优化器完成HQL查询语句从词法分析、语法分析、编译、优化以及查询计划的生成。生成的查询计划存储在HDFS中，并在随后有MapReduce调用执行。  
（4）Hive的数据存储在HDFS中，大部分的查询、计算由MapReduce完成（包含*的查询，比如select * from tbl不会生成MapRedcue任务）。

Operator操作符

编译器将一个Hive SQL转换操作符；
操作符是Hive的最小的处理单元；
每个操作符代表HDFS的一个操作或者一道MapReduce作业；

ANTLR词法语法分析工具解析hql

三、搭建

划分的维度：对关系型数据库的访问和管理来划分的
1、local模式：此模式连接到一个In-memory的数据库Derby，一般用于Unit Test。因为在内存不易维护，建议不用。

2、单用户模式：通过网络连接到一个数据库中，是最经常使用到的模式；

步骤一：安装mysql
参考：搭建Linux-java web运行环境之二：安装mysql

步骤二：解压apache-hive-3.1.1-bin.tar.gz 并设置环境变量
[root@PCS102 src]# tar -xf apache-hive-3.1.1-bin.tar.gz -C /usr/local

步骤三：修改配置

[root@PCS102 conf]# cd /usr/local/apache-hive-3.1.-bin/conf && ll
total
-rw-r--r--.  root root  Apr   beeline-log4j2.properties.template
-rw-r--r--.  root root  Oct  : hive-default.xml.template
-rw-r--r--.  root root  Apr   hive-env.sh.template
-rw-r--r--.  root root  Apr   hive-exec-log4j2.properties.template
-rw-r--r--.  root root  Oct  : hive-log4j2.properties.template
-rw-r--r--.  root root  Apr   ivysettings.xml
-rw-r--r--.  root root  Oct  : llap-cli-log4j2.properties.template
-rw-r--r--.  root root  Oct  : llap-daemon-log4j2.properties.template
-rw-r--r--.  root root  Apr   parquet-logging.properties

#复制配置文件

[root@PCS102 conf]# cp hive-default.xml.template hive-site.xml

#修改配置hive-site.xml
<!--hive数据上传到HDFS中的目录-->
<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>/root/hive_remote/warehouse</value>
</property>
<!--hive是否本地模式-->
<property>
  <name>hive.metastore.local</name>
  <value>false</value>
</property>
<!--hive连接mysql地址-->
<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://PCS101/hive_remote?createDatabaseIfNotExist=true</value>
</property>
<!--hive连接mysql驱动类-->
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
</property>
<!--hive连接mysql用户名-->
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>root</value>
</property>
<!--hive连接mysql 密码-->
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value></value>
</property>
 
:.,$-1d  删除当前行到倒数第二行

步骤四：拷贝mysql驱动包mysql-connector-java-5.1.32-bin
/usr/local/apache-hive-3.1.1-bin/lib

步骤五：初始化数据库  第一次启动之前需要hive元数据库初始化

[root@PCS102 bin]# /usr/local/apache-hive-3.1.-bin/bin/schematool -dbType mysql -initSchema

未初始化 会报错：

hive> create table test01(id int,age int);
FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

步骤六：启动 默认进入命令行

[root@PCS102 bin]# /usr/local/apache-hive-3.1.1-bin/bin/hive
which: no hbase in (/usr/local/jdk1..0_65/bin:/home/cluster/subversion-1.10./bin:/home/cluster/apache-storm-0.9./bin:/usr/local/hadoop-3.1./bin:/usr/local/hadoop-3.1./sbin:/usr/local/apache-hive-3.1.-bin/bin:/usr/local/jdk1..0_80/bin:/home/cluster/subversion-1.10./bin:/home/cluster/apache-storm-0.9./bin:/usr/local/hadoop-3.1./bin:/usr/local/hadoop-3.1./sbin:/usr/local/jdk1..0_80/bin:/home/cluster/subversion-1.10./bin:/home/cluster/apache-storm-0.9./bin:/usr/local/hadoop-3.1./bin:/usr/local/hadoop-3.1./sbin:/usr/local/jdk1..0_80/bin:/home/cluster/subversion-1.10./bin:/home/cluster/apache-storm-0.9./bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/openssh/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/apache-hive-3.1.-bin/lib/log4j-slf4j-impl-2.10..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.1./share/hadoop/common/lib/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = e08b6258-e2eb-40af-ba98-87abcb2d1728
 
Logging initialized using configuration in jar:file:/usr/local/apache-hive-3.1.-bin/lib/hive-common-3.1..jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive  and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive .X releases.
hive> create table test01(id int,age int);
OK
Time taken: 1.168 seconds
hive> desc test01;
OK
id int
age int
Time taken: 0.181 seconds, Fetched:  row(s)
hive> insert into test01 values(1,23);
Query ID = root_20190125164516_aa852f47-a9b0-4c59--efb557965a5b
Total jobs =
Launching Job  out of
Number of reduce tasks determined at compile time:
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1548397153910_0001, Tracking URL = http://PCS102:8088/proxy/application_1548397153910_0001/
Kill Command = /usr/local/hadoop-3.1./bin/mapred job -kill job_1548397153910_0001
Hadoop job information for Stage-: number of mappers: ; number of reducers:
-- ::, Stage- map = %, reduce = %
-- ::, Stage- map = %, reduce = %, Cumulative CPU 4.96 sec
-- ::, Stage- map = %, reduce = %, Cumulative CPU 8.01 sec
MapReduce Total cumulative CPU time:  seconds  msec
Ended Job = job_1548397153910_0001
Stage- is selected by condition resolver.
Stage- is filtered out by condition resolver.
Stage- is filtered out by condition resolver.
Moving data to directory hdfs://PCS102:9820/root/hive_remote/warehouse/test01/.hive-staging_hive_2019-01-25_16-45-16_011_1396999443961154869-1/-ext-10000
Loading data to table default.test01
MapReduce Jobs Launched:
Stage-Stage-: Map:  Reduce:  Cumulative CPU: 8.01 sec HDFS Read:  HDFS Write:  SUCCESS
Total MapReduce CPU Time Spent:  seconds  msec
OK
Time taken: 23.714 seconds
hive>

查看HDFS：

[root@PCS102 bin]# hdfs dfs -cat /root/hive_remote/warehouse/test01/*
123

查看插入数据MR：

查看mysql:

字段：

表：

步骤七：退出
hive>exit;
或者
hive>quit;

3、远程服务器模式/多用户模式：用于非Java客户端访问元数据库，在服务器端启动MetaStoreServer，客户端利用Thrift协议通过MetaStoreServer访问元数据库；

服务端需要启动metastore服务

[root@PCS102 conf]# nohup hive --service metastore &

[root@PCS102 conf]# jps
24657 RunJar
 Jps
 NameNode
 NodeManager
 DataNode
 JobHistoryServer
 RunJar
 ResourceManager
 SecondaryNameNode

1、服务端和客户端在同一个节点：
PCS101：mysql服务端
PCS102：hive服务端和客户端

PCS102 配置文件：hive-site.xml

<configuration>
<!--hive数据上传到HDFS中的目录-->
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/root/hive_remote/warehouse</value>
</property>
<!--hive是否本地模式-->
<property>
<name>hive.metastore.local</name>
<value>false</value>
</property>
<!--hive连接mysql地址-->
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://PCS101/hive_remote?createDatabaseIfNotExist=true</value>
</property>
<!--hive连接mysql驱动类-->
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<!--hive连接mysql用户名-->
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<!--hive连接mysql 密码-->
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
</property>
<!--hive meta store client地址-->
<property>
<name>hive.metastore.uris</name>
<value>thrift://PCS102:9083</value>
</property>
</configuration>

2、服务端和客户端在不同节点(客户端 服务端都要依赖hadoop)
PCS101：mysql服务端
PCS102：hive服务端
PCS103：hive客户端

PCS102：hive服务端配置文件：hive-site.xml

<configuration>
<!--hive数据上传到HDFS中的目录-->
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/root/hive_remote/warehouse</value>
</property>
<!--hive连接mysql地址-->
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://PCS101/hive_remote?createDatabaseIfNotExist=true</value>
</property>
<!--hive连接mysql驱动类-->
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<!--hive连接mysql用户名-->
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<!--hive连接mysql 密码-->
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
</property>
</configuration>

PCS103：hive客户端配置文件：hive-site.xml

<configuration>
<!--hive数据上传到HDFS中的目录-->
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/root/hive_remote/warehouse</value>
</property>
<!--hive是否本地模式-->
<property>
<name>hive.metastore.local</name>
<value>false</value>
</property>
<!--hive meta store client地址-->
<property>
<name>hive.metastore.uris</name>
<value>thrift://PCS102:9083</value>
</property>
</configuration>

问题：/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib   下jar包jline-0.9.94.jar 比较老导致  将hive下jline拷贝到hadoop下就可以了

[root@node101 bin]# schematool -dbType mysql -initSchema
// :: WARN conf.HiveConf: HiveConf of name hive.metastore.local does not exist
Metastore connection URL:     jdbc:mysql://node102/hive_remote?createDatabaseIfNotExist=true
Metastore Connection Driver :     com.mysql.jdbc.Driver
Metastore connection User:     root
Starting metastore schema initialization to 1.2.
Initialization script hive-schema-1.2..mysql.sql
[ERROR] Terminal initialization failed; falling back to unsupported
java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
    at jline.TerminalFactory.create(TerminalFactory.java:)
    at jline.TerminalFactory.get(TerminalFactory.java:)
    at org.apache.hive.beeline.BeeLineOpts.<init>(BeeLineOpts.java:)
    at org.apache.hive.beeline.BeeLine.<init>(BeeLine.java:)
    at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:)
    at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:)
    at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:)
    at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:)
    at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
    at java.lang.reflect.Method.invoke(Method.java:)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:)
 
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
    at org.apache.hive.beeline.BeeLineOpts.<init>(BeeLineOpts.java:)
    at org.apache.hive.beeline.BeeLine.<init>(BeeLine.java:)
    at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:)
    at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:)
    at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:)
    at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:)
    at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
    at java.lang.reflect.Method.invoke(Method.java:)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:)