HIVE学习（待更新）

1 安装hive

下载

http://mirrors.shu.edu.cn/apache/hive/hive-1.2.2/，红框中的不需要编译。

由于hive是默认将元数据保存在本地内嵌的 Derby 数据库中，但是这种做法缺点也很明显，Derby不支持多会话连接，因此本文将选择mysql作为元数据存储。

安装mysql

yum安装mysql
 
  wget -i -c http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm
 
 yum -y install mysql57-community-release-el7-.noarch.rpm 
 
 yum -y install mysql-community-server
 
启动MySQL
systemctl start  mysqld.service
查看MySQL运行状态
systemctl status mysqld.service
 
mysql -uroot -p     # 回车后会提示输入密码
此时MySQL已经开始正常运行，不过要想进入MySQL还得先找出此时root用户的密码，通过如下命令可以在日志文件中找出密码：
ALTER USER 'root'@'localhost' IDENTIFIED BY 'new password';
12345
 
具体请参考
https://www.cnblogs.com/brianzhu/p/8575243.html

tar -zxvf apache-hive-1.2.3-bin.tar.gz  后的内容如下，进入到conf

cp hive-default.xml.template  hive-site.xml

编辑文件hive-site.xml

<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
<!--
        Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at
 
       http://www.apache.org/licenses/LICENSE-2.0
 
   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-->
<configuration>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>xxxx</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>xxxx</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>mysql
    <value>jdbc:mysql://hostIP:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false&amp;nullNamePatternMatchesAll=true</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
  </property>
</configuration>

复制mysql的驱动程序到hive/lib下面

2 hive启动方式

运行hive之前首先要确保meta store服务已经启动，
 
nohup hive --service metastore > metastore.log >& &
 
如果需要用到远程客户端(比如 Tableau)连接到hive数据库，还需要启动hive service
 
nohup hive --service hiveserver2 > hiveserver2.log >& &
 
[sms@gc64 conf]$ hive --help
Usage ./hive <parameters> --service serviceName <service parameters>
Service List: beeline cli help hiveburninclient hiveserver2 hiveserver hwi jar lineage metastore metatool orcfiledump rcfilecat schemaTool version 
Parameters parsed:
  --auxpath : Auxillary jars 
  --config : Hive configuration directory
  --service : Starts specific service/component. cli is default
Parameters used:
  HADOOP_HOME or HADOOP_PREFIX : Hadoop install directory
  HIVE_OPT : Hive options
For help on a particular service:
  ./hive --service serviceName --help
Debug help:  ./hive --debug --help

hive2.0一下没有web查看

[sms@gc64 ~]$ hive
 
Logging initialized using configuration in jar:file:/home/sms/app/apache-hive-1.2.-bin/lib/hive-common-1.2..jar!/hive-log4j.properties
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
hive>
    >
    >
    > show databases;
OK
default
Time taken: 1.285 seconds, Fetched:  row(s)
hive>
    >
    >
    >
    >
    >
    >
    >
    >
    > 

from pyspark.sql import HiveContext,Row
from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("local").setAppName("count")
sc = SparkContext(conf=conf)
hiveCtx=HiveContext(sc)
hiveCtx.sql("show tables").show()
hiveCtx.sql("select count(1) from (select msid from raw_data group by msid) a").show()