hive的安装和升级

目录

目录

• 目录
• 安装
• 升级
  • 检查环境
• 参考

1. 安装
2. 升级

---

安装

nohup hive –service hiveserver  &

启动：

sudo nohup ./hive --service metastore &-----------Startup Hive Embedded(如果metastore配置成 remote方式 那就应该把metastore也启动) 测试standalone，必须用remote mysql那种设置

• 1，hive 命令行模式。
  • 直接输入/hive/bin/hive的执行程序
  • 或者输入 hive –service cli 用于linux平台命令行查询，查询语句基本跟mysql查询语句类似
• 2， hive web界面的启动方式，hive –service hwi
  
  用于通过浏览器来访问hive，感觉没多大用途
• 3， hive 远程服务 (端口号10000) 启动方式，nohup hive –service hiveserver &
  
  用java等程序实现通过jdbc等驱动的访问hive就用这种起动方式了，这个是程序员最需要的方式了.启动hive service ：$HIVE_HOME/bin/hive --service hiveserver 10001 >/dev/null 2>/dev/null &
  
  参考：调用javaAPI访问hive

参考：

hive-1.2.1安装步骤

升级

目标

• 对hive从0.14.0升级到1.2.1，
• 集群中使用了hue，升级后检查hue是否能够正常。

整个过程如下

• 检查环境、当前配置
• 当前运行情况
• 下载最新版
• 上传并替换现有目录
  • 用最新的密码。在金山云主机.XLS
  • 在登录生产线的时候选择sftp协议
• 修改配置文件
• 修改权限为hive
• 启动
• 升级参考

检查环境

hive、hue都是只用部署在一个节点上的，只考虑一个节点，确认下安装位置。

配置文件

确定hue安装节点

[kduser@master1 conf]$ netstat -ano | grep 9401
tcp        0      0 0.0.0.0:9401                0.0.0.0:*                   LISTEN      off (0.00/0/0)
tcp        0      0 10.63.7.30:9401             219.133.170.76:60402        TIME_WAIT   timewait (14.46/0/0)
tcp        0      0 10.63.7.30:9401             219.133.170.76:50811        ESTABLISHED off (0.00/0/0)
tcp        0      0 10.63.7.30:9401             219.133.170.172:8322        ESTABLISHED off (0.00/0/0)
tcp        0      0 10.63.7.30:9401             219.133.170.76:19791        TIME_WAIT   timewait (44.73/0/0)
tcp        0      0 10.63.7.30:9401             219.133.170.76:39595        ESTABLISHED off (0.00/0/0)
tcp        0      0 10.63.7.30:9401             219.133.170.76:2382         ESTABLISHED off (0.00/0/0)

确定hive的安装节点

[kduser@master1 conf]$ netstat -ano | grep 9300
tcp        0      0 10.63.7.30:9300             0.0.0.0:*                   LISTEN      off (0.00/0/0)
tcp        0      0 10.63.7.30:9300             10.63.7.30:48307            ESTABLISHED off (0.00/0/0)
tcp        0      0 10.63.7.30:9300             10.63.7.30:64415            ESTABLISHED off (0.00/0/0)
tcp        0      0 10.63.7.30:7435             10.63.7.30:9300             ESTABLISHED off (0.00/0/0)
tcp        1      0 10.63.7.30:41196            10.63.7.30:9300             CLOSE_WAIT  off (0.00/0/0)
tcp        0      0 10.63.7.30:64428            10.63.7.30:9300             ESTABLISHED off (0.00/0/0)
tcp        1      0 10.63.7.30:34734            10.63.7.30:9300             CLOSE_WAIT  off (0.00/0/0)
tcp        0      0 10.63.7.30:53330            10.63.7.30:9300             ESTABLISHED off (0.00/0/0)
tcp        0      0 10.63.7.30:48307            10.63.7.30:9300             ESTABLISHED off (0.00/0/0)
tcp        1      0 10.63.7.30:32012            10.63.7.30:9300             CLOSE_WAIT  off (0.00/0/0)
tcp        0      0 10.63.7.30:9300             10.63.7.30:64428            ESTABLISHED off (0.00/0/0)
tcp        0      0 10.63.7.30:64415            10.63.7.30:9300             ESTABLISHED off (0.00/0/0)
tcp        0      0 10.63.7.30:54545            10.63.7.30:9300             ESTABLISHED off (0.00/0/0)
tcp        0      0 10.63.7.30:9300             10.63.7.30:7435             ESTABLISHED off (0.00/0/0)
tcp        1      0 10.63.7.30:32153            10.63.7.30:9300             CLOSE_WAIT  off (0.00/0/0)
tcp        0      0 10.63.7.30:9300             10.63.7.30:53330            ESTABLISHED off (0.00/0/0)
tcp        0      0 10.63.7.30:9300             10.63.7.30:54545            ESTABLISHED off (0.00/0/0)

查看hue的目录

[kduser@master1 hue]$ cd hue
[kduser@master1 hue]$ ll
total 80
-rw-rw-r--  1 hue hue  2782 Jun 17  2015 app.reg
drwxrwxr-x 22 hue hue  4096 Jun 17  2015 apps
drwxrwxr-x  4 hue hue  4096 Jun 17  2015 build
drwxrwxr-x  5 hue hue  4096 Jun 17  2015 desktop
drwxrwxr-x  3 hue hue  4096 May  4  2015 ext
-rw-rw-r--  1 hue hue 11358 May  4  2015 LICENSE.txt
drwxrwxr-x  2 hue hue  4096 Apr  7 13:56 logs
-rw-rw-r--  1 hue hue  4715 May  4  2015 Makefile
-rw-rw-r--  1 hue hue    44 Jun 17  2015 Makefile.buildvars
-rw-rw-r--  1 hue hue  8505 May  4  2015 Makefile.sdk
-rw-rw-r--  1 hue hue  3498 May  4  2015 Makefile.vars
-rw-rw-r--  1 hue hue  2192 May  4  2015 Makefile.vars.priv
-rw-rw-r--  1 hue hue  1562 May  4  2015 README
drwxrwxr-x  4 hue hue  4096 Jun 17  2015 tools
-rw-rw-r--  1 hue hue   932 May  4  2015 VERSION
[kduser@master1 hue]$ cd desktop/
[kduser@master1 desktop]$ ll
total 244
drwxrwxr-x  2 hue hue   4096 Nov 28 17:56 conf
drwxrwxr-x  4 hue hue   4096 Jun 17  2015 core
-rw-r--r--  1 hue hue 227328 Jun 17  2015 desktop.db
drwxrwxr-x 11 hue hue   4096 Jun 17  2015 libs
-rw-rw-r--  1 hue hue   3400 May  4  2015 Makefile
[kduser@master1 desktop]$ cd  conf/
[kduser@master1 conf]$ ll
total 48
-rw-rw-r-- 1 hue hue 37237 Nov 21 20:06 hue.ini
-rw-rw-r-- 1 hue hue  1843 May  4  2015 log4j.properties
-rw-rw-r-- 1 hue hue  1550 May  4  2015 log.conf
[kduser@master1 conf]$ more hue.ini
# Hue configuration file
# ===================================
#
# For complete documentation about the contents of this file, run
#   $ <hue_root>/build/env/bin/hue config_help
#
# All .ini files under the current directory are treated equally.  Their
# contents are merged to form the Hue configuration, which can
# can be viewed on the Hue at
#   http://<hue_host>:<port>/dump_config

###########################################################################
# General configuration for core Desktop features (authentication, etc)
###########################################################################

[desktop]

  # Set this to a random string, the longer the better.
  # This is used for secure hashing in the session store.
  secret_key=

  # Webserver listens on this address and port
  http_host=0.0.0.0
  http_port=9401

  # Time zone name
  time_zone=UTC

  # Enable or disable Django debug mode.
  django_debug_mode=false

  # Enable or disable database debug mode.
  ## database_logging=false

  # Enable or disable backtrace for server error
  http_500_debug_mode=false

  # Enable or disable memory profiling.
  ## memory_profiler=false

  # Server email for internal error messages
  ## django_server_email='hue@localhost.localdomain'

  # Email backend

找hue和hive的关系，发现hue.ini中的beeswax就是hive相关的

[kduser@master1 hue]$  find . -name hive
./apps/oozie/examples/unmanaged/hive
./apps/oozie/examples/managed/hive
[kduser@master1 hue]$

[kduser@master1 conf]$ pwd
/mnt/vdc/hue/hue/desktop/conf
[kduser@master1 conf]$ ll
total 48
-rw-rw-r-- 1 hue hue 37237 Nov 21 20:06 hue.ini
-rw-rw-r-- 1 hue hue  1843 May  4  2015 log4j.properties
-rw-rw-r-- 1 hue hue  1550 May  4  2015 log.conf

 more hue.ini
[beeswax]

  # Host where HiveServer2 is running.
  # If Kerberos security is enabled, use fully-qualified domain name (FQDN).
  hive_server_host=master1

  # Port where HiveServer2 Thrift server runs on.
  hive_server_port=9300

  # Hive configuration directory, where hive-site.xml is located
  hive_conf_dir=/mnt/vdc/hive/conf

  # Timeout in seconds for thrift calls to Hive service
  ## server_conn_timeout=120

  # Choose whether to use the old GetLog() thrift call from before Hive 0.14 to retrieve the logs.
  # If false, use the FetchResults() thrift call from Hive 1.0 or more instead.
  ## use_get_log_api=false

  # Set a LIMIT clause when browsing a partitioned table.
  # A positive value will be set as the LIMIT. If 0 or negative, do not set any limit.
  ## browse_partitioned_table_limit=250

  # A limit to the number of rows that can be downloaded from a query.
  # A value of -1 means there will be no limit.
  # A maximum of 65,000 is applied to XLS downloads.
  ## download_row_limit=1000000

  # Hue will try to close the Hive query when the user leaves the editor page.
  # This will free all the query resources in HiveServer2, but also make its results inaccessible.
  ## close_queries=false

  # Thrift version to use when communicating with HiveServer2.
  # New column format is from version 7.
  thrift_version=7

解压上传的压缩包

tar -zxvf apache-hive-1.2.1-bin.tar.gz
mv apache-hive-1.2.1-bin hive

把旧版本中conf的配置文件复制到新版本目录中

cp  /mnt/vdc/hive/conf/hive-env.sh  .
cp  /mnt/vdc/hive/conf/hive-site.xml   .

添加其他配置文件

cp beeline-log4j.properties.template  beeline-log4j.properties
sudo cp beeline-log4j.properties.template  beeline-log4j.properties
sudo hive-default.xml.template  hive-default.xml
sudo cp  hive-default.xml.template  hive-default.xml
sudo cp  hive-log4j.properties.template  hive-log4j.properties
sudo cp hive-exec-log4j.properties.template  hive-exec-log4j.properties

因为配置的hive用mysql数据库，需要加连接驱动

cd hive/lib
sudo cp /mnt/vdc/hive_old/lib/mysql-connector-java-5.1.38.jar

修改权限

sudo chown -R  hive:hive hive

遇到的问题：

启动hive annot access /mnt/vdc/spark/lib/spark-assembly-*.jar: No such file

在命令行敲入hive启动的时候

• 无法访问/home/ndscbigdata/soft/spark-2.0.0/lib/spark-assembly-*.jar: 没有那个文件或目录。
• 无法找到hadoop相关的类
  
  hive默认了一些类库的位置。说明我在/etc/profile中配置的环境变量没起作用。
  
  解决方法：
  
  在hive/conf的中的hive.env中添加：

参考：无法访问../lib/spark-assembly-*.jar: 没有那个文件或目录的解决办法

参考

HUE配置HIVE - 单行道| - 博客园 http://www.cnblogs.com/one--way/p/5687857.html