hive on spark配置
1、安装java、maven、scala、hadoop、mysql、hive
略
2、编译spark
./make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-2.6,parquet-provided"
3、安装spark
tar -zxvf spark-1.6.0-bin-hadoop2-without-hive.tgz -C /opt/cdh5/
4、配置spark
:spark-env.sh
export JAVA_HOME=/opt/service/jdk1.8.0_151
export SCALA_HOME=/opt/service/scala-2.10.5
export HADOOP_HOME=/opt/cdh5/hadoop-2.6.0-cdh5.10.0
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HIVE_CONF_DIR=/opt/cdh5/hive-2.1.0/conf
export SPARK_WORKER_CORES=4
export SPARK_WORKER_INSTANCES=4
export SPARK_WORKER_MEMORY=1g
export SPARK_DRIVER_MEMORY=1g
export SPARK_MASTER_IP=chavin.king
export SPARK_LIBRARY_PATH=/opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive/lib
export SPARK_MASTER_WEBUI_PORT=8080
export SPARK_WORKER_WEBUI_PORT=8081
export SPARK_WORKER_DIR=/opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive/work
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_PORT=7078
export SPARK_LOG_DIR=/opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive/log
:spark-default.xml
#spark.master yarn
spark.master spark://chavin.king:7077
spark.home /opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive
spark.eventLog.enabled true
spark.eventLog.dir hdfs://chavin.king:8020/spark-log
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.executor.memory 1g
spark.driver.memory 1g
spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
:slaves
chavin.king
5、配置yarn
:yarn-site.xml
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
6、配置hive
<property>
<name>hive.execution.engine</name>
<value>spark</value>
</property>
<property>
<name>hive.enable.spark.execution.engine</name>
<value>true</value>
</property>
<property>
<name>spark.home</name>
<value>/opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive</value>
</property>
<property>
<name>spark.master</name>
<value>spark://chavin.king:7077</value>
</property>
<property>
<name>spark.enentLog.enabled</name>
<value>true</value>
</property>
<property>
<name>spark.enentLog.dir</name>
<value>hdfs://chavin.king:8020/spark-log</value>
</property>
<property>
<name>spark.serializer</name>
<value>org.apache.spark.serializer.KryoSerializer</value>
</property>
<property>
<name>spark.executor.memeory</name>
<value>1g</value>
</property>
<property>
<name>spark.driver.memeory</name>
<value>1g</value>
</property>
<property>
<name>spark.executor.extraJavaOptions</name>
<value>-XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"</value>
</property>
7、为hive添加spark jar包:
cp /opt/software/spark-1.6.0/core/target/spark-core_2.10-1.6.0.jar /opt/cdh5/hive-2.1.0/lib/
ln -s /opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive/lib/spark-assembly-1.6.0-hadoop2.6.0.jar /opt/cdh5/hive-2.1.0/lib/
bin/hdfs dfs -put /opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive/lib/spark-assembly-1.6.0-hadoop2.6.0.jar hdfs://chavin.king:8020/spark-assembly-1.6.0-hadoop2.6.0.jar
在hive-site.xml中添加:
<property>
<name>spark.yarn.jar</name>
<value>hdfs://chavin.king:8020/spark-assembly-1.6.0-hadoop2.6.0.jar</value>
</property>
8、验证hive on spark是否成功配置
$ bin/hive
which: no hbase in (/opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive/bin:/opt/service/maven-3.3.3/bin:/opt/service/scala-2.10.5/bin:/opt/service/jdk1.8.0_151/bin:/opt/service/jdk1.8.0_151/jre/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cdh5/hive-2.1.0/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cdh5/spark-1.6.0-bin-hadoop2-without-hive/lib/spark-assembly-1.6.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cdh5/hadoop-2.6.0-cdh5.10.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in file:/opt/cdh5/hive-2.1.0/conf/hive-log4j2.properties Async: true
hive (default)> show tables ;
OK
tab_name
t1
Time taken: 0.966 seconds, Fetched: 1 row(s)
hive (default)> select count(*) from t1;
Query ID = hadoop_20171204024017_cda99c42-21eb-480f-9d2a-e0dbb18a9b63
Total jobs = 1
Launching Job 1 out of 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Spark Job = e8b4ccc6-2dfa-43b9-99cc-7a066e2c0a0f
Query Hive on Spark job[0] stages:
0
1
Status: Running (Hive on Spark job[0])
Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]
2017-12-04 02:40:32,861 Stage-0_0: 0/1 Stage-1_0: 0/1
... ...
2017-12-04 02:44:11,388 Stage-0_0: 1/1 Finished Stage-1_0: 0(+1)/1
2017-12-04 02:44:50,826 Stage-0_0: 1/1 Finished Stage-1_0: 1/1 Finished
Status: Finished successfully in 268.11 seconds
OK
c0
3
Time taken: 338.493 seconds, Fetched: 1 row(s)
hive (default)> exit;
hive on spark配置的更多相关文章
- spark 2.0.0集群安装与hive on spark配置
1. 环境准备: JDK1.8 hive 2.3.4 hadoop 2.7.3 hbase 1.3.3 scala 2.11.12 mysql5.7 2. 下载spark2.0.0 cd /home/ ...
- Hive on Spark安装配置详解(都是坑啊)
个人主页:http://www.linbingdong.com 简书地址:http://www.jianshu.com/p/a7f75b868568 简介 本文主要记录如何安装配置Hive on Sp ...
- 大数据学习系列之九---- Hive整合Spark和HBase以及相关测试
前言 在之前的大数据学习系列之七 ----- Hadoop+Spark+Zookeeper+HBase+Hive集群搭建 中介绍了集群的环境搭建,但是在使用hive进行数据查询的时候会非常的慢,因为h ...
- Hive记录-Hive on Spark环境部署
1.hive执行引擎 Hive默认使用MapReduce作为执行引擎,即Hive on mr.实际上,Hive还可以使用Tez和Spark作为其执行引擎,分别为Hive on Tez和Hive on ...
- hive on spark:return code 30041 Failed to create Spark client for Spark session原因分析及解决方案探寻
最近在Hive中使用Spark引擎进行执行时(set hive.execution.engine=spark),经常遇到return code 30041的报错,为了深入探究其原因,阅读了官方issu ...
- Hive on Spark和Spark sql on Hive,你能分的清楚么
摘要:结构上Hive On Spark和SparkSQL都是一个翻译层,把一个SQL翻译成分布式可执行的Spark程序. 本文分享自华为云社区<Hive on Spark和Spark sql o ...
- 基于CDH 5.9.1 搭建 Hive on Spark 及相关配置和调优
Hive默认使用的计算框架是MapReduce,在我们使用Hive的时候通过写SQL语句,Hive会自动将SQL语句转化成MapReduce作业去执行,但是MapReduce的执行速度远差与Spark ...
- CM记录-配置Hive on Spark
默认hive on spark是禁用的,需要在Cloudera Manager中启用.1.登录CM界面,打开hive服务.2.单击 配置标签,查找enable hive on spark属性.3.勾选 ...
- Mac OSX系统中Hadoop / Hive 与 spark 的安装与配置 环境搭建 记录
Mac OSX系统中Hadoop / Hive 与 spark 的安装与配置 环境搭建 记录 Hadoop 2.6 的安装与配置(伪分布式) 下载并解压缩 配置 .bash_profile : ...
随机推荐
- Mathmatica简介
作者:桂. 时间:2018-06-27 21:53:34 链接:https://www.cnblogs.com/xingshansi/p/9236502.html 前言 打算系统学习一些数学知识,容 ...
- Atitit 乌合之众读后感attilax总结 与读后感结构规范总结
Atitit 乌合之众读后感attilax总结 与读后感结构规范总结 1. 背景概览与鸟瞰overview 1 1.1. 社会背景 与 历史事件背景 与历史时间背景 1 1.2. 书籍简绍 2 1. ...
- [db]mysql全量迁移db
机房要裁撤, 原有的老业务机的mysql需要迁移到新的. 方案1: 全量打包拷贝data目录, 发现拷过去各种毛病 方案2: mysqldump逻辑导出解决问题 新的db刚安装好. 步骤记录下. # ...
- CentOS5.9 编译Emacs 24
从Emacs官方网站下载最新版解压后,执行 ./configure 得到错误信息: configure: error: The following required libraries were no ...
- matplotlib绘图不显示问题解决plt.show()
最近在看<Python数据分析>这本书,而自己写代码一直用的是Pycharm,在练习的时候就碰到了plot()绘图不能显示出来的问题.网上翻了一下找到知乎上一篇回答,试了一下好像不行,而且 ...
- [python] ThreadPoolExecutor线程池 python 线程池
初识 Python中已经有了threading模块,为什么还需要线程池呢,线程池又是什么东西呢?在介绍线程同步的信号量机制的时候,举得例子是爬虫的例子,需要控制同时爬取的线程数,例子中创建了20个线程 ...
- 【iCore4 双核心板_ARM】例程三十八:DSP MATH库测试
实验现象: 核心代码: int main(void) { /* USER CODE BEGIN 1 */ int i,j; int res; ]; ; /* USER CODE END 1 */ /* ...
- C语言 · 超级玛丽
算法提高 超级玛丽 时间限制:1.0s 内存限制:256.0MB 问题描述 大家都知道"超级玛丽"是一个很善于跳跃的探险家,他的拿手好戏是跳跃,但它一次只能向前跳 ...
- linux服务查看
(1)#service servicename status比如查看防火墙:#service iptables status (2)#chkconfig --list |grep 服务名 比如查看te ...
- XMind8 破解激活教程(最详细,一定是有效的!!!)
下载安装包 首先去xmind国外官网下载对应操作系统的安装包,国内官网的那个是有残缺的,不支持破解. 点击链接进行下载安装包:https://pan.baidu.com/s/1VITXSEQvwGDi ...