问题

Drill最新版本是1.14,从1.13开始Drill支持hive的版本升级到2.3.2,详见1.13的release notes

  • The Hive client for Drill is updated to version 2.3.2. With the update, Drill supports queries on transactional (ACID) and non-transactional Hive bucketed ORC tables. The updated libraries are backward compatible with earlier versions of the Hive server and metastore. (DRILL-5978)

强行使用Drill1.14连接Hive2.1.1会由于metastore thrift接口变化导致问题,具体体现为 show tables是空,具体报错如下:

2018-10-10 13:03:54,355 [244277c5-ba8c-b6c8-8f99-2cdde9f3c4d8:frag:0:0] WARN  o.a.d.e.s.h.DrillHiveMetaStoreClient - Failure while attempting to get hive table. Retries once.

org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'

at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) ~[drill-hive-exec-shaded-1.14.0.jar:1.14.0]

at org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$TableLoader.load(DrillHiveMetaStoreClient.java:531) [drill-storage-hive-core-1.14.0.jar:1.14.0]

at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) [guava-18.0.jar:na]

at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) [guava-18.0.jar:na]

at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) [guava-18.0.jar:na]

at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) [guava-18.0.jar:na]

at com.google.common.cache.LocalCache.get(LocalCache.java:3937) [guava-18.0.jar:na]

at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) [guava-18.0.jar:na]

at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) [guava-18.0.jar:na]

at org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.getHiveReadEntry(DrillHiveMetaStoreClient.java:495) [drill-storage-hive-core-1.14.0.jar:1.14.0]

at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSelectionBaseOnName(HiveSchemaFactory.java:230) [drill-storage-hive-core-1.14.0.jar:1.14.0]

at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getDrillTable(HiveSchemaFactory.java:210) [drill-storage-hive-core-1.14.0.jar:1.14.0]

at org.apache.drill.exec.store.hive.schema.HiveDatabaseSchema.getTable(HiveDatabaseSchema.java:62) [drill-storage-hive-core-1.14.0.jar:1.14.0]

at org.apache.drill.exec.store.AbstractSchema.getTablesByNames(AbstractSchema.java:239) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.store.AbstractSchema.getTableNamesAndTypes(AbstractSchema.java:257) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator$Tables.visitTables(InfoSchemaRecordGenerator.java:301) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:216) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:209) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:209) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:196) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.store.ischema.InfoSchemaTableType.getRecordReader(InfoSchemaTableType.java:58) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:34) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:30) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:159) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:137) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:110) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:87) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:261) [drill-java-exec-1.14.0.jar:1.14.0]

at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.14.0.jar:1.14.0]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]

编译

于是尝试重新编译Drill1.14,将依赖的hive版本降到2.1.1,下载代码

http://mirror.bit.edu.cn/apache/drill/drill-1.14.0/apache-drill-1.14.0-src.tar.gz

POM

修改pom中的hive版本

<hive.version>2.3.2</hive.version>

修改为<hive.version>2.1.1</hive.version>

重新编译打包后发现问题依旧,经检查发现修改版本之后只有jars/3rdparty下的3个hive相关jar从2.3.2改为2.1.1

hive-contrib-2.1.1.jar
hive-hbase-handler-2.1.1.jar
hive-metastore-2.1.1.jar

报错的jar是drill-hive-exec-shaded-1.14.0.jar,这个jar包中包含包含hive-exec及依赖,

<artifactId>maven-shade-plugin</artifactId>

<configuration>

<artifactSet>

<includes>

<include>org.apache.hive:hive-exec</include>

并且没有使用配置的hive.version

<dependency>

<groupId>org.apache.hive</groupId>

<artifactId>hive-exec</artifactId>

<scope>compile</scope>

导致打进jar包中的hive-exec是2.3.2版本的,增加hive.version配置

<dependency>

<groupId>org.apache.hive</groupId>

<artifactId>hive-exec</artifactId>

      <version>${hive.version}</version>

<scope>compile</scope>

再打包,问题消失,show tables正常;

Hadoop Location

官方文档说明如下:

Apache Drill users must tell Drill-on-YARN the location of your Hadoop install. Set the HADOOP_HOME environment variable in $DRILL_SITE/drillenv.sh to point to your Hadoop installation:

export HADOOP_HOME= /path/to/hadoop-home  

但配置之后依然存在问题:

1)报错

Diagnostics: File file:/user/drill/site.tar.gz does not exist
java.io.FileNotFoundException: File file:/user/drill/site.tar.gz does not exist

需要添加link

ln -s $HADOOP_HOME/etc/hadoop/core-site.xml $DRILL_SITE/core-site.xml

2)在实际查询时会报错找不到hdfs_name,需要添加link

ln -s $HADOOP_HOME/etc/hadoop/hdfs-site.xml $DRILL_SITE/hdfs-site.xml

【原创】大数据基础之Drill(2)Drill1.14+Hive2.1.1运行的更多相关文章

  1. 【原创】大数据基础之Drill(1)简介、安装及使用

    https://drill.apache.org/ 一 简介 Drill is an Apache open-source SQL query engine for Big Data explorat ...

  2. 【原创】大数据基础之Zookeeper(2)源代码解析

    核心枚举 public enum ServerState { LOOKING, FOLLOWING, LEADING, OBSERVING; } zookeeper服务器状态:刚启动LOOKING,f ...

  3. 【原创】大数据基础之Benchmark(2)TPC-DS

    tpc 官方:http://www.tpc.org/ 一 简介 The TPC is a non-profit corporation founded to define transaction pr ...

  4. 【原创】大数据基础之词频统计Word Count

    对文件进行词频统计,是一个大数据领域的hello word级别的应用,来看下实现有多简单: 1 Linux单机处理 egrep -o "\b[[:alpha:]]+\b" test ...

  5. 【原创】大数据基础之Impala(1)简介、安装、使用

    impala2.12 官方:http://impala.apache.org/ 一 简介 Apache Impala is the open source, native analytic datab ...

  6. 大数据基础知识:分布式计算、服务器集群[zz]

    大数据中的数据量非常巨大,达到了PB级别.而且这庞大的数据之中,不仅仅包括结构化数据(如数字.符号等数据),还包括非结构化数据(如文本.图像.声音.视频等数据).这使得大数据的存储,管理和处理很难利用 ...

  7. 大数据基础知识问答----spark篇,大数据生态圈

    Spark相关知识点 1.Spark基础知识 1.Spark是什么? UCBerkeley AMPlab所开源的类HadoopMapReduce的通用的并行计算框架 dfsSpark基于mapredu ...

  8. 大数据基础知识问答----hadoop篇

    handoop相关知识点 1.Hadoop是什么? Hadoop是一个由Apache基金会所开发的分布式系统基础架构.用户可以在不了解分布式底层细节的情况下,开发分布式程序.充分利用集群的威力进行高速 ...

  9. hadoop大数据基础框架技术详解

    一.什么是大数据 进入本世纪以来,尤其是2010年之后,随着互联网特别是移动互联网的发展,数据的增长呈爆炸趋势,已经很难估计全世界的电子设备中存储的数据到底有多少,描述数据系统的数据量的计量单位从MB ...

随机推荐

  1. JQ面向对象的放大镜

    index.html <!DOCTYPE html><html> <head> <meta charset="utf-8" /> & ...

  2. dubbo在idea下的使用创建 服务者,消费者 注册中心

    1.基于windows 下  spring 下的dubbo  需要书写配置文件 (1).创建带有web工程的项目 创建一个服务者 package cn.edu.aynu.bean; import lo ...

  3. Python——装饰器

    1.装饰器形成的过程 2.装饰器的作用 3.原则:开放封闭原则 开放:对扩展是开放的 封闭:对修改是封闭的 4.装饰器的固定模式 def func(): time.sleep(0.01) ') def ...

  4. vhdl when else

    在VHDL中,IF...THEN...ELSE是顺序语句,只能出现在行为描述中(进程体或者子程序中):而WHEN...ELSE是并行语句,可以直接出现在结构体中,但却不能出现在行为描述中.WHEN.. ...

  5. [模板] 虚树 && bzoj2286-[Sdoi2011]消耗战

    简介 虚树可以解决一些关于树上一部分节点的问题. 对于一棵树 \(T\) 的一个子集 \(S\), 可以在 \(O(|S| \log |S|)\) 的时间复杂度内求出 \(S\) 的虚树. 虚树包括根 ...

  6. java基础1之引用数据类型

    5种引用类型(对象类型) 类 接口 数组 枚举 标注 类 类在JVM的内存空间的存储 (1). Heap 堆空间:分配对象 new Student() 存放引用数据类型的实例 (2). Stack 栈 ...

  7. stop()在animate中的用法

    stop 是jQuery中用于控制页面动画效果的方法.运行之后立刻结束当前页面上的动画效果.stop在新版jQuery中添加了2个参数:第一个参数的意思是是否清空动画序列,也就是stop的是当前元素的 ...

  8. 探索 Python 学习

    Python 是一种敏捷的.动态类型化的.极富表现力的开源编程语言,可以被自由地安装到多种平台上(参阅 参考资料).Python 代码是被解释的.如果您对编辑.构建和执行循环较为熟悉,则 Python ...

  9. Python import hook

    转自http://blog.konghy.cn/2016/10/25/python-import-hook/,这里有好多好文章 import hook 通常被译为 探针.我们可以认为每当导入模块的时候 ...

  10. spring security oauth2 client_credentials模

    spring security oauth2 client_credentials模 https://www.jianshu.com/p/1c3eea71410e 序 本文主要简单介绍一下spring ...