解决kylin sync table报错：MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found
在kylin-gui中sync表default.customer_visit时报错：
-- ::, ERROR [http-bio--exec-] controller.BasicController: :
java.lang.RuntimeException: cannot get HiveTableMeta
at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.rest.service.CubeService.reloadHiveTable(CubeService.java:)
at org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:)
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:)
at org.apache.kylin.rest.service.CubeService$$EnhancerBySpringCGLIB$$543fa169.reloadHiveTable(<generated>)
at org.apache.kylin.rest.controller.TableController.loadHiveTable(TableController.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result.read(ThriftHiveMetastore.java:)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_fields(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_fields(ThriftHiveMetastore.java:)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:)
at org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:)
如果此表已被别的project引入，在另一个project中sync时报错为：
java.lang.RuntimeException: Failed to fetch metadata of 'DEFAULT.CUSTOMER_VISIT': java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not fou
nd
at org.apache.kylin.source.hive.SchemaChecker$CheckResult.raiseExceptionWhenInvalid(SchemaChecker.java:)
at org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiveTables(HiveSourceTableLoader.java:)
at org.apache.kylin.rest.service.CubeService.reloadHiveTable(CubeService.java:)
at org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:)
下面是此表的定义，ClassNotFoundException中的类就是此表用到的JsonSerDe:
CREATE EXTERNAL TABLE `customer_visit`(
`age_m` int COMMENT 'from deserializer',
`discharge_date` date COMMENT 'from deserializer',
`address_code` string COMMENT 'from deserializer',
`admission_date` date COMMENT 'from deserializer',
`birth_date` date COMMENT 'from deserializer',
`outcome` string COMMENT 'from deserializer',
`age` int COMMENT 'from deserializer')
ROW FORMAT SERDE
'org.apache.hive.hcatalog.data.JsonSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'hdfs://xxx/user/data/abc.db/customer_visit'
TBLPROPERTIES (
'COLUMN_STATS_ACCURATE'='false',
'numFiles'='',
'numRows'='-1',
'rawDataSize'='-1',
'totalSize'='',
'transient_lastDdlTime'='')
解决方案：
相关jar需要放到hive metastore服务的$HIVE_HOME/lib/。从第一个报错的Caused by部分可以看到报错是从org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore抛出来的，提示了是metastore的报错。
2. 另外kylin服务器上的$HIVE_HOME/lib/也需要放置此jar，否则执行第一步报错，这是hive cli需要用的：
FAILED: RuntimeException MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found)
at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:92)
at org.apache.kylin.source.hive.CreateFlatHiveTableStep.createFlatHiveTable(CreateFlatHiveTableStep.java:90)
at org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork(CreateFlatHiveTableStep.java:121)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
执行build第一步时报如下错，需要配置hive-env.sh 中的HIVE_AUX_JARS_PATH，把用到的jar包路径配置到其中，这个是给MR用的，HIVE_AUX_JARS_PATH的说明是：Folder containing extra ibraries required for hive compilation/execution
2017-04-27 19:34:33,040 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140)
... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found
at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328)
at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109)
... 22 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141)
at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
... 24 more
一点收获：
1. kylin使用hive 依赖的是本地的hive客户端
2. 鉴于第一条，调试Kylin和hive相关的东西时，只需对本地hive客户端进行修改，不需要hive，这样可以节省不少时间
3. kylin sync table是通过hive metastore服务执行的，所以需要相关表在hive metastore服务上能正常访问，是否能正常访问可以在metastore服务的机器上执行相关hql（主要是元数据相关的hql）验证。
解决kylin sync table报错：MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found的更多相关文章
- 【原创】大叔问题定位分享（8）提交spark任务报错 Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException
spark 2.1.1 一 问题重现 spark-submit --master local[*] --class app.package.AppClass --jars /jarpath/zkcli ...
- javafx:JavaFX Scene Builder 2.0打开含有第三方jar包的fxml文件报错 Caused by: java.lang.ClassNotFoundException
报错如下: java.io.IOException: javafx.fxml.LoadException: /C:/User.................test.fxml at com.orac ...
- 【spring cloud】spring cloud 使用feign调用，1.fallback熔断器不起作用，2.启动报错Caused by: java.lang.ClassNotFoundException: com.netflix.hystrix.contrib.javanica.aop.aspectj.Hystri解决
示例GitHub源码地址:https://github.com/AngelSXD/springcloud 1.首先使用feign调用,需要配置熔断器 2.配置熔断器需要将熔断器注入Bean,熔断器类上 ...
- eclipse 报错问题：java.lang.ClassNotFoundException:
解决方法:https://www.cnblogs.com/whatlonelytear/articles/5921978.html
- Tomcat报错java.lang.ClassNotFoundException: 2localhost.org.apache.juli.FileHandler
Can't load log handler "1catalina.org.apache.juli.FileHandler" java.lang.ClassNotFoundExce ...
- 报错：maven:java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
运行报错: maven:java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets 找不到Charsets这个类 上网查了以后,是因为 ...
- Tomcat报java.lang.ClassNotFoundException: 1catalina.org.apache.juli.FileHandler
最近在生产环境部署Tomcat的时候,在启动的时候,在控制台报"java.lang.ClassNotFoundException: 1catalina.org.apache.juli.Fil ...
- 用JSON报的一个错误java.lang.ClassNotFoundException: org.apache.commons.lang.exception.NestableRuntimeExcep
以前在做项目的时候就曾接触过JSON的技术,但那个时候是项目经理把所有该配制的都配了,工具类也提供了,如何使用也跟我们说了,那个时候只是觉得很好用,倒没有研究过. 今天自己写了一个JSON的例子,可以 ...
- 解决dwr报错【 Error: java.lang.SecurityException: No class by name: service】
打开包含dwr的网页时后台报错: 警告: Names of known classes are: __System DwrQueryService 十二月 11, 2015 10:24:44 上午 o ...
随机推荐
- Redis数据类型之Set（四）
前言:set类似于数学上面的集合概念,包含的元素无序,不能重复,能进行交.并.差操作. 一.内部原理 set数据结构,也是随着元素数目的多少而变化.当set中添加 ...
- LoadRunner中文乱码问题解决方法
LoadRunner中文乱码问题解决方法 前段时间在录制,增强,整合LoadRunner脚本,期间两次遇到了中文乱码问题.在此记录一下中文乱码问题的解决办法. 一.录制回放中文乱码 我录制登陆的脚本, ...
- 8-7 Unique Snowflakes UVA11572
输入一个长度为n n<=10 6 的序列A 找到一个尽量长的连续子序列 使得该序列中没有相同的元素 用滑动窗口法 时间复杂度n 好神奇 此题非常经典 map 410ms #inc ...
- 《Android源码设计模式》--单例模式
No1: 懒汉单例模式优缺点分析 public class Singleton{ private static Singleton instance; private Singleton(){} pu ...
- NetworkX 使用（三）
官方教程 博客:NetworkX NetworkX 使用(二) Introduction to Graph Analysis with NetworkX %pylab inline import ne ...
- Python面向对象中super用法与MRO机制
1. 引言 最近在研究django rest_framework的源码,老是遇到super,搞得一团蒙,多番查看各路大神博客,总算明白了一点,今天做一点总结. 2. 为什么要用super 1)让代码维 ...
- （四）静态断言(上)，assert，NDEBUG， 以及通过宏定义处理文件包含关系
一.断言:运行时与预处理时 断言(assertion)是一种编程常用的手段.想必大家都见过 assert 吧.今天我们就来了解一下它. 通常情况下,断言就是将一个返回值总是需要为真的判别式放在语句中, ...
- shell 规范
代码风格规范 开头有“蛇棒” 所谓shebang其实就是在很多脚本的第一行出现的以”#!”开头的注释,他指明了当我们没有指定解释器的时候默认的解释器,一般可能是下面这样: #!/bin/bash 当然 ...
- [ 原创 ]Centos 7.0下安装 Tomcat8.5.15
Tomcat下载地址:http://tomcat.apache.org/download-80.cgi#8.5.15 上传到文件夹 并解压缩 出现问题: 解决方法: http://blog.csdn. ...
- 基于ONVIF协议的摄像头开发总结
<什么是ONVIF协议> 2008年5月,由安讯士(AXIS)联合博世(BOSCH)及索尼(SONY)公司三方宣布携手共同成立一个国际开放型网络视频产品标准网络接口开发论坛,取名为 ...