首次尝试SeaTunnel同步Doris至Hive？这些坑你不能不避
笔者使用SeaTunnel 2.3.2版本将Doris数据同步到Hive(cdh-6.3.2)首次运行时有如下报错，并附上报错的解决方案：
- java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
- java.lang.NoClassDefFoundError: org/apache/thrift/TBase
- java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
- java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
- java.lang.OutOfMemoryError: Java heap space
目录：
java.lang.NoClassDefFoundError:org/apache/hadoop/hive/metastore/api/MetaException
1.1 解决办法
java.lang.NoClassDefFoundError: org/apache/thrift/TBase
2.1 解决办法
java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
3.1 解决办法
java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
4.1 解决办法
java.lang.OutOfMemoryError: Java heap space
5.1 解决办法
1、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/metastore/api/MetaException
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.MetaException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 11 more
1.1 解决办法
原因是缺少对应的包，去Hive里面的lib将包复制到SeaTunnel的包下面即可：
hive-metastore-2.1.1-cdh6.3.2.jar # cdh版本的记得将这个也复制上
hive-metastore.jar
2、java.lang.NoClassDefFoundError: org/apache/thrift/TBase
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/thrift/TBase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.thrift.TBase
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
2.1 解决办法
是缺少对应的包，去Hive里面的lib将包复制到SeaTunnel的包下面即可
libthrift-0.9.3-1.jar
3、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:48)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
3.1 解决办法
是缺少对应的包，去Hive里面的lib将包复制到SeaTunnel的包下面即可
hive-common-2.1.1-cdh6.3.2.jar #cdh版本的记得复制过来
hive-common.jar
hive-exec-2.1.1-cdh6.3.2.jar #cdh版本的记得复制过来
hive-exec.jar
4、java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
Exception in thread "main" java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: com.facebook.fb303.FacebookService$Iface
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 25 more
4.1 解决办法
是缺少对应的包，这个我的Hive里面的libthrift-0.9.3-1.jar包里面没有，所以报了这个错，去下面这个地址直接下载即可
下载地址：https://repo1.maven.org/maven2/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar
5、java.lang.OutOfMemoryError: Java heap space
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:379)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4129)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4115)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:563)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:303)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:219)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
5.1 解决办法
去SeaTunnel的config里面增加对应值，截图如下：
将值修改大一点就可以了
原文链接：https://blog.csdn.net/qq_43224174/article/details/131430223
本文由 白鲸开源 提供发布支持！
首次尝试SeaTunnel同步Doris至Hive？这些坑你不能不避的更多相关文章
- Mongodb同步数据到hive（二）
Mongodb同步数据到hive(二) 1. 概述 上一篇文章主要介绍了mongodb-based,通过直连mongodb的方式进行数据映射来进行数据查询,但是那种方式会对线上的 ...
- 【Luogu P1714】切蛋糕（面向对象编程首次尝试？）
Luogu P1714 题目的大意就是给定一个长度为n的序列,求出这个序列中长度不超过m的子串的最大和 很容易想出的一个解法就是枚举起点终点,直接暴力扫一遍得出答案. 当然也很容易发现这种做法肯定会T ...
- Hive 填坑指南
Hive 填坑指南 目录 Hive 填坑指南 数据表备份 数据表备份 方法1:create table 表名_new as select * from 原表 create table 表名_new a ...
- Spiking-YOLO : 前沿性研究，脉冲神经网络在目标检测的首次尝试 | AAAI 2020
论文提出Spiking-YOLO,是脉冲神经网络在目标检测领域的首次成功尝试,实现了与卷积神经网络相当的性能,而能源消耗极低.论文内容新颖,比较前沿,推荐给大家阅读 来源:晓飞的算法工程笔记 公众 ...
- 臣妾做不到啊---记团队内首次尝试HOLACRACY管理会议
"易穷则变,变则通.通则久". --<系辞传>下篇的第二章 实践敏捷方法接近两年,瓶颈效应逐渐显现.计穷力竭之时.有幸了解到HOLACRACY.购买了市面上唯一一本中文 ...
- REmap首次尝试--PC端
在REchart项目中,实验此网址的内容,得到若干实例 针对此我针对北京市做了若干实验 第一:绘制北京市的填充地图 代码如下: >BJqu<-mapNames("beijing& ...
- python首次尝试——hello world
百忙之中抽空学习python,但是令人惊讶的是连hello world都写不出来.
- 首次尝试LINUX下的ssh命令：登录和退出
1:我现在本机安装了centos虚拟机,然后在windows桌面下使用SecureCRT ssh客户端登录我的本地虚拟机,再然后 通过centos下的ssh命令登录局域网内测试机192.168.0.1 ...
- 首次尝试Flink的一些感受
最近打算研究研究 Flink,根据官方文档写个 Hello,World.入门还是比较容易的,不需要复杂的安装环境.配置.这篇文章简单介绍 Flink 的使用感受以及入门. 感受 搭建环境方便:Flin ...
- Evolution of Image Classifiers，进化算法在神经网络结构搜索的首次尝试 | ICML 2017
论文提出使用进化算法来进行神经网络结构搜索,整体搜索逻辑十分简单,结合权重继承,搜索速度很快,从实验结果来看,搜索的网络准确率挺不错的.由于论文是个比较早期的想法,所以可以有很大的改进空间,后面的很大 ...
随机推荐
- 解决TypeError: 'NoneType' object is not subscriptable
1.捕获异常的方式try: img_list = img_list["name"]except: img_list = "" 2.对象进行判断if img_li ...
- 如何使用 Dump 文件？
引言 本文概述了使用 WinDbg 的一些必要步骤. 准备工作 第一步,你必须更改系统的配置使其能够生成 PDB 文件,包括 Release 版本.近期的 Visual C++ 编译器默认启用此配置, ...
- 手把手带你使用JWT实现单点登录
JWT(英文全名:JSON Web Token)是目前最流行的跨域身份验证解决方案之一,今天我们一起来揭开它神秘的面纱! 一.故事起源 说起 JWT,我们先来谈一谈基于传统session认证的方案以及 ...
- 使用64位Office2016处理万级数据的过程
先放下载和安装教程https://mp.weixin.qq.com/s/5ym9950_NZROlN0s2ZmLTg 由于同事电脑在使用Mysql for Excel插件处理十万级数据,如下图: 爆出 ...
- Excel 更改数据同步更新到Mysql数据库
刚上班,领导给我提出一个需求,想要每天更新Mysql数据库中的原有商品订单状态,添加新的商品订单状态.因为公司目前的数据库只能添加数据,不能更改数据,想要更改原有的数据,只能将原有的数据清空,再导入新 ...
- 如何让其他模型也能在SemanticKernel中调用本地函数
在SemanticKernel的入门例子中: // Import packages using Microsoft.SemanticKernel; using Microsoft.SemanticKe ...
- MySql常用日期时间查询
-- 某一天所在周的第一天: -- 我们知道国外的星期的第一天是从星期天开始的,所以DAYOFWEEK('2017-3-10')函数的第一天是星期天: SELECT CASE WHEN DAYNAME ...
- 2. 标签上 title 与 alt 属性的区别是什么？
1. alt 是给搜索引擎识别,在图像无法显示时的替代文本2. title 是关于元素的注释信息,主要是给用户解读.3.当鼠标放到文字或是图片上时有 title 文字显示.(因为 IE 不标准)在 I ...
- 1. 简述一下你对 HTML 语义化的理解？
用正确的标签做正确的事情.1.HTML 语义化让页面的内容结构化,结构更清晰,便于对浏览器.搜索引擎解析; 2.即使在没有样式 CSS 的情况下也能以一种文档格式显示,并且是容易阅读的; 3.搜索引擎 ...
- 很好用的SSH工具FinalShell
上图片:1.远程连接Linux 2.Linux:CentOS 3.虚拟机: