【原创】大叔问题定位分享(22)hive同时执行多个insert overwrite table只有1个可以执行
hive 2.1
一 问题
最近有一个场景,要向一个表的多个分区写数据,为了缩短执行时间,采用并发的方式,多个sql同时执行,分别写不同的分区,同时开启动态分区:
set hive.exec.dynamic.partition=true
insert overwrite table test_table partition(dt) select * from test_table_another where dt = 1;
结果发现只有1个sql运行,其他sql都会卡住;
查看hive thrift server线程堆栈发现请求都卡在DbTxnManager上,hive关键配置如下:
hive.support.concurrency=true
hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
配置对应的默认值及注释:
org.apache.hadoop.hive.conf.HiveConf
- HIVE_SUPPORT_CONCURRENCY("hive.support.concurrency", false,
- "Whether Hive supports concurrency control or not. \n" +
- "A ZooKeeper instance must be up and running when using zookeeper Hive lock manager "),
- HIVE_TXN_MANAGER("hive.txn.manager",
- "org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager",
- "Set to org.apache.hadoop.hive.ql.lockmgr.DbTxnManager as part of turning on Hive\n" +
- "transactions, which also requires appropriate settings for hive.compactor.initiator.on,\n" +
- "hive.compactor.worker.threads, hive.support.concurrency (true), hive.enforce.bucketing\n" +
- "(true), and hive.exec.dynamic.partition.mode (nonstrict).\n" +
- "The default DummyTxnManager replicates pre-Hive-0.13 behavior and provides\n" +
- "no transactions."),
二 代码分析
hive执行sql的详细过程详见:https://www.cnblogs.com/barneywill/p/10185168.html
hive中执行sql最终都会调用到Driver.run,run会调用runInternal,下面直接看runInternal代码:
org.apache.hadoop.hive.ql.Driver
- private CommandProcessorResponse runInternal(String command, boolean alreadyCompiled)
- throws CommandNeedRetryException {
- ...
- if (requiresLock()) {
- // a checkpoint to see if the thread is interrupted or not before an expensive operation
- if (isInterrupted()) {
- ret = handleInterruption("at acquiring the lock.");
- } else {
- ret = acquireLocksAndOpenTxn(startTxnImplicitly);
- }
- ...
- private boolean requiresLock() {
- if (!checkConcurrency()) {
- return false;
- }
- // Lock operations themselves don't require the lock.
- if (isExplicitLockOperation()){
- return false;
- }
- if (!HiveConf.getBoolVar(conf, ConfVars.HIVE_LOCK_MAPRED_ONLY)) {
- return true;
- }
- Queue<Task<? extends Serializable>> taskQueue = new LinkedList<Task<? extends Serializable>>();
- taskQueue.addAll(plan.getRootTasks());
- while (taskQueue.peek() != null) {
- Task<? extends Serializable> tsk = taskQueue.remove();
- if (tsk.requireLock()) {
- return true;
- }
- ...
- private boolean checkConcurrency() {
- boolean supportConcurrency = conf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY);
- if (!supportConcurrency) {
- LOG.info("Concurrency mode is disabled, not creating a lock manager");
- return false;
- }
- return true;
- }
- private int acquireLocksAndOpenTxn(boolean startTxnImplicitly) {
- ...
- txnMgr.acquireLocks(plan, ctx, userFromUGI);
- ...
runInternal会调用requiresLock判断是否需要lock,requiresLock有两个判断:
- 调用checkConcurrency,checkConcurrency会检查hive.support.concurrency=true才需要lock;
- 调用Task.requireLock,只有部分task才需要lock;
如果判断需要lock,会调用acquireLocksAndOpenTxn,acquireLocksAndOpenTxn会调用HiveTxnManager.acquireLocks来获取lock;
1)先看那些task需要lock:
org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer
- private void analyzeAlterTablePartMergeFiles(ASTNode ast,
- String tableName, HashMap<String, String> partSpec)
- throws SemanticException {
- ...
- DDLWork ddlWork = new DDLWork(getInputs(), getOutputs(), mergeDesc);
- ddlWork.setNeedLock(true);
- ...
可见DDL操作需要;
2)再看怎样获取lock:
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
- public void acquireLocks(QueryPlan plan, Context ctx, String username) throws LockException {
- try {
- acquireLocksWithHeartbeatDelay(plan, ctx, username, 0);
- ...
- void acquireLocksWithHeartbeatDelay(QueryPlan plan, Context ctx, String username, long delay) throws LockException {
- LockState ls = acquireLocks(plan, ctx, username, true);
- ...
- LockState acquireLocks(QueryPlan plan, Context ctx, String username, boolean isBlocking) throws LockException {
- ...
- switch (output.getType()) {
- case DATABASE:
- compBuilder.setDbName(output.getDatabase().getName());
- break;
- case TABLE:
- case DUMMYPARTITION: // in case of dynamic partitioning lock the table
- t = output.getTable();
- compBuilder.setDbName(t.getDbName());
- compBuilder.setTableName(t.getTableName());
- break;
- case PARTITION:
- compBuilder.setPartitionName(output.getPartition().getName());
- t = output.getPartition().getTable();
- compBuilder.setDbName(t.getDbName());
- compBuilder.setTableName(t.getTableName());
- break;
- default:
- // This is a file or something we don't hold locks for.
- continue;
- }
- ...
- LockState lockState = lockMgr.lock(rqstBuilder.build(), queryId, isBlocking, locks);
- ctx.setHiveLocks(locks);
- return lockState;
- }
可见当开启动态分区时,锁的粒度是DbName+TableName,这样就会导致多个sql只有1个sql可以拿到lock,其他sql只能等待;
三 总结
解决问题的方式有几种:
- 关闭动态分区:set hive.exec.dynamic.partition=false
- 关闭并发:set hive.support.concurrency=false
- 关闭事务:set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
三者任选其一,推荐第1种,因为在刚才的场景下,不需要动态分区;
【原创】大叔问题定位分享(22)hive同时执行多个insert overwrite table只有1个可以执行的更多相关文章
- 【原创】大叔问题定位分享(21)spark执行insert overwrite非常慢,比hive还要慢
最近把一些sql执行从hive改到spark,发现执行更慢,sql主要是一些insert overwrite操作,从执行计划看到,用到InsertIntoHiveTable spark-sql> ...
- 【原创】大叔问题定位分享(15)spark写parquet数据报错ParquetEncodingException: empty fields are illegal, the field should be ommited completely instead
spark 2.1.1 spark里执行sql报错 insert overwrite table test_parquet_table select * from dummy 报错如下: org.ap ...
- hive INSERT OVERWRITE table could not be cleaned up.
create table maats.account_channel ROW FORMAT DELIMITED FIELDS TERMINATED BY '^' STORED AS TEXTFILE ...
- 【原创】大叔问题定位分享(18)beeline连接spark thrift有时会卡住
spark 2.1.1 beeline连接spark thrift之后,执行use database有时会卡住,而use database 在server端对应的是 setCurrentDatabas ...
- 【原创】大叔问题定位分享(16)spark写数据到hive外部表报错ClassCastException: org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat cannot be cast to org.apache.hadoop.hive.ql.io.HiveOutputFormat
spark 2.1.1 spark在写数据到hive外部表(底层数据在hbase中)时会报错 Caused by: java.lang.ClassCastException: org.apache.h ...
- 【原创】大叔问题定位分享(31)hive metastore报错
hive metastore在建表时报错 [pool-5-thread-2]: MetaException(message:Got exception: java.net.ConnectExcepti ...
- 【原创】大叔问题定位分享(13)HBase Region频繁下线
问题现象:hive执行sql报错 select count(*) from test_hive_table; 报错 Error: java.io.IOException: org.apache.had ...
- 【原创】大叔问题定位分享(30)mesos agent启动失败:Failed to perform recovery: Incompatible agent info detected
mesos agent启动失败,报错如下: Feb 15 22:03:18 server1.bj mesos-slave[1190]: E0215 22:03:18.622994 1192 slave ...
- 【原创】大叔问题定位分享(28)openssh升级到7.4之后ssh跳转异常
服务器集群之间忽然ssh跳转不通 # ssh 192.168.0.1The authenticity of host '192.168.0.1 (192.168.0.1)' can't be esta ...
随机推荐
- Leetcode 88. Merge Sorted Array(easy)
Given two sorted integer arrays nums1 and nums2, merge nums2 into nums1 as one sorted array. Note:Yo ...
- 原生js设置rem
使用rem是为了界面响应不同尺寸的手机,引入下面的方法就可以使用rem了. setFontSize: function (doc, win) { var docEl = doc.documentEle ...
- [转帖] CA如何保护自己的私钥
作者:Gh0u1L5链接:https://www.zhihu.com/question/22260090/answer/648910720来源:知乎著作权归作者所有.商业转载请联系作者获得授权,非商业 ...
- [转帖]Linux中的15个基本‘ls’命令示例
Linux中的15个基本‘ls’命令示例 https://linux.cn/article-5109-1.html ls -lt 和 ls -ltr 来查看文件新旧顺序. list time rese ...
- springboot 打war
pom.xml <packaging>war</packaging> <!-- 打包设置 --> <plugins> <plugin> &l ...
- js和jquery设置css样式的几种方法
一.js设置样式的方法 1. 直接设置style的属性 某些情况用这个设置 !important值无效 element.style.height = '50px'; 2. 直接设置属性(只能用于某些 ...
- Magento 2 安装数据表
Magento 2 安装数据表 第1步:安装脚本 首先,我们将为CRUD模型创建数据库表.为此,我们需要插入安装文件 app/code/Mageplaza/HelloWorld/Setup/Insta ...
- 安卓微信连接fiddler等抓包工具无法抓取https
问题描述: 在手机连接pc的fiddler后,安卓微信打不开https页面,安卓的浏览器.qq等都可以正常访问https,ios也都可以,就只有安卓微信放问https是空白页面 解决思路: 一. 证书 ...
- GateOne Web SSH 环境搭建
环境配置安装python及tornadoyum -y install python-pippip install tornado GateOne安装下载源码:git clone https://git ...
- linux线程(一)
线程的优先级无法保障线程的执行次序.只不过优先级高的线程获取 CPU 资源的概率大一点而已. 线程相关函数(1)-pthread_create(), pthread_join(), pthread_e ...