报错栈:

-- ::, INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select "CTJX60","CTJX61","CTJX62","CTJX63","CTJX64","CTJX65","CTJX66","CTJX67","CTJX68","CTJX69","CTJX70","CTJX71","CTJX72","CTJX73","CTJX74","CTJX75","CTJX76","CTJX77","CTJX78","CTJX79","CTJX80","CTJX81","CTJX82","CTJX83","CTJX84","CTJX85","CTJX86","CTJX87","CTJX88","CTJX89","CTJX90","CTJX91","CTJX92","CTJX93","CTJX94","CTJX95","CTJX96","CTJX97","CTJX98","CTJX99","CTJX55","IID","IEXAM_IID","CTSBGLX","CTJX01","CTJX02","CTJX03","CTJX04","CTJX05","CTJX06","CTJX07","CTJX08","CTJX09","CTJX10","CTJX11","CTJX12","CTJX13","CTJX14","CTJX15","CTJX16","CTJX17","CTJX18","CTJX19","CTJX20","CTJX21","CTJX22","CTJX23","CTJX24","CTJX25","CTJX26","CTJX27","CTJX28","CTJX29","CTJX30","CTJX31","CTJX32","CTJX33","CTJX34","CTJX35","CTJX36","CTJX37","CTJX38","CTJX39","CTJX40","CTJX41","CTJX42","CTJX43","CTJX44","CTJX45","CTJX46","CTJX47","CTJX48","CTJX49","CTJX50","CTJX51","CTJX52","CTJX53","CTJX54","CTJX56","CTJX57","CTJX58","CTJX59" from RIS."ICNRIS_UIS_EXAM_TJX" tbl where ( = ) AND ( = )
-- ::, INFO [Thread-] org.apache.sqoop.mapreduce.AutoProgressMapper: Auto-progress thread is finished. keepGoing=false
-- ::, FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java:)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:)
at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:)
at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:)
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:)
at org.apache.hadoop.mapred.YarnChild$.run(YarnChild.java:)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:)
解决:调小sqoop参数:--fetch-size
解决过程,查看sqoop源码,看到fetchSize,想到调整这个参数:

ps: 调大mapper参数不管用,曾设置过参数:-D mapreduce.map.memory.mb=8192 -D yarn.app.mapreduce.am.resource.mb=6144。问题是在从db fetch数据的时候出现的,原表的列不算多,但每个列的长度很大:

 

解决sqoop报错:java.lang.OutOfMemoryError: Java heap space的更多相关文章

  1. 排查sqoop报错:Error running child : java.lang.OutOfMemoryError: Java heap space

    报错栈: -- ::, INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: = AND = -- ::, INFO [mai ...

  2. 【Xamarin报错】 COMPILETODALVIK : UNEXPECTED TOP-LEVEL error java.lang.OutOfMemoryError: Java heap space

    Xamarin Android 编译报错: COMPILETODALVIK : UNEXPECTED TOP-LEVEL error java.lang.OutOfMemoryError: Java ...

  3. Tomcat报java.lang.OutOfMemoryError: Java heap space错误停止运行如何解决

    最近开发的一个商业项目,部署完成后,经常出现Tomcat挂掉的现象,报的异常是:java.lang.OutOfMemoryError: Java heap space,上网google了一下,了解了一 ...

  4. Tomcat 启动项目报错 java.lang.OutOfMemoryError: Java heap space

    近日使用myeclipse 部署web项目,启动tomcat时报错: SEVERE: Error waiting for multi-thread deployment of directories ...

  5. Linux运行Java出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错

    在运行如下程序时出现“Exception in thread "main" java.lang.OutOfMemoryError: Java heap space”报错: java ...

  6. eclipse内存溢出报错:java.lang.OutOfMemoryError:Java heap space

    今天执行了一个比較大的程序,处理的数据达到126MB数据,将数据导入数据库中,用eclipse 来訪问时候,总是出现java.lang.OutOfMemoryError:Java heap space ...

  7. mycat 报错 java.lang.OutOfMemoryError: Java heap space

    今天排查mysql的错误日志发现  wrapper.log  中有如下错误日志 INFO   | jvm 1    | 2019/10/20 12:52:31 | java.lang.OutOfMem ...

  8. 解决eclipse maven install 造成JVM 内存溢出(java.lang.OutOfMemoryError: Java heap space)

    maven install 报错信息: The system is out of resources.Consult the following stack trace for details.jav ...

  9. JMeter内存溢出:java.lang.OutOfMemoryError: Java heap space解决方法

    一.问题原因 用JMeter压测,有时候当模拟并发请求较大或者脚本运行时间较长时,JMeter会停止,报OOM(内存溢出)错误. 原因是JMeter是一个纯Java开发的工具,内存由java虚拟机JV ...

随机推荐

  1. Java学习(Map接口)

    一.概述: 我们通过查看Map接口描述,发现Map接口下的集合与Collection接口下的集合,它们存储数据的形式不同,如下图. 1. Collection中的集合,元素是孤立存在的(理解为单身), ...

  2. 【LOJ】#2538. 「PKUWC2018」Slay the Spire

    题解 由于强化卡都是大于1的,我们分析一下就会发现,尽可能多的用强化卡,至少用一张攻击卡,一定是每组卡牌的最优选择 所以我们把攻击卡和强化卡从大到小排序 我们设\(g[i][j]\)表示前i张卡牌里选 ...

  3. Django实战(5):引入bootstrap,设置静态资源

    之前生成了Product类的scaffold,但是如同rails的开发者David所讲的那样,scaffold几乎没什么用.所以按照<Agile Web Development with Rai ...

  4. 8-3 4Values Whose Sum is Zero 和为0的四个值

    给定四个n元素集合 ABCD   要求分别从中取一个元素 abcd   使得他们的合为0   问有多少中取法 map果然炸了 #include<bits/stdc++.h> using n ...

  5. 洛谷——P1231 教辅的组成

    P1231 教辅的组成 题目背景 滚粗了的HansBug在收拾旧语文书,然而他发现了什么奇妙的东西. 题目描述 蒟蒻HansBug在一本语文书里面发现了一本答案,然而他却明明记得这书应该还包含一份练习 ...

  6. 第一次用python,成功的感觉不错。

    自己的作业: 1. count = 0 while count <= 9 : count += 1 if count == 7 : continue print (count) 2. count ...

  7. luoguP4492 [HAOI2018]苹果树 组合计数 + dp

    首先,每个二叉树对应着唯一的中序遍历,并且每个二叉树的概率是相同的 这十分的有用 考虑\(dp\)求解 令\(f_i\)表示\(i\)个节点的子树,根的深度为\(1\)时,所有点的期望深度之和(乘\( ...

  8. 【初识】KMP算法入门

    举个例子 模式串S:a s d a s d a s d f a s d 匹配串T:a s d a s d f 如果使用朴素匹配算法—— 1 2 3 4 5 6  8 9 a s d a s d a s ...

  9. java的反射机制(第三篇)

    本文转载自:http://c.biancheng.net/cpp/html/1782.html Person p=new Person();这是什么?当然是实例化一个对象了.可是这种实例化对象的方法存 ...

  10. SpringBoot静态资源目录

    在web开发中,静态资源的访问是必不可少的,如:图片.js.css 等资源的访问. SpringBoot对静态资源访问提供了很好的支持,基本使用默认配置就能满足开发需求. 在传统的web项目中静态资源 ...