原因分析

CDH 集群环境没有对 Container分配足够的运行环境（内存）

解决办法

需要修改的配置文件，将具体的配置项修改匹配集群环境资源。如下：
配置文件
配置设置
解释
计算值（参考）
yarn-site.xml
yarn.nodemanager.resource.memory-mb
分配给容器的物理内存数量
= 52 * 2 =104 G
yarn-site.xml
yarn.scheduler.minimum-allocation-mb
容器可以请求的最小物理内存量（以 MiB 为单位）
= 2G
yarn-site.xml
yarn.scheduler.maximum-allocation-mb
为容器请求的最大物理内存数量（以 MiB 为单位）。
= 52 * 2 = 104G
yarn-site.xml (check)
yarn.app.mapreduce.am.resource.mb
ApplicationMaster 的物理内存要求 (MiB)。
= 2 * 2=4G
yarn-site.xml (check)
yarn.app.mapreduce.am.command-opts
传递到 MapReduce ApplicationMaster 的 Java 命令行参数
= 0.8 * 2 * 2=3.2G
yarn-site.xml
yarn.nodemanager.vmem-pmem-ratio
容器内存限制时虚拟内存与物理内存的比率
默认是2.1，根据实际情况调整这个配置项的值
mapred-site.xml
mapreduce.map.memory.mb
为作业的每个 Map 任务分配的物理内存量(MiB)。
= 2G
mapred-site.xml
mapreduce.reduce.memory.mb
为作业的每个 Reduce 任务分配的物理内存量(MiB)。
= 2 * 2=4G
mapred-site.xml
mapreduce.map.java.opts
Map 进程的 Java 选项。
= 0.8 * 2=1.6G
mapred-site.xml
mapreduce.reduce.java.opts
Reduce 进程的 Java 选项。
= 0.8 * 2 * 2=3.2G

异常日志

'PHYSICAL' memory limit. Current usage: 2.1 GB of 2 GB physical memory used; 21.2 GB of 4.2 GB virtual memory used. Killing container.
 
Application application_1543392650432_0855 failed 2 times due to AM Container for appattempt_1543392650432_0855_000002 exited with exitCode: -104
Failing this attempt.Diagnostics: [2018-12-01 14:57:17.762]Container [pid=31682,containerID=container_1543392650432_0855_02_000001] is running 120156160B beyond the 'PHYSICAL' memory limit. Current usage: 2.1 GB of 2 GB physical memory used; 21.2 GB of 4.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_1543392650432_0855_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 1080 31768 31682 31682 (java) 2769 194 3968139264 299128 /usr/java/jdk1.8.0_141-cloudera/bin/java -Dproc_jar -Djava.net.preferIPv4Stack=true -Xmx2147483648 -Djava.net.preferIPv4Stack=true -Dlog4j.configurationFile=hive-log4j2.properties -Dlog4j.configurationFile=hive-log4j2.properties -Djava.util.logging.config.file=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/bin/../lib/hive/bin/../conf/parquet-logging.properties -Dyarn.log.dir=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/logs -Dyarn.log.file=hadoop.log -Dyarn.home.dir=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop-yarn -Dyarn.root.logger=INFO,console -Djava.library.path=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/lib/native -Dhadoop.log.dir=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop -Dhadoop.id.str=chenweidong -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender -Xmx2147483648 -Djava.net.preferIPv4Stack=true -Dlog4j.configurationFile=hive-log4j2.properties -Dlog4j.configurationFile=hive-log4j2.properties -Djava.util.logging.config.file=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/bin/../lib/hive/bin/../conf/parquet-logging.properties org.apache.hadoop.util.RunJar /opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/jars/hive-exec-2.1.1-cdh6.0.1.jar org.apache.hadoop.hive.ql.exec.mr.ExecDriver -libjars file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-hadoop2-compat.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-client.jar,file:/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hive/auxlib/hive-exec-2.1.1-cdh6.0.1-core.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-hadoop-compat.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-server.jar,file:/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hive/auxlib/hive-exec-core.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-protocol.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/lib/htrace-core.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-common.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hive/lib/hive-hbase-handler-2.1.1-cdh6.0.1.jar -localtask -plan file:/tmp/yarn/cfb1d927-a086-4b93-af4b-9816f2dc9f49/hive_2018-12-01_14-56-08_101_7346185201514786308-1/-local-10010/plan.xml -jobconffile file:/tmp/yarn/cfb1d927-a086-4b93-af4b-9816f2dc9f49/hive_2018-12-01_14-56-08_101_7346185201514786308-1/-local-10011/jobconf.xml
|- 31682 31680 31682 31682 (bash) 0 0 11960320 344 /bin/bash -c /usr/java/jdk1.8.0_141-cloudera/bin/java -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true -Dyarn.app.container.log.dir=/yarn/container-logs/application_1543392650432_0855/container_1543392650432_0855_02_000001 -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Dsubmitter.user=chenweidong org.apache.oozie.action.hadoop.LauncherAM 1>/yarn/container-logs/application_1543392650432_0855/container_1543392650432_0855_02_000001/stdout 2>/yarn/container-logs/application_1543392650432_0855/container_1543392650432_0855_02_000001/stderr
|- 31689 31682 31682 31682 (java) 355 28 14790037504 76787 /usr/java/jdk1.8.0_141-cloudera/bin/java -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true -Dyarn.app.container.log.dir=/yarn/container-logs/application_1543392650432_0855/container_1543392650432_0855_02_000001 -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Dsubmitter.user=chenweidong org.apache.oozie.action.hadoop.LauncherAM
|- 31768 31756 31682 31682 (java) 1750 114 4003151872 176993 /usr/java/jdk1.8.0_141-cloudera/bin/java -Dproc_jar -Djava.net.preferIPv4Stack=true -Xmx2147483648 -Djava.net.preferIPv4Stack=true -Dlog4j.configurationFile=hive-log4j2.properties -Dlog4j.configurationFile=hive-log4j2.properties -Djava.util.logging.config.file=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/bin/../lib/hive/bin/../conf/parquet-logging.properties -Dyarn.log.dir=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/logs -Dyarn.log.file=hadoop.log -Dyarn.home.dir=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop-yarn -Dyarn.root.logger=INFO,console -Djava.library.path=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/lib/native -Dhadoop.log.dir=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop -Dhadoop.id.str=chenweidong -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/bin/../lib/hive/lib/hive-cli-2.1.1-cdh6.0.1.jar org.apache.hadoop.hive.cli.CliDriver --hiveconf hive.query.redaction.rules=/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/bin/../lib/hive/conf/redaction-rules.json --hiveconf hive.exec.query.redactor.hooks=org.cloudera.hadoop.hive.ql.hooks.QueryRedactor --hiveconf hive.aux.jars.path=file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hive/lib/hive-hbase-handler-2.1.1-cdh6.0.1.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-hadoop-compat.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-hadoop2-compat.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-server.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/lib/htrace-core.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-protocol.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-common.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hbase/hbase-client.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/bin/../lib/hive/auxlib/hive-exec-2.1.1-cdh6.0.1-core.jar,file:///opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/bin/../lib/hive/auxlib/hive-exec-core.jar -S -v -e
|- 31756 31689 31682 31682 (initialization_) 0 0 11960320 371 /bin/bash ./initialization_data_step2.sh 20181123 20181129 dwp_order_log_process
 
[2018-12-01 14:57:17.770]Container killed on request. Exit code is 143
[2018-12-01 14:57:17.778]Container exited with a non-zero exit code 143.
For more detailed output, check the application tracking page: https://master.prodcdh.com:8090/cluster/app/application_1543392650432_0855 Then click on links to logs of each attempt.
. Failing the application.

引申参考

https://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits

https://yq.aliyun.com/articles/25470

troubleshooting-Container 'PHYSICAL' memory limit的更多相关文章

  1. hadoop的job执行在yarn中内存分配调节————Container [pid=108284,containerID=container_e19_1533108188813_12125_01_000002] is running beyond virtual memory limits. Current usage: 653.1 MB of 2 GB physical memory used

    实际遇到的真实问题,解决方法: 1.调整虚拟内存率yarn.nodemanager.vmem-pmem-ratio (这个hadoop默认是2.1) 2.调整map与reduce的在AM中的大小大于y ...

  2. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十三）kafka+spark streaming打包好的程序提交时提示虚拟内存不足（Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 G）

    异常问题:Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical mem ...

  3. spark运行任务报错：Container [...] is running beyond physical memory limits. Current usage: 3.0 GB of 3 GB physical memory used; 5.0 GB of 6.3 GB virtual memory used. Killing container.

    spark版本:1.6.0 scala版本:2.10 报错日志: Application application_1562341921664_2123 failed 2 times due to AM ...

  4. is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 2.6 GB of 40 GB virtual memory used

    昨天使用hadoop跑五一的数据,发现报错: Container [pid=,containerID=container_1453101066555_4130018_01_000067] GB phy ...

  5. hive： insert数据时Error during job, obtaining debugging information 以及beyond physical memory limits

    insert overwrite table canal_amt1...... 2014-10-09 10:40:27,368 Stage-1 map = 100%, reduce = 32%, Cu ...

  6. 运行hadoop的时候提示物理内存或虚拟内存溢出的解决方案running beyond physical memory或者beyond vitual memory limits

    当运行中出现Container is running beyond physical memory这个问题出现主要是因为物理内存不足导致的,在执行mapreduce的时候,每个map和reduce都有 ...

  7. Hive-Container killed by YARN for exceeding memory limits. 9.2 GB of 9 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.

    Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task times, most recen ...

  8. Min Stack (LeetCode) tweak it to avoid Memory Limit Exceeded

    class MinStack { public: void push(int x) { if(values.empty()) { values.push_back(x); min_indices.pu ...

  9. 1.Zabbix报错信息：It probably means that the systems requires more physical memory.

    点击返回:自学Zabbix之路 1.Zabbix报错信息:It probably means that the systems requires more physical memory. 1.报错信 ...

随机推荐

  1. ubuntu下绝对路径和相对路径的问题

     绝对路径与相对路径 除了需要特别注意的FHS目录配置外,在文件名部分我们也要特别注意.因为根据档名写法的不同,也可将所谓的路径(path)定义为绝对路径(absolute)与相对路径(relativ ...

  2. import tkinter与from tkinter import *的区别

    from tkinter import * class DirList(object): def __init__(self,initdir=None): self.top= Tk() ##可以不加t ...

  3. HDU 3746 - Cyclic Nacklace　&　HDU 1358 - Period　-　[KMP求最小循环节]

    题目链接:http://acm.hdu.edu.cn/showproblem.php?pid=3746 Time Limit: 2000/1000 MS (Java/Others) Memory Li ...

  4. pyAudio介绍

    概要 pyaudio有这么几个功能: - 提取特征 - 训练并且使用分类器 - 语音分割功能 - 内容关系可视化 python实现,好处有这么几个 - 适合做计算分析类型操作(编码少,效率不低) - ...

  5. Spark案例分析

    一.需求:计算网页访问量前三名 import org.apache.spark.rdd.RDD import org.apache.spark.{SparkConf, SparkContext} /* ...

  6. 开机自启：bat实现一次性打开win7中的常用软件和文件夹

    需求说明: 我们电脑(windows)办公,经常上班前一开机,就要手动打开很多文件夹和程序. 想节省时间,一键打开常用的文件夹和程序,研究了一下bat命令,于是mystart.bat产生了. myst ...

  7. python float()

    1.函数功能将一个数值或者字符转换成浮点型数值 >>> a = 12345 >>> amount = float(a) >>> print(amo ...

  8. vue使用resource传参数

    <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8&quo ...

  9. ubuntu vim python配置

    参考https://www.cnblogs.com/cjy15639731813/p/5886158.html 但是后面打开文件的时候会报错,参考https://blog.csdn.net/jeff_ ...

  10. The Air Jordan 11 Gym Red will be available December 9

    A few years ago Carmelo Anthony set the internet on fire when he was spotted rocking a never before ...