1. Dec  :: vgfs001 kernel: tiotest_AMD_x86 invoked oom-killer: gfp_mask=0x200da, order=, oom_adj=, oom_score_adj=
  2. Dec  :: vgfs001 kernel: tiotest_AMD_x86 cpuset=/ mems_allowed=
  3. Dec  :: vgfs001 kernel: Pid: , comm: tiotest_AMD_x86 Not tainted 2.6.-431.29..lustre.el6.x86_64 #
  4. Dec  :: vgfs001 kernel: Call Trace:
  5. Dec  :: vgfs001 kernel: [<ffffffff810d07b1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
  6. Dec  :: vgfs001 kernel: [<ffffffff81122b80>] ? dump_header+0x90/0x1b0
  7. Dec  :: vgfs001 kernel: [<ffffffff8122894c>] ? security_real_capable_noaudit+0x3c/0x70
  8. Dec  :: vgfs001 kernel: [<ffffffff81123002>] ? oom_kill_process+0x82/0x2a0
  9. Dec  :: vgfs001 kernel: [<ffffffff81122f41>] ? select_bad_process+0xe1/0x120
  10. Dec  :: vgfs001 kernel: [<ffffffff81123440>] ? out_of_memory+0x220/0x3c0
  11. Dec  :: vgfs001 kernel: [<ffffffff8112fd5f>] ? __alloc_pages_nodemask+0x89f/0x8d0
  12. Dec  :: vgfs001 kernel: [<ffffffff81167cea>] ? alloc_pages_current+0xaa/0x110
  13. Dec  :: vgfs001 kernel: [<ffffffff8111ff77>] ? __page_cache_alloc+0x87/0x90
  14. Dec  :: vgfs001 kernel: [<ffffffff81120c8e>] ? grab_cache_page_write_begin+0x8e/0xc0
  15. Dec  :: vgfs001 kernel: [<ffffffffa0a8f228>] ? ll_write_begin+0x58/0x1a0 [lustre]
  16. Dec  :: vgfs001 kernel: [<ffffffff811204f3>] ? generic_file_buffered_write+0x123/0x2e0
  17. Dec  :: vgfs001 kernel: [<ffffffff81078fd7>] ? current_fs_time+0x27/0x30
  18. Dec  :: vgfs001 kernel: [<ffffffff81121f50>] ? __generic_file_aio_write+0x260/0x490
  19. Dec  :: vgfs001 kernel: [<ffffffffa05211a5>] ? cl_env_info+0x15/0x20 [obdclass]
  20. Dec  :: vgfs001 kernel: [<ffffffff81122208>] ? generic_file_aio_write+0x88/0x100
  21. Dec  :: vgfs001 kernel: [<ffffffffa0aa3907>] ? vvp_io_write_start+0x137/0x2a0 [lustre]
  22. Dec  :: vgfs001 kernel: [<ffffffffa05301da>] ? cl_io_start+0x6a/0x140 [obdclass]
  23. Dec  :: vgfs001 kernel: [<ffffffffa05348e4>] ? cl_io_loop+0xb4/0x1b0 [obdclass]
  24. Dec  :: vgfs001 kernel: [<ffffffffa0a46306>] ? ll_file_io_generic+0x2a6/0x610 [lustre]
  25. Dec  :: vgfs001 kernel: [<ffffffffa0a47192>] ? ll_file_aio_write+0x142/0x2c0 [lustre]
  26. Dec  :: vgfs001 kernel: [<ffffffffa0a4747c>] ? ll_file_write+0x16c/0x2a0 [lustre]
  27. Dec  :: vgfs001 kernel: [<ffffffff81189298>] ? vfs_write+0xb8/0x1a0
  28. Dec  :: vgfs001 kernel: [<ffffffff81189c61>] ? sys_write+0x51/0x90
  29. Dec  :: vgfs001 kernel: [<ffffffff810e204e>] ? __audit_syscall_exit+0x25e/0x290
  30. Dec  :: vgfs001 kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
  31. Dec  :: vgfs001 kernel: Mem-Info:
  32. Dec  :: vgfs001 kernel: Node  DMA per-cpu:
  33. Dec  :: vgfs001 kernel: CPU    : hi:    , btch:    usd:
  34. Dec  :: vgfs001 kernel: CPU    : hi:    , btch:    usd:
  35. Dec  :: vgfs001 kernel: CPU    : hi:    , btch:    usd:
  36. Dec  :: vgfs001 kernel: CPU    : hi:    , btch:    usd:
  37. Dec  :: vgfs001 kernel: CPU    : hi:    , btch:    usd:
  38. Dec  :: vgfs001 kernel: CPU    : hi:    , btch:    usd:
  39. Dec  :: vgfs001 kernel: CPU    : hi:    , btch:    usd:
  40. Dec  :: vgfs001 kernel: CPU    : hi:    , btch:    usd:
  41. Dec  :: vgfs001 kernel: CPU    : hi:    , btch:    usd:
  42. Dec  :: vgfs001 kernel: CPU    : hi:    , btch:    usd:
  43. Dec  :: vgfs001 kernel: CPU   : hi:    , btch:    usd:
  44. Dec  :: vgfs001 kernel: CPU   : hi:    , btch:    usd:
  45. Dec  :: vgfs001 kernel: Node  DMA32 per-cpu:
  46. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  47. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  48. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  49. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  50. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  51. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  52. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  53. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  54. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  55. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  56. Dec  :: vgfs001 kernel: CPU   : hi:  , btch:   usd:
  57. Dec  :: vgfs001 kernel: CPU   : hi:  , btch:   usd:
  58. Dec  :: vgfs001 kernel: Node  Normal per-cpu:
  59. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  60. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  61. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  62. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  63. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  64. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  65. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  66. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  67. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  68. Dec  :: vgfs001 kernel: CPU    : hi:  , btch:   usd:
  69. Dec  :: vgfs001 kernel: CPU   : hi:  , btch:   usd:
  70. Dec  :: vgfs001 kernel: CPU   : hi:  , btch:   usd:
  71. Dec  :: vgfs001 kernel: active_anon: inactive_anon: isolated_anon:
  72. Dec  :: vgfs001 kernel: active_file: inactive_file: isolated_file:
  73. Dec  :: vgfs001 kernel: unevictable: dirty: writeback: unstable:
  74. Dec  :: vgfs001 kernel: free: slab_reclaimable: slab_unreclaimable:
  75. Dec  :: vgfs001 kernel: mapped: shmem: pagetables: bounce:
  76. Dec  :: vgfs001 kernel: Node  DMA free:15708kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15320kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned: all_unreclaimable? yes
  77. Dec  :: vgfs001 kernel: lowmem_reserve[]:
  78. Dec  :: vgfs001 kernel: Node  DMA32 free:53892kB min:19596kB low:24492kB high:29392kB active_anon:4kB inactive_anon:44kB active_file:1249260kB inactive_file:1249288kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3596496kB mlocked:0kB dirty:3436kB writeback:4180kB mapped:0kB shmem:0kB slab_reclaimable:24608kB slab_unreclaimable:689432kB kernel_stack:8kB pagetables:196kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned: all_unreclaimable? no
  79. Dec  :: vgfs001 kernel: lowmem_reserve[]:
  80. Dec  :: vgfs001 kernel: Node  Normal free:47588kB min:47900kB low:59872kB high:71848kB active_anon:4792020kB inactive_anon:685556kB active_file:943652kB inactive_file:944700kB unevictable:0kB isolated(anon):384kB isolated(file):0kB present:8791040kB mlocked:0kB dirty:160kB writeback:5188kB mapped:4128kB shmem:0kB slab_reclaimable:17948kB slab_unreclaimable:816972kB kernel_stack:5040kB pagetables:22256kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned: all_unreclaimable? no
  81. Dec  :: vgfs001 kernel: lowmem_reserve[]:
  82. Dec  :: vgfs001 kernel: Node  DMA: *4kB *8kB *16kB *32kB *64kB *128kB *256kB *512kB *1024kB *2048kB *4096kB = 15708kB
  83. Dec  :: vgfs001 kernel: Node  DMA32: *4kB *8kB *16kB *32kB *64kB *128kB *256kB *512kB *1024kB *2048kB *4096kB = 53892kB
  84. Dec  :: vgfs001 kernel: Node  Normal: *4kB *8kB *16kB *32kB *64kB *128kB *256kB *512kB *1024kB *2048kB *4096kB = 47084kB
  85. Dec  :: vgfs001 kernel:  total pagecache pages
  86. Dec  :: vgfs001 kernel:  pages in swap cache
  87. Dec  :: vgfs001 kernel: Swap cache stats: add , delete , find /
  88. Dec  :: vgfs001 kernel: Free swap  = 3377416kB
  89. Dec  :: vgfs001 kernel: Total swap = 4194300kB
  90. Dec  :: vgfs001 kernel:  pages RAM
  91. Dec  :: vgfs001 kernel:  pages reserved
  92. Dec  :: vgfs001 kernel:  pages shared
  93. Dec  :: vgfs001 kernel:  pages non-shared
  94. Dec  :: vgfs001 kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
  95. Dec  :: vgfs001 kernel: [  ]                             -         - udevd
  96. Dec  :: vgfs001 kernel: [ ]                           -         - auditd
  97. Dec  :: vgfs001 kernel: [ ]                                          rsyslogd
  98. Dec  :: vgfs001 kernel: [ ]                                           irqbalance
  99. Dec  :: vgfs001 kernel: [ ]                                          rpcbind
  100. Dec  :: vgfs001 kernel: [ ]                                           dbus-daemon
  101. Dec  :: vgfs001 kernel: [ ]                                          rpc.statd
  102. Dec  :: vgfs001 kernel: [ ]                                          cupsd
  103. Dec  :: vgfs001 kernel: [ ]                                            acpid
  104. Dec  :: vgfs001 kernel: [ ]                                         hald
  105. Dec  :: vgfs001 kernel: [ ]                                           hald-runner
  106. Dec  :: vgfs001 kernel: [ ]                                            hald-addon-inpu
  107. Dec  :: vgfs001 kernel: [ ]                                           hald-addon-acpi
  108. Dec  :: vgfs001 kernel: [ ]                                         automount
  109. Dec  :: vgfs001 kernel: [ ]                           -         - sshd
  110. Dec  :: vgfs001 kernel: [ ]                                          master
  111. Dec  :: vgfs001 kernel: [ ]                                        qmgr
  112. Dec  :: vgfs001 kernel: [ ]                                           abrtd
  113. Dec  :: vgfs001 kernel: [ ]                                          ksmtuned
  114. Dec  :: vgfs001 kernel: [ ]                                          crond
  115. Dec  :: vgfs001 kernel: [ ]                                            atd
  116. Dec  :: vgfs001 kernel: [ ]                                       python
  117. Dec  :: vgfs001 kernel: [ ]                                        libvirtd
  118. Dec  :: vgfs001 kernel: [ ]                                           rhsmcertd
  119. Dec  :: vgfs001 kernel: [ ]                                           dnsmasq
  120. Dec  :: vgfs001 kernel: [ ]                                           certmonger
  121. Dec  :: vgfs001 kernel: [ ]                                          gdm-binary
  122. Dec  :: vgfs001 kernel: [ ]                                            mingetty
  123. Dec  :: vgfs001 kernel: [ ]                                            mingetty
  124. Dec  :: vgfs001 kernel: [ ]                                            mingetty
  125. Dec  :: vgfs001 kernel: [ ]                                            mingetty
  126. Dec  :: vgfs001 kernel: [ ]                                            mingetty
  127. Dec  :: vgfs001 kernel: [ ]                            -         - udevd
  128. Dec  :: vgfs001 kernel: [ ]                            -         - udevd
  129. Dec  :: vgfs001 kernel: [ ]                                          gdm-simple-slav
  130. Dec  :: vgfs001 kernel: [ ]                                         Xorg
  131. Dec  :: vgfs001 kernel: [ ]                                        console-kit-dae
  132. Dec  :: vgfs001 kernel: [ ]                                           dbus-launch
  133. Dec  :: vgfs001 kernel: [ ]                                          dbus-daemon
  134. Dec  :: vgfs001 kernel: [ ]                                         gnome-session
  135. Dec  :: vgfs001 kernel: [ ]                                          devkit-power-da
  136. Dec  :: vgfs001 kernel: [ ]                                         gconfd-
  137. Dec  :: vgfs001 kernel: [ ]                                       gnome-settings-
  138. Dec  :: vgfs001 kernel: [ ]                                         at-spi-registry
  139. Dec  :: vgfs001 kernel: [ ]                                         bonobo-activati
  140. Dec  :: vgfs001 kernel: [ ]                                         gvfsd
  141. Dec  :: vgfs001 kernel: [ ]                                         metacity
  142. Dec  :: vgfs001 kernel: [ ]                                         gnome-power-man
  143. Dec  :: vgfs001 kernel: [ ]                                         polkit-gnome-au
  144. Dec  :: vgfs001 kernel: [ ]                                        gdm-simple-gree
  145. Dec  :: vgfs001 kernel: [ ]                                          polkitd
  146. Dec  :: vgfs001 kernel: [ ]                                          pulseaudio
  147. Dec  :: vgfs001 kernel: [ ]                                       rtkit-daemon
  148. Dec  :: vgfs001 kernel: [ ]                                          gdm-session-wor
  149. Dec  :: vgfs001 kernel: []                                         sshd
  150. Dec  :: vgfs001 kernel: []                                         bash
  151. Dec  :: vgfs001 kernel: [ ]                                   lwfsd
  152. Dec  :: vgfs001 kernel: [ ]                                         pickup
  153. Dec  :: vgfs001 kernel: [ ]                                          sleep
  154. Dec  :: vgfs001 kernel: [ ]                                        tiotest_AMD_x86
  155. Dec  :: vgfs001 kernel: Out of memory: Kill process  (lwfsd) score  or sacrifice child
  156. Dec  :: vgfs001 kernel: Killed process , UID , (lwfsd) total-vm:6846436kB, anon-rss:4742528kB, file-rss:20040kB

这里是从Lustre的入口导致的oom，但实际上，其他入口例如KVM管理程序也可能引起oom，即任何分配内存的可能点都可能引起oom。

从分析过程来看，确实是Lustre的Cache占用了大量内存，导致内存分配不足。

三个措施。
1、增大内存
从12GB增大到16GB。
virsh setmaxmem vgfsxxx 16GB --config
运行启动后
virsh setmem vgfsxxx 16GB
这个没有用，跑了几次测试后，仍然掉服务。

2、调整lwfsd的服务优先级
设置lwfsd的服务优先级为“-17”
PID=`ps | grep lwfs | grep -v grep | awk '{print $1}'`
echo -17 > /proc/$PID/oom_adj
echo -17 > /proc/$PID/task/$PID/oom_adj
这个好像有用。

3、修改内存分配策略
并且echo "2" >/proc/sys/vm/overcommit_memory，使得分配内存时，必须存在足够的空间用于映射。
这个好像也有一定的用处。再跑跑试试。

KVM虚拟机内存不足，调整参数的更多相关文章

  1. CentOS7中KVM虚拟机内存、CPU调整

    CentOS7中KVM虚拟机内存.CPU调整 1. 调小虚拟机内存 调小虚拟机内存可以动态实现,不用关机 1.1 查看当前内存大小 [root@kvm01 ~]# virsh dominfo vm1- ...

  2. CentOS7.6中 KVM虚拟机内存、CPU调整

    CentOS7.6中 KVM虚拟机内存.CPU调整 一.调小虚拟机内存 调小虚拟机内存可以动态实现,不用关机 1.查看当前内存大小 [root@heyong kvm]# virsh dominfo t ...

  3. KVM虚拟机管理——资源调整

    1. 概述2. 计算资源调整2.1 调整处理器配置2.2 调整内存配置3. 存储资源调整3.1 根分区扩展3.2 添加磁盘4. 网络资源调整 1. 概述 KVM在使用过程中,会涉及到计算(CPU,内存 ...

  4. Qemu创建KVM虚拟机内存初始化流程

    转载请注明:[转载自博客xelatex KVM],并附本文链接.谢谢. [注]文章中采用的版本: Linux-3.11,https://www.kernel.org/pub/linux/kernel/ ...

  5. kvm基础 虚拟机内存、CPU调整

    转自http://blog.csdn.net/hnhuangyiyang/article/details/50902223 一.调小虚拟机内存 调小虚拟机内存可以动态实现,不用关机1.查看当前内存大小 ...

  6. kvm虚拟机在线调整硬件配置

    #centos5.x版本不支持动态调整内存,CPU,以下是在centos6.x上测试 1.查看虚拟机信息 shell> virsh dumpxml cos_v1 | head -n 10 < ...

  7. KVM在线扩展虚拟机内存

    环境介绍 在KVM下有一台虚拟机内存不够需要扩展内存.宿主机地址是192.168.1.28.我需要扩展的虚拟机是centos1708vm03. 1.登陆上宿主机查看虚拟机配置 virsh dumpxm ...

  8. 阿里云下 centos7下启动程序总是被killed ,看内存占用情况以检查哪些服务存在问题并调整参数作调优

    很久不搭理自己的网站了,几天突然发现启动程序总是被killed, 于是查看了系统日志 vi /var/log/messages 发现出现 kernel: Out of memory: Kill pro ...

  9. 调整KVM虚拟机硬盘大小

    KVM虚拟机的硬盘映像默认存放在“/var/lib/libvirt/images/”中,先查看你的硬盘映像格式是不是RAW格式: qemu-img info /var/lib/libvirt/imag ...

随机推荐

  1. Codeforces Round #389 Div.2 E. Santa Claus and Tangerines

    time limit per test 2 seconds memory limit per test 256 megabytes input standard input output standa ...

  2. C/C++ Lua Parsing Engine

    catalog . Lua语言简介 . 使用 Lua 编写可嵌入式脚本 . VS2010编译Lua . 嵌入和扩展: C/C++中执行Lua脚本 . 将C++函数导出到Lua引擎中: 在Lua脚本中执 ...

  3. CSS 命名管理 之 BEM

    好吧,将 BEM 简单的解释为 “Block-Element-Modifier“, 其实是个不负责任的做法.鬼知道 Block 是什么啊?所以,看了一些似懂非懂的中文解释之后,自己还是得去找些英文来读 ...

  4. UVA1555-- Garland(推导+二分)

    题意:有n个灯,给定第一盏灯A的高度,接下去每盏灯的高度按照公式计算,求使所有灯都不会落在地上(允许碰触)的B的最低高度. uva 输出 double 用 %f,这一波坑的! #include < ...

  5. 重新认识unicode和utf8编码

    重新认识unicode和utf8编码 直到今天,准确的说是刚才,我才知道UTF-8编码和Unicode编码是不一样的,是有区别的囧 他们之间是有一定的联系的,看看他们的区别: UTF-8的长度是不一定 ...

  6. OpenGLES入门笔记三

    在入门笔记一中比较详细的介绍了顶点着色器和片面着色器. 在入门笔记二中讲解了简单的创建OpenGL场景流程的实现,但是如果在场景中渲染任何一种几何图形,还是需要入门笔记一中的知识:Vertex Sha ...

  7. 【项目】'NSRangeException', reason: '*** -[__NSArrayM removeObjectAtIndex:]: index 2 beyond bounds [0 .. 1]'

    问题代码: [self.assetsArray objectAtIndex:indexPath.row] 问题解决思路:这里

  8. BZOJ1798: [Ahoi2009]Seq 维护序列seq

    传送门 写这道题是为了get一个同时传送乘法下标和加法下标的小技巧,线段树模板题.不多说. 标记名字打错无限智力-- //BZOJ 1798 //by Cydiater //2016.9.13 #in ...

  9. zabbix 3.0快速安装简介（centos 6）

    zabbix快速安装 系统版本:centos 6 1.yum源配置和zabbix.msyql安装 rpm -ivh http://mirrors.aliyun.com/zabbix/zabbix/3. ...

  10. cobbler深入学习

    cobbler重要目录和cobbler各对象的关系 /var/www/cobbler/ks_mirror 存放操作系统镜像/var/www/cobbler/repo_mirror 存放仓库镜像/var ...