如何手动触发物理机panic，并产生vmcore

如何手动触发物理机panic，并产生vmcore?

1. 配置kdump

1.1 el6

如果是CentOS 6 则编辑/boot/grub/grub.conf配置在内核参数中添加 crashkernel=auto 类似如下

kernel /vmlinuz-2.6.32-xxx.el6.x86_64 ro root=LABEL=/ crashkernel=auto ...

1.2 el7

如果是CentOS 7 则编辑/etc/default/grub修改GRUB_CMDLINE_LINUX行添加crashkernel=auto类似如下

GRUB_CMDLINE_LINUX="crashkernel=auto ..."

1.3 只要修改/etc/default/grub修改GRUB_CMDLINE_LINUX行 ,就需要重新生成grub配置：

grub2-mkconfig -o /boot/grub2/grub.cfg

如果你遇到服务器极不稳定，需要在系统hung住时候立即crash掉系统生成kernel core dump，则需要使用NMI watchdog，则需要在内核参数中再加上 nmi_watchdog=1 激活watchdog（这样就不需要时时盯着服务器来手工触发core）

重启服务器使得以上配置生效

2. kdump涉及的sysctl 配置

查阅了网上很多有关kdump的资料，发现在配置kdump时，对sysctl.conf 内的一些配置也进行了调整。这里也列举下，可以根据具体的情况酌情进行修改。

如下参数也都可以在/etc/default/grub修改GRUB_CMDLINE_LINUX行添加

kernel.sysrq=1
kernel.unknown_nmi_panic=1
kernel.softlockup_panic=1

2.1 sysrq

kernel.sysrq=1，如果通过/proc文件配置 ，上面的配置等价于echo 1 > /proc/sys/kernel/sysrq .

默认SysRQ(/proc/sys/kernel/sysrq)设置值是16. 修改这个值为1激活SysRq来触发core dump

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

此时在带外可以看到

[1290981.013642] SysRq : Trigger a crash
[1290981.018405] BUG: unable to handle kernel NULL pointer dereference at           (null)
[1290981.028007] IP: [<ffffffff813ed756>] sysrq_handle_crash+0x16/0x20

然后切换到kdump内核并进行 vmcore 存储

         Starting Kdump Vmcore Save Service...
[    7.200189] BTRFS info (device sda4): disk space caching is enabled

kdump: saving to /kdumproot/data//127.0.0.1-2017-01-18-23:33:42/
kdump: saving vmcore-dmesg.txt
kdump: saving vmcore-dmesg.txt complete
kdump: saving vmcore
Copying data                       : [100.0 %] \
kdump: saving vmcore complete

打开sysrq键的功能以后，有终端访问权限的用户将会拥有一些特别的功能。如果系统出现挂起的情况或在诊断一些和内核相关，

使用这些组合键能即时打印出内核的信息。

因此，除非是要调试，解决问题，一般情况下，不要打开此功能。如果一定要打开，请确保你的终端访问的安全性。

2.2 kernel.unknown_nmi_panic

kernel.unknown_nmi_panic=1 ,如果系统已经是处在Hang的状态的话，那么可以使用NMI按钮来触发Kdump。

开启这个选项可以：echo 1 > /proc/sys/kernel/unknown_nmi_panic 需要注意的是，启用这个特性的话，是不能够同时启用NMI_WATCHDOG的！否则系统会Panic！

2.3 kernel.softlockup_panic

kernel.softlockup_panic=1,其对应的是/proc/sys/kernel/softlockup_panic的值，值为1可以让内核在死锁或者死循环的时候可以宕机重启。如果你的机器中安装了kdump，在重启之后，你会得到一份内核的core文件，这时从core文件中查找问题就方便很多了，而且再也不用手动重启机器了。如果你的内核是标准内核的话，可以通过修改/proc/sys/kernel/softlockup_thresh来修改超时的阈值，如果是CentOS内核的话，对应的文件是/proc/sys/kernel/watchdog_thresh。

2.4 nmi watchdog

2.4.1 编译支持

在很多x86/x86-64类型的硬件中提供了一个功能可以激活watchdog NMI interrupts（NMI：不可屏蔽中断在系统非常困难时依然可以执行）。这个功能可以用来debug内核异常。通过周期性执行MNI中断，内核可以见识任何CPU思索并且打印出相应的debug信息。

为了使用NMI watchdog，你需要在内核激活APIC支持。对于SMP内核，APIC支持已经自动编译进内核。即在内核配置中：

CONFIG_X86_UP_APIC (Processor type and features -> Local APIC support on uniprocessors) 

或

CONFIG_X86_UP_IOAPIC (Processor type and features -> IO-APIC support on uniprocessors) 

2.4.2 开启

在 x86 平台，nmi_watchdog默认是关闭的，你需要在启动参数中激活它。

动态修改：

也可以通过对/proc/sys/kernel/nmi_watchdog写入0来在运行时关闭NMI watchdog。而写入1这个文件则可以重新激活NMI watchdog。

永久修改：

需要在启动时使用nmi_watchdog=X参数来激活NMI watchdog，否则无法动态修改。

#grep NMI /proc/interrupts
 NMI:          2          1          0          5          1          0          1          1          0          1          2          0          0         12          1          1          0          0          0          5          1          1          0          0   Non-maskable interrupts

2.4.3 nmi watchdog触发kernel crash

当系统挂起并且通常中断都被禁止的故障时，可以通过不可屏蔽中断（non maskable interrupt, NMI）来触发一个panic以及获得crash dump。有两种方式来触发一个NMI，不过这两个方法不能同时使用。

3. 手动触发有两个方法：

3.1 方法1：使用/proc/sysrq-trigger

echo 1 | sudo tee /proc/sys/kernel/sysrq
echo c | sudo tee /proc/sysrq-trigger

3.2 方法2: 开启nmi

假如，希望机器A 接受nmi后panic

登录A：

echo 1 | sudo tee /proc/sys/kernel/unkown_nmi_panic

A的oob IP： 100.126.xx.xx

登录A的机房所在的oob跳板机：

接下来就可以通过网络使用IPMI远程发送unknown_nmi_panic信号给服务器触发kernel core dump

[root@oob1.xxx /root]
#ipmitool -I lanplus -U name -P xxxxx -H 100.126.xx.xx chassis power diag
Chassis Power Control: Diag

rt2m09613 login:
[  287.765130] Kernel panic - not syncing: An NMI occurred. Depending on your system the reason for the NMI is logged in any one of the following resources:
[  287.765130] 1. Integrated Management Log (IML)
[  287.765130] 2. OA Syslog
[  287.765130] 3. OA Forward Progress Log
[  287.765130] 4. iLO Event Log
[  287.926060] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           OE  ------------   3.10.0-327.xxxxx.x86_64 #1
[  287.990149] Hardware name: HP ProLiant DL380e Gen8, BIOS P73 08/20/2012
[  288.029748]  ffffffffa049e4b0 2df877b6d8f92a53 ffff88183f405de0 ffffffff81631816
[  288.074609]  ffff88183f405e60 ffffffff8162b0ef 0000000000000008 ffff88183f405e70
[  288.118449]  ffff88183f405e10 2df877b6d8f92a53 0000000000000000 ffffc9000c0d2072
[  288.162844] Call Trace:
[  288.177585]  <NMI>  [<ffffffff81631816>] dump_stack+0x19/0x1b
[  288.212435]  [<ffffffff8162b0ef>] panic+0xd8/0x1e7
[  288.240375]  [<ffffffffa049d8ed>] hpwdt_pretimeout+0xdd/0xe0 [hpwdt]
[  288.279340]  [<ffffffff8163a9f9>] nmi_handle.isra.0+0x69/0xb0
[  288.314476]  [<ffffffff8163ab66>] do_nmi+0x126/0x340
[  288.341792]  [<ffffffff81639e31>] end_repeat_nmi+0x1e/0x2e
[  288.374724]  [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10
[  288.409004]  [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10
[  288.442540]  [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10
[  288.476401]  <<EOE>>  [<ffffffff8101dd5f>] default_idle+0x1f/0xc0
[  288.510993]  [<ffffffff8101e666>] arch_cpu_idle+0x26/0x30
[  288.541880]  [<ffffffff810d3185>] cpu_startup_entry+0x245/0x290
[  288.577386]  [<ffffffff81621587>] rest_init+0x77/0x80
[  288.607035]  [<ffffffff81a89057>] start_kernel+0x429/0x44a
[  288.640036]  [<ffffffff81a88a37>] ? repair_env_string+0x5c/0x5c
[  288.675191]  [<ffffffff81a88120>] ? early_idt_handlers+0x120/0x120
[  288.713429]  [<ffffffff81a885ee>] x86_64_start_reservations+0x2a/0x2c
[  288.753833]  [<ffffffff81a88742>] x86_64_start_kernel+0x152/0x175
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu

4. crash进行结果分析

crash包需要yum -y install crash 单独安装过，另外crash 命令需要依赖kernel-debuginfo 包（该包又依赖kernel-debuginfo-common包），该包的下载地址：http://debuginfo.centos.org/6/x86_64/ 。

下载前先要确认下自己主机的内核版本。我在测试机上是通过下面的命令执行的：

el6

# uname -r
2.6.32-431.17.1.el6.x86_64
# wget http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-common-x86_64-2.6.32-431.17.1.el6.x86_64.rpm
# wget http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-2.6.32-431.17.1.el6.x86_64.rpm

el7

http://debuginfo.centos.org/7/x86_64/

crash分析案例1

下载完成后，通过rpm -ivh将这两个包安装。然后通过下面的命令进行crash分析

# pwd
/var/crash/127.0.0.1-2014-09-16-14:35:49
# crash /usr/lib/debug/lib/modules 2.6.32-431.17.1.el6.x86_64/vmlinux vmcore
crash 6.1.0-5.el6
Copyright (C) 2002-2012  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
      KERNEL: /usr/lib/debug/lib/modules/2.6.32-431.17.1.el6.x86_64/vmlinux
    DUMPFILE: vmcore  [PARTIAL DUMP]
        CPUS: 1
        DATE: Tue Sep 16 22:35:49 2014
      UPTIME: 00:05:33
LOAD AVERAGE: 0.00, 0.00, 0.00
       TASKS: 175
    NODENAME: localhost.localdomain
     RELEASE: 2.6.32-431.17.1.el6.x86_64
     VERSION: #1 SMP Wed May 7 23:32:49 UTC 2014
     MACHINE: x86_64  (3398 Mhz)
      MEMORY: 1 GB
       PANIC: "Oops: 0002 [#1] SMP " (check log for details)
         PID: 1412
     COMMAND: "bash"
        TASK: ffff88003d0b2040  [THREAD_INFO: ffff88003c33c000]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)
crash> bt
PID: 1412   TASK: ffff88003d0b2040  CPU: 0   COMMAND: "bash"
 #0 [ffff88003c33d9e0] machine_kexec at ffffffff81038f3b
 #1 [ffff88003c33da40] crash_kexec at ffffffff810c59f2
 #2 [ffff88003c33db10] oops_end at ffffffff8152b7f0
 #3 [ffff88003c33db40] no_context at ffffffff8104a00b
 #4 [ffff88003c33db90] __bad_area_nosemaphore at ffffffff8104a295
 #5 [ffff88003c33dbe0] bad_area at ffffffff8104a3be
 #6 [ffff88003c33dc10] __do_page_fault at ffffffff8104ab6f
 #7 [ffff88003c33dd30] do_page_fault at ffffffff8152d73e
 #8 [ffff88003c33dd60] page_fault at ffffffff8152aaf5
    [exception RIP: sysrq_handle_crash+22]
    RIP: ffffffff8134b516  RSP: ffff88003c33de18  RFLAGS: 00010096
    RAX: 0000000000000010  RBX: 0000000000000063  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 0000000000000063
    RBP: ffff88003c33de18   R8: 0000000000000000   R9: ffffffff81645da0
    R10: 0000000000000001  R11: 0000000000000000  R12: 0000000000000000
    R13: ffffffff81b01a40  R14: 0000000000000286  R15: 0000000000000004
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #9 [ffff88003c33de20] __handle_sysrq at ffffffff8134b7d2
#10 [ffff88003c33de70] write_sysrq_trigger at ffffffff8134b88e
#11 [ffff88003c33dea0] proc_reg_write at ffffffff811f2f1e
#12 [ffff88003c33def0] vfs_write at ffffffff81188c38
#13 [ffff88003c33df30] sys_write at ffffffff81189531
#14 [ffff88003c33df80] system_call_fastpath at ffffffff8100b072
    RIP: 00000036e3adb7a0  RSP: 00007fff22936c10  RFLAGS: 00010206
    RAX: 0000000000000001  RBX: ffffffff8100b072  RCX: 0000000000000400
    RDX: 0000000000000002  RSI: 00007fab7908b000  RDI: 0000000000000001
    RBP: 00007fab7908b000   R8: 000000000000000a   R9: 00007fab79084700
    R10: 00000000ffffffff  R11: 0000000000000246  R12: 0000000000000002
    R13: 00000036e3d8e780  R14: 0000000000000002  R15: 00000036e3d8e780
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b
crash>

上面，只是简单的通过打印堆栈信息，显示主机在出现kdump生成时，pid 为1412的bash进程操作。从上面的显示信息中也简单的看到有 write_sysrq_trigger 函数触发。crash在定位问题原因时，为我们提供了下面的命令：

crash> ?
*              files          mach           repeat         timer
alias          foreach        mod            runq           tree
ascii          fuser          mount          search         union
bt             gdb            net            set            vm
btop           help           p              sig            vtop
dev            ipcs           ps             struct         waitq
dis            irq            pte            swap           whatis
eval           kmem           ptob           sym            wr
exit           list           ptov           sys            q
extend         log            rd             task
crash version: 6.1.0-5.el6   gdb version: 7.3.1
For help on any command above, enter "help <command>".
For help on input options, enter "help input".
For help on output options, enter "help output".

crash分析案例2

2017-04-08 03:34:51    [29201245.714153] io-error-guard: catch 1 continuous bio error.
2017-04-08 03:34:51    [29201245.722046] Buffer I/O error on device sdc, logical block 0
...
2017-04-08 03:34:51    [29201245.781675] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
2017-04-08 03:34:51    [29201245.790380] IP: [] netoops+0x125/0x2a0
2017-04-08 03:34:51    [29201245.796262] PGD 371206d067 PUD 1c17f34067 PMD 0
2017-04-08 03:34:51    [29201245.801494] Oops: 0000 [#1] SMP
...
2017-04-08 03:34:51    [29201245.941654] Pid: 21606, comm: sh Tainted: GF          ---------------    2.6.32-358.23.2.ali1233.el5.x86_64 #1 Inspur SA5212M4/YZMB-00370-109
2017-04-08 03:34:51    [29201245.955245] RIP: 0010:[]  [] netoops+0x125/0x2a0
...
2017-04-08 03:34:51    [29201246.085431] Call Trace:
2017-04-08 03:34:51    [29201246.088422]
2017-04-08 03:34:51    [29201246.091092]  [] kmsg_dump+0x113/0x180
2017-04-08 03:34:51    [29201246.096875]  [] bio_endio+0x12a/0x1b0
2017-04-08 03:34:51    [29201246.102652]  [] req_bio_endio+0x90/0xc0
2017-04-08 03:34:51    [29201246.108601]  [] blk_update_request+0x262/0x480
2017-04-08 03:34:51    [29201246.115165]  [] blk_update_bidi_request+0x27/0x80
2017-04-08 03:34:51    [29201246.121984]  [] blk_end_bidi_request+0x2f/0x80
2017-04-08 03:34:51    [29201246.128542]  [] blk_end_request+0x10/0x20
2017-04-08 03:34:51    [29201246.134665]  [] blk_end_request_err+0x33/0x60
2017-04-08 03:34:51    [29201246.141140]  [] scsi_io_completion+0x2db/0x5b0
2017-04-08 03:34:51    [29201246.147699]  [] scsi_finish_command+0xc3/0x120
2017-04-08 03:34:51    [29201246.154258]  [] scsi_softirq_done+0x101/0x170
2017-04-08 03:34:51    [29201246.160734]  [] blk_done_softirq+0x83/0xa0
2017-04-08 03:34:51    [29201246.166944]  [] __do_softirq+0xbf/0x220
2017-04-08 03:34:51    [29201246.172895]  [] call_softirq+0x1c/0x30
2017-04-08 03:34:51    [29201246.178757]  [] do_softirq+0x65/0xa0
2017-04-08 03:34:51    [29201246.184451]  [] irq_exit+0x7c/0x90
2017-04-08 03:34:51    [29201246.189968]  [] smp_call_function_single_interrupt+0x34/0x40
2017-04-08 03:34:51    [29201246.198017]  [] call_function_single_interrupt+0x13/0x20
2017-04-08 03:34:51    [29201246.205438]
2017-04-08 03:34:51    [29201246.208106]  [] ? wait_for_rqlock+0x24/0x40
2017-04-08 03:34:51    [29201246.214403]  [] do_exit+0x5e6/0x8d0
2017-04-08 03:34:51    [29201246.220003]  [] do_group_exit+0x41/0xb0
2017-04-08 03:34:51    [29201246.225957]  [] sys_exit_group+0x17/0x20
2017-04-08 03:34:51    [29201246.231996]  [] system_call_fastpath+0x16/0x1b

找到对应的vmcore

cd tmp/linux-2.6.32-358.23.2.el5

gdb vmlinux

//转换Oops中的Call Trace中函数源代码位置

(gdb) l *bio_endio+0x12a

/// 就可以定位到出现故障时候源代码的位置

0xffffffff811b292a is in bio_endio (fs/bio.c:1474).
1469                }
1470            }
1471        }
1472        spin_unlock(&eio->lock);
1473        if (sysctl_enable_bio_netoops)
1474            kmsg_dump(KMSG_DUMP_SOFT, NULL);
1475    }
1476
1477    /**
1478     * bio_endio - end I/O on a bio
(gdb)
注意

REF

我需要内核的源代码