一个suse11 sp1的crash工具版本问题

这几年排查的各种类型的crash也比较多了，各种类型的也算见过，但是排查这个crash，走了不该走的弯路，事后显得很low，为了防止自己犯类似错误,也同时提醒后人，记录之。

内核是suse11，sp1，

uname -a
Linux Ftp1 2.6.32.59-0.7-default # SMP -- :: + x86_64 x86_64 x86_64 GNU/Linux

crash目录下有三个文件：

README.txt  vmcore  vmlinux-2.6.32.59-0.7-default

常规动作，编译vmlinux，然后看crash：

A10111916:~ # crash /home/caq/vmlinux /home/zxin11/vmcore

crash 4.0-7.6--------------------------------------------------------------------低版本
Copyright (C) , , , , , ,   Red Hat, Inc.
Copyright (C) , ,   IBM Corporation
Copyright (C) -  Hewlett-Packard Co
Copyright (C) ,   Fujitsu Limited
Copyright (C) ,   VA Linux Systems Japan K.K.
Copyright (C)   NEC Corporation
Copyright (C) , ,   Silicon Graphics, Inc.
Copyright (C) , , ,   Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb 6.1
Copyright  Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

crash: invalid structure size: x8664_pda
       FILE: x86_64.c  LINE:   FUNCTION: x86_64_cpu_pda_init()

[/usr/bin/crash] error trace:  => 4569bd => 4cb321 => 4e5300

  4e5300: SIZE_verify+
  4cb321: x86_64_init+
  4569bd: main_loop+
  : (undetermined)

我还以为是vmcore拷贝的有问题，检查了线上的vmcore和拷贝回来的vmcore，大小一样，md5值都是一样。然后检查编译的vmlinux，主要是检查.config文件 以及编译内核的

环境的gcc版本是否和线上出问题的gcc版本一致，也没有问题。过了好一会才开始怀疑，

是不是crash的版本有问题，为了验证这个想法，将vmlinux拷贝到线上去检查，线上环境的crash是5.0.1版本，就没有报错，看来真的跟crash版本有关系。这个也给自己上了一课，总共就

三个文件，crash，vmlinux，vmcore，解析出错，在保证vmlinux编译没问题和vmcore是完整的情况下，要仔细确认下crash的版本。

crash 5.0.1------------------------------------------------os自带版本
Copyright (C) -  Red Hat, Inc.
Copyright (C) , ,   IBM Corporation
Copyright (C) -  Hewlett-Packard Co
Copyright (C) ,   Fujitsu Limited
Copyright (C) ,   VA Linux Systems Japan K.K.
Copyright (C)   NEC Corporation
Copyright (C) , ,   Silicon Graphics, Inc.
Copyright (C) , , ,   Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.0
Copyright (C)  Free Software Foundation, Inc.
License GPLv3+: GNU GPL version  or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

      KERNEL: vmlinux
    DUMPFILE: vmcore
        CPUS:
        DATE: Thu Feb  ::
      UPTIME:  days, ::
LOAD AVERAGE: 0.08, 0.13, 0.10
       TASKS:
    NODENAME: Ftp1
     RELEASE: 2.6.32.59-0.7-default
     VERSION: # SMP -- :: +
     MACHINE: x86_64  ( Mhz)
      MEMORY:  GB
       PANIC: "[6186227.149497] Oops: 0000 [#1] SMP " (check log for details)
         PID:
     COMMAND: "swapper"
        TASK: ffffffff8180c020  ( of )  [THREAD_INFO: ffffffff81800000]
         CPU:
       STATE: TASK_RUNNING (ACTIVE)
     WARNING: panic task not found

居然显示"panic task not found"，常见的crash都如下所示，而这个crash解析多了一个warning，

      PANIC: "[335750.721156] Oops: 0002 [#1] SMP " (check log for details)
         PID:
     COMMAND: "bash"
        TASK: ffff88031b886380  [THREAD_INFO: ffff880319958000]
         CPU:
       STATE: TASK_RUNNING (PANIC)

比 crash 4.0-7.6 有进步，也算是个好兆头，下载 crash 5.0.1 的源码检查，发现这个warning关系也不大，但我犯了一个致命错误，就是对：

PANIC: "[6186227.149497] Oops: 0000 [#1] SMP " (check log for details)

这一行没有仔细看，高版本一些的内核，都是打印dmesg.txt在单独的一个文件，通过这个文件至少能快速地确认出panic的堆栈。而PANIC这行，

要求我去看log命令，我又没有去看，因为任务不多，直接去看各个进程的堆栈。导致又走了弯路。发现了两个堆栈比较可疑：

PID:   TASK: ffff88067bbc6080  CPU:    COMMAND: "SMSvr"
 # [ffff88067dbf5dc8] schedule at ffffffff813923c4
 # [ffff88067dbf5de0] sys_reboot at ffffffff8105e00d
 # [ffff88067dbf5e60] do_notify_resume at ffffffff810028c5
 # [ffff88067dbf5f30] sys_rt_sigreturn at ffffffff81002aa8
 # [ffff88067dbf5f50] ptregscall_common at ffffffff81003216
    RIP: 00007f017b6e8efd  RSP: 00007f0178b71dc0  RFLAGS:
    RAX: fffffffffffffdfc  RBX:   RCX: ffffffffffffffff
    RDX:   RSI: 00007f0178b71df0  RDI: 00007f0178b71df0
    RBP: 00007f0178b71e00   R8: fefefefefefeffff   R9:
    R10:   R11:   R12: 00007fff853b82f0
    R13: 00007f0178b72000  R14:   R15:
    ORIG_RAX:   CS:   SS: 002b

这个函数里面居然有一个sys_reboot调用，reboot导致panic我确实还没经历过，不死心，反汇编一下sys_reboot,打印如下：

rash> dis -l sys_reboot
/home/caq/usr/src/linux-2.6.32.59-0.7/kernel/sys.c:
0xffffffff8105df50 <sys_reboot>:        test   %edi,0x1(%rbx)
0xffffffff8105df53 <sys_reboot+>:      add    %al,(%rax)
0xffffffff8105df55 <sys_reboot+>:      cmp    $0x1f,%ebx
0xffffffff8105df58 <sys_reboot+>:      jg     0xffffffff8105df69 <sys_reboot+>
0xffffffff8105df5a <sys_reboot+>:     lea    -0x1(%rbx),%ecx
0xffffffff8105df5d <sys_reboot+>:     mov    $0x8430000,%eax
/home/caq/usr/src/linux-2.6.32.59-0.7/kernel/sys.c:
0xffffffff8105df62 <sys_reboot+>:     shr    %cl,%rax
0xffffffff8105df65 <sys_reboot+>:     test   $0x1,%al
/home/caq/usr/src/linux-2.6.32.59-0.7/kernel/sys.c: 

明显反汇编得不对啊，reboot的代码里面有很多case对应的魔术字，而这个却没有cmp指令，而且代码一开始进来也没有建立栈的过程，立马再次对这个crash的解析结果产生怀疑，因为按道理

crash从vmlinux取出响应的符号对应的地址，然后到vmcore中找到对应的地址展示出来，说明vmcore和vmlinux还是存在不对应。但这个crash工具居然没提示（我见过不一致的提示，类似于WARNING: kernel version inconsistency between vmlinux and dumpfile）

为了验证自己的想法，我到编译的vmlinux中找一下sys_reboot，

linux-h9c2:/home/caq # objdump -d vmlinux >caq.txt

linux-h9c2:/home/caq # grep sys_reboot caq.txt
ffffffff8105df50 <sys_reboot>:

linux-h9c2:/home/caq # nm vmlinux |grep -i sys_reboot
ffffffff8105df50 T sys_reboot

地址是：ffffffff8105df50，crash工具将这个地址去找sys_reboot,结果打印的却不是sys_reboot的反汇编，不可能crash工具出这么低级的问题啊，说明vmlinux和vmcore还是存在不对应。

想着reboot调用跟panic按道理风牛马不相及啊，放弃这条路，因为既然sys_reboot是错的，那么可能堆栈回溯都是错的了，

就剩下pid 38021了。

crash> bt -f
PID:   TASK: ffff88003531c340  CPU:    COMMAND: "sh"
 # [ffff880476051de8] schedule at ffffffff813923c4
    ffff880476051df0:
    ffff880476051e00:
    ffff880476051e10:
    ffff880476051e20:
    ffff880476051e30:
    ffff880476051e40:
    ffff880476051e50:
    ffff880476051e60:
    ffff880476051e70:
    ffff880476051e80:
    ffff880476051e90:
    ffff880476051ea0:
    ffff880476051eb0:
    ffff880476051ec0:
    ffff880476051ed0:
    ffff880476051ee0:
    ffff880476051ef0:
    ffff880476051f00:
    ffff880476051f10:
    ffff880476051f20:
    ffff880476051f30: 00000000006c9870 ffff88027dd62480
    ffff880476051f40: ffff88084c3a8d40
    ffff880476051f50: 00000000006a0dd0 00007fffbc69e690
    ffff880476051f60:  00000000006d3040
    ffff880476051f70:  00000000006d3ba0
    ffff880476051f80: ffffffff81002f7b
 # [ffff880476051f80] auditsys at ffffffff81002f7b
    RIP: 00007fb09a95b4f0  RSP: 00007fffbc69e6c0  RFLAGS:
    RAX:   RBX: ffffffff81002f7b  RCX:
    RDX: 00000000000001b6  RSI:   RDI: 00000000006d3040
    RBP: 00000000006d3ba0   R8:    R9: 6c6568732f6d732f
    R10:   R11:   R12:
    R13: 00000000006d3040  R14:   R15: 00007fffbc69e690
    ORIG_RAX:   CS:   SS: 002b

看着堆栈不太对啊，auditsys 不是一个系统调用的入口，按道理第一个压栈的函数应该是常见的system_call_fastpath ，直接查看一下这个地址：

Ftp1:/home # grep ffffffff81002f /proc/kallsyms
ffffffff81002f00 T system_call_after_swapgs
ffffffff81002f65 t system_call_fastpath
ffffffff81002f80 t ret_from_sys_call
ffffffff81002f85 t sysret_check
ffffffff81002fd8 t sysret_careful
ffffffff81002fe8 t sysret_signal

发现 ffffffff81002f7b 应该属于 system_call_fastpath 的地址范围。

看来这crash的工具用不了，映射是错的，于是找了个更新一点的crash工具，版本为7.0.9

crash 7.0.9---------------------------------------------------更高版本
Copyright (C) -  Red Hat, Inc.
Copyright (C) , , ,   IBM Corporation
Copyright (C) -  Hewlett-Packard Co
Copyright (C) , , ,   Fujitsu Limited
Copyright (C) ,   VA Linux Systems Japan K.K.
Copyright (C) ,   NEC Corporation
Copyright (C) , ,   Silicon Graphics, Inc.
Copyright (C) , , ,   Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

crash: vmlinux: no .gnu_debuglink section
GNU gdb (GDB) 7.6
Copyright (C)  Free Software Foundation, Inc.
License GPLv3+: GNU GPL version  or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel version inconsistency between vmlinux and dumpfile------------------------有告警

      KERNEL: vmlinux
    DUMPFILE: vmcore
        CPUS:
        DATE: Thu Feb  ::
      UPTIME:  days, ::
LOAD AVERAGE: 0.08, 0.13, 0.10
       TASKS:
    NODENAME: Ftp1
     RELEASE: 2.6.32.59-0.7-default
     VERSION: # SMP -- :: +
     MACHINE: x86_64  ( Mhz)
      MEMORY:  GB
       PANIC: "[6186227.149497] Oops: 0000 [#1] SMP " (check log for details)
         PID:
     COMMAND: "sh"-------------------------------------------------------找到对应的panic任务，比上一个版本靠谱
        TASK: ffff88003531c340  [THREAD_INFO: ffff880476050000]
         CPU:
       STATE: TASK_RUNNING (PANIC)

升级到7.0.9，然后敲入log命令：

对应的log中显示：

[6186227.149460] BUG: unable to handle kernel NULL pointer dereference at (null)
[6186227.149479] IP: [<ffffffff811e7752>] strlen+0x2/0x30
[6186227.149492] PGD 47b9be067 PUD 42e601067 PMD
[6186227.149497] Oops:  [#] SMP
[6186227.149502] last sysfs file: /sys/devices/pci0000:/::07.0/::00.1/host4/rport-:-/target4::/:::/state
[6186227.149510] CPU
[6186227.149513] Modules linked in: secureProof(N) iptable_filter ip_tables x_tables dm_round_robin dm_multipath scsi_dh ipv6 bonding microcode f
use loop dm_mod tpm_tis dcdbas(X) tpm qla2xxx usbhid tpm_bios hid iTCO_wdt scsi_transport_fc iTCO_vendor_support serio_raw sr_mod scsi_tgt ses cd
rom pcspkr enclosure bnx2 sg rtc_cmos rtc_core rtc_lib wmi power_meter button uhci_hcd ehci_hcd usbcore sd_mod crc_t10dif edd ext3 mbcache jbd fa
n processor ide_pci_generic ide_core ata_generic ata_piix libata megaraid_sas thermal thermal_sys hwmon mpdh(N) mpdt(N) scsi_mod [last unloaded:
secureProof]
[6186227.149571] Supported: Yes
[6186227.149577] Pid: , comm: sh Tainted: G          NX 2.6.32.59-0.7-default # PowerEdge R910
[6186227.149582] RIP: :[<ffffffff811e7752>]  [<ffffffff811e7752>] strlen+0x2/0x30
[6186227.149588] RSP: :ffff880476051280  EFLAGS:
[6186227.149592] RAX:  RBX: ffff8805f94ec000 RCX:
[6186227.149596] RDX:  RSI:  RDI:
[6186227.149600] RBP:  R08: ffff8804760511f8 R09: ffffffff81539570
[6186227.149604] R10:  R11: 0000000000000fff R12: ffff880476051d38
[6186227.149608] R13: ffff88067bbc6080 R14: ffffffffa03d8f79 R15:
[6186227.149612] FS:  00007fb09b21e700() GS:ffff880487400000() knlGS:
[6186227.149617] CS:   DS:  ES:  CR0:
[6186227.149621] CR2:  CR3: 0000000473f4d000 CR4: 00000000000006e0
[6186227.149625] DR0:  DR1:  DR2:
[6186227.149629] DR3:  DR6: 00000000ffff0ff0 DR7:
[6186227.149634] Process sh (pid: , threadinfo ffff880476050000, task ffff88003531c340)
[6186227.149638] Stack:
[6186227.149640]  ffffffffa03d5768 ffffffffa03d8e4c ffff88067bbc6080 ffff880476051d38
[6186227.149645] <> ffff8804760514c8  ffffffffa03d5b53 ffff880476051d58
[6186227.149650] <> ffffffffa03d8e4c 787a2f656d6f682f 7374642f30316e69 76534d532f6d732f
[6186227.149657] Call Trace:
[6186227.149678]  [<ffffffffa03d5768>] getprocpath+0xa8/0x150 [secureProof]
[6186227.149701]  [<ffffffffa03d5b53>] checkTrustProc+0x83/0x270 [secureProof]
[6186227.149710]  [<ffffffffa03d66ca>] checkProcAndFile+0x3da/0x890 [secureProof]
[6186227.149720]  [<ffffffffa03d7aba>] our_sys_open+0xfa/0x1d0 [secureProof]-----------我们模块接管的open
[6186227.149736]  [<ffffffff81002f7b>] system_call_fastpath+0x16/0x1b
[6186227.149745]  [<00007fb09a95b4f0>] 0x7fb09a95b4f0
[6186227.149749] Code:    c7  0f b6   c0  0c 0f b6 c0 f6  a0      e9   f8 c3     2e 0f 1f
  c0 <> 3f    fa    0f 1f      c2   3a
[6186227.149779] RIP  [<ffffffff811e7752>] strlen+0x2/0x30
[6186227.149784]  RSP <ffff880476051280>
[6186227.149787] CR2: 

这个打印和crash找的任务是一致的，都是sh进程，pid为38021。

然后查看strlen的代码：

/home/caq/usr/src/linux-2.6.32.59-0.7/lib/string.c:
0xffffffff811e1750 <strlen>:    ljmpq  *(%rcx)
0xffffffff811e1752 <strlen+>:  icebp

确定是由于rcx为NULL导致的，业务代码流程有问题，直接引用空指针，导致crash。

总结一下：

1.crash分析的时候，crash的版本尽量新一些，特别当某些crash工具解析有问题的时候，要果断换，出现的crash工具提醒的warning，要重视。

2.老司机也会翻车，编译vmlinx的gcc版本，最好和运行的内核的gcc版本一致。