Linux由于物理节点故障导致的异常重启-Case1

问题描述：Linux VM异常重启，需要排查问题原因

排查结果：

查询Messages日志获取到的信息

虚拟机内核版本：

Jun  :: test01 kernel: Linux version 2.6.-642.1..el6.x86_64 (mockbuild@worker1.bsys.centos.org) (gcc version 4.4.  (Red Hat 4.4.-) (GCC) ) # SMP Tue May  :: UTC 

查看到虚拟机重启时间约为：2019/6/20 03:34 CST

Jun  :: test01 kernel: hv_utils: Shutdown request received - graceful shutdown initiated
Jun  :: test01 init: tty (/dev/tty1) main process () killed by TERM signal
Jun  :: test01 init: tty (/dev/tty2) main process () killed by TERM signal
Jun  :: test01 init: tty (/dev/tty3) main process () killed by TERM signal
Jun  :: test01 init: tty (/dev/tty4) main process () killed by TERM signal
Jun  :: test01 init: tty (/dev/tty5) main process () killed by TERM signal
Jun  :: test01 init: tty (/dev/tty6) main process () killed by TERM signal
Jun  :: test01 init: serial (ttyS0) main process () killed by TERM signal
Jun  :: test01 abrtd: Got signal , exiting
Jun  :: test01 dnsmasq[]: exiting on receipt of SIGTERM
Jun  :: test01 acpid: exiting
Jun  :: test01 rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w"
Jun  :: test01 init: Disconnected from system bus
Jun  :: test01 auditd[]: The audit daemon is exiting.
Jun  :: test01 kernel: type= audit(1560972674.940:): audit_pid= old= auid= ses= subj=system_u:system_r:auditd_t:s0 res=
Jun  :: test01 kernel: type= audit(1560972675.039:): audit_enabled= old= auid= ses= subj=system_u:system_r:auditctl_t:s0 res=
Jun  :: test01 kernel: Kernel logging (proc) stopped.
Jun  :: test01 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="" x-info="http://www.rsyslog.com"] exiting on signal .
Jun  :: test01 kernel: imklog 5.8., log source = /proc/kmsg started.
Jun  :: test01 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="" x-info="http://www.rsyslog.com"] start
Jun  :: test01 kernel: Initializing cgroup subsys cpuset
Jun  :: test01 kernel: Initializing cgroup subsys cpu
Jun  :: test01 kernel: Linux version 2.6.-642.1..el6.x86_64 (mockbuild@worker1.bsys.centos.org) (gcc version 4.4.  (Red Hat 4.4.-) (GCC) ) # SMP Tue May  :: UTC
Jun  :: test01 kernel: Command line: ro root=UUID=adc76f7c-fef6--941e-e7ce50fb3e50 rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF- rd_NO_MD SYSFONT=latarcyrheb-sun16 console=ttyS0,115200n8 earlyprintk=ttyS0, rootdelay= rd_NO_LVM rd_NO_DM
Jun  :: test01 kernel: KERNEL supported cpus:
Jun  :: test01 kernel:  Intel GenuineIntel
Jun  :: test01 kernel:  AMD AuthenticAMD
Jun  :: test01 kernel:  Centaur CentaurHauls

查询到虚拟机重启之前出现了Kernel Panic的错误，提示：INFO: task jbd2/sda1-8:540 blocked for more than 120 seconds. 备注：jdb2进程是一个文件系统的写journal的进程

Jun  :: test01 auditd[]: Audit daemon rotating log files
Jun  :: test01 kernel: INFO: task jbd2/sda1-: blocked for more than  seconds.
Jun  :: test01 kernel:      Not tainted 2.6.-642.1..el6.x86_64 #
Jun  :: test01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun  :: test01 kernel: jbd2/sda1-   D                0x00000000
Jun  :: test01 kernel: ffff880433257b30   ffff880433336d80
Jun  :: test01 kernel: 000000002a5c72b4 ffffffffa307745d ffff880433257ae0 ffff880433257ad0
Jun  :: test01 kernel: ffffffffa0045958  ffff8804316505f8 ffff880433257fd8
Jun  :: test01 kernel: Call Trace:
Jun  :: test01 kernel: [<ffffffffa0045958>] ? read_hv_clock_tsc+0x38/0x80 [hv_vmbus]
Jun  :: test01 kernel: [<ffffffff8112e610>] ? sync_page+0x0/0x50
Jun  :: test01 kernel: [<ffffffff81547db3>] io_schedule+0x73/0xc0
Jun  :: test01 kernel: [<ffffffff8112e64d>] sync_page+0x3d/0x50
Jun  :: test01 kernel: [<ffffffff8154889f>] __wait_on_bit+0x5f/0x90
Jun  :: test01 kernel: [<ffffffff8112e883>] wait_on_page_bit+0x73/0x80
Jun  :: test01 kernel: [<ffffffff810a6b40>] ? wake_bit_function+0x0/0x50
Jun  :: test01 kernel: [<ffffffff811449c5>] ? pagevec_lookup_tag+0x25/0x40
Jun  :: test01 kernel: [<ffffffff8112ecab>] wait_on_page_writeback_range+0xfb/0x190
Jun  :: test01 kernel: [<ffffffff8112ed6f>] filemap_fdatawait+0x2f/0x40
Jun  :: test01 kernel: [<ffffffffa00bfe59>] jbd2_journal_commit_transaction+0x7e9/0x14f0 [jbd2]
Jun  :: test01 kernel: [<ffffffff8108fd4b>] ? try_to_del_timer_sync+0x7b/0xe0
Jun  :: test01 kernel: [<ffffffffa00c5a38>] kjournald2+0xb8/0x220 [jbd2]
Jun  :: test01 kernel: [<ffffffff810a6ac0>] ? autoremove_wake_function+0x0/0x40
Jun  :: test01 kernel: [<ffffffffa00c5980>] ? kjournald2+0x0/0x220 [jbd2]
Jun  :: test01 kernel: [<ffffffff810a662e>] kthread+0x9e/0xc0
Jun  :: test01 kernel: [<ffffffff8100c28a>] child_rip+0xa/0x20
Jun  :: test01 kernel: [<ffffffff810a6590>] ? kthread+0x0/0xc0
Jun  :: test01 kernel: [<ffffffff8100c280>] ? child_rip+0x0/0x20

查询了secure日志，仅仅发现有关机信号的记录，没有更加详细的信息了  备注：SIGTERM 15 A 终止信号

Jun  :: test01 sshd[]: Received signal ; terminating.
Jun  :: test01 sshd[]: Server listening on 0.0.0.0 port .
Jun  :: test01 sshd[]: Server listening on :: port .

查询了Linux问题虚拟机所在的物理节点，发现早在6/20 03:10 CST左右，物理节点已经出现了系统故障，虚拟机被自动迁移至其他可用节点，此过程造成了虚拟机的自动重启

总结：在某些情况下，服务器系统日志出现了call trace的报错，也不一定是操作系统层面引起的问题，还可能是物理节点（针对虚拟机）或硬件故障。