libvirtError: no connection driver available for qemu:///system 解决办法

今天部署了一套K版的OpenStack环境后，启动nova-compute后，日志里狂报如下错误（因为其内部有定时任务需要去连libvirtd进程）：

2015-08-14 10:01:00.061 28580 TRACE nova.virt.libvirt.host   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth
2015-08-14 10:01:00.061 28580 TRACE nova.virt.libvirt.host     if ret is None:raise libvirtError('virConnectOpenAuth() failed')
2015-08-14 10:01:00.061 28580 TRACE nova.virt.libvirt.host libvirtError: no connection driver available for qemu:///system

刚开始看到这个错误，首先第一反应是不是 libvirtd 进程没有启起来，于是 ps 查看确定下，发现已经启动起来了：

[root@localhost ~]# ps aux | grep libvirtd
root      2648  0.0  0.0 112640   980 pts/1    S+   10:34   0:00 grep --color=auto libvirtd
root     31850  0.1  0.0 1053776 7264 ?        Ssl  10:17   0:01 /usr/sbin/libvirtd --listen

后来，尝试直接执行下 virsh list 命令，同样得到如下错误：

[root@localhost ~]# virsh list
error: failed to connect to the hypervisor
error: no valid connection
error: no connection driver available for <null>
error: Failed to reconnect to the hypervisor

然后在想是不是URI不对，于是我指定URI再次尝试执行命令，同样返回错误：

[root@localhost ~]# virsh -c qemu:///system list
error: failed to connect to the hypervisor
error: no connection driver available for qemu:///system

为保持每次使用的URI都正确，后来我将 /etc/libvirt/libvirt.conf 配置文件里的 uri_default 直接设置为 qemu:///system：

[root@localhost ~]# cat /etc/libvirt/libvirt.conf
#
# This can be used to setup URI aliases for frequently
# used connection URIs. Aliases may contain only the
# characters  a-Z, -, _, -.
#
# Following the '=' may be any valid libvirt connection
# URI, including arbitrary parameters

#uri_aliases = [
#  "hail=qemu+ssh://root@hail.cloud.example.com/system",
#  "sleet=qemu+ssh://root@sleet.cloud.example.com/system",
#]

#
# This can be used to prevent probing of the hypervisor
# driver when no URI is supplied by the application.

uri_default = "qemu:///system"

然后重启 libvirtd 进程，通过 systemctl 命令查看下 libvirtd 进程的状态，发现有如下3个错误：

[root@localhost ~]# systemctl status libvirtd
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: active (running) since Fri -- :: CST; 4s ago
     Docs: man:libvirtd()
           http://libvirt.org
 Main PID:  (libvirtd)
   CGroup: /system.slice/libvirtd.service
           └─ /usr/sbin/libvirtd --listen

Aug  :: localhost.localdomain libvirtd[]: libvirt version: 1.2., package: .el7_1. (CentOS BuildSystem <http://bugs.centos.org>, 2015-05-12-20:12:58, worker1.bsys.centos.org)
Aug  :: localhost.localdomain libvirtd[]: failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so /usr/lib64/libvirt/connection-driver/...e reference
Aug  :: localhost.localdomain libvirtd[]: failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so /usr/lib64/libvirt/connection-driver/lib...eFileCreate
Aug  :: localhost.localdomain libvirtd[]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so not accessible
Aug  :: localhost.localdomain systemd[]: Started Virtualization daemon.
Hint: Some lines were ellipsized, use -l to show in full.

从上面错误来看，两个错误是由于libvirt_driver_storage.so、libvirt_driver_qemu.so加载失败，一个错误是由于没有libvirt_driver_lxc.so module。

于是查看下目录 /usr/lib64/libvirt/connection-driver/ 看看是否都有上面三个module：

[root@localhost ~]# ll /usr/lib64/libvirt/connection-driver/
total
-rwxr-xr-x  root root   May  : libvirt_driver_interface.so
-rwxr-xr-x  root root   May  : libvirt_driver_network.so
-rwxr-xr-x  root root   May  : libvirt_driver_nodedev.so
-rwxr-xr-x  root root   May  : libvirt_driver_nwfilter.so
-rwxr-xr-x  root root  May  : libvirt_driver_qemu.so
-rwxr-xr-x  root root    May  : libvirt_driver_secret.so
-rwxr-xr-x  root root   May  : libvirt_driver_storage.so

发现libvirt_driver_storage.so、libvirt_driver_qemu.so存在，而libvirt_driver_lxc.so确实不存在。

对于 libvirt_driver_lxc.so不存在 的情况，应该是相应的rpm包没有安装，我们只需要安装下即可：

[root@localhost ~]# yum install libvirt-daemon-lxc

安装完后，再次通过 systemctl 查看下 libvirtd 进程的状态，剩下两个错误还是没有得到解决：

[root@localhost ~]# systemctl status libvirtd
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: active (running) since Fri -- :: CST; 3s ago
     Docs: man:libvirtd()
           http://libvirt.org
 Main PID:  (libvirtd)
   CGroup: /system.slice/libvirtd.service
           └─ /usr/sbin/libvirtd --listen

Aug  :: localhost.localdomain libvirtd[]: libvirt version: 1.2., package: .el7_1. (CentOS BuildSystem <http://bugs.centos.org>, 2015-05-12-20:12:58, worker1.bsys.centos.org)
Aug  :: localhost.localdomain libvirtd[]: failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so /usr/lib64/libvirt/connection-driver/...e reference
Aug  :: localhost.localdomain libvirtd[]: failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so /usr/lib64/libvirt/connection-driver/lib...eFileCreate
Aug  :: localhost.localdomain systemd[]: Started Virtualization daemon.
Hint: Some lines were ellipsized, use -l to show in full.

于是查看 /var/log/message 关于libvirtd进程的系统日志，发现错误日志：

Aug 14 10:14:22 localhost journal: failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so: symbol dm_task_get_info_with_deferred_remove, version Base not defined in file libdevmapper.so.1.02 with link time reference
Aug 14 10:14:23 localhost journal: failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so: undefined symbol: virStorageFileCreate

从上面错误来看，undefined symbol，很明显，是因为driver所需要的引用未定义，于是想应该是 libdevmapper.so.1.02 库太老，与 libvirtd 库不匹配，于是就尝试更新下 device-mapper：

[root@localhost ~]# yum update device-mapper

更新后，重启 libvirtd 进程，再次查看其状态：

[root@localhost ~]# systemctl restart libvirtd
[root@localhost ~]# systemctl status libvirtd
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: active (running) since Fri -- :: CST; 2s ago
     Docs: man:libvirtd()
           http://libvirt.org
 Main PID:  (libvirtd)
   CGroup: /system.slice/libvirtd.service
           └─ /usr/sbin/libvirtd --listen

Aug  :: localhost.localdomain systemd[]: Started Virtualization daemon.

发现没有问题出现了，尝试执行 virsh list 命令，也OK了：

[root@localhost ~]# virsh list
 Id    Name                           State
----------------------------------------------------

最后，启动openstack-nova-compute，也不再出现最初的错误了。