Table of Contents
Hanging process 'libvirtd'
Error description
A running daemon 'libvirtd' is the process through which the command 'virsh' communicates with running vhosts. Sometimes the process becomes inaccessible:
root@kvm55:~# ps ax |grep libvirtd 7691 ? Zl 0:00 [libvirtd] <defunct>
As 'libvirtd' is inaccessible, no virsh command will be properly executed: it will hang forever. Thus all 'virsh' commands have to be avoided. Otherwise lots of hanging threads are generated. As a process libvirtd is running, no additional process 'libvirtd' can be started.
Failure cause
The error cause is unknown. It was once observed after an auto-restart of the service 'libvirt-bin' due to an auto-update of the package 'libvirt-bin'.
Administration experience on Systems with Ubuntu 14.04 indicated, that a hanging libvirtd is more likely when libvirt-bin is restarted like so:
service libvirt-bin restart
There are no experiences with Ubuntu 16.04 yet regarding this error.
Solution
To avoid the error on manual restart of libvirt-bin, at least on Ubuntu 14.04 restart libvirt-bin like so:
/etc/init.d/libvirt-bin restart
Hanging processes can be killed by killing their parent process unless the parent process is the init (1) process:
root@kvm55:~# ps -ef |grep [d]efunct UID PID PPID C STIME TTY TIME CMD root 7691 1 0 Jul21 ? 00:00:00 [libvirtd] <defunct>
The only known solution is a reboot of the KVM host thus interrupting all running vhosts.
The vhosts remain running while 'libvirtd' is inaccessible. Thus the reboot can be postponed to a time when the vhosts are not urgently needed. In the meantime avoid every action creating massive numbers of hanging virsh commands like puppet agents and backup processes.
During reboot of the kvm host the downtime for a vhost can be shortened by the following steps:
- Define the vhost on another KVM host with access to the virtual disk(s) of the vhost.
- Shut down the vhost from inside the vhost.
- Once the vhost is down start the vhost on the other KVM host.