The first hint to the error are often monitoring error messages stating that the vhost is down. Trials afterwards to access the vhost via network (e.g. ping) fail. After login to the vhost (e.g. from the serial console) it can be found that no outgoing network communication from the vhost is possible.
The libvirtd service of the KVM host cannot communicate with the vhost process '/usr/bin/kvm' of the vhost.
The error is caused by a bug. It is caused by a race condition during the access to the vhost process.
According to this source the error should be fixed with libvirtd-bin version v1.0.3 because beginning with this version the access to the vhost process is done with more granularity.
The error has indeed not been observed since libvirtd-bin versions v1.1 and above are used.
There are several solution opportunities. They are sorted by the severity of influence on the KVM host and the possible risks for the stability of the KVM host.
The easiest solution is the restart of the vhost. There are three opportunities to do this.
root@vhost:~# shutdown –r now
Restart the vhost from the KVM host afterwards:
root@kvm:~# virsh start vhost
This is only the second best option because the vhost is not properly shut down:
root@kvm:~# virsh destroy vhost root@kvm:~# virsh start vhost
If the libvirtd service cannot communicate with the kvm process all three trials will fail. On trial 1.) the vhost will not shut down, on trial 3.) nothimg will happen at all and on trial 2.) after some time (30-60s) the following error messages will occur:
root@kvm:~# virsh destroy vhost error: Timed out during operation: cannot acquire state change lock
The error message is a secure indication that the libvirtd service cannot communicate with the kvm process. Another secure indication to the problem is the fact that the affected vhost is displayed as running in the VMM, but the window for the 'CPU usage' of the vhost does not show any activity.
As a next step it can be tried to restart the libvirtd service to get the service to communicate with the vhost again:
root@kvm:~# /etc/init.d/libvirt-bin restart
Normally all running virtual hosts will keep running without interrupt when the libvirtd service is restarted, because the libvirtd service in fact monitors the vhost execution but the vhosts are running in their own processes.
In many cases the communication with the affected vhost or at least the restart of the vhost will be possible again after the restart of libvirtd. But in some cases the restart of libvirtd did not help.
Please note:
There exist no completely reliable statements, that the restart of the libvirtd service under all circumstances - e.g. also after an update of libvirtd - will certainly not interrupt the execution of vhosts. In order to be 100% secure, possibly move all other running vhosts to other KVM hosts before the restart of libvirtd. After move of all vhosts the third solution is unnecessary because the KVM host also could be rebooted.
As a last resort create a new vhost using the virtual hard disk of the failed vhost and abandon the failed vhost to his fate till the next reboot of the KVM host.
At first, save the definition of the old host:
root@kvm:~# virsh dumpxml vhost >vhost_new.xml
Afterwards the saved xml file has to be edited to change the following information:
A new and valid UUID can be generated with the command uuid from the software package 'uuid'.
In a next step the new vhost is defined and started:
root@kvm:~# virsh define vhost_new.xml root@kvm:~# virsh start vhost_new