Wiki

The first hint to the error are often monitoring error messages stating that the vhost is down. Trials afterwards to access the vhost via network (e.g. ping) fail. After login to the vhost (e.g. from the serial console) it can be found that no outgoing network communication from the vhost is possible.

The libvirtd service of the KVM host cannot communicate with the vhost process '/usr/bin/kvm' of the vhost.

The error is caused by a bug. It is caused by a race condition during the access to the vhost process.

According to this source the error should be fixed with libvirtd-bin version v1.0.3 because beginning with this version the access to the vhost process is done with more granularity.

The error has indeed not been observed since libvirtd-bin versions v1.1 and above are used.

There are several solution opportunities. They are sorted by the severity of influence on the KVM host and the possible risks for the stability of the KVM host.

The easiest solution is the restart of the vhost. There are three opportunities to do this.

root@vhost:~# shutdown –r now

Restart the vhost from the KVM host afterwards:

root@kvm:~# virsh start vhost

This is only the second best option because the vhost is not properly shut down:

root@kvm:~# virsh destroy vhost
root@kvm:~# virsh start vhost

• Start the VMM
• Select the KVM host and connect to it
• From the opening list of vhosts choose the affected vhost
• Start the VNC window of the vhost by double click
• Stop the vhost by the menu item 'Virtual Machine' → 'Shut Down' → 'Force Off'
• Start the vhost by the menu item 'Virtual Machine' → 'Run'

If the libvirtd service cannot communicate with the kvm process all three trials will fail. On trial 1.) the vhost will not shut down, on trial 3.) nothimg will happen at all and on trial 2.) after some time (30-60s) the following error messages will occur:

root@kvm:~# virsh destroy vhost
error: Timed out during operation: cannot acquire state change lock

The error message is a secure indication that the libvirtd service cannot communicate with the kvm process. Another secure indication to the problem is the fact that the affected vhost is displayed as running in the VMM, but the window for the 'CPU usage' of the vhost does not show any activity.

As a next step it can be tried to restart the libvirtd service to get the service to communicate with the vhost again:

root@kvm:~# /etc/init.d/libvirt-bin restart

Normally all running virtual hosts will keep running without interrupt when the libvirtd service is restarted, because the libvirtd service in fact monitors the vhost execution but the vhosts are running in their own processes.

In many cases the communication with the affected vhost or at least the restart of the vhost will be possible again after the restart of libvirtd. But in some cases the restart of libvirtd did not help.

Please note:
There exist no completely reliable statements, that the restart of the libvirtd service under all circumstances - e.g. also after an update of libvirtd - will certainly not interrupt the execution of vhosts. In order to be 100% secure, possibly move all other running vhosts to other KVM hosts before the restart of libvirtd. After move of all vhosts the third solution is unnecessary because the KVM host also could be rebooted.

As a last resort create a new vhost using the virtual hard disk of the failed vhost and abandon the failed vhost to his fate till the next reboot of the KVM host.

At first, save the definition of the old host:

root@kvm:~# virsh dumpxml vhost >vhost_new.xml

Afterwards the saved xml file has to be edited to change the following information:

• The name of the vhost (tags <name>…</name>)
• The UUID of the vhost (tags <uuid>…</uuid>)

A new and valid UUID can be generated with the command uuid from the software package 'uuid'.

In a next step the new vhost is defined and started:

root@kvm:~# virsh define vhost_new.xml
root@kvm:~# virsh start vhost_new

previous chapter | contents | next chapter