Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
===== iSCSI hard disk errors =====
==== Error description ====
When using kernel versions  of v3.19 and above multipath connection losses and scsi disk errors occur and are reported in log files. Affected virtual hosts can become inaccessible and virtual disks stored on iscsi devices can become corrupted.

As a result LVM devices cannot be properly read and communication with the libvirt-bin process e.g. from the command line or from the Virtal Machine Manager leads to time-outs. In the end the affected KVM host has to be rebooted thus loosing all running virtual machines. 
==== Failure cause ====
Most of the iSCSI devices report maximum block sizes (aka 'maximum transfer length') they can handle.\\ 
__Hint:__ For Ubuntu the command ''sg_vpd'' is part of the package 'sg3-utils'.
  root@kvm55:~# sg_vpd -p bl /dev/sdd
  Block limits VPD page (SBC):
    Write same no zero (WSNZ): 1
    Maximum compare and write length: 1 blocks
    Optimal transfer length granularity: 1 blocks
    Maximum transfer length: 4294967295 blocks
    Optimal transfer length: 4294967295 blocks
    Maximum prefetch length: 0 blocks
    Maximum unmap LBA count: 8388607
    Maximum unmap block descriptor count: 1
    Optimal unmap granularity: 16383
    Unmap granularity alignment valid: 0
    Unmap granularity alignment: 0
    Maximum write same length: 0xffffffff blocks

Some iSCSI devices report no maximum block size. Examples are known for Dell MD3200i (own experience), [[https://forum.synology.com/enu/viewtopic.php?f=147&t=100197&sid=d93a88568f62e60aa3a5af5f6d36b43f|Sinology DS-1813+]] or [[http://forum.qnap.com/viewtopic.php?t=108970|QNAP TS-669]]. Further details for Sinology devices can also be found [[https://groups.google.com/forum/#!topic/open-iscsi/6bp5FMM8DwI|here]]

In such a case kernels before v3.19 use a maximum block size of 512. Kernels of version v3.19 and above use a default block size of 32767 when no maximum block size is reported by the iSCSI device. Some iSCSI devices cannot handle such block sizes and generate the observed multipath and disk errors.
==== Solution ====
The old block size has to be set for all affected devices. This can be done like so:
   echo 512 >/sys/block/sdx/queue/hw_sector_size
In the example above 'sdx' is the name of an exisiting iSCSI disk device. All existing block device names (like e.g. 'dm-4') can be used instead.

**__Please note:__**
  - According to tests in production environments, the old block size has to be set **for all affected block devices** - also known as DM devices:
    * iSCSI disk devices (can be found by the command ''lsscsi'')
    * LVM physical devices on iSCSI devices (found by ''pvs'' and ''dmsetup ls'')
    * LVM logical volumes on iSCSI devices (found by ''lvs'' and ''dmsetup ls'')
  - The block sizes have to be set **before** the affected device is used.
  - Affected block devices can be created while a KVM host is running. After creation and before use of the device its block size has to be fixed.

[[public:trouble:network|previous chapter]] | [[public:start|contents]] | [[public:trouble:vmbuilder|next chapter]]