Wiki

Invisible LVM device

Error description

An LVM partition is visible on an iSCSI initiator:

root@kvm55:~# lvs
  LV              VG           Attr   LSize   Origin …
  images          vg           -wi-ao 200.00g
  lv_dbslave5501  vg           -wi-a-  20.00g
  lv_dbserver5501 vg3_dbserver -wi-ao   4.98g

On another iSCSI initiator of the same iSCSI device this LVM partition is not visible (in the example: 'lv_dbserver5501'):

root@kvm58:~# lvs
  LV              VG           Attr   LSize   Origin …
  images          vg           -wi-ao 200.00g
  lv_nas5801      vg           -wi-a- 200.00g

The DM tools prove, that this is a pure LVM problem: The corresponding DM device is visible on both hosts:

root@kvm55:~# dmsetup ls
350002ac21d622071  (252:4)
vg-images          (252:0)
vg3_nas-lv_nas5501 (252:5)
350002ac21d632071  (252:2)
vg3_dbserver-lv_dbserver5501   (252:3)
vg-lv_dbslave5501  (252:1)

root@kvm58:~# dmsetup ls
350002ac21d622071  (252:4)
vg-images          (252:0)
vg3_nas-lv_nas5501 (252:5)
350002ac21d632071  (252:2)
vg3_dbserver-lv_dbserver5501   (252:3)
vg-lv_nas5801      (252:1)

Failure cause

The error was observed on Debian 9 while the LVM chache service 'lvm2-lvmetad' was running on both hosts.

Solution

Don't change any further LVM devices on any iSCSI client using the affected iSCSI device. Further changes might render the LVM meta-data invalid that have to be used for restoring the LVM partition.

First solution step

Make sure the error must not occur again by disabling the LVM cache on all affected KVM hosts:

Set the option 'use_lvmetad = 0' in section 'global' of LVM configuration file '/etc/lvm/lvm.conf'.
Stop the service 'lvm2-lvmetad': service lvm2-lvmetad stop

On all affected KVM hosts reread the meta-data information with pvscan; vgscan; lvscan. Execute lvs on all affected hosts and check whether the missing LVM partition reappeared. If this is the case, you are done.

Second solution step

If the missing LVM partition did not reappear, it has to be re-added. Search for the latest definition of the partition in one of the LVM meta-data archive files in directory '/etc/lvm/archive'. The partition definition can look as follows:

vg3_dbserver {
...
  logical_volumes {
  ...
    lv_dbserver5501 {
      id = "lRbYQs-c7xf-vofG-JUsk-xX7s-PKGm-1P7NoV"
      status = ["READ", "WRITE", "VISIBLE"]
      flags = []
      creation_time = 1510388751	# 2017-11-11 09:25:51 +0100
      creation_host = "kvm55"
      segment_count = 1
      
      segment1 {
        start_extent = 0
        extent_count = 3840	# 15 Gigabytes
      
        type = "striped"
        stripe_count = 1	# linear
      
        stripes = [
          "pv0", 0
        ]
      }
    }
    ...
  }
}

The option 'creation_time' (see above) can help to identify the right partition definition.

Make a copy of the recent LVM metadata backup file '/etc/lvm/backup/vg3_dbserver' and add the LVM partition definition depicted above at the end of the file copy in the right place. Test the new file for syntax errors:

root@kvm55:~# vgcfgrestore -f ~/new_config -t vg3_dbserver
  TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.
  Restored volume group vg3_dbserver

If this was successful, apply the new configuration:

root@kvm55:~# vgcfgrestore -f ~/new_config vg3_dbserver
  Restored volume group vg3_dbserver

lvs should now display the missing partition.

previous chapter | contents | next chapter

Table of Contents