===== Invisible LVM device =====
==== Error description ====
An LVM partition is visible on an iSCSI initiator:
  root@kvm55:~# lvs
    LV              VG           Attr   LSize   Origin …
    images          vg           -wi-ao 200.00g
    lv_dbslave5501  vg           -wi-a-  20.00g
    lv_dbserver5501 vg3_dbserver -wi-ao   4.98g
On another iSCSI initiator of the same iSCSI device this LVM partition is not visible (in the example: 'lv_dbserver5501'):
  root@kvm58:~# lvs
    LV              VG           Attr   LSize   Origin …
    images          vg           -wi-ao 200.00g
    lv_nas5801      vg           -wi-a- 200.00g
The DM tools prove, that this is a pure LVM problem: The corresponding DM device is visible on both hosts:
  root@kvm55:~# dmsetup ls
  350002ac21d622071  (252:4)
  vg-images          (252:0)
  vg3_nas-lv_nas5501 (252:5)
  350002ac21d632071  (252:2)
  vg3_dbserver-lv_dbserver5501   (252:3)
  vg-lv_dbslave5501  (252:1)

  root@kvm58:~# dmsetup ls
  350002ac21d622071  (252:4)
  vg-images          (252:0)
  vg3_nas-lv_nas5501 (252:5)
  350002ac21d632071  (252:2)
  vg3_dbserver-lv_dbserver5501   (252:3)
  vg-lv_nas5801      (252:1)
==== Failure cause ====
The error was observed on Debian 9 while the LVM chache service 'lvm2-lvmetad' was running on both hosts.
==== Solution ====
Don't change any further LVM devices on any iSCSI client using the affected iSCSI device. Further changes might render the LVM meta-data invalid that have to be used for restoring the LVM partition.
=== First solution step ===
Make sure the error must not occur again by disabling the LVM cache on all affected KVM hosts:
  - Set the option 'use_lvmetad = 0' in section 'global' of LVM configuration file '/etc/lvm/lvm.conf'.
  - Stop the service 'lvm2-lvmetad': ''service lvm2-lvmetad stop''

On all affected KVM hosts reread the meta-data information with ''pvscan; vgscan; lvscan''. Execute ''lvs'' on all affected hosts and check whether the missing LVM partition reappeared. If this is the case, you are done.
=== Second solution step ===
If the missing LVM partition did not reappear, it has to be re-added. Search for the latest definition of the partition in one of the LVM meta-data archive files in directory '/etc/lvm/archive'. The partition definition can look as follows:
  vg3_dbserver {
  ...
    logical_volumes {
    ...
      lv_dbserver5501 {
        id = "lRbYQs-c7xf-vofG-JUsk-xX7s-PKGm-1P7NoV"
        status = ["READ", "WRITE", "VISIBLE"]
        flags = []
        creation_time = 1510388751	# 2017-11-11 09:25:51 +0100
        creation_host = "kvm55"
        segment_count = 1
        
        segment1 {
          start_extent = 0
          extent_count = 3840	# 15 Gigabytes
        
          type = "striped"
          stripe_count = 1	# linear
        
          stripes = [
            "pv0", 0
          ]
        }
      }
      ...
    }
  }

The option 'creation_time' (see above) can help to identify the right partition definition.

Make a copy of the recent LVM metadata backup file '/etc/lvm/backup/vg3_dbserver' and add the LVM partition definition depicted above at the end of the file copy in the right place. Test the new file for syntax errors:
  root@kvm55:~# vgcfgrestore -f ~/new_config -t vg3_dbserver
    TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.
    Restored volume group vg3_dbserver
If this was successful, apply the new configuration:
  root@kvm55:~# vgcfgrestore -f ~/new_config vg3_dbserver
    Restored volume group vg3_dbserver
''lvs'' should now display the missing partition.

[[public:trouble:dm|previous chapter]] | [[public:start|contents]] | [[public:trouble:comm|next chapter]]