Differences

This shows you the differences between two versions of the page.

--- public:trouble:cache [2017/11/12 03:06] – [Solution] Extended content wiki.tk
+++ public:trouble:cache [2018/01/11 13:21] (current) – renamed iscsi client as iscsi initiator wiki.tk
@@ Line 1: / Line 1: @@
+===== Invisible LVM device =====
+==== Error description ====
+An LVM partition is visible on an iSCSI initiator:
+  root@kvm55:~# lvs
+    LV              VG           Attr   LSize   Origin …
+    images          vg           -wi-ao 200.00g
+    lv_dbslave5501  vg           -wi-a-  20.00g
+    lv_dbserver5501 vg3_dbserver -wi-ao   4.98g
+On another iSCSI initiator of the same iSCSI device this LVM partition is not visible (in the example: 'lv_dbserver5501'):
+  root@kvm58:~# lvs
+    LV              VG           Attr   LSize   Origin …
+    images          vg           -wi-ao 200.00g
+    lv_nas5801      vg           -wi-a- 200.00g
+The DM tools prove, that this is a pure LVM problem: The corresponding DM device is visible on both hosts:
+  root@kvm55:~# dmsetup ls
+ac21d622071  (252:4)
+  vg-images          (252:0)
+  vg3_nas-lv_nas5501 (252:5)
+ac21d632071  (252:2)
+  vg3_dbserver-lv_dbserver5501   (252:3)
+  vg-lv_dbslave5501  (252:1)
+  root@kvm58:~# dmsetup ls
+ac21d622071  (252:4)
+  vg-images          (252:0)
+  vg3_nas-lv_nas5501 (252:5)
+ac21d632071  (252:2)
+  vg3_dbserver-lv_dbserver5501   (252:3)
+  vg-lv_nas5801      (252:1)
+==== Failure cause ====
+The error was observed on Debian 9 while the LVM chache service 'lvm2-lvmetad' was running on both hosts.
+==== Solution ====
+Don't change any further LVM devices on any iSCSI client using the affected iSCSI device. Further changes might render the LVM meta-data invalid that have to be used for restoring the LVM partition.
+=== First solution step ===
+Make sure the error must not occur again by disabling the LVM cache on all affected KVM hosts:
+  - Set the option 'use_lvmetad = 0' in section 'global' of LVM configuration file '/etc/lvm/lvm.conf'.
+  - Stop the service 'lvm2-lvmetad': ''service lvm2-lvmetad stop''
+On all affected KVM hosts reread the meta-data information with ''pvscan; vgscan; lvscan''. Execute ''lvs'' on all affected hosts and check whether the missing LVM partition reappeared. If this is the case, you are done.
+=== Second solution step ===
+If the missing LVM partition did not reappear, it has to be re-added. Search for the latest definition of the partition in one of the LVM meta-data archive files in directory '/etc/lvm/archive'. The partition definition can look as follows:
+  vg3_dbserver {
+  ...
+    logical_volumes {
+    ...
+      lv_dbserver5501 {
+        id = "lRbYQs-c7xf-vofG-JUsk-xX7s-PKGm-1P7NoV"
+        status = ["READ", "WRITE", "VISIBLE"]
+        flags = []
+        creation_time = 1510388751	# 2017-11-11 09:25:51 +0100
+        creation_host = "kvm55"
+        segment_count = 1
+        segment1 {
+          start_extent = 0
+          extent_count = 3840	# 15 Gigabytes
+          type = "striped"
+          stripe_count = 1	# linear
+          stripes = [
+            "pv0", 0
+          ]
+        }
+      }
+      ...
+    }
+  }
+The option 'creation_time' (see above) can help to identify the right partition definition.
+Make a copy of the recent LVM metadata backup file '/etc/lvm/backup/vg3_dbserver' and add the LVM partition definition depicted above at the end of the file copy in the right place. Test the new file for syntax errors:
+  root@kvm55:~# vgcfgrestore -f ~/new_config -t vg3_dbserver
+    TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.
+    Restored volume group vg3_dbserver
+If this was successful, apply the new configuration:
+  root@kvm55:~# vgcfgrestore -f ~/new_config vg3_dbserver
+    Restored volume group vg3_dbserver
+''lvs'' should now display the missing partition.
+[[public:trouble:dm|previous chapter]] | [[public:start|contents]] | [[public:trouble:comm|next chapter]]