Recently I have deployed a PVE cluster with a set of 4 hypervisors and an HP MSA2050 Fiber Channel SAN. Now I had to extend the volume because the organization needed more space.
The main problem is that you have to ensure that all the chain of connection is aware of the disk size change before being able to extend the PV can be extended. Here’s how I did.
Firs I had to check which disks were involved in the multipath device.
# multipath -ll 3600c0ff00050dacf6821485e01000000 dm-5 HPE,MSA 2050 SAN size=3.6T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 5:0:0:1 sde 8:64 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 3:0:0:1 sdc 8:32 active ready running
In bold you can find the information needed by the process. We have the serial number of the multipath device, its size and the SCSI bus and the device name of every device composing the multipath device.
So, first of all we have to check if the device is already updated with the new capacity, if not, we will force the rescan of the corresponding SCSI bus, and then we will resize of the multipath device, then we will resize the PV device.
# fdisk /dev/sde Welcome to fdisk (util-linux 2.33.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. The old LVM2_member signature will be removed by a write command. Device does not contain a recognized partition table. The size of this disk is 3.5 TiB (3848290697216 bytes). DOS partition table format cannot be used on drives for volumes larger than 2199023255040 bytes for 512-byte sectors. Use GUID partition table format (GPT). Created a new DOS disklabel with disk identifier 0xaeac5073. Command (m for help): q # fdisk /dev/sdc Welcome to fdisk (util-linux 2.33.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. The old LVM2_member signature will be removed by a write command. Device does not contain a recognized partition table. The size of this disk is 3.5 TiB (3848290697216 bytes). DOS partition table format cannot be used on drives for volumes larger than 2199023255040 bytes for 512-byte sectors. Use GUID partition table format (GPT). Created a new DOS disklabel with disk identifier 0xdf0c56c7. Command (m for help): q
!!!! RUN THE FOLLOWING 4 COMMANDS ON EACH HOST AND VERIFY THE RESULT BEFORE RESIZING THE MULTIPATH !!!! !!!! PAY ATTENTION TO HAVE THE COMMANDS READY TO BE RUN ON EACH HOST AND PAY ATTENTION TO ADAPT THE NUMBER OF THE SCSI HOST AND FIBERCHANNEL HOST THAT CAN VARY ON EACH HOST IT IS THE NUMBER WE FOUND BEFORE WITH multipath -ll (5:0:0:1 and 3:0:0:1)!!!!
# echo "1" > /sys/class/fc_host/host3/issue_lip
# echo "1" > /sys/class/fc_host/host5/issue_lip
# echo "- - -" > /sys/class/scsi_host/host3/scan
# echo "- - -" > /sys/class/scsi_host/host5/scan
# fdisk /dev/sde
Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
The old LVM2_member signature will be removed by a write command.
Device does not contain a recognized partition table.
The size of this disk is 4.7 TiB (5099996839936 bytes). DOS partition table format cannot be used on drives for volumes larger than 2199023255040 bytes for 512-byte sectors. Use GUID partition table format (GPT).
Created a new DOS disklabel with disk identifier 0xaeac5073.
Command (m for help): q
# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
The old LVM2_member signature will be removed by a write command.
Device does not contain a recognized partition table.
The size of this disk is 4.7 TiB (5099996839936 bytes). DOS partition table format cannot be used on drives for volumes larger than 2199023255040 bytes for 512-byte sectors. Use GUID partition table format (GPT).
Created a new DOS disklabel with disk identifier 0xdf0c56c7.
Command (m for help): q
# systemctl status multipathd
● multipathd.service - Device-Mapper Multipath Device Controller
Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-04-03 12:23:39 CEST; 5h 57min ago
Process: 976 ExecStartPre=/sbin/modprobe -a scsi_dh_alua scsi_dh_emc scsi_dh_rdac dm-multipath (code=exited, status=0/
Main PID: 986 (multipathd)
Status: "up"
Tasks: 7
Memory: 13.8M
CGroup: /system.slice/multipathd.service
└─986 /sbin/multipathd -d -s
Apr 03 12:23:38 pve3 systemd[1]: Starting Device-Mapper Multipath Device Controller...
Apr 03 12:23:38 pve3 multipathd[986]: --------start up--------
Apr 03 12:23:38 pve3 multipathd[986]: read /etc/multipath.conf
Apr 03 12:23:38 pve3 multipathd[986]: path checkers start up
Apr 03 12:23:38 pve3 multipathd[986]: 3600c0ff00050dacf6821485e01000000: load table [0 9960931328 multipath 1 queue_if_n
Apr 03 12:23:39 pve3 multipathd[986]: 3600c0ff00050dddddb51495e01000000: load table [0 1660149760 multipath 1 queue_if_n
Apr 03 12:23:39 pve3 systemd[1]: Started Device-Mapper Multipath Device Controller.
!!!! PAY ATTENTION NOT TO RUN THE BELOW COMMAND ON A MULTIPATH DEVICE WITH queue_if_no_path CONFIGURATION AND NO ACTIVE PATH TO PHISICAL DEVICE !!!!
!!!! RUN THE FOLLOWING COMMAND ON EACH HOST THE ID OF MULTIPATH DEVICE IS THE SAME ON EACH HOST !!!!
# multipathd -k"resize map 3600c0ff00050dacf6821485e01000000"
#### I HAVE NO OUTPUT FOR THIS, DIDN’T REMEMBER TO COPY/PASTE BUT I DON’T REMEMBER ANYTHING PARTICULAR ####
# multipath -ll 3600c0ff00050dacf6821485e01000000 dm-5 HPE,MSA 2050 SAN size=4.6T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 5:0:0:1 sde 8:64 active ready running `-+- policy='service-time 0' prio=10 status=enabled `- 3:0:0:1 sdc 8:32 active ready running
!!!! BETTER TO RUN THE BELOW COMMAND AFTER A DRY RUN ( -t option) !!!!
#### THIS HAS ONLY TO BE RUN ON A SINGLE HOST BECAUSE THE LVM DEVICE IS SHARED BUT BETTER TO CHECK THE RESULT ON EACH HOST ####
# pvresize -v /dev/mapper/3600c0ff00050dacf6821485e01000000
#### THE BELOW COMMANDS WERE USED ONLY TO CHECK THE RESULT ####
# pvs # vgs # lvs
Now we can use the newly available space on our pve cluster. Below you can find some useful resources for the entire process.
- https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/storage_administration_guide/ch37s03
- https://manpages.debian.org/testing/multipath-tools/multipath.conf.5.en.html
That’s all for now.