linux: lvm
DESCRIPTION
Linux LVM2, examplesTRANSCRIPT
2/29
Common problems
● Resize partition?● Use space on separated disks?● Use speed of multiple devices?● Write same data on more devices?● Backup used filesystem?
3/29
LVM
● Flexible capacity - multiple disks/partitions● Resizeable storage pools - extend/reduce● Online data relocation● Convenient device naming - names/tags● Disk striping ● Mirroring volumes● Volume Snapshots
4/29
Concept
5/29
PV
● partition, disk, LUN● LVM label - 2nd sector● 0, 1 or 2 copies of ASCII metadata● More PV on one disk?
# pvcreate /dev/sd[a-c]1
6/29
VG
# vgcreate -s 4M vg_test /dev/sd[a-c]1
● Pool of PVs● extent
● fixed for whole VG● smallest allocable space
● Size of logical extent = Size of physical extent● Map logical extents to physical extents
7/29
LV
● Linear● Striped● Mirrored● Snapshot● Virtual
8/29
Linear● concatenate physical
extents to logical volume
● flexible space allocation
● Example:● 2x99 physical extents● 198 logical extents
# lvcreate -L 50G -n lv_linear vg_test# lvcreate -l 100%FREE -n lv_linear vg_test
9/29
Striped● round robin● effective for large I/O
# lvcreate -L 50G -i 3 -I 128 -n lv_linear vg_test
10/29
Mirrored
● Identical copies● Mirror log on another
device or in memory● Mirror != backup● mirror_image_fault_p
olicy● allocate● remove
11/29
Mirror creation# lvcreate -m1 -L 1G -n lv_mirror vg_test# lvcreate -m1 -L 2T -R 2 -n lv_mirror vg_test# lvcreate -m1 -L 1G --mirrorlog core -n lv_mirror vg_test# lvcreate -m1 -L 1G -n lv_mirror -alloc anywhere vg_test# lvcreate -m1 -L 1G --mirrorlog mirrored -n lv_mirror vg_test# lvcreate -m1 -L 1G -n lv_mirror vg_test /dev/sda1 /dev/sdb1 /dev/sdc1
# lvcreate -m1 -l 5 --mirrorlog mirrored -n lv_mirror vg_test /> /dev/sda1:0-4 /dev/sdb1:0-4 /dev/sda1:5 /dev/sdb1:5
# lvcreate -m1 -i2 --mirrorlog core -n lv_lvm vg_four
12/29
Snapshot
● stores only differences● origin and snapshot, both are RW● snapshot != backup● 100% full is lost completely● Use
● backup● fsck● test/rollback on production data “--merge”● virtual machines
lvcreate --size 100M --snapshot --name lv_snap /dev/vg0/lv0
13/29
Virtual
● --virtualsize VirtualSize● sparse device
$ lvcreate --virtualsize 1T --size 100M --snapshot --name sparse vg1
14/29
Device mapper
● linear● striped● mirror● snapshot and snapshot-origin● error● zero● multipath● crypt
15/29
$ sudo dmsetup ls --treevg0-lv0 (253:5) ├─vg0-lv0_mimage_1 (253:2) │ ├─ (7:2) │ ├─ (7:1) │ └─ (7:0) ├─vg0-lv0_mimage_0 (253:1) │ ├─ (7:6) │ ├─ (7:5) │ └─ (7:4) └─vg0-lv0_mlog (253:0) └─ (7:3)
$ sudo dmsetup statusvg0-lv0_mlog: 0 8192 linear vg0-lv0_mimage_1: 0 2850816 striped 3 7:0 7:1 7:2 1 AAAvg0-lv0_mimage_0: 0 2850816 striped 3 7:4 7:5 7:6 1 AAAvg0-lv0: 0 2850816 mirror 2 253:1 253:2 2784/2784 1 AA 3 disk 253:0 A
16/29
$ sudo dmsetup infoName: vg0-lv0_mlogState: ACTIVERead Ahead: 256Tables present: LIVEOpen count: 1Event number: 0Major, minor: 253, 0Number of targets: 1UUID: LVM-ggZ5NqMTOZx43akuLZxxn03JdQGNta6gWlHnr0WT1Ua8Ek56ke6X9CdH8m3OVsAg
Name: vg0-lv0_mimage_1State: ACTIVERead Ahead: 768Tables present: LIVEOpen count: 1Event number: 0Major, minor: 253, 2Number of targets: 1UUID: LVM-ggZ5NqMTOZx43akuLZxxn03JdQGNta6gKIufDqbRgOaURN83IeA2AbYfH0zwTI0d
Name: vg0-lv0_mimage_0State: ACTIVERead Ahead: 768Tables present: LIVEOpen count: 1Event number: 0Major, minor: 253, 1Number of targets: 1UUID: LVM-ggZ5NqMTOZx43akuLZxxn03JdQGNta6g23mlbEbEk5r5l9Hy6YgRxuu2hd4DFg0Q
Name: vg0-lv0State: ACTIVERead Ahead: 256Tables present: LIVEOpen count: 1Event number: 1Major, minor: 253, 5Number of targets: 1UUID: LVM-ggZ5NqMTOZx43akuLZxxn03JdQGNta6g8uksf85SYd6BEr3fJvYbpjznRnuaVaBQ
17/29
Example: Striped Mirror
18/29
$ cd ~/tmp
$ for i in {0..6}; do dd if=/dev/zero of=lvm$i.img bs=1 count=0 / seek=505M; done
$ ls -lh lvm*-rw-rw-r-- 1 michal michal 505M 2011-10-20 22:36 lvm0.img-rw-rw-r-- 1 michal michal 505M 2011-10-20 22:36 lvm1.img-rw-rw-r-- 1 michal michal 505M 2011-10-20 22:36 lvm2.img-rw-rw-r-- 1 michal michal 505M 2011-10-20 22:36 lvm3.img-rw-rw-r-- 1 michal michal 505M 2011-10-20 22:36 lvm4.img-rw-rw-r-- 1 michal michal 505M 2011-10-20 22:36 lvm5.img-rw-rw-r-- 1 michal michal 505M 2011-10-20 22:36 lvm6.img
$ for i in {0..6}; do sudo losetup /dev/loop$i lvm$i.img; done
$ sudo losetup -a/dev/loop0: [0802]:1966260 (/home/michal/tmp/lvm0.img)/dev/loop1: [0802]:1966347 (/home/michal/tmp/lvm1.img)/dev/loop2: [0802]:1966350 (/home/michal/tmp/lvm2.img)/dev/loop3: [0802]:1966351 (/home/michal/tmp/lvm3.img)/dev/loop4: [0802]:1966354 (/home/michal/tmp/lvm4.img)/dev/loop5: [0802]:1966355 (/home/michal/tmp/lvm5.img)/dev/loop6: [0802]:1966356 (/home/michal/tmp/lvm6.img)
19/29
$ sudo pvcreate /dev/loop[0-6] Physical volume "/dev/loop0" successfully created Physical volume "/dev/loop1" successfully created Physical volume "/dev/loop2" successfully created Physical volume "/dev/loop3" successfully created Physical volume "/dev/loop4" successfully created Physical volume "/dev/loop5" successfully created Physical volume "/dev/loop6" successfully created
$ sudo pvs PV VG Fmt Attr PSize PFree /dev/loop0 lvm2 a- 500,00m 500,00m /dev/loop1 lvm2 a- 500,00m 500,00m /dev/loop2 lvm2 a- 500,00m 500,00m /dev/loop3 lvm2 a- 500,00m 500,00m /dev/loop4 lvm2 a- 500,00m 500,00m /dev/loop5 lvm2 a- 500,00m 500,00m /dev/loop6 lvm2 a- 500,00m 500,00m
$ sudo vgcreate vg0 /dev/loop[0-6] Volume group "vg0" successfully created
$ sudo vgs VG #PV #LV #SN Attr VSize VFree vg0 7 0 0 wz--n- 3,39g 3,39g
20/29
$ sudo vgdisplay --- Volume group --- VG Name vg0 System ID Format lvm2 Metadata Areas 7 Metadata Sequence No 22 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 7 Act PV 7 VG Size 3,39 GiB PE Size 4,00 MiB Total PE 868 Alloc PE / Size 0 / 0 Free PE / Size 868 / 3,39 GiB VG UUID ggZ5Nq-MTOZ-x43a-kuLZ-xxn0-3JdQ-GNta6g
$ sudo lvcreate -i2 -m1 -L1200m -n lv0 vg0 Using default stripesize 64,00 KiB Logical volume "lv0" created
21/29
$ sudo lvs -a -o +devices LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices lv0 vg0 mwi-a- 1,17g lv0_mlog 10,29 lv0_mimage_0(0),lv0_mimage_1(0) [lv0_mimage_0] vg0 Iwi-ao 1,17g /dev/loop0(0),/dev/loop1(0) [lv0_mimage_0] vg0 Iwi-ao 1,17g /dev/loop2(0),/dev/loop3(0) [lv0_mimage_1] vg0 Iwi-ao 1,17g /dev/loop4(0),/dev/loop5(0) [lv0_mimage_1] vg0 Iwi-ao 1,17g /dev/loop2(26),/dev/loop3(27) [lv0_mlog] vg0 lwi-ao 4,00m /dev/loop3(26)
$ sudo mkfs.ext4 /dev/vg0/lv0
$ sudo lvconvert -m0 --stripes 3 vg0/lv0 Logical volume lv0 converted.
$ sudo mount /dev/vg0/lv0 /mnt
$ sudo lvs -a -o +devices LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices lv0 vg0 -wi-a- 1,17g /dev/loop0(0),/dev/loop1(0) lv0 vg0 -wi-a- 1,17g /dev/loop2(0),/dev/loop3(0)
$ sudo lvconvert -m1 --stripes 3 vg0/lv0 Using default stripesize 64,00 KiB vg0/lv0: Converted: 0,3% vg0/lv0: Converted: 4,7% . . vg0/lv0: Converted: 100%
22/29
$ sudo lvs -a -o +devices LV VG Attr LSize Log Devices lv0 vg0 mwi-a- 1,17g lv0_mlog lv0_mimage_0(0),lv0_mimage_1(0) [lv0_mimage_0] vg0 Iwi-ao 1,17g /dev/loop0(0),/dev/loop1(0) [lv0_mimage_0] vg0 Iwi-ao 1,17g /dev/loop2(0),/dev/loop3(0) [lv0_mimage_1] vg0 Iwi-ao 1,17g /dev/loop4(0),/dev/loop5(0),/dev/loop6(0) [lv0_mlog] vg0 lwi-ao 4,00m /dev/loop3(26)
$ sudo lvconvert --splitmirrors 1 --name lv_split vg0/lv0 Logical volume lv0 converted.
$ sudo lvs -a -o +devices LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices lv0 vg0 -wi-a- 1,17g /dev/loop4(0),/dev/loop5(0),/dev/loop6(0) lv_split vg0 -wi-a- 1,17g /dev/loop0(0),/dev/loop1(0) lv_split vg0 -wi-a- 1,17g /dev/loop2(0),/dev/loop3(0)
$ sudo lvremove vg0/lv_splitDo you really want to remove active logical volume lv_split? [y/n]: y Logical volume "lv_split" successfully removed
$ sudo lvs -a -o +devices LV VG Attr LSize Origin Snap% Move Devices lv0 vg0 -wi-a- 1,17g /dev/loop4(0),/dev/loop5(0),/dev/loop6(0)
$ sudo lvconvert -m1 --stripes 3 vg0/lv0 Using default stripesize 64,00 KiB
23/29
$ sudo lvs -a -o +devices LV VG Attr LSize Log Devices lv0 vg0 mwi-a- 1,17g lv0_mlog lv0_mimage_0(0),lv0_mimage_1(0) [lv0_mimage_0] vg0 Iwi-ao 1,17g /dev/loop4(0),/dev/loop5(0),/dev/loop6(0) [lv0_mimage_1] vg0 Iwi-ao 1,17g /dev/loop0(0),/dev/loop1(0),/dev/loop2(0) [lv0_mlog] vg0 lwi-ao 4,00m /dev/loop3(0)
$ fsck /dev/vg0/lv0 fsck from util-linux 2.19.1e2fsck 1.41.14 (22-Dec-2010)/dev/mapper/vg0-lv0 is mounted.
WARNING!!! The filesystem is mounted. If you continue you ***WILL***cause ***SEVERE*** filesystem damage.
Do you really want to continue (y/n)? no
check aborted.
$ sudo lvcreate --size 100M --snapshot --name lv_snap /dev/vg0/lv0 Logical volume "lv_snap" created
$ sudo fsck /dev/vg0/lv_snap fsck from util-linux 2.19.1e2fsck 1.41.14 (22-Dec-2010)/dev/mapper/vg0-lv_snap: clean, 12/76800 files, 13475/307200 blocks
24/29
$ sudo lvs -a -o +devices LV VG Attr LSize Log Devices lv0 vg0 mwi-ao 1,36g lv0_mlog lv0_mimage_0(0),lv0_mimage_1(0) [lv0_mimage_0] vg0 Iwi-ao 1,36g /dev/loop4(0),/dev/loop5(0),/dev/loop6(0) [lv0_mimage_1] vg0 Iwi-ao 1,36g /dev/loop0(0),/dev/loop1(0),/dev/loop2(0) [lv0_mlog] vg0 lwi-ao 4,00m /dev/loop3(0)
$ df -h /mntSúborový systém Size Used Avail Use% Pripojený na/dev/mapper/vg0-lv0 1,2G 34M 1,1G 4% /mnt
$ sudo resize2fs /dev/vg0/lv0 resize2fs 1.41.14 (22-Dec-2010)Filesystem at /dev/vg0/lv0 is mounted on /mnt; on-line resizing requiredold desc_blocks = 1, new_desc_blocks = 1Performing an on-line resize of /dev/vg0/lv0 to 356352 (4k) blocks.The filesystem on /dev/vg0/lv0 is now 356352 blocks long.
$ df -h /mntSúborový systém Size Used Avail Use% Pripojený na/dev/mapper/vg0-lv0 1,4G 34M 1,3G 3% /mnt
25/29
$ sudo dd if=/dev/zero of=/dev/loop1 count=1010+0 záznamov prečítaných10+0 záznamov zapísaných5120 skopírovaných bajtov (5,1 kB), 0,00188767 s, 2,7 MB/s
$ sudo lvs -a -o +devices Couldn't find device with uuid hmPWAT-kPoM-yHZ8-Yh5M-m11G-yDjI-WLA28J. LV VG Attr LSize Log Devices lv0 vg0 mwi-ao 1,36g lv0_mlog lv0_mimage_0(0),lv0_mimage_1(0) [lv0_mimage_0] vg0 iwi-ao 1,36g /dev/loop4(0),/dev/loop5(0),/dev/loop6(0) [lv0_mimage_1] vg0 iwi-ao 1,36g /dev/loop0(0),unknown device(0),/dev/loop2(0) [lv0_mlog] vg0 lwi-ao 4,00m /dev/loop3(0)
$ sudo vgreduce --removemissing vg0 Couldn't find device with uuid hmPWAT-kPoM-yHZ8-Yh5M-m11G-yDjI-WLA28J. WARNING: Partial LV lv0 needs to be repaired or removed. WARNING: Partial LV lv0_mimage_1 needs to be repaired or removed. WARNING: There are still partial LVs in VG vg0. To remove them unconditionally use: vgreduce --removemissing --force. Proceeding to remove empty missing PVs.
$ sudo vgreduce --removemissing vg0 --force Couldn't find device with uuid hmPWAT-kPoM-yHZ8-Yh5M-m11G-yDjI-WLA28J. WARNING: Bad device removed from mirror volume, vg0/lv0 WARNING: Mirror volume, vg0/lv0 converted to linear due to device failure. Wrote out consistent volume group vg0
26/29
$ sudo lvs -a -o +devices LV VG Attr LSize Devices lv0 vg0 -wi-ao 1,36g /dev/loop4(0),/dev/loop5(0),/dev/loop6(0)
$ sudo pvcreate --uuid hmPWAT-kPoM-yHZ8-Yh5M-m11G-yDjI-WLA28J /dev/loop1 Physical volume "/dev/loop1" successfully created
$ sudo vgcfgrestore -f /etc/lvm/archive/vg0_00170.vg vg0 Restored volume group vg0
$ sudo lvs -a -o +devices LV VG Attr LSize Log Devices lv0 vg0 mwi-ao 1,27g lv0_mlog lv0_mimage_0(0),lv0_mimage_1(0) [lv0_mimage_0] vg0 iwi--- 1,27g /dev/loop4(0),/dev/loop5(0),/dev/loop6(0) [lv0_mimage_1] vg0 iwi--- 1,27g /dev/loop0(0),/dev/loop1(0),/dev/loop2(0) [lv0_mlog] vg0 lwi--- 4,00m /dev/loop3(0)
Corrupted metadata
27/29
New disk$ sudo lvs -a -o +devices LV VG Attr LSize Devices lv0 vg0 -wi-ao 1,36g /dev/loop4(0),/dev/loop5(0),/dev/loop6(0)
$ sudo pvcreate --restorefile /etc/lvm/archive/vg0_00170.vg --uuid "hmPWAT-kPoM-yHZ8-Yh5M-m11G-yDjI-WLA28J" Physical volume "/dev/loop1" successfully created
$ sudo pvs PV VG Fmt Attr PSize PFree /dev/loop0 vg0 lvm2 a- 496,00m 496,00m /dev/loop1 lvm2 a- 505,00m 505,00m /dev/loop2 vg0 lvm2 a- 496,00m 496,00m /dev/loop3 vg0 lvm2 a- 496,00m 496,00m /dev/loop4 vg0 lvm2 a- 496,00m 64,00m /dev/loop5 vg0 lvm2 a- 496,00m 64,00m /dev/loop6 vg0 lvm2 a- 496,00m 64,00m
$ sudo vgextend vg0 /dev/loop1 Volume group "vg0" successfully extended
$ sudo lvconvert -m1 --stripes 3 vg0/lv0
$ sudo lvs -a -o +devices LV VG Attr LSize Log Devices lv0 vg0 mwi-ao 1,27g lv0_mlog lv0_mimage_0(0),lv0_mimage_1(0) [lv0_mimage_0] vg0 Iwi-ao 1,27g /dev/loop4(0),/dev/loop5(0),/dev/loop6(0) [lv0_mimage_1] vg0 Iwi-ao 1,27g /dev/loop0(0),/dev/loop2(0),/dev/loop3(0) [lv0_mlog] vg0 lwi-ao 4,00m /dev/loop1(0)
28/29
Caveats
● Until Linux kernel 2.6.31[1], write barriers were not supported (fully supported in 2.6.33). This means that the guarantee against filesystem corruption offered by journaled file systems like ext3 and XFS was negated under some circumstances.[2]
● Device mapper does not honor barriers● Barriers are disabled by default● mount -t ext3 -o barrier=1 <device> <mount
point>
29/29
Thank you
Questions
30/29
Thank you
Questions
31/29
RHEL 6
● mirror_image_fault_policy=allocate/remove● data_alignment_detection● udev integration● lvconvert --repair● lvconvert --merge● lvconvert --splitmirrors● lvcreate --mirrorlog mirrored● lvextend --alloc cling● multiple --addtag and --deltag