Directly Attache Storage with redundancy and integrity using LVM

Directly Attached Storage (DAS) can be effectively used for frequent backups. Risk of data loss can be prevented by means of redundancy and integrity check mechanisms.

Data on disk can be lost due to:

• hardware error correction insufficiency or complete disk failure – which is obvious and can be remedied using multiple copies of same data on multiple disks (RAID). RAID for Logical Volumes is provided by dm-raid kernel module and offers functionality similar to mdadm utility,

• silent data corruption – when erros slip through disk’s error detection mechanisms and remain undetected. Risk can be considerably reduced by using checksums of data stored on additional integrity sub-LVs using dm-integrity module.

RAID and integrity can be combined: each RAID sub-LV has its per-sector checksums stored on integrity sub-LVs. Whenever checksum mismatch detects silent data corruption, restoration from uncorrupted source is possible thanks to disk redundancy. As of LVM tools 2.03.21 (2023-04-21), detecting which disk holds uncorrputed data would be impossible without dm-integrity. Even if 3-disk-RAID1 or RAID6 is used – which theoretically is enough to find source of single sector corruption event – the mechanisms are not implemented (see: man lvmraid, “Scrubbing Limitations”).

Setup

First create RAID1, then extend it with integrity layer. Integrity block size cannot be smaller than drive’s logical sector size and preferably should match file system block size. Even before sync is finished, filesystem can be created on LV:

# lvcreate --type raid1 --mirrors 1 -n backup -L 4t vgbackup /dev/sda /dev/sdb
# cat /sys/class/block/nvme0n1/queue/logical_block_size
512
# dumpe2fs -h /dev/vgbackup/backup | grep 'Block size'
Block size:               4096
# lvconvert --raidintegrity y --raidintegrityblocksize 4096 --raidintegritymode bitmap /dev/vgbackup/backup
# mkfs.ext4 -b 4096 -O ^has_journal /dev/vgbackup/backup

Both operations – creating RAID mirror and adding integrity layer – require initial sync, the progress of which can be monitored:

lvs -a -o name,segtype,devices,sync_percent

Drives can be detached for later synchronization at any time:

vgchange -an vgbackup