Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== RAID - mdadm - LVM on RAID ======

**LVM** can group multiple disks and RAID arrays into a Volume Group.

  * This Volume Group can be split into Logical Volumes; in essence different Partitions.
  * LVM provides the ability to resize these Logical Volumes very easily.
  * LVM is not a replacement for RAID, as LVM does not provide any options for redundancy or parity that RAID provides.
    * Therefore it is best to be have LVM in conjunction with RAID.

<code>
|--------------------------------------------------------------------------------------------|
|LVM Logical Volumes | / | /var | /usr | /home | /mnt                                        |
|--------------------------------------------------------------------------------------------|
|LVM Volume Group    |                            /dev/VolGroupArray                         |
|--------------------------------------------------------------------------------------------|
|RAID Arrays         |                   /dev/md0                     |      /dev/md1        |
|--------------------------------------------------------------------------------------------|
|Physical Partitions | /dev/sda1 | /dev/sda2 | /dev/sdb1 | /dev/sdb2 | /dev/sdc1 | /dev/sdc2 |
|--------------------------------------------------------------------------------------------|
|Devices             |         /dev/sda      |         /dev/sdb      |         /dev/sdc      | 
|--------------------------------------------------------------------------------------------|
|Hard Drives         |          Drive 1      |          Drive 2      |          Drive 3      |
|--------------------------------------------------------------------------------------------|
</code>

<WRAP info>
**NOTE:**  A filesystem that can grow must be used with LVM.
</WRAP>

----

===== Resizing a RAID array =====

To resize an existing RAID5:

<code bash>
mdadm --add /dev/md1 /dev/sdb1
mdadm --grow /dev/md1 --raid-disks=4 (new number of disks)
</code>

<WRAP info>
**NOTE:**  The **--raid-disks=4** is the new number of disks.

  * This will result in the raid having to restripe itself which can take a very long time.
</WRAP>

----

===== Prepare The Disks =====

<code bash>
fdisk /dev/sda
</code>

and:

  * Enter an **n** to create a new partition.
  * Enter an **e** for an extended partition.
  * Enter a **p** for a primary partition (1-4).
  * Enter a **1** for primary partition number 1.
  * Accept the default sizes.

  * Enter a **t** to change the partition type.
  * Enter **fd** to change to **Linux raid autodetect**.

  * Enter a **w** to write the changes.

----

===== Wipe everything =====

<code bash>
mdadm --stop /dev/md0
mdadm --zero-superblock /dev/sda1
mdadm --zero-superblock /dev/sdb1
mdadm --zero-superblock /dev/sdc1
</code>

----

===== Create the RAID Array =====

<code bash>
mdadm -v --create /dev/md0 --chunk=128 --level=raid5 --raid-devices=4 /dev/sda1 \
             /dev/sdb1 /dev/sdc1 missing
</code>

returns:

<code bash>             
mdadm: layout defaults to left-symmetric
mdadm: size set to 245111616K
mdadm: array /dev/md0 started.
</code>

<WRAP info>
**NOTE:**  /dev/sdc1 is marked as **missing**.

  * This will created the array as if one of the disks is dead.
  * At a later stage the disk can be hot-added; which will result in the array rebuilding itself.

</WRAP>

----

===== Check the RAID Array =====

<code bash>
mdadm --detail /dev/md0
</code>

returns:

<code bash>
/dev/md0:
        Version : 00.90.01
  Creation Time : Thu Jun  3 20:24:17 2004
     Raid Level : raid5
     Array Size : 735334656 (701.27 GiB 752.98 GB)
    Device Size : 245111552 (233.76 GiB 250.99 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Jun  3 20:24:17 2004
          State : clean, no-errors
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 128K 

    Number   Major   Minor   Raid Device State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       0        0       -1      removed
           UUID : d6ac1605:db6659e1:6460b9c0:a451b7c8
         Events : 0.5078
</code>

----

===== Update the mdadm config file =====

<code bash>
echo 'DEVICE /dev/sd*' >/etc/mdadm/mdadm.conf
echo 'PROGRAM /bin/echo' >>/etc/mdadm/mdadm.conf
echo 'MAILADDR some@email_address.com' >>/etc/mdadm/mdadm.conf
mdadm --detail --scan >>/etc/mdadm/mdadm.conf
cat /etc/mdadm/mdadm.conf
</code>

<WRAP info>
**NOTE:**  This should be done whenever the config is changed, including adding a spare disk, mark a disk as faulty etc.
</WRAP>

----

===== Create a LVM Physical Volume =====

<code bash>
pvcreate /dev/md0
</code>

returns:

<code bash>
No physical volume label read from /dev/md0
Physical volume "/dev/md0" successfully created
</code>

<WRAP info>
**NOTE:**  This makes the RAID Array usable by LVM.

The LVM Physical Volume can be checked by running <code bash>
pvdisplay
</code>

</WRAP>

----

===== Create a LVM Volume Group =====

<code bash>
vgcreate myVolumeGroup1 /dev/md0
</code>

returns:

<code bash>
Adding physical volume '/dev/md0' to volume group 'myVolumeGroup1'
Archiving volume group "myVolumeGroup1" metadata.
Creating volume group backup "/etc/lvm/backup/myVolumeGroup1"
Volume group "myVolumeGroup1" successfully created
</code>

<WRAP info>
**NOTE:**  This Volume Group can be used to make logical volumes from.

**NOTE:** The LVM Volume Group can be checked by running <code bash>
vgdisplay
</code>

</WRAP>

----

===== Create a LVM Logical Volume =====

<code bash>
lvcreate -L 100G --name myDataVolume myVolumeGroup1
</code>

returns:

<code bash>
Logical volume "myDataVolume" created
</code>

<WRAP info>
**NOTE:** The LVM Logical Volume can be checked by running <code bash>
lvdisplay
</code>

</WRAP>

----

===== Format the LVM Logical Volume =====

<code bash>
mkfs.ext4 /dev/myVolumeGroup1/myDataVolume
</code>

<WRAP info>
**NOTE:**  Other filesystems besides ext4 can be used if preferred.

  * Reiser <code bash>
mkfs -treiserfs /dev/myVolumeGroup1/myDataVolume
</code>
  * xfs <code bash>
mkfs.xfs /dev/myVolumeGroup1/myDataVolume
</code>

</WRAP>

----

===== Mount the LVM Logical Partition =====

<code bash>
mount /dev/myVolumeGroup1/myDataVolume /mnt/data
</code>

----

===== Check the mount =====

<code bash>
df -k
</code>

returns:

<code bash>
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda2             19283776    697988  18585788   4% /
/dev/hda1                97826     13003     79604  15% /boot
/dev/myVolumeGroup1/myDataVolume
                     629126396     32840 629093556   1% /mnt/data
</code>

<WRAP info>
**NOTE:**  To auto mount, add following to /etc/fstab:

<file bash /etc/fstab>
/dev/myVolumeGroup1-myDataVolume /mnt/data       ext4     defaults        1       2
</file>

</WRAP>


----

===== Performance Enhancements / Tuning =====

There may be performance issues with LVM on RAID.

A potential fix:

<code bash>
blockdev --setra 4096 /dev/md0
blockdev --setra 4096 /dev/myVolumeGroup1/myDataVolume
</code>

<WRAP important>
**WARNING:** This may lock up the machine and destroy data.  

  * Ensure data is backed up before running this!

</WRAP>

If the system has sufficient RAM the size of the software RAID MD cache can be increased:

<code bash>
echo 8192 > /sys/block/md0/md/stripe_cache_size
</code>


----

===== References =====

https://www.mythtv.org/wiki/LVM_on_RAID