====== Ubuntu - RAID - Software Raid Rebuilding Broken Raid 1 ======

The _ in the **cat /proc/mdstat** tells me the second disk (**/dev/sdb**) has failed:

<code bash>
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid1 sda1[0] sdb1[1]
      129596288 blocks [2/2] [U_]
</code>

U means up, _ means down [https://raid.wiki.kernel.org/index.php/Mdstat#.2Fproc.2Fmdstat]

First we remove the disk from the RAID array:

<code bash>
mdadm --manage /dev/md0 --remove /dev/sdb1
</code>

Make sure the server can boot from a degraded RAID array:

<code bash>
grep BOOT_DEGRADED /etc/initramfs-tools/conf.d/mdadm
</code>

If it says true, continue on.  If not, add or change it and rebuild the initramfs using the following command:

<code bash>
update-initramfs -u
</code>

We can now safely shut down the server:

<code bash>
shutdown -h 10
</code>

Replace the actual disk.  For hot swap disks this can be done while the server is on, but if a server has no hot swap disks then it should be shut down.

After that, boot the server from the first disk (via the BIOS/UEFI).  Make sure you boot to recovery mode.  Select the root shell and mount the disk read/write:

<code bash>
mount -o remount,rw /dev/sda1
</code>

Now copy the partition table to the new (in my case, empty) disk:

<code bash>
sfdisk -d /dev/sda > sfdisk /dev/sdb
</code>

This will erase data on the new disk.

Add the disk to the RAID array and wait for the rebuilding to be complete:

<code bash>
mdadm --manage /dev/md0 --add /dev/sdb1
</code>

This is a nice progress command:

<code bash>
watch cat /proc/mdstat
</code>

It will take a while on large disks:

<code bash>
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid1 sda1[0] sdb1[1]
      129596288 blocks [2/2] [U_]
      [=>...................]  recovery = 2.6% (343392/129596288) finish=67min speed=98840K/sec

unused devices: <none> 
</code>