Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== RAID - mdadm - Troubleshooting - Disk Failure ======

<code bash>
cat /proc/mdstat
</code>

returns:

<code bash>
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sde[4] sdd[2](F) sdc[1] sdb[0]
      3144192 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UU_U]

unused devices: 
</code>

<WRAP info>
**NOTE:**  The **(F)**:  Indicates the /dev/sdd device has failed.
</WRAP>

----

===== Confirm the failure =====

<code bash>
mdadm --detail /dev/md0
</code>

returns:

<code bash>
/dev/md0:
        Version : 1.2
  Creation Time : Tue Sep  6 18:31:41 2011
     Raid Level : raid5
     Array Size : 3144192 (3.00 GiB 3.22 GB)
  Used Dev Size : 1048064 (1023.67 MiB 1073.22 MB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Thu Sep  8 16:14:21 2011
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 1
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : raidtest.loc:0  (local to host raidtest.loc)
           UUID : e0748cf9:be2ca997:0bc183a6:ba2c9ebf
         Events : 75

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       2       0        0        2      removed
       4       8       64        3      active sync   /dev/sde

       2       8       48        -      faulty spare   /dev/sdd
</code>

----

===== Remove the failed device from the array =====


<code bash>
mdadm --manage --remove /dev/md0 /dev/sdd
</code>

returns:

<code bash>
mdadm: hot removed /dev/sdd from /dev/md0
</code>

----

===== Obtain the serial number of the failed device =====

This allows you to refer to your documentation to know which bay the failed disk in in.

  * Some systems do not have any visible indication of which bay a specific disk is loaded into, so unless this is documented at the time the RAID was created there is a risk that the wrong physical disk may be dealt with, instead of the actual broken disk.


<code bash>
smartctl -a /dev/sdd | grep -i serial
</code>

returns:

<code bash>
Serial Number:    VB455d882e-8013d7c9
</code>

----

===== Replace the failed drive =====

Physically remove the failed drive.

Replace this failed drive with a new one.

Add the replacement drive to the array:

<code bash>
mdadm --add /dev/md0 /dev/sdd
</code>

returns:

<code bash>
mdadm: added /dev/sdd
</code>

----

===== Check the status of the RAID =====

<code bash>
mdadm --detail /dev/md0
</code>

returns:

<code bash>
/dev/md0:
        Version : 1.2
  Creation Time : Tue Sep  6 18:31:41 2011
     Raid Level : raid5
     Array Size : 3144192 (3.00 GiB 3.22 GB)
  Used Dev Size : 1048064 (1023.67 MiB 1073.22 MB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Thu Sep  8 17:03:44 2011
          State : clean, degraded, recovering
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

 Rebuild Status : 36% complete

           Name : raidtest.loc:0  (local to host raidtest.loc)
           UUID : e0748cf9:be2ca997:0bc183a6:ba2c9ebf
         Events : 85

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       5       8       48        2      spare rebuilding   /dev/sdd
       4       8       64        3      active sync   /dev/sde
</code>

----

<WRAP info>
**NOTE:**  Wait for the RAID to rebuild.

  * This will result in a fully redundant array again.

</WRAP>

<WRAP important>
**WARNING:** If another drive fails before the point a new drive is added and the rebuild is complete, it could result in all data being lost!

There are several ways around this:

  * Add a hot spare, which will cause the rebuild process to start right away if a drive fails, minimizing the "danger time".
    * This is simply adding another drive while the array already has enough.  It will automatically get picked up and a rebuild will occur if a drive fails.

  * Have another raid level, such as raid 6, which can survive 2 failures.

<code bash>
mdadm --add /dev/md0 /dev/sdf
</code>

returns:

<code bash>
mdadm: added /dev/sdf
</code>
</WRAP>

----

===== Grow the array =====

<code bash>
mdadm --grow /dev/md0 --level=6 --raid-devices=5 --backup-file=/root/backup
</code>

returns:

<code bash>
mdadm level of /dev/md0 changed to raid6
</code>

<WRAP info>
**NOTE:**  Notice the **backup-file** argument.

Ensure:

  * The backup-file location must NOT be in the array.
  * There is sufficient space for this backup file.
    * This data will be deleted after the rebuild is complete.
</WRAP>

----

===== Check =====

<code bash>
cat /proc/mdstat
</code>

returns:

<code bash>
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdf[6] sdd[5] sde[4] sdc[1] sdb[0]
      3144192 blocks super 1.2 level 6, 512k chunk, algorithm 18 [5/4] [UUUU_]
      [==>..................]  reshape = 13.2% (139264/1048064) finish=4.4min speed=3373K/sec

unused devices: 
</code>

----

===== Another Check =====

<code bash>
mdadm --detail /dev/md0
</code>

returns:

<code bash>
/dev/md0:
        Version : 1.2
  Creation Time : Tue Sep  6 18:31:41 2011
     Raid Level : raid6
     Array Size : 3144192 (3.00 GiB 3.22 GB)
  Used Dev Size : 1048064 (1023.67 MiB 1073.22 MB)
   Raid Devices : 5
  Total Devices : 5
    Persistence : Superblock is persistent

    Update Time : Thu Sep  8 18:54:26 2011
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : raidtest.loc:0  (local to host raidtest.loc)
           UUID : e0748cf9:be2ca997:0bc183a6:ba2c9ebf
         Events : 2058

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       5       8       48        2      active sync   /dev/sdd
       4       8       64        3      active sync   /dev/sde
       6       8       80        4      active sync   /dev/sdf
</code>


<WRAP info>
**NOTE:**  If for whatever reason you want to go from 6 drives to 5, you can also do that.

  * This will then end up with a hot spare. 

</WRAP>