mdadm RAID5 won’t reassmble after 1 drive became temporarily unavailable

Recovering a Degraded mdadm RAID5 Array After Drive Disconnection: A Step-by-Step Guide

Introduction

Managing RAID arrays with mdadm provides a robust solution for data redundancy and performance. However, situations can arise where a drive temporarily disconnects or becomes unresponsive, leading to a degraded RAID5 array. Recovering such an array requires careful assessment and possibly some manual intervention. This article guides you through diagnosing the problem and restoring your RAID5 array, even if it involves minor data inconsistencies.

Understanding the Current Situation

In this scenario, your RAID5 array consists of three NVMe SSDs, configured via mdadm. One drive (nvme2) has temporarily disconnected, resulting in a degraded array. Despite efforts, mdadm refuses to reassemble the array automatically, treating the drive as a spare that is not sufficiently synchronized to rebuild the array.

Key observations from your assessment:

• Drives nvme1 and nvme3 are in sync and marked as active.
• Drive nvme2 is recognized but marked as spare, behind in synchronization.
• The array’s status indicates it is clean but degraded due to the missing or out-of-sync drive.

Troubleshooting and Recovery Steps

1. Confirm Drive Health and Connectivity

First, ensure all drives are physically connected and healthy:

bash
smartctl -a /dev/nvmeXn1p1

Check for SMART errors or predicted failures. If the drive appears healthy, proceed.

1. Examine RAID Metadata and State

Review detailed mdadm status:

bash
cat /proc/mdstat
mdadm --detail /dev/md0

This will give insights into the status of each device and the overall array health.

1. Attempt Re-Adding the Missing Drive

Since mdadm treats nvme2 as a spare, try to re-add it explicitly:

bash
sudo mdadm --add /dev/md0 /dev/nvme2n1p1

Monitor the re-synchronization process:

bash
cat /proc/mdstat

If mdadm refuses to add the device or the array does not start rebuilding, proceed with manual intervention.

1. Manually Rebuilding the Array

Given that your drives are in sync, but mdadm isn’t recognizing this, you can force a rebuild:

• First, stop the array:

bash
sudo mdadm --stop /dev/md0

• Then, assemble the array, explicitly specifying the