====== Optimizing Software RAID with External USB Drives ======

{{tag>raid mdadm usb storage linux optimization troubleshooting}}

===== Overview =====

This guide covers comprehensive optimization techniques for software RAID (mdadm) configurations using external USB drives. USB drives present unique challenges in RAID environments due to power management, timeouts, and connection stability issues.

<WRAP center round info 60%>
**Use Case:** This guide is essential for NAS setups, backup servers, and any RAID configuration using external USB/eSATA drives.
</WRAP>

===== The Problem =====

External USB drives in RAID arrays commonly face these issues:

  * **USB Autosuspend**: Linux power management can suspend USB ports, causing drives to disconnect
  * **Short SCSI Timeouts**: Default 30-second timeout is too short for USB drives
  * **Aggressive APM**: Advanced Power Management can spin down drives during RAID operations
  * **Spindown**: Drives entering standby mode cause delays and potential RAID failures

**Symptoms:**
  * Drives marked as "failed" during rebuild/resync
  * Random disconnections in logs
  * RAID degraded state without physical drive failure
  * Rebuild operations that fail or restart repeatedly

===== Prerequisites =====

  * Root/sudo access
  * ''hdparm'' package installed: ''yum install hdparm'' or ''apt install hdparm''
  * Active mdadm RAID array
  * External USB drives (HDD or SSD)

===== Step 1: Diagnosis =====

Before applying optimizations, check your current configuration:

<code bash>
# Check USB autosuspend status
cat /proc/cmdline | grep autosuspend

# Check disk timeouts (should be > 30 for USB)
cat /sys/block/sda/device/timeout
cat /sys/block/sdb/device/timeout

# Check APM levels (128 = aggressive power management)
hdparm -B /dev/sda
hdparm -B /dev/sdb

# Check for recent USB disconnections
journalctl -n 100 --no-pager | grep -i "usb.*reset\|disconnect"

# Check current GRUB configuration
grep GRUB_CMDLINE_LINUX /etc/default/grub

# Check RAID status
mdadm --detail /dev/md0
cat /proc/mdstat
</code>

<WRAP center round tip 60%>
**Optimal Values:**
  * Timeout: 180 seconds
  * APM: off (255)
  * USB Autosuspend: disabled (-1)
  * Spindown: disabled (0)
</WRAP>

===== Step 2: Immediate Optimizations (Temporary) =====

Apply these settings immediately (will reset after reboot):

<code bash>
# Disable USB autosuspend for all devices
for i in /sys/bus/usb/devices/*/power/autosuspend; do 
    echo -1 > $i 2>/dev/null
done

# Increase disk timeouts to 180 seconds
echo 180 > /sys/block/sda/device/timeout
echo 180 > /sys/block/sdb/device/timeout

# Disable APM (Advanced Power Management)
hdparm -B 255 /dev/sda
hdparm -B 255 /dev/sdb

# Disable spindown
hdparm -S 0 /dev/sda
hdparm -S 0 /dev/sdb
</code>

**Verify the changes:**

<code bash>
cat /sys/block/sda/device/timeout
cat /sys/block/sdb/device/timeout
hdparm -B /dev/sda
hdparm -B /dev/sdb
</code>

===== Step 3: Permanent Optimizations =====

==== Method 1: GRUB + systemd (Recommended) ====

This is the modern, distribution-agnostic method for RHEL/CentOS/Oracle Linux 7+, Ubuntu 16.04+, Debian 8+, Fedora, etc.

=== A. Configure GRUB ===

<code bash>
# Backup GRUB configuration
cp /etc/default/grub /etc/default/grub.backup

# Add USB autosuspend disable parameter
sed -i 's/quiet"/quiet usbcore.autosuspend=-1"/' /etc/default/grub

# Regenerate GRUB configuration
# For RHEL/CentOS/Oracle Linux/Fedora:
grub2-mkconfig -o /boot/grub2/grub.cfg

# For Ubuntu/Debian:
update-grub

# Verify the change
grep GRUB_CMDLINE_LINUX /etc/default/grub
</code>

=== B. Create systemd Service ===

<code bash>
cat > /etc/systemd/system/usb-raid-tweaks.service << 'EOF'
[Unit]
Description=USB RAID Power Management Optimizations
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'sleep 5; \
    hdparm -B 255 /dev/sda; \
    hdparm -B 255 /dev/sdb; \
    hdparm -S 0 /dev/sda; \
    hdparm -S 0 /dev/sdb; \
    echo 180 > /sys/block/sda/device/timeout; \
    echo 180 > /sys/block/sdb/device/timeout; \
    for i in /sys/bus/usb/devices/*/power/autosuspend; do \
        echo -1 > $i 2>/dev/null; \
    done'
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

# Reload systemd and enable service
systemctl daemon-reload
systemctl enable usb-raid-tweaks.service
systemctl start usb-raid-tweaks.service

# Check service status
systemctl status usb-raid-tweaks.service
</code>

<WRAP center round important 60%>
**Important:** Replace ''/dev/sda'' and ''/dev/sdb'' with your actual drive names. Use ''lsblk'' to identify them.
</WRAP>

==== Method 2: rc.local (Legacy Systems) ====

For older systems without systemd:

<code bash>
# Create/edit rc.local
cat >> /etc/rc.d/rc.local << 'EOF'
#!/bin/bash

# USB RAID Optimizations
sleep 5

# Disable APM
hdparm -B 255 /dev/sda
hdparm -B 255 /dev/sdb

# Disable spindown
hdparm -S 0 /dev/sda
hdparm -S 0 /dev/sdb

# Increase timeouts
echo 180 > /sys/block/sda/device/timeout
echo 180 > /sys/block/sdb/device/timeout

# Disable USB autosuspend
for i in /sys/bus/usb/devices/*/power/autosuspend; do
    echo -1 > $i 2>/dev/null
done
EOF

# Make executable
chmod +x /etc/rc.d/rc.local

# Enable rc-local service (RHEL/CentOS 7+)
systemctl enable rc-local
</code>

===== Step 4: Verification =====

After reboot, verify all settings are applied:

<code bash>
#!/bin/bash
echo "=== USB RAID Configuration Check ==="

echo -e "\n1. GRUB Configuration:"
grep usbcore.autosuspend /proc/cmdline && echo "  ✓ USB autosuspend disabled" || echo "  ✗ USB autosuspend NOT disabled"

echo -e "\n2. Disk Timeouts:"
for disk in sda sdb; do
    timeout=$(cat /sys/block/$disk/device/timeout 2>/dev/null)
    if [ "$timeout" = "180" ]; then
        echo "  ✓ /dev/$disk: $timeout seconds"
    else
        echo "  ✗ /dev/$disk: $timeout seconds (should be 180)"
    fi
done

echo -e "\n3. APM Levels:"
for disk in sda sdb; do
    apm=$(hdparm -B /dev/$disk 2>/dev/null | grep APM_level | awk '{print $3}')
    if [ "$apm" = "off" ] || [ "$apm" = "255" ]; then
        echo "  ✓ /dev/$disk: APM disabled"
    else
        echo "  ✗ /dev/$disk: APM = $apm (should be off)"
    fi
done

echo -e "\n4. Systemd Service:"
systemctl is-enabled usb-raid-tweaks.service >/dev/null 2>&1 && echo "  ✓ Service enabled" || echo "  ✗ Service NOT enabled"
systemctl is-active usb-raid-tweaks.service >/dev/null 2>&1 && echo "  ✓ Service active" || echo "  ✗ Service NOT active"

echo -e "\n5. Recent USB Issues:"
recent_issues=$(journalctl -n 100 --no-pager 2>/dev/null | grep -i "usb.*reset\|usb.*disconnect" | grep -v "postfix\|smtpd" | wc -l)
if [ "$recent_issues" = "0" ]; then
    echo "  ✓ No recent USB disconnections"
else
    echo "  ⚠ $recent_issues USB-related messages in recent logs"
fi

echo -e "\n6. RAID Status:"
mdadm --detail /dev/md0 2>/dev/null | grep -E "State|Failed Devices|Working Devices" | while read line; do
    echo "  $line"
done
</code>

Save as ''check-usb-raid.sh'' and run: ''bash check-usb-raid.sh''

===== Monitoring RAID Rebuild =====

During RAID rebuild/resync operations:

<code bash>
# Real-time monitoring (updates every 2 seconds)
watch -n 2 cat /proc/mdstat

# Detailed status with completion percentage
mdadm --detail /dev/md0 | grep -E "State|Rebuild Status|Working Devices"

# Estimate completion time
cat /proc/mdstat
</code>

**Expected output during rebuild:**
<code>
md0 : active raid1 sda1[2] sdb1[0]
      3906886464 blocks super 1.2 [2/1] [U_]
      [===>.................]  recovery = 15.2% (595123456/3906886464) 
      finish=245.3min speed=225024K/sec
</code>

===== Troubleshooting =====

==== Drive Marked as Failed During Rebuild ====

<code bash>
# Check system logs
journalctl -xe | grep -i "sda\|sdb\|md0"

# Check for USB resets
dmesg | grep -i "usb.*reset"

# Re-add the drive if it was incorrectly marked failed
mdadm --manage /dev/md0 --re-add /dev/sda1
</code>

==== Service Not Starting After Reboot ====

<code bash>
# Check service status
systemctl status usb-raid-tweaks.service -l

# View service logs
journalctl -u usb-raid-tweaks.service

# Manually test the service
systemctl start usb-raid-tweaks.service
</code>

==== APM Settings Not Persisting ====

Some USB enclosures ignore APM commands. Check with:

<code bash>
hdparm -I /dev/sda | grep -i "advanced power management"
</code>

If not supported, focus on USB autosuspend and timeout optimizations instead.

==== GRUB Changes Not Taking Effect ====

<code bash>
# Verify GRUB was regenerated correctly
grep usbcore.autosuspend /boot/grub2/grub.cfg
# or for Ubuntu/Debian:
grep usbcore.autosuspend /boot/grub/grub.cfg

# Check active boot parameters
cat /proc/cmdline
</code>

===== Best Practices =====

^ Practice ^ Description ^
| **Use USB 3.0+** | USB 2.0 is too slow for RAID rebuild operations |
| **Powered USB Hubs** | Ensure drives have stable power supply |
| **Quality Cables** | Use shielded, high-quality USB cables < 1.5m |
| **Monitor Temperatures** | Use ''hddtemp'' or ''smartctl'' to monitor drive temps |
| **Regular Scrubbing** | Run ''echo check > /sys/block/md0/md/sync_action'' monthly |
| **SMART Monitoring** | Enable ''smartd'' to catch drive issues early |
| **UPS Power** | Use UPS to prevent power-related disconnections |
| **Avoid USB Hubs** | Direct connection to motherboard USB ports is more stable |

===== Configuration Per Distribution =====

==== RHEL / CentOS / Oracle Linux / Rocky Linux / AlmaLinux ====

<code bash>
# Install required packages
yum install hdparm mdadm

# GRUB configuration file
/etc/default/grub

# GRUB regeneration command
grub2-mkconfig -o /boot/grub2/grub.cfg

# systemd service directory
/etc/systemd/system/
</code>

==== Ubuntu / Debian ====

<code bash>
# Install required packages
apt update
apt install hdparm mdadm

# GRUB configuration file
/etc/default/grub

# GRUB regeneration command
update-grub

# systemd service directory
/etc/systemd/system/
</code>

==== Fedora ====

<code bash>
# Install required packages
dnf install hdparm mdadm

# GRUB configuration file
/etc/default/grub

# GRUB regeneration command
grub2-mkconfig -o /boot/grub2/grub.cfg

# systemd service directory
/etc/systemd/system/
</code>

==== Arch Linux ====

<code bash>
# Install required packages
pacman -S hdparm mdadm

# GRUB configuration file
/etc/default/grub

# GRUB regeneration command
grub-mkconfig -o /boot/grub/grub.cfg

# systemd service directory
/etc/systemd/system/
</code>

===== Advanced: Dynamic Drive Detection =====

For systems where drive letters may change, use a more dynamic systemd service:

<code bash>
cat > /etc/systemd/system/usb-raid-tweaks.service << 'EOF'
[Unit]
Description=USB RAID Power Management Optimizations
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/usb-raid-optimize.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

# Create the optimization script
cat > /usr/local/bin/usb-raid-optimize.sh << 'EOF'
#!/bin/bash

sleep 5

# Disable USB autosuspend
for i in /sys/bus/usb/devices/*/power/autosuspend; do
    echo -1 > "$i" 2>/dev/null
done

# Find all drives in RAID arrays
for md_device in /dev/md*; do
    [ -b "$md_device" ] || continue
    
    # Get component drives
    for drive in $(mdadm --detail "$md_device" | grep -oP '/dev/sd[a-z]+' | sort -u); do
        # Get base device (remove partition number)
        base_drive=$(echo "$drive" | sed 's/[0-9]*$//')
        
        echo "Optimizing $base_drive for RAID..."
        
        # Disable APM
        hdparm -B 255 "$base_drive" 2>/dev/null
        
        # Disable spindown
        hdparm -S 0 "$base_drive" 2>/dev/null
        
        # Increase timeout
        drive_name=$(basename "$base_drive")
        if [ -f "/sys/block/$drive_name/device/timeout" ]; then
            echo 180 > "/sys/block/$drive_name/device/timeout"
        fi
    done
done

echo "USB RAID optimization complete"
EOF

chmod +x /usr/local/bin/usb-raid-optimize.sh

systemctl daemon-reload
systemctl enable usb-raid-tweaks.service
systemctl start usb-raid-tweaks.service
</code>

===== Expanding RAID After Rebuild =====

After replacing a drive and completing the rebuild, you may want to expand the RAID to use full drive capacity:

<WRAP center round important 60%>
**Warning:** Only proceed after rebuild is 100% complete and RAID status shows "clean, active"!
</WRAP>

<code bash>
# 1. Verify rebuild is complete
mdadm --detail /dev/md0 | grep State
# Should show: State : clean

# 2. Expand partitions on both drives
growpart /dev/sda 1
growpart /dev/sdb 1

# Or with parted:
parted /dev/sda
(parted) resizepart 1 100%
(parted) quit

# 3. Grow the RAID array
mdadm --grow /dev/md0 --size=max

# 4. Monitor the reshape operation
watch -n 10 cat /proc/mdstat

# 5. After reshape completes, resize the filesystem
# For ext4:
resize2fs /dev/md0

# For XFS:
xfs_growfs /mount/point

# 6. Verify new size
df -h /mount/point
</code>

===== FAQ =====

**Q: Will these optimizations affect system boot time?**

A: Minimal impact. The systemd service adds ~5 seconds delay (''sleep 5'') to ensure drives are detected before applying settings.

**Q: Can I use these settings with internal SATA drives?**

A: Some settings are beneficial (timeouts, APM), but USB autosuspend settings are unnecessary for internal drives.

**Q: What if I have more than 2 drives in RAID?**

A: Adjust the scripts to include all drive letters (sda, sdb, sdc, sdd, etc.) or use the dynamic detection script provided.

**Q: Do these settings increase power consumption?**

A: Yes, slightly. Drives will not spin down or enter power-saving modes. This is acceptable for a RAID array that should be always available.

**Q: Will this work with RAID 5/6/10?**

A: Yes! These optimizations work with any mdadm RAID level.

**Q: Can I revert these changes?**

A: Yes. Remove the systemd service (''systemctl disable usb-raid-tweaks.service''), revert GRUB changes, regenerate GRUB config, and reboot.

**Q: What about NVMe drives via USB?**

A: These optimizations work for NVMe drives in USB enclosures as well. Replace ''/dev/sd*'' with ''/dev/nvme*'' as needed.

===== Related Articles =====

  * [[raid_setup|Software RAID Setup with mdadm]]
  * [[raid_monitoring|RAID Monitoring and Alerts]]
  * [[raid_recovery|RAID Recovery Procedures]]
  * [[smart_monitoring|SMART Drive Monitoring]]

===== External Resources =====

  * [[https://raid.wiki.kernel.org/|Linux RAID Wiki]]
  * [[https://man7.org/linux/man-pages/man8/mdadm.8.html|mdadm man page]]
  * [[https://man7.org/linux/man-pages/man8/hdparm.8.html|hdparm man page]]
  * [[https://www.kernel.org/doc/Documentation/usb/power-management.txt|USB Power Management Documentation]]
  * [[https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_storage_devices/|RHEL Storage Management Guide]]