www.linux.org
Linux IDE-RAID Notes - IDE-I34RAID

February 2000

# df -H /mnt/tmp
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              236G    0G  236G   0% /mnt/tmp
# df -h /mnt/tmp
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              220G    0G  220G   0% /mnt/tmp

Hardware

Intel SE440BX-2 - with Pentium III 500MHz
Promise Technology Ultra66 - PCI Ultra ATA/66 Controller Card
8 IBM Deskstar34GXP - 34GB EIDE Ultra-ATA/66 7200RPM 9.0ms (23MB/s)
1 Western Digital Caviar 5400 - 10.2GB EIDE Ultra-ATA/66 5400RPM 9.5ms (29MB/s) [boot drive]
Enlight EN-89500SX4 - 9bay server case, 400W power supply, ATX
CTG (Cables To Go) 18729 - 24in int Ultra DMA/ATA ribbon 3 connector IDE 33/66 UDMA $11
 
controller channel disk  Ultra66
BIOS		 /dev/
Ultra66
0		 0 0 - master D0 hde
1 - slave D1 hdf
1 0 - master D2 hdg
1 - slave D3 hdh
Ultra66
1		 0 0 - master D4 hdi
1 - slave D5 hdj
1 0 - master D6 hdk
1 - slave D7 hdl

ei - hde+hdi - different controllers
eg - hde+hdg - same controller, different channels
ij - hdi+hdj - same controller, same channel
egij - hde+hdg+hdi+hdj - 4 disk RAID 4/5 made from this mix
e-l - hde+hdf+hdg+hdh+hdi+hdj+hdk+hdl - 8 disk RAID 4/5

Software

The recommended new software RAID patches are not available for kernels newer than 2.2.11, but they will be included in 2.4 due out in December 1999 (still waiting...).  However, RedHat 6.1 includes the 2.2.12 kernel with patches for new software RAID.  So I decided to patch RedHat /usr/src/linux-2.2.12 with the ide patches.  Use the patch that most closely matches the kernel version.

RedHat 6.1 Linux 2.2.12 kernel - Kernel-HOWTO
ide.2.2.14.20000124.patch - http://www.kernel.org/pub/linux/kernel/people/hedrick/

# umask 002
# mkdir /usr/src/linux-2.2.12-ide
# rm /usr/src/linux
# ln -s /usr/src/linux-2.2.12-ide /usr/src/linux
# cd /usr/src/linux-2.2.12
# find . -print | cpio -pdumv ../linux-2.2.12-ide
# cd ../linux-2.2.12-ide
download and uncompress ide.2.2.14.20000124.patch
# patch -p1 < ide.2.2.14.20000124.patch > patch.out
IMPORTANT - edit the FAILED hunks by hand
# make mrproper
# make xconfig
        make xconfig
            block devices
            y    Generic PCI bus-master DMA support
            y    Use DMA by default when available
            y        PROMISE PCD PDC20246/PDC20262 support
            y            Special UDMA Feature
# make dep clean bzImage modules modules_install
# sh scripts/MAKEDEV.ide
# cp arch/i386/boot/bzImage /boot/vmlinuz-2.2.12-ide
# cp System.map /boot/System.map-2.2.12-ide
# ed /etc/lilo.conf
image=/boot/vmlinuz-2.2.12-ide
    label=linux-ide
    initrd=/boot/initrd-2.2.14-5.0.img
    read-only
    root=/dev/hda5
# lilo    # LILO mini-HOWTO, BootPrompt-HowTo
# reboot
lilo: linux-ide

Disk Configuration

Standard ext2fs setup/test example

    # cfdisk /dev/hde
    # hdparm -Tt /dev/hde # repeat several times
    # mke2fs /dev/hde1
    # mount -t ext2 /dev/hde1 /mnt/tmp
    # cd /mnt/tmp
    # Bonnie -s 500 -html -m hde
    # cd /
    # umount /mnt/tmp

RAID setup/test example

    # cfdisk /dev/hde # make partitions
    # ed /etc/raidtab # see http://ostenfeld.dk/~jakob/Software-RAID.HOWTO/
    # mkraid /dev/md0
    # cat /proc/mdstat
    # hdparm -Tt /dev/md0 # repeat several times
    # mke2fs -m0 /dev/md0
    # mount -t ext2 /dev/md0 /mnt/tmp
    # cd /mnt/tmp
    # Bonnie -s 500 -html -m md0
    # cd /
    # umount /mnt/tmp
    # raidstop /dev/md0

Performance Summary

Since this was the first RAID system constructed, I ran a spectrum of tests for RAID 0, 1, 4 and 5 with various unit sizes.  For some yet unknown reason, mkraid linear fails.  While performance measurements are interesting for the various combinations of master and slave on IDE channels, my primary goal is maximum capacity for minimum cost with secondary concern for reliability - redundancy and recovery.  So big RAID5 is the focus for future tests.
 
key & configuration hdparm read
MB/sec		 Bonnie read
MB/sec		 Bonnie write
MB/sec		 Comment
PIO ex. I34GXP 4.0 4.1 4.3 Promise Ultra66
I34GXP P66 22.6 22.3 15.0 Promise Ultra66
RAID lin. NA NA NA  
I34RAID0-ei (16k stripe) 45.1 44.8 31.4  
I34RAID0-eg (16k stripe) 45.1 44.8 31.1  
I34RAID0-ij (16k stripe) 35.6 32.0 38.2  
I34RAID1-ei (4k mirror) 22.3 22.7 14.7  
I34RAID1-eg (4k mirror) 22.3 22.6 14.8  
I34RAID1-ij (4k mirror) 22.3 22.8 18.4  
I34RAID4-egij (32k chunk) 49.6 62.2 30.0  
I34RAID4-e-l (32k chunk) 116.4 68.1 33.6  
I34RAID5-egij (32k chunk) 46.4 49.3 31.0  
I34RAID5-e-l (32k chunk) 133.3  66.8  35.6 Promise Ultra66 

hdparm

    # top
    # hdparm -Tt /dev/hde # measure device reads, run at least 3x

Bonnie

http://www.textuality.com/bonnie/
   # Bonnie -s 500 -html -m ...
 
Sequential Output Sequential Input Random
Per Char Block Rewrite Per Char Block
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
I34GXP PIO 500 2845 64.1 4315 50.5 2053 10.5 2743 32.0 4114 5.4 86.9 2.1
I34GXP DMA 500 7789 98.8 14983 12.6 7739 13.5 7909 92.6 22316 12.9 132.8 1.7
IRAID0-ei 500 7813 99.1 31405 26.0 14617 29.1 8056 94.9 44759 26.0 151.9 1.5
IRAID0-eg 500 7819 99.1 31074 25.1 13849 26.5 8190 96.4 44843 29.9 154.1 1.8
IRAID0-ij 500 7815 98.8 38243 33.5 13525 18.4 8288 97.4 31972 14.7 79.1 1.0
IRAID1-ei 500 7649 97.7 14674 11.8 7800 14.9 7968 94.0 22675 13.8 129.9 1.9
IRAID1-eg 500 7622 98.0 14754 12.3 6705 12.6 7968 94.0 22592 14.0 134.2 2.0
IRAID1-ij 500 7615 97.7 18443 15.8 7960 11.1 8098 95.5 22786 12.7 134.7 1.4
IRAID4-egij 500 7095 90.5 29983 25.9 14763 28.4 8105 95.6 62170 42.1 205.1 2.5
IRAID4-e-l 500 7244 91.7 33596 29.1 18505 35.5 8133 96.0 68087 47.5 199.5 2.1
IRAID5-egij 500 7080 90.4 31009 27.9 13719 27.0 8092 95.9 49381 28.7 163.4 2.0
IRAID5-e-l 500 7251 91.9 35571 30.2 18232 35.0 8134 95.9 66774 46.8 207.6 3.0

NoBell Home - gjm - last update 10/3/2000