3ware RAID
A local copy of 3ware code, utilities, and documentation is on bursa:/ISO. 3ware downloads are available at http://www.3ware.com/support/download.asp (the CLI and 3DM2 for 9xxx are compatible with 7xxx/8xxx controllers). 3ware has good linux support and the drivers are integrated in the kernel.org tree.
In some cases you may need to use the BIOS or older utilities (7.7.x, or tw_cli help says version 2.00.00.032b) for functionality specific to the older controllers.
The below examples use the newer utilities.
Adding a volume
If you are replacing all the old drives with new larger ones, first delete the old volume, then remove and replace the drives with the new larger ones:
tw_cli maint deleteunit c1 u0 # not necessary, but minimizes error messages: tw_cli maint remove c1 p0 tw_cli maint remove c1 p1 tw_cli maint remove c1 p2 tw_cli maint remove c1 p3
When the 3ware card sees the new drives, it will add them as individual one-drive "JBOD" units. Delete the single units and create a RAID:
tw_cli maint rescan tw_cli maint deleteunit c1 u0 tw_cli maint deleteunit c1 u1 tw_cli maint deleteunit c1 u2 tw_cli maint deleteunit c1 u3 tw_cli maint createunit c1 rraid5 p0 p1 p2 tw_cli maint createunit c1 rspare p2
You now have a new volume, which the OS should see (check /var/log/messages). Assuming it is /dev/sdb, create a new disk label:
# adding a disk:
DISKDEV=/dev/sdb
PART=${DISKDEV}1
parted $DISKDEV mklabel msdos
parted $DISKDEV print
# get the end block number from above
ENDBLOCK=953875
And finally create the partition and filesystem and turn off the automatic fsck:
parted $DISKDEV mkpart primary 0 953875 mke2fs -O dir_index -j $PART tune2fs -c 0 -i 0 $PART
Replacing a drive
If a RAID unit is DEGRADED, a drive had died. Find out which port with tw_cli info c0:
Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC ------------------------------------------------------------------------------ u0 RAID-5 DEGRADED - 64K 701.284 ON - - Port Status Unit Size Blocks Serial --------------------------------------------------------------- p0 DEGRADED u0 233.76 GB 490234752 Y61DS7HE p1 OK u0 233.76 GB 490234752 Y63H22SE p2 OK u0 233.76 GB 490234752 Y61534QE p3 OK u0 233.76 GB 490234752 Y61ECSLE
Remove that port from the set:
[root@vagal ]# tw_cli maint remove c0 p0 Exporting port /c0/p0 ... Done.
Then pull and replace the drive (verify the Serial on the drive sticker), rescan, and rebuild. It probably would be best to do this via the 3ware BIOS (Alt-3 on boot) since last time Linux recognized the new physical drive as sda, making a mess of fstab.
[root@vagal ]# tw_cli maint createunit c0 rspare p0 Creating new unit on controller /c0 ... Done. The new unit is /c0/u0. [root@vagal ]# tw_cli maint rebuild c0 u1 p0 Sending Rebuild-Start request to /c0/u1 on 1 disk(s) [0] ... Done. [root@vagal ~]# tw_cli info c0 Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC ------------------------------------------------------------------------------ u1 RAID-5 REBUILDING 18 64K 701.284 ON - - Port Status Unit Size Blocks Serial --------------------------------------------------------------- p0 DEGRADED u1 233.76 GB 490234752 Y66G1R3E p1 OK u1 233.76 GB 490234752 Y63H22SE p2 OK u1 233.76 GB 490234752 Y61534QE p3 OK u1 233.76 GB 490234752 Y61ECSLE
Alternatively, if the DEGRADED drive isn't really bad (say, you pulled the wrong drive on a replace), you can just rescan the container and rebuild:
[root@vagal ]# tw_cli rescan Rescanning controller /c0 for units and drives ...Done. Found the following unit(s): [none]. Found the following drive(s): [none]. Rescanning controller /c1 for units and drives ...Done. Found the following unit(s): [none]. Found the following drive(s): [/c1/p0]. [root@vagal ~]# tw_cli maint rebuild c1 u0 p0 Sending Rebuild-Start request to /c1/u0 on 1 disk(s) [0] ... Done. [root@vagal ~]# tw_cli info c1 Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC ------------------------------------------------------------------------------ u0 RAID-5 REBUILDING 2 64K 701.284 ON - - Port Status Unit Size Blocks Serial --------------------------------------------------------------- p0 DEGRADED u0 233.76 GB 490234752 Y62K8SDE p1 OK u0 233.76 GB 490234752 Y61DS8NE p2 OK u0 233.76 GB 490234752 Y61DNJKE p3 OK u0 233.76 GB 490234752 Y61DS7DE
vastus example
For some functionality the older utilities must be used on vastus.
tw_cli info c0 u0 # The "N/A" means the drive is bad: # Subunit 0: CBOD: DEGRADED # Physical Port: N/A tw_cli maint remove c0 p0 # Now note the drive serial number (from "info c0") then power down and replace the drive tw_cli maint rescan c0 tw_cli maint createunit c0 rspare p0 tw_cli maint rebuild c0 u0 p0
There is a pretty good example at http://gentoo-wiki.com/HARDWARE_3ware_Raid_Array_Management
3dmd
The configuration file for 3dmd is /etc/3dmd.conf. By default it runs on port 1080 and listens to remote connections but refuses access to settings. It is capable of sending email alerts but I have not tested that feature.
The tw_cli (3ware command-line interface) tool
The User's Guide for the newer tw_cli is available from http://www.3ware.com/support/UserDocs/CLIGuide.pdf
[root@vagal root]# tw_cli show Ctl Model Ports Drives Units NotOpt RRate VRate BBU ------------------------------------------------------------------------ c0 8506-4LP 4 4 1 0 3 - - c1 8506-4LP 4 4 1 0 3 - - [root@vagal root]# tw_cli info c0 Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC ------------------------------------------------------------------------------ u0 RAID-5 OK - 64K 701.284 ON - - Port Status Unit Size Blocks Serial --------------------------------------------------------------- p0 OK u0 233.76 GB 490234752 Y61DS7HE p1 OK u0 233.76 GB 490234752 Y63H22SE p2 OK u0 233.76 GB 490234752 Y61534QE p3 OK u0 233.76 GB 490234752 Y61ECSLE [root@vagal root]# tw_cli info c1 Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC ------------------------------------------------------------------------------ u0 RAID-5 OK - 64K 701.284 ON - - Port Status Unit Size Blocks Serial --------------------------------------------------------------- p0 OK u0 233.76 GB 490234752 Y62K8SDE p1 OK u0 233.76 GB 490234752 Y61DS8NE p2 OK u0 233.76 GB 490234752 Y61DNJKE p3 OK u0 233.76 GB 490234752 Y61DS7DE
The User's Guide for the older tw_cli on vastus is available from http://www.3ware.com/support/UserDocs/CLI_UG.pdf
vastus:/# tw_cli info c0
Controller: c0
-------------
Driver: 1.02.00.037
Model: 7850
FW: FE7X 1.05.00.065
BIOS: BE7X 1.08.00.048
Monitor: ME7X 1.01.00.038
Serial #: F11602A2162007
PCB: Rev3
PCHIP: 1.30-33
ACHIP: 3.20
# of units: 1
Unit 0: RAID 5 1.04 TB ( 2241197056 blocks): OK
# of ports: 8
Port 0: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
Port 1: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
Port 2: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
Port 3: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
Port 4: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
Port 5: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
Port 6: Maxtor 4G160J8 G80438HE 152.67 GB (320173056 blocks): OK(unit 0)
Port 7: Maxtor 4G160J8 G80EQFYE 152.67 GB (320173056 blocks): OK(unit 0)
