wiki:3wareRaid
Last modified 9 years ago Last modified on 03/17/10 11:38:48

3ware RAID

A local copy of 3ware code, utilities, and documentation is on bursa:/ISO. 3ware downloads are available at http://www.3ware.com/support/download.asp (the CLI and 3DM2 for 9xxx are compatible with 7xxx/8xxx controllers). 3ware has good linux support and the drivers are integrated in the kernel.org tree.

In some cases you may need to use the BIOS or older utilities (7.7.x, or tw_cli help says version 2.00.00.032b) for functionality specific to the older controllers.

The below examples use the newer utilities.

Adding a volume

If you are replacing all the old drives with new larger ones, first delete the old volume, then remove and replace the drives with the new larger ones:

tw_cli maint deleteunit c1 u0
# not necessary, but minimizes error messages:
tw_cli maint remove c1 p0
tw_cli maint remove c1 p1
tw_cli maint remove c1 p2
tw_cli maint remove c1 p3

When the 3ware card sees the new drives, it will add them as individual one-drive "JBOD" units. Delete the single units and create a RAID:

tw_cli maint rescan
tw_cli maint deleteunit c1 u0
tw_cli maint deleteunit c1 u1
tw_cli maint deleteunit c1 u2
tw_cli maint deleteunit c1 u3
tw_cli maint createunit c1 rraid5 p0 p1 p2
tw_cli maint createunit c1 rspare p2

You now have a new volume, which the OS should see (check /var/log/messages). Assuming it is /dev/sdb, create a new disk label:

# adding a disk:
DISKDEV=/dev/sdb
PART=${DISKDEV}1
parted $DISKDEV mklabel msdos
parted $DISKDEV print 
# get the end block number from above
ENDBLOCK=953875

And finally create the partition and filesystem and turn off the automatic fsck:

parted $DISKDEV mkpart primary 0 953875
mke2fs -O dir_index -j $PART
tune2fs -c 0 -i 0 $PART

Replacing a drive

If a RAID unit is DEGRADED, a drive had died. Find out which port with tw_cli info c0:

Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  IgnECC
------------------------------------------------------------------------------
u0    RAID-5    DEGRADED       -      64K     701.284   ON     -        -        

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     DEGRADED         u0     233.76 GB   490234752     Y61DS7HE      
p1     OK               u0     233.76 GB   490234752     Y63H22SE      
p2     OK               u0     233.76 GB   490234752     Y61534QE      
p3     OK               u0     233.76 GB   490234752     Y61ECSLE      

Remove that port from the set:

[root@vagal ]# tw_cli maint remove c0 p0
Exporting port /c0/p0 ... Done.

Then pull and replace the drive (verify the Serial on the drive sticker) and rescan. As mentioned above, the 3ware card will add the new drive as a one-drive "JBOD" units, so delete it and make the new drive the hot spare:

tw_cli maint rescan
tw_cli maint deleteunit c1 u1
tw_cli maint createunit c1 rspare p1

The full process:

[root@vagal ~]# tw_cli maint rescan
Rescanning controller /c0 for units and drives ...Done.
Found the following unit(s): [none].
Found the following drive(s): [none].
Rescanning controller /c1 for units and drives ...Done.
Found the following unit(s): [/c1/u1].
Found the following drive(s): [none].


[root@vagal ~]# tw_cli info c1

Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  IgnECC
------------------------------------------------------------------------------
u0    RAID-5    OK             -      64K     931.521   ON     -        -        
u1    JBOD      OK             -      -       931.513   ON     -        -        

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     465.76 GB   976773168     9QG3NC5Z      
p1     OK               u1     931.51 GB   1953525168    5QJ0ZD6W      
p2     OK               u0     465.76 GB   976773168     9QG3NBNA      
p3     OK               u0     465.76 GB   976773168     9QG3NCQQ      


[root@vagal ~]# tw_cli maint createunit c1 rspare p1
Creating new unit on controller /c1 ...  Failed.

(0x0B:0x001F): One or more of the specified drives belong to another unit

[root@vagal ~]# tw_cli info c1

Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  IgnECC
------------------------------------------------------------------------------
u0    RAID-5    OK             -      64K     931.521   ON     -        -        
u1    JBOD      OK             -      -       931.513   ON     -        -        

[root@vagal ~]# tw_cli maint deleteunit c1 u1
Deleting unit c1/u1 ...Done.

[root@vagal ~]# tw_cli maint createunit c1 rspare p1
Creating new unit on controller /c1 ...  Done. The new unit is /c1/u1.

Alternatively, if the DEGRADED drive isn't really bad (say, you pulled the wrong drive on a replace), you can just rescan the container and rebuild:

[root@vagal ]# tw_cli rescan
Rescanning controller /c0 for units and drives ...Done.
Found the following unit(s): [none].
Found the following drive(s): [none].
Rescanning controller /c1 for units and drives ...Done.
Found the following unit(s): [none].
Found the following drive(s): [/c1/p0].


[root@vagal ~]# tw_cli maint rebuild c1 u0 p0
Sending Rebuild-Start request to /c1/u0 on 1 disk(s) [0] ... Done.


[root@vagal ~]# tw_cli info c1

Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  IgnECC
------------------------------------------------------------------------------
u0    RAID-5    REBUILDING     2      64K     701.284   ON     -        -        

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     DEGRADED         u0     233.76 GB   490234752     Y62K8SDE      
p1     OK               u0     233.76 GB   490234752     Y61DS8NE      
p2     OK               u0     233.76 GB   490234752     Y61DNJKE      
p3     OK               u0     233.76 GB   490234752     Y61DS7DE      


vastus example

For some functionality the older utilities must be used on vastus.

tw_cli info c0 u0
# The "N/A" means the drive is bad:
#       Subunit 0:    CBOD: DEGRADED
#	Physical Port: N/A
tw_cli maint remove c0 p0
# Now note the drive serial number (from "info c0") then power down and replace the drive
tw_cli maint rescan c0
tw_cli maint createunit c0 rspare p0
tw_cli maint rebuild c0 u0 p0

There is a pretty good example at http://gentoo-wiki.com/HARDWARE_3ware_Raid_Array_Management

3dmd

The configuration file for 3dmd is /etc/3dmd.conf. By default it runs on port 1080 and listens to remote connections but refuses access to settings. It is capable of sending email alerts but I have not tested that feature.

The tw_cli (3ware command-line interface) tool

The User's Guide for the newer tw_cli is available from http://www.3ware.com/support/UserDocs/CLIGuide.pdf

[root@vagal root]# tw_cli show

Ctl   Model        Ports   Drives   Units   NotOpt   RRate   VRate   BBU
------------------------------------------------------------------------
c0    8506-4LP     4       4        1       0        3       -       -  
c1    8506-4LP     4       4        1       0        3       -       -  

[root@vagal root]# tw_cli info c0

Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  IgnECC
------------------------------------------------------------------------------
u0    RAID-5    OK             -      64K     701.284   ON     -        -        

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     233.76 GB   490234752     Y61DS7HE      
p1     OK               u0     233.76 GB   490234752     Y63H22SE      
p2     OK               u0     233.76 GB   490234752     Y61534QE      
p3     OK               u0     233.76 GB   490234752     Y61ECSLE      

[root@vagal root]# tw_cli info c1

Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  IgnECC
------------------------------------------------------------------------------
u0    RAID-5    OK             -      64K     701.284   ON     -        -        

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     233.76 GB   490234752     Y62K8SDE      
p1     OK               u0     233.76 GB   490234752     Y61DS8NE      
p2     OK               u0     233.76 GB   490234752     Y61DNJKE      
p3     OK               u0     233.76 GB   490234752     Y61DS7DE      

The User's Guide for the older tw_cli on vastus is available from http://www.3ware.com/support/UserDocs/CLI_UG.pdf

vastus:/# tw_cli info c0
Controller: c0
-------------
Driver:   1.02.00.037
Model:    7850
FW:       FE7X 1.05.00.065
BIOS:     BE7X 1.08.00.048
Monitor:  ME7X 1.01.00.038
Serial #: F11602A2162007
PCB:      Rev3
PCHIP:    1.30-33
ACHIP:    3.20


# of units: 1
        Unit 0: RAID 5 1.04 TB ( 2241197056 blocks): OK

# of ports: 8
        Port 0: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
        Port 1: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
        Port 2: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
        Port 3: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
        Port 4: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
        Port 5: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
        Port 6: Maxtor 4G160J8 G80438HE 152.67 GB (320173056 blocks): OK(unit 0)
        Port 7: Maxtor 4G160J8 G80EQFYE 152.67 GB (320173056 blocks): OK(unit 0)