wiki:HardwareRaid

Version 1 (modified by trac, 12 years ago) (diff)

--

Several servers have hardware RAID controllers.

vagal has a 3ware Escalade 8506 vastus has a 3ware Escalade 7850 (a 7500 Series controller)

There are three ways to access information from the controller:

  • An on-board BIOS management tool when you boot the machine
  • A monitoring and configuration daemon which provides a web interface (3dmd)
  • A command-line interface tool (tw_cli)

A local copy of 3ware code, utilities, and documentation is on bursa:/ISO. 3ware downloads are available at http://www.3ware.com/support/download.asp (the CLI and 3DM2 for 9xxx are compatible with 7xxx/8xxx controllers). 3ware has good linux support and the drivers are integrated in the kernel.org tree.

/!\ In some cases you may need to use the BIOS or older utilities (7.7.x, or tw_cli help says version 2.00.00.032b) for functionality specific to the older controllers.

Replacing a drive

If a RAID unit is DEGRADED, a drive had died. Find out which port with tw_cli info c0:

Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  IgnECC
------------------------------------------------------------------------------
u0    RAID-5    DEGRADED       -      64K     701.284   ON     -        -        

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     DEGRADED         u0     233.76 GB   490234752     Y61DS7HE      
p1     OK               u0     233.76 GB   490234752     Y63H22SE      
p2     OK               u0     233.76 GB   490234752     Y61534QE      
p3     OK               u0     233.76 GB   490234752     Y61ECSLE      

Remove that port from the set:

[root@vagal ]# tw_cli maint remove c0 p0
Exporting port /c0/p0 ... Done.

Then pull and replace the drive (verify the Serial on the drive sticker), rescan, and rebuild:

/!\ Next time it probably would be best to do this via the 3ware BIOS (Alt-3 on boot) since Linux recognized the new drive as sda, making a mess of fstab.

[root@vagal ]# tw_cli maint createunit c0 rspare p0
Creating new unit on controller /c0 ...  Done. The new unit is /c0/u0.

[root@vagal ]# tw_cli maint rebuild c0 u1 p0
Sending Rebuild-Start request to /c0/u1 on 1 disk(s) [0] ... Done.

[root@vagal ~]# tw_cli info c0

Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  IgnECC
------------------------------------------------------------------------------
u1    RAID-5    REBUILDING     18     64K     701.284   ON     -        -        

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     DEGRADED         u1     233.76 GB   490234752     Y66G1R3E      
p1     OK               u1     233.76 GB   490234752     Y63H22SE      
p2     OK               u1     233.76 GB   490234752     Y61534QE      
p3     OK               u1     233.76 GB   490234752     Y61ECSLE      

Alternatively, if the DEGRADED drive isn't really bad (say, you pulled the wrong drive on a replace), you can just rescan the container and rebuild:

[root@vagal ]# tw_cli rescan
Rescanning controller /c0 for units and drives ...Done.
Found the following unit(s): [none].
Found the following drive(s): [none].
Rescanning controller /c1 for units and drives ...Done.
Found the following unit(s): [none].
Found the following drive(s): [/c1/p0].


[root@vagal ~]# tw_cli maint rebuild c1 u0 p0
Sending Rebuild-Start request to /c1/u0 on 1 disk(s) [0] ... Done.


[root@vagal ~]# tw_cli info c1

Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  IgnECC
------------------------------------------------------------------------------
u0    RAID-5    REBUILDING     2      64K     701.284   ON     -        -        

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     DEGRADED         u0     233.76 GB   490234752     Y62K8SDE      
p1     OK               u0     233.76 GB   490234752     Y61DS8NE      
p2     OK               u0     233.76 GB   490234752     Y61DNJKE      
p3     OK               u0     233.76 GB   490234752     Y61DS7DE      


3dmd

The configuration file for 3dmd is /etc/3dmd.conf. By default it runs on port 1080 and listens to remote connections but refuses access to settings. It is capable of sending email alerts but I have not tested that feature.

The tw_cli (3ware command-line interface) tool

The User's Guide for the newer tw_cli is available from http://www.3ware.com/support/UserDocs/CLIGuide.pdf

[root@vagal root]# tw_cli show

Ctl   Model        Ports   Drives   Units   NotOpt   RRate   VRate   BBU
------------------------------------------------------------------------
c0    8506-4LP     4       4        1       0        3       -       -  
c1    8506-4LP     4       4        1       0        3       -       -  

[root@vagal root]# tw_cli info c0

Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  IgnECC
------------------------------------------------------------------------------
u0    RAID-5    OK             -      64K     701.284   ON     -        -        

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     233.76 GB   490234752     Y61DS7HE      
p1     OK               u0     233.76 GB   490234752     Y63H22SE      
p2     OK               u0     233.76 GB   490234752     Y61534QE      
p3     OK               u0     233.76 GB   490234752     Y61ECSLE      

[root@vagal root]# tw_cli info c1

Unit  UnitType  Status         %Cmpl  Stripe  Size(GB)  Cache  AVerify  IgnECC
------------------------------------------------------------------------------
u0    RAID-5    OK             -      64K     701.284   ON     -        -        

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     233.76 GB   490234752     Y62K8SDE      
p1     OK               u0     233.76 GB   490234752     Y61DS8NE      
p2     OK               u0     233.76 GB   490234752     Y61DNJKE      
p3     OK               u0     233.76 GB   490234752     Y61DS7DE      

The User's Guide for the older tw_cli on vastus is available from http://www.3ware.com/support/UserDocs/CLI_UG.pdf

vastus:/# tw_cli info c0
Controller: c0
-------------
Driver:   1.02.00.037
Model:    7850
FW:       FE7X 1.05.00.065
BIOS:     BE7X 1.08.00.048
Monitor:  ME7X 1.01.00.038
Serial #: F11602A2162007
PCB:      Rev3
PCHIP:    1.30-33
ACHIP:    3.20


# of units: 1
        Unit 0: RAID 5 1.04 TB ( 2241197056 blocks): OK

# of ports: 8
        Port 0: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
        Port 1: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
        Port 2: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
        Port 3: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
        Port 4: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
        Port 5: Maxtor 4G160J8 G8000000 152.67 GB (320173056 blocks): OK(unit 0)
        Port 6: Maxtor 4G160J8 G80438HE 152.67 GB (320173056 blocks): OK(unit 0)
        Port 7: Maxtor 4G160J8 G80EQFYE 152.67 GB (320173056 blocks): OK(unit 0)