Replace RAID 1 harddisks with a bigger pair

I had replaced my 320GB RAID 1 harddisk pair with a new pair of 1TB harddisks. The steps are outlined as below.

Step 1: gather information on the existing partitions for reference. For example, I am using Linux software RAID + LVM2 on CentOS 5.2. I used the below commands to acquire the information I needed.

[root@server ~]# fdisk -l /dev/sda

Disk /dev/sda: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   fd  Linux raid autodetect
/dev/sda2              14        3929    31455270   fd  Linux raid autodetect
/dev/sda3            3930        4183     2040255   82  Linux swap / Solaris
/dev/sda4            4184       38913   278968725    5  Extended
/dev/sda5            4184       38913   278968693+  fd  Linux raid autodetect
[root@server ~]#

[root@server ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
      104320 blocks [2/2] [UU]

md2 : active raid1 sdb5[1] sda5[0]
      278968576 blocks [2/2] [UU]

md1 : active raid1 sdb2[1] sda2[0]
      31455168 blocks [2/2] [UU]

unused devices: <none>
[root@server ~]#

[root@server ~]# mount
/dev/md1 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/md0 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
/dev/mapper/vg0-lvvar on /var type ext3 (rw)
/dev/mapper/vg0-lvhome on /home type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)

[root@server ~]# vgdisplay -v
    Finding all volume groups
    Finding volume group "vg0"
  --- Volume group ---
  VG Name               vg0
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               266.03 GB
  PE Size               32.00 MB
  Total PE              8513
  Alloc PE / Size       8513 / 266.03 GB
  Free  PE / Size       0 / 0
  VG UUID               BL0sRH-Nq9y-rOTg-RRLF-GA4i-h3yh-yQdHvB

  --- Logical volume ---
  LV Name                /dev/vg0/lvhome
  VG Name                vg0
  LV UUID                Ro5PbX-R0M0-QJFD-B83v-Zs1n-E2Zt-UYmm1L
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                217.19 GB
  Current LE             6950
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/vg0/lvvar
  VG Name                vg0
  LV UUID                h58YvY-c30V-03W3-i5gO-gnuH-w17u-mE1Dzh
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                48.84 GB
  Current LE             1563
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

  --- Physical volumes ---
  PV Name               /dev/md2
  PV UUID               64bgxg-dB74-zEUX-upYI-C74v-6aSa-6AYo9R
  PV Status             allocatable
  Total PE / Free PE    8513 / 0

Step 2: shutdown the computer, disconnect the power, and replace /dev/sdb with one of  the new 1TB harddisk.

Step 3: create partitions on the new harddisk (i.e. /dev/sdb) with fdisk after reboot. The size of each partition should be the same to or greater than the old one.

Step 4: add the new partitions to the RAID array.

mdadm /dev/md0 --add /dev/sdb1
mdadm /dev/md1 --add /dev/sdb2
mdadm /dev/md2 --add /dev/sdb5

To monitor the progress of array reconstruction, I had used the below command.

while true; do cat /proc/mdstat ; sleep 30; done

My system had taken about 70 minutes for the RAID reconstruction.

Step 5: install GRUB onto the new harddisk /dev/sdb. Otherwise, the new harddisk cannot be used to boot the system. I had done the installation with the GRUB shell.

[root@server ~]#grub

grub> root (hd1,0)
grub> setup (hd1)
grub> quit

Step 6: turn off the system, remove the old /dev/sda harddisk, move the /dev/sdb harddisk to /dev/sda, and then install the other new 1TB harddisk as /dev/sdb.

Step 7: boot the system, create partitions on the new harddisk /dev/sdb. Since the two new harddisks are identitcal, I used the below command to copy the partition tables.

dd if=/dev/sda of=/dev/sdb bs=1k count=1

Then I used fdisk to create the extended partition /dev/sdb5.

Step 8: repeat step 4 & 5 above to add the new partitions to the array and install GRUB on the new harddisk /dev/sdb.

Now I have the two new harddisks in my system running in RAID 1. I have to proceed with the below steps to take advantage of the extra disk space.

Step 9: grow the partitions according to their new sizes.

mdadm --grow /dev/md0 --size=max
mdadm --grow /dev/md1 --size=max
mdadm --grow /dev/md2 --size=max

This process took another 3 hours. After it’s done, it looks something like:

[root@server ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[0] sda1[1]
      112320 blocks [2/2] [UU]

md2 : active raid1 sdb5[0] sda5[1]
      943168000 blocks [2/2] [UU]

md1 : active raid1 sdb2[0] sda2[1]
      31471232 blocks [2/2] [UU]

unused devices: <none>

Step 10: resize the filesystem on /dev/md0 and /dev/md1.

[root@server ~]# resize2fs /dev/md0
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/md0 is mounted on /boot; on-line resizing required
Performing an on-line resize of /dev/md0 to 112320 (1k) blocks.
The filesystem on /dev/md0 is now 112320 blocks long.

[root@server ~]# resize2fs /dev/md1
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/md1 is mounted on /; on-line resizing required
Performing an on-line resize of /dev/md1 to 7867808 (4k) blocks.
The filesystem on /dev/md1 is now 7867808 blocks long.

Step 11: grow the PV of LVM.

[root@server ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/md2
  VG Name               vg0
  PV Size               266.05 GB / not usable 14.25 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              8513
  Free PE               0
  Allocated PE          8513
  PV UUID               64bgxg-dB74-zEUX-upYI-C74v-6aSa-6AYo9R

[root@server ~]# pvresize /dev/md2
  Physical volume "/dev/md2" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

[root@server ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/md2
  VG Name               vg0
  PV Size               899.47 GB / not usable 6.31 MB
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              28783
  Free PE               20270
  Allocated PE          8513
  PV UUID               64bgxg-dB74-zEUX-upYI-C74v-6aSa-6AYo9R

Step 12: grow the extend and filesystem inside the LV.

[root@server ~]# lvextend -L+600G /dev/vg0/lvhome
  Extending logical volume lvhome to 817.19 GB
  Logical volume lvhome successfully resized

[root@server ~]# resize2fs /dev/vg0/lvhome
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/vg0/lvhome is mounted on /home; on-line resizing required
Performing an on-line resize of /dev/vg0/lvhome to 214220800 (4k) blocks.
The filesystem on /dev/vg0/lvhome is now 214220800 blocks long.

The on-line resizing of the 800+GB file system took another 20 minutes.

Now I have quite some space for my FLAC files, and I should be fine for a couple of years to come.

References:

http://portal.itauth.com/2007/10/07/linux-practical-introduction-linux-software-raid#start-5
http://www.howtoforge.com/software-raid1-grub-boot-debian-etch-p2
http://linux-raid.osdl.org/index.php/Growing