LVM adventures with SLES 10 SP2
Recently I was asked to move one database and few files systems from one storage system to another, they all resided on one server. The source storage system was EVA4400 with FATA and destination was again EVA4400, but running with FC drives. Both storage systems were having 8Gbits connection. The storage layout was: three physical volumes 2TB each and one PV 2.5 TB, these were separate in three volume groups and were created five logical volumes on top of them.
For completing the task I had several options:
- Regular file system copy (offline)
- Using dd (offline)
- Using BCV (almost online)
- LVM mirroring (online)
So I started testing and giving pros and cons of every option:
-
Because I had already moved files on other systems I knew the speed of coping files from one file system to another. Doing to calculations it turned out that we need three or at least two ways and half for coping data from one file system (FATA) to another (FC). Another good point was the count of files in one of the file systems, it was 3M (a lot of small) files! which means that probably three ways wasn't to be enough for process to complete. On top of this the process should be offline because of database consistency.
-
The other choice was doing dd for all of the volumes. Doing dd would be better than file system copy, but again it has to be offline and we have no control of the process. What's more some we had LUNs which are bigger than 2TB on the first system, but were unable to create bigger LUNs than 2TB on the second storage system, because of the firmware issue. It's something I'm going to blog latter, by the same reason were unable to use Busineess Copy (snapshots and snapclones).
-
We had an option to move data to the same storage system using BCV, with snapclone we could move data from one disk group to another. This would be definitely the fastest ways and a little downtime would be required just to remount the new file systems and start the database and applications. Because using the latest firmware we had LUNs which were bigger than 2TB we were unable to do any replication solutions with them. Again, I'll blog about this one soon.
-
So the last technology left was the LVM mirroring. I've had a lot of experience with LVM on the HP-UX systems and I really like it. I've decided to give it a try on the Linux, well I've worked with LVM on Linux, but nothing more than create/extend and resize. From here I started a month process with adventures:
The first difference with HP-UX was that I need one additional disk for mirror log (?). In Linux LVM if you want to create mirror, additional disk is need or else the log is written to memory and if the server is restarted, the process must be repeated. The error message is following:
sles10:/ # lvconvert -m1 vg01/lvtest /dev/sdc
Not enough PVs with free space available for parallel allocation.
Consider --alloc anywhere if desperate.
sles10:/ # lvconvert -m1 vg01/lvtest /dev/sdc /dev/sdd
Logical volume lvtest converted.
What's disturbing is that I wasn't able to find what should be the size of the mirror log. It's not in the docs, some folks at the forums said that it should be 1% of the size of the mirrored logical volume. Actually it's one extend big:
[lvtest_mlog] vg01 lwi-ao 1 linear 4.00M /dev/sdd(0)
After I spent one week in creating virtual disks and mirroring logical volumes and got to the point where I should break the mirror and remove the physical volumes from the first storage system. For this purpose I had two options:
4.1. lvconvert (online)
4.2. vgsplit (almost online)
4.1. Using lvconvert -m0 I was supposed to remove the first physical volume and leave the file system on the second storage system. With no obvious reason I got the following error when I try to break the mirror i.e. convert the logical volume to linear:
sles10:/ # lvconvert -m0 vg01/lvtest /dev/sdc
No free extents on physical volume "/dev/sdc"
No specified PVs have space available
Sure I don't have free extents, but why do I need them when I'm currently breaking and not creating the mirror ? I search a lot and didn't found any solution, probably this is a bug of the current version of the SLES. Either way I decided to test this in a lab environment and figure out what could be done to finish the process. I created one group with two physical volumes, 1GB each and then created logical volume with exact same size. It was the same, I wasn't able to break the mirror:
--- Physical volumes ---
PV Name /dev/sdb
Total PE / Free PE 256 / 0
PV Name /dev/sdc
Total PE / Free PE 256 / 0
I wasn't able to remove the first physical volume, if I execute just lvconvert -m0 /dev/sdb, without the third argument then I'm at starting point.
I've got it literally, I don't have enough physical extents, I've decided to test this in a lab environment and resizing the first physical volume just by one extent resolved the problem:
--- Logical volume ---
LV Name /dev/vg01/lvtest
Current LE 257
--- Physical volumes ---
PV Name /dev/sdb
Total PE / Free PE 258 / 1
PV Name /dev/sdc
Total PE / Free PE 257 / 0
sles10:~ # lvconvert -m0 vg01/lvtest /dev/sdb
Logical volume lvtest converted.
I was hopeful that I will get over this error by just resizing the LUNs with its minimal step (in EVA this is 1 GB), but this was not possible. Again because of firmware issues I was not able to extent the LUNs. At this point I decided to do it the hard way, which is unpresent the LUNs from the first storage system without breaking the mirror. This worked great, just by unpresenting the LUNs the LVM detected this and removed the failed physical volumes, all the file systems continued working without interruption.
This was possible with four of the LUNs, but the last one was bigger and it spanned across two logical volumes. Because of this or some other reason unpresenting just the LUN didn't worked out and I decided to go on with the last option.
4.2. Using vgsplit sounds promising, some of the manuals in Internet showed that this is the way to break a mirror. The steps are almost the same, using lvconvert remove the mirror from the second physical volume, delete any left lvol_mimage_X volumes and then using vgsplit create another volume group with the second physical volume. If the file systems are opened during lvconvert then lvol_mimage volume will reside for sure. After spliting the volume group the same logical volumes with exactly the same count of logical extents has to be created. At this point regular file system check would be enough and the file systems could be mounted. Well, it took me more than an hour to check 2TB the file system, but otherwise everything is fine.
As a conclusion I would say that BCV would be fastest for moving data within the same storage system. Of course we are only talking about online or almost online replication so the other option is using LVM mirroring. Depending on the Linux/Unix distribution this could be done with lvconvert to reduce the volume group with the first physical volume or using vgsplit to move the second physical volume to another volume group and recreate the logical volumes.
Regards,
Sve