Full Server Rsync Migrations

There are times when you need to do a one to one server migration.  This can seem like a daunting task, however with the magic of rsync, performing full server migrations can be a fairly painless task requiring little downtime.

Prerequisites

On both the old and new servers, you want to ensure the following requirements are met:

1. Confirm both the old and new servers are using the same hardware architecture. You cannot perform an rsync migration if one server is 32-bit, and the other is a 64-bit system. This can be verified by running the following command, and checking to see if it has “i386”, which means 32-bit, or if both have “x86_64”, which stands for 64-bit.

uname -a
Linux demo 2.6.18-308.el5xen #1 SMP Tue Feb 21 20:47:10 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

So in our example, I have verified that both the old and new servers are 64-bit.

2. Confirm they are both running the same exact version of the operating system. Normally this means simply confirming both servers are at the latest patch level which you can do by running:

yum update
cat /etc/redhat-release
CentOS release 5.8 (Final)

For this article, I will be using two servers that are running CentOS 5.8 (Final).

3. Confirm rsync is installed on both servers

yum install rsync

4. Clean up server before migration. Depending on the amount of files on your server, the initial rsync migration can take quite a while. So you will want to browse through your old server and remove any extraneous temporary, cache, or other large files that you no longer need. You should also check your logs and ensure that their sizes are reasonable, and archive or delete the older logs you no longer need.

5. If you are not going to be swapping IP’s, and simply updating DNS to point to the new server, confirm that all your domains on the old server have a very low TTL set in the zonefile. A TTL of 300 is usually considered the lowest acceptable TTL to set.

Begin server migration

The procedure I’ll be writing out below is a two step process. It is meant to help minimize the amount of downtime that is involved when you swap the IP’s or update DNS, assuming you have a low TTL set. The steps are below:
1. Perform initial rsync
2. Perform final rsync and ip swap (or DNS update)

The initial rsync is just used to get the majority of the static files over to the new server. The final rsync is meant to update anything that is dynamic, such as logs, updated web content, databases, etc.

So before we begin, you will want to create an excludes file on the old server. This file will tell rsync NOT to copy over system specific information that is not needed for your new system.

vi /root/rsync-exclude.txt
/boot
/proc
/sys
/tmp
/dev
/var/lock
/etc/fstab
/etc/mdadm.conf
/etc/mtab
/etc/resolv.conf
/etc/conf.d/net
/etc/network/interfaces
/etc/networks
/etc/sysconfig/network*
/etc/sysconfig/hwconf
/etc/sysconfig/ip6tables-config
/etc/sysconfig/kernel
/etc/hostname
/etc/HOSTNAME
/etc/hosts
/etc/modprobe*
/etc/modules
/etc/udev
/net
/lib/modules
/etc/rc.conf
/lock
/etc/ssh/ssh_host_rsa_key.pub
/etc/ssh/ssh_host_rsa_key
/etc/ssh/ssh_host_dsa_key.pub
/etc/ssh/ssh_host_dsa_key
/etc/network.d
/etc/network/*
/etc/machine-id
/usr/share/nova-agent*
/usr/sbin/nova-agent*
/etc/rc.d/init.d/nova-agent*
/etc/init.d/nova-agent*
/etc/rackspace-monitoring-agent*
/etc/rackspace
/etc/driveclient/bootstrap.json

The example above should cover a couple of different distros. But always review someone else’s example before applying it to your own systems!!

Now that we have the proper excludes in place, lets run a dry run of rsync to see what would have happened before we actually do it. Please note that this is to run on the old server. Replace xxx.xxx.xxx.xxx with the IP of the new server:

rsync -azPx -e ssh --dry-run -azPx --delete-after --exclude-from="/root/rsync-exclude.txt" / [email protected]:/

If all looks well, lets perform the actual initial rsync:

rsync -azPx -e ssh -azPx --delete-after --exclude-from="/root/rsync-exclude.txt" / [email protected]:/

Depending on how much data you have, this could take a few minutes, or many hours. Once this is complete, you will want to schedule a maintenance window to perform the final rsync and IP swap (or DNS update). You want to perform this during a maintenance window as you will need to stop any database services or anything else that has very dynamic data to avoid corruption. So in the example, I just have a basic LAMP server, so I will need to shut down MySQL before performing the final rsync. Here are the steps I’ll be using:
1. Stop MySQL on old server
2. Perform final rsync
3. On new server, reboot server and test everything
4. Swap IP from old server to new, or update your DNS accordingly.

On the old server:

service mysql stop
rsync -azPx -e ssh -azPx --delete-after --exclude-from="/root/rsync-exclude.txt" / [email protected]:/

Now we are ready to start testing our new server.

Testing And Go-Live

Lets wave that dead chicken over the alter, its time to see if your new server survives a reboot, and if all the services come back online properly. There are no guides for this. Just reboot your server, then test out your sites, databases, and keep a wary eye on your system logs.

Once you have confirmed that everything looks good, it will then be safe to swap the IP’s, or update DNS accordingly. In the event that a problem surfaces shortly after the migration, you always have the option of rolling back to your older server, assuming you won’t be losing any critical transactions.