Month: June 2012

Full Server Rsync Migrations

There are times when you need to do a one to one server migration.  This can seem like a daunting task, however with the magic of rsync, performing full server migrations can be a fairly painless task requiring little downtime.

Prerequisites

On both the old and new servers, you want to ensure the following requirements are met:

1. Confirm both the old and new servers are using the same hardware architecture. You cannot perform an rsync migration if one server is 32-bit, and the other is a 64-bit system. This can be verified by running the following command, and checking to see if it has “i386”, which means 32-bit, or if both have “x86_64”, which stands for 64-bit.

uname -a
Linux demo 2.6.18-308.el5xen #1 SMP Tue Feb 21 20:47:10 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

So in our example, I have verified that both the old and new servers are 64-bit.

2. Confirm they are both running the same exact version of the operating system. Normally this means simply confirming both servers are at the latest patch level which you can do by running:

yum update
cat /etc/redhat-release
CentOS release 5.8 (Final)

For this article, I will be using two servers that are running CentOS 5.8 (Final).

3. Confirm rsync is installed on both servers

yum install rsync

4. Clean up server before migration. Depending on the amount of files on your server, the initial rsync migration can take quite a while. So you will want to browse through your old server and remove any extraneous temporary, cache, or other large files that you no longer need. You should also check your logs and ensure that their sizes are reasonable, and archive or delete the older logs you no longer need.

5. If you are not going to be swapping IP’s, and simply updating DNS to point to the new server, confirm that all your domains on the old server have a very low TTL set in the zonefile. A TTL of 300 is usually considered the lowest acceptable TTL to set.

Begin server migration

The procedure I’ll be writing out below is a two step process. It is meant to help minimize the amount of downtime that is involved when you swap the IP’s or update DNS, assuming you have a low TTL set. The steps are below:
1. Perform initial rsync
2. Perform final rsync and ip swap (or DNS update)

The initial rsync is just used to get the majority of the static files over to the new server. The final rsync is meant to update anything that is dynamic, such as logs, updated web content, databases, etc.

So before we begin, you will want to create an excludes file on the old server. This file will tell rsync NOT to copy over system specific information that is not needed for your new system.

vi /root/rsync-exclude.txt
/boot
/proc
/sys
/tmp
/dev
/var/lock
/etc/fstab
/etc/mdadm.conf
/etc/mtab
/etc/resolv.conf
/etc/conf.d/net
/etc/network/interfaces
/etc/networks
/etc/sysconfig/network*
/etc/sysconfig/hwconf
/etc/sysconfig/ip6tables-config
/etc/sysconfig/kernel
/etc/hostname
/etc/HOSTNAME
/etc/hosts
/etc/modprobe*
/etc/modules
/etc/udev
/net
/lib/modules
/etc/rc.conf
/lock
/etc/ssh/ssh_host_rsa_key.pub
/etc/ssh/ssh_host_rsa_key
/etc/ssh/ssh_host_dsa_key.pub
/etc/ssh/ssh_host_dsa_key
/etc/network.d
/etc/network/*
/etc/machine-id
/usr/share/nova-agent*
/usr/sbin/nova-agent*
/etc/rc.d/init.d/nova-agent*
/etc/init.d/nova-agent*
/etc/rackspace-monitoring-agent*
/etc/rackspace
/etc/driveclient/bootstrap.json

The example above should cover a couple of different distros. But always review someone else’s example before applying it to your own systems!!

Now that we have the proper excludes in place, lets run a dry run of rsync to see what would have happened before we actually do it. Please note that this is to run on the old server. Replace xxx.xxx.xxx.xxx with the IP of the new server:

rsync -azPx -e ssh --dry-run -azPx --delete-after --exclude-from="/root/rsync-exclude.txt" / [email protected]:/

If all looks well, lets perform the actual initial rsync:

rsync -azPx -e ssh -azPx --delete-after --exclude-from="/root/rsync-exclude.txt" / [email protected]:/

Depending on how much data you have, this could take a few minutes, or many hours. Once this is complete, you will want to schedule a maintenance window to perform the final rsync and IP swap (or DNS update). You want to perform this during a maintenance window as you will need to stop any database services or anything else that has very dynamic data to avoid corruption. So in the example, I just have a basic LAMP server, so I will need to shut down MySQL before performing the final rsync. Here are the steps I’ll be using:
1. Stop MySQL on old server
2. Perform final rsync
3. On new server, reboot server and test everything
4. Swap IP from old server to new, or update your DNS accordingly.

On the old server:

service mysql stop
rsync -azPx -e ssh -azPx --delete-after --exclude-from="/root/rsync-exclude.txt" / [email protected]:/

Now we are ready to start testing our new server.

Testing And Go-Live

Lets wave that dead chicken over the alter, its time to see if your new server survives a reboot, and if all the services come back online properly. There are no guides for this. Just reboot your server, then test out your sites, databases, and keep a wary eye on your system logs.

Once you have confirmed that everything looks good, it will then be safe to swap the IP’s, or update DNS accordingly. In the event that a problem surfaces shortly after the migration, you always have the option of rolling back to your older server, assuming you won’t be losing any critical transactions.

Using PAM to enforce access based on time

Sometimes there is a need to restrict user access based on time. This could be access to one particular service, or all PAM enabled services. A common example is to only allow access for the user ‘bob’ monday through friday between 9:00AM – 6:00PM. This can be enforced by utilizing the pam_time module.

The pam_time module is an account module type. No arguments are passed directly to the module, but instead all configuration takes place within /etc/security/time.conf.

The time.conf operates based on rules, and each rule uses the following syntax:

services;ttys;users;times

Example Rules
Restrict SSHD access for bob to weekdays between 9:00AM – 7:00PM

sshd;*;bob;Wk0900-1900

Restrict ALL access for bob to weekdays between 9:00AM – 5:00PM

*;*;bob;Wk0900-1700

Restrict ALL access for ALL users except root to weekdays between 9:00AM – 5:00PM

sshd;*;bob;Wk0900-1700

Restrict SSH access for ALL users except bob and jane to weekdays between 9:00AM – 5:00PM

sshd;*;!bob|!jane;Wk0900-1700

To only allow bob to access SSH on Tuesdays between 3:23PM and 4:24PM:

sshd;*;bob;Tu1523-1624

Below is all the available abbreviates for the days of the week:

Mo : Monday Fr : Friday Wd : Sa/Su
Tu : Tuesday Sa : Saturday wk : Mo/Tu/We/Th/Fr
We : Wenesday Su : Sunday
Th : Thursday Al : All Days

Installation And Configuration
In our example, I am going be setting this up on a CentOS 5.x server. For the restricted user, the following variables will be used:

username: bob
allowed access times: 9:00AM - 6:00PM
restricted services: SSHD

First, add the user and time restriction to /etc/security/time.conf:

sshd;*;bob;Wk0900-1800

Now, update the pam module for login and sshd. You are including ‘account required pam_time.so‘. But I’ll post entire file for reference

cat /etc/pam.d/sshd
#%PAM-1.0
auth required pam_sepermit.so
auth include password-auth
account required pam_time.so
account required pam_nologin.so
account include password-auth
password include password-auth
# pam_selinux.so close should be the first session rule
session required pam_selinux.so close
session required pam_loginuid.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
session required pam_selinux.so open env_params
session optional pam_keyinit.so force revoke
session include password-auth
cat /etc/pam.d/system-auth
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth required pam_env.so
auth sufficient pam_unix.so nullok try_first_pass
auth requisite pam_succeed_if.so uid >= 500 quiet
auth required pam_deny.so
account required pam_time.so
account required pam_unix.so
account sufficient pam_localuser.so
account sufficient pam_succeed_if.so uid < 500 quiet
account required pam_permit.so
password requisite pam_cracklib.so try_first_pass retry=3 type=
password sufficient pam_unix.so md5 shadow nullok try_first_pass use_authtok
password required pam_deny.so
session optional pam_keyinit.so revoke
session required pam_limits.so
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so

And finally, restart SSH

service sshd restart

Creating table indexes in MySQL

You may ask, what is a table index and how will it help performance? Table indexes provide MySQL a more efficient way to retrieve records. I often like to use the following example to explain it:

Imagine you have a phone book in front of you, and there are no letters in the top right corner that you can reference if you are looking up a last name. Therefore, you have to search page by page through hundreds of pages that have tens of thousands of results. Very inefficient and intensive. Think of this as a full table scan.

Now picture the phone book having the letter references in the top right corner. You can flip right to section “La – Lf” and only have to search through a smaller result set. The time to find the results is must faster and easier.

Common symptoms where this logic can be applied is when you log onto a server and see MySQL frequently chewing up a lot of CPU time, either constantly, or in spikes. The slow-query-log is also a great indicator cause if the query is taking a long time to execute, chances are it was because the query was making MySQL work too hard performing full table scans.

The information below will provide you with the tools to help identify these inefficient queries and how to help speed them up.

There are 2 common ways to identify queries that are very inefficient and may be creating CPU contention issues:

View MySQL’s process list:

When entering into MySQL CLI, you will want to look for any queries that you see that are often running to evaluate. You can see the queries by:

mysql
show processlist;

View slow queries log:

To view this, first check to ensure the slow-query-log variables are enabled in the my.cnf:

log-slow-queries=/var/lib/mysqllogs/slow-log
long_query_time=5

Now, lets look at an example of a slow query that got logged. Please note, these queries got logged here cause they took longer to run then the max seconds defined on long_query_time:

# Time: 110404 22:45:25
# User@Host: wpadmin[wordpressdb] @ localhost []
# Query_time: 14.609104  Lock_time: 0.000054 Rows_sent: 4  Rows_examined: 83532
SET timestamp=1301957125;
SELECT * FROM wp_table WHERE `key`='5544dDSDFjjghhd2544xGFDE' AND `carrier`='13';

Here is a query that we know is know runs often, and takes over 5 seconds to execute:

SELECT * FROM wp_table WHERE `key`='5544dDSDFjjghhd2544xGFDE' AND `carrier`='13';

Within the MySQL cli, run the following to view some more details about this query:

explain SELECT * FROM wp_table WHERE `key`='5544dDSDFjjghhd2544xGFDE' AND `carrier`='13';
+----+-------------+------------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table      | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+------------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | wp_table   | ALL  | NULL          | NULL |    NULL | NULL | 83532 | Using where |
+----+-------------+------------+------+---------------+------+---------+------+-------+-------------+

The 2 important fields here are:

- Type: When you see "ALL", MySQL is performing a full table scan which is a very CPU intensive operation.
- Row: This is the total amount of rows returned in the table, so 83,000 results is a lot to sort through.

In general, when you are creating an index, you want to choose a field that has the highest amount of unique characters. In our case, we are going to use the field ‘key’ as shown below:

create index key_idx on wp_table(`key`);

Now, lets rerun our explain to see if the query is now returning less rows:

explain SELECT * FROM wp_table WHERE `key`='5544dDSDFjjghhd2544xGFDE' AND `carrier`='13';
+----+-------------+------------+------+---------------+------+---------+------+-------+-------------+
| id | select_type | table      | type | possible_keys | key  | key_len | ref  | rows  | Extra       |
+----+-------------+------------+------+---------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | wp_table   | ALL  | NULL          | NULL |    NULL | NULL | 13    | Using where |
+----+-------------+------------+------+---------------+------+---------+------+-------+-------------+

This is much better. Now each time that common query runs, MySQL will only have to go through 13 rows, instead of it having to check through 83,000.

Important note: Each time a table is updated, MySQL has to update the indexes, which could create some performance issues. Therefore, its recommended to keep the amount of indexes per table low, perhaps in the 4-6 range.

How to see what indexes already exist on a table and their cardinality:

show indexes from wp_table;

How to remove a table index:

delete index key_idx from wp_table;

RCS – Introduction

When there are 40+ admin’s logging into a client’s server, it can become difficult to keep track of who modified what. And more importantly, in the event that a change created an undesired result, being able to find out exactly what was changed so it can be quickly rolled back. This also becomes a critical component of change control if the client requires specific security requirements such as PCI-DSS 2.0.

This system of revision control is much cleaner to track changes rather creating a bunch of apache2.bak, apache2.20120212, apahce2.conf.031212, etc. Instead, you can view all the versions of the file available simply by:
rlog /etc/apache2/apache2.conf

RCS offers the following features in a very easy to use CLI:

- Store and retrieve multiple revisions of text
- Maintain a complete history of changes
- Maintain a tree of revisions
- Automatically identify each revision with name, revision number, creation time, author, etc
- And much more

For our specific use case, critical files to check into RCS would be configuration files such as /etc/sysctl.conf, /etc/ssh/sshd_config /etc/vsftpd/vsftpd.conf, /etc/httpd/conf/httpd.conf and stuff of that nature.

If RCS is not already installed, then simply run the following depending on your operating system:

yum install rcs
apt-get install rcs

Basic Use Case
The easiest way to learn RCS is to see it in action. So in the use case below, we are going to perform a series of changes to the httpd.conf file.  Before making changes to the file, check it into RCS first so we have a starting point:

root@web01:/etc/apache2# ci -l -wjdoe /etc/apache2/apache2.conf
/etc/apache2/apache2.conf,v  <--  /etc/apache2/apache2.conf
enter description, terminated with single '.' or end of file:
NOTE: This is NOT the log message!
>> Original Apache Configuration File
>> .
initial revision: 1.1
done

Now we can make our change to the config. As an example, we are going to be making some tuning changes to Apache.

vi /etc/apache2/apache2.conf

Once our changes are made, we check the changes in:

root@web01:/etc/apache2# ci -l -wjdoe /etc/apache2/apache2.conf
/etc/apache2/apache2.conf,v  <--  /etc/apache2/apache2.conf
new revision: 1.2; previous revision: 1.1
enter log message, terminated with single '.' or end of file:
>> Tuning changes per ticket #123456
>> .
done

Pretend a few days go by and you receive a call from the client reporting issues with Apache. You log into the server and checks to see if anyone recently made changes to Apache:

root@web01:/etc/apache2# rlog /etc/apache2/apache2.conf

RCS file: /etc/apache2/apache2.conf,v
Working file: /etc/apache2/apache2.conf
head: 1.2
branch:
locks: strict
        root: 1.2
access list:
symbolic names:
keyword substitution: kv
total revisions: 2;     selected revisions: 2
description:
Original Apache Configuration File
----------------------------
revision 1.2    locked by: root;
date: 2012/03/19 15:44:06;  author: jdoe;  state: Exp;  lines: +3 -3
Tuning changes per ticket #123456
----------------------------
revision 1.1
date: 2012/03/19 15:28:38;  author: jdoe;  state: Exp;
Initial revision
=============================================================================

So this tells us that user jdoe make changes to the apache2.conf on 3/19/2012 per ticket #123456. Lets see what changes he made by comparing version 1.1 to version 1.2:

root@web01:/etc/apache2# rcsdiff -r1.1 -r1.2 /etc/apache2/apache2.conf
===================================================================
RCS file: /etc/apache2/apache2.conf,v
retrieving revision 1.1
retrieving revision 1.2
diff -r1.1 -r1.2
77c77
< KeepAlive On
---
> KeepAlive Off
105,106c105,106
<     MaxSpareServers      10
<     MaxClients          150
---
>     MaxSpareServers      1
>     MaxClients          15
root@web01:/etc/apache2#

From the looks of this, it appears he may have typo’ed the MaxClient and MaxSpareServer variable when working that ticket. So lets roll back the configuration file to version 1.1 since that was the last known working version:

root@web01:/etc/apache2# co -r1.1 /etc/apache2/apache2.conf
/etc/apache2/apache2.conf,v  -->  /etc/apache2/apache2.conf
revision 1.1
writable /etc/apache2/apache2.conf exists; remove it? [ny](n): y
done

Then test Apache to confirm everything is working again. Be sure to commit your changes as a rollback is still a change:

root@web01:/etc/apache2# ci -l -wmsmith /etc/apache2/apache2.conf
/etc/apache2/apache2.conf,v  <--  /etc/apache2/apache2.conf
new revision: 1.3; previous revision: 1.2
enter log message, terminated with single '.' or end of file:
>> Rolling back changes made in ticket #123456 due to problems
>> .
done

When the next person logs in to see what changes have been made to the apache.conf, they will see the following:

root@web01:/etc/apache2# rlog /etc/apache2/apache2.conf

RCS file: /etc/apache2/apache2.conf,v
Working file: /etc/apache2/apache2.conf
head: 1.3
branch:
locks: strict
        root: 1.3
access list:
symbolic names:
keyword substitution: kv
total revisions: 3;     selected revisions: 3
description:
Original Apache Configuration File
----------------------------
revision 1.3    locked by: root;
date: 2012/03/19 16:00:38;  author: msmith;  state: Exp;  lines: +3 -3
Rolling back changes made in ticket #123456 due to problems
----------------------------
revision 1.2
date: 2012/03/19 15:44:06;  author: jdoe;  state: Exp;  lines: +3 -3
Tuning changes per ticket #123456
----------------------------
revision 1.1
date: 2012/03/19 15:28:38;  author: jdoe;  state: Exp;
Initial revision
=============================================================================