How to setup DRBD

Distributed Replicated Block Device (DRBD) mirrors block devices between multiple hosts. You can think of this loosely as network Raid 1.

DRBD is meant to run in a Active / Passive setup, meaning, you can only mount the disk on one node at a time. This is not a DRBD limitation, but rather a limitation of the common file systems (ext3, ext4, xfs, etc), since they cannot account for 2 or more servers accessing a single disk.

As with any form of data replication, always ensure you have good backups before you begin, and ensure that you have good backups throughout the life cycle of the setup. There is always a chance of data corruption or complete data loss due to some unforeseen situation, so make sure you have backups, and you have tested restoring from those backups!

Requirements

There are a few requirements that need to be met for DRBD to function properly and securely:

1. 2x servers with similar block devices
2. DRBD kernel module and userspace utilities
3. Private network between the servers
4. iptables port 7788 open between servers on the Private network
5. /etc/hosts configured
6. NTP synchronized

Preparation

For the purposes of this article, my two servers running CentOS 6 will be:

drbd01 192.168.5.2 | Cloud Block Storage 50G SSD
drbd01 192.168.5.3 | Cloud Block Storage 50G SSD

First, ensure that /etc/hosts are setup properly on both servers:

cat /etc/hosts
192.168.5.2 drbd01
192.168.5.3 drbd02

Next, open up iptables on both servers to allow communications across the private network:

cat /etc/sysconfig/iptables
-A INPUT -i eth2 -s 192.168.5.0/24 -p tcp --dport 7788 -m comment --comment "Allow DRBD on private interface" -j ACCEPT
...
service iptables restart

Finally, prep your block devices, but do not format them with a filesystem! For this guide, I am going to assume you are using separate disks for this, which are setup on /dev/xvdb:

fdisk /dev/xvdb
N
P 
1
enter
enter
t (choose 83)
w (write)
fdisk -l /dev/xvdb1 (confirm all looks well)

Install DRBD

CentOS requires the use of the RPM packages found in the repo, http://www.elrepo.org. This will provide the DKMS-based kernel module and userspace tools.

On both nodes:

rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
yum repolist
yum install drbd83-utils kmod-drbd83 dkms ntp ntpdate
service ntpd restart && chkconfig ntpd on
reboot

Configure DRBD

First, configure the global_common.conf

vi /etc/drbd.d/global_common.conf
# Change
usage-count no;
# To 
usage-count yes;

Then search for syncer {, and add rate 10M;. An example is posted below:

syncer {
# rate after al-extents use-rle cpu-mask verify-alg csums-alg
rate 10M;
}

Some important notes:

1. usage-count. The DRBD project keeps statistics about the usage of various DRBD versions. This is done by contacting an HTTP server every time a new DRBD version is installed on a system. This can be disabled by setting usage-count no;. The default is usage-count ask; which will prompt you everytime you upgrade DRBD.

2. rate 10M: This throttles the total bandwidth that DRBD will use to perform its tasks between the 2 nodes. A good rule of thumb for this value is to use about 30% of the available replication bandwidth. Thus, if you had an I/O subsystem capable of sustaining write throughput of 180MB/s, and a Gigabit Ethernet network capable of sustaining 110 MB/s network throughput (the network being the bottleneck), you would calculate: 110 x 0.3 = 33MB/s. I opted to go with 10M for this article. 10M is a bit on the low side, so read the following guide and increased your limits as needed depending on your available bandwidth: https://drbd.linbit.com/users-guide/s-configure-sync-rate.html

Resource Settings

Configure the 2 nodes so they can communicate with each other. On both servers, setup:

vi /etc/drbd.d/cent00.res
resource cent00 {
  protocol C;
  startup { wfc-timeout 0; degr-wfc-timeout 120; }
  disk { on-io-error detach; }
  net { cram-hmac-alg "sha1"; shared-secret "4ftl421dg987d33gR"; }
  on drbd01 {
    device /dev/drbd0;
    disk /dev/xvdb1;
    meta-disk internal;
    address 192.168.5.2:7788;
  }
  on drbd02 {
    device /dev/drbd0;
    disk /dev/xvdb1;
    meta-disk internal;
    address 192.168.5.3:7788;
  }
}

Now initialize the resources DRBD will be using, and set drbd01 to be primary. This is done by:

[[email protected] ~]# drbdadm create-md cent00
[[email protected] ~]# drbdadm create-md cent00
 
[[email protected] ~]# service drbd start; chkconfig drbd on
[[email protected] ~]# service drbd start; chkconfig drbd on
[[email protected] ~]# drbdadm -- --overwrite-data-of-peer primary cent00 

Once this is done, the disks will begin to sync up. This could take several hours. You can check the status by:

[[email protected] ~]# cat /proc/drbd 
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by [email protected], 2013-09-27 16:00:43
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:1124352 nr:0 dw:0 dr:1125016 al:0 bm:68 lo:0 pe:1 ua:0 ap:0 ep:1 wo:f oos:19842524
[>...................] sync'ed: 5.4% (19376/20472)M
finish: 0:31:21 speed: 10,536 (10,312) K/sec

Setup Filesystem

It is recommended to wait until the initial synchronization is complete. It simply depends on the size of the block storage, and the speed of the internal network connecting the 2 servers. You can check the status by running

[[email protected] ~]# cat /proc/drbd

Then before continuing, make sure you are on the primary node first:
** Due to WordPress, I had to put a space in the opening tags to avoid it being processed as markup.

[[email protected] ~]# drbdadm -- status cent00
< drbd-status version="8.3.16" api="88">
< resources config_file="/etc/drbd.conf">
< resource minor="0" name="cent00" cs="Connected" ro1="Primary" ro2="Secondary" ds1="UpToDate" ds2="UpToDate" />
< /resources>
< /drbd-status>

We’ll use the standard ext4 file system for this:

[[email protected] ~]# mkfs.ext4 /dev/drbd0
[[email protected] ~]# mkdir /data
[[email protected] ~]# mount -t ext4 /dev/drbd0 /data

Testing Scenarios

Below are some basic test scenarios you can simulate pretty easily. This goes without saying, but do not experiment with these scenarios on your production environment! Know what they do before you run them in production since they can cause problems if your not ready for it!

These are broken down into the following tests:
Test 1: Promote drbd02 to become primary
Test 2: Testing secondary node failure
Test 3: Testing primary node failure
Test 4: Recovering from split-brain

Test 1: Promote drbd02 to become primary

Unmount the partition and demote the current primary (drbd01) to secondary:

[[email protected] ~]# umount /data
[[email protected] ~]# drbdadm secondary cent00
On other server, drbd02, promote it to primary and mount the drbd device:
[[email protected] ~]# drbdadm primary cent00
[[email protected] ~]# mkdir /data
[[email protected] ~]# mount -t ext4 /dev/drbd0 /data
[[email protected] ~]# ls -d /data/*

At this time, drbd02 will now be the primary, and drdb01 will now be the secondary node.

Test 2: Testing secondary node failure

To see what happens when the secondary server goes offline:
Shutdown your secondary node, which in this case, is drbd02:

[[email protected] ~]# shutdown -h now

Now, back on the primary node drbd01, add a few files to the volume:

[[email protected] ~]# mkdir -p /data/test/
[[email protected] ~]# cp /etc/hosts /data/test/

Power back on the secondary node drbd02, and watch the system sync back up. Note, depending on how much data was written, it may take a bit of time for the volumes to become consistent again. You can check the status with:

[[email protected] ~]# cat /proc/drbd

Test 3: Testing primary node failure

This tests what happens when primary node goes offline, and someone promotes the secondary node before the primary comes online and can be demoted (split-brain).

If you want to simulate this worst case scenario, and you don’t care about your data, then perform the following:

[[email protected] ~]# echo 1 > /proc/sys/kernel/sysrq ; echo b > /proc/sysrq-trigger
[[email protected] ~]# reboot -f -n

Or just shutdown drbd01 (primary), and then log into drbd02 (secondary), and promote it to master:

[[email protected] ~]# drbdadm primary cent00
[[email protected] ~]# mkdir /data
[[email protected] ~]# mount -t ext4 /dev/drbd0 /data

Then boot drbd01 again and enjoy the split-brain scenario! For obvious reasons, do NOT do this on drives containing any data you need for anything! If the primary node loses the replication link, and you made the other node primary BEFORE connectivity is restored, you WILL have split-brain. Avoid that at all costs.

Test 4: Recovering from split-brain

In the event of split-brain, you may be able to correct it by performing the following, but do not do this blindly! Make sure you understand what this is doing before you run it on your production data, otherwise you may lose data you wanted! More information can be found at http://drbd.linbit.com/docs/working/

For reference:
– drbd01 : Primary node
– drbd02 : Secondary node

On secondary node

[[email protected] ~]# drbdadm secondary cent00
[[email protected] ~]# drbdadm -- --discard-my-data connect cent00

And back on the primary node

[[email protected] ~]# drbdadm connect cent00
[[email protected] ~]# cat /proc/drbd