VMware disk expansion

There are a number of ways to add more disk space to a VM on VMware. This guide will discuss 5 different ways to handle expanding the existing disk in VMware, which are:

- Expand existing disk to LVM (Not previously expanded)
- Expand existing disk to LVM (Previously expanded)
- Expand existing disk with LVM not in use (Dangerous)
- Add a new disk into an existing LVM Volume Group
- Add a new disk as a separate mount point

Many VMware solutions set their disk labels to MBR, so for this guide, we’ll be making extensive use of fdisk. If your disk label is set to GPT, please use caution when following this guide!

As with any disk expansion operations, always be sure you have solid backups in place in case something goes wrong!

Expand existing disk to LVM (Not previously expanded)

Assuming the VM’s disk has already been expanded within VMware, you have to rescan the specific SD device to detect the new properties. You can do this by:

[[email protected] ~]# echo 1 > /sys/block/sdX/device/rescan
or
[[email protected] ~]# echo 1 > /sys/bus/scsi/drivers/sd/SCSI-ID/block/device/rescan
or
[[email protected] ~]# echo 1 > /sys/bus/scsi/drivers/sd/SCSI-ID/block\:sdX/device/rescan

Whether the added a new disk, or expanded an existing one, you can usually detect the change by:

[[email protected] ~]# dmesg|tail
...
sd 2:0:0:0: [sda] 67108864 512-byte logical blocks: (34.3 GB/32.0 GiB)
sd 2:0:0:0: [sda] Cache data unavailable
sd 2:0:0:0: [sda] Assuming drive cache: write through
sda: detected capacity change from 17179869184 to 34359738368

Now you need to determine if the volume has actually be expanded. Check for the ‘resize_inode’ flag by:

[[email protected] ~]# tune2fs -l /dev/vglocal00/lvroot | grep -i "^filesystem features"

Check to see if storage has increased in size yet first by:

[[email protected] ~]# fdisk -cul /dev/sda
[[email protected] ~]# pvs
[[email protected] ~]# vgs
[[email protected] ~]# lvs
[[email protected] ~]# df -h

Once the root disk has been expanded in VMware, rescan the disk which should now show additional sectors have been added:

[[email protected] ~]# echo 1 > /sys/block/sda/device/rescan
[[email protected] ~]# fdisk -cul /dev/sda

Now we need to add a partition for the new space. As fdisk only allows 4 primary partitions, we are going to use extended partitions so we can create logical partitions to hold the new space:

[[email protected] ~]# fdisk -cu /dev/sda
p
n
e (extended)
3
enter
enter
n
l (logical)
enter
enter
p
w

Now rescan the partitions so the system can detect the new one without rebooting:

[[email protected] ~]# ls /dev/sda*
[[email protected] ~]# partx -l /dev/sda
[[email protected] ~]# partx -v -a /dev/sda # There may be some errors here, ignore.
[[email protected] ~]# partx -l /dev/sda
[[email protected] ~]# ls /dev/sda*

Now setup LVM on that new partition, and add it to the existing volume group and expand the logical volume:

[[email protected] ~]# pvcreate /dev/sda5
[[email protected] ~]# vgs
[[email protected] ~]# vgextend YOUR_VG_NAME /dev/sda5
[[email protected] ~]# pvdisplay /dev/sda5 | grep Free
  Free PE               4095
[[email protected] ~]# lvextend --extents +4095 -n /dev/YOUR_VG_NAME/lv_root

Finally, expand the filesystem doing an online resize:

[[email protected] ~]# resize2fs /dev/YOUR_VG_NAME/lv_root
[[email protected] ~]# df -h |grep root

Expand existing disk to LVM (Previously expanded)

If there is a VM where a previous expansion already took place, or otherwise is already on an extended partition with the first (only) logical partition taking up all the room, then this is the section you want.

Once the root disk has been expanded in VMware, rescan the disk which should now show additional sectors have been added:

# Print out disk information
[[email protected] ~]# fdisk -cul /dev/sda

# Then rescan the device
[[email protected] ~]# echo 1 > /sys/block/sdX/device/rescan
or
[[email protected] ~]# echo 1 > /sys/bus/scsi/drivers/sd/SCSI-ID/block/device/rescan
or
[[email protected] ~]# echo 1 > /sys/bus/scsi/drivers/sd/SCSI-ID/block\:sdX/device/rescan

# Print out disk information to confirm it detected the additional space
[[email protected] ~]# fdisk -cul /dev/sda

Expand the existing extended partition:

[[email protected] ~]# parted /dev/sda
unit s
pri
  Number  Start      End        Size       Type      File system  Flags
   1      2048s      41431039s  41428992s  primary                lvm
   2      41431040s  41943039s  512000s    primary   ext3         boot
   3      41943040s  52428799s  10485760s  extended
   5      41945088s  52428799s  10483712s  logical
resize 3 41943040s -1  (Take whatever the extended start value is, and the number)
pri
quit

Now partition the new space, setup LVM, expand and resize the filesystem:

[[email protected] ~]# fdisk -cu /dev/sda
p
n
l (logical)
enter
enter
p
w

[[email protected] ~]# ls -hal /dev/sda*
[[email protected] ~]# partx -l /dev/sda
[[email protected] ~]# partx -v -a /dev/sda # There may be some errors here, ignore.
[[email protected] ~]# partx -l /dev/sda
[[email protected] ~]# ls -hal /dev/sda*

[[email protected] ~]# pvcreate /dev/sda6 # Or whatever the new partition was
[[email protected] ~]# vgs
[[email protected] ~]# vgextend YOUR_VG_NAME /dev/sda6
[[email protected] ~]# pvdisplay /dev/sda6 | grep Free
  Free PE               4607
[[email protected] ~]# lvextend --extents +4607 -n /dev/YOUR_VG_NAME/lv_root
[[email protected] ~]# df -h
[[email protected] ~]# resize2fs /dev/YOUR_VG_NAME/lv_root
[[email protected] ~]# df -h

Expand existing disk to without LVM (Dangerous)

This section assumes that LVM was never setup for the disk. Therefore you would need to recreate the partitions to use the new space.

Re-creating partitions is a high risk operation as there is there is the potential for data loss, so make sure you have known good backups you can restore to. And at the very least, snapshot your VM! It also requires a reboot to occur on the VM. Ideally, you should first check to see if an additional disk can simply be mounted to a different mount point instead.

First, list the current partitions:

[[email protected] ~]# fdisk -l

Now within VMware or on the SAN presenting the disk, expand the disk. Once that is done, we need to rescan the volume and confirm the new space:

[[email protected] ~]# echo 1 > /sys/block/sda/device/rescan
[[email protected] ~]# fdisk -l
     Device Boot      Start         End      Blocks   Id  System
  /dev/sda1   *           1          13      104391   83  Linux
  /dev/sda2              14         274     2096482+  83  Linux
  /dev/sda3             275         796     4192965   82  Linux swap / Solaris
  /dev/sda4             797        2610    14570955    5  Extended
  /dev/sda5             797        2610    14570923+  83  Linux

Using the example above, you will notice that the new partitions (4 and 5) end on the same cylinder (2610). So the extended and logical partitions need to be set to use the new space. So to attempt to help you in the event everything goes wrong, list out the following information and store it somewhere safe so you can refer to it later:

[[email protected] ~]# fdisk -l /dev/sda
[[email protected] ~]# df -h
[[email protected] ~]# cat /etc/fstab

Now hold on to your butts and expand the disks by deleting the partitions (which *shouldn’t* affect the underlining data), then recreate the partitions with the new sizes:

[[email protected] ~]# fdisk /dev/sda
d
5
d
4
n
e
(Pick original extended position (Should be default, just hit enter)
(Pick the new, much larger cylinder ending position (for default all space until end, hit enter)
n
5 (or it should just assume right now here)
(Pick original logical partition started point (should be default next cylinder, hit enter)
(Pick new, much larger cylinder ending position (just default all space until end, hit enter)
p (Double check everything, ensure starting cylinders to extended partition 4 and logical partition 5 have the SAME starting cylinder number
w

Now reboot the system so it can use the new space:

[[email protected] ~]# shutdown -r now

Then expand the filesystem:

[[email protected] ~]# df -h | grep sda5
  /dev/sda5              14G  2.6G   11G  21% /
[[email protected] ~]# resize2fs /dev/sda5
[[email protected] ~]# df -h | grep sda5
  /dev/sda5              19G  2.6G   15G  15% /

Add a new disk or into an existing LVM Volume Group

This section assumes the new disk is the second disk on the VM, and is enumerated as /dev/sdb. The disk will be added to an existing Volume Group, and we’ll use all the new space on the disk for the volume group and logical volume.

[[email protected] ~]# parted -s -- /dev/sdb mklabel gpt
[[email protected] ~]# parted -s -a optimal -- /dev/sdb mkpart primary 2048s -1
[[email protected] ~]# parted -s -- /dev/sdb align-check optimal 1
[[email protected] ~]# parted /dev/sdb set 1 lvm on
[[email protected] ~]# parted /dev/sdb unit s print
[[email protected] ~]# pvcreate --metadatasize 250k /dev/sdb1
[[email protected] ~]# vgs
[[email protected] ~]# vgextend YOUR_VG_NAME /dev/sdb1
[[email protected] ~]# pvdisplay /dev/sdb1 | grep Free
  Free PE               4095
[[email protected] ~]# lvextend --extents +4095 -n /dev/YOUR_VG_NAME/lv_root
[[email protected] ~]# df -h
[[email protected] ~]# resize2fs /dev/YOUR_VG_NAME/lv_root
[[email protected] ~]# df -h

Add a new disk as a separate mount point

This section assumes the new disk is the second disk on the VM, and it enumerated as /dev/sdb. We are going to use GPT and LVM as a best practice (even if the root disk/partition has the disk label set to MBR or is non-LVM). This example also uses the whole disk in one partition.

# RHEL/CentOS 5:  Scan for new disk, check for existing partitions
# setup gpt, align, and partition:
[[email protected] ~]# for x in /sys/class/scsi_host/host*/scan; do echo "- - -" > ${x}; done
[[email protected] ~]# parted /dev/sdb unit s print
[[email protected] ~]# fdisk -l /dev/sdb
[[email protected] ~]# parted /dev/sdb
mktable gpt
quit
[[email protected] ~]# parted -s -- /dev/sdb mkpart primary 2048s -1
[[email protected] ~]# parted /dev/sdb set 1 lvm on
[[email protected] ~]# parted /dev/sdb unit s print

# RHEL/CentOS 6:  Scan for new disk, check for existing partitions
# setup gpt, align, and partition:
[[email protected] ~]# for x in /sys/class/scsi_host/host*/scan; do echo "- - -" > ${x}; done
[[email protected] ~]# parted /dev/sdb unit s print
[[email protected] ~]# fdisk -l /dev/sdb
[[email protected] ~]# parted -s -- /dev/sdb mklabel gpt
[[email protected] ~]# parted -s -a optimal -- /dev/sdb mkpart primary 2048s -1
[[email protected] ~]# parted -s -- /dev/sdb align-check optimal 1
[[email protected] ~]# parted /dev/sdb set 1 lvm on
[[email protected] ~]# parted /dev/sdb unit s print

Now on both OS’s, setup LVM, format, and mount the volume to /mnt/data:

[[email protected] ~]# VGNAME=vglocal$(date +%Y%m%d)
[[email protected] ~]# LVNAME=lvdata01
[[email protected] ~]# MOUNTPOINT="/mnt/data"
[[email protected] ~]# FILESYSTEM=`mount | egrep "\ \/\ " | awk '{print $5}'`
[[email protected] ~]# pvcreate --metadatasize 250k /dev/sdb1
[[email protected] ~]# vgcreate ${VGNAME} /dev/sdb1
[[email protected] ~]# lvcreate --extents 100%VG -n ${LVNAME} ${VGNAME}
[[email protected] ~]# mkfs.${FILESYSTEM} /dev/mapper/${VGNAME}-${LVNAME}
[[email protected] ~]# mkdir ${MOUNTPOINT}
[[email protected] ~]# echo -e "/dev/mapper/${VGNAME}-${LVNAME}\t${MOUNTPOINT}\t${FILESYSTEM}\tdefaults\t0 0" >> /etc/fstab
[[email protected] ~]# mount -a
[[email protected] ~]# df -hP | grep "${VGNAME}-${LVNAME}"
/dev/mapper/vglocal20160830-lvdata01   16G   44M   15G   1% /mnt/data

LVM basics

Logical Volume Management, or LVM for short, takes entire disks or individual partitions, and combines them together so the group can act as a single managable entity.

A few best practices to keep in mind when using LVM.

1. The Volume Group name should represent what kind of storage it exists on, such as vglocal00, vgsan00, vgdas00, vgiscsi00, vgraid00, etc.

2. The Logical Volume name should represent what the LV is being used for where possible, such as nfs00, data00, mysql00, var00, root00, etc. So the end result of a LV for MySQL running on SAN would be: /dev/vgsan00/mysql00

3. Never combine disks coming from different raids. In other words, don’t combine disks from a raid 1 and a raid 5 in the same Volume Group.

4. Never combine disks from different storage mediums, such as local storage and remote (SAN, DAS, iSCSI, etc).

5. Never combine non-partitioned and partitioned devices due to performance issues and general end user confusion.

6. To avoid end user confusion, a partition should be created on the new physical device as some tools may not be able to see that data already resides on the physical volumes when using tools like fdisk, parted, gdisk, etc.

Setup new disk

We are going to assume that your new disk is setup on /dev/sdb. First, determine if there is a disk label already set, and check for any existing information. You just want to avoid accidentally data loss:

[[email protected] ~]# parted /dev/sdb unit s print | grep Table
[[email protected] ~]# parted /dev/sdb unit s print

Set the disk label on the new disk to GPT:

[[email protected] ~]# parted -s -- /dev/sdb mklabel gpt

On the first partition only, start the partition on sector 2048 to follow generally accepted best practices to ensure partition alignment:

[[email protected] ~]# parted -s -a optimal -- /dev/sdb mkpart primary 2048s -1

Now confirm the starting sector of the partition is aligned for the disk:

[[email protected] ~]# parted -s -- /dev/sdb align-check optimal 1

Set the partition to use LVM:

[[email protected] ~]# parted /dev/sdb set 1 lvm on

Now review the disks newly created partition layout:

[[email protected] ~]# parted /dev/sdb unit s print

Setup the new disk with LVM:

[[email protected] ~]# pvcreate --metadatasize 250k /dev/sdb1

Create the volume group:

[[email protected] ~]# vgcreate vglocal00 /dev/sdb1

And now setup the logical volume to use all available disk space:

[[email protected] ~]# lvcreate -n data00 -l 100%FREE vglocal00

Format the logical volume with your filesystem:

[[email protected] ~]# mkfs.ext4 -v -m2 /dev/vglocal00/data00

And finally, mount the new volume:

[[email protected] ~]# mkdir /mnt/data
[[email protected] ~]# echo "/dev/vglocal00/data00   /mnt/data       ext4    defaults 0 0" >> /etc/fstab
[[email protected] ~]# mount -a
[[email protected] ~]# df -h

Shrink an existing Logical Volume

If you have to shrink an existing volume, there are a few steps that need to be taken. While its generally safe, you should always ensure that you have known good backups in place before proceeding.

Also note that you cannot shrink an existing volume while it is mounted. So this should be done during a scheduled maintenance window as you will need to stop any services that are using data from that volume.

First, unmount the logical volume:

[[email protected] ~]# umount /mnt/data

Run a file system check on the logical volume:

[[email protected] ~]# e2fsck -f /dev/vglocal00/data00

Now shrink the volume. In this example, we’re going to shrink it down to be 15G in size:

[[email protected] ~]# resize2fs /dev/vglocal00/data00 15G

Now reduce the file system to be 15G in size:

[[email protected] ~]# lvreduce -L 15G /dev/vglocal00/data00

Finally, mount the filesystem for normal use again:

[[email protected] ~]# mount -a

Shrink the root Logical Volume

As the / logical volume cannot be unmounted while the system is running, you need to boot the server off the distro’s cd, or boot in it a rescue environment if your running a Cloud server that supports this. While its generally safe to resize a volume, you should always ensure that you have known good backups in place before proceeding.

In this example, I’m running my server in VMware, so I can simply boot using a CentOS 6 installation cdrom. When the installation screen comes up, select:

Rescue installed system

When the screen asks if you would like the rescue environment to attempt to find your Linux installation and mount it under the directory /mnt/sysimage, select:

Skip

Now that your booted into the rescue enviroment, run the following commands so the system is aware of your LVM setup:

pvscan
vgscan
vgchange -a y
lvscan

In my case, my root logical volume is /dev/vg_local/lv_root. I want to shrink it from 60G down to 6G. I already confirmed that my data in the / partition does not exceed 6G.

First, run a file system check on the logical volume:

[[email protected] ~]# e2fsck -f /dev/vglocal00/lv_root

Now shrink the root volume. In this example, we’re going to shrink it down to be 6G in size:

[[email protected] ~]# resize2fs /dev/vglocal00/lv_root 6G

Then reduce the file system to be 6G in size:

[[email protected] ~]# lvreduce -L 6G /dev/vglocal00/lv_root

Finally, eject the CD, reboot the system, and check to ensure your / file system is now at 6G:

[[email protected] ~]# df -h /

Expand an existing Logical Volume

This operation can be done live with LVM2.

First confirm you have enough free space in the volume group by running:

[[email protected] ~]# vgs
[[email protected] ~]# vgdisplay vglocal00

Now lets expand the logical volume ‘data00’ from 15G to 25G total.

[[email protected] ~]# df -h
[[email protected] ~]# lvextend -L 25G /dev/vglocal00/data00
[[email protected] ~]# resize2fs /dev/vglocal00/data00
[[email protected] ~]# df -h

Add a new Logical Volume to an existing Volume Group

First confirm you have enough free space in the volume group by running:

[[email protected] ~]# vgs
[[email protected] ~]# vgdisplay vglocal00

Now create a new 5G logical volume called mysql00:

[[email protected] ~]# lvcreate -n mysql00 -L 5G vglocal00
[[email protected] ~]# mkfs.ext4 -v -m2 /dev/vglocal00/mysql00

Finally, mount the new logical volume:

[[email protected] ~]# mkdir /mnt/mysql-fs
[[email protected] ~]# echo "/dev/vglocal00/mysql00   /mnt/mysql-fs       ext4    defaults 0 0" >> /etc/fstab
[[email protected] ~]# mount -a
[[email protected] ~]# df -h

Remove a Logical Volume

First, unmount the volume:

[[email protected] ~]# umount /mnt/mysql-fs

Then remove the volume:

[[email protected] ~]# lvremove /dev/vglocal00/mysql00 
Do you really want to remove active logical volume mysql00? [y/n]: y
  Logical volume "mysql00" successfully removed

How to resize ext3 or ext4 filesystems

You have a 50G Amazon Elastic Block Storage device, or maybe a 50G Rackspace SSD Cloud Block Storage device that is running low on space. So you clone it and create a 100G drive. But when you mount it on your system, it still only shows 50G. What the #$%&!

This is because the partition and filesystem needs to be expanded to know it has more space!

Assuming that you have the larger drive already attached to your system, first verify the drive size vs the partition size:

[[email protected] ~]# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   20G  0 disk 
└─xvda1 202:1    0   20G  0 part /
xvdb    202:16   0  100G  0 disk 
└─xvdb1 202:17   0   50G  0 part

Checking the output above, you will see that xvdb has 100G available, but the partition, xvdb1, has only 50G. I’ll outline how to expand the filesystem to use all the new space on this ext4 volume.

Unmount the disk if its currently mounted:

[[email protected] ~]# umount /dev/xvdb1

Confirm the drive does not have any filesystem errors:

[[email protected] ~]# e2fsck /dev/xvdb1

Now comes the nail biting part. We need to remove the partition, then recreate it so it will see the entire disk. This guide assumes there is only 1 partition on this drive. This will not remove the data, however never trust notes that you didn’t verify yourself. Make sure you have backups before proceeding!

[[email protected] ~]# fdisk /dev/xvdb
d
n
p
1
enter
enter
t
83
w

Now, once the partition is recreated, you need to run another filesystem check:

[[email protected] ~]# e2fsck -f /dev/xvdb1
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/xvdb1: 13/3276800 files (0.0% non-contiguous), 251700/13107024 blocks

Then resize the filesystem:

[[email protected] ~]# resize2fs /dev/xvdb1
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/xvdb1 to 26214055 (4k) blocks.
The filesystem on /dev/xvdb1 is now 26214055 blocks long.

Finally, mount the volume and confirm it shows the correct space:

[[email protected] ~]# mount -a
[[email protected] ~]# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   20G  0 disk 
└─xvda1 202:1    0   20G  0 part /
xvdb    202:16   0  100G  0 disk 
└─xvdb1 202:17   0  100G  0 part /mnt

[[email protected] ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       20G  1.2G   18G   7% /
tmpfs           496M     0  496M   0% /dev/shm
/dev/xvdb1       99G   60M   94G   1% /mnt

Force fsck on next reboot

Ever needed to do a fsck on the root filesystem? I was happy to find that an old method of forcing a file system check on reboot still exists on most modern distributions. I tested this successfully on the following operating systems:

CentOS 6
Ubuntu 12.04
Ubuntu 14.04

Before proceeding, it is a really good idea to have a monitor and keyboard hooked up to the server just in case the fsck requires manual intervention due to bad blocks or something, therefore preventing normal boot up!

First, check to see when the last time the server had a file system check ran. This command should be against the device listing for /, such as /dev/sda1, or if using LVM, something like /dev/mapper/ubuntu–vg-root.

[[email protected] ~]# tune2fs -l /dev/mapper/ubuntu--vg-root |grep -iE "last|state"
Last mounted on:          /
Filesystem state:         clean
Last mount time:          Thu Sep 10 23:48:18 2015
Last write time:          Sun Mar  1 16:02:06 2015
Last checked:             Sun Mar  1 16:02:04 2015

As shown in the output above, a fsck has not been ran in quite some time! So to force a fsck at next boot, simply type:

[[email protected] ~]# touch /forcefsck

Now reboot your server:

[[email protected] ~]# shutdown -r now

Once the server comes back online, confirm the fsck ran by:

[[email protected] ~]# tune2fs -l /dev/mapper/ubuntu--vg-root |grep -iE "last|state"
Last mounted on:          /
Filesystem state:         clean
Last mount time:          Thu Feb 18 18:40:34 2016
Last write time:          Thu Feb 18 18:40:32 2016

Finally, check to confirm the system removed the file /forcefsck:

[[email protected] ~]# ls -al /forcefsck
ls: cannot access /forcefsck: No such file or directory

Bind mounts

While no one likes opening up /etc/fstab and seeing 30 bind mounts, (a sure sign of an over caffeinated sysadmin), bind mounts can have their place in the world of system administration.

To put it in english, a bind mount is simply the process of mounting a specific directory from one path into another. And to describe it another way, it allows you to mount part of a filesystem to another location, and have it accessible from both locations.

Some common uses for bind mounts include:
– Providing access to log files from within a users home directory
– Providing access to files that reside outside a chroot
– Setting up NFSv4 /exports
– Setup a link that crosses filesystems, for example, a directory that resides on drive /dev/xvda1 to another drive on /dev/xvdb1

Basic bind mount example

Setting up a bind mount is simple. In the example below, we are going to bind mount our websites folder /var/www/vhosts/domain.com into our user’s folder: /home/bob

[[email protected] ~]# mkdir /home/bob/domain.com
[[email protected] ~]# mount --bind /var/www/vhosts/domain.com /home/bob/domain.com

Then set the mount point to be persistent at boot:

[[email protected] ~]# vim /etc/fstab
/var/www/vhosts/domain.com  /home/bob/domain.com  none  bind  0 0

Read only bind mounts

Here is an example for setting up a ready only bind mount. In this example, we are going to bind mount our Apache logs in /var/log/httpd into our user’s folder: /home/bob, and only allow them read only permissions so they can troubleshoot their website:

[[email protected] ~]# mkdir /home/bob/logs
[[email protected] ~]# mount --bind /var/log/httpd /home/bob/logs
[[email protected] ~]# mount --bind -o remount,ro /home/bob/logs

Then set the mount point to be persistent at boot, retaining the read only permissions. This one is a bit more tricky as you cannot set the read only flag in /etc/fstab. Therefore, you need to for the remount command at startup by adding it into /etc/rc.local so it’ll run at boot time:

[[email protected] ~]# mkdir /home/bob/logs
[[email protected] ~]# vim /etc/fstab
/var/log/httpd  /home/bob/logs  none  bind  0 0

[[email protected] ~]# chmod +x /etc/rc.local
[[email protected] ~]# vim /etc/rc.local
[[email protected] ~]# mount --bind -o remount,ro /home/bob/logs

Then consider rebooting your system to ensure the directory mounts properly in read only mode. You will know its successful when you cannot delete a file, or write a new file to that directory, even as the root user. For example:

[[email protected] ~]# touch /home/bob/logs/mylogfile
touch: cannot touch `/home/bob/logs/mylogfile': Read-only file system

[[email protected] ~]# rm -f /home/bob/logs/messages
rm: cannot remove `/home/bob/logs/messages': Read-only file system

Looking at whats underneath a mount point

Sometimes a sysadmin may have setup a new drive as the existing one was full. Then they transferred over a large directory to the new disk and symlinked it back to the original location. However they forgot to cleanup the directory on the original disk, so how does one find whats using up all the space since du -sh will not be able to see whats in there?

In other words, if a filesystem is mounted on a directory that contains files, any files below the mount point are hidden by the mounted file system. So how can we easily see whats in that directory without un-mounting the filesystem?

You can use bind mounts to essentially ‘peek’ under a mount point. Lets say the mount point you wanted to peek under is in /home/mydirectory. You can peek under the /home/mydirectory mount point by:

[[email protected] ~]# mount --bind /home/mydirectory /tmp/mnt-peek

Now you can see what files are ‘hidden’ by checking in /tmp/mnt-new. Once your done, be sure to unmount it by:

[[email protected] ~]# umount /tmp/mnt-peek

Setup NFSv4 on CentOS

NFSv4 is a tried and tested method of allowing client servers to access files over a network, in a very similar fashion to how the files would be accessed on a local file system. As a very mature piece of software, it has been successfully developed and used on production environments for over 15 years, and it is still widely accepted and supported with a long life ahead of it.

Setting it up is pretty easy and straight forward. As this is a network file system, it is strongly recommended to setup a private switch or private network between to the servers to ensure the lowest latency, as well as better security.

NFS Server – Installation

Install the required packages on the NFS server:

# CentOS 5, CentOS 6 and CentOS 7
[[email protected] ~]# yum install portmap nfs-utils nfs4-acl-tools -y

NFS Server – Configuration

Out of the box, NFSv4 has the following option set which is getting outdated sorely at this time:
– Enables only 8 NFS threads

We are going to enable 64 NFS threads since you will most likely run into IO problems before you hit this limit as it was meant for much older systems.

[[email protected] ~]# vim /etc/sysconfig/nfs
RPCNFSDCOUNT=64

Next, set the domain as all servers and clients should resides within the same domain:

[[email protected] ~]# vim /etc/idmapd.conf
[General]
Domain = yourdomain.com

Open the firewall to allow your private network access to the NFS services. You may have to adjust your rules as my private network resides on eth2. Do not allow this on the public interface without adjusting the source IP’s accordingly!

# CentOS 5 and CentOS 6
[[email protected] ~]# vim /etc/sysconfig/iptables
-A INPUT -i eth2 -s 192.168.1.0/24 -p tcp -m tcp --dport 111 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p udp -m udp --dport 111 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p tcp -m tcp --dport 662 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p udp -m udp --dport 662 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p tcp -m tcp --dport 892 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p udp -m udp --dport 892 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p tcp -m tcp --dport 2049 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p udp -m udp --dport 2049 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p tcp -m tcp --dport 32803 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p udp -m udp --dport 32769 -j ACCEPT

# CentOS 7
[[email protected] ~]# vim /etc/firewalld/services/nfs.xml
<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>NFS
  <description>NFS service
  <port protocol="tcp" port="111"/>
  <port protocol="udp" port="111"/>
  <port protocol="tcp" port="662"/>
  <port protocol="udp" port="662"/>
  <port protocol="tcp" port="892"/>
  <port protocol="udp" port="892"/>
  <port protocol="tcp" port="2049"/>
  <port protocol="udp" port="2049"/>
  <port protocol="tcp" port="32803"/>
  <port protocol="udp" port="32803"/>
  <port protocol="tcp" port="38467"/>
  <port protocol="udp" port="38467"/>
  <port protocol="tcp" port="32769"/>
  <port protocol="udp" port="32769"/>
</service>

Then apply the rules by:
[[email protected] ~]# systemctl reload firewalld.service

Now add a zone to the private network interface and set PEERDNS to no:
[[email protected] ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth2
...
PEERDNS=no
ZONE=internal
...

Then apply the changes by:
[[email protected] ~]# ifdown eth2 && ifup eth2

Now add the NFS rules to the private network interface:
[[email protected] ~]# firewall-cmd --zone=internal --add-interface eth2
[[email protected] ~]# firewall-cmd --zone=internal --add-service=nfs
[[email protected] ~]# firewall-cmd --zone=internal --add-interface eth2 --permanent
[[email protected] ~]# firewall-cmd --zone=internal --add-service=nfs --permanent

NFSv4 uses a pseudo filesystem for the exports. A pseudo filesystem allows NFS clients to browse the hierarchy of exported file systems, but remains limited to paths that are actually exported. There are a number of ways to go about this, but for this guide, we’ll assume the pseudo filesystem root will be /exports, and we’ll simply bind mount the desired directories into the /exports folder.

For this example, I am looking to export in /data. So we’ll bind mount that to the /exports folder as follows:

[[email protected] ~]# touch /data/test-file
[[email protected] ~]# mkdir /exports
[[email protected] ~]# mkdir /exports/data
[[email protected] ~]# echo "/data  /exports/data  none  bind  0 0" >> /etc/fstab
[[email protected] ~]# mount -a
[[email protected] ~]# ls -al /exports/data
total 8
drwxr-xr-x 2 root     root     4096 Jan 11 22:19 .
drwxr-xr-x 3 root     root     4096 Jan 11 22:03 ..
-rw-r--r-- 1 root     root        0 Jan 11 22:03 test-file

If you can see the file, test-file, within /exports/data, then everything is setup properly.

Export the directory to be shared, along with its permissions, in /etc/exports:

[[email protected] ~]# vi /etc/exports
/exports      192.168.1.0/24(ro,no_subtree_check,fsid=0,crossmnt)
/exports/data 192.168.1.0/24(rw,no_subtree_check,no_root_squash)

Now start the services, and enable them to start at boot time:

# CentOS 5
[[email protected] ~]# service portmap start; chkconfig portmap on
[[email protected] ~]# service rpcidmapd start; chkconfig rpcidmapd on
[[email protected] ~]# service nfs start; chkconfig nfs on

# CentOS 6
[[email protected] ~]# service rpcbind start; chkconfig rpcbind on
[[email protected] ~]# service rpcidmapd start; chkconfig rpcidmapd on
[[email protected] ~]# service nfs start; chkconfig nfs on

# CentOS 7
[[email protected] ~]# systemctl start rpcbind nfs-idmap nfs-server
[[email protected] ~]# systemctl enable rpcbind nfs-idmap nfs-server

Check to make sure the services are running:

[[email protected] ~]# showmount -e
Export list for nfs01:
/exports/data 192.168.1.0/24
/exports     192.168.1.0/24

[[email protected] ~]# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  41418  mountd
    100005    1   tcp  50139  mountd
    100005    2   udp  50228  mountd
    100005    2   tcp  52070  mountd
    100005    3   udp  33496  mountd
    100005    3   tcp  54673  mountd
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049  nfs_acl
    100227    3   tcp   2049  nfs_acl
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    2   udp   2049  nfs_acl
    100227    3   udp   2049  nfs_acl
    100021    1   udp  38895  nlockmgr
    100021    3   udp  38895  nlockmgr
    100021    4   udp  38895  nlockmgr
    100021    1   tcp  39908  nlockmgr
    100021    3   tcp  39908  nlockmgr
    100021    4   tcp  39908  nlockmgr

NFS Client – Installation

Now that the NFS server is ready, the NFS clients now need to be setup to connect. Install the required packages on the NFS clients by:

# CentOS 5
[[email protected] ~]# yum install portmap nfs-utils nfs4-acl-tools -y

# CentOS 6 and CentOS 7
[[email protected] ~]# yum install rpcbind nfs-utils nfs4-acl-tools -y

Next, set the domain as all servers and clients should resides within the same domain:

[[email protected] ~]# vim /etc/idmapd.conf
[General]
Domain = yourdomain.com

Now start the services, and enable them to start at boot time.

# CentOS 5
[[email protected] ~]# service portmap start; chkconfig portmap on
[[email protected] ~]# service rpcidmapd start; chkconfig rpcidmapd on
[[email protected] ~]# chkconfig netfs on

# CentOS 6
[[email protected] ~]# service rpcbind start; chkconfig rpcbind on
[[email protected] ~]# service rpcidmapd start; chkconfig rpcidmapd on
[[email protected] ~]# chkconfig netfs on

# CentOS 7
[[email protected] ~]# systemctl start rpcbind nfs-idmap
[[email protected] ~]# systemctl enable rpcbind nfs-idmap

NFS Client – Configuration

Confirm the NFS clients can see the NFS server:

[[email protected] ~]# showmount -e 192.168.1.1
Export list for 192.168.1.1:
/data 192.168.1.0/24

Configure the mount point in /etc/fstab:

[[email protected] ~]# vim /etc/fstab
192.168.1.1:/  /data  nfs4  sec=sys,noatime  0  0

Now create the placeholder directory on the client, mount, and verify it works:

[[email protected] ~]# mkdir /data
[[email protected] ~]# mount -a
[[email protected] ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       14G  1.8G   11G  15% /
tmpfs                 939M     0  939M   0% /dev/shm
/dev/sda1             477M   74M  378M  17% /boot
192.168.1.1:/data      14G  1.9G   11G  15% /data
[[email protected] ~]#
[[email protected] ~]# grep /data /proc/mounts 
192.168.1.1:/data/ /data nfs4 rw,noatime,vers=4,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.164,minorversion=0,local_lock=none,addr=192.168.1.1 0 0
[[email protected] ~]#
[[email protected] ~]# touch /data/test-file01
[[email protected] ~]# ls -al /data/test-file01
-rw-r--r-- 1 root root 0 Dec 19 17:57 /data/test-file01

Finally, confirm the user mapping is the same on both servers. You can test to ensure both the server and the client’s show the same UID’s by:

# Create a user on the NFS server:
[[email protected] ~]# useradd -u 6000 testuser

# Create the same user on the NFS client:
[[email protected] ~]# useradd -u 6000 testuser

# Set the ownership of a test file on the NFS server:
[[email protected] ~]# touch /data/test-file01
[[email protected] ~]# chown testuser:testuser /data/test-file01

# Check the ownership of the test file on the NFS server:
[[email protected] ~]# ls -al /data/test-file01

# Confirm the client sees the same ownership:
[[email protected] ~]# ls -al /data/test-file01

Setup NFSv4 on Ubuntu or Debian

NFSv4 is a tried and tested method of allowing client servers to access files over a network, in a very similar fashion to how the files would be accessed on a local file system. As a very mature piece of software, it has been successfully developed and used on production environments for over 15 years, and it is still widely accepted and supported with a long life ahead of it.

Setting it up is pretty easy and straight forward. As this is a network file system, it is strongly recommended to setup a private switch or private network between to the servers to ensure the lowest latency, as well as better security.

NFS Server – Installation

Install the required packages on the NFS server:

# Ubuntu and Debian
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install rpcbind nfs-common nfs4-acl-tools nfs-kernel-server

NFS Server – Configuration

Out of the box, NFSv4 has the following option set which is getting outdated sorely at this time:
– Sets random ephemeral ports upon daemon startup.
– Enables only 8 NFS threads

To make things more easier for admin’s to lock down the firewalls, we are going to set static ports, and also enable 64 NFS threads since you will most likely run into IO problems before you hit this limit as it was meant for much older systems.

Stop the services so we can unload the lockd kernel module and configure the services. This step cannot be skipped!

# Ubuntu 12.04 and Ubuntu 14.04
[[email protected] ~]# service nfs-kernel-server stop
[[email protected] ~]# service statd stop
[[email protected] ~]# service idmapd stop
[[email protected] ~]# service rpcbind stop
[[email protected] ~]# modprobe -r nfsd nfs lockd

# Debian 7
[[email protected] ~]# service nfs-kernel-server stop
[[email protected] ~]# service nfs-common stop
[[email protected] ~]# service rpcbind stop
[[email protected] ~]# modprobe -r nfsd nfs lockd

Update the NFS thread count by:

[[email protected] ~]# vim /etc/default/nfs-kernel-server
...
RPCNFSDCOUNT=64
RPCNFSDPRIORITY=0
RPCMOUNTDOPTS="--manage-gids"
NEED_SVCGSSD="no"
RPCSVCGSSDOPTS=""
...

Next, set the domain as all servers and clients should resides within the same domain:

[[email protected] ~]# vim /etc/idmapd.conf
[General]
Domain = yourdomain.com

Update nfs-common to disable statd and rpc.gssd, then require idmapd:

# Ubuntu 12.04 and Ubuntu 14.04
[[email protected] ~]# vim /etc/default/nfs-common
NEED_STATD=no
STATDOPTS=
NEED_GSSD=no

# Debian 7
[[email protected] ~]# vim /etc/default/nfs-common
NEED_STATD=no
STATDOPTS=
NEED_IDMAPD=yes
NEED_GSSD=no

Open the firewall to allow your private network access to the NFS services. You may have to adjust your rules as my private network resides on eth2. Do not allow this on the public interface without adjusting the source IP’s accordingly!

[[email protected] ~]# ufw allow in on eth2 to 192.168.1.0/24 proto tcp
[[email protected] ~]# ufw allow in on eth2 to 192.168.1.0/24 proto udp

NFSv4 uses a pseudo filesystem for the exports. A pseudo filesystem allows NFS clients to browse the hierarchy of exported file systems, but remains limited to paths that are actually exported. There are a number of ways to go about this, but for this guide, we’ll assume the pseudo filesystem root will be /exports, and we’ll simply bind mount the desired directories into the /exports folder.

For this example, I am looking to export in /data. So we’ll bind mount that to the /exports folder as follows:

[[email protected] ~]# mkdir /data
[[email protected] ~]# touch /data/test-file
[[email protected] ~]# mkdir /exports
[[email protected] ~]# mkdir /exports/data
[[email protected] ~]# echo "/data  /exports/data  none  bind  0 0" >> /etc/fstab
[[email protected] ~]# mount -a
[[email protected] ~]# ls -al /exports/data
total 8
drwxr-xr-x 2 root     root     4096 Jan 11 22:19 .
drwxr-xr-x 3 root     root     4096 Jan 11 22:03 ..
-rw-r--r-- 1 root     root        0 Jan 11 22:03 test-file

If you can see the file, test-file, within /exports/data, then everything is setup properly.

Export the directory to be shared, along with its permissions, in /etc/exports:

[[email protected] ~]# vim /etc/exports

/exports      192.168.1.0/24(ro,no_subtree_check,fsid=0,crossmnt)
/exports/data 192.168.1.0/24(rw,no_subtree_check,no_root_squash)

Now start the services, and ensure they will start at boot time:

# Ubuntu 12.04 and Ubuntu 14.04
[[email protected] ~]# service rpcbind start
[[email protected] ~]# service idmapd start
[[email protected] ~]# service nfs-kernel-server start; update-rc.d nfs-kernel-server enable

# Debian 7
[[email protected] ~]# service rpcbind start; insserv rpcbind
[[email protected] ~]# service nfs-common start; insserv nfs-common
[[email protected] ~]# service nfs-kernel-server start; insserv nfs-kernel-server

Check to make sure the services are running:

[[email protected] ~]# showmount -e
Export list for nfs01:
/exports/data 192.168.1.0/24
/exports     192.168.1.0/24

[[email protected] ~]# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049
    100227    3   tcp   2049
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    2   udp   2049
    100227    3   udp   2049
    100021    1   udp  39482  nlockmgr
    100021    3   udp  39482  nlockmgr
    100021    4   udp  39482  nlockmgr
    100021    1   tcp  60237  nlockmgr
    100021    3   tcp  60237  nlockmgr
    100021    4   tcp  60237  nlockmgr
    100005    1   udp  39160  mountd
    100005    1   tcp  34995  mountd
    100005    2   udp  34816  mountd
    100005    2   tcp  56338  mountd
    100005    3   udp  49147  mountd
    100005    3   tcp  51938  mountd

NFS Client – Installation

Now that the NFS server is ready, the NFS clients now need to be setup to connect. Install the required packages on the NFS clients by:

# Ubuntu or Debian
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install rpcbind nfs-common nfs4-acl-tools

NFS Client – Configuration

Stop the services so we can unload the lockd kernel module and configure the services. This step cannot be skipped!

# Ubuntu 12.04 and Ubuntu 14.04
[[email protected] ~]# service nfs-kernel-server stop
[[email protected] ~]# service statd stop
[[email protected] ~]# service idmapd stop
[[email protected] ~]# service rpcbind stop
[[email protected] ~]# modprobe -r nfsd nfs lockd

# Debian 7
[[email protected] ~]# service nfs-kernel-server stop
[[email protected] ~]# service nfs-common stop
[[email protected] ~]# service rpcbind stop
[[email protected] ~]# modprobe -r nfsd nfs lockd

Next, set the domain as all servers and clients should resides within the same domain:

[[email protected] ~]# vim /etc/idmapd.conf
[General]
Domain = yourdomain.com

Update nfs-common to disable statd and rpc.gssd, then require idmapd:

# Ubuntu 12.04 and Ubuntu 14.04
[[email protected] ~]# vim /etc/default/nfs-common
NEED_STATD=no
STATDOPTS=
NEED_GSSD=no

# Debian 7
[[email protected] ~]# vim /etc/default/nfs-common
NEED_STATD=no
STATDOPTS=
NEED_IDMAPD=yes
NEED_GSSD=no

Now start the services, and ensure they will start at boot time:

# Ubuntu 12.04 and Ubuntu 14.04
[[email protected] ~]# service rpcbind start
[[email protected] ~]# service idmapd start

# Debian 7
[[email protected] ~]# service rpcbind start; insserv rpcbind
[[email protected] ~]# service nfs-common start; insserv nfs-common

Confirm the NFS clients can see the NFS server:

[[email protected] ~]# showmount -e 192.168.1.1
Export list for 192.168.1.1:
/var/www/vhosts 192.168.1.0/24

[[email protected] ~]# rpcinfo -p 192.168.1.1
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    662  status
    100024    1   tcp    662  status
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049
    100227    3   tcp   2049
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    2   udp   2049
    100227    3   udp   2049
    100021    1   udp  32769  nlockmgr
    100021    3   udp  32769  nlockmgr
    100021    4   udp  32769  nlockmgr
    100021    1   tcp  32803  nlockmgr
    100021    3   tcp  32803  nlockmgr
    100021    4   tcp  32803  nlockmgr
    100005    1   udp    892  mountd
    100005    1   tcp    892  mountd
    100005    2   udp    892  mountd
    100005    2   tcp    892  mountd
    100005    3   udp    892  mountd
    100005    3   tcp    892  mountd

Configure the mount point in /etc/fstab:

[[email protected] ~]# vim /etc/fstab

192.168.1.1:/data  /data  nfs4  sec=sys,noatime  0  0

Now create the placeholder directory on the client, mount, and verify it works:

[[email protected] ~]# mkdir /data
[[email protected] ~]# mount -a
[[email protected] ~]# df -h
Filesystem          Size  Used Avail Use% Mounted on
/dev/xvda1           20G  1.2G   18G   7% /
none                4.0K     0  4.0K   0% /sys/fs/cgroup
udev                484M  8.0K  484M   1% /dev
tmpfs                99M  404K   99M   1% /run
none                5.0M     0  5.0M   0% /run/lock
none                495M     0  495M   0% /run/shm
none                100M     0  100M   0% /run/user
192.168.13.1:/data   20G  1.2G   18G   7% /data
[[email protected] ~]#
[[email protected] ~]# grep /data /proc/mounts 
192.168.1.1:/data /data nfs rw,noatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.1,mountvers=3,mountport=892,mountproto=tcp,local_lock=none,addr=192.168.1.1 0 0
[[email protected] ~]#
[[email protected] ~]# touch /data/test-file
[[email protected] ~]# ls -al /data/test-file 
-rw-r--r-- 1 root root 0 Dec 20 01:45 /data/test-file

Finally, confirm the user mapping is the same on both servers. You can test to ensure both the server and the client’s show the same UID’s by:

# Create a user on the NFS server:
[[email protected] ~]# useradd -u 6000 testuser

# Create the same user on the NFS client:
[[email protected] ~]# useradd -u 6000 testuser

# Set the ownership of a test file on the NFS server:
[[email protected] ~]# touch /data/test-file01
[[email protected] ~]# chown testuser:testuser /data/test-file01

# Check the ownership of the test file on the NFS server:
[[email protected] ~]# ls -al /data/test-file01

# Confirm the client sees the same ownership:
[[email protected] ~]# ls -al /data/test-file01

Setup NFSv3 on Ubuntu or Debian

NFSv3 is a tried and tested method of allowing client servers to access files over a network, in a very similar fashion to how the files would be accessed on a local file system. As a very mature piece of software, it has been successfully developed and used on production environments for almost 20 years, and it is still widely accepted and supported with a long life ahead of it. Some could make the argument that NFSv4.1 is faster now with pNFS that is now available, but I personally still prefer NFSv3 on many environments.

Setting it up is pretty easy and straight forward. As this is a network file system, it is strongly recommended to setup a private switch or private network between to the servers to ensure the lowest latency, as well as better security.

NFS Server – Installation

Install the required packages on the NFS server:

# Ubuntu and Debian
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install rpcbind nfs-common nfs-kernel-server

NFS Server – Configuration

Out of the box, NFSv3 has the following options set which is getting outdated sorely at this time:
– Sets random ephemeral ports upon daemon startup.
– Enables only 8 NFS threads

To make things more easier for admin’s to lock down the firewalls, we are going to set static ports, and also enable 64 NFS threads since you will most likely run into IO problems before you hit this limit as it was meant for much older systems.

Stop the services so we can unload the lockd kernel module and configure static ports. This step cannot be skipped!

# Ubuntu 12.04 and Ubuntu 14.04
service nfs-kernel-server stop
service statd stop
service idmapd stop
service rpcbind stop
service portmap stop
modprobe -r nfsd nfs lockd

# Debian 7
service nfs-kernel-server stop
service nfs-common stop
service rpcbind stop
modprobe -r nfsd nfs lockd

Configure STATD and define the static ports:

# Ubuntu 12.04 and Ubuntu 14.04
echo "manual" > /etc/init/idmapd.override

vim /etc/default/nfs-common
NEED_STATD=yes
STATDOPTS="-p 662 -o 2020"
NEED_GSSD=no

# Debian 7
vim /etc/default/nfs-common
NEED_STATD=yes
STATDOPTS="-p 662 -o 2020"
NEED_IDMAPD=no
NEED_GSSD=no

Set the static port for LOCKD:

echo "options lockd nlm_udpport=32769 nlm_tcpport=32803" > /etc/modprobe.d/nfs-lockd.conf

Finally, update the NFS thread count by:

vim /etc/default/nfs-kernel-server
...
RPCNFSDCOUNT=64
RPCNFSDPRIORITY=0
RPCMOUNTDOPTS="--manage-gids -p 892"
NEED_SVCGSSD=no
RPCSVCGSSDOPTS=
...

Open the firewall to allow your private network access to the NFS services. You may have to adjust your rules as my private network resides on eth2. Do not allow this on the public interface without adjusting the source IP’s accordingly!

[[email protected] ~]# ufw allow in on eth2 to 192.168.1.0/24 proto tcp
[[email protected] ~]# ufw allow in on eth2 to 192.168.1.0/24 proto udp

Export the directory to be shared, along with its permissions, in /etc/exports:

[[email protected] ~]# vim /etc/exports

/data 192.168.1.0/24(rw,no_root_squash,no_subtree_check)

Now start the services, and ensure they will start at boot time:

# Ubuntu 12.04 and Ubuntu 14.04
service rpcbind start
service statd start
service nfs-kernel-server start; update-rc.d nfs-kernel-server enable

# Debian 7
service rpcbind start; insserv rpcbind
service nfs-common start; insserv nfs-common
service nfs-kernel-server start; insserv nfs-kernel-server

Check to make sure the services are running:

[[email protected] ~]# showmount -e
Export list for nfs01.domain.com:
/data 192.168.1.0/24

[[email protected] ~]# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    662  status
    100024    1   tcp    662  status
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049
    100227    3   tcp   2049
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    2   udp   2049
    100227    3   udp   2049
    100021    1   udp  32769  nlockmgr
    100021    3   udp  32769  nlockmgr
    100021    4   udp  32769  nlockmgr
    100021    1   tcp  32803  nlockmgr
    100021    3   tcp  32803  nlockmgr
    100021    4   tcp  32803  nlockmgr
    100005    1   udp    892  mountd
    100005    1   tcp    892  mountd
    100005    2   udp    892  mountd
    100005    2   tcp    892  mountd
    100005    3   udp    892  mountd
    100005    3   tcp    892  mountd

NFS Client – Installation

Now that the NFS server is ready, the NFS clients now need to be setup to connect. Install the required packages on the NFS clients by:

# Ubuntu or Debian
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install rpcbind nfs-common

Now start the services:

# Ubuntu 12.04 and Ubuntu 14.04
service rpcbind start
service statd start
service idmapd stop
echo "manual" > /etc/init/idmapd.override

# Debian 7
service rpcbind start; insserv rpcbind
service nfs-common start; insserv nfs-common
insserv mountnfs.sh

NFS Client – Configuration

Confirm the NFS clients can see the NFS server:

[[email protected] ~]# showmount -e 192.168.1.1
Export list for 192.168.1.1:
/var/www/vhosts 192.168.1.0/24

[[email protected] ~]# rpcinfo -p 192.168.1.1
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    662  status
    100024    1   tcp    662  status
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049
    100227    3   tcp   2049
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    2   udp   2049
    100227    3   udp   2049
    100021    1   udp  32769  nlockmgr
    100021    3   udp  32769  nlockmgr
    100021    4   udp  32769  nlockmgr
    100021    1   tcp  32803  nlockmgr
    100021    3   tcp  32803  nlockmgr
    100021    4   tcp  32803  nlockmgr
    100005    1   udp    892  mountd
    100005    1   tcp    892  mountd
    100005    2   udp    892  mountd
    100005    2   tcp    892  mountd
    100005    3   udp    892  mountd
    100005    3   tcp    892  mountd

Configure the mount point in /etc/fstab:

[[email protected] ~]# vim /etc/fstab

192.168.1.1:/data  /data  nfs  vers=3,proto=tcp,hard,intr,rsize=32768,wsize=32768,noatime  0  0

Now create the placeholder directory on the client, mount, and verify it works:

[[email protected] ~]# mkdir /data
[[email protected] ~]# mount -a
[[email protected] ~]# df -h
Filesystem          Size  Used Avail Use% Mounted on
/dev/xvda1           20G  1.2G   18G   7% /
none                4.0K     0  4.0K   0% /sys/fs/cgroup
udev                484M  8.0K  484M   1% /dev
tmpfs                99M  404K   99M   1% /run
none                5.0M     0  5.0M   0% /run/lock
none                495M     0  495M   0% /run/shm
none                100M     0  100M   0% /run/user
192.168.13.1:/data   20G  1.2G   18G   7% /data
[[email protected] ~]#
[[email protected] ~]# grep /data /proc/mounts 
192.168.1.1:/data /data nfs rw,noatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.1,mountvers=3,mountport=892,mountproto=tcp,local_lock=none,addr=192.168.1.1 0 0
[[email protected] ~]#
[[email protected] ~]# touch /data/test-file
[[email protected] ~]# ls -al /data/test-file 
-rw-r--r-- 1 root root 0 Dec 20 01:45 /data/test-file

Setup NFSv3 on CentOS

NFSv3 is a tried and tested method of allowing client servers to access files over a network, in a very similar fashion to how the files would be accessed on a local file system. As a very mature piece of software, it has been successfully developed and used on production environments for almost 20 years, and it is still widely accepted and supported with a long life ahead of it. Some could make the argument that NFSv4.1 is faster now with pNFS that is now available, but I personally still prefer NFSv3 on many environments.

Setting it up is pretty easy and straight forward. As this is a network file system, it is strongly recommended to setup a private switch or private network between to the servers to ensure the lowest latency, as well as better security.

NFS Server – Installation

Install the required packages on the NFS server:

# CentOS 5
[[email protected] ~]# yum install portmap nfs-utils -y

# CentOS 6 and CentOS 7
[[email protected] ~]# yum install rpcbind nfs-utils -y

NFS Server – Configuration

Out of the box, NFSv3 has the following options set which is getting outdated sorely at this time:
– Sets random ephemeral ports upon daemon startup.
– Enables only 8 NFS threads

To make things more easier for admin’s to lock down the firewalls, we are going to set static ports, and also enable 64 NFS threads since you will most likely run into IO problems before you hit this limit as it was meant for much older systems.

Uncomment or add the following variables in /etc/sysconfig/nfs

[[email protected] ~]# vim /etc/sysconfig/nfs

# CentOS 5 and CentOS 6
RPCNFSDCOUNT=64
RQUOTAD_PORT=875
LOCKD_TCPPORT=32803
LOCKD_UDPPORT=32769
MOUNTD_PORT=892
STATD_PORT=662
STATD_OUTGOING_PORT=2020

# CentOS 7
RPCRQUOTADOPTS="-p 875"
LOCKD_TCPPORT=32803
LOCKD_UDPPORT=32769
RPCNFSDCOUNT=64
RPCMOUNTDOPTS="-p 892"
STATDARG="-p 662 -o 2020"
GSS_USE_PROXY="no"

Open the firewall to allow your private network access to the NFS services. You may have to adjust your rules as my private network resides on eth2. Do not allow this on the public interface without adjusting the source IP’s accordingly!

# CentOS 5 and CentOS 6
[[email protected] ~]# vim /etc/sysconfig/iptables
-A INPUT -i eth2 -s 192.168.1.0/24 -p tcp -m tcp --dport 111 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p udp -m udp --dport 111 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p tcp -m tcp --dport 662 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p udp -m udp --dport 662 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p tcp -m tcp --dport 892 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p udp -m udp --dport 892 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p tcp -m tcp --dport 2049 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p udp -m udp --dport 2049 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p tcp -m tcp --dport 32803 -j ACCEPT
-A INPUT -i eth2 -s 192.168.1.0/24 -p udp -m udp --dport 32769 -j ACCEPT

# CentOS 7
# Note:  A space was added in the tags so WordPress wouldn't interpret as markup
vim /etc/firewalld/services/nfs.xml
<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>NFS
  <description>NFS service
  <port protocol="tcp" port="111"/>
  <port protocol="udp" port="111"/>
  <port protocol="tcp" port="662"/>
  <port protocol="udp" port="662"/>
  <port protocol="tcp" port="892"/>
  <port protocol="udp" port="892"/>
  <port protocol="tcp" port="2049"/>
  <port protocol="udp" port="2049"/>
  <port protocol="tcp" port="32803"/>
  <port protocol="udp" port="32803"/>
  <port protocol="tcp" port="38467"/>
  <port protocol="udp" port="38467"/>
  <port protocol="tcp" port="32769"/>
  <port protocol="udp" port="32769"/>
</service>

Then apply the rules by:
systemctl reload firewalld.service

Now add a zone to the private network interface and set PEERDNS to no:
vim /etc/sysconfig/network-scripts/ifcfg-eth2
...
PEERDNS=no
ZONE=internal
...

Then apply the changes by:
ifdown eth2 && ifup eth2

Now add the NFS rules to the private network interface:
firewall-cmd --zone=internal --add-interface eth2
firewall-cmd --zone=internal --add-service=nfs
firewall-cmd --zone=internal --add-interface eth2 --permanent
firewall-cmd --zone=internal --add-service=nfs --permanent

Export the directory to be shared, along with its permissions, in /etc/exports:

[[email protected] ~]# vim /etc/exports

/data 192.168.1.0/24(rw,no_root_squash)

Now start the services, and enable them to start at boot time:

# CentOS 5
[[email protected] ~]# service portmap start; chkconfig portmap on
[[email protected] ~]# service nfslock start; chkconfig nfslock on
[[email protected] ~]# service nfs start; chkconfig nfs on

# CentOS 6
[[email protected] ~]# service rpcbind start; chkconfig rpcbind on
[[email protected] ~]# service nfslock start; chkconfig nfslock on
[[email protected] ~]# service nfs start; chkconfig nfs on

# CentOS 7
[[email protected] ~]# systemctl start rpcbind nfs-lock nfs-server
[[email protected] ~]# systemctl enable rpcbind nfs-lock nfs-server

Check to make sure the services are running:

[[email protected] ~]# showmount -e
Export list for nfs01.domain.com:
/data 192.168.1.0/24

[[email protected] ~]# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    662  status
    100024    1   tcp    662  status
    100005    1   udp    892  mountd
    100005    1   tcp    892  mountd
    100005    2   udp    892  mountd
    100005    2   tcp    892  mountd
    100005    3   udp    892  mountd
    100005    3   tcp    892  mountd
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049  nfs_acl
    100227    3   tcp   2049  nfs_acl
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    2   udp   2049  nfs_acl
    100227    3   udp   2049  nfs_acl
    100021    1   udp  32769  nlockmgr
    100021    3   udp  32769  nlockmgr
    100021    4   udp  32769  nlockmgr
    100021    1   tcp  32803  nlockmgr
    100021    3   tcp  32803  nlockmgr
    100021    4   tcp  32803  nlockmgr

NFS Client – Installation

Now that the NFS server is ready, the NFS clients now need to be setup to connect. Install the required packages on the NFS clients by:

# CentOS 5
[[email protected] ~]# yum install portmap nfs-utils -y

# CentOS 6 and CentOS 7
[ro[email protected] ~]# yum install rpcbind nfs-utils -y

Now start the services, and enable them to start at boot time.

# CentOS 5
[[email protected] ~]# service portmap start; chkconfig portmap on
[[email protected] ~]# service nfslock start; chkconfig nfslock on
[[email protected] ~]# chkconfig netfs on

# CentOS 6
[[email protected] ~]# service rpcbind start; chkconfig rpcbind on
[[email protected] ~]# service nfslock start; chkconfig nfslock on
[[email protected] ~]# chkconfig netfs on

# CentOS 7
[[email protected] ~]# systemctl start rpcbind nfs-lock
[[email protected] ~]# systemctl enable rpcbind nfs-lock

NFS Client – Configuration

Confirm the NFS clients can see the NFS server:

[[email protected] ~]# showmount -e 192.168.1.1
Export list for 192.168.1.1:
/var/www/vhosts 192.168.1.0/24

[[email protected] ~]# rpcinfo -p 192.168.1.1
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    662  status
    100024    1   tcp    662  status
    100005    1   udp    892  mountd
    100005    1   tcp    892  mountd
    100005    2   udp    892  mountd
    100005    2   tcp    892  mountd
    100005    3   udp    892  mountd
    100005    3   tcp    892  mountd
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049  nfs_acl
    100227    3   tcp   2049  nfs_acl
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    2   udp   2049  nfs_acl
    100227    3   udp   2049  nfs_acl
    100021    1   udp  32769  nlockmgr
    100021    3   udp  32769  nlockmgr
    100021    4   udp  32769  nlockmgr
    100021    1   tcp  32803  nlockmgr
    100021    3   tcp  32803  nlockmgr
    100021    4   tcp  32803  nlockmgr

Configure the mount point in /etc/fstab:

[[email protected] ~]# vim /etc/fstab

192.168.1.1:/data  /data  nfs  vers=3,proto=tcp,hard,intr,rsize=32768,wsize=32768,noatime  0  0

Now create the placeholder directory on the client, mount, and verify it works:

[[email protected] ~]# mkdir /data
[[email protected] ~]# mount -a
[[email protected] ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       14G  1.8G   11G  15% /
tmpfs                 939M     0  939M   0% /dev/shm
/dev/sda1             477M   74M  378M  17% /boot
192.168.1.1:/data      14G  1.9G   11G  15% /data
[[email protected] ~]#
[[email protected] ~]# grep /data /proc/mounts 
192.168.1.1:/data /data nfs rw,noatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.1,mountvers=3,mountport=892,mountproto=tcp,local_lock=none,addr=192.168.1.1 0 0
[[email protected] ~]#
[[email protected] ~]# touch /data/test-file
[[email protected] ~]# ls -al /data/test-file 
-rw-r--r-- 1 root root 0 Dec 19 17:57 /data/test-file

How to setup DRBD

Distributed Replicated Block Device (DRBD) mirrors block devices between multiple hosts. You can think of this loosely as network Raid 1.

DRBD is meant to run in a Active / Passive setup, meaning, you can only mount the disk on one node at a time. This is not a DRBD limitation, but rather a limitation of the common file systems (ext3, ext4, xfs, etc), since they cannot account for 2 or more servers accessing a single disk.

As with any form of data replication, always ensure you have good backups before you begin, and ensure that you have good backups throughout the life cycle of the setup. There is always a chance of data corruption or complete data loss due to some unforeseen situation, so make sure you have backups, and you have tested restoring from those backups!

Requirements

There are a few requirements that need to be met for DRBD to function properly and securely:

1. 2x servers with similar block devices
2. DRBD kernel module and userspace utilities
3. Private network between the servers
4. iptables port 7788 open between servers on the Private network
5. /etc/hosts configured
6. NTP synchronized

Preparation

For the purposes of this article, my two servers running CentOS 6 will be:

drbd01 192.168.5.2 | Cloud Block Storage 50G SSD
drbd01 192.168.5.3 | Cloud Block Storage 50G SSD

First, ensure that /etc/hosts are setup properly on both servers:

cat /etc/hosts
192.168.5.2 drbd01
192.168.5.3 drbd02

Next, open up iptables on both servers to allow communications across the private network:

cat /etc/sysconfig/iptables
-A INPUT -i eth2 -s 192.168.5.0/24 -p tcp --dport 7788 -m comment --comment "Allow DRBD on private interface" -j ACCEPT
...
service iptables restart

Finally, prep your block devices, but do not format them with a filesystem! For this guide, I am going to assume you are using separate disks for this, which are setup on /dev/xvdb:

fdisk /dev/xvdb
N
P 
1
enter
enter
t (choose 83)
w (write)
fdisk -l /dev/xvdb1 (confirm all looks well)

Install DRBD

CentOS requires the use of the RPM packages found in the repo, http://www.elrepo.org. This will provide the DKMS-based kernel module and userspace tools.

On both nodes:

rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
yum repolist
yum install drbd83-utils kmod-drbd83 dkms ntp ntpdate
service ntpd restart && chkconfig ntpd on
reboot

Configure DRBD

First, configure the global_common.conf

vi /etc/drbd.d/global_common.conf
# Change
usage-count no;
# To 
usage-count yes;

Then search for syncer {, and add rate 10M;. An example is posted below:

syncer {
# rate after al-extents use-rle cpu-mask verify-alg csums-alg
rate 10M;
}

Some important notes:

1. usage-count. The DRBD project keeps statistics about the usage of various DRBD versions. This is done by contacting an HTTP server every time a new DRBD version is installed on a system. This can be disabled by setting usage-count no;. The default is usage-count ask; which will prompt you everytime you upgrade DRBD.

2. rate 10M: This throttles the total bandwidth that DRBD will use to perform its tasks between the 2 nodes. A good rule of thumb for this value is to use about 30% of the available replication bandwidth. Thus, if you had an I/O subsystem capable of sustaining write throughput of 180MB/s, and a Gigabit Ethernet network capable of sustaining 110 MB/s network throughput (the network being the bottleneck), you would calculate: 110 x 0.3 = 33MB/s. I opted to go with 10M for this article. 10M is a bit on the low side, so read the following guide and increased your limits as needed depending on your available bandwidth: https://drbd.linbit.com/users-guide/s-configure-sync-rate.html

Resource Settings

Configure the 2 nodes so they can communicate with each other. On both servers, setup:

vi /etc/drbd.d/cent00.res
resource cent00 {
  protocol C;
  startup { wfc-timeout 0; degr-wfc-timeout 120; }
  disk { on-io-error detach; }
  net { cram-hmac-alg "sha1"; shared-secret "4ftl421dg987d33gR"; }
  on drbd01 {
    device /dev/drbd0;
    disk /dev/xvdb1;
    meta-disk internal;
    address 192.168.5.2:7788;
  }
  on drbd02 {
    device /dev/drbd0;
    disk /dev/xvdb1;
    meta-disk internal;
    address 192.168.5.3:7788;
  }
}

Now initialize the resources DRBD will be using, and set drbd01 to be primary. This is done by:

[[email protected] ~]# drbdadm create-md cent00
[[email protected] ~]# drbdadm create-md cent00
 
[[email protected] ~]# service drbd start; chkconfig drbd on
[[email protected] ~]# service drbd start; chkconfig drbd on
[[email protected] ~]# drbdadm -- --overwrite-data-of-peer primary cent00 

Once this is done, the disks will begin to sync up. This could take several hours. You can check the status by:

[[email protected] ~]# cat /proc/drbd 
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by [email protected], 2013-09-27 16:00:43
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:1124352 nr:0 dw:0 dr:1125016 al:0 bm:68 lo:0 pe:1 ua:0 ap:0 ep:1 wo:f oos:19842524
[>...................] sync'ed: 5.4% (19376/20472)M
finish: 0:31:21 speed: 10,536 (10,312) K/sec

Setup Filesystem

It is recommended to wait until the initial synchronization is complete. It simply depends on the size of the block storage, and the speed of the internal network connecting the 2 servers. You can check the status by running

[[email protected] ~]# cat /proc/drbd

Then before continuing, make sure you are on the primary node first:
** Due to WordPress, I had to put a space in the opening tags to avoid it being processed as markup.

[[email protected] ~]# drbdadm -- status cent00
< drbd-status version="8.3.16" api="88">
< resources config_file="/etc/drbd.conf">
< resource minor="0" name="cent00" cs="Connected" ro1="Primary" ro2="Secondary" ds1="UpToDate" ds2="UpToDate" />
< /resources>
< /drbd-status>

We’ll use the standard ext4 file system for this:

[[email protected] ~]# mkfs.ext4 /dev/drbd0
[[email protected] ~]# mkdir /data
[[email protected] ~]# mount -t ext4 /dev/drbd0 /data

Testing Scenarios

Below are some basic test scenarios you can simulate pretty easily. This goes without saying, but do not experiment with these scenarios on your production environment! Know what they do before you run them in production since they can cause problems if your not ready for it!

These are broken down into the following tests:
Test 1: Promote drbd02 to become primary
Test 2: Testing secondary node failure
Test 3: Testing primary node failure
Test 4: Recovering from split-brain

Test 1: Promote drbd02 to become primary

Unmount the partition and demote the current primary (drbd01) to secondary:

[[email protected] ~]# umount /data
[[email protected] ~]# drbdadm secondary cent00
On other server, drbd02, promote it to primary and mount the drbd device:
[[email protected] ~]# drbdadm primary cent00
[[email protected] ~]# mkdir /data
[[email protected] ~]# mount -t ext4 /dev/drbd0 /data
[[email protected] ~]# ls -d /data/*

At this time, drbd02 will now be the primary, and drdb01 will now be the secondary node.

Test 2: Testing secondary node failure

To see what happens when the secondary server goes offline:
Shutdown your secondary node, which in this case, is drbd02:

[[email protected] ~]# shutdown -h now

Now, back on the primary node drbd01, add a few files to the volume:

[[email protected] ~]# mkdir -p /data/test/
[[email protected] ~]# cp /etc/hosts /data/test/

Power back on the secondary node drbd02, and watch the system sync back up. Note, depending on how much data was written, it may take a bit of time for the volumes to become consistent again. You can check the status with:

[[email protected] ~]# cat /proc/drbd

Test 3: Testing primary node failure

This tests what happens when primary node goes offline, and someone promotes the secondary node before the primary comes online and can be demoted (split-brain).

If you want to simulate this worst case scenario, and you don’t care about your data, then perform the following:

[[email protected] ~]# echo 1 > /proc/sys/kernel/sysrq ; echo b > /proc/sysrq-trigger
[[email protected] ~]# reboot -f -n

Or just shutdown drbd01 (primary), and then log into drbd02 (secondary), and promote it to master:

[[email protected] ~]# drbdadm primary cent00
[[email protected] ~]# mkdir /data
[[email protected] ~]# mount -t ext4 /dev/drbd0 /data

Then boot drbd01 again and enjoy the split-brain scenario! For obvious reasons, do NOT do this on drives containing any data you need for anything! If the primary node loses the replication link, and you made the other node primary BEFORE connectivity is restored, you WILL have split-brain. Avoid that at all costs.

Test 4: Recovering from split-brain

In the event of split-brain, you may be able to correct it by performing the following, but do not do this blindly! Make sure you understand what this is doing before you run it on your production data, otherwise you may lose data you wanted! More information can be found at http://drbd.linbit.com/docs/working/

For reference:
– drbd01 : Primary node
– drbd02 : Secondary node

On secondary node

[[email protected] ~]# drbdadm secondary cent00
[[email protected] ~]# drbdadm -- --discard-my-data connect cent00

And back on the primary node

[[email protected] ~]# drbdadm connect cent00
[[email protected] ~]# cat /proc/drbd