Testing out changes in a production environment is never a good idea. However prepping test servers can be tedious as you have to find the hardware and setup the operating system before you can begin. So I want a faster and more cost effective medium, turning a single Cloud Server into a virtualized host server for my test servers. Welcome LXC.
Taken from the providers site, LXC is an operating system-level virtualization technology for Linux. It allows a physical server to run multiple isolated operating system instances, called containers, virtual private servers (VPSs), or virtual environments (VEs.) LXC is similar to Solaris Containers, FreeBSD jails and OpenVZ.
To managed my LXC containers, I prefer to use Proxmox VE 5, which provides a clean control panel for managing my containers.
This guide will document how to install Proxmox on a 4G Rackspace Cloud Server running Debian 9. There will be a 50G SSD Cloud Block Storage volume attached to the server utilizing ZFS that will store the containers, which is outlined more below. The Proxmox installation will install everything needed to run LXC. The IP’s for the containers will be provided via NAT served from the server, therefore creating a self contained test environment.
Configure system for LXC according to best practices
Increase the open files limit by appending the following to the bottom of /etc/security/limits.conf:
[[email protected] ~]# vim /etc/security/limits.conf ... * soft nofile 1048576 unset * hard nofile 1048576 unset root soft nofile 1048576 unset root hard nofile 1048576 unset * soft memlock 1048576 unset * hard memlock 1048576 unset
Now setup some basic kernel tweaking at the bottom of /etc/sysctl.conf:
[[email protected] ~]# vim /etc/sysctl.conf ... # LXD best practices: https://github.com/lxc/lxd/blob/master/doc/production-setup.md fs.inotify.max_queued_events = 1048576 fs.inotify.max_user_instances = 1048576 fs.inotify.max_user_watches = 1048576 vm.max_map_count = 262144
Install Proxmox VE 5
For this to work, we need a vanilla Debian 9 Cloud Server and install Proxmox on top of it, which will install the required kernel.
To get things started, update /etc/hosts to setup your fqdn, and remove any resolvable ipv6 domains:
[[email protected] ~]# cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 123.123.123.123 proxmox01.yourdomain.com proxmox01-iad # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts
Test to confirm /etc/files is setup properly. This should return your servers IP address:
[[email protected] ~]# hostname --ip-address
Add the Proxmox VE repo and add the repo key:
[[email protected] ~]# echo "deb http://download.proxmox.com/debian/pve stretch pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list [[email protected] ~]# wget http://download.proxmox.com/debian/proxmox-ve-release-5.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpg
Update the package index and then update the system for Proxmox:
[[email protected] ~]# apt update && apt dist-upgrade * Select option for 'Install the package maintainer's version' when asked about grub
Install Proxmox VE and reboot:
[[email protected] ~]# apt install proxmox-ve postfix open-iscsi [[email protected] ~]# reboot
Once the cloud server comes back online, confirm you are running the pve kernel:
[[email protected] ~]# uname -a Linux proxmox 4.13.4-1-pve #1 SMP PVE 4.13.4-25 (Fri, 13 Oct 2017 08:59:53 +0200) x86_64 GNU/Linux
Setup NAT for the containers
As the Rackspace Cloud server comes with 1 IP address, I will be making use of NAT’ed IP addresses to assign to my individual containers. The steps are documented below:
Update /etc/sysctl.conf to allow ip_forwarding:
[[email protected] ~]# vim /etc/sysctl.conf ... net.ipv4.ip_forward = 1 ...
Then apply the new settings without a reboot:
[[email protected] ~]# sysctl -p
To setup the NAT rules, we need to setup a script that will start on boot. Two things need to be taken into consideration here:
1. Change IP address below (123.123.123.123) in the NAT rule to your Cloud server’s public IP address. 2. This assumes you want to use a 192.168.1.0/24 network for your VE’s.
The quick and dirty script is below:
[[email protected] ~]# vim /etc/init.d/lxc-routing #!/bin/sh case "$1" in start) echo "lxc-routing started" # It's important that you change the SNAT IP to the one of your server (not the local but the internet IP) # The following line adds a route to the IP-range that we will later assign to the VPS. That's how you get internet access on # your VPS. /sbin/iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j SNAT --to 123.123.123.123 # Allow servers to have access to internet: /sbin/iptables -A FORWARD -s 192.168.1.0/24 -j ACCEPT /sbin/iptables -A FORWARD -d 192.168.1.0/24 -j ACCEPT # Be sure to add net.ipv4.ip_forward=1 to /etc/sysctl.conf, then run sysctl -p # These are the rules for any port forwarding you want to do # In this example, all traffic to and from the ports 11001-11019 gets routed to/from the VPS with the IP 192.168.1.1. # Also the port 11000 is routed to the SSH port of the vps, later on you can ssh into your VPS through yourip:11000 #/sbin/iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 11000 -j DNAT --to 192.168.1.1:22 #/sbin/iptables -t nat -A PREROUTING -i eth0 -p udp --dport 11001:11019 -j DNAT --to 192.168.1.1 #/sbin/iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 11001:11019 -j DNAT --to 192.168.1.1 # In my case I also dropped outgoing SMTP traffic, as it's one of the most abused things on servers #/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 25 #/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 2525 #/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 587 #/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 465 #/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 2526 #/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 110 #/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 143 #/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 993 ;; *) echo "Usage: /etc/init.d/lxc-routing {start}" exit 2 ;; esac exit 0
Setup permissions, set to run on boot, and run it:
[[email protected] ~]# chmod 755 /etc/init.d/lxc-routing [[email protected] ~]# update-rc.d lxc-routing defaults [[email protected] ~]# /etc/init.d/lxc-routing start
When you go to start a new container, the container will not start as Proxmox will complain about an error similar to below:
-- Unit [email protected] has begun starting up. Nov 06 06:07:07 proxmox01.*********** systemd-udevd[11150]: Could not generate persistent MAC address for vethMVIWQY: No such file or directory Nov 06 06:07:07 proxmox01.*********** kernel: IPv6: ADDRCONF(NETDEV_UP): veth100i0: link is not ready
This can be corrected by:
[[email protected] ~]# vim /etc/systemd/network/99-default.link [Link] NamePolicy=kernel database onboard slot path MACAddressPolicy=none
Then reboot:
[[email protected] ~]# reboot
Navigate your browser to the control panel, login with your root SSH credentials, and setup a Linux Bridge
- Navigate your browser to: https://x.x.x.x:8006 - Click on System --> Network - On top, click 'Create' --> 'Linux Bridge' - Name: vmbr0 - IP address: 192.168.1.1 - Subnet mask: 255.255.255.0 - Autostart: checked - Leave everything else blank
Setup the 50G SSD Cloud Block Storage Volume with ZFS and add to proxmox. Assuming the device is already mounted, check to see what it got mapped to by:
[[email protected] ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 80G 0 disk └─xvda1 202:1 0 80G 0 part / xvdb 202:16 0 50G 0 disk <--- This is my new volume
First, install the ZFS utils for Linux, and enable the kernel module:
[[email protected] ~]# apt-get install zfsutils-linux [[email protected] ~]# /sbin/modprobe zfs
Then add the drive to the zpool:
[[email protected] ~]# zpool create zfs /dev/xvdb [[email protected] ~]# zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zfs 49.8G 97.5K 49.7G - 0% 0% 1.00x ONLINE -
Now add the new disk to proxmox:
- Navigate your browser to: https://x.x.x.x:8006 - Click on Datacenter --> Storage - On top, click 'Add' --> 'ZFS' - Name: zfs - ZFS Pool: zfs - Enable: Checked - Thin provision: Checked
Add Docker support to the containers
Docker can successfully run within a LXC container with some additional configuration. However, as the proxmox kernel may be older, the latest versions of Docker may fail to work properly. The versions of Docker you receive from the OS repos seem to work though.
First, create the containers as desired for Docker via Proxmox, then add the following to the bottom of containers LXC config file:
[[email protected] ~]# /etc/pve/lxc/100.conf ... #insert docker part below lxc.aa_profile: unconfined lxc.cgroup.devices.allow: a lxc.cap.drop:
After restarting that container, you will be able to install and configure Docker as normal on that container.
Add NFS support to the containers
NFS can successfully run within a LXC container with an additional configuration.
First, create an apparmor profile for NFS:
vim /etc/apparmor.d/lxc/lxc-default-with-nfs
# Do not load this file. Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc
profile lxc-container-default-with-nfs flags=(attach_disconnected,mediate_deleted) {
#include
# allow NFS (nfs/nfs4) mounts.
mount fstype=nfs*,
}
Then reload the LXC profiles by:
apparmor_parser -r /etc/apparmor.d/lxc-containers
You can explicitly allow NFS in containers by adding another apparmor profile for them
Finally, add the following to the bottom of containers LXC config file:
[[email protected] ~]# /etc/pve/lxc/100.conf ... #insert near bottom lxc.apparmor.profile: lxc-container-default-with-nfs
After restarting that container, you will be able to install and configure NFS as normal on that container.