MySQL 5.1 to MySQL 5.7 upgrade on CentOS

Upgrading MySQL is pretty straight forward, however there are some catches. Per MySQL’s official documentation, you must upgrade from 5.1 to 5.5, then upgrade from 5.5 to 5.6, and finally again from 5.6 to 5.7. You cannot upgrade from 5.1 directly to 5.7.

This guide will outline how to upgrade MySQL 5.1 to MySQL 5.7 in sequence and will assume you are using the IUS repositories for MySQL. If the IUS repositories are not already setup, you can install them by:

# CentOS 6
[root@db01 ~]# yum install epel-release
[root@db01 ~]# rpm -ivh https://dl.iuscommunity.org/pub/ius/stable/CentOS/6/x86_64/ius-release-1.0-15.ius.centos6.noarch.rpm
# CentOS 7
[root@db01 ~]# yum install epel-release
[root@db01 ~]# rpm -ivh https://dl.iuscommunity.org/pub/ius/stable/CentOS/7/x86_64/ius-release-1.0-15.ius.centos7.noarch.rpm

Upgrade from MySQL 5.1 to MySQL 5.5

Create a restore point. This will be backing up the databases, the my.cnf, as well as a user and database listing. This can be done by:

[root@db01 ~]# mysql -S mysqldump
[root@db01 ~]# mkdir -p /root/mysqlupgrade/mysql51
[root@db01 ~]# mysql -e 'show databases;' > /root/mysqlupgrade/mysql51/mysql-5.1.databases
[root@db01 ~]# mysql --silent --skip-column-names --execute "select concat('\'',User,'\'@\'',Host,'\'') as User from mysql.user;" | sort | while read u; do echo "-- $u"; mysql --silent --skip-column-names --execute "show grants for $u" | sed 's/$/;/'; done > /root/mysqlupgrade/mysql51/mysql-5.1.grants
[root@db01 ~]# cp /etc/my.cnf /root/mysqlupgrade/mysql51/mysql-5.1.cnf.orig
[root@db01 ~]# yum install xz
[root@db01 ~]# mysqldump --routines --all-databases | xz > /root/mysqlupgrade/mysql51/mysql-5.1.dump.sql.xz

Now stop MySQL and upgrade it to MySQL 5.5:

[root@db01 ~]# service mysqld stop
[root@db01 ~]# yum --disableexcludes=all shell
remove mysql mysql-server mysql-libs
install mysql55 mysql55-server mysql55-libs mysqlclient16
ts solve
ts run
exit

Generate a version of the my.cnf that is valid for MySQL 5.5. Some options get deprecated or changed between versions, so it is important that this is not skipped. To help with this, we’ll be using dbsake as shown below:

[root@db01 ~]# wget -O /root/mysqlupgrade/dbsake http://get.dbsake.net; chmod u+x /root/mysqlupgrade/dbsake
[root@db01 ~]# /root/mysqlupgrade/dbsake upgrade-mycnf --config /root/mysqlupgrade/mysql51/mysql-5.1.cnf.orig --target 5.5 > /root/mysqlupgrade/mysql-5.5.cnf
[root@db01 ~]# mv /etc/my.cnf /etc/mysql-5.5.cnf.orig
[root@db01 ~]# mv -f /root/mysqlupgrade/mysql-5.5.cnf /etc/my.cnf

Start MySQL without the grant tables to verify it is running MySQL 5.5 and all the databases loaded:

[root@db01 ~]# sed -i 's/\[mysqld\]/[mysqld]\nskip-grant-tables\nskip-networking/' /etc/my.cnf
[root@db01 ~]# service mysqld start
[root@db01 ~]# mysql -sse "select @@version"
[root@db01 ~]# mysql -e "show databases;" > /root/mysqlupgrade/mysql51/mysql-5.5.databases
[root@db01 ~]# diff -U0 /root/mysqlupgrade/mysql51/mysql-5.1.databases /root/mysqlupgrade/mysql51/mysql-5.5.databases

Now update the database schema. If mysql_upgrade says that a table needs to repaired, repair it with mysqlcheck -A before moving on:

[root@db01 ~]# mysql_upgrade
[root@db01 ~]# sed -i '/\(skip-grant-tables\|skip-networking\)/d' /etc/my.cnf
[root@db01 ~]# service mysqld restart

Finally, confirm MySQL is running version 5.5:

[root@db01 ~]# mysqladmin version

Upgrade from MySQL 5.5 to MySQL 5.6

Create a restore point. This will be backing up the databases, the my.cnf, as well as a user and database listing. This can be done by:

[root@db01 ~]# mysql -S mysqldump
[root@db01 ~]# mkdir -p /root/mysqlupgrade/mysql55
[root@db01 ~]# mysql -e 'show databases;' > /root/mysqlupgrade/mysql55/mysql-5.5.databases
[root@db01 ~]# mysql --silent --skip-column-names --execute "select concat('\'',User,'\'@\'',Host,'\'') as User from mysql.user;" | sort | while read u; do echo "-- $u"; mysql --silent --skip-column-names --execute "show grants for $u" | sed 's/$/;/'; done > /root/mysqlupgrade/mysql55/mysql-5.5.grants
[root@db01 ~]# cp /etc/my.cnf /root/mysqlupgrade/mysql55/mysql-5.5.cnf.orig
[root@db01 ~]# yum install xz
[root@db01 ~]# mysqldump --routines --all-databases | xz > /root/mysqlupgrade/mysql55/mysql-5.5.dump.sql.xz

Now stop MySQL and upgrade it to MySQL 5.6:

[root@db01 ~]# service mysqld stop
[root@db01 ~]# yum --disableexcludes=all shell
remove mysql55 mysql55-server mysql55-libs
install mysql56u mysql56u-server mysql56u-libs mysqlclient16
ts solve
ts run
exit

Generate a version of the my.cnf that is valid for MySQL 5.6. Some options get deprecated or changed between versions, so it is important that this is not skipped. To help with this, we’ll be using dbsake as shown below:

[root@db01 ~]# wget -O /root/mysqlupgrade/dbsake http://get.dbsake.net; chmod u+x /root/mysqlupgrade/dbsake
[root@db01 ~]# /root/mysqlupgrade/dbsake upgrade-mycnf --config /root/mysqlupgrade/mysql55/mysql-5.5.cnf.orig --target 5.6 > /root/mysqlupgrade/mysql-5.6.cnf
[root@db01 ~]# mv -f /root/mysqlupgrade/mysql-5.6.cnf /etc/my.cnf

Start MySQL without the grant tables to verify it is running MySQL 5.6 and all the databases loaded:

[root@db01 ~]# sed -i 's/\[mysqld\]/[mysqld]\nskip-grant-tables\nskip-networking/' /etc/my.cnf
[root@db01 ~]# service mysqld start
[root@db01 ~]# mysql -sse "select @@version"
[root@db01 ~]# mysql -e "show databases;" > /root/mysqlupgrade/mysql55/mysql-5.6.databases
[root@db01 ~]# diff -U0 /root/mysqlupgrade/mysql55/mysql-5.5.databases /root/mysqlupgrade/mysql55/mysql-5.6.databases

Now update the database schema. If mysql_upgrade says that a table needs to repaired, repair it with mysqlcheck -A before moving on:

[root@db01 ~]# mysql_upgrade
[root@db01 ~]# sed -i '/\(skip-grant-tables\|skip-networking\)/d' /etc/my.cnf
[root@db01 ~]# service mysqld restart

Finally, confirm MySQL is running version 5.6:

[root@db01 ~]# mysqladmin version

Upgrade from MySQL 5.6 to MySQL 5.7

Create a restore point. This will be backing up the databases, the my.cnf, as well as a user and database listing. This can be done by:

[root@db01 ~]# mysql -S mysqldump
[root@db01 ~]# mkdir -p /root/mysqlupgrade/mysql56
[root@db01 ~]# mysql -e 'show databases;' > /root/mysqlupgrade/mysql56/mysql-5.6.databases
[root@db01 ~]# mysql --silent --skip-column-names --execute "select concat('\'',User,'\'@\'',Host,'\'') as User from mysql.user;" | sort | while read u; do echo "-- $u"; mysql --silent --skip-column-names --execute "show grants for $u" | sed 's/$/;/'; done > /root/mysqlupgrade/mysql56/mysql-5.6.grants
[root@db01 ~]# cp /etc/my.cnf /root/mysqlupgrade/mysql56/mysql-5.6.cnf.orig
[root@db01 ~]# yum install xz
[root@db01 ~]# mysqldump --routines --all-databases | xz > /root/mysqlupgrade/mysql56/mysql-5.6.dump.sql.xz

Now stop MySQL and upgrade to MySQL 5.7:

[root@db01 ~]# service mysqld stop
[root@db01 ~]# yum --disableexcludes=all shell
remove mysql56u mysql56u-server mysql56u-libs mysql56u-common
install mysql57u mysql57u-server mysql57u-libs mysqlclient16
ts solve
ts run
exit

Generate a version of the my.cnf that is valid for MySQL 5.7. Some options get deprecated or changed between versions, so it is important that this is not skipped. To help with this, we’ll be using dbsake as shown below:

[root@db01 ~]# wget -O /root/mysqlupgrade/dbsake http://get.dbsake.net; chmod u+x /root/mysqlupgrade/dbsake
[root@db01 ~]# /root/mysqlupgrade/dbsake upgrade-mycnf --config /root/mysqlupgrade/mysql56/mysql-5.6.cnf.orig --target 5.7 > /root/mysqlupgrade/mysql-5.7.cnf
[root@db01 ~]# mv -f /root/mysqlupgrade/mysql-5.7.cnf /etc/my.cnf

Start MySQL without the grant tables to verify it is running MySQL 5.6 and all the databases loaded:

[root@db01 ~]# sed -i 's/\[mysqld\]/[mysqld]\nskip-grant-tables\nskip-networking/' /etc/my.cnf
[root@db01 ~]# service mysqld start
[root@db01 ~]# mysql -sse "select @@version"
[root@db01 ~]# mysql -e "show databases;" > /root/mysqlupgrade/mysql56/mysql-5.7.databases
[root@db01 ~]# diff -U0 /root/mysqlupgrade/mysql56/mysql-5.6.databases /root/mysqlupgrade/mysql56/mysql-5.7.databases

If MySQL fails to start, check the logs as it may be due to MySQL looking for a /var/lib/mysqltmp directory. You can verify and correct it by:

[root@db01 ~]# cat /var/log/mysqld.log |grep ERROR
[ERROR] InnoDB: Unable to create temporary file; errno: 2

[root@db01 ~]# cat /etc/my.cnf |grep tmpdir
tmpdir                          = /var/lib/mysqltmp

[root@db01 ~]# mkdir /var/lib/mysqltmp
[root@db01 ~]# chown mysql:mysql /var/lib/mysqltmp
[root@db01 ~]# service mysqld start

Now update the database schema. If mysql_upgrade says that a table needs to repaired, repair it with mysqlcheck -A before moving on:

[root@db01 ~]# mysql_upgrade
[root@db01 ~]# sed -i '/\(skip-grant-tables\|skip-networking\)/d' /etc/my.cnf
[root@db01 ~]# service mysqld restart

Finally, confirm MySQL is running version 5.7:

[root@db01 ~]# mysqladmin version

Rollback plan

What happens if you need to roll back? If you followed the instructions in this article to create the backups, restoration is simple. Just keep in mind that you also need to restore the databases themselves from the original backups. So anything that changed in the database since the upgrade will be lost. If this is not acceptable, do not use these rollback instructions!

It is going to be assumed that you are going to roll all the way back from MySQL 5.7 to MySQL 5.1. Simply adjust the instructions below accordingly if you are going to roll back to a different version.

Stop MySQL and rollback to MySQL 5.1 by:

[root@db01 ~]# service mysqld stop
[root@db01 ~]# yum --disableexcludes=all shell
remove mysql*
install mysql mysql-server mysql-libs mysqlclient16
ts solve
ts run
exit

Restore the original /etc/my.cnf by:

[root@db01 ~]# cp /root/mysqlupgrade/mysql51/mysql-5.1.cnf.orig /etc/my.cnf

Startup MySQL and restore the 5.1-formatted databases:

[root@db01 ~]# mv /var/lib/mysql /var/lib/mysql.bak
[root@db01 ~]# mkdir /var/lib/mysql
[root@db01 ~]# chown mysql:mysql /var/lib/mysql
[root@db01 ~]# service mysqld start
[root@db01 ~]# mysql_secure_installation
[root@db01 ~]# unxz -c /root/mysqlupgrade/mysql51/mysql-5.1.dump.sql.xz | mysql
[root@db01 ~]# service mysqld restart
[root@db01 ~]# mysqladmin version

Allowing multiple developers to read/write to website via SFTP or FTP

This article simply exists to serve as a visual reference when I’m explaining permissions to others. If you are looking to apply the concepts in this article on a live site, make sure you create a backup of the permissions and ownerships before proceeding as this article could break a pre-existing site!

This is one of those things where there is more than one way to go about it. The goal here is to allow multiple users the ability to work with the site via FTP/SFTP using basic permissions and groups.

First, create the shared group. In this case, as my domain is going to be example.com, I will call it exampleadmins:

[root@web01 ~]# groupadd exampleadmins

Now add the preexisting users to the group

[root@web01 ~]# usermod -aG exampleadmins user01
[root@web01 ~]# usermod -aG exampleadmins user02

Now change the group ownership recursively on the website directory:

[root@web01 ~]# chgrp -R exampleadmins /var/www/vhosts/example.com

Since we want users in the exampleadmins group to have write access, set the group write permissions on the website directory by:

[root@web01 ~]# chmod -R g+w /var/www/vhosts/example.com

To ensure that any new files or directory inherit the group ownership, use the SetGID bit on the directory recursively:

[root@web01 ~]# find /var/www/vhosts/example.com -type d -exec chmod g+s "{}" \;

To ensure that files or directories the user creates or uploads are group writable by default, you need to adjust the default umask for the FTP and SFTP server. For vsftpd which is generally the default FTP server, change the default umask from 022 to 002 by:

[root@web01 ~]# vim /etc/vsftpd.conf
...
local_umask = 002
...
[root@web01 ~]# service vsftpd restart

When using SFTP, update the sftp subsystem within /etc/ssh/sshd_config to set a umask of 0002 by:

[root@web01 ~]# vim /etc/ssh/sshd_config
...
Subsystem       sftp    /usr/libexec/openssh/sftp-server -u 0002
...
# Append to bottom of file:
Match Group exampleadmins
   ForceCommand internal-sftp -u 0002
[root@web01 ~]# service sshd restart

Now whenever you need to add additional users, simply create the user with a membership to exampleadmins

[root@web01 ~]# useradd -s /sbin/nologin -d /var/www/vhosts/example.com -G exampleadmins user03

And if the user already exists, simply run:

[root@web01 ~]# usermod -aG exampleadmins user03

Convert MyISAM tables to InnoDB in MySQL

Converting MyISAM tables over to InnoDB comes with several benefits such as row level locking, being able to commit and roll back transactions and using tractional logs for autorecovery.

While converting MyISAM over to InnoDB is usually straight forward, you must use caution:

- Always create a backup of the database before making any changes!
- Confirm you are running MySQL 5.6.4 or above as that was when full-text indexing was introduced to InnoDB.

You can check to see what tables are using MyISAM in your database by running the following:

[root@db01 ~]# mysql
mysql> SELECT TABLE_SCHEMA,TABLE_NAME,ENGINE,TABLE_ROWS FROM information_schema.TABLES WHERE  ENGINE = 'MyISAM' and TABLE_ROWS
+--------------+-----------------------+--------+------------+
| TABLE_SCHEMA | TABLE_NAME            | ENGINE | TABLE_ROWS |
+--------------+-----------------------+--------+------------+
| mysql        | db                    | MyISAM |          3 |
| mysql        | help_category         | MyISAM |         39 |
| mysql        | help_keyword          | MyISAM |        464 |
| mysql        | help_relation         | MyISAM |       1028 |
| mysql        | help_topic            | MyISAM |        508 |
| mysql        | proxies_priv          | MyISAM |          2 |
| mysql        | user                  | MyISAM |          5 |
| example01    | wp_commentmeta        | MyISAM |         15 |
| example01    | wp_comments           | MyISAM |         72 |
| example01    | wp_links              | MyISAM |         11 |
| example01    | wp_options            | MyISAM |        224 |
| example01    | wp_postmeta           | MyISAM |        260 |
| example01    | wp_posts              | MyISAM |        277 |
| example01    | wp_term_relationships | MyISAM |        178 |
| example01    | wp_term_taxonomy      | MyISAM |         62 |
| example01    | wp_terms              | MyISAM |         62 |
| example01    | wp_usermeta           | MyISAM |         22 |
| example01    | wp_users              | MyISAM |          1 |
+--------------+-----------------------+--------+------------+

Alternatively, if you wanted to see MyISAM tables that have more than 5,000 rows, run:

mysql> SELECT TABLE_SCHEMA,TABLE_NAME,ENGINE,TABLE_ROWS FROM information_schema.TABLES WHERE  ENGINE = 'myISAM' and TABLE_ROWS > 5000;
+--------------+---------------------+--------+------------+
| TABLE_SCHEMA | TABLE_NAME          | ENGINE | TABLE_ROWS |
+--------------+---------------------+--------+------------+
| example02    | employees           | MyISAM |      80257 |
| example02    | templates           | MyISAM |     155341 |
| example02    | zipcodes            | MyISAM |      33178 |
| example02    | providers           | MyISAM |      34833 |
| example02    | portal              | MyISAM |      25184 |
+--------------+---------------------+--------+------------+

Converting the tables is pretty simple, but keep in mind that this will lock the table while it is running. Make sure you backup your database BEFORE converting any tables! Things can go wrong at the worst times, so be sure to have a solid roll back plan before proceeding!

Convert the desired table from MyISAM to InnoDB by running:

[root@db01 ~]# mysql
mysql> use example02;
mysql> ALTER TABLE providers ENGINE=InnoDB;

How to install Elastic Stack

Your logs are trying to talk to you! The problem though is that reading through logs is like trying to pick out one conversation in a crowded and noisy room. Some people talk loud and others speak softly. With all this noise, how can you pick out the critical information? This is where Elastic Stack can help!

Elastic Stack is a group of open source products from Elastic designed to help users take data from any type of source and in any format and search, analyze, and visualize that data in real time. This is commonly referred to as an ELK stack (Elasticsearch, Logstash, and Kibana).

Setting up Elastic Stack can be quite confusing as there are several moving parts. As a very basic primer, logstash is the workhouse that applies various filters to parse the logs better. Logstash will then forward the parsed logs to elasticsearch for indexing. Kibana allows you to visualize the data stored in elasticsearch.

Server Installation

This guide is going to be based on CentOS/RHEL 7. Elasticsearch needs at least 2G of memory. So for the entire stack (Elasticsearch, Logstash and Kibana) to work, the absolute minimum required memory should be around 4G. Anything less than this may cause the services to become unstable or not start up at all.

Elastic Stack relies on Java, so install Java 1.8.0 by:

[root@elk01 ~]# yum install java-1.8.0-openjdk
[root@elk01 ~]# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

Elastic Stack packages all the needed software within their own repos, so to setup their repo by:

[root@elk01 ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[root@elk01 ~]# echo '[elasticstack-6.x]
name=Elastic Stack repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md' > /etc/yum.repos.d/elasticstack.repo

Now install the needed packages for Elastic Stack and set them to start on boot:

[root@elk01 ~]# yum install elasticsearch kibana logstash filebeat
[root@elk01 ~]# systemctl daemon-reload
[root@elk01 ~]# systemctl enable elasticsearch kibana logstash filebeat

Server Configuration

Setup Elasticsearch to listen for connects on the public IP of the server. Mine is also configured to listen on localhost as well since I am monitoring logs locally as well:

[root@elk01 ~]# vim /etc/elasticsearch/elasticsearch.yml
...
network.host: 123.123.123.123, localhost
...

Setup Elasticsearch to be able to use geoip and user-agent by:

[root@elk01 ~]# /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip
[root@elk01 ~]# /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent

Configure logstash with a basic configuration to accept logs from filebeats and forward them to elasticsearch by:

[root@elk01 ~]# echo 'input {
  beats {
    port => 5044
  }
}

# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }

filter {
  if [type] == "apache-access" {
    # This will parse the apache access event
    grok {
      match => [ "message", "%{COMBINEDAPACHELOG}" ]
    }
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" 
    document_type => "%{[@metadata][type]}" 
  }
}' > /etc/logstash/conf.d/logstash.conf

Start and test services by:

[root@elk01 ~]# systemctl start kibana elasticsearch logstash filebeat

Elasticsearch will take about 15 seconds or more to start. To ensure elasticsearch is running, check that the output is similar to the following:

[root@elk01 ~]# curl -XGET 'localhost:9200/?pretty'
{
  "name" : "Cp8oag6",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
  "version" : {
    "number" : "6.0.1",
    "build_hash" : "f27399d",
    "build_date" : "2016-03-30T09:51:41.449Z",
    "build_snapshot" : false,
    "lucene_version" : "7.0.1",
    "minimum_wire_compatibility_version" : "1.2.3",
    "minimum_index_compatibility_version" : "1.2.3"
  },
  "tagline" : "You Know, for Search"
}

Then log into Kibana by navigating your browser to:

http://localhost:5601

If this is installed on a remote server, then you can easily install Nginx to act as a front end for Kibana by:

# Install Nginx
[root@elk01 ~]# yum install nginx httpd-tools

# Setup username/password
[root@elk01 ~]# htpasswd -c /etc/nginx/htpasswd.users kibanaadmin

# Create Nginx vhost
[root@elk01 ~]# echo 'server {
    listen 80;

    server_name kibana.yourdomain.com;

    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.users;

    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;        
    }
}' > /etc/nginx/conf.d/kibana.conf

# Set services to start on boot and start nginx
[root@elk01 ~]# systemctl daemon-reload
[root@elk01 ~]# systemctl enable nginx
[root@elk01 ~]# systemctl start nginx

# Open up the firewall to allow inbound port 80 traffic from anywhere
[root@elk01 ~]# firewall-cmd --zone=public --add-port=80/tcp --permanent
[root@elk01 ~]# firewall-cmd --reload

# Allow nginx to connect to Kibana port 5601 if you’re using SELinux:
[root@elk01 ~]# semanage port -a -t http_port_t -p tcp 5601

# Navigate your browser to your new domain you setup, assuming you already setup DNS for it:
http://kibana.yourdomain.com

Client installation – Filebeat

The question now becomes, how can I get the log messages from other servers into our Elastic Stack server? As my needs are more basic since I am not doing any manipulation of log data, I can make use of filebeat and its associated plugins to get the Apache, Nginx, MySQL, Syslog, etc data I need over to the ElasticSearch server.

Assuming filebeat is not installed, ensure that you have the repos setup for it:

[root@web01 ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[root@web01 ~]# echo '[elasticstack-6.x]
name=Elastic Stack repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md' > /etc/yum.repos.d/elasticstack.repo

Then install filebeat by:

[root@web01 ~]# yum install filebeat
[root@web01 ~]# systemctl daemon-reload
[root@web01 ~]# systemctl enable filebeat

Setup filebeats to send the logs over to your Elastic Stack server:

[root@web01 ~]# vim /etc/filebeat/filebeat.yml
...
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["123.123.123.123:9200"]
...

Now setup the plugins for filebeat. Only setup the ones you need. Be sure to restart filebeat after you have your desired modules enabled.

To send over Apache logs

[root@web01 ~]# filebeat modules enable apache2
[root@web01 ~]# filebeat setup -e
[root@web01 ~]# systemctl restart filebeat

Note, you may need to modify the filebeat apache2 module to pickup your logs. In my case, I had to set the ‘var.paths’ for both the access and error logs by:

[root@web01 ~]# vim /etc/filebeat/modules.d/apache2.yml
- module: apache2
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/httpd/*access.log*"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/httpd/*error.log*"]

[root@web01 ~]# systemctl restart filebeat

To send over syslog data:

[root@web01 ~]# filebeat modules enable system
[root@web01 ~]# filebeat setup -e
[root@web01 ~]# systemctl restart filebeat

To handle MySQL data:

[root@web01 ~]# filebeat modules enable mysql
[root@web01 ~]# filebeat setup -e
[root@web01 ~]# systemctl restart filebeat

To send over auditd logs

[root@web01 ~]# filebeat modules enable auditd
[root@web01 ~]# filebeat setup -e
[root@web01 ~]# systemctl restart filebeat

To send over Nginx logs

[root@web01 ~]# filebeat modules enable nginx
[root@web01 ~]# filebeat setup -e
[root@web01 ~]# systemctl restart filebeat

Enable Docker log shipping to elasticsearch. There is no plugin for this, but its easy enough to configure:
Reference: https://www.elastic.co/blog/enrich-docker-logs-with-filebeat

[root@web01 ~]# vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
...
- type: log
  paths:
   - '/var/lib/docker/containers/*/*.log'
  json.message_key: log
  json.keys_under_root: true
  processors:
  - add_docker_metadata: ~
...

[root@web01 ~]# systemctl restart filebeat

Then browse to the Kibana dashboard to view the available dashboards for Filebeat, or create your own!

Client installation – Metricbeat

What about shipping metrics and statistics over to the Elastic Stack server? This is where Metricbeat comes into play. Metricbeat is a lightweight shipper that you can install on your client nodes that will collect metrics and ships them to Elasticsearch. There are modules for Apache, HAProxy, MySQL, Nginx, PostgreSQL, Redis, System and more. This can be installed on your client servers or on the ELK server itself if you like.

Assuming Metricbeat is not installed, ensure that you have the repos setup for it:

[root@web01 ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[root@web01 ~]# echo '[elasticstack-6.x]
name=Elastic Stack repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md' > /etc/yum.repos.d/elasticstack.repo

Then install Metricbeat by:

[root@web01 ~]# yum install metricbeat
[root@web01 ~]# systemctl daemon-reload
[root@web01 ~]# systemctl enable metricbeat

Setup Metricbeat to send the logs over to your Elastic Stack server:

[root@web01 ~]# vim /etc/metricbeat/metricbeat.yml
...
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["123.123.123.123:9200"]
...

Now setup the plugins for Metricbeat. Only setup the ones you need. Be sure to restart Metricbeat after you have your desired modules enabled.

To see the full listing of available modules and what is currently enabled:

[root@web01 ~]# metricbeat modules list

To send over Apache, MySQL, Nginx and System metrics:

[root@web01 ~]# metricbeat modules enable apache mysql nginx system
[root@web01 ~]# filebeat setup -e

After enabling each one, be sure to check out the modules associated config file as you may need to make changes to it so it will work with your environment. The modules config files can be found in:

[root@web01 ~]# cd /etc/metricbeat/modules.d

Once the configurations are updated accordingly, restart Filebeat by:

[root@web01 ~]# systemctl restart filebeat

Then browse to the Kibana dashboard to view the available dashboards for Metricbeat, or create your own!

Quick Kibana Primer

Now that you have data coming into Elasticsearch, you can use Kibana to generate some quick searches and visualizations. This is not meant to be a full fledge tutorial on how to use Kibana, just a way to jump start the learning process as Kibana can be somewhat complicated if you have never seen it.

To search Kibana looking for failed logins, type the following in the discover search box:

system.auth.ssh.event:Failed OR system.auth.ssh.event:Invalid

To see what packages have been recently installed, type the following in the discover search box:

source: "/var/log/messages" AND system.syslog.message: *Install*

What about visualizations? To see the top 5 countries accessing Apache:

- Click 'Visualizations' over on the left
- Select 'Vertical bar chart'
- Select 'filebeat-*' from the existing index
Click 'X-Axis'
Aggregation:  Terms
Field:  apache2.access.geoip.country_iso_code
Order by:  Metric: Count
Order Descending:  5

To break it down further by city:
- Click 'Add sub-buckets'
- Select 'Split series'
Sub Aggregation:  Terms
Field:  apache2.access.geoip.city_name
Order by:  metric: Count
Order Decending:  5
Click run

View the top 5 remote IP’s hitting Apache:

- Click 'Visualizations' over on the left
- Select 'Vertical bar chart'
- Select 'filebeat-*' from the existing index
- Click 'X-Axis'
Aggregation:  Terms
Field:  apache2.access.remote_ip
Size:  5

Click 'Add sub-buckets'
- Select 'Split series'
Sub Aggregation:  Terms
Field:  apache2.access.remote_ip
Order by:  metric: Count
Order: Descending
Size:  5

View the top 10 requested URL’s in Apache:

- Click 'Visualizations' over on the left
- Select 'Data Table'
- Select 'filebeat-*' from the existing index
- Under Buckets, click 'Split Rows'
Aggregation:  Terms
Field:  apache2.access.url
Order By:  metric: Count
Order:  Descending
Size:  10
Custom Label:  URL

Then click 'Split Rows'
Sub Aggregation:  Terms
Field:  apache2.access.body_sent_bytes
Order By:  metric: Count
Descending:  10
Custom Label:  Size
Click run

Create line chart for apache response codes:

- Click 'Visualizations' over on the left
- Select 'Line chart'
- Select 'filebeat-*' from the existing index
- Click X-Axis
Aggregation:  Date Histogram
Field:  @timestamp
Interval:  Minute

Click 'Split Series'
Sub Aggregation:  Terms
Field:  apache2.access.response_code
Oder by:  metric: Count
Order:  Descending
Size: 5
Click run

See which logs are receiving a lot of activity:

- Click 'Visualizations' over on the left
- Select 'Pie Chart'
- Select 'filebeat-*' from the existing index
- Click 'Split Slices'
Aggregation:  Terms
Field:  source
Order by:  metric: Count
Order: Descending
Size: 5

Show total traffic by domains:

- Click 'Visualizations' over on the left
- Select 'Line Chart'
Aggregation:  Date Histogram
Field:  @timestamp
Interval:  Auto

- Click 'Split Series'
Sub Aggregation:  Filters
Filter 1:  apache2.access.method:* AND source:"/var/log/httpd/domain01.com-access.log"
Filter 2:  apache2.access.method:* AND source:"/var/log/httpd/domain02.com-access.log"

Show GET request counts:

- Click 'Visualizations' over on the left
- Select 'Metrics'
- Select 'filebeat-*' from the existing index
- Click 'Metrics'
Aggregation:  Count

- Click Buckets
- Click 'Split Group'
Aggregation:  Filters
Filter 1 - GET:  apache2.access.method:"GET" AND source:"/var/log/httpd/domain01.com-access.log"

Show POST request counts:

- Click 'Visualizations' over on the left
- Select 'Metrics'
- Select 'filebeat-*' from the existing index
- Click 'Metrics'
Aggregation:  Count

Click Buckets
- Select 'Split Group'
Aggregation:  Filters
Filter 1 - GET:  apache2.access.method:"POST" AND source:"/var/log/httpd/domain01.com-access.log"

Show GET vs POST requests by domain:

- Click 'Visualizations' over on the left
- Select 'Line chart'
- Select 'filebeat-*' from the existing index
- Click X-Axis
Aggregation:  Date Histogram
Field:  @timestamp
Interval:  Auto

Click 'Split Series'
Sub Aggregation:  Filters
Filter 1:  apache2.access.method:"GET" AND source:"/var/log/httpd/domain01.com-access.log"
Filter 2:  apache2.access.method:"POST" AND source:"/var/log/httpd/domain01.com-access.log"

Show total requests on domain:

- Click 'Visualizations' over on the left
- Select 'Line chart'
- Select 'filebeat-*' from the existing index
- Click Y-Axis
Aggregation:  Count

Click 'Add sub-buckets'
- Aggregation:  Data Histogram
- Field:  @timestamp
- Interval:  Auto

Click 'Split Series'
- Sub Aggregation:  Filters
- Filter:  apache2.access.method:* AND source:"/var/log/httpd/domain01.com-access.log"

Display the current Apache error logs:

- Click 'Visualizations' over on the left
- Select 'Data Table'
- Select 'Apache error log [Filebeat Apache2] from Saved Search

View top 10 WordPress posts:

- Click 'Visualizations' over on the left
- Select 'Data Table'
- Select 'filebeat-*' from the existing index
In the search bar above, type:  apache2.access.url: like \/20*

- Under Buckets, click 'Split Rows'
Aggregation:  Terms
Field:  apache2.access.url
Order By:  metric: Count
Order:  Descending
Size:  10
Custom Label:  Posts

Purging all data from Elasticsearch indexes

Whatever the reason for wanting to completely purge the Elasticsearch indexes, its really simple to do as shown below. Just keep in mind you will lose all the data collected in the indexes! This example will clear out the filebeat indexes:

[root@elk01 ~]# curl -XDELETE 'http://localhost:9200/filebeat-*'

Purging MySQL binary logs

Binary logs contain a record of all changes to the database, both in data and structure. It does not keep track of simple SELECT statements. These are required for MySQL replication and can also be useful for performing point in time backups after a nightly database dump has been restored.

If binary logs are enabled within MySQL, but you are not expiring them after X amount of days, they will eventually cause your server to run out of disk space. Typically I see this value set to 5 days.

As a quick note before we begin, if the server is completely out of disk space, you may need to free up some space beforehand so you can work with the system. You can temporarily free up space by changing the filesystems reserved block setting from 5% to 1% by:

[root@db01 ~]# df -h /
/dev/xvda1       79G   76G     0 100% /
[root@web01 ~]# tune2fs -m 2 /dev/xvda1
[root@web01 ~]# df -h /
/dev/xvda1       79G   76G  1.8G  98% /

To determine how many days worth of logs you can purge, first identify if this server is Master MySQL running running MySQL replication to a Slave MySQL server. If it is, log onto the Slave MySQL server, and see which binary log it is currently reading from:

[root@db02 ~]# mysql
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.xx.xx.xx
Master_User: replication
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000244
...

So based off that, we can purge the binary logs up to mysql-bin.000244. So back on the master server, check the MySQL bin log status by:

[root@db01 ~]# mysql
mysql> show binary logs;
+------------------+------------+
| Log_name         | File_size  |
+------------------+------------+
| mysql-bin.000230 |  454230880 |
| mysql-bin.000231 |  511519497 |
| mysql-bin.000232 |  483552032 |
| mysql-bin.000233 |  472847181 |
| mysql-bin.000234 |  443236582 |
| mysql-bin.000235 |  408021824 |
| mysql-bin.000236 |  531519875 |
| mysql-bin.000237 |  468583798 |
| mysql-bin.000238 |  495423661 |
| mysql-bin.000239 |  474475274 |
| mysql-bin.000240 |  689162898 |
| mysql-bin.000241 | 1073749465 |
| mysql-bin.000242 |  939200512 |
| mysql-bin.000243 |  702552334 |
| mysql-bin.000244 |   40656827 |
+------------------+------------+

In my example, as I only want to keep 5 days worth of binary logs, we can purge all the prior bin logs by:

[root@db01 ~]# mysql
mysql> purge binary logs to 'mysql-bin.000240;
mysql> show binary logs;
+------------------+------------+
| Log_name         | File_size  |
+------------------+------------+
| mysql-bin.000240 |  689162898 |
| mysql-bin.000241 | 1073749465 |
| mysql-bin.000242 |  939200512 |
| mysql-bin.000243 |  702552334 |
| mysql-bin.000244 |   40656827 |
+------------------+------------+

As I only want to have a 5 day retention set in general, you can do this live without restarting mysql by:

[root@db01 ~]# mysql
mysql> show variables like '%expire_logs_days%';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| expire_logs_days | 0     |
+------------------+-------+

mysql> SET GLOBAL expire_logs_days = 5;
Query OK, 0 rows affected (0.00 sec)

mysql> show variables like '%expire_logs_days%';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| expire_logs_days | 5     |
+------------------+-------+
1 row in set (0.00 sec)

Then make the setting persistent across MySQL restarts by adding it to the /etc/my.cnf:

[root@web01 ~]# vim /etc/my.cnf
[mysqld]
...
expire_logs_days = 5
...

If you had to modify the filesystems block reservation, change it back to how it was beforehand, which is typically 5%. You can do this by:

[root@db01 ~]# tune2fs -m 5 /dev/xvda1

cURL Cheat Sheet

Curl is the swiss army knife for gathering information while troubleshooting websites. There are many ways to use curl and this guide is simply documenting the ones I often use, but can never seem to remember the specific flags.

General Usage

Test a page behind a username/password prompt:

[user@workstation01 ~]# curl --user name:password http://www.example.com

Download files from github:

[user@workstation01 ~]# curl -O https://raw.github.com/username/reponame/master/filename

Download content via curl to a specific filename:

[user@workstation01 ~]# curl -o archive.tgz https://www.example.com/file.tgz

Run a script from a remote source on server. Understand the security implications of doing this as it can be dangerous!

[user@workstation01 ~]# source <( curl -sL https://www.example.com/your_script.sh)

Have curl make the connection through a specific interface on the server:

[user@workstation01 ~]# curl --interface bond0 icanhazip.com

Website Troubleshooting

Receive output detailing the HTTP response and request headers, following all redirects, and discarding the page body:

[user@workstation01 ~]# curl -Lsvo /dev/null https://www.example.com

Test a domain hosted on the local server, bypassing DNS:

[user@workstation01 ~]# curl -sLH "host: www.example.com" localhost

Test a domain against specific IP, bypassing the need for modifying /etc/hosts:

[user@workstation01 ~]# curl -IL https://www.example.com --resolve www.example.com:443:123.123.123.123

or 

[user@workstation01 ~]# curl -Lsvo /dev/null --resolve 'example.com:443:123.123.123.123' https://www.example.com/

or

[user@workstation01 ~]# curl -Lsvo /dev/null --header "Host: example.com" https://123.123.123.123/

Send request using a specific user-agent. Sometimes a server will have rules in place to block the default curl user-agent string. Or perhaps you need to test using a specific user-agent. You can pass specific user-agent strings by running:

[user@workstation01 ~]# curl --user-agent "USER_AGENT_STRING_HERE" www.example.com

A comprehensive listing of available user-agent strings available resides at:
http://www.useragentstring.com/pages/useragentstring.php

For example, lets say you need to test a mobile device user-agent to see if a custom redirect works:

[user@workstation01 ~]# curl -H "User-Agent: Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19" -IL http://www.example.com/about/contact-us

HTTP/1.1 301 Moved Permanently
Date: Tue, 17 Nov 2015 18:10:09 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips
Location: http://www.example.com/mobi/contact.php
Content-Type: text/html; charset=iso-8859-1

SSL/TLS Testing

Test to see if the site supports specific SSL/TLS protocols:

[user@workstation01 ~]# curl --sslv2 https://www.example.com
[user@workstation01 ~]# curl --sslv3 https://www.example.com
[user@workstation01 ~]# curl --tlsv1 https://www.example.com
[user@workstation01 ~]# curl --tlsv1.0 https://www.example.com
[user@workstation01 ~]# curl --tlsv1.1 https://www.example.com
[user@workstation01 ~]# curl --tlsv1.2 https://www.example.com

Performance Troubleshooting

Load times can be impacted by a number of things, such as the TLS handshake, DNS lookup time, redirects, transfers, upload/downloads, etc. The curl command shown below will break down the times for each accordingly:

[user@workstation01 ~]# curl -Lsvo /dev/null https://www.example.com/ -w "\nContent Type: %{content_type} \
\nHTTP Code: %{http_code} \
\nHTTP Connect:%{http_connect} \
\nNumber Connects: %{num_connects} \
\nNumber Redirects: %{num_redirects} \
\nRedirect URL: %{redirect_url} \
\nSize Download: %{size_download} \
\nSize Upload: %{size_upload} \
\nSSL Verify: %{ssl_verify_result} \
\nTime Handshake: %{time_appconnect} \
\nTime Connect: %{time_connect} \
\nName Lookup Time: %{time_namelookup} \
\nTime Pretransfer: %{time_pretransfer} \
\nTime Redirect: %{time_redirect} \
\nTime Start Transfer (TTFB): %{time_starttransfer} \
\nTime Total: %{time_total} \
\nEffective URL: %{url_effective}\n" 2>&1

The example output is below:

...
HTTP Code: 200 
HTTP Connect:000 
Number Connects: 2 
Number Redirects: 1 
Redirect URL:  
Size Download: 136112 
Size Upload: 0 
SSL Verify: 0 
Time Handshake: 0.689 
Time Connect: 0.556 
Name Lookup Time: 0.528 
Time Pretransfer: 0.689 
Time Redirect: 0.121 
Time Start Transfer (TTFB): 0.738 
Time Total: 0.962 
Effective URL: https://www.example.com/

Another example for quickly seeing performance is below. For simplicity purposes, first create a file with our curl options already in it:

[user@workstation01 ~]# vim site-performance.cfg
\n
      DNS lookup                          :  %{time_namelookup}\n
      Connect to server (TCP)             :  %{time_connect}\n
      Connect to server (HTTP/S)          :  %{time_appconnect}\n
      Time from start until transfer began:  %{time_pretransfer}\n
      Time for redirection (if any)       :  %{time_redirect}\n
      Total time before transfer started  :  %{time_starttransfer}\n
\n
             Total time                   :  %{time_total}\n
             Size of download (bytes)     :  %{size_download}\n
             Average d/l speed (bytes/s)  :  %{speed_download}\n
\n

Then run it:

[user@workstation01 ~]# curl -w "@site-performance.cfg" -o /dev/null -s https://www.example.com
      DNS lookup                          :  0.138664
      Connect to server (TCP)             :  0.171131
      Connect to server (HTTP/S)          :  0.268969
      Time from start until transfer began:  0.269021
      Time for redirection (if any)       :  0.000000
      Total time before transfer started  :  0.532772

             Total time                   :  0.628730
             Size of download (bytes)     :  162510
             Average d/l speed (bytes/s)  :  258473.000

If a web server is running a bunch of sites and has a high load, how can you narrow down which site is likely causing the high load condition on the server? One way would be to see which site takes the longest to load, as that may indicate a resource intensive site. See the example below:

[user@workstation01 ~]# for i in www.example1.com www.example2.com www.example3.com; do echo -n "$i "; (time curl -IL $i -XGET) 2>&1 | grep -E "real|HTTP"; echo; done

www.example1.com HTTP/1.1 200 OK
real	0m0.642s

www.example2.com HTTP/1.1 200 OK
real	0m2.234s

www.example3.com HTTP/1.1 200 OK
real	0m0.421s

So www.example2.com takes 2 seconds to load. What happens to the load times on that domain during increased traffic. The example below will send 25 requests to the domain:

[user@workstation01 ~]# for i in {1..25}; do (time curl -sIL http://www.example2.com -X GET &) 2>&1 | grep -E "real|HTTP" & done

HTTP/1.1 200 OK
real	0m11.297s
HTTP/1.1 200 OK
real	0m11.395s
HTTP/1.1 200 OK
real	0m11.906s
HTTP/1.1 200 OK
real	0m12.079s
...
HTTP/1.1 200 OK
real	0m11.297s

Determining why this is happening will involve investigation outside the scope of this article. Mainly around investigating ways to cache the site or otherwise optimizing it. However at least now we know which site doesn’t perform well under increased requests, and may be causing the high server load.

How to setup Proxmox VE 5 with LXC containers on Rackspace Cloud

Testing out changes in a production environment is never a good idea. However prepping test servers can be tedious as you have to find the hardware and setup the operating system before you can begin. So I want a faster and more cost effective medium, turning a single Cloud Server into a virtualized host server for my test servers. Welcome LXC.

Taken from the providers site, LXC is an operating system-level virtualization technology for Linux. It allows a physical server to run multiple isolated operating system instances, called containers, virtual private servers (VPSs), or virtual environments (VEs.) LXC is similar to Solaris Containers, FreeBSD jails and OpenVZ.

To managed my LXC containers, I prefer to use Proxmox VE 5, which provides a clean control panel for managing my containers.

This guide will document how to install Proxmox on a 4G Rackspace Cloud Server running Debian 9. There will be a 50G SSD Cloud Block Storage volume attached to the server utilizing ZFS that will store the containers, which is outlined more below. The Proxmox installation will install everything needed to run LXC. The IP’s for the containers will be provided via NAT served from the server, therefore creating a self contained test environment.

Configure system for LXC according to best practices

Increase the open files limit by appending the following to the bottom of /etc/security/limits.conf:

[root@proxmox01 ~]# vim /etc/security/limits.conf
...
*       soft    nofile  1048576 unset
*       hard    nofile  1048576 unset
root    soft    nofile  1048576 unset
root    hard    nofile  1048576 unset
*       soft    memlock 1048576 unset
*       hard    memlock 1048576 unset

Now setup some basic kernel tweaking at the bottom of /etc/sysctl.conf:

[root@proxmox01 ~]# vim /etc/sysctl.conf
...
# LXD best practices:  https://github.com/lxc/lxd/blob/master/doc/production-setup.md
fs.inotify.max_queued_events = 1048576
fs.inotify.max_user_instances = 1048576
fs.inotify.max_user_watches = 1048576
vm.max_map_count = 262144

Install Proxmox VE 5

For this to work, we need a vanilla Debian 9 Cloud Server and install Proxmox on top of it, which will install the required kernel.

To get things started, update /etc/hosts to setup your fqdn, and remove any resolvable ipv6 domains:

[root@proxmox01 ~]# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
123.123.123.123 proxmox01.yourdomain.com proxmox01-iad

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Test to confirm /etc/files is setup properly. This should return your servers IP address:

[root@proxmox01 ~]# hostname --ip-address

Add the Proxmox VE repo and add the repo key:

[root@proxmox01 ~]# echo "deb http://download.proxmox.com/debian/pve stretch pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list
[root@proxmox01 ~]# wget http://download.proxmox.com/debian/proxmox-ve-release-5.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpg

Update the package index and then update the system for Proxmox:

[root@proxmox01 ~]# apt update && apt dist-upgrade
* Select option for 'Install the package maintainer's version' when asked about grub

Install Proxmox VE and reboot:

[root@proxmox01 ~]# apt install proxmox-ve postfix open-iscsi
[root@proxmox01 ~]# reboot

Once the cloud server comes back online, confirm you are running the pve kernel:

[root@proxmox01 ~]# uname -a
Linux proxmox 4.13.4-1-pve #1 SMP PVE 4.13.4-25 (Fri, 13 Oct 2017 08:59:53 +0200) x86_64 GNU/Linux

Setup NAT for the containers

As the Rackspace Cloud server comes with 1 IP address, I will be making use of NAT’ed IP addresses to assign to my individual containers. The steps are documented below:

Update /etc/sysctl.conf to allow ip_forwarding:

[root@proxmox01 ~]# vim /etc/sysctl.conf
...
net.ipv4.ip_forward = 1
...

Then apply the new settings without a reboot:

[root@proxmox01 ~]# sysctl -p

To setup the NAT rules, we need to setup a script that will start on boot. Two things need to be taken into consideration here:

1. Change IP address below (123.123.123.123) in the NAT rule to your Cloud server’s public IP address.
2. This assumes you want to use a 192.168.1.0/24 network for your VE’s.

The quick and dirty script is below:

[root@proxmox01 ~]# vim /etc/init.d/lxc-routing
#!/bin/sh
case "$1" in
 start) echo "lxc-routing started"
# It's important that you change the SNAT IP to the one of your server (not the local but the internet IP)
# The following line adds a route to the IP-range that we will later assign to the VPS. That's how you get internet access on # your VPS.
/sbin/iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j SNAT --to 123.123.123.123

# Allow servers to have access to internet:
/sbin/iptables -A FORWARD -s 192.168.1.0/24 -j ACCEPT
/sbin/iptables -A FORWARD -d 192.168.1.0/24 -j ACCEPT
# Be sure to add net.ipv4.ip_forward=1 to /etc/sysctl.conf, then run sysctl -p

# These are the rules for any port forwarding you want to do
# In this example, all traffic to and from the ports 11001-11019 gets routed to/from the VPS with the IP 192.168.1.1.
# Also the port 11000 is routed to the SSH port of the vps, later on you can ssh into your VPS through yourip:11000

#/sbin/iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 11000 -j DNAT --to 192.168.1.1:22
#/sbin/iptables -t nat -A PREROUTING -i eth0 -p udp --dport 11001:11019 -j DNAT --to 192.168.1.1
#/sbin/iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 11001:11019 -j DNAT --to 192.168.1.1

# In my case I also dropped outgoing SMTP traffic, as it's one of the most abused things on servers

#/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 25
#/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 2525
#/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 587
#/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 465
#/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 2526
#/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 110
#/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 143
#/sbin/iptables -A FORWARD -j DROP -p tcp --destination-port 993

;;

*) echo "Usage: /etc/init.d/lxc-routing {start}"
exit 2
;;

esac
exit 0

Setup permissions, set to run on boot, and run it:

[root@proxmox01 ~]# chmod 755 /etc/init.d/lxc-routing
[root@proxmox01 ~]# update-rc.d lxc-routing defaults
[root@proxmox01 ~]# /etc/init.d/lxc-routing start

When you go to start a new container, the container will not start as Proxmox will complain about an error similar to below:

-- Unit [email protected] has begun starting up.
Nov 06 06:07:07 proxmox01.*********** systemd-udevd[11150]: Could not generate persistent MAC address for vethMVIWQY: No such file or directory
Nov 06 06:07:07 proxmox01.*********** kernel: IPv6: ADDRCONF(NETDEV_UP): veth100i0: link is not ready

This can be corrected by:

[root@proxmox01 ~]# vim /etc/systemd/network/99-default.link
[Link]
NamePolicy=kernel database onboard slot path
MACAddressPolicy=none

Then reboot:

[root@proxmox01 ~]# reboot

Navigate your browser to the control panel, login with your root SSH credentials, and setup a Linux Bridge

- Navigate your browser to: https://x.x.x.x:8006
- Click on System --> Network
- On top, click 'Create' --> 'Linux Bridge'
	- Name:  vmbr0
	- IP address:  192.168.1.1
	- Subnet mask: 255.255.255.0
	- Autostart:  checked
	- Leave everything else blank

Setup the 50G SSD Cloud Block Storage Volume with ZFS and add to proxmox. Assuming the device is already mounted, check to see what it got mapped to by:

[root@proxmox01 ~]# lsblk 
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  80G  0 disk 
└─xvda1 202:1    0  80G  0 part /
xvdb    202:16   0  50G  0 disk   <--- This is my new volume

First, install the ZFS utils for Linux, and enable the kernel module:

[root@proxmox01 ~]# apt-get install zfsutils-linux
[root@proxmox01 ~]# /sbin/modprobe zfs

Then add the drive to the zpool:

[root@proxmox01 ~]# zpool create zfs /dev/xvdb 
[root@proxmox01 ~]# zpool list
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zfs   49.8G  97.5K  49.7G         -     0%     0%  1.00x  ONLINE  -

Now add the new disk to proxmox:

- Navigate your browser to: https://x.x.x.x:8006
- Click on Datacenter --> Storage
- On top, click 'Add' --> 'ZFS'
	- Name:  zfs
	- ZFS Pool:  zfs
	- Enable:  Checked
	- Thin provision:  Checked

Add Docker support to the containers

Docker can successfully run within a LXC container with some additional configuration. However, as the proxmox kernel may be older, the latest versions of Docker may fail to work properly. The versions of Docker you receive from the OS repos seem to work though.

First, create the containers as desired for Docker via Proxmox, then add the following to the bottom of containers LXC config file:

[root@proxmox01 ~]# /etc/pve/lxc/100.conf
...
#insert docker part below
lxc.aa_profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:

After restarting that container, you will be able to install and configure Docker as normal on that container.

Add NFS support to the containers

NFS can successfully run within a LXC container with an additional configuration.

First, create an apparmor profile for NFS:

vim /etc/apparmor.d/lxc/lxc-default-with-nfs
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default-with-nfs flags=(attach_disconnected,mediate_deleted) {
  #include 

# allow NFS (nfs/nfs4) mounts.
  mount fstype=nfs*,
}

Then reload the LXC profiles by:

apparmor_parser -r /etc/apparmor.d/lxc-containers

You can explicitly allow NFS in containers by adding another apparmor profile for them
Finally, add the following to the bottom of containers LXC config file:

[root@proxmox01 ~]# /etc/pve/lxc/100.conf
...
#insert near bottom
lxc.apparmor.profile: lxc-container-default-with-nfs

After restarting that container, you will be able to install and configure NFS as normal on that container.

Discrepancy on disk usage between df and du

When df is reporting your filesystem is almost full, but when you start using du to locate the offending directory and nothing lines up, what do you do? The example scenario is below:

[root@web01 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       40G   36G  1.5G  97% /

[root@web01 ~]# du -sh /* |grep G
1.8G    /home
1.8G    /root
1.4G    /usr
12G     /var

So there is almost 19G of space that is unaccounted for! This can happen when a large file is deleted, but the file itself is still held open by a running process. You can see a listing of open files on your server lsof. To find files that are held open by a process, but are marked as deleted, run:

[root@web01 ~]# lsof |grep del
vsftpd     1479    root  txt       REG              202,1      167536      90144 /usr/sbin/vsftpd (deleted)
mysqld     8432   mysql    4u      REG              202,1           0    1155075 /tmp/ib4ecsWH (deleted)
mysqld     8432   mysql    5u      REG              202,1           0    1155076 /tmp/ib8umUE8 (deleted)
mysqld     8432   mysql    6u      REG              202,1           0    1155077 /tmp/ib0CGmnz (deleted)
mysqld     8432   mysql    7u      REG              202,1           0    1155078 /tmp/ibGK9i6Z (deleted)
mysqld     8432   mysql   10w      REG              202,1 19470520347    1114270 /var/lib/mysql/mysql-general.log (deleted)
mysqld     8432   mysql   11u      REG              202,1           0    1155079 /tmp/ib4M9ZPq (deleted)

Notice the file: /var/lib/mysql/mysql-general.log (deleted). When doing the math on the file size, it works out to be about 18G. That would account for the ‘missing’ space that the command du is not seeing almost perfectly.

So in this case, the culprit is the mysql general log, which is typically used for debugging purposes only as it logs every query that MySQL uses. The problem with leaving this log enabled is that it can quickly fill up your disks. So MySQL simply needs to be restarted so the process will release the file, and therefore allow the space to be reclaimed by the file system. Of course, if the general log is no longer being used, it should probably be disabled in the my.cnf.

Nginx Load Balancer

Load balancing applications across multiple servers is a common technique for creating highly available, fault-tolerant solutions. Nginx can serve as a very light weight and efficient HTTP load balancer to distribute traffic to multiple servers.

This guide will outline one basic way Nginx can be deployed as a load balancer.

Install Nginx

First, install Nginx on your server. In many cases, the version of Nginx supplied by the OS repos is outdated, so this guide will be using Nginx’s official repos instead.

# CentOS 6
[root@lb01 ~]# rpm -i http://nginx.org/packages/centos/6/noarch/RPMS/nginx-release-centos-6-0.el6.ngx.noarch.rpm
[root@lb01 ~]# rpm --import http://nginx.org/keys/nginx_signing.key
[root@lb01 ~]# yum install nginx
[root@lb01 ~]# chkconfig nginx on

# CentOS 7
[root@lb01 ~]# rpm -i http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
[root@lb01 ~]# rpm --import http://nginx.org/keys/nginx_signing.key
[root@lb01 ~]# yum install nginx
[root@lb01 ~]# systemctl enable nginx

# Ubuntu 14.04 and 16.04
[root@lb01 ~]# nginx=stable # use nginx=development for latest development version
[root@lb01 ~]# apt-get install python-software-properties
[root@lb01 ~]# add-apt-repository ppa:nginx/$nginx
[root@lb01 ~]# apt-get update
[root@lb01 ~]# apt-get install nginx

Create Load Balancer config

As I have many servers running different domains, I will be setting up Nginx to load balance specific domains to specific servers. So in the example below, www.example.com will be load balanced to my 2 servers, 192.168.1.161 and 192.168.1.162.

The notable things defined in this example include:

- Uses Least Connections for routing traffic to the load balanced web servers
- Servers will automatically be disabled in the load balancer
- The load balancer will pass the x-forwarded-for header (real IP) to the load balanced web servers

Create the config by:

[root@lb01 ~]# vim /etc/nginx/conf.d/example.com.conf
upstream lb-example.com {

	# Load balancing method, only enable one.
	# blank        # Round robin, leave commented out as it is the default
        least_conn;    # Least connections
	# ip_hash;     # Sticky sessions 

	# Backend load balanced servers
        server 192.168.1.161 max_fails=3 fail_timeout=15s;
        server 192.168.1.162 max_fails=3 fail_timeout=15s;

	# Temporarily disable server in load balancer
        # server 192.168.1.163 down;
}

server {
        listen 80;
        server_name www.example.com example.com;
        access_log  /var/log/nginx/example.com.access.log  main;
        # Enforce https
        # return 301 https://$server_name$request_uri;
        # }

        # Pass to load balanced servers
        location / {
        proxy_pass http://lb-example.com;
	
	# Pass x-forwarded-for headers to backend servers
	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Proto $scheme;
        }
}

server {
        listen 443 ssl;
        server_name www.example.com example.com;
        access_log  /var/log/nginx/example.com.access.log  main;

        location / {
        proxy_pass https://lb-example.com;

        # Pass x-forwarded-for headers to backend servers
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        }
}

Now test the new configuration to ensure there are no errors and restart nginx:

[root@lb01 ~]# nginx -t
[root@lb01 ~]# service nginx restart

Ensure that you open the software firewall for 80 and 443 accordingly. Below is an example for CentOS 6:

[root@lb01 ~]# vim /etc/sysconfig/iptables
...
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT
...
[root@lb01 ~]# service iptables restart

Securing a site with Let’s Encrypt SSL certificates

Let’s Encrypt is a free, automated and open certificate authority for anyone that wants to secure a website with SSL. I recently had to setup Let’s Encrypt for a client, and found that it was absurdly simple to use with their Certbot ACME client.

WARNING: This guide may become quickly outdated and is really just for my own reference. If you are looking to use Let’s Encrypt, please review the following articles from Let’s Encrypt for the latest installation and setup instructions:
https://letsencrypt.org/getting-started/
https://certbot.eff.org

For this guide, I am assuming the server is running Apache. So to get started, I simply following the instructions provided on https://certbot.eff.org to get Certbot installed:

# CentOS 6
# There is currently no packaged version of Certbot for CentOS 6.  So you have to download the script manually by:
[root@web01 ~]# cd /root
[root@web01 ~]# wget https://dl.eff.org/certbot-auto
[root@web01 ~]# chmod a+x certbot-auto

# CentOS 7
[root@web01 ~]# yum install yum-utils
[root@web01 ~]# yum-config-manager --enable rhui-REGION-rhel-server-extras rhui-REGION-rhel-server-optional
[root@web01 ~]# yum install certbot-apache

# Ubuntu 14.04
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install software-properties-common
[root@web01 ~]# add-apt-repository ppa:certbot/certbot
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install python-certbot-apache 

# Ubuntu 16.04
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install software-properties-common
[root@web01 ~]# add-apt-repository ppa:certbot/certbot
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install python-certbot-apache

The command below will install or update the certbot script, and also modify your Apache configs accordingly as it automatically configures the SSL certificate. When you run the tool, it will ask you for your email address, review their terms of service, and will ask you to select which URL’s you want to have the SSL certificate generated for. Always be sure to include both the www and non-www domains unless you don’t need one of them for some reason.

[root@web01 ~]# certbot --apache

One of the great things about Let’s Encrypt certificates, asides the fact its free, is that you can add a cron job to automatically renew the SSL certificate so it doesn’t expire. Let’s Encrypt recommends running it twice daily. It won’t do anything until your certificates are due for renewal or revoked. Setup the cron job by running:

# CentOS 6
[root@web01 ~]# crontab -e
0 12/24 * * * /root/certbot-auto renew

# All other OS's:
[root@web01 ~]# crontab -e
0 12/24 * * * certbot renew