Securing a site with Let’s Encrypt SSL certificates

Let’s Encrypt is a free, automated and open certificate authority for anyone that wants to secure a website with SSL. I recently had to setup Let’s Encrypt for a client, and found that it was absurdly simple to use with their Certbot ACME client.

WARNING: This guide may become quickly outdated and is really just for my own reference. If you are looking to use Let’s Encrypt, please review the following articles from Let’s Encrypt for the latest installation and setup instructions:
https://letsencrypt.org/getting-started/
https://certbot.eff.org

For this guide, I am assuming the server is running Apache. So to get started, I simply following the instructions provided on https://certbot.eff.org to get Certbot installed:

# CentOS 6
# There is currently no packaged version of Certbot for CentOS 6.  So you have to download the script manually by:
[[email protected] ~]# cd /root
[[email protected] ~]# wget https://dl.eff.org/certbot-auto
[[email protected] ~]# chmod a+x certbot-auto

# CentOS 7
[[email protected] ~]# yum install yum-utils
[[email protected] ~]# yum-config-manager --enable rhui-REGION-rhel-server-extras rhui-REGION-rhel-server-optional
[[email protected] ~]# yum install certbot-apache

# Ubuntu 14.04
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install software-properties-common
[[email protected] ~]# add-apt-repository ppa:certbot/certbot
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install python-certbot-apache 

# Ubuntu 16.04
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install software-properties-common
[[email protected] ~]# add-apt-repository ppa:certbot/certbot
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install python-certbot-apache

The command below will install or update the certbot script, and also modify your Apache configs accordingly as it automatically configures the SSL certificate. When you run the tool, it will ask you for your email address, review their terms of service, and will ask you to select which URL’s you want to have the SSL certificate generated for. Always be sure to include both the www and non-www domains unless you don’t need one of them for some reason.

[[email protected] ~]# certbot --apache

One of the great things about Let’s Encrypt certificates, asides the fact its free, is that you can add a cron job to automatically renew the SSL certificate so it doesn’t expire. Let’s Encrypt recommends running it twice daily. It won’t do anything until your certificates are due for renewal or revoked. Setup the cron job by running:

# CentOS 6
[[email protected] ~]# crontab -e
0 12/24 * * * /root/certbot-auto renew

# All other OS's:
[[email protected] ~]# crontab -e
0 12/24 * * * certbot renew

Apache quick stats

When troubleshooting performance issues with Apache that happened earlier in the day or week, it is very useful to parse the logs quickly to determine quick facts about the inbound connection rates. It may reveal a period of increased traffic that needs to be investigated further, therefore giving you that thread to begin unraveling the problem.

To get the connections per hour for a specific day, run the following:

[[email protected] ~]# grep "27/Feb" /var/log/httpd/www.example.com-access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":00"}' | sort -n | uniq -c
  50205 03:00
  90516 04:00
  64837 05:00
  47410 06:00
  44876 07:00
  41098 08:00
  38996 09:00
  37234 10:00
  43704 11:00
  58702 12:00
  58922 13:00
  72592 14:00
  77792 15:00
  88882 16:00
  80815 17:00
  98287 18:00
 617857 19:00
  90507 20:00
  98568 21:00
 147584 22:00
 181814 23:00

Based off that output, there was a massive spike in connections during the 9:00PM hour (19:00). So now lets break the 9:00PM hour down to show the connections per minute:

[[email protected] ~]# grep "27/Feb/2017:19" /var/log/httpd/www.example.com-access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c | awk '{ if ($1 > 10) print $0}'
   1629 19:00
   1664 19:01
   1840 19:02
  10493 19:03
  13728 19:04
  17608 19:05
   1377 19:06
   2333 19:07
   1980 19:08
   2056 19:09
   2123 19:10
...
   1997 19:57
   1631 19:58
   1988 19:59

As shown above, there was some sort of traffic spike that occurred between 9:03PM – 9:05PM. As the window has been narrowed down to a 3 minute period, more specific analysis can be performed. The examples below will focus on what was happening around 9:03PM.

To list the top 10 IP’s accessing the site during around 9:03PM

[[email protected] ~]# grep "27/Feb/2017:19:03" /var/log/httpd/www.example.com-access.log | awk '{print $1}' | sort -nr | uniq -c |sort -nr | head

To list the top most called elements on the site:

[[email protected] ~]# grep "27/Feb/2017:19:03" /var/log/httpd/www.example.com-access.log | awk '{print $7}' | sort -nr | uniq -c | sort -nr | head

CVE-2016-2183 SWEET32 Birthday attacks

Lately, vulnerability scanners have been flagging servers that are susceptible to CVE-2016-2183. In a nutshell, you need to disable any TLS ciphers using 3DES. More detailed information about this vulnerability and why it exists can be found at the links below:
https://access.redhat.com/articles/2548661
https://sweet32.info

Mitigating this vulnerability within Apache is pretty straight forward. Below are the steps to confirm if you are actually affected by this vulnerability and how to remediate it.

First, confirm your Apache web server is actually vulnerable to this by seeing if the 3DES ciphers are returned in this nmap test:

[[email protected] ~]# nmap --script ssl-enum-ciphers -p 443 SERVER_IP_ADDRESS_HERE
Starting Nmap 5.51 ( http://nmap.org ) at 2016-11-30 17:57 EST
Nmap scan report for xxxxxxxx (xxx.xxx.xxx.xxx)
Host is up (0.018s latency).
PORT    STATE SERVICE
443/tcp open  https
| ssl-enum-ciphers: 
|   TLSv1.2
|     Ciphers (14)
|       TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
|       TLS_RSA_WITH_3DES_EDE_CBC_SHA
|       TLS_RSA_WITH_AES_128_CBC_SHA
|       TLS_RSA_WITH_AES_128_CBC_SHA256
|       TLS_RSA_WITH_AES_128_GCM_SHA256
|       TLS_RSA_WITH_AES_256_CBC_SHA
|       TLS_RSA_WITH_AES_256_CBC_SHA256
|       TLS_RSA_WITH_AES_256_GCM_SHA384
|     Compressors (1)
|_      uncompressed

As you can see in the output above, this server is affected by this vulnerability as its allowing for the following 3DES ciphers:

TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA

Disabling this in Apache is pretty easy. Simply navigate to where ever you have your SSLCipherSuite configuration defined and disable 3DES. Typically this should be in /etc/httpd/conf.d/ssl.conf, however some may also have this defined in each individual Apache vhost. If you are unsure where its configured, you should be able to locate it on your server by running:

# CentOS / Red Hat
[[email protected] ~]# egrep -R SSLCipherSuite /etc/httpd/*

# Ubuntu / Debian
[[email protected] ~]# egrep -R SSLCipherSuite /etc/apache2/*

Once you locate the config(s) that contain this directive, you simple add !3DES to the end of the SSLCipherSuite line as shown below:

[[email protected] ~]# vim /etc/httpd/conf.d/ssl.conf
...
SSLCipherSuite EECDH+AESGCM:EECDH+AES256:EECDH+AES128:EDH+AES:RSA+AESGCM:RSA+AES:!ECDSA:!NULL:!MD5:!DSS:!3DES
...

Once that is done, restart Apache by:

# CentOS / Red Hat
[[email protected] ~]# service httpd restart

# Ubuntu / Debian
[[email protected] ~]# service apache2 restart

Finally, retest using nmap to confirm no ciphers using 3DES show up:

[[email protected] ~]# nmap --script ssl-enum-ciphers -p 443 SERVER_IP_ADDRESS_HERE
Starting Nmap 5.51 ( http://nmap.org ) at 2016-11-30 18:03 EST
Nmap scan report for xxxxxxxx (xxx.xxx.xxx.xxx)
Host is up (0.017s latency).
PORT    STATE SERVICE
443/tcp open  https
| ssl-enum-ciphers: 
|   TLSv1.2
|     Ciphers (12)
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
|       TLS_RSA_WITH_AES_128_CBC_SHA
|       TLS_RSA_WITH_AES_128_CBC_SHA256
|       TLS_RSA_WITH_AES_128_GCM_SHA256
|       TLS_RSA_WITH_AES_256_CBC_SHA
|       TLS_RSA_WITH_AES_256_CBC_SHA256
|       TLS_RSA_WITH_AES_256_GCM_SHA384
|     Compressors (1)
|_      uncompressed

If no 3DES ciphers are returned like in the listing above, you should be good to rerun your vulnerability scan!

Basic Apache Hardening

Below are a couple of the more common best practices that should be used when hardening Apache. These are simply some basics for mitigating a few of the more common CVE’s that have been cropping up in Apache.

At the very minimum, disable the trace method and prevent information disclosure by updating the ServerToken and ServerSignature variables. This can be done within Apache by modifying the following file:

# CentOS 5 and 6
vim /etc/httpd/conf.d/security.conf

# Ubuntu 12.04
vim /etc/apache2/conf.d/security

Then set it accordingly as shown below:

# Disable access to the entire file system except for the directories that
# are explicitly allowed later.
#
# This currently breaks the configurations that come with some web application
# Debian packages. It will be made the default for the release after lenny.
#
#<Directory />
#       AllowOverride None
#       Order Deny,Allow
#       Deny from all
#</Directory>


# Changing the following options will not really affect the security of the
# server, but might make attacks slightly more difficult in some cases.

#
# ServerTokens
# This directive configures what you return as the Server HTTP response
# Header. The default is 'Full' which sends information about the OS-Type
# and compiled in modules.
# Set to one of:  Full | OS | Minimal | Minor | Major | Prod
# where Full conveys the most information, and Prod the least.
#
ServerTokens Prod 

#
# Optionally add a line containing the server version and virtual host
# name to server-generated pages (internal error documents, FTP directory
# listings, mod_status and mod_info output etc., but not CGI generated
# documents or custom error documents).
# Set to "EMail" to also include a mailto: link to the ServerAdmin.
# Set to one of:  On | Off | EMail
#
ServerSignature Off 

#
# Allow TRACE method
#
# Set to "extended" to also reflect the request body (only for testing and
# diagnostic purposes).
#
# Set to one of:  On | Off | extended
#
TraceEnable Off

Another common area to lock down further from the vendor defaults is the SSL configuration, which is located in:

# CentOS 5 and 6
vim /etc/httpd/conf.d/ssl.conf

# Ubuntu 12.04
vim /etc/apache2/mods-enabled/ssl.conf

The most common ones I see on security reports are:
Set SSLHonorCipherOrder to ‘on’
Restrict the allowed ciphers in SSLCipherSuite
Enable only secure protocols

The ciphers can be a bit tricky, especially if you have a WAF or IDS in front of your solution. There is not a one size fits all here, so please be sure to test your site after making these changes as they can cause you problems if set incorrectly for your solution. I’ll post some scenarios below.

For securing your ssl.conf against many of the current vulnerabilities posted at the time of this writing, disable TLSv1.0 which will be a requirement come June 2018, and enable forward security, you can use:

SSLCipherSuite EECDH+AESGCM:EECDH+AES256:EECDH+AES128:EDH+AES:RSA+AESGCM:RSA+AES:!ECDSA:!NULL:!MD5:!DSS:!3DES
SSLProtocol -ALL +TLSv1.1 +TLSv1.2
SSLHonorCipherOrder On

If you prefer to leave TLSv1.0 enabled for the time being as you still have clients connecting to your site with unsupported browsers from Windows XP that doesn’t support anything above TLSv1.0, then you can try the following:

SSLCipherSuite ALL:!EXP:!NULL:!ADH:!LOW:!SSLv3:!SSLv2:!MD5:!RC4:!DSS:!3DES
SSLProtocol all -SSLv2 -SSLv3
SSLHonorCipherOrder On

If you have an Imperva WAF or Alertlogic IDS in front your solution that needs to decrypt the SSL traffic for analysis, so you therefore can’t use forward security since they need to perform a man-in-the-middle on the traffic, but still want to disable insecure ciphers, then modify the variables in the ssl.conf as follows:

SSLCipherSuite HIGH:!MEDIUM:!AESGCM:!ECDH:!aNULL:!ADH:!DH:!EDH:!CAMELLIA:!GCM:!KRB5:!IDEA:!EXP:!eNULL:!LOW:!RC4:!3DES
SSLProtocol all -SSLv2 -SSLv3
SSLHonorCipherOrder On

As a final note, Mozilla also put out a config generator for this. It can just provide some additional view points of how you can go about the ciphers. The link is here.

WordPress setup on CentOS 6

Setting up WordPress is a pretty common task. However all too often I see people installing WordPress, and setting the ownership to ‘apache:apache’ recursively. While this makes life easier for the administrator, it opens up a host of security issues.

Taken directly from WordPress’s best practice guide on permissions:

Typically, all files should be owned by your user (ftp) account on your web server, and should be writable by that account. On shared hosts, files should never be owned by the web server process itself (sometimes this is www, or apache, or nobody user).

Most people know that using FTP is bad. However if you plan on using the wp-admin portal for media uploads, plugin updates, and core updates, you MUST have an FTP server installed and running. Using the Pecl SSH2 library looks like it would work in theory, but in reality, it doesn’t. Or at least, I haven’t found a way to make it work for the wp-admin portal without giving permission errors for this, that and everything in between since it needs weaker permissions. So while your users can use SSH/SCP to upload content via the command line, if they choose to do most of the WordPress tasks through wp-admin like most people would, use the FTP option from within /wp-admin.

This guide is going to show how you can setup WordPress properly accordingly to the note above from WordPress’s best practices guide on permissions. This guide will assume that you already have a working LAMP stack installed.

FTP Server Setup

First, install an FTP server called vsftpd:

[[email protected] ~]# yum install vsftpd
[[email protected] ~]# chkconfig vsftpd on

Now disable anonymous logins since vsftpd enables this by default for some reason:

[[email protected] ~]# vim /etc/vsftpd/vsftpd.conf
...
anonymous_enable=NO
...
[[email protected] ~]# service vsftpd restart

Then confirm you have a firewall in place that has a default to deny policy. So in the example below, I am only allowing in ports 80 and 443 from the world. Then I have SSH restricted to my IP address. Everything else is blocked, including that FTP server.

[[email protected] ~]# vim /etc/sysconfig/iptables
# Generated by iptables-save v1.4.7 on Fri Nov 13 19:24:15 2015
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2:328]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT 
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT 
-A INPUT -i eth0 -s xx.xx.xx.xx/32 -p tcp -m tcp --dport 22 -m comment --comment "Allow inbound SSH from remote ip" -j ACCEPT
-A INPUT -p icmp -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -j REJECT --reject-with icmp-host-prohibited 
-A FORWARD -j REJECT --reject-with icmp-host-prohibited 
COMMIT
# Completed on Fri Nov 13 19:24:15 2015

Database Setup

Create a database for your new WordPress site by:

[[email protected] ~]# mysql
mysql> create database your_database;

Now grant access for that database to a user:

[[email protected] ~]# mysql
mysql> grant all on your_database.* to 'your_db_user'@'localhost' identified by 'your_secure_db_password';
mysql> flush privileges;
mysql> quit

Apache Setup

First, create a FTP/SCP user:

[[email protected] ~]# mkdir -p /var/www/vhosts/example.com
[[email protected] ~]# chmod 755 /var/www/vhosts/example.com
[[email protected] ~]# useradd -d /var/www/vhosts/example.com example_site_user
[[email protected] ~]# passwd example_site_user

Now setup the Apache vhost:

[[email protected] ~]# vim /etc/httpd/vhost.d/example.com.conf
<VirtualHost *:80>
        ServerName example.com
        ServerAlias www.example.com
        #### This is where you put your files for that domain
        DocumentRoot /var/www/vhosts/example.com

        ### Enable this if you are using a SSL terminated Load Balancer
        SetEnvIf X-Forwarded-Proto https HTTPS=on

	#RewriteEngine On
	#RewriteCond %{HTTP_HOST} ^example.com
	#RewriteRule ^(.*)$ http://www.example.com [R=301,L]

        <Directory /var/www/vhosts/example.com>
                Options -Indexes +FollowSymLinks -MultiViews
                AllowOverride All
		Order deny,allow
		Allow from all
        </Directory>
        CustomLog /var/log/httpd/example.com-access.log combined
        ErrorLog /var/log/httpd/example.com-error.log
        # New Relic PHP override
        <IfModule php5_module>
               php_value newrelic.appname example.com
        </IfModule>
        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn
</VirtualHost>


##
# To install the SSL certificate, please place the certificates in the following files:
# >> SSLCertificateFile    /etc/pki/tls/certs/example.com.crt
# >> SSLCertificateKeyFile    /etc/pki/tls/private/example.com.key
# >> SSLCACertificateFile    /etc/pki/tls/certs/example.com.ca.crt
#
# After these files have been created, and ONLY AFTER, then run this and restart Apache:
#
# To remove these comments and use the virtual host, use the following:
# VI   -  :39,$ s/^#//g
# RedHat Bash -  sed -i '39,$ s/^#//g' /etc/httpd/vhost.d/example.com.conf && service httpd reload
# Debian Bash -  sed -i '39,$ s/^#//g' /etc/apache2/sites-available/example.com && service apache2 reload
##

# <VirtualHost _default_:443>
#        ServerName example.com
#        ServerAlias www.example.com
#        DocumentRoot /var/www/vhosts/example.com
#        <Directory /var/www/vhosts/example.com>
#                Options -Indexes +FollowSymLinks -MultiViews
#                AllowOverride All
#        </Directory>
#
#        CustomLog /var/log/httpd/example.com-ssl-access.log combined
#        ErrorLog /var/log/httpd/example.com-ssl-error.log
#
#        # Possible values include: debug, info, notice, warn, error, crit,
#        # alert, emerg.
#        LogLevel warn
#
#        SSLEngine on
#        SSLCertificateFile    /etc/pki/tls/certs/2016-example.com.crt
#        SSLCertificateKeyFile /etc/pki/tls/private/2016-example.com.key
#        SSLCACertificateFile /etc/pki/tls/certs/2016-example.com.ca.crt
#
#        <IfModule php5_module>
#                php_value newrelic.appname example.com
#        </IfModule>
#        <FilesMatch \"\.(cgi|shtml|phtml|php)$\">
#                SSLOptions +StdEnvVars
#        </FilesMatch>
#
#        BrowserMatch \"MSIE [2-6]\" \
#                nokeepalive ssl-unclean-shutdown \
#                downgrade-1.0 force-response-1.0
#        BrowserMatch \"MSIE [17-9]\" ssl-unclean-shutdown
#</VirtualHost>

Then restart Apache to apply the changes:

[[email protected] ~]# service httpd restart

WordPress Setup

Download a copy of WordPress, uncompress, and move the files into place by:

[[email protected] ~]# cd /var/www/vhosts/example.com
[[email protected] ~]# wget http://wordpress.org/latest.tar.gz && tar -xzf latest.tar.gz
[[email protected] ~]# mv wordpress/* ./ && rmdir ./wordpress && rm -f latest.tar.gz

Update the files and directories ownership to lock it down accordingly:

[[email protected] ~]# chown -R example_site_user:example_site_user /var/www/vhosts/example.com

Then open up a few files so wp-admin can manage the .htaccess, and so it can install plugins, upload media, and use the cache if you choose to configure it:

[[email protected] ~]# mkdir /var/www/vhosts/example.com/wp-content/uploads
[[email protected] ~]# mkdir /var/www/vhosts/example.com/wp-content/cache
[[email protected] ~]# touch /var/www/vhosts/example.com/.htaccess
[[email protected] ~]# chown apache:apache /var/www/vhosts/example.com/wp-content/uploads
[[email protected] ~]# chown apache:apache /var/www/vhosts/example.com/wp-content/cache
[[email protected] ~]# chown apache:apache /var/www/vhosts/example.com/.htaccess

And thats it! Once you have the domain setup in DNS, you should be able to navigate to the domain, and follow the WordPress installation wizard to complete the setup. Afterwards, log into wp-admin, and try to update a plugin, or install a new one. When it prompts you for the FTP information, be sure to use:

Hostname:  localhost
FTP Username:  example_site_user
FTP Username:  example_site_user_pw
Connection Type:  FTP

Apache Proxypass

Many solutions today are built using highly available configurations that can easily scale. Setting up a solution to scale is easy, but getting your web application to work correctly with a multi-server configuration can be difficult as not everyone has access to a quality shared storage solution that is fast and reliable.

In many web applications such as WordPress, you typically want all your wp-admin traffic to go to the master server. There are probably a dozen ways to go about this, many of which get very over complicated with wacky Varnish configurations handling the redirection, or even with Nginx.

These is where ProxyPass can offer a cleaner alternative. ProxyPass allows you to take a request for a specific URL, and forward it to another server, which would be known as your backend server, or your master web server.

This guide will assume that you are performing this on all web servers in the solution, unless otherwise specified. The specific examples are for a WordPress based solution, but it can be easily adapted for other CMS’s.

To get started, first ensure that mod_proxy is installed:

# CentOS 6
[[email protected] ~]# yum install mod_proxy_html
[[email protected] ~]# service httpd restart
[[email protected] ~]# httpd -M |grep proxy
 proxy_module (shared)
 proxy_balancer_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_connect_module (shared)
 proxy_ajp_module (shared)

# Ubuntu 12.04 and 14.04
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install libapache2-mod-proxy-html
[[email protected] ~]# a2enmod proxy proxy_http

There are several ways you can proceed from here. I’ll post them out as ‘options’ below. Each one basically accomplishes the same thing, but one may work better for your environment than another.

So no matter which of the 3 options you go with, always be sure to rigorously test it before implementing it in production!

Option 1: Easy – Define master server based off the URI in each Apache Vhost

This example is simple. In each Apache Vhost, add the following lines on each slave web server to point wp-admin and wp-login.php to your master server, which in this case is 192.168.2.1:

# CentOS 6
[[email protected] ~]# vim /etc/httpd/vhost.d/example.com.conf

# Ubuntu 12.04 and 14.04
[[email protected] ~]# vim /etc/apache2/sites-enabled/example.com.conf
...
ProxyPreserveHost On
        ProxyRequests Off
        ProxyPassMatch ".*/wp-admin.*" "http://192.168.2.1"
        ProxyPassMatch ".*/wp-login.php" "http://192.168.2.1"
...

Option 2: Advanced – Define master server based off URI using location blocks in each Apache Vhost

This example is slightly more advanced. In each Apache Vhost, add the following location blocks to point wp-admin and wp-login.php to your master server, which in this case is 192.168.2.1. We’re also manually defining the host header within these location blocks, which gives you the option to start excluding specific items if needed:

# CentOS 6
[[email protected] ~]# vim /etc/httpd/vhost.d/example.com.conf

# Ubuntu 12.04 and 14.04
[[email protected] ~]# vim /etc/apache2/sites-enabled/example.com.conf
...
ProxyRequests Off
  ProxyPreserveHost Off
  ProxyVia Off
  <Location "/wp-login.php">
    Header set "Host" "www.example.com"
    ProxyPass http://192.168.2.1/wp-login.php
    ProxyPassReverse http://192.168.2.1/wp-login.php
  </Location>
  <Location "/wp-admin">
    Header set "Host" "www.example.com"
    ProxyPass http://192.168.2.1/wp-admin
    ProxyPassReverse http://192.168.2.1/wp-admin
  </Location>

Option 3: Complex – Define master server in global Apache configuration, and only send over POST requests for wp-admin

This example is more complex. You are defining the master server (192.168.2.1) in your global Apache configuration, then configuring each Apache Vhost to only send over POST requests for wp-admin to the master server.

Setup proxypass so it knows which server is the master web server. Be sure to update the IP so its the IP address of your master web server:

# CentOS 6
[[email protected] ~]# vim /etc/sysconfig/httpd
...
OPTIONS="-DSLAVE"
export MASTER_SERVER="192.168.2.1"
...

# Ubuntu 12.04 and 14.04
[[email protected] ~]# /etc/apache2/envvars
...
export APACHE_ARGUMENTS="-DSLAVE"
export MASTER_SERVER="192.168.2.1"
...

Now on your slave web servers, we need to update the site’s vhost configuration to proxy the requests for /wp-admin so they will route to the master web server:

# CentOS 6
[[email protected] ~]# vim /etc/httpd/vhost.d/example.com.conf

# Ubuntu 12.04 and 14.04
[[email protected] ~]# vim /etc/apache2/sites-enabled/example.com.conf
...
<IfDefine SLAVE>
RewriteEngine On
ProxyPreserveHost On
ProxyPass /wp-admin/ http://${MASTER_SERVER}/wp-admin/
     ProxyPassReverse /wp-admin/ http://${MASTER_SERVER}/wp-admin/
RewriteCond %{REQUEST_METHOD} =POST
     RewriteRule . http://${MASTER_SERVER}%{REQUEST_URI} [P]
</IfDefine>
...

# CentOS 6
[[email protected] ~]# service httpd restart

# Ubuntu 12.04 and 14.04
[[email protected] ~]# service apache2 restart

That slave server(s) should now start proxying the /wp-admin requests and sending them over to the master web server. Please be sure to test this out and check your logs to ensure /wp-admin POST requests are now routing to the master web server.

Apache mod_status module

The Apache mod_status module is one that I don’t hear about much anymore, but it is something that can be very useful when troubleshooting high CPU or Memory usage with Apache.

Taken it directly from the Apache documentation, mod_status provide you with details such as:

- The number of worker serving requests.
- The number of idle worker.
- The status of each worker, the number of requests that worker has performed and the total number of bytes served by the worker.
- A total number of accesses and byte count served.
- The time the server was started/restarted and the time it has been running for.
- Averages giving the number of requests per second, the number of bytes served per second and the average number of bytes per request.
- The current percentage CPU used by each worker and in total by all workers combined.
- The current hosts and requests being processed.

Setting it up is simple. It only gets a bit complicated explaining it in a single blog post for multiple operating systems as it gets stored in different places depending on which distro your using.

I’ll outline the configuration location that it needs to be placed in below:

# CentOS 6 / CentOS 7
[[email protected] ~]# vim /etc/httpd/conf.d/status.conf

# Ubuntu 12.04
[[email protected] ~]# vim /etc/apache2/conf.d/status.conf

# Ubuntu 14.04
[[email protected] ~]# vim /etc/apache2/conf-available/status.conf

Using the correct location for your distro as shown above, use the following configuration to enable mod_status. Please be sure to update the AuthUserFile line accordingly for your distro:

<IfModule mod_status.c>
#
# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
ExtendedStatus On

# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Uncomment and change the ".example.com" to allow
# access from other hosts.
#
<Location /server-status>
     SetHandler server-status
     Order deny,allow
     Deny from all
     Allow from localhost ip6-localhost
     <IfModule mod_rewrite.c>
          RewriteEngine off
     </IfModule>
     Allow from 127.0.0.1

# On CentOS / RedHat systems, uncomment the following line
     AuthUserFile /etc/httpd/status-htpasswd

# On Debian / Ubuntu systems, uncomment the following line
#     AuthUserFile /etc/apache2/status-htpasswd

     AuthName "Password protected"
     AuthType Basic
     Require valid-user

     # Allow password-less access for allowed IPs
     Satisfy any
</Location>

</IfModule>

Once you have the configuration in place, you can secure it with a username and password, and then enable it by:

# CentOS 6 / CentOS 7
[[email protected] ~]# htpasswd -c /etc/httpd/status-htpasswd serverinfo
[[email protected] ~]# service httpd restart

# Ubuntu 12.04
[[email protected] ~]# htpasswd -c /etc/apache2/status-htpasswd serverinfo
[[email protected] ~]# service apache2 restart

# Ubuntu 14.04
[[email protected] ~]# htpasswd -c /etc/apache2/status-htpasswd serverinfo
[[email protected] ~]# a2enconf status.conf
[[email protected] ~]# service apache2 restart

Now that mod_status is enabled and working when going to http://serverip/server-status, how can it help with troubleshooting?

Lets say you look at top, and you consistently see an Apache process maxing out a CPU, or using up a ton of memory. You can cross-reference the PID of that Apache child process against the same PID that you find within the server-status page. The requests are constantly changing, so you may need to refresh the /server-status page a couple of times to catch it.

To aid in the troubleshooting as you are trying to match up pids against what is shown in top, you can have the /server-status page refresh automatically by using the following in the URL:

http://serverip/server-status?refresh=2

Once you do locate it, it may give you some idea of what client, or what types of requests, are causing the resource contention issues. Usually it is a specific web application misbehaving, or a specific client is attacking a site.

SSL terminated load balancer causing redirect loops

You have an environment that is terminating SSL on the load balancer for one reason or another. Your application, such as WordPress or Magento, is configured to force SSL. But when you go to test out the site or the admin portal, you get a redirect loop. What happened?

This is a very common issue. In most cases when you are terminating SSL at the load balancer, the load balancer will send the traffic over to your web server using HTTP. This can confuse the web application since it was expecting it to be over HTTPS, and the application will not be able to tell that the client’s browser was in fact using HTTPS, which will result in a redirect loop.

The solution to this is actually very simple. You need to ask your load balancer to send the X-Forwarded-Proto header. This can easily be done by adding a SetEnvIf directive into your .htaccess (assuming Apache here), which will set the header to be what your application was expecting.

To account for this, at the top of your site’s .htaccess file, add the following:

[[email protected] ~]# vim /var/www/vhosts/www.domain.com/.htaccess
...
# Detect the LB header and set the header accordingly for the application
SetEnvIf X-Forwarded-Proto https HTTPS=on
...

So in summary, this will prevent your application from getting confused regarding if the connection originated over HTTP or HTTPS since the load balancer is handling the SSL termination, not the server.

Apache Mod_Rewrite tutorial

Apache Mod_Rewrite provides the ability to rewrite incoming requests to different destinations. It can be a bit complicated to wrap your head around the syntax, but once you get used to it, it becomes very powerful.

Rewrite rules can be setup in 2 places, the Apache VirtualHost configuration, or within the website’s .htaccess file. Modifying the .htaccess file allows you to change the rewrite rules on the fly without restarting Apache. However it comes at a steep cost. When using an .htaccess file, each time someone visits your page, the .htaccess file must be processed. So for large rulesets, this could add a substantial CPU load on your web server.

Taken directly from Apache’s website:

You should avoid using .htaccess files completely if you have access to httpd main server config file. Using .htaccess files slows down your Apache http server. Any directive that you can include in a .htaccess file is better set in a Directory block, as it will have the same effect with better performance.

To avoid the performance penalty, I prefer to keep the rules within the Apache VirtualHost configuration. When Apache is restarted, the rules are read into memory, which allows for faster processing.

Rewrite rule example – Step by step

I prefer to learn by example. So lets take an example that is going to be slightly more complicated then it needs to be on purpose to illustrate the syntax.

All requests to http://www.domain.com should be redirected to http://192.168.1.100.  Concurrently, https should also redirect to https://192.168.1.100.

A possible solution to put into the Apache VirtualHost configuration would be:

RewriteEngine On
RewriteCond %{SERVER_PORT} ^80$
RewriteCond %{HTTP_HOST} ^domain.com$ [NC,OR]
RewriteCond %{HTTP_HOST} ^www.domain.com$ [NC]
RewriteRule ^/(.*) http://192.168.1.100/$1 [R,L]

RewriteCond %{SERVER_PORT} ^443$
RewriteCond %{HTTP_HOST} ^domain.com$ [NC,OR]
RewriteCond %{HTTP_HOST} ^www.domain.com$ [NC]
RewriteRule ^/(.*) https://192.168.1.100/$1 [R,L]

To explain some of the common regex used:

1.  ^ --> String starts with
2.  !^ --> String does not start with
3.  % --> server variable
4.  [NC,OR]  --> This means 'NC' Not case sensitive 'OR' It can either match this or the line below.  If you want it to match both rules, then just use 'NC'  Using an 'AND' statement would break the rule, as the policy already defaults to 'AND'.
5.  [R,L]  This means:  rewrite, last.  So basically it is doing the rewrite and it saying this is the last part of the condition, so end the loop.  
6.  ^/(.*) --> This means anything and everything matching the conditions above (used in this example in the rewriterule)

Easy right? Yea, I know it is not. But it makes more sense the more you have to use it. So let me explain the solution above line by line:

First, we need to enable the rewrite engine:

RewriteEngine On

Now put in the first test condition. For the purposes of this article, we are going to simplify this request by testing for anything coming in over port 80:

RewriteCond %{SERVER_PORT} ^80$

Then we need to test a condition to actually match the URL we are looking to do the rewrite on. In this case, it is domain.com. Now, you should be sure to set the ‘NC’ (non case) so we don’t break the rule if someone types in the request in capital letters. You want to test for both domain.com and www.domain.com. This is done by:

RewriteCond %{HTTP_HOST} ^domain.com$ [NC,OR]
RewriteCond %{HTTP_HOST} ^www.domain.com$ [NC]

Next we write the actual rewrite statement. This is saying, take the result of the above and point it to what we specify here. So to talk this out, it is rewriting anything that matches whats above ^/(.*) and redirecting it to the specified address. The [R,L] rewrite the url ‘R’, and end the statement as it is the last rule ‘L’.

RewriteRule ^/(.*) http://192.168.1.100/$1 [R,L]

Finally, don’t forget to apply all this to https on port 443! You use the same explanations as above for figuring this out.

Common rewrite examples

Enough of the obscene examples and explanations. Below is a bunch of common (and not so common) examples for quick reference:

Rewrite domain.com to www.domain.com:

RewriteEngine On
RewriteCond %{HTTP_HOST} !^www [NC]
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}$1 [R=301,L]

Rewrite www.domain.com to domain.com:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^www.domain.com [NC]
RewriteRule ^(.*)$ http://domain.com$1 [L,R=301]

Force SSL on your domain, meaning take any http requests and redirect them to https:

RewriteEngine On
RewriteCond %{SERVER_PORT} !^443$
RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R=301,L]

Force SSL on your domain, meaning take any http requests and redirect them to https when using an SSL terminated load balancer:

RewriteEngine On
RewriteCond %{HTTP:X-Forwarded-Proto} !https
RewriteRule ^(.*)$ https://www.domain.com$1 [R=301,L]

Rewrite one domain to another:

RewriteEngine on
RewriteCond %{HTTP_HOST} ^(www.)?domain1.com [NC]
RewriteRule ^(.*)$ http://domain2.com$1 [R=301,L]

Redirect a path to a new domain:

Redirect 301 /path http://new.domain.com
Redirect 301 /otherpath/somepage.php http://other.domain.com

Rewrite page with query string, and strip query string:

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-l
RewriteCond %{REQUEST_URI} ^/pages/pages\.php$
RewriteCond %{QUERY_STRING} ^page=[0-9]*$
RewriteRule ^.*$ http://www.domain.com/path/? [R=301,L]

Force all URL’s to be lowercase. Please note, this must be in the Apache VirtualHost config, it will not work in the .htaccess.

RewriteEngine On
RewriteMap lc int:tolower
RewriteCond %{REQUEST_URI} [A-Z]
RewriteRule (.*) ${lc:$1} [R=301,L]

Exclude phpmyadmin from being rewritten by existing rules (Place this above the problem rule):

RewriteCond %{REQUEST_URI} !=/phpmyadmin

Disable TRACE and TRACK methods:

RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]

Rewrite images to Cloud Files. Please note, this must be in the Apache VirtualHost config, it will not work in the .htaccess.
* Note: The RewriteRule and URL belong on the same line.

<Directory /var/www/vhosts/domain.com/content/images>
RewriteEngine On
RewriteRule ^(.*)$ http://c0000.cdn00.cloudfiles.rackspacecloud.com/$1 [R=302,L]
</Directory>

Rewrite all pages to a maintenance page

Options +FollowSymlinks
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_URI} !/maintenance.html$
RewriteRule .* /maintenance.html [L]

RHCSA Study Guide – Objective 8 : Web Services

############################
Everything below are my raw notes that I took while attending an unofficial RHCSA training session.  I am posting them here in hopes they will assist others who may be preparing to take this exam.  

My notes are my own interpretation of the lectures, and are certainly not a replacement to classroom training either through your company, or by taking the official RHCSA classes offered through Red Hat.  If you are new to the Red Hat world, I strongly suggest looking into their training courses over at Red Hat.
############################

Apache

Apache is the default web server on RHEL6. The default configuration file exists at:

/etc/httpd/conf/httpd.conf

In Apache2, the /etc/httpd/conf.d directory stores configuration that are specific to a particular Apache module. All files in this directory ending in .conf will be parsed as a configuration file.

Basic apache vhost default:

< VirtualHost blah.com>
        ServerName blah.com
        ServerAlias www.blah.com
        DocumentRoot /var/www/vhosts/www.blah.com
        CustomLog /var/log/httpd/blah.com.access
        ErrorLog /var/log/httpd/blah.com.error
< VirtualHost>

Apache supports 3 types of virtual hosting:

- IP based hosting : (All sites have different IP's)
- Port based virtual hosting : (Can use the port to tell the server were to go to.  ie. google.com:33333
- Name based virtual hosting : (most popular, as apache looks at the host header and directs the name to the vhost container and match)

Additional docs:

[[email protected] ~]# yum install httpd-manual
[[email protected] ~]# service httpd restart
[[email protected] ~]# firefox localhost:/manual

Lab

1.  Configure two websites on your server.  "X" represents your station #. 

2.  wwwX.example.com should be served from the /var/www/html and should also respond to requests for the short hostname wwwX.

3.  vhostX.example.com should be served from /home/linus/html and should also respond to requests for the short hostname vhostX.

4.  Both should be listening on your primary ip address, but wwwX.exmaple.com should be the default site.

** Too much to post answers here... but its really straight forward.  Just watch selinux, and perms on /home/linus

Securing Apache

2 directives for setting up access controls

-  allow from (host|network|ALL)
-  deny from (host|network|ALL)

These are applied in the given order:

1.  order allow,deny : Allows explicitly allowed clients and denies everyone else.  Anyone matching both deny and allow are denied.

2.  order deny,allow : Denies explicitly denied clients and allows everyone else.  Anyone matching both deny and allow are allowed.

These directives are placed inside one of the following tags:

< Directory>
< File>

In theory, its best to keep these as global variables in the httpd.conf. You have to remember that you are protecting your data, your files and directories, so its best to keep these secured against all vhosts… so you set them globally. In other words, set it OUTSIDE the vhost tag.