Month: March 2016

How to install Nginx and PHP-FPM on CentOS

Nginx is a very popular web server in recent years. It is very well known for being used as a high performance web server, a reverse proxy server, or even a load balancer. Some differences Nginx has from Apache:

- No built in PHP support (mod_php).
- Apache is process based while Nginx is event based.
- Modules in Nginx are compiled directly into the binary while Apache has installable modules that can be enabled or disabled quickly
- Nginx configuration files are a series of includes for each sites configuration, whereas Apache has global settings and a companion virtual host with overrides for customization.  So this means there is no override file like Apache's .htaccess in Nginx.
- There are no SSLCertificateChainFile in Nginx.  Intermediate CA certificates need to be appended directly to the site's SSL certificate.

At this time, the current best practice for installing Nginx is to use Nginx’s repos directly as they will contain the most recent and stable releases of Nginx. So install the repo package for Nginx by:

# CentOS 6
[root@web01 ~]# rpm -i http://nginx.org/packages/centos/6/noarch/RPMS/nginx-release-centos-6-0.el6.ngx.noarch.rpm
[root@web01 ~]# rpm --import http://nginx.org/keys/nginx_signing.key
[root@web01 ~]# yum install nginx

# CentOS 7
[root@web01 ~]# rpm -i http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
[root@web01 ~]# rpm --import http://nginx.org/keys/nginx_signing.key
[root@web01 ~]# yum install nginx

As Nginx does not have anything to natively process PHP, install php5-fpm by:

[root@web01 ~]# yum install php-fpm

By default, php-fpm will be listening for connections over TCP. This is incredibly slow when Nginx and php-fpm are on the same server. So modify the php-fpm configuration from listening over TCP to using sockets by:

[root@web01 ~]# vim /etc/php-fpm.d/www.conf
...
[www]
listen = /var/run/php5-fpm.sock
...

Now set php-fpm to run as user nginx:

[root@web01 ~]# vim /etc/php-fpm.d/www.conf
...
user = nginx
group = nginx
listen.owner = nginx
listen.group = nginx
...

Set php-fpm to start on boot, and then startup the service:

# CentOS 6
[root@web01 ~]# chkconfig php-fpm on
[root@web01 ~]# service php-fpm start

# CentOS 7
[root@web01 ~]# systemctl enable php-fpm.service
[root@web01 ~]# systemctl start php-fpm.service

Back in Nginx, configure it to send any PHP requests to the php-fpm socket inside of the http{} block:

[root@web01 ~]# vim /etc/nginx/nginx.conf
http {
...
upstream php5-fpm-sock {
     server unix:/var/run/php5-fpm.sock;
}
...
}

Now that Nginx and php-fpm are configured, setup your first site:

[root@web01 ~]# mkdir -p /var/www/vhosts/example.com
[root@web01 ~]# vim /etc/nginx/conf.d/example.com.conf
server {
     listen 80;
     server_name example.com www.example.com;

     root /var/www/vhosts/example.com;
     index index.php index.html index.htm;

     access_log /var/log/nginx/example.com-access.log;
     error_log /var/log/nginx/example.com-error.log;

     location ~ \.php$ {
          expires off;
          try_files $uri =404;
          include /etc/nginx/fastcgi_params;
          fastcgi_pass php5-fpm-sock;
          fastcgi_index index.php;
          fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
     }

     # caching of files
     location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
          expires max;
          log_not_found off;
     }
}

#server {
#      listen 443 ssl;
#      server_name example.com www.example.com;
#
#      root /var/www/vhosts/example.com;
#      index index.php index.html index.htm;
#
#      access_log /var/log/nginx/example.com-ssl-access.log;
#      error_log /var/log/nginx/example.com-ssl-error.log;
#
#      ssl_certificate /etc/pki/tls/certs/YYYY_example.com.crt;
#      ssl_certificate_key /etc/pki/tls/private/YYYY_example.com.key;
#
#      ssl_session_timeout 5m;
#
#      # Use PCI compliant SSL protocols and ciphers
#      ssl_protocols SSLv3 TLSv1;
#      ssl_ciphers HIGH:!kEDH:!ADH:!EXPORT56;
#      ssl_prefer_server_ciphers on;
#
#      location ~ \.php$ {
#           expires off;
#           try_files $uri =404;
#           include /etc/nginx/fastcgi_params;
#           fastcgi_pass php5-fpm-sock;
#           fastcgi_index index.php;
#           fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
#      }
#
#      # caching of files
#      location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
#           expires max;
#           log_not_found off;
#      }
# }

Finally, start Nginx and enable it to start on boot:

# CentOS 6
[root@web01 ~]# chkconfig nginx on
[root@web01 ~]# service nginx start

# CentOS 7
[root@web01 ~]# systemctl enable nginx.service
[root@web01 ~]# systemctl start nginx.service

NOTE : There is no SSLCertificateChainFile in nginx. CA Certs need to be appended to the certificate for the domain.

Troubleshooting

If you see a page that says ‘The page you are looking for is currently unavailable’, you’ll find Nginx is having trouble getting data from php-fpm (while using a socket). Make sure that the socket you specify in the nginx.conf and the listen line in www.conf pool file for PHP-FPM match.

If you get a ‘File Not Found’ page only on php pages (when using a TCP port for php-fpm) you can check the nginx logs, and you’ll probably see an error indicating it couldn’t connect on the port. Its possible this could be happening if you need to move the root and index definitions out of a location block (if they are defined there). This is because the php section has it’s own location, the documents won’t try to be found using definitions in the domain ‘location /’ block.

How to install Nginx and PHP-FPM on Ubuntu

Nginx is a very popular web server in recent years. It is very well known for being used as a high performance web server, a reverse proxy server, or even a load balancer. Some differences Nginx has from Apache:

- No built in PHP support (mod_php).
- Apache is process based while Nginx is event based.
- Modules in Nginx are compiled directly into the binary while Apache has installable modules that can be enabled or disabled quickly
- Nginx configuration files are a series of includes for each sites configuration, whereas Apache has global settings and a companion virtual host with overrides for customization.  So this means there is no override file like Apache's .htaccess in Nginx.
- There are no SSLCertificateChainFile in Nginx.  Intermediate CA certificates need to be appended directly to the site's SSL certificate.

At this time, the current best practice for installing Nginx is to use Nginx’s repos directly as they will contain the most recent and stable releases of Nginx. So install the repo package for Nginx by:

# Ubuntu 12.04 and 14.04
[root@web01 ~]# nginx=stable # use nginx=development for latest development version
[root@web01 ~]# apt-get install python-software-properties
[root@web01 ~]# add-apt-repository ppa:nginx/$nginx
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install nginx

As Nginx does not have anything to natively process PHP, install php5-fpm by:

# Ubuntu 12.04 and 14.04
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install php5-fpm php5-cli

By default, php-fpm will be listening for connections over TCP. This is incredibly slow when Nginx and php-fpm are on the same server. So modify the php-fpm configuration from listening over TCP to using sockets by:

[root@web01 ~]# vim /etc/php5/fpm/pool.d/www.conf
...
[www]
listen = /var/run/php5-fpm.sock
...

Now set php-fpm to run as user nginx:

[root@web01 ~]# vim /etc/php5/fpm/pool.d/www.conf
...
user = www-data
group = www-data
listen.owner = www-data
listen.group = www-data
...

Back in Nginx, configure it to send any PHP requests to the php-fpm socket inside of the http{} block:

[root@web01 ~]# vim /etc/nginx/nginx.conf
http {
...
upstream php5-fpm-sock {
     server unix:/var/run/php5-fpm.sock;
}
...
}

Now that Nginx and php-fpm are configured, setup your first site:

[root@web01 ~]# mkdir -p /var/www/vhosts/example.com
[root@web01 ~]# vim /etc/nginx/conf.d/example.com.conf
server {
     listen 80;
     server_name example.com www.example.com;

     root /var/www/vhosts/example.com;
     index index.php index.html index.htm;

     access_log /var/log/nginx/example.com-access.log;
     error_log /var/log/nginx/example.com-error.log;

     location ~ \.php$ {
          expires off;
          try_files $uri =404;
          include /etc/nginx/fastcgi_params;
          fastcgi_pass php5-fpm-sock;
          fastcgi_index index.php;
          fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
     }

     # caching of files
     location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
          expires max;
          log_not_found off;
     }
}

#server {
#      listen 443 ssl;
#      server_name example.com www.example.com;
#
#      root /var/www/vhosts/example.com;
#      index index.php index.html index.htm;
#
#      access_log /var/log/nginx/example.com-ssl-access.log;
#      error_log /var/log/nginx/example.com-ssl-error.log;
#
#      ssl_certificate /etc/pki/tls/certs/YYYY_example.com.crt;
#      ssl_certificate_key /etc/pki/tls/private/YYYY_example.com.key;
#
#      ssl_session_timeout 5m;
#
#      # Use PCI compliant SSL protocols and ciphers
#      ssl_protocols SSLv3 TLSv1;
#      ssl_ciphers HIGH:!kEDH:!ADH:!EXPORT56;
#      ssl_prefer_server_ciphers on;
#
#      location ~ \.php$ {
#           expires off;
#           try_files $uri =404;
#           include /etc/nginx/fastcgi_params;
#           fastcgi_pass php5-fpm-sock;
#           fastcgi_index index.php;
#           fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
#      }
#
#      # caching of files
#      location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
#           expires max;
#           log_not_found off;
#      }
# }

Finally, restart Nginx and PHP-FPM to apply the changes

# Ubuntu 12.04 and Ubuntu 14.04
[root@web01 ~]# service php5-fpm restart
[root@web01 ~]# service nginx restart

NOTE : There is no SSLCertificateChainFile in nginx. CA Certs need to be appended to the certificate for the domain.

Troubleshooting

If you see a page that says ‘The page you are looking for is currently unavailable’, you’ll find Nginx is having trouble getting data from php-fpm (while using a socket). Make sure that the socket you specify in the nginx.conf and the listen line in www.conf pool file for PHP-FPM match.

If you get a ‘File Not Found’ page only on php pages (when using a TCP port for php-fpm) you can check the nginx logs, and you’ll probably see an error indicating it couldn’t connect on the port. Its possible this could be happening if you need to move the root and index definitions out of a location block (if they are defined there). This is because the php section has it’s own location, the documents won’t try to be found using definitions in the domain ‘location /’ block.

Rackspace Cloud Monitoring syntax examples

This guide is just a quick reference article displaying the alarm criteria needed for certain checks. Most of these are available in the Rackspace control panel, and some are customized taken from the contrib github repo.

CPU

This check monitors the CPU for high CPU usage. The example below will send a warning message when the usage is at or over 90%, and will send a warning when the CPU usage is at or over 95%

Label: 

:set consecutiveCount=5

if (metric['usage_average'] > 95) {
  return new AlarmStatus(CRITICAL, 'CPU usage is #{usage_average}%');
}

if (metric['usage_average'] > 90) {
  return new AlarmStatus(WARNING, 'CPU usage is #{usage_average}%');
}

return new AlarmStatus(OK, 'CPU usage is #{usage_average}%');

Memory

This check monitors your memory and swap usage. If your system has less than 5% memory available, and less than 10% of swap available, it will throw an alarm.

Label: 

if (metric['swap_total'] > 0 && percentage(metric['swap_used'], metric['swap_total']) > 90
    && percentage(metric['actual_used'], metric['total']) > 95) {
  return new AlarmStatus(CRITICAL, 'Less than 5% of memory and 10% of swap available');
}

if (metric['swap_total'] == 0 && percentage(metric['actual_used'], metric['total']) > 95) {
  return new AlarmStatus(CRITICAL, 'Less than 5% of memory available');
}

return new AlarmStatus(OK, 'More than 5% of memory available');

Load Average

This checks the load average on the server. If the 15 minute load average is greater than 20, it will create an alarm.

Label: High Load Average

if (metric['15m'] > 20) {
  return new AlarmStatus(CRITICAL, '15 Minute Load Average is #{15m}');
}

return new AlarmStatus(OK, '15 Minute Load Average is #{15m}');

Filesystem

This one is broken down into 2 alarms. One for checking for avaiable space, and the other checking to see if the filesystem is in read only mode.

Please keep in mind that each alarm below should be a separate alarm!

Label: Low Filesystem Space

if (percentage(metric['used'], metric['total']) > 90) { 
    return new AlarmStatus(CRITICAL, 'Less than 10% free space available.'); 
    } 
if (percentage(metric['used'], metric['total']) > 80) { 
    return new AlarmStatus(WARNING, 'Less than 20% free space available.'); 
    } 
return new AlarmStatus(OK, 'Greater than 80% free space available.');

Label: Check for read only filesystem

if (metric['options'] regex ".*ro.*") {
return new AlarmStatus(CRITICAL, "Read-Only Filesystem");
}
return new AlarmStatus(OK, 'Filesystem in Read-Write mode.');

MySQL Replication Check

This is a custom agent plugin, so you need to download the plugin on your server first:

mkdir -p /usr/lib/rackspace-monitoring-agent/plugins
cd /usr/lib/rackspace-monitoring-agent/plugins/
wget https://raw.github.com/racker/rackspace-monitoring-agent-plugins-contrib/master/mysql_replication.py
chmod 755 mysql_replication.py

Now register the plugin with Cloud Monitoring:

curl -i -X POST -H 'Host: monitoring.api.rackspacecloud.com' -H 'Accept-Encoding: gzip,deflate' -H 'X-Auth-Token: YOUR_API_TOKEN' -H 'Content-Type: application/json; charset=UTF-8' -H 'Accept: application/json' --data-binary '{"label": "MySQL Replication Check", "type": "agent.plugin", "details": {"args": ["arg1"],"file": "mysql_replication.py"}}'  --compress 'https://monitoring.api.rackspacecloud.com:443/v1.0/YOUR_ACCOUNT_NUMBER/entities/ENTITY_ID/checks'

Finally, apply the alert criteria.

Label: MySQL Replication Check

if (metric['SLAVE_STATUS'] != 'ONLINE') {
  return new AlarmStatus(CRITICAL, 'MySQL Replication is OFFLINE.');
}

if (metric['SLAVE_STATUS'] == 'ONLINE' && metric['SECONDS_BEHIND_MASTER'] >= 120 && metric['SECONDS_BEHIND_MASTER'] < 300) {
  return new AlarmStatus(WARNING, 'MySQL Replication ONLINE but Slave is more than 2 minutes behind Master.');
}

if (metric['SLAVE_STATUS'] == 'ONLINE' && metric['SECONDS_BEHIND_MASTER'] >= 300) {
  return new AlarmStatus(CRITICAL, 'MySQL Replication ONLINE but Slave is more than 5 minutes behind Master.');
}

return new AlarmStatus(OK, 'MySQL Replication is ONLINE');

Holland Check

If you have Holland installed, you can monitor your nightly MySQL dumps to ensure no errors have been returned. It also checks to ensure MySQL is running, and that a valid /root/.my.cnf exists:

This is a custom agent plugin, so you need to download the plugin on your server first:

mkdir -p /usr/lib/rackspace-monitoring-agent/plugins
cd /usr/lib/rackspace-monitoring-agent/plugins/
wget https://raw.github.com/racker/rackspace-monitoring-agent-plugins-contrib/master/holland_mysqldump.py
chmod 755 holland_mysqldump.py

Now register the plugin with Cloud Monitoring:

raxmon-checks-create --entity-id=YOUR_ENTITY  --label=Holland --type=agent.plugin --username=YOUR_USERNAME --api-key=YOUR_API_KEY --details=file=holland_mysqldump.py

Finally, apply the alert criteria. Please keep in mind that each alarm below should be a separate alarm!

Label: Holland Log

if (metric['sql_ping_succeeds'] == 'true' && 
    metric['sql_creds_exist'] == 'true' && 
    metric['sql_status_succeeds'] == 'true' && 
    metric['dump_age'] < 172800 && 
    metric['error_count'] > 0) { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: #{last_error}.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: No errors found in most recent log entries.');

Label: MySQL Authenticates

if (metric['sql_ping_succeeds'] == 'true' && 
    metric['sql_creds_exist'] == 'true' && 
    metric['sql_status_succeeds'] == 'false') { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: MySQL credentials do not authenticate.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: MySQL credentials authenticate.');

Label: MySQL Credentials Exist

if (metric['sql_ping_succeeds'] == 'true' && 
    metric['sql_creds_exist'] == 'false') { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: MySQL credentials file does not exist.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: MySQL credentials file exists.');

Label: MySQL Running

if (metric['sql_ping_succeeds'] == 'false') { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: MySQL is not running.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: MySQL is running');

Label: Recent Backup Exists

if (metric['sql_ping_succeeds'] == 'true' && 
    metric['sql_creds_exist'] == 'true' && 
    metric['sql_status_succeeds'] == 'true' && 
    metric['dump_age'] > 172800) { 
  return new AlarmStatus(CRITICAL, 'holland-plugin: mysqldump file is older than 2d.'); 
} 
return new AlarmStatus(OK, 'holland-plugin: mysqldump file age is less than 2d.');

Process Check

This is a quick and dirty plugin I wrote for when you have to be notified if something is not running on the server, such as Lsyncd or Memcached since both these can’t (Lsyncd) or shouldn’t (Memcached) be listening on the public interface. Any process in the process list can be monitored with this.

This is a custom agent plugin, so you need to download the plugin on your server first:

mkdir -p /usr/lib/rackspace-monitoring-agent/plugins
cd /usr/lib/rackspace-monitoring-agent/plugins
wget https://raw.github.com/racker/rackspace-monitoring-agent-plugins-contrib/master/process_mon.sh
chmod 755 process_mon.sh

Now register the plugin with Cloud Monitoring:

curl -i -X POST -H 'Host: monitoring.api.rackspacecloud.com' -H 'Accept-Encoding: gzip,deflate' -H 'X-Auth-Token: YOUR_API_TOKEN' -H 'Content-Type: application/json; charset=UTF-8' -H 'Accept: application/json' --data-binary '{"label": "Process Check", "type": "agent.plugin", "details": {"args": ["PROCESS_NAME"],"file": "process_mon.sh"}}'  --compress 'https://monitoring.api.rackspacecloud.com:443/v1.0/YOUR_ACCOUNT/entities/YOUR_ENTITY/checks'

Finally, apply the alert criteria in the Rackspace control panel:

if (metric['process_mon'] == 0) {
return new AlarmStatus(CRITICAL, 'Process not running.');
}

return new AlarmStatus(OK, 'Process running normally.');

Nginx stub_status module

Like the Apache mod_status module, Nginx also has a module that can provide you with realtime data regarding its stats. While its not as detailed as Apache mod_status, the data is still useful, especially if you are using Nginx as a load balancer.

First, confirm that Nginx was compiled with the HttpStudStatusModule. If you installed this via yum or apt-get, it should have it:

[root@web01 ~]# nginx -V 2>&1 | grep -o with-http_stub_status_module
with-http_stub_status_module

To enable the module, set the following configuration:

[root@web01 ~]# vim /etc/nginx/conf.d/nginx_status.conf
server {

    listen 80 default_server;
    access_log off;
    server_name _;
    server_name_in_redirect off;
    root  /var/www/html;

        location /nginx_status {
                # Enable Nginx stats
                stub_status on;
                # Disable logging for stats
                access_log   off;
                # Security: Only allow access from authorized IP's
                allow 127.0.0.1;
                allow 192.168.1.100;
                # Deny everyone else
                deny all;
        }

}

Now navigate your browser to:

http://serverip/nginx_status

The results will look something like this:

Active connections: 1 
server accepts handled requests
 1 1 1 
Reading: 0 Writing: 1 Waiting: 0

Here is what these numbers mean, taken directly from Nginx’s documentation:

# Active connections
The current number of active client connections including Waiting connections.

# accepts
The total number of accepted client connections.

# handled
The total number of handled connections. Generally, the parameter value is the same as accepts unless some resource limits have been reached (for example, the worker_connections limit).

# requests
The total number of client requests.

# Reading
The current number of connections where nginx is reading the request header.

# Writing
The current number of connections where nginx is writing the response back to the client.

# Waiting
The current number of idle client connections waiting for a request.

Apache mod_status module

The Apache mod_status module is one that I don’t hear about much anymore, but it is something that can be very useful when troubleshooting high CPU or Memory usage with Apache.

Taken it directly from the Apache documentation, mod_status provide you with details such as:

- The number of worker serving requests.
- The number of idle worker.
- The status of each worker, the number of requests that worker has performed and the total number of bytes served by the worker.
- A total number of accesses and byte count served.
- The time the server was started/restarted and the time it has been running for.
- Averages giving the number of requests per second, the number of bytes served per second and the average number of bytes per request.
- The current percentage CPU used by each worker and in total by all workers combined.
- The current hosts and requests being processed.

Setting it up is simple. It only gets a bit complicated explaining it in a single blog post for multiple operating systems as it gets stored in different places depending on which distro your using.

I’ll outline the configuration location that it needs to be placed in below:

# CentOS 6 / CentOS 7
[root@web01 ~]# vim /etc/httpd/conf.d/status.conf

# Ubuntu 12.04
[root@web01 ~]# vim /etc/apache2/conf.d/status.conf

# Ubuntu 14.04
[root@web01 ~]# vim /etc/apache2/conf-available/status.conf

Using the correct location for your distro as shown above, use the following configuration to enable mod_status. Please be sure to update the AuthUserFile line accordingly for your distro:

<IfModule mod_status.c>
#
# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
ExtendedStatus On

# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Uncomment and change the ".example.com" to allow
# access from other hosts.
#
<Location /server-status>
     SetHandler server-status
     Order deny,allow
     Deny from all
     Allow from localhost ip6-localhost
     <IfModule mod_rewrite.c>
          RewriteEngine off
     </IfModule>
     Allow from 127.0.0.1

# On CentOS / RedHat systems, uncomment the following line
     AuthUserFile /etc/httpd/status-htpasswd

# On Debian / Ubuntu systems, uncomment the following line
#     AuthUserFile /etc/apache2/status-htpasswd

     AuthName "Password protected"
     AuthType Basic
     Require valid-user

     # Allow password-less access for allowed IPs
     Satisfy any
</Location>

</IfModule>

Once you have the configuration in place, you can secure it with a username and password, and then enable it by:

# CentOS 6 / CentOS 7
[root@web01 ~]# htpasswd -c /etc/httpd/status-htpasswd serverinfo
[root@web01 ~]# service httpd restart

# Ubuntu 12.04
[root@web01 ~]# htpasswd -c /etc/apache2/status-htpasswd serverinfo
[root@web01 ~]# service apache2 restart

# Ubuntu 14.04
[root@web01 ~]# htpasswd -c /etc/apache2/status-htpasswd serverinfo
[root@web01 ~]# a2enconf status.conf
[root@web01 ~]# service apache2 restart

Now that mod_status is enabled and working when going to http://serverip/server-status, how can it help with troubleshooting?

Lets say you look at top, and you consistently see an Apache process maxing out a CPU, or using up a ton of memory. You can cross-reference the PID of that Apache child process against the same PID that you find within the server-status page. The requests are constantly changing, so you may need to refresh the /server-status page a couple of times to catch it.

To aid in the troubleshooting as you are trying to match up pids against what is shown in top, you can have the /server-status page refresh automatically by using the following in the URL:

http://serverip/server-status?refresh=2

Once you do locate it, it may give you some idea of what client, or what types of requests, are causing the resource contention issues. Usually it is a specific web application misbehaving, or a specific client is attacking a site.

How to setup Munin on CentOS

Taken directly from Munin’s website:

Munin is a networked resource monitoring tool that can help analyze resource trends and "what just happened to kill our performance?" problems. It is designed to be very plug and play. A default installation provides a lot of graphs with almost no work.

As the paragraph above suggests, Munin is a really good tool for being able to spot anomalies in the servers performance that may help point a system administrator in the right direction for determining where the bottlenecks are coming into play.

Munin can be installed on each node as a stand alone instance, or you can setup a Munin master server, then simply have the clients (Munin nodes) report back to the Munin master.

If would would prefer to have a stand alone instance of Munin on each node, then follow the instructions for setting up the Munin Master server. This guide is currently making the assumption that you already have Apache setup and working on your servers.

Munin Master Setup

To install the latest stable version of Munin, run the following:

# CentOS 6 / RedHat 6
[root@munin-master ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[root@munin-master ~]# yum install munin

# CentOS 7 / RedHat 7
[root@munin-master ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
[root@munin-master ~]# yum install munin

To configure the Munin master, first uncomment the following lines:

[root@munin-master ~]# vim /etc/munin/munin.conf
...
dbdir   /var/lib/munin
htmldir /var/www/html/munin
logdir /var/log/munin
rundir  /var/run/munin
tmpldir /etc/munin/templates
...

Now setup the permissions so Munin can access the directory:

[root@munin-master ~]# mkdir -p /var/www/html/munin
[root@munin-master ~]# chown munin:munin /var/www/html/munin

To protect our performance metrics from being seen by the public, protect the web directory by:

[root@munin-master ~]# yum install httpd-tools
[root@munin-master ~]# htpasswd -c /etc/munin/munin-htpasswd serverinfo
New password: 
Re-type new password: 
Adding password for user serverinfo

Then uncomment, add, or modify the following fields:

# Apache only
[root@munin-master ~]# vim /etc/httpd/conf.d/munin.conf
...
Order allow,deny
Allow from all
Options None
...
AuthUserFile /etc/munin/munin-htpasswd
AuthName "Munin"
AuthType Basic
require valid-user
...

[root@munin-master ~]# service httpd restart

# Nginx only
[root@munin-master ~]# vim /etc/nginx/conf.d/default.conf
server {
...
location /munin/static/ {
        alias /etc/munin/static/;
        expires modified +1w;
}
location /munin {
        auth_basic            "Restricted";
        auth_basic_user_file  /etc/munin/munin-htpasswd;
        alias /var/www/html/munin/;
        expires modified +310s;
}
...
}
[root@munin-master ~]# service nginx restart

Finally, we need to update the name in the host tree to properly identify the Munin Master. So we will be changing the name from localhost.localdomain to MuninMaster. You could change to this to anything you like, such as a FQDN like web01.domain.com if you like.

[root@munin-master ~]# vim /etc/munin/munin.conf
...
[MuninMaster]
    address 127.0.0.1
    use_node_name yes
...

Restart Apache and Munin so the changes register:

[root@munin-master ~]# service httpd restart
[root@munin-master ~]# service munin-node restart

As Munin only refreshes the stats once every 5 minutes via cron, manually run it the first time so you can check out Munin without having to wait:

[root@munin-master ~]# su - munin -s /bin/bash -c 'test -x /usr/bin/munin-cron && /usr/bin/munin-cron'

Then view your Munin stats by navigating your browser to: 

http://serverip/munin

How to add a node to the Munin Master

Here is where we can install additional servers to the Munin Master, so you can view all the graphs from one interface.

First, install the munin-node package on your additional server:

# CentOS 6 / RedHat 6
[root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[root@web01 ~]# yum install munin-node

# CentOS 7 / RedHat 7
[root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
[root@web01 ~]# yum install munin-node

Now configure munin-node to accept inbound connections from your Munin Master server. About halfway down in the file, below the allow ^127\.0\.0\.1$, add the IP address of your Munin Master server. For example, if the IP address of the Munin Master is 192.168.1.100, then the entry would be:

[root@web01 ~]# vim /etc/munin/munin-node.conf
...
allow ^127\.0\.0\.1$
allow ^192\.168\.1\.100$
...

Then restart Munin:

[root@web01 ~]# service munin-node restart

Allow the Munin Master server to connect through the firewall:

[root@web01 ~]# vim /etc/sysconfig/iptables
...
-A INPUT -s 192.168.1.100/24 -p tcp --dport 4949 -m comment --comment "Allow Munin Master server access" -j ACCEPT
...
[root@web01 ~]# service iptables restart

Back on the Munin Master server, we need to tell Munin to query the new Munin Node for information. You can change the FQDN to anything you like, and also be sure to update the IP address to use the IP of your Munin Node:

[root@munin-master ~]# vim /etc/munin/munin.conf
...
[web01.domain.com]
    address 192.168.1.200
    use_node_name yes
...

Then restart Munin:

[root@munin-master ~]# service munin-node restart

Finally, view your Munin stats by navigating your browser to back to your Munin Master server.

Enable additional plugins

Munin comes preinstalled with a number of plugins, but many are disabled since every server runs different things.

To see what plugins can be used on your server, run:

[root@web01 ~]# munin-node-configure --suggest

Plugin                     | Used | Suggestions                            
------                     | ---- | -----------                            
acpi                       | no   | no [cannot read /proc/acpi/thermal_zone/*/temperature]
amavis                     | no   | no                                     
apache_accesses            | yes  | yes                                    
apache_processes           | no   | no                                     
apache_volume              | yes  | yes
...

To break this down:

- If 'Used' column says 'yes', then that means the plugin is already in use
- If 'Used' column says 'no' and 'Suggestions' says 'no', then the plugin cannot be used.
- If 'Used' column says 'no' and 'Suggestions' says 'yes', then you can enable that plugin

Enabling plugins is pretty easy. Lets say you want to enable the iostat plugin:

[root@web01 ~]# ln -s /usr/share/munin/plugins/iostat /etc/munin/plugins

Then restart Munin:

[root@web01 ~]# service munin-node restart

Enable additional plugins – Apache

Munin has the ability to graph Apache accesses, processes, and overall volume for Apache. In order for this to work, mod_status needs to be enabled with ExtendedStatus set to on, and needs to be accessible from the server at: http://localhost/server-status?auto. An example server-status config is shown below:

# RHEL / CentOS
[root@web01 ~]# vim /etc/httpd/conf.d/status.conf

<IfModule mod_status.c>
#
# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
ExtendedStatus On

# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Uncomment and change the ".example.com" to allow
# access from other hosts.
#
<Location /server-status>
     SetHandler server-status
     Order deny,allow
     Deny from all
     Allow from localhost ip6-localhost 127.0.0.1
     <IfModule mod_rewrite.c>
          RewriteEngine off
     </IfModule>

     # AuthUserFile /etc/httpd/status-htpasswd
     # AuthName "Password protected"
     # AuthType Basic
     # Require valid-user

     # Allow password-less access for allowed IPs
     Satisfy any

</Location>

</IfModule>

# RHEL / CentOS
[root@web01 ~]# service httpd restart

Now test to ensure the server-status module is working:

[root@web01 ~]# curl localhost/server-status
...
cache type: SHMCB, shared memory: 512000 bytes, current sessions: 0
subcaches: 32, indexes per subcache: 133
index usage: 0%, cache usage: 0%
total sessions stored since starting: 0
total sessions expired since starting: 0
total (pre-expiry) sessions scrolled out of the cache: 0
total retrieves since starting: 0 hit, 0 miss
total removes since starting: 0 hit, 0 miss
</td></tr>
</table>
</body></html>

Then confirm that the Munin plugin works by running the plugin manually. Confirm the output says ‘yes’ as shown below:

[root@web01 ~]# /usr/share/munin/plugins/apache_accesses autoconf
yes
[root@web01 ~]# /usr/share/munin/plugins/apache_processes autoconf
yes
[root@web01 ~]# /usr/share/munin/plugins/apache_volume autoconf
yes

Now enable the Apache plugins for Munin and restart the service:

[root@web01 ~]# ln -s /usr/share/munin/plugins/apache_accesses /etc/munin/plugins/
[root@web01 ~]# ln -s /usr/share/munin/plugins/apache_processes /etc/munin/plugins/
[root@web01 ~]# ln -s /usr/share/munin/plugins/apache_volume /etc/munin/plugins/
[root@web01 ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if they are now registering the new data. Otherwise you can test the plugins manually to confirm they are returning stats by:

[root@web01 ~]# /etc/munin/plugins/apache_accesses
accesses80.value 1300
[root@web01 ~]# /etc/munin/plugins/apache_processes
busy80.value 1
idle80.value 8
free80.value 58
[root@web01 ~]# /etc/munin/plugins/apache_volume
volume80.value 5093376

Enable additional plugins – Nginx

Munin has the ability to graph Nginx requests and nginx status for Nginx. In order for this to work, the Nginx stub_status module needs to be enabled so the following page will work: http://localhost/nginx_status. An example nginx_status configuration is shown below:

[root@web01 ~]# vim /etc/nginx/conf.d/nginx_status.conf
server {

    listen 80 default_server;
    access_log off;
    server_name _;
    server_name_in_redirect off;
    root  /var/www/html;

        location /nginx_status {
                # Enable Nginx stats
                stub_status on;
                # Disable logging for stats
                access_log   off;
                # Security: Only allow access from authorized IP's
                allow 127.0.0.1;
                # allow xx.xx.xx.xx;
                # Deny everyone else
                deny all;
        }

}
[root@web01 ~]# service nginx restart

Now test to ensure the stub_status module is working with curl:

[root@web01 ~]# curl localhost/nginx_status
Active connections: 1 
server accepts handled requests
 1 1 1 
Reading: 0 Writing: 1 Waiting: 0

Then confirm the Munin plugin works by running the plugin manually. Confirm the output says ‘yes’ as shown below:

[root@web01 ~]# /usr/share/munin/plugins/nginx_request autoconf
yes
[root@web01 ~]# /usr/share/munin/plugins/nginx_status autoconf
yes

Enable the Nginx plugins for Munin and restart the service:

[root@web01 ~]# ln -s /usr/share/munin/plugins/nginx_request /etc/munin/plugins/
[root@web01 ~]# ln -s /usr/share/munin/plugins/nginx_status  /etc/munin/plugins/
[root@web01 ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data. Otherwise you can test the plugins manually to confirm they are returning stats by:

[root@web01 ~]# /etc/munin/plugins/nginx_request
request.value 3

[root@web01 ~]# /etc/munin/plugins/nginx_status
total.value 1
reading.value 0
writing.value 1
waiting.value 0

Enable additional plugins – MySQL
Munin has the ability to graph a number of metrics for MySQL, including commands, caches, buffers, innodb stats, table locks, tmp tables, etc. In order for the plugin to work, Munin will need to be able to access MySQL, and there are also a few dependencies that need to be taken care of. First, install the needed perl modules on the Munin node:

# RHEL / CentOS 6 and 7
[root@db01 ~]# yum install perl-Class-DBI-mysql perl-DBD-mysql libdbi-dbd-mysql

Setup the MySQL user that Munin will use to log into MySQL to pull the stats:

[root@db01 ~]# mysql
mysql> GRANT PROCESS,SUPER on *.* to 'munin'@'127.0.0.1' IDENTIFIED BY 'YOUR_SECURE_PASSWORD_HERE';
mysql> GRANT PROCESS,SUPER on *.* to 'munin'@'localhost' IDENTIFIED BY 'YOUR_SECURE_PASSWORD_HERE';
mysql> flush privileges;
mysql> quit

Setup some configurations within Munin to let it know how to access MySQL:

[root@db01 ~]# vim /etc/munin/plugin-conf.d/munin-node
...
[mysql*]
env.mysqladmin /usr/bin/mysqladmin
env.mysqluser munin
env.mysqlpassword YOUR_SECURE_PASSWORD_HERE
...

[root@db01 ~]# vim /etc/munin/plugin-conf.d/mysql_
[mysql_*]
env.mysqlconnection DBI:mysql:information_schema;host=127.0.0.1;port=3306
env.mysqluser munin
env.mysqlpassword YOUR_SECURE_PASSWORD_HERE
env.cachenamespace munin_mysql_pri 

[root@db01 ~]# vim /etc/munin/plugin-conf.d/mysql_innodb 
[mysql_innodb]
env.warning 0
env.critical 0

Now enable the Munin MySQL plugins:

[root@db01 ~]# for i in `/usr/share/munin/plugins/mysql_ suggest`; do ln -s /usr/share/munin/plugins/mysql_ /etc/munin/plugins/mysql_$i ; done

Restart Munin so the new settings go into effect:

[root@db01 ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data.

Enable additional plugins – Memcached
Munin has the ability to graph commands, current values and network traffic for Memcached. There are also a few dependencies that need to be taken care of. First, install the needed perl modules on the Munin node:

# RHEL / CentOS 6 and 7
[root@memcache01 ~]# yum install perl-Cache-Memcached

Then setup some configurations within Munin to let it know how to access Memcached:

[root@memcache01 ~]# vim /etc/munin/plugin-conf.d/munin-node
...
[memcached_*]
env.host 127.0.0.1
env.port 11211
...

Now enable the Munin Memcached plugins:

[root@memcache01 ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_bytes
[root@memcache01 ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_counters
[root@memcache01 ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_rates

Restart Munin so the new setting take effect:

[root@memcache01 ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data.

Troubleshooting – Connectivity issues

If Munin is not graphing data, it could be because Munin cannot connect to the master server. Some basic things to check would be to ensure Munin is running on the master server and node server. Ensure that port 4949 is opened inbound to the Munin master from the node. Finally on the Munin master, confirm that /etc/munin/munin-node.conf has an allow block for the Munin node server.

You can test connectivity by running the following, substituting localhost out accordingly if your connecting from a Munin node server:

[root@web01 ~]# telnet localhost 4949
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
# munin node at localhost.localdomain

Troubleshooting – See what Munin plugins are enabled

To see which plugins are enabled on your Munin node, you can quickly test for it by issuing the ‘list’ command via telnet:

[root@web01 ~]# telnet localhost 4949
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
# munin node at localhost.localdomain
list
apache_accesses apache_processes apache_volume cpu df df_inode entropy forks fw_conntrack fw_forwarded_local fw_packets if_err_eth0 if_eth0 interrupts irqstats load memcached_bytes memcached_counters memcached_rates memory ...

Troubleshooting – Testing a plugin manually

If a graph is showing up blank, its useful to test the plugin directly to ensure it is able to grab the data. Using the ‘df’ plugin as the example, it can be ran manually by:

[root@web01 ~]# munin-run df
_dev_mapper_VolGroup_lv_root.value 16.9176920622854
_dev_sda1.value 27.6994797122402

Troubleshooting – Plugin not generating graph data

If a plugin is not generating stats within Munin, try running the plugin manually to see if it will return stats on the command line. Equally important, ensure that the application is getting some traffic for the plugin to graph. Finally, keep in mind that it can take up to 15 minutes for a plugin to start graphing, assuming that you see stats when running it manually.

How to setup Munin on Ubuntu

Taken directly from Munin’s website:

Munin is a networked resource monitoring tool that can help analyze resource trends and "what just happened to kill our performance?" problems. It is designed to be very plug and play. A default installation provides a lot of graphs with almost no work.

As the paragraph above suggests, Munin is a really good tool for being able to spot anomalies in the servers performance that may help point a system administrator in the right direction for determining where the bottlenecks are coming into play.

Munin can be installed on each node as a stand alone instance, or you can setup a Munin master server, then simply have the clients (Munin nodes) report back to the Munin master.

If would would prefer to have a stand alone instance of Munin on each node, then follow the instructions for setting up the Munin Master server. This guide is currently making the assumption that you already have Apache setup and working on your servers.

Munin Master Setup

First, we need to install some required packages so the graphs work properly.

# Apache only
[root@munin-master ~]# apt-get update
[root@munin-master ~]# apt-get install libcgi-fast-perl libapache2-mod-fcgid libio-all-lwp-perl 
[root@munin-master ~]# a2enmod fcgid
[root@munin-master ~]# service apache2 restart
[root@munin-master ~]# /usr/sbin/apachectl -M | grep -i cgi
fcgid_module (shared)

# Nginx only
[root@munin-master ~]# apt-get update
[root@munin-master ~]# apt-get install libcgi-fast-perl liblwp-protocol-https-perl

Now to install the latest stable version of Munin, run the following:

[root@munin-master ~]# apt-get update
[root@munin-master ~]# apt-get install munin

To configure the Munin master, first uncomment the following lines:

[root@munin-master ~]# vim /etc/munin/munin.conf
...
dbdir     /var/lib/munin
htmldir   /var/cache/munin/www
logdir    /var/log/munin
rundir    /var/run/munin
tmpldir /etc/munin/templates
...

Now setup the permissions so Munin can access the directory:

[root@munin-master ~]# mkdir -p /var/cache/munin/www
[root@munin-master ~]# chown munin:munin /var/cache/munin/www

To protect our performance metrics from being seen by the public, protect the web directory by:

[root@munin-master ~]# htpasswd -c /etc/munin/munin-htpasswd serverinfo
New password: 
Re-type new password: 
Adding password for user serverinfo

Then uncomment, add or modify the following fields based off the OS and software running:

# Apache - Ubuntu 14.04
[root@munin-master ~]# vim /etc/munin/apache.conf
...
#Order allow,deny
#Allow from localhost 127.0.0.0/8 ::1
#Options None
...
AuthUserFile /etc/munin/munin-htpasswd
AuthName "serverinfo"
AuthType Basic
require valid-user
...
[root@munin-master ~]# service apache2 restart

# Apache - Ubuntu 16.04
[root@munin-master ~]# mv /etc/munin/apache24.conf /etc/munin/apache24.conf.orig
[root@munin-master ~]# vim /etc/munin/apache24.conf
Alias /munin /var/cache/munin/www
<Directory /var/cache/munin/www>
 # Require local
 # Require all granted
 AuthUserFile /etc/munin/munin-htpasswd
 AuthName "Munin"
 AuthType Basic
 Require valid-user
 Options None
</Directory>

ScriptAlias /munin-cgi/munin-cgi-graph /usr/lib/munin/cgi/munin-cgi-graph
<Location /munin-cgi/munin-cgi-graph>
 # Require local
 # Require all granted
 AuthUserFile /etc/munin/munin-htpasswd
 AuthName "Munin"
 AuthType Basic
 Require valid-user
 <IfModule mod_fcgid.c>
 SetHandler fcgid-script
 </IfModule>
 <IfModule !mod_fcgid.c>
 SetHandler cgi-script
 </IfModule>
</Location>
[root@munin-master ~]# service apache2 restart

# Nginx - Ubuntu 14.04
[root@munin-master ~]# vim /etc/nginx/sites-available/default
server {
...
location /munin/static/ {
        alias /etc/munin/static/;
        expires modified +1w;
}
location /munin {
        auth_basic            "Restricted";
        auth_basic_user_file  /etc/munin/munin-htpasswd;
        alias /var/cache/munin/www/;
        expires modified +310s;
}
...
}
[root@munin-master ~]# service nginx restart

# Nginx - Ubuntu 16.04
# NOTE:  If you run into issues with the css not working within 
# Munin, you may have to disable the following within the site's
# Nginx config if it exists:
#  location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
#     expires max;
#     log_not_found off;
#  }

[root@munin-master ~]# vim /etc/nginx/sites-available/default
server {
...
location /munin/static/ {
        alias /etc/munin/static/;
}
location /munin {
        auth_basic            "Restricted";
        auth_basic_user_file  /etc/munin/munin-htpasswd;
        alias /var/cache/munin/www/;
}

Finally, we need to update the name in the host tree to properly identify the Munin Master. So we will be changing the name from localhost.localdomain to MuninMaster. You could change to this to anything you like, such as a FQDN like web01.domain.com if you like.

[root@munin-master ~]# vim /etc/munin/munin.conf
...
[MuninMaster]
    address 127.0.0.1
    use_node_name yes
...

Restart Apache and Munin so the changes register:

[root@munin-master ~]# service apache2 restart
[root@munin-master ~]# service munin-node restart

Then view your Munin stats by navigating your browser to: 

http://serverip/munin

How to add a node to the Munin Master

Here is where we can install additional servers to the Munin Master, so you can view all the graphs from one interface.

First, install the munin-node package on your additional server:

[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install munin-node

Now configure munin-node to accept inbound connections from your Munin Master server. About halfway down in the file, below the allow ^127\.0\.0\.1$, add the IP address of your Munin Master server. For example, if the IP address of the Munin Master is 192.168.1.100, then the entry would be:

[root@web01 ~]# vim /etc/munin/munin-node.conf
...
allow ^127\.0\.0\.1$
allow ^192\.168\.1\.100$
...

Then restart Munin:

[root@web01 ~]# service munin-node restart

Allow the Munin Master server to connect through the firewall:

[root@web01 ~]# ufw allow from 192.168.1.100 to any port 4949

Back on the Munin Master server, we need to tell Munin to query the new Munin Node for information. You can change the FQDN to anything you like, and also be sure to update the IP address to use the IP of your Munin Node:

[root@munin-master ~]# vim /etc/munin/munin.conf
...
[web01.domain.com]
    address 192.168.1.200
    use_node_name yes
...

Then restart Munin:

[root@munin-master ~]# service munin-node restart

Finally, view your Munin stats by navigating your browser to back to your Munin Master server.

Enable additional plugins

Munin comes preinstalled with a number of plugins, but many are disabled since every server runs different things. There are also extra plugins that may or may not be installed by default. So to get those, run:

[root@web01 ~]# apt-get install munin-plugins-extra

To see what plugins can be used on your server, run:

[root@web01 ~]# munin-node-configure --suggest

Plugin                     | Used | Suggestions                            
------                     | ---- | -----------                            
acpi                       | no   | no [cannot read /proc/acpi/thermal_zone/*/temperature]
amavis                     | no   | no                                     
apache_accesses            | yes  | yes                                    
apache_processes           | no   | no                                     
apache_volume              | yes  | yes
...

To break this down:

- If 'Used' column says 'yes', then that means the plugin is already in use
- If 'Used' column says 'no' and 'Suggestions' says 'no', then the plugin cannot be used.
- If 'Used' column says 'no' and 'Suggestions' says 'yes', then you can enable that plugin

Enabling plugins is pretty easy. Lets say you want to enable the iostat plugin:

[root@web01 ~]# ln -s /usr/share/munin/plugins/iostat /etc/munin/plugins

Then restart Munin:

[root@web01 ~]# service munin-node restart

Enable additional plugins – Apache

Munin has the ability to graph Apache accesses, processes, and overall volume for Apache. In order for this to work, mod_status needs to be enabled with ExtendedStatus set to on, and needs to be accessible from the server at: http://localhost/server-status?auto. An example server-status config is shown below:

# Debian / Ubuntu
[root@web01 ~]# vim /etc/apache2/mods-available/status.conf

<IfModule mod_status.c>
#
# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
ExtendedStatus On

# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Uncomment and change the ".example.com" to allow
# access from other hosts.
#
<Location /server-status>
     SetHandler server-status
     Order deny,allow
     Deny from all
     Allow from localhost ip6-localhost 127.0.0.1
     <IfModule mod_rewrite.c>
          RewriteEngine off
     </IfModule>

     # AuthUserFile /etc/httpd/status-htpasswd
     # AuthName "Password protected"
     # AuthType Basic
     # Require valid-user

     # Allow password-less access for allowed IPs
     Satisfy any

</Location>

</IfModule>

[root@web01 ~]# a2enmod status
[root@web01 ~]# service apache2 restart

Now test to ensure the server-status module is working:

[root@web01 ~]# curl localhost/server-status
...
cache type: SHMCB, shared memory: 512000 bytes, current sessions: 0
subcaches: 32, indexes per subcache: 133
index usage: 0%, cache usage: 0%
total sessions stored since starting: 0
total sessions expired since starting: 0
total (pre-expiry) sessions scrolled out of the cache: 0
total retrieves since starting: 0 hit, 0 miss
total removes since starting: 0 hit, 0 miss
</td></tr>
</table>
</body></html>

Then confirm that the Munin plugin works by running the plugin manually. Confirm the output says ‘yes’ as shown below:

[root@web01 ~]# /usr/share/munin/plugins/apache_accesses autoconf
yes
[root@web01 ~]# /usr/share/munin/plugins/apache_processes autoconf
yes
[root@web01 ~]# /usr/share/munin/plugins/apache_volume autoconf
yes

Now enable the Apache plugins for Munin and restart the service:

[root@web01 ~]# ln -s /usr/share/munin/plugins/apache_accesses /etc/munin/plugins/
[root@web01 ~]# ln -s /usr/share/munin/plugins/apache_processes /etc/munin/plugins/
[root@web01 ~]# ln -s /usr/share/munin/plugins/apache_volume /etc/munin/plugins/
[root@web01 ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if they are now registering the new data. Otherwise you can test the plugins manually to confirm they are returning stats by:

[root@web01 ~]# /etc/munin/plugins/apache_accesses
accesses80.value 1300
[root@web01 ~]# /etc/munin/plugins/apache_processes
busy80.value 1
idle80.value 8
free80.value 58
[root@web01 ~]# /etc/munin/plugins/apache_volume
volume80.value 5093376

Enable additional plugins – Nginx

Munin has the ability to graph Nginx requests and nginx status for Nginx. In order for this to work, the Nginx stub_status module needs to be enabled so the following page will work: http://localhost/nginx_status. An example nginx_status configuration is shown below:

[root@web01 ~]# vim /etc/nginx/conf.d/nginx_status.conf
server {

    listen 80 default_server;
    access_log off;
    server_name _;
    server_name_in_redirect off;
    root  /var/www/html;

        location /nginx_status {
                # Enable Nginx stats
                stub_status on;
                # Disable logging for stats
                access_log   off;
                # Security: Only allow access from authorized IP's
                allow 127.0.0.1;
                # allow xx.xx.xx.xx;
                # Deny everyone else
                deny all;
        }

}
[root@web01 ~]# service nginx restart

Now test to ensure the stub_status module is working with curl:

[root@web01 ~]# curl localhost/nginx_status
Active connections: 1 
server accepts handled requests
 1 1 1 
Reading: 0 Writing: 1 Waiting: 0

Then confirm the Munin plugin works by running the plugin manually. Confirm the output says ‘yes’ as shown below:

[root@web01 ~]# /usr/share/munin/plugins/nginx_request autoconf
yes
[root@web01 ~]# /usr/share/munin/plugins/nginx_status autoconf
yes

Enable the Nginx plugins for Munin and restart the service:

[root@web01 ~]# ln -s /usr/share/munin/plugins/nginx_request /etc/munin/plugins/
[root@web01 ~]# ln -s /usr/share/munin/plugins/nginx_status  /etc/munin/plugins/
[root@web01 ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data. Otherwise you can test the plugins manually to confirm they are returning stats by:

[root@web01 ~]# /etc/munin/plugins/nginx_request
request.value 3

[root@web01 ~]# /etc/munin/plugins/nginx_status
total.value 1
reading.value 0
writing.value 1
waiting.value 0

Enable additional plugins – MySQL
Munin has the ability to graph a number of metrics for MySQL, including commands, caches, buffers, innodb stats, table locks, tmp tables, etc. In order for the plugin to work, Munin will need to be able to access MySQL, and there are also a few dependencies that need to be taken care of. First, install the needed perl modules on the Munin node:

# Debian / Ubuntu
[root@db01 ~]# apt-get update
[root@db01 ~]# apt-get install libclass-dbi-mysql-perl libdbd-mysql-perl libdbi-perl libcache-perl libcache-cache-perl

Now enable the Munin MySQL plugins:

[root@db01 ~]# for i in `/usr/share/munin/plugins/mysql_ suggest`; do ln -s /usr/share/munin/plugins/mysql_ /etc/munin/plugins/mysql_$i ; done

Restart Munin so the new settings go into effect:

[root@db01 ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data.

Enable additional plugins – Memcached
Munin has the ability to graph commands, current values and network traffic for Memcached. There are also a few dependencies that need to be taken care of. First, install the needed perl modules on the Munin node:

# Debian / Ubuntu
[root@memcache01 ~]# apt-get update
[root@memcache01 ~]# apt-get install libcache-memcached-perl

Then setup some configurations within Munin to let it know how to access Memcached:

[root@memcache01 ~]# vim /etc/munin/plugin-conf.d/munin-node
...
[memcached_*]
env.host 127.0.0.1
env.port 11211
...

Now enable the Munin Memcached plugins:

[root@memcache01 ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_bytes
[root@memcache01 ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_counters
[root@memcache01 ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_rates

Restart Munin so the new setting take effect:

[root@memcache01 ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data.

Troubleshooting – Connectivity issues

If Munin is not graphing data, it could be because Munin cannot connect to the master server. Some basic things to check would be to ensure Munin is running on the master server and node server. Ensure that port 4949 is opened inbound to the Munin master from the node. Finally on the Munin master, confirm that /etc/munin/munin-node.conf has an allow block for the Munin node server.

You can test connectivity by running the following, substituting localhost out accordingly if your connecting from a Munin node server:

[root@web01 ~]# telnet localhost 4949
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
# munin node at localhost.localdomain

Troubleshooting – See what Munin plugins are enabled

To see which plugins are enabled on your Munin node, you can quickly test for it by issuing the ‘list’ command via telnet:

[root@web01 ~]# telnet localhost 4949
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
# munin node at localhost.localdomain
list
apache_accesses apache_processes apache_volume cpu df df_inode entropy forks fw_conntrack fw_forwarded_local fw_packets if_err_eth0 if_eth0 interrupts irqstats load memcached_bytes memcached_counters memcached_rates memory ...

Troubleshooting – Testing a plugin manually

If a graph is showing up blank, its useful to test the plugin directly to ensure it is able to grab the data. Using the ‘df’ plugin as the example, it can be ran manually by:

[root@web01 ~]# munin-run df
_dev_mapper_VolGroup_lv_root.value 16.9176920622854
_dev_sda1.value 27.6994797122402

Troubleshooting – Plugin not generating graph data

If a plugin is not generating stats within Munin, try running the plugin manually to see if it will return stats on the command line. Equally important, ensure that the application is getting some traffic for the plugin to graph. Finally, keep in mind that it can take up to 15 minutes for a plugin to start graphing, assuming that you see stats when running it manually.

SSL terminated load balancer causing redirect loops

You have an environment that is terminating SSL on the load balancer for one reason or another. Your application, such as WordPress or Magento, is configured to force SSL. But when you go to test out the site or the admin portal, you get a redirect loop. What happened?

This is a very common issue. In most cases when you are terminating SSL at the load balancer, the load balancer will send the traffic over to your web server using HTTP. This can confuse the web application since it was expecting it to be over HTTPS, and the application will not be able to tell that the client’s browser was in fact using HTTPS, which will result in a redirect loop.

The solution to this is actually very simple. You need to ask your load balancer to send the X-Forwarded-Proto header. This can easily be done by adding a SetEnvIf directive into your .htaccess (assuming Apache here), which will set the header to be what your application was expecting.

To account for this, at the top of your site’s .htaccess file, add the following:

[root@web01 ~]# vim /var/www/vhosts/www.domain.com/.htaccess
...
# Detect the LB header and set the header accordingly for the application
SetEnvIf X-Forwarded-Proto https HTTPS=on
...

So in summary, this will prevent your application from getting confused regarding if the connection originated over HTTP or HTTPS since the load balancer is handling the SSL termination, not the server.

Apache Mod_Rewrite tutorial

Apache Mod_Rewrite provides the ability to rewrite incoming requests to different destinations. It can be a bit complicated to wrap your head around the syntax, but once you get used to it, it becomes very powerful.

Rewrite rules can be setup in 2 places, the Apache VirtualHost configuration, or within the website’s .htaccess file. Modifying the .htaccess file allows you to change the rewrite rules on the fly without restarting Apache. However it comes at a steep cost. When using an .htaccess file, each time someone visits your page, the .htaccess file must be processed. So for large rulesets, this could add a substantial CPU load on your web server.

Taken directly from Apache’s website:

You should avoid using .htaccess files completely if you have access to httpd main server config file. Using .htaccess files slows down your Apache http server. Any directive that you can include in a .htaccess file is better set in a Directory block, as it will have the same effect with better performance.

To avoid the performance penalty, I prefer to keep the rules within the Apache VirtualHost configuration. When Apache is restarted, the rules are read into memory, which allows for faster processing.

Rewrite rule example – Step by step

I prefer to learn by example. So lets take an example that is going to be slightly more complicated then it needs to be on purpose to illustrate the syntax. 

All requests to http://www.domain.com should be redirected to http://192.168.1.100.  Concurrently, https should also redirect to https://192.168.1.100.

A possible solution to put into the Apache VirtualHost configuration would be:

RewriteEngine On
RewriteCond %{SERVER_PORT} ^80$
RewriteCond %{HTTP_HOST} ^domain.com$ [NC,OR]
RewriteCond %{HTTP_HOST} ^www.domain.com$ [NC]
RewriteRule ^/(.*) http://192.168.1.100/$1 [R,L]

RewriteCond %{SERVER_PORT} ^443$
RewriteCond %{HTTP_HOST} ^domain.com$ [NC,OR]
RewriteCond %{HTTP_HOST} ^www.domain.com$ [NC]
RewriteRule ^/(.*) https://192.168.1.100/$1 [R,L]

To explain some of the common regex used:

1.  ^ --> String starts with
2.  !^ --> String does not start with
3.  % --> server variable
4.  [NC,OR]  --> This means 'NC' Not case sensitive 'OR' It can either match this or the line below.  If you want it to match both rules, then just use 'NC'  Using an 'AND' statement would break the rule, as the policy already defaults to 'AND'.
5.  [R,L]  This means:  rewrite, last.  So basically it is doing the rewrite and it saying this is the last part of the condition, so end the loop.  
6.  ^/(.*) --> This means anything and everything matching the conditions above (used in this example in the rewriterule)

Easy right? Yea, I know it is not. But it makes more sense the more you have to use it. So let me explain the solution above line by line:

First, we need to enable the rewrite engine:

RewriteEngine On

Now put in the first test condition. For the purposes of this article, we are going to simplify this request by testing for anything coming in over port 80:

RewriteCond %{SERVER_PORT} ^80$

Then we need to test a condition to actually match the URL we are looking to do the rewrite on. In this case, it is domain.com. Now, you should be sure to set the ‘NC’ (non case) so we don’t break the rule if someone types in the request in capital letters. You want to test for both domain.com and www.domain.com. This is done by:

RewriteCond %{HTTP_HOST} ^domain.com$ [NC,OR]
RewriteCond %{HTTP_HOST} ^www.domain.com$ [NC]

Next we write the actual rewrite statement. This is saying, take the result of the above and point it to what we specify here. So to talk this out, it is rewriting anything that matches whats above ^/(.*) and redirecting it to the specified address. The [R,L] rewrite the url ‘R’, and end the statement as it is the last rule ‘L’. 

RewriteRule ^/(.*) http://192.168.1.100/$1 [R,L]

Finally, don’t forget to apply all this to https on port 443! You use the same explanations as above for figuring this out.

Common rewrite examples

Enough of the obscene examples and explanations. Below is a bunch of common (and not so common) examples for quick reference:

Rewrite domain.com to www.domain.com:

RewriteEngine On
RewriteCond %{HTTP_HOST} !^www [NC]
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}$1 [R=301,L]

Rewrite www.domain.com to domain.com:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^www.domain.com [NC]
RewriteRule ^(.*)$ http://domain.com$1 [L,R=301]

Force SSL on your domain, meaning take any http requests and redirect them to https:

RewriteEngine On
RewriteCond %{SERVER_PORT} !^443$
RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R=301,L]

Force SSL on your domain, meaning take any http requests and redirect them to https when using an SSL terminated load balancer:

RewriteEngine On
RewriteCond %{HTTP:X-Forwarded-Proto} !https
RewriteRule ^(.*)$ https://www.domain.com$1 [R=301,L]

Rewrite one domain to another:

RewriteEngine on
RewriteCond %{HTTP_HOST} ^(www.)?domain1.com [NC]
RewriteRule ^(.*)$ http://domain2.com$1 [R=301,L]

Redirect a path to a new domain:

Redirect 301 /path http://new.domain.com
Redirect 301 /otherpath/somepage.php http://other.domain.com

Rewrite page with query string to a different page and include query string. Please note, this must be in the Apache VirtualHost config, it will not work in the .htaccess.

# This will redirect www.example.com/products?sku=xxx over to www.example.com/products/sku/xxx
RewriteEngine On
RewriteCond %{REQUEST_URI} ^/products [NC]
RewriteCond %{QUERY_STRING} ^sku=(.*)
RewriteRule (.*) https://www.example.com/products/sku/%1? [R=301,L]

Rewrite page with query string, and strip query string:

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-l
RewriteCond %{REQUEST_URI} ^/pages/pages\.php$
RewriteCond %{QUERY_STRING} ^page=[0-9]*$
RewriteRule ^.*$ http://www.domain.com/path/? [R=301,L]

Force all URL’s to be lowercase. Please note, this must be in the Apache VirtualHost config, it will not work in the .htaccess.

RewriteEngine On
RewriteMap lc int:tolower
RewriteCond %{REQUEST_URI} [A-Z]
RewriteRule (.*) ${lc:$1} [R=301,L]

Exclude phpmyadmin from being rewritten by existing rules (Place this above the problem rule):

RewriteCond %{REQUEST_URI} !=/phpmyadmin

Disable TRACE and TRACK methods:

RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
RewriteRule .* - [F]

Rewrite images to Cloud Files. Please note, this must be in the Apache VirtualHost config, it will not work in the .htaccess.
* Note: The RewriteRule and URL belong on the same line.

<Directory /var/www/vhosts/domain.com/content/images>
RewriteEngine On
RewriteRule ^(.*)$ http://c0000.cdn00.cloudfiles.rackspacecloud.com/$1 [R=302,L]
</Directory>

Rewrite all pages to a maintenance page

Options +FollowSymlinks
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_URI} !/maintenance.html$
RewriteRule .* /maintenance.html [L]

Setup ClamAV for nightly scans

PCI-DSS 3.1 Requirement 5 states the following:

Protect all systems against malware and regularly update anti-virus software or programs.

There are commercial based solutions out there for Linux based systems, but costs can become an issue, especially for small companies with a small footprint within their card holder data environment (CDE). So can one satify this requirement without breaking the bank? I personally prefer ClamAV.

Taken from the projects website, ClamAV is an open source antivirus engine for detecting trojans, viruses, malware and other malicious threats.

My requirements:
1. I want to scan my entire system nightly.
2. All virus reports are emailed to me so I can archive them for a year offsite.
3. Have the antivirus definitions updated nightly before the scan.

Installing, running and maintaining ClamAV is very straight forward on Linux based systems. To get started, install ClamAV by:

# CentOS 6 / RedHat 6
[root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[root@web01 ~]# yum install clamav mailx

# CentOS 7 / RedHat 7
[root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
[root@web01 ~]# yum install clamav clamav-update mailx
[root@web01 ~]# sed -i '/^Example/d' /etc/freshclam.conf

# Ubuntu 12.04 / Ubuntu 14.04
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install clamav mailutils

Now update the virus definitions by running:

[root@web01 ~]# freshclam

Finally, configure the virus definitions to update nightly, and also scan the entire system and email a report:

[root@web01 ~]# crontab -e
00 2 * * *  /usr/bin/freshclam
00 3 * * * /usr/bin/clamscan -r -i / | mail -s "ClamAV Report : INSERT_SERVER_HOSTNAME_HERE" [email protected]

Posted below is an example report ClamAV would send me via email nightly:

----------- SCAN SUMMARY -----------
Known viruses: 4289299
Engine version: 0.99
Scanned directories: 51929
Scanned files: 808848
Infected files: 0
Total errors: 10982
Data scanned: 76910.89 MB
Data read: 83578.27 MB (ratio 0.92:1)
Time: 6641.424 sec (110 m 41 s)

How does one go about testing ClamAV to ensure its working? There is a known antivirus test file that was designed specifically for this purpose by www.eicar.org. To create this file, simply setup the following test file, then rerun your ClamAV scan:

[root@web01 ~]# vim /tmp/EICAR-AV-Test
...
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
...