How to setup Munin on CentOS

Taken directly from Munin’s website:

Munin is a networked resource monitoring tool that can help analyze resource trends and "what just happened to kill our performance?" problems. It is designed to be very plug and play. A default installation provides a lot of graphs with almost no work.

As the paragraph above suggests, Munin is a really good tool for being able to spot anomalies in the servers performance that may help point a system administrator in the right direction for determining where the bottlenecks are coming into play.

Munin can be installed on each node as a stand alone instance, or you can setup a Munin master server, then simply have the clients (Munin nodes) report back to the Munin master.

If would would prefer to have a stand alone instance of Munin on each node, then follow the instructions for setting up the Munin Master server. This guide is currently making the assumption that you already have Apache setup and working on your servers.

Munin Master Setup

To install the latest stable version of Munin, run the following:

# CentOS 6 / RedHat 6
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[[email protected] ~]# yum install munin

# CentOS 7 / RedHat 7
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
[[email protected] ~]# yum install munin

To configure the Munin master, first uncomment the following lines:

[[email protected] ~]# vim /etc/munin/munin.conf
...
dbdir   /var/lib/munin
htmldir /var/www/html/munin
logdir /var/log/munin
rundir  /var/run/munin
tmpldir /etc/munin/templates
...

Now setup the permissions so Munin can access the directory:

[[email protected] ~]# mkdir -p /var/www/html/munin
[[email protected] ~]# chown munin:munin /var/www/html/munin

To protect our performance metrics from being seen by the public, protect the web directory by:

[[email protected] ~]# yum install httpd-tools
[[email protected] ~]# htpasswd -c /etc/munin/munin-htpasswd serverinfo
New password: 
Re-type new password: 
Adding password for user serverinfo

Then uncomment, add, or modify the following fields:

# Apache only
[[email protected] ~]# vim /etc/httpd/conf.d/munin.conf
...
Order allow,deny
Allow from all
Options None
...
AuthUserFile /etc/munin/munin-htpasswd
AuthName "Munin"
AuthType Basic
require valid-user
...

[[email protected] ~]# service httpd restart

# Nginx only
[[email protected] ~]# vim /etc/nginx/conf.d/default.conf
server {
...
location /munin/static/ {
        alias /etc/munin/static/;
        expires modified +1w;
}
location /munin {
        auth_basic            "Restricted";
        auth_basic_user_file  /etc/munin/munin-htpasswd;
        alias /var/www/html/munin/;
        expires modified +310s;
}
...
}
[[email protected] ~]# service nginx restart

Finally, we need to update the name in the host tree to properly identify the Munin Master. So we will be changing the name from localhost.localdomain to MuninMaster. You could change to this to anything you like, such as a FQDN like web01.domain.com if you like.

[[email protected] ~]# vim /etc/munin/munin.conf
...
[MuninMaster]
    address 127.0.0.1
    use_node_name yes
...

Restart Apache and Munin so the changes register:

[[email protected] ~]# service httpd restart
[[email protected] ~]# service munin-node restart

As Munin only refreshes the stats once every 5 minutes via cron, manually run it the first time so you can check out Munin without having to wait:

[[email protected] ~]# su - munin -s /bin/bash -c 'test -x /usr/bin/munin-cron && /usr/bin/munin-cron'

Then view your Munin stats by navigating your browser to:

http://serverip/munin

How to add a node to the Munin Master

Here is where we can install additional servers to the Munin Master, so you can view all the graphs from one interface.

First, install the munin-node package on your additional server:

# CentOS 6 / RedHat 6
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[[email protected] ~]# yum install munin-node

# CentOS 7 / RedHat 7
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
[[email protected] ~]# yum install munin-node

Now configure munin-node to accept inbound connections from your Munin Master server. About halfway down in the file, below the allow ^127\.0\.0\.1$, add the IP address of your Munin Master server. For example, if the IP address of the Munin Master is 192.168.1.100, then the entry would be:

[[email protected] ~]# vim /etc/munin/munin-node.conf
...
allow ^127\.0\.0\.1$
allow ^192\.168\.1\.100$
...

Then restart Munin:

[[email protected] ~]# service munin-node restart

Allow the Munin Master server to connect through the firewall:

[[email protected] ~]# vim /etc/sysconfig/iptables
...
-A INPUT -s 192.168.1.100/24 -p tcp --dport 4949 -m comment --comment "Allow Munin Master server access" -j ACCEPT
...
[[email protected] ~]# service iptables restart

Back on the Munin Master server, we need to tell Munin to query the new Munin Node for information. You can change the FQDN to anything you like, and also be sure to update the IP address to use the IP of your Munin Node:

[[email protected] ~]# vim /etc/munin/munin.conf
...
[web01.domain.com]
    address 192.168.1.200
    use_node_name yes
...

Then restart Munin:

[[email protected] ~]# service munin-node restart

Finally, view your Munin stats by navigating your browser to back to your Munin Master server.

Enable additional plugins

Munin comes preinstalled with a number of plugins, but many are disabled since every server runs different things.

To see what plugins can be used on your server, run:

[[email protected] ~]# munin-node-configure --suggest

Plugin                     | Used | Suggestions                            
------                     | ---- | -----------                            
acpi                       | no   | no [cannot read /proc/acpi/thermal_zone/*/temperature]
amavis                     | no   | no                                     
apache_accesses            | yes  | yes                                    
apache_processes           | no   | no                                     
apache_volume              | yes  | yes
...

To break this down:

- If 'Used' column says 'yes', then that means the plugin is already in use
- If 'Used' column says 'no' and 'Suggestions' says 'no', then the plugin cannot be used.
- If 'Used' column says 'no' and 'Suggestions' says 'yes', then you can enable that plugin

Enabling plugins is pretty easy. Lets say you want to enable the iostat plugin:

[[email protected] ~]# ln -s /usr/share/munin/plugins/iostat /etc/munin/plugins

Then restart Munin:

[[email protected] ~]# service munin-node restart

Enable additional plugins – Apache

Munin has the ability to graph Apache accesses, processes, and overall volume for Apache. In order for this to work, mod_status needs to be enabled with ExtendedStatus set to on, and needs to be accessible from the server at: http://localhost/server-status?auto. An example server-status config is shown below:

# RHEL / CentOS
[[email protected] ~]# vim /etc/httpd/conf.d/status.conf

<IfModule mod_status.c>
#
# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
ExtendedStatus On

# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Uncomment and change the ".example.com" to allow
# access from other hosts.
#
<Location /server-status>
     SetHandler server-status
     Order deny,allow
     Deny from all
     Allow from localhost ip6-localhost 127.0.0.1
     <IfModule mod_rewrite.c>
          RewriteEngine off
     </IfModule>

     # AuthUserFile /etc/httpd/status-htpasswd
     # AuthName "Password protected"
     # AuthType Basic
     # Require valid-user

     # Allow password-less access for allowed IPs
     Satisfy any

</Location>

</IfModule>

# RHEL / CentOS
[[email protected] ~]# service httpd restart

Now test to ensure the server-status module is working:

[[email protected] ~]# curl localhost/server-status
...
cache type: SHMCB, shared memory: 512000 bytes, current sessions: 0
subcaches: 32, indexes per subcache: 133
index usage: 0%, cache usage: 0%
total sessions stored since starting: 0
total sessions expired since starting: 0
total (pre-expiry) sessions scrolled out of the cache: 0
total retrieves since starting: 0 hit, 0 miss
total removes since starting: 0 hit, 0 miss
</td></tr> </table> </body></html>

Then confirm that the Munin plugin works by running the plugin manually. Confirm the output says ‘yes’ as shown below:

[[email protected] ~]# /usr/share/munin/plugins/apache_accesses autoconf
yes
[[email protected] ~]# /usr/share/munin/plugins/apache_processes autoconf
yes
[[email protected] ~]# /usr/share/munin/plugins/apache_volume autoconf
yes

Now enable the Apache plugins for Munin and restart the service:

[[email protected] ~]# ln -s /usr/share/munin/plugins/apache_accesses /etc/munin/plugins/
[[email protected] ~]# ln -s /usr/share/munin/plugins/apache_processes /etc/munin/plugins/
[[email protected] ~]# ln -s /usr/share/munin/plugins/apache_volume /etc/munin/plugins/
[[email protected] ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if they are now registering the new data. Otherwise you can test the plugins manually to confirm they are returning stats by:

[[email protected] ~]# /etc/munin/plugins/apache_accesses
accesses80.value 1300
[[email protected] ~]# /etc/munin/plugins/apache_processes
busy80.value 1
idle80.value 8
free80.value 58
[[email protected] ~]# /etc/munin/plugins/apache_volume
volume80.value 5093376

Enable additional plugins – Nginx

Munin has the ability to graph Nginx requests and nginx status for Nginx. In order for this to work, the Nginx stub_status module needs to be enabled so the following page will work: http://localhost/nginx_status. An example nginx_status configuration is shown below:

[[email protected] ~]# vim /etc/nginx/conf.d/nginx_status.conf
server {

    listen 80 default_server;
    access_log off;
    server_name _;
    server_name_in_redirect off;
    root  /var/www/html;

        location /nginx_status {
                # Enable Nginx stats
                stub_status on;
                # Disable logging for stats
                access_log   off;
                # Security: Only allow access from authorized IP's
                allow 127.0.0.1;
                # allow xx.xx.xx.xx;
                # Deny everyone else
                deny all;
        }

}
[[email protected] ~]# service nginx restart

Now test to ensure the stub_status module is working with curl:

[[email protected] ~]# curl localhost/nginx_status
Active connections: 1 
server accepts handled requests
 1 1 1 
Reading: 0 Writing: 1 Waiting: 0

Then confirm the Munin plugin works by running the plugin manually. Confirm the output says ‘yes’ as shown below:

[[email protected] ~]# /usr/share/munin/plugins/nginx_request autoconf
yes
[[email protected] ~]# /usr/share/munin/plugins/nginx_status autoconf
yes

Enable the Nginx plugins for Munin and restart the service:

[[email protected] ~]# ln -s /usr/share/munin/plugins/nginx_request /etc/munin/plugins/
[[email protected] ~]# ln -s /usr/share/munin/plugins/nginx_status  /etc/munin/plugins/
[[email protected] ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data. Otherwise you can test the plugins manually to confirm they are returning stats by:

[[email protected] ~]# /etc/munin/plugins/nginx_request
request.value 3

[[email protected] ~]# /etc/munin/plugins/nginx_status
total.value 1
reading.value 0
writing.value 1
waiting.value 0

Enable additional plugins – MySQL
Munin has the ability to graph a number of metrics for MySQL, including commands, caches, buffers, innodb stats, table locks, tmp tables, etc. In order for the plugin to work, Munin will need to be able to access MySQL, and there are also a few dependencies that need to be taken care of. First, install the needed perl modules on the Munin node:

# RHEL / CentOS 6 and 7
[[email protected] ~]# yum install perl-Class-DBI-mysql perl-DBD-mysql libdbi-dbd-mysql

Setup the MySQL user that Munin will use to log into MySQL to pull the stats:

[[email protected] ~]# mysql
mysql> GRANT PROCESS,SUPER on *.* to 'munin'@'127.0.0.1' IDENTIFIED BY 'YOUR_SECURE_PASSWORD_HERE';
mysql> GRANT PROCESS,SUPER on *.* to 'munin'@'localhost' IDENTIFIED BY 'YOUR_SECURE_PASSWORD_HERE';
mysql> flush privileges;
mysql> quit

Setup some configurations within Munin to let it know how to access MySQL:

[[email protected] ~]# vim /etc/munin/plugin-conf.d/munin-node
...
[mysql*]
env.mysqladmin /usr/bin/mysqladmin
env.mysqluser munin
env.mysqlpassword YOUR_SECURE_PASSWORD_HERE
...

[[email protected] ~]# vim /etc/munin/plugin-conf.d/mysql_
[mysql_*]
env.mysqlconnection DBI:mysql:information_schema;host=127.0.0.1;port=3306
env.mysqluser munin
env.mysqlpassword YOUR_SECURE_PASSWORD_HERE
env.cachenamespace munin_mysql_pri 

[[email protected] ~]# vim /etc/munin/plugin-conf.d/mysql_innodb 
[mysql_innodb]
env.warning 0
env.critical 0

Now enable the Munin MySQL plugins:

[[email protected] ~]# for i in `/usr/share/munin/plugins/mysql_ suggest`; do ln -s /usr/share/munin/plugins/mysql_ /etc/munin/plugins/mysql_$i ; done

Restart Munin so the new settings go into effect:

[[email protected] ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data.

Enable additional plugins – Memcached
Munin has the ability to graph commands, current values and network traffic for Memcached. There are also a few dependencies that need to be taken care of. First, install the needed perl modules on the Munin node:

# RHEL / CentOS 6 and 7
[[email protected] ~]# yum install perl-Cache-Memcached

Then setup some configurations within Munin to let it know how to access Memcached:

[[email protected] ~]# vim /etc/munin/plugin-conf.d/munin-node
...
[memcached_*]
env.host 127.0.0.1
env.port 11211
...

Now enable the Munin Memcached plugins:

[[email protected] ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_bytes
[[email protected] ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_counters
[[email protected] ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_rates

Restart Munin so the new setting take effect:

[[email protected] ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data.

Troubleshooting – Connectivity issues

If Munin is not graphing data, it could be because Munin cannot connect to the master server. Some basic things to check would be to ensure Munin is running on the master server and node server. Ensure that port 4949 is opened inbound to the Munin master from the node. Finally on the Munin master, confirm that /etc/munin/munin-node.conf has an allow block for the Munin node server.

You can test connectivity by running the following, substituting localhost out accordingly if your connecting from a Munin node server:

[[email protected] ~]# telnet localhost 4949
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
# munin node at localhost.localdomain

Troubleshooting – See what Munin plugins are enabled

To see which plugins are enabled on your Munin node, you can quickly test for it by issuing the ‘list’ command via telnet:

[[email protected] ~]# telnet localhost 4949
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
# munin node at localhost.localdomain
list
apache_accesses apache_processes apache_volume cpu df df_inode entropy forks fw_conntrack fw_forwarded_local fw_packets if_err_eth0 if_eth0 interrupts irqstats load memcached_bytes memcached_counters memcached_rates memory ...

Troubleshooting – Testing a plugin manually

If a graph is showing up blank, its useful to test the plugin directly to ensure it is able to grab the data. Using the ‘df’ plugin as the example, it can be ran manually by:

[[email protected] ~]# munin-run df
_dev_mapper_VolGroup_lv_root.value 16.9176920622854
_dev_sda1.value 27.6994797122402

Troubleshooting – Plugin not generating graph data

If a plugin is not generating stats within Munin, try running the plugin manually to see if it will return stats on the command line. Equally important, ensure that the application is getting some traffic for the plugin to graph. Finally, keep in mind that it can take up to 15 minutes for a plugin to start graphing, assuming that you see stats when running it manually.

How to setup Munin on Ubuntu

Taken directly from Munin’s website:

Munin is a networked resource monitoring tool that can help analyze resource trends and "what just happened to kill our performance?" problems. It is designed to be very plug and play. A default installation provides a lot of graphs with almost no work.

As the paragraph above suggests, Munin is a really good tool for being able to spot anomalies in the servers performance that may help point a system administrator in the right direction for determining where the bottlenecks are coming into play.

Munin can be installed on each node as a stand alone instance, or you can setup a Munin master server, then simply have the clients (Munin nodes) report back to the Munin master.

If would would prefer to have a stand alone instance of Munin on each node, then follow the instructions for setting up the Munin Master server. This guide is currently making the assumption that you already have Apache setup and working on your servers.

Munin Master Setup

First, we need to install some required packages so the graphs work properly.

# Apache only
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install libcgi-fast-perl libapache2-mod-fcgid libio-all-lwp-perl 
[[email protected] ~]# a2enmod fcgid
[[email protected] ~]# service apache2 restart
[[email protected] ~]# /usr/sbin/apachectl -M | grep -i cgi
fcgid_module (shared)

# Nginx only
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install libcgi-fast-perl liblwp-protocol-https-perl

Now to install the latest stable version of Munin, run the following:

[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install munin

To configure the Munin master, first uncomment the following lines:

[[email protected] ~]# vim /etc/munin/munin.conf
...
dbdir     /var/lib/munin
htmldir   /var/cache/munin/www
logdir    /var/log/munin
rundir    /var/run/munin
tmpldir /etc/munin/templates
...

Now setup the permissions so Munin can access the directory:

[[email protected] ~]# mkdir -p /var/cache/munin/www
[[email protected] ~]# chown munin:munin /var/cache/munin/www

To protect our performance metrics from being seen by the public, protect the web directory by:

[[email protected] ~]# htpasswd -c /etc/munin/munin-htpasswd serverinfo
New password: 
Re-type new password: 
Adding password for user serverinfo

Then uncomment, add or modify the following fields based off the OS and software running:

# Apache - Ubuntu 14.04
[[email protected] ~]# vim /etc/munin/apache.conf
...
#Order allow,deny
#Allow from localhost 127.0.0.0/8 ::1
#Options None
...
AuthUserFile /etc/munin/munin-htpasswd
AuthName "serverinfo"
AuthType Basic
require valid-user
...
[[email protected] ~]# service apache2 restart

# Apache - Ubuntu 16.04
[[email protected] ~]# mv /etc/munin/apache24.conf /etc/munin/apache24.conf.orig
[[email protected] ~]# vim /etc/munin/apache24.conf
Alias /munin /var/cache/munin/www
<Directory /var/cache/munin/www>
 # Require local
 # Require all granted
 AuthUserFile /etc/munin/munin-htpasswd
 AuthName "Munin"
 AuthType Basic
 Require valid-user
 Options None
</Directory>

ScriptAlias /munin-cgi/munin-cgi-graph /usr/lib/munin/cgi/munin-cgi-graph
<Location /munin-cgi/munin-cgi-graph>
 # Require local
 # Require all granted
 AuthUserFile /etc/munin/munin-htpasswd
 AuthName "Munin"
 AuthType Basic
 Require valid-user
 <IfModule mod_fcgid.c>
 SetHandler fcgid-script
 </IfModule>
 <IfModule !mod_fcgid.c>
 SetHandler cgi-script
 </IfModule>
</Location>
[[email protected] ~]# service apache2 restart

# Nginx - Ubuntu 14.04
[[email protected] ~]# vim /etc/nginx/sites-available/default
server {
...
location /munin/static/ {
        alias /etc/munin/static/;
        expires modified +1w;
}
location /munin {
        auth_basic            "Restricted";
        auth_basic_user_file  /etc/munin/munin-htpasswd;
        alias /var/cache/munin/www/;
        expires modified +310s;
}
...
}
[[email protected] ~]# service nginx restart

# Nginx - Ubuntu 16.04
# NOTE:  If you run into issues with the css not working within 
# Munin, you may have to disable the following within the site's
# Nginx config if it exists:
#  location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
#     expires max;
#     log_not_found off;
#  }

[[email protected] ~]# vim /etc/nginx/sites-available/default
server {
...
location /munin/static/ {
        alias /etc/munin/static/;
}
location /munin {
        auth_basic            "Restricted";
        auth_basic_user_file  /etc/munin/munin-htpasswd;
        alias /var/cache/munin/www/;
}

Finally, we need to update the name in the host tree to properly identify the Munin Master. So we will be changing the name from localhost.localdomain to MuninMaster. You could change to this to anything you like, such as a FQDN like web01.domain.com if you like.

[[email protected] ~]# vim /etc/munin/munin.conf
...
[MuninMaster]
    address 127.0.0.1
    use_node_name yes
...

Restart Apache and Munin so the changes register:

[[email protected] ~]# service apache2 restart
[[email protected] ~]# service munin-node restart

Then view your Munin stats by navigating your browser to:

http://serverip/munin

How to add a node to the Munin Master

Here is where we can install additional servers to the Munin Master, so you can view all the graphs from one interface.

First, install the munin-node package on your additional server:

[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install munin-node

Now configure munin-node to accept inbound connections from your Munin Master server. About halfway down in the file, below the allow ^127\.0\.0\.1$, add the IP address of your Munin Master server. For example, if the IP address of the Munin Master is 192.168.1.100, then the entry would be:

[[email protected] ~]# vim /etc/munin/munin-node.conf
...
allow ^127\.0\.0\.1$
allow ^192\.168\.1\.100$
...

Then restart Munin:

[[email protected] ~]# service munin-node restart

Allow the Munin Master server to connect through the firewall:

[[email protected] ~]# ufw allow from 192.168.1.100 to any port 4949

Back on the Munin Master server, we need to tell Munin to query the new Munin Node for information. You can change the FQDN to anything you like, and also be sure to update the IP address to use the IP of your Munin Node:

[[email protected] ~]# vim /etc/munin/munin.conf
...
[web01.domain.com]
    address 192.168.1.200
    use_node_name yes
...

Then restart Munin:

[[email protected] ~]# service munin-node restart

Finally, view your Munin stats by navigating your browser to back to your Munin Master server.

Enable additional plugins

Munin comes preinstalled with a number of plugins, but many are disabled since every server runs different things. There are also extra plugins that may or may not be installed by default. So to get those, run:

[[email protected] ~]# apt-get install munin-plugins-extra

To see what plugins can be used on your server, run:

[[email protected] ~]# munin-node-configure --suggest

Plugin                     | Used | Suggestions                            
------                     | ---- | -----------                            
acpi                       | no   | no [cannot read /proc/acpi/thermal_zone/*/temperature]
amavis                     | no   | no                                     
apache_accesses            | yes  | yes                                    
apache_processes           | no   | no                                     
apache_volume              | yes  | yes
...

To break this down:

- If 'Used' column says 'yes', then that means the plugin is already in use
- If 'Used' column says 'no' and 'Suggestions' says 'no', then the plugin cannot be used.
- If 'Used' column says 'no' and 'Suggestions' says 'yes', then you can enable that plugin

Enabling plugins is pretty easy. Lets say you want to enable the iostat plugin:

[[email protected] ~]# ln -s /usr/share/munin/plugins/iostat /etc/munin/plugins

Then restart Munin:

[[email protected] ~]# service munin-node restart

Enable additional plugins – Apache

Munin has the ability to graph Apache accesses, processes, and overall volume for Apache. In order for this to work, mod_status needs to be enabled with ExtendedStatus set to on, and needs to be accessible from the server at: http://localhost/server-status?auto. An example server-status config is shown below:

# Debian / Ubuntu
[[email protected] ~]# vim /etc/apache2/mods-available/status.conf

<IfModule mod_status.c>
#
# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
ExtendedStatus On

# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Uncomment and change the ".example.com" to allow
# access from other hosts.
#
<Location /server-status>
     SetHandler server-status
     Order deny,allow
     Deny from all
     Allow from localhost ip6-localhost 127.0.0.1
     <IfModule mod_rewrite.c>
          RewriteEngine off
     </IfModule>

     # AuthUserFile /etc/httpd/status-htpasswd
     # AuthName "Password protected"
     # AuthType Basic
     # Require valid-user

     # Allow password-less access for allowed IPs
     Satisfy any

</Location>

</IfModule>

[[email protected] ~]# a2enmod status
[[email protected] ~]# service apache2 restart

Now test to ensure the server-status module is working:

[[email protected] ~]# curl localhost/server-status
...
cache type: SHMCB, shared memory: 512000 bytes, current sessions: 0
subcaches: 32, indexes per subcache: 133
index usage: 0%, cache usage: 0%
total sessions stored since starting: 0
total sessions expired since starting: 0
total (pre-expiry) sessions scrolled out of the cache: 0
total retrieves since starting: 0 hit, 0 miss
total removes since starting: 0 hit, 0 miss
</td></tr> </table> </body></html>

Then confirm that the Munin plugin works by running the plugin manually. Confirm the output says ‘yes’ as shown below:

[[email protected] ~]# /usr/share/munin/plugins/apache_accesses autoconf
yes
[[email protected] ~]# /usr/share/munin/plugins/apache_processes autoconf
yes
[[email protected] ~]# /usr/share/munin/plugins/apache_volume autoconf
yes

Now enable the Apache plugins for Munin and restart the service:

[[email protected] ~]# ln -s /usr/share/munin/plugins/apache_accesses /etc/munin/plugins/
[[email protected] ~]# ln -s /usr/share/munin/plugins/apache_processes /etc/munin/plugins/
[[email protected] ~]# ln -s /usr/share/munin/plugins/apache_volume /etc/munin/plugins/
[[email protected] ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if they are now registering the new data. Otherwise you can test the plugins manually to confirm they are returning stats by:

[[email protected] ~]# /etc/munin/plugins/apache_accesses
accesses80.value 1300
[[email protected] ~]# /etc/munin/plugins/apache_processes
busy80.value 1
idle80.value 8
free80.value 58
[[email protected] ~]# /etc/munin/plugins/apache_volume
volume80.value 5093376

Enable additional plugins – Nginx

Munin has the ability to graph Nginx requests and nginx status for Nginx. In order for this to work, the Nginx stub_status module needs to be enabled so the following page will work: http://localhost/nginx_status. An example nginx_status configuration is shown below:

[[email protected] ~]# vim /etc/nginx/conf.d/nginx_status.conf
server {

    listen 80 default_server;
    access_log off;
    server_name _;
    server_name_in_redirect off;
    root  /var/www/html;

        location /nginx_status {
                # Enable Nginx stats
                stub_status on;
                # Disable logging for stats
                access_log   off;
                # Security: Only allow access from authorized IP's
                allow 127.0.0.1;
                # allow xx.xx.xx.xx;
                # Deny everyone else
                deny all;
        }

}
[[email protected] ~]# service nginx restart

Now test to ensure the stub_status module is working with curl:

[[email protected] ~]# curl localhost/nginx_status
Active connections: 1 
server accepts handled requests
 1 1 1 
Reading: 0 Writing: 1 Waiting: 0

Then confirm the Munin plugin works by running the plugin manually. Confirm the output says ‘yes’ as shown below:

[[email protected] ~]# /usr/share/munin/plugins/nginx_request autoconf
yes
[[email protected] ~]# /usr/share/munin/plugins/nginx_status autoconf
yes

Enable the Nginx plugins for Munin and restart the service:

[[email protected] ~]# ln -s /usr/share/munin/plugins/nginx_request /etc/munin/plugins/
[[email protected] ~]# ln -s /usr/share/munin/plugins/nginx_status  /etc/munin/plugins/
[[email protected] ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data. Otherwise you can test the plugins manually to confirm they are returning stats by:

[[email protected] ~]# /etc/munin/plugins/nginx_request
request.value 3

[[email protected] ~]# /etc/munin/plugins/nginx_status
total.value 1
reading.value 0
writing.value 1
waiting.value 0

Enable additional plugins – MySQL
Munin has the ability to graph a number of metrics for MySQL, including commands, caches, buffers, innodb stats, table locks, tmp tables, etc. In order for the plugin to work, Munin will need to be able to access MySQL, and there are also a few dependencies that need to be taken care of. First, install the needed perl modules on the Munin node:

# Debian / Ubuntu
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install libclass-dbi-mysql-perl libdbd-mysql-perl libdbi-perl libcache-perl libcache-cache-perl

Now enable the Munin MySQL plugins:

[[email protected] ~]# for i in `/usr/share/munin/plugins/mysql_ suggest`; do ln -s /usr/share/munin/plugins/mysql_ /etc/munin/plugins/mysql_$i ; done

Restart Munin so the new settings go into effect:

[[email protected] ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data.

Enable additional plugins – Memcached
Munin has the ability to graph commands, current values and network traffic for Memcached. There are also a few dependencies that need to be taken care of. First, install the needed perl modules on the Munin node:

# Debian / Ubuntu
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install libcache-memcached-perl

Then setup some configurations within Munin to let it know how to access Memcached:

[[email protected]e01 ~]# vim /etc/munin/plugin-conf.d/munin-node
...
[memcached_*]
env.host 127.0.0.1
env.port 11211
...

Now enable the Munin Memcached plugins:

[[email protected] ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_bytes
[[email protected] ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_counters
[[email protected] ~]# ln -s /usr/share/munin/plugins/memcached_ /etc/munin/plugins/memcached_rates

Restart Munin so the new setting take effect:

[[email protected] ~]# service munin-node restart

Finally, wait about 5-10 minutes for the stats to be generated then check the Munin graphs to see if the graphs are now registering the new data.

Troubleshooting – Connectivity issues

If Munin is not graphing data, it could be because Munin cannot connect to the master server. Some basic things to check would be to ensure Munin is running on the master server and node server. Ensure that port 4949 is opened inbound to the Munin master from the node. Finally on the Munin master, confirm that /etc/munin/munin-node.conf has an allow block for the Munin node server.

You can test connectivity by running the following, substituting localhost out accordingly if your connecting from a Munin node server:

[[email protected] ~]# telnet localhost 4949
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
# munin node at localhost.localdomain

Troubleshooting – See what Munin plugins are enabled

To see which plugins are enabled on your Munin node, you can quickly test for it by issuing the ‘list’ command via telnet:

[[email protected] ~]# telnet localhost 4949
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
# munin node at localhost.localdomain
list
apache_accesses apache_processes apache_volume cpu df df_inode entropy forks fw_conntrack fw_forwarded_local fw_packets if_err_eth0 if_eth0 interrupts irqstats load memcached_bytes memcached_counters memcached_rates memory ...

Troubleshooting – Testing a plugin manually

If a graph is showing up blank, its useful to test the plugin directly to ensure it is able to grab the data. Using the ‘df’ plugin as the example, it can be ran manually by:

[[email protected] ~]# munin-run df
_dev_mapper_VolGroup_lv_root.value 16.9176920622854
_dev_sda1.value 27.6994797122402

Troubleshooting – Plugin not generating graph data

If a plugin is not generating stats within Munin, try running the plugin manually to see if it will return stats on the command line. Equally important, ensure that the application is getting some traffic for the plugin to graph. Finally, keep in mind that it can take up to 15 minutes for a plugin to start graphing, assuming that you see stats when running it manually.

Scrutiny

Being asked at 9AM to determine what caused a system to have problems at 2:30AM can be a weary task. If the normal system logs do not give us any real hints about what may have caused the issue, we oftentimes get trapped having to give the really poor answer of “We cannot replicate the issue that you experienced during the overnight, and the logs are not giving us enough information to go on. So we’ll have to watch for it tonight to see if it re-occurs.” Times like that makes a sysadmin feel completely helpless.

What if you could see what processes were running on the system at prescribed intervals? And not just processes, but what about what queries were running, how many people were hitting Apache, perhaps what types of network connections you were getting, on top of a bunch of other information that can be gathered from tools like vmstat, iostat, etc? Now you can draw better conclusions cause you will know what was happening at that single point in time.

Welcome Scrutiny! Located over on github. A tool based off of recap, rewriten to suit my own needs for portability between Red Hat, Debian, and FreeBSD based systems, as well as allowing for simple modifications of the metrics needed to best suit your own environment.

Features

– Simple code base for quick customizations
– Ability to enable/disable groups of checks
– Easy to add/modify/remove individual metric gathering
– Uses tools such as ps, top, df, vmstat, iostat, netstat, mysqladmin, and apache’s server-status module to help create a point in time snapshot of the systems events.

Configuration

The currently configurable options and thresholds are listed below:

# Enable / Disable Statistics
process_log=on
resource_log=on
network_log=on
mysql_log=on
apache_log=on

# Retention Days
retension=2

# Logs
basedir=/var/log/scrutiny

Implementation

Download script to desired directory and set it to be executable:

# Linux based systems
cd /root
git clone https://github.com/stephenlang/scrutiny/linux/scrutiny.sh

# FreeBSD based systems
git clone https://github.com/stephenlang/scrutiny/freebsd/scrutiny.sh
chmod 755 scrutiny.sh

After configuring the tunables in the script (see above), create a cron job to execute the script every 10 minutes:

crontab -e
*/10 * * * * /root/scrutiny.sh

Now days later, if a problem was reported during the overnight and you were able to narrow it down to a specific timeframe, you will be able to look at the point in time snapshots of system events that occurred:

ls /var/log/scrutiny

Server monitoring script

Without an agent based monitoring system, monitoring your servers internals for items such as CPU, memory, storage, processes, etc becomes very difficult without manually checking. There are many reputable monitoring services on the web such as Pingdom (www.pingdom.com), and most hosting providers provide a monitoring system, but they do not provide an agent. Therefore, you can only do basic external checks such as ping, port, and http content checks. There is no way to report if your MySQL replication has failed, some critical process has stopped running, or if your about to max out your / partition.

This simple bash script located on github, is meant to compliment these types of monitoring services. Just drop the script into a web accessible directory, configure a few options and thresholds, setup a URL content check that looks at the status page searching for the string ‘OK’, and then you can rest easy at night that your monitoring service will alert you if any of the scripts conditions are triggered.

Security note: To avoid revealing information about your system, it is strongly recommended that you place this and all web based monitoring scripts behind a htaccess file that has authentication, whitelisting your monitoring servers IP addresses if they are known.

Features

– Memory Check
– Swap Check
– Load Check
– Storage Check
– Process Check
– Replication Check

Configuration

The currently configurable options and thresholds are listed below:

 # Status page
status_page=/var/www/system-health-check.html

# Enable / Disable Checks
memory_check=off
swap_check=on
load_check=on
storage_check=on
process_check=on
replication_check=off

# Configure partitions for storage check
partitions=( / )

# Configure process(es) to check
process_names=( httpd mysqld postfix )

# Configure Thresholds
memory_threshold=99
swap_threshold=80
load_threshold=10
storage_threshold=80

Implementation

Download script to desired directory and set it to be executable:

 cd /root
git clone https://github.com/stephenlang/system-health-check
chmod 755 system-health-check/system-health-check.sh

After configuring the tunables in the script (see above), create a cron job to execute the script every 5 minutes:

 crontab -e
*/5 * * * * /root/system-health-check/system-health-check.sh

Now configure a URL content check with your monitoring providers tools check the status page searching for the string “OK”. Below are two examples:

 http://1.1.1.1/system-health-check.html
http://www.example.com/system-health-check.html

Testing

It is critical that you test this monitoring script before you rely on it. Bugs always exist somewhere, so test this before you implement it on your production systems! Here are some basic ways to test:

1. Configure all the thresholds really low so they will create an alarm. Manually run the script or wait for the cronjob to fire it off, then check the status page to see if it reports your checks are now in alarm.

2. To test out the process monitoring (assuming the system is not in production), configure the processes you want the script to check, then stop the process you are testing, and check the status page after the script runs to see if it reports your process is not running.

3. To test out the replication monitoring (assuming the system is not in production), log onto your MySQL slave server and run ‘stop slave;’. Then check the status page after the script runs to see if it reports an error on replication.