Securing a site with Let’s Encrypt SSL certificates

Let’s Encrypt is a free, automated and open certificate authority for anyone that wants to secure a website with SSL. I recently had to setup Let’s Encrypt for a client, and found that it was absurdly simple to use with their Certbot ACME client.

WARNING: This guide may become quickly outdated and is really just for my own reference. If you are looking to use Let’s Encrypt, please review the following articles from Let’s Encrypt for the latest installation and setup instructions:
https://letsencrypt.org/getting-started/
https://certbot.eff.org

For this guide, I am assuming the server is running Apache. So to get started, I simply following the instructions provided on https://certbot.eff.org to get Certbot installed:

# CentOS 6
# There is currently no packaged version of Certbot for CentOS 6.  So you have to download the script manually by:
[[email protected] ~]# cd /root
[[email protected] ~]# wget https://dl.eff.org/certbot-auto
[[email protected] ~]# chmod a+x certbot-auto

# CentOS 7
[[email protected] ~]# yum install yum-utils
[[email protected] ~]# yum-config-manager --enable rhui-REGION-rhel-server-extras rhui-REGION-rhel-server-optional
[[email protected] ~]# yum install certbot-apache

# Ubuntu 14.04
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install software-properties-common
[[email protected] ~]# add-apt-repository ppa:certbot/certbot
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install python-certbot-apache 

# Ubuntu 16.04
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install software-properties-common
[[email protected] ~]# add-apt-repository ppa:certbot/certbot
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install python-certbot-apache

The command below will install or update the certbot script, and also modify your Apache configs accordingly as it automatically configures the SSL certificate. When you run the tool, it will ask you for your email address, review their terms of service, and will ask you to select which URL’s you want to have the SSL certificate generated for. Always be sure to include both the www and non-www domains unless you don’t need one of them for some reason.

[[email protected] ~]# certbot --apache

One of the great things about Let’s Encrypt certificates, asides the fact its free, is that you can add a cron job to automatically renew the SSL certificate so it doesn’t expire. Let’s Encrypt recommends running it twice daily. It won’t do anything until your certificates are due for renewal or revoked. Setup the cron job by running:

# CentOS 6
[[email protected] ~]# crontab -e
0 12/24 * * * /root/certbot-auto renew

# All other OS's:
[[email protected] ~]# crontab -e
0 12/24 * * * certbot renew

CVE-2016-2183 SWEET32 Birthday attacks

Lately, vulnerability scanners have been flagging servers that are susceptible to CVE-2016-2183. In a nutshell, you need to disable any TLS ciphers using 3DES. More detailed information about this vulnerability and why it exists can be found at the links below:
https://access.redhat.com/articles/2548661
https://sweet32.info

Mitigating this vulnerability within Apache is pretty straight forward. Below are the steps to confirm if you are actually affected by this vulnerability and how to remediate it.

First, confirm your Apache web server is actually vulnerable to this by seeing if the 3DES ciphers are returned in this nmap test:

[[email protected] ~]# nmap --script ssl-enum-ciphers -p 443 SERVER_IP_ADDRESS_HERE
Starting Nmap 5.51 ( http://nmap.org ) at 2016-11-30 17:57 EST
Nmap scan report for xxxxxxxx (xxx.xxx.xxx.xxx)
Host is up (0.018s latency).
PORT    STATE SERVICE
443/tcp open  https
| ssl-enum-ciphers: 
|   TLSv1.2
|     Ciphers (14)
|       TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
|       TLS_RSA_WITH_3DES_EDE_CBC_SHA
|       TLS_RSA_WITH_AES_128_CBC_SHA
|       TLS_RSA_WITH_AES_128_CBC_SHA256
|       TLS_RSA_WITH_AES_128_GCM_SHA256
|       TLS_RSA_WITH_AES_256_CBC_SHA
|       TLS_RSA_WITH_AES_256_CBC_SHA256
|       TLS_RSA_WITH_AES_256_GCM_SHA384
|     Compressors (1)
|_      uncompressed

As you can see in the output above, this server is affected by this vulnerability as its allowing for the following 3DES ciphers:

TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA

Disabling this in Apache is pretty easy. Simply navigate to where ever you have your SSLCipherSuite configuration defined and disable 3DES. Typically this should be in /etc/httpd/conf.d/ssl.conf, however some may also have this defined in each individual Apache vhost. If you are unsure where its configured, you should be able to locate it on your server by running:

# CentOS / Red Hat
[[email protected] ~]# egrep -R SSLCipherSuite /etc/httpd/*

# Ubuntu / Debian
[[email protected] ~]# egrep -R SSLCipherSuite /etc/apache2/*

Once you locate the config(s) that contain this directive, you simple add !3DES to the end of the SSLCipherSuite line as shown below:

[[email protected] ~]# vim /etc/httpd/conf.d/ssl.conf
...
SSLCipherSuite EECDH+AESGCM:EECDH+AES256:EECDH+AES128:EDH+AES:RSA+AESGCM:RSA+AES:!ECDSA:!NULL:!MD5:!DSS:!3DES
...

Once that is done, restart Apache by:

# CentOS / Red Hat
[[email protected] ~]# service httpd restart

# Ubuntu / Debian
[[email protected] ~]# service apache2 restart

Finally, retest using nmap to confirm no ciphers using 3DES show up:

[[email protected] ~]# nmap --script ssl-enum-ciphers -p 443 SERVER_IP_ADDRESS_HERE
Starting Nmap 5.51 ( http://nmap.org ) at 2016-11-30 18:03 EST
Nmap scan report for xxxxxxxx (xxx.xxx.xxx.xxx)
Host is up (0.017s latency).
PORT    STATE SERVICE
443/tcp open  https
| ssl-enum-ciphers: 
|   TLSv1.2
|     Ciphers (12)
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
|       TLS_RSA_WITH_AES_128_CBC_SHA
|       TLS_RSA_WITH_AES_128_CBC_SHA256
|       TLS_RSA_WITH_AES_128_GCM_SHA256
|       TLS_RSA_WITH_AES_256_CBC_SHA
|       TLS_RSA_WITH_AES_256_CBC_SHA256
|       TLS_RSA_WITH_AES_256_GCM_SHA384
|     Compressors (1)
|_      uncompressed

If no 3DES ciphers are returned like in the listing above, you should be good to rerun your vulnerability scan!

Setup ClamAV for nightly scans

PCI-DSS 3.1 Requirement 5 states the following:

Protect all systems against malware and regularly update anti-virus software or programs.

There are commercial based solutions out there for Linux based systems, but costs can become an issue, especially for small companies with a small footprint within their card holder data environment (CDE). So can one satify this requirement without breaking the bank? I personally prefer ClamAV.

Taken from the projects website, ClamAV is an open source antivirus engine for detecting trojans, viruses, malware and other malicious threats.

My requirements:
1. I want to scan my entire system nightly.
2. All virus reports are emailed to me so I can archive them for a year offsite.
3. Have the antivirus definitions updated nightly before the scan.

Installing, running and maintaining ClamAV is very straight forward on Linux based systems. To get started, install ClamAV by:

# CentOS 6 / RedHat 6
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[[email protected] ~]# yum install clamav mailx

# CentOS 7 / RedHat 7
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
[[email protected] ~]# yum install clamav clamav-update mailx
[[email protected] ~]# sed -i '/^Example/d' /etc/freshclam.conf

# Ubuntu 12.04 / Ubuntu 14.04
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install clamav mailutils

Now update the virus definitions by running:

[[email protected] ~]# freshclam

Finally, configure the virus definitions to update nightly, and also scan the entire system and email a report:

[[email protected] ~]# crontab -e
00 2 * * *  /usr/bin/freshclam
00 3 * * * /usr/bin/clamscan -r -i / | mail -s "ClamAV Report : INSERT_SERVER_HOSTNAME_HERE" [email protected]

Posted below is an example report ClamAV would send me via email nightly:

----------- SCAN SUMMARY -----------
Known viruses: 4289299
Engine version: 0.99
Scanned directories: 51929
Scanned files: 808848
Infected files: 0
Total errors: 10982
Data scanned: 76910.89 MB
Data read: 83578.27 MB (ratio 0.92:1)
Time: 6641.424 sec (110 m 41 s)

How does one go about testing ClamAV to ensure its working? There is a known antivirus test file that was designed specifically for this purpose by www.eicar.org. To create this file, simply setup the following test file, then rerun your ClamAV scan:

[[email protected] ~]# vim /tmp/EICAR-AV-Test
...
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
...

Using AIDE for file integrity monitoring (FIM) on Ubuntu or Debian

PCI-DSS 3.1 section 10.5.5 has the following requirement:

Use file-integrity monitoring or change-detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert).

For large solutions, I would suggest using a well known tool such as Tripwire Enterprise. However many small to mid size companies that have a small footprint within their card holder data environment (CDE), may not be able to afford this. So what can companies use to meet this requirement? Implement AIDE (Advanced Intrusion Detection Environment).

Taken from the projects website, AIDE creates a database from the regular expression rules that it finds from the config file(s). Once this database is initialized it can be used to verify the integrity of the files.

AIDE is a very simple (yet powerful) program that runs from cron checking your files (typically once a night), and it will scan your system looking for any changes in the directories its monitoring. There are a number of different ways to use this program, but I’ll outline one that I like to use.

My requirements:
1. I want the reports to run nightly.
2. All change reports are emailed to me so I can archive them for a year offsite.
3. Have the database automatically commit the additions, deletions, and changes to baseline each time its ran.

In the event my system was compromised, I want to ensure that the malicious user was not able to modify, or delete my previous reports. Therefore, I choose not to store them on the machine. Its true that once the malicious user gained access to my system, they could change my AIDE config on me, but at least my previous reports will be intact which should help me when determining what malicious changes this user made to my server. Please note that I am making an assumption here that you are already backing up your system nightly, which would include your AIDE database! If you do not currently have a backup strategy in place, get one. Tools such as AIDE helps identify what files a malicious user may have changed, but if they completely crippled the system, you will need to restore from backups.

Setting up AIDE is fairly straight forward. It exists in most of package repositories out there including most variants of Linux and BSD.

On Ubuntu or Debian based systems, you can install it by:

[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install aide

Now to setup some basic configurations, such as the email notifications, update type, etc, modify the AIDE system configuration file according:

[[email protected] ~]# vim /etc/default/aide
...
FQDN=web01.domain.com
MAILSUBJ="Daily AIDE report for $FQDN"
[email protected]
QUIETREPORTS=no
COMMAND=update
COPYNEWDB=yes
...

Now that AIDE is installed, and the basic preferences are in place, its now time to check out the main configuration files. The default configuration from the upstream provider should give you a reasonable default configuration. But what if you wanted to add your website documentroot to this so you can keep track of what files are changing on your website? The Debian/Ubuntu way of configuring AIDE is a bit different from the CentOS/RHEL method.

All the configuration files resides in /etc/aide/aide.conf.d/. The number of the file appears to be used by the AIDE wrapper to use for deciding which order to process these files. The AIDE documentation seems to indicate that the most general rules should be processed last, so I’ll default to creating my servers profile with 50_aide_CUSTOM-RULES.

So lets say I want to monitor my documentroot, here is how this would be setup:

[[email protected] ~]# vim /etc/aide/aide.conf.d/50_aide_CUSTOM-RULES
...
/var/www/vhosts/domain.com FULL
...

Now AIDE will be keeping track of our website. But adding your site may lead to very noisy reports because most websites implement caching. So this now becomes a balancing act to exclude directories that change often, yet retain enough of your sites critical content. We could just leave the entire directory in AIDE, but I know I personally don’t want to read a change report that contains 1,000 changes every day. So in the case of this wordpress site, I exclude the cache directory by appending the following to my custom configuration:

[[email protected] ~]# vim /etc/aide/aide.conf.d/50_aide_CUSTOM-RULES
...
/var/www/vhosts/domain.com Full
!/var/www/vhosts/domain.com/web/wp-content/cache
...

The “!” means NOT to monitor that specific directory. You will need to run AIDE a few times and fine tune the configuration before you get a report that is useful for your specific needs.

Anytime a change is made to your AIDE configuration, you need to rebuild the AIDE run time configuration, and initialize the database. You do that by:

[[email protected] ~]# update-aide.conf
[[email protected] ~]# aideinit -y -f

Now, try making a basic change to /etc/hosts, then run a check on AIDE to see if it detects the change and emails out the report:

[[email protected] ~]# /etc/cron.daily/aide

If you wanted to just quickly test AIDE to ensure it picks up your changes, but won’t commit them to baseline, you can perform a one-time scan by:

[[email protected] ~]# aide.wrapper

To receive nightly AIDE reports, no further configuration is needed since Ubuntu/Debian already setup a cron job that will run AIDE automatically in /etc/cron.daily/aide. This will run whenever your system normally runs the cron.daily jobs, which is defined in /etc/crontab.

Posted below is an example report that AIDE would send me via email daily:

This is an automated report generated by the Advanced Intrusion Detection 
Environment on web01.domain.com started at 2016-03-07 13:16:35.

AIDE returned with exit code 7. Added, removed and changed files detected!
AIDE post run information
output database /var/lib/aide/aide.db.new was copied to /var/lib/aide/aide.db as requested by cron job configuration
End of AIDE post run information

AIDE produced no errors.

Output of the daily AIDE run (83 lines):
AIDE 0.15.1 found differences between database and filesystem!!
Start timestamp: 2016-03-07 13:16:35

Summary:
  Total number of files:	77937
  Added files:			2
  Removed files:		3
  Changed files:		7


---------------------------------------------------
Added files:
---------------------------------------------------

f++++++++++++++++: /var/log/aide/aide.log.0
d++++++++++++++++: /var/www/vhosts/domain.com/new

---------------------------------------------------
Removed files:
---------------------------------------------------

f----------------: /var/www/vhosts/domain.com/blah
f----------------: /var/www/vhosts/domain.com/test
d----------------: /var/www/vhosts/domain.com/test1

---------------------------------------------------
Changed files:
---------------------------------------------------

f   p.g    . A. .: /var/log/aide/aide.log
d =.... mc.. .. .: /var/spool/postfix/active
d =.... mc.. .. .: /var/spool/postfix/incoming
d =.... mc.. .. .: /var/spool/postfix/maildrop
F =.... mc.. ..  : /var/spool/postfix/public/pickup
F =.... mc.. ..  : /var/spool/postfix/public/qmgr
d =.... mc.. .. .: /var/www/vhosts/domain.com

---------------------------------------------------
Detailed information about changes:
---------------------------------------------------


File: /var/log/aide/aide.log
 Perm     : -rw-------                       , -rw-r-----
 Gid      : 0                                , 4
 ACL      : old = A:
----
user::rw-
group::---
other::---
----
                  D: 
            new = A:
----
user::rw-
group::r--
other::---
----
                  D: 

Directory: /var/spool/postfix/active
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23

Directory: /var/spool/postfix/incoming
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23

Directory: /var/spool/postfix/maildrop
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23

FIFO: /var/spool/postfix/public/pickup
 Mtime    : 2016-03-07 13:12:37              , 2016-03-07 13:17:37
 Ctime    : 2016-03-07 13:12:37              , 2016-03-07 13:17:37

FIFO: /var/spool/postfix/public/qmgr
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:36
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:36

Directory: /var/www/vhosts/domain.com
 Mtime    : 2016-03-07 13:03:25              , 2016-03-07 13:16:17
 Ctime    : 2016-03-07 13:03:25              , 2016-03-07 13:16:17

End of AIDE output.

The check was done against /var/lib/aide/aide.db with the following characteristics:
 Size     : 13041865
 Bcount   : 25480
 Mtime    : 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:13:23
 Inode    : 273628
 RMD160   : bIthG3Q5FiJmj4CIYdASjJx5Ygc=
 TIGER    : omto0nb3/oIqIiKHEjnbhjvXeGdfycbV
 SHA256   : VJPGKy61GxGfcSrjJFbrP879y/skJaiQ
 SHA512   : 7pz3FdYh8TvoNOqjxWBToZQNG6oxmrrp
 CRC32    : 1dYwqA==
 HAVAL    : LBFzyApqoYn7ogzoROG5FpneBO1s7R3p
 GOST     : iJ1tWPLtYaxxoFDHZEW8gxCS3/pVlS1G

The AIDE run created a new database /var/lib/aide/aide.db.new with the following characteristics:
 Size     : 13041834
 Bcount   : 25480
 Inode    : 273627
 RMD160   : 4TKRFSc0nt/VGDVvPEY8U6YNzaw=
 TIGER    : o4RzDHHWBlH+Zt3P7vI8GHHgGV1OecrC
 SHA256   : Gher/aINaU8r73/lQEWLQQSsKqP7sGjO
 SHA512   : D0/w3S6NOLZHw7D7dt1QxYBXe6miP5hF
 CRC32    : 5SRdpg==
 HAVAL    : pe7+ai57TPpW34NjJgTQxs+cQsFJ9zq0
 GOST     : RrIiyspbpKEb5wEGSG2HTYM7N6NUtKSv

End of AIDE daily cron job at 2016-03-07 13:18, run time 102 seconds

So this reports tells me that a log file for AIDE was rotated out, a new folder was created in my DocumentRoot called new, and the files/folders blah, test, and test1 where removed from my DocumentRoot.

Please remember that utilizing a tool to provide file integrity monitoring is only one part of a defense in depth strategy. There is no silver bullet for system security, but every layer you add will increase your security footprint which helps you with taking a proactive approach to security.

Automatic package updates on CentOS 7

Keeping on top of your systems security updates is one important step in a defense in depth strategy for your solution. Security best practices indicate security updates should be applied within 30 days of being released.

While I prefer to test any and all updates in a test environment before deploying to production just in case the updates break existing functionality, some system administrators prefer to have the updates applied automatically nightly or weekly. The school of thought behind this is automatic updates are applied quickly, more often, and generally more consistently than they would be if done manually.

Also, its important to ensure that you are excluding any packages that may be critical to your system, such as the kernel or perhaps MySQL. The package exclusion list is provided in the example below.

Please keep in mind, in rare cases, system updates have been known to cause problems. So you should be cautious with any type of automatic updates, especially on production systems, and fully understand the risks involved before proceeding.

To enable automatic updates, first update yum, then install the yum-cron package:

[[email protected] ~]# yum -y update yum
[[email protected] ~]# yum -y install yum-cron
[[email protected] ~]# systemctl enable yum-cron
[[email protected] ~]# systemctl start yum-cron

The configuration is pretty simplified on CentOS 7 as shown below:

[[email protected] ~]# vim /etc/yum/yum-cron.conf
...
#  What kind of update to use:
# default                            = yum upgrade
# security                           = yum --security upgrade
# security-severity:Critical         = yum --sec-severity=Critical upgrade
# minimal                            = yum --bugfix update-minimal
# minimal-security                   = yum --security update-minimal
# minimal-security-severity:Critical =  --sec-severity=Critical update-minimal
update_cmd = default

# Whether a message should be emitted when updates are available,
# were downloaded, or applied.
update_messages = yes

# Whether updates should be downloaded when they are available.
download_updates = yes

# Whether updates should be applied when they are available.  Note
# that download_updates must also be yes for the update to be applied.
apply_updates = yes

# The address to send email messages from.
email_from = [email protected]

# List of addresses to send messages to.
email_to = [email protected]
...

If you would like to exclude specific packages from receiving automatic updates, you can add exclusions. In this example, we are excluding updates to the mysql and the kernel. Please note this must be done in the ‘base’ section of the configuration as shown below:

[[email protected] ~]# vim /etc/yum/yum-cron.conf
...
[base]
# This section overrides yum.conf
exclude=mysql* kernel*
...

After you make your changes to the configuration file, restart yum-cron by:

[[email protected] ~]# systemctl restart yum-cron

Once that is complete, no further configuration should be needed as yum-cron will run when the daily cron jobs are set to run.

If you find that you need to roll back a package update, you can do that by:

[[email protected] ~]# yum history
Loaded plugins: fastestmirror
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
     9 | root               | 2016-02-09 17:47 | Install        |    1   
     8 | root               | 2016-02-09 17:47 | Update         |    1   
     7 | root               | 2015-08-18 03:19 | I, O, U        |  189 EE
     6 | root               | 2015-03-01 16:44 | Install        |    1   
     5 | root               | 2015-03-01 16:31 | Erase          |    1   
     4 | root               | 2015-03-01 16:30 | Install        |   49   
     3 | root               | 2015-03-01 16:28 | Install        |    1   
     2 | root               | 2015-03-01 16:25 | I, U           |   80   
     1 | System            | 2015-03-01 15:52 | Install        |  298  
[[email protected] ~]# yum history undo ID

Automatic package updates on CentOS 6

Keeping on top of your systems security updates is one important step in a defense in depth strategy for your solution. Security best practices indicate security updates should be applied within 30 days of being released.

While I prefer to test any and all updates in a test environment before deploying to production just in case the updates break existing functionality, some system administrators prefer to have the updates applied automatically nightly or weekly. The school of thought behind this is automatic updates are applied quickly, more often, and generally more consistently than they would be if done manually.

Also, its important to ensure that you are excluding any packages that may be critical to your system, such as the kernel or perhaps MySQL. Configuring the package exclusion list is shown below.

Please keep in mind, in rare cases, system updates have been known to cause problems. So you should be cautious with any type of automatic updates, especially on production systems, and fully understand the risks involved before proceeding.

To enable automatic updates, first install the yum-cron package:

[[email protected] ~]# yum -y install yum-cron
[[email protected] ~]# chkconfig yum-cron on
[[email protected] ~]# service yum-cron start

The main configuration file resides in /etc/sysconfig/yum-cron. Some common options for configuring how yum-cron works is documented below:

# Default - Check for updates, download and install:
[[email protected] ~]# vim /etc/sysconfig/yum-cron
...
CHECK_ONLY=no
DOWNLOAD_ONLY=no
...

# Download only and send an email report:
[[email protected] ~]# vim /etc/sysconfig/yum-cron
...
[email protected]
CHECK_ONLY=no
DOWNLOAD_ONLY=yes
...
 
# Send email report only, but do not download or install:
[[email protected] ~]# vim /etc/sysconfig/yum-cron
...
[email protected]
CHECK_ONLY=yes
DOWNLOAD_ONLY=no
...

You can also configure this to apply updates on specific days. This example below will check for updates, download and apply them on Sundays:

[[email protected] ~]# vim /etc/sysconfig/yum-cron
...
CHECK_ONLY=no
DOWNLOAD_ONLY=no
DAYS_OF_WEEK="0"
...

If you would like to exclude specific packages from receiving automatic updates, you can add exclusions. In this example, we are excluding updates to the mysql and the kernel. Please be sure to pay close attention to the single quotes and double quotes!

[[email protected] ~]# vim /etc/sysconfig/yum-cron
...
YUM_PARAMETER="--exclude='mysql*' --exclude='kernel*'"
...

Once that is complete, no further configuration should be needed as yum-cron will run when the daily cron jobs are set to run.

If you find that you need to roll back a package update, you can do that by:

[[email protected] ~]# yum history
Loaded plugins: fastestmirror
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
     9 | root               | 2016-02-09 17:47 | Install        |    1   
     8 | root               | 2016-02-09 17:47 | Update         |    1   
     7 | root               | 2015-08-18 03:19 | I, O, U        |  189 EE
     6 | root               | 2015-03-01 16:44 | Install        |    1   
     5 | root               | 2015-03-01 16:31 | Erase          |    1   
     4 | root               | 2015-03-01 16:30 | Install        |   49   
     3 | root               | 2015-03-01 16:28 | Install        |    1   
     2 | root               | 2015-03-01 16:25 | I, U           |   80   
     1 | System            | 2015-03-01 15:52 | Install        |  298  
[[email protected] ~]# yum history undo ID

Automatic package updates on Ubuntu

Keeping on top of your systems security updates is one important step in a defense in depth strategy for your solution. Security best practices indicate security updates should be applied within 30 days of being released.

While I prefer to test any and all updates in a test environment before deploying to production just in case the updates break existing functionality, some system administrators prefer to have the updates applied automatically nightly or weekly. The school of thought behind this is automatic updates are applied quickly, more often, and generally more consistently than they would be if done manually.

Also, its important to ensure that you are excluding any packages that may be critical to your system, such as the kernel or perhaps MySQL. Configuring the package exclusion list is shown below.

Please keep in mind, in rare cases, system updates have been known to cause problems. So you should be cautious with any type of automatic updates, especially on production systems, and fully understand the risks involved before proceeding.

To enable automatic updates, first install the unattended-upgrades package:

[[email protected] ~]# apt-get install unattended-upgrades

Now run the interactive dialog to configure /etc/apt/apt.conf.d/20auto-upgrades. Be sure to answer ‘Yes’ to automatically download and install stable updates.

[[email protected] ~]# dpkg-reconfigure --priority=low unattended-upgrades

The main configuration files resides in /etc/apt/apt.conf.d/50unattended-upgrades. You have a choice of what types of updates you want. The default are ‘security’ updates only. But your choices are:

security - just security related updates (default)
updates - other updates, for example bug fixes that don't affect security
proposed - "beta" channel for updates before they are considered final (not recommended)
backports - selected feature updates from later Ubuntu releases (not recommended)

So to enable both security updates, and bug fixes, enable them by:

[[email protected] ~]# vim /etc/apt/apt.conf.d/50unattended-upgrades
...
Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}-security";
        "${distro_id}:${distro_codename}-updates";
//      "${distro_id}:${distro_codename}-proposed";
//      "${distro_id}:${distro_codename}-backports";
};
...

If you wish to exclude packages from automatic updates, you can add the following to the configuration:

[[email protected] ~]# vim /etc/apt/apt.conf.d/50unattended-upgrades
...
Unattended-Upgrade::Package-Blacklist {
        "mysql55*";
        "kernel";
};
...

More information on the options available can be found within:

[[email protected] ~]# cat /etc/apt/apt.conf.d/20auto-upgrades
[[email protected] ~]# cat /etc/cron.daily/apt

Setup central log server on CentOS 6

For security purposes, it is best practice to store all critical logs to a secured centralized syslog server, and keep the retention rate set for at least 1 year.

In this example, there are 4 servers:
syslog01 (192.168.1.200)
web01-local (192.168.1.201)
web02-local (192.168.1.202)
web03-local (192.168.1.203)

The goal here is to store all the system’s logs, as well as the individual Apache vhost’s logs to the central log server.

Install and perform initial configuration of rsyslog

On syslog01, confirm rsyslog is installed and running by:

[[email protected] ~]# yum install rsyslog
[[email protected] ~]# chkconfig rsyslog on
[[email protected] ~]# service rsyslog start

Open the firewall to allow inbound connections over port 514 to syslog01 by:

[[email protected] ~]# vim /etc/sysconfig/iptables
...
-A INPUT -s 192.168.1.0/24 -p tcp -m tcp --dport 514 -j ACCEPT
...
[[email protected] ~]# service iptables restart

Finally, create the directory that will be storing the logs:

[[email protected] ~]# mkdir -p /var/log/remote
[[email protected] ~]# chown root:root /var/log/remote
[[email protected] ~]# chmod 700 /var/log/remote

Determine log storage style on central log server

Here is where it needs to be decided how to store the logs on the central logging server. I am unaware of any recommend best practices as there are many ways to go about this. I’ll outline 4 common ways below:
– Option 1: Store each server’s logs in its own directory.
– Option 2: Store each server’s logs in its own directory with natural log rotation
– Option 3: Store each server’s logs in its own directory, broken down by program name
– Option 4: Store all server’s logs in a single file

Option 1: Store each server’s logs in its own directory

This will configure rsyslog to store the servers logs in a directory named after the remote server’s hostname. Log rotation will happen daily via logrotate.

To implement this, first create a backup of the existing /etc/rsyslog.conf, and create a new one as shown below. The parts I added or modified are in bold for reference:

[[email protected] ~]# mv /etc/rsyslog.conf /etc/rsyslog.orig
[[email protected] ~]# vim /etc/rsyslog.conf
# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
# Moved below for RuleSets
#$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
# Moved below for RuleSets
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

$template RemoteHost, "/var/log/remote/%HOSTNAME%/syslog.log"

#### RULES ####

$RuleSet local

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

local0.*                                                /var/log/local0.log
local1.*                                                /var/log/local1.log
local2.*                                                /var/log/local2.log
local3.*                                                /var/log/local3.log
local4.*                                                /var/log/local4.log
local5.*                                                /var/log/local5.log
local6.*                                                /var/log/local6.log

# Log all messages also to the central log directory
*.* ?RemoteHost

# Bind the above ruleset as default
$DefaultRuleset local

# This uses the template above
$RuleSet remote
*.* ?RemoteHost

# Bind the above ruleset for remote logs
$InputUDPServerBindRuleset remote
$UDPServerRun 514
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

Then restart rsyslog:

[[email protected] ~]# service rsyslog restart

Now setup logrotate to rotate out the logs, compress them, and keep them on file for 1 year as shown below:

[[email protected] ~]# vim /etc/logrotate.d/centralrsyslog
/var/log/remote/*/*.log
{
daily
    rotate 365
    missingok
    daily
    compress
    delaycompress
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Finally, test the log rotation by running:

[[email protected] ~]# logrotate -f /etc/logrotate.d/centralrsyslog

Option 2: Store each server’s logs in its own directory with natural log rotation

Both the local and remote system logs will be stored in its own directory as follows:

/var/log/remote/2015-12_syslog01/syslog.conf
/var/log/remote/2015-12_web01-local/syslog.conf
/var/log/remote/2015-12_web02-local/syslog.conf
/var/log/remote/2015-12_web03-local/syslog.conf

As the months change, syslog will automatically create a new folder accordingly for that month, creating a type of natural log rotation.

To implement this, first create a backup of the existing /etc/rsyslog.conf, and create a new one as shown below. The parts I added or modified are in bold for reference:

[[email protected] ~]# mv /etc/rsyslog.conf /etc/rsyslog.orig
[[email protected] ~]# vim /etc/rsyslog.conf
# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
# Moved below for RuleSets
#$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
# Moved below for RuleSets
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

$template RemoteHost, "/var/log/remote/%$YEAR%-%$MONTH%_%HOSTNAME%/syslog.log"

#### RULES ####

$RuleSet local

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

local0.*                                                /var/log/local0.log
local1.*                                                /var/log/local1.log
local2.*                                                /var/log/local2.log
local3.*                                                /var/log/local3.log
local4.*                                                /var/log/local4.log
local5.*                                                /var/log/local5.log
local6.*                                                /var/log/local6.log

# Log all messages also to the central log directory
*.* ?RemoteHost

# Bind the above ruleset as default
$DefaultRuleset local

# This uses the template above
$RuleSet remote
*.* ?RemoteHost

# Bind the above ruleset for remote logs
$InputUDPServerBindRuleset remote
$UDPServerRun 514
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

Then restart rsyslog:

[[email protected] ~]# service rsyslog restart

Option 3: Store each server’s logs in its own directory, broken down by program name

This one will break down the logs to store them in a directory named after the remote server’s hostname, and further more break them down by program name.

To implement this, first create a backup of the existing /etc/rsyslog.conf, and create a new one as shown below. The parts I added or modified are in bold for reference:

[[email protected] ~]# mv /etc/rsyslog.conf /etc/rsyslog.orig
[[email protected] ~]# vim /etc/rsyslog.conf
# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
# Moved below for RuleSets
#$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
# Moved below for RuleSets
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

$template RemoteHost, "/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"

#### RULES ####

$RuleSet local

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

local0.*                                                /var/log/local0.log
local1.*                                                /var/log/local1.log
local2.*                                                /var/log/local2.log
local3.*                                                /var/log/local3.log
local4.*                                                /var/log/local4.log
local5.*                                                /var/log/local5.log
local6.*                                                /var/log/local6.log

# Log all messages also to the central log directory
*.* ?RemoteHost

# Bind the above ruleset as default
$DefaultRuleset local

# This uses the template above
$RuleSet remote
*.* ?RemoteHost

# Bind the above ruleset for remote logs
$InputUDPServerBindRuleset remote
$UDPServerRun 514
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

Then restart rsyslog:

[[email protected] ~]# service rsyslog restart

Now setup logrotate to rotate out the logs, compress them, and keep them on file for 1 year as shown below:

[[email protected] ~]# vim /etc/logrotate.d/centralrsyslog
/var/log/remote/*/*.log
{
daily
    rotate 365
    missingok
    daily
    compress
    delaycompress
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Finally, test the log rotation by running:

[[email protected] ~]# logrotate -f /etc/logrotate.d/centralrsyslog

Option 4: Store all server’s logs in a single file

This is useful is you want to be able to use grep to find the information you need from a single file. Just keep in mind that if you have 10-20+ servers, this log file could get very large, even with a daily log rotation in place.

This will tell rsyslog to store ALL your servers logs in /var/log/remote/syslog.log.

To implement this, first create a backup of the existing /etc/rsyslog.conf, and create a new one as shown below. The parts I added or modified are in bold for reference:

[[email protected] ~]# mv /etc/rsyslog.conf /etc/rsyslog.orig
[[email protected] ~]# vim /etc/rsyslog.conf
# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
# Moved below for RuleSets
#$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
# Moved below for RuleSets
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

$template RemoteHost, "/var/log/remote/syslog.log"

#### RULES ####

$RuleSet local

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

local0.*                                                /var/log/local0.log
local1.*                                                /var/log/local1.log
local2.*                                                /var/log/local2.log
local3.*                                                /var/log/local3.log
local4.*                                                /var/log/local4.log
local5.*                                                /var/log/local5.log
local6.*                                                /var/log/local6.log

# Log all messages also to the central log directory
*.* ?RemoteHost

# Bind the above ruleset as default
$DefaultRuleset local

# This uses the template above
$RuleSet remote
*.* ?RemoteHost

# Bind the above ruleset for remote logs
$InputUDPServerBindRuleset remote
$UDPServerRun 514
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

Then restart rsyslog:

[[email protected] ~]# service rsyslog restart

Now setup logrotate to rotate out the logs, compress them, and keep them on file for 1 year as shown below:

[[email protected] ~]# vim /etc/logrotate.d/centralrsyslog
/var/log/remote/*/*.log
{
daily
    rotate 365
    missingok
    daily
    compress
    delaycompress
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Finally, test the log rotation by running:

[[email protected] ~]# logrotate -f /etc/logrotate.d/centralrsyslog

Setup clients to forward their logs to your central log server

For this, I am opting to use TCP. Append the following at the bottom of /etc/rsyslog.conf on each client server:

[[email protected] ~]# vim /etc/rsyslog.conf
*.* @@192.168.1.200:514

Then restart Rsyslog:

[[email protected] ~]# service rsyslog restart

Apache logs

If there was a requirement within the organization to also send Apache error logs access logs to the central syslog server, that can be done pretty easily. However as these can get very large, make sure you have enough disk space on your central syslog server to handle this.

Apache does not have the ability to natively send over the logs, so logger must be used. The following would need to be done on each vhost configured for both 80 and 443. Be sure to change the tags for each site accordingly in the logger entry below:

[[email protected] ~]# vim /etc/httpd/vhost.d/example.com.conf
...
CustomLog "|/usr/bin/logger -p local0.info -t example.com-access" combined
ErrorLog "|/usr/bin/logger -p local0.error -t example.com-error"
...

Restart Apache when done

[[email protected] ~]# service httpd restart

Additional RemoteHost templates

As seen in the examples above, there are a number of ways you can work with the directories created. Some common ones that I know of are below:

Each folder is labeled with the hostname it came from:

$template RemoteHost, "/var/log/remote/%HOSTNAME%/syslog.log"

Each folder is labeled with the hostname and IP address it came from:

$template RemoteHost, "/var/log/remote/%HOSTNAME%-%fromhost-ip%/syslog.log"

Each folder is labeled with the hostname, and break down the logs within by program name:

$template RemoteHost, "/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"

Each folder is labeled with the year, month and hostname, creating a type of natural log rotation:

$template RemoteHost, "/var/log/remote/%$YEAR%-%$MONTH%_%HOSTNAME%/syslog.log"

SSH brute force prevention with fail2ban

Ever take a look at your server’s auth logs and do a quick count of how many failed SSH login attempts you had on your server last week? Its very common to see hundreds, if not thousands of attempts in a very short period of time. Assuming you cannot use a firewall to restrict SSH access to only authorized IP addresses, how do you mitigate these brute force attacks?

There are many tools out there to help with this. One I like is fail2ban. This program scans through log files and takes action against events such as repeated failed login attempts, and blocks the offending IP address for a set period of time.

Procedure

On CentOS systems, fail2ban can be installed from the EPEL repositories. If you do not have EPEL installed, you can get it setup by:

CentOS 5

rpm -ivh http://archives.fedoraproject.org/pub/archive/epel/5/x86_64/epel-release-5-4.noarch.rpm

CentOS 6

rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

CentOS 7

rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm

Now install fail2ban:

yum install fail2ban
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Now customize /etc/fail2ban/jail.local accordingly for your server. Posted below are some more commonly configured options for CentOS 6 servers. The defaults (at the time of this writing) should protect SSH by banning any 3 or greater failed login attempts for 5 minutes via iptables. So the defaults should be okay, but you may want to consider adding your workstations IP address to the ignore ip list below so you don’t lock yourself out by accident!

vi /etc/fail2ban/jail.local

# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
ignoreip = 127.0.0.1/8

# "bantime" is the number of seconds that a host is banned.
bantime  = 600

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime  = 600

# "maxretry" is the number of failures before a host get banned.
maxretry = 3

Finally, set fail2ban to start at boot, and start service:

chkconfig fail2ban on
service fail2ban start

As a quick side note, sometimes hosting providers will automatically install fail2ban for you. And depending on the host, they may configure it in such a way that it sends an email each time an IP address gets banned from SSH. This can quickly create a flood of email or email failures, especially if its not configured for a real email address.

If you have having issues like this and your fail2ban configuration was set to email you, you can prevent fail2ban from sending you emails for SSH bans by removing the line from the ssh-iptables block:

sendmail-whois[name=SSH, dest=root, [email protected]]

As a live example assuming it was previously configured, here is what it would look like before you make the change:

vim /etc/fail2ban/jail.conf
...
[ssh-iptables]

enabled  = true
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
           sendmail-whois[name=SSH, dest=root, [email protected]]
logpath  = /var/log/secure
maxretry = 5

And here is what it would look like after you make the change:

vim /etc/fail2ban/jail.conf
...
[ssh-iptables]

enabled  = true
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
logpath  = /var/log/secure
maxretry = 5

And be sure to restart fail2ban after the configuration update has been completed:

service fail2ban restart

Malware Detection – rkhunter

Following on my previous articles, there are several good malware detection tools out there. These scanners help notify you of malware, hopefully before your clients notify you. Some of the common ones include:

chkrootkit
Linux Malware Detect (maldet)
rkhunter

Each have their own strong points, and they certainly compliment each other nicely when using them together depending on the solutions security strategy.

Rkhunter is similar in nature to chkrootkit, and I feel that both complement each other nicely. Taken from wikipedia’s page:

rkhunter (Rootkit Hunter) is a Unix-based tool that scans for rootkits, backdoors and possible local exploits. It does this by comparing SHA-1 hashes of important files with known good ones in online database, searching for default directories (of rootkits), wrong permissions, hidden files, suspicious strings in kernel modules, and special tests for Linux and FreeBSD.

Procedure

On CentOS systems, rkhunter can be installed from the EPEL repositories. If you do not have EPEL installed, you can get it setup by:

Installing rkhunter is pretty straight forward as shown below:

# CentOS 5 / RedHat 5
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
[[email protected] ~]# yum install rkhunter mailx

# CentOS 6 / RedHat 6
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[[email protected] ~]# yum install rkhunter mailx

# CentOS 7 / RedHat 7
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
[[email protected] ~]# yum install rkhunter mailx

# Ubuntu / Debian
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install rkhunter mailutils

Now that the installation is out of the way, lets configure rkhunter to send email if warning is found during scan:

[[email protected] ~]# vim /etc/rkhunter.conf
# Change
MAIL-ON-WARNING=""
# To
MAIL-ON-WARNING="[email protected]"

Now fetch the latest updates, create a baseline, and run a on-demand scan:

[[email protected] ~]# rkhunter --update 
[[email protected] ~]# rkhunter --propupd
[[email protected] ~]# rkhunter -sk -c

On CentOS and RHEL, configure cron so this runs automatically:

First, confirm the cronjob exists:

[[email protected] ~]# cat /etc/cron.daily/rkhunter

Now, update the rkhunter configuration with your email address so you can receive the nightly reports:

[[email protected] ~]# vi /etc/sysconfig/rkhunter
# Change
[email protected]
# To
[email protected]

On Ubuntu based systems, confirm the cronjob exists:

[[email protected] ~]# cat /etc/cron.daily/rkhunter

Now, update the rkhunter configuration with your email address so you can receive the nightly reports:

[[email protected] ~]# vi /etc/default/rkhunter
# Change
APT_AUTOGEN="false"
REPORT_EMAIL="root"

# To
APT_AUTOGEN="true"
REPORT_EMAIL="[email protected]"

NOTE: See https://help.ubuntu.com/community/RKhunter for more information about APT_AUTOGEN.