Setup ClamAV for nightly scans

PCI-DSS 3.1 Requirement 5 states the following:

Protect all systems against malware and regularly update anti-virus software or programs.

There are commercial based solutions out there for Linux based systems, but costs can become an issue, especially for small companies with a small footprint within their card holder data environment (CDE). So can one satify this requirement without breaking the bank? I personally prefer ClamAV.

Taken from the projects website, ClamAV is an open source antivirus engine for detecting trojans, viruses, malware and other malicious threats.

My requirements:
1. I want to scan my entire system nightly.
2. All virus reports are emailed to me so I can archive them for a year offsite.
3. Have the antivirus definitions updated nightly before the scan.

Installing, running and maintaining ClamAV is very straight forward on Linux based systems. To get started, install ClamAV by:

# CentOS 6 / RedHat 6
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[[email protected] ~]# yum install clamav mailx

# CentOS 7 / RedHat 7
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
[[email protected] ~]# yum install clamav clamav-update mailx
[[email protected] ~]# sed -i '/^Example/d' /etc/freshclam.conf

# Ubuntu 12.04 / Ubuntu 14.04
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install clamav mailutils

Now update the virus definitions by running:

[[email protected] ~]# freshclam

Finally, configure the virus definitions to update nightly, and also scan the entire system and email a report:

[[email protected] ~]# crontab -e
00 2 * * *  /usr/bin/freshclam
00 3 * * * /usr/bin/clamscan -r -i / | mail -s "ClamAV Report : INSERT_SERVER_HOSTNAME_HERE" [email protected]

Posted below is an example report ClamAV would send me via email nightly:

----------- SCAN SUMMARY -----------
Known viruses: 4289299
Engine version: 0.99
Scanned directories: 51929
Scanned files: 808848
Infected files: 0
Total errors: 10982
Data scanned: 76910.89 MB
Data read: 83578.27 MB (ratio 0.92:1)
Time: 6641.424 sec (110 m 41 s)

How does one go about testing ClamAV to ensure its working? There is a known antivirus test file that was designed specifically for this purpose by www.eicar.org. To create this file, simply setup the following test file, then rerun your ClamAV scan:

[[email protected] ~]# vim /tmp/EICAR-AV-Test
...
X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
...

Using AIDE for file integrity monitoring (FIM) on Ubuntu or Debian

PCI-DSS 3.1 section 10.5.5 has the following requirement:

Use file-integrity monitoring or change-detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert).

For large solutions, I would suggest using a well known tool such as Tripwire Enterprise. However many small to mid size companies that have a small footprint within their card holder data environment (CDE), may not be able to afford this. So what can companies use to meet this requirement? Implement AIDE (Advanced Intrusion Detection Environment).

Taken from the projects website, AIDE creates a database from the regular expression rules that it finds from the config file(s). Once this database is initialized it can be used to verify the integrity of the files.

AIDE is a very simple (yet powerful) program that runs from cron checking your files (typically once a night), and it will scan your system looking for any changes in the directories its monitoring. There are a number of different ways to use this program, but I’ll outline one that I like to use.

My requirements:
1. I want the reports to run nightly.
2. All change reports are emailed to me so I can archive them for a year offsite.
3. Have the database automatically commit the additions, deletions, and changes to baseline each time its ran.

In the event my system was compromised, I want to ensure that the malicious user was not able to modify, or delete my previous reports. Therefore, I choose not to store them on the machine. Its true that once the malicious user gained access to my system, they could change my AIDE config on me, but at least my previous reports will be intact which should help me when determining what malicious changes this user made to my server. Please note that I am making an assumption here that you are already backing up your system nightly, which would include your AIDE database! If you do not currently have a backup strategy in place, get one. Tools such as AIDE helps identify what files a malicious user may have changed, but if they completely crippled the system, you will need to restore from backups.

Setting up AIDE is fairly straight forward. It exists in most of package repositories out there including most variants of Linux and BSD.

On Ubuntu or Debian based systems, you can install it by:

[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install aide

Now to setup some basic configurations, such as the email notifications, update type, etc, modify the AIDE system configuration file according:

[[email protected] ~]# vim /etc/default/aide
...
FQDN=web01.domain.com
MAILSUBJ="Daily AIDE report for $FQDN"
[email protected]
QUIETREPORTS=no
COMMAND=update
COPYNEWDB=yes
...

Now that AIDE is installed, and the basic preferences are in place, its now time to check out the main configuration files. The default configuration from the upstream provider should give you a reasonable default configuration. But what if you wanted to add your website documentroot to this so you can keep track of what files are changing on your website? The Debian/Ubuntu way of configuring AIDE is a bit different from the CentOS/RHEL method.

All the configuration files resides in /etc/aide/aide.conf.d/. The number of the file appears to be used by the AIDE wrapper to use for deciding which order to process these files. The AIDE documentation seems to indicate that the most general rules should be processed last, so I’ll default to creating my servers profile with 50_aide_CUSTOM-RULES.

So lets say I want to monitor my documentroot, here is how this would be setup:

[[email protected] ~]# vim /etc/aide/aide.conf.d/50_aide_CUSTOM-RULES
...
/var/www/vhosts/domain.com FULL
...

Now AIDE will be keeping track of our website. But adding your site may lead to very noisy reports because most websites implement caching. So this now becomes a balancing act to exclude directories that change often, yet retain enough of your sites critical content. We could just leave the entire directory in AIDE, but I know I personally don’t want to read a change report that contains 1,000 changes every day. So in the case of this wordpress site, I exclude the cache directory by appending the following to my custom configuration:

[[email protected] ~]# vim /etc/aide/aide.conf.d/50_aide_CUSTOM-RULES
...
/var/www/vhosts/domain.com Full
!/var/www/vhosts/domain.com/web/wp-content/cache
...

The “!” means NOT to monitor that specific directory. You will need to run AIDE a few times and fine tune the configuration before you get a report that is useful for your specific needs.

Anytime a change is made to your AIDE configuration, you need to rebuild the AIDE run time configuration, and initialize the database. You do that by:

[[email protected] ~]# update-aide.conf
[[email protected] ~]# aideinit -y -f

Now, try making a basic change to /etc/hosts, then run a check on AIDE to see if it detects the change and emails out the report:

[[email protected] ~]# /etc/cron.daily/aide

If you wanted to just quickly test AIDE to ensure it picks up your changes, but won’t commit them to baseline, you can perform a one-time scan by:

[[email protected] ~]# aide.wrapper

To receive nightly AIDE reports, no further configuration is needed since Ubuntu/Debian already setup a cron job that will run AIDE automatically in /etc/cron.daily/aide. This will run whenever your system normally runs the cron.daily jobs, which is defined in /etc/crontab.

Posted below is an example report that AIDE would send me via email daily:

This is an automated report generated by the Advanced Intrusion Detection 
Environment on web01.domain.com started at 2016-03-07 13:16:35.

AIDE returned with exit code 7. Added, removed and changed files detected!
AIDE post run information
output database /var/lib/aide/aide.db.new was copied to /var/lib/aide/aide.db as requested by cron job configuration
End of AIDE post run information

AIDE produced no errors.

Output of the daily AIDE run (83 lines):
AIDE 0.15.1 found differences between database and filesystem!!
Start timestamp: 2016-03-07 13:16:35

Summary:
  Total number of files:	77937
  Added files:			2
  Removed files:		3
  Changed files:		7


---------------------------------------------------
Added files:
---------------------------------------------------

f++++++++++++++++: /var/log/aide/aide.log.0
d++++++++++++++++: /var/www/vhosts/domain.com/new

---------------------------------------------------
Removed files:
---------------------------------------------------

f----------------: /var/www/vhosts/domain.com/blah
f----------------: /var/www/vhosts/domain.com/test
d----------------: /var/www/vhosts/domain.com/test1

---------------------------------------------------
Changed files:
---------------------------------------------------

f   p.g    . A. .: /var/log/aide/aide.log
d =.... mc.. .. .: /var/spool/postfix/active
d =.... mc.. .. .: /var/spool/postfix/incoming
d =.... mc.. .. .: /var/spool/postfix/maildrop
F =.... mc.. ..  : /var/spool/postfix/public/pickup
F =.... mc.. ..  : /var/spool/postfix/public/qmgr
d =.... mc.. .. .: /var/www/vhosts/domain.com

---------------------------------------------------
Detailed information about changes:
---------------------------------------------------


File: /var/log/aide/aide.log
 Perm     : -rw-------                       , -rw-r-----
 Gid      : 0                                , 4
 ACL      : old = A:
----
user::rw-
group::---
other::---
----
                  D: 
            new = A:
----
user::rw-
group::r--
other::---
----
                  D: 

Directory: /var/spool/postfix/active
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23

Directory: /var/spool/postfix/incoming
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23

Directory: /var/spool/postfix/maildrop
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23

FIFO: /var/spool/postfix/public/pickup
 Mtime    : 2016-03-07 13:12:37              , 2016-03-07 13:17:37
 Ctime    : 2016-03-07 13:12:37              , 2016-03-07 13:17:37

FIFO: /var/spool/postfix/public/qmgr
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:36
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:36

Directory: /var/www/vhosts/domain.com
 Mtime    : 2016-03-07 13:03:25              , 2016-03-07 13:16:17
 Ctime    : 2016-03-07 13:03:25              , 2016-03-07 13:16:17

End of AIDE output.

The check was done against /var/lib/aide/aide.db with the following characteristics:
 Size     : 13041865
 Bcount   : 25480
 Mtime    : 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:13:23
 Inode    : 273628
 RMD160   : bIthG3Q5FiJmj4CIYdASjJx5Ygc=
 TIGER    : omto0nb3/oIqIiKHEjnbhjvXeGdfycbV
 SHA256   : VJPGKy61GxGfcSrjJFbrP879y/skJaiQ
 SHA512   : 7pz3FdYh8TvoNOqjxWBToZQNG6oxmrrp
 CRC32    : 1dYwqA==
 HAVAL    : LBFzyApqoYn7ogzoROG5FpneBO1s7R3p
 GOST     : iJ1tWPLtYaxxoFDHZEW8gxCS3/pVlS1G

The AIDE run created a new database /var/lib/aide/aide.db.new with the following characteristics:
 Size     : 13041834
 Bcount   : 25480
 Inode    : 273627
 RMD160   : 4TKRFSc0nt/VGDVvPEY8U6YNzaw=
 TIGER    : o4RzDHHWBlH+Zt3P7vI8GHHgGV1OecrC
 SHA256   : Gher/aINaU8r73/lQEWLQQSsKqP7sGjO
 SHA512   : D0/w3S6NOLZHw7D7dt1QxYBXe6miP5hF
 CRC32    : 5SRdpg==
 HAVAL    : pe7+ai57TPpW34NjJgTQxs+cQsFJ9zq0
 GOST     : RrIiyspbpKEb5wEGSG2HTYM7N6NUtKSv

End of AIDE daily cron job at 2016-03-07 13:18, run time 102 seconds

So this reports tells me that a log file for AIDE was rotated out, a new folder was created in my DocumentRoot called new, and the files/folders blah, test, and test1 where removed from my DocumentRoot.

Please remember that utilizing a tool to provide file integrity monitoring is only one part of a defense in depth strategy. There is no silver bullet for system security, but every layer you add will increase your security footprint which helps you with taking a proactive approach to security.

Setup central log server on CentOS 6

For security purposes, it is best practice to store all critical logs to a secured centralized syslog server, and keep the retention rate set for at least 1 year.

In this example, there are 4 servers:
syslog01 (192.168.1.200)
web01-local (192.168.1.201)
web02-local (192.168.1.202)
web03-local (192.168.1.203)

The goal here is to store all the system’s logs, as well as the individual Apache vhost’s logs to the central log server.

Install and perform initial configuration of rsyslog

On syslog01, confirm rsyslog is installed and running by:

[[email protected] ~]# yum install rsyslog
[[email protected] ~]# chkconfig rsyslog on
[[email protected] ~]# service rsyslog start

Open the firewall to allow inbound connections over port 514 to syslog01 by:

[[email protected] ~]# vim /etc/sysconfig/iptables
...
-A INPUT -s 192.168.1.0/24 -p tcp -m tcp --dport 514 -j ACCEPT
...
[[email protected] ~]# service iptables restart

Finally, create the directory that will be storing the logs:

[[email protected] ~]# mkdir -p /var/log/remote
[[email protected] ~]# chown root:root /var/log/remote
[[email protected] ~]# chmod 700 /var/log/remote

Determine log storage style on central log server

Here is where it needs to be decided how to store the logs on the central logging server. I am unaware of any recommend best practices as there are many ways to go about this. I’ll outline 4 common ways below:
– Option 1: Store each server’s logs in its own directory.
– Option 2: Store each server’s logs in its own directory with natural log rotation
– Option 3: Store each server’s logs in its own directory, broken down by program name
– Option 4: Store all server’s logs in a single file

Option 1: Store each server’s logs in its own directory

This will configure rsyslog to store the servers logs in a directory named after the remote server’s hostname. Log rotation will happen daily via logrotate.

To implement this, first create a backup of the existing /etc/rsyslog.conf, and create a new one as shown below. The parts I added or modified are in bold for reference:

[[email protected] ~]# mv /etc/rsyslog.conf /etc/rsyslog.orig
[[email protected] ~]# vim /etc/rsyslog.conf
# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
# Moved below for RuleSets
#$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
# Moved below for RuleSets
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

$template RemoteHost, "/var/log/remote/%HOSTNAME%/syslog.log"

#### RULES ####

$RuleSet local

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

local0.*                                                /var/log/local0.log
local1.*                                                /var/log/local1.log
local2.*                                                /var/log/local2.log
local3.*                                                /var/log/local3.log
local4.*                                                /var/log/local4.log
local5.*                                                /var/log/local5.log
local6.*                                                /var/log/local6.log

# Log all messages also to the central log directory
*.* ?RemoteHost

# Bind the above ruleset as default
$DefaultRuleset local

# This uses the template above
$RuleSet remote
*.* ?RemoteHost

# Bind the above ruleset for remote logs
$InputUDPServerBindRuleset remote
$UDPServerRun 514
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

Then restart rsyslog:

[[email protected] ~]# service rsyslog restart

Now setup logrotate to rotate out the logs, compress them, and keep them on file for 1 year as shown below:

[[email protected] ~]# vim /etc/logrotate.d/centralrsyslog
/var/log/remote/*/*.log
{
daily
    rotate 365
    missingok
    daily
    compress
    delaycompress
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Finally, test the log rotation by running:

[[email protected] ~]# logrotate -f /etc/logrotate.d/centralrsyslog

Option 2: Store each server’s logs in its own directory with natural log rotation

Both the local and remote system logs will be stored in its own directory as follows:

/var/log/remote/2015-12_syslog01/syslog.conf
/var/log/remote/2015-12_web01-local/syslog.conf
/var/log/remote/2015-12_web02-local/syslog.conf
/var/log/remote/2015-12_web03-local/syslog.conf

As the months change, syslog will automatically create a new folder accordingly for that month, creating a type of natural log rotation.

To implement this, first create a backup of the existing /etc/rsyslog.conf, and create a new one as shown below. The parts I added or modified are in bold for reference:

[[email protected] ~]# mv /etc/rsyslog.conf /etc/rsyslog.orig
[[email protected] ~]# vim /etc/rsyslog.conf
# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
# Moved below for RuleSets
#$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
# Moved below for RuleSets
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

$template RemoteHost, "/var/log/remote/%$YEAR%-%$MONTH%_%HOSTNAME%/syslog.log"

#### RULES ####

$RuleSet local

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

local0.*                                                /var/log/local0.log
local1.*                                                /var/log/local1.log
local2.*                                                /var/log/local2.log
local3.*                                                /var/log/local3.log
local4.*                                                /var/log/local4.log
local5.*                                                /var/log/local5.log
local6.*                                                /var/log/local6.log

# Log all messages also to the central log directory
*.* ?RemoteHost

# Bind the above ruleset as default
$DefaultRuleset local

# This uses the template above
$RuleSet remote
*.* ?RemoteHost

# Bind the above ruleset for remote logs
$InputUDPServerBindRuleset remote
$UDPServerRun 514
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

Then restart rsyslog:

[[email protected] ~]# service rsyslog restart

Option 3: Store each server’s logs in its own directory, broken down by program name

This one will break down the logs to store them in a directory named after the remote server’s hostname, and further more break them down by program name.

To implement this, first create a backup of the existing /etc/rsyslog.conf, and create a new one as shown below. The parts I added or modified are in bold for reference:

[[email protected] ~]# mv /etc/rsyslog.conf /etc/rsyslog.orig
[[email protected] ~]# vim /etc/rsyslog.conf
# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
# Moved below for RuleSets
#$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
# Moved below for RuleSets
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

$template RemoteHost, "/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"

#### RULES ####

$RuleSet local

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

local0.*                                                /var/log/local0.log
local1.*                                                /var/log/local1.log
local2.*                                                /var/log/local2.log
local3.*                                                /var/log/local3.log
local4.*                                                /var/log/local4.log
local5.*                                                /var/log/local5.log
local6.*                                                /var/log/local6.log

# Log all messages also to the central log directory
*.* ?RemoteHost

# Bind the above ruleset as default
$DefaultRuleset local

# This uses the template above
$RuleSet remote
*.* ?RemoteHost

# Bind the above ruleset for remote logs
$InputUDPServerBindRuleset remote
$UDPServerRun 514
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

Then restart rsyslog:

[[email protected] ~]# service rsyslog restart

Now setup logrotate to rotate out the logs, compress them, and keep them on file for 1 year as shown below:

[[email protected] ~]# vim /etc/logrotate.d/centralrsyslog
/var/log/remote/*/*.log
{
daily
    rotate 365
    missingok
    daily
    compress
    delaycompress
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Finally, test the log rotation by running:

[[email protected] ~]# logrotate -f /etc/logrotate.d/centralrsyslog

Option 4: Store all server’s logs in a single file

This is useful is you want to be able to use grep to find the information you need from a single file. Just keep in mind that if you have 10-20+ servers, this log file could get very large, even with a daily log rotation in place.

This will tell rsyslog to store ALL your servers logs in /var/log/remote/syslog.log.

To implement this, first create a backup of the existing /etc/rsyslog.conf, and create a new one as shown below. The parts I added or modified are in bold for reference:

[[email protected] ~]# mv /etc/rsyslog.conf /etc/rsyslog.orig
[[email protected] ~]# vim /etc/rsyslog.conf
# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
# Moved below for RuleSets
#$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
# Moved below for RuleSets
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

$template RemoteHost, "/var/log/remote/syslog.log"

#### RULES ####

$RuleSet local

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

local0.*                                                /var/log/local0.log
local1.*                                                /var/log/local1.log
local2.*                                                /var/log/local2.log
local3.*                                                /var/log/local3.log
local4.*                                                /var/log/local4.log
local5.*                                                /var/log/local5.log
local6.*                                                /var/log/local6.log

# Log all messages also to the central log directory
*.* ?RemoteHost

# Bind the above ruleset as default
$DefaultRuleset local

# This uses the template above
$RuleSet remote
*.* ?RemoteHost

# Bind the above ruleset for remote logs
$InputUDPServerBindRuleset remote
$UDPServerRun 514
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

Then restart rsyslog:

[[email protected] ~]# service rsyslog restart

Now setup logrotate to rotate out the logs, compress them, and keep them on file for 1 year as shown below:

[[email protected] ~]# vim /etc/logrotate.d/centralrsyslog
/var/log/remote/*/*.log
{
daily
    rotate 365
    missingok
    daily
    compress
    delaycompress
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Finally, test the log rotation by running:

[[email protected] ~]# logrotate -f /etc/logrotate.d/centralrsyslog

Setup clients to forward their logs to your central log server

For this, I am opting to use TCP. Append the following at the bottom of /etc/rsyslog.conf on each client server:

[[email protected] ~]# vim /etc/rsyslog.conf
*.* @@192.168.1.200:514

Then restart Rsyslog:

[[email protected] ~]# service rsyslog restart

Apache logs

If there was a requirement within the organization to also send Apache error logs access logs to the central syslog server, that can be done pretty easily. However as these can get very large, make sure you have enough disk space on your central syslog server to handle this.

Apache does not have the ability to natively send over the logs, so logger must be used. The following would need to be done on each vhost configured for both 80 and 443. Be sure to change the tags for each site accordingly in the logger entry below:

[[email protected] ~]# vim /etc/httpd/vhost.d/example.com.conf
...
CustomLog "|/usr/bin/logger -p local0.info -t example.com-access" combined
ErrorLog "|/usr/bin/logger -p local0.error -t example.com-error"
...

Restart Apache when done

[[email protected] ~]# service httpd restart

Additional RemoteHost templates

As seen in the examples above, there are a number of ways you can work with the directories created. Some common ones that I know of are below:

Each folder is labeled with the hostname it came from:

$template RemoteHost, "/var/log/remote/%HOSTNAME%/syslog.log"

Each folder is labeled with the hostname and IP address it came from:

$template RemoteHost, "/var/log/remote/%HOSTNAME%-%fromhost-ip%/syslog.log"

Each folder is labeled with the hostname, and break down the logs within by program name:

$template RemoteHost, "/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"

Each folder is labeled with the year, month and hostname, creating a type of natural log rotation:

$template RemoteHost, "/var/log/remote/%$YEAR%-%$MONTH%_%HOSTNAME%/syslog.log"

SSH brute force prevention with fail2ban

Ever take a look at your server’s auth logs and do a quick count of how many failed SSH login attempts you had on your server last week? Its very common to see hundreds, if not thousands of attempts in a very short period of time. Assuming you cannot use a firewall to restrict SSH access to only authorized IP addresses, how do you mitigate these brute force attacks?

There are many tools out there to help with this. One I like is fail2ban. This program scans through log files and takes action against events such as repeated failed login attempts, and blocks the offending IP address for a set period of time.

Procedure

On CentOS systems, fail2ban can be installed from the EPEL repositories. If you do not have EPEL installed, you can get it setup by:

CentOS 5

rpm -ivh http://archives.fedoraproject.org/pub/archive/epel/5/x86_64/epel-release-5-4.noarch.rpm

CentOS 6

rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

CentOS 7

rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm

Now install fail2ban:

yum install fail2ban
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Now customize /etc/fail2ban/jail.local accordingly for your server. Posted below are some more commonly configured options for CentOS 6 servers. The defaults (at the time of this writing) should protect SSH by banning any 3 or greater failed login attempts for 5 minutes via iptables. So the defaults should be okay, but you may want to consider adding your workstations IP address to the ignore ip list below so you don’t lock yourself out by accident!

vi /etc/fail2ban/jail.local

# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
ignoreip = 127.0.0.1/8

# "bantime" is the number of seconds that a host is banned.
bantime  = 600

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime  = 600

# "maxretry" is the number of failures before a host get banned.
maxretry = 3

Finally, set fail2ban to start at boot, and start service:

chkconfig fail2ban on
service fail2ban start

As a quick side note, sometimes hosting providers will automatically install fail2ban for you. And depending on the host, they may configure it in such a way that it sends an email each time an IP address gets banned from SSH. This can quickly create a flood of email or email failures, especially if its not configured for a real email address.

If you have having issues like this and your fail2ban configuration was set to email you, you can prevent fail2ban from sending you emails for SSH bans by removing the line from the ssh-iptables block:

sendmail-whois[name=SSH, dest=root, [email protected]]

As a live example assuming it was previously configured, here is what it would look like before you make the change:

vim /etc/fail2ban/jail.conf
...
[ssh-iptables]

enabled  = true
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
           sendmail-whois[name=SSH, dest=root, [email protected]]
logpath  = /var/log/secure
maxretry = 5

And here is what it would look like after you make the change:

vim /etc/fail2ban/jail.conf
...
[ssh-iptables]

enabled  = true
filter   = sshd
action   = iptables[name=SSH, port=ssh, protocol=tcp]
logpath  = /var/log/secure
maxretry = 5

And be sure to restart fail2ban after the configuration update has been completed:

service fail2ban restart

Malware Detection – rkhunter

Following on my previous articles, there are several good malware detection tools out there. These scanners help notify you of malware, hopefully before your clients notify you. Some of the common ones include:

chkrootkit
Linux Malware Detect (maldet)
rkhunter

Each have their own strong points, and they certainly compliment each other nicely when using them together depending on the solutions security strategy.

Rkhunter is similar in nature to chkrootkit, and I feel that both complement each other nicely. Taken from wikipedia’s page:

rkhunter (Rootkit Hunter) is a Unix-based tool that scans for rootkits, backdoors and possible local exploits. It does this by comparing SHA-1 hashes of important files with known good ones in online database, searching for default directories (of rootkits), wrong permissions, hidden files, suspicious strings in kernel modules, and special tests for Linux and FreeBSD.

Procedure

On CentOS systems, rkhunter can be installed from the EPEL repositories. If you do not have EPEL installed, you can get it setup by:

Installing rkhunter is pretty straight forward as shown below:

# CentOS 5 / RedHat 5
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
[[email protected] ~]# yum install rkhunter mailx

# CentOS 6 / RedHat 6
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[[email protected] ~]# yum install rkhunter mailx

# CentOS 7 / RedHat 7
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
[[email protected] ~]# yum install rkhunter mailx

# Ubuntu / Debian
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install rkhunter mailutils

Now that the installation is out of the way, lets configure rkhunter to send email if warning is found during scan:

[[email protected] ~]# vim /etc/rkhunter.conf
# Change
MAIL-ON-WARNING=""
# To
MAIL-ON-WARNING="[email protected]"

Now fetch the latest updates, create a baseline, and run a on-demand scan:

[[email protected] ~]# rkhunter --update 
[[email protected] ~]# rkhunter --propupd
[[email protected] ~]# rkhunter -sk -c

On CentOS and RHEL, configure cron so this runs automatically:

First, confirm the cronjob exists:

[[email protected] ~]# cat /etc/cron.daily/rkhunter

Now, update the rkhunter configuration with your email address so you can receive the nightly reports:

[[email protected] ~]# vi /etc/sysconfig/rkhunter
# Change
[email protected]
# To
[email protected]

On Ubuntu based systems, confirm the cronjob exists:

[[email protected] ~]# cat /etc/cron.daily/rkhunter

Now, update the rkhunter configuration with your email address so you can receive the nightly reports:

[[email protected] ~]# vi /etc/default/rkhunter
# Change
APT_AUTOGEN="false"
REPORT_EMAIL="root"

# To
APT_AUTOGEN="true"
REPORT_EMAIL="[email protected]"

NOTE: See https://help.ubuntu.com/community/RKhunter for more information about APT_AUTOGEN.

Malware Detection – Linux Malware Detect (maldet)

Following on my previous articles, there are several good malware detection tools out there. These scanners help notify you of malware, hopefully before your clients notify you. Some of the common ones include:

chkrootkit
Linux Malware Detect (maldet)
rkhunter

Each have their own strong points, and they certainly compliment each other nicely when using them together depending on the solutions security strategy.

Maldet happens to be one of the more useful tools I use often for checking suspected compromised servers. This scanner is a very in depth tool that is great at locating backdoors within binaries and website content, as well as other malicious tools. Below is a quick introduction for installing and configuring Maldet, as well as setting up a nightly cron job.

Procedure

As a prerequisite to have Maldet’s utilize ClamAV’s engine, install ClamAV. This will significantly speed up the scan time, and it should be considered a hard requirement:

# CentOS / RedHat
[[email protected] ~]# yum install clamav

# Ubuntu / Debian
[[email protected] ~]# apt-get install clamav

Then install it from source by:

[[email protected] ~]# wget http://www.rfxn.com/downloads/maldetect-current.tar.gz
[[email protected] ~]# tar -xf maldetect-current.tar.gz
[[email protected] ~]# cd maldetect-*
[[email protected] ~]# ./install.sh

Now you can run the scan. You could run the scan against the / directory, but depending on how much data you have, this could take hours or days. Normally I use this tool to check my website documentroot’s as I have other tools installed to monitor other parts of my system. So assuming our website content is in /var/www, here is how you scan that directory:

[[email protected] ~]# maldet -u
[[email protected] ~]# maldet --scan-all /var/www

If you would like to scan the entire server, which is good to do if you suspect you have been compromised, then you can run:

[[email protected] ~]# maldet --scan-all /

To setup a nightly cronjob, scanning your content and emailing you a report if something is found, first we need to modify the conf.maldet file as follows:

[[email protected] ~]# vi /usr/local/maldetect/conf.maldet
...
email_alert=1
email_addr="[email protected]"
...

Maldet automatically installs a cronjob for this, and you can feel free to modify /etc/cron.daily/maldet to your liking. However I prefer to remove it and setup my own. Please choose whatever works best for your needs. Here is one method:

[[email protected] ~]# rm /etc/cron.daily/maldet
[[email protected] ~]# crontab -e
...
32 3 * * * /usr/local/maldetect/maldet -d -u ; /usr/local/maldetect/maldet --scan-all /var/www
...

Now if maldet detects anything that flags as malware, it will send you a report via email for you to investigate.

If you are like me and want Maldet to send you daily reports via email regardless of if it found something or not, then simply setup the following cron job:

[[email protected] ~]# rm /etc/cron.daily/maldet
[[email protected] ~]# crontab -e
...
32 3 * * * /usr/local/maldetect/maldet -d -u ; /usr/local/maldetect/maldet --scan-all /var/www | mail -s "Maldet Report : INSERT_HOSTNAME_HERE" [email protected]
...

The reports will look similar to the following:

Linux Malware Detect v1.5
           (C) 2002-2015, R-fx Networks 
           (C) 2015, Ryan MacDonald 
This program may be freely redistributed under the terms of the GNU GPL v2
maldet(18252): {scan} signatures loaded: 10822 (8908 MD5 / 1914 HEX / 0 USER)
maldet(18252): {scan} building file list for /var/www, this might take awhile...
maldet(18252): {scan} setting nice scheduler priorities for all operations: cpunice 19 , ionice 6
maldet(18252): {scan} file list completed in 35s, found 278654 files...
maldet(18252): {scan} found clamav binary at /usr/bin/clamscan, using clamav scanner engine...
maldet(18252): {scan} scan of /var/www (278654 files) in progress...
maldet(18252): {scan} scan completed on /var/www: files 278654, malware hits 0, cleaned hits 0, time 2355s
maldet(18252): {scan} scan report saved, to view run: maldet --report 151111-0420.12250

Malware Detection – Chkrootkit

Imagine one day you get a phone call from one of your customers. Great, a sale! Nope, just kidding! They are calling to tell you that your site has been hacked. What do you say? How did they find out before you did?

This is a scenario that every sysadmin dreads. How can you prevent something like this? A good defense in depth security strategy goes a long way in preventing this. This includes, WAF’s, firewalls, file integrity monitoring, IDS, two factor authentication, ASV scanning, event management, etc, etc.

The more layers you have, the better chance you have at either mitigating the attack, or being notified of the compromise shortly after it happens so you can react quickly and not have your clients inform you of the problem.

Below is another layer you can add: chkrootkit
Taken from the following wikipedia article (http://en.wikipedia.org/wiki/Chkrootkit):

chkrootkit (Check Rootkit) is a common Unix-based program intended to help system administrators check their system for known rootkits. It is a shell script using common UNIX/Linux tools like the strings and grep commands to search core system programs for signatures and for comparing a traversal of the /proc filesystem with the output of the ps (process status) command to look for discrepancies.

The document below will outline how to install and configure it for CentOS and Ubuntu, including how to run this nightly and email you the report for review.

Procedure

Installing chkrootkit is pretty straight forward as shown below:

# CentOS 5 / RedHat 5
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
[[email protected] ~]# yum install chkrootkit mailx

# CentOS 6 / RedHat 6
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[[email protected] ~]# yum install chkrootkit mailx

# CentOS 7 / RedHat 7
[[email protected] ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
[[email protected] ~]# yum install chkrootkit mailx

# Ubuntu / Debian
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install chkrootkit mailutils

Perform a one time scan by running:

[[email protected] ~]# chkrootkit

Finally, lets setup a nightly cronjob and receive the report via email for review:

[[email protected] ~]# crontab -e
12 3 * * * /usr/sbin/chkrootkit | mail -s "chkrootkit Report : hostname.yourservername.com" [email protected]

Keep in mind that just setting up these tools doesn’t take care of the problem itself. The sysadmin needs to actively review any and all logs, software, etc. So with this tool and any security tool, identify false positives early on and exclude them from your reports so they are easier to read. This will help malware from slipping through the cracks of your logs and daily reports.

Several more articles will be following this in a series of sorts for malware detection. Each performs essentially the same function, but they go about it in different ways, each having their own benefits. Find the one(s) that work best for your solution.
chkrootkit
Linux Malware Detect (maldet)
rkhunter

SSH – Two Factor Authentication

Many people are using Google Authenticator to secure their google apps such as gmail. However what if you wanted to be able to utilize two factor authentication (something you have, something you know) for your SSH logins? What if you want to protect yourself against accidently using weak passwords, which can lead to a successful brute force attack?

On both RedHat and Debian based systems, Google Authenticator’s one time passwords are pretty simple to implement. For the purposes of this guide, I’ll be using CentOS 6 and Ubuntu 12.04.

It should be noted that by using this guide, ALL your users (including root) will be required to use the google authenticator to SSH in unless you have SSH keys already in place. Please check with your administration teams before setting this up to ensure you don’t accidently disable their access, or lock yourself out from SSH!

Procedure

1. Install the module

# RedHat 6 based systems
rpm -ivh http://linux.mirrors.es.net/fedora-epel/6/x86_64/epel-release-6-7.noarch.rpm
yum install google-authenticator

# Debian based systems
aptitude install libpam-google-authenticator

2. Now update the /etc/pam.d/sshd file and add the following at the end of the ‘auth’ section:

auth required pam_google_authenticator.so

3. Then update your /etc/ssh/sshd_config

# Change
ChallengeResponseAuthentication no

# To
ChallengeResponseAuthentication yes

4. Restart sshd

# Redhat:  
service sshd restart

# Ubuntu:  
service ssh restart

5. Now, setup keys for your user

google-authenticator

It will ask you to update your ~/.google_authenticator file, answer yes to this question, and whatever you would like to use for the next three. Once complete, the following will be present to you:

    New Secret Key
    Verification Code
    Emergency Scratch Codes

You will use the new secret key for adding the account to your phone’s google authenticator app. The emergency scratch codes should be copied down and stored somewhere secure. They can be used if you ever lose your iphone, or otherwise need to get into your account without your phone’s google authenticator app.

Now when you log into your server using your user account, it will prompt you for your google auth token, followed by your normal password for the server. Any accounts that don’t have the this setup will not be allowed to log in.

Final thoughts

Remember, two factor authentication is only one part of a defense in depth strategy. No security management system is perfect, but each layer you add will help increase your solutions security footprint.

Dirty disks

Many companies these days are making use of hosting providers to house their critical compute infrastructure. There are many financial benefits to doing this as the costs of running a data center far exceeds the costs of leasing servers from a hosting provider.

But as with many things, there is always another side of the story that must be considered, your environments security. Dedicated and cloud hosting providers reuse their hardware often as clients may lease the hardware for only a short time. The question you must ask yourself is: What do these providers do with the hard drives once a server has been removed from the clients account before they lease that server to another client? Do they have documentation proving their compliance with the DOD_5220.22-M standard for secure data removal?

Just because an operating system has been re-installed, that does not mean the data is permanently wiped. In theory, one could argue that your data is never fully removed from a drive unless you properly destroy the drive, which can be an expensive operation, but the DOD_5220.22-M standard provides a set of reasonable guidelines for rendering the data unrecoverable for many situations.

Extracting the data from a recycled server is not all that complicated for your average system administrator. In fact, using one technique, its a simple one liner using dd and strings. So the next time you are curious about your hosting providers security practices, try running the following on the server or cloud server and see what fragments of data you can recover by:

dd if=/dev/sda1 bs=1M skip=5000 | strings

Review the output. If you see data on here from other users, the disks are dirty and that hosting provider is not properly sanitizing the hard drives before leasing the server to another client. This has huge implications, especially if you need to comply with security standards such as HIPAA, FISMA, PCI compliance or any of the other standards out there.

So the moral of the story is, always do your homework before choosing a dedicated or cloud hosting provider. Ask about their security procedures, ask them about their PCI, HIPAA, FISMA, ISO 27001, SSAE 16, etc. Never take anything at face value. Ask them for their report on compliance.

When it comes to your security stance, paranoia is your greatest defense.

Using AIDE for file integrity monitoring (FIM) on CentOS

PCI-DSS 3.1 section 10.5.5 has the following requirement:

Use file-integrity monitoring or change-detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert).

For large solutions, I would suggest using a well known tool such as Tripwire Enterprise. However many small to mid size companies that have a small footprint within their card holder data environment (CDE), may not be able to afford this. So what can companies use to meet this requirement? Implement AIDE (Advanced Intrusion Detection Environment).

Taken from the projects website, AIDE creates a database from the regular expression rules that it finds from the config file(s). Once this database is initialized it can be used to verify the integrity of the files.

AIDE is a very simple (yet powerful) program that runs from cron checking your files (typically once a night), and it will scan your system looking for any changes in the directories its monitoring. There are a number of different ways to use this program, but I’ll outline one that I like to use.

My requirements:
1. I want the reports to run nightly.
2. All change reports are emailed to me so I can archive them for a year offsite.
3. Have the database automatically commit the additions, deletions, and changes to baseline each time its ran.

In the event my system was compromised, I want to ensure that the malicious user was not able to modify, or delete my previous reports. Therefore, I choose not to store them on the machine. Its true that once the malicious user gained access to my system, they could change my AIDE config on me, but at least my previous reports will be intact which should help me when determining what malicious changes this user made to my server. Please note that I am making an assumption here that you are already backing up your system nightly, which would include your AIDE database! If you do not currently have a backup strategy in place, get one. Tools such as AIDE helps identify what files a malicious user may have changed, but if they completely crippled the system, you will need to restore from backups.

Setting up AIDE is fairly straight forward. It exists in most of package repositories out there including most variants of Linux and BSD.

On Linux based systems, you can install it by:

[[email protected] ~]# yum install aide

Once you have AIDE installed, the default configuration from the upstream provider should give you a reasonable default aide.conf. But what if you wanted to add your website documentroot to this so you can keep track of what files are changing on your website? Well, we simple add the directory to the aide.conf by including:

[[email protected] ~]# vim /etc/aide.conf
...
/var/www/vhosts/domain.com
...

Now AIDE will be keeping track of our website. But adding your site may lead to very noisy reports because most websites implement caching. So this now becomes a balancing act to exclude directories that change often, yet retain enough of your sites critical content. We could just leave the entire directory in AIDE, but I know I personally don’t want to read a change report that contains 1,000 changes every day. So in the case of this wordpress site, I exclude the cache directory by appending the following to my custom configuration:

[[email protected] ~]# vim /etc/aide.conf
...
/var/www/vhosts/domain.com
!/var/www/vhosts/domain.com/web/wp-content/cache
...

The “!” means NOT to monitor that specific directory. You will need to run AIDE a few times and fine tune the configuration before you get a report that is useful for your specific needs.

On CentOS, I had to change the following setting in /etc/aide.conf for the initialization to work:

[[email protected] ~]# vim /etc/aide.conf
...
# Whether to gzip the output to database
gzip_dbout=no
...

Once you have your configuration tuned for your specific purposes, you first my initialize the database to create a baseline before you can start getting change reports. You do this by:

[[email protected] ~]# aide --init
[[email protected] ~]# mv -f /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Now, try making a basic change to /etc/hosts, then run a check on AIDE to see if it detects the change:

[[email protected] ~]# aide --check

If you are like me and would prefer not to have to log into 10 servers a day to run and view the reports, you can configure cron to run the report, and email you the results daily, while committing the changes to baseline. If you choose to go this route, it is critical that you review your change reports as they come in because we are essentially committing every change to the baseline. Here is how I configure cron:

[[email protected] ~]# crontab -e
# Perform daily change report
0 3 * * * /usr/sbin/aide --update | mail -s "AIDE Audit Report : web01.example.com" [email protected]

# Initialize the AIDE database once a day:
30 3 * * *  nice -19 /usr/sbin/aide --init;mv -f /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Posted below is an example report that AIDE would send me via email daily:

AIDE found differences between database and filesystem!!
Start timestamp: 2012-09-13 01:24:05

Summary:
Total number of files: 57620
Added files: 1
Removed files: 1
Changed files: 1

---------------------------------------------------
Added files:
---------------------------------------------------

added: /var/spool/cron/root

---------------------------------------------------
Removed files:
---------------------------------------------------

removed: /etc/.aide.conf.swp

---------------------------------------------------
Changed files:
---------------------------------------------------

changed: /etc/aide.conf

--------------------------------------------------
Detailed information about changes:
---------------------------------------------------

File: /etc/aide.conf
Size : 2381 , 2390
Mtime : 2012-09-13 01:24:05 , 2012-09-13 01:24:05
Ctime : 2012-09-13 01:24:05 , 2012-09-13 01:24:05
MD5 : b+qbBDYEPesd+NCR1VRQHQ== , rG5pNPghdweedpU/c0ieHw==
RMD160 : T081ixhqik4efC3dfeCOBDCKpP4= ,
qe8MV0eteklAKmlZ5LTubaOUNKo=
SHA256 : g4jstEtfU8BNu+43jkrxJc9Cpr2SABZj ,
a65iaV54XR4vu8/zbA4Tdfe2U+W5uPNY

So this reports tells me that root’s crontab was added, a swap file for aide.conf was removed, and I updated the /etc/aide.conf recently.

Please remember that utilizing a tool to provide file integrity monitoring is only one part of a defense in depth strategy. There is no silver bullet for system security, but every layer you add will increase your security footprint which helps you with taking a proactive approach to security.