Malware Detection – Linux Malware Detect (maldet)

Following on my previous articles, there are several good malware detection tools out there. These scanners help notify you of malware, hopefully before your clients notify you. Some of the common ones include:

chkrootkit
Linux Malware Detect (maldet)
rkhunter

Each have their own strong points, and they certainly compliment each other nicely when using them together depending on the solutions security strategy.

Maldet happens to be one of the more useful tools I use often for checking suspected compromised servers. This scanner is a very in depth tool that is great at locating backdoors within binaries and website content, as well as other malicious tools. Below is a quick introduction for installing and configuring Maldet, as well as setting up a nightly cron job.

Procedure

As a prerequisite to have Maldet’s utilize ClamAV’s engine, install ClamAV. This will significantly speed up the scan time, and it should be considered a hard requirement:

# CentOS / RedHat
[root@web01 ~]# yum install clamav

# Ubuntu / Debian
[root@web01 ~]# apt-get install clamav

Then install it from source by:

[root@web01 ~]# wget http://www.rfxn.com/downloads/maldetect-current.tar.gz
[root@web01 ~]# tar -xf maldetect-current.tar.gz
[root@web01 ~]# cd maldetect-*
[root@web01 ~]# ./install.sh

Now you can run the scan. You could run the scan against the / directory, but depending on how much data you have, this could take hours or days. Normally I use this tool to check my website documentroot’s as I have other tools installed to monitor other parts of my system. So assuming our website content is in /var/www, here is how you scan that directory:

[root@web01 ~]# maldet -u
[root@web01 ~]# maldet --scan-all /var/www

If you would like to scan the entire server, which is good to do if you suspect you have been compromised, then you can run:

[root@web01 ~]# maldet --scan-all /

To setup a nightly cronjob, scanning your content and emailing you a report if something is found, first we need to modify the conf.maldet file as follows:

[root@web01 ~]# vi /usr/local/maldetect/conf.maldet
...
email_alert=1
email_addr="[email protected]"
...

Maldet automatically installs a cronjob for this, and you can feel free to modify /etc/cron.daily/maldet to your liking. However I prefer to remove it and setup my own. Please choose whatever works best for your needs. Here is one method:

[root@web01 ~]# rm /etc/cron.daily/maldet
[root@web01 ~]# crontab -e
...
32 3 * * * /usr/local/maldetect/maldet -d -u ; /usr/local/maldetect/maldet --scan-all /var/www
...

Now if maldet detects anything that flags as malware, it will send you a report via email for you to investigate.

If you are like me and want Maldet to send you daily reports via email regardless of if it found something or not, then simply setup the following cron job:

[root@web01 ~]# rm /etc/cron.daily/maldet
[root@web01 ~]# crontab -e
...
32 3 * * * /usr/local/maldetect/maldet -d -u ; /usr/local/maldetect/maldet --scan-all /var/www | mail -s "Maldet Report : INSERT_HOSTNAME_HERE" [email protected]
...

The reports will look similar to the following:

Linux Malware Detect v1.5
           (C) 2002-2015, R-fx Networks 
           (C) 2015, Ryan MacDonald 
This program may be freely redistributed under the terms of the GNU GPL v2
maldet(18252): {scan} signatures loaded: 10822 (8908 MD5 / 1914 HEX / 0 USER)
maldet(18252): {scan} building file list for /var/www, this might take awhile...
maldet(18252): {scan} setting nice scheduler priorities for all operations: cpunice 19 , ionice 6
maldet(18252): {scan} file list completed in 35s, found 278654 files...
maldet(18252): {scan} found clamav binary at /usr/bin/clamscan, using clamav scanner engine...
maldet(18252): {scan} scan of /var/www (278654 files) in progress...
maldet(18252): {scan} scan completed on /var/www: files 278654, malware hits 0, cleaned hits 0, time 2355s
maldet(18252): {scan} scan report saved, to view run: maldet --report 151111-0420.12250

Malware Detection – Chkrootkit

Imagine one day you get a phone call from one of your customers. Great, a sale! Nope, just kidding! They are calling to tell you that your site has been hacked. What do you say? How did they find out before you did?

This is a scenario that every sysadmin dreads. How can you prevent something like this? A good defense in depth security strategy goes a long way in preventing this. This includes, WAF’s, firewalls, file integrity monitoring, IDS, two factor authentication, ASV scanning, event management, etc, etc.

The more layers you have, the better chance you have at either mitigating the attack, or being notified of the compromise shortly after it happens so you can react quickly and not have your clients inform you of the problem.

Below is another layer you can add: chkrootkit
Taken from the following wikipedia article (http://en.wikipedia.org/wiki/Chkrootkit):

chkrootkit (Check Rootkit) is a common Unix-based program intended to help system administrators check their system for known rootkits. It is a shell script using common UNIX/Linux tools like the strings and grep commands to search core system programs for signatures and for comparing a traversal of the /proc filesystem with the output of the ps (process status) command to look for discrepancies.

The document below will outline how to install and configure it for CentOS and Ubuntu, including how to run this nightly and email you the report for review.

Procedure

Installing chkrootkit is pretty straight forward as shown below:

# CentOS 5 / RedHat 5
[root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
[root@web01 ~]# yum install chkrootkit mailx

# CentOS 6 / RedHat 6
[root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[root@web01 ~]# yum install chkrootkit mailx

# CentOS 7 / RedHat 7
[root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
[root@web01 ~]# yum install chkrootkit mailx

# Ubuntu / Debian
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install chkrootkit mailutils

Perform a one time scan by running:

[root@web01 ~]# chkrootkit

Finally, lets setup a nightly cronjob and receive the report via email for review:

[root@web01 ~]# crontab -e
12 3 * * * /usr/sbin/chkrootkit | mail -s "chkrootkit Report : hostname.yourservername.com" [email protected]

Keep in mind that just setting up these tools doesn’t take care of the problem itself. The sysadmin needs to actively review any and all logs, software, etc. So with this tool and any security tool, identify false positives early on and exclude them from your reports so they are easier to read. This will help malware from slipping through the cracks of your logs and daily reports.

Several more articles will be following this in a series of sorts for malware detection. Each performs essentially the same function, but they go about it in different ways, each having their own benefits. Find the one(s) that work best for your solution.
chkrootkit
Linux Malware Detect (maldet)
rkhunter

Investigating Compromised Servers

This guide is not a step by step tutorial on how to clean a compromised server, rather it is a reference to illustrate what tools are available for performing an analysis of the compromise. The goal of this guide is to show you what information is available to help you determine:

– Point of entry
– The origin of the attack
– What files were compromised
– What level access did the attacker obtain
– Audit trail of the attackers footprints

There are many different types of compromises available to exploit a UNIX server. Under many circumstances, a server is exploited using common techniques such as using a brute force attack, to guess a weak password, or attempting to use known vulnerabilities in software in hopes the server is not on a regular patch schedule. Whatever the method, it is important to understand how the machine got compromised so you can determine the extent of damage to your server and other hosts accessible to this machine.

During many root level compromises, the most straight forward approach to recovery is to perform a clean install of the server and restore any critical data from known good backups. However until the entry point of the compromise is known, this may not be enough as the compromise needs to be understood so that security hole can be properly closed.

Documentation

When you are notified that a system under your control may be compromised. You want to obtain as much information as possible from the complainant. This includes:

– How was the initial problem found?
– Can the time of the compromise be estimated?
– Has the server been modified since the compromise was detected?
– Anything else the complainant says that is important.

IMPORTANT NOTE: If you are planning on getting law enforcement involved, it is imperative that no additional actions are taken on the server. The server must remain in its current state for forensic and evidence collection purposes.

If you choose to proceed with the investigation, document anything you find on the server. It can be as simple as a copy/paste of the command and its results.

Tools used for investigation

In the attacker’s ideal scenario, all important log files would have been wiped so their tracks are clean. Oftentimes however, this doesn’t happen. This leaves some valuable clues in finding what was done. It may also help determine if this was a basic web hack, or a root level compromise. Below are some of the basic commands that I’ll look through when trying to find that one thread so I can unravel the rest of the compromise.

last

This will list the sessions of users that recently logged into the system and include the timestamps, hostnames and whether or not the user is still logged in. An odd IP address may be cross referenced against a brute force ssh attack in /var/log/messages or /var/log/secure which may indicate how the attacker gained entry, what user they got in as, and if they were able to escalate their privilege to root.

ls -lart /

This will provide a time ordered list of files and directories that can be correlated against the time of the compromise. This listing will help determine what has been added or removed from the system.

netstat -na

This will list the current listening sockets on the machine. Running this may reveal any back doors that are listening, or any errant services that are running.

ps -wauxef

This will be helpful in tracking down any errant processes that are listening, as well as help show other odd processes such as the user www running a bash process for example. lsof |grep can also be used to further find what open files this process is using. Concurrently, cat /proc//cmdline may also let you know where the file that controls this process exists.

bash_history

The history file often becomes the Rosetta stone of tracking down what took place during a compromise. Looking through the users .bash_history file will often show exactly what commands were executed, what malicious programs were downloaded, and possibly what directories they were focusing on.

top

Oftentimes, a malicious process will be causing CPU contention issues within the environment, and will usually show up near the top of the list. Any processes that are causing the CPU contention issues should be considered suspicious when tracking down a compromise.

strace

When running strace -p pid on a suspicious process, this may yield important insight into what the process is performing.

In some cases, the commands above may not provide many clues to what happened during the attack. This is where more fine grained tools must be used.

Before moving forward, it should be confirmed that the binaries you are using to investigate are not trojanned versions. These trojanned versions can perform whatever tasks the attacker wishes, which include not showing information that could trace what the compromise was trying to accomplish.

So to verify we have a good working set of tools:

rpm -Va

Verifying a package compares information about the installed files in the package with information about the files taken from the package meta data stored in the rpm database. Among other things, verifying compares the size, MD5 sum, permissions, type, owner and group of each file. Any discrepancies are displayed.

When running this command, it is important to note any packages that are flagged in the following directories may mean you are using a trojanned version of the binary, and therefore you cannot trust its output:

– /bin
– /sbin
– /usr/bin
– /usr/sbin

An example of what a trojanned file:

S.5….T /bin/login
rpm -qa

This can be used to show what rpm’s have been recently installed in chronological order. However, in the case of a root compromise, the rpm database could be altered and therefore not trusted.

lsattr

In cases where the attacker was able to get root access and trojan certain binaries, sometimes they will set that binary to be immutable so you cannot reinstall a clean version of that binary. Common directories to look in are:

– /bin
– /sbin
– /usr/bin
– /usr/sbin

An example of a file that had its attributes set to immutable:

-------i----- /bin/ps

Under normal circumstances in these directories, the rules should all look similar to:

------------- /bin/ps
find

Find is a UNIX tool that can be critical in finding files that have been recently modified. For instance, to find files that have been modified in the last file days, run:

find / -mtime 5

Common Directories Where Web Exploits Are Found
Check world writable directories that Apache would commonly write its temp files to. Locations such as:

– ls -al /tmp
– ls -al /var/tmp
– ls -al /dev/shm

If you have directories on your website that are chmod’ed 777, those are suspect as well.

When checking these directories, you are looking for any files that you don’t recognize, or look suspicious. Be on the lookout for hidden files or files that have execute permissions.

mailq
postcat -q [message_id] | grep X-PHP-Originating-Script
...
X-PHP-Originating-Script: 48:dirs60.php(243) : eval()'d code
...

As shown above, sometimes the compromise could reveal itself through Postfix. If you see your server sending out a lot of spam, or if you are having resource contention issues with CPU or Load, checking out mail queue could help locate the source of the malicious script on the server.

As dirs60.php doesn’t show you exactly where the file resides, you can usually track it down by:

updatedb
locate dirs60.php
# or
find /path/to/website/docroot -name dirs60.php
maldet --scan-all /var/www

Maldet can be very useful for scanning your site for potential malware. Instructions on how to install and run it can be found here.

clamscan -r -i /

ClamAV can also be extremely useful for scanning your entire server for malware. Instructions on how to install and run it can be found here.

Finding point of entry

If you found anything using the information above, that means that you most likely have a timestamp of when the malicious file(s) were installed on the server.

You can now use that timestamp to start combing through your website’s access logs, looking for any suspicious entries in the log during that time period. If you find something, you can cross reference it to where you found the malicious files, then you likely just narrowed down the point of entry.

While the large majority of compromises do come from exploitable code within your website, you cannot rule out other entry points. So be sure to dig through /var/log/* to see if there is anything suspicious during the reported time frame.

Example of investigation

Below is a real example of one of my investigations, documenting my thought process.

When I am investigating a suspected root level compromise, the first thing that needs to be verified was whether or not this was just a basic web hack, or if root privileges were really gained. 80% of the time, its just a simple web hack that can be safely cleaned.

Step 1: Quick and dirty check to see it root privileges were gained:

lsattr /usr/sbin | less
lsattr /usr/bin | less
lsattr /bin | less
lsattr /sbin | less

What to look for:

Your checking for modified attributes such as binaries being set immutable, etc.

Results:

s---ia------- /sbin/shs

^ When you strings that file, you see its a backdoor shell.

Step 2: See if the attacker cleaned his tracks.

Many times, these are script kiddies or dummies who just forgot to clean up after themselves.

What to look for:

All user accounts in /etc/passwd that a valid shell:

cat /home/$USER/.bash_history
Root's history:
history
cat /root/.bash_history

Results:

The /root/.bash_history revealed what the attacker did on the server, which includes:

– They downloaded some malicious tools to serve up via apache in /var/www/html/*.
– They also installed some IRC stuff in /var/tmp/.ICE-unix (as well as other tools).
– Modified root’s crontab to re-download the malicious tools if someone removes them from the server:
* * * * * /var/tmp/.ICE-unix/update >/dev/null 2>&1

Step 3: Check for basic web hacks

Normally if steps 1 and 2 do not show anything, most likely its just a simple web hack that CAN be cleaned easily without formatting the server or otherwise causing panic.

In this specific investigation, that logic is null and void since we know that root privileges were gained. However, just for reference, and since its relevant to this anyways cause I believe that the attacker exploited phpmyadmin. Once they had their backdoor phpshell loaded, they were able to perform a local root exploit to escalate their privileges.

What to look for:

Hidden files and directories, in world readable directories, that apache would normally write tmp files to:

ls -al /var/tmp |less
ls -al /tmp
ls -al /dev/shm

Results:

drwx------ 3 70 70 4096 Nov 19 02:00 /var/tmp/.ICE-unix

^ There is a whole bunch of fun stuff in there.

If items are found in here, you must attempt to track down the entry point so you can have the client take down the site, upgrade their site code, or otherwise fix the exploitable code. One quick and dirty way is by looking at step 5. However, if you see irc bots and stuff running in the output of ps -waux, then you can try to catch where the process is running from by using lsof, or ps -wauxxef |grep .

Step 4: Look for PID’s listening for incoming connections

What to look for:

netstat -natp : Looks for any suspicious connections running on odd ports
ps -wauxxef : look for suspicious files like bash running under www context
lsof : helps to determine where the pid above is running from

Results:

tcp 0 0 0.0.0.0:1144 0.0.0.0:* LISTEN 1008/bash
tcp 0 1 172.16.23.13:60968 22.22.22.22:7000 SYN_SENT 6860/sshd

There are also a fair amount of other ssh ESTABLISHED connections running from high level ports. This means the attackers are still connected to this machine. I can’t see them cause they probably modified the binaries to hide themselves.

[root@www tmp]# netstat -natp |grep sshd |awk '{print $4,$5,$6,$7}'
0.0.0.0:22 0.0.0.0:* LISTEN 1046/sshd
172.16.23.13:60986 22.22.22.22:6667 SYN_SENT 6860/sshd
123.123.123.123:22 22.22.22.22:59361 ESTABLISHED 22795/sshd
123.123.123.123:22 22.22.22.22:57434 ESTABLISHED 22796/sshd
123.123.123.123:57139 143.143.143.143:6667 ESTABLISHED 6860/sshd
123.123.123.123:57402 22.22.22.22:6667 ESTABLISHED 6860/sshd
123.123.123.123:22 143.143.143.143:49238 ESTABLISHED 8860/sshd
123.123.123.123:57134 22.22.22.22:6667 ESTABLISHED 6860/sshd
123.123.123.123:56845 22.22.22.22:6667 ESTABLISHED 6860/sshd
123.123.123.123:57127 143.143.143.143:6667 ESTABLISHED 6860/sshd

Step 5: Determine point of entry for original compromise

What to look for:

/var/log/[messages|secure] : check for brute forced ssh attempts.
apache access logs and error logs : May help determine which site is exploitable. Most attacks are linked against this.

When checking this, also cross reference IP’s against the logs if you think there is a chance it may have originated from there. Its a quick and easy way to trace down the origin point.

Simple ways of checking servers with a ton of web logs like the one used in this example:

cd /var/log/httpd
for i in `ls * |grep access`; do echo $i && grep wget $i; done
for i in `ls * |grep access`; do echo $i && grep curl $i; done

NOTE: wget was searched cause that was in root’s history file under what I believe may have been part of the entry point

Results:
Evidence found that the phpmyadmin installation in /var/www/html was exploited. The version of phpmyadmin was severely outdated. Keeping phpmyadmin patched on a regular schedule would have prevented this from happening.

Final thoughts

Investigating web or root level exploits is more of an art then a science. After you have investigating a few dozen, you will just ‘feel’ something is not right on the server. I cannot stress enough, when you are investigating a compromised system, you must do everything you can to determine exactly how the server got compromised in the first place. Once you have that information, you will then be able to successfully remediate the exploit.

SSH – Two Factor Authentication

Many people are using Google Authenticator to secure their google apps such as gmail. However what if you wanted to be able to utilize two factor authentication (something you have, something you know) for your SSH logins? What if you want to protect yourself against accidently using weak passwords, which can lead to a successful brute force attack?

On both RedHat and Debian based systems, Google Authenticator’s one time passwords are pretty simple to implement. For the purposes of this guide, I’ll be using CentOS 6 and Ubuntu 12.04.

It should be noted that by using this guide, ALL your users (including root) will be required to use the google authenticator to SSH in unless you have SSH keys already in place. Please check with your administration teams before setting this up to ensure you don’t accidently disable their access, or lock yourself out from SSH!

Procedure

1. Install the module

# RedHat 6 based systems
rpm -ivh http://linux.mirrors.es.net/fedora-epel/6/x86_64/epel-release-6-7.noarch.rpm
yum install google-authenticator

# Debian based systems
aptitude install libpam-google-authenticator

2. Now update the /etc/pam.d/sshd file and add the following at the end of the ‘auth’ section:

auth required pam_google_authenticator.so

3. Then update your /etc/ssh/sshd_config

# Change
ChallengeResponseAuthentication no

# To
ChallengeResponseAuthentication yes

4. Restart sshd

# Redhat:  
service sshd restart

# Ubuntu:  
service ssh restart

5. Now, setup keys for your user

google-authenticator

It will ask you to update your ~/.google_authenticator file, answer yes to this question, and whatever you would like to use for the next three. Once complete, the following will be present to you:

    New Secret Key
    Verification Code
    Emergency Scratch Codes

You will use the new secret key for adding the account to your phone’s google authenticator app. The emergency scratch codes should be copied down and stored somewhere secure. They can be used if you ever lose your iphone, or otherwise need to get into your account without your phone’s google authenticator app.

Now when you log into your server using your user account, it will prompt you for your google auth token, followed by your normal password for the server. Any accounts that don’t have the this setup will not be allowed to log in.

Final thoughts

Remember, two factor authentication is only one part of a defense in depth strategy. No security management system is perfect, but each layer you add will help increase your solutions security footprint.

Encrypting Block Storage In The Cloud

For sensitive information being stored in the cloud outside your direct control, it is critical to encrypt your data at rest. Full disk encryption helps to protect you against unauthorized indivuduals mounting your volume without a key.

It should be noted that LUKS encryption will not protect your data when it is mounted and viewable by your server. A malicious user could in theory break into your server and traverse to that mount point.

My requirements

When writing this guide, I am using the following thought process when implementing LUKS on my Cloud Block Storage volume:

– No keys are to be stored on the server. This is for security purposes since your keys shouldn’t be stored on the server. Would you tape the keys to your Porche on the hood? No. The same logic applies here.
– The volume will not be mounted at boot. This is to prevent the server from stopping the boot process for you to enter in the key. Please note that if the server reboots, you will have to manually log in, mount the volume, and type the passphase for the volume before the system can use it again!

Procedure

This example is going to be specific for CentOS 6.

1. Create your Cloud Block Storage Volumes
In this instance, I am going to be using 2x 100G volumes which are already mounted on my server, and will be setting them up in a RAID 1 configuration.

yum install mdadm cryptsetup-luks cryptsetup-luks-devel
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/xvdb /dev/xvdd

Now confirm the RAID is rebuilding itself by typing:

mdadm --detail /dev/md0

2. Now its time to setup LUKS. This will format your volume, so use caution.

cryptsetup luksFormat /dev/md0

Confirm the contents of the message, then type ‘YES’ in uppercase letters. Then enter a very secure passphrase, and store it somewhere safe. Never store the key on your server!

3. You can verify the results of the encryption process by typing the following:

cryptsetup luksDump /dev/md0

4. Time to mount the encrypted volume and give it a name:

cryptsetup luksOpen /dev/md0 mysecurevolume

5. Finally, lets put a filesystem on it and mount:

mkfs.ext4 /dev/mapper/mysecurevolume
mkdir /opt/mysecurevolume
mount /dev/mapper/mysecurevolume /opt/mysecurevolume

6. You can check the status to ensure its encrypted by typing:

cryptsetup status mysecurevolume

7. Now disable automount so your server won’t hang on boot waiting for the passphrase:

vi /etc/grub.conf
# Append the following at the end of all the kernel lines
rd_NO_LUKS

And your done!

To manually mount the volume after a reboot:

cryptsetup luksOpen /dev/md0 mysecurevolume
mount /dev/mapper/mysecurevolume /opt/mysecurevolume

To manually umount the volume, type:

umount /dev/mapper/mysecurevolume
cryptsetup luksClose mysecurevolume

Final thoughts

Remember, full disk encryption utilizing LUKS is only one part of a defense in depth strategy. No security management system is perfect, but each layer you add will help increase your solutions security footprint.

Dirty disks

Many companies these days are making use of hosting providers to house their critical compute infrastructure. There are many financial benefits to doing this as the costs of running a data center far exceeds the costs of leasing servers from a hosting provider.

But as with many things, there is always another side of the story that must be considered, your environments security. Dedicated and cloud hosting providers reuse their hardware often as clients may lease the hardware for only a short time. The question you must ask yourself is: What do these providers do with the hard drives once a server has been removed from the clients account before they lease that server to another client? Do they have documentation proving their compliance with the DOD_5220.22-M standard for secure data removal?

Just because an operating system has been re-installed, that does not mean the data is permanently wiped. In theory, one could argue that your data is never fully removed from a drive unless you properly destroy the drive, which can be an expensive operation, but the DOD_5220.22-M standard provides a set of reasonable guidelines for rendering the data unrecoverable for many situations.

Extracting the data from a recycled server is not all that complicated for your average system administrator. In fact, using one technique, its a simple one liner using dd and strings. So the next time you are curious about your hosting providers security practices, try running the following on the server or cloud server and see what fragments of data you can recover by:

dd if=/dev/sda1 bs=1M skip=5000 | strings

Review the output. If you see data on here from other users, the disks are dirty and that hosting provider is not properly sanitizing the hard drives before leasing the server to another client. This has huge implications, especially if you need to comply with security standards such as HIPAA, FISMA, PCI compliance or any of the other standards out there.

So the moral of the story is, always do your homework before choosing a dedicated or cloud hosting provider. Ask about their security procedures, ask them about their PCI, HIPAA, FISMA, ISO 27001, SSAE 16, etc. Never take anything at face value. Ask them for their report on compliance.

When it comes to your security stance, paranoia is your greatest defense.

Using AIDE for file integrity monitoring (FIM) on CentOS

PCI-DSS 3.1 section 10.5.5 has the following requirement:

Use file-integrity monitoring or change-detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert).

For large solutions, I would suggest using a well known tool such as Tripwire Enterprise. However many small to mid size companies that have a small footprint within their card holder data environment (CDE), may not be able to afford this. So what can companies use to meet this requirement? Implement AIDE (Advanced Intrusion Detection Environment).

Taken from the projects website, AIDE creates a database from the regular expression rules that it finds from the config file(s). Once this database is initialized it can be used to verify the integrity of the files.

AIDE is a very simple (yet powerful) program that runs from cron checking your files (typically once a night), and it will scan your system looking for any changes in the directories its monitoring. There are a number of different ways to use this program, but I’ll outline one that I like to use.

My requirements:
1. I want the reports to run nightly.
2. All change reports are emailed to me so I can archive them for a year offsite.
3. Have the database automatically commit the additions, deletions, and changes to baseline each time its ran.

In the event my system was compromised, I want to ensure that the malicious user was not able to modify, or delete my previous reports. Therefore, I choose not to store them on the machine. Its true that once the malicious user gained access to my system, they could change my AIDE config on me, but at least my previous reports will be intact which should help me when determining what malicious changes this user made to my server. Please note that I am making an assumption here that you are already backing up your system nightly, which would include your AIDE database! If you do not currently have a backup strategy in place, get one. Tools such as AIDE helps identify what files a malicious user may have changed, but if they completely crippled the system, you will need to restore from backups.

Setting up AIDE is fairly straight forward. It exists in most of package repositories out there including most variants of Linux and BSD.

On Linux based systems, you can install it by:

[root@web01 ~]# yum install aide

Once you have AIDE installed, the default configuration from the upstream provider should give you a reasonable default aide.conf. But what if you wanted to add your website documentroot to this so you can keep track of what files are changing on your website? Well, we simple add the directory to the aide.conf by including:

[root@web01 ~]# vim /etc/aide.conf
...
/var/www/vhosts/domain.com
...

Now AIDE will be keeping track of our website. But adding your site may lead to very noisy reports because most websites implement caching. So this now becomes a balancing act to exclude directories that change often, yet retain enough of your sites critical content. We could just leave the entire directory in AIDE, but I know I personally don’t want to read a change report that contains 1,000 changes every day. So in the case of this wordpress site, I exclude the cache directory by appending the following to my custom configuration:

[root@web01 ~]# vim /etc/aide.conf
...
/var/www/vhosts/domain.com
!/var/www/vhosts/domain.com/web/wp-content/cache
...

The “!” means NOT to monitor that specific directory. You will need to run AIDE a few times and fine tune the configuration before you get a report that is useful for your specific needs.

On CentOS, I had to change the following setting in /etc/aide.conf for the initialization to work:

[root@web01 ~]# vim /etc/aide.conf
...
# Whether to gzip the output to database
gzip_dbout=no
...

Once you have your configuration tuned for your specific purposes, you first my initialize the database to create a baseline before you can start getting change reports. You do this by:

[root@web01 ~]# aide --init
[root@web01 ~]# mv -f /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Now, try making a basic change to /etc/hosts, then run a check on AIDE to see if it detects the change:

[root@web01 ~]# aide --check

If you are like me and would prefer not to have to log into 10 servers a day to run and view the reports, you can configure cron to run the report, and email you the results daily, while committing the changes to baseline. If you choose to go this route, it is critical that you review your change reports as they come in because we are essentially committing every change to the baseline. Here is how I configure cron:

[root@web01 ~]# crontab -e
# Perform daily change report
0 3 * * * /usr/sbin/aide --update | mail -s "AIDE Audit Report : web01.example.com" [email protected]

# Initialize the AIDE database once a day:
30 3 * * *  nice -19 /usr/sbin/aide --init;mv -f /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Posted below is an example report that AIDE would send me via email daily:

AIDE found differences between database and filesystem!!
Start timestamp: 2012-09-13 01:24:05

Summary:
Total number of files: 57620
Added files: 1
Removed files: 1
Changed files: 1

---------------------------------------------------
Added files:
---------------------------------------------------

added: /var/spool/cron/root

---------------------------------------------------
Removed files:
---------------------------------------------------

removed: /etc/.aide.conf.swp

---------------------------------------------------
Changed files:
---------------------------------------------------

changed: /etc/aide.conf

--------------------------------------------------
Detailed information about changes:
---------------------------------------------------

File: /etc/aide.conf
Size : 2381 , 2390
Mtime : 2012-09-13 01:24:05 , 2012-09-13 01:24:05
Ctime : 2012-09-13 01:24:05 , 2012-09-13 01:24:05
MD5 : b+qbBDYEPesd+NCR1VRQHQ== , rG5pNPghdweedpU/c0ieHw==
RMD160 : T081ixhqik4efC3dfeCOBDCKpP4= ,
qe8MV0eteklAKmlZ5LTubaOUNKo=
SHA256 : g4jstEtfU8BNu+43jkrxJc9Cpr2SABZj ,
a65iaV54XR4vu8/zbA4Tdfe2U+W5uPNY

So this reports tells me that root’s crontab was added, a swap file for aide.conf was removed, and I updated the /etc/aide.conf recently.

Please remember that utilizing a tool to provide file integrity monitoring is only one part of a defense in depth strategy. There is no silver bullet for system security, but every layer you add will increase your security footprint which helps you with taking a proactive approach to security.

Using PAM to enforce access based on time

Sometimes there is a need to restrict user access based on time. This could be access to one particular service, or all PAM enabled services. A common example is to only allow access for the user ‘bob’ monday through friday between 9:00AM – 6:00PM. This can be enforced by utilizing the pam_time module.

The pam_time module is an account module type. No arguments are passed directly to the module, but instead all configuration takes place within /etc/security/time.conf.

The time.conf operates based on rules, and each rule uses the following syntax:

services;ttys;users;times

Example Rules
Restrict SSHD access for bob to weekdays between 9:00AM – 7:00PM

sshd;*;bob;Wk0900-1900

Restrict ALL access for bob to weekdays between 9:00AM – 5:00PM

*;*;bob;Wk0900-1700

Restrict ALL access for ALL users except root to weekdays between 9:00AM – 5:00PM

sshd;*;bob;Wk0900-1700

Restrict SSH access for ALL users except bob and jane to weekdays between 9:00AM – 5:00PM

sshd;*;!bob|!jane;Wk0900-1700

To only allow bob to access SSH on Tuesdays between 3:23PM and 4:24PM:

sshd;*;bob;Tu1523-1624

Below is all the available abbreviates for the days of the week:

Mo : Monday Fr : Friday Wd : Sa/Su
Tu : Tuesday Sa : Saturday wk : Mo/Tu/We/Th/Fr
We : Wenesday Su : Sunday
Th : Thursday Al : All Days

Installation And Configuration
In our example, I am going be setting this up on a CentOS 5.x server. For the restricted user, the following variables will be used:

username: bob
allowed access times: 9:00AM - 6:00PM
restricted services: SSHD

First, add the user and time restriction to /etc/security/time.conf:

sshd;*;bob;Wk0900-1800

Now, update the pam module for login and sshd. You are including ‘account required pam_time.so‘. But I’ll post entire file for reference

cat /etc/pam.d/sshd
#%PAM-1.0
auth required pam_sepermit.so
auth include password-auth
account required pam_time.so
account required pam_nologin.so
account include password-auth
password include password-auth
# pam_selinux.so close should be the first session rule
session required pam_selinux.so close
session required pam_loginuid.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
session required pam_selinux.so open env_params
session optional pam_keyinit.so force revoke
session include password-auth
cat /etc/pam.d/system-auth
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth required pam_env.so
auth sufficient pam_unix.so nullok try_first_pass
auth requisite pam_succeed_if.so uid >= 500 quiet
auth required pam_deny.so
account required pam_time.so
account required pam_unix.so
account sufficient pam_localuser.so
account sufficient pam_succeed_if.so uid < 500 quiet
account required pam_permit.so
password requisite pam_cracklib.so try_first_pass retry=3 type=
password sufficient pam_unix.so md5 shadow nullok try_first_pass use_authtok
password required pam_deny.so
session optional pam_keyinit.so revoke
session required pam_limits.so
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so

And finally, restart SSH

service sshd restart