Using AIDE for file integrity monitoring (FIM) on Ubuntu or Debian

PCI-DSS 3.1 section 10.5.5 has the following requirement:

Use file-integrity monitoring or change-detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert).

For large solutions, I would suggest using a well known tool such as Tripwire Enterprise. However many small to mid size companies that have a small footprint within their card holder data environment (CDE), may not be able to afford this. So what can companies use to meet this requirement? Implement AIDE (Advanced Intrusion Detection Environment).

Taken from the projects website, AIDE creates a database from the regular expression rules that it finds from the config file(s). Once this database is initialized it can be used to verify the integrity of the files.

AIDE is a very simple (yet powerful) program that runs from cron checking your files (typically once a night), and it will scan your system looking for any changes in the directories its monitoring. There are a number of different ways to use this program, but I’ll outline one that I like to use.

My requirements:
1. I want the reports to run nightly.
2. All change reports are emailed to me so I can archive them for a year offsite.
3. Have the database automatically commit the additions, deletions, and changes to baseline each time its ran.

In the event my system was compromised, I want to ensure that the malicious user was not able to modify, or delete my previous reports. Therefore, I choose not to store them on the machine. Its true that once the malicious user gained access to my system, they could change my AIDE config on me, but at least my previous reports will be intact which should help me when determining what malicious changes this user made to my server. Please note that I am making an assumption here that you are already backing up your system nightly, which would include your AIDE database! If you do not currently have a backup strategy in place, get one. Tools such as AIDE helps identify what files a malicious user may have changed, but if they completely crippled the system, you will need to restore from backups.

Setting up AIDE is fairly straight forward. It exists in most of package repositories out there including most variants of Linux and BSD.

On Ubuntu or Debian based systems, you can install it by:

[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install aide

Now to setup some basic configurations, such as the email notifications, update type, etc, modify the AIDE system configuration file according:

[root@web01 ~]# vim /etc/default/aide
...
FQDN=web01.domain.com
MAILSUBJ="Daily AIDE report for $FQDN"
[email protected]
QUIETREPORTS=no
COMMAND=update
COPYNEWDB=yes
...

Now that AIDE is installed, and the basic preferences are in place, its now time to check out the main configuration files. The default configuration from the upstream provider should give you a reasonable default configuration. But what if you wanted to add your website documentroot to this so you can keep track of what files are changing on your website? The Debian/Ubuntu way of configuring AIDE is a bit different from the CentOS/RHEL method.

All the configuration files resides in /etc/aide/aide.conf.d/. The number of the file appears to be used by the AIDE wrapper to use for deciding which order to process these files. The AIDE documentation seems to indicate that the most general rules should be processed last, so I’ll default to creating my servers profile with 50_aide_CUSTOM-RULES.

So lets say I want to monitor my documentroot, here is how this would be setup:

[root@web01 ~]# vim /etc/aide/aide.conf.d/50_aide_CUSTOM-RULES
...
/var/www/vhosts/domain.com FULL
...

Now AIDE will be keeping track of our website. But adding your site may lead to very noisy reports because most websites implement caching. So this now becomes a balancing act to exclude directories that change often, yet retain enough of your sites critical content. We could just leave the entire directory in AIDE, but I know I personally don’t want to read a change report that contains 1,000 changes every day. So in the case of this wordpress site, I exclude the cache directory by appending the following to my custom configuration:

[root@web01 ~]# vim /etc/aide/aide.conf.d/50_aide_CUSTOM-RULES
...
/var/www/vhosts/domain.com Full
!/var/www/vhosts/domain.com/web/wp-content/cache
...

The “!” means NOT to monitor that specific directory. You will need to run AIDE a few times and fine tune the configuration before you get a report that is useful for your specific needs.

Anytime a change is made to your AIDE configuration, you need to rebuild the AIDE run time configuration, and initialize the database. You do that by:

[root@web01 ~]# update-aide.conf
[root@web01 ~]# aideinit -y -f

Now, try making a basic change to /etc/hosts, then run a check on AIDE to see if it detects the change and emails out the report:

[root@web01 ~]# /etc/cron.daily/aide

If you wanted to just quickly test AIDE to ensure it picks up your changes, but won’t commit them to baseline, you can perform a one-time scan by:

[root@web01 ~]# aide.wrapper

To receive nightly AIDE reports, no further configuration is needed since Ubuntu/Debian already setup a cron job that will run AIDE automatically in /etc/cron.daily/aide. This will run whenever your system normally runs the cron.daily jobs, which is defined in /etc/crontab.

Posted below is an example report that AIDE would send me via email daily:

This is an automated report generated by the Advanced Intrusion Detection 
Environment on web01.domain.com started at 2016-03-07 13:16:35.

AIDE returned with exit code 7. Added, removed and changed files detected!
AIDE post run information
output database /var/lib/aide/aide.db.new was copied to /var/lib/aide/aide.db as requested by cron job configuration
End of AIDE post run information

AIDE produced no errors.

Output of the daily AIDE run (83 lines):
AIDE 0.15.1 found differences between database and filesystem!!
Start timestamp: 2016-03-07 13:16:35

Summary:
  Total number of files:	77937
  Added files:			2
  Removed files:		3
  Changed files:		7


---------------------------------------------------
Added files:
---------------------------------------------------

f++++++++++++++++: /var/log/aide/aide.log.0
d++++++++++++++++: /var/www/vhosts/domain.com/new

---------------------------------------------------
Removed files:
---------------------------------------------------

f----------------: /var/www/vhosts/domain.com/blah
f----------------: /var/www/vhosts/domain.com/test
d----------------: /var/www/vhosts/domain.com/test1

---------------------------------------------------
Changed files:
---------------------------------------------------

f   p.g    . A. .: /var/log/aide/aide.log
d =.... mc.. .. .: /var/spool/postfix/active
d =.... mc.. .. .: /var/spool/postfix/incoming
d =.... mc.. .. .: /var/spool/postfix/maildrop
F =.... mc.. ..  : /var/spool/postfix/public/pickup
F =.... mc.. ..  : /var/spool/postfix/public/qmgr
d =.... mc.. .. .: /var/www/vhosts/domain.com

---------------------------------------------------
Detailed information about changes:
---------------------------------------------------


File: /var/log/aide/aide.log
 Perm     : -rw-------                       , -rw-r-----
 Gid      : 0                                , 4
 ACL      : old = A:
----
user::rw-
group::---
other::---
----
                  D: 
            new = A:
----
user::rw-
group::r--
other::---
----
                  D: 

Directory: /var/spool/postfix/active
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23

Directory: /var/spool/postfix/incoming
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23

Directory: /var/spool/postfix/maildrop
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:23

FIFO: /var/spool/postfix/public/pickup
 Mtime    : 2016-03-07 13:12:37              , 2016-03-07 13:17:37
 Ctime    : 2016-03-07 13:12:37              , 2016-03-07 13:17:37

FIFO: /var/spool/postfix/public/qmgr
 Mtime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:36
 Ctime    : 2016-03-07 13:10:36              , 2016-03-07 13:13:36

Directory: /var/www/vhosts/domain.com
 Mtime    : 2016-03-07 13:03:25              , 2016-03-07 13:16:17
 Ctime    : 2016-03-07 13:03:25              , 2016-03-07 13:16:17

End of AIDE output.

The check was done against /var/lib/aide/aide.db with the following characteristics:
 Size     : 13041865
 Bcount   : 25480
 Mtime    : 2016-03-07 13:13:23
 Ctime    : 2016-03-07 13:13:23
 Inode    : 273628
 RMD160   : bIthG3Q5FiJmj4CIYdASjJx5Ygc=
 TIGER    : omto0nb3/oIqIiKHEjnbhjvXeGdfycbV
 SHA256   : VJPGKy61GxGfcSrjJFbrP879y/skJaiQ
 SHA512   : 7pz3FdYh8TvoNOqjxWBToZQNG6oxmrrp
 CRC32    : 1dYwqA==
 HAVAL    : LBFzyApqoYn7ogzoROG5FpneBO1s7R3p
 GOST     : iJ1tWPLtYaxxoFDHZEW8gxCS3/pVlS1G

The AIDE run created a new database /var/lib/aide/aide.db.new with the following characteristics:
 Size     : 13041834
 Bcount   : 25480
 Inode    : 273627
 RMD160   : 4TKRFSc0nt/VGDVvPEY8U6YNzaw=
 TIGER    : o4RzDHHWBlH+Zt3P7vI8GHHgGV1OecrC
 SHA256   : Gher/aINaU8r73/lQEWLQQSsKqP7sGjO
 SHA512   : D0/w3S6NOLZHw7D7dt1QxYBXe6miP5hF
 CRC32    : 5SRdpg==
 HAVAL    : pe7+ai57TPpW34NjJgTQxs+cQsFJ9zq0
 GOST     : RrIiyspbpKEb5wEGSG2HTYM7N6NUtKSv

End of AIDE daily cron job at 2016-03-07 13:18, run time 102 seconds

So this reports tells me that a log file for AIDE was rotated out, a new folder was created in my DocumentRoot called new, and the files/folders blah, test, and test1 where removed from my DocumentRoot.

Please remember that utilizing a tool to provide file integrity monitoring is only one part of a defense in depth strategy. There is no silver bullet for system security, but every layer you add will increase your security footprint which helps you with taking a proactive approach to security.

Chroot SFTP-only users

In an environment where you have multiple developers working on different sites, or perhaps you have multiple clients hosting their websites on your solution, restricting access for those users can become important.

Security becomes a concern in a normal FTP environment as it doesn’t take much for a user to simply ‘cd ..’ and see what other users are on your server. We want a way to simply lock those users only into their home directories so they cannot essentially ‘break out’.

It is important to consider how you are going to give those chrooted SFTP users access to their directories. In a chroot, simply using a symlink will not work as the filesystem will have no knowledge of the data outside that chroot. Therefore you would have to consider either:

1. Chrooting the user to their web sites home directory
2. Chrooting the user to their home directory, then create a bind mount to their website.

Both have their pro’s and con’s, however it could be argued that chrooting them to their home directory and using bind mounts is more secure since its offers an added layer of security since you are not relying solely on permissions, but also on the chroot itself. For the purposes of this article, we are going to default to chrooting users to their home directory, then creating a bind mount to their website.

To get started, first, create the restrict SFTP-only group

[root@web01 ~]# groupadd sftponly

Next, edit the sshd config to setup the internal-sftp subsystem. You will need to comment out the first entry as shown below:

[root@web01 ~]# vim /etc/ssh/sshd_config
...
# Subsystem       sftp    /usr/libexec/openssh/sftp-server
Subsystem     sftp   internal-sftp
...

Now at the very bottom of /etc/ssh/sshd_config, setup the following block. It is important that this is created at the very end of the file:

[root@web01 ~]# vim /etc/ssh/sshd_config
...
Match Group sftponly
     ChrootDirectory %h
     X11Forwarding no
     AllowTCPForwarding no
     ForceCommand internal-sftp
...

Then restart SSHD by:

[root@web01 ~]# service sshd restart

Now that the foundation is complete, we can add chrooted SFTP-only users. You will notice I will set the home directory to /home/chroot/bob. This is optional, but I prefer it so you can quickly tell the difference between regular users and SFTP-only users. To create a new user called bob with the proper group assignments and permissions:

[root@web01 ~]# mkdir -p /home/chroot
[root@web01 ~]# useradd -d /home/chroot/bob -s /bin/false -G sftponly bob
[root@web01 ~]# passwd bob
[root@web01 ~]# chmod 755 /home/chroot/bob
[root@web01 ~]# chown root:root /home/chroot/bob

Users will not be able to write any data within their home directory since the home directory MUST be owned by root. If they want to be able to write files, create them a writable directory by:

[root@web01 ~]# mkdir /home/chroot/bob/files
[root@web01 ~]# chown bob:bob /home/chroot/bob/files

Now to allow them to access their content in /var/www/vhosts/domain.com, you need to create a bind mount:

[root@web01 ~]# vim /etc/fstab
...
/var/www/vhosts/domain.com   /home/chroot/bob/domain.com        none    bind    0 0
...

Finally, create the placeholder folder, and mount the bind mount:

[root@web01 ~]# mkdir /home/chroot/bob/domain.com
[root@web01 ~]# mount -a

Confirm user bob is setup in the right group, and that the root directory ‘/home/chroot/bob/domain.com has group writable perms. In my specific example, as the directory has the ownership apache:apache, I had to do the following:

[root@web01 ~]# usermod -a -G apache bob
[root@web01 ~]# chmod 775 /var/www/vhosts/domain.com