Malware Detection – rkhunter

Following on my previous articles, there are several good malware detection tools out there. These scanners help notify you of malware, hopefully before your clients notify you. Some of the common ones include:

chkrootkit
Linux Malware Detect (maldet)
rkhunter

Each have their own strong points, and they certainly compliment each other nicely when using them together depending on the solutions security strategy.

Rkhunter is similar in nature to chkrootkit, and I feel that both complement each other nicely. Taken from wikipedia’s page:

rkhunter (Rootkit Hunter) is a Unix-based tool that scans for rootkits, backdoors and possible local exploits. It does this by comparing SHA-1 hashes of important files with known good ones in online database, searching for default directories (of rootkits), wrong permissions, hidden files, suspicious strings in kernel modules, and special tests for Linux and FreeBSD.

Procedure

On CentOS systems, rkhunter can be installed from the EPEL repositories. If you do not have EPEL installed, you can get it setup by:

Installing rkhunter is pretty straight forward as shown below:

# CentOS 5 / RedHat 5
[root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
[root@web01 ~]# yum install rkhunter mailx

# CentOS 6 / RedHat 6
[root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[root@web01 ~]# yum install rkhunter mailx

# CentOS 7 / RedHat 7
[root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
[root@web01 ~]# yum install rkhunter mailx

# Ubuntu / Debian
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install rkhunter mailutils

Now that the installation is out of the way, lets configure rkhunter to send email if warning is found during scan:

[root@web01 ~]# vim /etc/rkhunter.conf
# Change
MAIL-ON-WARNING=""
# To
MAIL-ON-WARNING="[email protected]"

Now fetch the latest updates, create a baseline, and run a on-demand scan:

[root@web01 ~]# rkhunter --update 
[root@web01 ~]# rkhunter --propupd
[root@web01 ~]# rkhunter -sk -c

On CentOS and RHEL, configure cron so this runs automatically:

First, confirm the cronjob exists:

[root@web01 ~]# cat /etc/cron.daily/rkhunter

Now, update the rkhunter configuration with your email address so you can receive the nightly reports:

[root@web01 ~]# vi /etc/sysconfig/rkhunter
# Change
MAILTO=root@localhost
# To
[email protected]

On Ubuntu based systems, confirm the cronjob exists:

[root@web01 ~]# cat /etc/cron.daily/rkhunter

Now, update the rkhunter configuration with your email address so you can receive the nightly reports:

[root@web01 ~]# vi /etc/default/rkhunter
# Change
APT_AUTOGEN="false"
REPORT_EMAIL="root"

# To
APT_AUTOGEN="true"
REPORT_EMAIL="[email protected]"

NOTE: See https://help.ubuntu.com/community/RKhunter for more information about APT_AUTOGEN.

Malware Detection – Linux Malware Detect (maldet)

Following on my previous articles, there are several good malware detection tools out there. These scanners help notify you of malware, hopefully before your clients notify you. Some of the common ones include:

chkrootkit
Linux Malware Detect (maldet)
rkhunter

Each have their own strong points, and they certainly compliment each other nicely when using them together depending on the solutions security strategy.

Maldet happens to be one of the more useful tools I use often for checking suspected compromised servers. This scanner is a very in depth tool that is great at locating backdoors within binaries and website content, as well as other malicious tools. Below is a quick introduction for installing and configuring Maldet, as well as setting up a nightly cron job.

Procedure

As a prerequisite to have Maldet’s utilize ClamAV’s engine, install ClamAV. This will significantly speed up the scan time, and it should be considered a hard requirement:

# CentOS / RedHat
[root@web01 ~]# yum install clamav

# Ubuntu / Debian
[root@web01 ~]# apt-get install clamav

Then install it from source by:

[root@web01 ~]# wget http://www.rfxn.com/downloads/maldetect-current.tar.gz
[root@web01 ~]# tar -xf maldetect-current.tar.gz
[root@web01 ~]# cd maldetect-*
[root@web01 ~]# ./install.sh

Now you can run the scan. You could run the scan against the / directory, but depending on how much data you have, this could take hours or days. Normally I use this tool to check my website documentroot’s as I have other tools installed to monitor other parts of my system. So assuming our website content is in /var/www, here is how you scan that directory:

[root@web01 ~]# maldet -u
[root@web01 ~]# maldet --scan-all /var/www

If you would like to scan the entire server, which is good to do if you suspect you have been compromised, then you can run:

[root@web01 ~]# maldet --scan-all /

To setup a nightly cronjob, scanning your content and emailing you a report if something is found, first we need to modify the conf.maldet file as follows:

[root@web01 ~]# vi /usr/local/maldetect/conf.maldet
...
email_alert=1
email_addr="[email protected]"
...

Maldet automatically installs a cronjob for this, and you can feel free to modify /etc/cron.daily/maldet to your liking. However I prefer to remove it and setup my own. Please choose whatever works best for your needs. Here is one method:

[root@web01 ~]# rm /etc/cron.daily/maldet
[root@web01 ~]# crontab -e
...
32 3 * * * /usr/local/maldetect/maldet -d -u ; /usr/local/maldetect/maldet --scan-all /var/www
...

Now if maldet detects anything that flags as malware, it will send you a report via email for you to investigate.

If you are like me and want Maldet to send you daily reports via email regardless of if it found something or not, then simply setup the following cron job:

[root@web01 ~]# rm /etc/cron.daily/maldet
[root@web01 ~]# crontab -e
...
32 3 * * * /usr/local/maldetect/maldet -d -u ; /usr/local/maldetect/maldet --scan-all /var/www | mail -s "Maldet Report : INSERT_HOSTNAME_HERE" [email protected]
...

The reports will look similar to the following:

Linux Malware Detect v1.5
           (C) 2002-2015, R-fx Networks 
           (C) 2015, Ryan MacDonald 
This program may be freely redistributed under the terms of the GNU GPL v2
maldet(18252): {scan} signatures loaded: 10822 (8908 MD5 / 1914 HEX / 0 USER)
maldet(18252): {scan} building file list for /var/www, this might take awhile...
maldet(18252): {scan} setting nice scheduler priorities for all operations: cpunice 19 , ionice 6
maldet(18252): {scan} file list completed in 35s, found 278654 files...
maldet(18252): {scan} found clamav binary at /usr/bin/clamscan, using clamav scanner engine...
maldet(18252): {scan} scan of /var/www (278654 files) in progress...
maldet(18252): {scan} scan completed on /var/www: files 278654, malware hits 0, cleaned hits 0, time 2355s
maldet(18252): {scan} scan report saved, to view run: maldet --report 151111-0420.12250

Malware Detection – Chkrootkit

Imagine one day you get a phone call from one of your customers. Great, a sale! Nope, just kidding! They are calling to tell you that your site has been hacked. What do you say? How did they find out before you did?

This is a scenario that every sysadmin dreads. How can you prevent something like this? A good defense in depth security strategy goes a long way in preventing this. This includes, WAF’s, firewalls, file integrity monitoring, IDS, two factor authentication, ASV scanning, event management, etc, etc.

The more layers you have, the better chance you have at either mitigating the attack, or being notified of the compromise shortly after it happens so you can react quickly and not have your clients inform you of the problem.

Below is another layer you can add: chkrootkit
Taken from the following wikipedia article (http://en.wikipedia.org/wiki/Chkrootkit):

chkrootkit (Check Rootkit) is a common Unix-based program intended to help system administrators check their system for known rootkits. It is a shell script using common UNIX/Linux tools like the strings and grep commands to search core system programs for signatures and for comparing a traversal of the /proc filesystem with the output of the ps (process status) command to look for discrepancies.

The document below will outline how to install and configure it for CentOS and Ubuntu, including how to run this nightly and email you the report for review.

Procedure

Installing chkrootkit is pretty straight forward as shown below:

# CentOS 5 / RedHat 5
[root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
[root@web01 ~]# yum install chkrootkit mailx

# CentOS 6 / RedHat 6
[root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[root@web01 ~]# yum install chkrootkit mailx

# CentOS 7 / RedHat 7
[root@web01 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
[root@web01 ~]# yum install chkrootkit mailx

# Ubuntu / Debian
[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install chkrootkit mailutils

Perform a one time scan by running:

[root@web01 ~]# chkrootkit

Finally, lets setup a nightly cronjob and receive the report via email for review:

[root@web01 ~]# crontab -e
12 3 * * * /usr/sbin/chkrootkit | mail -s "chkrootkit Report : hostname.yourservername.com" [email protected]

Keep in mind that just setting up these tools doesn’t take care of the problem itself. The sysadmin needs to actively review any and all logs, software, etc. So with this tool and any security tool, identify false positives early on and exclude them from your reports so they are easier to read. This will help malware from slipping through the cracks of your logs and daily reports.

Several more articles will be following this in a series of sorts for malware detection. Each performs essentially the same function, but they go about it in different ways, each having their own benefits. Find the one(s) that work best for your solution.
chkrootkit
Linux Malware Detect (maldet)
rkhunter

Duplicity manager

Coming up with a secure and cost effective backup solution can be a daunting task as there are many considerations that much be taken into account. Some of the more basic items to think about are:

- Where to store your backups?
- Is the storage medium redundant?
- How will data retention will handled?
- How will the data at rest be encrypted?

A tool that I prefer for performing encrypted, bandwidth efficient backups to a variety of remote backends such as Rackspace Cloud Files, Amazon S3, and many others is Duplicity.

Taken from Duplicity’s site, (http://duplicity.nongnu.org), Duplicity back directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.

Duplicity-manager was created to act as a wrapper script for the tasks I commonly perform with Duplicity.

Features

- Simple invocation from cron for nightly backups.
- All in one script for performing backups, restores, searching for content from specific time period.
- Provides an optional menu driven interface to make backups as painless as possible.

Configuration

The currently configurable options are listed below:

# Configuring either Rackspace Cloud Files or Amazon S3 backends

# List of directories to backup
INCLUDE_LIST=( /etc /var/www /var/lib/mysqlbackup )

# GPG Passphrase for encrypting data at rest
# You can use the following to generate a decent GPG passphrase, just be sure
# to store it someone secure off this server.
# < /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c64
export PASSPHRASE=YOUR_PASSPHRASE

# Backup Retention 
retention_type=remove-older-than
retention_max=14D
 
# Number of full backups to keep (alternative to above)
# retention_type=remove-all-but-n-full
# retention_max=3

# Force Full Backup Every XX Days
full_backup_days=7D

# Restore Directory
restore=/tmp

Usage

./duplicity-manager.sh 

Options:

--backup:                      runs a normal backup based off retention settings
--backup-force-full:           forces a full backup
--list-files [age]:            lists the files currently stored in backups
--restore-all [age]:           restores everything to restore directory
--restore-single [age] [path]: restores a specific file/dir to restore directory
--show-backups:                lists full and incremental backups in the archive
--menu:                        user friendly menu driven interface

Examples:

duplicity-manager.sh --list-files 0D              Lists the most recent files in archive
duplicity-manager.sh --restore-all 2D             Restores everything from 2 days ago
duplicity-manager.sh --restore-single 0D var/www/ Restores /var/www from latest backup

Implementation

Download script to desired directory and set it to be executable:

# Linux based systems
cd /root
git clone https://github.com/stephenlang/duplicity-manager

After configuring the tunables in the script (see above), create a cron job to execute the script one a day:

# Linux based systems
crontab -e
10 3 * * * /root/duplicity-manager/duplicity-manager.sh

As with any backup solution, it is critical that you test your backups often to ensure your data is recoverable in the event a restore is needed.