Apache quick stats

When troubleshooting performance issues with Apache that happened earlier in the day or week, it is very useful to parse the logs quickly to determine quick facts about the inbound connection rates. It may reveal a period of increased traffic that needs to be investigated further, therefore giving you that thread to begin unraveling the problem.

To get the total connections per day for a website, run the following:

[[email protected] ~]# cat /var/log/httpd/www.example.com-access.log | awk '{print $4}' | cut -d: -f1 |uniq -c
   1247345 [20/Feb/2017
   1331908 [21/Feb/2017
   1295677 [22/Feb/2017
   1435275 [23/Feb/2017
   1023423 [24/Feb/2017
   1342332 [25/Feb/2017
   1293422 [26/Feb/2017
   2131198 [27/Feb/2017

To get the total connections per day for each website on the server, run the following:

[[email protected] ~]# for i in `ls /var/log/httpd/*-access.log`; do echo $i && cat $i | awk '{print $4}' | cut -d: -f1 |uniq -c && echo ""; done
/var/log/httpd/www.example.com-access.log
   1247345 [20/Feb/2017
   1331908 [21/Feb/2017
   1295677 [22/Feb/2017
   1435275 [23/Feb/2017
   1023423 [24/Feb/2017
   1342332 [25/Feb/2017
   1293422 [26/Feb/2017
   2131198 [27/Feb/2017

/var/log/httpd/www.example02.com-access.log
   2542 [20/Feb/2017
   7586 [21/Feb/2017
   4776 [22/Feb/2017
   2975 [23/Feb/2017
  16756 [24/Feb/2017
   9874 [25/Feb/2017
   1638 [26/Feb/2017
   9654 [27/Feb/2017

To get the connections per hour for a specific day, run the following:

[[email protected] ~]# grep "27/Feb" /var/log/httpd/www.example.com-access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":00"}' | sort -n | uniq -c
  50205 03:00
  90516 04:00
  64837 05:00
  47410 06:00
  44876 07:00
  41098 08:00
  38996 09:00
  37234 10:00
  43704 11:00
  58702 12:00
  58922 13:00
  72592 14:00
  77792 15:00
  88882 16:00
  80815 17:00
  98287 18:00
 617857 19:00
  90507 20:00
  98568 21:00
 147584 22:00
 181814 23:00

Based off that output, there was a massive spike in connections during the 9:00PM hour (19:00). So now lets break the 9:00PM hour down to show the connections per minute:

[[email protected] ~]# grep "27/Feb/2017:19" /var/log/httpd/www.example.com-access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c | awk '{ if ($1 > 10) print $0}'
   1629 19:00
   1664 19:01
   1840 19:02
  10493 19:03
  13728 19:04
  17608 19:05
   1377 19:06
   2333 19:07
   1980 19:08
   2056 19:09
   2123 19:10
...
   1997 19:57
   1631 19:58
   1988 19:59

As shown above, there was some sort of traffic spike that occurred between 9:03PM – 9:05PM. As the window has been narrowed down to a 3 minute period, more specific analysis can be performed. The examples below will focus on what was happening around 9:03PM.

To list the top 10 IP’s accessing the site during around 9:03PM

[[email protected] ~]# grep "27/Feb/2017:19:03" /var/log/httpd/www.example.com-access.log | awk '{print $1}' | sort -nr | uniq -c |sort -nr | head

To list the top most called elements on the site:

[[email protected] ~]# grep "27/Feb/2017:19:03" /var/log/httpd/www.example.com-access.log | awk '{print $7}' | sort -nr | uniq -c | sort -nr | head

To show the bandwidth for a domain use the command below:

# Daily bandwidth total
[[email protected]eb01 ~]# grep '27/Feb/2017:' /var/log/httpd/www.example.com-access.log | grep -oP 'HTTP/1.[01]" [0-9]{3} [0-9]+' | awk '{SUM+=$3} END { print SUM / 1024 / 1024 / 1024 " GB" }'

# Monthly bandwidth total
[[email protected] ~]# grep '/Feb/2017:' /var/log/httpd/www.example.com-access.log | grep -oP 'HTTP/1.[01]" [0-9]{3} [0-9]+' | awk '{SUM+=$3} END { print SUM / 1024 / 1024 / 1024 " GB" }'

To get a count of status codes to identify any trends:

# Get all status codes
[[email protected] ~]# cat /var/log/httpd/www.example.com-access.log |awk '{print $9}' | sort -nr | uniq -c |sort -nr
  36355 200
   4896 304
   3942 404
   1599 302
    301 301
    195 403
      4 400
      3 401

# Get summary of top 10 404's:
[[email protected] ~]# awk '($9 ~ /404/)' /var/log/httpd/www.example.com-access.log | awk '{print $9,$7}' | sort -nr | uniq -c |sort -nr | head
   1369 404 /apple-touch-icon-precomposed.png
   1369 404 /apple-touch-icon.png
    502 404 /apple-touch-icon-120x120-precomposed.png
    502 404 /apple-touch-icon-120x120.png
     22 404 /apple-touch-icon-152x152-precomposed.png
     22 404 /apple-touch-icon-152x152.png
     21 404 /news/html
      5 404 /components/com_foxcontact/lib/file-uploader.php
      3 404 /blog/wp-login.php
      1 404 /author/wp-login.php