When troubleshooting performance issues with Apache that happened earlier in the day or week, it is very useful to parse the logs quickly to determine quick facts about the inbound connection rates. It may reveal a period of increased traffic that needs to be investigated further, therefore giving you that thread to begin unraveling the problem.
To get the total connections per day for a website, run the following:
[root@web01 ~]# cat /var/log/httpd/www.example.com-access.log | awk '{print $4}' | cut -d: -f1 |uniq -c
1247345 [20/Feb/2017
1331908 [21/Feb/2017
1295677 [22/Feb/2017
1435275 [23/Feb/2017
1023423 [24/Feb/2017
1342332 [25/Feb/2017
1293422 [26/Feb/2017
2131198 [27/Feb/2017
To get the total connections per day for each website on the server, run the following:
[root@web01 ~]# for i in `ls /var/log/httpd/*-access.log`; do echo $i && cat $i | awk '{print $4}' | cut -d: -f1 |uniq -c && echo ""; done
/var/log/httpd/www.example.com-access.log
1247345 [20/Feb/2017
1331908 [21/Feb/2017
1295677 [22/Feb/2017
1435275 [23/Feb/2017
1023423 [24/Feb/2017
1342332 [25/Feb/2017
1293422 [26/Feb/2017
2131198 [27/Feb/2017
/var/log/httpd/www.example02.com-access.log
2542 [20/Feb/2017
7586 [21/Feb/2017
4776 [22/Feb/2017
2975 [23/Feb/2017
16756 [24/Feb/2017
9874 [25/Feb/2017
1638 [26/Feb/2017
9654 [27/Feb/2017
To get the connections per hour for a specific day, run the following:
[root@web01 ~]# grep "27/Feb" /var/log/httpd/www.example.com-access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":00"}' | sort -n | uniq -c
50205 03:00
90516 04:00
64837 05:00
47410 06:00
44876 07:00
41098 08:00
38996 09:00
37234 10:00
43704 11:00
58702 12:00
58922 13:00
72592 14:00
77792 15:00
88882 16:00
80815 17:00
98287 18:00
617857 19:00
90507 20:00
98568 21:00
147584 22:00
181814 23:00
Based off that output, there was a massive spike in connections during the 9:00PM hour (19:00). So now lets break the 9:00PM hour down to show the connections per minute:
[root@web01 ~]# grep "27/Feb/2017:19" /var/log/httpd/www.example.com-access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c | awk '{ if ($1 > 10) print $0}'
1629 19:00
1664 19:01
1840 19:02
10493 19:03
13728 19:04
17608 19:05
1377 19:06
2333 19:07
1980 19:08
2056 19:09
2123 19:10
...
1997 19:57
1631 19:58
1988 19:59
As shown above, there was some sort of traffic spike that occurred between 9:03PM – 9:05PM. As the window has been narrowed down to a 3 minute period, more specific analysis can be performed. The examples below will focus on what was happening around 9:03PM.
To list the top 10 IP’s accessing the site during around 9:03PM
[root@web01 ~]# grep "27/Feb/2017:19:03" /var/log/httpd/www.example.com-access.log | awk '{print $1}' | sort -nr | uniq -c |sort -nr | head
To list the top most called elements on the site:
[root@web01 ~]# grep "27/Feb/2017:19:03" /var/log/httpd/www.example.com-access.log | awk '{print $7}' | sort -nr | uniq -c | sort -nr | head
To show the bandwidth for a domain use the command below:
# Daily bandwidth total
[root@web01 ~]# grep '27/Feb/2017:' /var/log/httpd/www.example.com-access.log | grep -oP 'HTTP/1.[01]" [0-9]{3} [0-9]+' | awk '{SUM+=$3} END { print SUM / 1024 / 1024 / 1024 " GB" }'
# Monthly bandwidth total
[root@web01 ~]# grep '/Feb/2017:' /var/log/httpd/www.example.com-access.log | grep -oP 'HTTP/1.[01]" [0-9]{3} [0-9]+' | awk '{SUM+=$3} END { print SUM / 1024 / 1024 / 1024 " GB" }'
To get a count of status codes to identify any trends:
# Get all status codes
[root@web01 ~]# cat /var/log/httpd/www.example.com-access.log |awk '{print $9}' | sort -nr | uniq -c |sort -nr
36355 200
4896 304
3942 404
1599 302
301 301
195 403
4 400
3 401
# Get summary of top 10 404's:
[root@web01 ~]# awk '($9 ~ /404/)' /var/log/httpd/www.example.com-access.log | awk '{print $9,$7}' | sort -nr | uniq -c |sort -nr | head
1369 404 /apple-touch-icon-precomposed.png
1369 404 /apple-touch-icon.png
502 404 /apple-touch-icon-120x120-precomposed.png
502 404 /apple-touch-icon-120x120.png
22 404 /apple-touch-icon-152x152-precomposed.png
22 404 /apple-touch-icon-152x152.png
21 404 /news/html
5 404 /components/com_foxcontact/lib/file-uploader.php
3 404 /blog/wp-login.php
1 404 /author/wp-login.php