Curl is the swiss army knife for gathering information while troubleshooting websites. There are many ways to use curl and this guide is simply documenting the ones I often use, but can never seem to remember the specific flags.
General Usage
Test a page behind a username/password prompt:
[[email protected] ~]# curl --user name:password http://www.example.com
Download files from github:
[[email protected] ~]# curl -O https://raw.github.com/username/reponame/master/filename
Download content via curl to a specific filename:
[[email protected] ~]# curl -o archive.tgz https://www.example.com/file.tgz
Run a script from a remote source on server. Understand the security implications of doing this as it can be dangerous!
[[email protected] ~]# source <( curl -sL https://www.example.com/your_script.sh)
Have curl make the connection through a specific interface on the server:
[[email protected] ~]# curl --interface bond0 icanhazip.com
Website Troubleshooting
Receive output detailing the HTTP response and request headers, following all redirects, and discarding the page body:
[use[email protected] ~]# curl -Lsvo /dev/null https://www.example.com
Test a domain hosted on the local server, bypassing DNS:
[us[email protected] ~]# curl -sLH "host: www.example.com" localhost
Test a domain against specific IP, bypassing the need for modifying /etc/hosts:
[[email protected] ~]# curl -IL https://www.example.com --resolve www.example.com:443:123.123.123.123 or [[email protected] ~]# curl -Lsvo /dev/null --resolve 'example.com:443:123.123.123.123' https://www.example.com/ or [[email protected] ~]# curl -Lsvo /dev/null --header "Host: example.com" https://123.123.123.123/
Send request using a specific user-agent. Sometimes a server will have rules in place to block the default curl user-agent string. Or perhaps you need to test using a specific user-agent. You can pass specific user-agent strings by running:
[[email protected] ~]# curl --user-agent "USER_AGENT_STRING_HERE" www.example.com
A comprehensive listing of available user-agent strings available resides at:
http://www.useragentstring.com/pages/useragentstring.php
For example, lets say you need to test a mobile device user-agent to see if a custom redirect works:
[[email protected] ~]# curl -H "User-Agent: Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19" -IL http://www.example.com/about/contact-us HTTP/1.1 301 Moved Permanently Date: Tue, 17 Nov 2015 18:10:09 GMT Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips Location: http://www.example.com/mobi/contact.php Content-Type: text/html; charset=iso-8859-1
SSL/TLS Testing
Test to see if the site supports specific SSL/TLS protocols:
[[email protected] ~]# curl --sslv2 https://www.example.com [[email protected] ~]# curl --sslv3 https://www.example.com [[email protected] ~]# curl --tlsv1 https://www.example.com [[email protected] ~]# curl --tlsv1.0 https://www.example.com [[email protected] ~]# curl --tlsv1.1 https://www.example.com [[email protected] ~]# curl --tlsv1.2 https://www.example.com
Performance Troubleshooting
Load times can be impacted by a number of things, such as the TLS handshake, DNS lookup time, redirects, transfers, upload/downloads, etc. The curl command shown below will break down the times for each accordingly:
[[email protected] ~]# curl -Lsvo /dev/null https://www.example.com/ -w "\nContent Type: %{content_type} \ \nHTTP Code: %{http_code} \ \nHTTP Connect:%{http_connect} \ \nNumber Connects: %{num_connects} \ \nNumber Redirects: %{num_redirects} \ \nRedirect URL: %{redirect_url} \ \nSize Download: %{size_download} \ \nSize Upload: %{size_upload} \ \nSSL Verify: %{ssl_verify_result} \ \nTime Handshake: %{time_appconnect} \ \nTime Connect: %{time_connect} \ \nName Lookup Time: %{time_namelookup} \ \nTime Pretransfer: %{time_pretransfer} \ \nTime Redirect: %{time_redirect} \ \nTime Start Transfer (TTFB): %{time_starttransfer} \ \nTime Total: %{time_total} \ \nEffective URL: %{url_effective}\n" 2>&1
The example output is below:
... HTTP Code: 200 HTTP Connect:000 Number Connects: 2 Number Redirects: 1 Redirect URL: Size Download: 136112 Size Upload: 0 SSL Verify: 0 Time Handshake: 0.689 Time Connect: 0.556 Name Lookup Time: 0.528 Time Pretransfer: 0.689 Time Redirect: 0.121 Time Start Transfer (TTFB): 0.738 Time Total: 0.962 Effective URL: https://www.example.com/
Another example for quickly seeing performance is below. For simplicity purposes, first create a file with our curl options already in it:
[[email protected] ~]# vim site-performance.cfg \n DNS lookup : %{time_namelookup}\n Connect to server (TCP) : %{time_connect}\n Connect to server (HTTP/S) : %{time_appconnect}\n Time from start until transfer began: %{time_pretransfer}\n Time for redirection (if any) : %{time_redirect}\n Total time before transfer started : %{time_starttransfer}\n \n Total time : %{time_total}\n Size of download (bytes) : %{size_download}\n Average d/l speed (bytes/s) : %{speed_download}\n \n
Then run it:
[[email protected] ~]# curl -w "@site-performance.cfg" -o /dev/null -s https://www.example.com DNS lookup : 0.138664 Connect to server (TCP) : 0.171131 Connect to server (HTTP/S) : 0.268969 Time from start until transfer began: 0.269021 Time for redirection (if any) : 0.000000 Total time before transfer started : 0.532772 Total time : 0.628730 Size of download (bytes) : 162510 Average d/l speed (bytes/s) : 258473.000
If a web server is running a bunch of sites and has a high load, how can you narrow down which site is likely causing the high load condition on the server? One way would be to see which site takes the longest to load, as that may indicate a resource intensive site. See the example below:
[[email protected] ~]# for i in www.example1.com www.example2.com www.example3.com; do echo -n "$i "; (time curl -IL $i -XGET) 2>&1 | grep -E "real|HTTP"; echo; done www.example1.com HTTP/1.1 200 OK real 0m0.642s www.example2.com HTTP/1.1 200 OK real 0m2.234s www.example3.com HTTP/1.1 200 OK real 0m0.421s
So www.example2.com takes 2 seconds to load. What happens to the load times on that domain during increased traffic. The example below will send 25 requests to the domain:
[[email protected] ~]# for i in {1..25}; do (time curl -sIL http://www.example2.com -X GET &) 2>&1 | grep -E "real|HTTP" & done HTTP/1.1 200 OK real 0m11.297s HTTP/1.1 200 OK real 0m11.395s HTTP/1.1 200 OK real 0m11.906s HTTP/1.1 200 OK real 0m12.079s ... HTTP/1.1 200 OK real 0m11.297s
Determining why this is happening will involve investigation outside the scope of this article. Mainly around investigating ways to cache the site or otherwise optimizing it. However at least now we know which site doesn’t perform well under increased requests, and may be causing the high server load.