Varnish 3 – Installation and configuration

Varnish is a HTTP reverse proxy that can be installed in front of the web server to provide caching. If the VCL’s are properly configured for your site, Varnish can greatly offset the backend server load many times over.

In this guide, it is assumed you already have a running LAMP stack, and Apache’s vhost configurations are stored in /etc/httpd/vhost.d/. At the end of this guide if all goes well, you will be running Varnish 3, with Varnish listening for inbound connections on port 80, and passing any backend connections that cannot be served via cache to Apache on port 8080.

CentOS 6 – Installation and initial configuration

Install the varnish-release package repository, then install:

[root@web01 ~]# rpm --nosignature -ivh https://repo.varnish-cache.org/redhat/varnish-3.0.el6.rpm
[root@web01 ~]# yum -y install varnish

Now update your Apache ports and vhosts to 8080 since Varnish will be listening on port 80:

[root@web01 ~]# sed -i "s/Listen 80\$/Listen 8080/g" /etc/httpd/ports.conf
[root@web01 ~]# sed -i "s/NameVirtualHost \*:80\$/NameVirtualHost \*:8080/g" /etc/httpd/ports.conf
[root@web01 ~]# sed -i "s/:80>/:8080>/g" /etc/httpd/vhost.d/*

Configure Varnish to pass connections back to Apache on port 8080:

[root@web01 ~]# sed -i 's/port = "80"/port = "8080"/g' /etc/varnish/default.vcl

Then update Varnish so it listens on port 80:

[root@web01 ~]# sed -i 's/VARNISH_LISTEN_PORT=6081$/VARNISH_LISTEN_PORT=80/g' /etc/sysconfig/varnish

Finally, restart Apache and Varnish:

[root@web01 ~]# service httpd restart
[root@web01 ~]# service varnish start
[root@web01 ~]# chkconfig varnish on

Ubuntu 12.04 / 14.04 – Installation and initial configuration

First, setup the Varnish repos:

# Ubuntu 12.04
[root@web01 ~]# curl -sL http://repo.varnish-cache.org/debian/GPG-key.txt | apt-key add -
[root@web01 ~]# echo "deb http://repo.varnish-cache.org/ubuntu/ precise varnish-3.0" > /etc/apt/sources.list.d/varnish.list

# Ubuntu 14.04
[root@web01 ~]# curl -sL http://repo.varnish-cache.org/debian/GPG-key.txt | apt-key add -
[root@web01 ~]# echo "deb http://repo.varnish-cache.org/ubuntu/ trusty varnish-3.0" > /etc/apt/sources.list.d/varnish.list

Now install Varnish:

[root@web01 ~]# apt-get update
[root@web01 ~]# apt-get install varnish

Next update your Apache ports and vhosts to 8080 since Varnish will be listening on port 80:

[root@web01 ~]# sed -i "s/Listen 80\$/Listen 8080/g" /etc/apache2/ports.conf
[root@web01 ~]# sed -i "s/NameVirtualHost \*:80\$/NameVirtualHost \*:8080/g" /etc/apache2/ports.conf
[root@web01 ~]# sed -i "s/:80>/:8080>/g" /etc/apache2/sites-available/*

Configure Varnish to pass connections back to Apache on port 8080:

[root@web01 ~]# sed -i 's/port = "80"/port = "8080"/g' /etc/varnish/default.vcl

Then update Varnish so it listens on port 80:

[root@web01 ~]# sed -i 's/^DAEMON_OPTS="-a :6081/DAEMON_OPTS="-a :80/g' /etc/default/varnish
[root@web01 ~]# sed -i 's/START=no/START=yes/' /etc/default/varnish
[root@web01 ~]# service apache2 restart
[root@web01 ~]# service varnish restart

Varnish VCL configuration examples

All of the tunings take place within the vcl’s. For the purpose of this guide, we are going to just use the default varnish configuration file in /etc/varnish/default.vcl for our examples.

How to enable basic caching of static resources:

sub vcl_recv {
...
if (req.url ~ "\.(html|gif|jpg|jpeg|png|js|css)$") {
         unset req.http.cookie;
         return(lookup);
     }
 return(pass);
 }
...

If the request is coming in from CloudFlare or a load balancer, here is how to set the real IP of the client:

sub vcl_recv {
...
     if (req.restarts == 0) {
        if (req.http.x-forwarded-for) {
            set req.http.X-Forwarded-For =
                req.http.X-Forwarded-For + ", " + client.ip;
        } else {
            set req.http.X-Forwarded-For = client.ip;
        }
     }
...

Here is an example of how to exclude things like phpmyadmin, apc.php, and server-status from being cached:

sub vcl_recv {
...
if (req.url ~ "(?i)/(phpmyadmin|apc.php|server-status)") {
      return(pass);
    }
...

Here is how you can exclude a specific URL from being cached:

sub vcl_recv {
...
     if (req.url ~ "^/example") {
     return (pass);
     }
...

Perhaps you have 30 domains on the server, and you need one of them to be excluded from the cache. Or maybe your actively working on the site. Here is how you can prevent the domain from being served through varnish:

sub vcl_recv {
...
    if (req.http.host ~ "^(www.)?domain.com") {
    return (pass);
    }
...

If you find a script running via your browser, and suspect it is timing out due to varnish, you can adjust the timeout on that specific script by:

sub vcl_recv {
...
if (req.url == "^/bigscript.php") {
    set bereq.first_byte_timeout = 10m;
}
...

Here is an example of how to never cache PUT and DELETE requests for a domain:

sub vcl_recv {
...
if ( req.http.host == "subdomain.domain.com" ) {
    if (req.method == "PUT" || req.method == "POST" || req.method == "DELETE")
    {
        return(pass);
    }
}
...

Varnish Troubleshooting

One of the most common errors I see on sites utilizing Varnish is a error message:

Error 503 Service Unavailable
Guru Meditation:
XID: 1234567

Typically Varnish is not the problem, but instead its something else such as Apache or PHP-FPM (aka the backend) not being available. If you can replicate the error in your browser, then run the following command so you can see if you can catch the issue as its happening in the logs:

[root@web01 ~]# varnishlog -d -c -m TxStatus:503

This will return a bunch of output. You are most interesting in the lines surrounding ‘FetchError’ as shown below:

   11 SessionOpen  c 192.168.1.56 60015 :80
   11 ReqStart     c 192.168.1.56 60015 311889525
   11 RxRequest    c GET
   11 RxURL        c /
   11 RxProtocol   c HTTP/1.1
   11 RxHeader     c Host: example.com
   11 RxHeader     c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
   11 RxHeader     c Connection: keep-alive
   11 RxHeader     c Cookie: wordpress_test_cookie=WP+Cookie+check; wp-settings-time-1=1455921695
   11 RxHeader     c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/601.4.4 (KHTML, like Gecko) Version/9.0.3 Safari/601.4.4
   11 RxHeader     c Accept-Language: en-us
   11 RxHeader     c DNT: 1
   11 RxHeader     c Accept-Encoding: gzip, deflate
   11 VCL_call     c recv pass
   11 VCL_call     c hash
   11 Hash         c /
   11 Hash         c example.com
   11 VCL_return   c hash
   11 VCL_call     c pass pass
   11 FetchError   c no backend connection
   11 VCL_call     c error deliver
   11 VCL_call     c deliver deliver
   11 TxProtocol   c HTTP/1.1
   11 TxStatus     c 503
   11 TxResponse   c Service Unavailable
   11 TxHeader     c Server: Varnish
   11 TxHeader     c Content-Type: text/html; charset=utf-8
   11 TxHeader     c Retry-After: 5
   11 TxHeader     c Content-Length: 418
   11 TxHeader     c Accept-Ranges: bytes
   11 TxHeader     c Date: Fri, 19 Feb 2016 23:04:03 GMT
   11 TxHeader     c X-Varnish: 311889525
   11 TxHeader     c Age: 0
   11 TxHeader     c Via: 1.1 varnish
   11 TxHeader     c Connection: close
   11 Length       c 418
   11 ReqEnd       c 311889525 1455923043.127803802 1455923043.128304243 0.000658751 0.000423908 0.000076532

And in the example above, where is has ‘FetchError’, it gave the 503 as Apache was not running. Which was why is says: “no backend connection’.