Nginx Load Balancer

Load balancing applications across multiple servers is a common technique for creating highly available, fault-tolerant solutions. Nginx can serve as a very light weight and efficient HTTP load balancer to distribute traffic to multiple servers.

This guide will outline one basic way Nginx can be deployed as a load balancer.

Install Nginx

First, install Nginx on your server. In many cases, the version of Nginx supplied by the OS repos is outdated, so this guide will be using Nginx’s official repos instead.

# CentOS 6
[[email protected] ~]# rpm -i http://nginx.org/packages/centos/6/noarch/RPMS/nginx-release-centos-6-0.el6.ngx.noarch.rpm
[[email protected] ~]# rpm --import http://nginx.org/keys/nginx_signing.key
[[email protected] ~]# yum install nginx
[[email protected] ~]# chkconfig nginx on

# CentOS 7
[[email protected] ~]# rpm -i http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
[[email protected] ~]# rpm --import http://nginx.org/keys/nginx_signing.key
[[email protected] ~]# yum install nginx
[[email protected] ~]# systemctl enable nginx

# Ubuntu 14.04 and 16.04
[[email protected] ~]# nginx=stable # use nginx=development for latest development version
[[email protected] ~]# apt-get install python-software-properties
[[email protected] ~]# add-apt-repository ppa:nginx/$nginx
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install nginx

Create Load Balancer config

As I have many servers running different domains, I will be setting up Nginx to load balance specific domains to specific servers. So in the example below, www.example.com will be load balanced to my 2 servers, 192.168.1.161 and 192.168.1.162.

The notable things defined in this example include:

- Uses Least Connections for routing traffic to the load balanced web servers
- Servers will automatically be disabled in the load balancer
- The load balancer will pass the x-forwarded-for header (real IP) to the load balanced web servers

Create the config by:

[[email protected] ~]# vim /etc/nginx/conf.d/example.com.conf
upstream lb-example.com {

	# Load balancing method, only enable one.
	# blank        # Round robin, leave commented out as it is the default
        least_conn;    # Least connections
	# ip_hash;     # Sticky sessions 

	# Backend load balanced servers
        server 192.168.1.161 max_fails=3 fail_timeout=15s;
        server 192.168.1.162 max_fails=3 fail_timeout=15s;

	# Temporarily disable server in load balancer
        # server 192.168.1.163 down;
}

server {
        listen 80;
        server_name www.example.com example.com;
        access_log  /var/log/nginx/example.com.access.log  main;
        # Enforce https
        # return 301 https://$server_name$request_uri;
        # }

        # Pass to load balanced servers
        location / {
        proxy_pass http://lb-example.com;
	
	# Pass x-forwarded-for headers to backend servers
	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Proto $scheme;
        }
}

server {
        listen 443 ssl;
        server_name www.example.com example.com;
        access_log  /var/log/nginx/example.com.access.log  main;

        location / {
        proxy_pass https://lb-example.com;

        # Pass x-forwarded-for headers to backend servers
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        }
}

Now test the new configuration to ensure there are no errors and restart nginx:

[[email protected] ~]# nginx -t
[[email protected] ~]# service nginx restart

Ensure that you open the software firewall for 80 and 443 accordingly. Below is an example for CentOS 6:

[[email protected] ~]# vim /etc/sysconfig/iptables
...
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT
...
[[email protected] ~]# service iptables restart

How to install Nginx and PHP-FPM on CentOS

Nginx is a very popular web server in recent years. It is very well known for being used as a high performance web server, a reverse proxy server, or even a load balancer. Some differences Nginx has from Apache:

- No built in PHP support (mod_php).
- Apache is process based while Nginx is event based.
- Modules in Nginx are compiled directly into the binary while Apache has installable modules that can be enabled or disabled quickly
- Nginx configuration files are a series of includes for each sites configuration, whereas Apache has global settings and a companion virtual host with overrides for customization.  So this means there is no override file like Apache's .htaccess in Nginx.
- There are no SSLCertificateChainFile in Nginx.  Intermediate CA certificates need to be appended directly to the site's SSL certificate.

At this time, the current best practice for installing Nginx is to use Nginx’s repos directly as they will contain the most recent and stable releases of Nginx. So install the repo package for Nginx by:

# CentOS 6
[[email protected] ~]# rpm -i http://nginx.org/packages/centos/6/noarch/RPMS/nginx-release-centos-6-0.el6.ngx.noarch.rpm
[[email protected] ~]# rpm --import http://nginx.org/keys/nginx_signing.key
[[email protected] ~]# yum install nginx

# CentOS 7
[[email protected] ~]# rpm -i http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
[[email protected] ~]# rpm --import http://nginx.org/keys/nginx_signing.key
[[email protected] ~]# yum install nginx

As Nginx does not have anything to natively process PHP, install php5-fpm by:

[[email protected] ~]# yum install php-fpm

By default, php-fpm will be listening for connections over TCP. This is incredibly slow when Nginx and php-fpm are on the same server. So modify the php-fpm configuration from listening over TCP to using sockets by:

[[email protected] ~]# vim /etc/php-fpm.d/www.conf
...
[www]
listen = /var/run/php5-fpm.sock
...

Now set php-fpm to run as user nginx:

[[email protected] ~]# vim /etc/php-fpm.d/www.conf
...
user = nginx
group = nginx
listen.owner = nginx
listen.group = nginx
...

Set php-fpm to start on boot, and then startup the service:

# CentOS 6
[[email protected] ~]# chkconfig php-fpm on
[[email protected] ~]# service php-fpm start

# CentOS 7
[[email protected] ~]# systemctl enable php-fpm.service
[[email protected] ~]# systemctl start php-fpm.service

Back in Nginx, configure it to send any PHP requests to the php-fpm socket inside of the http{} block:

[[email protected] ~]# vim /etc/nginx/nginx.conf
http {
...
upstream php5-fpm-sock {
     server unix:/var/run/php5-fpm.sock;
}
...
}

Now that Nginx and php-fpm are configured, setup your first site:

[[email protected] ~]# mkdir -p /var/www/vhosts/example.com
[[email protected] ~]# vim /etc/nginx/conf.d/example.com.conf
server {
     listen 80;
     server_name example.com www.example.com;

     root /var/www/vhosts/example.com;
     index index.php index.html index.htm;

     access_log /var/log/nginx/example.com-access.log;
     error_log /var/log/nginx/example.com-error.log;

     location ~ \.php$ {
          expires off;
          try_files $uri =404;
          include /etc/nginx/fastcgi_params;
          fastcgi_pass php5-fpm-sock;
          fastcgi_index index.php;
          fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
     }

     # caching of files
     location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
          expires max;
          log_not_found off;
     }
}

#server {
#      listen 443 ssl;
#      server_name example.com www.example.com;
#
#      root /var/www/vhosts/example.com;
#      index index.php index.html index.htm;
#
#      access_log /var/log/nginx/example.com-ssl-access.log;
#      error_log /var/log/nginx/example.com-ssl-error.log;
#
#      ssl_certificate /etc/pki/tls/certs/YYYY_example.com.crt;
#      ssl_certificate_key /etc/pki/tls/private/YYYY_example.com.key;
#
#      ssl_session_timeout 5m;
#
#      # Use PCI compliant SSL protocols and ciphers
#      ssl_protocols SSLv3 TLSv1;
#      ssl_ciphers HIGH:!kEDH:!ADH:!EXPORT56;
#      ssl_prefer_server_ciphers on;
#
#      location ~ \.php$ {
#           expires off;
#           try_files $uri =404;
#           include /etc/nginx/fastcgi_params;
#           fastcgi_pass php5-fpm-sock;
#           fastcgi_index index.php;
#           fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
#      }
#
#      # caching of files
#      location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
#           expires max;
#           log_not_found off;
#      }
# }

Finally, start Nginx and enable it to start on boot:

# CentOS 6
[[email protected] ~]# chkconfig nginx on
[[email protected] ~]# service nginx start

# CentOS 7
[[email protected] ~]# systemctl enable nginx.service
[[email protected] ~]# systemctl start nginx.service

NOTE : There is no SSLCertificateChainFile in nginx. CA Certs need to be appended to the certificate for the domain.

Troubleshooting

If you see a page that says ‘The page you are looking for is currently unavailable’, you’ll find Nginx is having trouble getting data from php-fpm (while using a socket). Make sure that the socket you specify in the nginx.conf and the listen line in www.conf pool file for PHP-FPM match.

If you get a ‘File Not Found’ page only on php pages (when using a TCP port for php-fpm) you can check the nginx logs, and you’ll probably see an error indicating it couldn’t connect on the port. Its possible this could be happening if you need to move the root and index definitions out of a location block (if they are defined there). This is because the php section has it’s own location, the documents won’t try to be found using definitions in the domain ‘location /’ block.

How to install Nginx and PHP-FPM on Ubuntu

Nginx is a very popular web server in recent years. It is very well known for being used as a high performance web server, a reverse proxy server, or even a load balancer. Some differences Nginx has from Apache:

- No built in PHP support (mod_php).
- Apache is process based while Nginx is event based.
- Modules in Nginx are compiled directly into the binary while Apache has installable modules that can be enabled or disabled quickly
- Nginx configuration files are a series of includes for each sites configuration, whereas Apache has global settings and a companion virtual host with overrides for customization.  So this means there is no override file like Apache's .htaccess in Nginx.
- There are no SSLCertificateChainFile in Nginx.  Intermediate CA certificates need to be appended directly to the site's SSL certificate.

At this time, the current best practice for installing Nginx is to use Nginx’s repos directly as they will contain the most recent and stable releases of Nginx. So install the repo package for Nginx by:

# Ubuntu 12.04 and 14.04
[[email protected] ~]# nginx=stable # use nginx=development for latest development version
[[email protected] ~]# apt-get install python-software-properties
[[email protected] ~]# add-apt-repository ppa:nginx/$nginx
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install nginx

As Nginx does not have anything to natively process PHP, install php5-fpm by:

# Ubuntu 12.04 and 14.04
[[email protected] ~]# apt-get update
[[email protected] ~]# apt-get install php5-fpm php5-cli

By default, php-fpm will be listening for connections over TCP. This is incredibly slow when Nginx and php-fpm are on the same server. So modify the php-fpm configuration from listening over TCP to using sockets by:

[[email protected] ~]# vim /etc/php5/fpm/pool.d/www.conf
...
[www]
listen = /var/run/php5-fpm.sock
...

Now set php-fpm to run as user nginx:

[[email protected] ~]# vim /etc/php5/fpm/pool.d/www.conf
...
user = www-data
group = www-data
listen.owner = www-data
listen.group = www-data
...

Back in Nginx, configure it to send any PHP requests to the php-fpm socket inside of the http{} block:

[[email protected] ~]# vim /etc/nginx/nginx.conf
http {
...
upstream php5-fpm-sock {
     server unix:/var/run/php5-fpm.sock;
}
...
}

Now that Nginx and php-fpm are configured, setup your first site:

[[email protected] ~]# mkdir -p /var/www/vhosts/example.com
[[email protected] ~]# vim /etc/nginx/conf.d/example.com.conf
server {
     listen 80;
     server_name example.com www.example.com;

     root /var/www/vhosts/example.com;
     index index.php index.html index.htm;

     access_log /var/log/nginx/example.com-access.log;
     error_log /var/log/nginx/example.com-error.log;

     location ~ \.php$ {
          expires off;
          try_files $uri =404;
          include /etc/nginx/fastcgi_params;
          fastcgi_pass php5-fpm-sock;
          fastcgi_index index.php;
          fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
     }

     # caching of files
     location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
          expires max;
          log_not_found off;
     }
}

#server {
#      listen 443 ssl;
#      server_name example.com www.example.com;
#
#      root /var/www/vhosts/example.com;
#      index index.php index.html index.htm;
#
#      access_log /var/log/nginx/example.com-ssl-access.log;
#      error_log /var/log/nginx/example.com-ssl-error.log;
#
#      ssl_certificate /etc/pki/tls/certs/YYYY_example.com.crt;
#      ssl_certificate_key /etc/pki/tls/private/YYYY_example.com.key;
#
#      ssl_session_timeout 5m;
#
#      # Use PCI compliant SSL protocols and ciphers
#      ssl_protocols SSLv3 TLSv1;
#      ssl_ciphers HIGH:!kEDH:!ADH:!EXPORT56;
#      ssl_prefer_server_ciphers on;
#
#      location ~ \.php$ {
#           expires off;
#           try_files $uri =404;
#           include /etc/nginx/fastcgi_params;
#           fastcgi_pass php5-fpm-sock;
#           fastcgi_index index.php;
#           fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
#      }
#
#      # caching of files
#      location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
#           expires max;
#           log_not_found off;
#      }
# }

Finally, restart Nginx and PHP-FPM to apply the changes

# Ubuntu 12.04 and Ubuntu 14.04
[[email protected] ~]# service php5-fpm restart
[[email protected] ~]# service nginx restart

NOTE : There is no SSLCertificateChainFile in nginx. CA Certs need to be appended to the certificate for the domain.

Troubleshooting

If you see a page that says ‘The page you are looking for is currently unavailable’, you’ll find Nginx is having trouble getting data from php-fpm (while using a socket). Make sure that the socket you specify in the nginx.conf and the listen line in www.conf pool file for PHP-FPM match.

If you get a ‘File Not Found’ page only on php pages (when using a TCP port for php-fpm) you can check the nginx logs, and you’ll probably see an error indicating it couldn’t connect on the port. Its possible this could be happening if you need to move the root and index definitions out of a location block (if they are defined there). This is because the php section has it’s own location, the documents won’t try to be found using definitions in the domain ‘location /’ block.

Nginx stub_status module

Like the Apache mod_status module, Nginx also has a module that can provide you with realtime data regarding its stats. While its not as detailed as Apache mod_status, the data is still useful, especially if you are using Nginx as a load balancer.

First, confirm that Nginx was compiled with the HttpStudStatusModule. If you installed this via yum or apt-get, it should have it:

[[email protected] ~]# nginx -V 2>&1 | grep -o with-http_stub_status_module
with-http_stub_status_module

To enable the module, set the following configuration:

[[email protected] ~]# vim /etc/nginx/conf.d/nginx_status.conf
server {

    listen 80 default_server;
    access_log off;
    server_name _;
    server_name_in_redirect off;
    root  /var/www/html;

        location /nginx_status {
                # Enable Nginx stats
                stub_status on;
                # Disable logging for stats
                access_log   off;
                # Security: Only allow access from authorized IP's
                allow 127.0.0.1;
                allow 192.168.1.100;
                # Deny everyone else
                deny all;
        }

}

Now navigate your browser to:

http://serverip/nginx_status

The results will look something like this:

Active connections: 1 
server accepts handled requests
 1 1 1 
Reading: 0 Writing: 1 Waiting: 0

Here is what these numbers mean, taken directly from Nginx’s documentation:

# Active connections
The current number of active client connections including Waiting connections.

# accepts
The total number of accepted client connections.

# handled
The total number of handled connections. Generally, the parameter value is the same as accepts unless some resource limits have been reached (for example, the worker_connections limit).

# requests
The total number of client requests.

# Reading
The current number of connections where nginx is reading the request header.

# Writing
The current number of connections where nginx is writing the response back to the client.

# Waiting
The current number of idle client connections waiting for a request.