Setting up nginx as a Reverse Proxy

While apache is dominant server it tends to load all configured modules for every request. Some modules, such as those for php, are by no means small, and certainly are not necessary for the vast majority of requests. One approach to addressing this problem, is to use a lighter front end server to handle requests for static files, and to proxy requests for dynamic files to apache. Two common servers used for this are nginx (pronounce EngineX) and lighttpd (pronounced Lighty).

The setup outlined below details the installation and configuration of nginx as a front end server for static files, with requests for php files proxied to apache running in the background, on a server running Amazon’s distribution of Linux. (The same basic steps would likely be applicable to CentOS and RHEL servers).

This procedure assumes a functioning EC2 setup currently running apache as a frontend server.

Since most steps require root privileges, let us get those out of the way:

sudo –i

Nginx setup

The next step is to obtain and extract the nginx setup (it isn’t offered in the amzn repository):

cd /usr/local/src
wget http://nginx.org/download/nginx-0.8.53.tar.gz
tar -xzvf nginx-0.8.53.tar.gz
cd nginx-0.8.53

The default compile options aren’t common for a RHEL derived system, so we will change them when we run configure. A full list of options can be seen by running ./configure –help

• Directories that don’t exist (below) are created during install
• The http_gzip_static_module will allow serving of precompressed files (ending with .gz), instead of compressing with each request.

./configure \
  --prefix=/etc/nginx \
  --sbin-path=/usr/sbin \
  --conf-path=/etc/nginx/nginx.conf \
  --pid-path=/var/run/nginx.pid \
  --lock-path=/var/lock/subsys/nginx \
  --error-log-path=/var/log/nginx/error.log \
  --http-log-path=/var/log/nginx/access.log \
  --http-client-body-temp-path=/var/tmp/nginx/client_body \
  --http-proxy-temp-path=/var/tmp/nginx/proxy \
  --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi \
  --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \
  --http-scgi-temp-path=/var/tmp/nginx/scgi \
  --with-http_gzip_static_module

You may need to install pcre-devel and zlib-devel (as well a the necessary ‘Development Tools’).

Compile and install:

make && make install

nginx does not install an init script, so we will use one derived from the Red Hat one on the nginx wiki.

cd /etc/init.d
vi nginx

Copy and paste the following:

#!/bin/sh
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig:   - 85 15
# description:  Nginx is an HTTP(S) server, HTTP(S) reverse \
#               proxy and IMAP/POP3 proxy server
# processname: nginx
# config:      /etc/nginx/nginx.conf
# config:      /etc/sysconfig/nginx
# pidfile:     /var/run/nginx.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0

nginx="/usr/sbin/nginx"
prog=$(basename $nginx)

NGINX_CONF_FILE="/etc/nginx/nginx.conf"

[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx

lockfile=/var/lock/subsys/nginx

make_dirs() {
   # make required directories
   user=`nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -`
   options=`$nginx -V 2>&1 | grep 'configure arguments:'`
   for opt in $options; do
       if [ `echo $opt | grep '.*-temp-path'` ]; then
           value=`echo $opt | cut -d "=" -f 2`
           if [ ! -d "$value" ]; then
               # echo "creating" $value
               mkdir -p $value && chown -R $user $value
           fi
       fi
   done
}

start() {
    [ -x $nginx ] || exit 5
    [ -f $NGINX_CONF_FILE ] || exit 6
    make_dirs
    echo -n $"Starting $prog: "
    daemon $nginx -c $NGINX_CONF_FILE
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}

stop() {
    echo -n $"Stopping $prog: "
    killproc $prog -QUIT
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

restart() {
    configtest || return $?
    stop
    sleep 1
    start
}

reload() {
    configtest || return $?
    echo -n $"Reloading $prog: "
    killproc $nginx -HUP
    RETVAL=$?
    echo
}

force_reload() {
    restart
}

configtest() {
  $nginx -t -c $NGINX_CONF_FILE
}

rh_status() {
    status $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart|configtest)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
            ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
        exit 2
esac

Allow execution of the script:

chmod +x nginx

Nginx Configuration

We will now configure nginx. This configuration is for a setup that uses VirtualHosts.
Firstly, edit the main configuration file (/etc/nginx/nginx.conf) – this only has a few changes:

cd /etc/nginx
vi nginx.conf

/etc/nginx/nginx.conf:

user  apache;
worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    client_max_body_size    10m;
    client_body_buffer_size 128k;
    client_header_buffer_size 2k;
    large_client_header_buffers 4 16k;

    sendfile        on;
    keepalive_timeout  65;

    gzip  on;
    gzip_min_length 1100;
    gzip_buffers 4 32k;
    gzip_types text/plain text/html application/x-javascript text/xml text/css;
    index index.php index.htm index.html

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/conf.d/proxy.inc;
    include /etc/nginx/sites-enabled/*.conf;
}

A few brief notes on the configuration:

• Since the files ownership already has apache in the group, and no write access is needed by nginx (it is intended to only serve static files), setting the user to apache might simplify things.
• The buffers were set to be on the safe side and are probably larger than needed.
• Gzip compression was turned on (it would be preferable to use the static gzip module that was included above, but the site itself must be configured to create the precompiled files.
• Finally, we include any configuration settings in conf.d and the site-specific configurations in sites-enabled.

We now need to create the directories specified above:

mkdir conf.d
mkdir sites-enabled

We will add the proxy settings to two files in conf.d
Firstly the numeric settings (buffers and timeouts):

/etc/nginx/conf.d/proxy.conf:

proxy_redirect          off;
proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;
proxy_buffer_size   16k;
proxy_buffers       32   16k;
proxy_busy_buffers_size 64k;

Now a second file for the headers to pass to apache.

/etc/nginx/conf.d/proxy.inc:

proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

The setup, above (nginx.conf) includes both these files in the ‘http’ section of the config. This reduces on the number of times it must be processed. However, if any headers are set in server or location blocks in the nginx configuration, none of the headers specified above will be inherited in that block. (As a side note, for reasons that escape me, when the headers were put directly into proxy.conf they were not passed to apache).

We will now switch to the sites-enabled directory and add sites that are found under apache’s VirtualHost blocks.

Since most simple sites are fairly similar in terms of their configuration, the following was setup to be included in most sites:

/etc/sites-enabled/default.inc:

location / {
    try_files $uri $uri/ /index.php?q=$request_uri;
}

location = /robots.txt {
    allow all;
    log_not_found off;
    access_log off;
    break;
}

location ~  \.(jpg|jpeg|png|gif|swf|ico|js|css|txt|htm|html|xml|bmp|pdf|doc|docx|ppt|pptx|zip)$     {
    expires     30d;
    break;
}

location ~ \.php$ {
    access_log off;
    proxy_pass   http://127.0.0.1:8080;
}

location ~ /\.ht {
    deny  all;
}

A few quick notes on the above:

• This will try to serve the exact file, a directory, or pass the request to index.php (e.g. for WordPress, Joomla, etc).
• If the location is exactly robots.txt we will not log the file and (because of the equal sign) will stop search for further matches and serve the file.
• Static files (with extensions matching those listed), will be served by nginx, with a 30 day expires header.
• PHP files will be passed to apache running on port 8080. Additionally, since apache will log these files, we will turn of logging from nginx.
• Finally, since nginx doesn’t use .htaccess files, we will deny access to them

The last step of the nginx setup is to actually create configuration files for each domain. The following should serve as a starting point. Additional options may be added as needed.

Create a file for the domain, ending in .conf (e.g. example.com.conf) and copy and paste the following (modify the paths and server name):
/etc/nginx/sites-enabled/example.com.conf:

server {
    server_name  *.example.com example.com;
    access_log  /var/www/html/example.com/logs/nginx_access.log;
    error_log  /var/www/html/example.com/logs/nginx_error.log debug;
    root   /var/www/html/example.com/public_html/;
    include /etc/nginx/sites-enabled/default.inc;
}

This setup will accept any subdomain (e.g. www) or no subdomain and serve content from the location specified under root. It will also provide site-specific access and error logs. Finally, it includes the other settings created above (default.inc).

To add additional sites, simply copy and paste the entire server block above, and correct the values (server name and paths) for the new site (either in the same file, or a new file).

That is the end of the setup for nginx.

Apache Configuration

We will now modify our apache config to listen on the port specified above (don’t forget to backup your configuration files before proceeding).

Set the Listen and NameVirtualHost directives, in /etc/httpd/conf/httpd.conf, to only listen to the port specified above (8080) on the loopback interface (127.0.0.1):

sed -i 's/^Listen.*/Listen 127.0.0.1:8080/g' httpd.conf
sed -i 's/^NameVirtualHost.*/NameVirtualHost 127.0.0.1:8080/g' httpd.conf

Edit all VirtualHost blocks to listen to 127.0.0.1:8080 instead of the default *:80. (ensure you apply this to the right directory – typically, either the httpd.conf file itself, or files in conf.d, or sites-enabled):

cd /etc/httpd/conf.d
find . -type f ! -name "ssl.conf" -exec sed -i 's/^<VirtualHost .*>/<VirtualHost 127.0.0.1:8080>/g' '{}' \;

(Note the line above is assuming you have a file ‘ssl.conf’ that contains a VirtualHost directive for port 443, which we do not want to change).

Lastly, edit your php.ini file, to prevent PHP from trying to guess which file to execute when it is passed a non-existent file:
Set (in /etc/php.ini):

cgi.fix_pathinfo=0

This should provide a functional setup, however, apache will log all requests as coming from 127.0.0.1.

mod_rpaf Setup

To log the correct IPs we need to install the reverse proxy add forward module (mod_rpaf).

Download and extract:

cd /usr/local/src
wget http://stderr.net/apache/rpaf/download/mod_rpaf-0.6.tar.gz
tar -xzvf mod_rpaf-0.6.tar.gz
cd mod_rpaf-0.6

On some setups, you might need to edit the Makefile (line 2) to APXS2=$(shell which apxs).

Compile and install mod_rpaf:

make rpaf-2.0 && make install-2.0

Create a configuration file for apache, and set a few directives:

/etc/httpd/conf.d/rpaf.conf:

LoadModule rpaf_module modules/mod_rpaf-2.0.so
RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1

Going Live

That’s it – restart apache, start nginx, and check that everything works.

service httpd restart
service nginx start

You probably also want to set nginx to load at startup:

chkconfig --levels 235 nginx on

You will need to rewrite your .htaccess files for nginx (although, if a request is sent to apache, apache will still process the .htaccess file). You should keep the new port apache is running on firewalled to prevent direct access to it.

Crude, preliminary tests with apache bench, showed about a 25% drop in processing time for static content (a 15ms mean for nginx vs a 20ms mean for apache (on 100 requests, with 10 concurrent)), which can hopefully be improved further. There was a slight increase in processing time observed for dynamic content.
Check out the nginx wiki for specific configuration options and examples.