Nginx vs Litespeed

It has been a while since my last update here, so I thought I would publish my latest win with a client's overloaded server.

Now, before we begin, ill explain the situation, this client, who will remain nameless due to privacy reasons, hosts a very popular site that is used by thousands of people daily, one of the servers primarily serves files from an APT like repository.

The host that is serving these files is by no means slow, it is an 8 core 64bit with 16GB of ram running CentOS 5. What makes this more then mere file serving is that every request is re-written to a php script that checks for download authorization with a remote service and also logs stats on each file download.

The server was configured with Litespeed, but over the new year break some new software was released which has caused the load on this server to increase over 15 times. In an attempt to reduce load I tried just about everything, tuning Litespeed, installing eAccelerator into PHP, tuning the script that handles the downloads. These things helped, the server load was up over 150-160 before these changes and this reduced it to around 70-80.

This was still not good enough, I advised the client that there is nothing more we can do and that he would have to purchase more servers to deal with the load... but I was not at all happy with this solution.

So, over the next few days I tried a few more things, such as a reverse Nginx proxy in front of litespeed, this helped by further reducing the load to around 30-40, but again, the server was still way overloaded.

Further profiling and examination identified some very poor code in the stats script that was quickly corrected, such as fetching a file from the local host via the fopen url wrapper instead of from the hard disk directly, and using Memcache to cache hits on files and only update the database every 30 seconds. This did not reduce load but prevented Litespeed from running out of connections that were causing 500 errors.

After discussing options with the client we decided to swap out Litespeed with Nginx completely, so over the next day I re-wrote the rewrite rules into Nginx and setup PHP FastCGI workers ready to handle requests. This was all done limiting these new rules to my IP address so I could test these changes, and the public would still get passed to the Litespeed backend.

Once everything was confirmed I stopped Litespeed and started up Nginx as the stand alone webserver, with full logging to ensure that things were running smoothly. Immediately the server load dropped to 2-3, initially I thought that something was broken, PHP was not running or some other fault, but testing proved that it was all running as it should.

After a few minior tweaks to the configuration such as number of PHP workers and Nginx workers, the servers load dropped below 3 permanently. I did not expect to see such a huge performance increase by just moving away from Litespeed, I was under the false impression that Litespeed is about as fast as Nginx, and faster with PHP... but this experience has without a doubt proved that Litespeed just cant keep up with Nginx + PHP FastCGI when configured properly.

cPanel & nginx

Well, its about time I posted something new and interesting. I have come to love nginx, its the best reverse proxy I have used, and it has some very nice features, but most importantly, its FAST.

Now, I know there are other sites that show you how to setup nginx on a cPanel server, but they don't go all the way, they still rely on the admin running a script to update the nginx configuration, and also manual configuration of mod_rpaf for Apache.

The problem is say admin X adds a new domain to the server, it wont work until nginx has been updated. Or, if a new IP is added to the server, it wont get logged correctly until the Apache configuration has been updated.

So here is how you do it properly

Install nginx

# yum install nginx

/etc/nginx/nginx.conf:

user                    nginx;
worker_processes        3;
error_log               /var/log/nginx/error.log;
pid                     /var/run/nginx.pid;

events {
        worker_connections      1024;
        use                     epoll;
}

http {
        server_names_hash_max_size      2048;
        include                         /etc/nginx/mime.types;
        default_type                    application/octet-stream;
        server_tokens                   off;

        sendfile                        on;
        tcp_nopush                      on;
        tcp_nodelay                     on;
        keepalive_timeout               10;
        ignore_invalid_headers          on;

        gzip                            on;
        gzip_http_version               1.0;
        gzip_comp_level                 1;
        gzip_min_length                 1100;
        gzip_buffers                    4 32k;
        gzip_types                      text/plain application/x-javascript text/xml text/php text/css application/javascript text/js;
        gzip_proxied                    any;
        gzip_disable                    msie6;

        client_header_timeout           3m;
        client_body_timeout             3m;
        send_timeout                    3m;
        connection_pool_size            256;
        client_header_buffer_size       4k;
        large_client_header_buffers     4 32k;
        request_pool_size               4k;
        output_buffers                  4 32k;
        postpone_output                 1460;

        server {
                listen          80;
                server_name     _;
                rewrite         ^(.*)$ http://primary_website.com$1 permanent;
        }

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/vhosts.d/*.conf;
}

You must update the bolded parts...

• worker_processes should be set to the same number of CPUs you have for best performance.
• primary_website.com should be changed to your primary website address, this will re-write direct IP address access to your website, instead of presenting the cPanel default page.

Make the vhosts.d path:

# mkdir -p /etc/nginx/vhosts.d

The build script (/usr/local/sbin/build-nginx-config):

#!/bin/bash

rm -rf /etc/nginx/vhosts.d
mkdir -p /etc/nginx/vhosts.d

pushd /var/cpanel/users &>/dev/null
for USER in *; do
 IP="`grep '^IP=' $USER | cut -d'=' -f2`"
 DNS=""
 NAMES=""
 while [ 1 ]; do
  RECORD=`grep "^DNS$DNS=" $USER`
  if [ $? -ne 0 ]; then break; fi
  FQDN=`echo $RECORD | cut -d'=' -f2`
  if [ -z "$FQDN" ] && [ -z "$DNS" ] ; then
   DNS=1;
   continue;
  fi

  if [ -z "$DNS" ]; then
   NAMES=".$FQDN"
   DNS=1;
  else
   if [ -z "$NAMES" ]; then
    NAMES=".$FQDN"
   else
    NAMES="$NAMES .$FQDN"
   fi
   let DNS=$DNS+1
  fi
 done

cat > "/etc/nginx/vhosts.d/$USER.conf" <
server {
 access_log off;
 error_log off;
 listen  80;
 server_name $NAMES;
 location / {
  proxy_pass  http://$IP:81/;
  proxy_set_header Host   \$host;
  proxy_set_header X-Real-IP  \$remote_addr;
  proxy_set_header X-Forwarded-For  \$proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Port \$server_port;
  proxy_hide_header X-Powered-By;
 }
}
EOF

 if [ -z "$IPLIST" ]; then
  IPLIST="$IP"
 else
  echo $IPLIST | grep -q $IP
  if [ $? -ne 0 ]; then
   IPLIST="$IPLIST $IP"
  fi
 fi
done
popd &>/dev/null

cat > "/usr/local/apache/conf/includes/pre_main_global.conf" <
LoadModule        rpaf_module modules/mod_rpaf.so
RPAF_Enable       On
RPAF_ProxyIPs     127.0.0.1 $IPLIST
RPAF_SetHostName  On
RPAF_SetHTTPS     On
RPAF_SetPort      On
EOF

service nginx reload
service httpd restart

WARNING: this will overwrite pre_main_global.conf, be sure that you copy anything that is in that file into this script just before the last bold EOF.

Make sure this file is executable:

chmod +x /usr/local/sbin/build-nginx-config

mod_rpaf

Please note that the settings and configuration here relies on a new version of mod_rpaf you can obtain from my git repository here: https://github.com/gnif/mod_rpaf. Click the top right "Download" button.

Now we just have to build and install it:

# tar -xvzf master
# cd gnif-mod_rpaf-*
# make
# make install

This version of mod_rpaf has been endorsed by the original author and accepted as the next version, it now has a few fixes for ipv6 and security, fixes the port that apache sees to the one the proxy is using, and sets the HTTPS environment variable if it is set in the headers.

Final steps:

• In cPanel under "Tweak Settings" change the HTTP port to 80 (not the HTTPS port!) and save.
• run: /usr/local/sbin/build-nginx-config
• run: chkconfig nginx on
• run: service nginx start

Now all thats left to do is make apache run the build-nginx-config script each time a domain is added or removed, this is through a script hook:

/scripts/postupdateuserdomains:

#!/bin/bash
/usr/local/sbin/build-nginx-config

Create the file if it does not exist, cPanel provides this file for exactly this kind of thing. Make sure that it has the execute bit set (chmod +x /scripts/postupdateuserdomains)