Nginx vs Litespeed

It has been a while since my last update here, so I thought I would publish my latest win with a client's overloaded server.

Now, before we begin, ill explain the situation, this client, who will remain nameless due to privacy reasons, hosts a very popular site that is used by thousands of people daily, one of the servers primarily serves files from an APT like repository.

The host that is serving these files is by no means slow, it is an 8 core 64bit with 16GB of ram running CentOS 5. What makes this more then mere file serving is that every request is re-written to a php script that checks for download authorization with a remote service and also logs stats on each file download.

The server was configured with Litespeed, but over the new year break some new software was released which has caused the load on this server to increase over 15 times. In an attempt to reduce load I tried just about everything, tuning Litespeed, installing eAccelerator into PHP, tuning the script that handles the downloads. These things helped, the server load was up over 150-160 before these changes and this reduced it to around 70-80.

This was still not good enough, I advised the client that there is nothing more we can do and that he would have to purchase more servers to deal with the load... but I was not at all happy with this solution.

So, over the next few days I tried a few more things, such as a reverse Nginx proxy in front of litespeed, this helped by further reducing the load to around 30-40, but again, the server was still way overloaded.

Further profiling and examination identified some very poor code in the stats script that was quickly corrected, such as fetching a file from the local host via the fopen url wrapper instead of from the hard disk directly, and using Memcache to cache hits on files and only update the database every 30 seconds. This did not reduce load but prevented Litespeed from running out of connections that were causing 500 errors.

After discussing options with the client we decided to swap out Litespeed with Nginx completely, so over the next day I re-wrote the rewrite rules into Nginx and setup PHP FastCGI workers ready to handle requests. This was all done limiting these new rules to my IP address so I could test these changes, and the public would still get passed to the Litespeed backend.

Once everything was confirmed I stopped Litespeed and started up Nginx as the stand alone webserver, with full logging to ensure that things were running smoothly. Immediately the server load dropped to 2-3, initially I thought that something was broken, PHP was not running or some other fault, but testing proved that it was all running as it should.

After a few minior tweaks to the configuration such as number of PHP workers and Nginx workers, the servers load dropped below 3 permanently. I did not expect to see such a huge performance increase by just moving away from Litespeed, I was under the false impression that Litespeed is about as fast as Nginx, and faster with PHP... but this experience has without a doubt proved that Litespeed just cant keep up with Nginx + PHP FastCGI when configured properly.

cPanel & nginx

Well, its about time I posted something new and interesting. I have come to love nginx, its the best reverse proxy I have used, and it has some very nice features, but most importantly, its FAST.

Now, I know there are other sites that show you how to setup nginx on a cPanel server, but they don't go all the way, they still rely on the admin running a script to update the nginx configuration, and also manual configuration of mod_rpaf for Apache.

The problem is say admin X adds a new domain to the server, it wont work until nginx has been updated. Or, if a new IP is added to the server, it wont get logged correctly until the Apache configuration has been updated.

So here is how you do it properly

Install nginx

# yum install nginx

/etc/nginx/nginx.conf:

user                    nginx;
worker_processes        3;
error_log               /var/log/nginx/error.log;
pid                     /var/run/nginx.pid;

events {
        worker_connections      1024;
        use                     epoll;
}

http {
        server_names_hash_max_size      2048;
        include                         /etc/nginx/mime.types;
        default_type                    application/octet-stream;
        server_tokens                   off;

        sendfile                        on;
        tcp_nopush                      on;
        tcp_nodelay                     on;
        keepalive_timeout               10;
        ignore_invalid_headers          on;

        gzip                            on;
        gzip_http_version               1.0;
        gzip_comp_level                 1;
        gzip_min_length                 1100;
        gzip_buffers                    4 32k;
        gzip_types                      text/plain application/x-javascript text/xml text/php text/css application/javascript text/js;
        gzip_proxied                    any;
        gzip_disable                    msie6;

        client_header_timeout           3m;
        client_body_timeout             3m;
        send_timeout                    3m;
        connection_pool_size            256;
        client_header_buffer_size       4k;
        large_client_header_buffers     4 32k;
        request_pool_size               4k;
        output_buffers                  4 32k;
        postpone_output                 1460;

        server {
                listen          80;
                server_name     _;
                rewrite         ^(.*)$ http://primary_website.com$1 permanent;
        }

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/vhosts.d/*.conf;
}

You must update the bolded parts...

• worker_processes should be set to the same number of CPUs you have for best performance.
• primary_website.com should be changed to your primary website address, this will re-write direct IP address access to your website, instead of presenting the cPanel default page.

Make the vhosts.d path:

# mkdir -p /etc/nginx/vhosts.d

The build script (/usr/local/sbin/build-nginx-config):

#!/bin/bash

rm -rf /etc/nginx/vhosts.d
mkdir -p /etc/nginx/vhosts.d

pushd /var/cpanel/users &>/dev/null
for USER in *; do
 IP="`grep '^IP=' $USER | cut -d'=' -f2`"
 DNS=""
 NAMES=""
 while [ 1 ]; do
  RECORD=`grep "^DNS$DNS=" $USER`
  if [ $? -ne 0 ]; then break; fi
  FQDN=`echo $RECORD | cut -d'=' -f2`
  if [ -z "$FQDN" ] && [ -z "$DNS" ] ; then
   DNS=1;
   continue;
  fi

  if [ -z "$DNS" ]; then
   NAMES=".$FQDN"
   DNS=1;
  else
   if [ -z "$NAMES" ]; then
    NAMES=".$FQDN"
   else
    NAMES="$NAMES .$FQDN"
   fi
   let DNS=$DNS+1
  fi
 done

cat > "/etc/nginx/vhosts.d/$USER.conf" <
server {
 access_log off;
 error_log off;
 listen  80;
 server_name $NAMES;
 location / {
  proxy_pass  http://$IP:81/;
  proxy_set_header Host   \$host;
  proxy_set_header X-Real-IP  \$remote_addr;
  proxy_set_header X-Forwarded-For  \$proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Port \$server_port;
  proxy_hide_header X-Powered-By;
 }
}
EOF

 if [ -z "$IPLIST" ]; then
  IPLIST="$IP"
 else
  echo $IPLIST | grep -q $IP
  if [ $? -ne 0 ]; then
   IPLIST="$IPLIST $IP"
  fi
 fi
done
popd &>/dev/null

cat > "/usr/local/apache/conf/includes/pre_main_global.conf" <
LoadModule        rpaf_module modules/mod_rpaf.so
RPAF_Enable       On
RPAF_ProxyIPs     127.0.0.1 $IPLIST
RPAF_SetHostName  On
RPAF_SetHTTPS     On
RPAF_SetPort      On
EOF

service nginx reload
service httpd restart

WARNING: this will overwrite pre_main_global.conf, be sure that you copy anything that is in that file into this script just before the last bold EOF.

Make sure this file is executable:

chmod +x /usr/local/sbin/build-nginx-config

mod_rpaf

Please note that the settings and configuration here relies on a new version of mod_rpaf you can obtain from my git repository here: https://github.com/gnif/mod_rpaf. Click the top right "Download" button.

Now we just have to build and install it:

# tar -xvzf master
# cd gnif-mod_rpaf-*
# make
# make install

This version of mod_rpaf has been endorsed by the original author and accepted as the next version, it now has a few fixes for ipv6 and security, fixes the port that apache sees to the one the proxy is using, and sets the HTTPS environment variable if it is set in the headers.

Final steps:

• In cPanel under "Tweak Settings" change the HTTP port to 80 (not the HTTPS port!) and save.
• run: /usr/local/sbin/build-nginx-config
• run: chkconfig nginx on
• run: service nginx start

Now all thats left to do is make apache run the build-nginx-config script each time a domain is added or removed, this is through a script hook:

/scripts/postupdateuserdomains:

#!/bin/bash
/usr/local/sbin/build-nginx-config

Create the file if it does not exist, cPanel provides this file for exactly this kind of thing. Make sure that it has the execute bit set (chmod +x /scripts/postupdateuserdomains)

nVidia EDID override and HDMI

I have a media center that is running Debian with XBMC on it, as I am a developer for XBMC I have just receved a new HDMI 1.4 capable AVR. I also have a large monitor that lies about its native resolution, reporting 1280x1024 instead of 1280x720. Normally I have been using a modified EDID file and the CustomEDID option in the nVidia driver to override the one the monitor supplies, which works really well IMHO, but there is a downside it took me 4 hours to track down, if you do this via a HDMI audio capable device, your HDMI audio will not work.

So, I had to find another option, took a few hours to figure out what worked, and I eventually came up with this. If you are allready using a custom/modified EDID keep reading, if not, figure that part out first over here.

To make use of your EDID information without using an EDID file is fairly simple, first you will need to install the package "read-edid", which comes with the "parse-edid" program. This program will generate an x conf "Display" section from the EDID information specifically for your monitor with all the modes and timing information.

Paste this section into your x conf file and add the following two options to the "Device" section to stop your video card from using the EDID frequency and timing information.

"UseEdidFreqs" "false"
Option "ExactModeTimingsDVI" "true"

And thats it, you get your custom resolutions, no more need for the CustomEDID option or file, and your HDMI audio will now work as it is not getting ignored by your video card.

For reference, here is my xorg.conf file, obviously this is specifically for my monitor (which is rather rare, so dont try it on yours).

# xorg.conf (X.Org X Window System server configuration file)
#
# This file was generated by dexconf, the Debian X Configuration tool, using
# values from the debconf database.
#
# Edit this file with caution, and see the xorg.conf manual page.
# (Type "man xorg.conf" at the shell prompt.)
#
# This file is automatically updated on xserver-xorg package upgrades *only*
# if it has not been modified since the last upgrade of the xserver-xorg
# package.
#
# If you have edited this file but would like it to be automatically updated
# again, run the following command:
#   sudo dpkg-reconfigure -phigh xserver-xorg

Section "InputDevice"
        Identifier      "Generic Keyboard"
        Driver          "kbd"
        Option          "XkbRules"      "xorg"
        Option          "XkbModel"      "pc104"
        Option          "XkbLayout"     "us"
EndSection

Section "InputDevice"
        Identifier      "Configured Mouse"
        Driver          "mouse"
EndSection

Section "Device"
        Identifier      "Configured Video Device"
        driver          "nvidia"

        Option          "UseEdidFreqs"          "false"
        Option          "ExactModeTimingsDVI"   "true"
EndSection

Section "Monitor"
        Identifier      "LG"
        VendorName      "GSM"
        ModelName       "LG"
        HorizSync       31-86
        VertRefresh     56-100

        Mode "1280x720"
                DotClock        108.000000
                HTimings        1280 1328 1440 1688
                VTimings        720 721 724 762
                Flags           "-HSync" "-VSync"
        EndMode
EndSection

Section "Screen"
        Identifier      "Default Screen"
        Monitor         "LG"
        SubSection "Display"
                Depth   24
                Modes   "1280x720"
        EndSubSection
EndSection

cPanel secondary master DNS without cPanel

At work I manage a cPanel server for over 200 clients that we needed to setup a secondary DNS server for. Since we wanted to use the machine for more then just backup DNS, and we wanted to run Debian, the cPanel DNS Only solution is no solution for us.

Here is how to slave cPanel to a Bind9 server without breaking cPanel or having cPanel undo the sync changes.

On the slave server I installed bind (apt-get install bind9) and made a directory "/Scripts" where I write two scripts, one is a class to wrap up cPanel API functions, the other is the actual DNS zone sync script. Both were written in PHP since it is my language of choice and is very easy to interface to the cPanel API with.

The class
=================================
<?PHP
        class cPanel {
                private $host;
                private $user;
                private $hash;

                private function __construct($host, $user, $hash) {
                        $this->host = $host;
                        $this->user = $user;
                        $this->hash = preg_replace("/[\t\r\n ]/", "", $hash);
                }

                private static $instance;
                public static function getInstance($host, $user, $hash) {
                        if (!(cPanel::$instance instanceof cPanel))
                                cPanel::$instance = new cPanel($host, $user, $hash);
                        return cPanel::$instance;
                }

                private function getJSON($request) {
                        $query = "https://{$this->host}:2087/json-api/" . $request;
                        $curl = curl_init();
                        curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 1);
                        curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 1);
                        curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
                        $header = array("Authorization: WHM {$this->user}:{$this->hash}");
                        curl_setopt($curl,CURLOPT_HTTPHEADER,$header);
                        curl_setopt($curl, CURLOPT_URL, $query);
                        $result = curl_exec($curl);
                        curl_close($curl);
                        if ($result == false)
                                throw new Exception("curl_exec threw error \"" . curl_error($curl) . "\" for $query");
                        return json_decode($result);
                }

                public function listzones() {
                        $info = $this->getJSON('listzones');
                        $zones = array();
                        foreach($info->zone as $zone)
                                $zones[$zone->domain] = $zone->zonefile;
                        return $zones;
                }
        }
?>
=================================
 
And the script:
=================================
#!/usr/bin/php
<?PHP ob_start(); ?>
PUT YOUR cPanel ACCESS HASH HERE 
<?PHP
        $hash   = ob_get_clean();
        $master = '123.123.123.123'; //YOUR MASTER SERVER IP
        require('/Scripts/cPanel.php');
        $cpanel = cPanel::getInstance('www.YOUR_HOST.com', 'root', $hash);
        $zones = $cpanel->listzones();
        if (count($zones) == 0)
                throw new Exception('Unable to retrieve the server\'s zone list');

        $fp = fopen('/etc/bind/cpanel.zones', 'w');
        foreach($zones as $zone => $file) {
                $zone =
                        "zone \"{$zone}\" {\n" .
                        "       type slave;\n" .
                        "       file \"{$file}\";\n" .
                        "       masters { {$master}; };\n" .
                        "};\n";

                fwrite($fp, $zone);
        }
        fclose($fp);
        shell_exec('/usr/sbin/rndc reload');
?>
================================= 

I then ran the script to create the zone file, and then created a symlink in /etc/cron.hourly to the sync script, so every hour the zones are fetched from the server, the configuration is re-built and bind is reloaded.

Then I modified /etc/bind/named.conf.options with the following two options, the first for security, the second to accept notify events from the master server to tell us to update our records.

options {

        ........ default options ........

    recursion no;

    allow-notify { 123.123.123.123; };

};

replace 123.123.123.123 with the master server's ip address. Then I just had to make a change to /etc/bind/name.conf.local to include the zone list built from the master server.

include "/etc/bind/cpanel.zones"

And thats it for the slave... now the master needs a minor modification to tell the client to update and to allow zone transfers to the slave. Add the following two lines to the options section in /etc/named.conf

allow-transfer { 123.123.123.123; };

also-notify { 123.123.123.123; };

Again, replacing the 123.123.123.123 with the IP address of the slave server.

Reduce your server's bandwidth consumption with cPanel

For those that are interested, I manage a server running cPanel with over 200 accounts, on average we are using around 220GiB per month traffic server wide.

Turns out, that cPanel does not enable any form of HTTP compression for general requests, even though the WHM supports it, pretty dumb IMHO.

Anyway, here is how to turn it on... go to "Apache Configuration" and choose "Include Editor"
Select "All Versions" under "Pre Main Include"
Paste the following in:

AddType application/vnd.ms-word.document.macroEnabled.12 .docm
AddType application/vnd.openxmlformats-officedocument.wordprocessingml.document .docx
AddType application/vnd.ms-word.template.macroEnabled.12 .dotm
AddType application/vnd.openxmlformats-officedocument.wordprocessingml.template .dotx
AddType application/vnd.ms-powerpoint.slideshow.macroEnabled.12 .ppsm
AddType application/vnd.openxmlformats-officedocument.presentationml.slideshow .ppsx
AddType application/vnd.ms-powerpoint.presentation.macroEnabled.12 .pptm
AddType application/vnd.openxmlformats-officedocument.presentationml.presentation .pptx
AddType application/vnd.ms-excel.sheet.binary.macroEnabled.12 .xlsb
AddType application/vnd.ms-excel.sheet.macroEnabled.12 .xlsm
AddType application/vnd.openxmlformats-officedocument.spreadsheetml.sheet .xlsx
AddType application/vnd.ms-xpsdocument .xps

DeflateCompressionLevel 9
AddOutputFilterByType   DEFLATE text/html text/plain text/xml text/css text/javascript text/vbscript text/x-html text/iuls text/richtext text/scriptlet text/tab-separated-values text/webviewhtml text/x-component text/x-setext text/x-vcard application/x-javascript application/javascript image/bmp

<IfModule mod_expires.c>
    ExpiresActive on
    ExpiresDefault A600
    ExpiresByType image/gif "access plus 1 day"
    ExpiresByType image/jpeg "access plus 1 day"
    ExpiresByType image/png "access plus 1 day"
    ExpiresByType image/x-icon "access plus 1 day"
    <FilesMatch "\.(php|php4)$">
        ExpiresByType text/html "now"
    </FilesMatch>
</IfModule>

This will also add mime type for the Microsoft formats that Apache does not know about by default, and set any image content to expire +1 day, so that they are not refreshed over and over.

This has reduced the servers consumption to around 80GiB per month, less then half.

OSX on a PC

My kind boss bought me an iPod Touch as an end of year gift, and naturally I wanted to write software for it, but (there is always a but), you need a Mac to write software for the iPod/iPhone, which sucks. There is two solutions to this problem, one is to buy a Mac, which I don't have the cash for, or get OSX running on my PC.

Due to financial reasons the latter was the only option, so I obtained an iAtkos install CD for Leopard. Following a few YouTube videos I did a base install, only to discover that my PC is not very mac friendly.

My PC has the following hardware:

• AMD Phenom II X2 550 @ 3.1Ghz


• ASUS M2N32-SLI Deluxe Motherboard


• PCIe nVidia GeForce 8600GT


• 2GB DDR2 Ram

After a day of messing around with install options it became aparrent that things just would not work on this motherboard until I discovered an obscure post of a forum somewhere where someone just moved the video card into the 2nd PCIe slot on the board. I initially dismissed it, but in the end, out of desperation, I tried it, and it fixed it!.

For others that have the following symptoms, try moving the video card.

• Boots to a blue screen, then goes black


• Mouse is inverted (up is down, left is right)


• Mouse is VERY jerky

Once that was sorted out, I soon discovered that the on-board ethernet would not function, being a nForce chipset, and after reading numerous posts, there was only a partial working kext for it that did not support SMP, and would crash under heavy load. Again, I was going to give up, but found another obscure post referencing an updated ethernet driver, which I tried, and the system became rock solid with both cores enabled.

I do not use the onboard audio, I use a SBLive, so I installed the Kx driver for Mac OSX and it started to work instantly!

The last thing to get working was the nForce SATA, and PATA. The iAtkos came with a driver that worked, but not too great. I found an updated driver called SuperNForceATA which was compiled for Snow Leopard (10.6), and would not load on my sytem, it did however come with the source. So I downloaded xCode and compiled it for 10.5, installed it, and off it went.

I now have a FULLY functional Mac OSX install on a PC

I have decided to start a page dedicated to WORKING osx kexts in the next few weeks. Enough for now, have to go and help out the wife :)

The ultimate media storage solution

I am the kind of person that has trouble making decisions in little things, and I am also the kind of person that hates a messy file system, so when I ran out of room on my media PC, I had to add another HDD to the machine to store more data, which presented a problem for me.

You see, I already had two other disks in the machine for storage, a 500GB and a 1TB disk, which were already fairly well sorted with all of my movies on the 500GB, and TV Shows on the 1TB. But adding a third disk would mean that I would have to split my TV Shows onto the third disk making it harder to manage since I would have to hunt for the show I would want to watch between the two disks.

The only solution I knew of to this problem was a RAID1 (stripe) array, which would combine all three disks into one huge disk... but there are issues with this, the partitions could only be as large as the smallest disk (500GB), and if one partition/disk is lost, all the data is lost.

I was just about to give up when I remembered something I read somewhere a while ago called LVM, or Logical Volume Management. I was investigating it for server backup since I manage multiple servers at work with 99.99% uptime requirements. LVM is very promising for that too, but it was perfect for this as well.

You see, LVM is a layer of abstraction on top of the physical hard disks, sort of like Virtual Memory, where the operating system and software see 2GB of RAM available, but you only have 1GB of physical ram, and a 1GB swap partition on your HDD.

I was able to combine all three disks into one huge volume, and it was so easy to do too. Here is the df output of my media machine now (note, as of writing, 2TB is the largest HDD available on the market).

# df -H
Filesystem             Size   Used  Avail Use% Mounted on
/dev/hda1               39G    13G    27G  32% /
tmpfs                  1.1G      0   1.1G   0% /lib/init/rw
udev                    11M   762k   9.8M   8% /dev
tmpfs                  1.1G   4.1k   1.1G   1% /dev/shm
none                   1.1G   4.1k   1.1G   1% /dev/shm
/dev/mapper/skx--vol-BulkData
                       2.5T   1.2T   1.3T  49% /media/BulkData

To set this up is very simple... instead of re-writing a whole bunch of info though, I will point you to an excellent beginners guide to setting this up on a machine that has already been installed and configured.

I highly recommend LVM for a media machine, it is so nice to have all my data in one location, and not have to think about where to put things, or where I out them when I want to watch them.