Nginx vs Litespeed

It has been a while since my last update here, so I thought I would publish my latest win with a client's overloaded server.

Now, before we begin, ill explain the situation, this client, who will remain nameless due to privacy reasons, hosts a very popular site that is used by thousands of people daily, one of the servers primarily serves files from an APT like repository.

The host that is serving these files is by no means slow, it is an 8 core 64bit with 16GB of ram running CentOS 5. What makes this more then mere file serving is that every request is re-written to a php script that checks for download authorization with a remote service and also logs stats on each file download.

The server was configured with Litespeed, but over the new year break some new software was released which has caused the load on this server to increase over 15 times. In an attempt to reduce load I tried just about everything, tuning Litespeed, installing eAccelerator into PHP, tuning the script that handles the downloads. These things helped, the server load was up over 150-160 before these changes and this reduced it to around 70-80.

This was still not good enough, I advised the client that there is nothing more we can do and that he would have to purchase more servers to deal with the load... but I was not at all happy with this solution.

So, over the next few days I tried a few more things, such as a reverse Nginx proxy in front of litespeed, this helped by further reducing the load to around 30-40, but again, the server was still way overloaded.

Further profiling and examination identified some very poor code in the stats script that was quickly corrected, such as fetching a file from the local host via the fopen url wrapper instead of from the hard disk directly, and using Memcache to cache hits on files and only update the database every 30 seconds. This did not reduce load but prevented Litespeed from running out of connections that were causing 500 errors.

After discussing options with the client we decided to swap out Litespeed with Nginx completely, so over the next day I re-wrote the rewrite rules into Nginx and setup PHP FastCGI workers ready to handle requests. This was all done limiting these new rules to my IP address so I could test these changes, and the public would still get passed to the Litespeed backend.

Once everything was confirmed I stopped Litespeed and started up Nginx as the stand alone webserver, with full logging to ensure that things were running smoothly. Immediately the server load dropped to 2-3, initially I thought that something was broken, PHP was not running or some other fault, but testing proved that it was all running as it should.

After a few minior tweaks to the configuration such as number of PHP workers and Nginx workers, the servers load dropped below 3 permanently. I did not expect to see such a huge performance increase by just moving away from Litespeed, I was under the false impression that Litespeed is about as fast as Nginx, and faster with PHP... but this experience has without a doubt proved that Litespeed just cant keep up with Nginx + PHP FastCGI when configured properly.

cPanel secondary master DNS without cPanel

At work I manage a cPanel server for over 200 clients that we needed to setup a secondary DNS server for. Since we wanted to use the machine for more then just backup DNS, and we wanted to run Debian, the cPanel DNS Only solution is no solution for us.

Here is how to slave cPanel to a Bind9 server without breaking cPanel or having cPanel undo the sync changes.

On the slave server I installed bind (apt-get install bind9) and made a directory "/Scripts" where I write two scripts, one is a class to wrap up cPanel API functions, the other is the actual DNS zone sync script. Both were written in PHP since it is my language of choice and is very easy to interface to the cPanel API with.

The class
=================================
<?PHP
        class cPanel {
                private $host;
                private $user;
                private $hash;

                private function __construct($host, $user, $hash) {
                        $this->host = $host;
                        $this->user = $user;
                        $this->hash = preg_replace("/[\t\r\n ]/", "", $hash);
                }

                private static $instance;
                public static function getInstance($host, $user, $hash) {
                        if (!(cPanel::$instance instanceof cPanel))
                                cPanel::$instance = new cPanel($host, $user, $hash);
                        return cPanel::$instance;
                }

                private function getJSON($request) {
                        $query = "https://{$this->host}:2087/json-api/" . $request;
                        $curl = curl_init();
                        curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 1);
                        curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 1);
                        curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
                        $header = array("Authorization: WHM {$this->user}:{$this->hash}");
                        curl_setopt($curl,CURLOPT_HTTPHEADER,$header);
                        curl_setopt($curl, CURLOPT_URL, $query);
                        $result = curl_exec($curl);
                        curl_close($curl);
                        if ($result == false)
                                throw new Exception("curl_exec threw error \"" . curl_error($curl) . "\" for $query");
                        return json_decode($result);
                }

                public function listzones() {
                        $info = $this->getJSON('listzones');
                        $zones = array();
                        foreach($info->zone as $zone)
                                $zones[$zone->domain] = $zone->zonefile;
                        return $zones;
                }
        }
?>
=================================
 
And the script:
=================================
#!/usr/bin/php
<?PHP ob_start(); ?>
PUT YOUR cPanel ACCESS HASH HERE 
<?PHP
        $hash   = ob_get_clean();
        $master = '123.123.123.123'; //YOUR MASTER SERVER IP
        require('/Scripts/cPanel.php');
        $cpanel = cPanel::getInstance('www.YOUR_HOST.com', 'root', $hash);
        $zones = $cpanel->listzones();
        if (count($zones) == 0)
                throw new Exception('Unable to retrieve the server\'s zone list');

        $fp = fopen('/etc/bind/cpanel.zones', 'w');
        foreach($zones as $zone => $file) {
                $zone =
                        "zone \"{$zone}\" {\n" .
                        "       type slave;\n" .
                        "       file \"{$file}\";\n" .
                        "       masters { {$master}; };\n" .
                        "};\n";

                fwrite($fp, $zone);
        }
        fclose($fp);
        shell_exec('/usr/sbin/rndc reload');
?>
================================= 

I then ran the script to create the zone file, and then created a symlink in /etc/cron.hourly to the sync script, so every hour the zones are fetched from the server, the configuration is re-built and bind is reloaded.

Then I modified /etc/bind/named.conf.options with the following two options, the first for security, the second to accept notify events from the master server to tell us to update our records.

options {

        ........ default options ........

    recursion no;

    allow-notify { 123.123.123.123; };

};

replace 123.123.123.123 with the master server's ip address. Then I just had to make a change to /etc/bind/name.conf.local to include the zone list built from the master server.

include "/etc/bind/cpanel.zones"

And thats it for the slave... now the master needs a minor modification to tell the client to update and to allow zone transfers to the slave. Add the following two lines to the options section in /etc/named.conf

allow-transfer { 123.123.123.123; };

also-notify { 123.123.123.123; };

Again, replacing the 123.123.123.123 with the IP address of the slave server.