I contemplated titleing this post “How to Totally b0rk Your Server”, but decided instead to go with a more informative one instead of an editorial one. The following is a story about how to totally trash a cPanel/WHM server environment and then get it back to stable in no time. In our case, it took us about ten hours, but YOU my dear reader can bask in our silliness and be back in no time, assuming you’ve found this post that is. And oh how sorry I feel for those who don’t. Anyhow…
As many of you may know, I work for a hosting company – Mindstorm Hosting and my major responsibility is to keep the server running smoothly and address customer support issues directly related to technical problems. We run a dedicated “unmanaged” server, meaning that our hardware provider is responsible for just that – the hardware – and don’t lend very much support at all in relation to software and stability issues. So yesterday, I was attempting to move our final domain that hosts our billing/helpdesk package over to our dedicated server and stumbled upon the fact that the Zend Optimizer wasn’t installed. A quick search on some popular cPanel/WHM forums yielded me this results: run the ‘/scripts/installzendopt’ and follow the prompts. OK, piece of cake. A few strokes of the keyboard later, phpinfo(); is reporting that Zend Optimizer 3.0 is now installed.
As I started tweaking the billing software, I noticed the server getting extremely sluggish. A few apache restarts later, its not much better, so I moved on to a full system reboot. Great! Things are back to normal now. Fast forward 30 minutes and things are pretty much dead – I can’t get access to a remote shell and any page requiring PHP is a red light. Another hard reboot later (thank goodness for remote reboot capabilities), I get shell access and begin watching running processes and ‘top’ to see whats going on behind the scenes. It didn’t take but a few minutes to find the culprit – PHP! The longer I watched, the more PHP processes appeared. At first, it was five using 19.9% of the CPU each, then it was ten using 9.8% each – yikes! For those not in the know technically, 100% CPU utilization is always bad – but its especially not fun in the web hosting business. On your personal computer, you’re probably the only person using it, and you can sit and wait for a minute for your machine to catch up – every minute that passes on a dead web server is hundreds and hundreds of unserved web pages.
OK, so now we’ve identified the cause of the sluggishness, so how do we fix it? Well, my instant fix was to run the command ‘killall -9 php’ which worked, but only temporarily, so I setup a cron job to run that command every five minutes while I sorted it out. My first instinct was to recompile apache/PHP now that Zend Optimizer was installed. This took about two hours via the WHM interface thanks to those stuck PHP proceses before the cron job was fired off.
But still no joy. Finally, I smartened up and reverted my php.ini file to the pre-Zend Optimizer one. It was as simple on my server as going into /usr/local/lib, removing the symbolic link created by Zend and renaming the php.ini-zend_optimizer.bak to php.ini, and restarting apache.
Bingo! We have a fully working server now. But I still can’t run ModernBill because it’s complaining now that Zend Optimizer isn’t installed. So what’s a server admin to do? Sleep. Thats what.
After a good night’s rest – I set out this morning to make it happen – without bringing the server to it’s knees. It didn’t take long after my coffee to find the critical difference between the server it was currently being hosted on and our dedicated setup. Everything appeared to be identical at first glance – a few different options enabled/disabled on PHP, but nothing related. Both machines were running Apache 1.3.36 and PHP 4.4.2, but then it jumped off the page at me: Zend Optimizer v2.6.2.
So I went over to http://downloads.zend.com/optimizer/ and grabbed the latest 2.x installer, extracted it, installed it, restarted apache, and ta-da!
If you’re reading this – it’s still running strong. If not – I’m fired.