It seems like it’s been an extremely long day today, and it’s not even over. I stayed up late last night doing nothing but watching bad TV. Clash of the Titans was on, and it’s such a horribly bad movie that I needed very much to watch it. Anyway, not very long after I went to sleep, my cell phone was ringing. It was 8 in the morning, and it was Dave, one of my clients. I had agreed to look after his server while he was away in Mexico. Usually, this means logging into it to check if it’s secure and operating correctly, and rebooting it if necessary. Anyway, I wasn’t scheduled to start doing this until tonight, so I was a bit surprised to hear from him.
Turns out that his server picked last night, of all nights, to die. He tried to check on it this morning, and found it unresponsive. This was extremely bad news, since he has 30 or 40 corporate web sites hosted on this box, and it also serves email for about 300 people. Every minute it was down meant more email getting lost, and more business disappearing. So, I told him to go have a look at it at the co-location facility where it’s connected to the Internet, and to phone me if I could help. I tried to go back to sleep, but I felt way too guilty for sending him there to struggle with something that I could probably fix pretty easily. So, I crawled into some clothes and drove up to the co-location facility. Sure enough, Dave was there with some server admin guy, and the both of them were scratching their heads over the thing. Since I built the server quite a few years ago, I had been warning him that it should be replaced. In fact, he had bought a new server from me last year, but hadn’t had the time to hook it up.
The damage to the sever was severe, but we had installed mirrored hard drives in it, so it wasn’t hard to restore. I got those 30 or 40 websites back up and running in a couple of hours, probably before many people noticed they were down. The email was another story. Dave has this really crappy email software running on the machine, and it was crashing like crazy. I tried everything to stablize it, but it just wasn’t going to happen. I decided that rebuilding the configuration from scratch would be a lot faster and more reliable than trying to repair it. Fortunately, Dave had good records of the configuration, so it just meant a lot of clicking and typing for me today.
In the end, Dave got to go to Mexico with his wife instead of staying home to worry about his server, possibly destroying his marriage in the process. And I got a crash course on server disaster recovery. There will be a lot of extra work on my plate this week, dealing with Dave’s customers who might have been affected by last night’s crash. But, I’ll make a pile of money from this.
I’m now too tired to do anything fun tonight. I cancelled my plans to go out, and I’ll just sit home and watch more bad TV. Everything has its tradeoff, I guess. It’s all about balance. 🙂