Emergency Response
Posted on 2007-03-28 (Updated on 2019-01-22)
I'm at work, but I'm not at work. I'm sitting in the Public Safety building, in a meeting room, with other essential staff. Certain public offices, like Public Health and the County Clerk, have a skeleton crew occupying a couple of seats. Computers and phones are set up where they're neccessary, and people come and go like it's business as usual.
Yesterday, the main electrical point of entry caught fire and started arcing all over the basement. The backup generator kicked in, which caused more fireworks. The building lost power, and acrid smoke filled the halls. Our network coordinator scrambled to power down the servers and devices before he was forced to leave by the firefighters on scene.
Shortly after this all happened, I received a phone call from a coworker telling me that I shouldn't bother coming in today. Luckily, this all occurred around 9:00 AM, and I'm scheduled to start work at 10:00 AM.
After some time spent home catching up on some personal web stuff, I received another call: it's our network coordinator. We had a backup generator on scene, and were ready to turn things back on...
Shortly after I got to the office, we had almost all primary systems up and running. The only device that did not bounce back was our RAID array. If you work in Warren County, your "My Documents" folder points to that device. That's no bueno. Our network coordinator hopped on the support line with Gateway (they sold us the device), and we overnighted a new one to the municipal center.
Back to today - we have the new device, and from what I hear they're putting it in right now and rebuilding the array. We may have to restore from backup, but at the very worst that means we lose two hours of work. Our main systems were down for around five hours - not too bad if you consider the circumstances.
This week has emphasized the importance of having a good backup plan, as well as a plan for emergency response. In the event of an emergency, having a set procedure saves a tremendous ammount of time and money, and allows you to bring everything back to operation.
Now, if only the plan didn't involve me coming in two hours early...
Tags: server