On June 29th, 2012 at 5:15PM a brief but severe front came through the Midwest and in a span of under twenty minutes cut electricity, phone and data lines to millions of homes and businesses. My business was one. Fortunately it was Friday and just after the standard work day and our battery backups switched on and beeped loudly as they strained to keep up our servers and phone system. Immediately our phone lines filled up with customers calling in with questions on how to shut down servers or what to do next when the power shut off at their business. We were able to support our customers for another 30 minutes until our batteries drained low and cleanly shut down our servers. As we listened to the radio and grasp the breath of this storm we realized we were looking at a multiple day if not weeklong power outage for most of our customers.
Let’s apply this knowledge while the experience is fresh. What is the disaster recovery check list that every business should consider?
1) Business location, not everyone can change this but just as the high ground is always advantageous in war, where you locate your business has advantages. My business is located in Westerville and happens to have their own city power, independent of the big power companies like AEP. Just like most of the cooperative power companies, they seem to be more responsive than AEP. Westerville City Power had the lights back on in my office building at 4:00AM on the following Saturday morning. Additional is your building high and well drained? Could it stay dry with rainfall of 4 to 8 inch per hour produced by a slow moving thunderstorm? Are the power and data lines run under ground to your building so they are more resistant to trees toppling over on power lines? Is your building near the main drag in your city? The main routes and business centers often get power first and have the most upgraded electric and data lines. Think through these strategic advantages when you resign that lease or consider a business move. Are you on the first or second floor? Do you need an elevator to get to your office? How is security in your area? Riots can sometimes break out in a power outage. Put on your thinking cap and think through the unique possibilities for your area and how you would avoid them.
2) How are your basic procedures? Do you have a good backup system that is tested and do you rotate a backup off site? Have you tested your battery backups? Do you have enough battery power to keep your servers and phone system up for more than 30 minutes? Are they smart so they will shut down the servers cleanly before they run out of juice? Do you have everything you need to run a network on a UPS? That means hubs, switches, firewalls, phone, and workstations. Do you have a flashlight or other battery powered light in the server room? I find customers often neglect workstations, a small workstation UPS costs only $80. It gives you time to do 30 minutes of work in an emergency, it prevents the workstation from turning off in a brown out and protects your computer from the spikes and surges that often come with power failures. Do you have a laptop with a longer life battery that you can use to access the server from the server room? Do you know what to do to shut down your server in a power failure? Do you have a list of important people to call for help and do you have a cell phone contact? This will include IT support, electricians, HVAC, gas, electric, plumbing, tree removal, landlord, snow removal, Datacom support, police and fire? Do you know where to get documentation on your network?
3) Once you have the basics covered, I would encourage you to consider a backup internet service. Internet for most of us has become a mission critical tool. One of the ways to make your internet more reliable is to purchase internet from two providers. For example our business has three T1’s but we also have a DSL from AT&T. The DSL uses different wire into our building and has different equipment and only cost $70 additional a month. This gives me some redundancy. We also use a Watchguard firewall that allows multiple internet connections to be plugged into it and it balances the load between the two providers. So not only do I get reliability in an emergency, I also get additional bandwidth for everyday use.
For the future…
This recent windstorm has confirmed my decision to move our servers and phones to the new Westerville Data Center. Westerville has invested our tax dollars to build a state of the art datacenter. I am talking redundant power supply, diesel engine power backup, air and humidity controlled, a building that will stand up to an F4 tornado. The city has also invested in burying fiber to all the major office buildings. My fiber will be connected to our building this fall. Once connected I will have access to 100 Gigabyte connection to the datacenter, a rack, 10 megabytes of internet speed and an emergency office space in a disaster for about the same price I am paying for datacomm today. I believe this is the next step to making my business more bullet proof. If you want to talk more about disaster recovery for your small business network, I would welcome the call.
Michael Milligan, CEO AccuNet Inc.
firstname.lastname@example.org 614-899-9900 x 119