A lot of what we do here at Transparent revolves around protecting data and systems and ensuring business continuity for our clients.  We’ve seen our fair share of failed servers, crashed desktops and fried routers.  Almost all the new clients we meet have experienced some form of interruption, whether that is from a break-in, virus, hardware failure etc.  They’ve felt the pain.  They don’t want it to ever happen again.  We hear that message loud and clear!

We’re not immune to it ourselves.  You name it, I think we’ve experienced it.  At one point in the early days of the company we had a neighbouring business that was an incense factory.  They had a fire that ripped through the entire complex and basically destroyed everything, including our offices.  Some of our old paper records STILL smells like lavender from the incense fire.  We moved out of that office (or what was left of it) into our new home.  Within two weeks of getting setup, we were broken into and cleaned out.  All of our laptops, desktops, monitors, the works.  What a horrible feeling.  As we grew, we needed to move again.  Three months into a yet again new offices, and the floor above us sprung a leak and  water poured all through our shop all through the weekend.  Anything electronic was destroyed.   How much downtime did we experience or data did we loose?  Zero.

It’s a given that most businesses have some form of backup (sadly there are still some we encounter that have none).  However what they really need, is Recovery.  Too much emphasis is put on backup, it’s really should be called recovery.  That is the goal after all – to quickly recover.

Imagine this scenario – your server experiences a critical hardware failure and crashes.  Won’t come back to life – your not sure what’s wrong with it at this point.  You quickly get your IT manager (or whomever was handling your backups) to bring in the last backup tape.  Now what?  Hopefully all of the latest data is on that tape.  What do you do with it?

Well first before you can do anything, you need to get your server up and running before you can even start the restore process.  If that server is new, then you should have a hardware warranty with it.  Call in the technicians.  If you’re lucky they are there the next day to diagnose the issue, find out what parts need to be replaced or repaired.  If you’re lucky, it’s fixed a day or two later.  If it’s not under warranty, your shopping around for a new server, and the pressure is on to get it ordered and delivered in record time.  Before you can start to restore any critical data, first you have to get the operating system up and running with the right software and drivers.  Finally, you can start to restore the data.  You are hoping that the tapes have the latest valid data, but the reality is that tapes fail to restore far too often.  That is compounded by the age of the tapes.  While all of this is going on, your staff is without access to their data, email, calendars etc.

The end result is 2-3 days of downtime, and a if you’re backing up nightly, probably an entire day of lost data.  The high cost of rushed emergency labour and replacement of hardware is a drop in the bucket next to the lost wages, lost productivity, lost revenue, lost sales and potentially a tarnished reputation from frustrated customers.

All this assumes that everything went as planned, and that all data was on the tapes and tested.  In our experience that’s rarely the case.  In fact majority of issues that we’ve come across could have easily been prevented by simply testing the backup systems and trying to restore something.  Unfortunately most businesses wait until disaster strikes before they put their backup system to the test.

This begs the question – ‘How should businesses be preparing themselves for the inevitable crash or downtime?’.  I’m glad you asked

Here’s some minimum requirements  that every disaster recovery solution should have:

• Backs up everything automatically. Rather than trying to decide on what is important and what is not – assume everything on your server needs to be backed up.
• Backs up very frequently. Once a day is quite often just not enough.  Loosing a full days worth of data is a significant loss.
• Backs up both locally and off site.  Data needs to be kept off site in case of fire/flood/theft, however it also needs to be kept locally for rapid restoration.
• Easy to test restore. You shouldn’t have to bring down your entire system, or wait hours, or search for tapes to try to test the restore functionality.
• Restores all data, or just a single file quickly. Sometimes you don’t need to do a full recovery.  You may just need a single file or email.  It should be just as easy in either case.
• Provides the hardware required for continuity. Here’s a key item – your backup device should be able to act as an on-site standby server.  Ready to kick in at a moments notice.  That’s true recovery.
• Requires almost no human intervention. Your backup system should not rely on somebody having to remember to change tapes or drives.  It should work every time without a manual process.

That sums up the minimum requirements in a nutshell.  In reality we’ve just scratched the surface.  There are a lot more questions that need to be answered, such as:

• What is an acceptable amount of data loss for our company?  A day?  A week?  A month?
• What is our recovery time objective?  Do we have to get things back up and running in an hour?  A day?  A week?
• What happens to our email and  phone systems?  Do calls get forwarded?  To whom?
• Is there a manual process in place to complete orders and make sales?
• Who makes the decision to invoke the disaster recovery plan (ie who can pull the fire alarm)?
• Is the plan documented, and more importantly are key staff members aware of the plan and is it practiced?
• If the office is not accessible, where do staff work?  Do they take the day off?  Do they work from home?
• What business functions need to continue, what will get put aside temporarily?

As you can see there is lots to think about here.  Is it worth it?  You better bet your business on it!  I can personally say, without a shadow of a doubt that having the above systems in place, and a plan saved our business from being shut down, not once, but three separate occasions.  We experience the fire, the theft, the flood.  Is something else waiting for us around the corner?  Probably. But I’m not loosing sleep over it.  We have a plan and a system.  Its tried, it’s true and more importantly – it’s tested!

As always if you have any input or questions about this, please contact us, or leave a comment.