In-Depth

The Heart of Your DRBC Plan

Second in a series: Let's look beyond software and hardware issues to look at what elements to factor into your business continuity plan. Also, what makes a warm and hot site?

• By Bill Heldman
• 05/02/2006

Whether or not they live in New Orleans, Hurricane Katrina has caused a lot of people to start thinking much more seriously about DRBC than prior to the disaster. Consider the scenario for a moment, putting yourself in the place of Greg Meffert, the city's technology chief. You've got a myriad of applications and systems you need to be worried about:

• Building permits
• Police, fire, sheriff and first responder record tracking
• Revenue databases
• Airport systems
• Water and wastewater systems
• Parks applications
• Environmental health
• And others.

Additionally, think of all the servers, network equipment, PCs, printers, telephony systems and other specialized gear the city used on a daily basis…lots of it underwater. Filthy, greasy, sewage-infested water.
Now, donning your Monday-morning quarterback helmet for a minute, try to imagine the things you'd do differently, given a recurring scenario.

What are the systems most vital to a city? Having worked for a large city before, I can tell you anything having to do with revenue generation is right up near the top. (Of course, that's the case in any organization, isn't it?) So you would expect taxation, permitting, and other systems actually bringing in money to be the ones you'd make your highest priority in a DRBC scenario.

But what about the airport? Human services (food stamps, welfare, and housing)? Parks? Was there anyone using the parks immediately after the storm?

How about the emergency services folks like the police, fire, sheriff, ambulance drivers and paramedics? They have radio systems, wireless networks, record-tracking databases, jail systems and a host of other requirements. How about the coroner and environmental protection services? Not to be too grotesque, but from a practical standpoint, someone has to do something with those who perished.

Further, what would have happened had a bunch of city workers been killed? Suppose, for example, you lost a core building housing numerous city workers. Some sort of event killed a couple thousand while they were working away, a transformer explosion or building collapse maybe. What kind of issues would city leaders face in finding people who were savvy about various in-place, highly esoteric production systems? Imagine trying to bring in temp workers to labor on a complex, custom-built revenue and taxation system. Would the tax bills would go out accurately and on time?

Doubtful.

This is the very heart of DRBC. And, like the heart, it has two segments to it:

• What are the systems that we, without exception, must protect, else our company will go out of business? (Or, our government entity will cease to function, at least for a prolonged period.)
• Provided we are able to effectively duplicate those systems, but their normal users are no longer available, how do we come back with at least a minimum of human functionality?
  

And, there's a third important element that affects both of these questions: How quickly can we recover, given a catastrophic event? The speed element of business continuity is something we'll cover in the next edition of this series.

Laissez-faire Business Continuity Maybe for you we've entered the "Why do I care?" zone. After all, devastating hurricanes don't happen often, do they? And it was decades ago that Mount Saint Helens erupted in Washington State, making the entire mountain look like something a five-year-old might shape out of a mud puddle. But consider: On the tail of Katrina, recent devastating wildfires in Texas have led Governor Rick Perry to beg the federal government for more money to manage the devastation Texas has experienced, not just from the fires, but also from Katrina. Alabama and Mississippi are well behind New Orleans in recovering from the same hurricane, and the new hurricane season is just getting started. Mountainous snow storms occurred in the Northeast this year. Floods, wildfires, and landslides have had effects all up and down California. Heat so hot in Phoenix Satan checked out early. Not to mention in just one day in March of this year, the U.S. set a record for the number of tornadoes moving across southern states. One hundred tornadoes, numerous dead, all within a few terrifying early morning hours. And we're all tired of the constant onslaught of terrorist warnings. It seems as though people are either so jaded they don't want to think about it, or they're resolved the shoe is about to drop at any time. In either case, the result is inaction. The point is this: Even though you may think nothing will happen, there are fairly substantial odds something will happen. Would you rather have taken the time to think through the possibilities, proactively planning your reaction to them, or would you prefer be standing by the side of your boss saying "I didn't think something like this would ever happen!"

It's Risky
In both project management and risk management circles we use a technique called Risk Identification, Response and Mitigation. Let's talk through each of these elements, as they have practical import in our DRBC work.

Risk Identification/Prevention
The idea in risk identification is to get in front of a whiteboard with other decision-makers and stakeholders to try to imagine everything that could possibly happen. You want as comprehensive a list as possible. While going through your whiteboard exercise nothing is too silly -- you can always erase the silly stuff (though our friend Murphy has a way of making the silly stuff turn into the risk that actually happens). Your primary goal is to find ways to prevent risks from occurring in the first place.

The importance of a Risk Identification team cannot be understated. You cannot possibly think of every possible risk by yourself. The higher up you go in the corporate food chain for input, the more clear the scope of your efforts will become. Risk identification is an exercise requiring input from a lot of stakeholders to gain a clear picture.

Next you and the team prioritize these risks in order of the likelihood you think they might occur. For example, maybe you'll give each risk a rating between 1 and 100. The more likely something is to occur, the higher the rating.

Table 1. Risks should be assessed based on their possibility of occurring in your specific region. Earthquake might get a higher rating if you're in California, for example, than flooding, especially if your company is situated far from dams, the coast and low lying valleys.

Risk Possibility of Occurrence Fire 90 Flood 80 Earthquake 70 Landslide 60 ... ...

Risk Response
Once done with risk identification, you and your team will hammer out responses to each risk. What do we mean by response? It depends on the risk, of course, but you want to come up with a playbook for each of the risks identified. You want to know exactly how you'll respond, and what each involved party will be doing during the response. Of course, once written up and published, you'll give a copy of these risk responses to everyone involved, along with some training to help each one understand their place should a risk arise. And, you'll practice your risk responses at least once a year.

One tuning element for Risk Response is setting up a practice mechanism where you "tabletop" a risk. Remember in the last article we said a tabletop is a practice exercise where everyone plays as though the event has occurred. The idea behind tabletops isn't to see how fit someone is, or even how ready you are for an event to occur: It is to make everyone think about those things that are needed but were not initially included. You're looking for the Ah-hah factor in your tabletops.

Noteworthy: Risk Response implies buy-in from those above you. It is critical that DRBC planning include sponsorship, buy-in and participation from the top (or as close to the top as you can get). Also, it should be pointed out that leaders need to be ready to put money behind risk mitigation strategies. For example, if you identify a fire risk, your leadership needs to commit the bucks needed to prevent the occurrence from happening. In a case like this, risk mitigation doesn't say "Buy a better extinguisher", it says "Do something about this before anything catches fire." This is a subtle differentiation, one that you should pay attention to with every risk.

Risk Mitigation/Remediation
Finally, should a risk rear its ugly head, you pull out the risk response book and go through the mechanisms to mitigate (e.g. shorten its duration and strength), and remediate the damage done.

Continuity of People Skills When I was a senior manager working on a DRBC plan at a former place of employment, we were squarely hit with the "lost the people" phenomenon. If we lost just two people who were responsible for the day-to-day financial calculations for the organization, we found that there was no one in a building of 1,200 people who could successfully sit down and re-create the business process these two had done so much they could practically do it in their sleep. After some research into the issue, we found a couple of people who had previously worked in the position years ago, and who retained some faint recollection about the process. In a pinch we felt we could put together enough of a team to re-create the business process at a hot site -- never mind that it was a lame and feeble re-creation. We ran into issues with their current supervisors. They did not want their employees taking valuable time off their current job in order to retrain on activities that had only a miniscule chance of being required. While everyone totally understood the hesitancy, we also understood the expediency of the need.

Finalizing the process
Once this process has been accomplished and the risk response books are published (expect this to take days to weeks), you can't just let everything go, never revisiting the process again. Successful risk response teams revisit things annually, updating and modifying as required to make the risk response mechanisms sensible for the current environment.

Warm Site, Hot Site
All this discussion of risk and response and who should be in place leads us to the meat of this article, and back around to our original two questions: What systems, which people?

Once you've identified all mission-critical systems in the enterprise, you must now figure out a way to duplicate those systems elsewhere. Recall in my previous article that I said a warm site is simply the data replicated elsewhere for easy retrieval. It's warm because we still have to do something to be able to access the data -- whether "do something" means hire contractors to set up systems so we can access it, or we pack equipment over to the warm site for retrieval of the data, or some other mechanism.

The point is this: A warm site implies we're half-way there, not all the way. A warm site in no way fully protects your organization! It's like buying a $10,000 term life insurance policy, when you need $500,000 to pay off your house plus make sure your family can continue to live a normal lifestyle in the event of your death.

The Gear
A hot site, on the other hand, is a site that is completely ready to go. The data is being replicated over to this site on a routine basis (timing is everything, as we'll discuss momentarily). But also, computers similar to the ones used in production are set up at the hot site. They have the required applications installed on them, they have been connected to the data, and their functionality has been tested. In the event of a catastrophe, all people need to do is drive to the site, log on and begin working.

This sounds easy of course, but it is incredibly difficult to accomplish. There are numerous considerations:

• Will you outsource the hot-site hosting? There are a lot of great providers who can deliver this service to you, for a fee. They will likely not provide similar computers for you. While they may provide space for you on a SAN for your data requirements, you are responsible for providing the equipment to connect data and apps.
• Will you develop a hot site at another place in your corporate environment, or with a corporate partner? In this case, how can you be sure that the disk space, servers and computers required will be adequate?
• The "similar computers" you provide at your hot site may not be as robust as the computers your users currently have.
• Conversely, the host (whether servers or SAN) for your data may not be as robust.

In either case (non-robust client computers and servers or data repositories) means reduced computing cycles, less throughput and an overall reduction in business performance.

Most importantly, all of this effort comes at a cost…potentially a tremendous cost. So-called "C-band" folks (CFOs, CIOs) may not be inclined to be hugely generous in developing hot-site deployments.

Noteworthy: The number of times a day that you replicate data to the hot site closely corresponds to the number of hours (or minutes, or seconds) that your company is behind the eight-ball, in terms of catching back up after a disaster. If you have your SAN set to copy a snapshot of the transaction deltas over to the hot site once an hour, and workers commit 1,000 transactions every hour, at the time of a disaster and bringing up of the hot site, you are, at a minimum, 1,000 transactions behind. Someone will need to somehow re-key or re-enter those 1,000 transactions into the system.

Copyright © ScienceCartoonsPlus.com

The Folks
In a famous Sidney Harris cartoon two scientists are discussing some complex formulas written on the chalkboard. In the middle there is some writing: "Then a miracle occurs". The senior scientist says to the junior: "I think you should be more explicit here in step two." Nowhere is the A Miracle Happens Here expectation more prevalent than in DRBC.

Even though you have a hot site, if the people who had the technical expertise to understand and run the applications are gone, you haven't accomplished a complete DRBC plan. Businesses can't rely on "click-experimentation" (also known as the "by gosh and by golly" method) -- e.g. novice application users clicking through various menu options in an effort to figure out how the application works -- for day-to-day operations. You have to have people who can sit down and "drive" the application effectively so the business can rebound from the catastrophe.

You can outsource SQL Server DBAs, but you cannot easily outsource customized enterprise application expertise.

In the next article in this series, we'll explore the development of a hot-site even further, as there are far-reaching ramifications revolving around its development. "Hot site" is easy to say, but quite difficult to develop.