In Depth

Disaster Recovery: Write People into the Plot

Business continuity and disaster recovery plans truly work only if they take employee needs into account

By Malcolm Wheatley

Read on for tips from companies with lessons to impart on ensuring employees are as well taken care of as IT systems or other business assets.

Employees and Disaster Recovery: Rising to the Occasion

First, a caveat. No two disasters are the same, and employees responses will differ. While blanket no-shows certainly happen, experience suggests that this is exceptional. "In the face of adversity, human beings will usually step up to the plate," asserts John Wisbey, head of business continuity services for Europe, Middle East and Africa at Computer Sciences Corp. And Wisbey knows of what he speaks: He was personally involved in the aftermath of two high-profile terrorist atrocities in the United Kingdom—the Manchester city center bombing of June 1996, and the bombing of Londons Moorgate in April 1993. The Moorgate bomb, which struck right at the heart of Londons financial district and caused almost $1 billion of damage, has especially colored Wisbeys views. An employee of financial institution J.P. Morgan at the time, he witnessed firsthand how people and organizations pitched in to help shattered businesses get back on their feet. Thanks to borrowed equipment and freely given help to install it, the trading floor of the Saudi International Bank, for example, was back in action just 48 hours after the bomb detonated, recalls Wisbey, noting that similar reports emerged from New York City in the aftermath of 9/11.

"People are far more resilient than is often imagined," he says. Yet while recognizing that resiliency, Computer Sciences Corp.s own business continuity plans make practical allowance for the unpredictability of disaster. Given an event that necessitates employee relocation, he explains, the company predicts that the majority of its employees would rise to the occasion, even in circumstances of widespread disruption. Despite this, there is an acceptance that some employees would have other priorities. "A certain percentage would put their families first, sure," says Wisbey, but for genuine reasons, which would be accepted as such. "It could be you next time; people recognize this," he says.

So while employee resilience is a useful bonus to compensate for vagaries in response, its better regarded as an adjunct to the plan, rather than the plan itself. A similar seasoned pragmatism underpins the business continuity philosophy of consultant Michael Arata, who until this spring served as director of corporate security and facilities for a large construction company based on the West Coast. "Here in California, earthquakes and natural disasters are a fact of life," he says. "To employees, the family comes first; people just arent going to show up. You can order them to, but they still arent going to show." And even if employees wanted to head for the office, they might be prevented from reaching it. "When an earthquake strikes, the authorities dont want people on the roads," says Arata. "Bridges and underpasses are closed until they can be inspected for damage, and roads are kept clear for emergency vehicles. You just cant assume that people will be able to make the journey." Likewise, he adds, you cant assume that any employees at work when an earthquake strikes will be able to get home. "We have a provision within our disaster plans to keep stranded employees safe and fed, even in the absence of power supplies—you cant just throw people out onto the street," he says. Consequently, business continuity plans at Aratas former employer have a two-pronged approach. A small corps of people at corporate headquarters is nominated to keep critical services like IT working—if possible. But fairly implicit in the assumptions that underpin the plan, adds Arata, is that this will in practice prove problematic. Accordingly, the construction companys offices in other California locations are primed to take over if required. Important though this flexible approach is, this aspect of business continuity planning isnt what most engages Arata. A personal experience cemented Aratas more closely held view. Visiting another company facility in midtown Manhattan on the morning of September 11, 2001—one with an excellent view of the twin towers—it was to be eight weeks before he returned home. Employees had gathered to watch the smoke and flames billowing from the first attack, he explains. "Then the second plane hit, and everyone went to pieces. We didnt have anybody killed, but we certainly had people affected through knowing other people who had been killed," he says. Basic needs were simply met. As New York isnt an earthquake zone, there were no stocks of provisions, says Arata. The solution: The cafeteria stayed open, feeding people until they were able to make their journey home through the chaos. Longer-term needs, though, have been met by counseling. "Its one thing thats been learned from September 11th: Your plan must include a provision for counseling," Arata says. "People really are the business, and you cant ignore them." Keeping in Touch Few businesses factor their people into business continuity planning as thoroughly as Morristown, N.J.-based telecommunications giant AT&T. When disaster strikes, says Director of Business Continuity and Recovery Services Jerry Shammas, the company aims to be ready to respond—and to that end it not only holds full-scale practices four times a year but also subjects those practice events to deliberate extremes of climate. "We need to know that our plans will work anywhere in the U.S., under any weather conditions—irrespective of whether theres three feet of snow in Pittsburgh, or its a hot humid day in Los Angeles," he says. "We also invite state and local disaster recovery agencies to participate, as well as local telcos. They like the opportunity to practice, and it helps make the scenarios as realistic as possible." Although the details of AT&Ts plans differ slightly depending on whether it is a data center or a communications facility that has been affected by a disaster, explains Shammas, the basic procedures are the same. At four locations spread across the continental United States, 120 trailer-loads of network recovery equipment lie ready to be sent to a disaster-affected site—diesel generators, network equipment, tools and replacement communications equipment. (And yes, those four full-scale practice events each year do involve actually dispatching a subset of the trucks to the site of the mock disaster.) "The idea is that the trailers can be onsite within 24 hours, subject to federal and state authorities allowing access," Shammas says. Upon arrival, he explains, the trailers are met by a team of network recovery engineers who have been flown in to get the equipment up and running. Critically, the starting premise is that there wont be anyone else around to help. "We dont rely on local employees at all," says Shammas. "We assume they arent there." But it isnt a case of "out of sight, out of mind." Even as the trucks are getting under way, an incident recovery team starts efforts to make contact with every employee who works at the affected site, using their home phone, cell phone or other known means of contact. And if they cant make contact that way, an employee from the network recovery team—when it arrives onsite—will be dispatched to their home address. "We try very hard to get in contact with everybody," says Shammas. "We want to make sure that they are all right, and identify what kinds of support they might need in terms of food, clothing and shelter. If they need cash, well lend them cash. If they need accommodation, well put them in a hotel or motel, or put them in contact with the Red Cross. We also have trailers that can be fitted out as conference rooms and offices, but which can also, if required, be fitted out with beds. The basic idea is to help people get back on their feet." Which is always easier if people havent been knocked down in the first place, of course. Shammas says the company tries to anticipate likely disasters and prepare accordingly. If a hurricane were to move along the Florida coast, for example, AT&T facilities in its likely path would transfer as much as possible of their workload to other locations, and send their workforces home. "If we know that something like that is going to happen, we have our people go home and look after their families, or board their windows, or go somewhere else," says Shammas. Many of these recovery planning and coordination functions were centralized under a three-person Incident Management Operations Center group, reporting to the COO; recently AT&Tdisbanded that center and absorbed the functions into other groups. What of relocating employees to backup facilities? The answer depends on the scale of any disruption, says Shammas. If a disaster is highly localized—a fire or flood, for example—then employees will be expected to relocate temporarily to the backup site. AT&T will provide transport if required, or employees can use their own. But if the disruption is wider, then other employees are repositioned to take up the load. "It depends on the scenario involved and if the local environment and community are affected," says Shammas. Going to Extremes AT&Ts approach to human factors ought to pass muster with John Medaska. His gripe, youll recall, was that too many companies never even consider the human factor in their contingency plans and are consequently caught out when disaster strikes. But Medaskas own definition of a gold standard is a tough one to beat. An unnamed financial institution, it turns out, has a data center in Florida. Concerned about the difficulties of relocating employees to a backup facility in a region known for hurricanes, the company analyzed historical meteorological data for risk factors. The outcome: The probability that two locations a hour and a halfs drive time apart would be affected by the same storm was extremely remote. And an hour and a half, it figured, was not only a distance that seemed about right in terms of managing, keeping current and testing a backup facility, it also was about the farthest that people would want to commute. Not that commuting is the preferred option. Near the location of the backup facility, the company also has in place contracts with local hoteliers. Not only can rooms for key staff be made available at very, very short notice at prearranged rates but they are also suitable for long stays, families and pets.As disaster planning goes, says Medaska, this companys plan is pretty hard to fault. Trouble is, as best practices go, its also pretty rare.

Other stories by Malcolm Wheatley

• Email
• Print
• Comments
• Digg This
• SlashDot