a business continuity plan for government

A Business Continuity Planfor GovernmentA Business Continuity Planfor GovernmentGeorge BomarDianne CaseyTexas Department of Licensing and Regulation

A practiced logistical plan for how an organization will recover and restore

partially or completely interrupted critical functions within a predetermined time after a disaster or extended disruption.

The Focus on PeopleThe Focus on People

“For the main event, CIO Steve Yates wanted to test more than the company's technology procedures; he wanted to incorporate the most unpredictable element in any contingency planning exercise: the people.”

USAA Insurance Company

Legacy of Y2k - Computer failures in banking, power, health, telecommunications and financial institutions

September 11, 2001– “Worst case” scenario concept shifted

80% of companies worldwide are not prepared for a pandemic or a natural disaster

U.S. DOL estimates over 40% of businesses never reopen following a disaster

Of the remaining 60%, 25% close within 2 years.

Selected StatsSelected Stats

Fires permanently close 44% of businesses affected

90% of companies that lose data are forced to shut down within 2 years

1993 World Trade Center bombing 150 of 350 affected businesses failed

Selected StatsSelected Stats

More Arkansas Poultry Flocks Checked For Bird Flu (UPDATED SATURDAY, JUNE 14, 2008 5:55 PM CDT IN NEWS)

By The Associated Press

“Within a few days all commercial chicken houses in the area had been tested and the 15,000 birds affected were killed and buried. The next step was for the commission to go door-to-door, checking for other cases.”

The Food and Drug Administration is expanding its warning to consumers nationwide that a salmonellosis outbreak has been linked to consumption of certain raw red plum, red Roma, and red round tomatoes, and products containing these raw, red tomatoes.

June 5, 2008

The Emergency Email and Wireless Network

What does BCP “look like”What does BCP “look like”

Formal printed manual

Full access by employees

Stored in multiple locations

Secondary work center

Copies of critical materials

Relationship to Relationship to Disaster Recovery PlanDisaster Recovery Plan

DR - focused on information technology applications domain

Overlap with BCP

Crisis mgmt structure

Secondary work center

Data requirements between primary and secondary work centers:

Telecommunications architecture;

Data replication methodology;

Application and software availability;

Any physical data requirements at secondary site.

Recommended BCP approachRecommended BCP approach

Smaller ones always contain partial elements of larger disasters

BCP should be broader than disaster recovery alone or in case of emergency (“ICE”) procedures

Plan for the BIG disastersPlan for the BIG disasters

BCP PurposeBCP Purpose

To enable leaders to

maintain essential business

processes and practices

and equip the organization

with means of becoming

less vulnerable to incidents

The TDLR PlanThe TDLR Plan

Identifies management team members

Designates remote site(s)

Enumerates four (4) major scenarios

Itemizes recovery steps to be taken within

five (5) primary business functions

Loss of key personnel

Weather-related

Infrastructure-related

Internal system breakdowns

EventsThat might trigger an interruptionEventsThat might trigger an interruption

Failure of an external business partner

Health crisis impacting the work force

A cyber attack

An act of terrorism

EventsThat might trigger an interruptionEventsThat might trigger an interruption

Rating the TriggersRating the Triggers

1- Least likely to happen

4 - Most likely to happen

Probabilities of occurrenceProbabilities of occurrence

ImpactsImpacts

DURATIONWill the effects be short-term, or longer?

EXTENTHow much of work force is impacted?

Devising a TemplateDevising a Template

A questionnaire was circulated to capture:

Recovery procedure

Recovery time objective

Recovery location

Dependencies

Other considerations

Summary of recovery steps

The ProcessThe Process

Solicit written input from key personnel

via templates

Interview managers

Prepare draft for each business function

Obtain review comments and incorporate

into revised draft

How About Prevention?How About Prevention?

Mitigate the impact of a disaster:

Practice good housekeeping

Adhere to security procedures

Observe information security procedures

Maintain up-to-date operating guidelines

An Emergency Management Team An Emergency Management Team Convenes to decide: Convenes to decide:

Implement the BCP?

Activation prompted by Team Lead

Alternate Location(s)Alternate Location(s)

Primary Site

Alternate Site

BCP provides directions to the sites

Scenario IScenario I

The population of possible causes was condensed into four (4) major scenarios:

Loss of key executive personnel for a protracted period due to accident or other unforeseen event;

Scenario IIScenario II

Loss of building access because of weather (or other natural disaster)-related event;

Scenario IIIScenario III

Contractor default, or other supplier of a critical service to the agency, abruptly goes out of business without warning; and,

Scenario IVScenario IV

Health crisis (or act of terrorism) leads to an exorbitant rate of employee absenteeism (and temporary replacements are unavailable).

Functions ImpactedFunctions Impacted

The plan identifies five (5) main business functions adversely affected by the crisis:

Licensing of individuals and businesses

Education and examination activities

Measures to ensure compliance

Administrative support

Technological support

Initial ApproachInitial Approach

For each of the five (5) business functions,

Identify impact,

Recovery procedures, and

Dependencies

Redundancy

Adopted ApproachAdopted Approach

For each of the four (4) scenarios:

Identify how each business function

would be adversely impacted

Example IExample I

If key personnel were lost (Scenario I)

Notify the agency’s directors

Convene emergency meeting of the Commission

Formulate short-term succession plan

Notify Governor’s office and key legislators

Designate primary agency contacts

Implement plans to notify the public, equip customer service, respond to complaints

Example IIExample II

If building was inaccessible (Scenario II)

Licensing

Education and Examinations

Compliance

Administrative Support

Technological Support

Example IIIExample III

If major contractor failed (Scenario III)

Identify affected functions

Marketplace alternatives?

Make temporary process changes

Procure new/other contractor

Example IVExample IV

If a health crisis decimated the work force (Scenario IV)

Identify skills of available staff

Can skills be realigned?

Determine what functions (e.g. inspections) can be postponed or suspended

Consider tapping into regulated industries for temporary expertise

A Summary of Recovery StepsA Summary of Recovery Steps

Plan must specify:

Key actions to be taken,

By whom,

In what order,

For each business function.

Important AddendaImportant Addenda

Identify in an Appendix

BCP Team Lead and Members

with current contact information

Name and address

Phone number(s)

E-mail address(es)

Include:

a Phone Tree listing - who will contact whom;

Identify how information will be disseminated to employees;

List first group(s) to report to alternate site.

Periodically,

re-assess your BCP

and update as needed!

TestingTesting

Purpose:

Achieve organizational acceptance

Determine that the BCP solution is appropriate for recovery requirements

Identify and correct design flaws

Identify and correct implementation errors

After 9/11, those companies with

tested BCP manuals had business

resumption within days.

Selected StatsSelected Stats

45% of companies with a BCP do not test it annually

80% of companies have not developed an IT crisis management function

40% of companies that have a crisis management plan do not have a dedicated crisis management team

Mistakes and PitfallsMistakes and Pitfalls

Failing to gain senior level management support

Not identifying all critical systems (including laptop data)

Failing to bring the entire business into planning and testing

Not identifying and planning for all gaps in recovery objectives

Insufficient funding for testing

USAA StoryUSAA Story

20,000+ employees - needed HazMat training, an evacuation plan and a recovery plan

Live exercises were confined to technology assets - recovering data from backup data

Otherwise, passive exercises – tabletop and paper simulations, role-play, guessing how people would react

Post 9/11, built alternative center 200 miles away from San Antonio, on different power grid and water supply

Steve Yates designed large scale continuity exercises

At the first one, USAA discovered:

The setup process for computers and phones took nearly two hours leaving employees standing in the hot Texas sun.

USAA StoryUSAA Story

USAA ‘take-away’ from testing:

Those who walked through the simulation were in the best position to find flaws and offer suggestions.

Those who practice emergency situations are less likely to panic and are more likely to remember the plan.

USAA StoryUSAA Story

Plan Maintenance CyclePlan Maintenance Cycle

Revisit annually or biannually

Confirm information; roll out to all staff

Perform staff training

Test and verify technical solutions for recovery

Test organization recovery procedures

Questions

????

Presenters:

George Bomar – 512-936-4313

GBomar@license.state.tx.us

Dianne Casey – 512-463-7182

Dianne@license.state.tx.us

Texas Department of Licensing and Regulation