0

WATERSECTORBLACK

SKYPLAYBOOKV3.0

Draft/SectorSteeringCommitteeReviewedonddmmm

2017.

ABSTRACTThisSectorBlackSkyPlaybookreflectscollectiveinput

fromnumerouspartners,aswellasoperationalindustry

technicalpersonnel,asarecommendedframeworkfor

planningresilienceinvestments,restorationplanningand

cross-sectorcoordinationneededforlongduration,multi-

regionpoweroutages.Thispeer-revieweddocumentis

designedasaresourcefortheelectricsubsector,

addressingcriticalBlackSkyresilienceneedsfor

Preparation/Mitigation,Response,Restorationand

Recoverymeasures.

JohnOrganek,PE

V3.0

1

Table of Contents RoleoftheEPROSectorBlackSkyPlaybook................................................................................................3

SectorBackground...................................................................................................................................3

SectorBlackSkyEnvironment..................................................................................................................4

SectorModelOverview................................................................................................................................5

SectorModelGraphic..............................................................................................................................6

SectorBlackSkyStrategicMissionStatement.............................................................................................7

SectorBlackSkyStrategicMissionPrioritiesMatrix................................................................................8

BlackSkyAssumptions...............................................................................................................................11

DHSPowerOutageIncidentAnnex:.....................................................................................................12

NationalInfrastructureAdvisoryCouncilWaterSectorResilience:....................................................12

BlackSkyDecisionsOverview....................................................................................................................12

BlackSkyDecisionsMatrix.....................................................................................................................12

SectorBlackSkySituationalAwarenessOverview.....................................................................................13

PriorityInformationRequirementsMatrix............................................................................................13

SectorInitialActions..................................................................................................................................14

SectorInitialActionsMatrix(V3)...........................................................................................................14

InternalSectorRequirements....................................................................................................................15

InternalSectorRequirementsMatrix.....................................................................................................22

ExternalandCrossSectorDependenciesOverview...................................................................................23

ExternalandCrossSectorRequirementsMatrix....................................................................................28

SectorSpecializedResourceRequirementsOverview...............................................................................29

SectorCommoditySpecificListMatrix...................................................................................................29

SectorBlackSkyCommunicationsOverview.............................................................................................30

SectorCommunicationsMatrix..............................................................................................................30

SectorBlackSkyAssessmentTool(s)Overview.........................................................................................31

SectorBlackSkyPlanningRequirements(On-going).................................................................................32

SectorBestPracticesMatrix(On-going)................................................................................................33

IntegratedandSharedPlanningActions(3.5/V4)......................................................................................34

2

PlanningandCoordinationActionsMatrix............................................................................................35

SectorBlackSkyResilienceConsiderationsOverview(3.5/V4).................................................................36

ResilienceInitiativesMatrix...................................................................................................................36

SectorBlackSkyRegulatoryImpactsandIssuesOverview(On-Going).....................................................37

SectorRegulatoryMatrix.......................................................................................................................37

SectorBlackSkyEssentialCriticalInfrastructure(MC)Overview(3.5/V4)................................................38

SectorCriticalInfrastructureMatrix(V3.5/V4)......................................................................................38

SectorBlackSkySpecializedSkillTrainingRequirementsOverview(V3.5/V4/V5)....................................39

SectorSpecializedSkillTrainingRequirementsMatrix(3.5/V4/V5)......................................................39

AnnexA–Assessments(On-going)............................................................................................................40

SectorOverallResilienceAssessment....................................................................................................40

AnnexB–RegulatoryIssuesDetailStatements(On-Going)........................................................................2

IssueStatement1:TestingEmergencyGenerators.................................................................................2

IssueStatement2:ReducedWaterQualityLevels..................................................................................2

AnnexC–CommunicationsRequirements(V3/V4).....................................................................................2

CommunicationsRequirement1:InternalCoordination.........................................................................2

CommunicationsRequirement2:ExternalSupportRequirements.........................................................2

CommunicationsRequirement3:StatusofServices...............................................................................3

CommunicationsRequirement4:DamagetoComponents.....................................................................3

CommunicationsRequirement5:RequestforPersonnel........................................................................3

CommunicationsRequirement5:RequestforResupplyofTreatmentChemicals..................................4

AnnexD:ResilienceRequirementsbyLayer................................................................................................5

3

Role of the EPRO Sector Black Sky Playbook TheapproachtobuildingthisPlaybookisdesignedtocontinuouslyengageWaterSectorutilitymembers

tocontributetoanevolvingframeworkofrecommendedguidelinestomanagerisksoflongduration,

multi-regionpoweroutagesassociatedwithemerging“BlackSky”hazards.

ThisPlaybookwillbecontinuallyupdatedandreviewedusingtheEPROSECTORsteeringcommittee

process,throughconsultationwithWaterSectorutilityprofessionalsandmanagers.Itrepresentsthe

consolidatedrecommendationsofthesesectormanagersandprofessionalstoaddresstheunique

challengesposedbywidearea,longdurationoutages.Itprovidesguidelinestohelpindividualentities

strengthentheirownresiliencemeasures,developfocusedoperationalplansandidentifyvitalexternal

supportneededintheeventoftheseseverehazardscenarios.

Sector Background Waterisalifelinesectorthatservesbusinessesandcommunitiesonadailybasisandbringsthemback

tonormalafteradisaster,whichmakesmaintainingwaterservicesandquicklyrestoringthemapriority.

Maintaininganddistributinganadequatesupplyofbothpotableandwaterusedforfirefighting,isvital

toensuringsuccessofashelter-in-placestrategy.

• Roughly85percentofallwaterandwastewatersystemsarepubliclyownedandoperatedby

municipalitiesandmostaresmall;morethan80percentofcommunitywatersystemsand

publiclyownedtreatmentworksservepopulationsoflessthan3,300.

• MostStateandmunicipaldecision-makersareconstrainedbylong-heldexpectationsby

customersthatwaterisalow-cost,affordableserviceandthatthecostpaidbycustomersdoes

notaccountfortruelife-cyclecosts.

• Nearlyallwaterinfrastructureassetsareoutofpublicsightandhistoricallyreliable,leadingto

anunderappreciationofthecriticalityofwaterservicesandtheinfrastructurethatdelivers

them.

• Likeothersectors,waterhasanaginginfrastructurethatrequiresmassivereinvestmentto

upgradepipes,mains,andequipment.Manyassetsarenearingorbeyondtheirexpected

lifespan,leadingtoroughly240,000watermainbreaks,andbetween23,000and75,000

sanitarysewageoverflowsperyearintheUnitedStates.Theestimatedinvestmentgapranges

fromabout$400billiontonearly$1trillion,justtomaintaincurrentlevelsofwaterservice.

• Communitywatersystemsarenotconnectedtoadjacentsystems,unlikeelectricityand

transportationinfrastructure,whichareinterconnectedintonationalnetworks.Thismeansthat

watersystemscannottakefulladvantageofnetworkeconomiesthatenablemutualassistance

anditmeansthatvirtuallyeachsystemmustbetreatedindividually,ona‘oneof’basis.

• UnliketheEnergyandTransportationSectors,whicheachhaveaFederaldepartmentand

Cabinetpositiondedicatedtotheirsectorsandinfrastructure,theWaterSectorhasno

correspondingFederaldepartmentdedicatedtoitssector.TheU.S.EnvironmentalProtection

Agency(EPA),whichservesastheSector-SpecificAgency(SSA)fortheWaterSector,regulates

andenforcestheCleanWaterActandtheSafeWaterDrinkingAct.Whileithasprograms

designedtoimprovethesecurityandresilienceoftheNation'sdrinkingwaterandwastewater

4

infrastructure,itsprimarymissionisensuringwaterqualityfromaregulatoryperspectiveand

notwithaperspectiveofoveralloperationandresilienceofthesystems.

Figure1:WaterSectorSnapshot(Source:EPA,2015WaterandWastewaterSystemsSector-SpecificPlan,(2015SSP),2016)

Sector Black Sky Environment WaterSectorutilitiesrequireaconsiderableamountofenergytoeffectend-to-endflow,processingand

distribution/collectionofwaterandwastewater.ThismakestheWaterSectorutilitiesparticularly

vulnerabletoBlackSkyhazardsandincreasestheneedtobuildinresiliencetothesehazards.Beingable

tomaintainsocietalintegrityfollowingaBlackSkyeventwillbevitallydependentonmaintaining

5

sufficientwatersupplyanddistributionsystemintegritytomeetrequirementsforpotable,firefighting

andotherusesofwater,suchasforbusinesses.Inordertoachievethegoalofshelter-in-place,the

WaterSectorutilitiesmustabletooperatesystemswhileoffoftheElectricGridandmeetminimal

sustainablelevelsofservice.Tosupportevenminimalservices,alargenumberofemergencypower

unitswillbeneedtobeacquiredand/orcapableofbeingrelocated,andtheiraccompanyingsupply

chainsforfuelandtechnicalsupportadjustedaccordingly.Neededchemicals,particularlyfor

disinfection,willbequicklydepletedandtheirresupplywillrequireamajorlogisticseffort,overa

deterioratedtransportationnetwork.TheSectormustquicklypivottoBlackSkyprotocolsthatwould

enablesustainingminimalservicelevelstothelargestnumberofpeople.

Figure2:WaterSectorImpactfromLossofElectricity(Source:NIACWaterSectorResilience:FinalReportand

Recommendations)

Sector Model Overview TheWaterSectorisdividedintotwomajorsetsofservices,WaterandWastewater.Waterservices

includesupportfor:drinking/cooking/bathing;firefighting;industrialuse;nationaldefensemission

assurance;andwastewatersystems.Wastewaterservicesincludesupportfortreatinghouseholdand

industrialwaste,aswellasforstormflow,inthecaseofacombinedsewersystem.

Waterissourced,treated,distributed(includingstorage),managedandreusedincompliancewith

standards,e.g.,qualityandpressure,toassuresocietalwellbeing.Wastewateroperationsinclude

sanitaryandstormwatercollectionandtreatment,aswellasthetreatmentanddisposalofthesludge

(biosolids)producedbytheremovalprocesses.

Figure3describestheend-to-endWaterSectorprocess,acrossbothwaterandwastewater.

6

Waterinfrastructurecomprisesthephysicalandcyberassetsofdrinkingwaterandwastewatersystems,

asdefinedbyHomelandSecurityPresidentialDirective7(HSPD-7),the2013NationalInfrastructureProtectionPlan(NIPP2013),andthe2015WaterandWastewaterSystemsSector-SpecificPlan(2015SSP).

Sector Model Graphic

Figure 3: End to End Overview of Water/Wastewater Process (reference: NIAC Water Sector Report)

TheotherLifelineSectorsalsodependheavilyontheWaterSectorutilitiesforbeingabletomaintain

theirsectoroperations,andsocietyasawholedependsontheeffectiveinteroperabilityamongallof

theseLifelineSectors.Figure4,fromtheNationalInfrastructureAdvisoryCouncilReportonWater,

depictsthedependencythatotherLifelineSectorshaveontheWaterSector.

7

Figure4:LifelineSectorInterdependencywithWaterSector(NIACWaterSectorResilience:FinalReportand

Recommendations,2016)

Sector Black Sky Strategic Mission Statement Intheseverelydisrupted,multi-region,long-durationpoweroutagesassociatedwithBlackSky

scenarios,themostcriticalgoalsforpublichealthandsafety,andsocietalcontinuitywillbesystematic,

timelyandwell-prioritizedpowergridrestoration,andmaintainingthelargestpossiblenumberof

peopleina“shelter-in-place”statusduringthatmulti-weekorlongerrestoration.

FortheWaterSectorutilities,thesecondofthesetwogoalsrepresentstheprimarymissionforthe

utility’sBlackSkyPlaybook:

Toachievethismission,minimalservicelevelsmustbedefined,especiallyforutilitiesservinglargecities

andcriticalfacilities,becausepopulationslivingindenseurbanenvironmentswillbetheleastableto

shelter-in-placeifwaterserviceisseverelydisrupted.The410verylargewatersystemsintheU.S.

(thoseserving100,000ormorecustomers)offeranespeciallyleveragedopportunitytomaximizethe

WaterSectorBlackSkyMission:Maintainingatleastminimalwaterandwastewaterservicefor

thelargestpossiblenumbersofresidentsandcriticalfacilities,inaffectedregions,forlong

durations(atleast30days).

8

impactofBlackSkypreparednessinitiatives.Ifthewaterutilitiesinmajorurbanareascansustain

reducedlevelsofservicetoenablemostoftheircustomerstoshelterinplace,thebenefitsgainedfrom

avertingunplannedmassmigrationswouldbeenormous.Focusingonthesmallnumberofverylarge

systemsalsoboundstheplanningandlogisticsproblemsthatWaterSectorutilitypartnerswillconfront.

Forgovernmentagenciesandprivatecontractorsresponsibleforsupplyinggenerators,fuel,and

chemicalsinBlackSkyevents,beingabletotargettheiroperationsonafewhundredofthelargest

systems(versusthemorethan40,000smallandverysmallsystems)wouldsignificantlynarrowtheir

challenge.Whileallwatersystemsandtheircustomersareimportant,thelimitedresourcesandthe

enormouschallengesofBlackSkyenvironmentsrequirethatresilienceeffortsfocusonthe(currently)

greatestvulnerabilitytounplannedmassmigration.

PreparingtoimplementtheseservicelevelsinBlackSkyscenariosrequires,inturn,developmentof

expandedresilienceandoperationalplans,aswellasspecificexternalsupportmeasuresfrom

governmentandmass-careNGOpartners.

TheseBlackSkyservicelevelsandtherequirementsthatcharacterizetheexpandedsectorplanningand

externalsupportneededtoprepareforthem,maybesummarizedinthreelevels—NearNormal,

Reduced,andNoService--andaredescribedindetailbelow.

Sector Black Sky Strategic Mission Priorities Matrix Meetingthesector’smissionrequirement–maintainingatleastminimalwaterandwastewaterservice

forthelargestpossiblenumberofresidentsandcriticalfacilities,forlongdurations–requirescareful

definitionofserviceprioritiesasafunctionofcustomertype,geographicregionandrelated

characteristics.DecisionsonBlackSkyservicelevels—madeincoordinationwithregulatorybodies--

alsovaryfromutilitytoutility.

Figure5:WatersystemservicelevelsinBlackSkyemergencies

Agivenutility,dependingontheoverallsystemserviceleveltheychoose,maydefinedifferent

servicesforcustomers,basedondifferentprioritycriteria.

Overallsystemservicelevelswillvaryasafunctionoftimeandmilestone.Forexample,

immediatelyafteraregionalblackoutbegins,waterutilitieswilltypicallybeginoperatingaccording

9

tonormal“GraySky1”playbooks,optimizedfornormaloperationduringshortdurationpower

outages.TofacilitateplanningforemergencydrinkingwatersupplyduringaGraySkyevent,EPA

andAWWAjointlydeveloped“PlanningforEmergencyDrinkingWaterSupply”in2011,foruseby

watersystemsofallsizes.Amongthemanyexternalrequirementsdefinedbelow,apriorityforthe

WaterSectorutilitieswillbetoreceive,asearlyaspossible,adefinitiveestimateoftheoutageclass

(i.e.,GraySky/BlackSky)toenablethemtoconvertovertoBlackSkyPlaybookprotocolsand

operationsattheearliestpossibleopportunity.

OncewaterutilitiesbecomeawarethattheoutagerepresentsalongdurationBlackSkyscenario,

eachutilitywouldthenbeginoperatingaccordingtoitsuniqueimplementationoftheBlackSky

Playbookrecommendations.AsshowninFigure5,thiswouldtypicallyinvolveshifting,notto

reducedwaterservicelevelsassociatedwithconventionalscenarios,buttolevelsassociatedwith

BlackSkyscenarios.Asreviewedbelow,thesewouldincludepreplannedchangestooperateat

minimumpressurelevels,alongwithapotentialforgeographically-variedservice,andreductionsin

waterquality.

a. System-widequalityreductions:Insteadofmaintainingdistributionsystempressuresandother

functionsnecessarytoprovidewaterthatmeetsnormaldrinkingwaterqualitystandards,

systemsshoulduselowerquality,still-potablewater,orprovidewaterthatcanbemade

drinkablebycustomers,givenadequatecommunitypreparation,suchasprovidinginstruction

about‘boiledwateradvisory’.Becausepowerandnaturalgaswillnotbegenerallyavailableto

customersinthesescenarioshowever,theprimarymeanswillincludeuseofbleach,iodineor

otherdisinfectants,ormanualfilters.

Asanessentialelementofthisproposedrequirement,recommendedemergencypurification

meanswillbecoordinatedwithgovernmentandNGOpartners,aswellasdirectlytocustomers,

toensureamaximumdegreeofadvancepreparationbythecommunity.EPAhasrecently

updatedguidanceonemergencydisinfection:(SeeEPAinstructionsonemergencydisinfection

ofdrinkingwater,whichshouldbedistributedtoallcustomers):

https://www.epa.gov/sites/production/files/2015-11/documents/epa816f15003.pdf

b. Geographically-variedservicelevels:Ratherthanseekingtomaintainasetservicelevelovera

utility’sentiredistributionsystem,utilitiesshoulddevelopplansthatallowforselectively

targetingareas,forexample,pressurezones,forsustainedservicebasedonpopulationdensity,

topography,criticalcustomers,orothercriteria.

Asastartingpoint,thiswouldinvolveassessmentofwhetheritismoreeffectivetooperatewith

reducedwaterqualityacrossthewholeserviceareaortoscalebackserviceinselectedareasif

pressurezoneisolationisfeasible.Utilitiesobtainingwaterfrommultiplewatersourcesmay

needtoexamineeachsourcetodeterminewhichneedstheleastamountofpowerand

chemicalstotreatanddeliver.

Ifgeographiccutbackswouldbeoptimumforaparticularutilityinthesescenarios,thesearea-

basedservicelevelreductionswouldbeidentifiedbyconductingadvancedplanningand

exercisesforthesescenarios.Theseoperationalplanswouldrequireadvancecoordinationwith

1Theterm“graysky”isusedhereasacounterpointtoBlackSkyevents.Itisnotintendedtobeadiminutiveterm.

Grayskyeventscanalsobeverysevere–suchasSuperstormSandyortheJapanesetsunamithatprecipitatedthe

Fukushimadisaster–buttheyaremorelocalized(sub-regional)andofshorterduration(lessthanonemonth)than

BlackSkyevents.

10

sectorpartnerssince,itmaybenecessarytomoveaportionofthepopulationwithinthe

utility’sservicearea.Inputsintotheplanningprocesswouldinclude:

1. Assessmentofwhetheronlycertainzonescanhavewaterserviceforthedurationofthe

event,orwhetherpressurezonescanbealternatedforseveralhoursatatime.

2. Assessmentoftheviabilityofusingemergencycontingencyequipment,suchasmobile

pumpingstations,toprovideintermittentservicetopressurezones.

c. System-widepressurereduction:Currently,20poundspersquareinch(PSI)ofpressureisa

typical(nearlycanonical)minimumstandardrequirementforpressureinwatersystems.Black

Skyplanningshouldconsiderthepossibilityofoperatingatevenlowerpressures—in

consultationwithregulators.Utilitiesshouldanalyzetheiroperationsanddeterminewhethera

lowerpressureforBlackSkyoperationscanbefeasiblyachievedandwhetheritwouldstillbe

safe(fordrinking)andfunctional(forfirefightingandsewage),andnotnegativelyimpactthe

physicalintegrityoftheirsystems.

d. Preplanningtoreducedemand:Meetingdemandwillbeeasierifdemandisitselfreduced.

Whilesomenaturaldemandreductionwilllikelyoccurduringemergencies,utilitiesshouldwork

withgovernmentandNGOpartnerstoprovidetheircommunitieswithadvanceinformation,

periodicmediapublicserviceannouncements,andothermeanstoencouragetheircustomers

toactivelyreducewateruseduringemergencies.2

Advanceplanningforreduced-usageagreementswithhigh-usecriticalfacilitiesthatwouldstill

beoperatinginthesescenariosisalsoapriority.Forlargewatercustomers,suchashospitals,

HVACsystemsarealargewaterconsumer.Pre-negotiatedplansforalternatewatersupplyfor

thesesystemsmaybeanoption.

e. Non-PotableWaterPlanning:Incaseswhereutilitiesareunabletoprovidereducedpressure,

minimal-puritypotablewater–orwaterthatconsumerscanmakepotable–maintainingthe

flowofwaterforfiresuppression,sewerageandothernon-drinkingpurposeswillbevitalfor

avoidingtheunplannedmigrationtippingpointinaBlackSkyevent.Utilitiesshouldensurethat

asafallbackplan,theyareabletosustainwaterflowssufficienttomeetnon-drinkingneeds.

f. Preplanningforcontrolledwaterandwastewatersystemshutdownandrestoration:Despite

implementingallofthesePlaybookoptions,someresidualriskof“completefailure”willremain.

Toprovideforaworstcasescenario,utilitiesshoulddevelopplanstoacceleratecomplete“dry

start”restorationofservicewhenthepowergridreturnstooperation.

U.S.waterutilitieshavelittleexperienceinrestoringwaterflowsafteracompleteservice

shutdown.Giventhedifficultiesofdoingso,especiallyintermsofreplacingpipesthatwill

breakoncewaterpressureislostortooquicklyrestored,itwillbeimperativeforsystemsto

avoidtotalshutdownbyseverelyreducing(andharshlyprioritizing)levelsofserviceinBlackSky

events.Nevertheless,utilitiesmaywanttohelpemergencymanagersandotherresilience

stakeholdersunderstandrestorationplans,priorities,andtimelines,andconsiderinfrastructure

investmentsthatmighthelpaccelerateservicerestoration.

g. Alternativehydro-systems:Ahydro-systemnetworkprovidesapotentialcapabilityfor

emergencymanagementagencies,inpartnershipwithwaterutilities,toprovideemergency

2ReductionindemandplanningandpublicoutreachisthesubjectofexistingguidanceissuedbyAWWAinthe

formof“M60:DroughtPreparedness&Response”,designedtoassistautilityindeterminingtriggerlevelsfora

demandreductionprogram.

11

potablewatersuppliestopopulationsthatcannotbeservedbythewaterdistributionsystem.

Analternativepotablewaterdistributionsystem,thesehydro-systems(equipmentand

operations)couldbemodeledafterIsrael’sconceptofoperationsforfurnishingpotablewater

duringwatercrises.

Althoughthiswouldnotprovideanadequatesolutionforlargecities,forsmallercommunities–

whicharemorelikelytofacesuchcompletewaterserviceshutdowns–thisapproachcould

provideavalidbackupoption.Thesealternativedistributionsystemswouldbedesigned,

configured,operatedandmaintainedtoprovidepotablewaterunder“NoWaterService”or

“FireandSanitationServicesOnly”conditions.Note:otheremergencywaterdistributiontactics,

suchasbottledorbulkwater,areaddressedunderSection3.,ExternalRequirements.

h. KeyResources:WARNandEmergencyWaterSupplyGuidance

1. TheAmericanWaterWorksAssociationhasdevelopedtheWater/WastewaterAgency

ResponseNetwork(WARN),amutualassistancenetworkfortheWaterSectorutilities

toprepareforandrespondtodisastersandotheremergencies.3

2. TheEnvironmentalProtectionAgencyandAWWAhavedevelopedguidancetofacilitate

planningandconsiderationofneedsunderemergencyconditions.4

EPROHandbookII,Chapter2(Water)andthisPlaybookprovideasolidfoundationforemergency

planningtoaddresstopicsa.throughh.aboveandcanbeexpandedupontoaddressBlackSky

scenarios.Missionprioritiesbyphasearelistedbelow.

Black Sky Assumptions BecauseoflimitedexperiencewithBlackSkyenvironments,planningintheWaterSectormust

necessarilybebasedonassumptions.Theprincipalassumptionisoneofanelectricgridoutageacrossa

verylargefootprintandthatlastsoveramonthormore.Thoughwegenerallyconsideralargefootprint

torepresentoneoftheinterconnections,itisconceivable,inacyberattackforexample,thatmultiple

3http://www.awwa.org/NationalWARN.org

4 Planning for an Emergency Drinking Water Supply http://www.awwa.org/Portals/0/files/resources/water%20knowledge/rc%20emergency%20prep/Emergencywater.PDF

Phase Priority Missions

Mitigation 1 Conductcross-sectorplanning

Mitigation 2 Participateincross-sectorBlackSkyexercises

Response 2 Reducequality,systemwide

Response 3 Varyservicelevelsgeographically

Response 4 Reducepressuresystem-wide

Response 1 Reducedemand

Response 5 Conduct“BoilWater”operations

Recovery `1 CoordinatewithElectricSubsector

Recovery 2 Restartlong-startupfacilities,suchassludgedigestors

Recovery 2 ExecutecoordinatedrestartwithElectricSubsector

12

interconnectionsmightbeaffected,ifnotdirectly,thenasaresultofbeingrestrainedfromproviding

resourcestootherinterconnections.

Tosharpentherelevanceoftheseassumptions,thefollowingassumptionsaretakenfromtwosources:

DHS Power Outage Incident Annex: • CauseAgnostic

• MultipleStates/Regions;millionsofpeoplewithoutpowerforlongperiod

• Largeportionofpopulationrequiressustainedmasscaresupport

• Lossofcriticallifelinefunctionsresultsinriskstohealth,personalsafety,nationalsecurity,and

economicviability

• ImpactstootherCriticalInfrastructuresectorscausessignificantlossofservicesorfunctionsand

createsneedforsustainedoperationalcoordination

National Infrastructure Advisory Council Water Sector Resi l ience: • BlackSkyeventsarehigh-impact/uncertainprobability

• UtilitieshavebuiltsomeresilienceforBlackSkyevents,butadditionalpreparednessisneeded

• PowerfailureisthelargestriskdependencytotheWaterSector

• Availablegenerators:

o Maynotbelargeenoughtoaddresspowerrequirements

o Requireextensivequantitiesoffuelandcontinuousmaintenancetosustainoperations

Black Sky Decisions Overview Waterandwastewatermanagersandoperatorswillberequiredtomakenumerouscriticaldecisions

followingaBlackSkyevent,mostlikelyunderaseverelylimitedcommunicationsenvironment.Detailed

planningandexerciseofinternaloperations,aswellascoordinationwithexternalpartners,would

mitigatetherisksassociatedwithbeingunabletoobtaincriticalinformationfollowingaBlackSkyevent.

PreparationsshouldfocusonreducingthetimerequiredtoeffectdecisionsinthePlan-Do-Check-Act

cycle.OnemethodistodevelopStandardOperatingProcedures,basedontheBlackSkyConceptof

Operations,toidentifydecisionsandtoderiveandminimizecriticalinformationrequirements.In

addition,utilitiescouldusehydraulicandcross-sectormodeling,insupportofBlackSkyplanning,to

identifyvariantscenariosanddevelopcontingencyplanswhichcouldthenbeexecutedcentrallywith

minimalinformation,orautonomously,bottomup,attheoperatorlevel.

Black Sky Decisions Matrix DecisionsaffectingWaterSectorutilitieswilltakeplaceat3basiclevels,ifaBlackSkyeventoccurs:

acrosssystems(Regional/State/Federal);atsystem;andbelowsystem.Furthermore,whilehierarchical

decisionmakinghasbeenproventobemostefficientapproachduringemergencies,thedecisions

acrosstheselevelswillbeaffectedbythedegreetowhichinformation,authority,expertiseand

capability-to-executeexistatanymomentintime.Asarule,decisionsduringcrisesshouldbemadeat

thelowestlevel,whileeachincreasinglevelshouldbefocusedoncoordinatingandfacilitatingthe

actionstakingplacebelowtheirlevel.Such‘federated’approachtodecisionmakingshouldbewell

coordinated,plannedandexercised.

Intheeventofamajorlossofelectricalpower,awaterutilitymustfirstdeterminewhetheraBlackSky

eventhastakenplaceandwhetherthesystemoperatormustinitiateResponseundertheirpre-

13

determined,pre-plannedBlackSkyProtocols.Ifcommunicationshavefailed,thisdecisionmayrevert

downtoindividualsubcomponentsofthesystem.

Sector Black Sky Situational Awareness Overview FollowingaBlackSkyevent,WaterSectorutilitiesmustdeterminethefollowingandincorporatethe

informationintheirEmergencyResponsePlan(ERP):

• WHAT:servicesmustIprovideandatwhatservicelevels

• WHERE:doIprovideand/orshutservicesdown,bothexternallytothepublicandinternally,to

thecomponentsofthesystem

• WHO:doIprovideservicestoandobtainservicesfrom

• WHEN:doIinitiateBlackSkyProtocols

• HOW:doIsustainminimalistservicelevels

Priority Information Requirements Matrix

Phase Priority Decision

Response 1 WhethertoinitiateBlackSkyServiceLevels

Response 2 Whatchangesaretobemadetosystemoperationstosustainservicesas

longaspossible

Mitigation,

Response

3 Wheretoemployemergencypower

Response 4 Whentorecallandwheretopositioncriticalpersonnel

Information Source Priority ConfidenceLevel

StatusofSystem SCADAandon-site

visits

1 LOW,initially,untilall

componentshavebeen

inspected

HIGH,subsequently

Consequences,e.g.,numbers

andareasaffectedbysystem

Impactanalyses

conductedby

individualsystems

andtheir

components

2 LOW,initially,untilall

componentshavebeen

inspected

HIGH,subsequently

StatusofSystemsacross

Region

IndividualSystems 3 LOW,initially,untilall

componentshavebeen

inspected

HIGH,subsequently

Consequencesofoutage

acrossRegion

WARN,with

informationfrom

individualsystems

4 LOW,initially,untilall

componentshavebeen

inspected

HIGH,subsequently

StatusofSystemsacross

OutageArea

WARN,with

informationfrom

individualsystems

5 LOW,initially,untilall

componentshavebeen

inspected

14

Sector Initial Actions WaterSectorutilityactionsinresponsetoaBlackSkyeventwillvaryovertime,followingtheevent.

Theywillalsovaryoverindividualsystems,whichdiffermarkedlyinhowtheysource,treatand

distributewaterandwastewater.Generally,utilitieswillactivatetheirEmergencyResponsePlans,

beginningwithstartingemergencypowerunitsandcontinuetoprovidewater/wastewaterservices

accordingtopre-plannedpriorities.Whilemaintainingnormalservicesundercurtailedpowersupport,

systemandsub-systemoperatorswillconductasituationalassessment,sothattheycandetermine

whereandtowhatdegreetoreduceservicelevels.Effectivepre-planning,underavarietyofscenarios,

wouldgreatlyimprovetheoutcomeoftheseinitialactions.

Sector Initial Actions Matrix (V3)

HIGH,subsequently

Consequencesofoutage

acrossOutageArea

WARN,with

informationfrom

individualsystems

6 LOW,initially,untilall

componentshavebeen

inspected

HIGH,subsequently

Priority InitialAction Desired/RequiredOutcome

1 AssessCapabilitytoMaintainCriticalServiceLevelsfor

systemandsub-systems

Accuratesituationawareness

andestimateoftimeto

servicerestoration(ETS)

2 ContactElectricUtilitytodetermineDate/Timeof

Restoration

Situationawarenessand

determinationofwhetherto

gotoBlackSkyprotocol

3 AssessConsequencestoPopulation/AreaServed Maintain,orrestorein

minimaltime,serviceto

maximumnumberofpeople

4 SwitchtoBlackSkyProtocolsforOperations Maintainsustainablelevelof

services

5 SwitchtoEmergencyPower Maintainservicesatpre-

designatedlevels

6 RecallKeyPersonnel ProvideResponseand

Recoverycapability

7 EstablishSectorLiaisontoEmergencyOperationsCenters Sharesituationawareness

acrossLifelineSectors

8 CommunicateRecoveryStatustoEOC Sharesituationawareness

acrossLifelineSectorsand

provideinformationusedto

coordinaterecoveryactions

15

Internal Sector Requirements InBlackSkyoutages,eventhebest-preparedwaterandwastewatersystemswillbeunabletooperate

alloftheirwaterpressurezonesattheirtypicalqualitystandards,andtoprovidenormalwastewater

flowandtreatmentlevels,foraperiodofweekswithoutexternalpower.BlackSkyrequirementswill

typicallyincludehardwareresilienceinvestmentsandcomprehensiveoperationalplanningtoenable

thesesystemstooperateindegradedpressureandtreatmentmodes,forlongdurations,with

infrequentanduncertaindeliveriesofreplacementdieselfuelandtreatmentchemicals.

A. WATERANDWASTEWATERSYSTEM:RECOMMENDEDBLACKSKYHARDWAREANDFACILITYRESILIENCE

REQUIREMENTS

1. EmergencyPower

Waterandwastewaterutilitiesshouldassumethattheywillneedtheirownemergencypower

suppliesinordertofunctionduringaBlackSkyevent,andthatevenmultiplepowerfeedswill

proveinadequatetohelpthemsustainservice.

DuringemergencypowerplanningforBlackSkywaterandwastewatersystemoperations,

plannersshouldemployhydraulicmodelingtoolsandavarietyofscenariostodeterminethe

mosteffectiveallocationofemergencypowerunits,andthendevelopoperationsplansto

execute,accordingtotheemergencypowerunitallocation.

Emergencypowerconsiderationsincludeacquisitionofsufficientinventoriesofemergency

generatorstosupportwatertreatmentandpumpingrequirementsforadesignatedportionof

theareaserved,andincluderequirementsforsparegenerators,toreplacethosethatbecome

inoperable.Whengeneratorsarenewlyacquired,procurementspecificationsshouldcallfor:

long-durationoperation,EMPandCyberprotection,provisionsformobility,andcapabilitiesto

enableeithercontinuous,shortdurationorperiodicoperation.

a. Planforadequate,onsiteemergencygeneration

“Justintime”strategiesforacquiringemergencypowerwillneedthoroughrevampingand

reassessment.WaterSectorutilitiesshouldbuildstrategiestoaddresstheenormous

mismatchthatwillemergeinBlackSkyeventsbetweenthenumberofwaterand

wastewatersystemsrequestingjust-intimeinstallationofemergencygenerators,available

generatorsuppliesandinstallationtechniciansfamiliarwiththeinstallationrequirements

specifictothatsite.

Utilitiesshouldplantoacquire,ataminimum,generatorsforwatersupplyandminimal

treatmentrequirements,forliftstations,andtoensureatleastonegeneratorisavailable

forpumpingtoeachpressurezonetomaintainthetargetservicelevel,asdeterminedby

theutility.Asstatedabove,pre-planningandfundingrestraintsmightdisclosetheneedto

shutdownoneormorepressurezones,andtoplanformigrationofthepopulationto

anothersupportablelocation.

b. Preparationsandinformationneededtosupport(limited)emergencymanagement

agencygeneratoracquisition

Whilemajorwaterutilities–particularlythoseservinglargecities,shouldworkwith

regulatorybodiestoensureadequateemergencygeneratorsareinstalledandavailable

onsite,manysmallerutilitiesmaybeunabletoacquiresufficientgenerators.Ifsuchutilities

16

anticipaterequestingreal-timedeliveryandinstallationofemergencygeneratorsduringa

BlackSkyoutage,prearrangedconnectionsfortheappropriatefacilitieswillbeessential.

RegistrationofneedsintheUSArmyCorpsofEngineers’EmergencyPowerFacility

AssessmentTool(EPFAT)ishighlyrecommended.

UntilbetterdataexistsonWaterSectorutilities’emergencypowerplansandcapabilities,it

willbeimpossibleforemergencymanagementagenciestodevelopstrategiestomitigate

shortagesingeneratorsuppliesandinstallationtechnicians.WaterSectorutilities,

associationsandtheirpublicsectorpartners(includingtheEPA)shouldconductsurveysto

determinetheexistingversusrequiredemergencygeneratorinventoryofU.S.waterand

wastewatersystemsintheUnitedStates.Informationwillberequiredontheavailableand

requiredgeneratorinventory–numbersandsizesofgenerators–infourcategories:

1. Utilitiesthathaveadequateon-siteemergencypowergeneratorsandfuelstorage

tanks,orplantoacquirethem,includinghavinggeneratorinventoryinformation

(numbersandsizes)onhand.

2. Utilitieswithliftsorotherfacilitiesrequiringemergencygeneration,preconfigured

forgeneratorconnection.

3. Utilitieswithneithergeneratorsnorpreconfiguredconnectionpoints.

4. Utilitieswithamixofpumps,liftsandfacilitiesrequiringgenerators–somewith

generators,somepreconfigured,somewithneither.

Anassessmentshouldbemadeofthisdatatoidentifypotentialshortfalls,bylocation,and

toidentifypotentialpointsofcompetitionforresources,especiallysinceaBlackSkyevent

willgeneratewidespreaddemandacrossalllifelinesectors,andsuppliersgenerallydonot

stockenoughassetstocoverpeakdemand.

Inaddition,acomprehensiveassessmentshouldbemadeacrossallregulatorybodyrulesto

determinehowoperationsplanswillbeimpacted,preparingforandoperatingunderaBlack

Skyenvironment.Primeareasinclude:full-loadtesting,transportationrestrictions,access

restrictions.

Utilitieswithhydraulicmodelingcapabilitiescanusethesetoolstomodel‘whatif’scenarios

thatvarybythenumberandlocationofemergencypowergeneratorsacrosstheirsystem.

Runninganumberofscenarioswithonlypartialemergencypowercoveragewill:enable

optimalpre-placementoflimitedresources;familiarizepersonnelwithpotential

vulnerabilitiesforresilienceinvestment;andpromoteadaptabilitybycreatingavarietyof

operationalcontingenciesthatcouldbeusedtocutdowndecisiontimeduringresponseand

recovery.

c. Generatorrequirementsshouldtakeintoaccountsparingrequirements

Mostbackuppowerunitsarenotdesignedtoruncontinuously,underload.Utilitiesshould

developdynamic(versusstatic)assessmentsoftheirgeneratorrequirements,andaccount

forlikelyreplacementrequirementsovertime.Watersystemsthatexperiencedgenerator

failuresduringSandy,forexample,haveincreasedtheirstockpilesofreplacement

generatorssincethatevent.Otherutilitiesshouldconsideradoptingasimilarapproach.

17

Whencontemplatingtheacquisitionofspares,utilitiesshouldevaluatethealternativeof

procuringamoreexpensivegeneratorcapableofcontinuousoperation.

d. Procurementguidelinesforemergencygenerators

Owners/Operatorsofsomefacilitieswithoutemergencybackupgeneratorsmaynotknow

exactlywhattheiremergencybackuppowergeneratorrequirementsare.Key

determinationsinclude:5

• Generatorsizerequired(inkWorMW)

• Doesthegeneratorneedtoprovidefullload,orpartialload?Ifpartial,itis

importanttolockoutotherbreakerssoasnottooverloadthegenerator.

• GeneratorOutputVoltageRequired(e.g.,480,120/208,4160)

• GeneratorPhaseOutput(singleorthree-phase)

• FacilityElectricalConnectionPoint:Isthereanexistingmanualtransferswitch?Will

thegeneratorconnecttothemainelectricalpanel,oranotherconnectionpoint?

Cantheconnectionpointbepre-installed?

• BasicOrderofMaterials(BOM)requiredtophysicallyconnectthegenerator:

Cable/Conductor(size,length,numberofruns);ConnectionLugs;GroundingPoint

(existingorneedstobeinstalled?);GroundingCablesizeandlength

• CRITICAL:Facility’sCommercialPowerGridIsolationPointLocation(importantfor

safetyofanyelectricallinemenoffsite,andalsotoensurebackuppoweris

dedicatedtothefacility)

• BlackSky-compatibleenhancementsthatneedtoberequestedwhenprocuring

units:Designedforlongduration,continuousoperation,withexpandedfueltank,

anddesignedwithenhancedEMPandcyberprotection.

UtilitiesshouldassessandregistertheiremergencypowerrequirementsbyusingtheCorps

ofEngineers’EmergencyPowerFacilityAssessmentTool:http://epfat.swf.usace.army.mil/

e. Specialoptionsforlowpowerequipment

Fortelemetrysystemsandotherlower-powerconsumptioncomponents,someutilities

havebegunusingsolarpowerandbatterybackupsystems.TheuseofsensorsinWater

SectorutilitiesisexpandingveryrapidlyandEISCouncilisresearching

sensor/communicationscapabilitiesthataffordBlackSky-securedsituationawareness,atan

affordablecost.

f. Planningforemergencypowertoensurewastewaterthroughput

Waterandwastewaterutilities--whetherseparateorcombined--shouldcoordinate

regardingbothrequiredflowsandthenecessaryemergencypowersupport,toensurethat

minimalwastewaterflowsaremaintainedforsewagedisposal.

g. Power-conservativewastewatertreatmentoptions

Whereutilitiesuseaeratedsludgeoperationsforwastewatertreatment,thisapproach

typicallyconsumesmorethan75%ofwastewaterutilitypowerrequirements.Inaddition,

5InformationprovidedbytheU.S.ArmyCorpsofEngineers.

18

withoutpowerforaeration,withina48-72hourwindow,microbesforsewagedigestionwill

die,andre-establishmenttimesareaweekorlonger.Workingwiththeirrateregulators,

utilitiesshouldevaluateandpotentiallyimplementalternative,lesspower-intensive

wastewatertreatmentmethods,ifavailable.

h. Minimalsetofnecessarychemicals

Significantopportunitiesexistforwaterandwastewaterutilitiestodownsizetheirchemical

storageandresupplyrequirementsforBlackSkyoperations

Watersystemsmaintainsuppliesofchemicalsandother“consumable”suppliesforwater

treatmentthatareprimarilyusedtomeetregulatoryrequirementsfordisinfectantresidual

infinishedwaterandtoachievegoalsforcolor,odor,taste,andothercharacteristicsthat

aremostlyaesthetic(andnotenforcedbyregulation),ratherthannecessaryforpreventing

seriousillness.AchievingtheseaestheticgoalswillbecomelessurgentinBlackSkyoutages.

Instead,sustainingadequatequantitiesofwaterthatmeetbareminimum,health-related

requirements–orthatcanbetreatedbycustomerstoachievethatlevelofquality–willbe

ofparamountimportanceinsuchevents.

AswaterutilitiesdetermineBlackSkyservicelevelswithreducedqualityofwaterthey

deliver,theyshouldstringentlyprioritizethechemicalsthattheyneedtostoreandresupply.

Suchprioritizationwillneedtobedevelopedinagreementwithappropriateregulatory

entitiestoprovidevariancesfromchronichealthstandardswhileleavingthenecessary

acutehealthstandardsinplace.6SeeSection3.B.below.

i. Microgrids:Operatingwithinamicrogrid–asmallsectionoftheelectricgridthatcanbe

electricallyisolatedfromthelargerpowergrid–isanadvancedtechniquebeingpursuedby

ahandfulofwaterutilities,spurredbytherapidandextensiveprogressbeingmadewith

useofrenewables.

• Majorwastewatersystemshaveespeciallypromisingopportunitiestoestablish

microgridsforBlackSkyoperations.Anaerobicsludgedigestioncangenerate–via

capturedmethaneemissions–enoughpowertooperateawastewaterfacility,and

evengivebackexcesspower.However,fortheutilitytooperateasafunctional

microgrid,therequisiteelectricalinfrastructuremustalsobeinstalled.As

wastewaterutilitiesconsiderthedevelopmentofCombinedHeatandPower(CHP)

systems,utilityleaders,electedofficials,andotherresiliencestakeholdersshould

carefullyconsideropportunitiestoprovideformicrogridoperations,andbalance

theincrementalcostsofdoingfullCHPagainstthepotentialbenefitsforsustaining

wastewaterserviceinlongdurationblackouts.

• Overthelongerterm,microgrids(orsecurepowerenclaves)canprovideaholistic

meansofstrengtheningwaterandcross-sectorresilience.Whileelectricutilities

willneedtomakemanyoftheinvestmentsnecessarytocreatemicrogrids,water

andwastewatersystemsshouldhelpleadcross-sectordiscussionontheirdesignto

ensurethatminimalistwaterandwastewaterservicegoalscanbesustainedinBlack

Skyevents.

6SuchwasthecaseduringtheaftermathofHurricaneKatrina,wherethefocuswasonsafeguardingagainstacutemicrobialriskwhileobtainingavariancefromchronic70-yearlifetimeexposurestandardstoarsenicinwater.

19

2. EmergencyFuelTankage

Increasedonsiteemergencyfueltankage,orprovisionsforonsiteornearbystorageand

managementoftankertrucks,isneeded.Thisisafundamentalrequirementtoallowoperations

tocontinuedespiteinfrequentanduncertaindeliveriesofdieselfuel.Waterutilitiesshouldwork

withemergencymanagementagenciesandotherwaterutilities,todevelopregionalplans

(coordinatedwithfuelstorageanddeliverycompaniesforBlackSkyconditions)toensurethe

deliveryfrequenciesandvolumestheyanticipatewillberequiredinaBlackSkyscenarioaremet.

Asadefaultplan,wheresuchplansarenotyetavailable,sufficientfuelfor10daysofoperation

isrecommended(i.e.,approximatelyone9000gal.tankertruckcapacityforfullloadoperationof

a0.5megawatt(MW)generatorfor10days).UnderBlackSkyconditions,backuppower

generatorswillberunningunderfullload,whichwilldramaticallyincreasefuelrequirementsand

outstriptypicallevelsofon-sitefuelstorage.Note:Regulatoryrequirementswillinmostcases,

needtobeaddressedtoallowforadditionalfueltankage.Manyof thewatersectorutilities’BlackSkyresilienceinvestmentsandoperationalplanningwillrequireadditionalfundingthat

mustbeapprovedbyutilityboardsofdirectorsorgovernmentofficials.Otherinvestments,

includingthoseinwaterandwastewatertreatment,andinpowergeneratorsandfuelstorage,

mayrequireregulatorypolicydevelopmentandassociatedpre-orpost-outagewaiversofSafe

DrinkingWaterorCleanAirActstandardsandotherregulatorypolicychanges.Developingnew

approachestoaccountfortradeoffsbetweenBlackSkyresilienceandregulatoryobjectiveswill

beessentialforprogress.

3. Implementationapproachesandconsiderations:

a. Centralizedfuelstorage,oruseofpurchasedfueldeliverytrucks

Supportingalandscapewithmultiple,scatteredbackupgeneratorsandassociatedfuel

storagefacilitiespresentsmajorchallengesforexpandingfuelstorage,andforconducting

resupplyoperationsoveradisruptedtransportationnetwork.Anumberofwaterutilities

arecentralizingtheirstoragefacilitiestomeetthefuelneedsoflargenumbersof

generators,andarepurchasingorcontractingforfueldeliverytrucks.Contracting

arrangementsneedpre-coordinationwithemergencymanagementagencies,however,

sincetheymightcreateamajorcompetitionforresources,especiallyproblematicif

discoveredonceaBlackSkyeventtakesplace.

Theseoptionsshouldbeexaminedsector-wide,withtheaspirationalgoalofhavinganon-

site10-to30-dayfuelsupply,aswellasmaintainingarrangementswithmultiplevendorsfor

fuelresupply.Furthermore,utilitiesshouldtracetheirfuelsupplychainevenasfarbackas

therefineryorportandidentifythevulnerabilitiesthroughoutthesupplychain.

b. Fuelstoragepartnerships

FuelstorageandtransportationaremajorchallengesinmitigatingBlackSkyhazards.In

ordertoreducethenumberofstorageunitsandresupplytripsanddestinations,

partnerships(forlocaldeploymentofadequatefuel)withothercriticalusers/sectors

(emergencymanagement,hospitals,etc.)maypresentidealopportunitiesforfacilitysharing

andmutual-assistanceagreementsonfuelstorageandresupply.

20

c. Fuelsystemmaintenance

Acommonproblemfordiesel-rungeneratorsisthatdieselsludgewillbuildupovertime.

Fueltanksandlinesmustberoutinelycleanedtwiceperyear,andfuelfiltersmonitoredand

changedasneeded.Filtersmustbedraineddaily,especiallyincoldclimates,topreclude

waterbuildupandfreezing.

d. Alternativefuels

UnderemergencysituationssuchasthosefollowingaBlackSkyevent,itmaybenecessary

tousealternativefuelstooperateemergencypowergenerators.Oneapproachisto

acquiremulti-fuelbackuppowergenerators,buttheirpurchasecostishigher.Another

approachistouseaviation-gradefuel--whichwillpotentiallybemoreavailableinaBlack

Skyenvironment—butonlyafterconsultationwiththemanufacturer,distributororleasing

agent.TheuseofaviationfuelasanalternatefuelsourceisbeinginvestigatedbytheEIS

CouncilandthefindingswillbeincorporatedinasubsequentversionofthisPlaybook.

4. Facilitypreparationsforreducedtreatmentlevels:Waterandwastewatersystemsshouldbe

preconfiguredtoallowforswitchovertoBlackSkytreatmentlevels,designedtoassure

designatedtreatmentqualitywhileminimizingelectricityandconsumablesrequirements.Each

utilitywillhavetoestablishlevelsspecifictotheirsystem,conducttheassociatedplanningand

coordinationanddeveloptheconceptofoperationsforreducedlevels,andgaintheapprovalof

theappropriateregulatoryagencies,fortheseBlackSkyservicelevels.

5. Spares:Utilitiesmustplanforandmakeinvestmentsingeographically-stagedsparesfor

vulnerable/frequentlyreplacedcomponents,includingSCADAcontrollers,valves,and

geographicallydistributedstoresofbasictools.Utilitiesmustmaintainawarenessofspareparts

availabilityandsupplychainvulnerabilities.

B. WATERSYSTEM:RECOMMENDEDBLACKSKYOPERATIONALPLANNINGREQUIREMENTS

Operationalplansforoptimizeduseofemergencyfueltoextendoperationsbetweenrefueling,for

operationatreducedwatertreatmentqualitylevels,andotherchangesarerequiredtosupport

BlackSkyServiceLevels.Tobeeffective,theseplansshouldbefullycoordinatedwithsupporting

sectorsandexercisedfrequently.

1. OperationalPlansforTransitiontoBlackSkyServiceLevels:Utilitieswillneedtodevelopoperationalplanstoimplementeachoftheservicelevelchangesthatmayberequiredduring

BlackSkyemergencies.Theseshouldinclude:

a. Transitiontoreduceddrinkingwatertreatmentoperations,whereprocessesforenhanced

waterquality(clarity,odor)aresuspendedduringBlackSkyoperations.

b. Transitiontolimitedzonalservice,toincludeshuttingdownpressurezonestokeepservice

tohigher-priorityormoreeasilyserviced(lowerelevation)zones.

c. Transitionfromfull-powertoreduced-orintermittent-poweroperationsfortreatmentand

pumpingthroughoutthe(possiblyreduced)servicearea.

d. Transitionto“FireSuppressionOnly”operations.

21

e. Transitiontorecoverablesystemshutdown.Forthemostdiresituations,utilitiesshould

examinemethodstoshutthesystemdownwhilemaintainingpressurewithinthesystemto

allowformorerapidpost-eventrecoverybylimitingthedamagethatafulllossofpressure

wouldentail.Thisplanningshouldaddresspotentialpipebreaksduetolossofpressureor

totransientsproducedduringrestart.

2. OperationalplansforBlackSkyService:OncethetransitiontoBlackSkyoperationsismade,

thoseoperationsmayneedtostayineffectforamonthormore.StandingBlackSkyOperating

Procedureswillneedtobeadoptedasthe“newnormal”duringtheemergency.

3. OperationalPlansforRecoverytoNormalServiceLevels:Rapidsystemrecoverywillbe

enhancedifrecoveryplansfromeachofthediminishedBlackSkyservicelevelsbacktonormal

levelsareplannedinadvance,toensurethatthesystemsorcomponentsarenotinadvertently

damagedoncepowerisrestored.Closecoordinationwithsupportingelectricutilitiesisrequired,

inordertominimizedisruptivestarts/stopsthatmayoccurduringblackstart/crankingpath

operations.

4. Training:TrainingisrequiredtomakecriticalpersonnelawareofallBlackSkyoperationalplans,

toaddresseachoftheBlackSkyserviceleveloptions.Trainingneedsshouldaddressnon-typical

areassuchasuseofemergencypowerandcommunicationsmeans,aswellasmaintaining

reducedwaterqualitylevels.

5. CriticalPersonnelandFamilySupport:Utilitieswillneedtosupplementtheirowncritical

personnelandfamilysupport,toassuretheirlaborforcewillbeavailable.Criticalskillsand

proficiencylevels,aswellasfamilysupportrequirements,needtobespecificallyaddressedand

provisionsplannedforsecuringthemintheeventofanextendedemergency.

a. Identificationofkeypersonnelisessential,butforBlackSkyemergencies,itisimportantto

recognizethatsomekeypersonnelmaynotbeavailableastheymaybecaughtupinthe

emergencythemselves.Contingencyplansareneededtoensurethatoperationscanbe

maintainedwithoutthefullcomplementofkeypersonnel.

b. Utilitiesmayneedtoconsiderinternalrelocationofkeypersonnel.

c. Utilitiesmayneedtodevelop,coordinateandexerciseemergencytransportationplansthat

minimizethenumberofvehiclesandthatusepre-designatedemergencyroutesusedto

accesstheirplaceofduty.Carpoolingandpre-planningtheuseof4-wheeldrivevehiclesare

bestpractices.

d. Pre-eventcoordinationwiththeRedCrossorotherdisasterresponseandsupport

organizationisrecommendedtoensurethatcriticalpersonnelandtheirfamiliesaresafe

andsupportedduringanevent.

6. Emergencydatabase:Developmentandmaintenanceofapower-secureemergencydatabase

thatprovideskeyelementsofdataaboutthewaterandwastewaterassets,isvitaltoresponse

andrestorationdecisionsthatfollowaBlackSkyevent.

7. Emergencycommunications:Planningforuseofanappropriateemergencycommunication

systemforinternalandexternalcommunicationanddata,anduseofasynergisticemergency

situationalawarenessanddecisionsupporttoolwillbeessentialtoaccommodatebothinternal

operationalplans,andcoordinationwithexternalsectors.Asanexample,utilitiesmayreview

22

andutilize,asabasisforsupportingtheseplans,theavailableBlackSkyCoordinationandControl

System(BSX)architecture,beingdevelopedforthispurpose.7

8. Rigorousgeneratorserviceandmaintenance:BlackSkyemergencymaintenanceprocedures.

Generatorsaregenerallydesignedforshort-durationemergencyoperations,notlong-term24by

7operation.Planningmustconsidermaintenanceandfailurefactors,andactionstaken

accordingly.Generatorsmustbeexercisedunderfullloadforafewhours,oradayatatime,

everymonth.Utilitiesshouldmaintainamasterlistoftheavailabilitystatusofallbackup

generatorsforuseinBlackSkyoperationalplanning.

9. Critical(External)personnel:Utilitiesmustplanforobtainingpersonnelhavingcriticalsubject

matterexpertiseneededtorestoreormaintainoperations,suchasgeneratormaintenanceand

repair,electricalhookup,specializedplantoperators,etc.Oneexampleistheconceptof

CertifiedPowerRecoveryEngineeringTeams8(CPR)teams.Inthecaseofwatersystems,these

wouldbeemergencypowertechnicians,watersystemcivilengineeringsupportteamsand

cyber/industrialcontrolsystemexperts.

C. WASTEWATERSYSTEM:RECOMMENDEDBLACKSKYOPERATIONALPLANNINGREQUIREMENTS

WastewaterservicemaybeevenmoreimportantinachievingthekeyBlackSkygoalofreducing

massmigrationthandrinkingwaterservice–especiallyinurbanareas.Recommendedoperational

planningrequirementsforwastewaterutilitiesmirrorthoseforthedrinkingwaterutilitieslisted

above.Utilitieswillneedtodevelopoperationalplanstoimplementeachoftheservicelevel

changesthatmayberequiredduringBlackSkyemergencies.Additionalconsiderationsincludethe

following:

1. Transitiontolesspower-intensivewastewatertreatmentoperations,suchasfiltrationrather

thanaeratedsludge,wherefeasible.

2. Transitiontoareducedsetofwastewatertreatmentchemicals.

3. Conductalternativeoperationsthatensurewastewaterflowsaremaintained.

Internal Sector Requirements Matrix

7EISCouncil,BlackSkyCoordinationandControlSystem(BSX):http://eiscouncil.org/Protection/ItemDetails/62

8ForadditionalinformationontheCPREngineeringTeamconcept,see:

http://eiscouncil.org/Protection/ItemDetails/63

Phase Priority Requirement

Water

Response 4 OperationalPlansforTransitiontoBlackSkyServiceLevels

Response 1 OperationalplansforBlackSkyService

Response 8 OperationalPlansforRecoverytoNormalServiceLevels

Mitigation 3 TrainingandExercises

Response 5 CriticalPersonnelandFamilySupport

Mitigation 7 Emergencydatabase

Response,

Recovery

2 Emergencycommunications

Response, 9 Rigorousgeneratorserviceandmaintenance

23

External and Cross Sector Dependencies Overview WaterSectorutilitieswillbeabletomeettheirmissionrequirementonlyiftheyhaveassuredsupport

fromothersectors,tosupplythoseservicesandresourcesthatlieoutsideitsnormalcapabilities.

Definingthesector’srequestedexternalrequirement–andcoordinationwiththeappropriatesectors–

isacrucialpartofdevelopmentoftheBlackSkyProtocol.

A. Government–Non-

RegulatoryAuthoritiesandEmergencyManagers

1. NationalPlanningforEmergencyPower:Prioritizationofemployment,emergencyfuel

deliveries,replacementgeneratorsandgeneratortechsupportwillberequired,andappropriate

managementandcoordinationprocessesassociatedwithfederalandstateemergency

managementagenciesandothergovernmentstakeholderorganizationswillbeessential.TheEIS

Council’sNationalEmergencyPowerCouncil(NEPC)Initiative9providesanexampleofsuch

capability.

2. NationalPlanningforEmergencyUtilityConsumables:Areducedsuiteoftreatmentchemicals

willbeneededfordrinkingwaterandwastewatertreatmentatlevelsrequiredforminimal

health-basedquality.Foranextendedoutage,withnormaloperationofchemicalsupplyand

transportationseverelydisrupted,deliveryofsuchchemicalswilltakeplaceonlyifstateand

federalgovernmentagenciespreplananorganizedprocesstoassureeffectiveexecution.The

NationalEmergencyUtilityConsumablesCouncilInitiative10providesoneexampleofhowsucha

structuremightbeformulated.Fromthewaterutilityperspective,itwillbeimportanttoensure

thisrequirementisclearlyarticulatedasasectorexternalrequirement,tohelpgovernment

agenciesunderstandtherequirementthatsuchaprocesswillneedtoaddress.Key

considerationsinclude:

a. Typicalon-sitechlorinestorageis1–2weeksofneededsupply.Increasingon-sitestorage

however,posesmanyspaceandregulatorychallenges,whichneedtobeaddressedinBlack

Skyplanning.

9Emergencyfueldeliveries,replacementgeneratorsandgeneratortechsupportwillberequiredfromtheemergingNational

EmergencyPowerCouncilorequivalentorganization,onaschedulethatisnegotiatedbetweentherelevantsectors.See:

http://eiscouncil.org/Protection/ItemDetails/64.10Aminimalsetoftreatmentchemicalswillbeneeded,fromtheemergingNationalEmergencyUtilityConsumablesCouncilor

equivalentorganization,providingapre-negotiatedsetofchemicalsandquantities.See:

http://eiscouncil.org/Protection/ItemDetails/65.

Recovery

Mitigation,

Response,

Recovery

6 Critical(External)personnelsuchasgeneratormaintenancepersonneland

cyberexperts

Wastewater

Response 2 Transitiontolesspower-intensivewastewatertreatmentoperations

Response 3 Transitiontoareducedsetofwastewatertreatmentchemicals

Response 1 Conductalternativeoperationsthatensurewastewaterflowsare

maintained

24

b. Onsitechlorinegenerationgreatlyincreaseselectricpowerneeds,soitisnotagoodoption

forBlackSkyoperation.

c. Hypochloritebleachtypicallyrequires12timesthestorage/resupplyrequirementcompared

togaseouschlorineandmaythereforebeimpracticalwhenspaceislimited.

d. Onlyahandful(~5)facilitiesnowproducechlorineproductsintheU.S.Conductingregion

orindustry-widediscussionswiththesesuppliersisimperativeforBlackSkyplanning.

e. Forjointwater/wastewaterutilities,itmaybepossibletosacrificechlorineuseonthe

wastewatersidetoextendchlorinefordrinkingwatertreatment.Utilitiesneedto

understandandplanforthetradeoff,andcoordinatewiththeirwaterorwastewater

counterpart,ifthesefunctionsareperformedbyseparateutilities.

f. Otheressentialchemicalssuchascoagulantsmustbeconsideredinresupplyandstorage

planning.

g. CertifiedchemicaloperatorsandwaterqualityanalystsmaybeinshortsupplyinaBlackSky

emergency,andmustbeplannedfor.

3. CommunityDisasterPreparation:Mediaoutreach,communityforumsandotherformsof

advancecommunitypreparationforlongdurationoutageswillbeessentialtosuccessful

implementationofmanylifelineutilityBlackSkyplans.Forexample,governmentagencies,

utilitiesandmass-careNGOswillneedtoplayapivotalroleineducatingthepublicabout

preparingforhome-basedwaterdisinfection--byuseofchlorinebleachoriodinetablets,for

example--intheeventofsevere,longdurationoutages.

Similarly,governmentagencieswillneedtoprovideadvanceinformationtocommunitiesabout

boththecriticalityofreducingwaterconsumptioninsuchdisasters,andprovidebestpractices

fordoingso.TheEnvironmentalProtectionAgencyrecentlylaunchedtheCommunity-Based

WaterResiliency(CBWR)Tooltohelpfacilitatepartnershipsbetweenwaterutilitiesandthe

communitiestheyserve.11Finally,utilitieswillneedtocoordinatewithemergencymanagersand

otherresiliencestakeholdersregardingtheutility’srestorationplans,priorities,andtimelinesfor

limitedorrotatingservicewithinservicezones.

4. CommunityShelters:Preparationswillberequiredfromgovernmentagencies,incoordination

withtheNationalGuardandmasscareNGOs,tomovepeopletosheltersfromareas

predesignatedbyalocalwatercompanyasan“unserved”location,intheeventofaBlackSky

hazard.Governmentagencies,workingwithwaterandpowercompaniesandwithmasscare

NGOs,shouldpre-planshelterlocationsinareaswhereelectricityserviceandwaterservicewill

bemosteasilysustainedorrestoredfollowingaBlackSkyevent.

5. Limitedemergencywatersupplysupport–AlternativeHydro-Systems:Whilebottledwater

andotherformsofemergencywatersupplywillnotbeavailabletoreplacewaterservicefora

majorcity,governmentagenciesshouldplantoprovidelimitedemergencywatersupportfor

localuse,bothinsmallcommunitieswithoutothermeanstoobtainwater,andforlocalusein

unpowered“pockets”thatremainwithoutpowerandwater,astheoverallpowerrestoration

processproceedsintheaftermathofaBlackSkyevent.

11https://www.epa.gov/communitywaterresilience

25

Analternativedistributionsystem(hydro-systems,includingequipmentandoperations)couldbe

modeledafterIsrael’sconceptofoperationsforfurnishingpotablewaterduringwatercrises.

Theseemergencysystems,properlycoordinatedwithemergencymanagersandlocal

governmentofficials,wouldbedesigned,configured,operatedandmaintainedtoprovide

potablewaterunderNoWaterServiceorFireandSanitationServicesOnlyconditions.12

6. SecurityandTransportationSupport:Provisionswillneedtobemadeforfacilityandpersonnel

security,aswellasfortransportation,indisrupted,BlackSkyenvironments.Furthermore,if

securityisnotaddressed,unauthorizedtamperingwithhydrantsandstandpipescouldleadto

criticallosssupplyand/orpressure.Plansshouldincludecoordinationwithemergencymanagers

andtheNationalGuardtoassureaccessbycriticalrestorationpersonnelandtoprotectthe

distributionsystemfromtampering.

7. NationalSecurityConsiderations:Waterandwastewaterserviceisessentialfornational

security.Civiliancriticalinfrastructure–especiallyinfrastructurethatdirectlysupportsmilitary

installationsorcommandandcontrolelements–isapotentialtargetforanyadversary.Water

SectorutilityleadersshouldpartnerwithFederalandstateagenciestoaddressresilience

initiativeswithregardtonationalsecurity-relatedpriorities.Thiswillnotonlyenhanceresilience

ofthewaterutilityitself,butenhancedeterrenceontwolevels:

a. Activedeterrence--ournationalsecurityapparatuswillremainfunctional,ensuring

appropriatepoliticalandmilitaryresponse;and

b. Deterrencebydenial–reducingvulnerabilitytoattackandtheconsequencesthatanattack

wouldyield;adversarieswouldfindutilitiesalessdesirabletargetandbelesslikelytoattack

inthefirstplace.

B. Government–

RegulatoryBodies

RegulatoryPolicyAdaptations:Utilitieswillneedregulatorychangestorecognizetheneedsfor

BlackSkyresilienceinvestmentandoperations,inordertoprocurerequiredequipmentandbuild

appropriateplanstoimplementtheInternalRequirementsreflectedinthisBlackSkyPlaybook.

Thesewouldincludechangesinwaterpressure,waterquality,wastewatertreatment,airquality

(whenusingemergencybackupgenerators)andotherrelevantfeaturesofplanningforBlackSky

operation.Provisionsmayalsobeneededtoincreaserateschargedtocustomers,whensignificant

additionalinvestmentisrequiredtoimprovesystemresilience.Pointstobeconsideredarelisted

belowandexplainedindetailinAnnexB.

1. RegulatoryMechanisms:UtilitiesshouldlooktotheirAssociationstoidentifyandunderstand

regulatoryrequirementsthatmayneedtobechangedorthatrequirevariances,duetoBlackSky

BlackSkyoperations.

2. EmergencyStandards:Authorizationtomeetonlyacute–butnotchronic–waterquality

standardsduringBlackSkyemergencies.

12AbroaderstrategyforsuchactionisdefinedinPlanningforanEmergencyWaterSupply,issuedbyEPAand

AWWAin2011.

http://www.awwa.org/Portals/0/files/resources/water%20knowledge/rc%20emergency%20prep/Emergencywate

r.PDF

26

3. BlackSkyServiceLevelApproval:BlackSkyservicelevelswillrequireregulatorybodypolicydecisionsandcorrespondingruleadaptations,basedoncoordinatedreviewwiththeutilities

involved.Tofacilitatesuchcoordination,utilitiesarerecommendedtodevelopaBlackSky

CONOPSorPowerOutageAnnexthataddressesBlackSkyprotocols.

4. AirQualityStandards:Airqualityandotherregulatorystandardscreatechallengesforexpandedstorageoffuelandchemicals.Improvedairqualityisanimportantgoal.However,giventhe

absolutelyvitalrolethattheWaterSectorutilitywillplayinsavinglivesduringaBlackSkyevent,

sectorleadersshouldengagewithregulatorstoexplicitlyexaminethetradeoffsbetween

improvedsectorresilience(throughexpandedstorageandotherinfrastructureinvestments)and

environmentalregulations.AttheFederallevel,theEPAisboththeFederalsector-specific

agencyresponsiblefortheWaterSector,andistheFederalleadformakingandenforcing

environmentalregulations.BringingthosetwoportionsoftheAgencytoconductsuchatradeoff

analysisbetweensectorresilienceandemissionsgoalscouldprovideamodelfordoingsoat

stateandlocallevelsaswell.Limitedenvironmentalqualitywaiversshouldbedevelopedfor

emergencygeneratoroperation,torecognizetheneedsforongoingmaintenanceuse,aswellas

preplannedcontinuoususeinsevere,extendedpoweroutages

5. FuelStorage:Whileitisestimatedthat75%ofwaterandwastewaterfacilitieshaveinstalled

emergencypowergenerators,onsiteemergencydieselfuelstoragegenerallydoesnotexceed

suppliesadequatefor48to72hours.InaBlackSkyevent,whereelectricpoweroutagesare

expectedtolastweeksorlonger,backupfuelwillbedepletedquicklyandfuelresupplywillbe

extremelychallenging,anddeliveriesinfrequent.Regulatorybodiesshoulddeveloppoliciesthat

permitandencourageexpansionoftheseemergencyfuelreserves,eitheronsiteatwater

utilities,orstoredinaccessiblelocationswhereitcanalsobesharedwithotherlifelinesectors

anddisasterresponders.

6. ChemicalStorage:Atleastaspressingastheneedforadequateonsiteemergencyfuelstorage,

adequateavailabilityofchemicalsneededtotreatwaterandwastewaterwillbeessential.While

thesechemicalsareroutinelyresuppliedatthesefacilitiesundernormalconditions,availability

anddeliverywillbedrasticallyreducedinseverelydisrupted,BlackSkyscenarios.

TheU.S.EPAandresponsiblestatedepartments,incoordinationwithutilities,shouldexamine

optionsforexpandedonsitestorageofthesubsetoftreatmentchemicalsthatcorrespondto

designated,BlackSkytreatmentlevels,todevelopcorrespondingnewregulatorypolicyforsuch

chemicalstorage.

7. FuelandChemicalTransportation:Deliveryoffuelandwatertreatmentchemicalsisoftena

challengeforutilitiesandtheirregulators,duetoHazMatandothertransportationregulations.

Whilesuchregulationsarenecessarytolimitrisksandprotectpublicsafety,underBlackSky

conditions,thecontinuedfunctioningoflifelineinfrastructureswillbecomeacrucialrequirement

forsavingandsustaininglives.Unlesstreatmentchemicalsareaddressedasapriorityitemtobe

allowedin--particularlyinlargecities---ahealthandsafetycatastrophecouldresult.However,

advanceplanningforrequiredtransportationofthesechemicals–whetherbyutilitiesorby

governmentemergencymanagementagencies–cantakeplaceonlyifsuchtransportation

requirementsareembodiedinprearrangedregulatorypolicyadaptationsandrecognizedbythe

regulatoryagencyresponsible.

27

TheU.S.EPA,DepartmentofTransportation,andstateenvironmentaldepartmentsandother

regulatorybodiesshoulddeveloppoliciesfortransportationofessentialchemicalsunderBlack

Skyconditions,asanessentialframeworkforusebyothergovernmentagenciesandutilitiesin

theirplanningforsustainedwaterandwastewatersystemoperationinBlackSkyscenarios.

8. EmergencyGeneratorProcurement:Waterandwastewaterutilities,incoordinationwith

regulatorycommissions,shouldexaminetheirexistingbackuppowercapability.Forthose

utilitiesthatdonotcurrentlyhavesufficientemergencygenerationtoallowforsustained

operationatBlackSkyservicelevelsthataredesignatedforthatutility,theU.S.EPAshould

developpoliciesthatprovideforcostrecoverytoacquire,installandmaintaintheminimum

additionalgeneratorinventory.

Asanintermediatestep,infrastructureinvestmentinelectricalconnectionsandmanualtransfer

switcheswillallowforconnectionof“trucked-in”emergencygeneratorsduringseverehazards.

Thissolutionisnotideal,sinceavailabilityofsuchunitsisanticipatedtobeverylimited.

Nevertheless,itdoestakeanessentialsteptowardlateracquisitionandinstallationofdedicated

emergencygenerators.

9. Microgrids:Opportunitiesexistforsomewastewaterutilitiesthatuseanaerobicsludge

treatmentprocessestoself-power,reclaimingenergythroughtheuseofcombinedheatand

power(CHP)units,fueledbycapturedmethanefromthewastewatertreatmentprocess.Sucha

processisnotonlyenvironmentallyfriendlyandreducesenergycostsforthewastewaterutility

duringnormaloperations,butcanalsobeapowerfuldriverforsystemresilienceiftheadditional

investmentinelectricalinfrastructureismadetoallowtheutilitytofunctionasapowerislandor

microgrid.

C. MassCareNGOs

1. CriticalPersonnelandFamilySupport:Utilitieswillneedtosupplementtheirowncritical

personnelandprovidefamilysupport,toassuretheirlaborforcewillbeavailable.Masscare

NGOscanplayacrucialroleinthisarea,andwillrequiresuchinformationfromwaterutilitiesto

enablethemtoplanandtraintoprovidesuchsupport.

2. CommunityCoordination:MasscareNGOsshouldbeaskedtohelpcommunicatewiththe

public,tonotifythemofwatersafetyandusagelimitations,includingbothpre-eventandpost-

eventcoordination.NGOscanalsoplayanimportantroleinbothpreparingcommunitiesfor

waterconservationinsuchemergencies,andinreal-timecommunicationoftheurgencyofsuch

conservationduringamajoroutageemergency.

OneexampleisthatMassCareNGOscouldbetaskedwithdistributionofiodinetablets(pre-

eventpreferred),withinstructionalmaterialontheiruseforhomewaterpurification.

Finally,aswithemergencymanagers,utilitieswillneedtocoordinatewithmasscareNGOsand

otherresiliencestakeholdersontheutility’srestorationplans,priorities,andtimelinesforlimited

orrotatingservicewithinservicezones.

3. Shelters:Withwaterutilitiesbuildingplansforgeographicservicelimitations,NGOswillbean

importantpartofplanningtorelocatepeoplefrompredesignated,non-servedareas.Thisneed

shouldbeidentifiedbylocalutilitiesasanimportantexternalrequirementandcoordinatedwith

28

NGOs.Sheltersshouldbepreplanned,locatedinareaswhereelectricityserviceandwater

servicewillbemosteasilysustainedorrestoredinBlackSkyemergencies.Incollaborationwith

emergencymanagersandNGOs,utilitiesshoulddevelopplansformassevacuationstoshelters

havingassuredwaterandwastewaterservices.

D. Electricity

Subsector

1. Prioritizedpowerrestoration:Prioritizedpowerrestorationwillbeessential,tominimizethe

durationofoperationofemergencygeneratorsandrelatedequipment.TheWaterSector

utilitiesmustidentifyprioritizedrequirementsandcollaboratewithpowerproviderstoensure

theseprioritiesareunderstood,addressed,andsynchronized.Currently,mostpowerutilities

prioritizepowerrestorationona“customersserved”basis.Eventhoughwaterutilitiesareavery

largepowerconsumer,theyareoftenviewedbypowercompaniesas“justonecustomer(or

meter).”Furthermore,theyaretypicallylocatedoutsideofthemainpopulationcenters,and

oftenonanelectricalspurline,whichlowerstheirrestorationpriority.WaterSectorutilities

shouldcoordinatewithelectricpowerutilitiesandemergencymanagerstoensurethey

understandthatreturningpowertothewaterandwastewaterutilitiesmustbeapriority,as

waterservicetohomesandbusinesseswillallowthepopulationto“shelterinplace”untilpower

isrestored.

2. Earlypowerrestoration:Insomecases,prioritizednear-immediaterestoration(ifavailable)will

beneededforutilitieswhoseemergencygeneratorsfailorarenotoperable,andcannotfind

adequatespares.Waterandwastewatersystemsalsoprovidealargeloadthatcouldpotentially

beleveragedaspartofthesupportingelectricutility’sBlackStart/CrankingPathConceptof

Operations.Bothoftheseopportunitiesshouldbemutuallyplannedbytheelectricandwater

utilities.

External and Cross Sector Requirements Matrix Requirement

Area

Priority Requirement

Manpower 5 Maintainingallcriticalskillsneededtooperatethesystemat

sustainableminimallevels.ContinuallyimproveWARN.

Transportation 7 Assesstransportrequirementsforallsupportingsupplychains,

extendingbackthroughmainsupplierstotheirsuppliers.Factorinto

planningandmaintainsituationawarenessofprimaryandalternate

meansoftransportationsupportingcriticalsupplychains

EmergencyPower 6 Identifyandmaintainalistofrequiredbackuppowerunits,

includingreplacementsandmaintenancerequirements,provide

thoserequirementstoemergencyplannersandestablishthe

requiredconnectionpointsandoperatingprocedurestoreceive,

connect,operateandmaintainemergencypowerunits.(Consideran

EmergencyPowerPlanasanannextotheEmergencyResponsePlan

thatprovidesacomprehensiveplanofoperations)

Security 1 Providesecurityforfacilities(includingdistributionsystem)and

personnel.Whilepartofthedistributionsystem,hydrantsand

29

Requirement

Area

Priority Requirement

standpipesdeservespecialsecurityplanningtoavoidlossassociated

withunauthorizedaccess

Communications 2 Capabilitytocommunicatewithkeysubsystemstoreceivesituation

reportsandtoeffectcontrolofthesystem.Thisincludesmaintaining

SCADAcapabilities,butataminimum,oralcommunicationswithkey

personnel.

Information

Sharing

3 Capabilityforwaterandwastewaterutilitymanagerstoobtainfrom,

andprovideinformationtoWaterISACaboutpotentialthreatsand

riskstothenation’swaterinfrastructurefromallhazards.This

includesaccessthroughWaterISACtootherISACssuchasE-ISACand

FS-ISAC.

Communications

(Physical)

4 BlackSky-securecommunicationscapableofextendingtoallinternal

componentsandsubsystems.

Fuel 8 Resupplyoffuelnecessarytomaintainwater/wastewater

operationsatdesignatedBlackSkylevelofservice

Sector Specialized Resource Requirements Overview Waterandwastewatertreatmentrequiresthefollowingtypesofspecializedresourcestosustain

servicesfollowingaBlackSkyevent:

1. Skilledmanpower:treatmentplantoperators;distributionsystemrepairandmaintenance

personnel;cyber/ICSexperts;generatoroperatorsandmaintenancespecialists.

2. Disinfectantchemicals:continuingsupplytopreventtheoutbreakofillnessanddisease.

Additionalrequirementsexistforchemicalsusedtoachievelowerprioritytasteandodor

standards.

3. Fuel:foremergencypower,beyondthatstoredonsite.

4. BottledWater:asupplementalsupplyofwaterprovidedbyexternalpartiestomeettheneedsof

thosepeoplelosingtheiraccesstopotablewater.

Sector Commodity Specific List Matrix

Phase Commodity Estimated

Quantity

PotentialSource

Mitigation,Response,

Recovery

Skilledgeneratoroperators Variable WARN,Ex-

Military

Mitigation,Response,

Recovery

Skilledgeneratormaintenancepersonnel Variable WARN,Ex-

Military

All Skilledtreatmentplantandsystem

operators

Variable WARN

All DisinfectantChemicals Securean

initial30-

daysupply

WARN,Chemical

Plants,Food

Stores

30

Sector Black Sky Communications Overview Communicationsareveryimportanttotheoperationoflarge-scalewaterandwastewatersystems,

becausetheysupportpersontopersoncoordinationandtheincreasingadoptionofIndustrialControl

andSCADAsystemstomanagesystemoperations.Thesecommunicationsarenecessary tomonitorthe

systemandprovidereadingstotheSystemOperatorviasomeformofcommunication(BSXwouldbean

optionhere)hourlyinordertomaintainsustainableminimalistlevelsofservice.Communicationswith

WARNpartners,particularlyiftherehasbeenextensivepriorcollaboration,couldbeconsideredas

extendedinternalcommunications.

CommunicationswithEmergencyOperationsCentersandLawEnforcementwouldbenextinpriority

andcommunicationswithcomponentsofthesupplychainwouldfollow.Communicationswith

GovernmentauthoritieswouldgenerallybewithStates,toprovidestatusandtorequestintervention

neededtofacilitateoperations.Communicationwithpublicwillbevitalforongoingconservation

efforts,hometreatmentneeds,andpotentialrelocationrequirements.

Sector Communications Matrix

All Skilledtreatmentplantrepairpersonnel TBD WARN;local

trades

All CoagulantChemicals 30-day

supplyon

site

WARN,Chemical

Plants

All Fuels Securean

initial30-

daysupply

Contractswith

currentsupply

chain;alternate

fuels,when

necessary

Mitigation,Response BottledWater TBDby:

Pressure

Zoneshut

downand

where

quality

levelsare

lowered

FoodStores,

NGOs

Phase CommunicationsRequirement CoordinatedCrossSector

Element

Mitigation,

Response,

Recovery

WithPublic External

Mitigation,

Response,

BetweenOperatorsofthesub-systems(including

WARNpartners)

Internal

31

Sector Black Sky Assessment Tool (s) Overview EPA,withAmericanWaterWorksAssociationandDepartmentofHomelandSecurity,hasdevelopeda

numberofusefultoolsthatcanbeusedtoassessoverallresilienceandtheresilienceofcomponentsof

theWaterSector.

• RiskandResiliencyManagementofWaterandWastewaterSystems(2014)VSATVersion6.0

o http://water.epa.gov/infrastructure/water_security/techtools/vsat.cfm

o Vulnerabilityself-assessmenttoolbasedonANSI-AWWAJ100-10Standard(RAMCAP).

Thistoolisdesignedtoassistwaterandwastewaterutilitieswithdetermining

vulnerabilitiestoallhazardsandwithevaluatingpotentialimprovementstoenhance

theirsecurityandresilience.Itcanbeusedtoconductorupdateanall-hazardsrisk

assessment.(trainingvideosarealsoavailableatthissite).

• WaterSectorUtilityIncidentActionChecklists(2015)

o www.epa.gov/safewater

o ‘RipandRun’stylecheckliststofacilitateemergencypreparedness,responseand

recoveryactionsfortennaturaldisasterthatcansignificantlyimpactwaterand

wastewaterutilities:

§ DroughtEPA817-F-15-001

§ Earthquake:EPA817-F-15-002

§ ExtremeCold/WinterStorm:EPA817-F-15-003

§ ExtremeHeat:EPA817-f-15-004

§ Flooding:EPA817-f-15-005

§ Hurricane:EPA817-F-15-006

§ Tornado:EPA817-f-15-007

§ Tsunami:EPA817-f-15-008

§ VolcanicActivity:EPA817-F-15_009

• Wildfire:EPA817-F-15-010EmergencyWaterSupplyPlanningGuideforHospitalsandHealth

CareFacilities(2012)

Recovery

All BetweenElectricSubsectorandWaterSector External

Protection,

Mitigation,

Response,

Recovery

Sharethreatandhazardinformationacrosssectorsby

usingtheInformationSharingandAnalysisCenter

networkofSectorISACs

External

Protection,

Mitigation,

Response,

Recovery

MaintainICS/SCADAfunctions Internal

Mitigation,

Response,

Recovery

BetweenWaterSystemsandEmergencyOperations

Centers

External

Mitigation,

Response,

Recovery

BetweenWaterSystemsandLawEnforcement External

32

o www.cdc.gov/healthywater/pdf/emergency/emergency-water-supply-planning-

guide.pdf

o CentersforDiseaseControlandAmericanWaterWorksAssociationGuidelinestoassist

healthcarefacilitiesdevelopanEmergencyWaterSupplyPlan(EWSP)toenablethe

facilitytocontinueoperationswhenthedrinkingwatersupplyhasbeeninterruptedfor

anyreason.

• HazardMitigationforNaturalDisasters(AStarterGuideforWaterandWastewaterUtilities)

(2016)EPA810-B-16-001

o www.epa.gov/safewater

o Thisguideprovidespracticalsolutionstohelpwaterandwastewaterutilitiesmitigate

theeffectsofnaturaldisasters.

• AWWACybersecurityGuidanceTool(2014)

o www.awwa.org/cybersecurity

o Thistoolappliesonlytoprocesscontrolsystemsandnotbusinesssystems.Thoughnot

anassessmenttool,itusesanumberofquestionstocharacterizetheutility’sSCADA

system,andthenoffersanumberof‘controls’(suggestions)tohardentheprocess

controlsystem.Thistoolisconsistentwiththefederal‘CyberSecurityFramework’

issuedbytheNationalInstituteofStandardsandTechnologyin2014.

• WaterResearchFoundationBusinessContinuityPlanGuidanceDocument(2013)

o www.waterrf.org

o Thistoolassistsutilitiesinpreparingaplantoensurecontinuationofutilitybusiness

activitiesfollowingadisaster.Thedocumentdirectstheutilitytoidentifykey

parameterssuchas:missionessentialfunctions,criticalresources,andatimeframefor

reachingobjectives.

• SomeutilitiesarebeginningtoincorporatehydraulicmodelingandanalysisusingGeographic

InformationSystems(GIS)intoBlackSkyassessmentsandplanning

o Multipletools:hydraulicmodeling,GIS,simulationmodeling

EISCouncilworkswithU.S.EPA,hydraulicandGISmodelingtoolcompaniesandselectutilitiesto

developrecommendationsregardinghowtousehydraulicmodelingcombinedwithGIStoprovide

operatorsadeepunderstandingofwaterorwastewatersystembehaviorunderavarietyofBlackSky

scenarios,andtoenabledevelopmentof‘whatif’strategiesthatcouldbeusedtobuildadaptabilityinto

ResponseandRecoveryactivities.

Sector Black Sky Planning Requirements (On-going) AsnotedthroughoutthesectiononInternalRequirementsabove,extensiveplanningforBlackSky

operationsisveryimportanttomaintainingadequatewaterandwastewaterservicesvitaltopreventing

widespreadlossoflifeandthecontinuityofsociety.Planningmustaddresstherequirements

proprietarytoeachofthephases,andshouldcontinuetoberefinedinastructuredapproachthatseeks

thatleverages“everythingislearning”,aswellasmodelingandsimulation’tobuildandevolvea“body

ofknowledge”.Mostimportantly,whenperformedcollaborativelyacrosssectors,BlackSkyplanning

forcesthescopetobebroadened,generatesdiscoveryandservestoimprovecurrentoperations.

33

Inadditiontotherequirementsstatedabove,therearethreeotherareasthatneedtobeaddressed:

• CriticalSupportRequirements.BlackSkyeventswillprecipitateworkrequirementsfar

exceedingcapabilities,requiringresponseandrecoveryactionstobeprioritized.EPRO

HandbookIII,BlackSkyCoordinationandCommunication,advocatesforidentifying/designating

criticalfacilitiestoprovidefocusandprioritizationforresponseandrecoveryefforts.Listsof

designatedcriticalfacilitiesshouldformacommonoutcome-basedsetoftargetsusedforBlack

Skyplanning.

• ModelingandSimulation.Modelingandsimulationshouldbeusedextensivelyasanadjunctto

supportBlackSkyplanning.Eachsectorhasdevelopedintra-sectorcapabilitiestomodelits

systemsandsimulateoperations.IntheWaterSector,forexample,EPAhasdevelopedatool

calledWNTRthatutilitiescanusetomodelavarietyofscenariostodeterminearichsetof

vulnerabilitiesandtheirimpactonpressureandwater/wastewaterquality.Modelingand

simulationcanbeusedforexample,todetermineoptimaldispositionofalimitedsupplyof

emergencygenerators,workforce,andequipment.Itcanbeusedalsotoassessthe

effectivenessofplannedsequencingandresourcingofrestorativeactions.Mostimportantly,

whenmodelingandsimulationincorporatescross-sectoroperations,suchaswithGINOM,itcan

beusedtoassesstheimpactsofinterdependenciesthroughoutallphases.Finally,whenusedin

conjunctionwiththemetricsappropriatetoeachreportingactivity,modelandsimulation

providesaverypowerfulmeansofvalidatinginvestmentsinBlackSkyresilience.

• Water/WastewaterBlackStartProjectPlans.OneofthekeyelementsofcompleteBlackSky

planningisrecoveryplanning,thatis,autility’splanforrestoringservicestopre-event—or

otherwisespecified---levels.ThesizeandcomplexityofWater/WastewaterSectorfacilities

requireswell-plannedactivities,awell-consideredresourceallocationplanthatincorporates

contingentresourceavailability,andrisk-basedscheduling.Eachofthesecomponents—

activities,resources,schedule—arenotonlyinter-relatedbutalsomustbe“risk-adjusted”

meaningeachprojectplanmustincorporateprobabilitiestoprovidearangeofdurations,in

additiontomaximumlikelihoodofresourceavailabilityorduration.Theprobabilitiescanbe

usedinmodelingandsimulationtoolssuchasGINOMtoprovide“worst/bestcase”planning

parametersessentialtodecidingamongcoursesofaction,andtoestablishtheirsequencing.

Thesetoftheseprojectplansconstitutea“runningestimate”ofrecoverydurationswithinand

acrosssectors.

Sector Best Practices Matrix (On-going)

Areaof

Operations

Recommendation ExpectedImprovement

Waterand

Wastewater

andElectrical

DevelopcomprehensivePowerOutageIncident

AnnextoEmergencyOperationsPlans

Greaterawarenessand

understandingofwater/electric

nexuswillresultinimproved

resiliencetoBlackSkyhazards

Waterand

Wastewater

Developcomprehensiveemergencypowerplan Optimizescapabilitytoidentify,

planforandmaintain

34

Integrated and Shared Planning Actions (3.5/V4) MaintainingWaterSectorservicesfollowingaBlackSkyevent,evenatreducedlevels,requires

extensiveplanning,collaboratingandexercisingacrossallsectors.Integrationwithothersectors,

classifiedbytypeisasfollows:

andElectrical sustainableminimalistlevelsof

service

Water UsehydraulicmodelingandGIStosupport

resilienceplanningandinvestment

Improvedsituationawareness,

responseplanningandresilience

investments

Waterand

Wastewater

MaximizeWARNopportunities Enhancesresiliencebyadding

additionalcapabilitiesat

reducedlevelsofcost.Improves

thequalityofresilienceby

leveragingthe‘twoheadsare

betterthanone’principle

All Coordinateandmanageinterdependencies

amongsectorsintensively.

Improvedresilienceandgreater

efficiencyintheallocationof

resources

35 Planning and Coordination Actions M

atrix

Water

ElectricFederal

StateRegulatory

Telecommunications

NGO

Water

Support

• PriorityforRestorationofG

ridPow

er• TechnicalKnow

ledge• InclusioninPlanning,Training,Exercises

• CoordinatedBlackStartplanning

• SupportforMicrogridsand

SecureEnclaves• CoordinatedModeling

• ProvidedFederalEm

ergencyAssistance

• WaterInfrastructure

FinanceandIm

provementAct

(WIFIA)Funding

• Tools• M

odelingSupport• SetDesignStandardsforSustainedEm

ergencyOperations

• NERC-equivalent

PrimacyAgency

Support• Com

plianceGuidance

• LaboratoryServices• Dam

ageAssessm

ents• Technicalknow

ledge• RepresentationinEO

C• IncidentFieldSupport

• SupporttoWARN

• Financing(SRF)Em

ergencyManagem

entSupport• AccesstoSites• Credentialing• EvacuationPlans• ProtectionofAssets• State-levelCoordination

• InclusioninPlanning,Training,andExercises

• RequestingFederalAssistance

• SetDesignStandardsforEm

ergencyOperations

• RelieffromW

aterQualityStandards

(SDWA)

• RelieffromCleanAir

Act• Relieffrom

FuelStorageLim

itations• Relieffrom

highway

andInter-statetransportationrestrictions

• Plansforfacilitatingmovem

entofequipm

entandpersonnel

• • IncorporateW

aterSectorintoallEvacuationPlanning,TrainingandExercises

• Identifyevacuationlocations

Information

• Estimated

RestorationTime

• • Em

ergencyResponsePlans

• EOClocation

• PlannedEvacuationSites

• •

•

36

Sector Black Sky Resilience Considerations Overview (3.5/V4) TheWaterSectorhasidentifiedanumberofpotentialresilienceactions,activitiesandinvestmentsthatwouldstrengthenthesector’sresiliencetoBlackSkyhazardsandarelistedinthetablebelow.

Ingeneral,WaterSectorutilitiesaremunicipallyownedandratesareregulatedtokeepthemlow.ThissituationhinderstheirabilitytomakethetypesofinvestmentinresiliencethatwouldberequiredforBlackSkyhazards.

Resilience Initiatives Matrix Toolingandcapabilitiespreparedinadvance:Definethosecategoriesofadvanceplanning,investmentandeffortrequiredtomakeinfrastructurerestorationandpopulationsustainmentpossible,whenthenationfindsitselfthrustintoahighlydisrupted,BlackSkyscenario.

InitiativeTitle InitiativeDescription/Cost ExpectedOutcome

Water/EnergyNexus ParticipateinDOEinitiativetoanalyzetherelationshipsbetweenenergyandwateruseandconductresearchonwaterandenergysystems.`

DeepercollaborativerelationshipbetweenWaterSectorandElectricSubsector

EPRI ReviewandleverageEPRI’spublishedresearchonWaterSustainabilityandseektoestablishapartnershipbetweenEPRIandtheWaterSector.

DeeperrelationshipbetweenWaterSectorandElectricSubsector

HydraulicModeling/GeographicInformationSystem(GIS)

PromotehydraulicmodelingandGIStosupportBlackSkyplanning.Hydraulicmodelingcanassistautilityingainingafullrealizationofcapabilities,vulnerabilitiesandresponses,toavarietyof‘whatif’scenarios,whileGIScanprovideinformationonlanduse,population,criticalcustomers,shelterlocations,etc.

ImprovedBlackSkyplanningandSituationAwareness

EmergencyPowerAnnextoEmergencyResponsePlan

Conductanin-depthanalysisofemergencypowerrequirements;incorporatehydraulicmodelingtodeterminenumbersandplacementofemergencypowerunits.Assesssupportingrequirementssuchasfuel,maintenance,andunitreplacements.

Detailedexecutableplanforrestoringsustainableminimalistservicelevels,andbeyond.

Cyber-SecureWaterandWastewaterOperations

Developplansforacyber-basedattackonWaterandWastewatersystems,particularlyinconjunctionwithanaccompanyingattackontheElectricGrid

Reducedtimetorecover

37

Sector Black Sky Regulatory Impacts and Issues Overview (On-Going) BlackSkyregulatoryissuesaredescribedabove,inExternalandCrossSectorDependenciesOverview,PartB.,Government-RegulatoryBodies,andaresummarizedasfollows.Theyarefurtherdetailed,onanitembyitembasis,inAnnexB,RegulatoryIssuesDetailStatements.

Sector Regulatory Matrix Areaof

Operations

Issue RecommendedSolution/Resolution

WaterTreatment

Limitsonuseofspecificwatersources Identify,andcoordinatewithRegulatoryofficialstouse,specificsourcesofwaterduringBlackSkyresponseandrecovery

WaterTreatment

WaterQualityStandardsunderBlackSkyConditions

IdentifyandcarefullyconsiderimpactsofstandardstoberelaxedunderBlackSkyenvironment(e.g.,tasteandodor,andacutevschronicexposure)

WastewaterTreatment

Relaxdischargestandardsforwastewatersystems

IdentifyandcarefullyconsiderimpactsofstandardstoberelaxedunderBlackSkyenvironment(e.g.,tasteandodor,andacutevschronicexposure)

Waterand Responseandrecoverypersonnelshouldbe

SupplyChainResilienceAssessment(commodities)

Developadeepunderstandingofthecapabilitiesandvulnerabilitiesthesupplychainsthatsupportcontinued

Withanimprovedunderstandingofthevulnerabilities,threats,andconsequences,ownersandoperatorsofutilitiescancontinuetothoroughlyexamineandimplementrisk-basedapproachestoprotect,detect,respondto,andrecoverfromallhazardsbetter

FinancingResilienceInitiatives

Identifymeanstofinanceresiliencemeasuresintonewandexistingcapabilities.

Overcomecurrentobstaclesposedbyrequirementtouseratestopayforresilience

NIACWaterSectorResilienceMeasures

AdoptandsupporttheresiliencerecommendationscontainedintheNIACWaterSectorResilienceFinalReportandRecommendations,datedJune2016.

BlendsNIACandEISCouncileffortsandopenstheopportunitytobroadenBlackSkyconsiderationsinfutureNIACrecommendations.

38

Areaof

Operations

Issue RecommendedSolution/Resolution

Wastewater providedFirstResponderstatusEmergencyPower

LimitsPlacedonRunningEmergencyPower PartialExemptionfromCleanAirAct

WaterandWastewaterTreatment

Limitsonon-sitechemicalstorage IdentifyandcarefullyconsiderimpactsofstandardstoberelaxedunderBlackSkyenvironment(e.g.,tasteandodor,andacutevschronicexposure)

EmergencyPower

LimitsPlacedonon-sitefuelstorage IdentifyandcarefullyconsiderimpactsofstandardstoberelaxedunderBlackSkyenvironment(e.g.,tasteandodor,andacutevschronicexposure)

Sector Black Sky Essential Critical Infrastructure (MC) Overview (3.5/V4) Themostessentialelementsofwaterutilities’infrastructuretoachieveresiliencearedetailedbelow.TheseincludeitemsthatareuniqueormissionessentialtotheWaterSector.

Sector Critical Infrastructure Matrix (V3.5/V4) Element Function

Pumps Drawfromsource,convey,lift,maintainpressure:highenergyDistributionPipelines

Conveywater:notolowenergy

StorageTanks

Store:energytoliftbutnotoncestored

WaterTreatmentSettlingTanks

Clarificationbymixingandsettling;notolowenergy(mixingandsettledsolidremoval)

WaterOzonation

Useozonetokillmicro-organisms:highenergy

WaterFiltration:membrane

Improvequality;desalination:highenergy

WaterFiltration:mediafilter

Improvequality:notolowenergy

WaterDisinfection

Disinfectwater:lowenergy

WastewaterPrimary

PrimarySewageTreatmentcapabilities:lowtonoenergy

WastewaterSecondary

Secondarysewagetreatmentcapabilities,especiallybiosolidremoval:mediumtohighenergy

WastewaterTertiary

Sewagedischargetreatmentcapabilities:lowtomedium

39

Element Function

SludgeProcessing

Sludgeconveyanceanddisposal:highenergy

SCADA Controlsystemoperations:lowtomediumenergy

Sector Black Sky Specialized Skill Training Requirements Overview (V3.5/V4/V5) TobeBlackSkyresilient,theWaterSector,includingitswaterandwastewatersystemsshouldidentifymissioncriticalskills/positionsthatmustbetrainedtoaccomplishBlackSkyresilience,startingwithAssessmentandmovingthrough,Response,RestorationandRecovery.TheAmericanWaterWorksAssociationhasoutstandingtrainingprogramsforWaterandWastewateroperators,andconsiderationshouldbegiventoeitherincorporatespecialBlackSkyskillrequirementsintoexistingcoursesor2)startaseparatetrackforBlackSkyoperations.

AWWAhasdevelopedaresourcetypingcatalogforWARNthatcanservethedualpurposeofspecifyingtheskillsrequiredduringanemergencies.Resourcetypingisthecategorizationanddescriptionofresponseresourcesthatarecommonlyexchangedindisastersthroughmutualaidandassistanceagreements.Resourcetypingdefinitionscangiveutilitiestheinformationtheyneedtoensurethattheyrequestandreceivetheappropriateresourcesduringanincident.TheresourcetypingprotocolprovidedbytheNationalIncidentManagementSystemorNIMSdescribesresourcesusingtheparametersofcategory,kind,components,metrics,andtype.Utilityandsystemplannersshouldconsultthistypingcatalog,identifytheirspecializedskillrequirementsandincludethemintheirrespectiveBlackSkyemergencyandcontinuityofoperationsplans.http://www.awwa.org/Portals/0/files/resources/water%20knowledge/rc%20emergency%20prep/rc%20warn%20resources%20pdf/typingmanual.pdf

Sector Specialized Skill Training Requirements Matrix (3.5/V4/V5) Phase Position/Skill Training/CertificationRequirement

All EmergencyPlanner TBDMitigation HydraulicModeler TBDPrevention,ProtectionMitigation

VulnerabilityAssessment TBD

Response EmergencyPowerOperator TBDResponse EmergencyPowerMechanic TBDAll SCADATechnicalArchitect TBD

40

Annex A – Assessments (On-going) Sector Overall Resilience Assessment MostofthelargerwaterandwastewatersystemshaveconductedavulnerabilityassessmentusingtheVulnerabilitySelf-AssessmentTool(VSAT)6.0.VSATVersion6.0complieswiththeWaterSectorriskassessmentstandardandcanofferliabilityprotectionundertheDepartmentofHomelandSecurity’sSupportAnti-TerrorismbyFosteringEffectiveTechnologies(SAFETY)Actprogram.

TheEISCouncilisevaluatingthecurrenttooltodeterminethemodificationsrequiredforBlackSkyresilience.ItisunlikelythatmorethanafewsystemshavemadeaBlackSkyresilienceassessment,thoughLasVegasValleyWaterDistricthasbeguntoconductBlackSkyplanning.LessonslearnedfromtheLVVWDexperiencewillbeusedtorefineassessmentrequirementsinfutureupdatesofthisPlaybook.

2

Annex B – Regulatory Issues Detail Statements (On-Going) Issue Statement 1: Testing Emergency Generators

• Statement:CleanAirActrestrictsthetestingofemergencypowerunits.Reliefissoughttoallowpropertesting.

• DecisionAuthority:EnvironmentalProtectionAgency• RequiredDocumentation:obtain/compilemanufacturers’literature

regardingtesting• ResiliencyInvestmentstatement:generatorsmustbeperiodicallytested

underfullloadforextendeddurationtoassuretheywillproperlyfunctionduringemergencyoperations

• PlanRequirements:

• TrainingRequirements:

• LiabilityStatement/3rdPartyProtectionIssue:

• Explicitrequestedlegislativechanges/Insurance/Assurance/3rdParty

Indemnification:

Issue Statement 2: Reduced Water Quality Levels • Statement:SafeDrinkingWaterActestablisheswaterqualitystandards,

manyofwhicharedeterminedbyeffectsoflong-termuse.ReliefissoughtforBlackSkyconditions.

• DecisionAuthority:EnvironmentalProtectionAgency• RequiredDocumentation:listofvariances

• ResiliencyInvestmentstatement:reducingwaterqualitylevelsforBlackSkyoperationsmayextendthesustainabilityofwaterandwastewateroperations

• PlanRequirements:

• TrainingRequirements:• LiabilityStatement/3rdPartyProtectionIssue:

• Explicitrequestedlegislativechanges/Insurance/Assurance/3rdParty

Indemnification:

2

Annex C – Communications Requirements (V3/V4) Communications Requirement 1: Internal Coordination

• Internal/PlannedFormat/Path:Internalcommunications• External/PlannedFormat/Path:N/A• ExplicitModel

o Who:designatedsitesandmobileteamsoftheutilityo What:coordinationwithinternalresources

§ Statusofkeycomponents§ Damagetoassets§ Emergencypowergeneratorstatus—includingmaintenance§ Fuelstatus§ Personnelstatus§ Tanklevels§ Pressureatkeypoints§ Qualitymeasurements§ Firehydrantoperations§ Operationsorders§ Valveoperations

o When:asoccurs;followupuntilprovidedo Strategies(backup):walkie/talkieo Bandwidthrequirement(actualandnotional):HF/UHF/VHFo Format:voiceo Priority:High

Communications Requirement 2: External Support Requirements • Internal/PlannedFormat/Path:Internalcommunications• External/PlannedFormat/Path:BSX• ExplicitModel

o Who:designatedsitesandmobileteamso What:InterventionrequestedfromEmergencyOperationsCenter

§ Accesstosites§ LawEnforcement§ Generators§ TechnicalSupportbyType§ Fuel§ TreatmentChemicals§ BottledWater§ Other

o When:asoccurs;followupuntilprovidedo Strategies(backup):physicalpresenceo Bandwidthrequirement(actualandnotional):HF/UHF

3

o Format:Voiceo Priority:High

Communications Requirement 3: Status of Services • Internal/PlannedFormat/Path:SCADA• External/PlannedFormat/Path:BSX• ExplicitModel

o Who:SystemOperatoro What:SystemMapupdatedwithSCADA-reportedStatuso When:Initial,within1hourofBlackSkyincident;followup,twicedailyo Strategies(backup):CouriertoEmergencyOperationCentero Bandwidthrequirement(actualandnotional):Lowo Format:

§ NumberofPeoplewithoutwaterservice§ Pressurezones(designators)withoutservice(preregisteredwithEOC)§ Criticalfacilities(designators)withoutadequatefireflow/pressure§ Reductionorforecastedlossofserviceovernext24hours

o Priority:High

Communications Requirement 4: Damage to Components • Internal/PlannedFormat/Path:SCADA• External/PlannedFormat/Path:BSX• ExplicitModel

o Who:SystemOperatoro What:SCADA-reportedStatuso When:Initial,within1hourofBlackSkyincident;followup,twicedailyo Strategies(backup):Voice,followingphysicalinspectiono Bandwidthrequirement(actualandnotional):TBDo Format:

§ Assetsinoperable(selectfromprepositionedlistofsystem’sassets)§ Prioritizedlistofassetreplacements(requiredtooptimizelevelofservice):use

AWWAResourceTypeo Priority:High

Communications Requirement 5: Request for Personnel • Internal/PlannedFormat/Path:Supervisorreportsusinginternal

communications• External/PlannedFormat/Path:BSXtoWARN• ExplicitModel

o Who:UtilityManagero What:Requestforadditionalsupport,byResourceTypeo When:Initial,withinTBDhoursofBlackSkyincident;followup,twicedailyo Strategies(backup):Voice,followingphysicalinspectiono Bandwidthrequirement(actualandnotional):TBDo Format:

4

§ Prioritizedlistofassetreplacements(requiredtooptimizelevelofservice):useAWWAResourceType

o Priority:High

Communications Requirement 5: Request for Resupply of Treatment Chemicals • Internal/PlannedFormat/Path:Internal• External/PlannedFormat/Path:BSX• ExplicitModel

o Who:LogisticsManagero What:Treatmentresupplyrequirementso When:3daysandwheneversuppliesfallbelowreorderpoint(continuously

updatedbasedonexperiencegainedduringBlackSkyresponseandrecoveryo Strategies(backup):Preplanned‘push’supplyo Bandwidthrequirement(actualandnotional):TBDo Format:

§ Estimatedchemicalsupplybyquantity,projectedovera30-dayperiodo Priority:Medium

5

Annex D: Resilience Requirements by Layer Identifyrequirementsforeachphase,includingboth(A)GenericrequiredelementsforallBlackSkyhazards,and(B)Hazard-specificrequiredrequirements.Note,however,thatmanyrequirementswillexhibitheavyoverlapamongsectors,withmanyinfrastructuresectors,andtheirpartners,requiringthesameresiliencemeasures.

Area Phase BlackSkyGeneric BlackSkyHazardSpecific

Notes/Status

Emergency

CommunicationMitigate

Respond

Recover

• Planforcommunicationswithoutcellular

• Planfornon-gridpowerforcommunications

• Train/rehearseusingfrequenciesavailableduringBlackSky

• BSXcapabilities(EMPhardenedandprolongednon-gridpower)

• Planforcommunicationswithoutcellularandsatellite(EMP)

• StoreequipmentinFaradayboxes(EMP)

MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission

Cross-sector

situational

awareness,

coordinationand

decisionsupport

requirements

Mitigate

Respond

Recover

• Identifykeymetricsformonitoringinternalsectoroperations

• Identifykeymetricsformonitoringcross-sectoroperations

• Performall-hazard,multiple-scenariomodelingandsimulationtosupportBlackSkyplanning

• IncorporateGINOM-furnishedinformationintoallplanning

• EnsureBlack-Skysecure(EMP/GMDhardenedandlong-termpowersupply)toGINOMcapabilities

MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission

EmergencyFuel Mitigate

Respond

Recover

• Developfuelplanforextendedpoweroutagecontingencyo Coordinateplanwithsuppliers

o Identifypotentialcompetitionforresourceswithsuppliers

o Investinfueltankerstosupportlong-termfuelingoperations

• Maintainatleast7daysoffuelforfleetandemergencypower

• Planfordisruptedfuelandchemicaltransportationintheeventoffloodingorearthquake

MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission

OperationalFuel

(e.g.,naturalgas)

Requirements

Mitigate

Respond

Recover

• Plan/investinfuelingcapabilitiesforemergencyvehicles,usingonhandcapabilities

• Planfordisruptedfueltransportationduetofloodingorearthquake

MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission

Emergency

Generators(Black

Mitigate • Plan/investinsufficientemergencypowercapabilitytomaintain

• InvestinEMP/GMD-hardenedgenerators

MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission

2

Skyhazard-

hardened,designed

forlongduration

continuous

operation)

Respond

Recover

minimalservicelevels• Usehydraulicmodelingtoidentifyoptimaldeploymentofemergencypower

• RegisterrequirementsinEPFAT

• Provisionhookupsformobileandexternally-providedunits

• Investingeneratorsusedforlong-duration,continuousoperation

• Investinalternativeenergysources(wind,PVandco-generation)

• Planforspecializedmaintenancesupport

• Investinmulti-fuelcapability

• Securetheavailabilityofreplacementgenerators

Protectivemeasures:Hardware,software,operationalprocedures

Mitigate

Respond

Recover

• Protectelectricfacilitiesfromflooding

• Strengthendistributionsystemtominimizeearthquakedamage

• ProtectSCADAfromEMP• ProtectemergencygeneratorsandvehiclesfromEMP/GMD

MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission

Health/damage

assessment

diagnostics:

Mitigate

Respond

Recover

• Develop/exercisecomprehensivedamageassessmentCONOPS/CONPLAN

• IncorporateassetmanagementintoBlackSkyplanning

• Performcomprehensivemodelingwithmultiplescenarios(allhazards)toassesstherangeofpotentialdamagetosystem

• LeverageIoTtodesignacomprehensiveandintegratedsensorplanbasedonsituationawarenessgoals

• Plan/installhealthchecksforemergencypowerunits

• ProtectdiagnosticcapabilitiesfromEMP/GMD

MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission

Pre-deployed,automatedself-poweredremotereportingtocentralassessmentcontrollers

Mitigate

Respond

Recover

• Pumpstationpoweroutage

• Tank-levelindicator• Pressureindicatoratcriticalpointsindistributionsystem

• InvestinEMP/GMD-securesensing/communicationscapabilities

MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission

On-sitediagnostics Mitigate • Manual • N/A Meetingallrequirementswill

3

providingeasy1storderdamageassessmentforanonsite,deployedrestorationteam

Respond

Recover

water/wastewaterqualitytestingkits

haveMAJORimpactonmeetingsystem’smission

Diagnostictoolingdesignedformanualusebyadeployedrestorationteam

Mitigate

Respond

Recover

• Planforenhancedwater/wastewaterqualitytesting

• Planforwater/wastewaterqualitytestingcapability

• Providecapabilitytoreconfigurehydraulicmodelingtoolsinresponsetodamage

• SecureEMP/GMD/vulnerablediagnosticequipment

MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission

Restoration/

emergencytoolingMitigate

Respond

Recover

• Developblackstartplanforsystem–forseveraloutagescenarios

• Identifyspecializedsupportforrestoringsystemfollowingaprolongedpoweroutage

• Developresourcedprojectplanforsystemrestart

• IncorporatemodeledEMP/GMD/earthquakeandflooddamageforecastintorecoveryplans

•

MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission

BlackSkyExercise

andTraining

Requirementsand

Plans

Mitigate

Respond

Recover

• DevelopPowerOutageIncidentAnnex(POIA)CONOPSorCONPLANforconductingoperationsfollowingextensivegridoutage

• Plan/conductsectorPOIAexercise

• Plan/conductcross-sectorPOIAexercise

• IdentifyanddocumenttrainingrequirementsforPOIAoperations

• Incorporatehazard-causeddamage(inadditiontoeffectsofpoweroutage)intowater/wastewatersectorresponseandrecoveryplans

MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission

Materiel Mitigate

Respond

Recover

• Enhancedwater/wastewaterqualitytestingcapability

• Treatmentchemicalsandsupplies

• Suppliestomaintainworkforce

• ProtectelectronicequipmentfromEMP/GMD

• Protectelectricalequipmentfromflooding

• Investinhardeningmeasuresforearthquakeandflooding

MeetingallrequirementswillhaveMAJORimpactonmeetingsystem’smission

4

1. Spares,withadvancefielddeploymentrequirements:Inthreecategories

a. Inexpensive,commonusehardwarethatwillbehardtofindinBlackSkyconditionsunlesspre-deployed(i.e.,“Forwantofanail…”)

b. Inexpensivetomoderatelyexpensivehazard-vulnerablehardwareelementsfordifferentBlackSkyhazardscenarios

c. Expensive,longleadhazard-vulnerablehardwareandassociatedinstallationtoolingfordifferentBlackSkyhazardscenarios

2. Personnelsupportrequirements:Whatteams,andhowmanyteams,willberequiredbyeachstakeholderorganization,withwhatteammakeupandwhatdeployment,foreachhazardtypeandforeachphase?

Note1.Thisassessmentwilldrive,forexample,therequirementsforprearranged,pre-certified,externalCertifiedPowerRecovery(CPR)EngineeringTeamstoaddsubstantiallytothecoreorganizationalteamsforthatsector.

Note2.Asupplementalrequirement:giventhenumber,makeupanddeploymentoftherequisiteteams,whatwillberequiredforassociatedfamilysupport?

Note3:Thesepersonnel/teamrequirementsshouldbeprovidedasafunction,atleastcrudely,ofthelevelofavailabilityofdiagnostics(in(g)i.->iii.Above).E.g.,ifthereareremotelypre-deployed,BlackSkyhardened,self-powered,embeddedandremotelyreportingdiagnosticsinallimportantpowergirdsubstationsandgeneratingsubstationsandwatersystemandgaspipelinekeyfacilities,vastlyfewerrestorationteamswouldneedtobedeployed,thosethataredeployedwouldbefarmoreeffective,andtheentiredamageassessmentphasewouldbefarquicker.

3. BlackSkyOperationalPlansandProcedures:Supplementingexistinghazardplansandproceduresforconventionalhazards

4. BlackSkyExerciseandTrainingRequirementsandPlans:Supplementingexistinghazardexerciseandtrainingrequirements