incident response ”oh no, we’ve been hacked! now what do we do?” sec 211

Incident Incident ResponseResponse”Oh no, we’ve been hacked!”Oh no, we’ve been hacked!Now what do we do?”Now what do we do?”

SEC 211SEC 211

Why?Why?

Typical incident response Typical incident response processprocess1.1. Oh no, we got hacked!Oh no, we got hacked!2.2. Look for the easy solutionLook for the easy solution3.3. Failing that, observe the damage for a timeFailing that, observe the damage for a time4.4. Update resume and await executionUpdate resume and await execution

There’s a better wayThere’s a better wayWhy wait until your first attack?Why wait until your first attack?

Affects decision-makingAffects decision-makingCosts moreCosts more

Make it part of your security policy and risk Make it part of your security policy and risk mitigation strategymitigation strategyReal benefitsReal benefits

No panic attacks—process guides responseNo panic attacks—process guides responseFinancial—discounts from insurance companyFinancial—discounts from insurance companyService provider—help win businessService provider—help win business

Taxonomy of security workTaxonomy of security work

PreventionPreventionPreventionPrevention DetectionDetectionDetectionDetection

IT budgetIT budget

ReactionReactionReactionReaction

The processThe processMinimize the number and severity of Minimize the number and severity of incidentsincidentsAssemble the core computer security incident Assemble the core computer security incident response teamresponse teamDefine an incident response planDefine an incident response plan

MinimizeMinimizeIncidentsIncidents

Know your stuff!Know your stuff!Where are your backups?Where are your backups?Are they good?Are they good?What assets are you trying to protect?What assets are you trying to protect?What are the threats against them?What are the threats against them?What vulnerabilities might the assets have?What vulnerabilities might the assets have?How likely are those threats to materialize?How likely are those threats to materialize?What is “normal” traffic on your network?What is “normal” traffic on your network?How fast should the network typically How fast should the network typically respond?respond?Who sends us email? Who should be?Who sends us email? Who should be?Are your computers protected from theft?Are your computers protected from theft?

Where does this fit?Where does this fit?


IT budgetIT budget


Risk assessmentRisk assessment

LowLow HighHigh

Ris

kR

isk

Asset ValueAsset Value

HighHigh

Risk tolerance

Risk tolerance

What?What?Me worry?Me worry?

Yes!Yes!We worry!We worry!

It’s got to cover all layersIt’s got to cover all layers

People, policies, and process

People, policies, and process

Physical securityPhysical security

PerimeterPerimeter

Internal networkInternal network

HostHost

ApplicationApplication

DataData

Sample classification Sample classification schemesschemesPhysicaPhysicall

Where is the Where is the asset?asset?

How is access obtained?How is access obtained?

Public areaPublic areaEmployee-onlyEmployee-onlyControlledControlled

Available during business hoursAvailable during business hoursCard-key readersCard-key readersCard-key, PIN, and palm printCard-key, PIN, and palm print

NetworNetworkk

Access from Access from where?where?

How to authenticate?How to authenticate?

Wired corpnetWired corpnetWireless Wireless corpnetcorpnet

VPNVPNKiosksKiosksInternetInternet

Domain logon (human and PC)Domain logon (human and PC)Domain logon plus certificatesDomain logon plus certificates(human and computer)(human and computer)Domain logon, smartcard, Domain logon, smartcard, quarantinequarantineDisallowedDisallowedDisallowed except from corp PCDisallowed except from corp PC

Start with the “soft” stuffStart with the “soft” stuffEstablish, enforce, and measure policiesEstablish, enforce, and measure policies

If you can’t measure it, drop itIf you can’t measure it, drop itA lot of incidents happen “by accident”A lot of incidents happen “by accident”

Get management supportGet management supportBegin regular security trainingBegin regular security training

ILOVEYOU!ILOVEYOU!Most email worms target carbon, not siliconMost email worms target carbon, not silicon

Think about security bannersThink about security bannersThat they stop prosecution is an urban legendThat they stop prosecution is an urban legendBut they remind people of their responsibilitesBut they remind people of their responsibilites

Don’t neglect periodic Don’t neglect periodic maintenancemaintenanceConduct regular vulnerability assessmentsConduct regular vulnerability assessments

Do it yourself or hire a consultant, your choiceDo it yourself or hire a consultant, your choiceRun away from checklist slavesRun away from checklist slavesAre they bondable? Do you trust them? Are they bondable? Do you trust them? (the “daughter (the “daughter test”)test”)

Don’t forget to test social engineeringDon’t forget to test social engineeringGet permission!Get permission!

Don’t neglect periodic Don’t neglect periodic maintenancemaintenanceKeep your systems patched and up-to-dateKeep your systems patched and up-to-date

Clients: come on, start using a patch management Clients: come on, start using a patch management tooltoolServers: your choice, be mindful of rebootsServers: your choice, be mindful of reboots

Important technical controlsImportant technical controlsStrong password policiesStrong password policies

Passphrases are better thoughPassphrases are better though

Monitor and analyze network traffic and Monitor and analyze network traffic and system performancesystem performance

Learn what “normal” means for youLearn what “normal” means for you

Important technical controlsImportant technical controlsRoutinely check all logsRoutinely check all logs

But they’re useful only after you’ve already But they’re useful only after you’ve already learned “normal”learned “normal”

Verify backupsVerify backupsDo restores actually work?Do restores actually work?Is the media still functioning?Is the media still functioning?Who can perform?Who can perform?

Assemble theAssemble theCore CSIRTCore CSIRT

The core CSIRTThe core CSIRTThese are the people who respond to all These are the people who respond to all incidentsincidentsRequire responsibility Require responsibility andand authority authority

Clearly-defined duties: eliminates “not my job!”Clearly-defined duties: eliminates “not my job!”Who pulls the LAN cable? Under what conditions?Who pulls the LAN cable? Under what conditions?

Build this Build this beforebefore you get attacked you get attackedMake it part of their regular job descriptionMake it part of their regular job descriptionInclude in job performance goalsInclude in job performance goalsGive them periodic drills for practiceGive them periodic drills for practice

Successful teamsSuccessful teamsMonitor for security breachesMonitor for security breachesAct as “communications central”Act as “communications central”

Receive reports of incidentsReceive reports of incidentsDisseminate information about incidentsDisseminate information about incidents

Document incidentsDocument incidentsPromote security awareness inside the Promote security awareness inside the companycompanySupport system and network auditingSupport system and network auditing

Vulnerability assessments, penetration testingVulnerability assessments, penetration testing

Remain abreast of new vulnerabilities and Remain abreast of new vulnerabilities and attacksattacksResearch software patchesResearch software patchesAnalyze and implement new processes and Analyze and implement new processes and technologies for reducing vulnerabilities and technologies for reducing vulnerabilities and riskrisk

Team preparationTeam preparationTrain to use good toolsTrain to use good tools

Where are they?Where are they?How to use them?How to use them?Rapidly available—Rapidly available—specialized laptops used specialized laptops used only for thisonly for thisBe sure to protect them Be sure to protect them when not in use!when not in use!

Team preparationTeam preparationTrain to use good toolsTrain to use good toolsAssemble all relevant Assemble all relevant communication infocommunication info

Contact names and Contact names and numbersnumbers

CSIRT teamCSIRT teamAdmins and ownersAdmins and ownersLegalLegalPublic/media relationsPublic/media relationsISPISPLaw enforcementLaw enforcement

Involve legal in any Involve legal in any dealings with law dealings with law enforcement and when enforcement and when gathering evidencegathering evidence

Team preparationTeam preparationTrain to use good toolsTrain to use good toolsAssemble all relevant Assemble all relevant communication infocommunication infoKeep emergency info in Keep emergency info in central offline storagecentral offline storage

PasswordsPasswordsIP addressesIP addressesRouter configurationsRouter configurationsFirewall rulesetsFirewall rulesetsCertification authority keysCertification authority keysContact names and Contact names and numbersnumbersEscalation proceduresEscalation proceduresIf electronic, encrypt it If electronic, encrypt it then lock it up!then lock it up!

Team rolesTeam roles

In charge of the team’s activitiesIn charge of the team’s activitiesCoordinates reviews of team’s actionsCoordinates reviews of team’s actionsAuthorized to change policies and proceduresAuthorized to change policies and procedures

TeamTeamLeadLeadTeamTeamLeadLead


Do the actual workDo the actual work


IncidentIncidentHandlerHandlerIncidentIncidentHandlerHandler







Owns a particular incidentOwns a particular incidentCoordinates all communication about the incidentCoordinates all communication about the incidentRepresents entire CSIRT to those outsideRepresents entire CSIRT to those outsideMight vary, depending on incident particularsMight vary, depending on incident particulars



IncidentIncidentLeadLead

IncidentIncidentLeadLead





From various departments throughout your companyFrom various departments throughout your companySpecialize in areas affected by security incidentsSpecialize in areas affected by security incidentsParticipate in incidents or delegate to another in Participate in incidents or delegate to another in their areatheir area







AssociateAssociateMemberMember








Associate membersAssociate membersIT contactIT contact Coordinates communications between incident Coordinates communications between incident

lead and rest of IT departmentlead and rest of IT department

Legal Legal representatirepresentati

veve

Lawyer familiar with incident response policies; Lawyer familiar with incident response policies; determines how to proceed. Involved before determines how to proceed. Involved before incidents, evaluates response policies to ensure incidents, evaluates response policies to ensure you aren’t at legal risk during containmentyou aren’t at legal risk during containment

If we shut down, do we violate service agreements?If we shut down, do we violate service agreements?If we don’t shut down, are we liable if someone else If we don’t shut down, are we liable if someone else gets attacked from the compromised system?gets attacked from the compromised system?

PR officerPR officer Crafts message for the public and handles all Crafts message for the public and handles all media inquiriesmedia inquiries

ManagemenManagementt

Either departmental or company-wide; determines Either departmental or company-wide; determines total impact (financial and otherwise) to the total impact (financial and otherwise) to the company; directs communications officer and company; directs communications officer and interaction with law enforcement agenciesinteraction with law enforcement agencies

Response rolesResponse roles

Inc. leadInc. lead IT contactIT contactLegal Legal repr.repr. PR officerPR officer Mgmt.Mgmt.

Initial assessmentInitial assessment Owns Advises

Initial responseInitial response Owns Implements

Updates Updates Updates

Collects forensic Collects forensic evidenceevidence

Implements

Advises Owns

Implements Implements temporary fixtemporary fix

Owns Implements

Updates Updates Advises

Sends Sends communicationcommunication

Advises Advises Advises Implements

Owns

Checks with law Checks with law enforcementenforcement

Updates Updates Implements

Updates Owns

Implements Implements permanent fixpermanent fix

Owns Implements

Updates Updates Updates

Determines Determines business impactbusiness impact

Updates Updates Advises Updates Owns

Define an IncidentDefine an IncidentResponse PlanResponse Plan

Where does this fit?Where does this fit?


IT budgetIT budget


Incident response is not…Incident response is not…PanicPanicParanoiaParanoiaFrustrationFrustrationGiving upGiving up

Plan componentsPlan componentsMake an initial assessmentMake an initial assessmentCommunicate the incidentCommunicate the incidentContain the damage and minimize the riskContain the damage and minimize the riskIdentify the type and severity of the Identify the type and severity of the compromisecompromiseProtect evidenceProtect evidenceNotify external agencies (if appropriate)Notify external agencies (if appropriate)Recover systemsRecover systemsCompile and organize incident Compile and organize incident documentationdocumentationAssess incident damage and costAssess incident damage and costReview the response and update policiesReview the response and update policies

Test this process regularly!Test this process regularly!It’s the only way you can be sure that it will work when the time It’s the only way you can be sure that it will work when the time

comescomes

Test this process regularly!Test this process regularly!It’s the only way you can be sure that it will work when the time It’s the only way you can be sure that it will work when the time

comescomes

Not purely sequentialNot purely sequentialThroughout Throughout incidentincident

DocumentationDocumentationCommunicationCommunication

In conjunctionIn conjunction Initial assessment + damage Initial assessment + damage containmentcontainment

Some Some sequencesequence

1.1.Identify type and severityIdentify type and severity2.2.Contain damage and minimize riskContain damage and minimize risk

Make an initial assessmentMake an initial assessmentIs it really a bad guy?Is it really a bad guy?

An admin doing his/her job might appear maliciousAn admin doing his/her job might appear malicious

Is it a configuration problem?Is it a configuration problem?Causing the IDS to report too many false positivesCausing the IDS to report too many false positives

Start trying to determine type and severityStart trying to determine type and severityGet enough info for further study and Get enough info for further study and communicationcommunicationHow will you contain it?How will you contain it?

Record everything you doRecord everything you doNot acting on a real incident is worse than Not acting on a real incident is worse than acting on a false positive, but don’t take too acting on a false positive, but don’t take too much time to figure it outmuch time to figure it out

Communicate the incidentCommunicate the incidentIf it’s real then communicate to entire CSIRTIf it’s real then communicate to entire CSIRTIdentify an incident lead and appoint Identify an incident lead and appoint handling team membershandling team membersDetermine who outside CSIRT to contactDetermine who outside CSIRT to contact

Maintains coordinationMaintains coordinationMinimizes damageMinimizes damageHeadline in newspaper could be Headline in newspaper could be moremore damaging… damaging…Don’t want to tip off the attackerDon’t want to tip off the attacker

Contain the damageContain the damageActing quickly and decisively can make the Acting quickly and decisively can make the difference between a minor attack and a difference between a minor attack and a major onemajor oneHelpful priorities—Helpful priorities—1.1.Protect human life and peoples’ safetyProtect human life and peoples’ safety2.2.Protect classified and sensitive dataProtect classified and sensitive data3.3.Protect other data (proprietary, scientific, Protect other data (proprietary, scientific,

managerial)managerial)4.4.Protect hardware and softwareProtect hardware and software5.5.Minimize disruption of computing resourcesMinimize disruption of computing resources

The goal:The goal: get back online as soon as get back online as soon as possiblepossible

while protecting people and preservingwhile protecting people and preservingthat which keeps us in businessthat which keeps us in business

The goal:The goal: get back online as soon as get back online as soon as possiblepossible

while protecting people and preservingwhile protecting people and preservingthat which keeps us in businessthat which keeps us in business

Contain the damageContain the damageDon’t let the bad guy know you’re on to himDon’t let the bad guy know you’re on to him

A wholesale password change, while necessary, A wholesale password change, while necessary, will also be a give-awaywill also be a give-away

Do you unplug or not?Do you unplug or not?Compare cost of yes or noCompare cost of yes or noWill you violate an SLA? Which is more expensive?Will you violate an SLA? Which is more expensive?Next time: incorporate such decision into the SLA Next time: incorporate such decision into the SLA itselfitself

Disable bad guy’s ingress pointDisable bad guy’s ingress pointModem…firewall rule…physical entryModem…firewall rule…physical entry

Rebuild new system with new hard drivesRebuild new system with new hard drivesLock up existing ones to preserve forensic Lock up existing ones to preserve forensic evidenceevidenceChange passwords, especially administrativeChange passwords, especially administrative

Determine nature of the Determine nature of the attackattackMight be different from initial assessmentMight be different from initial assessment

What’s the origin?What’s the origin?What’s the intent—What’s the intent—

Are we a specific target?Are we a specific target?Just a random victim?Just a random victim?Why us? (information, bandwidth, …)Why us? (information, bandwidth, …)

Which systems are compromised?Which systems are compromised?Which files have been accessed? How Which files have been accessed? How sensitive are they?sensitive are they?Helps direct how you will recoverHelps direct how you will recover

Incident response plan should guide your Incident response plan should guide your responses as you learn more about the attackresponses as you learn more about the attack

Determine severity of the Determine severity of the attackattackWork with other CSIRT membersWork with other CSIRT members

Do they agree with your assessment?Do they agree with your assessment?

Any unauthorized physical access?Any unauthorized physical access?Any unauthorized hardware suddenly appearing?Any unauthorized hardware suddenly appearing?Any new, unexpected members in admin groups?Any new, unexpected members in admin groups?Any new startup programs?Any new startup programs?Any gaps in logs? Or completely missing? Anything else weird in Any gaps in logs? Or completely missing? Anything else weird in them?them?

Unexpected or unusual access failures or successesUnexpected or unusual access failures or successesStrange times (nonworking hours)Strange times (nonworking hours)Permissions changes or elevationsPermissions changes or elevations

What’s different now compared to previous integrity check?What’s different now compared to previous integrity check?Any non-business data? (porn, music, warez)Any non-business data? (porn, music, warez)Any employee data now in a bad place?Any employee data now in a bad place?

You might have to deal with privacy issues nowYou might have to deal with privacy issues now

Any change in performance?Any change in performance?

Comparing against a baselineComparing against a baselineBest way to know what’s changedBest way to know what’s changedWorks only if you know your previously-Works only if you know your previously-recorded baseline hasn’t already been recorded baseline hasn’t already been compromisedcompromisedMy favorite tool: TripWireMy favorite tool: TripWireOthersOthers

EventCombMTEventCombMTDumpELDumpELMicrosoft Operations ManagerMicrosoft Operations Manager

Collect and protect evidenceCollect and protect evidenceProsecution should be the least of your Prosecution should be the least of your worries… but make backups anywayworries… but make backups anywayMake two bit-for-bit backupsMake two bit-for-bit backups

First: on write-once media (DVD±R)First: on write-once media (DVD±R)Use in case you decide to prosecute; keep physically Use in case you decide to prosecute; keep physically securesecure

Second: on brand-new hard driveSecond: on brand-new hard driveUse for data recoveryUse for data recovery

Document everything you do with themDocument everything you do with them

Physically secure original compromised disksPhysically secure original compromised disksWill become evidence if you prosecuteWill become evidence if you prosecuteRebuild system with new drivesRebuild system with new drives

Collect and protect evidenceCollect and protect evidenceThere’s always that trade-offThere’s always that trade-off

Does the cost of preserving data outweigh the cost Does the cost of preserving data outweigh the cost of delaying response and recovery?of delaying response and recovery?Is rapid recovery the most important thing for you?Is rapid recovery the most important thing for you?

Comprehensive backups might be impossible Comprehensive backups might be impossible for very large systemsfor very large systems

Limit to system state, logs, breached portions of Limit to system state, logs, breached portions of systemssystemsIf you can figure it out!If you can figure it out!

Document document Document document document!document!If you do prosecute, questions about your If you do prosecute, questions about your

evidence evidence willwill arise ariseEvery jurisdiction has its own requirements Every jurisdiction has its own requirements for acceptable evidencefor acceptable evidenceMaintain detailed and complete Maintain detailed and complete documentationdocumentation

Who…did what…when…and howWho…did what…when…and howSign and date every pageSign and date every page

Notify external agenciesNotify external agenciesPotential agencies—Potential agencies—

Local and national law enforcement (especially if Local and national law enforcement (especially if loss is financial)loss is financial)External security agencies (their experience is External security agencies (their experience is helpful)helpful)Malware expertsMalware experts

Coordinate with your legal representativeCoordinate with your legal representativeWhat kind of public notification?What kind of public notification?

Depends on your industryDepends on your industryDepends on whether customers were affectedDepends on whether customers were affected

Media attentionMedia attentionIf you’re a high-profile company, expect If you’re a high-profile company, expect attention!attention!Rarely desirable, often unavoidableRarely desirable, often unavoidableIncident response plan describes—Incident response plan describes—

Who’s allowed to interactWho’s allowed to interactWhat they’re allowed to sayWhat they’re allowed to sayWhether you notify media or wait for their callWhether you notify media or wait for their call

Speaking of notification…Speaking of notification…Consider how to spin it to your advantageConsider how to spin it to your advantageBeing honest, showing how you’re improving could Being honest, showing how you’re improving could actually win customersactually win customersDon’t lie about it, of course—reputation damageDon’t lie about it, of course—reputation damage

Compile and organize Compile and organize documentationdocumentationWhat was the attack?What was the attack?

How did we respond?How did we respond?WhoWhoWhenWhenWhyWhy

Organize, sign, then review with Organize, sign, then review with management and legal representativemanagement and legal representativeConsider dual sign-offs…increases likelihood Consider dual sign-offs…increases likelihood of evidence acceptanceof evidence acceptanceAll this absolutely critical if you suspect an All this absolutely critical if you suspect an insiderinsider

Assess damage and costAssess damage and costDirect and indirect costsDirect and indirect costsLoss of competitive edge (because of release Loss of competitive edge (because of release of confidential information)of confidential information)Legal costsLegal costsLabor costs—incident analysis and recoveryLabor costs—incident analysis and recoveryDowntime costsDowntime costs

Lost productivityLost productivityLost salesLost salesReplaced hardware, software, other propertyReplaced hardware, software, other property

Costs of updating physical securityCosts of updating physical securityConsequential damages—reputation, trustConsequential damages—reputation, trust

Review and updateReview and updateAfter you’ve cleaned it all up, review your After you’ve cleaned it all up, review your responseresponse

What went well?What went well?What needs improvement?What needs improvement?How will we get better next time?How will we get better next time?

Update policiesUpdate policiesCan we make them better?Can we make them better?Opportunities to streamline?Opportunities to streamline?Anything we need to strengthen?Anything we need to strengthen?

Consider new technologiesConsider new technologiesCan we improve our prevention mechanisms?Can we improve our prevention mechanisms?

More InformationMore Information

Learn moreLearn moreHandbook for Computer Security Incident Response Teamsby Moira J. West Brown, et alhttp://www.sei.cmu.edu/publications/documents/03.reports/03hb002.html

Forum of Incident Response and Security Teamshttp://www.first.org/

© 2006 Microsoft Corporation. All rights reserved.© 2006 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Steve RileySteve [email protected]@microsoft.comhttp://blogs.technet.com/http://blogs.technet.com/

sterileysteriley

incident response ”oh no, we’ve been hacked! now what do we do?” sec 211

Documents

incidents slide

business slide

execution slide

responsibilites slide

budget reaction slide

incident response plan

corp pc slide

security policy