After the After the BreachBreachDennis SchmidtDennis Schmidt

Director, Office of Information Systems Director, Office of Information Systems HIPAA Security OfficerHIPAA Security Officer

UNC School of MedicineUNC School of Medicine

OMG, We have a breach!OMG, We have a breach!

In late July, 2009, UNC Information Technology employees discovered that a server which contained sensitive information on 180,000 research subjects, including 114,000 Social Security Numbers, had been the target of a computer hack in 2007. The compromised server was taken down and the data on the server were removed.

Incident DiscoveryIncident Discovery

OIS receives call from departmental OIS receives call from departmental server admin reporting that a server server admin reporting that a server would not reboot after power failure. would not reboot after power failure.

OIS technician suspects virus and OIS technician suspects virus and performs full virus scan on machine. Virus performs full virus scan on machine. Virus detected. detected.

Technician is told by department that Technician is told by department that server may contain sensitive information.server may contain sensitive information.

Server turned over to OIS Information Server turned over to OIS Information Security for forensic analysis. Security for forensic analysis.

Forensic AnalysisForensic Analysis-- A Long, Painful -- A Long, Painful

Process --Process -- Verification – Verify the incident occurred

Interview the SysAdmins and other users involved

Examine system and application logs (Snort, Tipping Point, etc.)

Check volatile information using forensic tools System Description

Physical observation, forensic tools Interview SysAdmins and users, determine use Hardware and software system characteristics Hard disk geometry

Forensic Analysis (cont.)Forensic Analysis (cont.)

Evidence Collection All available computer information

(volatile and non-volatile) is collected and transferred to external media or forensic workstation to perform analysis tasks.

Data must be collected in order of volatility and data integrity safeguarded by hash signature, MD5

Forensic Analysis (cont.)Forensic Analysis (cont.)

Timeline Creation & Analysis – Use time-stamps from internal and external sources to correlate into timeline that traces back the system activity.

Media Analysis – Thorough examination of the media layers (physical, data, metadata, file system and file name) searching for evidence.

Forensic Analysis (cont.)Forensic Analysis (cont.)

Data Recovery – extracting unallocated data in order to recover any deleted files. File fragments could represent a critical piece of information relevant to the case

String Search – searching for specific strings or keywords contained inside files to reveal useful information relevant to the case.

Reporting -- detailed report(s) of the forensic process explaining the evidence found, together with the techniques and methodology used.

 

Houston, We have a Houston, We have a problem!problem!

Virus/worm/trojan infection for 2 Virus/worm/trojan infection for 2 yearsyears

26 files containing over 500,000 26 files containing over 500,000 recordsrecords 180,000 unique research subjects180,000 unique research subjects 114,000 Social Security Numbers114,000 Social Security Numbers

Qualys Scan ResultsQualys Scan Results

But, did they get But, did they get anything?anything?

When did compromise occur? Is it still When did compromise occur? Is it still active?active?

When were the sensitive files put on When were the sensitive files put on the machine? When were they last the machine? When were they last accessed?accessed?

Was it during the compromise window?Was it during the compromise window? Is there any corroborating evidence on Is there any corroborating evidence on

the network of file downloads from the the network of file downloads from the server?server?

The Antivirus Dilemma The Antivirus Dilemma

Full virus scan changes the last accessed Full virus scan changes the last accessed time on time on everyevery file. file.

It now becomes impossible to determine if It now becomes impossible to determine if the malware actually accessed specific the malware actually accessed specific files. files. e.g., If compromise occurred one week ago, e.g., If compromise occurred one week ago,

and last access of sensitive file was one month and last access of sensitive file was one month ago, you know the data was not likely accessed ago, you know the data was not likely accessed by the malware. by the malware.

If virus scan was done yesterday, you no longer If virus scan was done yesterday, you no longer know when the file was last accessed. know when the file was last accessed.

No Smoking GunNo Smoking Gun

There was no way to prove that data There was no way to prove that data on the server was accessed on the server was accessed inappropriately.inappropriately.

And… there was no way to prove And… there was no way to prove that data on the server was that data on the server was notnot accessed inappropriately.accessed inappropriately.

The doors were unlocked and people The doors were unlocked and people were in the house, but we couldn’t were in the house, but we couldn’t prove that they stole anything.prove that they stole anything.

Second OpinionSecond Opinion

Magnitude of potential breach Magnitude of potential breach warranted additional opinionswarranted additional opinions

ITS Security conducted parallel ITS Security conducted parallel investigation to verify or refute investigation to verify or refute initial findingsinitial findings

Additional corroborating data Additional corroborating data searchedsearched Network traffic logs (only last 90 days)Network traffic logs (only last 90 days)

Notification is not an IT Notification is not an IT DecisionDecision

University Counsel makes final University Counsel makes final recommendation based on inputs from:recommendation based on inputs from: IT Security (OIS & ITS)IT Security (OIS & ITS) University RelationsUniversity Relations UNC Health Care UNC Health Care

Communications/MarketingCommunications/Marketing UNC Health Care CounselUNC Health Care Counsel HIPAA Privacy and HIPAA Security OfficersHIPAA Privacy and HIPAA Security Officers

How do we notify How do we notify 180,000 people?180,000 people?

Is their address current? Do we Is their address current? Do we have an address?have an address?

Are they still alive?Are they still alive? Who writes the letters?Who writes the letters? Who addresses the envelopes? Licks Who addresses the envelopes? Licks

the stamps?the stamps? Who handles phone calls from Who handles phone calls from

concerned recipients?concerned recipients?

The Notification ProcessThe Notification Process

UNC Hired Rust Consulting to assistUNC Hired Rust Consulting to assist Consultation servicesConsultation services Mailed notification lettersMailed notification letters Established and staffed Call CenterEstablished and staffed Call Center Responded to calls; referred problem Responded to calls; referred problem

calls to UNCcalls to UNC Received 4,144 callsReceived 4,144 calls 450 calls referred to UNC450 calls referred to UNC

Technical ResponseTechnical Response

Major concern: Uncontrolled server Major concern: Uncontrolled server proliferationproliferation

Determine scope of problemDetermine scope of problem Protect high risk machines firstProtect high risk machines first Develop long term strategy to Develop long term strategy to

mitigate riskmitigate risk

The Scope of the The Scope of the ProblemProblem

500+ machines with server OS’s on 500+ machines with server OS’s on SOM networkSOM network

2200 machines running a service2200 machines running a service 2068 File Server / File Services2068 File Server / File Services 1989 Remote Access / Remote 1989 Remote Access / Remote

ManagementManagement 762 Web Servers762 Web Servers 194 Database Servers194 Database Servers

Manual Data CollectionManual Data Collection

Mandatory self reporting of serversMandatory self reporting of servers 433 servers reported433 servers reported 98 server admins98 server admins 47 different OS flavors and 47 different OS flavors and

versionsversions Qualys scans on all servers reporting Qualys scans on all servers reporting

sensitive information (200 machines)sensitive information (200 machines)

Long Range StrategyLong Range Strategy

IT Simplification and Security RFP (Dell)IT Simplification and Security RFP (Dell) Develop Plan for streamlining IT resources Develop Plan for streamlining IT resources

in SOMin SOM Develop strategic virtualization architectureDevelop strategic virtualization architecture Develop enterprise storage architectureDevelop enterprise storage architecture Develop security umbrella to cover Develop security umbrella to cover

centralized operationcentralized operation Goal: Provide robust central Goal: Provide robust central servicesservices

that will get end users out of that will get end users out of serverserver businessbusiness

Recovery from the Recovery from the breachbreach

Moved data to centrally managed serversMoved data to centrally managed servers Database encrypted behind hardware firewallDatabase encrypted behind hardware firewall All working files encrypted with PGP Net ShareAll working files encrypted with PGP Net Share All machines, including desktops, scanned with All machines, including desktops, scanned with

Qualys Qualys Well defined procedures documented, Well defined procedures documented,

approved by IRBapproved by IRB Two person rule for manual movement of data Two person rule for manual movement of data

filesfiles Update software to automate processesUpdate software to automate processes

How much did it cost?How much did it cost? Average breach reportedly costs $204 per nameAverage breach reportedly costs $204 per name

$204 X 180,000 = $36.7 Million!$204 X 180,000 = $36.7 Million! Other references state that a major breach Other references state that a major breach

costs an organization a costs an organization a minimumminimum of $1 Million. of $1 Million. Postage alone cost $75,000.Postage alone cost $75,000. Rust Consulting cost $260,000Rust Consulting cost $260,000 Thousands of person hours spent on the projectThousands of person hours spent on the project

OIS Security, ITS Security, OUC, P&A, HIPAA OIS Security, ITS Security, OUC, P&A, HIPAA Privacy, senior leadership, etc. etc. etc.Privacy, senior leadership, etc. etc. etc.

Lessons LearnedLessons Learned

Implementation of IT Governance is Implementation of IT Governance is criticalcritical

Decentralized server environment is high Decentralized server environment is high riskrisk

New procedures for virus investigations New procedures for virus investigations involving sensitive datainvolving sensitive data Disconnect from networkDisconnect from network Do not shut downDo not shut down Do not perform virus scanDo not perform virus scan Notify IT SecurityNotify IT Security

Questions?Questions?