life after the hack

Life after the hack

OSInetFrédéric G. MARAND (fgm)

2/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr

Topics• 1 Intro : setting the stage• 2 Snapshotting• 3 Maintaining presence• 4 Crisis communication• 5 Rebuild, don’t repair• 6 Using forensics tools• 7 Back online


1.1 Some fact checking first• In this room …• Who has been hacked already ?• Who feels ready to face a hacked server ?• Who actually has a contingency plan ?• Who read node 2365547 ?


1.2 Can you say that again ?

I.A.N.A.L.So be sure to get one !


1.3 Whence do I speak ?• Drupal org member since 2005 (fgm)• Drupal consultant, not a site building agency• Worked on fixing broken (in) sites since 2008• Auditing• Fixing technical flaws• Addressing intrusions / exploits

• Mostly Media and Government sites (.fr)• Provisional member of the Security Team


1.4 Setting the stage • 10:00 The daily scrum has just begun.• 10:01 Phones rings : someone noticed your site

has been defaced and is warning you• 10:02 Twitter and Reddit start buzzing• 10:05 Phones ring all over the place, with

journalists and the various C-level execs onthe other end, your mailbox is filling withwarnings• What is your next step ?


1.5 Get ready• Pad 1 : discovery log• all your work steps• all your findings / observations• with timestamps and numbers

• Pad 2 : remedies ideas• cross-refer pad 1 numbers• all your ideas for fixing the breach• all your ideas for further hardening


2.1 Forensic copy : why ?• First temptation : restore and resume• But you’re still vulnerable• So you need to diagnose

• Analyzing means modifying• So preserve the « crime scene »• Snapshot everything


2.2 Snapshots : pull the plug• Prevents interference• Shutdown

handlers, SIGPWR• Self-destructing

code on networkloss

• Easy on VMs

But…• Bare remote servers• Further data loss• Journaled FS• Databases

• Service interruption


2.3 Snapshots : what ?Not just the main DB

• Reverse Proxy logs• Web fronts• DB servers• File servers

And also…

• External logs (SaaS)• External transactions• IDS/firewall logs

The site may just be an attack vector


3.1 Maintaining presence 1

• Yes• Don’t tip off

hackers• Keep generating

short-term value

• No• Increasing

damage• Responsibility• Legal• Financial• Moral

As though intrusion had not been detected


3.2 Attacker workflowEvolved• Break in• Dig for gold• Implant zombie• Wait for implant

migration to archives• Activate• Profit

• Alt : Need for Speed• Use exploit ASAP• While it lasts• Usually least loss

• Alt : hidden steal• Valuable content• Identity data• Close the door


3.3 Maintaining presence 2

• Limited static site• Best with prior

work• Minimal subset• Possibly taken

from RP cache• Very little load : can

run on RP heads

• Working limited site• Alternate infra• Alternate tech

• Updates ?• Content created

during this step

Safe fallback mode


3.4 Maintaining presence 3When all else fails

• Social networks• Always there• Also authoritative for audience

• Still needs some preparation :• Accounts access• Include them in long-term communication


4.1 Communicating : from tech• Stakeholders• Chain up to CxO level in most cases• Prepare next steps, do not overreach

• Fear of reprisal ? Gag orders, SLAPP…• Protection• France : whistleblower protection (Sapin 2)• Italy : Dec. 385 01/09/93 sect 52bis (banks)• US : Anti-SLAPP• Many other countries have similar rules


4.2 : Communication : C-level• Legal counsel (first)• Crisis Management specialists• Law enforcement• EU countries typically have specialized

units for « cybercrime »• Other sites• On same server• On same network• Online business partners


4.3 Communication : privacy• In many cases personal data leaks• will happen, or...• unprovable they did not happen

• Operational constraints• Commerce : PCI/DSS (12 steps etc)• Health : (US) HIPAA Subtitle D E2.80.93

• Public image damage control• A french example


5.1 Rebuild : keep, rollback or ?• Restore and restart same ?• Still just as vulnerable

• Keep and fix ?• lots of time and effort reviewing• never completely trusted : not just Drupal

• Throw away ?• Event sites, past lines of biz, post-M&A...• Can a static version suffice ?• From RP snapshots : recent content


5.2 Rebuild : restore• Needs backups from before the hack• Do you know when it happened ?• Remember attacker workflow « wait »

• GFS, continuous incremental, 15 min ?• How much can you lose ?

• FLOSS solutions : Amanda, Bacula, custom• Unprepared emergency ?• Preproduction, CI builds...


5.3 Rebuild : sources + export• Easy and reliable, but assumes :• Code-driven development process• Reliable data export system in place• Flat content exports• Content + assets repositories

• Still need to add the fixes• Delay can be a problem on high-volume sites• Bulk handling, Incremental loading


5.4 Rebuild : other cases• Ad hoc « traditional » build process• Longer, less reliable• Too long to be a chance to fix the process

• From scratch• Too long in most cases• Do it as a complement after the fix• Not NOW


6 Forensics : switching hats


6.1 Forensics : first, think !• How did you become aware of hack ?• What did it take to succeed ?• Cast your net wide, think big• « Unlikely » vs « impossible »

• Priority :• Easiest attacks first• OWASP 10• GIYF : search your Pad 1 patterns


6.2 Forensics : keep in mind• /anything/ may be erased after success• But most of the time, not /everything/ will

• Anything you do leaves its own traces• Work on copies of the snapshots• You can restart from fresh copies anytime

• There maybe more than one exploit


6.3 Forensics : classics• Code files :• lax permissions• filesystem traversal issues• Remote payload execution by upload

• Nginx without extra hardening• .htaccess won’t do much good

• In-DB PHP• PHP module• Eval-uated code


6.4 Forensics : non-Drupal• Filesystem : • <user>/www-data outside /sites• www-data/www-data suspicious• x bit on files below docroot

• timestamps • outside sites/*/files = install• exploits > install

• meld with fresh build from sources• Also check outside docroot


6.5 Forensics : Drupal modules• Code signing/diffing :• Hacked!• D7 : md5check, file_integrity

• Finding DB PHP• QA (github)

• Misc• security_review


6.6 Forensics : DB• Quick wins :• users.email!= users.init• review roles, accounts with admin roles• On corp. sites, users.email domains• match users accounts with SSO data

• Diff DB snapshot with live• Especially menu_router :

file_put_contents, assert• Altova DatabaseSpy content compare


6.7 Forensics : sessions• Sessions should be in persistent storage• Remember when you pulled the plug• Were your sessions in Memcache ?

• sessions.timestamp vs users_field_data :created/changed/access/login• for intranets : sessions.hostname


6.8 Forensics : logs• You use off-site logs, right ?• SaaS : Loggly, Logmatic, Logsene, Logz.io,

Papertrail, Scalyr….• Remote ELK

• On site ?• dblog {watchdog}• syslog → follow the redirects• mongodb_watchdog

• Application/WS logs


6.9 Forensics : sleuth tools• Software• Guidance

Software : Encase• AccessData :

UltimateForensics Toolkit(FTK)

• Consider certifiedconsultants


7.1 Live again : restoring prod• Recheck Pad 1 findings vs new build• Usually, reset passwords. On D7 :• update users set pass = concat('ZZZ', sha(concat(pass, md5(rand()))));

• Prepare marketing/social copy


7.2 L8R : future-readiness


7.3 L8R : disaster prevention• Developer education on security• Security Team mailing list• https://twitter.com/drupalsecurity • https://www.drupal.org/security/rss.xml• http://crackingdrupal.com/


7.4 L8R : disaster prevention• Security process• Analyse sec. releases to understand fixes• Look for similar flaw in custom code• Take part in contrib for more expertise

• Quality process• Systematic peer code reviews• Code-driver maintenance + dev process• Automatic quality tools in CI• Contrib updates scheduling


7.5 Continuous improvement• You can’t improve what you don’t measure• Get time metrics from Pad 1

• Build contigency plan from Pad 2• Plan for periodic intrusion simulations