splunk at corporation service company at corporation service company ... • internal and external...

Splunk at

Corporation Service Company

Scott Plichta Chief Information Security Officer

Background – Scott Plichta

•  Chief Information Security Officer •  Development background •  Over 20 years developing/operating SaaS solutions •  Startups +


•  Established in 1899 •  With 2,000+ employees worldwide, CSC: –  Services more than 180,000 businesses,

including 90% of the Fortune 500® –  Supports nearly 10,000 law firms and the largest

U.S. law firms –  Provides Solutions to more than 3,000 financial institutions –  Protects the top three and more than half of the 100 Best

Global Brands (Interbrand ®)

Our Products & Services

We manage more than 1.5 million corporate domain names, search the internet for brand abuses and provide online security assets including SSL digital certificates and anti-phishing services.

We help the general counsel efficiently handle and store sensitive data including service of process, contracts, and M&A documents.

Ranked #1 for Matter Management and Entity Management by The New York Law Journal

We enable banks and commercial lenders to conduct UCC searches and filings, prepare and record real estate documents, and manage titles and real property.

Information Technology at CSC

•  Service business first and foremost •  High availability Software as a Service component (SaaS) •  Internal and external apps •  Security is a top priority •  Heterogeneous environment

–  .Net / SQL Server –  Java / Oracle / SOLR / Hadoop –  Cold Fusion, PHP, Salesforce.com integration

Operational Environment

•  Splunk environment -  About 400 distinct sourcetypes. -  400GB App ingest, 1 TB security ingest -  Production and pre-prod

•  Security team structure –  2 Operational –  2 Senior engineers

•  Part of IT Operations –  ~ 100 people

Four Scenarios

1.  Audit privileged access 2.  Chasing ransomware 3.  Phishing site targeting payments 4.  APDEX customer experience

Auditing Privileged Access 1.  Audit that only active employees

utilize privileged access (monthly) 2.  Audit that appropriate individuals

access (quarterly) •  Challenge:

–  Time consuming à aggregate manual

–  Tedious à manual correlation –  Not consistently run à have to

remember

Privileged Access: •  Allow user to modify client

permissions •  Password reset •  Perform functions on

behalf of clients •  Only allowed by specific

active employees

Solution Design

•  Evaluate all access –  Do you have access? –  Are you in the database? –  Are you still employed? –  Need to know admin functions

•  Function tagged (MM_Admin) –  Pre-assess threat – –  And NOW we can alert on it

Tools

•  Person DB –  Database of all employees and contractors

•  Splunk access logs –  Tag assigned to all admin functions

(tag=MM_adminFunction)

DB Connect

•  Active connection to production DB

•  Allows lookup against active DB in real-time from Splunk

•  Easy configuration and setup

•  Lookup command syntax to access

The Results Output

•  All relevant employee data: –  Manager –  Full Name –  Active/Term – Term date –  Last Access –  Green – Yellow - Red

Green/Yellow/Red indicator

Breaking Down the Query sourcetype="prod-MM-perf" tag=MM_adminFunction | lookup local=1 Employees USERID as MM_UserName OUTPUT NAME, SUP_NAME, STATUS_CODE,TERMINATION_DATE | fillnull VALUE=null NAME,SUP_NAME,TERMINATION_DATE,STATUS_CODE | eval Date=strftime(_time,"%Y-%m-%d") |stats max(Date) as LastAccess, count by SUP_NAME, NAME, MM_UserName, MM_uri, STATUS_CODE, TERMINATION_DATE | eval flag=if(STATUS_CODE="null","Yellow", if(STATUS_CODE="A","Green",if(TERMINATION_DATE="null","Yellow",if(TERMINATION_DATE>LastAccess, "Green", "Red"))) )

Get values

Assign Alert level

Cleanup

Link to employee DB

Calc last access

Access Audits Simplified

•  Alert –  Any non-active employee access realtime

•  Monthly audit of usage –  Operations Manager reviews list of access

•  Quarterly management audit –  Email manager of employees using functions

Value Review

•  Challenges: –  Time consuming à

aggregate manual –  Tedious à manual

correlation –  Not consistently run à

have to remember

•  Solution: –  Splunk does all aggregation –  Splunk highlights anomalies –  Splunk sends finalized report

Three Scenarios

1.  Validating privileged access 2.  Chasing ransomware 3.  Phishing site targeting payments 4.  APDEX customer experience

Ransomware

•  Blocks access on infected computer •  Demands ransom to unlock •  Payment *might* release access •  Delivery mechanism vary

•  CryptoLocker (and variants) –  Encrypt all files on all drives –  Infect other computers –  Leaves ransom note

That “Oh, Crap” Moment

Malware Remediation

•  Isolate “the patient(s)” •  Fix the “the patient(s)” •  How did it get in? •  What’s the link? •  Who else has the link? (vector) •  Who else has clicked the link?

Isolate/Fix

•  Disconnect shared drives •  Find the program –  but its gone by now…

•  Assess the damage •  Start the restore

Ø  Let’s go find patient zero

Find the Source

•  We know: –  User IDs of encrypted files on shared drives

•  We assume: –  Payload/link was delivered by e-mail

•  What does Splunk know: –  Mail header logs –  Proxy logs

Check Email… Email

Senders Each Infected

User

Every user received this email Same process for proxy logs

Find the Email

sourcetype="MSExchange:2010:MessageTracking” !recipients="*user1*” OR recipients="*user2*" OR recipients="*user3*" OR recipients="*user4*" ! !|eval user1=if(searchmatch(”user1"),1,0)!

|eval user2=if(searchmatch(”user2”),1,0)!|eval user3=if(searchmatch(”user3"),1,0) !|eval user4=if(searchmatch(”user4"),1,0) ! !| stats count,sum(user1),sum(user2), ! sum(user3), sum(user4) by !

sender_domain,sender!

Specify email

Show me a table of senders and receivers

Mark each email based on who received it

Select users

Who Else Got This email?

•  Pull a sample email •  Search by subject and time sourcetype="MSExchange:2010:MessageTracking" message_subject="*Statament*" OR message_subject="*August 2014*” OR sender=“[email protected]”

| stats count by message_subject, recipients

•  Send copy to spam protection •  Remove the emails from Exchange

Who Clicked the Link?

•  So who has been to the link? •  Search the proxy for all references to all URL’s in email

Ø Now we have users to contact

What if This Didn't Work

•  Technique works –  Splunk sources à more forensics

•  Not always email –  Could be links – follow the proxy

•  Check proxy against threat lists

Chasing Ransomware

•  Without Splunk –  4 day event, with re-infection –  Multiple people, many hours

•  With Splunk –  Multiple hour –  Small staff –  Minimal WIP

Four Scenarios


Phishing Sites

•  Do you take payments online? •  Do you worry about phishing? •  Do hacker have good separation of duties? •  Do hackers test in production?

incorporate.com

lncorporate.com

•  Marketing monitors referrers on webservers

sourcetype="prod_apache_logs" source="/opt/httpd/logs/www.incorporate.com-access.log" | rex field=http_referrer "http[s]://(?<ref_hostname>[^/]+)"| timechart count by ref_hostname

Lncorporate.com •  Is something strange here?

0 2000 4000 6000 8000 10000 12000

incorporate.com

lncorporate.com

cscglobal.com

google.com

advendor1.com

advendor2.com

Incorporate.com Referrers

5/6/14

5/5/14

5/4/14

5/3/14

5/2/14

5/1/14

•  lncorporate.com à Lncorporate.com

Fix the Problem & Monitor

•  After a few whois/dns searches –  This is not right

•  Contacted CSC’s Take Down service –  DNS, website offline in 30 minutes –  Phone number disabled within 45 minutes

•  New search to detect phishing attacks –  Scan for new referrers and alert [lookup]

Fraudulent Site Process

•  Phishers –  Develop in production –  Use your images

•  Everyone should: –  Baseline your referrers –  Check your actual referrer

Lncorporate

•  Conclusion: –  Use Splunk to analyze web traffic –  Anomaly detected and remediated

•  EVERYONE is in security –  Problem fixed while still in development

Four Scenarios


User Experience

“Know before the customer does”

Monitoring is all green

Have them screen share Get their

credentials

Performance is awful

Report won’t finish 50 Employees

crawling Works on test acct

The Monitoring Dilemma What level of performance are

my users seeing today?

•  Sitescope™ – How’s my server? •  BAC™ – Synthetic read-only transactions •  Gomez™ – Synthetic read-only transactions over the

Internet But what do our users really see (and why…) …

Apdex

•  Application Performance Index •  Sample Range –  Satisfied = 0-1.5s –  Tolerating = 1.5s – 7.0s –  Frustrated = >7.0s

•  Your ranges may vary by work type

Approach

•  Gather render times •  Bucket according to Apdex •  Monthly reports/reviews by app/page •  Trending reports •  Baseline •  Alerts for abnormal behavior

•  | rangemap field=microseconds Sa3sfied=0-‐1500000 Tolera3ng=1500001-‐7500000 Frustrated=7500001-‐999999999999 default=Unknown …

•  ….|chart values(pct) over Applica3on by range

41

Satisfied = 0-1.5s Tolerating = 1.5s – 7.0s Frustrated = >7.0s

Apdex

Application Real Time Dashboard

42

Splunk & Apdex

sourcetype="prod_apache_logs" NOT file=*.js NOT file=*.jpg httpstatus=200 | rangemap field=microseconds Satisfied=0-1500000 Tolerating=1500001-7500000 Frustrated=7500001-999999999999 default=Unknown | eval apdexScore=if(range=="Satisfied", 1, if(range=="Tolerating", 0.5, 0)) | streamstats count as Num, sum(apdexScore) as apScore window=500 global=f | eval apdex=apScore/Num | timechart avg(apdex)

Apdex =Satisfied+ Tolerating

2!

"#

$

%&

SampleSize

Splunk & Apdex

•  Drill down to page level •  Correlate to user – customer data –  DB lookup to bring in customer stats –  Proactive customer service

•  Performance over time •  Historic or real time metric

What Can I Do Tonight? •  1. Privileged access

–  Employee database linked to Splunk –  Compare access vs. HR records –  Create and automate it

•  2. Malware Infections –  Don’t stop with a re-image – Keep going –  Find all possible infections

•  3. Internet sites –  No reason not to look for invalid referrers

•  4. Apdex –  Score your site against Apdex –  Baseline and trend

Splunk at


Scott Plichta [email protected]

Splunk at


Appendix

What Can I Do Tonight?

1. Privileged access Employee database linked to Splunk Compare access vs. HR records Create and automate it

2. Malware Infections Don’t stop with a re-image – Keep going Find all possible infections

3. Internet sites No reason not to look for invalid referrers

4. Apdex Score your site against Apdex Baseline and trend

Put Totals in Your Searches?

Command: appendpipe Let’s count all our production sourcetypes sourcetype=prod_*

|stats count as Lines by sourcetype

|appendpipe [stats count as Types, sum(Lines) as Lines |eval sourcetype="Totals ...." ] | sort -Types

Recipes

•  Can’t remember a sourcetype name: •  |metadata type=sourcetypes | search

sourcetype=“prod*”

•  Count by ip/user/host/dest •  sourcetype=“prod_apache”| stats count by c_ip |

sort 20 –count

•  Anonymizing Data •  eval NAME=

substr(NAME,1,5). substr("**************",1,len(NAME)-5)

51

Security office hours: 11:00 AM – 2:00 PM @Room 103 Everyday

Geek out, share ideas with Enterprise Security developers Red Team / Blue Team - Challenge your skills and learn new tricks Mon-Wed: 3:00 PM – 6:00 PM @Splunk Community Lounge Thurs: 11:00 AM – 2:00 PM

Learn, share and hack

Birds of a feather- Collaborate and brainstorm with security ninjas Thurs: 12:00 PM – 1:00 PM @Meal Room

splunk at corporation service company at corporation service company ... • internal and external...

Documents