osint - sans.org · © 2019 david mashburn defensive osint hibp api access via curl 10 $ curl...

OSINTNot Just For Attackers

Applied Open Source Intelligence

David Mashburn, @d_mashburn

• SANS Certified Instructor

• GSE #157• Security mule• Family guy

What is OSINT?

Open-Source Intelligence (OSINT)• Harvesting and analysis of data from publicly available

resources

OSINT typically performed without directly interacting with the target

Going beyond the search• Key is in the analysis to transform from data to information to

intelligence

OSINT Resources

Numerous OSINT resources on the web

Web-based resources are often free,

but may be rate limited

Many offer APIs, which facilitate scripting & automation

May need to leverage a paid service to unlock features or to access more detailed information

OSINT as attack reconnaissance

Penetration testing frameworks have dedicated phases for reconnaissance activities

• PTES (Pentest Execution Standard)1

Attack models such the Lockheed-Martin Cyber Kill Chain2

include recon phase

Image source: https://www.oreilly.com/library/view/practical-cyber-intelligence/9781788625562/37a5852b-ef31-4b1e-a184-93ea7cf5cd75.xhtml

OSINT as a defensive resource

Blue needs to know what Red sees

The same OSINT tools used by pen testers, network admins, and security researchers can be leveraged by defenders

Let’s explore the application of OSINT by looking at a few defensive scenarios

Defensive OSINT use cases

Compromised Credentials

• Identities involved in some sort of data breach

Data Leakage

• Sensitive company information posted to the web

Encryption Certificates

• Track certificates without having network visibility

Social Media

• Basis for social engineering attacks, map relationships

Compromised credentials

Password re-use is an unfortunate reality

Our user communities may will reuse passwords across different sites

May lead to compromise even though the source was a third-party

Keeping Things Simple

A simple web lookup is a good start, but will not scale effectively

https://haveibeenpwned.com/

HIBP API access via curl

$ curl --insecure

https://haveibeenpwned.com/api/v2/breachedaccount/dXXXX@gmail.com | json_pp

"Description" : "In August 2017, a spambot by the name of <a

href=\"https://benkowlab.blogspot.com.au/2017/08/from-onliner-spambot-to-

millions-of.html\" target=\"_blank\" rel=\"noopener\">Onliner Spambot was ...

<a href=\"https://www.troyhunt.com/inside-the-massive-711-million-record-

onliner-spambot-dump\" target=\"_blank\" rel=\"noopener\">Inside the Massive

711 Million Record Onliner Spambot Dump</a>.",

"Domain" : "",

"Name" : "OnlinerSpambot", ...

"Domain" : "data4marketers.com",

"Description" : "In early 2015, a spam list known as <a

href=\"http://www.data4marketers.com/2015APRspecials.html\" target=\"_blank\"

rel=\"noopener\">SC Daily Phone</a> emerged containing almost 33M identities.

The data includes personal attributes such as names, ...

HIBP domain monitoring

An Open Source OSINT Framework

Recon-ng by Tim Tomes

Python-based framework

Performs data transformation, using input data ‘seeds’ to harvest new information

http://recon-ng.com

Automating the Search

Framework lookup for breached credentials (slow, rate limited)• recon/contacts-credentials/hibp_breach

Operational Efficiencies

How can we further automate OSINT?• What if we could script the framework?

The recon-ng framework has two scripting options• Provide a resource file via the –r option• Can use an interactive session to create the resource file using the record command

• Use the scripting interface recon-cli.py• Supports shell scripting

Merges automation with structured data collection and storage

Leveraging the Findings

HIBP service allows you to do regular monitoring of something that you can’t control

Automation and different output formats make it possible to ingest this data into your SIEM

HIBP will not provide the actual passwords

A slightly grayer path

Choose your own adventure, but you can find resources beyond the clearly white hat arena

Paid service with tiers, API access is only via paid service

Staying on top of things

HIBP is focused on credentials

Checks for potential credential dumps by consuming the @dumpmon Twitter feed

Open source project

https://github.com/jordan-wright/dumpmon

What exactly is being monitored?

Beyond credential dumps

Data loss is more significant than password loss

Challenging to track and control data in authorized systems

What about our data where it doesn’t belong?

Pastebin

Pastebin is one of the better-known paste sites

Provides API access and email alerts to monitor for keywords

Pro version (paid)

Monitoring across multiple paste sites

Michael Bazzell runs a fantastic website for OSINT

IntelTechniques.com

One of the tools is a search across 57 paste sites

Leverages Google Custom Search

Google Custom Search Engines

Create your own CSE at https://cse.google.com/cse/

OpSec may be a significant consideration

How sensitive are the keywords you are monitoring?

Google Alerts

Create alerts for specific searches

Alerts delivered via email

How much do you want to entrust to any third party?

Certificate Tracking

Encrypted traffic is the norm

Certificate issuance is free and requires only proof of domain control

Can you identify certificates in use that aren’t hitting your sensors that use your domains?

Certificate Transparency

“… provides a way for every certificate issued by any publicly trusted CA to be publicly logged, monitored, and audited”

This means that we can monitor in near real-time certificates that have been issued for your domains

Certificate Transparency Sources

Many options for online searching of CT logs

• Digicert SSL tools, Entrust Datacard, Cas generally

Can also download the CT logs directly and ingest

Other Certificate Monitoring options

Censys.io and crt.sh are 0ther options for tracking certificates

Web interface for simple searching

Censys.io API for automation

Search output

The “rest of the web”

What about the layer 8 connection?

Social media mining

Oversharing on social media has significant personal implications

Social media goes far beyond the personal realm

Who is claiming to be part of our organization?

Understanding the Social Media surface area

Who looks interesting in this list?

Social media helps make the connections with suggestions for who might be of interest to you

Profile information allows relationships and lines of reporting to be determined

Who needs that extra awareness training?

Nice to meet you

So about what do we know about Roma Blaser?

Reverse image search may be a good starting point*

Conclusions

OSINT has wide applicability for defenders

Helps find things where they don’t belong

Provides us with the same knowledge as an external party

Can help deal with some issues that are difficult for our typical logging and visibility setups

Questions

The floor is open

for questions

Domain Shadowing

Domain registrar accounts compromised

Attacker creates subdomains to support exploit kits

Excellent blog from Cisco Talos on domain shadowing

OSINT unshadows?

GoDaddy specifically named as a targeted environment

Blog post focused on detecting at scale

What about focusing on your domains?

GoDaddy DNS behavior

Automatically updates the zone serial number based on the date of the last change to the name server

Simple detect for zone changes

Automate the change monitoring

GoDaddy specifically named as a targeted environment

Blog post focused on detecting at scale

What about focusing on your domains?

osint - sans.org · © 2019 david mashburn defensive osint hibp api access via curl 10 $ curl...

Documents

127964_061311 europe osint guidance

spectrum osint monitoring system

osint council decision

61438_101208 europe osint guidance

osint and beyond

offensive osint

osint report 3/2010 - startseite - css ·...

styleclub curl - 株式会社アリミノ curl for curl for...

great overview - social osint

old.iss.itold.iss.it/binary/hibp/cont/iop_meto_04_ar_rev._05_del_26.10.09.pdf ·...

the future of osint - wikileaks

open-source intelligence osint osint - owaspopen-source...

quarterly marketplace report north curl curl 3rd quarter...

blackmagic open source intelligence osint

emerging trends and issues open source intelligence (osint...

2- osint 15-01

open source intelligence (osint)

osint overview 26 mar 2015

scientific approach on osint training program development...

writing sample_isis osint report