osint tools for security auditing with python
TRANSCRIPT
![Page 1: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/1.jpg)
José Manuel Ortega
@jmortegac
PYCONES 7-9 OCT 2016
![Page 2: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/2.jpg)
![Page 3: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/3.jpg)
https://github.com/jmortega/osint_tools_security_auditing
![Page 4: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/4.jpg)
▪ OSINT introduction ▪ Server information(Censys,Shodan) ▪ OSINT tools developed with python ▪ Geolocation,Metadata ▪ Twitter,Footprinting,FullContact
![Page 5: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/5.jpg)
▪ Define a specific target and data you wish to obtain
▪ Technical-Accounts,servers,services,software ▪ Social-Social Media,Email,Photos ▪ Physical-Address,Home IP address,Footprinting ▪ Logical-Network,Operational intelligence
![Page 6: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/6.jpg)
▪ GeoLocation ▪ IP address ▪ Email address ▪ Telephone Number ▪ Usernames in social network profiles ▪ Metadata information from images ▪ Server information & vulnerabilities
![Page 7: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/7.jpg)
![Page 8: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/8.jpg)
▪ https://www.censys.io/api/v1/view/ipv4/<ip_address> ▪ https://www.censys.io/api/v1/view/websites/<domain>
![Page 9: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/9.jpg)
![Page 10: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/10.jpg)
![Page 11: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/11.jpg)
![Page 12: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/12.jpg)
![Page 13: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/13.jpg)
▪ Checking data with ip address ▪ https://www.shodan.io/host/144.76.246.116
![Page 14: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/14.jpg)
![Page 16: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/16.jpg)
▪ https://bitbucket.org/LaNMaSteR53/recon-ng ▪ Open Source OSINT toolkit written in python ▪ Actively maintained ▪ Uses modules and saves all recollected
information in databases
![Page 17: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/17.jpg)
▪ dnspython - http://www.dnspython.org/
▪ dicttoxml - https://github.com/quandyfactory/dicttoxml/
▪ jsonrpclib - https://github.com/joshmarshall/jsonrpclib/
▪ lxml - http://lxml.de/
▪ slowaes - https://code.google.com/p/slowaes/
▪ XlsxWriter - https://github.com/jmcnamara/XlsxWriter/
▪ Mechanize
▪ PyPDF2
▪ sqlite3
![Page 18: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/18.jpg)
![Page 19: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/19.jpg)
![Page 20: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/20.jpg)
![Page 21: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/21.jpg)
![Page 22: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/22.jpg)
![Page 23: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/23.jpg)
![Page 24: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/24.jpg)
![Page 25: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/25.jpg)
▪ https://github.com/laramies/theHarvester
![Page 26: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/26.jpg)
![Page 27: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/27.jpg)
▪ httplib
▪ socket
▪ requests
▪ shodan
![Page 28: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/28.jpg)
![Page 29: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/29.jpg)
▪ pip install osrframework
▪ Developed in python 2.7
▪ Integrates with maltego transforms
▪ https://pypi.python.org/pypi/osrframework/0.13.2
▪ https://github.com/i3visio/osrframework
![Page 30: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/30.jpg)
▪ BeautifulSoup
▪ Requests
▪ Mechanize
▪ pyDNSresolving name servers
▪ python-whoisto recover the whois info from a domain
▪ tweepyfor connecting with Twitter API
▪ Skype4Py for connecting with Skype API
▪ Python-emailahoyfor checking email address
▪ Multiprocessingimport Process, Queue, Pool
![Page 31: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/31.jpg)
![Page 32: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/32.jpg)
![Page 33: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/33.jpg)
![Page 34: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/34.jpg)
![Page 35: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/35.jpg)
![Page 36: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/36.jpg)
![Page 37: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/37.jpg)
![Page 38: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/38.jpg)
![Page 39: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/39.jpg)
Source Location Notes
abuse.ch http://www.abuse.ch Various malware trackers.
AdBlock https://easylist-
downloads.adblockplus.org/easylist.txt
AdBlock pattern matches
AlienVault https://reputation.alienvault.com AlienVault’s IP reputation database.
Autoshun.org http://www.autoshun.org Blacklists.
AVG Site Safety Report http://www.avgthreatlabas.com Site safety checker.
Bing http://www.bing.com Scraping but future version to also use API.
Blocklist.de http://lists.blocklist.de Blacklists.
Checkusernames.com http://www.checkusernames.com Look up username availability on popular sites.
DNS Your configured DNS server. Defaults to your local DNS but can be configured to
whatever IP address you supply SpiderFoot.
DomainTools http://www.domaintools.com
DroneBL http://www.dronebl.org
Facebook http://www.facebook.com Scraping but future version to also use API.
FreeGeoIP http://freegeoip.net
Google http://www.google.com Scraping but future version to also use API.
Google+ http://plus.google.com Scraping but future version to also use API.
Google Safe Browsing http://www.google.com/safebrowsing Site safety checker.
LinkedIn http://www.linkedin.com Scraping but future version to also use API.
malc0de.com http://malc0de.com Blacklists.
malwaredomainlist.com http://www.malwaredomainlist.com Blacklists.
![Page 40: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/40.jpg)
Source Location Notes
malwaredomains.com http://www.malwaredomains.com Blacklists.
McAfee SiteAdvisor http://www.siteadvisor.com Site safety checker.
NameDroppers http://www.namedroppers.org
Nothink.org http://www.nothink.org Blacklists.
OpenBL http://www.openbl.org Blacklists.
PasteBin http://www.pastebin.com Achieved through Google scraping.
PGP Servers http://pgp.mit.edu/pks/ PGP public keys.
PhishTank http://www.phishtank.org Identified phishing sites.
Project Honeypot http://www.projecthoneypot.org Blacklists. API key needed.
RIPE/ARIN http://stat.ripe.net/
Robtex http://www.robtex.com
SANS ISC http://isc.sans.edu Internet Storm Center IP reputation database.
SHODAN http://www.shodanhq.com API key needed.
SORBS http://www.sorbs.net Blacklists.
SpamHaus http://www.spamhaus.org Blacklists.
ThreatExpert http://www.threatexpert.com Blacklists.
TOR Node List http://torstatus.blutmagie.de
TotalHash.com http://www.totalhash.com Domains/IPs used by malware.
UCEPROTECT http://www.uceprotect.net Blacklists.
VirusTotal http://www.virustotal.com Domains/IPs used by malware. API key needed.
Whois Various Whois servers for different TLDs.
Yahoo http://www.yahoo.com Scraping but future version to also use API.
Zone-H http://www.zone-h.org Easy to get black-listed. Log onto the site in a
browser from the IP you’re scanning from first and
enter the CAPTCHA, then it should be fine.
![Page 41: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/41.jpg)
▪ Python 2.7
▪ BeautifulSoup
▪ DNSPython
▪ Socks
▪ Socket
▪ SSL
▪ CherryPy
▪ M2MCrypto
▪ Netaddr
▪ pyPDF
![Page 42: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/42.jpg)
from bs4 import BeautifulSoup, SoupStrainer
![Page 43: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/43.jpg)
![Page 44: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/44.jpg)
![Page 45: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/45.jpg)
![Page 46: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/46.jpg)
▪ PDFPyPDF2,PDFMiner
▪ ImagesPillow,pyexiv2(python 2.7),gexiv2(python 3)
![Page 47: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/47.jpg)
import geoip2
import geoip2.database
http://dev.maxmind.com/geoip/geoip2/geolite2/
![Page 48: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/48.jpg)
▪ Orb(Python 2.x)
• https://github.com/epsylon/orb
• python-whois - Python module for retrieving WHOIS information
• python-dnspython - DNS toolkit for Python
• python-nmap - Python interface to the Nmap port scanner
• InstaRecon(Python 2.x)
• https://github.com/vergl4s/instarecon
• dnspython,ipaddress
• ipwhois,python-whois
• requests,shodan
![Page 49: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/49.jpg)
![Page 50: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/50.jpg)
![Page 51: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/51.jpg)
▪ BeautifulSoup for parsing web information
▪ Requests,urllib3 for synchronous requests
▪ Asyncio,aiohttp for asynchronous requests
▪ Robobrowser,Scrapy for web crawling
▪ PyGeoIP,geoip2,geojson for GeoLocation
▪ python-twitter,tweepy for connecting with twitter
▪ Shodan for obtain information for servers
▪ DNSPython,netaddr for resolving ip address
![Page 52: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/52.jpg)
![Page 53: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/53.jpg)
python tinfoleak.py pycones -i -s --sdate 2016/01/01 --hashtags --mentions --meta --media [d] --geo
GEOFILE --top 10 -o report.html
![Page 54: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/54.jpg)
![Page 55: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/55.jpg)
▪ import tweepyTwitter API library for Python
▪ from PIL import Image, ExifTags, ImageCmsmetadata from
images
▪ import pyexiv2metadata from images
▪ import urllib2requests
▪ from OpenSSL import SSL
▪ from jinja2 import Template, Environment,
FileSystemLoaderreport
![Page 56: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/56.jpg)
![Page 57: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/57.jpg)
![Page 58: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/58.jpg)
![Page 59: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/59.jpg)
![Page 60: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/60.jpg)
![Page 61: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/61.jpg)
▪ We know we have a valid email address ▪ What other profiles are associated with this
address? ▪ Go to fullcontact.com for an API key…..
![Page 62: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/62.jpg)
![Page 63: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/63.jpg)
![Page 64: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/64.jpg)
![Page 66: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/66.jpg)
![Page 67: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/67.jpg)
▪ https://sourceforge.net/projects/spiderfoot
▪ http://www.edge-security.com/theharvester.php
▪ https://developer.shodan.io/api
▪ http://www.clips.ua.ac.be/pattern
▪ http://www.pentest-
standard.org/index.php/PTES_Technical_Guidelines#OSINT
▪ http://www.vicenteaguileradiaz.com/tools
▪ https://github.com/automatingosint/osint_public
▪ http://www.automatingosint.com/blog/
![Page 68: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/68.jpg)
![Page 69: OSINT tools for security auditing with python](https://reader030.vdocuments.us/reader030/viewer/2022013123/587059a41a28aba2118b62bd/html5/thumbnails/69.jpg)
Thanks!
@jmortegac
AMSTERDAM 9-12 MAY 2016