osint – open source intelligence -...

Pattern Recognition and Applications Lab

Universitàdi Cagliari, Italia

Dipartimento di Ingegneria Elettrica

ed Elettronica

OSINT – Open Source Intelligence

Ing. Davide Ariu

[email protected]

May 18, 2017

http://pralab.diee.unica.it

Intelligence Defined

Simply defined, intelligence is information that has been analyzed

and refined

so that it is useful to policymakers in making decisions —specifically, decisions about potential threats to national security

1. Intelligence is a product that consists of information that has been refined to meet the needs of policymakers

2. Intelligence is also a process through which that information is identified, collected, and analyzed.

3. And intelligence refers to both the individual organizations that shape raw data into a finished intelligence product for the benefit of decision makers, and the larger community of these organizations.

https://www.fbi.gov/about-us/intelligence/defined


Intelligence Cycle

The process of developing unrefined data into polished intelligence for the use of policymakers

The process has a circular nature, although movement between the steps is fluid.

Intelligence uncovered at one step may require going back to an earlier step before moving forward.

6 steps:1. Requirements

2. Planning and Direction

3. Collection

4. Processing and Exploitation

5. Analysis and Production

6. Dissemination

https://www.fbi.gov/about-us/intelligence/intelligence-cycle


Intelligence Collection Disciplines (INTs) - 1

Intelligence is not only gathered through secret or covert meansSome intelligence is collected through clandestine operations and known only at the highest levels of government

Other intelligence consists of information that is widely available

Intelligence can be used for both legitimate and nefarious purposes.

Currently 5 intelligence collection disciplines exists1. HUMan INTelligence (HUMINT).

Is the process of gaining intelligence from humans or individuals by analyzing behavioralresponses through direct interaction.

Openly vs. Clandestine/Covert (Espionage)

2. SIGnal INTelligence (SIGINT) refers to electronic transmissions that can be collected by ships, planes, ground sites, or satellites.

Communications Intelligence (COMINT) refers to the interception of communications between two parties.

https://www.fbi.gov/about-us/intelligence/disciplines


Intelligence Collection Disciplines (INTs) - 2

[…]3. IMagery INTelligence (IMINT) is sometimes also referred to as Photo Intelligence (PHOTINT).

4. Measurement And Signatures INTelligence (MASINT) includes the advanced processing and use of data gathered from overhead and airborne IMINT and SIGINT collection systems.

TELemetry INTelligence (TELINT) is sometimes used to indicate data relayed by weapons during tests.

ELectronic INTelligence (ELINT) can indicate electronic emissions picked up from modern weapons and tracking systems

MASINT can be used, for example, to help identify chemical weapons or pinpoint the specific features of unknown weapons systems.

5. Open Source INTelligence (OSINT) is the process of gathering intelligence from publicly available resources (including Internet and others)

*https://www.fbi.gov/about-us/intelligence/disciplines


OSINT - Definitions

Open Source Information (OSINF) is data which is available publicly – not necessarily free

Open Source Intelligence (OSINT) is proprietary intelligence recursively derived from OSINF

OSINF Collection consists in monitoring, selecting, retrieving, tagging, cataloguing, visualising & disseminating data

OSINT is the result of expert analysis of OSINF

Slide Credit: C.H. Best, JRC – European Commission


OSINT – Origins - 1

Term Originates from Security Services

The practice of using open source information to build intelligence is indeed not new

In Italy OVRA (Organizzazione per la Vigilanza e la Repressione dell'Antifascismo) reported the use of OSINF since 1930

“Gli anonimi informatori, secondo l’ex-prefetto di Brescia Arturo Bocchini (capo indiscusso sia dell’OVRA che della Polizia sino al 1940, anno della sua morte) dovevano fornire elementi per “…sondare con ogni mezzo e continuamente la pubblica opinione”, in modo che Mussolini potesse “…rendersi conto della temperatura del paese”.

During the cold war, American and German secret services heavily analysed the Russian press to gather information about their Russian enemies

Nevertheless, open source information has been traditionally considered definitely less valuable than classified information

“Anonymous Informer Report”, OVRA Region 1 – Milano, 1939 - http://gnosis.aisi.gov.it/Gnosis/Rivista2.nsf/ServNavig/15


OSINT – Origins - 2

Paradigm change after 9/11 (shock to the system of old style intelligence)

Pre 9/11 intelligence services were closed as they relied almost only on HUMINT, SIGINT and classified information Realisation that open source information could have been use to foresee attacks “Failure to connect the dots” reassessment in use of OSINT, & in sharing intel between agencies. Terrorists skilled use of internet was an eye opener

The fast growth of the Internet and the appearance of Social Networks have further pushed the paradigm change

“The need to restructure the intelligence community grows out of six problems that have become apparent before and after 9/11

Structural barriers to performing joint intelligence workLack of common standards and practice across the foreign-domestic divideDivided management of national intelligence capabilitiesWeak capacity to set priorities and to move resourcesToo many jobsToo complex and secret”

The 9/11 Commission Report


OSINT – Who is Involved?

Tool Builder/Develo

per

MinisterGeneral

CommissionerCEO

Analyst

Classified Information

OSINF Collector /Researcher



Who uses OSINT?

Security Services, Law Enforcement & Military Bodies

Governmental OrganisationsEU, NATO, AU Situation Centre

IAEA – Nuclear Safeguard

UN Department for Peacekeeping Operations

World Health Organisation

NGOs

Large CompaniesOil/Gas Industries

Multinationals


OSINT Sources of Information - 1

MediaNewspapers, magazines, radio, television, etc.

The InternetNews, Social Networks, Blogs, Video sharing sites, Thematic sites. etc.DeepWeb (not indexed by traditional search engines)

Dynamic Web Pages Sites behind Log-inSytes with a ROBOT.txt file properly configured

Dark Nets/Web (TOR, I2P)

Subscription ServicesLexisNexis (http://www.lexisnexis.com) is a corporation providing computer-assisted legal research as well as business research and risk management services. During the 1970s, LexisNexis pioneered the electronic accessibility of legal and journalistic documents.Factiva (http://www.dowjones.com/products/product-factiva/) is the world’s leading source of premium news, data and insight, with access to thousands of premium news and information sources on more than 22 million public and private companiesJane's (www.janes.com) Information Group is a British publishing company specialising in military, aerospace and transportation topics.BBC Monitoring (http://www.bbc.co.uk/monitoring) includes news, information and comment gathered from the mass media around the world for service subscribers.


OSINT Sources of Information - 2

Commercial Satelliteshttp://www.euspaceimaging.com/applications/fields/security-defense-intelligence

https://www.digitalglobe.com/industries/defense-and-intelligence

Public Datagovernment reports, budgets, demographics, hearings, legislative debates, press conferences, speeches, marine and aeronautical safety warnings, environmental impact statements and contract awards.

Professional and Academicconferences, professional associations, academic papers, and subject matter experts.

Open Datahttps://open-data.europa.eu/en/data

http://www.dati.gov.it

http://www.datiopen.it

Geospatial Data ProvidersAn exhaustive list is available here https://en.wikipedia.org/wiki/List_of_GIS_data_sources


*INT Target Modes

Intelligence activities can target both

IndividualsIntelligence gathering refers to querying online public resources that provide information specific to the targeted individuals.

Corporates and organizationsIntelligence gathering refers to the process of collecting information about target organizations.


*INT Target Modes – Targeting Individuals

When individuals are targeted, the following information may be of interestPhysical locations of the individuals

OSN profiles for checking on relationships, contacts, content sharing, preferred web sites, etc.

E-mail addresses, users’ handles and aliases available on the Internet including infrastructure owned by the individual such as domain names and servers.

Associations and historical perspective of the work performed including background details, criminal records, owned licenses, registrations, etc. This data is categorized into public data provided by official databases and private data provided by professional organizations.

Released intelligence such as content on blogs, journal papers, news articles, and conference proceedings.

Mobile information including phone numbers, device type, applications in use, etc.

Source: Targeted Cyber Attacks Multi-staged - Attacks Driven by Exploits and Malware, Elsevier, 2014


*INT Target Modes – Targeting Corporates and Organisations

When Corporates and Organisations are targeted, the following information may be of interest

Determining the nature of business and work performed by target corporates and organizations to understand the market vertical.

Fingerprinting infrastructure including IP address ranges, network peripheral devices for security and protection, deployed technologies and servers, web applications, informational web sites, etc.

Extracting information from exposed devices on the network such as CCTV cameras, routers, and servers belonging to specific organizations.

Mapping hierarchical information about the target organizations to understand the complete layout of employees at different layers including ranks, e-mail addresses nature of work, service lines, products, public releases, meeting, etc.

Collecting information about the different associations including business clients and business partners.

Extracting information out of released documents about business, marketing, financial, and technology aspects.

Gathering information about the financial stand of the organization from financial reports, trade reports, market caps, value history, etc.

Source: Targeted Cyber Attacks Multi-staged - Attacks Driven by Exploits and Malware, Elsevier, 2014


Domains registered by criminals forCounterfeit goods

Data exfiltration

Exploit attacks

Illegal pharma

Infrastructure (ecrime name resolution)

Malware C&C

Malware distribution, ransomware

Phishing, Business Email Compromise

Scams (419, reshipping, stranded traveler…)

*INT Target Modes – Investigating a cyber-attack


OSINT Processes

Collect Transform AnalyseVisualise& Report

Collaborate


Multilingual Information Retrieval

SearchCrawlNews feeds

Machine Translation

Geo-taggingTranslationEntity Extraction

Entity Resolution

Link AnalysisRelationshipsGeolinkingTrends

Statistics

NetworksRelationsTime graphs

Maps

Intel WikiIMCase DB

Publish

TECHNICAL ISSUESData MappingData DedupingData CleansingData ConversionData LinkingData Normalisation


Before you begin an investigation, ask


Recon-Ng

Recon-ng is a full-featured Web Reconnaissance framework written in Python. Complete with independent modules, database interaction, built in convenience functions, interactive help, and command completion, Recon-ngprovides a powerful environment in which open source web-basedreconnaissance can be conducted quickly and thoroughly.

Recon-ng is a completely modular framework and makes it easy for even the newest of Python developers to contribute. Each module is a subclass of the "module" class. The "module" class is a customized "cmd" interpreterequipped with built-in functionality that provides simple interfaces to common tasks such as standardizing output, interacting with the database, making web requests, and managing API keys. Therefore, all the hard work has been done. Building modules is simple and takes little more than a fewminutes.


Recon-Ng - Modules

Discoverydiscovery/info_disclosure/interesting_files

Exploitationexploitation/injection/command_injectorexploitation/injection/xpath_bruter

Importimport/csv_fileimport/list

Recon (circa 60 moduli in tutto)recon/companies-multi/whois_minerrecon/domains-credentials/pwnedlist/leak_lookuprecon/hosts-hosts/ipinfodbrecon/profiles-profiles/twitter

Reportingreporting/csvreporting/htmlreporting/jsonreporting/list


Google “Nintendo Co. Ltd. Board”http://quotes.wsj.com/JP/7974/company-people


Data.com - Connect


Data.com – Nintendo Co. Ltd. Info


Data.com - Nintendo Co. Ltd. Locations


Data.com - Nintendo Co. Ltd. Internet Domains


Data.com - Nintendo Co. Ltd. Personnel ContactInfo


About Data.com Points Earning


About Data.com Points Earning


Data.com - Nintendo Co. Ltd. Personnel


Data.com - Nintendo Co. Ltd. Personnel


Radaris.com - Kathryn Rigney

Phones First reported Last reported

(253) 813-5796 06/01/1989 04/05/2016

(253) 277-0426 01/16/2011 02/15/2011

(206) 854-6297 – –

(760) 749-8776 – –

(714) 749-8776 – –

(253) 854-6297 – –

Emails

[email protected], [email protected], [email protected], [email protected]

Addresses First reported Last reported

226 R St, Auburn, WA 98002 > 06/30/2012 04/04/2016

802 45Th St # 11-20, Auburn, WA 98002 > 01/16/2011 02/15/2011

802 45Th St Apt 11-203, Auburn, WA 98002 > 04/01/1999 08/01/2010

Kathryn RigneyAuburn, WA

Advanced People Search ReportReport date: May 14, 2016

1 person found.

Born: Nov 11, 1957 - 58 years old

1 Kathryn L Rigney


Spokeo.com - Kathryn Rigneyhttp://www.spokeo.com/Kathryn-Rigney/Washington/Auburn/p18244555111


recon-NGrecon/contacts-profiles/fullcontact


recon-NGrecon/profiles-profiles/profiler


Recon-Ng: Show Contacts


Recon-Ng: Show Profiles


recon-NGrecon/profiles-profiles/twitter


The Eyepiramid CyberEspionage Case

Source: F. Maggi, Uncovering the Inner Workings of EyePiramid, TrendLabs Security Intelligence Blogs, 2017

eyepyramid[.]com millertaylor[.]com wallserv[.]com enasrl[.]com marashen[.]com ayexisfitness[.]com

hostpenta[.]com occhionero[.]com westlands[.]com eurecoove[.]com millertaylor[.]com occhionero[.]info


A case study: Dashbida.comCredits: TISCALI ITALIA S.p.A., PRALab

16531 unique Ips have contacted37 domains under dashbida.com

during the last 7 days.

an5172.dashbida.com events.dashbida.com nevov-pavon.dashbida.com

bepun-riryt.dashbida.com fyryd-dexyx.dashbida.com romos-recek.dashbida.com

bidel-cosyt.dashbida.com hocap-datyn.dashbida.com rtb-sync.dashbida.com

cdn2.dashbida.com hotem-calub.dashbida.com s.update.dashbida.com

cdn3.dashbida.com hozed-midet.dashbida.com v-imp.dashbida.com

cimys-kylob.dashbida.com hunuc-pebur.dashbida.com vastservice.dashbida.com

civic-sisim.dashbida.com hunyf-xesof.dashbida.com vecar-cevyb.dashbida.com

dixex-xeles.dashbida.com huzav-dydib.dashbida.com vexigo.dashbida.com

docoz-limup.dashbida.com lafac-logis.dashbida.com ximaf-fukun.dashbida.com

dofob-gixux.dashbida.com mobib-pivis.dashbida.com xovov-holuv.dashbida.com

dypyg-zafyk.dashbida.commukyb-

sycep.dashbida.com dbam.dashbida.com



osint – open source intelligence -...

Documents