collecting and processing of data from security tools in ... · system for efficient information...

Collecting and processing of data from security tools in CESNET

CESNET, z. s. p. o.

Andrea Kropáčová[email protected]

CNMS2016, Prague, 25 Apr

● Operates Czech Educational and Scientific Network● Established in 1996● Has 27 members and ~300 participants ● Main goals:

– research and development of information and communication technologies

– construction and development of eCESNET infrastructure for research and education

– promotion and dissemination of education, culture and knowledge

● 2011 – 2015: Project „Big infrastructure CESNET“● 2015 – 2020: Project „Einfrastructure CESNET“● Operates security teams CESNETCERTS

– accredited team by Trusted Introducer in 2008 (established in 2003)

– responsible for solving security issues in CESNET2 (AS2852)

– http://csirt.cesnet.cz/, [email protected]

https://www.cesnet.cz/


Network management

Transparent

No restriction of legitimate traffic (until the problem came up)

Connected networks can use tools developed and operated by CESNET for their own selfprotection and selfregulation

1)

2)

3)

CESNET2

- HW accelerated probes- large scale (backbone-wide) flow based monitoring (NetFlow data sources)- Honey Pots- IDS, IPS, tar pit based systems, etc.. - SNMP based monitoring


Begining ... ● Administrators often run their own IDS`, security probes, central

syslog, honeypots, IPS ...– For networks and services monitoring– Finding compromised machines, botnet activity, malware, antispam– Detection of networks anomalies and attacks ...

● Problem – they pick just the data important for them, what to do with the rest?– Throw away?

➔ Noo, it is information wasting ...– Make report?

➔ Too much work ... recipients may need help, another information ...

➔ How? Data format? Protocol? Data clasification? Protection? Policy?

SHARE!!!


S● System for efficient information sharing● Client/server architecture (transport, not storage)● Community (aka „let's build security together")

– Reciprocity – all your data is available to the whole Warden community...

– … and all the community data is available to you

● Sending and receiving clients● Format: IDEA (https://idea.cesnet.cz)

● Protocol: JSON/HTTPS

● Sec/auth: TLS/X509

● Platform: Python/WSGI

● Bulk operations, incoming filtering

● Security (X509, encryption, “sanity” checks, peer review)

Dionea

Kippo

Dionea

IDSLaBrea NEMEA 3rd

Shadow, N6,X2, X4

NSHARP

FTASCESNET-CERTS

NOCPSSCSIRT.SK

VŠBVUTBR

Dionea

Kippo

Data flow (sending client)

Data flow (download client)


Lesson learned I

Connected organizations do not have sufficient human resources to use the open community approach to

=

They can not download and processe data themselves.

But they want to obtain this data, the data is useful

=

it is necessary to deliver the processed data.

Dionea

Kippo

Dionea

IDSLaBrea NEMEA 3rd

Shadow, N6,X2, X4

NSHARP

FTASCESNET-CERTS

NOCPSS

Data flow (sending client)

Data flow (download client)

CSIRT.SKVŠB

VUTBR

Dionea

Kippo


S


● Mentat is downloading client in Warden architecture.

● SIEM

● Data storage

● Divides events according to end networks (creating reports)

● Send reports to the end networks (abuse @ ...)

– RIPE DB


Lesson learned II

● Too little info, we do not know what to do.

● I do not want a email report, I wont structured data format.

● How is the severity?

● We do not want this information, we get it from the source.

● Data from 3rd parties have different quality

● NAT, FW, DHCP …

● Big networks like universities …

– we must divide the information in the report and create the new reports.

● Why do I receive the same report? I solved it yesterday.

... report recipients say ...


IDEA Format

● JSON (NoSQL friendly), but mostly flat and typed structure (SQL friendly)● Extensibility (producers can use their own keys and tags)● Marking of anonymised, imprecise, forged data● Able to distinct third party events, correlated events, updated/referenced events● Taxonomies (mkII categories, tag based Source/Target/Detector description)● https://idea.cesnet.cz

https://idea.cesnet.cz/


S


Filtering● Endnetworks admin may set up reporting

– Ignore one IP address– Ignore one source of data– Ignore some types of events


inetnum 147.32.1.0 – 147.32.50.255remarks Report network abuse --> [email protected]

inetnum 147.32.1.0 – 147.32.50.255netname CVUT-TCZdescr Praha 1remarks Report network abuse --> [email protected]







Lesson learned III

● We can gather data into one place and report them.

● BUT!

– Share primary data (via report) is not enough!

– Data obtained from security tools in one network is not enough!

– Share data in one and from one network is not sufficient!

● Why?

– Primary data are many and have different information value.

– We do no see some problems.

– Missing context, we do not see the big picture.

... present & future ...


What next?● New and more sources of primary data in CESNET.● New and more sources of primary data out of CESNET.● New source from 3rd parties.● Better validation and classification.● Data enrichment.● Inteligent analysis and data correlation.● Information and data sharing at national and international level.

„ ... more, better, faster...“


SABU

● Project “Sharing and Analysis of Security Events“

● 2016 – 2020, funded by Ministry of Interior of Czech Republic

● CESNET, Masaryk University in Brno

● https://sabu.cesnet.cz – in development

● sabu[email protected]

● Partners:

– CSIRT.SK

– ISP

– Bank sector

– Invea Technologies

(Sdílení a analýza bezpečnostních událostí)

https://sabu.cesnet.cz/


Thank you for your attention!

collecting and processing of data from security tools in ... · system for efficient information...

Documents