beyond the soc as a detection center: holistic cybersecurity … · 2018-05-17 · beyond...

BEYOND THE SOC AS A DETECTION CENTER: HOLISTIC CYBERSECURITY OPERATIONS IN THE COMING AGE

WHITE PAPER 2017 — Thales — Critical Information Systems and Cybersecurity

OUR VISION OF THE NEW GENERATION OF CYBERSECURITY OPERATIONS 4

NEW IT CHALLENGES CALL FOR A CYBERSECURITY PARADIGM SHIFT 6

BUILDING ON SOLID FOUNDATIONS 2.1 FOUNDATION #1: DETECTING & MANAGING VULNERABILITIES 8

2.2 FOUNDATION #2: MANAGING INCIDENT RESPONSE OVERLOAD 10

2.3 FOUNDATION #3: CONCENTRATING ON PEOPLE AND TALENTS 11

THE PARADIGM SHIFT3.1 CYBERSECURITY OPERATIONS WHILE MOVING TO PUBLIC CLOUDS 13

3.2 CYBERSECURITY OPERATIONS FOR OT ENVIRONMENTS 14

3.3 THE SOC AS A CORNERSTONE OF THE CYBER-DEFENSE CHAIN 16

KEY BUILDING BRICKS FOR CYBERSECURITY OPERATIONS IN COMPLEX ENVIRONMENTS

4.1 THREAT INTELLIGENCE 19

4.2 BEHAVIORAL ANALYSIS, MACHINE LEARNING AND ARTIFICIAL INTELLIGENCE 21

CONCLUSION AND KEY TAKEAWAYS 24

TABLE OF CONTENTS

1

2

3

4

5

3

Thales — 2017 BEYOND THE SOC AS A DETECTION CENTER: HOLISTIC CYBERSECURITY OPERATIONS IN THE COMING AGE

OUR VISION OF THE NEW GENERATION OF CYBERSECURITY OPERATIONS

With the digital transformation, IT and security professionals are facing one of the most challenging moments in their history, with trends including a massive move to public clouds, the adoption of Big Data analytics and Artificial Intelligence, and the emergence of Industry 4.0 and the (industrial) Internet of Things. Data is becoming the core engine of businesses and the center of organizations. IT is

becoming more than a tool, establishing itself as one of the key foundations of organizations, inclu-ding industrial operations.

This transformation comes with specific security challenges in a context of ever-accelerating change in terms of threat landscape, attack surface, ecosystems, solutions and regulations. Based on our expe-rience in building and running cybersecurity operations for a number of private, public-sector as well as defense customers, we at Thales have forged strong convictions on how to step up to these critical challenges. Although it is key to embrace new technologies, we are convinced that it is even more important to adopt a pragmatic and operational approach in line with the realities of today’s world.

• First, this means having a number of basics fixed, beyond baseline security supervision. Real-life experience with customers has shown that deploying new detection tools serves no useful purpose when vulnerability management is inadequate, when security incident teams are overwhelmed with low-priority issues or false positives, and when professionals suffer from «alert fatigue». Before moving forward, organizations first need to ensure that these foundations are at an acceptable level.

• Second, there is a need for extensive visibility and control over today’s organizations and assets, including not only on-premise IT and network infrastructures, but also assets deployed in the cloud and at industrial sites.

• Third, the two most important new technologies for cybersecurity operations are threat intelligence and artificial intelligence leveraging Big Data platforms. They offer effective ways to improve detec-tion rates and enable analysts to work more efficiently. But new technologies are not magic wands. They complement more traditional approaches such as SIEM-based solutions.

• Fourth, both the move to public clouds and OT environments call for specific approaches. Moving to public clouds does not mean transferring all security responsibilities to public cloud providers. Public cloud customers have significant security challenges to overcome with a combination of tools, processes and expertise. A specific approach is also required for OT environments, where cultural and organizational aspects are as important as technical factors.

• Fifth, none of these key points can be achieved without a clear focus on expertise and talents. We know in particular that AI is not going to diminish the need for high-end expertise in cybersecurity, unlike certain other white-collar activities, due to the fundamental nature of problems to be solved.

• Last, we believe that all these points are a matter of mastering complexity: laying the right founda-tions, prioritizing, and focusing on talents and processes and not only on tools and technologies.

Based on our experience, we can see that security operations are engaged in a deep transformation, moving from a centralized breach detection and response role to a cornerstone position with a holistic view of cyberdefense and the ability to manage overall cybersecurity complexity consistently with business risks and high-level governance.

4

BEYOND THE SOC AS A DETECTION CENTER: HOLISTIC CYBERSECURITY OPERATIONS IN THE COMING AGE Thales — 2017

BUILDING ON SOLID FOUNDATIONS

ENCOMPASSING CLOUD AND OT

LEVERAGING THREAT INTELLIGENCE AND ARTIFICIAL

INTELLIGENCE AS APPROPRIATE

HOLISTIC, AGILE, EXPERTISE BASED• SIEM heuristic based detection

• Augmented AI & TI based detection• Advanced investigation

• Comprehensive Vulnerability Mgt.• Business Risk Mgt.

& Governance

THE SOC AS THE CORNERSTONE

OF CYBERSECURITY OPERATIONS

OUR VISION OF THE NEW GENERATION OF CYBERSECURITY OPERATIONS

Public Clouds security issues

Do your part, e.g. Identity & Access Management, Data Security Leverage specific security design & control mechanismsGet visibility and ensure compliance with security policy

Unsecured OT & industrial environments

Concentrate on safety and other specific OT

requirementsUse a step-by-step

approachLeverage OT probes &

mechanismsOvercome the cultural and

organizational OT – IT gapGet visibility and ensure end-to-

end security across both IT and OT

190 days average mean-time to

identify breaches*

Threat Intelligence augment detection

capabilities through IoC searches

Artificial Intelligence detects breaches through unusual

behavior analysis

Analysts focusing on low-value tasks

Threat Intelligence brings immediate and actionable

informationAI will rise efficiency by

up to 40%(*)

30 daysVS

245 days

Average time to create an exploit for a new vulnerability*

Average time to patch a vulnerability in IT systems*

Deploy a business driven Vulnerability Management

From 18% to 40%Average rate of cyber detection

false positives in the industry*

Solve the incident response overload by: - Focusing on business related risks - Tuning the system to reach

<5% false positives

No tool and technology silver bullet

Focus on people and talents

1

2

3

THE NEW GENERATION OF CYBERSECURITY OPERATIONS

* Sources: Verizon, WhiteHat Security, Gartner, Ponemon & GS Global Investment Research, Infosecurity Magazine, Crowd research partners

5


NEW IT CHALLENGES CALL FOR A CYBERSECURITY PARADIGM SHIFT

While the world of information technology has seen a number of revolutions in the past decades, today’s shift is rather drastic and radical. IT people and organizations are going through an extre-mely deep transformation which is rewriting the ground rules. The key elements of this transformation include:

• The massive move to the cloud, relying on large public cloud providers such as Amazon, Google or Microsoft

• The leverage of data value, leading to the emergence of new business fields and models based on Big Data and Artificial Intelligence technologies applied to field such as customer care, predictive maintenance or physical security

• The increasing need for velocity and agility, driven by operational groups and business units which are willing to quickly implement IT solutions by relying on SaaS providers, often bypassing central IT groups and creating shadow IT environments

• The “ITfication” of industrial environments, leading to the emergence of the Industrial Internet of Things (IioT) and Industry 4.0 and connecting industrial assets such as turbines or trains

• The evolution of the regulatory environment, impacting the IT and security world (e.g. GDPR, NIS or specific national security requirements)

• The increasing dependency of businesses or organizations on IT infrastructures interconnected with the outside world.

Although we can debate about the pace and depth of this new revolution, for sure it is happening. We actually are at the beginning of it and can expect it to bring a deep transformation of the societies we live in. For security professionals, it is combined with a continuous increase in threat level and attack surface, an evolution of both threat actors and cybersecurity mechanisms, and exploding demand for skilled cybersecurity resources. We believe these factors do not constitute a short and temporary transition. They are structural, bringing an unprecedented and long-lasting level of complexity and instability. As a consequence, Chief Security Officers (CSO) and Security Operation professionals face tremendous difficulties in mastering the emerging complexity and reducing the “unknown” part of the security equation.

The bottom line is that we are entering the age of data. This means we have to protect data of any kind, anywhere and for any type of businesses. This calls for a profound paradigm shift on how to manage cybersecurity and prepare for the coming years. Although no single actor can pretend to hold the truth about the future of cybersecurity, we see clear trends emerging today for modern cyber-security operations.

From a technological standpoint, detection and response mechanisms are not only based on selected and preformatted events, but rather on the collection of as much structured or unstructured information as possible from a growing number of devices and applications. This volume of data, captured in data lakes, can only be relevant when coupled with powerful analytics engines leveraging Artificial Intelligence and Threat Intelligence technologies, among others.

Beyond technology, we see the cybersecurity operations roles and responsibilities significantly reconfi-gured in the new era. The typically centralized operational cybersecurity team now interfaces and shares responsibilities with a growing number of stakeholders, including cloud service providers, Factories and OT groups, third-party MSSP providers and regulatory authorities.

The SOC as a whole is becoming the cornerstone of the overall cybersecurity architecture.

16


Together with the technological choices, this leads to different approaches, which can be centralized or distributed, automated or expertise-based, and focusing on various aspects of the prevent – detect – remediate – govern continuum.

Based on these complex trends, operational expertise and feedback, Thales sees some key success factors for cyber-securing businesses while moving to secured Digital Transformation. Focusing on cybersecurity operations, which cover threat detection, analysis, response and remediation phases as well as compliance, our top three convictions are:

• First, it is essential to build on solid foundations and fix the basics. We are all too often obsessed with the latest futuristic technologies. However, the reality of most organizations today is that many basic security prevention and response mechanisms are either not in place, or not adequately deployed, leaving significant room for improvement.

• Second, the role of the Security Operation Center (SOC) is evolving significantly, including its core toolset, the Security Information and Event Manager (SIEM). At Thales, we have been running 24x7 SOCs using various SIEM technologies since 2000. Although we see articles in the press predicting the end of the SIEM , we are convinced that:

1. The SIEM will continue to play a key role in threat detection, analysis and response in the fore-seeable future.

2. The SIEM will have to increasingly interwork with other enabling technologies such as Threat Intelligence, Big Data or Artificial Intelligence.

3. The SOC as a whole is becoming the cornerstone of the overall cybersecurity architecture, ensuring the cybersecurity consistency of the organization, across the business and the infrastructure, encom-passing on-premise IT and network, but also cloud services and Operational Technologies (OT).

• Third, technical (r)evolutions in cybersecurity are making significant contributions, but none can be considered a silver bullet. Technologies such as Threat Intelligence, Big Data and Artificial Intelli-gence all add value and need to be embraced, but only as part of a holistic approach which takes into account the complexity of the tools, processes and environments involved. In other words, the core value is not so much in impressive new technologies, but rather in the ability to manage com-plexity.

Based on these convictions, this white paper examines how cybersecurity operations will respond to the key challenges posed by the ongoing Digital Transformation.

TECHNOLOGIES & TOOLS

EXPERTISE

GOVERNANCE

METHODS & ORGANIZATION

The SOC core value is not so much in new fancy technologies but in its ability to manage complexity.

KEY DIMENSIONS FOR EFFICIENT CYBERSECURITY OPERATIONS

7



Cybersecurity being in constant and rapid evolution, professionals face tremendous difficulties in coping with the changes and keeping their heads above water. Working with many customers and organizations, we think that the first priority is to fix the fundamentals. The key here is to set up a coherent cybersecurity protection chain, including prevention, detection, analysis, response, remediation as well as compliance and governance. These basics are not going to protect organizations against the most advanced or state-sponsored attacks exploiting zero-day vulnerabilities, for example. However, they constitute the foundations required to build more advanced defense mechanisms.

In many organizations, the first basics to get fixed relate to the detection and patching of known vulnerabilities. This issue is widely recognized by both IT and security professionals, and became mainstream in May 2017 when the infamous WannaCry ransomware exploited vulnerabilities for which Microsoft had released a patch two months earlier. Even so, NotPetya made the headlines a month later by targeting the exact same vulnerability.

While all IT and security operation teams are aware of the issue, fixing it is another story. Modern vulnerability management requires access to a reliable source of information on vulnerabilities, corre-lation with an up-to-date asset database or with results from scan campaigns, and setting the right prio-rities based on business criticality, system exposure and operational patch management constraints.

While patch management typically remains within the scope of responsibility of IT operations, there are significant advantages in having the SOC taking the lead on vulnerability management:

• Using vulnerability intelligence, the SOC can detect assets that might be impacted by an ongoing attack and help to prioritize patching or containment actions.

• When critical patches cannot be applied for operational reasons, the SOC can put in place mecha-nisms to detect or stop potential exploits of the related vulnerabilities.

• By launching scans, the SOC can act as a control authority to validate the effectiveness of patch management.

Put simply, one of the key pillars of cybersecurity lies in a solid vulnerability management program. This should not mean scanning and patching everything immediately, but rather making the right choices based on a global view of the risk involved.

2.1

FOUNDATION #1: DETECTING & MANAGING VULNERABILITIES

30 daysis the median time to create an exploit once a vulnerability has been released **

245 daysis the average time to patch a vulnerability in the IT industry (150 days across all industries) **

90% of orgsrecorded exploits for vulnerabilities that were three or more years old ***

60% of orgsstill see related attacks 10+ years after a flaw release ***

VULNERABILITY MANAGEMENT GAPS BY THE NUMBERS

Sources: *Verizon Data Breach Investigation Report 2016, **WhiteHat Web Security Statistics Report 2016 and ***Fortinet Global Threat Landscape report Q2 2017

One of the key pillars of cybersecurity lies in a solid vulnerability management program. This should not mean scanning and patching everything immediately, but rather making the right choices.

28


VULNERABILITY INTELLIGENCE

SCAN RESULTS

RAW RISK

NET RISK

ASSET DATABASE

BUSINESS CRITICALITY

SYSTEM EXPOSURE

OPERATIONAL PATCH MANAGEMENT CONSTRAINTS

PATCH MANAGEMENT PRIORITIZATION

CSOS BASED DETECTION & CONTROL

Through the Cert-IST (Computer Emergency Response Team - Industry, Services, Tertiary), Thales provides vulnerability intelligence services for efficient patch prioritization as well as detection and analysis support.The Cert-IST organization was established in 1999 by a consortium of French and multinational companies. It is operated by Thales and offers a variety of services including vulnerability intelligence.As part of this vulnerability intelligence service, the Cert-IST publishes more than 1,000 new vulnerability advisories and 3,000 updates a year, covering over 2,000 products and 16,000 versions.

Yet, despite those numbers, the value of the Cert-IST does not actually reside in the quantity, but rather the quality of vulnerability advisories it delivers.Indeed, there are more than 10,000 vulnerabilities published per year from various sources in the field. With such volumes, CISOs and security operations need to access information that is qualified and actionable. This is precisely what Cert-IST delivers with a manageable volume of advisories, which are filtered, classified, prioritized and enriched with risk ratings and recommendations.Based on this information, Cert-IST customers are protecting more than a million assets in industries across all

critical sectors such as finance, energy and telecoms.Thales SOCs also leverage Cert-IST intelligence to improve the supervision of customer infrastructures. When combined with vulnerability scans, the SOCs can help to prioritize patches and determine other mitigation actions. One of these consists in defining specific rules to detect potential exploits of known – yet unpatched – vulnerabilities.

THE EFFECTIVE VULNERABILITY MANAGEMENT WORKFLOW

ACTIONABLE VULNERABILITY INTELLIGENCE FROM THALES

These decisions can only be made when the various pieces — vulnerability scans, vulnerability intel-ligence, asset management, risk management, patch management and security monitoring — are integrated together in a holistic approach to cybersecurity operations.

9


2.2

FOUNDATION #2: MANAGING INCIDENT RESPONSE OVERLOAD

Pushing rules into a SIEM or an IDS device, for example, is easy. However, this approach typically leads to generating large numbers of alerts that the incident response team can have trouble coping with. Many organizations have subcontracted cybersecurity detection to third parties which basically generate alerts 24x7 using a one-size-fits-all set of rules and scenarios. This typically leads to an overwhelmed 8x5 incident response team. Again, generating alerts is easy, but focusing incident response analysts on the right incidents is another story. Rules need to be tuned and alerts need to be qualified to take into account the organization’s business risks, the specifics of the infrastructure and applications, the cybersecurity policy and perceived vulnerabilities in order to derive sound scenarios, use cases and decide which rules to use.

Of course, there are alternative approaches such as the use of Artificial Intelligence as discussed later in this paper. However, those technologies alone cannot solve the incident response overload problem, and they can even make it worse if they are not well-tuned and end up generating a lot of false positives.

Instead, Thales experience demonstrates that a key success factor in cybersecurity detection and res-ponse resides in substantial, continuous customization and fine-tuning of the detection chain, along with highly skilled analysts who can perform a fully qualified analysis.

In addition, incident response teams need to balance their efforts between:

• Stopping and limiting attack impacts with short term actions

• Remediating to go back to a normal situation

• Analyzing and fixing the root cause of the breach, which can lead to basic patch management or new rules in firewalls, or to more fundamental actions such as network redesign or new protection mechanism

• Analyzing and limiting user behaviors that increase the security risks of the organization (such as opening remote connections from the Internet or opening attachments received from unknown third parties), which can be done either through technical means or through training and education (for example, to raise awareness of scams targeting executives).

These various tasks often compete for the same resources. The key is to be able to prioritize the tasks and to have access to a pool of resources of the right size and with the right skills, whether in-house or through an MSSP provider.


From 18% to 40%is the average rate of false positives in the industry.*

<5%is what we experience at Thales on well-tuned systems with mature customers.

2

Generating security alerts is easy, but focusing on the right incidents is another story.

OVERWELMED WITH FALSE POSITIVES?

(Source: Infosecurity Magazine, Ponemon)

10


2.3

FOUNDATION #3: CONCENTRATING ON PEOPLE AND TALENTSAs mentioned earlier, cybersecurity protection can be depicted as a set of (1) technologies and tools, (2) processes, methods and organizations, and (3) human expertise. While a lot of focus is put on technologies and tools, people and talents are equally if not more important.

Indeed, the quality of cybersecurity operations is highly dependent on the capability to attract and retain highly skilled professionals in a market where the demand is exploding and resources are very scarce. Based on our experience, the challenges are the following:

• Recruiting external talents

• Recruiting internal talents, bringing to cybersecurity other IT professionals, such as network or system architects

• Keeping expertise up-to-date in a very fast-moving environment

• Retaining talents.

On this last point, turnover increases when talents are only given routine tasks that generate so-called alert fatigue. To counter this risk, two key trends need to be considered:

• The move to a higher level of cybersecurity expertise while low-value tasks get more and more auto-mated, either in the SOC or in the infrastructure itself. Reducing the proportion of repetitive tasks frees up expertise to focus on higher added-value activities such as incident analysis and threat hunting.

• The emergence of new profiles, such as Threat Intelligence analysts and Data Scientists who are exploiting Big Data analytics technologies.

Cyber-security organizations definitely need to embrace these trends in order to bring more value and to offer more attractive environments to cybersecurity professionals.

In this overall transition, we strongly believe that technologies will not displace human expertise in the foreseeable future. Nor they will solve the resource scarcity issue that organizations are facing. Focusing on people and talents will remain an absolute necessity, in terms of recruitment, training, retention and skills development.

CYBERSECURITY TALENT FACTORY @ THALES

While a lot focus on techno and tools, people and talents are equally if not more important.

With “I Love Cybersecurity”, Thales has launched an extensive program to attract new cybersecurity talents.This includes the opening of a number of positions ranging from security analysts and integrators, to management and consulting specialists, as well as strong incentives to encourage employees to co-opt fellow professionals to take up these positions.In addition, Thales has developed long-term partnerships with top cybersecurity

engineering universities. Thales also provides a one-year dedicated training for in-house employees wishing to make a career move into cybersecurity.Finally, Thales has announced the launch of the Digital Factory, an initiative to put together skills and projects related to key enabling technologies for the digital transformation, such as a big data and cybersecurity.As a result of this, Thales was selected by STATION F – the biggest startup campus

in the world – to manage its cybersecurity program and hence drive the emergence of new cybersecurity champions.

11


2

1. HUMANS ARE OVERWHELMED WITH SECURITY ALERTS. ONLY ARTIFICIAL INTELLIGENCE (AI) WILL BE ABLE TO DO THE JOB IN THE MID-TERM.False! Most security response teams are indeed overloaded with security alerts. However, AI is not the silver bullet in this instance. AI plays a significant role in helping to detect new threat signals, and increases detection accuracy and incident response efficiency. Yet, AI presents a number of drawbacks that can only be overcome with human expertise. In the cat-and-mouse game of cybersecurity, both parties leverage more and more sophisticated tools, including AI. But AI alone won’t do the job.

2. THE MOST IMPORTANT POINT IS TO BE ABLE TO DETECT ATTACKS IN A VERY SHORT TIMEFRAME. False! While many players are advertising the capability to detect attacks in a matter of seconds, solely focusing on this metric is like measuring aircraft performance based only on the time the aircraft needs to reach 10,000 feet. In cybersecurity, the ultimate metric is the business impact of an attack, whether tangible, such as operation cost or business loss, or more intangible, such as reputational damage or loss of intellectual property. The objective here is to act on the full cost chain. In that context, response time is at least as important as detection time.

3. THE INFRASTRUCTURE IS BECOMING SELF-RESILIENT, DETECTING THREATS AND REMEDIATING AUTOMATICALLY.False!Although a number of network- or end-point-based security systems have capabilities to stop known attacks using pre-defined rules and markers, the dynamics of the threat landscape make it impossible to automatically detect all the threats and define the appropriate answers. To overcome these limitations, some security

systems develop smart mechanisms to detect and mitigate zero-day attacks using behavioral analysis, machine learning or sandbox techniques. Yet these solutions are prone to false positives, which could lead to inappropriate responses and even service outages if they are not recognized as such. At the end of the day, there will always be a need for human expertise to analyze and, more importantly, to determine the relevant answer to a threat.

4. THE SOC IS USELESS AS IT DOES NOT DETECT ANYTHING IMPORTANT.False! Several Thales customers have gone through a first SOC implementation consisting in installing a SIEM with pre-defined rules or subcontracting it to a low-end third party. The end result is that they receive a huge amount of alerts, most of them being false positives or already blocked through other security mechanisms. However, there is no fatality here. The key resides in extensive customization to tune the system correctly, concentrating on the right events and bringing real value. In addition, the SOC should not be reduced to surveillance activities but rather extend its role across the prevent – detect – remediate - govern continuum.

5. THE SOC OF THE FUTURE IS GOING TO BE FULLY AUTOMATED.False!A fully automated SOC would mean that any type of incident detected could be qualified and remediated without human intervention. This would clearly imply that machine learning and artificial intelligence can drive such a SOC, since it would be impossible to pre-define and hard-code all possible attack scenarios. Even if it was possible with the most advanced AI techniques, hackers would end up finding out how these mechanisms work and would craft attacks that circumvent them. Again, this is a game of cat-and-mouse, and only smart humans, assisted by the latest technologies, can take part.

FIVE MISCONCEPTIONS ABOUT CYBERSECURITY OPERATIONS


12


THE PARADIGM SHIFT

3

PUBLIC CLOUD SECURITY AND CUSTOMER RESPONSIBILITY

3.1

CYBERSECURITY OPERATIONS WHILE MOVING TO PUBLIC CLOUDSWith the growing adoption of public cloud services, organizations of all sizes and sectors have to dramatically reconsider the way they approach cybersecurity. Indeed, while they move part of their IT to the cloud, they also shift part of their security responsibilities to cloud service providers. Following leaks from whistle blowers and high-profile data breaches, all the major cloud players know that their reputation is at stake and therefore take security very seriously with means and standards that go far beyond what most of companies can afford on-premise.

At the same time, cloud providers are pushing part of the responsibility onto their customers, via shared security responsibility models that define the responsibility of both parties. For example, an organization subscribing to an IaaS offering with a major player will benefit from highly secured data centers, yet it remains responsible for the security of the virtualized systems and applications they deploy on them. Conversely, an organization subscribing to a SaaS offering will not have to worry about scanning and patching end-points for the latest specific application vulnerabilities, yet it must take care of the security of sensitive information in the cloud, such as intellectual property or Personally Identifiable Information (PII) data.

Hence, organizations using cloud services no longer need to worry about lower-level network and IT assets previously handled on-premise, but rather need to focus on threats to business critical assets such as identities or sensitive information that become more exposed when hosted in the cloud. These threats come from both external hackers who may leverage public cloud exposure to breach accounts, elevate their privileges and eventually leak data, and internal users who use the cloud to perform mali-cious activities or access unauthorized IT services, compromising the organization’s security.

ACCOUNT BREACH

ELEVATION OF PRIVILEGES

DATA LEAK

SHADOW IT

MALICIOUS INSIDERS

CSOC

IAM

DATA SECURITY

THREAT PROTECTION

COMPLIANCE

VISIBILITY

Organizations need to focus on threats to business critical assets that become more exposed when hosted in the cloud.

13


THE PARADIGM SHIFT3

Having said that, the traditional boundaries and control points found in on-premise IT vanish with the cloud usage. This can transform the cloud into a smokescreen that diminishes an organization’s visibility, control and ability to react.

In order to deal with these new security challenges, cybersecurity operations need a specific “cloud-ready” approach, supporting multiple cloud providers, to complement the approach traditionally used for monitoring on-premise infrastructures. This cloud-ready approach may leverage one or more of the following capabilities to achieve visibility and control over cloud usage:

• Security features made available by the Cloud Service Providers themselves

• Third-party security solutions specifically designed to secure cloud usage (e.g. Cloud Access Security Brokers - CASB)

• More traditional security solutions adapted to the cloud (e.g. NGFW, Web proxies, virtualized security functions).

Based on these, cyber-security operations need to extend their prevention, detection, response and compliance capabilities to the cloud. This requires:

• Extracting data from the cloud tenant, using APIs provided by cloud vendors and collectors deployed in the cloud or on SOC premises. By correlating events collected from both the on-premise and the cloud environment, cybersecurity operations can detect threats targeting the extended enterprise.

• Leveraging threat intelligence focused on cloud-specific threats and vulnerability scans adapted to virtualized and container environments.

• Performing incident response and compliance services, using the security controls, audit logs and reports made available by the cloud or third-party security vendors, in order to perform incident forensics and track deviations from security policies.

Ultimately, cybersecurity operations need to act as a supervisory authority which has visibility over both on-premise and cloud usage, in order to ensure compliance with the policies defined by the organization and regulators, and leveraging security controls made available by the cloud vendors.

3.2

CYBERSECURITY OPERATIONS FOR OT ENVIRONMENTSWhether used in industrial environments or in other environments, such as aircraft cockpits, combat systems, railway signaling systems or airborne radars, Operational Technologies (OT) present a new set of cybersecurity issues.

Indeed, for many years, OT used to live in isolation: they were disconnected from the Internet and based on non-IP technologies. OT environments then started to converge with IT technologies, adop-ting standard IP protocols and opening up towards external networks. This shift is now leading to the Industry 4.0 revolution, which relies on fully connected devices (IIoT), Big Data and other IT technolo-gies applied to OT environments. From a cybersecurity standpoint, these IT systems present tremendous challenges, putting the OT infrastructure at risk, for example by providing remote access to an OT device to perform routine maintenance while neither the OT system nor the network were designed to enable such remote connection in a secure way.

14


As compared to IT, OT does not share the same objectives, processes or culture. Cybersecurity for OT environments therefore requires a specific approach.

As compared to IT, OT does not share the same objectives, processes or culture. In particular, the top priorities of OT are typically personal and environmental safety, and production continuity. In addition, the OT environment is usually extremely stable, predictive and sensitive to real-time events. Based on specific technologies and protocols, the OT infrastructure design is often certified and must remain stable to avoid long and difficult recertification procedures. Last but not least, with the transition to the IIoT and Industry 4.0, the volume of data increases dramatically as more connected devices come on stream. All these compelling characteristics call for specific cybersecurity detection and response mechanisms. As an example, in the OT world, the reaction to suspicious activity cannot be a brutal system reconfiguration, for example, or a process shut-down, disconnection, makeshift patching or hasty redesign.

Cybersecurity for OT environments therefore requires a specific approach. Focusing first on the points of control in the infrastructure, this approach can be described as a set of logical steps aimed at harde-ning and monitoring specific points of the infrastructure, namely:

1. Access points to the Industrial Control Systems (ICS)

2. IT assets which are part of the ICS

3. ICS network flows

4. OT command and control traffic

5. Field devices (e.g. Programmable Logic Controllers - PLCs).

FIVE STEPS FOR SECURING OT ENVIRONMENTS

SOC MONITORING SECURITY INCIDENT RESPONSE OT SUPERVISION

Data basesSensors, pumps, actuators, etc

PLC, RTU, etc

Probe

Web servers

Mail servers

DNS

HistorianOperators Consoles

Control unit (MTU)

Application servers

Switch Firewalls

Data bases

ITOT

FIELD UNITS INDUSTRIAL CONTROL SYSTEMS

BUSINESS, NETWORK, HR, R&D, ETC

1

2

453

These steps may be taken separately or combined depending on the environment.

15


3

Beyond such an approach, the cybersecurity of OT environments faces significant organizational and cultural issues. In particular, the traditional split of responsibility between the IT and the OT teams somehow needs to be overcome so that cybersecurity is approached in a consistent way from secu-rity-by-design to detection and incident response. Who is responsible for defining the cybersecurity strategy? For establishing detection use cases? For embedding cybersecurity by design? For detecting and analyzing cybersecurity issues? For responding to them?

There is no single answer to these questions. One possibility is to have the IT-centric SOC also monitor the OT environment, while the responsibility for the incident response is shared between the IT res-ponse team (to propose technical solutions) and the OT operations (to ensure safety and production continuity). This calls for Level 3 specialists who understand both cybersecurity and OT. In fact, the key challenge is to have cybersecurity experts working in close relationships with the OT groups and overcoming the cultural differences together.

3.3

THE SOC AS A CORNERSTONE OF THE CYBER-DEFENSE CHAINIn the early days, the role of the SOC was basically to detect cybersecurity incidents, reducing the false positives and false negatives rates. For that purpose, the SOC would typically use a SIEM tool as a core engine to filter and correlate events from the monitored infrastructure, then would send a ticket to a Computer Security Incident Response Team (CSIRT aka CERT).

This surveillance function still exists and provides baseline visibility on explicit threat scenarios, such as well-known attacks or security policy breaches. However, we are seeing a significant shift in res-ponsibilities, with the SOC now becoming the cornerstone of the overall cybersecurity defense chain. This development has several drivers.

• Centralized vs. distributed detection and response: there has always been and continues to be a ten-sion between distributed detection and response mechanisms versus more centralized approaches. Although distributed automated tools are able to handle more and more threats, the SOC will conti-nue to provide high-value expertise to cope with more sophisticated attacks and issues.

• Holistic and actionable focal point of control: neither the SOC nor the CSIRT have disappeared, and they will not in the foreseeable future. However, their responsibilities are been significantly extended. With the expanding number of tools and security mechanisms involved today, the SOC is becoming the focal point of cybersecurity control for the overall organization. This is where all technologies and processes converge to provide a holistic, real-time and actionable view of cyber-security protection, integrating information from in-the-field detection and response tools, Threat Intelligence, as well as audits, vulnerability scans and pen tests. As an example, one Thales cus-tomer was hit by an email campaign targeting tens of thousands of mailboxes. Here, the role of the SOC was not so much to filter the infected mails, but rather to raise an alert and investigate the level of protection provided by the automated and distributed mechanisms, such as mail gateways, anti-viruses or EDR. What percentage of infected mails might have reached their targets? What version of anti-virus signature covered all the variants of the payload? Were some end-points more vulnerable than others, requiring more in-depth analysis? The mission of the SOC here consists in providing a complete view of the defense chain and actionable recommendations on where to focus on to tackle the attack.

The SOC is now becoming the cornerstone of the overall cybersecurity defense chain.

THE PARADIGM SHIFT

16


• Governance, risk management and compliance: in coordination with the Chief Security Officers (CSO), the SOC is evolving to ensure consistency between the identified business risks, the security policy and what is seen in reality. Which threats is the organization exposed to? Which ones can have significant business impact? Which ones are really seen? This leads to executive level gover-nance and provides key input to cybersecurity network and system design. Eventually, the SOC becomes a key contributor to the Governance, Risk Management and Compliance (GRC) function.

BUSINESS RISK MANAGEMENT

THREATINTELLIGENCE

EXECUTIVE GOVERNANCE

DISTRIBUTED DETECTION & RESPONSE

CENTRALIZED DETECTION

& RESPONSE

HOLISTIC & ACTIONABLE VIEW

COMPLIANCE

VULNERABILITYMANAGEMENT

THE HOLISTIC SOC

CENTRALIZED DETECTION

& RESPONSE

DISTRIBUTED DETECTION & RESPONSE

TRADITIONNAL SOCNEW

GENERATION SOC

In 2017, massive ransomware attacks such as WannaCry and NotPetya led to huge revenue losses and operational costs, which were recognized publicly by a number of global industry players such as FedEx ($300m), Maersk ($200-300m) or Saint-Gobain (€330m).Thales was on the front line to protect its customers, with extensive support from the Cert-IST Threat Intelligence (TI) services and SOC operations. In the case of the WannaCry attack for example, Thales SOC leveraged TI, vulnerability management and supervision together:• Prior to the attack, the Cert-IST raised

specific high-priority alerts. This triggered scan campaigns performed by Thales to help customers to prioritize patches.

• During the attack, specific markers were searched both in real time and in log history to detect potential infections.

WANNACRY: A RETURN ON EXPERIENCE

14th March 2017

12th May 2017

13th May 2017

14th May 2017

June 2017

THREAT INTELLIGENCE

VULNERABILITY MANAGEMENT

SECURITY MONITORING

INCIDENT RESPONSE

Vulnerability advisory (MS17-010)

Preventive scans

WannaCry Ransomware

Attack

Attack IOC Investigation

Detection scans

Attack TTP Exploit detection

Verification scans

Vigilance

• During the attack as well, information about the attack Threats, Tactics and Procedures (TTP) was leveraged to put in place specific mechanisms in the network and in the SOC in order to stop or detect potential exploits on unpatched systems,

• After the attack, specific scan campaigns were performed to control the robustness of the systems.

Thanks to this continuum between threat intelligence, vulnerability management, security monitoring and incident response, Thales was able to have a clear and holistic view of the malware propagation across its customer base. As a result, amongst the 100+ Thales SOC and IT outsourcing customers, only one infection was discovered and contained straight away.

17


THREAT HUNTING

KEY BUILDING BRICKS FOR CYBERSECURITY OPERATIONS IN COMPLEX ENVIRONMENTS4

As we have seen previously, the new generation of SOCs must bring added value that goes far beyond basic incident detection and ticket generation. To get this right, a number of basics first need to be fixed with respect to vulnerability management, incident response and talent management in order to build the necessary foundations for delivering the expected added value. Then, the SOC must also revamp the approach used to operate on-premise IT and network infrastructures, in order to extend its reach to new types of environments such as public cloud and industrial operations. Finally, the next generation SOC needs to embrace Threat Intelligence, Big Data and Artificial Intelligence technologies, which all have roles to play in the cybersecurity defense chain, as summarized in the diagram below.

VULNARIBILITY MANAGEMENT

INCIDENT ANALYSIS

REAL TIME DETECTION

INCIDENT RESPONSE

THREAT INTELLIGENCE, BIG DATA, ARTIFICIAL INTELLIGENCE: VALUE IN THE CYBERSECURITY DEFENCE CHAIN

THREAT INTELLIGENCE

BIG DATA

ARTIFICIAL INTELLIGENCE

Assess and reduce risksPrioritize remediationReduce time-to-patch

Improve detection rate on know attacksReduce false positives

Enrich incident information

Improve analysis efficiency

Improve efficiency of initial analysis Reduce breach dwell timeImprove 0-day based breach detection

Improve breach detection Reduce breach dwell time

Improve efficiencythrough

enriched alerts

Facilitate communication between teams

Shorten Build lead timeImprove L1 operationsImprove detection rate on both known and 0-days

18



COLLECTS INFORMATIONS INTELLIGENCES CONTEXTUALIZED INTELLIGENCE

CYBER DEFENSE SYSTEMS

BUILDING THREAT INTELLIGENCE TO FEED CYBERSECURITY OPERATIONS

4.1

THREAT INTELLIGENCEIn a complex, changing and interconnected world, understanding the rapidly evolving threat lands-cape is key to adapting cyberdefense mechanisms and operations. Defending an infrastructure against zero-day exploits and Advanced Persistent Threats (APT) remains an increasingly difficult challenge that requires an understanding of the adversary’s goals, mode of operations, markers and potential impacts. Based on accurate and easy-to-access information, defenders can very rapidly define effi-cient mitigation and remediation actions.

For most organizations, gathering, qualifying and organizing this information on their own would be too costly. This is why they need a trusted partner who can provide qualified and actionable Threat Intelligence data that can be used to take both operational and strategic decisions.

From a daily operational standpoint, when triggered by suspicious events from the infrastructure (e.g. IDS, Firewalls, AD, anti-virus), security analysts need to understand the magnitude of the breach, the set of compromised assets as well as the objectives and the characteristics of the attack. Their goals are both to assess the priority based on potential business impact and to propose relevant responses.

Therefore, from an operational standpoint, the value of Threat Intelligence can be summarized as follows:

• Effective Threat Detection: by leveraging TI Indicators of Compromises (IoC), fewer attacks go unde-tected.

• Efficient Threat Analysis: TI provides quick access to relevant information so as to help analysts understand the magnitude and severity of the breach, and the known mitigation or remediation techniques.

• Efficient Threat Response: Threat Intelligence provides immediate information on the known res-ponses to a threat, such as patch information or workarounds.

• Efficient communication and coordination between security teams, whether they are part of a SOC, a CERT/CIRT, IT or network operations, company subsidiaries or third-party providers or partners. By sharing information and accessing the same level of information, all stakeholders understand the context and obtain what they need to perform a specific task.

If you know the enemy and know yourself you need not fear the results of a hundred battles.SunTzu, The art of war

REACTIVITY

DETECTIONDECISION

19



4

At a more strategic level, Threat Intelligence provides information such as threat evolutions, targeted sectors and geographies and evolutions of the attackers’ Threats, Tactics and Procedures (TTP). This information is used by various stakeholders such as:

• Executives, who need information about activities threatening their organization in order to asset the cybersecurity risk and manage related strategic decisions.

• Security managers, CISOs and CSOs, who are in charge of defining and updating security policies and security operational plans.

• Architects, who need to improve the by-design defense of the infrastructure. Threat Intelligence contributes to building security specifications and related test plans.

As we see here, Threat Intelligence is not only one element among others in modern cybersecurity operations — it constitutes a core and fundamental brick structuring all teams, processes and mecha-nisms. Threat Intelligence and cybersecurity operations grow and mature together, by feeding each other constantly with information shared inside and outside of the organization.

All Threat Intelligence implementations are not made equal! As for most of the aspects mentioned in this paper, it is a matter of technology, organization and people. The key here is to ensure smooth and controlled information sharing between all stakeholders.From a technological standpoint, Thales has selected the Threat Quotient technology for its Cyber Threat Intelligence Platform (CTIP). It is the central point of information which aims at serving all Thales groups and SOCs. Thales also provides a Threat Intelligence solution to its customers, where the customer CTIP is connected to the Thales platform, providing information exchange capabilities based on strict confidentiality rules.The CTIP platform contains Threat Intelligence feeds. Thales integrates external or generic Threat Intelligence feeds, such as the one provided by the Cert-IST, operated by Thales.

THREAT INTELLIGENCE IN ACTION

EXTERNAL FEEDS INTERNAL FEEDS

DEDICATED LIBRARY

The key success factor here is not so much the size, but rather the quality and contextualization of those feeds. Customers can benefit from this added value. They can also add their own threat intelligence to the CTIP solution deployed for them, and share part of it back with the Thales experts if they choose.

As we can see, Threat Intelligence is organized to be at the center of cybersecurity operations, thanks to system interfaces with tools such as SIEM, probes and forensics, as well as user interfaces with relevant operation staff and groups.

20


4.2

BEHAVIORAL ANALYSIS, MACHINE LEARNING AND ARTIFICIAL INTELLIGENCE The traditional approach to detecting threats involves applying explicit rules to data gathered from the monitored infrastructure. In this context, an efficient detection mechanism relies on a good set of cus-tomized and tuned rules, based on clear threat scenarios. This approach is based on the fundamental assumption that we have enough knowledge about what we are looking for and about the attackers’ modes of operations, leveraging Threat Intelligence as an example.

However, alternative approaches to threat detection exist. Threat Hunting is one of them. This involves conducting in-depth searches for indicators of compromise in a huge amount of collected data.

Other complementary approaches are based on Artificial Intelligence and Machine Learning techno-logies. For years now, they have been used in cybersecurity tools such as spam filtering, anti-malware and sandboxes. More recently, they have reached the SOC operations, where they make it possible to:

• Reduce the number of undetected breaches (i.e. the number of false negatives),

• Improve human efficiency.

AI makes it possible to detect attacks which have not been thought of explicitly: an attack is detected based on a deviation from the average or normal behavior of the infrastructure. Weekend activities, data exfiltration on a suspicious internet server or attempts to connect as an administrator can typically be detected because they represent unusual behaviors, and not because there are rules in the system to explicitly detect them, as in a SIEM tool. This behavioral analysis is done both in real time and in batch mode on historical data.

AI AS COMPLEMENTARY DETECTION TECHNOLOGIES

HEURISTIC BASED DETECTION

HIGH MATURITY

THREAT HUNTINGD DETECTION

MATURE

BEHAVIORAL ANALYSIS DETECTION

GROWING MATURITY

SIEM

Reliable, low false positive rate

Cannot find what is not looked for

Requires significant turning to be efficient

Finds unexpected issues and unknown threats

Expertise intensive

Finds unusual behaviors

High # of false positives

Relatively easy to evade

Analysis is difficult

DATA LAKE & THREAT INTEL

MACHINE LEARNING & AI

21


AI technologies also help to improve human efficiency on several fronts:

• During the build phase: AI-based detection systems are typically easy to set up as they do not require specific rules to be configured or costly build phases. Likewise, they quickly adapt to new perimeters or evolving environments by leveraging their continuous online learning capabilities.

• During the detection phase: through Active Learning algorithms, human expertise used during a threat investigation is ingested by the system for faster and more accurate detection (see insert on Guavus below).

• During the investigation phase: security analysts use AI technologies applied to Threat Intelligence to enrich alerts and investigate the magnitude of a breach more quickly. This also leads to incident response efficiency improvement.

On the other hand, with the current state-of-the-art, AI technologies bring a number of drawbacks and difficulties. The three main ones are:

• Number of false positives: for example, system administrators, who typically have irregular beha-viors due to their job activities, generate a number of false positives which must be analyzed by human experts.

• Ability to evade: an attack that stays within the normal behavior, such as long and slow attacks, goes undetected. In addition, it is possible to poison the system: an attacker can take advantage of the online learning capabilities of the system to ensure malicious behavior is learned as being normal.

• Difficulties to provide actionable information to analyze potential attacks. The detection system provides a weighted differential between a normal behavior and an actual behavior. However, the information available to start analyzing the reasons and to drill down further is rather limited. This point leads to another difficulty when the supervision system is committed to specific SLAs or must obtain certification.

These drawbacks are due to the uniqueness and asymmetrical nature of cybersecurity from an AI point of view: breach-related events are very rare and hackers are focused at bypassing detection systems.

In conclusion, and beyond the hype generated by these technologies, Thales’s convictions are:

• Introducing AI in cybersecurity operations is not a matter of “if” or “when”. It is a matter of “how”. In that respect, not all AI technologies are made equal.

• AI will not displace rule-based technologies, but rather complement them. Rule-based and SIEM-based detection is and will remain very effective for normal security detection, including security policy control and the definition of scenarios and use cases.

• AI does not reduce the need for human expertise. As a matter of fact, AI helps to find new incidents and improves low-level task efficiency. On the other hand, higher-level expertise is required, for example for analyzing false positives and tuning the system to reduce them.

• Attackers are also start to use AI technologies, for instance to model victims’ behavior and better mimic it. The endless cat-and-mouse game continues!

4

Introducing AI in cybersecurity operations is not a matter of “if” or “when”, it is a matter of “how”.


22


Five petabytes of data per day! That is what the Guavus Big Data analytics platform is ingesting, day after day. Based on this platform, Guavus provides services typically to telecom operators, in particular for customer care, fraud detection and marketing.Guavus is the name of a company based in San Mateo (California) which employs 250 people and was acquired by Thales in 2017. For Thales, the acquisition of Guavus represents a tremendous accelerator of the company’s digital strategy for the benefit of all customers, whether in aeronautics, space, rail, signaling, defense or security.The true value of the company lies not so much in its Big Data platform, but rather in the analytics and Artificial Intelligence algorithms running on top of it — combined with the scarce human expertise of its Data Scientists.

GUAVUS: WHEN BIG DATA IS REALLY SMART!

Addressing cybersecurity detection and response needs, Guavus has developed a specific Adaptive Learning cybersecurity application. While the platform collects as much information as possible, cybersecurity experts are guided by the algorithm to detect potential threats. Conversely, the algorithm automatically learns from the experts about the true positives in order to enhance future manual or automatic hunting.These technologies have already been tested in a Thales Cybersecurity Operations Center’s environment with promising results and will progressive be made available in production environments.The Thales-Guavus solution will show measurable results in detection and response efficiency, while significantly improving the risk landscape.

23


CONCLUSION AND KEY TAKEAWAYS

5

At the end of the day, there is no choice: organizations need to deeply transform their IT and leverage the Digital Transformation to evolve their business. And they need to do it securely. Deciding not to transform because of a lack of security or to transform without taking cybersecurity seriously are two recipes for business failure.

Ensuring cybersecurity in such environment remains a challenge. At Thales, we believe that there is no single silver bullet solution. Our conviction is that the key to successful cybersecurity operations relies on a combination of a number of best practices:

• Fixing certain fundamentals beyond basic security supervision and operations. This includes effec-tive vulnerability management and incident response. It also means focusing on people, talents and expertise as well as on organization, processes and methods.

• Adopting a new approach to address the paradigm shift to Cloud and new OT environments. In this respect, the SOC evolves to become the cornerstone of holistic cybersecurity operations.

• Incorporating disruptive technologies such Threat Intelligence and Machine Learning in order to improve operation efficiency and continue to raise levels cyberdefense.

All in all, the new generation of cybersecurity operations is about mastering the complexity of a num-ber of components, environments and requirements. It is about integrating those components in an efficient and agile manner, in order to address the full cyberdefense continuum in a holistic manner. It is about customizing solutions and adapting them to specific and ever-moving environments, as there is no one-size-fits-all solution.

Managing this complexity is the challenge but also the value of a successful cybersecurity organization.

The new generation of cybersecurity operations is about mastering complexity.

24


THE COMPLEXITY OF CYBERSECURITY OPERATIONS

GOVERN

PREVENT

DETECT

REMEDIATE

Corporate Governance

Business Risk Management

Compliance Management

Red Team

Prevention through user Awareness & Training

Security by Design, incl. IAM, PIM/PAM

Network / Infrastructure based Prevention, Detection & Remediation

(FW, EDR, IDS/IPS, AV, Sandboxes, etc.)

Pen Testing

Security Operations

Incident Response

Breach & Incident Analysis

Threat Intelligence

Forensic Analysis

Crisis MgtAsset Management

Security Policy & Processes

Security Audits: organizations,

network, applications

Explicit Rule based

detection

Threat Hunting

Behavioral Analysis

(AI based)

Vulnarility & Patch Management

Active Scanning

25


Thales is a global technology leader for the Aerospace, Transport, Defence and Security markets. With 64,000 employees in 56 countries, Thales reported sales of €14.9 billion in 2016.

With over 25,000 engineers and researchers, Thales has a unique capability to design and deploy equipment, systems and services to meet the most complex security requirements. Its exceptional

international footprint allows it to work closely with its customers all over the world.

Drawing on renowned cryptographic capabilities, Thales is the European leader in cybersecurity products, solutions and services, and the worldwide leader in data protection.

Its 2,000 cybersecurity specialists meet the information system security and data security requirements of the most demanding government and enterprise customers including critical infrastructure providers.Thales occupies a central role in the digital transformation of society,

with a presence throughout the information security chain. The Group offers a comprehensive range of services and solutions from security consulting and audits, data protection, digital trust

management, cybersecured sytem design, development, integration, certification and through-life management to cyber-threat intelligence, intrusion detection and security supervision with Security Operation Centres in France, the United Kingdom, The Netherlands, Canada and Hong-Kong.

cyber security operation center (CSOC)

computer emergency response team (Cert-IST)

Serving organizations from

countries

THALES SECURITY OPERATIONS IN A NUTSHELL

dedicated SOCs for Defense customers

SOC Specialists

cyber security experts

Leading r&d for advanced technologies in

Recent acquisition of

(realtime big data analytics)

5 1 20

2000AI & Big Data

Guavus120+

3

Thales — 2017BEYOND THE SOC AS A DETECTION CENTER: HOLISTIC CYBERSECURITY OPERATIONS IN THE COMING AGE

Thales20-22 rue Grange Dame Rose 78 141 Vélizy Cedex

beyond the soc as a detection center: holistic cybersecurity … · 2018-05-17 · beyond...

Documents