A Three-Stage Analysis of IDS for Critical Infrastructures

Lorena Cazorla, Cristina Alcaraz, Javier Lopez

Computer Science Department,University of Malaga, Spain

{lorena,alcaraz,jlm}@lcc.uma.es

Dated: November, 2015

Abstract

The correct operation of Critical Infrastructures(CIs) is vital for the well being of society, howeverthese complex systems are subject to multiple faultsand threats every day. International organizationsaround the world are alerting the scientific commu-nity to the need for protection of CIs, especiallythrough preparedness and prevention mechanisms.One of the main tools available in this area is theuse of Intrusion Detection Systems (IDSs). However,in order to deploy this type of component within aCI, especially within its Control System (CS), it isnecessary to verify whether the characteristics of agiven IDS solution are compatible with the specialrequirements and constraints of a critical environ-ment. In this paper, we carry out an extensivestudy to determine the requirements imposed by theCS on the IDS solutions using the Non-FunctionalRequirements (NFR) Framework. The outcome ofthis process are the abstract properties that theIDS needs to satisfy in order to be deployed withina CS, which are refined through the identificationof satisficing techniques for the NFRs. To providequantifiable measurable evidence on the suitabilityof the IDS component for a CI, we broaden our studyusing the Goal Question Metric (GQM) approach toselect a representative set of metrics. A requirementsmodel, refined with satisficing techniques and sets ofmetrics which help assess, in the most quantifiable

way possible, the suitability and performance of agiven IDS solution for a critical scenario, constitutethe results of our analysis.

Keywords: Control Systems, Critical Infrastruc-tures, Requirements, Satisficing Techniques, Metrics.

1 Introduction

Practically all our Critical Infrastructures (CIs) aretoday under the supervision and are dependent onother additional systems whose underlying infras-tructures in turn, rely heavily on the new Informationand Communication Technologies (ICTs) for control.Indeed, ICTs have now become essential elementsin our society because they offer significant bene-fits to improve efficiency, cost reduction and enhancequality of life. Mobile computing technologies, dis-tributed systems, smart devices, wireless communi-cation or cloud-computing are becoming the majordriving forces behind the management of diverse in-formation, allowing a quicker operation of the greatmajority of today’s competitors’ infrastructures andtheir services.

In fact, most of these physical facilities are highlyinterconnected to other national (and international)infrastructures through communication systems, andare managed through ICTs [10]. This new way ofmonitoring makes the present Control Systems (CSs)critical in themselves, where the notion of critical-

1

A. Nieto, R. Roman, and J. Lopez, “Digital Witness: Safeguarding Digital Evidence by using Secure Architectures in Personal Devices”, IEEENetwork, pp. 12-19, 2016.http://doi.org/10.1109/MNET.2016.1600087NMNICS Lab. Publications: https://www.nics.uma.es/publications

ity is intertwined with the nature of the system andits sensitivity to adverse events caused by unforeseenfaults or intentional threats. This also means thatCIs and their minimum services (e.g., water, energyor transport) are also dependent on the effectivenessof the ICTs integrated inside CSs in charge of col-lecting, distributing and processing the correct func-tional performance of resources and the provision ofservices.

Examples of CSs are Distributed Control Systems(DCSs) or Supervisory Control and Data Acquisition(SCADA) systems, and both belong to the categoryof Industrial Control Systems (ICS) [8]. SCADA sys-tems, in particular, are composed of hybrid integralsystems in which a set of control processes is widelydistributed over large geographical areas and the in-formation is centralized at a single point, the SCADACenter. Generally speaking, SCADA systems arecommonly sensitive to a number of threatening fac-tors: (deliberate/unintentional) faults and existingvulnerabilities related to access control, communica-tion or control. All of them may imply not only thedegradation of the minimum monitoring services butalso the neglect of other essential services for society,which could even result in the well-known cascadingeffect between infrastructures [48].

The proof of this is found in annual reports pub-lished by different governments through specific or-ganizations such as the European Union Networkand Information Security Agency (ENISA) [25] andthe Industrial Control System Cyber Emergency Re-sponse Team (ICS-CERT) [52, 53], respectively. Bothreflect the current situation and the severity of poten-tial threats, where the number of specific incidentsapparently continues to grow. This requires an im-mense effort to design protection measures withoutinfringing the five basic control principles defined in[11]: real-time operational performance, dependabil-ity, survivability, sustainability and safety critical.

These five requirements are basic because they en-compass a further set of important conditions, suchas availability, integrity, access to component, com-ponent lifetime, change management and reliability,among others [27]. Many of these requirements arealso included in the guidelines published by the Na-tional Institute of Standards and Technology (NIST)

in [4] for ICSs and for Smart Grids in [7], and evenin the guidelines published by the North AmericanElectric Reliability Corporation (NERC) through itsNERC-CIP series (002-009) [5].

On the other hand, we have so far identified threechief vulnerabilities in CSs: (i) weaknesses in the con-figurations of the CIs (network configurations, secu-rity policies, defense mechanisms, etc.); (ii) architec-tural design of the system; and (iii) errors in the de-velopment, which are generally related to software(SW) and hardware (HW) elements. Many of thesevulnerabilities come from modernizing the TCP/IP-based technologies, increasing, on the one hand, thecomplexities of the CS and, on the other hand, addingnew vulnerabilities to the control.

Indeed, existing SCADA architectures, their de-vices and their protocols have to adjust to thenew technological changes to offer network architec-tures which are distributed, secure and autonomousfor data management in real time, interoperabil-ity between protocols (e.g., Modbus or DNP3 withISA100.11a, WirelessHART or ZigBee PRO) andscalability for the cooperation and collaboration withthe new control technologies. A clear example of thisnew change is found in the new electrical distributiongeneration, i.e, in the Smart Grid generation.

In the Smart Grid, CSs serve as the central edgeof supervision and data acquisition from thousandsto millions of smart devices with direct connection todiverse networks: backhaul, Wide Area, Field Area,Neighborhood Area, and Local Area Networks. Inthis context, backhaul and the Internet are the chiefsources that connect the different sub-domains withthe rest of the networks, including Advanced Meter-ing Infrastructures that characterize the bidirectionalinterfaces between the real world, and the acquisitionand control world.

Through these interfaces it is possible to man-age and interact with smart meters and utility busi-ness systems, substituting the traditional one-wayadvanced meters. This technological evolution alsoshows how CSs are becoming more complex at the dif-ferent levels (functionally, architecturally), and hencealso require special care when adapting new technolo-gies. In other words, finding a perfect connection be-tween systems and guaranteeing secure monitoring at

2

all times requires, for each adapted technology, a setof minimum requirements which should not interferewith the basic control principles.

Specifically, this paper looks at those requirementsthat Intrusion Detection Systems (IDSs) should con-sider since they act as the main way of filtering anddefense in CSs. We also explore the set of metricsthat help determine the compliance level of the re-quirements of the IDSs in relation to the five controlrequirements. It is necessary to focus on detectiontechnologies due to the current problems in findinga suitable IDS capable of offering sufficient meansto not only understand any SCADA vulnerabilityand all SCADA traffic (i.e., property protocols), butalso to guarantee sustainability, coexistence and scal-ability when other types of TCP/IP protocols (e.g.,6LowPAN for Internet of Things) have to be inte-grated within the system.

To the best of our knowledge, the current SCADAliterature does not contain any extensive theoreti-cal study on identifying those IDS requirements thathave to be considered when implementing IDS solu-tions for ICSs. There are only specific-purpose ap-proaches restricted to a set of factors [58],1805, e.g.,the type of observed protocol (e.g., the SCADA IDSdefined within the SPARKS European project canidentify anomalies associated with IEC-61850 andIEC-60870-5 networks [32]), the acquisition architec-ture (centralized, distributed), the level of scalabilityand detection granularity, the type of response (pas-sive, active), or the degree of interoperability.

Given this, the main contribution in our paper isthe provision of an implementation guideline for IDSsolutions for CSs based on the intrinsic characteris-tics of IDSs and in relation to the context features.We also try to assist in the elaboration of futureadaptable intrusion detection applications for thoseICSs that are becoming increasingly complex and dy-namic.

This paper is organized as follows, Section 2 re-vises the special requirements and constraints of CSs.Based on this analysis, in Section 3 we explore therequirements imposed by this critical scenario on agiven IDS solution that would be deployed withinthis environment. In Section 4, we further developour study by determining satisficing techniques that

would help fulfill the previously identified require-ments. In Section 5 we outline useful metrics whichcan help quantitatively evaluate the suitability of thegiven IDS solution for the CSs. Finally, Section 6concludes the paper and highlights future work.

2 The Special Requirementsand Constraints of CriticalControl Systems

As stated, CSs are systems that manage other crit-ical systems (also known as “systems of systems”).Specifically, ICSs are a type of control system, de-ployed to aid in the operation of industrial infrastruc-tures, the services of which are also essential to socialand economic welfare. This feature means that CSscan also be considered as Critical Control Systems(CCSs), where the correct operation of their controlservices against unforeseen and/or dynamic changesis of paramount importance.

2.1 CCS Requirements

CCS are complex systems, and this complexity is dueto several factors: (i) ICSs are composed of multi-ple networks with thousands of nodes in them; (ii)the heterogeneity of the network is high, integrat-ing multiple types of nodes and technologies, i.e.,legacy equipment (e.g., old Remote Terminal Units(RTUs) and industrial sensors) using protocols suchas Modbus/TCP [40] running alongside modern de-vices (e.g., cloud servers, wireless sensors, mobile de-vices, etc.) which use technologies such as Bluetooth[14] or ISA100.11a [45]; (iii) ICS components and sub-systems have many dependencies between them, andCCSs also have interdependencies with other CIs [48].

These factors make CCSs challenging systems tomanage, moreover, they also make the ICSs targetsvulnerable to attacks [25]. In recent years, cyber at-tacks targeting CIs have increased exponentially, asshown in the cases of Stuxnet [38], Duqu [34], theNitro attacks [18] and others [22, 51].

Due to their complexity, CCSs need to be analyzedin order to better understand their requirements, to

3

provide them with adequate protection against themultiple threats they face because of their charac-teristics. In [11], the authors compile the require-ments that a CCS must comply with to achieve theright levels of security and performance. We describethem here, as these requirements form the basis ofour study:

• Real-time performance: CCSs have hard real-time constraints regarding communications, exe-cution processes and system upgrading, as noneof them should cause delays in the system. Thecommunications’ response time is heavily con-strained, sometimes tightened to a maximum ofone millisecond [8]. Additionally, naturally oc-curring faults in CIs, or malicious activities canintroduce delays in the system, something thatneeds to be palliated and reduced as soon as pos-sible.

• Dependability : is “the ability of a system to prop-erly offer its services on time, avoiding frequentand severe internal faults” [11], thus a controlsystem must provide its service despite fault oc-curring. Dependability comprises five attributesthat absolutely have to be observed: availability,reliability, maintainability, safety and security[11].

• Sustainability : as defined in [11], sustainabilityis “the ability of a system to meet the needs ofthe present without compromising its ability tomeet future needs”, i.e., the system must con-tinue to function like the day it was deployeddespite any later updates, upgrades or modifica-tion of its components (hardware and software).

• Survivability : is “the capability of a system tofulfill its mission and thus to face malicious, de-liberate or accidental faults in a timely manner”[11]. Survivability is composed of three mainelements: unsusceptibility, resilience and recov-erability (defined below).

• Safety critical : this is safety related to criticalenvironments; its implementation makes it pos-sible to prevent unplanned effects that the fail-ure of a critical system could inflict on society.

It also relates to the protection against faultscascading from critical infrastructure to another,the so-called “cascading effect” [11, 48].

These requirements are common to all critical en-vironments. These scenarios are highly complex andheterogeneous and, as we have mentioned, they com-bine multiple technologies and are subject to manyrestrictions and constraints. This makes it difficult touse the same state-of-the-art in ICTs as in regular,non-critical information systems. In Figure 1, we rep-resent the aforementioned CCS requirements, graph-ically. In this figure, not only do we consider the hi-erarchies of the requirements and how they are com-posed, we also consider that all these requirementshave the same level of importance or criticality.

Sustainability Safety-critical Dependability

Real Time Performance

Survivability

Figure 1: SCADA Requirements

The reason for the hierarchization of these require-ments is purely semantic, according to the definitionand concepts of safety, security and survivability en-gineering [26]. We understand that if one of theSCADA requirements is not met, the CCS fails toperform its tasks and is incapable of providing therequired service. This failure to meet the SCADArequirements causes failures in the affected CI whichcould also possibly cause a cascading effect, affectingthose related and interdependent infrastructures thatrely on the service affected.

2.2 CCS Constraints

SCADA systems are one of the main types of CCSsused for large, geographically-dispersed distributionoperations, such as electrical power grids, petroleumand gas pipelines, water or waste-water systems[58]. The special characteristics of CCSs, especiallySCADA, as seen in [11], result in special constraintsthat control HW and SW elements deployed within

4

them have to comply with. Fleury et al. [27] iden-tify these constraints and classify them in 5 differentcategories:

• Performance and availability : critical data mustbe available at all times, without delays or jit-ter in data delivery, and it must be reliable andhave high integrity [27]. In addition to theseconstraints, any (cyber) security mechanism im-plemented must be fail-safe so that the failuresof such mechanisms do not result in the failureof the CI.

• Deployment and management : CCSs need tobe highly stable with respect to failures beforethey can be deployed because they govern phys-ical systems with equipment deployed to lastdecades. What is more, their operation cannotsuffer down-time for system maintenance andupgrades in the way that is common to tra-ditional Information Communications Technol-ogy (ICT) systems [27]. Thus, practices suchas SW patching are not trivial in CSs, since it isnot practical (and sometimes impossible) to takedown CCSs to apply security patches [42].

• There are strong computation, space and storageconstraints in CCSs because they were adoptedin the 1960s and although their architecture hasevolved, legacy equipment, SW and protocols arestill working in today’s networks and need to betaken into account [8].

• A common constraint found in control systems isthe strict application timing requirements, someof which require a message delivery time of nomore than 2 ms [47].

• The extra costs associated with security compu-tations, i.e., the ones performed solely to achievea device’s security goals, do not scale well in criti-cal environments, due to the diversity of its manyembedded systems [47].

Due to the above-mentioned constraints, it is nec-essary to provide CCS with special security measures,in a way that is compatible with their requirements.

A. Nicholson et al. [42] defend that Anti-Virus, Fire-walls, Intrusion Detection Systems (IDSs) and Intru-sion Prevention Systems (IPSs) solutions found ingeneral information technology networks are equallyeffective when employed to protect SCADA networks,but they must be tailored to the types of data usedin this environment.

Apart from the use of different communication pro-tocols and data types, tailored security solutions areparticularly critical wherever the resources are con-strained and the security measures applied competewith the SCADA software to perform their tasks.These security measures must respect the responsive-ness aspect of the system, (e.g., a command fromthe controller to actuator should be executed in real-time by the latter), and the timeliness of any relateddata being delivered within its designated time pe-riod, also meaning freshness of data, i.e., the data isonly valid for its assigned time period [58].

3 Requirements for ProtectionSolutions Deployed withinCritical Systems

Since CCSs, particularly SCADA systems, are con-sidered critical assets with a great potential impactfor the society, it is vital to protect their correctoperation and integrity [11]. Traditional means ofprotection for CSs are usually based on the triadfirewalls, Demilitarized Zones (DMZs) and antivirus[8, 58]. However, it is necessary to go a step furtherand deploy monitoring tools specifically designed andadapted to critical contexts [20, 24] which will aidprevention through automatic detection, creating anenvironment of awareness and alert [9, 8].

These mechanisms are the IDSs [8], a security layer(HW or SW), designed to detect ongoing intrusiveactivities in computer systems and networks [57]. Inthe context of CCSs, IDSs are designed to monitornetwork or system activities for suspicious activitiesand produce reports to a management station. TheIDS is also used for other purposes, such as identi-fying problems with security policies, documentationof threats and to dissuade individuals from violating

5

security policies [36].IDS solutions, as part of the SW deployed within

the CCS, are subject to special requirements and con-straints. To provide good IDS solutions for CCSs is agoal that a lot of studies try to approach from manydifferent fields and areas of knowledge, but to thebest of our knowledge, still there is no study thatprovides a holistic point of view of the suitability ofthese solutions to CIs, by contrasting them againstthe requirements of a CCS.

In an effort to deliver such point of view, we havemodeled the requirements that any IDS solution hasto fulfill to be deployed within a CI. This model takesinto account the need to comply with the CCS re-quirements identified in [11], and the constraints ofCCS as described in Section 2.2. These constraintson the CCS impose powerful restrictions on the IDSsolutions which is deployed in the critical environ-ment, then our model will help build a more securecontrol system and satisfy its requirements, throughthe identification and compliance with the identifiedrequirements for the IDS.

3.1 NFR Model Framework

Since this work covers the requirements of a systemat a very high level of abstraction, the modeling iscarried out in terms of non-functional requirements.A Non-Functional Requirement (NFR) is defined as“a SW requirement that describes not what the SWwill do, but how the SW will do it” [19]. Examples ofthese NFRs are SW performance requirements, SWexternal interface requirements, and SW quality at-tributes.

NFRs are difficult to test, therefore, they are usu-ally evaluated subjectively [19]. It is our aim, how-ever, to develop our study further, translating thesehigh-level requirements into quantitative informationwith respect to the IDS solution and its suitabilitywithin the critical environment where it will be de-ployed. To this end, we base our analysis on differentframeworks and guidelines to be able to model ourscenario.

In the first stage of this modeling, we address thestudy of the requirements of the system. To this end,we use the NFR Framework, as described in [19].

This framework is used to model qualitative process-oriented goals, dividing them into non-functional re-quirements, satisficing techniques and claims. Sinceour proposed model implies high-level non-functionalrequirements, instead of goals, we will model soft-goals.

A softgoal is defined as a goal with “no clear-cutdefinition and or criteria as to whether it is satisfiedor not, since NFRs are subjective, relative, and in-terdependent” [19]. For example, Figure 1 representsthe requirements or softgoals for the SCADA system;whereas Figure 2 illustrates two main areas. The tophalf contains the CCS softgoals as identified in Fig-ure 1, while the bottom half of the figure shows theIDS softgoals, needed in a critical scenario.

In other words, Figure 2 shows the application ofthe NFR Framework to describe the non-functionalrequirements or softgoals that the IDS needs to sat-isfy in order to comply with the CCS requirements.All the requirements are organized hierarchically andconnected with other softgoals according to the defi-nition and concepts of safety, security and survivabil-ity engineering [26]. The identification of the soft-goals is a first step towards modeling the require-ments of a critical complex system. The next stageof our study is the definition of the NFR softgoals forthose IDS to be configured within a CCS.

3.2 NFR Requirements for ProtectionSystems

In this section, we define the concepts involved inthe NFRs identified in the model, in order to providemore information about the non-functional require-ments selected for the IDS so as to comply with therequirements of the CCS (cf. Figure 2). These defi-nitions determine the scope of the requirements andwill be used as a reference to later identify the tech-niques capable of satisfying the requirements.

• Responsiveness: is the ability of a functional sys-tem to perform an assigned task within the re-quired time interval [56]. For our purpose, wewill consider that a system is responsive whenit is capable of performing its functions in the

6

Real-time Performance

Scalability

Robustness

Observability

Recoverability

Task conformance

Reliability

Availability

Maintainability

Repairability

Security

IntegrityConfidentiality

Dependability

Fault Tolerance

Accessibility

Performance

Sustainability

Extensibility

Survivability

Unsusceptibility

Resilience

Safety critical

SW Quality

IDS Soft Goals - Critical Context

Critical Control System Soft Goals

Correctability

SafetyCorrectness

Efficiency

Configurability

Operational Environment Compatibility

HW Dependability

Interoperability

Performability

Responsiveness

Figure 2: SCADA and IDS Softgoals

required time intervals, with no (or at least nosignificant) delays.

• Safety : From the point of view of reliability engi-neering, safety is defined as the “degree to whichaccidental harm is prevented, reduced and prop-erly reacted to” [26]. A safety system has themechanisms to prevent, reduce and react to ac-cidental harm that could affect society in someway. A safety critical system is a particulariza-tion of a safety system, where the environmentor the system itself is critical to social and eco-nomic welfare. Safety requirements in CIs mustbe always be complied with, otherwise malfunc-tions in the infrastructures operation could affectsociety and even endanger human life [11].

• Security : is the “degree to which malicious harmis prevented, reduced and properly reacted to”[26]. In CCSs it is assumed that this propertymust be a primary requirement for defense andprotection of control assets, since the securementof CCSs is vital to maintain the normal opera-tion of CIs [20], which in turn, is essential tosociety. It is necessary to consider the security

variables of resources and data:

– Availability : is “the degree to which a sys-tem is in a specified operable and commit-table state at the start of a mission” [50].We also consider the availability of infor-mation, i.e., the information required bythe system is always obtainable or acces-sible. Therefore, availability is defined asthe availability of a system (asset), andthe availability of the information. In ourstudy, we refer to the availability of theIDS and the control resources, and to theavailability of the information when it is re-quired, regardless the filtering and process-ing of the IDS.

– Integrity : is the quality of being honest andincorruptible [6], in computer science it re-lates to the consistency or lack of corruptionin electronic data.

– Confidentiality : describes something pri-vate, or secret [6]. In computer science, it isthe way to prevent the disclosure of privateor secret information.

7

• Fault Tolerance: the system that is fault toler-ant is capable of continuing its operation despitethe occurrence of failures; a desirable conditionwhich guarantees the resilience or self-healing ofthe underlying CCS.

• Scalability : is the ability of something, especiallyan ICT system, to adapt to increased demands(e.g., control of Smart Grid environments) [6].In computer science, it is the ability of a system,network, or process to handle a growing amountof work in a capable manner or its ability to beenlarged to accommodate that growth [15].

• Extensibility : an extensible system “is able tosupport new control components such as newtechnologies, protocols, HW and SW compo-nents, and security services” [11]. Scalabilityand extensibility tend to be mixed up and con-fused, however they represent two very impor-tant challenges for complex heterogeneous sys-tems like CCSs. Extensibility is a maintenanceand updating characteristic which is especiallydesirable in a CI, where the arrival of SW ap-plications and new technologies to CIs such asthe Internet of Things (IoT) or Wireless SensorNetworks (WSN), greatly increases the need toincorporate new interoperable devices and pro-tocols.

• Interoperability : is “the ability of two or moresystems or components to exchange informationand to use the information that has been ex-changed” [28]. Thus we consider interoperable, aheterogeneous set of systems that are capable ofexchanging useful information with each other.

• Maintainability : also serviceability, is the capa-bility of the system to be maintained over timein order to continuously improve it and keep itfree from defects and errors. Maintainability isinfluenced by three factors [26]: (1) correctabil-ity, the ease with which minor defects can becorrected between major changes, while the sys-tem is still in use; (2) extensibility, defined above;and (3) repairability, the ability of a damaged or

failed system to be restored to acceptable op-erating conditions, within a specified period oftime (repair time) [6].

• Resilience: is the capability of a system to main-tain a proper service in the face of faults, as wellas being able to return to normal operation assoon as possible. In terms of survivability, re-silient is the antonym of vulnerable.

• (Un)Susceptibility : is the state of being likelyor liable to be influenced or harmed by a par-ticular thing [6]. In the context of CIs and theIDS deployed within them, unsusceptibility is adesirable requirement.

• Recoverability : refers to the ability to recoverquickly from a system failure or disaster, to thepoint and (if it is possible) to the state of thesystem at which the failure occurred [26]. It isclosely related to survivability, but it alludes tothe capabilities of the system or deployment.

• Efficiency : is “the degree to which something ef-fectively uses (i.e., minimizes its consumptionof) its resources (computing, machinery, facil-ities, and personnel)” [26]. Related to efficiencyis the relative efficiency of two procedures, de-fined as the ratio of their efficiencies. It is fre-quently calculated as the comparison made be-tween a given procedure and a notional “bestpossible” one.

• Correctness: is “the degree to which a work prod-uct and its outputs are free from defects once it isdelivered” [26]. Three main factors influence cor-rectness, namely: accuracy, currency and preci-sion, which are central in CSs in order to guar-antee reliability of data and operations.

• Operational Environment Compatibility : is “thedegree to which a system functions correctly un-der specified conditions of the physical environ-ment(s) in which it is intended to operate” [26].The IDS must be operationally compatible in or-der to be deployed within a CCS, where machin-ery and environmental radiation and noise might

8

interfere with the operation of unprepared sys-tems.

• Reliability : is “the degree to which a work prod-uct operates without failure under given condi-tions during a given time period” [26]. It re-lates to the costs produced by hazards turninginto incidents, and the level of loss of revenuefor the company or the customer. It differs fromsafety, as safety deals with dangerous hazardswhich could lead to severe accidents with an im-pact on society.

• Performance: according to [26] is “the degreeto which timing characteristics are adequate”.Measurements of the quality of the performancecan be jitter, latency, response time, sched-uleability and throughput.

• Performability : is the characteristic of the per-formance of a system, viewed from the point ofview of dependability, i.e., the performance of asystem in the presence of faults over a specifiedperiod of time [39].

• Robustness: is “the degree to which an executablework product continues to function properly un-der abnormal conditions or circumstances” [26].Fault tolerance is one of its main sub-quality fac-tors. Robustness, from a usability point of viewhas four related criteria: responsiveness and re-coverability (defined above), observability (con-sistency and inferability of the internal states ofa system from the external outputs), and taskconformance (support for the tasks establishedby design).

• Configurability : is “the degree to which some-thing can be configured into multiple forms (i.e.,configurations)” [26]. Configurability includesinternationalization, personalization, subsetabil-ity and variability.

• Accessibility : is the degree to which the user in-terface enables users to perform their specifiedtasks [26]. IDSs for CCS must be accessible tothe system administrators to receive updates andmodifications.

• Software Quality : refers to the compliance of theSW with its design and established functional re-quirements [26]. SW quality is guided by goodpractices and guidelines [29], responsible for en-suring a quality SW development process andthat the resulting SW complies with determinedmeasurements and metrics of performance.

• HW Dependability : are the characteristics thatmake the HW of the system dependable (see def-inition above). Three characteristics indicate de-pendable HW [49]: reliability of the HW, avail-ability of the HW and maintainability of theHW. All these characteristics are defined in asimilar way to the same concepts applied to soft-ware.

The previous sections have described the scopeand definitions of the NFRs identified for an IDS forCCSs. We have established and defined the charac-teristics present in a critical environment, and thoserequirements and constraints that restrict the inclu-sion of new components within such a scenario. Inthe next section, we will further develop our studyin order to determine how to comply with the NFRsdefined in this section.

4 NFR Satisficing Techniquesfor Protection Systems

In Section 3 we identified the requirements presentin our scenario, taking into account both the require-ments in CCSs and the ones that constrain the de-ployment of an IDS solution within a critical envi-ronment. The NFR Framework has provided us withtools to represent the NFRs, or softgoals, present inthis scenario. However, we now need to go a step fur-ther in order to find specific ways to analyze whetherthese softgoals can be satisfied, and to find deter-mined techniques or tools to help the IDS achievethis compliance.

Since we are working with NFRs, we have to ad-dress their characteristics to carry out our study. Theproblem is that these goals or properties lack a cleardefinition, as they are usually based on abstract terms

9

which is not very useful from a measurement perspec-tive [43]. It is therefore difficult to assess whether ornot the NFRs have been satisfied, because there is noclear-cut criteria for this evaluation.

Our approach to tackle this problem is inspiredby the Goal Question Metric (GQM) approach [54].The GQM approach is a goal-oriented methodologyfor the identification of measurements in SW engi-neering. It is built upon the idea of decomposingthe problem into several goals, which are further re-fined by questions and metrics for answering them.We follow this idea to continue our study on the IDSsoftgoals for CIs, in order to bind these abstract char-acteristics to specific practices.

Instead of refining our analysis in terms of ques-tions at the operational level of the GQM, and inline with the NFR Framework, we reflect on the op-erations taken for reaching the identified softgoals interms of satisficing techniques. Thus, in this section,we aim to identify those techniques that can be imple-mented by the system (in our case, the IDS), capableof satisficing the established NFRs. Table 1 presentsthe simplified matching of the satisficing techniquesfound for the softgoals of the IDS, directly linked tothe requirements of the CCS.

Table 1: Satisficing techniques

CCS Requirements Techniques

Dependability

Real-TimePerformance

SW Optimization HW OptimizationDesegmentation PrioritizationLoad Balancing TestingUse of good practices

Survivability

Redundancy DiversityReplication ReactionRestoration IntelligenceSelf-Healing Self-Consciousness

Safety Critical

Isolation RedundancyReplication PrioritizationDesegmentation RestorationUse of good practices

SustainabilityStandardization TestingModularization Design for assuranceUse of good practices

This simplified matching to the NFRs of the CCScan be done because the use of these satisficing tech-niques for the IDS will, in turn, make the IDS complywith the requirements of the CCS. To better under-stand the scope of the satisficing techniques, we pro-vide a brief description of each of them:

• Isolation: isolated systems are not connected toother systems or networks (e.g., the Internet).Currently, actual isolation is almost always im-possible, since new technologies (e.g., the In-ternet, remote management, etc.) are incorpo-rated to help manage CCS. Relative isolation isachieved through the deployment of protectionlayers (e.g., firewalls, DMZs) to only allow per-mitted communication traffic to the most criticalsystems of the CI.

• Redundancy : is the inclusion of extra compo-nents that are not strictly necessary for the nor-mal operation, in case of failure. When the pri-mary devices stop working due to a fault, thesecondary components are activated to main-tain the normal operation of the system, whilethe primary ones are under repair. Redundancycan be implemented by introducing exact copiesof primary components in the system, or us-ing components of different natures as redundantones in order to maximize diversity.

• Replication: replication implies providing mul-tiple identical instances of the same system (ortask), all of them running in parallel. Replica-tion benefits performance and availability (seeSection 2.2), since replicated components helpwhen there are peaks of activity. The use ofreplication implies that all the replicated systemsare always running to balance their workload,in contrast to redundancy, where the additionalcomponents are put in place to ensure the con-tinuation of the operation even if a system isbrought down by a failure.

• Prioritization: is the establishment of prioritiesamong processes in a system, to ensure that themost critical ones always have available the as-sets they need to operate properly. In CCSs,critical tasks need to be taken care of as soon aspossible, this is made possible through the orga-nization of tasks according to their priority.

• Desegmentation: implies uncoupling the pro-cesses in a system, so that they are not interde-pendent. This technique builds robustness into a

10

system, as independent processes are less likelyto spread a cascading effect.

• Restoration: means to return a system to a for-mer or original condition after the occurrenceof a failure. Restoration is central to the needfor mitigation and recovery techniques. Inter-national organizations provide guidelines to im-prove recovery capabilities in CIs [21, 41].

• Development guided by good practices: compli-ance with good practices [55] and standards[16] when designing and developing components(e.g., the IDS) for CCSs is of great importance,since the new system has to adapt to a complexenvironment without introducing new risks.

• SW Optimization: is the application of optimiza-tion techniques to the SW that is to be deployedin a critical context. Optimization should beguided by good practices.

• HW Optimization: implies that the deployedequipment has sufficient capabilities to performthe required tasks, and that they are used prop-erly. All HW deployed in a critical context haveto comply with the requirements of the environ-ment, and be operationally compatible.

• Load balancing : is the distribution of the work-load across multiple resources to maximize thethroughput, minimize the response time and pre-vent overloads. The use of techniques such asreplication can help balance the workload forCCSs’ centralized systems and prevent any vi-olations of their real-time performance require-ments.

• Testing : provides the certainty that the testedsystem has determined capabilities. All SWcomponents deployed within CIs must have beentested for errors to assess the compatibility withthe environment.

• Diversity : implies providing different implemen-tations of the same specification (HW or SW),and using them as replicated systems to cope

with errors in a specific implementation. Diver-sity complements the techniques of replicationand redundancy.

• Reaction: a system is reactive when it provides“a response to some foregoing action or stimu-lus” [6]. IPSs are specialized in reaction, and usethe information offered by the IDS to decide thebest response strategies. IPSs for CCSs are stillsubject to research, since automatic reaction ina critical environment could introduce new risksinto the system and cause cascading effects.

• Intelligence: generates solutions capable ofadapting to new circumstances and acting moreefficiently against any anomalous occurrence.Machine learning can provide the IDS with intel-ligence, a vital capability to implement efficientand accurate IPSs.

• Self-Consciousness: allows the system to con-tinuously monitor itself and its internal states,building security into the system. This charac-teristic helps provide early detection capabilities,even in the presence of sophisticated or stealthyattacks against the IDS.

• Self-Healing : self-healing systems are able to de-tect a malfunction and to react to it, returningto their normal status and operation. It comple-ments self-consciousness, helping build robust-ness and preventing faults in the IDS that couldaffect the normal operation of the CCS.

• Standardization: makes the system or processcompliant with standards, and is aligned withthe development guided by good practices. Theimportance of assuring the deployment of good(certified) quality components within CIs is vitalto guarantee that no IDS’s failures will compro-mise the normal operation of the surveilled CCS.

• Modularization: is the design of a whole systemas a set of different modules. It makes the sys-tem versatile, providing the means for modify-ing the structure and functionality of the systemby adding or removing modules. A modularizedsystem is also easier to maintain, given that the

11

complete functionality of the system is not com-promised when a module needs modifications.

• Design for Assurance: refers to provisioning evi-dence for compliance to governing rules, and thatthe governing rules provide appropriate groundsfor trustworthiness [17]. It is based on assurancecases, which make easier the accountability andthe evaluation of the compliance to good prac-tices and standards easier.

Redundancy DiversityReplication ReactionRestoration IntelligenceSelf-

healingSelf-

Consciousness

Resilience Reliability SecurityPerformabilityRecoverability Unsusceptibility

Fault Tolerance

Availability

Integrity Confidentiality

Maintainability Accessibility

Extensibility Repairability CorrectabilityHW

Dependability

Survivability

Figure 3: Survivability, softgoals and satisficing tech-niques

Table 1 describes the simplified matching of thesatisficing techniques to CCS requirements, now wemodel how these techniques satisfice the IDS soft-goals, which, in turn is related to the SCADA re-quirements. Figures 3 through 7 represent these rela-tionships. We have divided the satisficing techniquesand IDS softgoals taking into account the CCS re-quirements they are related to for the sake of clarity.

Figure 3 represents the relationship between thesatisficing techniques and the NFRs related to thesurvivability of the CCS. The satisficing techniqueswhich help the system to comply with these require-ments are those that build resilience into the system,e.g., redundancy. Figure 4 presents the satisficingtechniques linked to the real-time performance of theCCS. Some of the corresponding softgoals are relatedto the efficiency, performance and availability of thesystem, they try to optimize and balance the generalperformance of the system, to avoid peaks of demandthat the system cannot respond to, e.g., SW and HWoptimization and load balancing.

SW Optimization

HW Optimization

Desegmentation PrioritizationLoad

BalancingTesting

Development guided by good practices

Responsiveness ScalabilityPerformance Availability

Efficiency Performability

SW Quality

Maintainability Accessibility

Extensibility Repairability CorrectabilityHW

Dependability

Real-time

Performance

Figure 4: Real-time performance, softgoals and sat-isficing techniques

Isolation RedundancyReplication PrioritizationDesegmentationRestorationDevelopment guided

by good practices

Fault tolerance

Availability

CorrectnessSafety

SW Quality

Accessibility Maintainability

Extensibility Repairability CorrectabilityHW

Dependability

Safety-Critical

Figure 5: Safety critical, softgoals and satisficingtechniques

In Figure 5 we represent those IDS requirementslinked to the safety-critical of the CCS. IDS softgoalssuch as fault tolerance, safety or availability have agreat impact on the safety-critical of the surveilled in-frastructure. Thus, techniques focused on improvingthe IDS’s safety and correct operation have a goodinfluence on the general safety of the system, e.g.,redundancy, replication, and development guided bygood practices.

Figure 6 illustrates those IDS NFRs related to thesustainability of the CCS. Here, the IDS should com-ply with requirements such as maintainability, inter-operability or scalability. Therefore the IDS shouldbe designed in such a way as to make configuring andmodifying its functionality easy. It should also makerepairs or updates of their SW components easier soas to fix any problems of the system, as in modular-ized systems.

12

Standardization TestingModularizationDevelopment guided

by good practicesDesign for Assurance

ScalabilityInteroperability

Maintainability

Operational Environment Compatibility

Configurability

Extensibility Repairability CorrectabilityHW

Dependability

Sustainability

Figure 6: Sustainability, softgoals and satisficingtechniques

Robustness

Development guided by good practices

ObservabilityTask

ConformanceResponsiveness Recoverability

Fault tolerance

Availability

Accessibility Maintainability

Extensibility Repairability CorrectabilityHW

Dependability

Robustness

Figure 7: Robustness, softgoals and satisficing tech-niques

Finally, in Figure 7, we present those IDS sat-isficing techniques corresponding to the robustnessof CCSs. Techniques that help achieve robustnessare based on the design and development process ofthe IDS, especially the development guided by goodpractices. We can see in the figures that most ofthe aforementioned requirements and satisficing tech-niques overlap in the diagrams presented. This is dueto the separation we previously introduced, to moreclearly visualize our study.

From this study, it is possible to identify the greatneed for standardization and good practices when de-ploying new components within a critical scenario.These practices ensure that the CCSs are not affectedby the addition of IDSs, and that new risks will notbe introduced into the system as a source of unpre-dictable events or faults.

Additionally, as mentioned, there is an identifiedneed for the CI to be protected. Some of the pillarsthat support this protection are the mitigation andresponse capabilities of the infrastructure [21, 41].CCS are able to provide early detection and responseto threats if their IDSs are equipped with sufficientlyintelligent capabilities. The intelligence of the system

will determine its adaptability to new circumstancesand dynamics, providing the CCS with tools to reactto unknown threatening events and restore the CI toits normal operation.

5 Metrics for Protection Sys-tems

Throughout our approach, we have concentrated ourefforts on studying the requirements present in CCSwhich influence and constrain the deployment of anIDS within the infrastructure (see Section 2). Oncewe have studied the NFRs and constraints of CCSs,we can identify the requirements an IDS solution hasto comply with to be deployed in this environment(see Section 3). Our approach has followed the NFRFramework [19], which allows us to model the soft-goals of the system.

The main problem of NFRs is that it is difficultto ascertain whether they have been satisfied or not.Their definitions have a high level of abstraction, sothe constraints they impose on the system are notquantifiable. Since it is our aim to bring our study toa more specific and quantifiable area, as mentionedin Section 4, we have based our analysis on meth-ods such as the NFR Framework [19] and the GQMapproach [54].

In Section 4, we have outlined a set of satisficingtechniques, inspiring the materialization of the oper-ational level on the GQM approach [54]. Followingthis methodology, we now address the next level ofthe GQM, the quantitative level. This stage tries toidentify different metrics to evaluate the suitability ofthe IDS for critical environments. Metrics, as denom-inated in the field of SW engineering, are quantitativemeasures of the degree to which a system or processhas a given property.

For our purposes, we consider a metric as an eval-uation method for assessing the level of satisfactionof certain non-functional properties in a quantitativeor qualitative way, on the basis of evidence and con-textual input, like, for example, stakeholders criteria[43]. In our analysis, we provide examples of setsof metrics for evaluating some NFRs of the IDS and

13

CCS. This connection between sets of metrics andsets of softgoals is a variation of the GQM approach,because in GQM a set of metrics is used to evaluateeach operational question of the model.

We make, however, this high-level connection ofsets of metrics and softgoals, in relation to the satis-ficing techniques. Otherwise, following the in-depthGQM analysis, the extension of this study would ex-ceed the scope of a scientific paper. Our different setsof metrics are mere suggestions and examples, we un-derstand that we have not been exhaustive, howeverwe do refer to different standards and guides whereit is possible to find extensive information and dif-ferent implementations and formulae expressing de-tailed metrics within these sets.

5.1 Reliability and Availability Met-rics

In this subsection, we present metrics related to thereliability and availability of a system. These metricscan help determine and quantify the availability andreliability parameters of given IDS solution. Withthese measurements, we try to provide quantifiableevidence for several parameters of the system thatcan help determine whether or not the IDS is suffi-ciently reliable and available to be deployed within aCCS.

• Diversity : measures the number of differentimplementations of the same specification, themore diverse a system is, the more resilient tofailures it is. Examples are: number of platformswhere the IDS can run, and number of commu-nication protocols the IDS can understand.

• Replication: measures the number of replicatedsystems that are present in the system. It canalso be applied to a component, in order to iden-tify its level of replication, e.g., RAID systems:level 0 to level 7.

• Uptime: is the measure of the time a systemis working and available, representing the timeit can work non-stop and without maintenance.Examples are: the percentage of time the system

is running and number of hours uptime versusnumber of hours of outage/downtime.

• Downtime: opposite to uptime, it represents theperiods of time that the system is unavailable oroff-line because of unplanned events or mainte-nance routines. An example is the percentage oftime the system is down.

• Availability : is the time that the system is notfailed and not under repairs [33]. In this set weusually find metrics related to maintainability,such as the mean time between failures, whichare discussed below.

5.2 Performance and ResponsivenessMetrics

Metrics related to performance and responsivenessare also useful for assessing IDS within CCSs. Wehave selected some of them as examples, describingthe ones we think are representative of our scenario.

• Jitter : is “the precision of the time when oneor more events occur” [26]. An example is theabsolute jitter, that measures the difference be-tween the instant one event occurs and the idealinstant when it should have happened.

• Latency : is the time it takes to provide a re-quested service or allow access to resources [26],which provides information about the delay ofthe system. Examples are the communicationlatency or operational latency.

• Response time: is the time it takes to initiallyrespond to a request for a service or to accessresources [26]. It is vital for the CCS surveilledby the IDS.

• Schedulability : is “the degree to which events canbe scheduled and then occur at their scheduledtimes” [26]. An example is the acceptance ratio,i.e., number of accepted task with respect to thefeasible ones.

• Throughput : is “the number of times that a ser-vice can be provided within a specified unit of

14

time” [26]. An example is the IDS’s through-put, i.e., the number of items processed by theIDS over a defined period of time.

• Compression Ratio: refers to the reduction ofthe size of the data, performed by a compressionalgorithm. It is usually defined as the ratio be-tween the uncompressed size and the compressedsize.

• Speedup: speedup is the increase of performance(speed) of a parallel algorithm versus its sequen-tial version. In CCSs, it can be useful to mea-sure the performance in the presence of replica-tion mechanisms, where identical instances of acomponent are running in parallel.

• Service Time: measures the time it takes for thesystem to deliver a service to the actor who re-quests it. It is useful to measure the time neededto upgrade the IDS, the time the IDS needs torespond to a query from the CCS, etc.

• Instruction Path Length: measures the numberof machine code instructions required to executea section of a computer program, providing in-formation about the relative efficiency of a sys-tem. It helps assess the complexity versus theefficiency of the IDS modules.

• Completion Time: measures the amount of timerequired to perform and complete a given task.It is aligned with metrics such as response timeand latency. It helps assess the suitability of theIDS for the CCS.

• Channel Capacity : refers to the maximum rateof information that can be transmitted reliablyover a communications channel. It indicateswhether the CCS’s channels have sufficient ca-pacity to include the IDS’s traffic, or if additionaldedicated resources are needed.

• Performance per Watt : is the rate of computa-tion that can be delivered for each watt of energyconsumed. A low ratio of performance per wattis an indicator of the suitability of a componentfor the CCS.

• Relative Efficiency : the relative efficiency of twoprocedures, known as the ratio of their efficien-cies is frequently calculated as the comparisonmade between a given procedure and a notional“best possible” procedure.

• Bandwidth: is a measurement of the data com-munication resources available, expressed in bitsper second or multiples of it. It is related tochannel capacity metrics.

5.3 Correctness Metrics

This section discusses examples of the metrics wehave identified related to correctness. These metricsare especially important for CCSs, since they helpmeasure how good the IDS’ detection process is intodetecting the threatening events, and not confusingthem with harmless system dynamics.

• Accuracy : is “the degree of closeness of mea-surements of a quantity to that quantity’s actual(true) value” [13]. In IDSs, accuracy has to dowith the rate of False Positives (FP) and FalseNegatives (FN) of the system, i.e., the IDS canonly be considered accurate when its rate of FPand FN is minimum or negligible.

• Precision: is a characteristic inherent to themeasurement system, statistically it is definedas the dispersion of quantitative data, regardlessof its accuracy [26]. It is, therefore, the degreeto which repeated measurements, taken in un-changed conditions, show the same results.

• Currency : in terms of correctness, currency isdefined as the degree to which data remain cur-rent, i.e., not obsolete. It is also known as thefreshness of the data[26].

In the same context of correctness, there are twospecific metrics that are truly interesting in the caseof an IDS: sensitivity and specificity. They are sta-tistical measures related to both the accuracy andprecision of a classification system, and are definedas follows [12]:

15

• Sensitivity : also “true positive rate”, measuresthe proportion of actual positives correctly iden-tified as such. Sensitivity shows how good a testactually is by calculating how often the test willcorrectly identify a positive.

• Specificity : measures the proportion of negativesthat are correctly identified as such. This mea-surement shows how accurate the test is withfalse positives, and can be considered as the per-centage of times a test will correctly identify anegative result.

5.4 Maintainability Metrics

This section presents those metrics related to main-tainability. As we have discussed, those metrics havea strong relationship with the set of metrics in chargeof assessing the availability of the system. In this setof metrics we find very specific metrics that are com-monly used in ICT systems, and which can be applieddirectly to CCS or CIs.

• Planned Maintenance: refers to the preventivemaintenance events that are programmed to ver-ify the correct operation of a system. Theseplanned maintenance times either for the CI orthe IDS should be keep to a minimum and alwaysensure the continuous operation of the CCS.

• Repair Time: also “Mean Time To Repair”(MTTR1) is the average time required to repaira system that has failed. MTTR1 has to be keptto a minimum in critical systems, which imple-ment redundancy and replication mechanisms tocompensate the failure of a single component.

• HW Costs: refers to the average cost of replacinga determined component or system. HW costsfor complex CCS are usually elevated, in the caseof deploying an IDS with its own dedicated HW,it must provide high compatibility with the sub-jacent system, in order to provide a sustainableservice for years.

• SW Costs: refers to the average cost that thereplacement of given SW incurs. In the case of

an IDS SW component, it is important to verifyits compatibility and capability of providing asurvivable service over time.

• Restoration Time: is the estimated time re-quired for a system to be restored to its origi-nal operation, in the case of failure. This metrichas an important impact on availability, thus CIsshould have support mechanisms put in place toreduce its time and impact as much as possible.

• Failures Over Time: or failure rate is the fre-quency with which a system fails. When deploy-ing a new component (e.g., an IDS), it is impor-tant to ensure that its failure rate is within anacceptable threshold for the CCS, and that itsfaults will not have an impact on the subjacentCI.

• Maintenance Man-Hours: refers to the man-power needed to maintain the system. In thecase of IDS components, the evaluation of thismetric depends on their deployment (dedicatedHW or integrated SW), since the computationof time will be dependent (or not) on the infras-tructure.

• Upgrade Events: refers to the estimated fre-quency of upgrade events needed in a system.The IDS component should not need frequentupgrades, in order to avoid generating too muchtraffic or demanding too many computation re-sources, in a critically constrained environment.

• Recovery Time: also “Mean Time To Recovery”(MTTR2), refers to the average time a given sys-tem will take to recover from a failure. Similarto MTTR1, it indicates the time lapsed beforethe system returns to its normal operation. Ithas to be kept under a given threshold, in orderto avoid failures cascading to other dependentsystems.

• Mean Time Between Failures (MTBF): is thepredicted lapse of time that occurs between in-herent failures of a system during operation.This metric gives an estimation on when the fail-ures will occur.

16

• Mean Downtime: is the average time the systemis not operative, due to maintenance or a fail-ure. Any down time suffered by a CI is criticalfor interdependent systems, and for society. Inthe case of the IDSs deployed within CCS, it isvital that any faults occurring in the IDS do notcascade to the subjacent CS and the CI.

5.5 Dependability and Safety Metrics

Dependability metrics and safety metrics are reallyrepresentative of CCSs. In this section we presentan example of some of these dependability and safetymetrics that can be useful to evaluate in CCSs. Inthe field of safety, we have found metrics that aremostly based on statistics that provide insight intothe probable occurrence of certain events over time.Dependability metrics have a strong link to those ofsurvivability, mostly related to availability, maintain-ability, etc. For a more comprehensive list of this typeof metric, we refer the interested reader to the IEC61508 Standard [1].

• Mean Time To Unsafe Failure (MTTUF): rep-resents the average time that a system will oper-ate safely before the occurrence of a failure thatproduces an unsafe system state [23]. MTTUFshould be as high as possible, indicating thatthere are few probable unsafe failures for thewhole system, and thus a robust dependable sys-tem.

• Reliability : as a function of time, or survivorfunction R(t), is the probability of a systemwhich does not fail in a determined time inter-val [23]. From R(t) it is possible to compute thereliability of the whole system.

• Stability : metrics, such as Mean Square Stability(MSS) [30], refer to the equilibrium and stabilityproperties of a system. Systems with high sta-bility will suffer less faults and provide a betterservice.

• Safety specific metrics: compliance with metricssuch as the safety design stability metric, or thesafety requirements traceability metric [35] can

ensure that the IDS component deployed withina CCS is a safe system.

• Safety Integrity Level (SIL): metrics that areavailable at IEC 61508 [1] allow developing sys-tems observing the level of safety of the system.IDS solutions complying with the specificationsof this standard will ensure that the SIL of thesubsystem is adequate for the CCS.

• Percent System Safety Hazards: measures thesafety hazards of the system against the system’shazards [35]. The lower this percentage is, thesafer and more robust the CI is; and thus, itscomponents (e.g., an IDS).

5.6 Security Metrics

The last set of metrics we have identified are thosemetrics related to security. These metrics are usuallystrongly linked to the specific context where the se-curity is to be implemented. In our study, they aredistributed among the other sets of metrics. Accord-ing to the NISTIR 7564 [31], the security metrics arebased on two aspects: correctness and effectiveness.

• Correctness: usually related to the process of thedevelopment of the system, and to the compli-ance of its operation with the expected behav-ior. Standardization, quality assurance or devel-opment guided by good practices are targetedto improve the correctness of the system, andtherefore to improve their security.

• Effectiveness: measuring effectiveness is usuallybased on the security-enforcing mechanisms, de-termining how well they function and if the sys-tem shows signs of vulnerabilities. The afore-mentioned metrics can help evaluate the effec-tiveness of the system, and thus help gain insightinto the security of the system.

5.7 Discussion

The main objective of this study is to provide a struc-tured analysis which illustrates the steps to follow to

17

determine if an IDS solution is suitable for deploy-ment within CCSs. Furthermore, with little modi-fication, it is possible to extend this work in orderto determine whether or not any given componentis suitable for deployment within a CI. To this end,we have structured our analysis based on the guide-lines of two main methodologies, the NFR Frame-work [19] and the GQM approach [54]. This studyhas been divided into three stages: the analysis of re-quirements, the identification of satisficing techniquesand the identification of representative metrics.

The first stage has been developed in Section 2,where the CCS requirements and special constraintsare identified. In Section 3 this analysis has been ex-tended in order to discern the requirements imposedon an IDS solution we wish to deploy within CCS,always taking into account the previously analyzedCCS requirements. The requirements have been an-alyzed following the NFR Framework guidelines [19].Here we found that the IDS is required to complywith numerous softgoals which constrain the featuresof the current IDS. The compliance with the NFRsprovides, indicates an IDS that is suitable for deploy-ment in critical environments such as CCSs.

These NFRs, in contrast to functional require-ments, represent high-level characteristics and infor-mation about the system under study. In SW engi-neering, the capture and definition of requirements isan iterative process that is refined in each cycle. Inour work, we have performed several iterations to de-termine the NFRs that are most suitable for a genericCI (see Figure 2), stopping our analysis at this point.If this process is continued, given a concrete CI (e.g.,Smart Grid) and an instance of the IDS tool to eval-uate, a further cycle of refinement focused on thisconcrete scenario would provide the final NFRs thatdefine the needs of this particular infrastructure andIDS.

Our analysis, however, stops before this last cycleof specialization, remaining purely theoretical, sincewe would like for our study to remain as open and ver-satile as possible to be applied to the different typesof CIs in existence. Once the NFRs have been refinedaccording to a given concrete scenario, a validationprocess should be done to determine the completenessand validity of the NFRs selected. However, as previ-

ously stated, due to their abstract nature, NFRs aredifficult to define and test, therefore, they are usuallyevaluated subjectively [19].

Validation of some NFRs can be done through the-oretical approaches, e.g., network security require-ments can be evaluated from a game-theory point ofview, in the form of a game between attacker and de-fenders using the Nash Equilibria [44]. Or through anexpert group interview process (Delphi method [37],checklists [46]) to determine the adequacy of the se-lected NFRs for the problem at hand, and to haveaccess to feedback about the completeness of the re-quirement set.

Since we consider that the final refinement cycleof NFRs should be done taking into account the realcharacteristics of a CI, we provide our study as aguideline for future reference when studying and an-alyzing the requirements and constraints of concretecritical scenarios. Thus the final refinement cycle ofthe NFRs and, in consequence, the validation processof the set of requirements selected remain as futurework.

The second stage of our study provides insights intothe way to comply with the identified NFRs (see Sec-tion 4). We have outlined the satisficing techniquesproposed to achieve and satisfice the softgoals identi-fied in the first stage of the study. We have thereforedescribed sets of techniques that help achieve certaingoals for the IDS, e.g., standardization, testing anddevelopment guided by good practices are techniquesthat help the IDS be more robust and sustainable,hence helping the system achieve sustainability.

In an effort to concretize the results of a studybased on abstract characteristics (NFRs), we use theGQM approach [54] to identify sets of metrics thatcould help analyze from a quantitative point of viewthe selected set of requirements. The metrics we haveidentified for our study are merely examples of im-portant quantitative information that can be used toevaluate a critical system, and any component thatwe want to add to this system, especially to ensurethat the newly added sub-system will not introducenew risks and threats into the CI. This part of ouranalysis is developed in Section 5.

We have separated our metrics into five large setsof metrics for the sake of clarity, however, as we have

18

shown, they can overlap. On the other hand, thesefive large sets have been characterized from a high ab-straction level due to the scope of this research, whichends in a refinement cycle before the concretizationof the scenario. In practice, it is necessary to for-malize the exercise by considering, for example, thetemplates offered by ISO 27004 [3] or NIST SP 800-55 [2]. Both standards provide sufficient guidanceto compute, through more tangible calculation, thequantitative level of a study resulting in concrete setsof well defined metrics. In this way, the refinement ofthe metrics offers the means to develop and use as-sessment measures to determine the effectiveness ofan information system and its controls.

It is important to note, that any metric we wantto apply to a given system (e.g., CIs, the cloud, theIoT) has to be validated in order to learn whetherthese metrics are suitable for this system’s evalua-tion, i.e., if it makes sense to use these measurementsin this scenario, and whether or not they are repre-sentative for what we want to validate. However, thispart of the study needs the refined final set of NFRsthat takes into account the actual CI and the con-crete IDS solution, which compose the final scenarioof application. In our study, we have not included thevalidation stage of our proposed metrics. For each ofthem, we have discussed and exemplified the useful-ness of the measurements for our specific scenario,the IDS for CCS.

It is possible to tackle the validation of the met-rics using two methods: through a formal process ofvalidation, or through an expert group interview pro-cess, such as the Delphi method [37]. In such a com-plex scenario, where numerous actors and interde-pendencies are in place, we consider the most viablemethod of validation for these metrics is through theexpert group interview process, a task that addition-ally can result in very valuable feedback for the worldof academia and also for the CI’s management. Asupplementary study which could provide additionalinformation in the validation process consists in test-ing the selected metrics within other non-critical en-vironments.

The results of this experimentation would provideadditional insight to the experts when reviewing themetrics for the given IDS within a CI. This informa-

tion would provide the experts an overview of, forexample, the performance of the IDS in general net-works, indicating whether the IDS solution is a prioritoo costly for a critical setting or not. However, theseexperiments are difficult to set, and of course, remov-ing the IDS from a critical scenario would only be rel-atively useful in terms of evaluating the metrics. Thisis because the requirements of the CI could constrainthe application of the IDS too much, which wouldperform perfectly in a general environment. Thus,due to the need to refine and validate NFRs accord-ing to a determined scenario, and the need of con-cretize the metrics as previous steps, this last stageof the study also remains as future work.

6 Conclusions

In our work, we have performed an analysis of therequirements and constraints that CIs force on anysecurity component that we want to include to theCCS of the infrastructure. We have particularly fo-cused on the inclusion of IDS solutions, and the char-acteristics such a tool should have in terms of NFRsin order to be added to a critical scenario. To thisend we have followed a three-stage analysis where wehave defined the NFRs that characterize the IDS forCCSs, a set of satisficing techniques that helps tai-lor an IDS solution to the critical setting, and lastlyseveral sets of metrics that quantitatively assess thesuitability of the component for critical scenarios.

To the best of our knowledge, this is the first at-tempt to formalize from an SW engineering point ofview the requirements (NFRs) that have to be con-sidered when implementing IDS solutions for criticalsettings, while attempting to provide quantifiable in-sight about the adequacy of the solutions. We believethat this is a very useful approach to this problem,in an environment where legacy systems are contin-uously patched to improve their functionalities, usu-ally without taking into account sound SW engineer-ing design principles.

Our analysis stops at a theoretical point, where theactual scenario (concrete CI and IDS solution) havenot yet been defined. This implies that our studyis applicable to different settings, but also that fur-

19

ther refinement of the NFRs and metrics is neededto take into account the particularities of each sce-nario. Moreover, since the refinement cycles are notfinished at this stage, the necessary validation proce-dures of NFRs and metrics have not been performedin our study, they remain as future work. Valida-tion of NFRs and metrics are difficult, we believe avalid strategy would be the expert group interviewprocesses, such as the Delphi method, providing ex-perts with the sufficient insight into the concrete sce-nario through simplified simulations. From our pointof view, our three-stage analysis and the outcome ofthis study as a model would help sustain the improve-ment of security within CIs by ensuring the introduc-tion of efficient and compatible components withinthe CCSs.

Acknowledgment

This work has been partially supported by theSpanish Ministry of Economy and Competitivenessthrough the project PERSIST (TIN2013-41739-R),and by the Andalusian government through theproject PISCIS (P10-TIC-06334). The first authorhas been funded by a FPI fellowship from the Juntade Andalucıa through the project FISICCO (P11-TIC-07223). The second author has received fund-ing from the Marie Curie COFUND programme “U-Mobility” co-financed by Universidad de Malaga, theEC FP7 under GA No. 246550 and the Spanish Min-istry of Economy and Competitiveness (COFUND2013-40259).

References

[1] IEC 61508 -3 - Functional Safety of Electrical/ Electronic / Programmable Electronic Safety-Related Systems, 1998.

[2] NIST SP 800-55 - Performance MeasurementGuide for Information Security, 2008.

[3] ISO 27004 - Information Technology SecurityTechniques Information Security ManagementMeasurement, 2009.

[4] Guide to Industrial Control Systems (ICS) Se-curity, 2011.

[5] NERC CIP Compliance, 2011.

[6] Collins Concise English Dictionary. Harper-Collins Publishers, 2013.

[7] Guidelines for Smart Grid Cybersecurity: Vol.1, Smart Grid Cybersecurity Strategy, Architec-ture, and High-Level Requirements, 2013.

[8] C. Alcaraz, G. Fernandez, and F. Carvajal. Se-curity Aspects of SCADA and DCS Environ-ments. Critical Infrastructure Protection, pages120–149, 2012.

[9] C. Alcaraz and J. Lopez. Wide-Area SituationalAwareness for Critical Infrastructure Protection.IEEE Computer, 46(4):30–37, 2013.

[10] C. Alcaraz and S. Zeadally. Critical Control Sys-tem Protection in the 21st Century. Computer,46(10):74–83, 2013.

[11] Cristina Alcaraz and Javier Lopez. Analysisof Requirements for Critical Control Systems.International Journal of Critical InfrastructureProtection, 2012.

[12] Pierre Baldi, Søren Brunak, Yves Chauvin,Claus AF Andersen, and Henrik Nielsen. As-sessing the Accuracy of Prediction Algorithmsfor Classification: An Overview. Bioinformat-ics, 16(5):412–424, 2000.

[13] IEC BIPM, ILAC IFCC, IUPAP IUPAC, andOIML ISO. International Vocabulary of Metrol-ogy, Basic and General Concepts and AssociatedTerms. JCGM, 2008.

[14] SIG Bluetooth. Bluetooth Specification, 2007.

[15] Andre B Bondi. Characteristics of Scalabilityand their Impact on Performance. In Proceedingsof the 2nd international workshop on Softwareand performance, pages 195–203. ACM, 2000.

[16] Joel Brenner. ISO 27001: Risk Managementand Compliance. RISK MANAGEMENT-NEWYORK-, 54(1):24, 2007.

20

[17] Daniele Catteddu, Massimo Felici, Giles Hog-ben, Amy Holcroft, Eleni Kosta, Ronald Leenes,Christopher Millard, Maartje Niezen, DavidNunez, Nick Papanikolaou, et al. Towards aModel of Accountability for Cloud ComputingServices. In Intr. Workshop on Trustworthiness,Accountability and Forensics in the Cloud, 2013.

[18] E. Chien and G. OGorman. The Nitro At-tacks, Stealing Secrets from the Chemical Indus-try. Symantec Security Response, 2011.

[19] Lawrence Chung, BA Nixon, E Yu, and J My-lopoulos. Non-Functional Requirements in Soft-ware Engineering. 2000.

[20] European Commission. COM(2009) 149 - Pro-tecting Europe from Large Scale Cyber-Attacksand Disruptions: Enhancing Preparedness, Se-curity and Resilience. Publications Office, 2009.

[21] European Commission. COM(2011) 163 -Achievements and Next Steps: Towards GlobalCyber-Security. Publications Office, 2011.

[22] Computer News. Chinese Hacking Team CaughtTaking Over Decoy Water Plant. Online News,August 2013. Last Accessed May 2014.

[23] Todd A DeLong, D Todd Smith, and Barry WJohnson. Dependability Metrics to AssessSafety-Critical Systems. IEEE Transanctions onReliability, 54(3):498–505, 2005.

[24] M. Dunn Cavelty and M. Suter. The Art ofCIIP Strategy: Tacking Stock of Content andProcesses. Critical Infrastructure Protection,7130:15–38, 2012.

[25] ENISA. Analysis of Annual Incident Reports2012. Annual Incident Reports, 13:1–30, 2012.

[26] Donald G Firesmith. Common Concepts Under-lying Safety Security and Survivability Engineer-ing. Technical report, DTIC Document, 2003.

[27] Terry Fleury, Himanshu Khurana, and VonWelch. Towards A Taxonomy Of AttacksAgainst Energy Control Systems. Critical In-frastructure Protection II, 290:71–85, 2009.

[28] Anne Geraci, Freny Katki, Louise McMonegal,Bennett Meyer, John Lane, Paul Wilson, JaneRadatz, Mary Yee, Hugh Porteous, and FredrickSpringsteel. IEEE Standard Computer Dictio-nary: Compilation of IEEE Standard ComputerGlossaries. 1991.

[29] Alan Gillies. Software Quality: Theory andManagement. Lulu. com, 2011.

[30] Oscar R Gonzalez, Jorge R Chavez-Fuentes, andW Steven Gray. Towards a Metric for the Assess-ment of Safety Critical Control Systems. In Proc.2008 AIAA Guidance, Navigation and ControlConference, 2008.

[31] Wayne Jansen. NISTIR 7564. Directions in Se-curity Metrics Research. NIST. gov-ComputerSecurity Division-Computer Security ResourceCenter, April 2009.

[32] K. McLaughlin. Intrusion Detection for SCADASystems - the Smart Grid Protection AgainstCyber Attacks. US CERT, 2015.

[33] Balajee Kannan and Lynne E Parker. Fault-Tolerance Based Metrics for Evaluating SystemPerformance in Multi-Robot Teams. In Per-formance Metrics for Intelligent Systems Work-shop, 2006.

[34] Kaspersky Lab Expert. Duqu: Steal Everything,2011. Last accessed, April 2014.

[35] S Phani Kumar, P Seetha Ramaiah, andV Khanaa. Identification of Software SafetyMetrics for Building Safer Software based Crit-ical Computing Systems. International Journalof Computer Science and Application, 2010.

[36] Hui Lin, Adam Slagell, Catello Di Mar-tino, Zbigniew Kalbarczyk, and Ravishankar KIyer. Adapting Bro into SCADA: Building aSpecification-based Intrusion Detection Systemfor the DNP3 Protocol. In 8th Cyber Securityand Information Intelligence Research Work-shop, 2013.

21

[37] Harold A Linstone, Murray Turoff, et al. TheDelphi Method: Techniques and Applications,volume 29. Addison-Wesley Reading, MA, 1975.

[38] Aleksandr Matrosov, Eugene Rodionov, DavidHarley, and Juraj Malcho. Stuxnet Under theMicroscope. ESET LLC, 2010.

[39] John F Meyer and William H Sanders. Specifica-tion and Construction of Performability Models.In 2nd Intr. Workshop on Performability Mod-eling of Computer and Communication Systems,pages 28–30, 1993.

[40] Modbus-IDA. Modbus Application ProtocolSpecification, 2006.

[41] National Institute of Standards and Technology(NIST). Framework for Improving Critical In-frastructure Cybersecurity, February 2014.

[42] Andrea Nicholson, S Webber, S Dyer, T Pa-tel, and Helge Janicke. SCADA Security in theLight of Cyber-Warfare. Computers & Security,31(4):418–436, 2012.

[43] David Nunez, Carmen Fernandez-Gago, SianiPearson, and Massimo Felici. A Metamodelfor Measuring Accountability Attributes in theCloud. In IEEE 5th International Conference onCloud Computing Technology and Science, 2013.

[44] Vicky Papadopoulou and Andreas Gregoriades.Nonfunctional Requirements Validation UsingNash Equilibria. SCIYO. COM, page 41, 2010.

[45] Stig Petersen and Simon Carlsen. Wire-lessHART versus ISA100. 11a: The Format WarHits the Factory Floor. IEEE Industrial Elec-tronics Magazine, 5:23–34, 2011.

[46] A Ananda Rao and Merugu Gopichand. FourLayered Approach to Non-Functional Require-ments Analysis. arXiv preprint arXiv:1201.6141,2012.

[47] J. Reeves, A. Ramaswamy, M. Locasto, S. Bra-tus, and S. Smith. Intrusion Detection for

Resource-Constrained Embedded Control Sys-tems in the Power Grid. Intr. Journal of CriticalInfrastructure Protection, 2012.

[48] S.M. Rinaldi. Modeling and Simulating Criti-cal Infrastructures and their Interdependencies.In System sciences, 2004. Proceedings of the37th annual Hawaii international conference on,pages 8–pp. IEEE, 2004.

[49] Daniel P Siewiorek and Robert S Swarz. Reli-able Computer Systems: Design and Evaluation,volume 3. AK Peters Massachusetts, 1998.

[50] Federal Standard. 1037C, Telecommunications:Glossary of Telecommunication Terms. Institutefor Telecommunications Sciences, 7, 1996.

[51] M. Thompson. Mariposa Botnet Analysis. Tech-nical report, Technical report, Defence Intelli-gence, 2009.

[52] US DHS ICS-CERT. ICS-Monitor Malware In-fections in the Control Environment. US CERT,December 2012. Last accessed, April 2014.

[53] US DHS ICS-CERT. ICS-Monitor Incident Re-sponse Activity. National Cybersecurity andCommunications Integration Center, April 2014.Last accessed, May 2014.

[54] Rini Van Solingen, Vic Basili, GianluigiCaldiera, and H Dieter Rombach. Goal Ques-tion Metric (GQM) Approach. Encyclopedia ofSoftware Engineering, 2002.

[55] Kenneth R Van Wyk. Secure Coding: Principlesand Practices. ” O’Reilly Media, Inc.”, 2003.

[56] Martin Weik. Computer Science and Communi-cations Dictionary. Kluwer Academic Pub, 2000.

[57] Z. Yu, J.J.P. Tsai, and T. Weigert. An AdaptiveAutomatically Tuning Intrusion Detection Sys-tem. ACM Transactions on Autonomous andAdaptive Systems (TAAS), 3(3):10, 2008.

[58] B. Zhu and S. Sastry. SCADA-Specific Intru-sion Detection/Prevention Systems: A Surveyand Taxonomy. In Proceedings of the 1st Work-shop on Secure Control Systems (SCS), 2010.

22