ÉCOLE DE TECHNOLOGIE SUPÉRIEURE UNIVERSITÉ DU QUÉBEC

LITERATURE REVIEW PRESENTED TO ÉCOLE DE TECHNOLOGIE SUPÉRIEURE

IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY

BY ANDERSON RAVANELLO

MODELING END USER PERFORMANCE PERSPECTIVE FOR CLOUD COMPUTING SYSTEMS USING DATA CENTER LOGS ON BIG DATA TECHNOLOGY: A

LITERATURE REVIEW

MONTREAL, ,

This Creative Commons licence allows readers to dowload this work and share it with others as long as the author is credited. The content of this work can’t be modified in any way or used commercially.

MODELING END USER PERFORMANCE PERSPECTIVE FOR CLOUD

COMPUTING SYSTEMS USING DATA CENTER LOGS ON BIG DATA

TECHNOLOGY: A LITERATURE REVIEW

Anderson RAVANELLO

ABSTRACT

End user performance management for information systems is a complex task that involves engineering and business perspectives that are not always aligned. The software engineering perspective approaches performance management supported by the ISO/IEC 25000 family of standards, whereas the business perspective lacks such a formal standardization. Business perspective, on the other hand, encompasses socio-anthropologic factors such as user training, familiarity to technology, fitness to task and adhesion to technology. All these components are added by the recent adoption, by the companies, of the cloud computing technology that is architecturally more complicated, containing more and inter-related components. This research aims design a measurement model that, employing a performance management framework, is able to measure the end user performance perception of a system when operating on a cloud computing technology. This objective is to be achieved via the analysis of data center logs with the utilization of Big Data technology.

INTRODUCTION

Performance1 measurement of information systems is a challenging research topic.

Measuring quality of information systems has been a concern, for organizations, for a long

time, and, measuring the quality of software, systems and Information Technology (IT)

services is challenging (Juran J, 2010). This is, in part, caused by both of the immaturity of

IT as a science and also because the organizations seldom are able to follow up with the

rapidly evolving field. (HP, 2013)

Software measurement can be conducted from different perspectives: the internal

perspective, measures the quality of how well built and maintainable is the systems and; the

external perspective that is interested on how well the system fits its technological

environment; and its quality when used, being the actual utilization of the system by users in

achieving their particular business goals (ISO/IEC, 2005) These perspectives are documented

in the ISO/IEC 25000 family of standards.

There is a difference between the software engineering perspectives of software performance

measurement and the organizations – or business – perspective to software and IT

performance management. The Software engineering perspective leans toward software

quality, using internal, external and quality in use concepts states that a software created with

high internal quality has a better potential for offering high external quality, as long as it is

well integrated to its operating environment. If this is achieved, it then has a better potential

for achieving a high quality in use. It also adds that if the users are well trained and

comfortable with its use the potential for a quality system will be high. (Stavrinoudis, 2008).

We can see that, for high quality to be achieved, a number of factors must be controlled and

measured to ensure success.

1 Performance is the ability of completing a given task measured against preset known standards of accuracy, completeness, cost, and speed.

2

Alternatively, the business perspective considers software to be a part of the service it

renders to its customers; it is either useful or not to the organization in fulfilling its goals

(Bundschuh, 2008). This perspective, developed by businesses is focused mainly on the end

result of the use of software where end-user satisfaction, where its performance is key: is the

main criteria that the organizations use to state if a software is useful or not. (Glass,

1998)

Software systems performance measurement is currently conducted in many forms. One

popular approach is to use the logs readily available in different operational systems,

applications, computers and IT infrastructure components. Logs are binary files that collect

data from different components in a system and store this data in a file or database for

posterior use. Many commercial, and easily accessible tools, are available for collecting,

analyzing and generating performance dashboards that present technical measures of

different system components that are used by a software (Microsoft, Microsoft Perfmon,

2013), (Blog H. S., 2011), (Omniti, 2013), (Agendaless, 2013), (Tidelash, 2013), (Massie,

2012), (Munin, 2013), (Cacti, 2013), (Nagios, 2013), (Zabbix, 2013), (Observium, 2013),

(Zenoss, 2013), (Forster, 2013), (Weisberg, 2013). How these measures are analyzed and

interpreted, on the other hand, and the measure results on the organizations goals, which is

the business perspective, are still to be resolved and part of the objectives of this research.

(Blog R. U., 2011)

Measuring performance using measures issued from logs can only measure the internal, and

very technical, perspectives of an IT system. This is why the end user2 performance

perspective is often inferred – or estimated, approximated and even guessed - from logs that

affect or not user performance according to the observer’s perspective and experience

(Microsoft, Performance Analysis of Logs Tool, 2012), (Friedl & Ubik, 2008), (Kufrin,

2005). Analysts will agree that whenever a desktop’s processor is showing a measure of

100% of utilization, the end user experience – what the user feels on using that specific

2 The end user is one of the stakeholders of the software, the one who uses it to perform a task.

3

system on that specific moment – is degraded (Bundschuh, 2008). This is not a guarantee to

reflect the true performance experience of the end user; for example scenarios where the user

isn’t interacting with the system, but the performance is deemed as “bad” would represent a

false positive. Scenarios where the experience is considered as degraded, by the user, but are

not properly measured by the internal and external measures can also be measured as false

negatives (Mahmood, 2010), (Tullis & Albert, 2010).

ISO 9141-10 defines user experience as "a person's perceptions and responses that result

from the use or anticipated use of a product, system or service" (ISO, 2009). That relates to

the human emotions, evidencing that user experience is dynamic, context-dependent and

subjective (Law, Roto, Hassenzahl, Vermeeren, & Kort, 2009). Systems performance

measurement, on the other hand, focuses on collecting quantic3 data to determine how the

software and hardware are employed by the users. This data can then be interpreted, what can

include the utilization of benchmarks (Castor, 2006) and the analyst’s empiric, personal

experience in order to determine the occurrence or not of degraded performance ex post

facto. The “bad” or “good” measures conducted by any of the many known tools need to be

interpreted by different stakeholders, exposing something that is (the measure) to how it feels

(the experience). This is a great source of dispute in the performance management discussion

(Law, Roto, Hassenzahl, Vermeeren, & Kort, 2009)

One characteristic that performance logs portrait that should also be discussed is that there is

little or no control over the quality of the design of the measure, whilst pertaining this

research. The design phase of the measurement states that the measure should measure either

the measurand – the perceptible portion of the software execution – or a model of its

interaction with the real world. Additionally, measurement should aim to build consensus on

what will and will not be measured, describing the entity, the attribute, and the adequate

model that characterizes that attribute. In this research the quality of the measure design will

3 Quantitative, measurable and scalable data

4

be achieved by following the activities proposed by (Abran, Software Metrics and Software

Metrology, 2010).

The broad adoption of cloud computing (Weinman, 2009) by the organizations presents

challenges to measuring performance. Cloud computing are complex systems that depend on

different infrastructures that include components that are often dispersed geographically,

with shared elements and running diverse applications (Mirzaei, 2008), (Yiduo Mei, 2010)].

This technology employs hardware and software to deliver ubiquitous, resilient, scalable,

billed-by-use, application agnostic systems (Prasad, 2010). Cloud computing technology is

often categorized in 3 different service models: Infrastructure as a Service (Iaas), Platform as

a Service (PaaS) and Software as a Service (SaaS). Finally, these formats can be hosted

within organizations or supplied by 3rd parties.

These formats bring, to the research, the challenges of collecting data that is physically and

logically displaced, amongst different service providers and over different hardware. A very

common infrastructure for a standard Internet Based application could be:

- A Web page.

- Running on a distributed web server.

- Hosted on a clustered, multi homed hardware.

- Accessing a database that has local and remote contents.

When using cloud computing, issues like the location of the data, the ownership of the

servers, the accessibility of the logs, the security and privacy on the shared resources and the

quality of the service provided became pressing concerns (Prasad, 2010) (Dillon, 2010).

This technology, when measured, generates large amounts of internal and external measures

(Buyya, Yeo, Venugopal, Brober, & Brandic, 2009). When an end user reports degraded

performance of his software operating on Cloud computing, how can a performance analyst

diagnose the problem? What are the techniques and technologies that allow a better

understanding of performance monitoring and performance management for the cloud

computing technology? (Jackson, 2010)

5

Cloud computing performance measurement is currently discussed by different authors.

There are empirical approaches that employ automated software to simulate access to

services, then measuring response times (Sinung Suakanto, 2012); third-party performance

evaluation services (Cloudsleuth, 2013), (Cloudharmony, 2013), comparative tests amongst

different providers (Abel, 2010), internal measurement of different service configurations

over the same infrastructure (Meijer, 2012). Others suggest that cloud performance should be

approached from a business perspective, then going into the internal measures (Croll, 2013).

Each of the above discoveries added to the knowledge base for this research, but the effects

of the performance on the end user perspective are still not measured for cloud computing

applications. Following a Performance Measurement Framework (Bautista, 2012), this

research aims to develop a model that measures the end user perspective in cloud based

systems. This model will include both the internal measures, collected from performance

logs, as well as context-dependent, user interactive measures as a suggestion of previous

researchers (Law, Roto, Hassenzahl, Vermeeren, & Kort, 2009), (Marshall, 2008), (Etezadi-

Amoli, 1996) (Baer, 2011). This is potentially a more complete solution to the problem as an

evolution of previous works.

The log data to be collected in this research tends to be large and increasingly bigger, as each

interaction of measurement should increase the size of the performance databases. One

possible solution for processing large quantities of data is the utilization of Big Data

technology (Cohen, 2009), (Trelles, 2011). This approach is being discussed and has been

proven useful for processing performance logs (Rabl, 2012) and large cluster data processing

(Dean, 2008).

The objective of this research is to design a measurement model that, employing a

performance management framework is able to measure the end user performance perception

of a system when operating on a cloud computing technology. This model shall employ, as

much as it is possible, only data center logs measures due to the industry’s use, convenience

6

of its large availability and technical familiarity that and to its large availability; to .

Additionally, Big Data technology will be leveraged in order to process the large quantity of

data that is generated real-time from the many systems and infrastructural components that

compose the cloud computing environment.

CHAPTER 1

Research Presentation

1.1 Motivation

Managing Information Technology (IT) infrastructure has been a challenge since the first

information systems (technology, data, knowledge and the users) where implemented in the

organizations (Laudon, 2013). Recent pressures such as the emergence of new and highly

mobile technologies, distributed knowledge, real time collaboration and growing competition

have increased its, or, more specifically, have increased the complexity of the decision

environment: in order to be able to compete, are the companies leveraging their information

systems in a way that enables the users to be as productive as possible? The investments that

are required to keep increasingly complex infrastructures are being really executed in a way

that bolsters the firm’s competitiveness?

As we approach information systems as an ensemble of technology, information, knowledge

and people, performance measurement becomes increasingly fuzzier: When performance is

measured, the goal is the discovery of a measure, usually mathematical or percentile, that

explains how a system performs, in the forms of 0% - 100% of a N dimension utilization:

what does 0% utilization means? What does 30% utilization means? And what does 100%

utilization means? Is it better to have a measurement that indicates 0 or 100?

A management’s approach would it be that resources should be applied in such a fashion that

users should be able to fulfill their task based on the Fitness to Task Theory (Goodhue &

Thompson, 1995) while using the least possible resources with the help of Resource

7

allocation matrix theory (Martensson, 2006). For example, measuring in an interval, i.e. 0 –

100 would be just a quantitative way of measuring if a user is capable of completing the

required tasks using the available resources. Finally, this research is motivated by

considering that both resource availability and user capability are directly dependent of the

user’s motivation to actually fulfill tasks as described in the End User Acceptance Theory

(Davis, 1989).

The emergence of cloud computing technologies adds complexity to this situation. Figure 1

and 2 demonstrate the difference between a classical IT client-server versus a cloud

computing technology for an information system environment.

Figure 1 – Common three-tiered client-server architecture (IBM, 2013)

8

Figure 2 – Cloud computing architecture (Lemay, 2012)

In a cloud computing environment, after an end user accepts the task to be performed and

engages in its execution, what components between the user interface and the data

repositories and processors should be gauged in order to actually measure the end user

performance perspective? How to achieve this goal with the least possible customization,

meaning in the case of this specific research, trying to analyze only data center logs.

Performance measurement frameworks for cloud computing applications (CCA) are still in

early research stages (Bautista, 2012). Adoption of cloud computing technology by the

industry is also in its early stages (Phaphoom, 2013) (US General Service Admnistration,

2010). The study of cloud computing management has the potential for innovative research,

particularly with the utilization of recent large volume data processing technologies such as

Big Data. (Lin, 2010)

Understanding this, the motivation for this research is defined by the opportunity of

designing a performance measurement model for cloud computing applications, with the

utilization of Big Data technology. The model will try to include the software application,

infrastructure, network and end user perspectives of performance. To achieve this goal, it will

9

be necessary to identify performance measurement measures, issued from the data center

logs, required to experiment the subject of ‘end user performance measurement for cloud

computing applications’.

1.2 Problem definition

Measuring end user performance is a complex task. First, internal software performance,

measured through a series of quantitative measure, must be correctly designed and validated

in the most efficient way possible. Secondly, these measures must be applied and yield

results within the decision time specified. Third, the measurement result should be exploited/

interpreted by some form of intelligent mechanism that may either be machine or human in

order to infer significance to the measurement. Finally, end user performance is a mix of the

available resources for performing a set of tasks, user motivation and engagement (Hutchins,

1985) (Davis S. &., 2001) and the factors such as training (Marshall, 2008), perceived

usefulness and ease of use (Davis, 1989), support, anxiety and experience towards

technology (Fagan, 2004) that influence the user’s ability to actually perform the task given

the resources available; it’s the translation of quantitative metrics into qualitative

measurements.

Considering such a challenging scenario, the problem definition of this research can be

summarized as: a proposition of a performance measurement model that reflects the end

user experience for an application operating on a cloud computing environment. This

model will use data currently available from data center logs and, because of their large size,

will require use Big Data technology for its capture and experimentation.

1.3 Research question

Given the ample opportunities for discovery in the field of software performance

measurement from an end user perspective using cloud computing technology, this

research focuses on proposition of a performance measurement model, considering

two main perspectives: 1) it is possible to measure and analyze performance of an

10

application operating on the cloud, from an end user perspective, using only data

center logs data; 2) how do we integrate the many internal quality measures reported

in these logs, into measurement model, that would reflect the performance as

perceived by end users. Finally, this measurement model would be explored and

validated using different case studies.

Based on these perspectives, the main research question can be formulated as: How

can end user performance of an application be measured in a cloud computing

environment?

This question is declined in the following specific research questions:

- What defines a cloud computing environment?

- What influences end user performance perspective measurement in a cloud

computing environment?

- Which performance measures, found in existing data center logs, best relate to the

end user experience of a specific application?

- Which performance measurement framework can be used for the creation of a

performance model that represents the end user performance of an application that

uses cloud computing technology?

1.4 Methodology

In order to answer the research questions outlined in section 1.3, the author resorts to the

Basili framework (Basili, Selby, & Hutchens, 1986) used for experimentation in software

engineering, to plan the research and organize its four main research activities phases: 1)

definition, 2) planning, 3) development and operation and 4) interpretation as presented in

the following sections: 1.4.1 to 1.4.4.

11

1.4.1 Definition of the research

This phase, presented in table 1.1, identifies the research motivation, objective, goal and

users.

Table 1.1 - Research Definition

Motivation Objective Goal Users The design of a performance measurement model that reflects the end user experience of an application operating on a cloud computing environment using only data that is currently available from the data center logs

. Define/clarify the notion of end user performance perspective . Define/Clarify the cloud computing technology . Design a measurement model and its toolset to support the infrastructure specialist in proactively managing the cloud infrastructure to identify the performance issues from the end user’s perspective. . Identify the data center logs direct measures that best reflect the end user perspective of an application operating in a cloud;

Design a performance measurement model and its toolset that is generalizable and is still is capable of representing the end user experience of an application operating in a cloud.

Students, researchers, IT professionals and managers.

This next planning phase helps on determining the research problem as well as the specific

research activities that have to be achieved in order to answer the research questions.

1.4.2 Planning

The planning phase contains the description of any deliverables that are required for solving

the problem and answering the research questions. For supporting these findings, the required

literature reviews are also described. Table 1.2 contains the appropriate inputs and outputs of

each.

12

Table 1.2 – Research planning

Planning Steps Inputs Outputs Sate of the art of the concept of user perception of application software performance

Literature review of : . Software Engineering Performance . End user expectation and perception of application performance . End user performance perception, and other psychosocial entities that affect end user performance perception

-Literature review of the state of the art of the end-user performance standards, models, techniques and methods; -State of the art of the end user performance perspective for cloud based computing systems

State of the art of Cloud computing and BigData technology for data center logs processing

Literature review of : . Cloud computing technology, components, types and utilization. . Existing Data Center logs data analysis . Hadoop and Hbase project documentation . REAP project data

- Literature review of existing data center logs use and techniques for its analysis, open source BigData technology and corroboration of the Cloud computing syllabus by matching of components with the experiment’s infrastructure -First publication: Paper on how to measure performance as perceived by the end users that uses cloud applications

1.4.3 Development of theory and experimentation

The development phase of the research presents the activities were new knowledge and

theories are created, the definition and preparation for the experimentation and validations as

well as the main components that foster the answer to the main research question, as

presented in table 1.4.3.

Table 1.3 - Research Development and Operation

13

Development   Validation   Analysis  Design of a measurement model and method specific to the research’s objective

. Identification of individual log measures that best reflect the end-user perspective . Design of the measurement method associated with these measures

Fitness to the framework . Applicability and comparison with commercial performance measurement tools

Identification of an experimentation and validation for the model

. Identification of analysis techniques and feedback mechanism to adequate best with an end-user perspective

. Fitness to framework

. Applicability and comparison with similar approaches according to the literature review . Definition of the scope of the experiments

Application of the performance measurement method rules (framework) over different laboratory test scenarios

. Validation of the measurement method application . Secondary performance measurement data

. Validation of the methodology

. Discovery of structural equations that could explain performance on the studied environment. . Application of big data exploration techniques on the collected data (Paper on application of big data over IT performance indexes)

Results verification and evaluation

. Measurement result analysis

. Adequacy and overall fitness of data to the frameworks . Results of the end user performance perception instruments

. Validation of the framework

. Addition to the framework with the end user perspective Publication of the findings

1.4.4 Interpretation of the results

This section contains the information required for properly understanding the methods, use

cases and scenarios that are explored during the research, as well as providing grounds for

the future research that could be conducted.

Table 1.4 – Interpretation of the results

14

Interpretation  Context                    Extrapolation  of  results                              Future  research  This research is conducted by evaluating the data contained datacenter performance log files; these files might be generated by physical or virtual machines in shared or dedicated cloud workspace. Application clusters are assigned according to the specific use cases tested, for example “all Outlook 2010 users”.

. Different case studies, for larger audiences. . Different sets of measurement variables . Discovery of symbiotic applications in shared workspaces . Machine learning approaches for dynamic work distribution based on end user performance measurement fluctuations

. Can machine learning prevent degradation and resource misallocation? . Is it possible to locate clusters of symbiotic applications (applications that consume different sets of resources, thus optimizing resource utilization)? . Is it possible to locate clusters of symbiotic users? . Can machine learning dynamically assign workloads according to symbiotic profiles?

CHAPTER 2

Literature review

This section presents the topics of end user performance management (xx13) and cloud

computing. The first topic is approached from two perspectives, 1) the software engineering

perspective and 2) the business perspective. Software quality models have long been

discussed (Mccall, 1977) (Boehm, 1976, 1978) (Dromey, 1995) (Grady, 1992) (Jacobson,

1999) (ISO/IEC, 2003) (ISO/IEC, 2005) with the researchers and practitioners gravitating

towards internal and external performance characteristics that should be satisfied in order to

obtain a software product that displays high quality. On the other hand, the business

perspective often relies on the concepts of key performance indicators (Kaplan & Norton,

1992) and Service Level Agreements (ISACA, 2012), focusing in efficiency and user

satisfaction; these two perspectives are overlapping and complementary, both required for the

creation a broad model for performance measurement that is able to measure end user

performance of cloud computing applications. Figure 3 demonstrates a generic ISO/IEC

25000 measure paired with an equivalent strategic map that contains a KPI. Finally, the cloud

computing topic is presented with the state of the art of the literature.

Figure 3 – ISO/IEC 25000 compliant measure versus BSC & KPI compliant measure

Quality model

Characteristic

Sub-Characteristic

Measure / Attribute

Quality in use

Efficiency

Task Efficiency: Time that the user spends on “open file – print”

task

Name: Total Time Numeric goal: How long does it take to

print? Formula: A+ B + C

A: client time B: print server time C: printing device

Theme  :  Contract  Output   Objective  :    

 

Financial:   more   contracts   signed   per  

financial  adviser  work  hour  

Lowered   service   time   /  

client  

Costumer:  less  wait  time  to  signature   Faster   response  

between   deal   and  

signature  

Internal:   Fast   printer   response,   less  

printer  errors,  less  downtime  

Less   event   viewer  

entries  for  printer  error,  

less  service  desk  ticket  

Learning:   Send   the   job   to   the   correct  

printer  

Less  recycled  paper  

Profit improvement via printer performance

16

2.1 Performance management

The software engineering perspective of performance measurement is presented in section

2.1.1 summarizes a review of the most recent ISO reference models. This review is based on

the international standards as well as different issues and limitations known for their

applications. Then, the business perspective of the end user performance measurement

frameworks is described in section 2.1.2. This topic has been popular during the 80’s and its

evolution, current trends and performance measurement tools are presented. Methodologies,

research conducted and their results are discussed in order to uncover potential research and

applicability of the techniques in lieu of the proposed cloud based computing research.

Finally, the limitations and difficulties for these methodologies are discussed.

2.1.1 Performance Measurement – Software Engineering Perspective

This section presents the ISO 25000 family of standards, ISO 15939, the subject of metrics

validation and the difficulties of applying such standards in the organizations. The objective

is the documentation of the completeness of the contemporary ISO 25000 standard as the

confluence of previous standards, the coverage of the ISO15939 measurement process and

the caveats that involve the selection, election and evaluation of the metrics. Finally, an

evaluation of the performance measurement process is executed for the demonstration of the

efforts and challenges involved on applying such standards on the organizations.

What is the quality for a software product? Many authors define and debate quality:

(Shewhart, 1980), (Deming, 2000), (Feigenbaum, 1991), (Juran J, 2010) and others all have

contributed to the creation of a broad definition, reflected in ISO/IEC 9001, where quality is

the characteristic that a product or a service has that defines it as satisfactory to its required

intents.

Measuring quality then requires validated and widely accepted measurement models like

ISO/IEC 9126 (ISO/IEC, 2003) and its superseding ISO/IEC 25000 series (ISO/IEC, 2005)

17

of standards named SQuaRE. Systems and Software Engineering – Systems and software

Quality Requirements and Evaluation (SQuaRE) aims to harmonize many other standards on

software quality: ISO/IEC 9126, 14598 and 15939, complementing and addressing the gaps

between them.

SQuaRE has many groups of documents for different audiences. They are: Quality

Management (ISO/IEC 2500n), Quality Model (ISO/IEC 2501n), Quality Measurement

(ISO/IEC 2502n), Quality Requirements (ISO/IEC 2503n), Quality Evaluation (ISO/IEC

2504n) and the Extensions (ISO/IEC 25050 - 25099). The 5 groupings and its 14 documents

are listed in the next subsection (section 2.1.1.1).

2.1.1.1 ISO25000 (SQuaRE) Grouping and Documents.

This subsection describes briefly the 5 groupings and 14 documents that compose the

SQuaRE international standard on software and systems quality. Figure 3 demonstrates the

groups and documents, as explained through this section.

Figure 4 – ISO/IEC 25000 - Groups and documents, adapted from (ISO/IEC, 2005)

- ISO/IEC 2500n – Quality Management. These are the International Standards

for common models, terms and definitions that are referred by the other

documents of the SQuaRE series. It contains only two documents: 1) 25000,

Extension Division 2505- - 25099

Quality Model Division

2501n

Quality Management

Division 2500n

Quality

Evaluation

Division

2504n

Quality Measurement

Division 2502n

Quality

Requirements

Division

2503n

18

Guide to SQuaRE, pertaining the architecture, terminology overview, parts and

references; and 2) 25001, Planning and management, with the requirements and

guidance for supporting the specification and evaluation of software and system

products.

- ISO/IEC 2501n – Quality Model. Quality models for systems and software

products, quality in use and data, including practical guidance for its utilization. It

contains only two documents: 1) 25010 – Quality model: characteristics and sub

characteristics for product quality and quality in use, derived from ISO/IEC 9126-

1 and 14598-1; and 2) 25012 – Data Quality model: definitions of general data

quality models within computer systems, for data quality requirements, measures,

planning and quality evaluations.

- ISO/IEC 2502n – Quality Measurement. Reference model, mathematical

definitions and practical guidance for quality measurement. The five documents

contained in this division are: 1) 25020 – Measurement reference model and

guide, introductory explanation and reference model for the application of

performance measurement from the International Standards; 2) 25021 – Quality

measure elements, recommended base and derived measures to be used during the

system or software development life cycle; 3) 25022 – Measurement of quality in

use: a set of measures for quality in use; 4) 25023 – Measurement of system and

software product quality: quantitative measures for system and software products

according to the characteristics defined in ISO/IEC 25010; and 5) Measurement

of data quality: quantitative measures for utilization with ISO/IEC 25012.

- ISO/IEC 2503n – Quality Requirements. Specification of quality requirements

to be used in the elicitation for product requirements and inputs for evaluations. It

contains only one document: 25030 – Quality requirements: guidance,

requirements and recommendations for quality requirements based on ISO/IEC

9126-[1-4], 14598-[1, 3-5].

- ISO/IEC 2504n – Quality Evaluation. Requirements, guidelines and

recommendations for product evaluation. It contains four documents: 1) 25050 –

Evaluation reference model and guide: requirements and process description for

19

evaluating system or software products; 2) 25041 – Evaluation guide for

developers, acquirers and independent evaluators: Specific recommendations for

these 3 types of actors; 3) Evaluation modules: structure and contents for

documentation of evaluation modules; and 4) Evaluation modules for

recoverability: external measures for software and systems resiliency and

autonomic recovery.

- ISO/IEC 25050 to 25099 – SquaRE extensions. International Standards and/or

Technical reports addressing specific application domains or complementary to

one or more SQuaRE standards. There are seven document in this series: 1) 25051

– Requirements for quality of commercial off-the-shelf(COTS) software product

and instructions for testing: quality, documentation, test requirements, conformity

and evaluation of COTS software according to the ISO/IEC 12119; 2) 25060 –

General Industry format (CIF) for usability test reports: General framework for

usability-related information: Potential standards for specification and evaluation

of the usability of interactive systems; 3) 25062 – Common industry format (CIF)

for usability test reports: format for reporting measures from usability tests

according to ISO 9241-11; 4) 25064 - Common industry format (CIF) for

usability: User needs report: CIF for reporting user needs with specifications for

the contents and sample format of user needs reports; 5) 25063 – Common

Industry Format (CIF) for usability: Context of use description: high level and

detailed description format for existing or future systems; 6) 25065 – Common

industry format (CIF) for usability: User requirements specification: CIF for user

requirements specifications with relationship in between the specified

requirements; and 7) 25066 – Common industry format (CIF) for usability:

Evaluation Report: specifications of the contents of evaluation reports.

2.1.1.2 ISO/IEC 25010 – Quality in use and Product Quality Models

The quality in use of a system is the result of the internal quality of the software, the

hardware and its operation environments, as well as the interactions between the users and

20

the system. It is influenced and influences the users, the tasks and the social environment that

is created by leveraging the use of the system. The five characteristics that compose the ISO

software product quality model are: effectiveness, efficiency, satisfaction, freedom from risk

and context coverage. Figure 5 demonstrates some of the characteristics and sub

characteristics in a graphical manner for clearer understanding of the internal and external

quality model

21

Figure 5 - Quality in Use and Product Quality models

Effectiveness Efficiency

Satisfaction R

isk Mitigation

Coverage

Quality In Use

Effectiveness

Efficiency

Usefulness Trust

Pleasure Comfort

Economic risk

mitigation Health and safety risk mitigation

Environmental risk

mitigation

Context completeness

Flexibility

Suita

bilit

y

Effic

ienc

y C

ompa

tibili

ty

Usa

bilit

y R

elia

bilit

y

Product Quality

Completeness Correctness

Appropriateness

Time-Behaviour Resource utilization Capacity

Co-existence Interoperability

Appropriateness Recognisability

Learnability Operability User Error Protection

User Interface Acessibility

Maturity Availability

Fault Tolerance Recoverability

Co-existence Interoperability

Adaptability Instalability

Replaceability

Confidentiality Integrity

Non-repudiation Accountability Authenticity

Modularity Reusability

Analysability Modifiability Testability

Maintainability

Portability

22

Sub characteristics are derived from these broader categories. Usefulness, trust, pleasure

and comfort are sub characteristics of satisfaction. Economic, health and safety,

environmental are sub characteristics of freedom from risk. Flexibility and context

completeness are sub characteristics of context coverage.

The product quality model focuses on the intrinsic qualities of the software products, the

computer system and the sub characteristics that integrate the system. The quality in use

model focuses on the interaction between the user and the system and how this interaction

affects the outcomes and operation of a system, whereas the product quality model focuses

on the software and system components and their interaction to influence the results achieve

by the system. One such measure is demonstrated in figure 6.

Figure 6: Quality in use: New Invoice Submission efficiency measure

As an example of this approach to quality, it is possible to abstract a mental model of an

information system: the system being composed by directly related hardware and software,

as well as unrelated software (applications installed on the same machine that are not part of

the information system, for example) and unrelated hardware (other machines that consume

the same network as the target system). The actual information system, composed of

machines, information and people, encompasses both the product quality target as well as the

Quality model

Characteristic

Sub-Characteristic

Measure / Attribute

Quality in use

Efficiency

Task Efficiency: Time the user takes to

submit a new invoice in the web system

Name: Total Time Numeric goal: How long does it take to type and submit?

Formula: A+ B + C + D +E

A: user time B: Local Workstation

C: Network time D: Web Server E: Database

23

scopes of utilization, requirements and evaluation of the users, with the stakeholders directly

influencing on the perception of a system’s quality.

The quality in use model proposed by ISO/IEC 25000, is defined by five characteristics:

effectiveness, efficiency, satisfaction, freedom from risk and context coverage. The

product quality model is characterized by eight properties: functional suitability,

performance efficiency, compatibility, usability, reliability, security, maintainability and

portability. These properties are extensively described on the ISO/IEC 25010 document.

In this research the focus will be efficiency, usability and maintainability particularly time

behavior, task efficiency, resource consumption, end-user time and error occurrence. This

refines the focus and the objectives of the research.

Different stakeholders bring different perspectives on the perceived quality. The stakeholders

can be characterized as primary (direct interaction with the system in order to achieve

primary goals), secondary (content providers, managers, maintainers and installers) and

indirect (output consumers). It is important to differentiate the stakeholders’ approach on

determining the scope of a quality system because the intrinsic differences between

perspectives, knowledge and expectation will define different measures, and resulting

measured results of each one of the characteristics and sub characteristics; while at a given

moment, a different user, like a data center support technician, might be satisfied with a

server performance, it is not guaranteed that an end user, using an application on that server,

will be just as satisfied at the same time. Table 1 describes this relation in regards to the

measures, the mesurands and the expected outcomes.

Stakeholder satisfaction can be greatly influenced by external elements such as user’s

predisposition towards technology, learning, stress levels, comfort, environment of

utilization, and cooptation levels towards the system’s goals. Quality measures, in this case,

might be influenced by the external elements’ influence over its stakeholders. This ‘noise

24

level’ should be explored at the exploitation of the measurement result step in order to decide

if its influence could influence the performance measurement process. (Marshall, 2008)

Table 1: Different stakeholder perspectives for the quality of Time Effectiveness

Stakeholder   Measure  Mesurand  

Expected  outcome  

Primary   user:  

Direct  

interaction   to  

the  system  

Effectiveness:   Time  

to   complete   and  

submit  form  

Browser’s  

response   time  

“document:  

done”  

Typing,   clicking   “submit”   and  

receiving  confirmation   should  be  

completed   without   errors   and  

delays.  

Secondary  User  

Content  

Provider   or  

application  

owner  

Effectiveness:   time  

for  processing  form  

Processor   time  

and   utilization,  

process  stack,    

User   will   provide   proper   data  

that   will   be   processed  

accordingly   to   previous  

benchmarks,   no   extraneous  

influences  on  the  system  

Secondary  User  

Maintainer   or  

support  

technician  

Effectiveness:   time  

for  processing  form  

Actively  

collected  logs  

There   will   be   no   internal   errors,  

crashes,   user   errors   or  

exceptions   that   cause   the  whole  

system  to  be  unstable  

Indirect  user  

Manager  

Effectiveness:   form  

processing  

effectiveness  

Number   of  

processed  

forms   versus  

work  hours   and  

infrastructure  

investment  

The   number   of   processed   forms  

must   increase   whereas   work  

hours   and   investment   in  

infrastructure   lower   per  

processed  form.    

25

The quality in use and product quality models are described by interactive quality

characteristics. These characteristics can be represented from different stakeholders’

perspectives. The stakeholder bias and predisposition towards a system can influence the

fulfillment of the system’s primary goals, thus influencing the quality measure. Quality in

use and product quality are both the ability of a system on satisfying the stakeholder’s needs

as well as the result of the interaction of the aforementioned stakeholders with the system. A

user that is personally unsatisfied with organizational aspects will often present poor

satisfaction with any aspects of the organization, including its systems. (Baer, 2011)

(Buyya, Yeo, Venugopal, Brober, & Brandic, 2009) (Davis S. &., 2001) (Etezadi-Amoli,

1996) (Law, Roto, Hassenzahl, Vermeeren, & Kort, 2009)

2.1.1.3 ISO/IEC 15939:2007 – Systems and software engineering – Measurement

process.

Measuring is an important part of the quality process. It is the measurement process that sets

the objectives, and where progress may be assessed towards the fulfillment of the set

requirements. It is also with the help of measures that it is possible to observe changes like

“improvement” and “deterioration” of the status of quality measures.

The objective of a measurement process is to collect, analyze and report data for decision

making as recommended by the international standards. A successful measurement process

should observe the following stages: organizational commitment towards measuring;

identification of information needs; identification or development of measure sets;

identification of measuring activities; planning for measurement; data collection, storage and

analysis; utilization of the information for better decisions and communication; evaluation of

the measurement process and communication of the improvements on the measurement

process to the process owner. The core activities of the measurement process, as

recommended by ISO 15939, are planning and performing the measurement process itself.

The other activities establish, sustain measure commitment and evaluate measurement

support and extend the core measurement activities.

26

Figure 1 presents the measurement process proposed by ISO. The driver of the process is the

organization’s information needs, whereas the products of the process are the information

products that satisfy the said needs, aiming to support better decision-making. The items

numbered on the figure as 5.1, 5.2, 5.3, 5.4 refer to the activities referred to on the pages 10-

11 of the International Standard, under the topic “3.3 – Organization of this International

Standard”.

Figure 7 – ISO/IEC 15939:2007 - Measurement process

Discovery, creation or election of measures is a process that requires careful validation.

Jacquet and Abran present the validity issues while proposing a process model for software

measurement methods (Jacquet & Abran, 1997). The validation is addressed by three

different approaches: validation of the design of the measurement method, validation of the

application of the measurement method and validation of the use of the measurement results

in a predictive system. This measure validations method is further discussed on 2.1.1.5.

Measurement process is well defined in multiple literature entries and from different

perspectives (ISO/IEC, 2003), (Kaplan & Norton, 1992), (ALinezhad, Masaeli, &

Esfandiari, 2010)). It is one of the axiomatic components of PDCA cycle defined by the

Core Measurement Process

Information Needs

Information Products

Measurement User Feedback

Requirements for Measurement

Information Products &

Evaluation Results

Planning

Information

Information

Products &

Performance

Measures

Commitment

Technical Management Process

Establish & Sustain Measurement Process

Commitment (5.1) Plan the

Measurement Process (5.2)

Perform the MeasurementProcess (5.3)

Evaluate Measurement

(5.2)

Measurement Experience Knowledge Data Base

Improvement Actions

27

ISO/IEC9000, therefore being of great importance for any engineering processes that follows

that standard. Via the application of measurement methods and exploitation of measurement

results it is possible to define improvement points for processes. The design of the measures

must be validated in order to guarantee that the measurements yield pertinent and relevant

outcomes that relate to what is expected to measure (Jacquet J. P., 1998)

2.1.1.4 ISO/IEC 25020 – Software product Quality Requirements and Evaluation

(SQuaRE) – Measurement reference model and guide

The scope of this standard is the selection and construction of quality measures for software

products. Based on the Software Product Quality Measurement Reference Model (SPQM-

RM), software product quality is composed of quality characteristic and sub characteristics

that are demonstrated by software quality measures, acquired from measurement functions

that apply previously defined quality measure elements. Internal, external and quality in use

measures are referred as part of the software product quality life cycle.

Internal software quality measures are defined and implemented during its development.

External software quality measures are related to the behavior of the system where the

specific software product is inserted. Quality in use measures come from the product’s

ability to meet the user’s needs. All these measures should be applied during the software life

cycle to achieve effective software quality management.

Quality measures should contain the following properties: Name, corresponding

characteristic and sub characteristic, measurement focus, purpose statement, decision criteria

for interpretation and action and identification of the quality measure elements used to

construct it. Performance measurement metrics should be validated and have its reliability

assessed. Validation should be inferred from correlations, tracking, consistency,

predictability and discrimination. Reliability and repeatability measure the variations in a

measurement method, both direct and those caused by external sources.

28

2.1.1.5 Software Product Measurement and Measure Validation

ISO/IEC 25000 determines that there are 3 forms of quality measurement for software

performance: Internal, External and Quality in use. Each form possesses different, inter

complimentary primitives and measurement methods, and all of them require validation.

Quality measures are exemplified in ISO/IEC 25022, ISO/IEC 25023 and ISO/IEC25024 for

internal, external and quality in use perspectives.

Internal software quality is related to the intrinsic characteristics of the coding, assembling,

testing, project management, documentation and reporting that is present on a system. It can

be assessed during the early software lifecycle through numerous software engineering

measurement techniques (Haldestead, 1975), (McCabe, 1976), (Tsai, 1986). It does not

allow the inferring of future good software quality, but it allows the early discovery of

software defect and malpractice.

External software quality measures are related to the outcomes of the software development,

deployment, learning, operation, maintainability and adaptability. These measures can be

acquired by third-party applications and external observations over the operation behavior,

often via automated run-time data collection, questionnaires, surveys and interviews. Authors

suggest that high internal quality can influence in higher external quality, whereas low

internal quality will always impact negatively on external quality. External quality is only

measurable when considering the software as part of a system.

Quality in use measures relate to the user’s ability to fulfill their goals by employing the

software and it can be assessed by observing users in real or simulated work conditions. That

can be achieved by simulation of a realistic working environment or by observation of the

operational use of the product. Whereas internal quality measure can be obtained early on the

lifecycle and external quality is measurable on run-time, quality in use can only be

approached by a broader perspective that encompasses both the technical elements of the

software development, deployment and customization as well as the non-technical human-

Secu

rity

29

related factors, such as learning, comfort, satisfaction and trust. An extended list of measures

is presented in ISO/IEC 25020, 25022, 25023, 25024. Refer to Annex 1 and 2 for the

measures currently present on the ISO standard as well as the ones being discussed in the

workgroups.

After analyzing the literature related to metrics validation and scientific measurement, Abran

and Jacqet (Jacquet & Abran, 1997) proposed a process model for software measurement

methods that is defined by 4 steps: Design of the measurement method, Application of the

measurement method rules, Measurement result and Exploitation of the measurement result.

The Design of the measurement method and application of measurement method steps are

subdivided into sub-steps that contain the required tasks for each step.

The first set of sub steps relates to the Design phase: Definition of the objectives, where

what is going to be measured is declared; Characterization of the concept to be measured,

with the definition of the most concrete possible attribute for that concept; Design or

selection of the meta-model, where it is possible to find the description of the entity types

that will be used to describe the software and the rules that allow their identification and

Definition of the numerical assignment rules which will allow the definition of a formal

relation system.

Sub steps are also contained in the application phase: Software documentation gathering

that will allow information gathering over the studied system; Construction of the software

model where the entities for the measurement are referenced according to the meta-model

and the application of the numerical assignment rules.

30

Figure 8 – Detailed Model – Measurement Process (Jacquet and Abran 1997)

During the process model for software measurement lifecycle, the measures must be

validated in the different steps of the process, in different ways. The validation of the design

of the measurement model is required in order to guarantee that the measurement method is

capable of verifying the representation theorem. The validation of the application of a

measurement method can be conducted both a priori et posteriori, relating to the steps 2

and 3 of the process and with the objective of guaranteeing that there is enough information

for carrying out the process as well as technical understanding of the technology and rules

applied. Finally, the more aptly know validation of the predictive system is to be applied in

relation to the step 4 of the model, via the empirical corroboration of relationships amongst

attributes in specific contexts or by the process and theoretical conditions of measurement

methods as predictive systems.

31

2.1.1.6 Limitations and difficulties of using ISO/IEC software engineering quality

models in a typical organization

Many different authors discuss the difficulties of implementing different ISO standards in

different industries (Sousa-Poza, 2009) (CAgnazzo, 2010), (Poksinska, 2002) (Gotzamani,

2005) amongst many others. Most of them gravitate towards a number of challenges that

permeate all industries and all standards: lack of financial and human resources,

inadequate technical knowledge of quality management, lack of knowledge of

formalized systems and lack of ability and experience for conducting internal audits.

Wherever the literature was approached, though, the conclusive results where that, no matter

the effort involved into standardization, the outcomes where positive for the organizations

and the associated stakeholders (Lamport, 2010).

It is interesting to notice these challenges still exist in mature industries such as energy, mass

production and extraction, historically the most mature applications of contemporary

engineering. It is also intriguing that such “simple” factors such as knowledge, ability and

investment are the recurring ones in different ISO/IEC standards and industries. The

discipline of software engineering, when approached from an epistemological perspective,

demonstrate additional challenges, mostly related to its immaturity.

Software engineering as a term became prominent due to a NATO workshop in 1968 (Naur,

1969) where the expression was minted to bring attention to the shortcomings of developers.

According to Mary Shaw (Shaw, 1990), software engineering has been following the

historical evolution of other engineering disciplines as described in figure 9 (Finch, 1951):

artisanal, commercial, scaled, scientific and finally professional engineering. Software

engineering came into existence as an ad-hoc approach to solving problems and then

progressively became more systematic as more “artisans” improved the practice.

The artisanal approach to problem solving, focuses on implementing a solution in any way

possible. It would then become a set of internal knowledge that each artisan would own

differently about how to solve a specific ad hoc challenge. Within time, the peers would

agree in a set of beliefs that, when used against specific problems, would yield known results.

These beliefs, or “best practices” then become part of the practitioners’ folklore, leading to

32

formal codification that would then be turned into models and theories that could, potentially,

lead to improved practices.

Figure 9 – Evolution of engineering disciplines (Finch, 1951)

The great difficulties of utilizing standards like ISO / IEC 25000 on a discipline such as

software engineering are then a combination of the challenges of immaturity with those

innate of utilizing ISO in more mature industries:

- Lack of knowledge of formalized systems influences and is influenced by the

immature state of knowledge in the field; most of the software engineering best

practices are not widely adapted because they aren’t widely known by the

practitioners.

- Inadequate knowledge in quality management causes and is caused by the

difficulties on creating a high quality software product; as demonstrated in table

1, different stakeholders have different expectations on the outcomes of the

software, so even the measurement of “good enough” is elusive. (Bach, 1997)

- Lack of financial and human resources are a cause-and-consequence in its own; if

the software engineering lifecycle cannot make clear how high quality software

systems improve the organization’s outcomes, there will be lesser organizational

commitment in high quality.

33

- Lack of ability and experience on performing quality audits is the result of not

knowing the standards and norms that already exist as well as the new developing

models.

This research aims to address a few of these points: first, the approach of both the business

and the engineering perspectives aim to bridge the gap between business and science, thus

displaying the value, for the organizations, of software engineering high quality standards.

The utilization of big data technology is expected to simplify the comprehension of the

results to the deciding parties. The proposition of a performance measurement model for

cloud computing applications using big data is an initiative that aims to simplify, popularize

and expand the application of software engineering high quality practices.

2.1.1.7 Section conclusion

This section has presented the internal, external and quality in use measurement process,

steps and validation. It is noticeable that, the more external is the measure, the more complex

it is to acquire data to its measurement. Additionally, the validation of measures must be

conducted in order to guarantee that the mesurands are related to the measures and to the

desired outcomes.

The whole performance measurement process is a complex approach to the problem of

understanding the expectations, needs and desires of the organizations and it is not widely

accepted nor employed by the organizations. Its hermetic, engineering specific language does

not easily translate into business, which hinders its broader application in the organizations.

Also, quality improvement and achieving high quality on the software products and systems

don’t always receive the commitment, from the organizations, that is necessary for such a

quality system as ISO/IEC 25000 to be effectively implemented.

It is important to note that different individual stakeholders on the same role can bring

different mental models in relation to quality, satisfaction and success measures. Considering

these perspectives and external influences, modeling end user performance is a challenge on

34

technical and non-technical aspects. Complex systems with interactive elements are operated

by different individuals that fill interchangeable roles towards the systems and these

interchanges might influence the expected quality outcome of a system; whereas a

stakeholder is satisfied when facing a particular quality result, another individual stakeholder

might not be.

Software engineering is still an immature science, and it is evolving in a way that is

analogous to other engineering disciplines. It shares most of the standardization challenges

and difficulties as other, more mature ones, such as lack of financial and human resources,

inadequate technical knowledge of quality management, lack of knowledge of formalized

systems and lack of ability and experience for conducting internal audits. Additionally,

incomplete and evolving formalisms, paradigms and artisanal best practices also help to

increase the efforts of standardization.

2.1.2 Performance Measurement – Business perspective (xx14)

We have seen that performance management, from an engineering perspective, focuses

mainly on designing/identifying valid measures, selecting/designing valid measurement

methods, collecting relevant data using these methods and exploiting the result of the

measures properly. Performance management, from a business perspective, on the other

hand, focuses on the individual managers being able to decide and act, within desired time

frames, upon the results of the measures. Performance measurement for the business

perspective aims to provide information that allows the stakeholders to plan and react

accordingly to scenarios that can be unfavorable to the organization to which they relate.

Whereas the software engineering perspective is interested in the intrinsic quality of the

software product or service, the business perspective measures the effects of the

previous quality upon the organizations ability in achieving its goals.

It is no longer debatable if computer based information technology is an asset or not for the

contemporary organization. Employing, exploiting and controlling the organizations

35

resources via IT is simply inescapable, rendering the organizations increasingly dependent on

its Information Technology infrastructure. As the organizations face their daily challenges for

remaining competitive within the ever shifting panorama that compose the global economics,

managing its resources and, by consequence, its IT infrastructure, becomes a quotidian task.

In an analog way to the measures presented by the ISO/IEC standard, performance

measurement from the business perspective can utilize Key Performance Indicators (KPI) as

one of the many alternatives for management challenges (xx50). Multiple theories approach

its definition, development, creation, documentation and analysis (xx47, xx 48, xx49),

whereas it is largely defined as being an abstract construct, derived from quantitative

measures, that indicates the proximity of the quality level of a working process towards a

desired goal. This definition is divergent from the software engineering perspective as

illustrated in the table 2.1 and further discussed in this section.

There are also different management techniques that can employ KPI’s in order to achieve

better organizational results (xx50, xx51, xx52). One of the most popular techniques is the

Balanced Score Card (BSC), a performance measurement framework that adds strategic non-

financial performance measures to traditional financial metrics that are broadly used by

managers. The objective of the (BSC) is tying in the different measures that, combined,

document and identify an organization’s success while allowing executive action on the

results of individual KPIs. (xx53)

BSC’s employ 4 different – or balanced - perspectives that demonstrate the organization’s

performance: Business process, Customer, Financial, Learning and Growth. Business process

focuses on the internal quality of the processes and how well the outcomes conform to

customer needs. Customer perspective relates to the level of customer satisfaction and or

potential for yet undiscovered needs; it represents how big the organization is and its

potential growth. Financial perspective is the more orthodox approach to performance

management that has been used historically to measure organization’s outcomes. Learning

and Growth includes employee training and corporate cultural attitudes towards the company

performance; its objective is to foster the environment where users – both as stakeholders and

as important corporate resources – are continuously learning and increasing in value.

36

Both KPI’s and BSC’s don’t have set international standards, but both have well accepted

characteristics.

KPIs should be (xx53):

1- non-financial measures;

2- measured frequently;

3- acted by the senior management;

4- clearly indicate what action is required;

5- tied to an specific team for action and remediation (are “owned”);

6- have a significant impact in the organizational performance;

7- respond to action and remediation.

BSC’ should be

1 – widely adopted on the organization;

2 - sources of objective data for business decisions;

3 – adopted and sponsored by the top management;

4 – used to fundament employee training;

5 – drivers of reward and recognition

6 – facilitators for implementing change

7 – analytic sources of information for acting upon corporate problems;

8 – allow the organizations performance management through the organizations

component’s performance measurement.

37

Figure 3 - Example of a Balanced Score Card Strategy map - Kaplan &Norton (1997)

KPI’s are often displayed in an interactive dashboard to foster ease of use and quick

response. Figure 4 presents a generic dashboard. It is important to abstract how each one of

the gauges actually contributes to the organization’s objectives.

38

Figure 4 – Generic Dashboard with KPI’s

Different authors discuss the alignment between BSC and ISO/IEC 9000 family of standards

(xx54, xx55, xx56), identifying positive correlation between the improvement of quality, the

adherence towards the BSC strategy plan and better results achieved. Other authors describe

the harmonization between ISO/IEC 9001 and ISO/IEC 25000. This literature review didn’t

identify, so far, efforts for mapping BSC to ISO/IEC 25000, which could be an opportunity

for furthering the researches, albeit not the objective of this specific work.

There are efforts towards mapping Information Technology Service Management (ITSM) to

the organization’s strategy and international standards, whereas two common approaches are

the best practices contained in Information Technology Infrastructure Library (ITIL) and

Control Objectives for IT and related technologies (COBIT). Most of the literature relate to

bridging the gap between ISO/IEC 20000, ITIL, Cobit and BSC. This indicates that the

organizations identify quality in software products as part of quality of IT service, thus

relating to the quality in use scenario of ISO/IEC 25000 (xx57, xx58, xx59)

39

There are different approaches to performance measurement for IT services and applications

according to the explored literature. Measurement for applications can be performed in a real

time manner or in an off-time manner (xx16). Measurements can be intrusive or non-

intrusive (xx17). Also, measurements can be performed over the different components that

compose the whole information system, inferring a performance level for the whole system

based on its components (xx18).

Real time performance measurement is performed using software components that are

specifically created to allow performance to be captured. That states that the even though a

hardware component is expected to be measured, it is always a software like component that

will enable that measurement to take place, be it implemented on the component or in the

local operation system context. More complex software will often embed its own

performance measurement facilities, aiming to provide improved management. This

approach has the drawback of consuming run time resources while generating performance

measurements that are not mandatorily nor intensively used, albeit useful, in managing the

said components. This approach has the benefit of enabling the management of performance

thresholds as the low performance events occur, allowing a prompt response to such events.

Off-time (or batch) performance measurement is undertaken over any format of registries

(xx19) that the software and the operating system (OS) generate during run-time. These

registries are collected and processed in a diverse time frame, ex post facto. This processing

often takes place in different hardware, thus un-taxing the observed system. The side effects

of off-time performance measurement are the requirement of a secondary set of resources

and the inability to quickly respond to timely performance events; even if a performance

processing infrastructure is conceived in order to quickly process the registries, the gap

between performance event and response will never closed. This is the measurement

technique that is adopted on this research, allowing for the creation of the Big Data sets that

will be discussed further on the literature review.

40

Intrusive and non-intrusive observations are related to the level of resources they take

from the examined system. Upon generating performance measures, it has been observed that

at least some of the processing and IO generated is going to be consumed by the actual

measurement process (xx20). The more resources consumed, the more intrusive the

monitoring is. If possible, performance measurement processing should take into

consideration this aspect in order to be able to differ what is actual measurement and what is

noise generated by the performance measurement tactics.

Finally, measurement is performed over individual components or subsets of a system. These

components can be such as processors, network bandwidths, memory and hard disks, or even

whole sub systems within a greater system, such as a data base server that performs queries

for a web-based application. On these cases, performance measurement is executed over the

components, but the actual construct measured is non-existent and prone to imprecisions on

the interpretation of the data (xx21).

The measurement process relies in a series of broad and often dangerous assumptions: 1 –

managers are sufficiently knowledgeable that they are capable of understanding what is a

KPI and will be able to elect a bundle of KPI’s that is significant for the business and

technically feasible; 2 – the KPI’s are revised and revisited in a regular basis in order to

reassess its validity; 3 – the targeted component possesses the possibility of being reported,

noted and analyzed; 4 – the KPI has a quantic counterpart that enables direct or indirect

measurement(xx22).

From this section it is possible to identify that there are differences for quality measurement

for software products between the perspective presented by software engineering and

business. Software engineering considers both internal, external and in use, whereas business

largely ignore internal and external software quality, focusing on the effects of software

product quality in use.

41

2.2 Cloud computing

Cloud computing is in order to document the contemporary research, its trends of use and

emerging problems. Its technical components and applications are also presented in order to

clarify the complex layers of performance-impacting constructs that may or may not

influence in the end user performance perspective. Additionally, performance log generation

and analysis for cloud computing is discussed, as well as the possible applications of Bigdata

on these analyses.

2.3 Big Data

CHAPTER 3

<TITLE>

3.1 <Title>

<Text to insert>

CHAPTER 4

<TITLE>

4.1 <Title>

<Text to insert>

CHAPTER 5

<TITLE>

5.1 <Title>

<Texte à insérer >

48

CONCLUSION

<Text to insert>

LIST OF BIBLIOGRAPHICAL REFERENCES

<Text to insert>

Bibliography

Abel, A. (2010). Benchmarking 5 Cloud Platforms.

http://www.infoq.com/news/2010/07/Benchmarking-5-Cloud-Platforms.

Abran, A., & Al-Qutaish, R. E.-M. (2005). “Harmonization Issues in the Updating of the ISO

Standards on Software Product Quality”. Magdeburg, Germany: Otto-von-Guericke

University.

Abran, A., & Al-Qutaish, R. E.-M. (2005). An Information Model for Software Quality

Measurement with ISO Standards.

Agendaless. (2013). Supervisor Process Control. http://supervisord.org/.

ALinezhad, A., Masaeli, A., & Esfandiari, N. M. (2010). "Evaluation of Effectiveness of

Implementing Quality Management System (ISO9001:2000) Using BSC Approach in

NIGC". Journal of Industrial Engineering 6, 33-42.

Blog, H. S. (2011). Troubleshoot Response Time Problems.

http://highscalability.com/blog/2011/5/11/troubleshooting-response-time-problems-

why-you-cannot-trust.html.

Blog, R. U. (2011). The Complete List of End User Experience Monitoring Tools. Récupéré

sur http://www.real-user-monitoring.com/the-complete-list-of-end-user-experience-

monitoring-tools/

Bundschuh, M. D. (2008). The IT Measurement compendium. Berlin: Springer-Verlag.

Bush, E. (2004). Schema Based Computer System Health Monitoring, United States Patent

6754664 b1.

Buyya, R., Yeo, C., Venugopal, S., Brober, J., & Brandic, I. (2009). Vision, hype, and reality

for delivering computing as the 5th utility. . FGCS, Volume 25, I 6, June 2009, 599-

616.

Cacti. (2013). Cacti RRDTool. http://www.cacti.net.

Castor, K. (2006). Testing and benchmarking methodology.

http://donutey.com/hardwaretesting.php.

Cloudharmony. (2013). Cloudharmony Project. http://cloudharmony.com.

Cloudsleuth. (2013). Cloudsleuth Project. http://cloudsleuth.net.

53

Compuware. (2013). Lifecycle Performance Management.

http://www.compuware.com/en_us/application-performance-

management/products/lifecycle-performance-management.html.

Croll, A. (2013). How Should we Measure clouds? .

http://www.informationweek.com/cloud-computing/software/how-should-we-

measure-clouds/240151231.

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of

information technology. MIS Quarterly, 13(3), 319-339.

Drucker, P. (1991). "The New Productivity Challenge". Harvard Business Review, vol. 69,

pp. 69-76; .

Eric E. Schadt1, M. D. (2010). Computational solutions to large-scale data management and

analysis. Nature Reviews Genetics 11, 647-657| doi:10.1038/nrg2857.

Forster, F. (2013). Collectd Open source project. Munich: http://www.collectd.org.

Friedl, A., & Ubik, S. (2008). Perfmon and Servmon: Monitoring Operational Status and

Resources of Distributed Computing Systems. CESNET Technical report 1/2008.

Glass, R. (1998). Software runaways: Lessons Learned from Massive software project

failures. Englewood Cliffs, NJ: Prentice Hall.

Goodhue, D. L., & Thompson, R. L. (1995). Task-technology fit and individual performance.

MIS Quarterly, 19, 2, 213-236.

HP. (2013). HP ITSM Transforming IT organizations into service providers. Récupéré sur

http://h20427.www2.hp.com/program/ngdc/cn/zh/file/fuwu/management/ITSMBusin

essWP.pdf

ISO. (2009). ISO FDIS 9241-210:2009. Ergonomics of human system interaction - Part 210:

Human-centered design for interactive systems (formerly known as 13407).

International Organization for Standardization (ISO).

ISO/IEC. (2005). ISO/IEC 25000:2005 Software Engineering -- Software product Quality

Requirements and Evaluation (SQuaRE) -- Guide to SQuaRE, ISO/IEC JTC 1/SC 7.

J. Cohen, B. D. (2009). MAD Skills: New Analysis Practices for BigData. PVLDB.

54

Jackson, K. e. (2010). Performance Analysis of High Performance Computing Applications

on the Amazon Web Services Cloud. Cloud Computing Technology and Science

(CloudCom). 2010 IEEE Second International Conference.

Jeffrey dean, S. g. (2008). mapReduce: simplified data processing on large clusters.

Communications of the ACM: Volume 51 Issue 1, January.

Juran J, D. F. (2010). Juran's Quality Handbook: The Complete Guide to Performance

Excellence, . McGraw-Hill.

Kufrin, R. (2005). Measuring and improving application performance with Perfsuite.

http://perfsuite.ncsa.illinois.edu/publications/LJ135/.

L. Bautista, A. A. (2012). "Design of a Performance Measurement Framework for Cloud

Computing". Journal of Software Engineering and Applications, Vol. 5 No. 2, pp. 69-

75. doi: 10.4236/jsea.2012.52011.

Law, E., Roto, V., Hassenzahl, M., Vermeeren, A., & Kort, J. (2009). Understanding scoping

and defining user experience: a survey approach. Proceedings of Human Factors in

computing Systems,. CHI' 09, pp 719-728. .

Mahmood, M. B. (2010). Variables affecting information technology end-user satisfaction: a

meta-analysis of the empirical literature. IJHCS, Vol 54, I 4, 751-771.

Massie, M. e. (2012). Monitoring with Ganglia . O'Reilly Media.

Meijer, G. (2012). How do you Measure Cloud Performance.

http://www.cloudproviderusa.com/how-do-you-measure-cloud-performance/.

Microsoft. (2012). Performance Analysis of Logs Tool. http://pal.codeplex.com/.

Microsoft. (2013). Microsoft Perfmon. http://technet.microsoft.com/en-

us/library/cc749249.aspx.

Mirzaei, N. (2008). cloud computing . Institute Report, Community Grids Lab, Indiana.Edu.

Mohammed Alhamad, T. D. (2011). A Survey on SLA and Performance Measurement in

Cloud Computing. Lecture Notes in Computer ScienceVolume 7045, 2011, pp 469-

477 .

Munin. (2013). Munin Monitoring Open Project. http://munin-monitoring.org/.

Nagios. (2013). Nagios IT Infrastructure Monitoring. www.nagios.org.

55

Observium. (2013). Observium: autodiscovering SNMP network monitoring.

www.observium.org.

Omniti. (2013). Reconnoiter Fault Detection and Trending.

https://labs.omniti.com/labs/reconnoiter.

Oswaldo Trelles, P. P. (2011). Big Data, but are we ready? Nature Reviews Genetics 12, 224

| doi:10.1038/nrg2857-c1.

Rabl, T. (2012). solving big data challenges for enterprise application performance

management. Proceedings of the VLDB Endowment, Vol. 5, No. 12.

Sinung Suakanto, S. H. (2012). Performance Measurement of Cloud Computing Services.

International Journal on Cloud Computing: Services and Architecture (IJCCSA),

April 2012, Volume 2, Number 2.

Stavrinoudis, X. (2008). “Comparing internal and external software quality measurements”.

(I. Press, Éd.) Piraeus, Greece: Proccedings of the 8th Joint Conference on

Knowledge-Based Software Engineering.

Tidelash. (2013). Monit process monitor. http://mmonit.com/monit/.

Tullis, T., & Albert, B. (2010). Measuring the User Experience: Collecting, Analyzing, and

Presenting Usability Metrics. Morgan Kaufmann.

Weinman, J. (2009). Cloudonomics.com. http://joeweinman.com.

Weisberg, J. (2013). Argus TCP Monitor. http://argus.tcp4me.com.

Yiduo Mei, L. L. (2010). Performance Measurements and Analysis of Network I/O

Applications in Virtualized Cloud.

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.165.6742&rep=rep1&type

=pdf .

Zabbix. (2013). Zabbix: Enterprise Class Open source Distributed Monitoiring Solution.

www.zabbix.com.

Zenoss. (2013). Zenoss Unified IT Operations Monitor. www.zenoss.com.

<Texte à insérer. Note : retrait de première ligne activée>

56

[JUR2010] Juran J, De Feo, J. Juran's Quality Handbook: The Complete Guide to

Performance Excellence, McGraw-Hill Professional; 6 edition (May 19, 2010)

http://www.pqm-online.com/assets/files/lib/juran.pdf

[HP2003] HP IT Service Management: Transforming IT organizations into service providers,

http://h20427.www2.hp.com/program/ngdc/cn/zh/file/fuwu/management/ITSMBusin

essWP.pdf, 8/8/2013.

[ISO2005] ISO/IEC 25000:2005 Software Engineering -- Software product Quality

Requirements and Evaluation (SQuaRE) -- Guide to SQuaRE, ISO/IEC JTC 1/SC 7

[ABR2005] Abran, Alain; Al-Qutaish, Rafa E. and Desharnais, Jean-Marc, “Harmonization

Issues in the Updating of the ISO Standards on Software Product Quality”, Metrics

News Journal, Vol. 10, No. 2, Otto-von-Guericke University of Magdeburg,

Germany, December, 2005. pp. 35-44. (ISSN: 1431-8008)

[ALI2010] ALinezhad, A; Masaeli, A; Esfandiari, N, Mirhadi, M "Evaluation of

Effectiveness of Implementing Quality Management System (ISO9001:2000) Using

BSC Approach in NIGC" Journal of Industrial Engineering 6 (2010), 33-42

[ABR2005-2] Abran, Alain; Al-Qutaish, Rafa E. and Desharnais, Jean-Marc, An

Information Model for Software Quality Measurement with ISO Standards

[STA2008] Stavrinoudis, Xenos, “Comparing internal and external software quality

measurements”, Proccedings of the 8th Joint Conference on Knowledge-Based

Software Engineering, IOS Press, pp. 115-124, Piraeus, Greece, August 25-28, 2008.

[BUN2008] Bundschuh, M, Dekkers, C. The IT Measurement compendium. Berlin,

Springer-Verlag 2008.

[GLA1998] Glass, R. Software runaways: Lessons Learned from Massife software project

failures. Englewood Cliffs, NJ: Prentice Hall, 1998.

[REA2011] The Complete List of End User Experience Monitoring Tools http://www.real-

user-monitoring.com/the-complete-list-of-end-user-experience-monitoring-tools/

[TUL2010] Tullis, T; Albert, B: Measuring the User Experience: Collecting, Analyzing, and

Presenting Usability Metrics. Morgan Kaufmann; 2 edition (2010)

57

[MAH2010] Mahmood, M. Burn, J, Gemoets, L, Jacquez, C: Variables affecting information

technology end-user satisfaction: a meta-analysis of the empirical literature. IJHCS,

Vol 54, I 4, Apr 200, 751-771.

[BUY2009] Buyya, R; Yeo, C; Venugopal, S; Brober, J; Brandic, I: Cloud computing and

emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th

utility. FGCS, Volume 25, I 6, June 2009, 599-616.

[JAC2010] Jackson, K. et al; Performance Analysis of High Performance Computing

Applications on the Amazon Web Services Cloud. Cloud Computing Technology and

Science (CloudCom), 2010 IEEE Second International Conference on Nov 30 2010.

Xx29 http://royal.pingdom.com/2009/03/19/10-historical-software-bugs-with-extreme-consequences/ Xx31 http://www.prweb.com/releases/2013/1/prweb10298185.htm

58

Annex 1 - (to be worked)

Performance  Efficiency     performance  relative  to  the  amount  of  resources  used  under  

stated  conditions    • Time-­‐behavior     degree  to  which  the  response  and  processing  times  and  

throughput  rates  of  a  product  or  system,  when  performing  its  functions,  meet  requirements  (benchmark)    

• Resource  utilization     degree  to  which  the  amounts  and  types  of  resources  used  by  a  product  or  system  when  performing  its  functions  meet  requirements    

• Capacity     degree  to  which  the  maximum  limits  of  a  product  or  system  parameter  meet  requirements    

   Functional  Suitability        

degree  to  which  a  product  or  system  provides  functions  that  meet  stated  and  implied  needs  when  used  under  specified  conditions    

• Functional  completeness     degree  to  which  the  set  of  functions  covers  all  the  specified  tasks  and  user  objectives    

• Functional  correctness     degree  to  which  a  product  or  system  provides  the  correct  results  with  the  needed  degree  of  precision    

• Functional  appropriateness     degree  to  which  the  functions  facilitate  the  accomplishment  of  specified  tasks  and  objectives.  As  an  example  :  a  user  is  only  presented  with  the  necessary  steps  to  complete  a  task,  excluding  any  unnecessary  steps    

   Compatibility     degree  to  which  a  product,  system  or  component  can  exchange  

information  with  other  products,  systems  or  components,  and/or  perform  its  required  functions,  while  sharing  the  same  hardware  or  software  environment    

• Coexistence     degree  to  which  a  product  can  perform  its  required  functions  efficiently  while  sharing  a  common  environment  and  resources  with  other  products,  without  detrimental  impact  on  any  other  product    

• Interoperability     degree  to  which  two  or  more  systems,  products  or  components  can  exchange  information  and  use  the  information  that  has  been  exchanged    

   Usability     degree  to  which  a  product  or  system  can  be  used  by  specified  

users  to  achieve  specified  goals  with  effectiveness,  efficiency  and  satisfaction  in  a  specified  context  of  use    

• Appropriateness    recognisability    

degree  to  which  users  can  recognize  whether  a  product  or  system  is  appropriate  for  their  needs.  Appropriateness  recognizability  will  depend  on  the  ability  to  recognize  the  appropriateness  of  the  product  or  system’s  functions  from  initial  impressions  of  the  product  or  system  and/or  any  associated  documentation.    

59

• Learnability     degree  to  which  a  product  or  system  can  be  used  by  specified  users  to  achieve  specified  goals  of  learning  to  use  the  product  or  system  with  effectiveness,  efficiency,  freedom  from  risk  and  satisfaction  in  a  specified  context  of  use    

• Operability     degree  to  which  a  product  or  system  has  attributes  that  make  it  easy  to  operate  and  control    

• User  error  protection     degree  to  which  a  system  protects  users  against  making  errors    

• User  interface  aesthetics     degree  to  which  a  user  interface  enables  pleasing  and  satisfying  interaction  for  the  user    

• Accessibility     degree  to  which  a  product  or  system  can  be  used  by  people  with  the  widest  range  of  characteristics  and  capabilities  to  achieve  a  specified  goal  in  a  specified  context  of  use    

   Reliability     degree  to  which  a  system,  product  or  component  performs  

specified  functions  under  specified  conditions  for  a  specified  period  of  time    

• Maturity     degree  to  which  a  system  meets  needs  for  reliability  under  normal  operation    

• Availability     degree  to  which  a  system,  product  or  component  is  operational  and  accessible  when  required  for  use    

• Fault  Tolerance       degree  to  which  a  system,  product  or  component  operates  as  intended  despite  the  presence  of  hardware  or  software  faults    

• Recoverability     degree  to  which,  in  the  event  of  an  interruption  or  a  failure,  a  product  or  system  can  recover  the  data  directly  affected  and  re-­‐establish  the  desired  state  of  the  system    

   Security     degree  to  which  a  product  or  system  protects  information  and  

data  so  that  persons  or  other  products  or  systems  have  the  degree  of  data  access  appropriate  to  their  types  and  levels  of  authorization    

• Confidentiality     degree  to  which  a  product  or  system  ensures  that  data  are  accessible  only  to  those  authorized  to  have  access    

• Integrity     degree  to  which  a  system,  product  or  component  prevents  unauthorized  access  to,  or  modification  of,  computer  programs  or  data.    

• Non-­‐repudiation     degree  to  which  actions  or  events  can  be  proven  to  have  taken  place,  so  that  the  events  or  actions  cannot  be  repudiated  later    

• Accountability     degree  to  which  the  actions  of  an  entity  can  be  traced  uniquely  to  the  entity    

• Authenticity     degree  to  which  the  identity  of  a  subject  or  resource  can  be  proved  to  be  the  one  claimed    

   Maintainability     degree  of  effectiveness  and  efficiency  with  which  a  product  or  

system  can  be  modified  by  the  intended  maintainers    • Modularity     degree  to  which  a  system  or  computer  program  is  composed  of  

discrete  components  such  that  a  change  to  one  component  has  minimal  impact  on  other  components    

60

• Reusability     degree  to  which  an  asset  can  be  used  in  more  than  one  system,  or  in  building  other  assets    

• Analyzability     degree  of  effectiveness  and  efficiency  with  which  it  is  possible  to  assess  the  impact  on  a  product  or  system  of  an  intended  change  to  one  or  more  of  its  parts,  or  to  diagnose  a  product  for  deficiencies  or  causes  of  failures,  or  to  identify  parts  to  be    

  modified    • Modifiability     degree  to  which  a  product  or  system  can  be  effectively  and  

efficiently  modified  without  introducing  defects  or  degrading  existing  product  quality    

• Testability     degree  of  effectiveness  and  efficiency  with  which  test  criteria  can  be  established  for  a  system,  product  or  component  and  tests  can  be  performed  to  determine  whether  those  criteria  have  been  met    

   Portability     degree  of  effectiveness  and  efficiency  with  which  a  system,  

product  or  component  can  be  transferred  from  one  hardware,  software  or  other  operational  or  usage  environment  to  another    

• Adaptability     degree  to  which  a  product  or  system  can  effectively  and  efficiently  be  adapted  for  different  or  evolving  hardware,  software  or  other  operational  or  usage  environments    

• Installability     degree  of  effectiveness  and  efficiency  with  which  a  product  or  system  can  be  successfully  installed  and/or  uninstalled  in  a  specified  environment    

• Replaceability     degree  to  which  a  product  can  be  replaced  by  another  specified  software  product  for  the  same  purpose  in  the  same  environment    

61

Annex 2 – (to be worked)

Characteristic Subdivision Primitive

Effectiveness Task completion Proportion of tasks completed correctly.

Effectiveness Task effectiveness Proportion of the goals of the tasks achieved

Effectiveness Error frequency Frequency of User errors

Efficiency Time efficiency Time for task completion against a desired

target

Efficiency Relative task time Relative time for task completion in comparing

a regular user to an expert

Efficiency Task efficiency Percentage of goals achieved per unit of time

Efficiency Relative task efficiency Goals achieved per unit of time against desired

target

Efficiency Economic productivity User’s cost effectiveness

Efficiency Productive proportion Percentage of time units dispensed on

productive actions

Efficiency Relative number of

actions

Amount of user actions required to perform a

task

Satisfaction Satisfaction scale Grade of user satisfaction

Satisfaction Specific satisfaction

scale

Grade of user satisfaction in relation to specific

system features

Satisfaction Discretionary usage Proportion of users that elect to use the system

Satisfaction Specific discretionary

usage

Proportion of users that elect to use specific

functions in the system

Satisfaction User complaints Proportion of users that report dissatisfaction

Satisfaction Trust scale Scale of declared user trust on the system

Satisfaction Pleasure scale Scale of declared user pleasure on the system

Satisfaction Comfort scale Scale of declared user comfort

Freedom from Risk mitigation Percent of risk that can be averted via system

62

risk quality

Freedom from

risk

Return of investment

(ROI)

Rate of return

Freedom from

risk

Time to ROI Time scale versus percentage of ROI

Freedom from

risk

Relative business

performance

Average comparison between top class

companies of the same industry – benchmark

Freedom from

risk

Business alignment How aligned to the business objectives is the

system

Freedom from

risk

Delivery time Time scale versus expected targets

Freedom from

risk

Revenue for each

customer

Value per customer

Freedom from

risk

Errors with economic

impact

Frequency and criticality of errors, human or

systemic, with economic consequences

Freedom from

risk

Software corruption Frequency and criticality of errors, human or

systemic, that cause damage to other software

components

Freedom from

risk

User health and safety Frequency and criticality of health problems on

using the system

Freedom from

risk

User health and safety

impact

Effect of utilization of the system on user’s

health

Freedom from

risk

Safety of people

affected by the use of

the system

Hazard incidence over system users

Freedom from

risk

Environmental impact Environmental impact of developing,

manufacturing, installing, operating the system

Context

completeness

Context completeness Proportion of intended contexts that provide

acceptable usability

Flexibility Flexible context of use Percentage of additional contexts of use that

63

provide acceptable usability

Flexibility Flexible design

features

Percentage of different user needs that can be

satisfied via product adaptation