human centered design process assessment
TRANSCRIPT
HUMAN CENTERED DESIGN PROCESS ASSESSMENT
Rüdiger Heimgärtner Intercultural User Interface Consulting (IUIC)
Lehderstraße 71, 13086 Berlin, Germany
ABSTRACT
Process assessment is a disciplined project-oriented evaluation of an organizational unit’s processes with respect to a process assessment model. In this paper, the approach according to ISO 33000 is presented of how to assess HCD processes. In detail, the standard approaches according to ISO 9241-210 and to ISO 9241-220 are considered. It is explained, how an assessment of HCD processes should be prepared, executed, and followed up.
KEYWORDS
Capability, Maturity, Process Assessment Model, Process Reference Model, ISO 9241-220, Human Centered Design Processes
1. INTRODUCTION
Experience from various fields of technology shows that it is possible for a development process recognized as mature to produce products of higher quality. Process assessments and improvement are an everyday business. Each project team member must know the current process and its optimization potential. Self-assessments can be used for continuous process improvement. Each project team member can subsequently recognize what is important for successful projects and what is also asked in assessments. Therefore, time is saved sustainably, and productivity is increased in projects. However, assessing processes is a complex endeavor and comprises those integrative parts necessary to obtain valid objective and comparable results. In this paper, the structure and content of a process assessment model (PAM) is explained and exemplified to yield an impression of how to determine the capability of the human centered design (HCD) processes.
The term "process" can be understood at three levels of abstraction. These levels of abstraction are not meant to define a strict black-or-white split, nor is it the aim to provide a scientific classification schema – the message here is to understand that, in practice, when it comes to the term "process" there are different abstraction levels, and that a PAM resides at the highest.
Capturing experience acquired during product development (i.e., at the DOING level) to share this experience with others means creating a HOW level. However, a HOW is always specific to a context such as a company, an organizational unit, or a product line. For example, the HOW of a project, organizational unit, or company A is potentially not applicable to a project, organizational unit, or company B. However, both might be expected to adhere to the principles represented by PAM indicators for process outcomes and process attribute achievements. These indicators are at the WHAT level while deciding on solutions for concrete templates, proceedings, and tooling, etc. is left to the HOW level.
2. PROCESS CAPABILITY DETERMINATION
The concept of process capability determination by using a process assessment model (PAM) is based on a two-dimensional framework. The process dimension (first dimension) is provided by processes defined in a process reference model (PRM). The capability dimension (second dimension) consists of capability levels that are further subdivided into process attributes. The process attributes provide the measurable characteristics of process capability. The PAM selects processes from a PRM and supplements it with indicators. These indicators support the collection of objective evidence which enable an assessor to assign ratings for processes according to the capability dimension.
ISBN: 978-989-8704-31-3 © 2021
116
2.1 Process Reference Model
Processes are grouped by process category and at a second level into process groups according to the type of
activity they address. In general, there are at least three process categories: primary life cycle processes,
organizational life cycle processes and supporting life cycle processes. Each process is described in terms of a
purpose statement. The purpose statement contains the unique functional objectives of the process when
performed in an environment. For each purpose statement a list of specific outcomes is mapped, as a list of
expected positive results of the process performance. For the process dimension, the HCD PRM provides the
set of processes in ISO 9241-220:2019-03, whereas HCP.3 is divided into further process groups according to
the structured main HCD activities described in ISO 9241-210 and considered in ISO 9241-220:2019-03 as
shown in Figure 1.
Figure 1. Relationship between the HCD processes (including the sub-processes in HCP.3) from ISO 9241-220:2019-03
2.1.1 Organizational and Supporting Life Cycle Processes Category
The organizational life cycle processes category consists of processes that develop process, product, and
resource assets which, when used by projects in the organization, will help the organization achieve its business
goals. The organizational life cycle processes category consists of the following groups:
• the HCD.1 ensure process group;
• the HCD.2 enable process group.
The management processes may be used by anyone who manages any type of HCD project or process
within the life cycle belonging to the supporting life cycle process category.
2.1.2 Primary Life Cycle Processes Category
The primary life cycle processes category consists of processes that may be used by the customer when
acquiring products from a supplier, and by the supplier when responding and delivering products to the
customer including the engineering processes needed for specification, design, development, integration, and
testing. The primary life cycle processes category consists of the following groups:
• the HCP.3 Execute process group;
• the HCP.4 Introduce & operate process group.
The HCP.3 Execute process group consists of processes addressing the elicitation and management of
customer and internal requirements, the definition of the system architecture and the integration and testing on
the system level.
International Conferences Interfaces and Human Computer Interaction 2021; and Game and Entertainment Technologies 2021
117
3. MEASUREMENT FRAMEWORK
The measurement framework provides the necessary requirements and rules for the capability dimension. It
defines a schema which enables an assessor to determine the capability level of a given process. These
capability levels are defined as part of the measurement framework. To enable the rating, the measurement
framework provides process attributes defining a measurable property of process capability. Each process
attribute is assigned to a specific capability level. The extent of achievement of a certain process attribute is
represented by means of a rating based on a defined rating scale. The rules from which an assessor can derive
a final capability level for a given process are represented by a process capability level model. The HCD PAM
uses the measurement framework defined in ISO/IEC 33020:2015.
3.1 Process Capability Levels
Process capability levels and process attributes are identical to those defined in ISO/IEC 33020:2015. Rough
descriptions of the capability levels and the corresponding process attributes can be found in section 5 in the
paper at hand. Process attributes are features of a process that can be evaluated on a scale of achievement,
providing a measure of the capability of the process. They are applicable to all processes. A capability level is
a set of process attribute(s) that work together to provide a major enhancement in the capability to perform a
process. Each attribute addresses a specific aspect of the capability level. The levels constitute a rational way
of progressing through the improvement of the capability of any process. According to ISO/IEC 33020:2015
there are six capability levels (Figure 2), incorporating nine process attributes (PA) according to the levels.
Figure 2. Process capability levels according to ISO/IEC 33020:2015
3.2 Process Attribute Rating
To support the rating of process attributes, the ISO/IEC 33020:2015 measurement framework provides a
defined rating scale with an option for refinement, different rating methods and different aggregation methods
depending on the class of the assessment (e.g., required for organizational maturity assessments). Within this
process measurement framework, a process attribute is a measurable property of process capability. A process
attribute rating is a judgement of the degree of achievement of the process attribute for the assessed process.
The rating scale is defined by ISO/IEC 33020:2015 as shown in Figure 3.
ISBN: 978-989-8704-31-3 © 2021
118
Figure 3. Rating scale according to ISO/IEC 33020:2015
The rating and aggregation method in ISO/IEC 33020:2015 provides the following definitions. A process
outcome is the observable result of successful achievement of the process purpose. A process attribute outcome
is the observable result of achievement of a specified process attribute. Process outcomes and process attribute
outcomes may be characterized as an intermediate step to providing a process attribute rating. When performing
rating, the rating method employed shall be specified relevant to the class of assessment. The use of rating
method may vary according to the class, scope, and context of an assessment. The lead assessor shall decide
which (if any) rating method to use. The selected rating method(s) shall be specified in the assessment input
and referenced in the assessment report. Further information can be taken from ISO/IEC 33020:2015.
4. PROCESS ASSESSMENT MODEL
A process assessment model (PAM) offers indicators to identify whether the process outcomes and the process
attribute outcomes (achievements) are present or absent in the instantiated processes of projects and
organizational units. These indicators provide guidance for assessors in accumulating the necessary objective
evidence to support judgments of capability. They are not intended to be regarded as a mandatory set of
checklists to be followed. To judge the presence or absence of process outcomes and process achievements an
assessment delivers objective evidence. All such evidence comes from the examination of work products and
the repository content of the assessed processes, and from testimony provided by the performers and managers
of the assessed processes. This evidence is mapped to the PAM indicators to allow the establishment of a
correspondence to the relevant process outcomes and process attribute achievements. There are two types of
indicators. Process performance indicators, which apply exclusively to capability level 1, provide an indication
of the extent of fulfillment of the process outcomes. Process capability indicators, which apply to capability
levels 2 to 5, provide an indication of the extent of fulfillment of the process attribute achievements.
Assessment indicators are used to confirm that certain practices were performed, as shown by evidence
collected during an assessment. All such evidence comes either from the examination of work products of the
processes assessed, or from statements made by the performers and managers of the processes. The existence
of base practices and work products pro-vide evidence of the performance of the processes associated with
them. Similarly, the existence of process capability indicators provides evidence of process capability. The
evidence obtained should be recorded in a form that clearly relates to an associated indicator, in order that
support for the assessor’s judgment can be confirmed or verified as required by ISO/IEC 33002:2015.
4.1 Process Performance Indicators
Types of process performance indicators are base practices (BP) and work products (WP). Both BPs and WPs
relate to one or more process outcomes. Consequently, BPs and WPs are always process-specific and not
generic. BPs represent activity-oriented indicators. WPs represent result-oriented indicators. Both BP and WP
are used for judging objective evidence that an assessor is to collect, and accumulate, in the performance of an
assessment. In that respect BPs and WPs are alternative indicator sets the assessor can use. A PAM offers a set
of work product characteristics (WPC) for each WP. These are meant to offer a good practice and
International Conferences Interfaces and Human Computer Interaction 2021; and Game and Entertainment Technologies 2021
119
state-of-the-art knowledge guide for the assessor. Therefore, WP and WPC are supposed to be a quickly
accessible information source during an assessment. In that respect WPs and WPCs only represent an example
structure. They are neither a "strict must" nor are they normative for organizations. Instead, the actual structure,
form and content of concrete work products and documents for the implemented processes must be de-fined
by the project and organization, respectively. The project and/or organization ensures that the work products
are appropriate for the intended purpose and needs, and in relation to the development goals.
4.2 Process Capability Indicators
Types of process capability indicators are generic practice (GP) and generic re-source (GR). Both GPs and
GRs relate to one or more PA achievements. In contrast to process performance indicators, however, they are
of generic type, i.e., they apply to any process. The difference between GP and GR is that the former represents
activity-oriented indicators while the latter represent infrastructure-oriented indicators for judging objective
evidence. An assessor must collect and accumulate evidence supporting process capability indicators during
an assessment.
5. PROCESS CAPABILITY LEVELS AND PROCESS ATTRIBUTES
Process capability indicators are the means of achieving the capabilities addressed by the considered process
attributes. Evidence of process capability indicators supports the judgment of the degree of achievement of the
process attributes. The capability dimension of the PAM consists of six capability levels matching the
capability levels defined in ISO/IEC 33020:2015. The process capability indicators for the 9 process attributes
included in the capability dimension for process capability level 1 to 5 are described in section 5.3. The generic
practices address the characteristics from each process attribute. The generic resources relate to the process
attribute. Process capability level 0 does not include any type of indicators, as it reflects a non-implemented
process or a process which fails to partially achieve any of its outcomes.
5.1 Process Capability Level 0: Incomplete Process
The process is not implemented or fails to achieve its process purpose. At this level there is little or no evidence
of any systematic achievement of the process purpose.
5.2 Process Capability Level 1: Performed Process
The implemented process achieves its process purpose. The following process at-tribute demonstrates the
achievement of a performed process representing process capability level 1. The process performance process
attribute for level 1 is a measure of the extent to which the process purpose is achieved. If the process achieves
its defined outcomes requested by the generic practice GP 1.1.1 “Achieve the process outcomes”, then level 1
is reached. This means, i.e., that the intent of the process specific base practices is met, and the work products
are produced (evidencing the process outcomes) using the generic resource “Resources are used to achieve the
intent of process specific base practices”.
5.2.1 Process Reference Model and Performance Indicators (Level 1)
The processes in the process dimension can be drawn from the HCD process reference model (based on ISO
9241-220:2019-03). Each table related to one process in the process dimension contains the process reference
model and the process performance indicators necessary to define the processes in the process assessment
model (cf. Figure 4). The process references model describes the processes by ID, name, purpose, benefit, and
outcomes. The process performance indicators consist of base practices and output work products.
ISBN: 978-989-8704-31-3 © 2021
120
Figure 4. Template for the process description
5.2.2 Output work Products (Level 1)
Work product characteristics (WPC) can be used when reviewing potential outputs such as work products of
process implementation. The characteristics are provided as guidance for the attributes to look for, in a sample
work product (WP) and to provide objective evidence supporting the assessment of a process. A documented
process and assessor judgment are needed to ensure that the process context (application domain, business
purpose, development methodology, size of the organization, etc.) is considered when using this information.
WPs and their characteristics should be considered as a starting point for considering whether, given the
context, they con-tribute to the intended purpose of the process, and not as a checklist of what every
organization must have.
5.3 Process Reference Model and Performance Indicators (Level 2-5)
In contrast to level 1, the performance indicators on level 2-5, represented by generic practices and resources,
relate to all process in the process reference model.
If the performed process is implemented in a managed fashion (planned, monitored, and adjusted) and its
work products are appropriately established, controlled, and maintained, then it is called a “managed process".
Thereby the performance management process attribute (PA 2.1) and the work product management process
attribute (PA 2.2), together with the previously defined performance process attribute (PA 1.1), demonstrate
the achievement of a managed process representing process capability level 2.
If the managed process is implemented using a defined process that can achieve its process outcomes, then
it is called a “established process”. The process definition process attribute (PA 3.1) and the process
deployment process attribute (PA 3.2), together with the process attributes for the performed (PA 1.1) and the
managed process (PA 2.1 and PA 2.2), demonstrate the achievement of an established process representing
process capability level 3.
If the established process operates predictively within defined limits to achieve its process outcomes, then
it is called a “predictive process”. Quantitative management needs are identified, measurement data are
collected and analyzed to identify assignable causes of variation. Corrective action is taken to address
assignable causes of variation. The quantitative analysis process attribute (PA 4.1) as well as the quantitative
control process attribute (PA 4.2), together with the process attributes for the performed (PA 1.1), managed
(PA 2.1 and PA 2.2) and established (PA 3.1 and PA 3.2) process, demonstrate the achievement of a predictable
process representing process capability level 4.
If the predictable process is continually improved to respond to change aligned with organizational goals,
then it is called a “innovating process”. The process innovation process attribute (PA 5.1) as well as the process
innovation implementation process attribute (PA 5.2), together with the process attributes for the performed
(PA 1.1), managed (PA 2.1 and PA 2.2), established (PA 3.1 and PA 3.2) and predicted (PA 4.1 and PA 4.2)
process, demonstrate the achievement of an ‘innovating process’ representing process capability level 5.
International Conferences Interfaces and Human Computer Interaction 2021; and Game and Entertainment Technologies 2021
121
6. RATING IN THE PROCESS ASSESSMENT
The assessment procedure contains at least the following steps:
• Explain process purpose and benefit and most important requirements of the process to the audience,
• Interview the audience regarding process life based on purpose, base practices and work products and
collect documented evidence,
• Rate the evidence according to the rating scale,
• Accumulate the ratings for one process (e.g., HCP.3.4.2) to an overall rating considering:
• Achievement of process purpose,
• Achievement of all base practices,
• Achievement of work products.
As an example, the summary of the results of the rating up to level 2 could appear as presented in Figure 5.
Depending on the rating, the HCD processes of the executing process group (HCP.3) are ratings up to level 2
in this figure addressing the performance process attribute (level 1) and the management process attributes
(level 2). The only process that reached level 2 in this example is HCP.3.4.2 (Produce and refine user interface
design solutions) indicating that all base practices are achieved and the work products of this process (such as
system interface requirement, architectural design description and user interface specification) are available
and reviewed, and the process is managed in a proper way. Other processes such as HCP.3.1.1 (Establish
human-centered quality objectives) were not performed and managed properly, and the work products are
mostly missing.
Figure 5. Example Rating of the HCP.3 Process Group from ISO 9241-220:2019-03
ISBN: 978-989-8704-31-3 © 2021
122
7. DISCUSSION
With a seamless HCD development process, mistakes and unproductivity can be prevented and product quality
rises with high quality process output contributing to a high-quality interactive product. Behind such
experiences a causal relationship be-tween process and product quality is assumed, which consists in the fact
that a more mature process with a lower probability of error produces higher quality products than a less mature
process. Since quality cannot be achieved without a sustainable focus on user requirements, usability
engineering is also referred to as a human-centric development process. Requirements for a human-centered
development process are defined in DIN EN ISO 9241-210:2011-01. The aim of usability engineering is to do
everything necessary in a development process to steer it towards an optimal quality of use of the final product.
Following a PRM like that derived from ISO 9241-220:2019-03 ensures the involvement of users as
requirement donors and as evaluators of
a) the achieved quality of use,
b) the application of suitable usability engineering methods,
c) the documentation of design decisions with relevance for the quality of use,
d) the establishment of a quality management system for the quality of use and
e) setting up process roles with suitably qualified personnel (cf. DIN SPEC 92412:2015-11).
8. CONCLUSION
The presented PAM for the development process of interactive products serves to assess the process quality in
development projects based on objective indicators for process capability and thereby to achieve quality of use
referring to the HCD processes defined in ISO 9241-220:2019-03. Even if the presented HCD-PAM is still in
a working draft state and, hence, must be improved and worked out in detail in accordance with all other
relevant standards, the principles of this HCD-PAM and the presented assessment procedure have been
confirmed for years by assessments in the automotive industry regarding software engineering processes
(cf. VDA QMC WG 13, 2017). The presented HCD-PAM can be applied in conjunction with the process
reference model (PRM) for human centered design (HCD) processes derived from the processes in
ISO 9241-220:2019-03 to a similar benefit. The refinement and application of the suggested HCD-PAM will
show if it can be used to prove that the process has the capability of justifying trust in the HCD processes by
all stakeholders and causes that the assessments can be productively conducted.
REFERENCES
DIN EN ISO 9241-210:2011-01, Ergonomics of human-system interaction - Part 210: Hu-man-centred design for
interactive systems.
DIN SPEC 92412:2015-11, Ergonomics of human-system interaction - Auditing procedure for the development of
interactive products based on DIN EN ISO 9241-210.
ISO 9241-220:2019-03, Ergonomics of human-system interaction - Part 220: Processes for enabling, executing and
assessing human-centred design within organizations.
ISO/IEC 15504-5:2006, Information Technology – Process assessment – Part 5: An exem-plar Process Assessment Model.
ISO/IEC 33002:2015, Information technology -- Process assessment – Requirements for performing process assessment.
ISO/IEC 33020:2015, Information technology -- Process assessment – Process measure-ment framework for assessment
of process capability.
VDA QMC Working Group 13 / Automotive SIG, 2017, Automotive SPICE Process Assessment / Reference Model,
URL=http://www.automotivespice.com/fileadmin/software-download/AutomotiveSPICE_PAM_31.pdf, last access
06-30-2021.
International Conferences Interfaces and Human Computer Interaction 2021; and Game and Entertainment Technologies 2021
123