Development of Indicators for Integrated System Validation

Leena Norros & Maaria Nuutinen & Paula Savioja VTT Industrial Systems: Work, Organisation and System Usability Research

20.1.2005

2

Outline of the Presentation

• NPP control room modernizations• Integrated System Validation (ISV)• Performance indicators in validation• Development of the evaluation framework for intelligent

environments• Conclusions

3

4

NPP Control Room Modernizations

• Current control and automation systems are being modernized• No changes to the degree of automation • Technological rationale for the change

• Maintenance costs• Lack of spare parts• Technological possibilities exist

• Different strategies adopted by the utilities• Some human centered design principles implicitly adopted in the

projects• Happening at the same time

• OL3• Generation change within the personnel of existing NPPs

5

NPP Control Room Modernizations: The Effective Changes

• Loss of individual data points and controls in the information panels and desks• The decrease in peripheral information Tacit knowledge, Process feel and

awareness• Spatial memory memorability, skill based behavior, response times• Co-operation within the crew communication, group awareness

• Adoption of individual information displays• Sequential use of information instead of parallel, “key hole effect” Windows

and dialogs might hide information• Active searching required understanding of the available resources• Secondary tasks from manipulating the interface possibility of confusion,

response times• Higher abstraction level in the information orientation, constraints and

possibilities

• Adoption of large screen displays• Basis for shared co-operation group SA• Higher abstraction level in the information

6

NPP Control Room Modernizations: Model of the Change

Control and

automation system UI

User practices

Process performanc

e

Control and

automation system UI

User practices

Process performanc

e

7

How do we know that a complex system can be safely operated?

8

Integrated System Validation

• Performance based evaluation of the integrated design, in order to ensure that the human system interface supports the safe operation of the plant

• Use of full scope simulator

• The effect of contextual and situational factors to the safety of operation must be evaluated

• Towards the end of the design process

• The total system is available

• After the training period of the operators

• Use of actual crews, representative sample of the population

• Use of normal conditions, specific failures, accidents, beyond design basis events

• Compare the selected measures with the predefined acceptance criteria

9

Integrated System Validation: Current Problems

1. Which indicators to use• Which measures reflect the safety• Which measure are relevant in the change situation• Which measures reflect performance in a way that can be

generalized2. How to set the criteria

• What is the acceptable level of performance with the selected indicators

3. The effort needed, the amount of testing required4. Generalization of the results

Norros & Savioja 2004, Heimdal et. al. 2004

10

Validation: The Problem of the ε – case

How to predict what will happen in a very rarely occurring beyond design basis, beyond validation possibilities, event that nobody predicted ever to happen?

Predictive capabilities of validation procedures?

11

Performance Indicators: Development Challenges

• Process performance• Do not really differentiate enough

• High degree of automation• Complex defenses within the system• Thorough training process

• Difficult to anchor to the HF-related changes taking place in modernization

• Not predictive of future performance in the conditions not tested

• Human performance• Do not describe how and based on what underlying assumptions the

crew acts in the situation• Not predictive of future performance in the conditions not tested

12

The Development of the Evaluation Framework

Evaluation Framework Development

The Design Process

Current control room

Evaluation frameworkpreliminary version

Simulation &Evaluation

BASELINE

Evaluation frameworkversion 1

Evaluation frameworkversion 2

Modernisation phase 1 Modernisation phase 2

Simulation &Evaluation

Simulation &Evaluation

13

Concept of System Usability

• System Usability: The effect of the emerging technology on the whole activity system

• In NPP modernizations: the effect on process performance, user practices, user acceptance

• System Usability denotes how the system works as a

• Material• Cognitive• Communicative

tool in an organization promoting the fulfillment of the core task

Control and

automation system UI

User practices

Process performanc

e

14

DataEmpirical- orientation interview- simulator run- stimulated process tracing interviews- interface interviewsCourse of actionanalysis- goals,perceptions & actions- communications- resource utilisation

Indicators

- trust-utilisation of functional possibilities

Indicators

Outcome- process measures- error- work load- procedure following

Way of acting

- orientation- way of perception and action- way of collaboration - way of communication- way of using procedures

Assessment Criteria

Effectiveness & efficiency- process parameters- number of errors- TLX- number of deviations

Core-task oriented appropriateness

- realistic-objectivistic- reactive-interpretative - transparency of actions - shared horizon and meaning- understanding the rationale as making sense

ModellingDomain- motives &objectives- functionsSituation- constraints &possibilities- resourcesComplexity- interactions- dynamics- uncertainty

Internal

good

External

good

Criteria- Evidence of the possibility for creating new usage practices and culture

PRACTICE

Si t u

atio

n al c

r iter

ia

Ass

essm

ent

of s

yste

m u

sabi

li ty

EXPERIENCED APPROPRIATENSS

15

Conclusions

• Traditional scientific performance measures do not differentiate between UIs in a highly automated environment more profound criteria in assessment are needed

• A system with high system usability induces good working practices on the users

• With practices individual users cope with system uncertainty which is a critical demand in the NPP environment

• In validation practices within a new system will be compared to the practices in the baseline evaluation within the valid traditional system

• Further work: Connect the practice-driven performance indicators to the changes in the modernization

16

Thank You!

17

18

19

20

21

Classification of User Practices

• Reactive• Repetition of pre-learned• Not understanding the reasoning behind i.e. procedures

• Diffuse• Characteristics of both reactive and interpretative

• Interpretative• Takes into account the situational variation in objectives• Attempts to interpret what contextual factors have an effect• Understands the trade off between the actions for acute and chronic

cures; the effect of one’s own actions to the overall performance goals of operation

22

Practice Related Criteria in Validation

Baseline Validation

Diffuse

Interpretative

Fail Fail Pass

Acceptable: Rea validation ≤ Reabaseline Λ Int validation ≥ Int baseline

Reactive