human machine cooperation on moon or mars … eva suit can ... each meca unit will alert the crew...

http://www.CrewAssistant.com

Human Machine Cooperation on Moon or Mars

Mark Neerincx, Jasper Lindenberg, Nanja Smets, Tim Grant, André Bos, Leo Breebaart, Antonio Olmedo Soler, Uwe Brauer, Mikael Wolff


Objective & Vision

Objective: support mission goals (without injury or loss of life) by• empowering the cognitive capacities of human-machine

teams during planetary exploration missions • in order to cope autonomously with unexpected, complex and

potentially hazardous situations.

Vision: crew support that • acts in a ubiquitous computing environment • as “electronic partner”, helping the crew

– to assess the situation, – to determine a suitable course of actions to solve a problem, – to refresh skills and knowledge,– to improve resilience.


Illustrative space scenario

Benny and Brendaare on a rock collecting procedure.

Suddenly,Benny’s

spacesuit fails

A Rover from another team comes to pick up Benny

Benny is brought to the habitat for recovery


Situated Cognitive Engineering

Prototype

HitL-test

UX

Simulation

Sim-Assess

Sim Results

OperationalDemands

Human FactorsKnowledge

EnvisionedTechnologyDerive

Refine

Review

Comments

Test

Refine

Requirements

Use Cases Claims Core Functions

Specify

contextualizeorganize justify


Operations: Abstraction Decomposition Space (ADS) Claire Baker

Outpost Spaces/ Systems

Subspaces/ Subsystems

Components

Mission Priorties

Crew&Vehicle Safety

Values & Prio Measures

Minimise crew system maint…

Prio-related Functions

Surface Exploration

Maintenance & repair

Information dissemination

communication

Physical Capability

Base cap. to support n astronauts…

Prod. levels of ISRU plant

EVA suit can carry…

EVA suit can maintain temp. at x for y hours

Physical Objects

Mars surface outpost

ISRU plant EVA suits EVA suit thermal control…

=> decomposition

abst

ract

ion

<=


Human Factors

• Cognitive Task Load • Situation Awareness• Sense Making• Decision Making• Dynamic Human Capacities• Trust and Emotion• Collaboration• Crew Resource

Management


Technology

• Smart Task Environment with automatic distribution of data, knowledge, software and reference documents (can cope with infrastructure failures)

• Agent, Web technology and smart sensors• Model-based reasoning and health management• Human-machine (e.g. robot) collaboration and mixed

reality• Boundaries of the technology:

– maturity, – graceful degradation, – maintainability and fault tolerance, and – constraints of the hostile environment.


ePartner Concept that integrates operations, human factors and technology

Has information of its hPartner, e.g.• permanent characteristics (e.g., personality) • dynamic characteristics (e.g., experience) • base-line state (e.g., “normal” heart rate) • momentary state (e.g., current heart rate)• tasks to do (e.g., alarm handling)• task performance (e.g. time)• current context (e.g., location)

And interprets this information to• assess human’s condition for current context• identify critical situations (e.g. panic)• apply mitigation strategies to reduce the negative

effects (e.g. reschedule tasks, notify colleague, …)


ePartner’s knowledge

• Easy to share with its hPartner• Trustworthy• Based on situated cognitive theories:

– cognitive task load– emotional state– fitness– team involvement

• Continuously updating the models via human input, and automatic sensing of human behavior, physiology and context



Prototype

HitL-test

UX

Simulation

Sim-Assess

Sim Results

OperationalDemands



Refine

Review

Comments

Test

Refine

Requirements


Specify



Theory-driven requirements engineering

requirementsrequirements

use casesuse cases claimsclaims

justifycontextualizeorganize

Situated Cognitive Theory

Requirements



ePartner Requirement Specification

Identify critical states and start mitigation:• Dialogue Style• Feedback• Crew Notification• Information Filter• Task Allocation• Automation Level• …

Example specification (simplified)

RF2024 MECA shall communicate with the crew about important events.

Claim C064 Each MECA unit will alert the crew member regarding for instance scheduled events, low-frequency nominal events and off-nominal events.+ Helps maintain high situation

awareness for crew members. Consistent notifications help maintain sufficient trust.

- May interrupt with current activities, increasing cognitive task load, and hence decreasing effectiveness or efficiency.

Use-Cases UC077, UC078, UC080, UC083

ePartner concept


Justification and Refinement

Claims should provide an adequate justification: • Truthful:

– information is factual – upsides, downsides & trade-offs occur as in reality – new facts => requirement is modified or removed

• Exclusive: – explains why current (not another) requirement is optimal. – alternative requirement with same upsides, downsides and

trade-offs => generalization until further research reveals which of its instantiations is the best candidate.

=> Refinement of requirements and the corresponding claims iteratively proceeds from general to specific, carefully justifying at each step the refinement made.


Level of Specification and Assessment

Task level• based on users’ goals and information needs, the

system’s functions and information provision are specified or assessed

• functional requirements with corresponding claims

Communication level • the control of the functions and the presentation of

the information are specified or assessed (i.e. the “look-and-feel”)

• User Interface features with corresponding claims


Empirical-driven refinement

The requirements derivation process:• iterative• incremental • top-down

Task level: • Adaptive notification => increased

awareness and minimal interruption

Communication level:• avatar facial expressions => valence effect• avatar speech expressions => arousal effect

“Your space suit shows a malfunction concerning the

temperature regulation. I informed Brenda and

Herman…”

E.g., adaptive notification of an ePartner


Use-Case-based Reqs Structuring

UC083 Alarm handling

Goal To bundle low-level alarms into meaningful events.

Actors Astronaut and MECA in habitat.

…

Reqs RF2024, RF2050, RF2080,…

Sequen ce

1. Low-level smoke and IR alarms. 2. MECA combines the low-level alarms to determine location/scope of fire. 3. MECA gets attention of astronaut 4. MECA provides procedures 5. Astronaut fights fire 6. Fire is put out.

Example use-case (simplified).

A

B

C

D

EF

G

H

A B C D E HGF

(b)

(a)

Clustering of 8 reqs

hierarchical clustering: similarity based on use-cases to which the reqs apply

hierarchical clustering: similarity based on use-cases to which the reqs apply


Use-Case-based clustering

Alternative or complement for functional decomposition• really user-centered• dynamic and flexible• can highlight functional-usage structure mismatches

Optimizing evaluation for testability and empirical value:• The more requirements in a hypothesis, the more closely it

resembles the design specification => the higher its empirical value, but the lower its testability.

• Related requirements have interdependencies and apply to similar situations => increases its empirical value and testability

Selecting hypotheses for empirical evaluation:• Select an “under-evaluated” hypothesis• If the hypothesis is very difficult to test, move down • If the hypothesis is very easy to test, move up


Dendogram

Nodes represent clusters/hypotheses• Red: not plausible yet • Green: plausible

Nodes represent clusters/hypotheses• Red: not plausible yet • Green: plausible



Prototype

HitL-test

UX

Simulation

Sim-Assess

Sim Results

OperationalDemands



Refine

Review

Comments

Test

Refine

Requirements


Specify



Evaluation

Fidelity • adequate representation of relevant rules, particularly

dependencies, in a human-agent team

Realism • varies from one extreme—the real environment—to the

other—a virtual environment.

Evaluation Options• Computer simulations• Mixed Reality lab experiments• Real life experiments in analogue environments• Real life Experiments in a 500 day isolation experiment.


Increasing fidelity and reality

Reality

Fide

lity

Virtual Real

Low

Hig

h

Computer simulated

Computer simulated & policies

Human-in-

VR-loop

Moon/Mars

WOz

Analogue environment


Ontologies

Policies

agent 1agent 2agent 3

… KAOS: Simulation of Collaboration.• Obligations, authorizationsBrahms: Simulation of Work Practice.• Agents, …

Computer Simulations

http://www.CrewAssistant.com 2008Human Factors24

WOz Evaluation with Virtual Environments


Evaluations in analogue environments/ isolation

MARS500: gaming as diagnostic and training tool for resilienceAnalogue environments (e.g.,

Eifel volcanic area): ePartner helps to cope with (social, cognitive and affective) demanding situations.

Concordia: maintaining physical and mental fitness

DoneDone

In ProgressIn Progress

PlanPlan


MARS-500

• Collaborative Trainer (COLT)– Multi-user – 1 instructor and 2 trainees– Learn to use systems (water tank,

cardiopres) with a procedure viewer

• Colored Trails– Multi-user – Proposers and responders – Position pieces onto or as close as

possible to a goal location

• Lunar Lander– Single-user– Land a Lunar Lander on the Moon

A M H AAutomatic Mental Health Assistance


Cognitive Task Load

Level of Information Processing:How complex is the task for the actor?• Low: routine (“automatic”)• High: non-routine (“intensive

problem-solving”)

Task-Set Switches:How often must the actor switch?• Low: one task after the other• High: continuous interleaving or

interruption

Time occupied:How much time for small pauses?• Low: slowing down/ pauses are

allowed• High: must perform at maximum

speed to complete the tasks


Emotion: Valence and Arousal

Arousal

Valence

high

positivenegative

low

Relaxed/soothing

Delight/rejoice

gloomy

Terrified/restless

Arousal (exciting/calm)Valence (displeasure/pleasure)


Modeling Cognitive Task Load and Emotion (e.g. with Bayesian Networks)

LIP

U/Oload

Events …

Cognitive Task Load

Performance

Emotional State

TO

Tasks

Happiness

Speech…

Face

Valence ArousalTSS

CogLockUp…

…Anger

observables

classe

s

dimensions

Identify critical states:• intra & inter models


Bayesian Networks

• Analysis existing data sets– Process control tasks in laboratory

• 12 students: platform supervision, damage control & navigation

• 1407 cases (60 sec with an overlap of 50 sec)• Performance prediction: 84.8%

– Process control task on sailing ship• 24 participants: 3 scenario’s on sailing ships• 3478 cases (60 sec with an overlap of 50 sec)• Performance prediction: 74.2%

• MECA m500 software and protocol– 24 participants performed 306 cases– Performance prediction:

• CTL: 59.2%• ES: 57.8%• CTL + ES: 67.0%


Timeline Tool and General Feedback


Session Feedback and Annotation

2


Conclusions

• The MECA project is providing a Requirements Baseline (RB) with prototypes for enhancing human-machine resilience– Several refinements have been established

• The situated Cognitive Engineering Method helps to establish a theoretical and empirical sound RB– Tools are needed for updating and maintaining the RB

• Experiments with prototypes are providing – Performance data and user feedback on the ePartner functions– Data to train the ePartner models on Cognitive Task Load, Emotion,

Social Networks and Performance.

• MECA provided a software platform that combines simulation and Human-in-the-Loop evaluation in a flexible way– Combination with Virtual Reality to be done

• Scientific progress is being shown in numerous publications


Next Steps

• Requirements baseline tools for– sharing (e.g. for implementation)– maintenance (e.g. for version management)– refinement (e.g. for traceability)– organization (e.g. for evaluation guidance)

• Experiments– The 520-days study in MARS-500– Concordia– International Space Station

• Prototypes– Automatic state sensing– User modeling– Adhering to policies for human-machine collaboration– Persuasion of desirable behaviour– Supporting of refreshment training


Questions?

human machine cooperation on moon or mars … eva suit can ... each meca unit will alert the crew...

Documents