1

Human Factors Research

Some OSU examples

2

Users,Operators,

Subject MatterExperts

Human Factors Research to Inform the Human-Machine Systems Engineering Process

Needs,Problems,

Opportunities

Operation,Test& Evaluation Analysis

DesignImplementation

DesignSpecifications

RequirementsHMS: Humans,Machines,Processes(Model, Mockup,Prototype, Product)

Data Collection

Data Analysis &Hypothesis Testing

Interpretation &Application of Results

GeneralizableResearch

Question(s)

HFE Principles& Guidelines

Research Design

Hypothesis Formulation

3

Human Factors Research:An Overview

● Experimental Methods● Relationships studies

– independent variables → dependent variables● Comparative studies

● Descriptive Methods● Literature Review● Observation● Surveys and Questionnaires● Incident and Accident Analysis● Modeling and Simulation● Meta-Analysis

● Always involve human subjects/participants (directly or indirectly)

4

Human Factors and Aviation Safety

Source: Boeing Commercial Airplanes

Primary Causes of Aircraft AccidentsHull Loss Accidents – Worldwide Commercial Jet Fleet – 1994 Through 2005

Maintenance

Airport/ATC

Misc./Other

Weather

Airplane

Flight Crew

0%10%

20%30%

40%50%

60%70%

80%90%

100%

3%

5%

7%

13%

17%

55%

5

Human Factors Research Methods In OSU's Cockpit Task Management (CTM) Research

CTM: Process by which pilots selectively attend to multiple, concurrent flight tasks to safely and effectively complete a flight.

Lockheed L1011

Boeing 777

6

Developing a Conceptual Framework For CTM: Literature Review, Analysis, and Modeling

● Literature Review● Cockpit Resource Management (e.g., Lauber, 1986)● Human error in aviation (e.g, Nagel, 1988; Wiener, 1987; Ruffel-Smith, 1979)● Cognitive psychology (e.g., Navon & Gopher, 1979; Wickens, 1984)● Systems theory (e.g., Padulo & Arbib, 1979)

● A Model of CTM● initiate tasks to achieve goals● assess status of all tasks● terminate completed tasks● prioritize remaining tasks based on

– importance:1. aviate2. navigate3. communicate4. manage systems

– urgency– other factors (?)

● allocate resources (attend) to tasks in order of priority

Funk, K.H. (1991). Cockpit Task Management: Preliminary Definitions, Normative Theory, Error Taxonomy, and Design Recommendations, The International Journal of Aviation Psychology, Vol. 1, No. 4, pp. 271-285.

7

Determining the Significance of CTM:Accident Analysis

● CTM Error Taxonomy● Task Initiation: early / late / incorrect / lacking● Task Prioritization: incorrect● Task Termination: early / late / incorrect / lacking

● Method:● Reviewed 324 National Transportation Safety Board (NTSB) Aircraft Accident Reports

(1960 – 1989)● Developed pre-impact timelines, classified CTM errors

● Findings: 80 CTM errors in 76 (23%) of the accidents

Chou, C.D., D. Madhavan, and K.H. Funk (1996). Studies of Cockpit Task Management Errors, International Journal of Aviation Psychology, Vol. 6, No. 4, pp. 307-320.

CTM Error # Accidents% CTM

Accidents # CTM Errors% of All CTM

Errors

Task Initiation 35 46 35 44

Task Prioritization 24 32 24 30

Task Termination 21 28 21 26

8

Determining the Significance of CTM:Incident Analysis

● Method:● Reviewed 470 Aviation Safety Reporting System (ASRS) incident reports:

– Controlled Flight Toward Terrain incidents– In-flight engine emergency incidents– Terminal flight phase incidents

● Identified concurrent tasks, classified CTM errors● Findings: 231 (49%) of the incidents involved CTM errors

Chou, C.D., D. Madhavan, and K.H. Funk (1996). Studies of Cockpit Task Management Errors, International Journal of Aviation Psychology, Vol. 6, No. 4, pp. 307-320.

Conclusion: CTM is a significant factor in flight safety.

CTM Error # Incidents% CTM

Incidents # CTM Errors% of All CTM

Errors

Task Initiation 137 59 145 42

Task Prioritization 133 58 122 35

Task Termination 83 36 82 23

9

Understanding CTM:Incident Analysis

● Does cockpit automation level affect task performance?

● Method: ● Reviewed 420 NASA ASRS

incident reports– 210 advanced technology +

210 conventional technology ● large commercial transport

aircraft● 2 pilots● 1988-89, 1990-91, 1992-93

● Reviewed narratives● Constructed task models● Classified errors● Comparison with t-tests

● Findings:● Error rate higher for advanced

technology aircraft (p = 0.036)● Error rate decreasing (p = 0.032)

Task Prioritization Error Frequency

Total Errors by Submission Period

Ad

van

ced

Tec

hno

logy

Tra

dit

iona

l T

ech

nolo

gy

Submission Period

1988-1989 13 7 20

1990-1991 11 5 16

1992-1993 4 3 7 Total Errors by

Aircraft Technology 28 15

Wilson, J. and K. Funk (1998). The Effect of Automation on the Frequency of Task Prioritization Errors on

Commercial Aircraft Flight Decks: An ASRS Incident Report Study, Proceedings of the Second Workshop on

Human Error, Safety, and System Development, Seattle, WA, April 1-2, 1998, pp. 6-16.

10

Understanding CTM:Simulator Study

● What are the factors that affect task prioritization in the CTM process?

● Method: simulator study● Professional pilot participants● Difficult San Francisco approach scenarios● Task prioritization Challenge Probe Points (CPPs)● Stop sim or record & replay for interviews on CPPs:

“Why did you ...?”● Analysis with ANOVA

● Findings: Prioritization Factors1. Procedural compliance2. Task importance3. Task salience4. Task status5. Time/Effort requirements6. Task urgency

C A R M E

A N J E E

S K U N K

B O L D R

M E N L O

O A K V O R

S F O 2 8 R I L S

B i g S u r V O R

E 1 - C A R M E( S c e n a r i o E v e n t )

T u r n f r o m V 2 7 t o 3 3 1 R a d i a lD M E = 8 3 . 9

E 2 - D M E 4 2( A T C E v e n t )

V e c t o r a n d A l t . I n s t r u c t i o n sD M E = 4 2

E 3 - B O L D R( M a l f u n c t i o n E v e n t )

B u s T i e C o n t a c t o r ( M 4 )D M E = 3 4

E 4 - V e c t o r 3 6 0( A T C E v e n t )

V e c t o r i n s t r u c t i o nD M E = 2 5

E 5 - L o c a l i z e r( S c e n a r i o E v e n t )

L o c a l i z e r N e e d l e " S w i n g s "D M E = 1 7 . 6

E 6 - F i n a l( M a l f u n c t i o n E v e n t )

B o o s t P u m p F a i l u r e ( M 5 )D M E = 1 3

F l i g h t P a t h

S t u d y 1 B r a v o E x p e r i m e n t a l S c e n a r i o w i t h D a t a C o l l e c t i o n P o i n t s

S t a r t t u r n a b o u t 8 3 . 9 f r o m O A K

S p e e d = 3 0 0a l t = 1 0 , 0 0 0f r e q = 1 3 4 . 5

S p e e d = 2 1 0a l t = 8 0 0 0

f l a p s = 1a p p r o a c h / d e s c e n t c h e c k l i s t

a l t = 6 0 0 0

2 5 n m f r o m O A KS p e e d = 1 9 0

f l a p s = 5

S p e e d = 1 6 5f l a p s = 2 5

F i n a l D e s c e n t c h e c k l i s t

E 1 - S c e n a r i o E v e n tE 2 - A T C E v e n tE 3 - M a l f u n c t i o n E v e n tE 4 - A T C E v e n tE 5 - S c e n a r i o E v e n tE 6 - M a l f u n c i t o n E v e n t

3 6 0 °

3 2 5 °

D i r e c t i o n o f F l i g h t

Colvin, K., K. Funk, & R. Braune (2005). Task Prioritization Factors: Two Part-Task Simulator

Studies, International Journal of Aviation Psychology, Vol. 15, No. 4, pp. 321–338.

11

Improving CTM:Experimental Study (Training)

● Can task prioritization be trained?● APE Mnemonic: Assess, Prioritize, Execute● Simulator Experiment

● Licensed pilot participants● Independent variable: training (Descriptive,

Prescriptive, None/Control)● Dependent Variables

– Task Prioritization Error Rate– Prospective Memory Recall

● Flight – training / no training – flight● ANOVA of results●

–

–

–

–

–

Bishara, S. and K. Funk (2002). Training Pilots to Prioritize Tasks, Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting, Baltimore, MD, September 30-October 4, 2002, pp. 96-100.

Interaction Plot

Flight

Mem

ory

Task

s

GroupControlDescriptivePrescriptive

0.5

0.6

0.7

0.8

0.9

1

Pre Training Post Training

Pros

pect

ive

Mem

ory

Per

f.

Prescriptive

Descriptive

Control

12

Improving CTM:Experiment (System Comparison)

● Can CTM be facilitated by a cockpit aid?● AgendaManager (CTM aid) vs. EICAS (conventional pilot warning/alerting system)● Simulator Experiment

● Professional pilot participants● Independent Variables: Alerting (AMgr vs. EICAS), Scenario● Dependent Variables: CTM metrics● Flight 1 (EICAS/AMgr) – Flight 2 (Amgr/EICAS)● ANOVA of results

Funk, K. and Braune, R. (1999). The AgendaManager: A Knowledge-Based System to Facilitate the Management of Flight Deck Activities, SAE 1999-01-5536. 1999 World Aviation Congress, 19-21 October 1999, San Francisco, CA.

Dependent Variable AMgr EICAS sig.

Within subs. correct prioritization 100% 100% NS

Subs. fault correction time (sec) 19.5 19.6 NS

A/F programming time (sec) 7.9 5.9 NS

goal conflicts % corrected 100% 70% 0.10

goal conflict resolution time (sec) 34.7 53.6 0.10

Subs./Aviate correct prioritization 72% 46% 0.05

Mean # unsatisfactory tasks 0.64 0.85 0.05

% time all tasks satisfactory 65% 52% 0.05

Mean participant rating (-5 - +5) 4.8 2.5 0.05

Findings

13

Flight Deck Automation Issues Research:Literature Review, Surveys, Accident/Incident Analyses, Meta-Analysis

Funk, K., B. Lyall, J. Wilson, R. Vint, M. Niemczyk, C. Suroteguh, and G. Owen (1999). Flight Deck Automation Issues, International Journal of Aviation Psychology, Vol. 9, No. 2, pp. 109-123.

1. Automation may demand attention.

2. Automation behavior may be unexpected and unexplained.

3. Pilots may be overconfident in automation.

4. Behavior of automation may not be apparent.

5. Failure assessment may be difficult.

6. Mode transitions may be uncommanded.

7. Mode awareness may be lacking.

8. Mode selection may be incorrect.

9. Situation awareness may be reduced.

10. Understanding of automation may be inadequate.

Top 10:

L1011 vs. B777

14

Operating Room Human Factors Research:Observation and Modeling

● Observed surgical procedures in Oregon hospital.● Interviewed surgeons, nurses, assistants.● Developed IDEF0 process (functional) model of

laparoscopic cholecystectomy (gall bladder removal using minimally invasive procedures) for future error identification.

4

check needle

3

insert needle

2

elevateabdominal w all

1

plan & assessV erres needle

4Ps/ needledisplacement

4Ps/Verres needle

4Ps/abdominal w all4Ps/ needle

insert

OR Sys factors/needle displacement

OR Sys f actors/Verres needle

OR Sys f actors/abdominal w all

Pt factors/abdominal w all

4Ps / Ve rres needle

OR Sys factors/ V erres needle ins er t

Pt: initial incision m ade

insert verres needletools & matls: used

check needle displacementtools & matls: used

verres needle tools & mat ls: us ed

elevate abdom inalw all tools & m atls: used

Pt: Verres needleinserted info

Pt: abdominalw all elevated info

Pt: infoPt: Verres

needle in info

Pt: Verresneedle positioned

Pt: Verresneedle in

Pt: abdominalw all elevated

elevate abdominal w alltool & matls: ready to use

check needle displacement tools &matls: ready to use

insert verres needle tools &matls: ready to use

check needledisplacement goal

insert V erresneedle goal

elevate abdominalw all goal

insert Verresneedle subgoals

Pt factors/needle displacement

Pt factors/verres needle

w aste

surg specimens

OR Sys : use d

insert V er re s nee dle goal

Pt: Verres needle inserted

request for support

verres needle tools &m atls: ready to us e

Pt factors / verres nee dle

S FA

Funk, K.H., T.L. Doolen, R. Botney, and J.D. Bauer, “A Functional Model of the Operating Room,” Proceedings of the Human

Factors and Ergonomics Society 47th Annual Meeting, October 13-17, 2003, Denver, CO, pp. 1569-1573.

15

Operating Room Human Factors Research:Systemic Vulnerabilities Analysis

● Systemic vulnerabilities to Verres Needle insertion

● Improperly visualize underlying anatomy.● Miss important cue for needle

placement.● Choose wrong insertion angle.● Fail to stabilize abdominal wall.● Misinterpret the degree of needle

resistance ...● Err in sensing the click of the needle● etc.

● Validated by post hoc literature review.

Funk, K.H., J.D. Bauer, T.L. Doolen, D. Telasha, R.J. Nicolalde, M. Reeber, N.

Yodpijit, and M. Long (2010). The use of modeling to identify vulnerabilities to

human error in laparoscopy, The Journal of Minimally Invasive Gynecology, Vol.

17, No. 3, pp. 311-320.

FMEA+

16

Operating Room Distractions and Interruptions

Research in collaboration with OHSU Department of Surgery

● Simulated laparoscopic cholecystectomy● 18 OHSU 2nd & 3rd year surgical residents● Independent Variable: Distracted vs non-

distracted● Dependent Variables:

● Damage to organs● Collateral blood loss● Remembering to announce closure● Total and cauterizing times

● Results: 8 out of 18 committed errors when distracted versus 1 out of 18 when not distracted

Distractions/Interruptions # Errors

Visual movement 0

Ringing cell phone 1

Question about “crashing” patient 4

Side conversation 3

Question about choice of profession 2

Dropped metal tray 0

Feuerbacher, R.L.,Funk II, K.H., Spight, D.H., Diggs, B.S., Hunter, J.G. (2012). Realistic distractions and interruptions impair simulated surgical performance by novice surgeons, Archives of Surgery,

http://archsurg.jamanetwork.com/article.aspx?articleid=1216543.

17

A Comparison of Human and Near-Optimal Task Management Behavior

● Objectives● Framework for studying human TM behavior &

performance● Study human TM strategy, tactics● Compare human TM performance with near-

optimal heuristic (tabu search)● Background

● Common TM situations● Engineering models of TM

● Method: “Simulator” Experiment● Apparatus: Tardast TM “game”● Participants: 10 OSU students● 5 randomized scenarios ● IVs: DRs, CRs, Ws● DVs: scores, strategies, tactics ● Compared with tabu search

heuristic● Results

● Human < tabu (not by much)● Different strategies, tactics

● Conclusions● Too many tasks → too few● Over-attention to salient stimuli● Tardast a useful framework

Shakeri, S., Funk, K. (2007). A comparison of human and near-optimal task management behavior, human

factors, vol. 49, No. 3, pp. 400–416.

18

Human Factors Research:Summary

● Experimental Methods● Relationships studies

– independent variables → dependent variables● Comparative studies

● Descriptive Methods● Literature Review● Observation● Surveys and Questionnaires● Incident and Accident Analysis● Modeling and Simulation● Meta-Analysis

● Always involve human subjects/participants (directly or indirectly)

19

Users,Operators,

Subject MatterExperts

Human Factors Research to Inform the Human-Machine Systems Engineering Process

Needs,Problems,

Opportunities

Operation,Test& Evaluation Analysis

DesignImplementation

DesignSpecifications

RequirementsHMS: Humans,Machines,Processes(Model, Mockup,Prototype, Product)

Data Collection

Data Analysis &Hypothesis Testing

Interpretation &Application of Results

GeneralizableResearch

Question(s)

HFE Principles& Guidelines

Research Design

Hypothesis Formulation