reliance on decision aids results from 15 studies using a soldier- detection task april 12, 2006

Reliance on Decision AidsReliance on Decision Aids

Results from 15 Studies Using a Soldier-Detection Task

April 12, 2006

CollaboratorsCollaborators• Linda PierceLinda Pierce• Hall (Skip) BeckHall (Skip) Beck• Scott PetersonScott Peterson• Regina PomrankyRegina Pomranky• Lori PurcellLori Purcell• Lloyd DaweLloyd Dawe• Catherine FieldCatherine Field• Angela PriceAngela Price

Cameron University

• Jennifer BatkaJennifer Batka• Emily BeadlesEmily Beadles• Paul ClarkPaul Clark• Tamara YoungTamara Young• Paul ClarkPaul Clark• Rachael RossRachael Ross• Adrian GohlAdrian Gohl• Emily DesorentoEmily Desorento

How can one best design How can one best design a system that will a system that will

encourage humans to encourage humans to appropriately use an appropriately use an

automated decision aid?automated decision aid?

Research QuestionResearch Question

Human Factors Literature:Human Factors Literature:Inappropriate Use of AutomationInappropriate Use of Automation

Parasuraman & Riley (1997)Parasuraman & Riley (1997)

Misuse: Misuse: Overreliance on automationOverreliance on automation

Disuse: Underutilization Disuse: Underutilization of automationof automation

Study 1Study 1

In Study 1, the reliability of the aid was In Study 1, the reliability of the aid was manipulated in a Visual Detection manipulated in a Visual Detection

Task and automation use was Task and automation use was examinedexamined

Study 1 Research DesignStudy 1 Research Design

• Four Conditions:Four Conditions:– AloneAlone– Computer with 90% Accuracy Computer with 90% Accuracy – Computer with 75% AccuracyComputer with 75% Accuracy– Computer with 60% AccuracyComputer with 60% Accuracy

• Dependent VariablesDependent Variables– PerformancePerformance

– p(incorrect)p(incorrect)– Inappropriate Automation UseInappropriate Automation Use

– Misuse = p(incorrect | aid incorrect)Misuse = p(incorrect | aid incorrect)– Disuse = p(incorrect | aid correct)Disuse = p(incorrect | aid correct)

p(US | CS)p(US | CS)

Inappropriate Automation UseInappropriate Automation UseRescorla-Wagner ModelRescorla-Wagner Model

p(US | no CS)p(US | no CS)

inhibitory association

excitatory association

no a

ssoc

iation

Inappropriate Automation UseInappropriate Automation Use

p(error | aid error)p(error | aid error)

p(p(errorerror | aid correct | aid correct))

Opposite ofAid

Reliance onAid

Igno

ring

of a

id

Procedure

Two Hundred Trials

View Slide

Make Decision

Indicate Confidence Level

View Aid’s Decision

Contrast Detector

Continue

PRESENT

Response Panel

Was the enemy in the precedingphotograph?

YES NO

Continue

Decision: YES

Continue

Highly Confident

Confident

Somewhat Confident

Slightly Confident

Not At All Confident

How confidentare you thatyou have madethe correctdecision?

AnalysisAnalysisSignal Detection TheorySignal Detection Theory

Soldier Soldier PRESENTPRESENT

Soldier Soldier ABSENTABSENT

Participant Participant responds responds PRESENTPRESENT

HitHit False AlarmFalse Alarm

Participant Participant responds responds ABSENTABSENT

MissMiss Correct Correct RejectionRejection







MissMiss Correct Correct RejectionRejection

If you use p(error), you miss potentially useful informationIf you use p(error), you miss potentially useful information







MissMissCorrect Correct

RejectionRejection

TotalTotal Signal TotalSignal Total Noise TotalNoise Total






TotalTotal Signal TotalSignal Total Noise TotalNoise Total

Hit Rate = p(S says present | soldier is present)Hit Rate = p(S says present | soldier is present)False Alarm Rate = p(S says present | soldier is absent)False Alarm Rate = p(S says present | soldier is absent)


• Subject’s SensitivitySubject’s Sensitivity– How well the participant discriminated between the How well the participant discriminated between the

soldier present and absent trialssoldier present and absent trials– A subject with PERFECT sensitivity would haveA subject with PERFECT sensitivity would have

• Hit Rate = 1Hit Rate = 1• False Alarm Rate = 0False Alarm Rate = 0

– A subject with NO sensitivityA subject with NO sensitivity• Hit Rate = False Alarm RateHit Rate = False Alarm Rate

– Usually subjects fall between these two extremesUsually subjects fall between these two extremes– d’ = z(Hit Rate) – z(False Alarm Rate)d’ = z(Hit Rate) – z(False Alarm Rate)


Soldier AbsentSoldier Absent Soldier PresentSoldier Present

What is d’?What is d’?



d’d’

What is d’?What is d’?



What is bias?What is bias?C = -.5[z(Hit Rate) + z(False Alarm Rate)]C = -.5[z(Hit Rate) + z(False Alarm Rate)]

C=0C=0c = participants bias to say absent or presentc = participants bias to say absent or present


ROC Curve (Receiver Operating Characteristic)ROC Curve (Receiver Operating Characteristic)

False Alarm RateFalse Alarm Rate

Hit

Rat

eH

it R

ate

d’=0d’=0d’=.5d’=.5

0

0.5

1

1.5

2

2.5

3

3.5

-2.1 -1.6 -1.1 -0.6 -0.1 0.4 0.9 1.4

Hits

Fal

se A

larm

s

ROC of Standardized Cumulative Probabilities ROC of Standardized Cumulative Probabilities for Subject #37 for Trials When Machine is Correctfor Subject #37 for Trials When Machine is Correct

A(z) = the complement of the area under the ROC curveA(z) = the complement of the area under the ROC curve

Study 1 ResultsOverall Performance

0

0.1

0.2

0.3

0.4

0.5

Alone 60% 75% 90%

ProbabilityProbabilityIncorrectIncorrect

A(z)A(z)

ConditionCondition

Study 1 ResultsExtent of Automation Use

DISUSEDISUSEProbability ofProbability of

Incorrectly Incorrectly IgnoringIgnoring

the Systemthe System

p(incorrect | aid correct)p(incorrect | aid correct)

MISUSEMISUSEProbability ofProbability of

Incorrectly Relying Incorrectly Relying onon

the Systemthe System

p(incorrect | aid incorrect)p(incorrect | aid incorrect)

.128.128 .265.265

Study 1 ResultsEffect of Reliability on Automation Use

0.21

0.13

0.25

0.12

0.34

0.13

0

0.1

0.2

0.3

0.4

0.5

60% 75% 90%

Aid Incorrect Aid Correct

Reliability of Automated AidReliability of Automated Aid

ProbabilityProbabilityIncorrectIncorrect

A(z)A(z)

Conclusions from Study 1Conclusions from Study 1

• Machine accuracy (or automation reliability) did Machine accuracy (or automation reliability) did not affect reliance on automated aid or overall not affect reliance on automated aid or overall performanceperformance

• Participants in the aided conditions relied on the Participants in the aided conditions relied on the aid more than they ignored the aidaid more than they ignored the aid

Study 1 ImplicationsStudy 1 Implications

• BottomBottom Line• Improving the reliability of the automated decision Improving the reliability of the automated decision

aid does NOT improve the performance of the aid does NOT improve the performance of the human-automation teamhuman-automation team

• People relied on the unreliable decision aid almost People relied on the unreliable decision aid almost as much as they relied on the reliable decision as much as they relied on the reliable decision aid...aid...

• QuestionQuestion• Why do people inappropriately use automated Why do people inappropriately use automated

aids?aids?

Why do people inappropriately useautomated aids?

Group DynamicsGroup DynamicsLiteratureLiterature

Human FactorsHuman FactorsLiteratureLiterature

Cognitive Cognitive ProcessesProcesses

Cognitive MiserCognitive MiserHypothesisHypothesis

MotivationalMotivationalProcessesProcesses

Diffusion ofDiffusion ofResponsibilityResponsibility

Social ProcessesSocial Processes Authority HypothesisAuthority Hypothesis

Cognitive Processes

• People use the information provided by the automated aid in a heuristic manner

• The automated aid’s decision may be given undue weight in the human’s decision making process

Model of Automation Reliance:Cognitive Processes

Cognitive ProcessesCognitive Processes

Provision of Aid’sProvision of Aid’sDecision Prior toDecision Prior toDecision MakingDecision Making

AutomationAutomationUseUse

To eliminate this process, provide the aid’s decisionTo eliminate this process, provide the aid’s decisiononly after the participant has indicated his decision.only after the participant has indicated his decision.

Motivational Processes:Diffusion of Responsibility

Shepperd’s Expectancy-Value Theory

• Expectancy• Instrumentality• Outcome Value

Motivation is a function ofMotivation is a function of

Motivational Processes:Shepperd’s Expectancy-Value Theory

• Expectancy – The extent to which members feel their

efforts are necessary for the group to succeed

• Instrumentality– The extent to which members feel that the

group’s successful performance will lead to a positive outcome

• Outcome Value – Importance of Outcome - Costs of Outcome

Model of Automation Reliance:Motivational Processes

Motivational ProcessesMotivational Processes

AutomationAutomationUseUseEffortEffort

ExpectancyExpectancy

DispensabilityDispensability

Perceived UtilityPerceived Utilityof Automated Aidof Automated Aid

Task DifficultyTask Difficulty

InstrumentalityInstrumentality

Outcome ValueOutcome ValueCostsCosts

ImportanceImportance

WorkloadWorkload

FatigueFatigue

Intrinsic InterestIntrinsic Interest in the Taskin the Task

PenaltiesPenalties

RewardsRewards To eliminate this process, make the effort expendedTo eliminate this process, make the effort expendedfor self-reliance equal to that for automation reliance.for self-reliance equal to that for automation reliance.

Motivational Processes

AutomationRelianceEffort

Expectancy

Dispensability

Perceived Utilityof Automated Aid

Task Difficulty

Instrumentality

Outcome ValueCosts

Importance

Workload

Fatigue

Intrinsic Interest in the Task

Penalties

Rewards

Social Processes:Authority Hypothesis

TRUSTTRUST

Model of Automation Use:Social Processes

Automation Automation UseUse

Social ProcessesSocial Processes

RelativeRelative TrustTrust

PerceivedPerceivedUtility ofUtility of

AutomatedAutomatedAidAid

Cognitive ProcessesCognitive Processes

Perceived ReliabilityPerceived Reliabilityof Automatedof Automated

AidAid

Perceived ReliabilityPerceived Reliabilityof Manual of Manual OperationOperation

ActualActualReliabilityReliability

of of AutomatedAutomated

AidAid

ActualActualReliabilityReliability

of of Manual Manual

OperationOperation

Bias TowardBias TowardAutomationAutomation

Self-Serving BiasSelf-Serving Bias

Study 2Study 2

Study 2 examined social processes (TRUST)Study 2 examined social processes (TRUST)while eliminating the cognitive processes while eliminating the cognitive processes

and holding constant the motivational and holding constant the motivational processesprocesses

Procedure

Two Hundred Trials

View Slide

Make Decision

Indicate Confidence Level

View Aid’s Decision

Response Panel

Was the enemy in the precedingphotograph?

YES NO

Continue

Decision: YES

Continue

Highly Confident

Confident

Somewhat Confident

Slightly Confident

Not At All Confident

How confidentare you thatyou have madethe correctdecision?

View Feedback (some participants)

Contrast Detector

Continue

PRESENT

Study 2 DesignStudy 2 Design

• Independent VariablesIndependent Variables– Partner’s Performance LevelPartner’s Performance Level– Provision of Cumulative FeedbackProvision of Cumulative Feedback– Reward Based on Future Trials…Reward Based on Future Trials…

InstructionsInstructions

……You will perform an additional ten trials. The amount You will perform an additional ten trials. The amount of extra credit that you will earn will entirely depend of extra credit that you will earn will entirely depend on the outcome of these next ten trials…on the outcome of these next ten trials…

The amount of extra credit...will either be based on The amount of extra credit...will either be based on your performance or on the performance of the your performance or on the performance of the contrast detector. It is your choice.contrast detector. It is your choice.


• Independent VariablesIndependent Variables– Partner’s Performance LevelPartner’s Performance Level– Provision of Cumulative FeedbackProvision of Cumulative Feedback– Reward Based on Future or Past TrialsReward Based on Future or Past Trials


……Ten trials will be randomly selected from Ten trials will be randomly selected from the preceding 200. The amount of extra creditthe preceding 200. The amount of extra creditthat you will earn will entirely depend upon the that you will earn will entirely depend upon the outcome of those trials.…outcome of those trials.…

The amount of extra credit...will either be based on The amount of extra credit...will either be based on your performance or on the performance of the your performance or on the performance of the contrast detector. It is your choice.contrast detector. It is your choice.


• Independent VariablesIndependent Variables– Partner’s Performance LevelPartner’s Performance Level– Provision of Cumulative FeedbackProvision of Cumulative Feedback– Reward Based on Future or Past TrialsReward Based on Future or Past Trials

• Dependent VariableDependent Variable– Choice of Self or Aid’s Performance to Determine Choice of Self or Aid’s Performance to Determine

RewardReward

Study 2 HypothesesStudy 2 Hypotheses

• Those working with an aid superior to the Those working with an aid superior to the participant would trust the automated systemparticipant would trust the automated system

• Those working with an aid inferior to the participant Those working with an aid inferior to the participant would not trust the automated systemwould not trust the automated system

• If self-serving biases exist, above pattern of results If self-serving biases exist, above pattern of results would appear only among participants given would appear only among participants given feedbackfeedback

Study 2 ResultsStudy 2 Results

ConditionCondition SelfSelf AidAid

Future Trials-No Feedback-InferiorFuture Trials-No Feedback-InferiorFuture Trials-No Feedback-EqualFuture Trials-No Feedback-EqualFuture Trials-No Feedback-Future Trials-No Feedback-SuperiorSuperior

998888

001111



Future Trials-No Feedback-InferiorFuture Trials-No Feedback-InferiorFuture Trials-No Feedback-EqualFuture Trials-No Feedback-EqualFuture Trials-No Feedback-Future Trials-No Feedback-SuperiorSuperiorFuture Trials-Feedback-InferiorFuture Trials-Feedback-InferiorFuture Trials-Feedback-EqualFuture Trials-Feedback-EqualFuture Trials-Feedback-SuperiorFuture Trials-Feedback-Superior

998888999999

001111000000



Future Trials-No Feedback-InferiorFuture Trials-No Feedback-InferiorFuture Trials-No Feedback-EqualFuture Trials-No Feedback-EqualFuture Trials-No Feedback-Future Trials-No Feedback-SuperiorSuperiorFuture Trials-Feedback-InferiorFuture Trials-Feedback-InferiorFuture Trials-Feedback-EqualFuture Trials-Feedback-EqualFuture Trials-Feedback-SuperiorFuture Trials-Feedback-SuperiorPast Trials-No Feedback-InferiorPast Trials-No Feedback-InferiorPast Trials-No Feedback-EqualPast Trials-No Feedback-EqualPast Trials-No Feedback-SuperiorPast Trials-No Feedback-Superior

998888999999998899

001111000000001100



Future Trials-No Feedback-InferiorFuture Trials-No Feedback-InferiorFuture Trials-No Feedback-EqualFuture Trials-No Feedback-EqualFuture Trials-No Feedback-Future Trials-No Feedback-SuperiorSuperiorFuture Trials-Feedback-InferiorFuture Trials-Feedback-InferiorFuture Trials-Feedback-EqualFuture Trials-Feedback-EqualFuture Trials-Feedback-SuperiorFuture Trials-Feedback-SuperiorPast Trials-No Feedback-InferiorPast Trials-No Feedback-InferiorPast Trials-No Feedback-EqualPast Trials-No Feedback-EqualPast Trials-No Feedback-SuperiorPast Trials-No Feedback-SuperiorPast Trials-Feedback-InferiorPast Trials-Feedback-InferiorPast Trials-Feedback-EqualPast Trials-Feedback-EqualPast Trials-Feedback-SuperiorPast Trials-Feedback-Superior

998888999999998899998866

001111000000001100001133


• When controlling motivational and cognitive When controlling motivational and cognitive processes, find disuse of automated aidprocesses, find disuse of automated aid

• Even when given feedback that the computer was Even when given feedback that the computer was superior, most participants chose to rely on their superior, most participants chose to rely on their performanceperformance


• Bottom LineBottom Line• In this paradigm, people did In this paradigm, people did

NOT appropriately rely on a NOT appropriately rely on a superior automated decision superior automated decision aidaid

• QuestionQuestion• Are people biased against Are people biased against

automated decision aids?automated decision aids?


• Independent VariablesIndependent Variables– Partner’s Performance LevelPartner’s Performance Level– Provision of Feedback Provision of Feedback – Type of PartnerType of Partner

• Dependent VariableDependent Variable– Choice of Self or Partner’s Performance during 10 Choice of Self or Partner’s Performance during 10

Randomly Chosen PAST Trials to Determine RewardRandomly Chosen PAST Trials to Determine Reward

Study 3 ProcedureStudy 3 Procedure

• Two Hundred TrialsTwo Hundred Trials– View SlideView Slide– Make DecisionMake Decision– Indicate Confidence LevelIndicate Confidence Level– View Other’s DecisionView Other’s Decision

• View Feedback/InstructionsView Feedback/Instructions• Decide Reward BasisDecide Reward Basis


……Ten trials will be randomly selected from Ten trials will be randomly selected from the preceding 200. The amount of money (in the preceding 200. The amount of money (in coupons) that you will obtain will entirely coupons) that you will obtain will entirely depend upon the outcome of those trials.…depend upon the outcome of those trials.…

The amount of money...will either be based The amount of money...will either be based on your performance or on the performanceon your performance or on the performanceof the prior student. It is your choice.of the prior student. It is your choice.


00

2020

4040

6060

8080

100100

FdbkFdbk No FdbkNo Fdbk FdbkFdbk No FdbkNo FdbkSuperiorSuperior EqualEqual

Human AidHuman AidAutomated AidAutomated Aid

PercentPercentChose Chose to Rely to Rely on Selfon Self

chancechance


00

2020

4040

6060

8080

100100

FdbkFdbk No FdbkNo Fdbk FdbkFdbk No FdbkNo FdbkSuperiorSuperior EqualEqual

PercentPercentChose Chose to Rely to Rely on Selfon Self

chancechance



• As in Study 2, a strong bias to rely on self was found As in Study 2, a strong bias to rely on self was found when cognitive and motivational processes were when cognitive and motivational processes were controlledcontrolled

• Evidence of a bias against automationEvidence of a bias against automation– In the feedback-superior condition: In the feedback-superior condition:

53% participants relied on a human aid53% participants relied on a human aid<6% participants relied on the automated aid<6% participants relied on the automated aid

Why did you choose to rely on aid?Why did you choose to rely on aid?Cognitive Processes: Logical ExplanationsCognitive Processes: Logical Explanations

00

22

44

66

88

1010

FdbkFdbk No FdbkNo Fdbk FdbkFdbk No FdbkNo Fdbk SuperiorSuperior Equal Equal

NumberNumberofof

LogicalLogicalStatementsStatements

ConditionConditionHuman AidHuman AidAutomated AidAutomated Aid

Why did you choose to rely on yourself?Why did you choose to rely on yourself?Social Processes: Assessing ReliabilitySocial Processes: Assessing Reliability

0

1

2

3

4

5

6

Fdbk No Fdbk Fdbk No Fdbk

SuperiorSuperior Equal Equal

ConditionCondition

NumberNumberofof

StatementsStatementsAssessing Assessing ReliabilityReliability


Why did you choose to rely on yourself?Why did you choose to rely on yourself?Social Processes: Relative TrustSocial Processes: Relative Trust

012345678



NumberNumberofof

CommentsCommentsConcerningConcerning

RelativeRelativeTrustTrust


Why did you choose to rely on yourself?Why did you choose to rely on yourself?Cognitive Processes: Aid Made a MistakeCognitive Processes: Aid Made a Mistake

0

2

4

6

8

10

12



NumberNumberofof

CommentsCommentsConcerningConcerning

Errors Errors Made byMade by

AidAid



• Consistent with the model, people reported Consistent with the model, people reported assessing the reliability of the aid, their own ability, assessing the reliability of the aid, their own ability, and the difference between the twoand the difference between the two

• People remembered mistakes made by automated People remembered mistakes made by automated aids more than they remembered mistakes made by aids more than they remembered mistakes made by human aids and used this information to affect their human aids and used this information to affect their decision to rely on the aiddecision to rely on the aid


• Bottom LineBottom Line• People were more likely to appropriately rely on a People were more likely to appropriately rely on a

human aid than on an automated decision aidhuman aid than on an automated decision aid• They were more likely to justify their disuse of the They were more likely to justify their disuse of the

automated aid than the human aid by describing automated aid than the human aid by describing instances in which they knew the aid made an errorinstances in which they knew the aid made an error

• QuestionQuestion• How well do participants expect the automated How well do participants expect the automated

decision aid to perform?decision aid to perform?


• Told They Will Perform Two Hundred TrialsTold They Will Perform Two Hundred Trials– View SlideView Slide– View Other’s DecisionView Other’s Decision– Make DecisionMake Decision– Indicate Confidence LevelIndicate Confidence Level

• Perform Four Practice SlidesPerform Four Practice Slides


• Independent VariableIndependent Variable– Type of AidType of Aid

– Prior ParticipantPrior Participant– Contrast DetectorContrast Detector

• Dependent VariableDependent Variable– Estimated Number of Errors in Upcoming 200 TrialsEstimated Number of Errors in Upcoming 200 Trials


0

10

20

30

40

Automated Aid Human Aid

EstimatedErrors

Type of PartnerType of PartnerAidSelf

How many errors do you think the contrast How many errors do you think the contrast detector/prior participant will make? (blank line)detector/prior participant will make? (blank line)


• Bottom LineBottom Line• Participants expect automated aids to perform Participants expect automated aids to perform

better than human aidsbetter than human aids

• QuestionQuestion• What happens during the interaction with the What happens during the interaction with the

automated system that turns the bias toward automated system that turns the bias toward automation into a bias against automation?automation into a bias against automation?

Perhaps...

Expectations

Inconsistent

Very well rememberedSmith & Graesser (1981)

Inaccurate Relative Perception


• Independent VariablesIndependent Variables– Framing of Past PerformanceFraming of Past Performance

– Positively (“Aid makes 1/2 as many errors as most students”)Positively (“Aid makes 1/2 as many errors as most students”)– Negatively (“Aid makes 10 errors”)Negatively (“Aid makes 10 errors”)– Unspecified (“Aid is not perfect”)Unspecified (“Aid is not perfect”)

– Type of PartnerType of Partner




• Two Hundred TrialsTwo Hundred Trials– View SlideView Slide

– Make DecisionMake Decision

– Indicate Confidence LevelIndicate Confidence Level

– View Other’s DecisionView Other’s Decision

• View FeedbackView Feedback

• Decide Reward BasisDecide Reward Basis


0

20

40

60

80

100

Negatively Positively Unspecified

Human Aid Automated Aid

Self-

Rel

ianc

eSe

lf-R

elia

nce


• Consistent with past research, disuse was commonConsistent with past research, disuse was common

• Consistent with “Perfect Automation Schema” Consistent with “Perfect Automation Schema” hypothesis, those who expected a less perfect hypothesis, those who expected a less perfect performance from their automated decision aid performance from their automated decision aid were more likely to rely on the aidwere more likely to rely on the aid


• Bottom LineBottom Line• People are more likely to appropriately rely on an People are more likely to appropriately rely on an

automated decision aid if the aid’s errors are salient…automated decision aid if the aid’s errors are salient…• Reducing people’s initial trust in an automated Reducing people’s initial trust in an automated

decision aid may lead to more appropriate reliancedecision aid may lead to more appropriate reliance

• QuestionQuestion• What other types of intervention will lead to more What other types of intervention will lead to more

appropriate automation reliance?appropriate automation reliance?


• Scenario TrainingScenario Training





• View Feedback (5 seconds)View Feedback (5 seconds)


Scenario TrainingScenario Training

• A stockbroker was giving investment advice. Research showed an average gain of 7% per year, when the broker made suggestions based on personal study. In contrast, recommendations made from a computer program yielded an annual average gain of 4%.

• Is the better investment strategy to rely on recommendations based on the stockbroker’s own study or on the computer program?


0102030405060708090

100


Scenario No Scenario

Sel

f-R

elia

nce

Superior InferiorSuperior Inferior


• Bottom LineBottom Line• People are more likely to appropriately rely on a People are more likely to appropriately rely on a

superior automated decision aid if they are given superior automated decision aid if they are given some training about probability.some training about probability.

• QuestionQuestion• What other types of intervention will lead to more What other types of intervention will lead to more

appropriate automation reliance?appropriate automation reliance?• Will feedback reduce disuse of automation? Will feedback reduce disuse of automation?


• Independent VariablesIndependent Variables– Cumulative Results FeedbackCumulative Results Feedback– Individual Trial FeedbackIndividual Trial Feedback– Prior Results FeedbackPrior Results Feedback








– View Feedback (some participants)View Feedback (some participants)

• View Feedback View Feedback (some participants)(some participants)

• Prior Results Feedback Prior Results Feedback (some participants)(some participants)



0

20

40

60

80

100

0 1 2 3

Self

- Rel

ianc

eSe

lf - R

elia

nce

Number of Sources of FeedbackNumber of Sources of Feedback


• Bottom LineBottom Line• Providing participants with various types of Providing participants with various types of

information concerning their and their information concerning their and their automated aid’s performance reduces the automated aid’s performance reduces the bias toward disusebias toward disuse

• QuestionQuestion What other techniques will reduce disuse of What other techniques will reduce disuse of

automated aids?automated aids?


• Independent VariablesIndependent Variables– Provision of Aid’s DecisionProvision of Aid’s Decision– Aid’s Performance LevelAid’s Performance Level– Type of FeedbackType of Feedback

• ContinuousContinuous

0

5

10

15

20

25

30

35

# of Errors Made

Contrast Detector

YOU

Continuous FeedbackContinuous Feedback

Remains on screen Remains on screen during all 200 trialsduring all 200 trials

Increases accuracyIncreases accuracyof perceived utilityof perceived utility

Updated every 5Updated every 5trialstrials

EliminatesEliminatesdetection of detection of

obvious errorsobvious errors


• Independent VariablesIndependent Variables– Provision of Aid’s DecisionProvision of Aid’s Decision– Aid’s Performance LevelAid’s Performance Level– Type of FeedbackType of Feedback

• ContinuousContinuous• CumulativeCumulative• NoneNone

• Dependent VariableDependent Variable– Choice of Self or Partner’s Performance during 10 Randomly Choice of Self or Partner’s Performance during 10 Randomly

Chosen PAST Trials to Determine RewardChosen PAST Trials to Determine Reward

Study 8 Performance DataStudy 8 Performance Data

0

20

40

60

80

SuperiorSuperior

No Feedback No Decision Continuous Feedback Decision

No Feedback Decision Cumulative Feedback Decision

Cumulative Feedback No Decision Continuous Feedback No Decision

Dis

use

Providing Continuous Feedback andProviding Continuous Feedback andPreventing Participants from Viewing Obvious Preventing Participants from Viewing Obvious Errors Made by the Automated Aid Reduced Errors Made by the Automated Aid Reduced

Disuse without Increasing MisuseDisuse without Increasing Misuse

0

20

40

60

80

100

SuperiorSuperior InferiorInferior

% o

f th

ose

wh

o

cho

seto

rel

y o

n s

elf

Study 8 ResultsStudy 8 ResultsJustificationsJustifications

• 85% of all justifications fell into one of four categories85% of all justifications fell into one of four categories• Trust in computersTrust in computers

– I don’t trust computers that much. I know a lot about their I don’t trust computers that much. I know a lot about their tendency for errorstendency for errors

• Detection of obvious errorsDetection of obvious errors– There were a few times that I’m pretty sure I saw the soldier, There were a few times that I’m pretty sure I saw the soldier,

but the program said he was absentbut the program said he was absent• Confidence in selfConfidence in self

– I was not confident in what I sawI was not confident in what I saw• Relative performanceRelative performance

– The computer made more mistakes compared to mineThe computer made more mistakes compared to mine


• Questions asked to Decision PresentQuestions asked to Decision Present– How much did you trust the decisions made by the How much did you trust the decisions made by the

contrast detector (very little – a great amount)contrast detector (very little – a great amount)• Overall, below the midpointOverall, below the midpoint• Superior aids were rated more trustworthy than inferior Superior aids were rated more trustworthy than inferior

aidsaids– To what extent did you lose trust in the contrast To what extent did you lose trust in the contrast

detector when you noticed it made an error? (very detector when you noticed it made an error? (very little – a great amount)little – a great amount)

• Overall, above the midpointOverall, above the midpoint


• Questions asked to Decision PresentQuestions asked to Decision Present– Compared with those who chose to rely on self, Compared with those who chose to rely on self,

those who chose to rely on the automated decision those who chose to rely on the automated decision aidaid

– Trusted the aid moreTrusted the aid more– Noticed the aid made fewer errorsNoticed the aid made fewer errors– Indicated they lost LESS trust in the aid when they Indicated they lost LESS trust in the aid when they

noticed an error the aid madenoticed an error the aid made

Study 8 ImplicationsStudy 8 Implications • Bottom LineBottom Line• Providing participants with continuous feedback and Providing participants with continuous feedback and

eliminating the possible detection of obvious errors drastically eliminating the possible detection of obvious errors drastically reduces disuse without causing misusereduces disuse without causing misuse

• More theoretical than appliedMore theoretical than applied• Trust may play a role (Lee & Moray’s (1992) performance, Trust may play a role (Lee & Moray’s (1992) performance,

process, and purpose)process, and purpose)

• QuestionQuestion• Will providing a reason as to why the aid might err lead Will providing a reason as to why the aid might err lead

to more appropriate reliance? to more appropriate reliance?


• Independent VariableIndependent Variable– ReasonReason– RationaleRationale



I need to inform you that this machine is I need to inform you that this machine is NOTNOT perfect. The contrast detector will indicate the perfect. The contrast detector will indicate the soldier is present if it detects a form that humans soldier is present if it detects a form that humans often take. often take. Since non-humans (e.g. shading Since non-humans (e.g. shading from a tree) can sometimes take human-like from a tree) can sometimes take human-like forms, mistakes can be made.forms, mistakes can be made. Therefore, if your Therefore, if your decision and the decision reached by the program decision and the decision reached by the program are not the same, do NOT assume that your decision are not the same, do NOT assume that your decision is incorrect.is incorrect.

ReasonReason






• View Feedback that Aid Is SuperiorView Feedback that Aid Is Superior


Study 9 Results Study 9 Results

0

5

10

15

20

25

30

35

40

45

No Reason Reason No Reason Reason

Dis

use

No Rationale RationaleNo Rationale Rationale


• Bottom LineBottom Line• In this artificial paradigm (in which the automation bias is In this artificial paradigm (in which the automation bias is

eliminated and the expended effort is controlled), neither eliminated and the expended effort is controlled), neither providing participants with a reason (for false alarms) nor a providing participants with a reason (for false alarms) nor a rationale for why an aid might err reduced disuserationale for why an aid might err reduced disuse

• QuestionQuestion• Would providing a reason as to why the aid might err lead to Would providing a reason as to why the aid might err lead to

more appropriate reliance in a more realistic paradigm (i.e., more appropriate reliance in a more realistic paradigm (i.e., one in which the automation bias is not eliminated and effort one in which the automation bias is not eliminated and effort is not controlled)? is not controlled)?


• Independent Variables Independent Variables – Aid’s Relative PerformanceAid’s Relative Performance

– Provision of a Reason for Aid’s ErrorsProvision of a Reason for Aid’s Errors • Dependent VariablesDependent Variables

– Reliance on Aid:Reliance on Aid:– Transformed p(incorrect | incorrect)Transformed p(incorrect | incorrect)– Transformed p(incorrect | correct)Transformed p(incorrect | correct)






• View FeedbackView Feedback– Inferior AidInferior Aid

– Superior AidSuperior Aid

• One Hundred TrialsOne Hundred Trials– View SlideView Slide– View Other’s DecisionView Other’s Decision– Make DecisionMake Decision– Indicate Confidence LevelIndicate Confidence Level

• Complete QuestionnaireComplete Questionnaire


00.20.40.60.8

11.21.41.6

No Reason Reason No Reason Reason

Aid Incorrect

Aid Correct

tran

sfor

med

tran

sfor

med

p(e

rror

)p(

erro

r)

Target Absent Target PresentTarget Absent Target Present

Study 10 ResultsStudy 10 ResultsTrust in Aid is Greater for Trust in Aid is Greater for

those Working with a Superior Aidthose Working with a Superior Aid

1

2

3

4

5

6

7

Superior Inferior

Self-

Rep

orte

d Tr

ust

Self-

Rep

orte

d Tr

ust

Study 10 ResultsStudy 10 ResultsTrust in Aid is Greater for Trust in Aid is Greater for

Those Provided with a ReasonThose Provided with a Reason

1

2

3

4

5

6

7

No Reason Reason

Self-

Rep

orte

d Tr

ust

Self-

Rep

orte

d Tr

ust

Three Conditions to Show Trust Mediates the Three Conditions to Show Trust Mediates the Relationship between Reason and RelianceRelationship between Reason and Reliance

Baron and Kenny (1986)Baron and Kenny (1986)

– Effect of reason on automation reliance is significantEffect of reason on automation reliance is significant– b1 = .55; t(1) = 2.39, p < .03 b1 = .55; t(1) = 2.39, p < .03

– Effect of reason on trust is significantEffect of reason on trust is significant– b1 = 1.30; t(1) = 2.12, p < .05b1 = 1.30; t(1) = 2.12, p < .05

– When controlling for trust, the effect of reason on When controlling for trust, the effect of reason on automation reliance is reducedautomation reliance is reduced– b1 = .52; t(1) = 1.81, p > .05b1 = .52; t(1) = 1.81, p > .05

Study 10 ImplicationsStudy 10 Implications • Bottom LineBottom Line• People were more likely to rely on a superior automated decision People were more likely to rely on a superior automated decision

aid if they were provided with a reason why it might err aid if they were provided with a reason why it might err • The relationship between reason and reliance was mediated by The relationship between reason and reliance was mediated by

trust trust – Those provided with a reason why the aid might err trusted the aid moreThose provided with a reason why the aid might err trusted the aid more– This increased trust led them to rely on the aid more oftenThis increased trust led them to rely on the aid more often

• QuestionQuestion• Will training in the conditions in which the aid is reliable and Will training in the conditions in which the aid is reliable and

unreliable lead to appropriate reliance?unreliable lead to appropriate reliance?

Individual CombatIdentification System

ICIDS

Battlefield Combat Identification System

BCIS

If the target returns a signal, soldiers will

receive the message: “FRIENDLYFRIENDLY”

Combat Identification SystemsCombat Identification Systems

If the target does not return a signal, soldiers

will receive the message:

“UNKNOWNUNKNOWN”

Absent 50 times

100%

50%

50%

Contrast Detector

Continue

ABSENT

Contrast Detector

Continue

PRESENT

Aid Says

Study 11 TrainingStudy 11 Training

Reality

Present 50 times

Absent

Present

Reality

Contrast Detector

Continue

PRESENT

Contrast Detector

Continue

ABSENT

Aid Said

75 times

25 times

100%

66.6%

33.3%

25 times

50 times

25 times

Study 11 TrainingStudy 11 Training

05

101520253035404550

Aid Correctly says ABSENT

Aid Incorrectlysays PRESENT

Aid Correctlysays PRESENT

Alone No Training Training

% o

f R

eloo

ksStudy 11 ResultsStudy 11 Results

0

0.1

0.2

0.3

0.4

0.5

TargetPresent

(MISUSE)

TargetAbsent

(DISUSE)

TargetAbsent

(MISUSE)

TargetPresent

(DISUSE)

Alone No Training Training

Aid CorrectlySays ABSENT

Aid SaysPRESENT

Tra

ns

form

ed p

(err

or)



• Bottom LineBottom Line• Those provided with the automated decision aid Those provided with the automated decision aid

performed WORSE than those without itperformed WORSE than those without it• Misuse exceeded disuse--even when the reliability of Misuse exceeded disuse--even when the reliability of

the automated system was very low!the automated system was very low!• Training of the conditions in which the aid would be Training of the conditions in which the aid would be

reliable did little to improve appropriate reliance on reliable did little to improve appropriate reliance on the simple automated decision aidthe simple automated decision aid

• Question: What if you gave them a perfect aid?Question: What if you gave them a perfect aid?


• Independent VariableIndependent Variable– Provision of a Perfect AidProvision of a Perfect Aid– No AidNo Aid

• For 200 slidesFor 200 slides– View SlideView Slide– Make DecisionMake Decision– View Aid’s Decision (if View Aid’s Decision (if

Perfect Aid Condition)Perfect Aid Condition)

• Key 100 trialsKey 100 trials– View SlideView Slide– View Aid’s Decision (if View Aid’s Decision (if

Perfect Condition)Perfect Condition)– View Slide Again (if want)View Slide Again (if want)– Make DecisionMake Decision– Indicate ConfidenceIndicate Confidence

0

20

40

60

80

100

Number of Relooks

Alone Perfect Aid

Nu

mb

er

of

Re

loo

ks


0

0.1

0.2

0.3

0.4

0.5

Target Absent (DISUSE) Target Present (DISUSE)

Alone Perfect Aid

Aid Correctly Says ABSENT

Aid Correctly SaysPRESENT

Tra

ns

form

ed p

(err

or)



• Bottom LineBottom Line• People did not always rely on the perfect People did not always rely on the perfect

automated aidautomated aid• Perhaps the need to control led people to ignore Perhaps the need to control led people to ignore

some of the perfect aid’s decisionssome of the perfect aid’s decisions• QuestionQuestion• What would happen if you gave people an aid What would happen if you gave people an aid

that was certain to be WRONG?that was certain to be WRONG?


• Automated is ALWAYS Automated is ALWAYS wrongwrong

• For 200 slidesFor 200 slides– View SlideView Slide– Make DecisionMake Decision– View Aid’s DecisionView Aid’s Decision

• Key 100 trialsKey 100 trials– View SlideView Slide– View Aid’s DecisionView Aid’s Decision– View Slide Again (if want)View Slide Again (if want)– Make DecisionMake Decision– Indicate ConfidenceIndicate Confidence

0

20

40

60

80

100

Aid Always Wrong Aid Always Right No Aid

Nu

mb

er

of

Re

loo

ks

Study 13 Results (with Study 12)Study 13 Results (with Study 12)

Study 13Study 13 Study 12Study 12

0

0.1

0.2

0.3

0.4

0.5

Target Absent (DISUSE) Target Present (DISUSE)Aid INCorrectly

Says PRESENT

Aid INCorrectly Says

ABSENT

Tra

ns

form

ed p

(err

or)


Study 13 ImplicationsStudy 13 Implications • Bottom LineBottom Line• People did not always ignore an automated aid People did not always ignore an automated aid

known to be wrong on every trialknown to be wrong on every trial• Therefore, the need for control is not the cause of the Therefore, the need for control is not the cause of the

inappropriate resistance of the perfect aidinappropriate resistance of the perfect aid• Perhaps the experience with the aid led participants to Perhaps the experience with the aid led participants to

distrust the aiddistrust the aid• Question: What would happen if you gave people a perfect Question: What would happen if you gave people a perfect

automated aid but did not give them much experience with automated aid but did not give them much experience with the aid?the aid?


• All participants are told their automated All participants are told their automated aid is perfectaid is perfect

• Key 100 trialsKey 100 trials– View SlideView Slide– View Aid’s Decision View Aid’s Decision – View Slide Again (if want)View Slide Again (if want)– Make DecisionMake Decision– Indicate ConfidenceIndicate Confidence

0

20

40

60

80

100

Wrong AidExperience

Correct AidExperience

Correct AidNo

Experience

No Aid

Nu

mb

er

of

Re

loo

ks

Study 14 Results (with Studies 12&13)Study 14 Results (with Studies 12&13)

Study 13 Study 12 Study 14 Study 12Study 13 Study 12 Study 14 Study 12

0

0.1

0.2

0.3

0.4

0.5

Target Absent (DISUSE) Target Present (DISUSE)Aid Correctly Says

ABSENTAid Correctly Says

PRESENT

Tra

ns

form

ed p

(err

or)


Study 14 ImplicationsStudy 14 Implications • Bottom LineBottom Line• People did not always rely on a perfect People did not always rely on a perfect

automated aid—even without experience with automated aid—even without experience with the aidthe aid

• Removing experience with the perfect aid Removing experience with the perfect aid slightly decreased the number of re-looks, but it slightly decreased the number of re-looks, but it also decreased reliance on the perfect aidalso decreased reliance on the perfect aid

• Question: What if the automated aid was presented as Question: What if the automated aid was presented as an expert?an expert?


• Independent VariablesIndependent Variables– Presumed Expertise of the Aid (between)Presumed Expertise of the Aid (between)– Type of Aid (within)Type of Aid (within)

• HumanHuman• AutomatedAutomated

• Dependent VariablesDependent Variables– Transformed p(error |aid incorrect) = MISUSETransformed p(error |aid incorrect) = MISUSE– Transformed p(error|aid correct) = DISUSETransformed p(error|aid correct) = DISUSE


• 100 trials100 trials– View SlideView Slide– ChooseChoose

• View Automated Aid’s DecisionView Automated Aid’s Decision• View Human Aid’s DecisionView Human Aid’s Decision• Make DecisionMake Decision

– View Aid’s Decision (if requested) View Aid’s Decision (if requested) – Make DecisionMake Decision– Indicate ConfidenceIndicate Confidence

AnalysesAnalyses• 2 (Expert vs Not) x 3 (Aid: None vs Human vs

Auto) ANOVAs were performed for errors made when target was present and absent.

• When target was absent– Main effect for Aid: Participants made more errors if

they did not ask for aid’s help than if they did

• When target was present:– Main effect for Aid: Participants made more errors if

they did not ask for the aid’s help than if they did– Main effect for Expertise of Aid: Participants with

expert aids made fewer errors

0

0.2

0.4

0.6

0.8

1

No Aid Human Aid Automated Aid

p(error|aid error) p(error|aid correct)



• Examination of the number of trials participants asked to view the decision made by the automated aid, human aid, or no aid revealed some individual differences– 16 participants did not view an aid’s decision for

over half of the trials; 4 of these NEVER viewed an aid’s decision

– 6 participants viewed the human aid’s decision for over half the trials

– Only one participant viewed the automated aid’s decision for over half the trials

Study 15Study 15

Please describe the usefulness of the Please describe the usefulness of the automated/human aid. Pretend you automated/human aid. Pretend you are writing to the next participant are writing to the next participant coming into this experiment. Give the coming into this experiment. Give the next person some advice concerning next person some advice concerning this automated/human aid.this automated/human aid.

ResultsResults

• Can linguistic analysis identify the people who are likely to ignore an aid or prefer a human aid over an automated aid?

Category Category (Examples) (Examples)• pronouns pronouns (I, me, we, you) (I, me, we, you) • positive emotions positive emotions (happy, pride, good)(happy, pride, good)• negative emotions negative emotions (hate, afraid, sad)(hate, afraid, sad)• insight insight (think, know, consider)(think, know, consider)• time time (past, present, future)(past, present, future)• communication communication (talk, share, converse)(talk, share, converse)• anxiety words anxiety words (nervous, afraid, tense)(nervous, afraid, tense)

LIWC analyzes text word-by-word and categorizes LIWC analyzes text word-by-word and categorizes the text into 74 different linguistic dimensionsthe text into 74 different linguistic dimensions

LIWC VariablesLIWC Variables

0

2

4

6

8

10

12

Article Use Inclusive Words

Human Automated

People Used Less Complex Language People Used Less Complex Language When Writing about an Automated Aid When Writing about an Automated Aid

than a Human Aidthan a Human Aid

0

2

4

6

8

10

12

Causal Words Social Words

Human Automated

Operators Used More Cognitive and Social Operators Used More Cognitive and Social Processes When Writing about the Human Processes When Writing about the Human

than Automated Aidthan Automated Aid

0

0.5

1

1.5

2

2.5

3

Second Person Pronouns References to Other People

Human Automated

Pronoun Use Varied with Type of AidPronoun Use Varied with Type of Aid

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Anxiety Words Anger

Human Automated

Negative Emotions Were Expressed More Negative Emotions Were Expressed More When Discussing the Automated than When Discussing the Automated than

Human AidHuman Aid

The more participants followed the human aid into The more participants followed the human aid into an error (i.e., misused the human aid):an error (i.e., misused the human aid):

• The more they described the automated aid with– Discrepancy words

– Future tense

• The more they described the human aid with – Negate words

– Fewer cognitive mechanism words

• The bigger the difference between auto and human aid descriptions for– Negate words

– Discrepancy words

The more participants followed the non-expert The more participants followed the non-expert human aid into an error (i.e., misused the human human aid into an error (i.e., misused the human

aid):aid):• The more they described the automated aid with– Discrepancy words

• The more they described the human aid with – Communication words– Negate words– Fewer affect words

• The bigger the difference between auto and human aid descriptions for– Discrepancy words – Negate words – First person pronouns (human > auto)– First person singular pronouns (human > auto)

Study 15 ImplicationsStudy 15 Implications • Bottom LineBottom Line• People show a preference for self-reliancePeople show a preference for self-reliance• People were more likely to view decisions People were more likely to view decisions

of human rather than automated aidsof human rather than automated aids• People were equally likely to rely on human People were equally likely to rely on human

and automated aidsand automated aids• Linguistic analysis was able to detect Linguistic analysis was able to detect

differences in people’s conceptions of differences in people’s conceptions of human and automated aidshuman and automated aids

ConclusionsConclusions

–Supplying automated systems will not always Supplying automated systems will not always improve performanceimprove performance

–Predicting automation usage decisions is Predicting automation usage decisions is complex even with a simple taskcomplex even with a simple task

Future ResearchFuture Research

– Replicating the results in a combat-like environmentReplicating the results in a combat-like environment

– Examining the effect of trainingExamining the effect of training

– Further testing of the modelFurther testing of the model– Determine Effect SizesDetermine Effect Sizes– Examine PathwaysExamine Pathways

– Applying results to system developmentApplying results to system development

What We Know for System DevelopmentWhat We Know for System Development• Sometimes, people provided with an automated decision aid Sometimes, people provided with an automated decision aid

performed WORSE than those without itperformed WORSE than those without it• Improving the reliability of the automated decision aid did NOT Improving the reliability of the automated decision aid did NOT

improve the performance of the human-automation teamimprove the performance of the human-automation team• Training in conditions in which the aid will and will not be reliable Training in conditions in which the aid will and will not be reliable

was not effective in promoting appropriate automation usewas not effective in promoting appropriate automation use• However, information as to why the automated aid might err led However, information as to why the automated aid might err led

people to trust the aid more and appropriately rely on the aidpeople to trust the aid more and appropriately rely on the aid

In addition… these techniques may lead to more In addition… these techniques may lead to more appropriate automation relianceappropriate automation reliance

• Reduce people’s initial trust in the automated aidReduce people’s initial trust in the automated aid• Provide participants with various types of Provide participants with various types of

information concerning their and their automated information concerning their and their automated aid’s performanceaid’s performance

• Provide participants with continuous feedback and Provide participants with continuous feedback and eliminate the possibility of the detection of obvious eliminate the possibility of the detection of obvious errorserrors

Questions??Questions??

reliance on decision aids results from 15 studies using a soldier- detection task april 12, 2006

Documents