automated assessment for adaptive learning of complex tasks alan koenig markus iseli allen munro cct
TRANSCRIPT
![Page 1: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/1.jpg)
Automated Assessment for Adaptive Learning of Complex Tasks
Alan Koenig
Markus Iseli
Allen Munro
CCT
![Page 2: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/2.jpg)
Characteristics of Cognitively Complex Tasks
Often require multiple, non-trivial steps to complete
Performance can be highly variable
Often include interdependent task features
Often well suited to representation in games & simulations
– Multiple scenarios
– Low cost
Examples:
Tactical Planning Simulation Damage Control Simulation
– Low risk
– But… assessment can be difficult
![Page 3: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/3.jpg)
Anti-Submarine Warfare
Anti-submarine warfare (ASW) is a branch of naval warfare that uses surface warships, aircraft, or other submarines to find, track and deter, damage, or destroy enemy submarines.
– Carried out by teams of people within and across ships
– Requires synthesis of large quantities of information in short periods of time
– Heavy emphasis on situation awareness and decision making skills (especially among TAO’s)
![Page 4: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/4.jpg)
Assessment Methodology Overview
![Page 5: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/5.jpg)
ASW Ontology Construction
ExpertsRules /Doctrine
Procedures /Processes
Facts /Observations
DOMAIN
![Page 6: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/6.jpg)
Ontology Excerpt
![Page 7: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/7.jpg)
Bayesian Network Development
OntologyNodes
Directionalityof Links
Observables /SME Input
Strength of Relationships
Characterizing the nature of relationships is essential
Observable Actions
Latent Variables
SonarPlanning
PassiveSonar Planning
ActiveSonar
Planning
UseShipPassiveSonar DropPassiveSonobuoys
DropActiveSonobuoys UseDippingSonar UseShipActiveSonar
![Page 8: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/8.jpg)
ASW Skilled
Torpedo CounterMeasures
MEUProtectionSub Hunting
Plan for Opposition
Intell on Opposition
Opposition Estimation
ShowMissionInfo ShowBriefing
EstLLSubSpeed
EstLLTorpedoRange
0.5
SonarPlanning
TDA AvoidManeuvers
PassiveSonar Use
ActiveSonar Skill
AlterCourseTo AvoidFast
AlterCourseTo AvoidSlow
UseShipPassiveSonarDropPassiveSonobuoy
s
DropActiveSonobuoys UseDippingSonar UseShipActiveSonar
TDA Avoidance
DontEverEnterTDALimLines
Estimation
EstDatumSubSpeed EstDatumTorpedoRange
Datum Estimation
TorpedoDeception
TorpedoEscape
NixieDropped MICDropped
TorpedoEvasion
TorpedoCounter
Bayesian Network for TDA Avoidance
![Page 9: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/9.jpg)
ASW Skilled
Torpedo CounterMeasures
MEUProtectionSub Hunting
Plan for Opposition
Intell on Opposition
Opposition Estimation
0.5
TDA AvoidManeuvers
DropActiveSonobuoys UseDippingSonar UseShipActiveSonar
TDA Avoidance
LimLines Estimation
Datum Estimation
TorpedoDeception
TorpedoEscape
Bayesian Network for TDA Avoidance
ShowMissionInfo ShowBriefing
EstLLSubSpeed
EstLLTorpedoRange
SonarPlanning
PassiveSonar Use
ActiveSonar Skill
AlterCourseTo AvoidFast
AlterCourseTo AvoidSlow
UseShipPassiveSonarDropPassiveSonobuoy
s
DontEverEnterTDA
EstDatumSubSpeed EstDatumTorpedoRange
NixieDropped MICDropped
TorpedoEvasion
TorpedoCounter
![Page 10: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/10.jpg)
Conditions for TDA Avoidance to be considered by Bayesian network
![Page 11: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/11.jpg)
Observable Actions
Latent Variables
The Advantage of Bayesian Networks
SonarPlanning
PassiveSonar Planning
ActiveSonar
Planning
UseShipPassiveSonar DropPassiveSonobuoys
DropActiveSonobuoys UseDippingSonar UseShipActiveSonar
Probability of mastery of the latent variables is inferred from the scored observable actions
P(ASP): Probability of mastery of concept Active Sonar Planning
P(UDS | ASP): Conditional probability. Probability of mastery of concept UseDippingSonar, given information about mastery of concept Active Sonar Planning
Example: Determining probabilities
P(UDS):Probability of mastery of concept UseDippingSonar
P(ASP) x P(UDS | ASP)
P(UDS)P(ASP | UDS) =
![Page 12: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/12.jpg)
Real-Time Formative Assessment
•Provide performance feedback
•Provide practice / resources
•Add / change tasks
•Add / change affordances
FORMATIVE ASSESSMENT
Simulation(Sandbox)
ADAPTIVE OPTIONS
Observable Actions & Events
CAA(with Bayesian Network)
![Page 13: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/13.jpg)
![Page 14: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/14.jpg)
Check the Briefing—Good!
1st PROBLEM: Chinese Kilos Threat Problem
00:00:00 .k.Announce. ShowBriefing 0.95
The fact that we’re up against Kilos implies a silent sub speed of 6 Kt.and a torpedo range of 12,000 yds.
![Page 15: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/15.jpg)
Failed to Correct Estimates—Bad!
1st PROBLEM: Chinese Kilos Threat Problem
07:26:32 .k.Announce. EstSubSpeed 0.207:26:32 .k.Announce. EstTorpedoRange 0.2
The estimated sub speed was left at 3 Kt. So the limit lines were not accurate, relative to the briefing.
Scored at end, because these values were never changed.
![Page 16: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/16.jpg)
Failed to Correct Estimates—Bad!
1st PROBLEM: Chinese Kilos Threat Problem
07:26:32 .k.Announce. DontEverEnterTDA 0.1
Uh-oh! The MEU entered the TDZ.
![Page 17: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/17.jpg)
17
Problem 1
![Page 18: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/18.jpg)
18
Problem 1
![Page 19: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/19.jpg)
19
Problem 1
![Page 20: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/20.jpg)
20
Problem 1
![Page 21: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/21.jpg)
21
Problem 1
![Page 22: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/22.jpg)
Check the Briefing—Good!
2nd PROBLEM: Threat West of Gibraltar
This time the mission brief gives specific estimates of hostile sub speed and torpedo range.
00:00:00 .k.Announce. ShowBriefing 0.95
![Page 23: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/23.jpg)
Datum Estimates2nd PROBLEM: Threat West of Gibraltar
The datum estimates forsub speed and torpedorange don’t match the briefing.
![Page 24: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/24.jpg)
Datum Estimates Corrected—Good!
2nd PROBLEM: Threat West of Gibraltar
Set to the correct values.
01:14:03 .k.Announce. EstSubSpeed 0.9501:14:03 .k.Announce. EstTorpedoRange 0.95 Scored at end.
![Page 25: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/25.jpg)
Check Limit Lines2nd PROBLEM: Threat West of Gibraltar
Perhaps not a wide enough berth…
![Page 26: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/26.jpg)
Plot a New Course2nd PROBLEM: Threat West of Gibraltar
A safer course?
![Page 27: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/27.jpg)
Check Limit Lines Again2nd PROBLEM: Threat West of Gibraltar
Looks safer
![Page 28: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/28.jpg)
Successful Avoidance2nd PROBLEM: Threat West of Gibraltar
Success!
01:14:03 .k.Announce. DontEverEnterTDA 0.95
![Page 29: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/29.jpg)
29
Problem 2
![Page 30: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/30.jpg)
30
Problem 2
![Page 31: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/31.jpg)
31
Problem 2
![Page 32: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/32.jpg)
32
Problem 2
![Page 33: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/33.jpg)
33
1 2 3 4 50
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Problem 1Problem 2
1 2 3 4 50
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Problem 1Problem 2
Plan for Opposition
TDA Avoidance
Sub speed: Wrong estimate
Entered TDA
Prob
abili
ty o
f Mas
tery
Prob
abili
ty o
f Mas
tery
time
time
![Page 34: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/34.jpg)
ASW Skilled
Torpedo CounterM.
MEUProtectionSub Hunting
Plan for Opposition
Intelligence on Opposition
Opposition Estimation
GetMissionInfo GetBriefing EstSubSpeed EstTorpedoRange
SonarPlanning
TDA AvoidManeuvers
SenseQuietly
SenseLoudly
ManeuverFastManeuverSlow
ShipsPassivSonar PassiveSonobuoys ActiveSonobuoys DippingSonar ShipsActivSonar
TDA Avoidance
B20: score 0.95D20: score 0.6F20: score 0.1
B16: score 0.95D16: score 0.7F16: score 0.05 B5: score 0.9
D5: score 0.7B6: score 0.95D6: score 0.7
B11: score 0.95D11: score 0.8F11: score 0.2
B12: score 0.95D12: score 0.8F12: score 0.2
B36: score 0.9E36: score 0.3F36: score 0.2
B40: score 0.95C40: score 0.4E40: score 0.3
C44: score 0.9D44: score 0.6F44: score 0.05
B28: score 0.95D28: score 0.6
B32: score 0.95E32: score 0.3
DontEnterTDAIfPos
B24: score 0.95C24: score 0.9F24: score 0.1
TDA avoidanceBayesian network sub-net
![Page 35: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/35.jpg)
Initial Evaluation Feedback
CAA Debugging OutputInstructional Feedback High Level Assessments
![Page 36: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/36.jpg)
First Successful Problem
![Page 37: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/37.jpg)
Into the Next Problem
![Page 38: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/38.jpg)
Adaptive Problem Selection
A new problem is advised.
![Page 39: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/39.jpg)
Loading the Problem
The advised problem is selected
![Page 40: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/40.jpg)
Initial Actions in the New Problem
![Page 41: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/41.jpg)
Preparing to Find the Sub
Launching a helicopter
![Page 42: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/42.jpg)
Detecting the Sub
Sonobuoys detect sub
![Page 43: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/43.jpg)
Recording of Performances & Assessments
Simulation(Sandbox)
Log Files(Actions and Events)
These performance recordscan be replayed.
Report Generation(CAA Monitor)
Assessment History(Per Student)
Report
CAA
![Page 44: Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT](https://reader035.vdocuments.us/reader035/viewer/2022081516/56649cfd5503460f949cd6be/html5/thumbnails/44.jpg)
Questions / Comments?
CCT