bisantz and michael jenkins, university at buffalonagi/muri/muri/year_2_files/pdf presentati… ·...
TRANSCRIPT
Human Source Characterization & Human in the Fusion Loop
Ann Bisantz and Michael Jenkins, University at BuffaloTel. (716) 645‐4714, E‐Mail: [email protected]
Objectives• Expand on source characterization efforts to include additional categories
• Gain an understanding of the cognitive requirements, complexities and constraints of the intelligence analysis (IA) domain
• Identify key human‐fusion system interaction touch points for the MURI system
Scientific/Technical Approach• Literature based review of existing cognitive research in IA
• Adaptation of proven HF techniques to develop method for touch point identification
• Application of developed method to MURI fusion system
Accomplishments• Characterized 8 additional soft observation categories
• Created report characterizing the IA domain based on review of research literature
• Developed adaptable method to identify 6 recommended interaction touch points for the MURI fusion system
•Challenges• Software state to conduct H‐I‐L experiments• Lack of direct access to Intelligence Analysts or SMEs
Main Scientific/Technical AccomplishmentsHuman Source Characterization & Human in the Fusion Loop
Characterized 8 additional soft observation categories – Based on existing empirical literature– Categories selected based on availability of literature + relevance to COIN IA
Created extensive report (+150 p.) characterizing the IA domain – Based on review of research literature– Focus on developing an understanding of the cognitive requirements,
complexities and constraints of the intelligence analysis domain
Developed methodology identify fusion system design components which support human operators in the fusion loop
– Based on a range of human factors engineering techniques– Method designed to be adaptable across fusion systems and design life‐cycle
stages– Utilized method to identified 6 recommended human‐fusion system interaction
touch points for the MURI fusion system2 of 21
Personnel supported:• Ph.D. Student: Michael Jenkins• Faculty: Ann Bisantz
Publications:• Journal papers: 1 (submitted)• Conference papers: 2 (published/presented)• Book chapters: 1 (in process)
Project Statistics and SummaryHuman Source Characterization & Human in the Fusion Loop
3 of 21
M. Jenkins, G. Gross, A. Bisantz, and R. Nagi. “Toward context‐aware hard/soft information fusion: Incorporating situationally qualified human observations into a fusion process for intelligence analysis,” IEEE Conference on Cognitive Methods in Situation Awareness and Decision Support, 2011, 22‐24 February, Miami Beach, FL. (8 pp).
M. Jenkins, G. Gross, A. Bisantz, R. Nagi “Towards Context Aware Data Fusion: Modeling and Integration of Situationally Qualified Human Observations into a Fusion Process for Intelligence Analysis,” IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, (Submitted, May 2011), 12 p.
M. Jenkins, A. Bisantz “Identification of Human‐Interaction Touch Points for Intelligence Analysis Information Fusion Systems,” Fusion 2011, 2011.Chicago, IL, July 5 ‐ 8.
M. Jenkins, A. Bisantz, J. Pfautz, (in preparation). Human Engineering Factors in Distributed and Net Centric Fusion Systems. In Distributed data fusion for net‐centric operations. Boca Raton: CRC Press.
PublicationsHuman Source Characterization & Human in the Fusion Loop
4 of 21
Overall Goal:Develop context‐aware error models for the remaining observation categories identified in year 1
– Only categories with existing empirical literature available to extract contextual influences & error characteristics were completed
– Approach was the same as was utilized for characterizing 4 categories completed in year 1
Categories Included:1. Memory based traversed distance estimation 2. Visual gender classification3. Visual quantitative estimation of object dimensions4. Memory based quantitative estimation of duration5. Visual quantitative estimation of weight6. Auditory based voice recognition (does not imply positive identification)7. Visual based facial recognition (does not imply positive identification)
Technical ApproachHuman Source Characterization
5 of 21
Overall Goal:Develop study to validate the benefits of providing context‐aware soft sensor source characterization models for information fusion systems
– No literature exists validating the fusion improvements when leveraging soft sensor source characterization models
– Experiment focused on comparing fusion output of simulated truthedand observed data with/without the context‐aware models
– Results to be analyzed using Signal Detection Theory
Potential Study Variables:1. Number of entity attributes2. Percentage of population with known attributes3. Fusion “Hit” threshold4. Comparison Format (Truthed vs. Observed & Observed vs. Observed)
Technical ApproachHuman Source Characterization
6 of 21
Overall Goal:Develop generalizable and adaptable method to support the design of fusion systems which support the human operator in the fusion loop.
Overall Approach:1. Characterize IA domain in terms of factors such as constraints,
actors, resources, dynamics, workflows, actors, etc.2. Apply HF and other techniques to the design of a theoretical fusion
system, based on predicted MURI project output, to:• Support human operator in the fusion loop• Enhance overall human‐machine system capabilities• Increase likelihood of successful system integration to existing IA workflows
3. Expand/Augment methodology to be adaptable to design of different fusion systems and across multiple domains
Technical ApproachHuman in the Fusion Loop
7 of 21
Intelligence analysis (IA) is a broad domain that in its most basic form is reasoning over available information with the goal of making a coherent whole of the past, present, and/or future states of some real‐world environment or situation.
– IA is a complex task of filtering, validating, associating and summarizing massive amounts of information relevant to a real‐world scenario
– IA is unlike most other analytical processes in that making a decision and taking action based on that decision is not the immediate goal
– IA outputs are passed to often remote consumers who will take action (and potentially face severe consequences) based on the provided summary
– IA is often considered to be unbounded as scenarios being studied must be considered on a dynamic and comprehensive scale, “the world is literally its province”
Intelligence analysts are challenged with this task of illuminating the future to provide “actionable intelligence,” by:
– Compiling, reviewing, and processing as much information as they can • Based on externally imposed deadlines and availability of data
– Providing an interpretation of some, often ambiguous, substantive problem • Based on their domain experience and hypotheses
Technical ApproachHuman in the Fusion Loop
Characterizing the IA Domain
8 of 21
Common Workflow:Needed to understand typical analyst workflow in order to define how, when & where a fusion system will aid analysts
• IA process is an iterative cycle; however 7 stages are typically present•Greater # iterations = more robust analysis
•Number of iterations is dependent on availability:• Of Time• Of New or updated information
Technical ApproachHuman in the Fusion Loop
Characterizing the IA Domain
9 of 21
Creation / Presentation of IA Artifact
Additional data collection & hypothesis testing
Moment of hypothesis
Evaluation of Data
Collection of data
Analysis of substantive problem
Appearance of substantive problem
Intelligence Analysis Complexities: • Leveraging an understanding of the IA domain & typical
workflow, common complexities were identified that presented challenges to analyst performance.
• Complexities were mapped to the stages of analysis based on when they are likely to occur. Likely Stage(s) of analysis where
Challenges will ariseCommon Challenges to IA Performance 1 2 3 4 5 6 7Ambiguous Requests for Information (RFI) XCharacteristics of the Data X X XDistributed Structure of IA Domain X XLack of Data Needed for Analysis X X XLack of Consumer and/or Source Feedback X XMultiple ‘Worlds’ of Consideration X X X XPotential for Deception X XDynamics of the Real‐World X XUnbounded Problem Space X X
Technical ApproachHuman in the Fusion LoopIA Complexities Mapping
Leveraging identified IA complexities, potential fusion system capabilities were identified to mitigate contributing factors.– Capabilities that focus on alleviating common IA complexities
are more likely to be leveraged by analysts than capabilities which simply automate a task that the analyst can perform manually with little effort or difficulties.
Technical ApproachHuman in the Fusion Loop
Identification of System Capabilities
IA Complexity 1 2 3
Ambiguous RFIs
Provide feedback channels to requesting source
Support multiple earches (stored or ad‐hoc) for situations of interest
Data CharacteristicsSupport manual or automated data association
Suppoautomassess
Distributed Structure
‐ Store access credentials to multiple data sources‐ Automate language translation
Provide ‐ Queue incoming data for Mainta
S
IA Complexity 1 2 3 4 5 6 7
Ambiguous RFIs
Provide feedback channels to requesting source
Support multiple earches (stored or ad‐hoc) for situations of interest
Provide feedback channels to requesting source
Data CharacteristicsSupport manual or automated data association
Support manual or automated situation assessment
Support manual or automated SA & DA
Distributed Structure
‐ Store access credentials to multiple data sources‐ Automate language translation
Lack of Data
Provide feedback channels to requesting source
‐ Queue incoming data for processing‐ Setup custom search alerts to notify when data or situation available appears
Maintain and fuse pedigree data during data association processes
‐ Support what‐if scenarios‐ Support automated pattern identification to highlight potential sit. of interest
‐ Queue incoming data for processing‐ Setup custom search alerts to notify when data or situation available appears‐ Maintain and fuse pedigree data during data association processes
Potential Deception
Auto‐validate credentials of request
Auto‐validate credentials of source
Support what‐if scenarios ‐ Support what‐if scenarios‐ Support automated pattern identification to highlight potential sit. of interest
‐ Support what‐if scenarios‐ Support automated pattern identification to highlight potential sit. of interest
Real‐World Dynamics
Maintain temporal meta‐data of requests
‐ Update KdBase in near real‐time with most recent data‐ Create temporal boundaries on SA & DA‐ Maintain temporal reference meta‐data
‐ Update KdBase in near real‐time with most recent data‐ Create temporal boundaries on SA & DA‐ Maintain temporal reference meta‐data
Unbounded Problem SpaceSupport custom boundary ranges
Support custom boundary ranges
Support custom boundary ranges
Stage of Intelligence Analysis Process
The capabilities mapping table is used to:• Track the development of each capability• Ensure the final fusion system provides coverage to mitigate all the common complexities• Highlight areas where capabilities are needed (pink cells with no text)• Highlight areas where potentially excessive capabilities are being included (grey cells with
text)
Technical ApproachHuman in the Fusion Loop
Complexities:Fusion Sys. Capabilities Mapping
12 of 21
Human‐System touch points were identified to facilitate the incorporation of the identified system capabilities into the larger fusion architecture.– Touch points selected given considerations of user, system and
workflow/domain characteristics
Technical ApproachHuman in the Fusion Loop
Identification of Recommended Touch Points
User Requirements &
Limitations
System Requirements & Limitations
Domain / Workflow Dynamics & Constraints
Overlapping regions between User & System are where potential touch points would be required.
13 of 21
Touch Point
Touch Point
Touch Point
Touch Point
Touch Point
Touch Point
Technical ApproachHuman in the Fusion Loop
Architecture Touch Point Locations
14 of 21
Individual touch point features were identified to begin defining an effective & efficient human‐computer interaction for each respective touch point.
• Features selected to instantiate the pre‐defined capabilities
Touch point definitions were limited by the early stage of the fusion system architecture.
• User interface mediums were not defined• System input/output formats were not defined• Availability of source/data meta‐information not defined
As the system backend and architecture become better defined, further effort will be dedicated to defining the touch points
• Focused on the human‐computer interaction
Technical ApproachHuman in the Fusion Loop
Defining Fusion Sys. Touch Points
15 of 21
Location: After data association processing, but prior to incorporation of processed data into the overall fusion entity‐association database
Expected IA Stage(s) When Accessed:Stage 3 & 4 – Collection & Evaluation of Data
Complexities being at least partially addressed:• Distributed Structure of IA Domain• Potential for Deception• Real‐World Dynamics
Required Features to Allow For:• Browsing/reviewing of incoming data sets’ entity‐association network• Selection of entities within the network to highlight or annotate prior to
fusing with the overall fusion database• Review of executed data merges• Search input to search the incoming data sets’ network for situations of
interest• Drill‐down to entity/association meta‐data, source data, and data
association log
Example Touch Point
Technical ApproachHuman in the Fusion Loop
Defining Fusion Sys. Touch Points
Touch Point
16 of 21
Technical ApproachHuman in the Fusion LoopModel Generalization
Methodology was adapted to provide:– Generalized methods applicable to alternative domain & system architectures– HF engineering techniques to apply at different stages of system development
Early Development Stage
1: Characterize Domain
2: ID Potential Sys Features to Overcome Domain Challenges
3: Characterize Intended User
Mid‐to‐Late Development Stage
4: Designate Touch Points to Control,
Monitor, Access, etc. Sys Capabilities
5: Apply HCI Methods to Design of Touch Point Dialogues
6: Plan/Conduct human‐in‐the‐loop XPs to Validate Touch
Point Selections
Post‐Release Stage
7: User Training
17 of 21
1. Generated context‐aware error characteristics for 8 additional soft observation categories
2. Created extensive report characterizing the IA domain based on review of research literature
3. Developed adaptable methodology to apply human factors engineering to the design and/or improvement of fusion systems across life‐cycle stages– Applied method to identify & initially define requirements for 6 human‐
fusion system interaction touch points for the MURI fusion system
Challenges: • MURI system maturity not yet supportive of high fidelity HIL studies• Lack of direct access to Intelligence Analysts (or other SMEs)
SummaryHuman in the Fusion Loop
18 of 21
Capability Goal: Context‐Aware Human Source CharacterizationsTo validate the benefits of providing soft sensor context‐aware error characteristics for fusion processes, an experiment will be designed and conducted leveraging a subset of the 12 observation category error models.
Research Goals: Validate Human Source Characterization EffortsDetermine the degree of fusion system improvement gained from leveraging the source characterization models for both:– Truthed to Observed References & Associations– Observed to Observed References & Associations
2011‐2012 PlansHuman Source Characterization
19 of 21
2011‐2012 PlansHuman in the Fusion Loop
20 of 21
Capability Goal: Human‐in‐the‐LoopGiven that adequate system components are unlikely to exist for model prototyping, plan and conduct human‐in‐the‐loop experiments focused on topics generalizable to MURI fusion system planned capabilities and employment
Research Goals: Concept‐of‐EmploymentTouch Point 5: •Determine effectiveness of input formats for operators querying fusion systems
Touch Point 6:•Determine effectiveness of output formats for operators leveraging fusion system• Explore the benefits (& drawbacks) of integrating level 2 fusion systems into IA workflows
Touch Point 5
Touch Point 6
Capability Goal: Human Operator SupportHard‐soft fusion system should be supportive of multiple forms of human operator interaction as identified by methodology employed in 2010‐11 program year.
Research Goals: Human‐in‐the Loop experimentation• Leverage results of experimental & theoretical studies to support design of human touch
points within MURI system under development to deploy:• Touch Point 3: Validated control actions to support user during data association processes
• Touch Point 4: Validated control actions to support user during graph merging and enhancement operations
• Extend experimental methodology to encompass multiple touch points as well as to use the capabilities of the MURI systems as those capabilities become available
2012 – 2014 PlansHuman in the Fusion Loop
21 of 21
Touch Point 3
Touch Point 4
APPENDIX
Kent, S. (1965). Special Problems of Method in Intelligence Work. Strategic Intelligence for American World Policy. Hamden, CT, Archon Books: 159‐179.
• Intelligence analysts receive problem of focus most often as direct instance of a consumer request (e.g., Tell me about this)
•Problems, or sub‐problems, can also arise as result of:• Emergence of something unusual (e.g., What is that?)
• An effort to anticipate future problems (e.g., What if this were the case?)
• Source of the problem often affects the complexity & degree of difficulty of the subsequent analysis.
Stage 1: Appearance of substantive problem
Technical ApproachHuman in the Fusion Loop
Characterizing the IA Domain
23 of
Kent, S. (1965). Special Problems of Method in Intelligence Work. Strategic Intelligence for American World Policy. Hamden, CT, Archon Books: 159‐179.
• This stage is based on framing & understanding the problem to serve as a starting point for evidence gathering & evaluation.
• The goal here is to reduce the ambiguity of the problem facing the analyst
• The degree of difficulty is often based on type of request & the degree of ambiguity:•Overly general requests are more common in terms of challenging requests (e.g., Tell me everything about XYZ?)
•Overly narrow requests can also represent a potential challenge (e.g., Where is person A right now?)
Stage 2: Analysis of substantive problem
Technical ApproachHuman in the Fusion Loop
Characterizing the IA Domain
24 of 26
• Rounding up ALL available materials the analyst believes may be related to the problem environment.
• Depending on the availability of information this can result in the potential for:
• Data overload • Challenging scenarios where there is a large gap in available information.
• Due to distributed nature of IA, problems can arise if analysts need data:
• From another organization• From another expert• On a domain he/she is not familiar with
• Across a language barrier
• Collaboration can further add to the complexity of the issue due to the analyst needing to quickly and effectively convey the context of his/her request(s).
Stage 3: The collection of data
Technical ApproachHuman in the Fusion Loop
Characterizing the IA Domain
25 of 26
Kent, S. (1965). Special Problems of Method in Intelligence Work. Strategic Intelligence for American World Policy. Hamden, CT, Archon Books: 159‐179.
• The task of reading and understanding the collected information to determine how it compares to the analyst’s hypotheses and previously considered information.
•A “criticism of data,” as it should consist of the analyst questioning the information’s: •Pedigree •Relevance in relation to all other available data
Stage 4: The evaluation of data
Technical ApproachHuman in the Fusion Loop
Characterizing the IA Domain
26 of 26
Kent, S. (1965). Special Problems of Method in Intelligence Work. Strategic Intelligence for American World Policy. Hamden, CT, Archon Books: 159‐179.
• This is a stage or moment where the analyst puts together the pieces to form the inklings of an explanation for the evaluated data.
• Ideally there will be numerous hypotheses that the analyst considers in parallel•Due to limitations of human cognitive capabilities ideal comparison set is rarely considered
Stage 5: The moment of hypothesis
Technical ApproachHuman in the Fusion Loop
Characterizing the IA Domain
27 of 26
Kent, S. (1965). Special Problems of Method in Intelligence Work. Strategic Intelligence for American World Policy. Hamden, CT, Archon Books: 159‐179.
• Iteration of the previous stages based on analyst’s current hypotheses & the marshaling of evidence to confirm and/or refute them.
• This stage of the process is often dependent on the availability of:• Time •New or updated information
• Evaluation of hypotheses shown to be susceptible to common issues due to human cognitive capabilities. Ex:
• Confirmation bias• Tunnel vision
Stage 6: More collecting/testing of hypotheses
Technical ApproachHuman in the Fusion Loop
Characterizing the IA Domain
28 of 26
Kent, S. (1965). Special Problems of Method in Intelligence Work. Strategic Intelligence for American World Policy. Hamden, CT, Archon Books: 159‐179.
• Final stage of the analyst’s process is creation of an artifact to communicate the established hypothesis (or ideally competing hypotheses) to the IA consumer(s)
•Analysts are not decision makers, their job is to develop & successfully transfer a “new and better approximation of the truth” with regards to the situation of interest
•Distributed nature of domain often makes consumer feedback inaccessible so analysts do not know if their analysis was on target & actionable
Stage 7: Creation / Presentation of IA Artifact
Technical ApproachHuman in the Fusion Loop
Characterizing the IA Domain
29 of 26
Location: • Prior to incoming data set processing
Expected IA Stage(s) When Accessed:• Stage 2 – Analysis of Substantive Problem
Complexities being at least partially addressed:• Ambiguous requests for information• Lack of Data
Required Features:• Allow for custom data sets to be created• Allow for multiple data sets to be prioritized for
processing• Requires ability to scan/review data sets pre‐processing
Touch Point 1:
Technical ApproachHuman in the Fusion Loop
Defining Fusion Sys. Touch Points
Location: • After initial processing & uncertainty alignment, but prior to data
association
Expected IA Stage(s) When Accessed:• Stage 3 – Collection of Data
Capability Intended to Support:• Support multiple searches (stored or ad‐hoc) for situations of
interest
Required Features to Allow For:• Review of data sets being considered for data association• Filtering of data sets to determine custom boundaries to utilize for
data association• Selection of additional data sets (previously processed) to be
included in data association• Custom threshold levels to be set to determine when manual
approval is needed for a system proposed merge to be carried out• Custom uncertainty values to be assigned to individual data
elements (e.g., what if scenarios)
Touch Point 2:
Technical ApproachHuman in the Fusion Loop
Defining Fusion Sys. Touch Points
Location: • During data association processing
Expected IA Stage(s) When Accessed:• Stage 3 – Collection of Data
Complexities being at least partially addressed:• Data Characteristics• Potential for Deception
Required Features to Allow For:• Review of system proposed merges based on pre‐determined threshold level
• Approval/Rejection assignment to system proposed merges
Touch Point 3:
Technical ApproachHuman in the Fusion Loop
Defining Fusion Sys. Touch Points
Location: • After data association processing, but prior to incorporation of processed
data into the overall fusion entity‐association database
Expected IA Stage(s) When Accessed:• Stage 3 & 4 – Collection & Evaluation of Data
Complexities being at least partially addressed:• Distributed Structure of IA Domain• Potential for Deception• Real‐World Dynamics
Required Features to Allow For:• Browsing/reviewing of incoming data sets’ entity‐association network• Selection of entities within the network to highlight or annotate prior to
fusing with the overall fusion database• Review of executed data merges• Search input to search the incoming data sets’ network for situations of
interest• Drill‐down to entity or association meta‐data, source data, and data
association log
Touch Point 4:
Technical ApproachHuman in the Fusion Loop
Defining Fusion Sys. Touch Points
Location: • After any update to the fusion system information database
Expected IA Stage(s) When Accessed:• Stage 3 & 4 – Collection & Evaluation of Data
Complexities being at least partially addressed:• Distributed Structure of IA Domain• Potential for Deception• Real‐World Dynamics• Unbounded Problem Space
Required Features to Allow For:• Browsing/reviewing of the fusion database• Selection of database boundaries with respect to search/browsing
capabilities• Review of highlighted/annotated entities and/or associations• Manual editing/addition of entities/associations/attributes• Search input to search the database using custom criteria• Creation of entity/association placeholders that indicate expected
hypotheses not yet incorporated/observed• Drill‐down to entity or association meta‐data, source data, update log,
weighting, edit precedence, data association log
Touch Point 5:
Technical ApproachHuman in the Fusion Loop
Defining Fusion Sys. Touch Points
Location: • After any update to the fusion system information database
Expected IA Stage(s) When Accessed:• Stage 6 – Additional data collection & hypothesis testing
Complexities being at least partially addressed:• All, with the exception of ambiguous requests for information
Required Features to Allow For:• Review of data sets being considered for data association• Filtering of data sets to determine custom boundaries to utilize for
data association• Selection of additional data sets (previously processed) to be
included in data association• Custom threshold levels to be set to determine when manual
approval is needed for a system proposed merge to be carried out• Custom uncertainty values to be assigned to individual data
elements (e.g., what if scenarios)
Touch Point 6:
Technical ApproachHuman in the Fusion Loop
Defining Fusion Sys. Touch Points