ncw/nec metrics workshop 17-19 december, 2002
DESCRIPTION
NCW/NEC Metrics Workshop 17-19 December, 2002. Report from Work Group 4 Evidence and Assessment. Work Group 4 - Objectives. To discuss ways to develop a knowledge base relevant to NCW/NEC metrics. Key tasks: Identify and evaluate the utility of sources of evidence; - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/1.jpg)
NCW/NEC Metrics Workshop17-19 December, 2002
Report from Work Group 4
Evidence and Assessment
![Page 2: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/2.jpg)
Work Group 4 - Objectives
• To discuss ways to develop a knowledge base relevant to NCW/NEC metrics.
• Key tasks:– Identify and evaluate the utility of sources of evidence;
– Identify the “meta-data” needed to organize and integrate the knowledge base.
![Page 3: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/3.jpg)
Work Group 4 - Strategy
• Journalist’s questions– Who? Why? What? When? Where? How?
• Who? - Identified range of knowledge base user groups.
• Why? - For each, identified why they need the knowledge...
• What? - …and what types of knowledge they need.
• Used NCW Framework to identify NCW-specific knowledge needs.
• When? Considered qualities of evidence for different needs.
• Where? Considered qualities provided by different type of evidence source.
• How? Considered constraints on format/management of KB.
![Page 4: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/4.jpg)
Themes for Evidence Workgroup• Source of evidence
– history, experiment, model, judgement,..
– Multi-national availability
• What you want evidence for?– persuade participation/commitment,
– inform investment (BOI, individual cases, substantiate/test claims),
– inform planning (strategic, campaign, mission),
– generate usable heuristics.
• What are the claims of NCW/NEC?
• Nature of evidence– Strength,credibility, reliability, fidelity, specific/ generic, cost to gather
– Risk of deciding the “wrong” thing
– Types of evidence
• What is this “knowledge base” (and why do we need it)?– What type of knowledge? (claims, evidence, …) - part of a process
– What characteristics does the KB have.
• Who, What, When, Where, Why?
![Page 5: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/5.jpg)
Who? and Why?• Overall WHY? - Justify decisions, generate of commitment to action.
• Policy and Force Structure folks– WHY? - Mission/Context drivers for capabilities. Asymmetric threats. Policy
effectiveness. Economics.
• Materiel folks – WHY? - Where are the new needs? What technologies? Trade-off assessment,
effectiveness - against baseline of current/legacy. Efficiency/cost.
• Doctrine developers – WHY? - Limits of capability. What is effective? Areas for promising
experimentation.
• Operators – WHY? - What have we discovered? ( heuristics). Under what conditions was this
knowledge derived? (reliability). What doesn’t work?
• Trainers– WHY? - What technologies are in-service or emerging? Awareness of current
capabilities, concepts of use, emerging doctrine. [National differences here?]
![Page 6: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/6.jpg)
What?
• Evidence about the nature of NCW/NEC itself.
• Evidence to:– Inform the generation of ideas, goals, concepts.
– Justify decision and persuade others to endorse/participate.
– Enable bringing into being, engineering, production.
![Page 7: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/7.jpg)
Policy and Force Structure
• WHY - Mission/Context drivers for capabilities. Asymmetric threats. Policy effectiveness. Economics.
• WHAT - Evidence to inform, justify, enable...
• Mission requirements - What missions? Where/when? Constraints?
• Capability requirements
• Desired force structure attributes, e.g.– Ability to fight for decision superiority
– Speed of command
– Information access + shared awareness
– Dispersed forces in non-contiguous/continuous operations
– “De-massification”
– Elimination of process lines (boundaries and comm chains)
– Elimination of structural lines
– Dynamic, self-synchronisation of forces
– Compressed levels of operation (collapse of the levels)
![Page 8: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/8.jpg)
Materiel folks
• WHY - Where are the new needs? What technologies? Trade-off assessment, effectiveness - against baseline of current/legacy. Efficiency/cost.
• WHAT - Evidence to inform, justify, enable…• Operational requirements (implies goals, threats, contexts, constraints)
• Systemic requirements (addressable accessible entities, level of sophistication of information exchange allowed)
• Capability gaps (implies baseline capability and capability needs, value gap)
• Technologies, etc. to close gaps (maturity, costs, risks, potential synergies)
• People issues (manning, skills, link to training and doctrine/processes)
• Networking concepts (synergies, concepts for exploitation, potential to enable operations/activities, net-readiness requirements)
![Page 9: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/9.jpg)
Doctrine developers
• WHY - Limits of capability. What is effective? Areas for promising experimentation.
• WHAT Thing itself, and Evidence to inform, justify, enable…
• National Military strategy
• Operational concept/concept of operations; joint doctrine
• Organisational structure
• Specific processes: effects based planning, information sharing, subscription lists, post and pull
• Coevolution of doctrine/materiale by experimentation
• TTPs
![Page 10: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/10.jpg)
Operators
• WHY - What have we discovered? ( heuristics). Under what conditions was this knowledge derived? (reliability). What doesn’t work?
• WHAT - Evidence to inform, justify, enable…
– Effectiveness of different NCW methods under different circumstances
– Range of applicability--circumstances under which types of equipment should/should not be used
– Robustness--
– Resource and information requirements for different kinds of actions
– Rules of thumb/stories/heuristics for suggesting methods and judging applicability; execution level of detail
![Page 11: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/11.jpg)
Trainers
• WHY - What technologies are in-service or emerging? Awareness of current capabilities, concepts of use, emerging doctrine. [National differences here?]
• WHAT - Evidence to inform, justify, enable…
– Doctrine, TTP
– Technologies; when to apply, how to use, human impact
![Page 12: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/12.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
![Page 13: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/13.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
NCW knowledge needs for Trainers
Nothing much
![Page 14: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/14.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
NCW knowledge needs for Trainers
Nothing much
When to apply, how to use different info sources
Nothing much
Nothing much
When/how to exploit sharing capabilities•Consistent ontology, epistemology, context
![Page 15: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/15.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
NCW knowledge needs for Trainers
Nothing much
When to apply, how to use different info sources
Factors for“Net-ready people”•Frames of mind, heuristics•Sharing behaviour modes•Knowledge of networks and how to exploit them•Holistic habits, supra-task views
Nothing much
Nothing much
When/how to exploit sharing capabilities
![Page 16: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/16.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
NCW knowledge needs for Trainers
Nothing much
When to apply, how to use different info sources
Factors for“Net-ready people”
Factors for“Net-ready people”
Nothing much
Nothing much
When/how to exploit sharing capabilities
Factors for effective interaction•Trust, confidence, comfort, … (esp. in computer-mediated interaction)•Ownership of enterprise goals•Feedback and motivation, altruism?•Holistic habits, Basic business rules•Generational changes?
![Page 17: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/17.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
NCW knowledge needs for Trainers
Nothing much
When to apply, how to use different info sources
Factors for“Net-ready people”
Factors for“Net-ready people”
Nothing much
Nothing much
When/how to exploit sharing capabilities
Factors for shared sense-making•Consistent mental constructs•Cultural affinity•Prior interaction experience•Consistent Doctrines
Factors for effective interaction
Factors for collaborative DM•Consistent Doctrines, TTP•Consistent goals/objectives•Will to collaborate•Requisite explicitness•Feedback, confirmation
![Page 18: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/18.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
NCW knowledge needs for Trainers
Nothing much
When to apply, how to use different info sources
Factors for“Net-ready people”
Factors for“Net-ready people”
Nothing much
Nothing much
When/how to exploit sharing capabilities
Factors for shared sense-making
Factors for effective interaction
Factors for collaborative DM
“Gearing” from decision quality to synchronization•What quality factors and how much of each are needed to achieve the behaviour goal
![Page 19: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/19.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
NCW knowledge needed for Policy & Force Structure folks
![Page 20: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/20.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
Doctrine developers
![Page 21: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/21.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
Operators
![Page 22: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/22.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
Materiel
![Page 23: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/23.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
NCW knowledge needs for Materiel
Nothing much•Individual sensor information that is not shared
•State of legacy network•Info systems/comms•Human co-location•Equipment capability gaps (implies requirements)
•Emerging/available technologies•Cost/capability trade-offs
![Page 24: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/24.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
•Information user groups•Security architecture requirements•Interoperation doctrine/TTP•Organisation agility requirements
NCW knowledge needs for Material
•Information service requirements•Driven by TTP? Driving TTP?
•Emerging service concepts•e.g. addressable accessible entities
•Criteria for “net-readiness”•…and for “net-dependency”
•speaks to quality of service and priority for net-readiness
![Page 25: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/25.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
•Human factors/design•Non-organic information requirements•Cognitive process requirements
NCW knowledge needs for Material
![Page 26: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/26.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
NCW knowledge needs for Material
•Legacy collaboration capabilities•Human factors/design for collaboration
•Collaboration Technologies•Information requirements for collaboration•Collaborative cognitive process requirements
![Page 27: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/27.jpg)
NCW Conceptual Framework
AwarenessUnderstanding
Degree of Decision/ Plan Synchronization
Degree of Actions/ Entities Synchronized
Degree of Effectiveness/ Agility
Degree of Information “Share-ability”
Degree of Networking
Force
Quality of Individual Information Degree of Shared Information
Quality of Individual SensemakingShared Awareness
Shared Understanding
Degree of Shared Sensemaking
Operating Environments
Quality of Organic Information
C2 EffectorsValue Added
Services
Quality of Collaborative DecisionsQuality of Individual Decisions
Physical Domain
Social Domain
Information Domain
Cognitive Domain
Quality of
Inter-actions
Information Sources
… ?
NCW knowledge needs for Material
![Page 28: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/28.jpg)
When?• Qualities of evidence for different user needs.• Need for good evidence
– Manage the risk of reaching incorrect conclusion or knowledge claims.
• Dimensions of the evidence:– Strength - the extent to which the evidence supports the hypothesis claim
(the probability of the hypothesis given the evidence).• Credibility – the extent to which the decision-maker is predisposed to believe
the evidence based in the source and prior experience with evidence. The extent to which it speaks to his understanding or is coherent with his criteria (or comes from sources he trusts).
• Reliability – reproducibility under insignificantly context changes (e.g., different observer)
• Fidelity – depth of detail or fineness of focus• Specific/ generic – relevance of evidence to wide range of context, robustness
across variable conditions, applicability to the future context
• Case to case comparability• Cost to gather evidence traded with quality
– minimum acceptable quality– maximum effective quality
![Page 29: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/29.jpg)
Qualities of evidence required
Evidence user
Strength Credibility Reliability Specific/Generic
Fidelity Risk of "error"
Policy/ Force Structure
HIGH HIGH HIGH Generic LOW LOW
Doctrine developers
LOW HIGH MEDIUM Generic LOW MEDIUM
Materiel folks
MEDIUM HIGH HIGH Specific HIGH MEDIUM
Operators MEDIUM HIGH HIGH? Specific HIGH LOW
Trainers ? MEDIUM MEDIUM Generic HIGH MEDIUM
Researchers LOW LOW LOW ? HIGH HIGH
![Page 30: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/30.jpg)
Where?
• ALL evidence, of whatever quality, can be used provided one knows the quality
• Sources of evidence– Historical records, personal experiences, synthetic
experiences, expert opinion, controlled experiments, “forensic” analysis.
![Page 31: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/31.jpg)
Qualities of evidence sources
Evidence source
Strength Credibility Reliability Specific/Generic Fidelity Risk of "error"
Historical experiences
LOW LOW MED Specific ? HIGH
Personal experiences (Post deployment hot wash)
MED MED LOW Specific HIGH? ?
Synthetic experiences (modelling, exercises, experiments)
MED MED MED Specific HIGH MED
Expert opinion MED MED MED Generic MED MED
Controlled experiments
MED MED HIGH Specific HIGH LOW
"Forensic" Analysis
HIGH MED MED Generic HIGH LOW
![Page 32: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/32.jpg)
How?
• Network-centric knowledge representation – Knowledge distributed and self-synchronised?
– Maintenance policy by central dictat or enlightened self-interest?
– Using NCW principles in relation to creating/exploiting evidence
– Dissemination
• National differences?
![Page 33: NCW/NEC Metrics Workshop 17-19 December, 2002](https://reader034.vdocuments.us/reader034/viewer/2022052603/5681484b550346895db55ff7/html5/thumbnails/33.jpg)
Summary
• Knowledge relevant to NCW serves diverse user communities (e.g., policy makers, force structure, operators, trainers, materiel people, doctrine developers)– Information in the knowledge base can be mapped on to the NCW
framework
– Information needs to be organized to support each user community
– There are technical and national policy issues to build a knowledge base
• Evidence (meta-data) to substantiate the information in the knowledge base is available from multiple sources– Historical experiences, personal experience, synthetic, expert opinion,
controlled experiments, analysis & synthesis
– Information from multiple sources can increase strength
– Required strength of evidence varies with how the information is going to be used