towards appropriate selection of analysis tools and methods
DESCRIPTION
Towards Appropriate Selection of Analysis Tools and Methods. Overview. Research to determine the appropriate tools and methods for effective application within a broad range of analysis categories. Longer term goal to complete whole activity Summarised later This paper focuses on - PowerPoint PPT PresentationTRANSCRIPT
Towards Appropriate Selection of Analysis Tools
and Methods
Overview
• Research to determine the appropriate tools and methods for effective application within a broad range of analysis categories.
• Longer term goal to complete whole activity– Summarised later
• This paper focuses on – categorisation of study and problem types
• Initial activities description
– Discussion on derivation of metrics • for assessing the value of study methods and tools vs
problem and study types.
Aim
• Overall Aim and Introduction• The aim of the whole research is to determine the
appropriate tools and methods for effective application within a broad range of analysis categories.
• The work will– investigate and analyse a broad range of differing analysis study
types and – research optimal methods and tools for utilisation within these
types. • The analysis study types
– will be example operational and system analysis studies – investigated to determine
• the approaches used, • the constraints and issues associated with them and • the overall success and outcomes.
Activities to date
• Initial Literature Review– OR Society– College Library– MORS Material
• Methods Handbook• Model Taxonomy
– SME Contact
• No evidence of complete approach • But
– Problem Structuring Methods – Paradigms
Problem Types
• Obeng’s Problem types– Fog – Open– Movie – Semi Open– Quest – Semi Closed – Paint by Numbers – Closed
• Mike Pidd– Problems– Puzzles– Messes
• Obeng preferred– More refined
– Organisational Level• Corps• Div• Bn
– Combat Type• High Intensity• PK• Hybrid
– Timescale of Battle• Hour• Day• Days
Proposed Factors affecting the problem
• The Question– Cost Effectiveness Comparison– Force Mix– Battlefield Effectiveness– Capability Gap– Tactics comparison– Numbers
• The Customer
• Soft Issues
Proposed Factors affecting the problem II
Factors affecting the solution• Skill set• Analysis Timescales• Analysis Budget• Model Types
– Hard– Soft– etc
Factors affecting the solution• Skill setDefence EnvironmentAnalysis within Acquisition and Systems Process
UsesCapability AnalysisAcquisition Processes
Business CaseBOICOEIA
Operational SupportWargamingSeBATraining
Systems ApproachValidation
Factors affecting the solution – Skill set
Tools and Methods OR Techniques
Soft MethodsSpreadsheetsMathematical ModellingEngineering Modelling
Modelling and SimulationVirtualLiveConstructiveStochasticDeterministic etcGamesSimulationsAnalytical methods – lanchester etcSEs
Model Development Process
Factors affecting the solution – Skill setDesign of Experiments
Analysis and interpretation of resultsDesign of Experimental Studies
ConfidenceVerificationCommunication and Understanding
Model Components Networking/ Distribution of SimulationHCIProtocolsBehavioural Representation/Intelligent Systems
DataData Analysis & Databases
Combat Sys, TerrainScenariosSoftware Development
Skill set
• Level of Skill required for differing roles– Senior Decision Maker– Problem Owner– Study Leader– Analyst– Developer– Player– Project Manager– Technical Manager
Success Criteria - Measuring how well the method(s) Measuring how well the method(s) fits the problemfits the problem• A. Answer obtained
– Within time/budget/other constraints
• B Answer accepted – Validated
• C Answer Used
• D Repeat work
Success CriteriaMeasuring how well the method(s) fits the problemMeasuring how well the method(s) fits the problem
• A. Answer obtained– Within time/budget/other constraints– Scale
• On budget/time• Ahead of budget/time• 20% slip• 50% slip• 100% slip
Success CriteriaMeasuring how well the method(s) fits the problemMeasuring how well the method(s) fits the problem
• B Answer accepted – Validated– By peer/QA review– By acceptance– Accepted after changes
Success CriteriaMeasuring how well the method(s) fits the problemMeasuring how well the method(s) fits the problem
• C Answer Used– By one user– By several– Citations
Success CriteriaMeasuring how well the method(s) fits the problemMeasuring how well the method(s) fits the problem
• D Repeat work– Once– Extension– Frequent
Next StepsStudy Examples - • Drawn from industry and Dstl where possible. • For each study
– What was done, – what techniques used – resources used – Output measured
Study Analysis• Studies categorised according to the range of criteria or dimensions,
– Problem type – Effort levels available– Timescales– Tools available– Skills available– Audience/Customer– Other Soft issues – political
Comparative Analysis of UK MoD Guidelines for conduct of Analysis
Outline• Evaluation of Study Approaches
– Identification of a full range of approaches including tools and methods for potential use within these types of analyses.
– Identification of the approaches and the associated tools and methods used within each example study.
– Derivation of metrics for assessing the value of each of approaches and the tools and methods in achieving successful outcomes in the studies.
• Derivation of Optimising Approaches– Derivation of theoretical/initial rule sets for identifying
the optimal approaches and tools and methods for use within study categories.
Outline• Analytical Review of Example Study
– Testing of the initial theoretical rule sets against the example studies to determine where non-optimal methods were applied and identification of the improved approaches which might have been employed and the impact these would have had on the study outcome.
• Test phase using New Studies– Experimental phase using real study examples to
develop approaches and test the hypothesised rule sets. • Conclusions
– Identified optimised approaches including the risk areas and constraints and the ‘region of validity’ of the conclusions.
Towards Appropriate Selection of Analysis Tools
and Methods