![Page 1: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/1.jpg)
Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace ResearchersSharon Monica Jones NASA Langley
Rafael E. Landaeta, C. Ariel Pinto and Resit Unal Old Dominion University
James T. Luxhøj Rutgers University
Hampton Roads Area INCOSE Conference on Decision Analysis and Its Applications to Systems Engineering, Newport News, VA (November 17-18, 2009)
National Aeronautics and Space Administration
www.nasa.gov
![Page 2: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/2.jpg)
National Aeronautics and Space Administration Jones,, et al. (2009) 2
Outline• Background/Definitions
• Data Collection Process
• Results
• Concluding Remarks
![Page 3: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/3.jpg)
National Aeronautics and Space Administration
Problem Definition• Aerospace technology managers need:
To predict future technologies
To assess progress toward R&D goals
• Aerospace technology portfolio decisions are difficult because:
Very little time to acquire background data
Uncertainty factors (e.g., politics, global economy, environment, funding)
• Prescriptive decision analysis methods
Have been used for financial portfolio assessment
Value for policy related decisions has been questioned
Jones, et al (2009) 3
![Page 4: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/4.jpg)
National Aeronautics and Space Administration
Ralph Keeney’s Suggestions for Making Better Decision Makers*• Develop concepts, tools and procedures to help decision makers
“My experience is that many people, including well-educated people, have a very difficult time in structuring their decisions. They can get mixed up about the difference between fundamental concepts such as alternatives and objectives.”
• Use real decisions, not just laboratory problems in decision research
“We have learned a great deal from all the laboratory settings where decision experiments have been conducted. There have also been some research studies of real decision problems. I feel there is much more to be gained by having more of this type of research.”
• Teach people what they can and will learn and use
“…hundreds and thousands of people have had at least a course that included a substantial part on decision analysis and very few have probably ever conducted a formal decision analysis. Once we find out what people can and will learn and use, that should constitute the basis for much of our teaching of decision analysis.”
*Source: Ralph L. Keeney, “Making Better Decision Makers”, Decision Analysis, 1:4 (2004)
Jones,, et al. (2009) 4
![Page 5: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/5.jpg)
National Aeronautics and Space Administration
Decision Analysis Usage in Aerospace Portfolio Development• Aerospace managers have investigated the use of decision analysis methods for portfolio
investment decisions:
Commercial Aviation Safety Team (CAST)
NASA Aviation Safety Program
Future Aviation Safety Team (FAST)
• These technology assessments involved:
Resource commitments (e.g., employee time, travel money, software acquisition, training)
Assumption that decision analysis methods would improve aerospace technology
assessment process
Jones,, et al. (2009) 5
![Page 6: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/6.jpg)
National Aeronautics and Space Administration
Technology Assessment
“ A process for measuring the impact of established or new technologies”
*Hans Mohr, “Technology Assessment in Theory and Practice”, Society for Philosophy and Technology, 4:4 (Summer 1999)
Jones, et al (2009) 6
![Page 7: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/7.jpg)
National Aeronautics and Space Administration
Aerospace Technology AssessmentThree different processes for examining impact of a set of technologies
Technology assessment
Technology forecasting
Technology foresight
Aerospace Technology Assessment
Jones, et al (2009) 7
![Page 8: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/8.jpg)
National Aeronautics and Space Administration
Conceptual Model *
1. Identify Context & Understand Objectives
2. Identify Alternatives
Aerospace Technology Assessment
Decision Analysis Methods
3. Decompose & Model the Problema. Model of Problem Structureb. Model of Uncertaintyc. Model of Preferences
4. Choose the Best Alternative
5. Sensitivity Analysis
6. Implement the Chosen Alternative
*Adapted from Robert T. Clemen, Making Hard Decisions, 2nd Edition (1995)Jones, et al (2009) 8
![Page 9: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/9.jpg)
National Aeronautics and Space Administration
Purpose of StudyWhat is known:
Decision analysis methods in financial portfolio assessment
Decision experiments in laboratory settings
Technology assessment in medical R&D
What is unknown:Decision analysis methods for policy related decisions
Decision analysis methods in real decision problems
Technology assessment in aerospace
Jones, et al (2009) 9
![Page 10: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/10.jpg)
National Aeronautics and Space Administration
Study Overview• Population was aerospace researchers with experience in one or more of the following:
Aerospace program/project management
Aerospace technology assessment
Aerospace technology selection
Aerospace R&D portfolio development
• Methods that were investigated in the study
Decision trees
Influence diagrams
Criteria aggregation methods
Explicit tradeoff approaches
• Participants were questioned about their usage of these methods for aerospace technology
assessment
Jones, et al (2009) 10
![Page 11: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/11.jpg)
National Aeronautics and Space Administration
Decision Trees
11
Buy shares of XYZ stock
Don’t buy XYZ stock
Value of XYZ stock increases
Value of XYZ stock unchanged
Value of XYZ stock decreases
Earn money
Total amount of money unchanged
Lose money
Total amount of money unchanged
= decision node
= chance node
= consequence node
Jones, et al (2009)
![Page 12: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/12.jpg)
National Aeronautics and Space Administration
Influence Diagrams
12
Buy XYZ stock? Change in amount of money
Price of XYZ stock at
future point
decision node
chance node
consequence node
Jones, et al (2009)
![Page 13: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/13.jpg)
National Aeronautics and Space Administration
Criteria Aggregation Methods• Methods in which two sets of aggregated indices are developed and used to evaluate the
alternatives in the decision problem
• Methods in the category include:
Analytical Hierarchy Process (AHP)
Weighted Sum Model (WSM)
13Jones, et al (2009)
*E. Triantaphyllou, Multi-Criteria Decision Making Methods: A Comparative Study(1995)
Example of Simple Weighted Sum Model *
n
A*WSM-score =
max
aij,wj, for 1=1,2,3,….n i
j = 1 where, A*WSM-score = the WSM of the best alternative n = the total number of criteria
aij = the score of the i-th alternative in terms of the j-th criterion wj = the weight of importance of the j-th criterion
![Page 14: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/14.jpg)
National Aeronautics and Space Administration
Explicit Tradeoff Approaches• Decision analysis methods that are based on value functions
• Methods in this category include:
Multi-Attribute Utility Theory (MAUT)
Simplified Multi-Attribute Rating Approach (SMART)
14Jones, et al (2009)
![Page 15: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/15.jpg)
National Aeronautics and Space Administration
Excluded Decision Analysis Methods • Avoided decision analysis methods that are not popular in U.S.
• Real world applications are complex with large amounts of uncertainty
• Specific decision analysis methods that were excluded from study:
Outranking methods (e.g., ELECTRE, TOPSIS)
Optimization methods
Analytic network process (ANP)
Jones, et al (2009) 15
![Page 16: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/16.jpg)
National Aeronautics and Space Administration
Data Collection and Analysis ProcessRefine List of Candidate Survey Participants
Develop Web-Based Survey Instrument
Develop List of Pilot Participants
Conduct Pilot Survey
Review & Analyze Pilot Results
Modify Survey Instrument
AnalyzeResults
Conduct Survey
Legend Survey Development Data Collection Data Analysis
Jones, et al (2009) 16
![Page 17: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/17.jpg)
National Aeronautics and Space Administration
Web-Based Instrument Development• Several web-based services examined
• Questions developed based on several sources:
Short surveys at professional meetings
Validated research in decision analysis literature
• Identities of survey participants remained anonymous
Jones, et al. (2009) 17
![Page 18: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/18.jpg)
National Aeronautics and Space Administration
Pilot Survey• Conducted with subset of population (10 persons)
• Think aloud cognitive interviewing techniques used
Solicitation of all thoughts and comments
Manual recording of information during completion of online survey
De-identification of subjects in final documentation
18Jones, et al. (2009)
![Page 19: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/19.jpg)
National Aeronautics and Space Administration
Survey Instrument Modification• Questions were modified, added or eliminated from the survey based on input from:
Pilot survey comments
Data analysis of pilot survey data
Additional comments from other reviews (e.g., ODU IRB)
• Number of survey questions reduced from 70 to 65
19Jones, et al. (2009)
![Page 20: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/20.jpg)
National Aeronautics and Space Administration
Data Collection Overview• Approval to conduct survey was obtained from NASA Langley and ODU Institutional Review
Boards (IRB’s)
• E-mail invitation was sent to 260 persons
• 154 total visits to survey website
16 partial responses
99 completes surveys
• Out of the 99 completed surveys
76% male, 24% female
Highest degree level was 60% Masters, 21% Bachelors, 18% Doctorate, 1% Associates
72% employed as government civil servants
Jones, et al. (2009) 20
![Page 21: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/21.jpg)
National Aeronautics and Space Administration
Job Functions
Jones, et al. (2009) 21
![Page 22: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/22.jpg)
National Aeronautics and Space Administration
Aerospace Experience
Jones, et al. (2009) 22
![Page 23: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/23.jpg)
National Aeronautics and Space Administration
Decision Trees
23Jones, et al. (2009)
![Page 24: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/24.jpg)
National Aeronautics and Space Administration
Influence Diagrams
24Jones, et al. (2009)
![Page 25: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/25.jpg)
National Aeronautics and Space Administration
Criteria Aggregation Methods
25Jones, et al. (2009)
![Page 26: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/26.jpg)
National Aeronautics and Space Administration
Explicit Tradeoff Approaches
26Jones, et al. (2009)
![Page 27: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/27.jpg)
National Aeronautics and Space Administration Jones, et al. (2009) 27
Usage of Decision Analysis Methods Outside of a Classroom Environment
![Page 28: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/28.jpg)
National Aeronautics and Space Administration Jones, et al (2009) 28
Categories of Non-ATA Usage of DA Outside of a Classroom Environment
![Page 29: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/29.jpg)
National Aeronautics and Space Administration
Additional Questions
29Jones, et al. (2009)
![Page 30: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/30.jpg)
National Aeronautics and Space Administration
Additional Questions (cont’d)
30Jones, et al. (2009)
![Page 31: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/31.jpg)
National Aeronautics and Space Administration 31Jones, et al. (2009)
Additional Questions (cont’d)
![Page 32: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/32.jpg)
National Aeronautics and Space Administration
Additional Questions (cont’d)
32Jones, et al. (2009)
![Page 33: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/33.jpg)
National Aeronautics and Space Administration
Additional Questions (cont’d)
33Jones, et al. (2009)
![Page 34: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/34.jpg)
National Aeronautics and Space Administration
Likelihood of Future Usage of DA Methods
34Jones, et al. (2009)
![Page 35: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/35.jpg)
National Aeronautics and Space Administration
Concluding Remarks• This is a subset of the total data
There are many additional questions in the study
More formal analysis of the data was conducted using structural equation modeling techniques to test a set of hypotheses
• Survey participants believed that the successful use of decision analysis methods depends on:
Selection criteria in the decision model
Experience of the person that implements the method
Reliability of the input data
• Training/education does not guarantee future use of a decision analysis method
35Jones, et al. (2009)
![Page 36: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/36.jpg)
National Aeronautics and Space Administration
Questions?
36
![Page 37: Usage of Decision Analysis Methods Outside of a Classroom Environment by Aerospace Researchers Sharon Monica Jones NASA Langley Rafael E. Landaeta, C](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514a7bf550346b2598b5e31/html5/thumbnails/37.jpg)
National Aeronautics and Space Administration Your Title Here 37