beyond trl: a revised model of technology development and ... · beyond trl: a revised model of...
Post on 21-Aug-2019
215 Views
Preview:
TRANSCRIPT
Beyond TRL: A Revised Model of Technology Development and Considerations for
Programmatic Analysis Prof. Zoe Szajnfarber
Assistant Professor of Engineering Management & Systems Engineering
and Space Policy, The George Washington University
Research Affiliate, MIT Engineering Systems Division
NASA Cost Symposium
NASA Langley – August 14, 2014
On Technology Cost Estimating
• Technology development– or the focused long term development of – is a major part of R&D at NASA.
• Technology Cost Estimating is a relatively unexamined field in the academic literature on cost estimating
• NASAs recent efforts to fund technology cost estimating research have been helpful in understanding how technology develops (Cole et al 2013, 2014)
• Our focus is not on technology cost estimating: we study the process of technology development itself
• However, we hope our research can provide insight for the cost and scheduling community
© 2014 SzajnLab
© 2014 SzajnLab
21995 1990 1985
History of Shifts in R&D Strategy
(1986-1992)
R&T in Office of Aeronautics
and Space Technology
(OAST) = Basic, cross-
cutting research (~250M)
(1992-1994)
Aeronautics
to Office of
Aeronautics
(1992-1994)
Space to Office
of Advanced
Concepts
(OSAT, ~200M)
(1996-1999)
Cross-Enterprise
Technology
Development
(under Code S)
(1996-1998)
Office of Space
Science
(~100M)
(1999-2001)
Office of
Aerospace
Technology
(OAT) ~$40M NRA
~$100M
cross-cutting
(2001-2004)
“Pioneering
Revolutionary
Technology” ~$100M
Advanced
~$180M
IT/Comm.
(1998-2007) NIAC
(2004+)
Exploration
Techn ology
Development
program
~75% cut in
2005
Basic
P
roje
ct-
specific
?
(Based on data collected for NASA R&T Study and NRC study of NIAC)
NEED: To control the system better, we need to understand it better.
© 2014 SzajnLab
Guiding Research Questions
NEED: To control the system better, we need to understand it better.
1. How do new capabilities traverse the innovation system as they are matured and infused into flight projects?
• Empirically grounded models of the innovation process
• Considers technical, social and political factors
• Can this process be predicted/estimated?
2. To what extent can the process be improved through feasible management interventions?
• Exploring organization configuration as a design lever
• Design for evolvability/tinkerability
• Improved incentive systems, based on valid preference structures.
• Balanced technology investment strategies that acknowledge key attributes of space innovation ecosystem
© 2014 SzajnLab
NASA Innovation Landscape
Political-level context
Agency-level planning
Project-level Development & Implementation
Technology-level Research & Development
Scientific and Technical State-of-the-art
© 2014 SzajnLab
NASA (Space Science) Innovation Landscape
Political-level context
New mission
concepts
Prioritization: E.g., Decadal
Survey
Selection: E.g., 1
Flagship; 4 Ex
Concept Studies
Phase A: Concept & Tech Development
Phase B: Prelim design
NRA
Center IRAD
Branch
Center IRAD
Branch
Center IRAD
Branch
Center IRAD
Branch
Center IRAD
Branch
Center IRAD
Branch
Center IRAD
Branch
Center IRAD
Branch
Center IRAD
Branch
NRA NRA NRA NRA
Center IRAD
Branch
Center IRAD
Branch
NRA
Scientific and Technical State-of-the-art
© 2014 SzajnLab
Current Conceptualization: Stage-Gates
Concepts
Basic R&D Applied R&D Project-specific
Tech Dev.
Shelved
capabilities Shelved
capabilities
Shelved
concepts
Flight
Innovation as an Optimization Problem Relative resource allocation problem (how much money in each bucket?) Resources spacing problem (how many buckets? Gate criteria definition problem (how many should be advanced, and by what criteria?)
*Synthesized from NASA strategic planning documents 1990-2006
Maturity (TRL)
© 2014 SzajnLab
Applied R&D
RTOP RTOP
Project-specific
Tech Dev.
Basic R&D
DDF
IRAD
SBIR P1
SBIR P2
Office of
Commercialization
NRA
ROSES
ESTO
MPTO
Pre/
Phase A
Pathfinder
missions
SBIR P2
DDF
IRAD
SBIR P1
Office of
Commercialization
NRA
ROSES
ESTO
MPTO
Pre/
Phase A
Pathfinder
missions
Concepts
Shelved
capabilities Shelved
capabilities
Flight
Shelved
concepts
3
4
5
6
7
8
9
1
9a
Legend New capability
Event
Action
Decision
Contract
New technical concept
First detector
arrays
New Camera Prototype
Space-qualified QWIPs
Chance
encounter
Contacts
Called
Readiness
Communicated
Technical
problems on
project
Congress:
must fly IR
camera!
2 Collaboration
policy
Actual Complexity of Process
Scientist
recruited
0
Maturity (TRL)
© 2014 SzajnLab
Stage-Gate Assumptions
Concepts Basic R&D Applied R&D Project-specific
Tech Dev.
Shelved
capabilities Shelved
capabilities
Shelved
concepts
Flight
Maturity (TRL)
*Synthesized from NASA strategic planning documents 1990-2006
Underlying assumptions:
(1) Technologies mature from left to right over time;
(2) Stages are mutually exclusive (at a given time);
(3) Shelving is an active process, controlled by decision makers;
(4) Shelf life is passive and a function of technical obsolescence.
© 2014 SzajnLab
19
90
20
00
20
10
198
2
198
3
198
4
198
5
198
6
198
7
198
8
198
9
199
1
199
2
199
3
199
4
199
5
199
6
199
7
199
8
199
9
200
1
200
2
200
3
200
4
200
5
200
6
200
7
200
8
200
9
DDF
CETDP
DDF
DDF
DDF IRAD IRAD IRAD
Con-X Con-X
CTD
IPP Seed
Astro-H
PIXIE
AXAF Astro EAstro E2
RTOPS
DDF
SR&T SR&T
DDF DDF
P1
SBIR P2
NMSC
BASIS
BASIS
NDE
P1
SBIR P2
SWIFT
DDF
APPRA
DDFDDFDDF
SR&T
DDF DDF
DDF
IRAD
IRAD IRAD IRAD
APPRA
Cat “3” PhA
AXP GEMS
P1
SBIR P2
DDF
AXAF-NRA
AXAF “-S” Astro-E Astro-E2 Astro-H
Con-X NRA
Con-X IXO
NRA NRANRANRA
NeXT
NRA NRA NRA
CETDPCCT
DDF DDF IRAD IRAD
NRA
DDF DDF IRAD IRAD
IRAD
IRAD
Switchbacks in Maturity
Applied R&D
Project-
specific Tech
Dev.
Basic R&D
Time
vs. Observation:
switchbacks
Matu
rity
(T
RL)
© 2014 SzajnLab
19
90
20
00
20
10
198
2
198
3
198
4
198
5
198
6
198
7
198
8
198
9
199
1
199
2
199
3
199
4
199
5
199
6
199
7
199
8
199
9
200
1
200
2
200
3
200
4
200
5
200
6
200
7
200
8
200
9
DDF
CETDP
DDF
DDF
DDF IRAD IRAD IRAD
Con-X Con-X
CTD
IPP Seed
Astro-H
PIXIE
AXAF Astro EAstro E2
RTOPS
DDF
SR&T SR&T
DDF DDF
P1
SBIR P2
NMSC
BASIS
BASIS
NDE
P1
SBIR P2
SWIFT
DDF
APPRA
DDFDDFDDF
SR&T
DDF DDF
DDF
IRAD
IRAD IRAD IRAD
APPRA
Cat “3” PhA
AXP GEMS
P1
SBIR P2
DDF
AXAF-NRA
AXAF “-S” Astro-E Astro-E2 Astro-H
Con-X NRA
Con-X IXO
NRA NRANRANRA
NeXT
NRA NRA NRA
CETDPCCT
DDF DDF IRAD IRAD
NRA
DDF DDF IRAD IRAD
IRAD
IRAD
Applied R&D
Project-
specific Tech
Dev.
Basic R&D
Time
vs. Observation:
switchbacks
Matu
rity
(T
RL)
Switchbacks in Maturity
DDF DDF
DDF
DDF IRAD IRAD IRAD
CTD
DDF DDF DDF
P1 P1
Con-X IXO
© 2014 SzajnLab
Passive Gates, Active Shelves
• Expectation (assumptions #3 and 4):
3. Rejection at Gate => Shelving
4. Similar shelf lives for similar technologies
• Observation:
Case Rejected
+ Shelf
Rejected
+ !Shelf
!Rejected
+ Shelf
Duration on
Shelf
Tech A 1 1 1 8 /1yrs
Tech B 0 2 1 5 yrs
Tech C 0 3 0 N/A
Tech D 0 2 1 2 yrs
Tech E 1 Multiple 1 2 / 5 yrs
Tech F 0 multiple 0 N/A
Szajnfarber, Z., and Weigel, A. L. (2012). "Managing Complex Technology Innovation: the need to move
beyond stages and gates" International Journal of Space Technology Management and Innovation, 2(1), 30-48
Need: More nuanced understanding of underlying processes
© 2014 SzajnLab
Building Theory from Case Studies
Process
Data
~100 hrs interviews
~150 archival
documents
~2 months informal
observation
Within-case
Analytical Chronologies
(Pettigrew 1990)
1 1997
2 10/1/1998 1998
3 1999
4 9/1/1999 1999
5 10/1/1999 1999
6 10/1/1999 1999
7 9/1/2000 2000
8 10/1/2000 2000
9 10/1/2000 2000
10 late 2000
11 2000
2001d
y
t
re
Interview se 12 early 2001
with KB ng
to
ey
ke
13 6/7/2001 2001
14 4/6/2001 2001
“sense-making” Cross-case theory
building
Structured Visual Map (per Langley 1999)
Characteristic Epochs
Transition inducing Shocks
Technology
Exploration
Architectural
Exploration
Treading
Water and
Branching Out
Exploitation
DevO
p
TechB
Context
Join (-)
Context
Funding (+/-)
DevOp
Approved
DevO
pN
eed
TechA
TechA
Con
text
Fundin
g (+
/-)
DevOpNeed
TechA
Technology
graveyard
Flight
Gestation
Period
DevOpNeed
TechA
Con
text
TechB
TechB
Path Initiation Path Termination
Epoch-Shock Model
SR&T, int with
KB and KJ
MOXE ongoing, LOBSTER just s tarting, us ing gas proportional counters to do a l l sky
monitoring. KB gets connected with KJ (LOBSTER) and SH (interested in Gamma-ray
telescope)
DDF proposal
Awarded: Proposal to explore use of "wel l" microstructure recently patented by Solberg,
Pi tts and Walsh for astronomica l gas detectors . Appl ication: a l l sky monitors ; "The micro-
wel l detector replaces the thin wires of the traditional multi -wire proportional counter
with metal electrodes mounted on oppos i te s ides of a polymer fi lm. The polymer i s then
micro-machined to expose the anodes to the cathodes , establ ishing a two-dimens ional
array of anode-cathode pairs that act as gas proportional counters . The anodes and
cathodes provide an orthogonal coordinate readout with equal resolution in each
dimens ion."
Interview
with KB
KB s truggl ing with making detector work, someone says PDJ good at that - tel l s him to
clean i t, they s tart working together
DDF reportChange: Found that idea wasn't feas ible, instead focused on TFT readouts - >
col laboration with PSU
SR&T
proposal ,
interview
with KB
Rejected: track immaging technology for gamma-ray telescopes . Us ing TFTs col laboration
with PSU
DDF proposal
Awarded: proposal to explore use of sol id-s tate instead of imaging gas micro-wel l
detector. "Construction of a sui table thin window for such a gas detector remains a
technica l chal lenge. We therefore propose to investigate an a l ternative development
path: a sol id-s tate substi tute for the gas in the microwel l detector"
DDF report
Bui l t TFT and integrated i t with micro-wel l detector: S. D. Hunter, et a l . 1999, “Imaging
Micro-wel l Proportional Counters Fabricated with Masked UV Laser Ablation,” Proc. 5th
Int. Conf. on Pos i tion-Sens i tive Detectors , to appear in Nucl . Inst. Meth.
K. Black, et a l . 2000, “Imaging Micro-wel l Detectors for X and g-ray Appl ications ,”
SPIE. 4140-33, in press .
J.R. Huang, et a l . 2000, "Active-Matrix Pixel i zed Wel l Detectors on Polymeric
Substrates , " Proc. National Aerospace and Electronics Conf., Oct 10-12, 2000, Dayton
OH, NAECON 2000. (best paper)
DDF proposaldirect continuation of DDF 1999, but focus on pourous dielectric and colaboration with
Adelphi more than PSU. PSU continued with SR&T
SR&T
proposal ,
interview
with KB
Awarded: same as above. Funding for 2000-2002; never managed to get the TFTs working
though
interview
with KB
memory of conversation with PDJ: thought these detectors were important but needed to
justi fy investment with future appl ications . Pursuing Lobster and gamma-ray miss ion.
Thought polarimeter could be good (science director asking us to solve that problem
every year!). Did some back of the envelop ca lculations and proved to ourselves that i t
wouldn't work for polarimetry... so we dropped that
(Martoff et.
a l . 2000)
NIM A - demonstrated an innovative technique for di ffus ion suppress ion in multi -wire
dri ft chambers with electronegative gas additives
DDF report Adelphi got SBIR through FY2003 to pursue purous dia lectric for thin fi lm window
"It’s kinda funny. That group has been working on polarimeters for a whi le and ha
been doing i t with s trip-readout (one dimentional read out) and they’d never rea l l
gotten anywhere. In their papers i t a lways sa id that i f we had a pixi lated readout i
would be great. And I had never pa id any attention … Then they submitted this natu
paper and somebody here – who I knew – was asked to review the paper… and becau
of my expertise they gave i t to me to take a look… Assuming i t was yet another nothi
result, I took i t home and forgot about i t unti l i t was bed time... when I got around
reading i t. That was a mistake because i t got me so exci ted that I couldn’t s leep! Th
had fantastic results reading out these detectors in a pixel i zed fashion. We could ma
polarimetry work!"
(Costa et a l
2001)
Demonstrate polarimeter based on photoelectric effect - use micropattern gas counters
read-out in pixel i zed way. (Neon-based gas , GEM, MPGC digi ta l read-out)
APRA
proposal
Acepted: develop polarimeter based on Costa (result not yet publ ished?) Increased
active area by factor of 50 (KB says i t's wasn't funded, but found reference in 2006 APRA
that says i t was)
Incident# Date Year Data Source Description
Event Database (Van de Ven et al
1990; 2000)
Case Funding Personnel Technology
Co
mp
on
ent
Ex
plo
rati
on
CADR#1 4xCenter team + Inst - Tech parallel component paths
CZT#2 3xCenter + 3xNASA +
Balloon
team +4xTech
+Inst multiple technique strategies
Pol#3 Brainstorm + 2xCenter +
3xNASA team + Tech multiple readout strategies
Si#4 NASA + Project team + 3xInst +
Tech - 3xObs
multiple materials and techniques
tried
Si#5 2xCenter + 2xNASA +
Sounding Rocket + Project team + Tech
multiple materials and techniques
tried
Si#6 2xCenter + NASA + SR
+2xProject no change
multiple readout strategies and
techniques tried
TES#7 Branch +3xCenter + 2xNASA
+ SR + Project team + Tech
Exploration of new materials and
techniques
Case Funding Personnel Technology
Co
mp
on
ent
Ex
plo
rati
on
CADR#1 4xCenter team + Inst - Tech parallel component paths
CZT#2 3xCenter + 3xNASA +
Balloon
team +4xTech
+Inst multiple technique strategies
Pol#3 Brainstorm + 2xCenter +
3xNASA team + Tech multiple readout strategies
Si#4 NASA + Project team + 3xInst +
Tech - 3xObs
multiple materials and techniques
tried
Si#5 2xCenter + 2xNASA +
Sounding Rocket + Project team + Tech
multiple materials and techniques
tried
Si#6 2xCenter + NASA + SR
+2xProject no change
multiple readout strategies and
techniques tried
TES#7 Branch +3xCenter + 2xNASA
+ SR + Project team + Tech
Exploration of new materials and
techniques
Case Funding Personnel Technology
Co
mp
on
ent
Ex
plo
rati
on
CADR#1 4xCenter team + Inst - Tech parallel component paths
CZT#2 3xCenter + 3xNASA +
Balloon
team +4xTech
+Inst multiple technique strategies
Pol#3 Brainstorm + 2xCenter +
3xNASA team + Tech multiple readout strategies
Si#4 NASA + Project team + 3xInst +
Tech - 3xObs
multiple materials and techniques
tried
Si#5 2xCenter + 2xNASA +
Sounding Rocket + Project team + Tech
multiple materials and techniques
tried
Si#6 2xCenter + NASA + SR
+2xProject no change
multiple readout strategies and
techniques tried
TES#7 Branch +3xCenter + 2xNASA
+ SR + Project team + Tech
Exploration of new materials and
techniques
Case Funding Personnel Technology
Co
mp
on
ent
Ex
plo
rati
on
CADR#1 4xCenter team + Inst - Tech parallel component paths
CZT#2 3xCenter + 3xNASA +
Balloon
team +4xTech
+Inst multiple technique strategies
Pol#3 Brainstorm + 2xCenter +
3xNASA team + Tech multiple readout strategies
Si#4 NASA + Project team + 3xInst +
Tech - 3xObs
multiple materials and techniques
tried
Si#5 2xCenter + 2xNASA +
Sounding Rocket + Project team + Tech
multiple materials and techniques
tried
Si#6 2xCenter + NASA + SR
+2xProject no change
multiple readout strategies and
techniques tried
TES#7 Branch +3xCenter + 2xNASA
+ SR + Project team + Tech
Exploration of new materials and
techniques
© 2014 SzajnLab
Treading Water and
Branch Out
Epoch-Shock Model: Track View
• System exhibits epochs of persistent stable (and identifiable) behaviors punctuated by transition inducing shocks
– Epochs are illustrated as boxes, and roughly map to stages
– Shocks induce transitions following arrows from one box to another
Treading
Water and
Branch Out
Technology
Exploration
Architectural
Exploration Exploitation
Technology
Exploration
Architectural
Exploration Exploitation
Technology
graveyard
Gestation
Technology
graveyard
Flight
Basic
R&DApplied
R&D
Project-
specific
Shelf ShelfShelf
Flight
Gestation
Path Initiation Path Termination
Szajnfarber, Z., and Weigel, A. L. (2013). "A process model of technology innovation in governmental
agencies: insights from NASA's science directorate" Acta Astronautica, 84(3-4), 56-68
© 2014 SzajnLab
Treading Water and
Branch Out
Epoch-Shock Model: Track View
• System exhibits epochs of persistent stable (and identifiable) behaviorspunctuated by transition inducing shocks
Technology
Exploration
Technology
graveyard
Initiation Path Termination Basic R&D STAGE • Low TRL
• <$100K• Cen ter-level
Technology Exploration EPOCH • Patchwork of funding sources• Small core team; ad hoc collaborations• Multiple parallel technology paths
Case Funding Personnel Technology
Tec
hn
olo
gy
Ex
plo
rati
on
CADR#1 4xCenterteam + Inst -
Techparallel component paths
CZT#23xCenter + 3xNASA +
Balloon
team +4xTech
+Instmultiple technique strategies
Pol#3Brainstorm + 2xCenter +
3xNASAteam + Tech multiple readout strategies
Si#4 NASA + Projectteam + 3xInst +
Tech - 3xObs
multiple materials and
techniques tried
Si#52xCenter + 2xNASA +
Sounding Rocket + Projectteam + Tech
multiple materials and
techniques tried
Si#62xCenter + NASA + SR
+2xProjectno change
multiple readout strategies and
techniques tried
TES#7Branch +3xCenter +
2xNASA + SR + Projectteam + Tech
Exploration of new materials and
techniques
Architectural Exploration Exploitation
Gestation
Flight atio
nor
plEx
y og
olTe
chn
Case PersonnelFunding Technology
CADR#1 4xCenter team + Inst -Tech parallel component paths
CZT#2 3xCenter + 3xNASA + Balloon
team +4xTech +Inst multiple technique strategies
Pol#3 Brainstorm + 2xCenter +3xNASA team + Tech multiple readout strategies
Si#4 NASA + Project team + 3xInst + Tech - 3xObs
multiple materials and techniques tried
Si#5 2xCenter + 2xNASA + Sounding Rocket + Project team + Tech multiple materials and
techniques tried
Si#6 2xCenter + NASA + SR +2xProject no change multiple readout strategies and
techniques tried
TES#7 Branch +3xCenter + 2xNASA + SR + Project team + Tech Exploration of new materials and
techniques
© 2014 SzajnLab
Epoch-Shock Model: Track View
• System exhibits epochs of persistent stable (and identifiable) behaviorspunctuated by transition inducing shocks
– Epochs are illustrated as boxes, and roughly map to stages
– Shocks induce transitions following arrows from one box to another
– Innovation pathways start in gestation and move through the system.
Technology
Exploration
Architectural
Exploration
Gestation
Technology
graveyard
Flight Exploitation
Path Initiation Path Termination
Readiness
Communicated
Technical
problems on
project
Congress:
must fly IR
camera!
Chance
encounterContacts
Called
QWIPs detector arraysCollaboration
policy
© 2014 SzajnLab
Epoch-Shock Model: Paths Traveled
• Overlay of ALL the transitions from the pathways studied
– Bi-directional and heavy flow between Technology and Architectural exploration.
– Flow through Exploitation forks between Treading Water and Flight
Technology
Exploration
(8)
Architectural
Exploration
(11)
Exploitation
(11)
Treading
Water
(4)
Path Initiation Path Termination
Gestation
(5)
Technology
graveyard
Flight (4)
© 2014 SzajnLab
Epoch-Shock Model: Paths Traveled
• Overlay of ALL the transitions from the pathways studied
– Colors differentiate different types of shocks, some of which are more controllable by management interventions
– Combined shocks are possible (e.g., red + blue = purple)
Architectural
Exploration
(11)
Exploitation
(11)
Treading
Water
(4)
Path Initiation Path Termination
Gestation
(5)
Technology
graveyard
Flight (4)
Technology
Exploration
(8)
Technology
Missions
Context
Actions
© 2014 SzajnLab
Implications:
Stage-Gate-based management strategies suppress
important dynamics. The Epoch-Shock view
provides a basis for feasible, productive intervention.
© 2014 SzajnLab
Treading Water and Branch Out
Treading
Water and
Branch Out
Technology
Exploration
Architectural
Exploration Exploitation
Technology
Exploration
Architectural
ExplorationExploitation
Technology graveyard
Gestation
Technology graveyard
Flight
Why Stage-Gates Can’t Work
Stage-Gate
Current control mechanisms
1. Proportionally morefunding for basic R&D toincrease pool of early-stage concepts.
2. Used gate decisions tocontrol % progression tonext stage.
Epoch-Shock
Gestation
Path Initiation Path Termination
Basic
R&DApplied
R&D
Project-
specific
Shelf ShelfShelf
Flight
Assessment based on Epoch-Shock model
1. Resources can’t be earmarked for “earlystage/basic.” In practice that funding stream issplit between basic concepts and others that aretreading water and branching out.
2. Actively controllable gates don’t exist. Winnowinghappens based on the co-timing of a technicalbreakthrough (unpredictable) and the nextrelevant mission call (semi-cyclical).
© 2014 SzajnLab
Rethinking the Management Problem
• Basic insight:
– As long as innovation occurs at multiple technical levelssimultaneously, and innovating teams can choose to drawresources from multiple institutional levels
– Current management strategies can’t work as intended!
• Epoch-Shock formulation provides a basis for rethinking themanagement problem:
– Some shocks can be harnessed as management levers: exploringpredictability and influenceability.
– The work environment can be designed, to encourage desirableinteractions and collaborations: exploring incentive systems andorganizational/architectural interactions
© 2014 SzajnLab
Implications for Cost & Schedule Analysts
•A key part of TRL analysis depends on the stage gate model of innovation
–Thinking in terms of the epoch shock model may help point analysts to more complex nuances that they need to study and evaluate.
•A key part of estimating an individual technology depends on the depends on the broader tech ecosystem
–Our cases showed that funding for these projects came from a variety of funding sources at multiple levels
•The process of technology development takes much longer than expected
© 2014 SzajnLab
Thanks for your attention. Comments welcome.
Professor Zoe Szajnfarber
E-mail: zszajnfa@gwu.edu
Web: www.seas.gwu.edu/~zszajnfa
Thanks to our sponsors:
MS: Ademir Vrolijk
PhD: Isabel Bignon
PhD: Alex Burg
PhD: Jason Crusan
PhD: Zachary Pirtle
PhD: Amy Cox
PhD: Sam Marquart
top related