tia consulting march 2002 estimating the contribution of a government program rosalie t. ruegg tia...

TIA ConsultingMarch 2002

Estimating the Contribution of a Government Program

Rosalie T. RueggTIA Consulting

[email protected]

Conference on Estimating the Benefits of Government-Sponsored Energy R&D

Crystal City, VAMarch 5, 2002


Attributing Benefits to a Government Program

inputs outputs outcomes impacts

• What would have happened without the gov’t effort?[counterfactual studies]

• What happened in similar cases where there was no gov’t intervention?

[control group and comparison studies]

It’s not enough to measure these effects; evaluation seeks to determine how much is attributable (caused by) the gov’t intervention.


Overall Net Benefits to Innovation X

Government +Industry R&D Investment

NPV Benefits computed against a baseline


To estimate the return on government investment, we need to isolate its effect

Benefits w/o gov’tmeasured against the

baseline

Private Actions w/o Gov’t Intervention


A Gov’t Program Might Affect Benefits By:

• Accelerating development & deployment- start earlier- progress faster- pursue what otherwise would not be done

• Increasing scale raising likelihood of success

• Increasing scope widening applications

• Building new innovation networks/pathways

• Other effects

(It might also substitute for private investment)


Impact of Acceleration on Social Returns

Source: RTI, NIST GCR 97-737, 1998, fig. 2-3


Illustrative Approaches to Estimating Gov’t Impact

(examples from ATP)

Purposes:1. To support specific project benefit-cost case studies2. To provide program-wide evidence of attribution

Study Examples:• Survey Method e.g., Silber, Powell, Laidlaw –counterfactual analysis Feldman/Kelley--control group study

• Case Study Method e.g., Gompers/Lerner, RTI, Pelsoci, 50 completed Projects, etc.

• Econometric/Sociometric Methodse.g., Branstetter/Sakakibara, Zucker/Darby, Fogarty/Sinha/Jaffe, etc.


Acceleration: Laidlaw Survey of 28 ATP Projects

Applied research cycle time Reductions:

Median reported values- 50% reduction- 3 year cut

Range of reported values- % reduction: 25% to infinity- years cut: 1-2 to unbounded


Other Relevant Effects: Laidlaw Survey

• 86% expected flow-thru of time cuts to later stages

• 79% gave “ballpark estimates” of the value of cutting time by 1 year

• Respondents listed ways the program cut time- upfront planning and mangt requirements- achieving a critical funding mass- enhancing ability to attract additional funding- increasing project stability- encouraging collaboration

• 86% reported carry-over of acceleration to other projects


Feldman/Kelley Non-Winner “Comparison Group” Analysis as Evidence that Gov’t Program (ATP) Makes a Difference

Extent of Follow-up Activity Percentage

Did not proceed with the project at all

61.4

Began project on a much smaller scale

16.4

Began project on a somewhat smaller scale

11.7

Began project at about the same scale as proposed to ATP

5.3

Began project at somewhat larger scale

2.9

Began project on a much larger scale

0.6

Refused to answer/don’t know

1.8

Source: Feldman & Kelley, NRC Report on ATP, 2001, p. 134.


Discussion Topic: Use of a 5-Year Acceleration Rule for DOE Program Evaluation

• NRC basis for?

• Is it “conservative”?

• Is it generalizable?

• Is it feasible to determine specific time lags on a program-by-program basis or should it be on a project-by-project basis?

(ATP’s experience suggests no fixed Rule)

tia consulting march 2002 estimating the contribution of a government program rosalie t. ruegg tia...

Documents