Estimating the Impact of Hypothetical Portfolio Reductions on

Production of Major Discoveries Funded by the National Institute of

Allergy and Infectious Diseases

Kevin WrightAEA Annual Conference

Nov 3, 2011

Co-AuthorsBrandie K. Taylor (NIH), Jamie Mihoko Doyle (STPI), Brian Zuckerman (STPI)

Overview

• Task description and approach• Part I Exploratory Analysis: Data and results• Part II Counterfactual Analysis: Data, methods,

results, and assumptions• Conclusions

2

Question

• What might the impact be on the research community and public health if NIAID received a budget reduction?

• What would have been the impact had NIAID received smaller budgets?

3

Approach

4

• Two sets of analyses were conducted:1. An initial exploratory analysis to characterize the

discoveries 2. A comparison of characteristics of R01s

associated with discoveries with those that were not

Part 1Exploratory Analysis

Data

6

• 138 unique discoveries (or “advances”) between 2005 and 2009 were identified by NIAID and a list was provided to STPI

• A complex script was written to scan the attributes of each discovery for grant numbers, query them in QVR, and populate a database

• A total of 10,323 awards were pulled from QVR– These records contained all award types (ex. 1, 2, 5, etc.)

Data

7

• Variables that were downloaded from QVR include:– Priority scores and percentiles (if available)– Award amount and year– Study section– Full grant number– Institution and institution state– Principal investigator (PI) name

Data

8

Exclusions• Only the most recent Type 1 or 2 was retained• R37 (i.e. MERIT) and R56 (i.e. High Priority,

Short-term Project) awards were excluded, but the associated R01s were kept

• Contracts and intramural awards were excluded from the analysis

Data

9

• 138 unique discoveries were associated with 305 unique awards (247 of which were NIAID)

• 106 out of the 138 of the discoveries were funded exclusively through NIAID awards (77%)

Results and Findings

Distribution of Discoveries by Year

11

2005 2006 2007 2008 20090

5

10

15

20

25

30

35

40

All Discoveries NIAID-funded only

Num

ber

of D

isco

verie

s

(N=138 discoveries) (N=106 discoveries)

Non-NIAID Award Funding Sources

12

NIAMS NCI NCID NIDA NIGMS NICHD NHLBI NLM NIMH NINDS NCRR FIC0

5

10

15

20

25

Num

ber

of A

war

ds

Number of Awards Cited per Discovery

13

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 160

10

20

30

40

50

60

70

80

All Awards NIAID Only

Num

ber

of D

isco

verie

s

(N=138 discoveries)

Mean: 2 awards

Distribution of Activity Code

14

All Awards, All ICs(N=305 awards)

NIAID Only (N=247 awards)

N %R01 117 38.4R03 1 0.3R21 12 3.9Other R 5 1.6U01 73 23.9Other U-awards 24 7.9P01 22 7.2Other P-awards 16 5.2K-awards 12 3.9T-awards 6 2.0Other 17 5.6

N %R01 97 39.3R03 0 0R21 11 4.5Other R 5 2.0U01 71 28.7Other U-awards 20 8.1P01 20 8.1Other P-awards 9 3.6K-awards 10 4.0T-awards 4 1.6Other 0 0

Part 2 Counterfactual Analysis

Counterfactual Analysis

16

1. A comparison of R01 associated with discoveries with those that were not

2. An analysis of discovery-associated awards affected by reductions in appropriations

R01s Only

17

• Highest percentage of awards funding NIAID discoveries

• Primary funding mechanism for basic and applied research

• Percentile and payline data easily accessible

Reference Sample

18

• All Type 1 and Type 2 R01s were extracted from QVR for FY 2004-2008

• R01s associated with the discoveries were then removed

• R01s with percentiles that were missing or outside of the payline for that particular FY were also excluded

• A total of 2,320 awards are contained in the reference set

Sample

19

• The analysis is restricted to the 40 discoveries associated with R01s– Of the original 138 discoveries, 59 were omitted because

they were not supported by at least 1 R01– 29 were omitted because none of R01s associated with the

discovery were within 2004-2008 period– 10 discoveries excluded because the percentile

information for the R01s were missing (N=4) or outside of the payline (N=6) for that FY

• 38 R01s are considered for the analysis

Methods

20

• The amount of appropriated funds were allowed to vary, but several factors within each FY were held constant. These factors include: – Percentage of allocated funds dedicated to extramural

awards– The percentage of extramural funds dedicated to Type 1

and Type 2 R01s– Cost per R01

• The cost per R01 estimated by the budget office (based on previous years) is equal to the actual cost per R01 reported for that FY

Distribution of R01 Awards

21

Discovery-assoc. R01 Non-discovery assoc. R010%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

20082007200620052004

Pe

rce

nta

ge o

f Aw

ard

s

Distribution of Percentiles

22

R01s associated with discoveries R01s not associated with discoveries0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

19+ 14-1812-1411-125-101-5<1

(N= 38 awards) (N= 2,320 awards)

Pe

rce

nta

ge o

f Aw

ard

s

Procedure for Counterfactual

23

1. For each FY, calculate the reduction in extramural allocations, total funding for Type 1 and Type 2 R01s as a function for each percent reduction in overall appropriations

2. Calculate the number of awards lost by holding the cost per R01 constant

3. Order all R01s by percentile (high to low) and eliminate awards according to number of awards lost at each percent reduction in overall appropriations

R01s Affected by Appropriation Reduction

24

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%0

102030405060708090

100

Discovery-assoc. R01 % Other R01 affected

% R

01s

Affe

cted

Conclusions

25

• Results from the descriptive analysis demonstrate the complexities of how discoveries are generated and reinforce the challenges of attributing major advances to a single award

• Even small reductions in the budget could result in the loss of discoveries

• Applications having percentiles outside of the payline, but were funded, were no less likely to yield a discovery than those within a payline

Contact Info

Kevin D. Wright, MPADeputy ChiefStrategic Planning and Evaluation BranchOSPFM, OSMO, NIAID, NIH, DHHS(p) 301-402-3574wrightk@mail.nih.gov

Thank You

Mapping of award percentiles by discovery (N=40)

Awards Above the Payline

29

Below Payline Above Payline Total

Non-Discovery 85.5 14.5 100

2,320 394 2,714

Discovery 86.4 13.6 100

38 6 44

• Whether select pay or other awards outside of the payline would have been lost is uncertain. The assumption is that these awards would not be affected and therefore excluded from the analysis.

• An ancillary analysis to determine whether discoveries had higher percentages of awards outside of the payline was conducted