Probability Elicitation and Calibrationin a Research & Development Portfolio

A 13-Year Case Study

• Dimensions of value for R&D projects• Probability of technical success as a metric• Assessing probabilities• A review of thirteen years of data

Jay AndersenEli Lilly and Company

June, 2012

R&D Portfolio Model Structure

ProjectValue

TechnicalFeasibility

MarketOpportunity

Cost Timing

PreclinicalTech. Feas.

Phase 1Tech. Feas.

Phase 3+Tech. Feas.

Phase 2Tech. Feas.

CapitalCosts

ClinicalCosts

DevelopmentCosts Regulatory

Review

PreclinicalStudies

ClinicalStudies

Patients Competition

All CostsTechnicalFeasibility

PreclinicalTech. Feas.

Phase 1Tech. Feas.

Phase 3+Tech. Feas.

Phase 2Tech. Feas.

The focus of today’s discussion 2

Measuring Technical Feasibility

• Qualitative descriptions of uncertainty suffer from vagueness and lack of collective agreement on useful definitions.

• A subjective probability represents the degree of belief in an event by an individual.

• Quantification of this uncertainty allows other business metrics to be specified (e.g, probabilized NPV, cash flows, expenses).

3

A probability of technical success can be elicited by type of uncertainty

TechnicalSuccess

ToxicologyResults

ClinicalEfficacyResults

ClinicalSafetyResults

RegulatoryResults

4

A probability of technical success can be elicited by stage of development

TechnicalSuccess

PreclinicalSuccess

Phase 1Success

Phase 2Success

Phase 3 &Registration

Success

5

Each stage of development can be broken down by type of uncertainty

Phase 2Success

ToxicologyResults

TumorResponse

Rate

Time toProgression

AdverseEvents

6

Probability ElicitationProcess Alternatives

• Self-assessed by project team • Using a trained facilitator• Independent review board• Independent review board with a trained

facilitator

7

Facilitators are trained to deal with bias

• Anchoring and Adjustment• Availability• Conditioning• Motivational• Representativeness

8

A historical assessment ofprobability projections

• At Lilly, an independent review board (PAG) has been charged with the responsibility of objectively assessing the P(TS) of R&D portfolio projects since early 1997.

• These assessments have been partitioned by stage of development:– P(preclinical success)– P(phase 1 success given preclinical success)– P(phase 2 success given phase 1 success)– P(phase 3 & registration success given phase 2 success)

• Our database has over 730 probability estimates over the past 13 years. We have been able to couple these estimates with actual success and failures to determine the accuracy of probability assessments.

9

0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

PAG probability assessment

Act

ual S

ucce

ss R

ate

146

PAG predicted a probability of 0.80146 times over the last 13 years. 118 of those events were successes, for an observed success rate of 0.81.

PAG Performance

10

Figure 1

0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

PAG probability assessment

Act

ual S

ucce

ss R

ate

Most of the “high-sample size” observations lie within a +/- 0.10 band about the target line

134

146

70

111

1

45

23

3022

2411

8

PAG Performance, all raw data2

2

2 2

4

2

2

2 2 1

12 12 1 1 1 2 1 1

223

1

1

21

11

11 3 110

318

1

11 111

Figure 2

0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

PAG probability assessment

Act

ual S

ucce

ss R

ate

19

139

In this analysis, “nearby” observations were combined together into adjacent intervals centered at the 2½% points.

For example, there were:• 5 observations at 0.48 (4 successes)• 45 observations at 0.50 (24 successes)• 2 observations at 0.51 (2 successes)• 1 observation at 0.52 (0 successes)• 2 observations at 0.53 (0 successes)• 1 observation at 0.55 (0 successes)• 2 observation at 0.57 (0 successes)These combined for:• 58 observations at 0.525 (30 successes)

41

49 56

58

117

96153

5

Collecting Nearby Observations into Buckets, view 1

12

Figure 3

0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

PAG probability assessment

Act

ual S

ucce

ss R

ate

16

22

52

55

78

119

75

173

140

3

For example, there were:• 3 observations at 0.34 (1 success)• 18 observations at 0.35 (6 successes)• 2 observations at 0.36 (1 success) • 1 observation at 0.37 (0 success)• 30 observations at 0.40 (13 successes)• 1 observation at 0.42 (0 success)These combined for:• 55 observations at 0.375 (21 successes)

Collecting Nearby Observations into Buckets, view 2

13

Figure 4

In this analysis, “nearby” observations were combined together into adjacent intervals centered at the 7½% points.

0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

PAG probability assessment

Act

ual S

ucce

ss R

ate

In this analysis, intervals were centered about the deciles, and observations at 0.05, 0.15, …, 0.95 were split half/half between adjacent intervals.

For example, there were:• 23 observations at 0.45 (7 successes)These were split “evenly” between the

intervals centered at 0.40 & 0.50.

Collecting Nearby Observations into Buckets, view 3

17½

33

49

67

84½

164

2

118½

138½

14

2½

56½

Figure 5

Summary

• The PAG assessments have been remarkably accurate – no matter how the data are grouped the actual results are usually within 5% of the predictions, and never more than 10% away.

• Our experience has shown that a well-planned process for probability assessment can provide executives with reliable measurements of technical feasibility.– A careful consideration of technical feasibility is key to portfolio

management.– Probability is an excellent language for quantifying this

uncertainty.

15