evaluating research consortium

Evaluating the value of

research-by-consortium

Mark David Lim, PhD

September 4, 2014

actionFasterCures is an “action tank” driven by a singular

goal – to save lives by speeding up and

improving the medical research system.

A center of the Milken Institute, we are a nonprofit

and nonpartisan organization that works with all the

sectors of the medical research and development

ecosystem.

bringing a new discovery from lab to market is a

long, expensive and risky road

innovationMilken Institute partnered with

Sanofi to host an Innovation

Retreat in 2011

Meeting yielded 40+ policy and

R&D recommendations including:

• Open innovation and

cooperation among competitors

• Collaborating in the

precompetitive space

• Defining metrics of success

Shared scientific challenge

Widely-usable tool

Virtual team to create / qualify

research-by-consortium

Academia /

Clinical

IndustryGovernment

Patient

groups

research-by-consortium

Temporary association

of researchers that share

resources and effort for

a common objective.

Consortia integrate

multiple types of

knowledge, data from

multiple sources, and

align different interests.

Sci. Trans. Medicine, June 2014

http://bit.ly/STMConsortia

Operational Framework Landscape

consortiapedia.fastercures.org

• Mission/governance

• Financing

• Data-sharing

• Intellectual property

• and others…

Database

369 consortia

• Disease focus

• Types of tools

• Where and who

• Why

Planned release:

end of 2014

• Consortium-provided

content

• Cross-comparison of

consortia

• Point-of-contact

objectives• Share findings from analysis of

the consortia landscape

• Propose a new framework for

measuring the value of

research-by-consortia efforts

• Have an open dialogue around

the utility and feasibility of

measuring consortia value

Metrics

- collaboration and partnerships

- framework of consortia

What is important to you?

- output, efficiency

Who and what

Sci. Trans. Medicine, June 2014

http://bit.ly/STMConsortia

More than half focused on disease/condition

Sci. Trans. Medicine, June 2014

http://bit.ly/STMConsortia

Sharing comparator arm

data from clinical trials

Research assays,

animal models

Genomic/clinical

databaseT2D patients

AgedBrainSYSBIO

Age-associated

pathways

Breadth-of-scale: Innovative Medicines Initiative

€1 952 573 292

€ 756 906 619

Infectious diseases - 39%

€ 213 636 872

Drug discovery - 11%

€ 186 102 324

Brain disorders - 10 %

€ 118 189 462

Metabolic disorders - 6%

€ 116 287 312

Drug safety - 6%

€ 76 872 548

Stem cells - 4%

€ 74 004 854

Cancer € 74 345 401

Data management 4%

€ 68 069 432

Inflammatory disorders

€ 55 930 954

Biologicals

€ 49 310 000

Geriatrics

€ 39 901 138

Lung diseases

€ 38 994 284

Education and training

€ 30 531 192

Sustainable chemistry

€ 20 462 255

Drug delivery

€ 18 118 249

Drug kinetics

€ 14 910 397

Relative effectiveness

IMI

fundingCorporate

contribution

IMI Report: May 2014 Highlights

Consortium lifespan: 5 - 6 years

Inception

Ramp up

Mid-stream

Wind down

Closure

| 1 year | 2 - 3 years | 1 year |

Scientific challenge

Sponsor engagement

Governance

Agreements

Tool concept

Engaging tool-builders

Project plan

Project launch

Team culture

Infrastructure

Project execution

MilestonesDeliverables

Licensing/IP

Dissemination Data management

Licensing/IP

Dissemination

Royalties

Evaluations should be simple

Hub-and-spoke – central source of information

Innovative Medicines Initiative, Critical Path Initiative,

Foundation for the National Institutes of Health,

Health & Environmental Sciences Institute

Formalized agreements and governance

transparency

Established timelines and milestones

project management

Evaluation = Support

Inception

Ramp up

Mid-stream

Wind down

Closure

| 1 year | 2 - 3 years | 1 year |

Financial and in-kind commitment

Monitoring & EvaluationSteering

Committee

Board of

Directors

Many formal evaluationsSteering Committee Board of Directors

Sponsors Consortium Staff

Research Team

Many informal evaluationsSteering Committee Board of Directors

Sponsors Consortium Staff

Research Team

How do you evaluate

consortia?

What do you value?

Efficiency

- Convening

- Executing

- Managing

- Concluding

Output

- Level of adoption

- Business strategy alignment

- Government roles

- Creating opportunities

- R&D cost/time/efficiency

Output: eye of the beholder

Government • public health

• regulatory science

• de-risk innovation

• economic growth

• state-of-science research guidance documents

Industry • accelerate pipeline

• new therapeutic area

• access resources

• de-risk innovation

• access intellectual

capital

Academia • access resources

• opportunities for

publications

• training opportunities

• identify collaborators

Patient

organizations

• accelerate pipelines

• advance basic

research

• de-risk medical

product development

Consortium

researchers

• simplify day jobs

• access resources

• networking

• training / education

Bibliometrics

• By the end of 2013, IMI projects had

delivered over 600 scientific

publications in over 300 journals

• The citation index of papers from IMI

projects is twice the world average,

and higher than the EU average..

Data & analysis: Thomson Reuters (Custom Analytics & Engineered Solutions), 2013

Bibliometrics and collaboration

Pre IMI funding award Post IMI funding award

Data & analysis: Thomson Reuters (Custom Analytics & Engineered Solutions), 2013

Collaborations – who / what

Co-authorship – 69%Cross-sector collaboration – 42%Cross-project collaboration – 37%Cross-disease collaboration – 31%

IMI researcher networks by sector

Data & analysis: Thomson Reuters (Custom Analytics & Engineered Solutions), 2013

Value of consortiaHow will the output be used? Is consortium on-track?

Therapeutic area core strategy vs opportunistic

Platform methods / tools clinical trials, personalized medicine,

data standards / exchange, assays

Others?

Project Name Outcome Output Area

IMIDIA Smaller clinical trials and Personalized

medicine; Faster development times,

Reduced attrition, and Predictive

models

Biomarkers and

personalized medicine;

Efficacy

Diabetes

COMPACT Faster development times, Reduced

attrition, and Predictive models

Efficacy Biologicals

Safe-T Smaller clinical trials and Personalized

medicine; Faster development times,

Reduced attrition, and Predictive

models

Biomarkers and

personalized medicine

Drug Safety

Examples of IMI consortia

Not all consortium outputs are publishable – licenses, databases

Publications are retrospective, rarely primary/secondary deliverable

Different stakeholders = different expectations on output

Bias: "Sexiness" of the science

Virtual collaborations - no dedicated laboratory/workspace

Semi-committed teams - not their day jobs

Human capital - turnover, advancement

Numerous consortia, different operational models

- cross comparison?

Complexities for evaluation by output

#

What do you value?

Efficiency

- Convening

- Executing

- Managing

- Concluding

Output

- Level of adoption

- Business strategy alignment

- Government roles

- Creating opportunities

- R&D cost/time/efficiency

Evaluating efficiency

Tracking progress - convene to perform

Coordinating virtual teams• Within work streams

• Across work streams

• With governing bodies

Resolving bottlenecks• Maintaining scope

• Appropriate expertise / resources

• Communications

• Conflicts / adaptability

• Team member turnover

Dynamics of teamwork

Phase of Research

Stage of Team

Development

Phase of Team

Adaptation

Wooten, U. Houston, Science of Team Science conference

Phase of Research

Stage of Team

Development

Phase of Team

Adaptation

Development- goals, mission

Conceptualization- research question, framework

Implementation- launch, conduct

Translation- application

Wooten, U. Houston, Science of Team Science conference

Hall et al, Trans Behavioral Med (2012)

Phase of Research

Stage of Team

Development

Phase of Team

Adaptation

Assess situation- recognition

Plan formulation- goal setting, expectations

Plan execution

- monitoring, communication, coordination

Team learning- lessons learned

Wooten, U. Houston, Science of Team Science conference

Burke et al., J. Applied Psychology (2006)

Phase of Research

Stage of Team

Development

Phase of Team

Adaptation

Wooten, U. Houston, Science of Team Science conference

Tuckman & Jensen, Group and Organizational Studies (1977)

Forming- tasks, strategy, team

Storming- roles and interactions

Norming- rules, roles, expectations

Performing- tasks, implementation

Adjourning- finalizing

Tracking consortium progress via metrics

Inception

Ramp up

Mid-stream

Wind down

Closure

Collective orientation

Interpersonal relations

Goal setting

Teamwork concept

Knowledge

consideration

Role clarification

Team subgroups

Cohesion / collective efficacy

Evolved interpersonal

relations

Maintaining shared vision

Problem solving / adaptability

Knowledge accommodation

Evolved role clarification

Autonomy & interdependence

Collective knowledge

transformation

Evolved interpersonal relations

Defining accomplishments

Problem solving

Mediated information

exchange

Autonomy & interdependence

Convene Perform Transition

Collective orientation

Interpersonal relations

Goal setting

Teamwork concept

Knowledge

consideration

Role clarification

Team subgroups

Cohesion / collective efficacy

Evolved interpersonal

relations

Maintaining shared vision

Problem solving / adaptability

Knowledge accommodation

Evolved role clarification

Autonomy & interdependence

Collective knowledge

transformation

Evolved interpersonal relations

Defining accomplishments

Problem solving

Mediated information

exchange

Autonomy & interdependence

Which phase?

Convene Integrate Implement

Collective orientation

Interpersonal relations

Goal setting

Teamwork concept

Knowledge

consideration

Role clarification

Team subgroups

Cohesion / collective efficacy

Evolved interpersonal

relations

Maintaining shared vision

Problem solving / adaptability

Knowledge accommodation

Evolved role clarification

Autonomy & interdependence

Collective knowledge

transformation

Evolved interpersonal relations

Defining accomplishments

Problem solving

Mediated information

exchange

Autonomy & interdependence

Leveraging human capital

Periodic survey of team dynamics

Steering Committee Board of Directors Research Team

Consortium Staff

62%

coherence in mission

35%

contribution

Correctional action:

- Increased face-to-face interaction

- Document-sharing technology

- Conflict resolution

Framework for reports

Sponsors

Operational

efficiency

Alignment to

strategy

Consortium Staff

Metrics = better communication?

• Output - Did the team deliver?

• Technical milestones – binary

• Team dynamics - Could the team have done better?

• Leverage resources and expertise

• Adaptability

Mid-term report overview

Technical progress: 4 / 5 milestones accomplished

Stage of team: Perform

Team integration across disciplines

Document sharing / development

Researcher engagement

Steering committee alignment

Interdependency defined

Conflicts resolved

How was output used?

IMI Executive Office and other consortia

Business strategy

• Open new therapeutic approaches and research avenues

• Reduce R&D costs, time to market and development risk

• Increase the efficacy and/or safety of existing drugs

Indirect benefits

• Education and training

• Spin off companies

• New partners (patients, foundations, academic, SME)

• Increased interest in geographic investments

• Implementation of standards / best practices / tools into strategy

• Informing regulatory science (policy / guidelines)

• Publication output and extent of collaboration

• Intellectual Property metrics

Others

Design• Consortium management

• Consortium participants

• Sponsor/stakeholder

Refine• Other consortia – managers/participants

• Other sponsor within same sector

Pilot• Several consortia

Optimize• Analyze / Evaluate

• Optimize survey vehicles

• Re-pilot

?

Utility• Need for these evaluations?

• Inform best practices?

• Other non-consortium partnerships?

Approach• Right approach?

• Aligning consortia with business strategy necessary

after concept development?

• Generalizable?

• Other key elements to measure?

• Indirect effects?

• Who/how to pilot?

Implementation• How to measure (surveys, etc)?

• Who measures?

Seeking insights & expertise

consortiapedia.fastercures.org

mlim@fastercures.org