Download - Evaluating research consortium
![Page 1: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/1.jpg)
Evaluating the value of
research-by-consortium
Mark David Lim, PhD
September 4, 2014
![Page 2: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/2.jpg)
actionFasterCures is an “action tank” driven by a singular
goal – to save lives by speeding up and
improving the medical research system.
A center of the Milken Institute, we are a nonprofit
and nonpartisan organization that works with all the
sectors of the medical research and development
ecosystem.
![Page 3: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/3.jpg)
bringing a new discovery from lab to market is a
long, expensive and risky road
![Page 4: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/4.jpg)
innovationMilken Institute partnered with
Sanofi to host an Innovation
Retreat in 2011
Meeting yielded 40+ policy and
R&D recommendations including:
• Open innovation and
cooperation among competitors
• Collaborating in the
precompetitive space
• Defining metrics of success
![Page 5: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/5.jpg)
![Page 6: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/6.jpg)
Shared scientific challenge
Widely-usable tool
Virtual team to create / qualify
research-by-consortium
![Page 7: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/7.jpg)
Academia /
Clinical
IndustryGovernment
Patient
groups
research-by-consortium
Temporary association
of researchers that share
resources and effort for
a common objective.
Consortia integrate
multiple types of
knowledge, data from
multiple sources, and
align different interests.
![Page 8: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/8.jpg)
Sci. Trans. Medicine, June 2014
http://bit.ly/STMConsortia
Operational Framework Landscape
consortiapedia.fastercures.org
• Mission/governance
• Financing
• Data-sharing
• Intellectual property
• and others…
Database
369 consortia
• Disease focus
• Types of tools
• Where and who
• Why
Planned release:
end of 2014
• Consortium-provided
content
• Cross-comparison of
consortia
• Point-of-contact
![Page 9: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/9.jpg)
objectives• Share findings from analysis of
the consortia landscape
• Propose a new framework for
measuring the value of
research-by-consortia efforts
• Have an open dialogue around
the utility and feasibility of
measuring consortia value
![Page 10: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/10.jpg)
Metrics
- collaboration and partnerships
- framework of consortia
What is important to you?
- output, efficiency
![Page 11: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/11.jpg)
Who and what
Sci. Trans. Medicine, June 2014
http://bit.ly/STMConsortia
![Page 12: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/12.jpg)
More than half focused on disease/condition
Sci. Trans. Medicine, June 2014
http://bit.ly/STMConsortia
Sharing comparator arm
data from clinical trials
Research assays,
animal models
Genomic/clinical
databaseT2D patients
AgedBrainSYSBIO
Age-associated
pathways
![Page 13: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/13.jpg)
Breadth-of-scale: Innovative Medicines Initiative
€1 952 573 292
€ 756 906 619
Infectious diseases - 39%
€ 213 636 872
Drug discovery - 11%
€ 186 102 324
Brain disorders - 10 %
€ 118 189 462
Metabolic disorders - 6%
€ 116 287 312
Drug safety - 6%
€ 76 872 548
Stem cells - 4%
€ 74 004 854
Cancer € 74 345 401
Data management 4%
€ 68 069 432
Inflammatory disorders
€ 55 930 954
Biologicals
€ 49 310 000
Geriatrics
€ 39 901 138
Lung diseases
€ 38 994 284
Education and training
€ 30 531 192
Sustainable chemistry
€ 20 462 255
Drug delivery
€ 18 118 249
Drug kinetics
€ 14 910 397
Relative effectiveness
IMI
fundingCorporate
contribution
IMI Report: May 2014 Highlights
![Page 14: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/14.jpg)
Consortium lifespan: 5 - 6 years
Inception
Ramp up
Mid-stream
Wind down
Closure
| 1 year | 2 - 3 years | 1 year |
Scientific challenge
Sponsor engagement
Governance
Agreements
Tool concept
Engaging tool-builders
Project plan
Project launch
Team culture
Infrastructure
Project execution
MilestonesDeliverables
Licensing/IP
Dissemination Data management
Licensing/IP
Dissemination
Royalties
![Page 15: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/15.jpg)
Evaluations should be simple
Hub-and-spoke – central source of information
Innovative Medicines Initiative, Critical Path Initiative,
Foundation for the National Institutes of Health,
Health & Environmental Sciences Institute
Formalized agreements and governance
transparency
Established timelines and milestones
project management
![Page 16: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/16.jpg)
Evaluation = Support
Inception
Ramp up
Mid-stream
Wind down
Closure
| 1 year | 2 - 3 years | 1 year |
Financial and in-kind commitment
Monitoring & EvaluationSteering
Committee
Board of
Directors
![Page 17: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/17.jpg)
Many formal evaluationsSteering Committee Board of Directors
Sponsors Consortium Staff
Research Team
![Page 18: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/18.jpg)
Many informal evaluationsSteering Committee Board of Directors
Sponsors Consortium Staff
Research Team
![Page 19: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/19.jpg)
How do you evaluate
consortia?
![Page 20: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/20.jpg)
What do you value?
Efficiency
- Convening
- Executing
- Managing
- Concluding
Output
- Level of adoption
- Business strategy alignment
- Government roles
- Creating opportunities
- R&D cost/time/efficiency
![Page 21: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/21.jpg)
Output: eye of the beholder
Government • public health
• regulatory science
• de-risk innovation
• economic growth
• state-of-science research guidance documents
Industry • accelerate pipeline
• new therapeutic area
• access resources
• de-risk innovation
• access intellectual
capital
Academia • access resources
• opportunities for
publications
• training opportunities
• identify collaborators
Patient
organizations
• accelerate pipelines
• advance basic
research
• de-risk medical
product development
Consortium
researchers
• simplify day jobs
• access resources
• networking
• training / education
![Page 22: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/22.jpg)
Bibliometrics
• By the end of 2013, IMI projects had
delivered over 600 scientific
publications in over 300 journals
• The citation index of papers from IMI
projects is twice the world average,
and higher than the EU average..
Data & analysis: Thomson Reuters (Custom Analytics & Engineered Solutions), 2013
![Page 23: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/23.jpg)
Bibliometrics and collaboration
Pre IMI funding award Post IMI funding award
Data & analysis: Thomson Reuters (Custom Analytics & Engineered Solutions), 2013
![Page 24: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/24.jpg)
Collaborations – who / what
Co-authorship – 69%Cross-sector collaboration – 42%Cross-project collaboration – 37%Cross-disease collaboration – 31%
IMI researcher networks by sector
Data & analysis: Thomson Reuters (Custom Analytics & Engineered Solutions), 2013
![Page 25: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/25.jpg)
Value of consortiaHow will the output be used? Is consortium on-track?
Therapeutic area core strategy vs opportunistic
Platform methods / tools clinical trials, personalized medicine,
data standards / exchange, assays
Others?
Project Name Outcome Output Area
IMIDIA Smaller clinical trials and Personalized
medicine; Faster development times,
Reduced attrition, and Predictive
models
Biomarkers and
personalized medicine;
Efficacy
Diabetes
COMPACT Faster development times, Reduced
attrition, and Predictive models
Efficacy Biologicals
Safe-T Smaller clinical trials and Personalized
medicine; Faster development times,
Reduced attrition, and Predictive
models
Biomarkers and
personalized medicine
Drug Safety
Examples of IMI consortia
![Page 26: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/26.jpg)
Not all consortium outputs are publishable – licenses, databases
Publications are retrospective, rarely primary/secondary deliverable
Different stakeholders = different expectations on output
Bias: "Sexiness" of the science
Virtual collaborations - no dedicated laboratory/workspace
Semi-committed teams - not their day jobs
Human capital - turnover, advancement
Numerous consortia, different operational models
- cross comparison?
Complexities for evaluation by output
#
![Page 27: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/27.jpg)
What do you value?
Efficiency
- Convening
- Executing
- Managing
- Concluding
Output
- Level of adoption
- Business strategy alignment
- Government roles
- Creating opportunities
- R&D cost/time/efficiency
![Page 28: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/28.jpg)
Evaluating efficiency
Tracking progress - convene to perform
Coordinating virtual teams• Within work streams
• Across work streams
• With governing bodies
Resolving bottlenecks• Maintaining scope
• Appropriate expertise / resources
• Communications
• Conflicts / adaptability
• Team member turnover
![Page 29: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/29.jpg)
Dynamics of teamwork
Phase of Research
Stage of Team
Development
Phase of Team
Adaptation
Wooten, U. Houston, Science of Team Science conference
![Page 30: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/30.jpg)
Phase of Research
Stage of Team
Development
Phase of Team
Adaptation
Development- goals, mission
Conceptualization- research question, framework
Implementation- launch, conduct
Translation- application
Wooten, U. Houston, Science of Team Science conference
Hall et al, Trans Behavioral Med (2012)
![Page 31: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/31.jpg)
Phase of Research
Stage of Team
Development
Phase of Team
Adaptation
Assess situation- recognition
Plan formulation- goal setting, expectations
Plan execution
- monitoring, communication, coordination
Team learning- lessons learned
Wooten, U. Houston, Science of Team Science conference
Burke et al., J. Applied Psychology (2006)
![Page 32: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/32.jpg)
Phase of Research
Stage of Team
Development
Phase of Team
Adaptation
Wooten, U. Houston, Science of Team Science conference
Tuckman & Jensen, Group and Organizational Studies (1977)
Forming- tasks, strategy, team
Storming- roles and interactions
Norming- rules, roles, expectations
Performing- tasks, implementation
Adjourning- finalizing
![Page 33: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/33.jpg)
Tracking consortium progress via metrics
Inception
Ramp up
Mid-stream
Wind down
Closure
Collective orientation
Interpersonal relations
Goal setting
Teamwork concept
Knowledge
consideration
Role clarification
Team subgroups
Cohesion / collective efficacy
Evolved interpersonal
relations
Maintaining shared vision
Problem solving / adaptability
Knowledge accommodation
Evolved role clarification
Autonomy & interdependence
Collective knowledge
transformation
Evolved interpersonal relations
Defining accomplishments
Problem solving
Mediated information
exchange
Autonomy & interdependence
![Page 34: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/34.jpg)
Convene Perform Transition
Collective orientation
Interpersonal relations
Goal setting
Teamwork concept
Knowledge
consideration
Role clarification
Team subgroups
Cohesion / collective efficacy
Evolved interpersonal
relations
Maintaining shared vision
Problem solving / adaptability
Knowledge accommodation
Evolved role clarification
Autonomy & interdependence
Collective knowledge
transformation
Evolved interpersonal relations
Defining accomplishments
Problem solving
Mediated information
exchange
Autonomy & interdependence
Which phase?
![Page 35: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/35.jpg)
Convene Integrate Implement
Collective orientation
Interpersonal relations
Goal setting
Teamwork concept
Knowledge
consideration
Role clarification
Team subgroups
Cohesion / collective efficacy
Evolved interpersonal
relations
Maintaining shared vision
Problem solving / adaptability
Knowledge accommodation
Evolved role clarification
Autonomy & interdependence
Collective knowledge
transformation
Evolved interpersonal relations
Defining accomplishments
Problem solving
Mediated information
exchange
Autonomy & interdependence
Leveraging human capital
![Page 36: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/36.jpg)
Periodic survey of team dynamics
Steering Committee Board of Directors Research Team
Consortium Staff
62%
coherence in mission
35%
contribution
Correctional action:
- Increased face-to-face interaction
- Document-sharing technology
- Conflict resolution
![Page 37: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/37.jpg)
Framework for reports
Sponsors
Operational
efficiency
Alignment to
strategy
Consortium Staff
![Page 38: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/38.jpg)
Metrics = better communication?
• Output - Did the team deliver?
• Technical milestones – binary
• Team dynamics - Could the team have done better?
• Leverage resources and expertise
• Adaptability
Mid-term report overview
Technical progress: 4 / 5 milestones accomplished
Stage of team: Perform
Team integration across disciplines
Document sharing / development
Researcher engagement
Steering committee alignment
Interdependency defined
Conflicts resolved
![Page 39: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/39.jpg)
How was output used?
IMI Executive Office and other consortia
Business strategy
• Open new therapeutic approaches and research avenues
• Reduce R&D costs, time to market and development risk
• Increase the efficacy and/or safety of existing drugs
Indirect benefits
• Education and training
• Spin off companies
• New partners (patients, foundations, academic, SME)
• Increased interest in geographic investments
• Implementation of standards / best practices / tools into strategy
• Informing regulatory science (policy / guidelines)
• Publication output and extent of collaboration
• Intellectual Property metrics
Others
![Page 40: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/40.jpg)
Design• Consortium management
• Consortium participants
• Sponsor/stakeholder
Refine• Other consortia – managers/participants
• Other sponsor within same sector
Pilot• Several consortia
Optimize• Analyze / Evaluate
• Optimize survey vehicles
• Re-pilot
![Page 41: Evaluating research consortium](https://reader034.vdocuments.us/reader034/viewer/2022042816/559a562e1a28abe9788b47b4/html5/thumbnails/41.jpg)
?
Utility• Need for these evaluations?
• Inform best practices?
• Other non-consortium partnerships?
Approach• Right approach?
• Aligning consortia with business strategy necessary
after concept development?
• Generalizable?
• Other key elements to measure?
• Indirect effects?
• Who/how to pilot?
Implementation• How to measure (surveys, etc)?
• Who measures?