involving end users in research proposal evaluation: a case study with the dutch heart foundation
TRANSCRIPT
![Page 1: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/1.jpg)
Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation
Ingeborg MeijerCentre for Science and Technology Studies, Leiden University
QMM Workshop, Brunel University, London1-2 October 2015
![Page 2: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/2.jpg)
2
Outline
• Societal quality: Ensuring use of of research • DHF and the end user panel• Evaluation in practice - triangulation• Methods
– Observation of peer review– Questionnaire– Interview
• Results• Conclusion
![Page 3: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/3.jpg)
Scientific & societal quality
• Economic returns
• Cultural returns
• Social returns
• Scientific returns
Scientific interaction
sProfessional interactions
Private interaction
s
Public interaction
s
3
![Page 4: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/4.jpg)
Science and society
Evaluation scienceMonitoringInvolving End
users
ProgrammingSocietal Demand driven
• The process of value creation is by transferring knowledge from a research institute into society (private or public parties, or general public), and includes (demand-driven, user-inspired) research programming, and interaction with potential users during the research.
![Page 5: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/5.jpg)
5
Dutch Heart Foundation objectives
![Page 6: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/6.jpg)
Dutch Heart Foundation (DHF)• Charity Fund > 50% of budget to scientific research • Mission: 3 times less….
– Less people get a cardiovascular disease;– Less people suffer from cardiovascular disease;– Less people die from cardiovascular disease.
• Change in objective:– Closer connected to donators,– Faster translation from results science to patients– More participation from stakeholders
• Goal: to experiment with processes in the core of the DHF > evaluation of research proposals.• End User panel (EP) & criteria to evaluate
societal aspects
6
![Page 7: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/7.jpg)
7
Societal quality operationalised5 Criteria• Relevance of the health
problem• Contribution of the research to
solving the health problem• To the next step in the
research (or development) process.
• Focus on activities towards (eventual) application of results in healthcare– CV of applicant– Objectives, strategy and
actions• Participation of stakeholders
• Relation between criteria
Relevance
Activity towards actors
Societal quality
Participation
![Page 8: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/8.jpg)
8
Phase 2 Practicing in reality• Separate calls
– Young investigator personal grant– Focused topic: women and cardiovascular disease– Focused topic: congenital heart disease in children
• End user panel evaluation in parallel to scientific evaluation panel
• Different types of information sharing• No formal input of evaluation of societal aspects
on decision by the board of DHF (despite formal advice)
• Evaluation of the whole process by CWTS
![Page 9: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/9.jpg)
9
Overview of calls and EP input
Societal aspects in evaluation process
Personal Women Congenital
CTMM
Information in call text + + + -Intermediate advice to scientists - + - +Final advice to scientists + + + -Advice to ISC as referent - + - -Advice to ISC in person - - + -Interaction with EP on same day as ISC + + NAInteraction with EP on another day as ISC - - + NA
![Page 10: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/10.jpg)
10
Research questions• Is the EP able to play a role in evaluation of
research proposals, and what kind?• Is there a difference in focus between ISC and EP?• How do scientists value the feedback from
external stakeholders?• Are scientists aware of societal relevance or
quality aspects of their work?• What do they think about the aim and set up of
the DHF?
![Page 11: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/11.jpg)
Evaluation in practice: triangulation
11
Questionnaire Interview
EP Societal
ISC
EPISC
Interaction
Observation
![Page 12: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/12.jpg)
12
Observations
![Page 13: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/13.jpg)
Observations of peer review
• Aim: comparing distribution of arguments between ISC and EP
• 2x ISC and 1x EP • Session was chaired• Only arguments
were scored, not final judgement
13
![Page 14: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/14.jpg)
Observations – Scores• Use of observation score form• Scoring arguments
– Positive– Negative– Person– Before or after presentation– Content of remarks
• Counting occurrence of arguments• Limitation: Validation of scores by independent
other; field knowledge
14
![Page 15: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/15.jpg)
15
Result observation Peer review – congenital EP vs ISC
EP most used“Next step” MA3 > positive “Activity” MA4 > negativeOften in combination with: “Feasibility” WA4 positive“Solution” WA2 negativeNegative goes downAP and CL arguments used
ISC most used“Feasibility” WA4 both in positive and negative contextAll others: very limited
![Page 16: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/16.jpg)
16
Result observation Peer review –ISC congenital & women
Comparison two calls:Feasibility is dominant argument in discussionSecondly, the chosen approach (WA2) and workplan (WA5) are discussed most and it relates to how realistic plan isNote: ISC has not been asked to discuss societal aspects, and contribution of EP in person (congenital) has little effect.
![Page 17: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/17.jpg)
17
Preliminary conclusion
• EP and ISC use different arguments, where the EP uses more scientific argumemts than the ISC societal arguments.
• Suggests that EP has a broader vision compared to ISC which has little attention for applicant or context related aspects.
• EP peer review could have added value
![Page 18: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/18.jpg)
18
Questionnaire
![Page 19: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/19.jpg)
Questionnaire• Aimed at scientists involved in session with End
user panel (n=20)• Goal: Establishing perceptions of “societal” and
opinion on capabilities of End User Panel• Three main questions, worked out through
statements to agree upon (4-point Likert scale)– Vision of scientists on the concept of societal quality– Vision of scientists on the role of the End Users and their
feedback– Weighing the contribution of the EP and societal criteria on the
outcome of the evaluation
19
![Page 20: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/20.jpg)
20
![Page 21: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/21.jpg)
21
Q: What is societal quality? Yes, agree No, don’t agree
Process of knowledge valorisation leads to societal benefit.
I think that societal benefit is mainly related to communication with users.
Societal benefit has social, economic and cultural aspects
I think that valorisation has mainly to do with economic innovation (some doubt)
Societal quality of research relates to value creation through the interaction of the research group with the societal environment
I think that societal quality of research is independent of the scientific quality and interaction with fellow scientists.
Societal quality is related to focused activities towards diverse users.
Paying attention to societal quality in an early stage of a research project increases the chance that results are indeed transferred to the right actor much quicker.
It is important that scientists take into account the context of users, and adjust the way they transfer knowledge to that user.
![Page 22: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/22.jpg)
22
Q: Contribution of EP to assessmentStatement All
(median 2)
personal EP
The judgement of my scientific peers on my proposal is sufficient for me.
2,5 3 2 (2)
I understand that when a proposal is scientifically ‘state of the art’, it may be rejected based on insufficient societal quality.
2,32 1,75 2,63 (3)
I understand that only proposal that have sufficient societal quality proceed to scientific evaluation process.
2,21 1,75 2,5 (2)
I think that scientific and societal quality should count equally
2,18 1,5 2,38 (2)
I find all this attention for societal quality or benefit unnecessary and too much
1,95 2,75 2,25 (2)
![Page 23: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/23.jpg)
Preliminary conclusion• Scientists and EP largely agree on what societal quality is. • Young researchers have less interest in application of
knowledge, don’t want to pay attention to it in an early stage, and think it has mainly to do with communication.
• Societal quality cannot be leading in deciding what proposals to fund (scientists and EP)
• Method wise, also look at distribution of answers since n=low
23
![Page 24: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/24.jpg)
24
Interviews
![Page 25: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/25.jpg)
Interviews – semi-structured• N=20• 3 topics:
– Preparing for the interaction with EP– The actual interaction with the EP– Feedback from EP
• Interview usually 30 minutes, after questionnaire• All comments collected, not counted, broad
picture
25
![Page 26: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/26.jpg)
26
Interviews – snapshot of results
PresentationProfessionals in EPRole in decision
PositiveNext step
CommunicationCareer/grant
Feedback vagueNot transparentPersonal grants
![Page 27: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/27.jpg)
27
Mixed method message
![Page 28: Involving end users in research proposal evaluation: A case study with the Dutch Heart Foundation](https://reader035.vdocuments.us/reader035/viewer/2022070523/58ed00621a28ab10768b45f9/html5/thumbnails/28.jpg)
28
Preliminary conclusion• The contribution, assessment, and feedback of the
EP has added value, but in general scientists do not hear anything new. It is a different emphasis.
• Scientsist think that End users have a different view, which is true in some aspects but not in others.
• Interaction EP with ISC could be valuable• Young scientists: different approach necessary
because of funding pressure and scientific careers (MM)
• Procedure needs to be transparent• Feedback from EP more explicit