stakeholder inv olvement in kce working esses...pres ceo insur pres food presi pres secu gene medi...
TRANSCRIPT
2012
KCE REPORTS
STAKEPROC
174C
EHOLDESSES
ER INV
VOLVEMMENT INN KCE WWORKINNG
www.kce.fgoov.be
Belgian He
Executive B
ealth Care K
Board
Knowledge CThe Bof DeKCE health PresCEOInsurPresFoodpresiPresSecuGeneMediRepr Repr Repr Interm ProfephysProfenurseHosp Socia Hous
Centre Belgian Health Cecember 2002 un
is in charge of h insurance
ident O - National Institrance (vice presidident of the Fed
d Chain Safetyident) ident of the Fed
urity (vice presideneral Administratoricines and Health resentatives of the
resentatives of the
resentatives of the
mutualistic Agenc
essional Organisicians
essional Organises pital Federations
al Partners
se of Representat
are Knowledge Cder the supervisconducting stu
tute for Health andent) deral Public Servy and Environm
deral Public Sernt) r of the Federal Products
e Minister of Publi
e Minister of Socia
e Council of Minis
cy
sations - represe
sations - represe
tives
Centre (KCE) is asion of the Ministdies that suppo
AcPi
nd Disability Jo
vice Health, ment (vice
Di
rvice Social Fr
Agency for Xa
ic Health BeM
al Affairs ORi
sters JeDaMPaXa
entatives of MJe
entatives of MMJoJeRiPaLi
an organization ter of Public Hea
ort the political d
tual Members ierre Gillet o De Cock
irk Cuypers
rank Van Massen
avier De Cuyper
ernard Lange arco Schetgen liver de Stexhe i De Ridder ean-Noël Godin aniel Devos ichiel Callens atrick Verertbruggavier Brenez arc Moens
ean-Pierre Baeyenichel Foulon yriam Hubinon
ohan Pauwels ean-Claude Praetita Thys aul Palsterman eve Wierinck
of public interesalth and Social Adecision making
Substit Benoî
Chris
hove Jan B
Greet
Franç Annic Karel Lambe Frédé Bart O Frank
gen Yolan Geert
ns Rolan
Rita C Ludo
Olivie Katrie Pierre Leo N Celien
st, created on theAffairs. g on health care
tute Members
ît Collin
Decoster
ertels
Musch
çois Perl k Poncé Vermeyen ert Stamatakis
éric Lernoux Ooghe k De Smet de Husden Messiaen
nd Lemye Cuypers Meyers r Thonon
en Kesteloot e Smiets Neels n Van Moerkerke
e 24th
e and
Control
Manageme
Contact
ent
Gove
ChiefAssisMana
Belgi
DoorbBouleB-100Belgi
T +32
F +32
info@
http://
ernment commissi
f Executive Officestant Chief Executagers Program Ma
an Health Care K
building (10th Floevard du Jardin B00 Brussels um
2 [0]2 287 33 88
2 [0]2 287 33 85
@kce.fgov.be
/www.kce.fgov.be
ioner
r tive Officer anagement
Knowledge Centre
or) otanique, 55
e
Yv
RaJeChKr
(KCE).
ves Roger
af Mertens ean-Pierre Closonhristian Léonard ristel De Gauquierr
2012
KCE REPORTS METHOD
STAKEPROC
JULIEN PIERA
174C
EHOLDESSES
ART, CHRISTIAN
ER INV
LEONARD, PATR
VOLVEM
RICE CHALON, F
MENT IN
FRANÇOIS DAUE
N KCE W
E, RAF MERTENS
WORKIN
S
NG
www.kce.fgoov.be
COLOPHOTitle :
Authors :
Acknowledgem
External Valida
Conflict of intere
Layout :
Disclaimer :
Publication date
Domain:
MeSH:
NLM Classificat
Language:
Format:
ON
ents:
tors :
est :
e:
tion:
Stake
JulieHealt
Laure
Bert Sant
None
Ine V
The comnece
Subsa corepo
Fina
Onlyalso
Marc
Meth
Stake
Q 18
Engli
Adob
eholder Involveme
n Piérart (KCE), Cthcare Projects), R
ence Kohn (KCE)
Boer (College voé Publique - UCL
e Declared
Verhulst, Sophie V
external expertments were disc
essarily agree wi
sequently, a (finansensus or a vo
ort and did not ne
lly, this report ha
y the KCE is respunder the full re
ch 28th 2012
hods
eholder involveme
80.55
ish
be® PDF™ (A4)
ent in KCE workin
Christian Léonard Raf Mertens (KCE
)
oor Zorgverzeker)
Vaes
ts were consultcussed during mth its content.
al) version was soting process beecessarily all thr
as been approve
ponsible for erroesponsibility of th
ent ; Organization
ng processes
(KCE), Patrice X E)
ringen, Netherland
ted about a (pmeetings. They
submitted to theetween the validaee agree with its
ed by common as
ors or omissionshe KCE
nal Objectives
Chalon (KCE), F
ds), François Mé
preliminary) versdid not co-auth
e validators. The ators. The validas content.
ssent by the Exe
that could pers
rançois Daue (Op
élard (ULg), Alain
sion of the sciehor the scientific
validation of theators did not co-
ecutive Board.
ist. The policy re
ptifa - Optimum fo
n Deccache (Ecol
entific report. Tc report and did
e report results f-author the scien
ecommendations
r
le de
Their d not
from ntific
s are
Legal depot:
Copyright:
How to refer to
this document ?
D/20
KCE http:/
Pieraproce174C
This
12/10273/11
reports are publis//kce.fgov.be/cont
art J, Leonard Cesses. Method. B
C. D/2012/10.273/
document is avai
shed under a “by/tent/about-copyrig
, Chalon P, DauBrussels: Belgian/11
lable on the webs
/nc/nd” Creative Cghts-for-kce-repor
ue F, Mertens R Health Care Kn
site of the Belgian
Commons Licencets.
. Stakeholder Invnowlegde Centre
Health Care Kno
e
volvement in KC(KCE). 2012. KC
owledge Centre
CE working CE reports
KCE Reports 17
FORE
74C
EWORD
In 20KCEwhichcommclose
On thfield,agenprojestudi
We anecetakencomealternelderthe fi
In thikey eKCEpracttechnOthe
But methdefinwe cstudy
Jean
Assis
S
009 the KCE com’s scientific excellh progress couldmunication with the involvement in th
he one hand there by involving the
nda. On the otheects as well couldes.
are now over two essary emphasis n, including a newes to stakeholdenative medicine, orly, to name but aield.
is report we want elements of stake’s specific duties tical “process noniques. The first
ers will follow in th
the essence of hodological handbnes the KCE’s tascannot disregard wy provides a majo
n-Pierre CLOSON
stant Managing D
Stakeholder Involv
mmissioned an exence and credibil
d still be made. he target groups che KCE study pro
e was the concernstakeholders earlr hand it appeare
d contribute to a
years further dowon these issues.
w editorial approar involvement, won burnout amonga few, each had a
to place this appeholder involvemeand possibilities.tes” on the concof these processe coming years.
an effective invbooks. What is bsk on the basis of what all those othr impetus for this
irector
vement
xternal audit of itsity were generallyFor example, th
could be further imocesses.
n to have the subjly in the projects, ed that a good inbetter impact of t
wn the line. The neIn the field of c
ach to the studieswe haven’t been g general practitioa sizeable compo
proach within a moent and reflect on The document a
crete terms and notes deals with
volvement of thebasically involved
social relevance her players in the approach.
s impact on healty recognised, the he interviewed stmproved, and they
jects under study and in particular
nteraction with ththe messages an
ew management communication, a s and a thorough
sitting around doners and on the nent of intensive
ore scientific and n how these can aims to be the refconditions governh qualitative meth
e interested parti is a vision and and added valuehealth system ha
thcare policy in thaudit pointed outtakeholders werey also clearly expr
better attuned toin the drafting of
he target groups nd recommendatio
plan of 2010 of cohost of initiative
revamping of thedoing nothing, eit
future need for reinteraction with th
conceptual framebest be structure
ference frameworning specific stakhods, and is curre
ies is much mo organisational c
e. And when bringave to say. We the
he country. Whilet a number of field
e of the opinion ressed a call for m
the complaints inthe specific resein the final phas
ons formulated in
ourse also placeds have already b
e website. But whther. The studiesesidential care forhe groups involve
ework. We look ated and geared tork for a series of keholder involvemently being prepa
ore than a seriesculture that expreging this into pracerefore hope that
Raf MERTENS
Managing Directo
i
e the ds in that
more
n the arch
se of n the
d the been en it s on r the ed in
t the o the very
ment ared.
s of essly ctice, t this
or
ii
SUMMMARY
SStakeholder Involv
Sths
Inpsdis
Thasdin“wla
BIncthebere
Wrescs
vement
Stakeholder involvhat are potentiallysince the KCE was
n the vast majorityprojects, care prostudy subject madistinction betwees not always easy
The involvement however, has beea comprehensive stakeholders in odetailed methodonvolvement into pwhen” and “why”atter point will be
BASIC PRIn the second chconceptual and emhe specialist liteexperiences from be meaningful aexperiences of oesearch.
We first outline thesearch field. Nex
stakeholder involvcompletion of a studies:
vement, i.e. the ay or actually affecs set up.
y of the KCE projeoviders and polictter on a day-to-n their contributio
y.
of other groupsn much less systereflection on wheour processes,
ological handbookpractice. This repo”, without going inthe subject of a s
INCIPLESapter we look at
mpirical standpoinerature in the fother sectors whnd relevant. In
other agencies a
he origin of stakext, we examine th
vement in greater project and diss
active involvemencted by our studi
ects, and in the vacy-makers who h-day basis will b
on as an expert ve
than the providematic. Hitherto wen and why best neither have weks on how bestort focuses primanto depth on the eries of methodol
t stakeholder invnt. In the former wield, and in the
here stakeholder so doing we d
and players, but
holder involvemehe various aspects
depth, from the isemination of the
KCE Reports 1
nt of groups or pes, has been fos
arious phases of tave dealings wit
be consulted. Yetersus as a stakeh
ders or policy-mawe have not carrie
to involve the vae developed spet to put this kin
arily on the questimatter of “how”.
ogical process no
olvement from bwe based ourselvee latter on publinvolvement provraw not only on
also on sociolo
ent in the public hs of and approachnitial phase throue results. This r
174C
eople stered
those h the t, the
holder
akers, ed out arious ecific, nd of ion of . This otes.
oth a es on ished
ved to n the ogical
health hes to ugh to report
KCE Reports 17
• The possibgenuine “partners in
• How to ide
• The imporphases of a
• The challen
• The skillsinvolvemen
FROM THRESEARIn this methodostakeholders inopportunities anbased themselvdown during foresearchers tomoderator.
Among other ttraining, and aneeds of each one that was nwas that the KCinvolvement rerespect.
74C
ble objectives, fro“democratisation”the study who ac
ntify and describe
rtance of open a project;
nges and risks pre
s and principlent.
HE ANGLECHERS
ological research pn past and currend success factorves here on their ur discussion foraok part and the
things, the KCE n adapted way oproject. Patients
not always easy CE ought not to loquired time and
om mere informa, whereby the
ctively contribute t
e stakeholders;
communication
esented by contro
s needed for
E OF THE K
project we have aent KCE projectsrs according to thown experience,
a in October and e meetings were
experts pointed of involving stakes were named as
to approach. Onose its independe
could only occu
S
tion gathering thrstakeholders
to the results;
and transparenc
oversial subjects;
successful stak
KCE
also looked at the s, and at the obe KCE researcheand their view waNovember 2011. chaired by an
to the need for eholders accordins an important groe oft expressed nce, and that stakr in a climate of
Stakeholder Involv
rough to become
cy in all
keholder
place of bstacles, rs. They as noted All KCE external
suitable g to the oup, but concern keholder f mutual
HIKOpeit
Wins
1
2
3
vement
HOW TO PNVOLVEMKCE? On the basis of thepresents a numbeensuring a more ets projects.
We identified five mnvolvement couldsuccessive phases
. Make the subis useful to faThis can be patients and cmatter on a da
2. Define the rito address thseen by the cwe must try towill help us values, and sgo into certaparticipative fothe project amunderstandingimplications.
3. Ensure the sinsofar as is approach. If nuseful for obta
PUT STAKEMENT INTO
e above-mentioneer of approaches teffective involvem
major strategic obd be useful. Bys of a research pr
bject tangible. Bemiliarise oneself wdone, for exam
care providers whaily basis.
ght scope and re really importan
care providers, pao expose the undto understand th
sometimes also thain study questioormulation of thesmong the end usg of the regula
study structure apossible, that the
necessary, add infaining a broad vie
EHOLDER O PRACTIC
ed insights, the lasthat could increas
ment of various re
bjectives in respecy and large, theroject:
efore commencinwith the subject aple, by asking f
ho are confronted
research questiot problems and c
atients, the publicderlying problems he more deep-sehe reasons for a ons in greater dse questions can sers. It is also imatory context, in
and methods are e main stakeholdformation sources
ew.
CE AT THE
st chapter of the rse the KCE’s impaelevant stakeholde
ct of which stakehey correspond to
g a research projand to make it tanfor the contributio
with the study su
ons. If our studiesconcerns, as thesc and/or policy-ma
or controversieseated motivationspossible resistan
depth. A carefulalso create suppo
mportant to have ncluding its fina
acceptable. Seeders accept the Ks or data that mig
iii
report act by ers in
holder o the
ect, it gible. on of ubject
s aim se are akers, . This s and nce to l and ort for a full
ancial
e to it, KCE’s ght be
iv
4. Arrive at good view to be ableclear and erealistic, anleast in the
5. Find effecto obtain thof the repo
These objectivevalues in our objectivity, mod
In practice, theinvolvement. Thto the above-minterlocutors. Ccorresponding the results we e
The report endsinto formal KCEinto the KCE’s r
clear-cut resultsof contrasting sta
e to draft your owexplicit terms. Evend therefore also e long term.
ctive communicahat the target gro
orts.
es can only be acontacts with t
desty and sincere
ere is no single she question has mentioned object
Consequently, we to the different s
expect it to yield.
s with a list of tecE process notes, research methods
s and realistic aandpoints and thewn conclusions aery recommendatifeasible – perhap
ation channels aoups themselves b
achieved if we bethe stakeholderscuriosity.
standard way of inumerous dimenstives, the nature will have to deve
situations of stak
chniques that neewhich, ultimately
s.
S
answers. Try to oe main argumentand recommendaion should in any ps not immediatel
and spokespersobecome active ad
ear in mind a nu: respect, transp
mplementing staksions that vary ac
of the project elop an array of mkeholder involvem
d to be further dey, should be inco
Stakeholder Involv
obtain a ts, so as ations in case be y, but at
ons. Try dvocates
umber of parency,
keholder ccording and the
methods, ment and
eveloped rporated
vement KCE Reports 1174C
KCE Reports 17
SCIETABLE O
74
NTIFIC ROF CONTEN
REPORTNTS
LI1.1.1.1.2.2.
2.
2.
2.
2.
S
T ST OF ABBREVI
RATIONA1. POLICY C2. SCIENTIF3. STRUCTU
KEY PRIN1. THE RISE
2.1.1. W2.1.2. T2.1.3. T2.1.4. W2.1.5. C2.1.6. W
2. STAKEHO2.2.1. E2.2.2. B2.2.3. O
3. COMMUN2.3.1. A2.3.2. A
4. STAKEHO2.4.1. R2.4.2. R2.4.3. C
5. DIALOGU
Stakeholder Involv
IATIONS .............ALE OF STAKEHOCHALLENGES ANFIC CHALLENGESURE OF THIS STUNCIPLES OF STAE OF STAKEHOLDWho are stakeholdThe circle of stakeThe patient and theWhy stakeholder inCriteria of effectiveWhat are the key cOLDER IDENTIFICExplore the world :Be prepared beforeOrganize your stakNICATION AT AN Analyse the risks aAnticipate time-conOLDER CONSULTRefer to standardsRefine your procesChoose your methE IN CONTROVE
vement
............................OLDER INVOLVEND OBJECTIVESS AND OBJECTIVUDY REPORT ....
AKEHOLDER INVDER THINKING ..ders? ...................holders ...............e citizen ..............nvolvement? .......e stakeholder invocomponents of staCATION AND AN: from unfolding coe the journey ......keholder identificaEARLY STAGE ..
and benefits linkednsuming communTATION ALL ALOs ...........................ss ........................od .......................
ERSIES ...............
............................EMENT AT KCE ..............................VES .................................................
VOLVEMENT ...................................................................................................................................................
olvement ..............akeholder involvemALYSIS ...............omplexity to orde............................
ation and analysis............................d to communicatio
nication activities ..ONG THE PROJE
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................ment? .............................................ring complexity ...............................
s .......................................................on ....................................................CT DEVELOPME................................................................................................................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................ENT PROCESS ...................................................................................................................
1
...... 7
...... 8
...... 8
...... 8
...... 8
...... 9
...... 9
.... 11
.... 12
.... 13
.... 14
.... 15
.... 15
.... 16
.... 17
.... 20
.... 20
.... 25
.... 25
.... 27
.... 27
.... 27
.... 28
.... 29
.... 34
2
2.
2.
2.
2.
2.3.3.3.3.
3.
S
2.5.1. D2.5.2. D2.5.3. D
6. MANAGEM2.6.1. A2.6.2. U
7. STAKEHO2.7.1. C2.7.2. C
8. REGULAR2.8.1. P2.8.2. D2.8.3. A2.8.4. U2.8.5. G
9. SKILLS TO2.9.1. B2.9.2. B2.9.3. B
10. GUIDING CONSULT
1. RATIONA2. METHOD3. RESULTS
3.3.1. L3.3.2. R3.3.3. P3.3.4. T
4. CONCLUS
Stakeholder Involv
Dialogue beyond eDialogue to producDialogue beyond cMENT OF STAKE
Acknowledge comUse forms and foruOLDER INVOLVEConsider added-vaConsider the way tR FEED-BACK TOPlan your communDisplay relevant inAdapt your feed-baUse electronic medGive regular feed-bO MANAGE THE
Be trustworthy andBe involved ...........Be expert ..............
PRINCIPLES ......TATION ROUND LE AND OBJECT............................
S ...........................Lessons learned ...Requests ..............Possible avenues .Treats ...................SION ....................
vement
effectiveness .......ce knowledge syncontroversies .......EHOLDERS’ COMment and respondums .....................MENT IN PROJEalues to involve stto deal with controO STAKEHOLDERnication ................formation ............ack to the level of dia and meetingsback ....................PROCESS OF S
d respectful ..............................................................................................OF KCE EXPERT
TIVES ......................................................................................................................................................................................................................
............................nthesis ..............................................
MMENTS ..............d ......................................................
ECT MONITORINGtakeholders in prooversy..................RS ...............................................................................
f stakeholder invo........................................................TAKEHOLDER IN................................................................................................................TS .......................................................................................................................................................................................................................................................
............................
............................
............................
............................
............................
............................G .........................oject monitoring ...................................................................................................................lvement ......................................................................NVOLVEMENT ...............................................................................................................................................................................................................................................................................................................................................................................
KCE Reports
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
............................
s 174
.... 34
.... 36
.... 37
.... 37
.... 37
.... 39
.... 39
.... 39
.... 42
.... 43
.... 43
.... 43
.... 43
.... 43
.... 43
.... 44
.... 44
.... 45
.... 46
.... 46
.... 47
.... 47
.... 47
.... 48
.... 48
.... 49
.... 50
.... 51
.... 52
KCE Reports 17
74
4.4.
4.4.4.4.
5.
6.
S
STAKEHO1. RATIONA
4.1.1. Im4.1.2. F
2. THE FIVE3. THERE AR4. FROM ST5. STAKEHO
4.5.1. P4.5.2. C4.5.3. R4.5.4. FAPPENDIXEXPERTSREFEREN
Stakeholder Involv
OLDER INVOLVELE : ENHANCINGmpact of the KCE Five strategic obje KEY VALUES FORE MANY DIFFE
TRATEGY TO COOLDER INVOLVEPrerequisites ........Choice of the mostRequired competeFurther elaborationX: APPROACHE
S ...........................NCES ...................
vement
EMENT IN KCE PG THE RELEVAN
reports: levers anctives to overcomOR FRUITFUL STRENT WAYS TO NCRETE ACTIONMENT IN PRACT............................t appropriate meth
encies ..................n of methods that S ILLUSTRATED........................................................
ROCESSES .......CE AND IMPACTnd barriers ...........
me the impact barrTAKEHOLDER ININVOLVE STAKE
NS .......................TICE .................................................hod(s) ..............................................were considered
D DURING THE C........................................................
............................T OF KCE REPOR............................riers and operate tNTERACTION .....EHOLDERS ....................................................................................................................................................as eligible for KC
CONSULTATION ........................................................
............................RTS .................................................the levers ................................................................................................................................................................................................................
CE .........................ROUND OF KCE........................................................
3
.... 53
.... 53
.... 53
.... 54
.... 55
.... 55
.... 56
.... 59
.... 59
.... 59
.... 59
.... 59 E .... 60 .... 70
4
LIST OF F
FIGURES FiFiFiFiFiweFiFiFiFiFiFiFiFiFiFiFiFi
S
gure 1: Stakeholdgure 2: Bridging bgure 3: Key compgure 4: Primary idgure 5: Actors-neebsites links ........gure 6 : Actors-negure 7: Timeline ogure 8 : Example gure 9: Core elemgure 10 : Health Cgure 11: The NICgure 12: The IQWgure 13: The VIKCgure 14: The Cocgure 15: Comparagure 16: HAS progure 17: The ena
Stakeholder Involv
der involvement inbetween decision-ponents of stakehodentification of staetworks diagram, c............................etwork diagram, reof a KCE project ..of table of interpr
ments of a partnerCanada’s Public I
CE key stages of cWIG production proC key stages of g
ck Burning & Honiative chart for par
ocedure followed iblers/resistors spe
vement
n research-use the-making and reseaolder involvement
akeholders in legaconnections betwe............................epresentation of B............................retive frames of acrship process .......nvolvement Conti
clinical developmeocess of reports ..uideline developmngh decision tree
rticipatory methodn formal consensectrum ................
eoretical models ..arch domains ......t ...........................
al texts ..................een osteopathic u............................
Belgian osteopath............................ctors involved ..................................nuum ..................
ent ....................................................
ment used in KCE to identify relevas..........................us method .......................................
............................
............................
............................
............................unions in Belgium ............................
hic unions and sch........................................................................................................................................................................
E project on Prostant participatory m....................................................................................
KCE Reports
............................
............................
............................
............................through their ............................
hools in 2010 ...............................................................................................................................................................................ate cancer ...........ethods ....................................................................................................
s 174
.. 10
.. 11
.. 16
.. 18
.. 19
.. 22
.. 23
.. 24
.. 26
.. 28
.. 31
.. 32
.. 33
.. 33
.. 34
.. 42
.. 44
KCE Reports 17
LIST
74
T OF TABLES TaTaTaTaTapaTaTaTaTaTa
S
able 1 : EBM agenable 2: Six questioable 3 : Consultatable 4 : Dimensionable 5: The potentarticipatory researable 6: Rationale fable 7: Levers andable 8. Dimensionable 9. Timing of table 10: Impact en
Stakeholder Involv
ncies’ stakeholderons to identify releion approaches ans for analyzing ptial added value orch process ..........for stakeholder ind barriers with resns of stakeholder ithe different interanhancing strategie
vement
r involvement’ tasevant consultationnd application in K
public policies ......of participatory res............................volvement ...........
spect to the impacinvolvement ........actions with stakees ........................
sks and group typen methods with staKCE projects ...................................
search approache........................................................
ct of KCE reports .............................holders ............................................
es ........................akeholder involve........................................................
es across the three........................................................................................................................................................................
............................ment ...........................................................................e phases of the ........................................................................................................................................................................
5
.. 29
.. 30
.. 31
.. 35
.. 40
.. 45
.. 53
.. 56
.. 57
.. 58
6
LIST OF B
BOXES BoBoBoBoBoBoBoBoBoBoBo
S
ox 1: Stakeholder ox 2: Evolution of ox 3: an epistemoox 4: Diving in maox 5: Information dox 6: The ADAPTox 7: Acceptabilityox 8 : The Deliberox 9: DACEHTA rox 10: Some guidox 11: List of ques
Stakeholder Involv
r engagement .......knowledge-to-act
ological shift from agma, the cartogradisclosure ............E assessment of y as the most comrative Process – Arecommendationsing principles to sstions for the cons
vement
............................tion thinking and smatter of facts to aphy of controvers............................acceptability and
mplex dimension oA Snapshot ..........s to create the bassupport the use ofsultation round of
............................semantics ............matter of concernsies .................................................applicability of re
of the analysis ..................................
sis for a good andf research in practKCE experts .......
............................
............................ns ................................................................................commendations ......................................................... rewarding coopetice ..................................................
KCE Reports
............................
............................
............................
............................
............................
............................
............................
............................eration ..........................................................................
s 174
.... 9
.. 10
.. 11
.. 16
.. 26
.. 28
.. 35
.. 36
.. 45
.. 46
.. 47
KCE Reports 17
LIST OF A
74
ABBREVIAATIONS
S
ABBREVIATION
ADAPTE AGREE
AHRQ
CCOHTA
CIHR
DACEHTA
EBM EUnetHTA
HAS
HTA IQWiG
KBF
NCCHPP
NICE
(P&)CP VIKC
Stakeholder Involv
N DEFINITI
http://wwwAppraisahttp://wwwAgency fohttp://wwwCanadianhttp://wwwCanadianhttp://wwwDanish Chttp://wwwEvidenceEuropeanhttp://wwwHaute Auhttp://wwwHealth TeInstitut German King Bauhttp://wwwNational http://wwwNational http://www(Patient &Verenigin
vement
ION
w.adapte.org l of Guidelines Rew.agreetrust.org/or Healthcare Resw.ahrq.gov/ n Coordinating Ofw.cadth.ca/ n Institutes of Heaw.cihr-irsc.gc.ca/
Centre for Health Tw.sst.dk/English/D
e-Based Medicinen Network for Heaw.eunethta.eu/
utorité de Santé / w.has-sante.fr/ echnology Assessfür Qualität uInstitute for Qualit
udouin Foundationw.kbs-frb.be/ Collaborating Cenw.ncchpp.ca/ Institute for Healtw.nice.org.uk/ &) Citizen-Patientng van Integrale K
esearch & Evaluat
search and Qualit
ffice for Health Te
alth Research
Technology AssesDACEHTA.aspx
alth Technology A
French National A
sment und Wirtschaftlicty and Efficiency in
ntre for Healthy P
h and Clinical Exc
Kankercentra
tion
ty
chnology Assessm
ssment
Assessment
Authority for Healt
chkeit im Gesn Health Care htt
ublic Policy
cellence
ment
th
sundheitswessen tps://www.iqwig.de
7
/ e/
8
1. RATIINVO
Although stakeprocesses sincwillingness of diversified way‘when’ to involvsocietal pertineonly linked to tthe policy and p
1.1. Policy cThe ‘raison dfundamentally pbe seen as ‘emfinite resourceshealthcare neechallenge requunambiguous reObviously, for ain several othealso have to anthropology. perceptions, exindependence a
1.2. ScientiCombining a truof scientific rigoas the achievestudy processeThis must, withoWhen addressinvolve the diff
IONALE OFOLVEMENT
eholder involvemece its early daysthe organization . The primary aim
ve stakeholders, wence and the imphe scientific qualpractical challenge
challenges andd’être’ of scientifpolitical and pragm
mployed’ by societys of the healthcareds of the populauires that we foresults and proposaddressing this brer disciplines than
mobilize methodWe must find ad
xperiences and opand objectivity.
fic challenges ue openness to por and objectivity
ement of valid reses, including qualiout any doubt, being this challengferent groups of
F STAKEHT AT KCEent has been pas, this methodoloto address it in
m of this work is twith as an ultimatepact of the KCE rity of the researces are met.
d objectives fic advice institumatic. In this sensy to help and findre system for meation. Taking up
rmulate relevant se realistic recommroader perspectivn medicine and hds specific to sodequate ways to pinions of stakeho
and objectivesarticipative appro
y poses serious ssults by means oitative research a
e part of the core bge, i.e. learning tstakeholders, we
S
HOLDER
art of the KCE rogical study refle
a more systemato reflect about ‘we objective to increreports. The lattech, but also to the
utions such as se, KCE collaborad the best ways toeeting the virtually this fundamentaresearch questiomendations.
ve, we need comphealth economics;ociology, law, an
deal with and vaolders, without lo
s oaches with the sacientific challengeof systematic and
and sociological mbusiness of KCE. to master valid
e can tap into a
Stakeholder Involv
research ects the atic and
why’ and ease the er is not e degree
KCE is ators can o use the y infinite al policy ons, find
petences ; we will nd even alue the sing our
afeguard es, such d robust
methods. ways to body of
kse
1InWinInKpfothinFineas
vement
knowledge availasimilar institutionsexperiences from o
1.3. Structuren chapter 2, we eWe address thenvolvement but fron chapter 3, we pKCE researchers principles could orour group discusheir perception ofnvolving stakeholdFinally, chapter 4 nvolvement at Kendeavour fruitfuappendix, illustratestudy processes.
ble in the scients in other counour own work.
of this study reexplore the key pre ‘who’, ‘why’,’ om a theoretical apresent how we themselves, in t
r could not mean ssions, the experf the threats and tders in their day tsynthesizes the s
KCE, including thl. A number of e how we involved
tific literature andtries, but also g
eport rinciples of the sta
how’ and ‘wheand empirical poininvolved our ownthe reflection aboin practice for th
rts have shared the barriers but alo day work. strategic objectivehe key values nf succinct technid stakeholders at
KCE Reports
d the know-how get inspiration in
akeholder involveen’ questions ofnt of view. n stakeholders, i.eout what the idene KCE studies. Dtheir experienceslso the opportunit
es of the stakehoneeded to makeical sheets, givet different stages i
s 174
from n the
ment. f this
e. the ntified During s and ties of
lder’s e this en in in our
KCE Reports 17
2. KEY INVO
2.1. The risStakeholder thevolved1: • from stak
stakeholdeprocess-or
• from a budynamic inwell as influ
A set of similarconceived to dpossibilities to iRational-linearmakers rarelycommunicatio• Supply-sid
of research• Demand-s
research u(rational-lin
These models emphasize linkaInteractive mrelationshipsresearch: multicontext of its us
a See the Sb See the IN
74
PRINCIPLOLVEMENTse of stakeholdeinking grew up i
keholders vieweders as individuariented view; uffered dependennterdependency uential stakeholder theoretical modeescribe whether ncrease the resear models focusey used researcon between researde research initih by adapted publside research iusers about thenear model with fewere abandonedages between resodels, recently between the ke
ple actors, multipse. These variable
SUPPORT program NVOLVE program d
ES OF STAT er thinking n the economic
d as subjects als in a networ
ncy on stakeholdbetween a firm
ers. els has emerged and how researcarch impact2. d on disseminatio
ch it was becrchers and policy atives were deveication formats annitiatives were ir needs and p
eedback loop)b. in favour of mor
searchers and decdeveloped, put
ey variables thaple exchanges, naes are no longer c
description in Lavis
description4.
S
AKEHOLD
c area in the 193
to be manark-based, relatio
ders towards a and its surround
in the knowledgch is used and to
on of research. Icause of problemakers:
eloped to support nd effective packa
developed to qrovide tailored r
re interactive modcision makers5.
a focus on cat shape policy ature of the reseaconsidered in isola
s, 20093
Stakeholder Involv
ER
30s and
aged to nal and
bridging dings as
ge area, explore
f policy-ems of
the use aginga. question research
dels that
complex use of
arch and ation but
inraaeth
B
Inassininsda
c
vement
n interaction. Andationalc. These in
and rational modeexperience in the he postmodern cr
Box 1: Stakehold
n 1984, Freemaacademic attentiostakeholder. The subsequent ones nfluences. Thesenfluence and howstakeholder engadiffusion of these a good practice ha
Recently, ano
to establish nprocess by process (Nut
d the ways researnteractive modelsels, incorporate the
research use proritique (and the re
der engagement
an pushed a ston and provided
core idea that is that manageme
e theories aim tow firms respond tagement. The Woideas by publishinandbook in 20117.
other set of initiativenew kinds of relatiodeveloping intermeley 2008, p246)2.
rch is used are ns, in comparison te role of researchocess, and ask thal possibility to em
takeholder theory boundaries of underlies Freem
ent choice is a fuo understand theto those influenceorld Bank Group ng a good practice.
es between supply onship between resediary broker orga
no more considereto the traditional users’ knowledgee question of powmpower partners)
y to the forefrowhat could defi
man’s theory andnction of stakeho types of stakehes with what is chas contributed te manual in 1998
and demand have bsearchers and the nizations into the
9
ed as linear e and wer in .
nt of ine a d the
olders’ holder called to the
86 and
begun policy policy
10
Figure 1: Stake
Figure inspired byCanada 2000, p.
eholder involvem
y “Health Canada’s128 and “Spectrum
ment in research
s public involvementof stakeholder enga
S
-use theoretical
t continuum » in Heagement” in IFC 20
Stakeholder Involv
models
alth 11, p.3 7
B
Bthc
LRvT“
R
TdKrere“
STmitthRsin
vement
Box 2: Evolution
Best and colleaguhe knowledge-to-cited in Nutley, 20
Linear models (19Research is dissevarious settings. WThe predominantknowledge trans
Relationship mod
The key processedevelop within neKnowledge produelationships and elationships is theknowledge exch
System models (The way knowledgmost important fact needs not only the priorities, cuResearch use is system. The langntegration”, “tran
of knowledge-to
ues provide a fram-action thinking i10, p.135)9:
960s to mid-1990eminated as resulWhether it gets ust language of sfer” and “dissem
dels (mid-1990s t
es for improving reetworks of collaboucts are defined improving the inte key. The langua
hange”.
more recently) ge is embedded wctor in improving rto be embedded iultures and cont
thus a dynamicuage of research
nslation” and “mo
o-action thinking
mework that descrn three phases (
0s) ts are handed ovsed is a function oresearch use f
mination”.
to present)
esearch used areorating research and utilized in
terpersonal commage of research u
within organizatioresearch use. Forn relationships butexts of organizc process withinh use for these mobilization”.
KCE Reports
g and semantics
ribes the generati(Best et al frame
ver to others for uof effective packafor these mode
e the relationshipsproducers and uthe context of t
munication within use for these mod
ons and systems r knowledge to be ut also interwovenzations and sysn a complex adamodels is “knowl
s 174
ion of ework
use in aging. els is
s that users. these these
dels is
is the used
n with tems. aptive ledge
KCE Reports 17
Box 3: an episconcerns
The knowledgemodels towardepistemologicaresearch knowResearch-policysystems thinkinbecome blurred
• In the tradprovide dstraightforw(Nutley 200
• In interactarrive as udecisions, its use thr2008, p.25
EXAMPLE: Th(DACEHTA) DACEHTA pubnew scientific monly with elemeand ethics, but
HTA may be coand the researcfocus of an HTdocumented ba
74
stemological shif
e-to-action thinkids more interal shift which req
wledge as somey relations becom
ng, the boundariesd”9.
ditional linear andiscrete, unambwardly be transfe08, p.123)2.
tive models, we runcomplicated “fabut may be trans
rough ongoing di1)2
he Danish Centr
blished in 2008 amethods and apprents of HTA, i.e. talso with the step
onsidered as bridch domain. In ordA must be based
asis for decisions
ft from matter of
ing has historicactive models inquires us to ret
ething separate me at least two-was between these
nd rational modebiguous, factuaerred and applie
recognise that “reacts” to be weighslated and reconsalogue with rese
re for Health Te
a new handbook roaches to HTA10.technology, patienps in the HTA proc
ging between theer to fulfil such a
d on the need of tabout the use of h
S
facts to matter o
ally evolved fromn association wthink our concefrom policy kno
ay and in some vetwo realms of kno
els research was l accounts thed to policy or
esearch evidence hed up in makinstructed in the proearch producers”.
echnology Asse
presenting upda This handbook d
nt, organisation, ecess.
e decision-makingpurpose, the prob
the decision makehealth technology
Stakeholder Involv
of
m linear with “an ption of
owledge. ersion of owledge
seen to at can practice
may not ng policy ocess of (Nutley
essment
ated and deals not economy
g domain blems in ers for a
y.
F
Tomdb2
2In
T
Ecin
d
e
vement
Figure 2: Bridgin
The DACEHTA modone-way flow represmaking domain. It sedecision-makersd bubetween the policy-m2008, p15)10
2.1.1. Who aren the economic ar
“any group orachievement o
The International F“those who minfluence its “stake” that eproject or inve
EUnetHTA recentlcharacteristics ofnterested in and
See the “two (Freeman 19
g between decis
del is inspired by thesentation, where theeparates two commut allows some spacmaking paradigm an
e stakeholders?rea, Freeman defr individual who cof the firm’s objecFinance Corporati
may have interestoutcome, either ach of these diffeestment will vary” ly referred to thesf stakeholders : d/or potentially infl
communities” thesi84, p25)12 cited in (
sion-making and
e rational-linear mode research domain ta
munities, one of resece for interactions innd the HTA paradig
fined stakeholderscan affect or who ctives”e. ion’s definition adts in a project anpositively or neg
erent individuals o(IFC 2011, p.10-1
se two definitionsthey are affec
luent.
s of Caplan11. Andriof 2002, p30)1
research domai
del of research use argets the decision-
earchers and the othn the overlapping zogm. From (Kristense
s as is affected by the
ds nd/or the ability togatively. […] Theor groups has in a11)7. to underline the cted by the pr
1.
11
ns
in a -her of one en
e
o e a
three roject,
12
When comparilists, the most c• industry • policy mak• patient/use• health care• health relat• academia• commercia
2.1.2. The cDepending on wis more or less
2.1.2.1. WidFollowing the G(IQWiG) has cinstitutions, or the preliminarypreliminary repinterim results13
restricted. Prinstitutes, and c
2.1.2.2. CloIn contrast, thapproach in wstakeholders. Srepresentativeshealth professpractices. Thesentire productirepresentativesincluded in thes
ng several EBMcommonly cited st
ers and/or instituter representativese professionals ted media
al enterprises
circle of stakeholwhich stakeholderopen to these sev
de-open consultaGerman Institute chosen an apprcommercial ente
y report plans, oports and thereby
3. The circle of perivate persons, companies, may s
osed monitoring ghe French Natiohich all actors ha
Stakeholders may s. An equilibriumionals, types of
se stakeholders aion process of s, health insuranse groups ; they m
M or health-relatetakeholders appea
tions
lders r involvement metven types of stake
ation for Quality and E
roach in which aerprises may submon amendments y comment on thersons entitled toas well as pr
submit comments
groups nal Authority forave to candidatebe health profess
m is searched beopinions, types
are involved in direcommendation
nce companies may be consulted
S
ed agencies’ stakar to be :
thod is chosen, theholders.
Efficiency in Heaall interested pmit written commto report plans
he methods or p submit commentrofessional asso.
r Health (HAS) e or to be designsionals and patienetween types of s of practice, plfferent groups dus. But public aand industries on a more ad hoc
Stakeholder Involv
keholder
he circle
lth Care persons, ments on
and on ublished ts is not ciations,
has an nated by nt/users’ involved aces of
uring the agencies'
are not c basis.
EWda
Sba
EEF
•
•
•
•
•
•
f
vement
EXAMPLE in the When identifying sdefinition as in thea consultative role
Stakeholders are gby, or have an intactions or aims of
EUnetHTA has chEuropean level, anFollowing groups a
Policy makers a
Policy makers a
Patient organiz
Healthcare pro
Industry
Health related
Definition cite
research area : Estakeholders, the e economic area be:
groups or organizterest in and mayan organization, p
hosen to focus on nd on generic ratare targeted:
at national/region
at hospitals/statut
zations
fessional organisa
media
ed by Nielsen 20091
EUnetHTA definiEUnetHTA Collab
but restrains stake
zations which potey in a consultativeproject or policy d
umbrella organizher than disease
al level
tory health insuran
ations
14.
KCE Reports
ition of stakeholdboration used a seholder involveme
entially will be affee role influence odirectionsf.
zations operating aspecific organiza
nce/HMOs
s 174
ders similar ent to
ected on the
at the ations.
KCE Reports 17
2.1.3. The p
2.1.3.1. A pThere exist macorresponding consider that wapplication: (1) the involvementhe involvemenhealthcare instinvolvement of the P&CP in ththese 5 domainhis or her opiniothe first domainpublic health. There exist difranging from mapproach, bothare needed. Yeand execute sTronto17.
2.1.3.2. A tr‘car
Tronto suggesthuman activity and repair our world includes oseek to interwp.143)17. It h‘capacitating’ bA liberty to exphope18-20. This individual relati
74
patient and the c
patchwork of posany forms of patie
to different objewe can divide th
the involvement nt of the patient int of the patienttitutions (hospitalCP in the financie determination o
ns, the P&CP has on, to co-decide on, the research ac
fferent models tomere information
h generic and speet, each of these should be chara
ransversal qualitre’ ts that on the m“that includes ev‘world’ so that w
our bodies, ourseweave in a comhas been suggeecause it could ‘r
press an opinion, condition of ‘car
ionship between
itizen
ssibilities ent and citizen-patectives and domis kind of involveof the P&CP in thn its relationship t in its relationshl, mutuality’s, prng options to ado
of health prioritiesto be informed, to
or to decide. We lictivities and more
o involve the cin up to actual decific technical aprocedures that w
acterized by the
ty of the involvem
ost general levelverything that we we can live in it aelves, and our envmplex, life-sustain
sted that this preveal’ a certain lan anxiety, a wisre’ seems to be the patient and
S
tient (P&CP) invomains of decisioement into 5 domhe research activwith her practitio
hip with the heaivate insurance) opt (5) the involves and choices. Ino give informationimit the present seprecisely the res
tizens and the pdecision15, 16. Fnd human compwe have to conce
‘care’ defined b
ment: the practic
, caring be viewdo to maintain, c
as well as possibvironment, all of wning web.”(Trontopractice of care liberty for the ben
sh, an expectationquite trivial in ththe practitioner.
Stakeholder Involv
olvement n. We mains of vities, (2) oner, (3) alth and
(4) the ement of each of
n, to give ection to
search in
patients, or each etencies eptualize by Joan
ce of
ed as a continue, ble. That which we o, 2009,
can be neficiary. n and an he inter-But it is
min
2NinifAfogpawW(cininddimaFhsfop‘i
2Wththe
vement
much less straightn public health.
2.1.3.3. InvolvNevertheless, wenvolvement, effecf this concept wasA first example of or normal low-risgroup of gynaecoplain-language veall socioeconomicwith the input fromWe also find exNational Institute
cited example, thenform the researnstitution is able developed a systdetails can be foumpressive list ofanxiety25 or the AlzFurthermore, the helpful to confirmsomewhat unexpeor every patientpermanent presennvasion’ of their l
2.1.3.4. EvaluaWhereas publicatihe pertinence of he patient involvevaluation remain
tforward to concre
vement of the pae can find sevectively characterizes not explicitly ‘namsuch type of patie
sk childbirth, develogists and midwrsion has been de categories of wo
m women of differexamples of patie
for Health and Cle information follorch institution andto provide better
tematic procedureund on the NICEf guidances for zheimer disease26
information origi research hypoth
ectedly, peritoneat. It appeared nce of medical eqives by their disea
ation of the patieons attest the inflthe research28-31,
vement32, 33. Theto be defined.
etize in other setti
tient in the ‘reseeral examples oed by the practicemed’ or ‘used’. ent involvement iseloped by KCE iives21. On the baerived, in order toomen. This exercent education levent involvement ilinical Excellence,ows a ‘reciprocal’ d, given this infoinformation to th
e to involve patiE website 23, whipatients (e.g. on6). inating from the heses. This is hoal dialysis was no
that some of tquipment in theirase27.
ent and citizen-pluence of the way, there is also a actual format a
ings, such as rese
earch activities’ of successful pe of ‘care’ indeed,
s the practice guidn collaboration w
asis of this guidelio be understandabcise has been reaels22. in the work of , UK). As in the ab pathway: the pa
ormation, the resehe patients. NICEents and citizensch is also offerin
n caesarian sect
patients can alsow NICE learnedot the preferred othem experiencer home as a com
patient involvemy to involve patienneed for evaluati
and modalities o
13
earch
atient even
deline with a ine, a ble by alized
NICE bove-tients earch E has s; full ng an tion24,
so be that,
option e this mplete
ent nts for ion of f this
14
2.1.4. Why sReasons to invnumerous, anddescribed as us• provide leg• promote EB• prevent dis• improve ac• enhance re• reduce dupStakeholder inv• Transparen• Accountab• Participatio• Objectivity • Evidence-b• Patient-orie• Scientific wThe ‘objectivity of a richer picpractices, in a s
EXAMPLES ofEBM and healtEUnetHTA: becsense that theimprove the helegitimacy hasrelation to the transparency o
stakeholder invovolve stakeholded vary betweenseful to : gitimacy or credibiBM practices and sagreement or concceptance of resuelevance of reseaplication. volvement is seenncy ility
on
based outcome ented outcome work.
and scientific wocture and the oso-called second-d
f declared reasoth-related agenccause HTA agen
ey are (often) puealthcare systems to be considescientific quality
of the production
olvement? rs in a research
n agencies. Stak
lity to the agency EBM culture ; nflicts ; lts and increase imrch ;
n as serving the pr
ork mentioned herbjective renderindegree heuristic o
ons to use stakies ncies resemble public, tax-finance
m by their activitieered. Legitimacy y of HTA productprocesses14.
S
h production prockeholder involve
;
mpact ;
rinciples of:
re point to the acg of all viewpoi
objectivity.
keholder involvem
political institutiond institutions seees, the question must be unders
ts and the fairne
Stakeholder Involv
cess are ment is
cquisition nts and
ment in
s in the eking to of their
stood in ess and
Ca
DcT
Aha
C
Fw
Stap
Nasb
IQto
Hpre
vement
CCOHTA: stakehoan enhanced awa
DACEHTA: stakehcourse by prevenTo improve the po
AHRQ: stakeholdehelps avoid prioritand reduces poten
CIHR: there are tw
First, the creationwidespread adopt
Second, all of the axpayers’ dollarspractice/action (ac
NICE: comments assurance and pstakeholders offebalance the scienc
QWIG: stakeholransparency wh
objective and evid
HAS: stakeholder principles which aepresentatives), t
older consultationareness and cult
holder involvemennting the occurrossible acceptanc
er involvement pizing topics that hntial duplication.
wo primary reason
of new knowledtion or have an im
interested partiess invested in heaccountability, Tet
from stakeholdeer-review procesr specialist expece of the guideline
lder involvemenhich is one of thience-based, patie
involvement meare : participatiotransparency and
s are important ature of best pract
nt is to ensure therence of disagrece and applicatio
rovides credibilithave no relevanc
ns.
ge often does nompact on patients
s want to see the alth research by troe 2007, p.1)34.
ers are “a vitalsses” (NICE 2007ertise and a prae methodology.
nt corresponds is agency’s princent-oriented and s
thodologies are gon (of professionad independency.
KCE Reports
as it helped in cretices.
e most efficient peement and confon of the HTA resu
ty to AHRQ resece to real-world is
ot, by itself, lead s’ health.
benefits reaped bmoving research
part of the qu7, p.80)35. In this actical perspectiv
to its principlciples36 : indepenscientific.
grounded in the als and patients/u
s 174
eating
roject flicts. ults.
earch, sues,
to its
by the h into
uality way,
ve to
e of ndent,
same users’
KCE Reports 17
2.1.5. Criteri
2.1.5.1. InvoDACEHTA undinvolved from thfeel co-responacceptance of tcooperative behhigh work ethics
2.1.5.2. InvoIQWIG, NICE involvement protasks to do andthe amount of w
2.1.5.3. InvoThe CIHR poinoccurs in the inresulting in mucomplexity, andresearch result2007, p.1)34. Siresearch-use is“dissemination”“translation” or
2.1.5.4. InvoAs influence anattention to staprocess and itabout the purpoof the outcome.
74
ria of effective st
olved at an earlyderlines that it ishe beginning in thnsible for the pthe different occuhaviour is requireds among the proje
olved in transpaand HAS have
ocess to clarify cd planning. Stake
work needed (the
olved in mutual lnts out the imponteraction betweeutual learning. Td level of engagts and on the nence stakeholderss no more descri” but “knowledg“mobilization”.
olved with satisfnd impact are eluakeholders’ satists outcome. Satiose and power of.
takeholder involv
y stage of the prs important that he project. It helpsproject and creapational backgroud in stakeholder iect participants.
rency e developed speconfidentiality issueholders have to number of meetin
learning ortance of the knn the knowledge
These interactionsgement, dependineeds of the partics are involved in thibed in terms likege exchange”,
faction usive concepts, Nfaction with the isfaction relates f stakeholders, th
S
vement
roject all relevant bod
s to make the parates understandiunds. Implicit is thnvolvement with t
ecific guidelines ues and publicatio
be informed earngs) and tasks to
nowledge exchanuser and the res
s may vary in ing on the naturecular stakeholder he research-use pe “knowledge tran“knowledge inte
NICE recommendsstakeholder invoto realistic expee usability and re
Stakeholder Involv
dies are rticipants ing and hat good trust and
on the on rules, rly about do.
nge that earcher, ntensity, e of the r (Tetroe process, nsfer” or gration”,
s to pay olvement ectations elevance
2Sd••••••••
vement
2.1.6. What areStakeholder involvdeveloped in furth Stakeholder id Communicatio Stakeholder c Dialogue in ca Management Stakeholder in Regular feed- Skills to mana
e the key compovement encompaer sections. dentification and aon at an early stagconsultation all aloase of controversiof stakeholders’ cnvolvement in pro-back to stakeholdage the process o
onents of stakehasses a range of
analysis ge ong the project deies comments oject monitoring ders f stakeholder invo
holder involvemef activities that w
evelopment proces
olvement
15
ent? will be
ss
16
Figure 3: Key c
Figure inspired by
components of s
y Key component o
stakeholder invo
of stakeholders enga
S
olvement
agement in IFC 200
Stakeholder Involv
07, p.127.
2Assinoa12
Tino
B
“vNsuoMg
S
vement
2.2. StakeholdAlso called envistakeholder identstakeholder involvnterested or infloutcome of your analysis has two a
. to clarify who 2. to assess whi
identify which This task is compnterconnections bof stakeholders to
“Collective sitconcerned, th
Box 4: Diving in m
“The cartographyvisualize issues, dNetwork Theory tsocio-technical deunfortunately, not of controversies hMACOSPOL (Magathers eight Euro
Source : Venturini 20
der identificatioronmental scantification and avement. To negleuential stakeholdproject. As men
aims (Kristensen 2are the stakeholdch stakeholders a function to assignplex as one neebefore being able
involve. tuations are alwayhe more intricate th
magma, the carto
y of controversiesdeveloped by Brunto train studentsebate. Today it a much documenas also become tpping COntrover
opean universities
010, p137
on and analysisn when it is qunalysis is the
ect or to miss ideders may impactioned by DACE2007, chapter 1)10
ders; are the most impon them to in the cods to explore theto order it and to
ys intricate and thhey can get.”(Ven
ography of contr
s is a set of techno Latour as a dids in the investiga
is a full reseanted one. […] Recthe object of the rsies on Sciences and research cen
KCE Reports
s uickly performedfirst crucial ste
entification of affect negatively theHTA, this stakeh
0:
ortant to include, aontext of the projee social complexproduce one’s ow
he more actors arenturini 2009, p.6)37
roversies
niques to exploredactic version of Aation of contemparch method, thocently, the cartogEU funded conso
e for POLitics), wnters.”
s 174
, the ep of ected, e end holder
and to ect. xity of wn list
e 7
e and Actor-porary ough, raphy
ortium which
KCE Reports 17
2.2.1. Explo
compl“Social cadescription
2.2.1.1. Obs“Following statementsconsider cStatementsbeings, tecand so on”
Starting by readon websites, inopportunity to them. Figure 4mental health sa careful scrutinPerplexity is reconnections. Incondition for thapproach. But,quickly ordered
2.2.1.2. DesNotes and mapvisibility to diffeto: 1. representa2. influence : 3. relevance
original per
74
re the world : frolexity rtographers shou
ns at once”.(Ventu
servation : from the webs of
s, social cartogconnections that s are always part chnical objects, na” (Venturini 2010, pding texts about t
n press, in legal tidentify potential
4 illustrates in a services for childrny of the successiequired in such en this way, perplhe second-order h quick increase in notes and map
scription : from aps have to rendeerent stakeholder
ativeness : how maactors on influent: even if stake
rspectives and qu
om unfolding com
uld work out theurini 2010, p.2)38
texts to actors relations surrou
graphers are inespread beyond t
t of larger networkatural organisms, p.2)38. the issue under sexts or in scientifl stakeholders an
graphical way hren and adolescenive legal text on thexercise to take ilexity becomes aheuristic objectiviin complexity of
ps.
actors to texts er the complexity rs we need to be
any actors subscrtial positions deseholders are mino
uestion what is giv
S
mplexity to orde
eir observations
unding controverevitably brought the textual univeks comprising hummetaphysical enti
scrutiny (in grey litfic literature) offernd connections bhow the stakehonts can be identifhe subject. into account a lota research tool, ty that characterif connections ha
legible. To give e proportional ac
ribe to their viewperve special attentoritary, they mig
ven for granted.
Stakeholder Involv
ering
and
rsial t to erse. man ities
terature, rs a first between
olders in fied from
t of new and the izes this
as to be
different ccording
oint. tion ght offer
vement 17
18
Figure 4: Prim
Working docume
ary identification
ent from KCE Repor
n of stakeholders
rts 170 “Organisatio
S
s in legal texts
on of child and adole
Stakeholder Involv
escent mental health
vement
h care: study of the literature and an innternational overview
KCE Reports
w” 39
s 174
KCE Reports 17
Figure 5: Acto
Mapping of the in“[Ostheopathy an
74
rs-networks diag
ntensity (or paucity) nd chiropractic: state
gram, connection
of the connections e of affairs in Belgiu
S
ns between oste
between osteopathum]” 40.
Stakeholder Involv
eopathic unions i
hic unions in Belgium
vement
in Belgium throu
m through their web
ugh their website
bsites links. Working
es links
g document from KC
CE reports 148
19
20
2.2.2. Be preA social cartogtrying to identp.7)38: 1. you shall lis2. you shall o3. you shall
methodolog4. you shall a5. you shall s6. you shall a7. you shall
flexible.
2.2.3. OrganUsing the basEuropean MACstakeholder ide1. The glossa2. The docum3. The analys4. The review5. The tree of6. The scale o7. The diagra8. The chrono9. The table o10. The forum
epared before thgrapher has to fify and analyze
sten to actors’ voiobserve from as m
not restrain yougy;
adjust your descripimplify complexity
attribute to each acprovide descrip
nize your stakehsic framework of COSPOL project, sentification and anary of non-controvmentation repositosis of scientific litew of media and puf disagreement of controversies
am of actors-netwoology of dispute of interpretive fram(see the negotiat
he journey follow all these stakeholders’ po
ices more than to many viewpoints as
ur observation t
ptions and observy respectfully; ctor a visibility protions that are a
holder identificaticontroversy ma
several tasks mayalysis, including t
versial elements ory erature blic opinions
orks
mes ion and partnersh
S
recommendationsositions (Venturin
your own presums possible; to any single th
vations recursively
oportional to its weadapted, redunda
ion and analysispping, developedy be required for he production of:
hips chapter)
Stakeholder Involv
s before ni 2010,
mptions;
heory or
y;
eight; ant and
s d in the relevant
2Inopreid
E
FH
vement
2.2.3.1. The gln HTA, GCP or organizations of spictures or diagramesponsible of thdentified, these ac
EXAMPLE : Sourc
Figure used in KCE Hulpmiddelen Komp
lossary of non-cHSR projects, ne
services or practicms to underline their construction ctors may be invo
ces of oxygen for
report 156C “Homepas 2004, Zuurstofa
controversial elemew technologies ces, may be adeqheir various compor implementatio
olved in the study.
home therapy
e oxygen therapy”, ppparatuur – ISBN 9
KCE Reports
ments and devices, comquately describedponents and the aon. After having
page ii41. Source: 90-70918-38-2
s 174
mplex d with actors been
KCE Reports 17
2.2.3.2. TheFrom observatcomplexity of simplifications, documents, i.e.Publication of tkeeping regardconsidered caexperts and sta
2.2.3.3. The“Scientometricsthe analysis oresearch centeideas through literature may identify pools o
2.2.3.4. TheSince articles ano longer neceto quickly scan See the figure the KCE study “
2.2.3.5. The“Cartograpconnected
74
e documentationtion to descriptiointerrelationship.field notes, in
all traces shouldthis material musing information d
arefully to guaraakeholders (see ne
e analysis of scies can reveal the nof co-authorship, rs, journals…) thrlexicographic anbe investigated r ‘schools’ of scie
e review of mediand press releasesessary. Useful too
debates and/or co6 with connectio“Osteopathy and
e tree of disagrephers should not r
and structured i
n repository on, analysis is To assure the
nterview recordin be kept and offer
st be discussed inisclosure and/or
antee trustworthyext chapter).
entific literature networks of scien
the relative autrough citation ananalysis”(Venturini not only to collentists and stakeho
ia and public opis are digitally pub
ols such as internontroversies to idns between osteochiropractic : stat
ement renounce to tracein discourses. A
S
needed to simpe reversibility ongs, raw data, red to public examn the project teampublication rules y relationships b
: bibliometric antific collaboration thority actors (scalysis, and the diff
2010, p.14)38. Sect evidence but olders.
inions blished, hand-seaet crawlers may entify stakeholderopathy websites, e of affairs in Belg
e how arguments position taken o
Stakeholder Involv
plify the of these
archive mination. m. Gate-must be between
nalysis through
cientists, fusion of Scientific
also to
rching is be used rs. used in
gium.”40
are on a
2
TpththdSinc
2
DSuin
vement
specific issueissues. This r(Venturini 201
2.2.3.6. The sc“Cartographerinvestigation, in the scale of
This scale has tpotentially involve he opportunity toheir European udisputes that ariseSee the internanterventions for Gcan it be prevente
2.2.3.7. The di“[…] every aevery networsingle actor. Win an explosioconstellation o(Venturini 201
During a researchSome stakeholderumbrella of organnteresting ways o
e limits the positiramification can b10, p16)38.
cale of controvers are free to but they must bef disputes where ito be chosen to
in the study. Inteo follow connectioumbrella organize at that level of diational confrontatGP’s burnout in “d and how can it b
iagram of actorsctor can be dec
rk can be conneWhat used to be aon of conflicting aof agents can soli10, p.16)38. process, it’s not rs become closer nizations may expf representation o
ions that could bbe represented in
ersies choose the gra
able to situate thit belongs” (Ventu limit the numbeernational compa
ons between natiozation; and to discussion. tion of philosopBurn-out of genebe cured ?”42
s-networks : mapomposed into a
ected tightly enoua single actor can gents and what uidify into a unique
rare that stakeholthan before, othe
plode. Evolutive of these magmatic
be taken on othen numerous ways
anularity of theiheir object of studyurini 2010, p.16)38.er of stakeholde
arisons in projectsonal stakeholdersdescribe coalition
phies underlyingeral practitioners :
pping burnout network and […
ugh to become asuddenly dissolve
used to be a loosee source of action.
lders’ positions evers become distangraphs and map
c movements.
21
er s”
ir y .
ers to s offer s and ns or
g the : how
…] a e e .”
volve. nt. An s are
22
Figure 6 : Acto
Working docume
2.2.3.8. The“Obviously, thecontroversies rsudden cascadmost events ar(Venturini 2010
ors-network diag
ent from KCE Repor
e chronology of de evolution of cremain dormant fe of quarrels. Thisre packed in shor0, p.17)38. But the
gram, representa
rts 148 “[Ostheopath
dispute controversies is for years or decas makes timelinesrt periods separa
ese timelines offe
S
tion of Belgian o
hy and chiropractic:
not uniform: somades and then bus difficult to draw bated by long empr renewed possib
Stakeholder Involv
osteopathic unio
: state of affairs in B
metimes urst in a because ty gaps”
bilities to
dKa
vement
ns and schools
Belgium]”40.
distinguish ‘permaKCE appears in tha flow of interactio
in 2010
anent’ stakeholdehe timeline, it’s inons that may have
ers from intermittteresting to positi
e a long history.
KCE Reports
tent ones. And, ion its intervention
s 174
when n into
KCE Reports 17
Figure 7: Time
Working docume
74
eline of a KCE pro
ent from KCE Repor
oject
rts 165 “[Burnout am
S
mong general practit
Stakeholder Involv
tioners: prevention a
vement
and management]”442
23
24
2.2.3.9. The“To handle thsimplified interinterpretive fra
Figure 8 : Exam
From Moret-Hartm
e table of interpre growing comprpretation grids” ames should r
mple of table of i
man 2007, p.32043.
retive frames plexity of social
(Venturini 2010epresent all th
interpretive fram
S
life, all actors , p.18)38. The t
hese differing p
mes of actors invo
Stakeholder Involv
develop table of point of
vinac
olved
vement
views, showing wnterpretative framand the solution, corresponding wor
where they diverme describes for in
and how is theirrldview43.
rge and where tnstance how actorr underlying syste
KCE Reports
hey may overlaprs express the proem of values, wi
s 174
p. An oblem ith its
KCE Reports 17
2.2.3.10. TheStakeholder idobserving andinformation forarenas where rand the KCE aalso provide a sSee the use of in Belgium (cha
EXAMPLE : DAStakeholders adegree, have ingiven HTA. To during the stake
• Who is th
• Who are
• Who has
• Who pay
• Who is inconven
• Who has
The answers comprising botmanager who involve.
Two specific tyrecommends th
• stakeholdteam (we
74
e forum dentification and d describing”. Sr this environmenelationships betwre lively and vividspace to perform tDebategraph© to
apter Dialogue in c
ACEHTA stakehoare defined by Dnfluence on or aridentify stakeholdeholder analysis (
he initiator?
the users of the r
s to accept the res
ys for the work and
affected: Who bnienced by the res
s the knowledge a
to these questioth individuals anassesses which
ypes of stakehohe use of specific
ders with obviouse call them “oppos
analysis may bStakeholder meent scanning may
ween stakeholdersdly performed. Cothese public deba
o perform public dcase of controvers
older identificatioDACEHTA as gro
e influenced by tders, the following(see below) (Kriste
results?
sults?
d the results?
benefits/profits/hasults?
nd resources, or c
ons will producend institutions. It h stakeholders a
lders are identifimethods to involv
s opposed interesed stakeholders”
S
become more thetings used to y transform into s or between stakeontroversy-websiteates. debate around ostsies)
on oups which, to a he possible chang questions can bensen 2007, p.24
as drawbacks/risk
contributes?
e a list of stakeis primarily the
re the most rele
ed by DACEHTAve them (see belo
ests to the actua);
Stakeholder Involv
an “just gather
political eholders es might
teopathy
marked ges in a
be asked )10 :
ks or is
eholders e project evant to
A which w).
l project
•
2
2LcincliresalicnBse9wsme
vement
stakeholderthe decisioimplementinthem “end s
2.3. Commun2.3.1. Analyse Like identification challenge driven bnteractive model conceived as circnear models, buteached by puttin
sociology of innoactors: informationnk up the netw
conception, sharenetwork. But, while transpstakeholder involvelements in collab9). And these are when sensitive infstakeholders. Litemutual respect aestablished, can n
rs who – at the coon or planning ng the results andstakeholders”).
ication at an eathe risks and beof stakeholders,
by principles of tof stakeholder invulation of messagt as opportunities ng go-betweens ovation means en, things, money,ork actors, they ed knowledge is
parency and accvement, mutual borative processehighly vulnerable formation is disclo
erature on particiand trust takes never be taken for
ompletion of the prprocesses and
d recommendation
arly stage enefits linked to
communication wtransparency andvolvement, commges only, as is thto strengthen a n'into circulation'. verything that ci people and theirhelp to rebind
s a go-between
countability are twrespect and trus
es (Cargo 2008, pto information dis
osed (un-)intentiopatory research time and patiegranted.
roject – will be clowho can assi
ns of the HTA (w
communication with stakeholders
d accountability. Imunication is no lohe case within ranetwork. This couBy go-betweens
irculates betweenr skills44. Go-betwand bind. In suthat strengthens
wo core elemenst are two other p.336)45 (see FIGsclosure problemsonally by experts stresses that bunce and, even
25
ose to st in
we call
s is a n the onger tional
uld be s, the n the
weens uch a s the
nts of core
GURE s, e.g. or by
uilding once
26
Figure 9: Core
From Cargo 2008
e elements of a p
8, p.32945.
artnership proce
S
ess
Stakeholder Involv
STos
2SossthacB
“inm(aaspt
S
2InsInpdInc
vement
Secure your comTo resolve problemother EBM or heasolutions, impleme
2.3.1.1. PublicScientists, industryopportunity to substages in the projestakeholders havehese institutions’ acknowledged anchapter ManagemBox 5: Informatio
Disclosure is a fonterested and afmanner that is un(and ongoing) steactivities, from cand resolution stakeholders […project, its impacthem”
Source: IFC 2007, p
2.3.1.2. Confidn the HAS commsigned by all stanformation disclopublication. Particdisagree with the fn its Guide to thconsultation proce
mmunication ms of perceived blth-related agenciented before publi
c communicationy, professional sobmit comments oects. In the IQWI
e to give their conswebsites. As a
d answered as fment of stakeholdeon disclosure
ormal-sounding teffected parties. Cnderstandable to ep in the process onsultation and
of grievances…] have accurate
cts, and any othe
p.277.
dential communimunication strateakeholders involvosure is prohibitecipants have the final version of thehe methods of teess where eviden
betrayal linked to ies are using two ication of the repo
n ocieties, doctors, on IQWiG's or NIG and NICE comsent to publication
counterpart, eacfully and as factu
ers’ comments).
rm for making infCommunicating syour stakeholderof stakeholder einformed partici
s, will be moe and timely iner aspects that m
ication egy, confidentialityved in guideline ed, both before right to mention
e report46. echnology appraisnce submitted in
KCE Reports
information disclotypes of very diff
ort.
and patients havCE’s work at diff
mmunication straten of their commench comment muually as possible
formation accessibsuch information rs is an importanengagement. All oipation to negotiore constructiv
nformation aboumay have an effe
y clauses have tdevelopment grand after the r
n in the report if
sal, NICE describ confidence is in
s 174
osure, ferent
ve the ferent egies, nts on st be (see
ble to in a
nt first other iation ve if ut the ect on
to be oups. report f they
bes a nvited
KCE Reports 17
from manufactuappraised. Undaccept unpublexample, if thconfidence’) or(‘academic in c
2.3.2. AnticipA 2004 evaluindicated that sof all the larger are re-deployedtheir job descractivities may rsuch as web-coPersonal telepinformation dismoods of interlrespectful andactivities. How these activities“opposed stakewith others, asfocused (see vproject monitori
«These daphone) to various expstakeholde(KCE expetranslated f
74
urers and sponsoder exceptional clished evidence he information isr if its use might onfidence’)” (NICE
pate time-consuation of the sta
stakeholding activorganizations inv
d to cope with theiptions (Ilott 2004require much moonsultation procesphone contacts, closure are not ocutors, developm
d maybe trustwoto deal with it a
s, are two queseholders”, their ins this group is evarious options ining). ays, it takes me
pass all the nexplanation. It almoers] » ert during preparfrom French).
rs of the technolocircumstances, it
under agreemens commercially st adversely affectE 2008, p.22)47.
uming communicakeholder involvevities are now intevolved in this proce workload, and s4, p.3)48. This mere time, even out
sses or stakeholdepersonal mail
unfrequent. Manament of convincinorthy relationshi
and how much timtions that shouldvolvement may b
expected to be vn chapter 2.7. Sta
about one hour ecessary diplomaost seems like we
ration phase of c
S
ogy or technologieis stated that NInt of confidentia
sensitive (‘commet future publicatio
cation activitiesement process iegrated in businescess. New or existtakeholding is inceans that commutside dedicated mer meetings.
exchanges, ofagement of feelinng arguments, bups are time-con
me has to be devd be anticipatedbe more problemavery labour-intensakeholder involve
a day (by e-maiatic steps, this e ‘frighten’ them
consultation proce
Stakeholder Involv
es being ICE “will ality; for ercial in on rights
n NICE ss plans ting staff cluded in unication moments
ff-record ngs and
uilding of nsuming voted to
d. As to atic than sive and ement in
il or with [the
ess,
2
2SdinbtoeFoIninainpth•
•••
vement
2.4. Stakeholddevelopm
2.4.1. Refer to Stakeholder consdecision-making anformation level (between stakeholdo do more than gevolve towards a dFrameworks for aoffer standards to n the case of a nnstance, is usedaddressed adequnternally and extep.2)49. This meanhat the process of The guideline
relevant profe The patients’ The target use The guideline
der consultatioment process
standards sultation is situaactivities, below (see FIGURE 10)ders and experts
give or receive infdeeper engagemeassessing the quaassess the stakeh
new guideline ded to check wheuately and wheternally valid, and s that stakeholdef their involvemene development gessional groups; views and prefereers of the guidelin has been piloted
n all along the
ated at mid-levelthe partnership
). At this consultamay occur as st
formation. Sometient of stakeholderality of clinical prholder involvemen
evelopment, the Ather the potentiather the recommare feasible for pers have been cont may meet thesegroup includes in
ences have been ne are clearly defin among target use
project
l on a continuulevel but above
ation level, discutakeholders are inimes, discussionsrs in the project. ractice guidelinesnt process. AGREE instrumenal biases have mendations are
practice (AGREE 2orrectly consultede requirements : ndividuals from a
sought; ned; ers.
27
m of e the ussion nvited s may
s now
nt, for been both
2003, d and
ll the
28
Figure 10 : Hea
From Health Can
In the case of to assess accein existing guidkey stakeholde• clinical kno• personal ex• policy and • methodolog
appraisal, • information• implementa• and manag
alth Canada’s Pu
nada 2000, p.128
guideline adaptaeptability and appldelines. This task rs. Necessary ski
owledge in the topxperiential expertadministrative exgical expertise
n retrieval expertisation expertise, gerial and facilitati
ublic Involvemen
ation, the ADAPTlicability of the rehas to be performlls include (ADAP
pic area, ise (patient viewspertise, in guideline de
se,
ion skills.
S
nt Continuum
TE instrument is ucommendations amed by a panel i
PTE 2007, p.6)50 :
and preferences)
evelopment and
Stakeholder Involv
used e.g. as found ncluding
),
critical
Br
Anq
•
•
•
•
•
•
•
2Eogrefo
vement
Box 6: The ADAPrecommendation
Assessing whethenot is done by discquestions:
Does the popwhich the (acceptable)?
Does the intcontext of use
Are the interv(applicable)?
Is the necesscontext of use
Are there anyand/or resourthe implemen
Is the recommsetting where
Does the benmake it worth
2.4.2. Refine yEBM agencies pioown methodologicguidelines develoeport publication,orms along the pr
PTE assessment ns
er a recommendacussing each reco
pulation describedrecommendation
?
tervention meet e (acceptable)?
vention and/or eq
sary expertise (ke (applicable)?
y constraints, orgarces in the healthtation of the recom
mendation compait is to be used (a
efit to be gained fimplementing (ac
your process oneering in stakehcal handbooks thapment (see abov, stakeholder conroject developmen
of acceptability
ation is acceptablommendation in th
d for eligibility mais targeted i
patient views an
uipment available
knowledge and s
anisational barrier care setting of ummendation (app
atible with the culacceptable and ap
from implementingcceptable)?
holder involvemenat are quite similave). From projectnsultation occurs nt process.
KCE Reports
and applicability
e and/or applicabhe light of the follo
atch the populatin the local s
nd preferences in
e in the context o
skills) available in
rs, legislation, pouse that would imlicable)?
ture and values ipplicable)?
g this recommend
nt have producedar to standards ust scoping meetinin a limited numb
s 174
y of
ble or owing
on to etting
n the
of use
n the
licies, mpede
in the
dation
d their sed in gs to
ber of
KCE Reports 17
2.4.3. ChoosThe King Baudstep by step pconsultation. Faccount when c1. Objectives2. Topic: The3. Participan4. Time: Amo5. Budget: AvThen, the objeimpose to answneeded. Thesebe asked usingrefer to six quParticipatory Mbelow:
74
se your method ouin Foundation’sprocedure to ideirst of all, the fochoosing your mes: Reasons for inve nature and scopnts: Who is affecteount of time availavailability of resouective of the conwer some questioe questions aboutg a decision tree uestions synthesiethods Toolkit (K
s Participatory Meentify the relevanllowing five elemthod (KBF 2005,
volvement and expe of the issue ed, interested or cable urces nsultation and thons before evaluat nature and obje(see Figure 14). zed in The King
KBF 2005, p.13)51
S
ethods Toolkit hant method to cho
ments must be tap.12-15)51: pected outcomes
can contribute to s
he nature of the ating the time andctives of the projThese decisions
g Baudouin Foun and detailed in t
Stakeholder Involv
as built a oose for ken into
solutions
subject d budget ject may globally
ndation’s the table
Tty
G
Pg
In
Dg
R
Pg
g
h
vement
Table 1 : EBM agypes
Group types
Project scoping group
nventory groupg
Development group
Rating group46 ;
Peer-review grouph
See the VIKC See the NICE
encies’ stakehol
Size
Small, with physical presence
Small, with physical presence
Small, with physical presence
Medium, with physical presence and web-consultation Very large, with web-consultation (“hearings”)
C methodology in FIE and IQWIG flow c
lder involvement
Timespan
At the beginning
At the beginning
All along the project development process
In case of disensus
Before finalization, once or several times, during 4 to 8 weeks.
IGURE 13 charts in FIGURES 1
t’ tasks and grou
Task
To help framingproject, scopingresearch questiochoosing the appropriate methodology To identify a limset of questionsdevelop in the project To develop a project or guidefrom the selectioa topic to the identification of specific questioTo formalize thedegree of agreement needto select proposguidelines, etc. To comment scoreport plan, preliminary reposet of guidelines
11 and 12
29
up
g the g the on,
mited s to
line on of
ns e
ded sals,
ope,
ort or s
30
Table 2: Six qustakeholder in
Motivation
Targeted outp
Knowledge
Maturity
Complexity
Controversy
From: KBF 2005,
2.4.3.1. TheWhat is your mo• “The purpo
their own kdirectly con
• “The purpknowledgedecision-m
What is your ta• “The purpo
to enable explicit) or(mapping
• “The purpoon an issue
uestions to identvolvement
Democratiz
ut Mapping ou
A lot of cocommon kn
Most peopsubject or ttheir opinio
The topic iscomplex no
The topic controversi
, p.13 51
e objective of theotivation for undeose of using the mknowledge to creancern them” (dempose of using te, values and idemaking” (advising)
rgeted output ? ose is to generate a group to discl
r test alternative out diversity).
ose is to enable ae” (reaching cons
tify relevant cons
zation or advising
ut diversity or reac
ommon knowledgenowledge ?
le have already the subject is newns ?
s highly complex or technical ?
is highly contal ?
e consultation rtaking this stakeh
method is to enabate options for ta
mocratization). the method is eas that are rele).
a spectrum of oplose information strategies in a
a group to reach asensus).
S
sultation method
?
ching consensus
e exists or there
formed opinions w; people are still f
or technical or n
troversial or no
holder consultatioble participants tockling (policy) iss
to reveal stakeevant to the pro
ptions and informa(making tacit knpermissive envir
a single informed
Stakeholder Involv
ds with
?
is little
on the forming
not very
ot very
on? o employ sues that
eholders’ ocess of
ation and owledge
ronment”
decision
2T•
•
•
•
2AisTDK
vement
2.4.3.2. The naThink about the to “To what ex
knowledge of “To what exte
legislation on emergent that
“Is the subjecinformation is
“Is the issuepolarised, suc
2.4.3.3. The apAn extended set os available in thToolkit 51 and in thDecision Making 8
KCE projects and
ature of the subjpic or nature of thxtent does the f the subject?” (knent has the socie
the subject ? Dot norms have not ct highly complexrequired?” (comp
e highly controvch that consensus
pproaches of consultation aphe King Baudouihe Health Canada8. Some of these may offer good ex
ject he subject:
society alreadynowledge) ety already develoo strong views exbecome establish
x, such that a greplexity) versial and has s is difficult to reac
pproaches with stain Foundation’s a Policy Toolkit forapproaches havexamples.
KCE Reports
y possess a ge
oped opinions or xist or is the issu
hed?” (maturity) reat deal of (tech
the debate bech?” (controversy
akeholder involveParticipatory Mer Public Involvemee already been us
s 174
eneral
even ue so
hnical)
come y)
ement thods ent in sed in
KCE Reports 17
Table 3 : Cons
APPROACHES
Participatory Monitoring and Scenario BuildiConsensus conDeliberative poSearch confereStudy circles Study groups Think tanks Charrette Constituent assDelphi process
Retreats Round tables Advisory commplanning cell Interactive wwwOnline discussi
Televoting or wIssue conferencNominal group Workshops
i Ongoing p
managem
74
sultation approac
S
AssessmEvaluation (PAMng Exercise
nference lling
ence
sembly
mittee, board/coun
w/e-conferencing on groups/list ser
websurvey ces process
project : 2011-01 G
ment of localised pro
ches and applica
KCE PRO
ment, E)
Child Men Breast canGP burnou GP burnou
cil or All reports
rvers Alternative
osteopathProstate c GP burnou
CP A national clinic
ostate cancer
S
ation in KCE proj
OJECTS
ntal Health39
ncer screening ut42
ut42 s
e Medicines : y40
cancer screeningi
ut42
cal practice guidelin
Stakeholder Involv
jects
ne on the
F
F
vement
Figure 11: The NI
From NICE 2009, p.
ICE key stages o
1052
of clinical development
31
32
Stakeholder orgguideline at vabrings his/her increases the empirical evideof stakeholderNCCs to assimThe flow chartreports and itspreliminary repsubmitted commassessed with Institute will hosubmitted in wdebate are invitIn the VIKC mquestions (“knecomprehensiveassessed. Recquestions, selecIn the KCE Orecommendatiopotentially probby means of a cancer patients
j Ongoing p
managem
ganizations can carious stages duown sectional i
potential for connce for many comrelations for NICilate. t below describes two hearing moports plans or repments during thesregard to their r
old a scientific debwriting. This debated personally53. ethodology, the ielpunten analyse”e list with needscommendations wcted by web-cons
Ongoing Project ons to be scrutinblematic recomme
websurvey to pras.
project : 2011-01 G
ment of localised pro
contribute to and uring its developinterests. The m
nflict. This, combmmon interventionE and to the volu
es the productionoments inviting sports and amendse hearing momenrelevance to the bate on unclear aate is not public.
inventory group s”). This inventorys and sticking p
will have to answsultation.
2011-01 GCPj, ized by KCE exp
endations was duractitioners involve
CP A national clinic
ostate cancer
S
comment on thepment. Each stakmultitude of stake
ined with the pans, adds to the coume of comments
n process of thestakeholders to cdments to it. All cnts will be considereport. If necess
aspects of the co The participants
selects only contry group then propoints which sher a limited set o
to limit the numperts, an inventoring one month pred in the care for
cal practice guidelin
Stakeholder Involv
e clinical keholder eholders aucity of mplexity s for the
e IQWIG comment correctly ered and sary, the omments s of the
roversial duces a ould be of these
mber of ry of 19 resented prostate
ne on the
F
“F
vement
Figure 12: The IQ
Flow chart for repor
QWIG production
rts”, available on the
n process of repo
e IQWIG website 54
KCE Reports
orts
s 174
KCE Reports 17
Figure 13: TheKCE project on
Flow chart based
74
e VIKC key stagen Prostate cance
d on VIKC methodol
s of guideline deer
logy55.
S
evelopment used
Stakeholder Involv
d in
Fr
F
vement
Figure 14: The Corelevant participa
From KBF 2007, p.3
ock Burning & Hatory methods
3215
Honingh decisionn tree to identify
33
34
Figure 15: Com
From KBF 2005,
mparative chart f
p.27 51
for participatory
S
methods
Stakeholder Involv
2
Sinnepre•
•
2AimweimAcsa
vement
2.5. Dialogue “Social cartocomplicated, knots.” (Ventu
Stakeholder consnvolvement. It henetwork, to nurtuexpectations. A sprocess will be chesearch area: In the busine
characterized In the resea
experts, decisan issue and problem (NCC
2.5.1. DialogueAs stakeholders mplementation of with the effectiveeffects, their effecmplementation (coAmongst these dcomplex one. Mscrutinized but alabout acceptability
in controversieographers knowsubtle and ever-c
urini 2009, vol1, psultation is only elps to establish ure a dialogue stakeholder dialoharacterized diffe
ess area, a dialog by negotiation (IFrch area, a dialosion makers and discuss solutions
CHPP 2010, p.14-
e beyond effectivare actors in
f a policy 57, dialogness of public pcts on equity, anost, feasibility, andimensions of pu
Moreover, acceptalso may be “pery may evolve duri
es w that issues
changing to be s.11)37 one preliminarytrust and credibiprocess and to
ogue organized erently in the bus
gue that follows tFC 2007, p.63-68ogue that can beother actors, aim
s they judge suita-15)56.
veness nvolved with thgue with stakeho
policies but also nd mainly with isd acceptability) (Tblic policies, accability is a dimrformed”, i.e. judng stakeholder di
KCE Reports
are always toosliced like Gordian
y step in stakehility, to strengthe
foster more reaafter the consul
siness area and i
he consultation st)7. e established bet
ms to critically exaable for addressin
he objectives alders will not onlywith their uninte
ssues related to Table 4). ceptability is the
mension that magments and posalogue.
s 174
o n
holder n the alistic tation n the
tep is
tween amine ng the
and/or y deal ended
their
most y be
sitions
KCE Reports 17
Table 4 : Dime
Effects
Implementation
From NCCHPP, p
74
ensions for analy
Effectiveness
Unintended effects
Equity
n Cost
Feasibility
Acceptability
p.456
yzing public polic
s What effectstudy have
What are tthe policy?
What are groups?
What are tpolicy?
Is the policy
y Do the relethe policy a
S
cies
ts does the policon the targeted p
he unintended ef
the effects on
he financial cost
y technically feasib
evant stakeholdes acceptable?
Stakeholder Involv
cy under roblem?
ffects of
different
s of the
ble?
ers view
B
F
Sb
Feke
E
•
•
•
•
•
•
•
F
vement
Box 7: Acceptabi
First, it involves su
Secondly, it is inflbeing studied, amo
Finally, the accepexternal to it: theknowledge, belieetc.).
Examples of Key
Which actorsconsidered?
Is the problemissue that me
Is it on the dis
What are stakthis problem?
What type of issue?
What do theyunintended efdegree of coe
What do theyimplementatio
From NCHPP 2010,
ility as the most
ubjective elements
luenced by all theong other things.
ptability of a pole position of eacefs, values and in
y Questions: Acc
s are or will be
m the policy aimsrits intervention?
scussion agenda?
keholders’ reactio
intervention do st
y think of the proffects, its effects oercion it involves?
y think of the coon of this policy?
p.9-1056
complex dimens
s (the judgment of
e other dimensio
licy also dependch actor is deternterests (politica
ceptability
affected by the
s to address con
?
ons to the idea of
takeholders propo
oposed policy? Oon equity, its costs
onditions surround
sion of the analy
f actors).
ns of the public p
s on factors tharmined by that al, economic, sym
e public policy
sidered to be a s
intervening to ad
ose for addressin
Of its effectiveness, its feasibility? O
ding the adoption
35
ysis
policy
at are ctor’s
mbolic,
being
social
dress
g this
ss, its Of the
n and
36
2.5.2. Dialog
“Knowledgresearch wsystematic knowledge
Recent developthese judgmentfor political dep.10)56. Withinprocesses areother means (litdeliberative pro15)56: • Combining• Contextual• GeneratingBox 8 : The De
“A typical delismall group oprocesses havemaker communthe public, pabackgrounds, ina specific issuethe issue is usubefore the seParticipants areask further quesmall sub-grothemselves, witnot a consensudeliberative prodays.”
gue to produce kge synthesis is a swith information froc and transparente” (Bos 2007, p.8)5
pment in knowledts because “thesecision makers thn the context e set up to enrichterature, consulta
ocesses can fulfil
different forms ofizing data drawn
g new knowledge eliberative Proce
iberative processof participants (soe been used to nities but are alsoatients and othernterests and valuee and to debate ually provided in aession or throue often given the
estions before begoups, participantth the goal of dev
us, to inform decisocess exercise c
knowledge synthstrategy for combinom policymakers t way in order to 58. dge synthesis atte perceptions oftehan objective evi
of knowledge h and contextualiation, comments m
at least three role
f “evidence” from the literature
ess – A Snapshot
s exercise involveomewhere betwebring together th
o relied upon to brrs. Group membes, meet face-to-fpotential options
advance, either thugh presentations opportunity to ch
ginning deliberatiots then debateveloping some forsion-making. Depcan range from o
S
hesis ning information fand practitioners promote the use
tach much importen carry more impdence” (NCCHPsynthesis, delib
ize the data collemanagement, etc.es (NCCHPP 201
e on the subject
t
es engaging a reen 10 and 40)he scientific and ring together members, who bring face to weigh evid. Information perhrough assigned rs by relevant hallenge the expe
ons. In a large groe the issues armal recommenda
pending on its struone day to three
Stakeholder Involv
from in a e of
tance to portance P 2010, berative ected by ). These
10, p.14-
relatively . These decision
mbers of diverse
dence on rtinent to readings experts. erts and oup or in amongst ations, if ucture, a e or four
F
E
DreBeforofiafe
Fm
vement
From CHRSF 20095
EXAMPLE : The d
During one dayecommendations
Belgium.. These experts and wereoreign experienceounds with GPs inal workshop wiafter which the staeasibility.
From KCE Reports management] “41
59
deliberative proc
y, stakeholders to prevent and
preliminary recome based on literaes, qualitative inteand stakeholdersth stakeholders,
akeholders assess
165 “[Burnout amon
cess in KCE proj
have deliberated treat burnout mmendations weature review, a serviews with GPss and expert coneach recommendsed its level of pri
ng general practition
KCE Reports
ect
ed about prelimamong the GP
ere proposed by study of nationa, a Delphi study insultations. Durindation was discuiority, acceptability
ners: prevention and
s 174
minary P’s in
KCE l and n two g the ssed, y and
d
KCE Reports 17
2.5.3. DialogBesides observspaces to perfoDebategraph©)arrangedk in co
EXAMPLE : DeThe Debategrastakeholders inmonths debateaccess to profregulations. Pa
From KCE ReporBelgium]40
k As the co
in (Ventur
74
gue beyond conving and represeorm them. Places)- where disputes-production activit
ebategraph© useaph© was used tn osteopathic fielde about issues fession, patient rticipants were tra
rts 148 “[Ostheopat
ontroversy-websites rini 2010, p.18)38.
troversies enting public des –it may be digits are collectivelyties.
ed in KCE projeco perform a debds. This was intelike specificity osafety, competen
ained to use this to
thy and chiropractic:
in the cartography
S
bates, we may tal ones (see exay elaborated and
ct ate on the web bended to produceof osteopathic mnce profile and ool.
: state of affairs in
y of controversies d
Stakeholder Involv
imagine ample of d maybe
between e a two-
medicine, financial
described
2
2AcqawTc•
•
•
IQcInec•••••
vement
2.6. Managem“A good prosatisfaction thoutcome is les
2.6.1. AcknowlAs stated in thcomments, commquality-assurance are addressed apwhen responding tThe following key comments from sta Each commen
factually as po If changes are
must be made Responses a
whole GuideliQWIG has also comments during n these two guidelements in the considered: transparency publication ru accessibility to response time good records
ment of stakehoocess can enhahat their complaiss than optimal” (
ledge comment e NICE protoco
ments received froand peer-review
ppropriately (NICEto comments on tpoints should be akeholders: nt must be acknoossible. e made to the gue clear in the resp
and changes musne Development developed spec
“Hearing” momenelines and in themanagement of
of the process les o comment procee and feed-back
olders’ commeance outcomes ints have been h(IFC 2007, p.68)7.
and respond ol to manage gom stakeholders processes, and it
E 2007, p.123)35. he draft scope antaken into accou
owledged and ans
uideline as a resuponse. st be made with Group before pubcific guidelines onts13 (see examplee IFC good pract
stakeholders’ co
ss
nts and give people
heard, even if the
guidance consultare a vital part ot is important thatNICE principles
nd later. nt when respond
swered as fully an
lt of the comment
the agreement oblication. on the submissioe). ice handbook, seomments have t
37
e e
tation of the t they apply
ing to
nd as
t, this
of the
on of
everal to be
38
EXAMPLE : ThIt is mentioned
• the person(publication o
• all authors potential cosummarized
• the deadline
• the proceduin a separareport plan.meeting minunpublishedpublished.
From IQWIG 200
he IQWIG managin IQWIG guidelin
s) submitting comon the IQWIG web
of comments shonflicts of intered in the documenta
e is 4 weeks after
re regarding the sate document pu. Comments fulfinutes of the sciend documents attac
0753
ement of stakehnes on the submis
mments explicitlybsite;
hould complete aest. The connecation of comment
publication;
submission of comublished simultanilling the formal ntific debate, will ched to comments
S
olders’ commenssion of comment
y give(s) their con
a form for discloctions disclosed s and published.;
mments will be deneously with the
criteria, as wellbe published. Pr
s as evidence will
Stakeholder Involv
nts s that:
nsent to
osure of will be
escribed revised
as the reviously l also be
Er
F
vement
EXAMPLE : Exreceived on the N
From NICE 2007, p.
xamples of resNICE guideline fo
8135
sponses to staor schizophrenia
KCE Reports
akeholder comma
s 174
ments
KCE Reports 17
2.6.2. Use foWeb-based proor IQWIG procea standardizedp.80)35, all comin a Word file, responses fromfollowing inform• Organisatio• Chapter/se
consultatiothe identific
• Commentsunchanged
• ResponsesIn addition, literutilization highlithrough forumssuch a way, thhas to assesscollected in muback to the paplanned in thescientific debatecomments.
74
orms and forumsocedures of stakeedures) propose fd way. As specifments received bwhich is sent to
m the Guideline Dmation. on – name of orgaection – dependinon. This column cacation of commens – comments d. s – blank column frature on the efficights the importans aimed to facilite ADAPTE metho
s acceptability anltiple existing guid
anel to be discuse IQWIG processe may be held to
s eholder commentsforms to collect afied in the NICE
by NICE are enterthe National Collevelopment Grou
anisation that subg on the documean be sorted by th
nts by section. received from
for the developerscacy of interactionnce of face-to-factate interpretationodology proposesnd applicability odelines; and a forssed. Discussion s after “hearing”
discuss any uncl
S
s management (aassessment informE guideline (NICEred into a commenlaborating Centre
up. The table cont
mitted the commeent that has been he developers to
stakeholders,
s to complete. n approaches in rce contact and intn of research ress a form when thof the recommenrum where resultsabout commentsmoments where ear aspects of the
Stakeholder Involv
as NICE mation in E 2009, nts table
e to elicit tains the
ents. sent for facilitate
entered
research teraction sults5. In he panel ndations s are fed s is also
an oral e written
2Wsp123AtoothcC
2
AcupreresToc
l
vement
2.7. StakeholdWe may talk abostakeholders are projectl :
. shaping the p2. implementing 3. and interpretinAt this level of invo define stakeholdof this report) becheoretical modecorresponds to lContinuum (see F
2.7.1. Considemonitorin
At such a level ofcompared to partiused “as an umbrphilosophy of inclesearch processesearch) those w
stakeholders of theThe added value oof a research projcritical review of th
Phases desc
der involvemenout stakeholder in
involved through
urpose and scopethe research and
ng and applying thvolvement, the “brder involvement in
comes a lively nols, this is the level 5 in the igure 10).
r added-values ng f stakeholder invocipatory researchrella term for a susivity and of rec (rather than incwho are intendee research” (Cargof stakeholder invject could be sumhe PR literature. (C
ribed in Cargo 2008
nt in project monvolvement in prohout the entire p
e of the research, d considering conthe research outcoridging dynamic in the economic ation in the researhighest level oHealth Canada’s
to involve stake
olvement, a reseah. “Participatory reschool of approaccognizing the valcluding them ond to be the ben
go 2008, p.326)45. volvement throughmmarized in tableCargo 2008 ,p.33
8, p.33745.
onitoring oject monitoring
process of a rese
textual factors, omes. nterdependency” rea (in the first chrch area. As figurof involvement ws Public Involve
holders in projec
arch project has esearch” (PR) maches that share aue of engaging ily as subjects oneficiaries, users,
hout the entire pro 5 that is based 8-339)45
39
when earch
used hapter red in which ement
ct
to be ay be
a core n the
of the , and
ocess on a
40
Table 5: The p
Level of stakeh
Shaping the sresearch
Research impl
otential added v
holder involveme
scope and purp
lementation and
value of participa
ent Po
pose of the •
••
context Co•••
• Ca•••••••
S
atory research ap
otential added va
Enhanced relehealth systemResearch is reInitiation of owresearch
ontextual advantaResearch is lesEnhanced credLinkage of stubuilding Ethical agreem
apacity, empowerMore targeted Strengthened sIncreased capaAcquisition of sEconomic deveAcquisition of mDevelopment o
Stakeholder Involv
pproaches acros
alue
evance and import
sponsive to the cownership, empow
age: ss disruptive to imdibility for other acdy participants w
ments negotiated w
rment and ownersand efficient plansense of ownershacity of non acadespecialized researelopment through management and of decision-making
vement
s the three phas
tance of research
ommunity of interewerment, and ca
mplementing contectivities due to par
with needed health
with academic par
ship: ning and problemip through active
emic partners to drch knowledge, skemployment oppleadership skills
g skills
ses of the particip
h questions to the
est apacity building t
exts rticipation in PR ph care resources
rtners address co
m solving participation in re
do PR kills, and experienportunities and loc
patory research
e organization, co
through active p
projects by treating resea
ncerns of the com
esearch activities
nce al resource utiliza
KCE Reports
process
ommunity, or publ
participation in th
arch as awarenes
mmunity of interes
ation
s 174
ic
he
ss
t
KCE Reports 17
Level of stakeh
Interpretation research outco
From Cargo 2008
74
holder involveme
and applicatiomes
8, p.338-33945
ent Po
on of the Ca••
•
•• In••• Pa•
Su•
•
•
S
otential added va
apacity, empowerTimely feedbacEnhanced captranslation Enhanced undact Increased capaEnhanced med
strumental use ofEnhanced cultuCreation of invImproved form
articipation: Potential for hdevelopment
ustaining the partnMore effective capacity Augmented intcatalyzed by paImproved linka
Stakeholder Involv
alue
rment, and ownersck of research res
pacity, empowerm
erstanding of hea
acity for health prodia and education
f scientific knowledural and contextuaentories, training ulation of policy re
higher interventio
nership and reseaapplications for f
tersectoral mobiliarticipation in PRges among comm
vement
ship: sults to non acade
ment, and ownersh
alth problems, the
omotion al capabilities
dge: al relevance of demanuals, and hanecommendations
on participation r
arch products: funding and lever
zation of leaders
munity-, state-, and
emic partners and hip from participa
ir root causes, an
eveloped interventndbooks to informand policy chang
ates when end
raging of resource
s, volunteers, age
d federal-level ag
the community oating in research
nd solutions can g
tions, program plam practice
es
users are involv
es due to establis
encies, institutions
encies
f interest dissemination an
galvanize people t
anning, and action
ved in interventio
shed credibility an
s, and businesse
41
nd
to
n
on
nd
es
42
2.7.2. Consi“Being atteveryone th
Stakeholder invEBM agencies’involvement mthroughout thechapter), involvcomments (SeeYet, the stakehEBM agencies • either by li
everyone’s• or by exte
around maof concern
Agencies pionereveal matters consequences,way they deal wof controversy, highest levels o(see examples)
m A distinct
objectivity
ider the way to dtentive to all vihe same status” (volvement in pro processes scruti
may be performede entire processvement is usuallye the HAS figure).holder involvemehave chosen to dmiting their missio
s agreement; nding their missio
atters of concern a. eering in stakehoof concern (stak …) beyond mattwith these matterthe core philoso
of involvement m).
ion made by Bruno
y and the second-de
deal with controviewpoints does (Venturini 2010, poject monitoring hnized for this repd at an early st
s of project devy limited to cons nt level may varyeal with controveron to identify the
on to reveal the and by choosing t
older involvement keholder intereststers of fact (literas of concern mayphy of inclusion tay be weakened
o Latour between
egree objectivity (Ve
S
versy not mean gran.4)38.
has never been fport. Although staktage of the proj
velopment (see tsultation, i.e. che
y depending on rsym matters of fact th
full range of oppto deal with these
enlarge their mis, anticipation of ature evidences). y differ extremely.that is at the heaby exclusion pro
the positivistic ‘firsenturini 2010, p.3)38
Stakeholder Involv
nting
found in keholder ect and the first cks and
the way
hat meet
positions e matters
ssion to possible But the
In case art of the ocedures
st-degree’ 8
F
F
vement
Figure 16: HAS p
From HAS 2010, p.4
procedure followe
446.
ed in formal con
KCE Reports
sensus method
s 174
KCE Reports 17
2.8. RegulaA follow-througkeep accountaon “communicamore constructiabout the projeeffect on them.
2.8.1. Plan yDetermine at an• what inform• to which st• by what me• and how fre
2.8.2. DisplaInternational strecommend to the research ar2008, p.37)60 :• the purpos• the nature • how these • what policie• what the E
questions.
2.8.3. Adaptinvolv
Information reinvolvement: • for intere
involvemeinvolvemen
74
ar feed-back to h principle has to
ability at its higheation at an earlyive if stakeholdersect, its impacts, a
your communican early stage of thmation needs to beakeholders, ethod, equently.
ay relevant informandards in stakebrief stakeholders
rea, this relevant i
e and scope of thof the research qquestions are cures and services aEBM agency can
t your feed-backvement elevancy also d
ested stakeholdent process as ant (IFC 2007, p.88
stakeholders o be applied in stst level. As alread
y stage”, stakehos […] have accuraand any other as
ation he project (IFC 20e reported,
rmation holder engagemes with relevant infinformation should
e stakeholder invuestions; rrently managed w
are already in placn and wants to d
k to the level of s
depends on the
ders, informationa whole has to be8)7;
S
takeholder involvedy stressed in theolder involvementate and timely infospects that may
07, p.88)7:
ent in the economformation. Translad address (Accou
olvement;
within the EBM agce; do about these r
stakeholder
e level of stak
n on the stake displayed to en
Stakeholder Involv
ement to e chapter t will be ormation have an
mic area ated into untAbility
gency;
research
keholder
keholder ncourage
•
•
•
2EfeowE2acSbs
2Rppa
n
o
vement
for consulteresponses tochapter Mana
for main staksufficient levresponsibility;
for end stakprocess is neneed for the themn.
2.8.4. Use elecElectronic mediaeedback informatof stakeholder. EUwebsite, to give sEUnetHTA and e2009, p.88)14. NICas a task to sendcurrent publicationStakeholder meebriefings are othestakeholders (Acco
2.8.5. Give regRegularity of feedproduction proceproviding clear andalso corresponds
Following th
stakeholders decision or presults and re
This was onnew program
ed stakeholderso their comment agement of stakehkeholders being el of details is
keholders, informeeded to make suresults, so that t
ctronic media ana are widely usion on the project
UnetHTA has put takeholders targe
establish a platfoCE and IQWIG had to all interestedns and new commetings, one-to-oer means recommountAbility 2008,
gular feed-backd-back is fundamesses (see figurd ongoing commus to stakeholde
he definition of D
who – at the complanning processesecommendations (Ke result of the CC
m COMPUS61.
s, appropriate ais needed to susholders’ commentrepresented in th
s required to su
mation about the ure that they undhey expect them
nd meetings ed to collect cot development proa Stakeholder Op
eted information arm for virtual coave both an inford parties informa
missions advertisedone meetings, mended to give cp.41)60.
ental in IQWIG, res). To maintaunications is a cleers’ needso. As
DACEHTA, “end smpletion of the projes and who can asKristensen 2007, p.2OHTA stakeholders
acknowledgment stain involvementts); e project monitorustain feeling o
project developerstand that thereand will make u
omments and to ocess to large numpen Forum page about developmenommunication (Nirmation service hation on new prod. follow-up telep
consistent feedba
NICE and HAS rain transparencyearly stated objects underlined in
stakeholders” are ect – will be close sist in implementin
25)10. s consultations abo
43
and t (see
ing, a f co-
pment e is a
use of
give mbers on its nts in elsen aving
ojects,
phone ack to
report y by tive. It the
those to the
ng the
out its
44
EUNETHTA stabe the only wa(Nielsen 2009, In participatory activity, the chasustainability design of parpartnership willaction and instit
2.9. Skills toHow to play aPositive and guaranteed, neenablers are foa status quo, faand many othertowards stakeh• from res
multistakeh• to enablers
involvemenAttitudes, behavneeded to overc
akeholder involvey to “prove” that p.86)14. research, regular
allenges of which (Cargo 2008, p.3
rticipatory effortsl continue and thtutionalized” (Carg
o manage the a significant role proactive attitudeeither by stakeh
ound both sides, dailure to recognizr reasons (Griffitholder involvement
sistors who dholder working s who have identnt (see FIGURE 1viours and rationacome the barrier o
ement policy, “trano interests were
r feedback is consare to maintain t
334)45. “[…] consids can enhance hat research resugo 2008, p.337)45
process of stakin a stakeholde
es in stakeholdeholders, nor by due to vested intee the potential ofs 2007, p.20)62. Tt is very large :
don’t even cre
ified a clear role f8).
ales may evolve wof inability.
S
ansparent processe inappropriately
sidered as a mainrust and respect dering sustainabilthe likelihood tlts will be transla
5.
keholder involvr involvement prer involvement experts. Resisto
erests, interest in f stakeholder invoThe spectrum of a
eate opportunit
for themselves in
with experience. S
Stakeholder Involv
ses may favored”
ntenance to foster ity in the that the ated into
vement ocess ? are not ors and keeping
olvement attitudes
ies for
terms of
Skills are
F
F
2Treesbusa2Doep
vement
Figure 17: The en
From Griffiths 2008,
2.9.1. Be trustwTrust and mutuaesearch (see figu
elements have to stakeholder involvbeginning of the understand the cosidestepped withoand respect, which2007, p.335)45. DACEHTA stressoutcome require ethics among theproject manageme
nablers/resistors
p.2062
worthy and respl respect are th
ure in chapter Combe established anvement. This can
study. “Taking tontext, the peopleout compromisingh, if violated, can
es that construcgood cooperative
e project participent skills are requ
s spectrum
ectful e core elements
mmunication at annd maintained thron be time-consumthe time to becoe, their culture, an the fragile foundhamper methodo
ctive meetings ane behaviour withpants (Kristensenired (see BOX 9).
KCE Reports
s of any participn early stage)45. Toughout the proce
ming, especially aome familiar withnd priorities canndations of mutualological quality” (C
nd, ultimately, a h trust and high 2007, p.27)10. .
s 174
patory These ess of at the h and not be trust
Cargo
good work
Good
KCE Reports 17
Box 9: DACEHand rewarding
It is important in
• Problem fopolicy ques
• Clarificatio
• informationevidence?
• Establishm
• Planning o
• Planning oresults to b
• Planning ohow do we
• Strategy re
From Kristensen
74
HTA recommendag cooperation
nitially to agree on
ormulation and destion) and what do
n of the alternativ
n search strateg
ment of time sched
f the work phase –
of the completion –be presented?
of the implementae do it?
egarding follow-up
2007, p.2710
ations to create t
n:
elineation – what o we want to study
es to be studied
gy – what is av
dule/meeting plan
– who does what
– how, in which fo
ation – if changes
p and problem solv
S
the basis for a g
is the main quesy?
vailable and how
for the project pe
and when?
orm and to whom
s are to take plac
ving of derived qu
Stakeholder Involv
ood
tion (the
w is its
eriod
m are the
ce, then,
uestions
2Wr(sT
A
In
CLo
F
Teqin•••
TEm•••
vement
2.9.2. Be involvWhy should any rationales for stasee Table).
Table 6: Rational
Altruism
nvestment
Compulsion Lost opportunity
From Griffiths 2007,
These rationalesexpert/researcher questions, issuesnvolvement2: the degree of the politicizati the establish
policy is a chaTo answer these qEBM agencies pimeans : clearly stated publication of research (col
EUnetHTA Pr
ved potential stakeho
akeholder involve
e for stakeholde
we get involvedto do irrespectivwe get involveda return on our we get involvedwe get involvedgreat that we competitors aremaintain our po
p.962
s may be fogroups. But we
s and challeng
independence reon of research is ment of differentallenge. questions, solve toneering in stake
rules in standardsgood practices orlaboration) dedicaroject WP6).
older wish to bement are multiple
er involvement
d because we belive of cost d because we perinvestment because we hav
d because the pocannot afford
e doing it, thus weosition
und in a sime have to take ges when talkin
searchers can sean issue, t relationships b
these issues and eholder involvem
s, guides, protocor lessons learned;ated to stakehold
come involved ?e and probably m
ieve it is the right
rceive that there w
e been told we haotential benefits a
not to, or thate must do the sam
milar manner winto account sp
ng about resea
cure is a question
between research
win these challement are using se
ols or handbooks; ; der involvement (
45
? The mixed
thing
will be
ave to are so t our me to
within pecific archer
n,
h and
nges, everal
(as in
46
2.9.3. Be exTo support stathis report, specomponents ofbelow : • to recogniz
be weighed• to take into
and perple• to overcom
(traditore=t• to accept p
(in chapterprocess);
• to review dialogue (in
• to hear, acand querie
• to maintainStakeholde
• to developexperts’ intstakeholde
These skills can
xpert keholder involvemecific skills havef stakeholder inv
ze that research ed up in making poo account a lot of xity (see chapter me inherent probtraditere) (in chapphysical presencer Stakeholder con
one’s own judgn chapter Dialogucknowledge and ms (in chapter Mann a good power er involvement in p an interesting sterest) at its high
er). n be improved by
ment requires moe been overviewvolvement. These
evidence may not olicy decisions (senew connections Stakeholder identblems of betraya
pter Communicatioe of stakeholderssultation all along
ments and positue in controversiesmanage stakeholdnagement of stake
balance, even inproject monitoringstudy to keep staest level (in chap
experience or tra
S
ore expertise. Thrwed for each of e skills are sum
arrive as simple “ee introductory cha
by maintaining optification and analal in translation on at an early stags without any screg the project deve
tions during staks); ders' comments, eholder's commenn controversy (in g); akeholders’ interepter Regular feed
aining.
Stakeholder Involv
roughout the key
mmarized
“facts” to apter); penness lysis); process
ge); een filter elopment
keholder
critiques nts);
chapter
est (and -back to
2HqspthoBp
•
•
•
•
•
•
•
•
F
vement
2.10. guiding pHow can researcquestion regardisummarized a ranprinciples, that cohe backbone of thof stakeholder invoBox 10: Some gupractice
Research mus
Ownership is
The need for e
Conduct a con
Ensure credib
Provide leade
Give adequate
Develop integ
From Nutley 2008, p
rinciples ch inform public ng evidence-bange of guiding prntribute to the likehe first part of thisolvement.
uiding principles
st be translated
key
enthusiasts
ntextual analysis
bility
ership
e support
gration
p.3122
services ? To sed policy, Nurinciples (see BOelihood of evidens report presentin
to support the u
KCE Reports
answer this prautley and al.
OX 10). These guce-based practice
ng the key compo
use of research in
s 174
actical have
uiding e, are nents
n
KCE Reports 17
3. CONEXPE
3.1. RationaBefore embarkprocedures, theprevailing opiniothe opportunityintegrate stakespecifically, thefears, and askIncidentally, theinvolvement, sitheir daily woinvolvement poThe ultimate oapproach in mainto the own potential internmeasures or ad
3.2. MethodThe format choallow for eachexchange betwviewpoints andallow every sinwe opted for a The meetings tNovember 2011In preparation owas drawn umanagement a
74
SULTATIOERTS ale and objectiing into new and e management ofons and attitudes y to contribute ieholder involveme researchers wered to share theire very exercise itsnce the KCE expork by whatevelicy as an organis
objective of this atters of stakeholdcollective knowle
nal barriers and dapt the stakehold
d osen for this conh participant to eween participants,
opinions, and cgle KCE researchseries of discuss
took place in the1. of the discussion up, after severand project team. (
ON ROUND
ves more formalised
f KCE wanted to of the KCE reseaideas and sugge
ment in the resre also polled abr own positive anself was a tangiblperts will be the fir decision to c
sation. consultation rounder involvement medge and experi
threats and byder involvement p
sultation round wexpress him or so as to bring uplarify them in a dher to optimally psion groups with e last half of Octo
groups, a list of qal discussion se(see BOX 11).
S
D OF KCE
stakeholder invoget a better insigarchers, and to ofestions on how search processesout their resistannd negative expee instance of stakrst ones to be affchange our stak
nd is to make tmore effective, byence, by identify
y taking the approcedures accord
was discussion grherself, and for
p in the discussiondialectic way. In participate and comax 8 participanober and the firs
questions to be dessions with th
Stakeholder Involv
olvement ht in the ffer them
to best s. More ces and
eriences. keholder fected in keholder
he KCE y tapping ying the propriate dingly.
roups to a lively
n shared order to
ontribute, nts each. st half of
ealt with he KCE
B
Q
•
•
•
•
•
Q
•
•
•
•
•
vement
Box 11: List of qu
Question 1 – WHO
What are theAuthorities, PAcademia?
What is yostakeholders?
What do yostakeholders?
Do you have ones and why
Do you have relations and
Question 2 - HOW
Many differenis your knowle
Which onesdisadvantages
These methoparticularly pr
Who must inexternalization
Is training or s
uestions for the
O?
e different types Payers, Private In
ur knowledge o?
ou think about ?
fears about the y?
any suggestions tprocesses connec
W?
nt types of methodedge of and exper
s seem more s?
ods require somroblematic to you?
nvest in these mn?
support necessary
consultation rou
of stakeholdersndustry, Patients-
of / experience
the involvemen
involvement of s
to improve your ccted with these?
ds are mentioned rience with these
useful to you
me specific skills? Why?
methods? Internal
y? Your suggestio
und of KCE expe
s : Providers , P-Citizens + Media
with these diff
nt of these diff
some of them? W
capacity to manag
in the document. different methods
u? Advantages
s. Which ones
specialization a
ons?
47
rts
Public a and
ferent
ferent
Which
ge the
What s?
and
seem
and/or
48
• Question 3
• The reporGCP - andyour opiniosubjects? Isome of the
• Why? No Timing? Hu
Question 4 – D
• Different d(see chaptdimensions
Question 5 – O
The participantreport in preparThe moderator knowledge of thof observer andsessions were tAt the beginninwere briefly remexplained and cwas guaranteeresults would cindividual particAfter each sesobserver. The min depth listeniobserver. No fu
3 – WHY?
rts produced by Kd a large spectruon about the staIs it relevant for aem?
added value? Tuman resources a
IMENSIONS OF I
dimensions of invter 4). What ares?
THER SUGGEST
ts were asked to ration to the discu
of the discussionhe KCE and the ‘ed note taker was gtape recorded, anng of the sessionminded to the paconsent was aske
ed, in the sense contain no names cipants. ssion, there was moderator drafteding to the record
ull transcripts were
KCE cover three um of subjects inakeholder’s involall types of subje
Too dangerous? available ... Any su
INVOLVEMENT
volvement are dee your opinions/s
TIONS AND REM
read a draft verussion. n was an externaenvironment’ in wgiven to the KCE nd had a duration n, the objectives articipants, the rued about the tape that the reportinor other recognis
a short debriefind detailed minutesdings, complemene made.
S
fields - HTA, HSside these fieldsvement for the
ects or must be li
Risk for indepenuggestions?
escribed in the dosuggestions abou
ARKS.
rsion of chapter 4
al consultant with which it operates.
knowledge manaof 2 hours. of the discussio
ules of the exercirecording. Confidg and publicationsable details perta
ng of the moderas of each session,nted by the note
Stakeholder Involv
SR, and . What’s different mited to
ndency?
ocument ut these
4 of this
a prior The role ger , All
on group se were dentiality n of the aining to
ator and through s of the
Fli
3TethmmtrMW7pthin
3MTninilApAdMTewpre
vement
Finally, the four sesting all key mess
3.3. Results The groups discuexpress them is sohe key points amessages, but onmessages are simranslations of the Messages have Weaknesses, Opp7, 9, 14, 15, 16, 1person inside this he group. Other mn the four groups
3.3.1. LessonsMessage 1. StakeThe four groups enot new for KCEnvolvement. Sevelustrate methods
A grid has been practical way, thiAppendix providediscussions. Message 2. BoardThe KCE board except for a direcway to automaticaphase of the projecommendations
ession minutes wesages coming from
ussions resulted ometimes differenare the same. Enly selected exammilar. The provide
original messagebeen grouped in
portunities and Tre19 were expressegroup, but withou
messages were ex(1, 2, 6, 8, 10, 12
s learned eholder involvemxpressed the sam
E. KCE has alreaeral examples weor report some leelaborated to p
s grid will be ues grids build o
d's involvement has representativ
ct representation ally get some formects. Yet, the intis not without stir
ere synthesised im the discussion
in several messnt from group to gExamples are g
mples are includeed examples are es (expressed in Dn 4 categories weats) approach ined only by one grut contradiction byxpressed in 2 to 3, 13, 18).
ment is a long stame opinion: stakehady much experiere provided duri
essons learned. present the diffesed in the forthcn the examples
is not optimal ves of the main of the patients.
m of stakeholder iterference of the rring some disquie
KCE Reports
n a final overall regroups.
sages. The way roup, but concept
given to illustrated if the content oas truthful as pos
Dutch or French). with SWOT (Stren mind. Messagesroup and often byy the other membe3 groups (5, 11, 1
anding practice holders’ involvemence of stakehoing the discussio
rent approaches coming process
provided during
stakeholders grIt is perceived asnvolvement at theboard with the p
etude.
s 174
eport,
they tually, e the of the ssible
ength, s 3, 4, y one ers of 7), or
ent is olders’ ons to
in a note.
g the
roups, s one e end policy
KCE Reports 17
“The mostwhere all opportunityabout their “The steprecommen“How to iaccepted. Sbecome ridthe recomm
Message 3: InvSome collabordecision making
“The Commof directorsbeginning competenc
Message 4: To“To be und“Be undersinvolvemen
Message 59: DThe experienceas not experts regularly percei
“The probleexperts. Thbetween eThe stakeh“The objecdifferent. experiencegenerated
74
t important groupthe decision m
y to comment or tor implementation.”p between the dations is not easimprove this? It Some recommenddiculous. Sometimmendations.” volvement of comators stressed thg munities and Regs. They should be
of the projeccies.” o be understood derstood by the othstood by is a misnt. This is importaDistinction betwee with so-called etend also to be
ived as rather uncem is to make a he best way shou
experts and stakeholders are often tctives of expert mIt is important
e, knowledge ... by these meeting
p of stakeholdersmakers are prese
o reject recomme”
conclusion of sy – This is a diffic
is complex; somdations are so mu
mes we are disapp
mmunities he importance to
gions are not repre acknowledged acts. They have
by her ones is cruciassing dimension i
ant.” een stakeholdersexpert meetings astakeholders to a
comfortable or disdistinction betwe
uld be to organizeholders. But it isthe experts for someetings and stake
to define exacand to think abo
gs.”
S
s is the KCE boent. They have endations and to th
a study and cult issue.” me reports are uch reduced that tpointed ourselves
o involve other le
resented in the boand involved from
more and m
al ...” in the dimensions
s and experts at KCE is that moa varying degreesturbing. een stakeholders ze separate meetis difficult in practme topics.” eholder meetings ctly the objectivout the expectati
Stakeholder Involv
oard the
hink
the
not they s by
evels of
oard the
more
s of
ore often . This is
and ings tice.
are ves: ions
3MTfrsth
MTsmcs
MTo
vement
3.3.2. RequestMessage 6. IndepThe concern aborequently expresshould at least unhem, but not to su
“Independenc“We must be the stakeholdstakeholders have a scienti“We must be game. It is ntake their opin“We should restakeholders.“How is it posbe perceived
Message 7: ClearThe KCE researcstakeholders woumechanism of oneclearly earmark ascientific findings p
“It is probablybetween whacoming from opinions anddistinction bet
Message 8 : TrainThe researchers feof stakeholder invo
“For smaller gskills: how to
ts pendence is a prout the risk of sed, in a varietynderstand clearlyubjugate ourselvece must be a prior
cautious that recders. The recomm
don’t agree. Thific background.” very clear from thecessary to annonions as such.” einforce this perce”
ssible to stay indeas independent?”r distinction to inchers are excludild be taken overe sort or another
any input from stproper. y necessary to maat’s coming from
other meetingsd contributions tween scientific asning is necessareel they lack approlvement without groups, it would bmanage the mee
iority losing one’s indy of different way that we are pres to their opinions
rity.” commendations amendations must he recommendati
he beginning aboounce very clearl
eption of independ
ependent? And ho” nclude in the reping by all meansr in the conclusior. Hence, they strakeholders and d
ake a clear distinthe scientific lite
s or inputs. Themust be clearlyspect and meetingry ropriate training toadequate trainingbe useful to be tr
eting, how to give
dependence was ays. The stakehoepared to interacts.
re not oriented bynot change if the
ions must always
out the rules of thely that we will no
dence amongst a
ow is it possible to
port s that the opinioons by a ‘democress the importandistinguish it from
nction in the reporrature and what’se different partsy identified. Thegs is important.”
o engage in new fg. trained for specifithe floor, to boos
49
very olders t with
y e
ys
e ot
ll
o
ns of cratic’ nce to m the
rt ’s s, e
forms
c st
50
a debate (.important pimportant, the group a“To apply t
This perceived actually conducdeal with thecharacterising t
“Involvemeemotions, would be ifirst need to
Finally, accordthemselves wou
“The stakefor all the need to tra
Message 9: EvInsofar as the increase the imevaluate its effe
“Evaluationexpected im“The targeimpact. Thcompared
Message 10. A “It is very to have a This databa “Not an exthe contact
...) Sometimes wepersons. It is miso as to avoid th
and that the patienthe right method foneed of training
cting interviews, e interpersonal, this type of interacent generates emto respect the othimportant. As of yo acquire new comding to the KCuld benefit from so
eholders don’t havprojects. If we w
ain them.” valuation of impaprimary reason t
mpact of the KCE wect. n of the impact mpact is not there
et of the stakehohis way of imprwith other method
A knowledge basdifficult to identifytool, a database ase must be up-toxhaustive databats...”
e don’t dare whenissing and would hat some individunts for instance do
for the right topic, not only pertains meetings, etc…,
relational and ctions.
motions. A traininther ones and to yet, we are not yempetences.”
CE researchers, ome sort of trainin
ve the scientific bawant to involve th
act is necessaryto invest in stakework, the researc
is important: wee.” lder’s involvemenroving the impads. Is it the best ae of stakeholder
fy the stakeholderto identify the a
o-date.” se but just the ex
S
n the participants be useful.” “Thi
ual experts dominon’t dare to speaktraining is neededto the technical but also to the a
psychological
ng to manage thconduct the procet ready for this.
also the stakeng ackground necesshem at all levels,
eholder involvemechers express the
e should stop if
nt is to increase act will have to pproach?”
rs would be usefrs. It would be us
associations by to
xperiences we ha
Stakeholder Involv
are is is nate k.” d.” skills for ability to aspects
hese cess We
eholders
sary we
ent is to need to
the
the be
ul seful opic.
ave,
So
3MAsthfo
Me
Ts
vement
Such a database often solicited duri
3.3.3. PossibleMessage 11: MotAlthough the resestakeholders, the he whole span ofocus on one or tw
“Why to involformulate an stakeholder’s with the quest“The ultimate of the KCE.” “Do we want the case we m
Message 12: Invoeach project
“It is necessaBut we will nintensity. It wi“Involvement project for tinvolvement toperception of “Involvement sure to answetype of subjec
The researchers asteps of a study.
“The involvemthe research, the post-publi
could also help toing the same perio
e avenues tivations of Stakeearchers identify a
discussions alsof possible objectiv
wo goals. lve? Is it more to
advice? What involvement? Do
tions or the methotarget of the man
to increase the amust find new toololvement is alwa
ary to involve the not involve them ill be different andis not monolithic.
the recommendao define the reseaindependence wilwill be different fr
er to the needs. It ct, type of researcacknowledge that
ment of stakeholdat the end to co
ication phase.”
o ”avoid that the od.”
eholders' involvea range of differe
o reveal that theyves and motivatio
o install a democrdo we want to
o we want that eodology?” nagement is to in
appropriation by thls.” ays important bu
stakeholders in ein each step, n
d specific from stu The involvement
ations is quite darch questions. Till be variable.“ from project to pro
depends of the ach, type of questio
the needs may v
ders is important aommunicate the c
KCE Reports
same persons ar
ement ent reasons to eny seldom acknowons, but rather te
ratic process or too reach with thiseverybody agrees
ncrease the impac
he citizens? If it is
ut must be adapt
every single studyor with the samedy to study.” t at the end of thedifferent from an
The results and the
oject. We must beavailable evidenceon ...” vary along the diff
at the beginning oconclusions and in
s 174
re too
ngage ledge
end to
o s s
ct
s
ed to
y. e
e n e
e e,
ferent
of n
KCE Reports 17
Message 13: partnership Involvement is levels of intensi
“ ‘Discuss wwe should “The scienpragmatic: all the staanalyse theaspects cowant to imp“We must implementamust comethem.” “This is anexample oliterature gcountries atrue partnedecided bereforms.”
Message 14: KWhen stressingmean that theyand independeviewpoints only
“If these kerejected. Tmutual resp
Message 15: S“It would beThis steerin
74
Involvement is
perceived to beity, up to the levelwith’ is the minimanalyse if we go ftific character of oit must be implem
akeholders aroune reaction on the oming from the flplement new guidshow them that wation. We have toe from them. We
n internal decisioof a study in twogives only a few care bad. Second pers to define the reefore. Only this wa
Key values are crg the importance oy expect to be rence. They are wi
y if this respect is rey values are no
This respect of valupect of key values
Steering committe useful to set up ing committee wo
s possible unt
e both important l of co-creation an
mum level, and thefurther.” our work is prioritmented. We accepnd the table ... w
floor: this would loor into our reco
delines this is relevwe are ready to tho plan time to do th
must show that w
on of the manageo parts. First paconclusions and thpart: the stakehol
reform.” “This musay, they are true p
rucial of our key values,espected in their lling to respect threciprocal. t respected the sues is also imports.” tee a steering commi
ould gather at the
S
il implementatio
and possible at nd partnership en, project by proj
ty, but we have topt to be influencedwhy not to see allow to include n
ommendations. If vant and importanhink with them abhis, but the questiwe are ready to h
ement. We haveart of the study: he results from olders are involvedst be transparent partners to define
the researchers pown values of ob
he stakeholders a
stakeholders musttant for KCE: this
ittee for each projdifferent stages.
Stakeholder Involv
on and
different
ject,
o be d by and new f we nt.” bout ions help
e an the
ther d as and
e the
primarily bjectivity and their
t be is a
ject. We
MSdEs
3MTsrep
MAisinreo
vement
could use thisvalidate the m
Message 16: InvoSome collaboratodecision making aEuropean Medicinsuggested.
“We should in"This group cinvolvement knowledge ofcould be paid
3.3.4. Treats Message 17: TimThere is a widesprstakeholders will cesearchers ask to
projects. “The problempossible to usat the end of t“The choices we must chanecessary and
Message 18: PatiAlthough patients s very little, if nvolvement is peeasons. There
objectivity, of feas“I think that thvery difficult tthem. It is an
s group to identifymethodology, the rolvement of EMAors stressed the above the Belgia
nes Agency) ; and
nvolve people fromcould be used fois important. T
f the methodologthrough attendan
ing is an obstacread concern thatcost a substantialo acknowledge th
m with these methse these methodsthe project“. are often dictate
ange this way od then request theients' involvemenare invariably seeany, experience
erceived as particare issues of ribility…
he patients are theto involve the patiimportant challen
y the societal trenrecommendationsA and citizens' pa
importance to aan federal structud a fixed panel o
m European Medicfor all projects wThey can acquirgies and the way nce fees."
le t adding more thol amount of extra
his extra workload
hods is the time ts if we don’t have
d by the time thaof working. We me necessary budgnt is the most difen as a central sta
with this groupcularly difficult to representativity,
e most important stients and to workge to imply them
nds of the societys...” anels also involve leveures (EU commisf five to ten citize
cines Agency.” where the citizensre a fairly good
of working. They
rough interactionsa time. Converselyd in the planning o
they take. It is noenough time, e.g
at is available, bumust plan what’s
get.” fficult akeholder group, p at KCE, and implement for seof perceived lac
stakeholders. It isk scientifically withefficiently.”
51
y,
els of ssion, ens is
s’ d y
s with y, the of the
ot g.
ut ’s
there their
everal ck of
s h
52
“The mostwith the passociationinterests... “There arefor instanctheir own a“Patients aTheir opinio“A distincassociationorganisatioknowledge“The identiare not alinfluenced
Furthermore, thwell perceived.
“A distinctcitizens whchallenge i
Message 19: AIn one of the dsensitive data, and collaboratioopenness from
“Many studcancer surpatients wamany requthey are 5 “We considexpectationbarriers. W
t difficult ones arepatients themselvn – There are “
e no such probleme, even if it is nec
agenda.” are only experts inon will not be use
ction must be ns and the gons. The speciale than the general ification’s phase lways patient asby industry.”
he need to disting
tion must be maho are not ill. Theis to identify peop
Are we ready to giscussion groupse.g. on hospital
on from patients KCE in return.
dies publish resulrvival rates. But ant to know which
uests of this type :good hospitals... der that it is notns of the patient
What is our respon
re the patients. Twves or to speak
problems of a
ms with other stacessary to filter a
n their own treatmeful for each reseamade between eneral patient lized associationsones.“ of the patients is
ssociations and t
guish between pa
de between patieey tend to see thile having the cap
give all informati, the issue was rquality indicatorsor citizens, they
lts on the quality these results are
h hospitals are the:’My father has a where must I go?t our role to makts are legitimate.se?”
S
wo options: to spk with the patieavailability, perso
akeholders – induand even if they h
ment. There is a barch question.”
specialized patassociation cup
s have much m
s also difficult. Ththey are sometim
atients and the cit
ents and tax-payings differently.” “acity to discuss.”on to the patientraised of public acs. If we expect opy may well expec
of hospitals, e.ge anonymous. Me best ones. We h
cancer: you say ?’ “ ke rankings. But . We have no le
Stakeholder Involv
peak ents’ onal
ustry have
bias.
tient pola
more
here mes
tizens is
yers/ “The
ts ccess to penness
ct similar
. 5y Many have that
the egal
3TwmAgcTin
vement
3.4. ConclusioThe four discussiwere quite prolifimotivation, frankneAll relevant aspecgroups have beechapter 4. The actual bodynvolvement of the
on ion groups organc and generatedess and dynamismcts of the messaen integrated into
y of experience e stakeholders is g
nized in October d a real added m of the participanges that emergeo the KCE polic
originating fromgiven in appendix
KCE Reports
and November value, thanks to
nts. d from the discu
cy, that is detaile
m past projects .
s 174
2011 o the
ussion ed in
with
KCE Reports 17
4. STAKPROC
4.1. RATIOKCE re
The ultimate oevidence-basedof advice andpractitioners, redegree to whiliterature reviemodeling is notAn external audthe healthcare be more involvdissemination. different stagesconsequently, tFrom a scientielements of thbased upon stawere applied, a
4.1.1. ImpacIn a general mreports, a seriecan be identifieoptimal impact.
74
KEHOLDERCESSES
ONALE : enhanceports objective of KCEd healthcare and d guidelines, p
espectively. The imich this objectiveews and state-ot sufficient. dit, performed in 2policies63 showed
ved, from the veryBeyond scientific
s of the projects, ihe potential impafic point of view
he study results,akeholder input, inand their limitation
ct of the KCE repmanner, over andes of levers potened. On the other h
R INVOLVE
cing the releva
E is to contributehealthcare policy
primarily aimed mpact of the KCEe is reached. Cof-the-art data a
2009 to evaluate d the explicit demy onset of the stuc excellence, stais a means to enhct of the KCE rec, it is important , conclusions anncluding a descris and risks.
ports: levers andd above the scientially increasing
hand, there are als
S
EMENT IN
ance and impac
e in an effective y, through the format policy make
E can be measureClearly, performinanalyses and e
the impact of the mand from stakeho
udies up to the pkeholder involvemhance the relevanommendations. to indicate clearl
nd recommendatiption of the meth
d barriers entific value of tthe impact of KC
so potential barrie
Stakeholder Involv
KCE
ct of
way to mulation ers and ed by the ng good conomic
KCE on olders to phase of ment, at nce and,
ly which ons are ods that
he KCE CE work ers to an
Tr
L
TcSRfeAhORcPMb
vement
Table 7: Levers areports
Levers
The real questionsconcerns are dealtSuggested solutioRecommendationseasible A priori impartialityheard Ownership by stakRecognised scientcomprehensive apPublic support Message (can be)by credible actors
and barriers with
s and relevant t with ns are acceptables are operational,
y: all groups are
keholders tific excellence;
pproach
adopted/relayed
respect to the im
Barriers
e
Mismatch betparadigms of those of stakeViolation of ininterests Real-life cont(financial, orgare ignored KCE perceiveoffending, disPrevailing cohidden agendComplexity isPerception ofjudged
mpact of KCE
tween values andf researchers vs. eholders ndividual/group
text and constrainganisational, other
ed as taking srespectful stancentroversies, tensioda’s are ignored s (partly) disregardf being accused o
53
nts r…)
e ons,
ded or
54
4.1.2. Five sand op
As a researchstrategies to othese actions n 1. Make your sKCE researchemethods, but mof the actual hbefore embarkiget acquainted this specific heand feel? Whatone’s own eyehave had a first 2. Get your scoEvidently, relevall! Our studiesas perceived bymakers. This retensions, contrvalues… Addrecan also be an among end-recEqually importaregulatory cont(‘cui prodest’), c
strategic objectivperate the leverer or research ivercome these beed the involveme
subject tangible ers are supposedmore often than nhealthcare subjecng into a researcwith the subject althcare approact are the troubles s, to have heardt-hand experience
ope and researcvancy is an absos should address y the providers, tequires to get beoversies or conflessing these queimportant step in ipients. ant at this stageext with its financcompeting interes
ves to overcomers nstitution, we ca
barriers or to opeent of stakeholde
d to be specialistot, they have onlct on which they
ch project on a givand to make it soh organized in prwith it on the fie
d from the mouthe, is worth several
h questions righlute prerequisite the really importahe patients, the peyond first impresicts, to understan
estions in a carefthe creation of ac
e, is to get a fucial implications. Wsts, barriers, comp
S
e the impact barr
an mobilize a nuerate the levers. rs, one way or an
ts in matters of ry limited prior kny are appointed. ven subject, it is uomehow tangibleractice? How doeld? … To have se of a patient, or l days of reading w
ht for getting any imant issues and copublic and/or the ssions, to identifynd deep motivatioful, participatory cceptance and ow
ull understandingWhere are the inplexities …?
Stakeholder Involv
riers
mber of Most of other.
research owledge Hence,
useful to . How is
es it look een with even to
work!
mpact at oncerns, decision
y hidden ons and manner,
wnership
g of the centives
3PwashthWigwsa
4TthcWscaaththdojurewaBosa
vement
3. Gain acceptanPeople are rarely wwhen they come oanalyses will not should be aware ohow do they see hey see as a solWhen we ignore sgnored ourselves we should adopt oshould not evade and take a look at
4. Reach clear resThis is all about that will be formconsequence fromWe very often prstakeholder groupcourse also our awareness of oppand discuss them.hat you have listehem, can certadisrespectfully rejonly cause anger ust pertain to thecommendation,
words, if people aas a ground for furBesides being acopinion), as discushould also be reaat least in the long
ce of your reseawilling to readily a
out of a black boxso easily yield v
of the conceptual the problems andution; what are th
strong voices ‘out when we come w
other paradigms our duty to engagother sources of
sults and acceptthe acceptability mulated, and, n
m the preceding poroduce answers
ps would have exp‘raison d’être’.
posing viewpoints. Showing respecened to them in aainly help to ecting other viewand hostility towa
he mere fact theven if they differ
accept our role in rther discussion.ceptable (even ifussed above, whalistic in terms of g term.
arch setup and maccept conclusionx. Likewise, ‘esotevery convincing aframework used bd in what terms dheir underlying vathere’, what else with our solution?and jeopardize Ege into a dialogue‘evidence’
table recommendand feasibility of
normally, this shoints. that are not ma
pected, let alone pEven so, we shs and their mostt for other viewpo
a honest way, eveget respect in wpoints without pards KCE. The ahat KCE formular from those of ththe system, we h
f only acceptablehatever recommefeasibility – mayb
KCE Reports
methods ns or recommendaeric’ or little underanswers. Researby the target audido they spell out alues and worldvican we expect tha This is not to say
EBM; it means thae with other viewp
dations the recommenda
hould be the na
tching what impopreferred, but thishould keep an t important argumoints, by demonstren if you did not f
return. Conveproper argument
acceptability couldated conclusionse stakeholder. In
have at least this
e as having a diffendation we formbe not immediately
s 174
ations stood
rchers ence: what
iews? an be y that at we points
ations atural
ortant s is of acute
ments rating follow
ersely, s will
d thus s and
other asset
ferent mulate y, but
KCE Reports 17
5. Get effectiveThe very best actively dissemreport has gunderstanding, certain, the prevrequire addition
4.2. The fivThe rationale benhance the evthey are also healthcare syst 1. Respect To involve thossimply a matterIt is based on input to offer antaking an omnidisrespect. When showing to expect recipand for the legit 2. TranspareInvolving stakemean, nor requmade perfectly the system demdistance, so aevidence. Morwhichever stakeach of the acto
74
e communicationproof of impact i
minate and get to wgone throughout
acceptance andvious steps shoul
nal efforts of trans
e key Values fobehind each of tventual impact of the translation o
tem and the role K
se who will be mor of respect, and, the premises th
nd that KCE has siscient or paterna
respect towards procal respect in timacy of our rese
ency holders, taking thuires that we takclear from the on
mands that, at a ces to reach valid reover, there sheholder being invors is playing.
n channels and ris when the key work with our rep the long wa
d eventually adopd greatly facilitatelation, disseminat
or fruitful stakethe above-cited othe work of KCE
of a number of KCE is playing in t
ost affected by thin principle, shouat these key stasomething to learalistic stance, cou
one’s interlocutorreturn for our ex
earch.
heir views and opike over these vienset. The specificertain stage in theconclusions in t
hould be a mutvolved in the cour
S
relays target actors theorts. This means y from awaren
ption and ownerse true adoption, bution and advocacy
eholder interactobjectives is not , but, by the samvalues underpinnthis system.
e outcome of ouruld be beyond disakeholders have vrn from them. Conuld be seen as a
r, one is evidentlyxpertise in EBM m
inions seriously, dewpoints. This shc role KCE has toe research, we takthe view of all atual understandirse of a study, of
Stakeholder Involv
mselves that this
ness to ship. For ut it may y.
tion only to
e token, ning our
r work is cussion. valuable nversely, form of
y entitled methods
does not hould be o play in ke some available ng with the role
Tp
3TreLintaao
4Eontoinbis
5Cdeacweo
vement
This goes along process, and abou
3. Objectivity Taking people’s oender them truthf
Likewise, the selnfluenced by pre-akes some judgmand weight he orobjective grounds
4. Modesty Even if KCE is objectivity, it cannonot even to have ao differ in opinionterlocutor. Recobehind deviant opssues that are rea
5. Curiosity Curiosity is a valudirectly aimed at extend to the psyand policies undeconsequence of thwhen venturing epidemiological, eonto shifting groun
with a clear cut the study progre
opinions and viewfully, i.e. without slection of the st-existing convictio
ment, though, to atr she deserves, whenever possib
and should be ot pretend to haveanswered the righon, while refraininognition and apprpinions, can only ally at stake.
uable character trprocuring healthc
ychology, sociologer study. This fohe respectful and
out of the baeconomical certitunds!
communication aess.
ws creates the oubjective interprettakeholders to inons or preferencettribute to each stabut this judgmele.
keeping up its e the definitive an
ht questions. The ng from any formreciation of the vlead to a deeper
rait for a scientistcare policy advicegy and politics beorm of opennessmodest attitude aastion of EBM udes, one better
bout the involve
obligation to somtation or filtering. nvolve should noes of the researchakeholder the attent should be bu
scientific rigournswers to all queschallenge is to be
m of disdain for values and worldr understanding o
. But, when sciene, this curiosity sehind the technols should be a naadvocated above.
with its statisr be prepared to
55
ement
ehow
ot be her. It ention ilt on
r and stions, e able one’s dview of the
nce is hould ogies atural Yet, stical, step
56
4.3. There aWhen decidingprofessional intto mind is a meis just one of thand for each ocould be approprecise set of systematic wayeach stakeholdchoose the mos
are many differg to involve a terest lobby groupeeting with five to he many ways onbjective there mig
opriate and effectcircumstances an
y, one can distinguder interaction shst suitable approa
rent ways to ingroup of key
p of the doctors, tten persons at thee could seek to inght be a number ive. The actual cnd on the expectuish a number of hould be situatedach.
S
volve stakeholstakeholders, e
the first approache KCE premises. nteract with stakeof different meth
choice will dependted outcomes. In dimensions alon
d in order to be
Stakeholder Involv
ders e.g. the h coming Yet, this
eholders, ods that d on the
a more ng which
able to
T
D
Win
R
Cs
LinD
MF
InnLc
C
vement
Table 8. Dimensio
Dimensions
Who to nvolve?
Representation
Circle of stakeholders
Level of nvolvement Desired output
Motivation Format
nformation need Level of controversy
Confidentiality
ons of stakehold
Possible choice
Patients; Genergeneral); Providers; Potecommercial actoDecision makersDirect contact w(representative aActive exclusion(reference) grouselected) Obtain informatioObtain engagemFactual informcontroversies) ↔solutions Advising ↔ DemFace to face ↔ Use of web toolsStakeholders hastake ↔ StakehoSubject is highlyDepends also othe stakeholdersEverything can bcannot be said in
der involvement
es/situations
ral population; M
ntial future proviors, Foreign expers; Administration; with ‘the field’ ↔associations) n of certain groupup ↔ Open,
on from ↔ Listenment ↔ Be partnermation ↔ De↔ Consensus ↔
mocratisation Forum ↔ Survey
s, of social media…ave poor prior knoolders are well inf
y controversial ↔ on the objective os be openly discussn public
KCE Reports
Media (specialise
ders; Industry; Orts in the domain Insurers; Academ
↔ Via represent
s ↔ Closed; seleunselected (i.e.
n to ↔ Discuss wirs escription (e.g. ↔ Co-constructio
y… … wledge of the issuformed Subject is consenof the interaction
sed ↔ Some elem
s 174
d or
Other
mia ation
ected self-
th ↔
of on of
ue at
nsual with
ments
KCE Reports 17
4.4. From sAs apparent froto involve stakeeach of the strmethods) shouare more arbitexperience (seeFurthermore, eaat specific mom Table 9. Timing
1. Make subje2. Get scope a
questions r3. Gain accep
methods 4. Reach reali
5. Get effectivtion channe
74
strategy to concom the first part ofeholders in the strategic objectivesuld be selected. Strary and will poe Table 10, next pach of the objectiv
ments during the p
g of the different
ect tangible and research right ptance of
stic answers
ve communica-els
crete actions f this report, thereudy work of advis, the most approSome choices arossibly have to bpage). ves requires inter
project (Table 9)
t interactions wit
At the very onsBefore finalisi(projectfiche-ficEarly in the reto be repeated Is an ongoidifferent types the project; pparticularly cruFrom pre-finalbut facilitated b
S
e is a wide variety sory bodies like Kopriate method (ore straightforwardbe revised with
ractions with stake
th stakeholders
set of the project.ng the study p
che projet) alisation phase, ponce or twice ng concern, reof interaction thro
pre-finalisation phcial isation phase on
by earlier interactio
Stakeholder Involv
of ways KCE. For or mix of d, others
growing
eholders
protocol
possibly
equiring oughout hase is
nwards, ons
vement 57
58
Table 10: Impa
Stakeholder in
Search of medOn-site visitsmanufacturer, Individual inteIndividual inteproposal, engaFocus groupsDiscussion fodiscussion of inventory grouDelphi methodScenario BuildConsensus coDeliberative poOnline discussSurvey (Web, tUse of social mWorkshop K= To be applied
A number of ostudy processetanks, Charrette
act enhancing st
nvolvement appro
dical and lay press (to healthcpatient…), rviews with ‘typi
erviews with keyaged stakeholde
orum (meeting f contentious poups; rating groupds ding Exercise onference olling sion groups/list stelephone, papermedia
d/developed by KCE
other participatoryes : Search confere, Constituent as
rategies
oaches
ss care institution
ical’ stakeholdery informants (a
er, policy watche
with 8-20 stakoints; project scps.
servers r self-administer
E √
y methods may brences, Study circ
ssembly, Retreats
S
or surgery,
rs author of study r)
keholders), for coping groups;
red, ...)
√ = Other eligible me
be less relevant cles, Study groups, Round tables, A
Stakeholder Involv
1.Make subject tangible
K
K
+
K
+
ethod; could be subc
for KCE ps, Think Advisory
cOp
vement
2.Get scope and research
questions right
+
+
K
K
+
K
√ √ √ √ √ K √ +
contracted; + = M
committee, Board/Online discussionprocess (see KBF
3.Gain
acceptance omethods
+
+
K
+ + +
+ Method could be of h
/council or plannin groups/list serve200551 and Healt
of 4.Reach realistic answers
+
+
K
x
K
√ √ √ √ + + + K
help or contribute
ng cell, Interactiveers, Issue confereth Canada 20008)
KCE Reports
5.Get effectcommunica
n channels
+
+
+ +
e www/e-conferenences, Nominal g).
s 174
ive atios
ncing, group
KCE Reports 17
4.5. Stakeh4.5.1. PrereqAs a prerequisnumber of unavSome of themdimensions liste• Who are th
Given the drepresentaindustry cobe built upneeded in groups, of
• What are ton the stra
• What are th• SWOT ana
o What ao What
involveo What o
The requirementhe type of studwill need otherdebated subjec
4.5.2. ChoicDepending on specific strateglisted in table 1one might deciare perceived tdominance by l
74
holder involvemquisites ite for every stakvoidable question
m correspond to ed in table 8: he key stakeholddifficulty of finding
atives of patient ontacts, key persop. Even so, the order to get the pcitizens, of the hethe significant istegic objective thahe real goals of thalysis (Dimensionsare the strengths
risks are assoement and with a opportunities are nts and expectatidy and subject. Er types of contac
ct.
ce of the most apthe answers to
gic objective, one 10 which one(s) isde not to engageto outweigh the plobby groups. The
ment in Practice
keholder involvems to be answeredchoices that ha
ers to involve ? (Dg the appropriate
organizations, ons in the administhelp of external
perceptions and oealthcare workersssues for the staat is pursued) he involvement ? s 9 and 10) and weaknesses
ociated with thepoor involvementassociated with aons might be qui
E.g. a guideline incts and inputs th
ppropriate methothe above-mentiowill have to cho
s (are) the most ae into stakeholderotential advantage choice will also
S
e
ment exercise, the by the KCE reseave to be made
Dimensions1, 2 anindividuals, a dataof professional tration… should gsubcontractors mpinions of specific‘on the floor’… keholders ? (Will
(Dimensions 4, 5
of the involvemene involvement, wt
a complete involvete different depen
n a non-controverhan a HTA on a
od(s) oned questions,
oose among the mappropriate. Alterr requirement, if tes, e.g. for fear ohave to take into
Stakeholder Involv
re are a earchers. e in the
nd 3) abase of groups,
gradually might be c patient
depend
and 6)
nt ? with no
ement nding on sial field fiercely
and the methods rnatively, the risks of undue account
ma
4SaporekdTidthFs
4
Ad2eTaduIninwaed
vement
more practical issand timing conside
4.5.3. RequiredSome of these cattention should bpatients or with soother parties with esearcher. He or
keeping the righdynamics… Thorough debriefindentifying the “dohe gradual buildinFor other compesubcontractors wh
4.5.4. Further eeligible f
A number of the during the interna2011 (see Chapteelaborated into deThey are briefly advantages and mdescribed explicitlyuse in other contexnsofar as KCE nvolvement methwith much certaintand institutions caexperience, this cduring the coming
ues like the avaierations.
d competenciescompetencies shbe paid to the pometimes exigent,vested interests she may benefit
ht distance, in
ngs after completo’s and don’t’s” ong-up of an experietencies, we co
ho have a specific
elaboration of mfor KCE
methods listed ial expert consulter 3) will in the tailed, operationadescribed in app
merits, as perceivy target their applxts is outside the has still a limiteods, key successty at this stage, aannot readily be tchapter will have few years.
lability of resourc
ould be developsychological asp, suspicious or evmay entail psychofrom training in cothe managemen
tion of a researchof specific approacience and knowleould make use experience in the
methods that wer
in table 10 that tation organizedcoming months
al KCE Process Nopendix, with a foved by KCE reseication to stakehoscope of this repo
ed experience wis factors and riskand the experienctransposed eitherto be updated a
ces and competen
ped internally. Spects. Interactions
ven hostile provideological distress ionflict management of complex g
h project should hches and contribudge base at the K
of the servicee field.
re considered as
have been mentin October-Noveand years be fu
otes. ocus on their sparchers. The me
older involvementort. ith formal stakehks cannot be idence from other cour. Hence, with groand further elabo
59
ncies,
pecial s with ers or in the ent, in group
elp in ute to
KCE. es of
s
ioned ember urther
pecific thods ; their
holder ntified ntries owing orated
60
5. APPEILLUCONEXPE
APPROACHinstitutions
Example(s)
Objective
Timing Who to involve
Selection
Level of involvement Motivation Desired outpu
ENDIX: APSTRATED SULTATIOERTS
1 – On-site vis
KCE Report2005 64 KCE ReportPrevention o65 Make the suclinical and At the onset
e? Providers/hetechnology (under studyguideline. Not necessarather via pecentres andInformation
Avoiding a pt Good grasp
technology, approach, inparticularitie
PPROACHEDURING T
ON ROUND
sits to healthca
ts 20 [Molecular D
ts 46 [Home Moniof Sudden Infant D
ubject tangible (wipractical aspects)t of the project. ealthcare institutio(HTA) or providing
y, or caring for the
arily via representersonal contacts; quality-consciousof the researcher
purely theoretical p of the concrete (c
service or diagnoncluding the regules (if debatable).
S
ES THE D OF KCE
are providers o
Diagnostics in Belg
itoring of Infants inDeath Syndrome]
ith the focus on th).
on currently usingg the service (HS condition covered
tative organizationlook for large-volus providers. rs.
stance. clinical) practicalit
ostic or therapeutiatory/financial
Stakeholder Involv
r
gium],
n , 2006
he
the R) d by the
ns; ume
ties of a c
Afo
DPinn
PmD
FLc
C
L
vement
Appropriate ormat
Duration Prior nformation need
Preparatory material Deliverable
Follow-up Level of controversy
Confidentiality
Limits
Non-interferingprocesses, alteproviders and (any case, inforpatient. Ideally, all reseparticipate. In pediting of the d½ to 1 day. Stakeholders nResearchers cwith the technoregulatory framNone, or a set preparatory reaBrief descriptiono other specifof the final studtechnology or sobservation noIn principle notIn theory, out overy fact that thcreate specific bring out potenvisits. Hence, tthe function in All observationinformation shoAs representatof the researchresults, and doreport.
g observation of thernated/followed b(if needed and fearmed consent is to
earchers involved practice, at least tdiscussion and con
need little prior infoould benefit from ology, medical con
mework. of remaining quesading work. on of the meeting fic deliverable; andy report, describiservice, may beneotes of the researct applicable. of scope for this mhe researches beperceptions amo
ntially controversiathe nature of the vthe project should
ns, and, a fortiori, aould remain strictltivity is not aimed hers cannot be coo, as such, not app
KCE Reports
he actual clinical by discussion withasible) with patieno be obtained from
in the study shouthose involved in tnclusions.
ormation. prior acquaintancndition, jargon,
stions after
(who, when , whe introductory chaping the problem, efit from the persochers.
method; in practicelong to the KCE mng the providers, al aspects during visit, its objectivesd be made very clall patient-relatedy confidential. for, the observatinsidered to be forpear in the study
s 174
h the nts. In m the
uld the
ce
ere), pter
onal
e, the may and the
s and lear.
ons rmal
KCE Reports 17
Risks of the method
How to evaluate?
74
Perception bbased uponpersonal exthese potenother actorsAbsence of know what tsuggestions
bias and preconce a very limited, buperience. This cotial preconception
s, preferentially wisignals that “rese
they are talking abs.
S
eptions in the reseut potentially stronuld require to expns and test them wth opposing views
earchers do not rebout” or similar
Stakeholder Involv
earcher, ng press with s.
eally
As
E
O
TW
S
LinM
D
vement
APPROACH 2 stakeholders
Example(s)
Objective
Timing Who to involve?
Selection
Level of nvolvement Motivation
Desired output
– Individual in
KCE Reports 4Prevention of S65 Understand the‘in the field’. At the onset ofProviders/healtechnology (HTunder study, oguideline; Patients confro‘Normal’ providrepresentativesPatients could social media, spartners speciaInformation of
Avoiding a purmiss importantpatient and clinquestions, but recommendatioRicher view of under study Gopracticalities oftherapeutic apRelevant resea
nterviews with ‘
46 [Home MonitorSudden Infant Dea
e subject also in t
f the project. thcare institution cTA) or providing thr caring for the co
onted with the issuders and patients,s; be recruited via p
sickness funds, oralised in participathe researchers.
rely theoretical stat aspects linked tonician when decidalso in view of forons at the end of the different dimeood grasp of the cf a technology, seproach; arch questions.
typical’
ring of Infants in ath Syndrome], 20
he way it is perce
currently using thehe service (HSR) ondition covered b
ue under study. not their political
patient organisatior with the help of otory work.
ance; make sure no the perception oding on the researrmulating relevantthe project.
ensions of the proconcrete (clinical) ervice or diagnosti
61
006
eived
e
by the
ons, other
not to of the rch t
oblem
ic or
62
Appropriate format
Duration
Prior informatineed
Preparatory material
Deliverable
Follow-up
Level of controversy Confidentiality
Limits
Risks of the method
In depth intePreferentialcollaboratorlocation of cby telephonApproximateby telephon
ion StakeholderResearcherwith the clinSet of half-oIntervieweelines of the Tape/video-consent) anis not requirA synthesis mandatory)publication oIn principle
y All observatconfidentialAs represenobservationto be formalthe study reindividual peInformants mdesirable faPerception bbased upon
erviews. ly face-to-face (1,rs or subcontractinchoice of the stakee, after having fixely ¾ to 1H per ine). rs need little prior rs could benefit fronical/medical aspeopen or open quess should be pre-ininterview. -recording of the fnd notes (full transred); synthesis anof the interview (
. Active informatioof the report low.
tions and recordin. ntativity is not strics of the researchel results, and do, a
eport as qualitativeersons’ views. may retain informcts or opinions).bias and preconce
n a limited number
S
, or max 2 KCE ng researchers), aeholder; second c
xed an appointmennterview (20 to 30
information. om prior acquaintaects. stions. nformed of the ge
focus groups (aftescription of the recd analysis for validation, not
on of participants
ngs should remain
ctly aimed for, theers cannot be conas such, only appe observations of
ation (e.g. socially
eptions in the reser of contacts. Tea
Stakeholder Involv
at the choice : nt.. min. if
ance
neral
er cordings
of the
n strictly
e nsidered pear in
y less
earcher, m
He
vement
How to evaluate?
discussions shscientific objecNo new issuesprojects; main duly identified.
hould address thectivity. s appear at the latpoints of debate a.
KCE Reports
ese challenges to
er stages of the and controversy w
s 174
were
KCE Reports 17
APPROACH(author of stuwatcher)
Example(s)
Objective
Timing
Who to involve
Selection
Level of involvement
74
3 – Individualudy proposal, e
KCE Reportfor diabetesKCE Reportprocesses oKCE ReportPrevention oUnderstand aspects (regVery early, bin the pre-finconclusions
e? The initiatorkey-personsProviders/hetechnology (under studyguideline; Foreign expPatients; paIndustry Providers anorganizationinterests; buInformation
l interviews witengaged stakeh
ts 27 [Quality ands 2], 2006 66 ts 133C Optimisatof the Special Solits 46 [Home Moniof Sudden Infant D the subject also igulatory, financialbefore final choicenal phase, to test s and recommendars of the research s in the administraealthcare institutio(HTA) or providing
y, or caring for the
perts in the domainatient organization
nd patients : via rens; look for balancut limit to 6 à 8 in tof the researcher
S
h key informanholder, policy
organization of th
tion of the operatidarity Fund 67 itoring of Infants inDeath Syndrome]in its more ‘politica, other interests). e of research queacceptability of ations topic, key policy m
ation; ons currently using the service (HS condition covered
n; ns (if applicable).
epresentative ce of viewpoints atotal. rs.
Stakeholder Involv
nts
he care
onal
n , 200665 al’ stions;
makers,
g the R) d by the
and
M
D
Afo
D
Pinn
Pm
D
F
Lc
vement
Motivation
Desired output
Appropriate ormat
Duration
Prior nformation need
Preparatory material
Deliverable
Follow-up
Level of controversy
Avoiding ‘to midiscussions onIdentification oGood grasp of regulatory and potential controRelevant reseaand/or recommIn depth interviPreferentially facollaborators), stakeholder; sevideoconferenc1H to 1.30H petelephone). Stakeholders ndiscussion of cbe sent > 1 weResearchers cwith the regulaexpressed viewSet of half-opeInterviewees slines of the inteTape/video-recconsent) and nis not required)A synthesis of mandatory). Acthe publication Can be high. Eobjectives and made very clea
ss the point’ befon the recommendaf the key questionthe ‘political’ stakfinancial aspects
oversies betweenarch questions; ac
mendations. iews. ace-to-face (1, or at the location of
econd choice : by cing, after having er interview (30 to
need little prior infoconclusions / recoeek in advance. ould benefit from
atory framework, fiwpoints (medical aen or open questiohould be pre-inforerview. cording of the focunotes (full transcrip); synthesis and athe interview (for ctive information oof the report
Even so, the naturthe function in the
ar.
re the final ations. ns to be answeredkes, including the under debate an interest groups. cceptable conclus
max 2 KCE choice of the telephone or fixed an appointm
o 45 min. if by
ormation, except mmendations : te
prior acquaintancinancial implicatioand lay press). ons. rmed of the gener
us groups (after ption of the record
analysis validation, not
of all participants o
re of the visit, its e project should b
63
d.
d the
sions
ment..
if for ext to
ce ons,
ral
dings
of
be
64
Confidentiality
Limits
Risks of the method
How to evaluate?
y All observatconfidential.As represenobservationto be formalthe study reindividual pePerception bbased upondiscussions scientific obNo new issuprojects; maduly identifie
tions and recordin. ntativity is not strics of the researchel results, and do, a
eport as qualitativeersons’ views.. bias and preconce a limited numbershould address t
bjectivity. ues appear at the ain points of debaed..
S
ngs should remain
ctly aimed for, the ers cannot be conas such, only appe observations of
eptions in the reser of contacts. Teamhese challenges t
later stages of thete and controvers
Stakeholder Involv
n strictly
nsidered pear in
earcher, m to
e sy were
A
EO
T
W
S
LinM
D
A
DPn
vement
APPROACH 4
Example(s) Objective
Timing
Who to involve?
Selection
Level of nvolvement Motivation
Desired output
Appropriate form
Duration Prior informationneed
Focus groups
Understandaspects (regorder to formproject. To be plannTo be conduPatients; Gegeneral); Providers; POther commDecision maAcademia Providers aorganizationinterests Listen to
Avoiding ‘todiscussionsIdentificatioInventory ofand / or conregarding th
mat Several meeincludes onBetween 2.
n Several optobjective of
the subject also gulatory, financialmulate the resear
ned at the beginniucted at the approeneral population;
Potential future promercial actors akers; Administrat
nd patients : via rns; look for balanc
o miss the point’ bes on the recommen of the key questf knowledge, percnsensus elementshe topics exploredetings (6-12 peoply one type of stakhours and 3 hourion possible, depethe focus groups
KCE Reports
in its more ‘politic, other interests) ich question(s) of
ng of the project opriate time. ; Media (specialis
oviders; Industry;
tion; Insurers;
epresentative ce of viewpoints a
efore the final ndations. tions to be answe
ceptions, controves of the participantd by the focus grople), each focus grkeholder . rs ending on the .
s 174
cal’ in the
ed or
and
ered. ersy ts ups. roup
KCE Reports 17
Preparatory material
Deliverable
Follow-up
Level of controversy
Confidentiality
Limits
Risks of the method How to evalua
74
A script irelated toincludingTape/vidconsent) recordingA synthemandatothe publicFocus grpeople wopinions to minimtopic of d
y All obserstrictly co Learningbias (avalimited toof debateorganisaParticipa
te? No new iprojects; were duly
n which the topicso a general and og an indication of teo-recording of th and notes (full trags is not required)
esis of focus groupory). Active informacation of the reporoups methodologwith previously ide
in separated homize the level of codiscussion. rvations and recoronfidential. gs will be limited bailability, interest to the perceptions oe is strongly relatetional aspects can
ants do not share t
ssues appear at tmain points of dey identified..
S
s to be addressedr particular questithe timing. he focus groups (aanscription of the ); synthesis and ap (for validation, nation of all particip
ort. gy recommend to ntified heterogene
mogenous groups ontroversy on the t
rdings should rem
by a de facto seleco participate, ....),of the participantsed to the animatorn be very limiting.the real informatio
the later stages ofebate and controv
Stakeholder Involv
d, are on,
after
analysis. ot pants of
invite eous in order targeted
main
ction , are s; quality r skills,
on
f the versy
A
E
O
T
W
S
Lin
M
D
AfoD
vement
APPROACH 5
Example(s)
Objective
Timing
Who to involve?
Selection
Level of nvolvement
Motivation
Desired output
Appropriate ormat
Duration
Survey (Web s
Study 2011-1the supportiveSelect and prproject. EvalurecommendatTo be plannedTo be conducPatients; Gengeneral); Providers; Pocommercial aDecision makNormal providrepresentativeviewpoints anPatients couldsocial media, other partnersObtain informListen to Avoiding ‘to mdiscussions oIdentification oGet quantitatigroups of stakDepending on
Must be adaprespondents (
survey, telepho
0 (GCP) Clinical pe treatment for caioritize the resear
uate the acceptabitions d at the beginningcted at the appropneral population; M
otential future provctors, Foreign exp
kers; Administratioders and patients,es organizations; nd interests; but limd be recruited via sickness funds, o
s specialised in pamation
miss the point’ befon the recommendof the key questiove data, possibly keholders n the project : We
pted regarding ava(think about holida
one survey ...)
practice guidelinencer patients
rch question(s) of ility of proposed
g of the project priate time. Media (specialised
viders; Industry; Operts in the domaon; Insurers; Acad, not their: politicalook for balance o
mit to 6 à 8 in totapatient organisat
or with the help of articipatory work.
fore the final dations. ons to be answereto compare differ
b, Phone, paper,
ailability of potentiays)
65
e for
the
d or
Other in
demia al via of al ions, f
ed. rent
....
ial
66
Prior informatineed Preparatory material Deliverable
Follow-up
Level of controversy ConfidentialityLimits
Risks of the method How to evalua
ion Participantcomplete tQuestionnacall centre,Quantitativtext,...) A synthesiscomment, participantsLow
y All questioGathered dbias (availaselected m...). Low participarticipants
te? No new issprojects; mduly identif
ts must receive dihe survey by advaaire, directive, tec, ..)
ve data (raw data,
s of results (for innot mandatory). As of the publicatio
nnaires should redata will be limitedability, interest to
media format (acce
ipation rate, bias is sues appear at the
main points of debfied..
S
rective on how to ance.. chnical media (we
tables, structured
formation and furtActive informationon of the report.
main strictly confid by a de facto separticipate, ....), a
ess to, acceptabili
in representativity
e later stages of thate and controver
Stakeholder Involv
bsite,
d
ther of all
idential. election and the ity of
y of the
he rsy were
As
E
O
T
W
S
LinM
D
Afo
DPinn
vement
APPROACH 6 stakeholders)
Example(s)
Objective
Timing
Who to involve?
Selection
Level of nvolvement Motivation
Desired output
Appropriate ormat
Duration Prior nformation need
Discussion for
KCE Reports in Belgium: ProStudy 2011-10the supportive Confront draft rresults or recomstakeholders, shopefully moreDepending on the start, duringRepresentativeindustry,…. For patients, prrepresentative Consulting and
Make sure thatstakeholder grounderstood. Identification ofreconsidered, rFace to face mCan be in the K2 hours Participants recdiscussed cleashould be limite
rum (meeting w
167 [Residential cojections 2011 – 2 (GCP) Clinical prtreatment for canresearch questionmmendations to thso as to obtain a me acceptable end pthe points to be dg, or shortly befores of patients, pro
roviders and induorganisations.
d discussion
t the viewpoints ofoups have been id
f elements in the reformulated, bett
meeting of max 15 KCE premises.
ceived document arly identified. Reqed to less than 1
KCE Reports
with 8-20
care for older pers2025] 68 ractice guideline fcer patients
ns or preliminary he criticism of relemore robust and product. discussed, shortly re the end of the s
oviders, policy mak
stry: through
f the principal dentified and
study that need toter argued. to 20 persons.
with points to be quired reading timhour.
s 174
sons
for
evant
after study. ker,
o be
e
KCE Reports 17
Preparatory material Deliverable
Follow-up
Level of controversy ConfidentialityLimits
Risks of the method
How to evaluate?
74
Questions/s
Minutes (at acceptable –minutes; notActive informof the reportCan be very
y In principle nInformation necessarily Researcherindependenat the begingame shoulParticipantsof the officiaFollow-up ofpublication ocould possib
statements that wi
least 2 persons) a– tape recording (t to be transcribedmation of all partict. y high. A strong m
not applicable. obtained are ‘officreflecting the full s could feel to be cy of KCE may nening of the meetind be clear to all p could provide pe
al viewpoint they af reactions of stakof report and reflebly have been avo
S
ll be discussed.
and – if feasible a(for verification of d or kept) cipants of the pub
moderator is advisa
cial viewpoints’, nimage of an issueput under pressu
eed to be clearly sng, i.e. the rules oarticipants. rsonal viewpoint i
are excepted to trakeholders after ection on how proboided.
Stakeholder Involv
nd
blication
able.
ot e. ure; stressed
of the
instead ansmit.
blems
A
E
O
T
W
SLinM
DAfoDPinnPmDF
vement
APPROACH 7
Example(s)
Objective
Timing
Who to involve?
Selection Level of nvolvement Motivation
Desired output Appropriate ormat
Duration Prior nformation need Preparatory material Deliverable Follow-up
: Delphi Metho
KCE Reports 1practitioners: pKCE Reports 1Primary Care pReach a consecontent of recoDepending on the start, duringPractitioners, pindustry “Experts” in theConsulting and
Make the resulimplementableConsensus Web, e-mail or
Several roundsNot necessary
Questionnaire.the aim of the mSemi-quantitatActive informatof the report
od
165 [Burnout amoprevention and ma125 Impact of Acaphysicians 69 ensus about reseaommendations …the points to be dg, or shortly befor
patients, providers
e field d discussion
lts and recommene, get effective com
r postal survey
s over several we
Briefing about thmethod tive data, prioritiestion of all participa
ng general anagement], 2011ademic Detailing o
arch questions,
discussed, shortly re the end of the ss, policy maker,
ndations realistic ammunication chan
eks
he expectancies a
s lists ants of the publica
67
42 on
after study
and nnels
and
ation
68
Level of controversy Confidentiality
Limits
Risks of the method
How to evaluate?
Probably higprocess
y Responses anonymousThe results questions asInstrumentaArtificial conNumber andpart of the re
gh at baseline, an
of the participants depend on the chsked.
alisation by some nsensus d/or content of theeport
S
nd low at the end o
s have to remain
hosen experts, and
participants
e critics concernin
Stakeholder Involv
of the
d of the
ng this
A
E
OT
W
S
LinMD
Afo
DPinn
Pm
vement
APPROACH 8
Example(s)
Objective Timing
Who to involve?
Selection
Level of nvolvement Motivation Desired output
Appropriate ormat
Duration Prior nformation need
Preparatory material
: Workshop
KCE Reports 1practitioners: pStudy 2010-25health servicesBelgium (seconCo-creation of During the rese(pre)-final phasStakeholder grthe most knowCombination oinvitations High: stakeholdpartners. Gain maximumRelevant and athe potential toMax 40 persongroups of max table configuranumber of facilAt least 3 ½ to Series of informquestions or prparticipants willeast 1 week bSee above.
165 [Burnout amoprevention and ma5 (HSR) The organs for children and nd phase, ongoincertain elements earch process, pose roups who will be
wledgeable expertsf ‘official’ represen
ders are deeply e
m relevance, acceacceptable recomo effectively boost ns. Most of the wo6 to 7 persons. R
ation /meeting roolitators for sub-gro4 hours; preferen
mation cards or shroposals on whichll be solicited shouefore the meeting
KCE Reports
ong general anagement], 2011nisation of mentaladolescents in g) of the study
ossibly rather in th
most affected, ans ntatives and perso
ngaged or real
ptability, impact. mendations that hchange.
ork to be done in sRequires appropriam(s), flip charts, oups, … ntially whole day. heets describing th input from the uld be circulated a
g.
s 174
42
he
nd/or
onal
have
small ate
the
at
KCE Reports 17
Deliverable
Follow-up
Level of controversy ConfidentialityLimits
Risks of the method
How to evaluate?
74
Written mateminutes (miplenary discrecording neAllow re-reaparticipants editing) Can be relaadvisable.
y In principle Product will word; to be (explicit disc‘Politics’ invexpectationsfeel to be pushould be mEnd productapplied by ta
erial from ‘rapportn. 2 persons) and
cussions. No full treeded. ading of final resul
for validation (bu
tively high. A stro
not applicable. not be ‘scientific’ clearly identified aclaimer…) vading the study; gs among stakehout under pressure
made clear to all pt is effectively adoarget group.
S
teurs’ of the subgd tape recording oranscript of tape
lt of workshop by t avoid extensive
ng moderator is
in the strict senseas such in the fina
generation of falselders; researchers; the rules of the garticipants. opted, disseminat
Stakeholder Involv
roups; of
re-
e of the al report
e s could game
ed or
vement 69
70
6. REFE1. Andriof J
Unfoldingengagem
2. Nutley S,inform pu
3. Lavis J, Oevidence2009;7(S
4. INVOLVEhealth an2012-01-
5. Ginsburgin knowle
6. Riddle MConsultaInternatiofrom: http://ww$FILE/Pu
7. SequeiraHandbooWashingfrom: http://wwngageme
8. CorporatCommunInvolvem662-2924asc/alt_foeng.pdf
ERENCES , Waddock S. Unf
g stakeholder thinment. Sheffield: Gr, Walter I, Davies ublic services. BrisOxman A, Lewin Se-informed health Suppl 1):S1. E | INVOLVE Supnd social care rese-11]. Available frog L, Lewis S, Zackedge translation. I. Doing Better Bu
ation and Disclosuonal Finance Corp
ww.ifc.org/ifcext/enublicConsultation.a D, Warner M. Staok for Companies ton: International
ww.ifc.org/ifcext/enent_full/$file/ifc_stte Consultation Senications Branch.
ment in Decision M43-X Available froormats/pacrb-dga
folding stakeholdenking. Theory, respreenleaf; 2002. p.H. Using evidencstol: The Policy PS, Fretheim A. SUPolicymaking. He
pporting public invoearch [Eastleigh: m: http://www.invokheim L, Casebeemplementation S
usiness through Efure: A good practicporation; 1998. (0
nviro.nsf/Attachmepdf akeholder EngageDoing Business iFinance Corporat
nviro.nsf/attachmetakeholderengageecretariat, Health Health Canada P
Making. Ottawa: Hom: http://www.hc-apcr/pdf/public-con
S
er engagement. Inponsibility and 19-42.
ce. How research ress; 2008.
UPPORT Tools forealth Res Policy S
olvement in NHS,INVOLVE;2011 [co.org.uk/
er A. Revisiting intcience. 2007;2:34ffective Public ce Manual. Washi0-8213-4342-4) A
entsByTitle/p_pub
ement: A Good Pn Emerging Marktion; 2011. Availa
entsbytitle/p_stakeement.pdf Policy and olicy Toolkit for Pealth Canada; 20-sc.gc.ca/ahc-nsult/2000decision
Stakeholder Involv
n:
can
r Syst.
, public cited
teraction 4.
ington: Available
bconsult/
ractice ets. able
eholdere
ublic 00. 0-
n-
9
1
1
1
1
1
1
1
1
vement
9. Nutley S, MEuropean cEvidence&P
0. Kristensen FHandbook. AssessmenAvailable frohttp://www.sfinal.pdf
1. Caplan N. TAmerican B
2. Freeman R.Boston: Pitm
3. SubmissionIQWIG;2011from: https:/
4. Nielsen C, Kstakeholderthe EuropeaEUnetHTA. 91.
5. Leys M, Reypolitique deinitiatives inBaudouin; 2frb.be/uploaFRB/05%29ications/FR_
6. Becher G, Vprenante devoix. BruxelAvailable froFRB/05%29ications/Pat
7. Tronto JC. ULa Découve
orton S, Jung T, Bountries: diverse
Policy. 2010;6(2):1FB, Sigmund H. HCopenhagen: Dant, National Board om: sst.dk/publ/Publ20
The two-communitehavioral Scientis. Strategic managman; 1984. of comments | Co1 [updated 2011-1//www.iqwig.de/inKristensen F, Bistrs and developing an network for HeInt J Technol Ass
yntens S, Gobert s soins de santé: ternationales et b
2007. 978-2-87212adedFiles/KBS-9_Pictures,_docum_Participatie3.pdfVandenbroeck P, We la politique des sles: Fondation Room: http://www.kb9_Pictures,_documtients_partie_prenUn monde vulnéraerte; 2009.
Boaz A. Evidenceapproaches and c131-44. Health Technologynish Centre for Heof Health; 2008.
008/MTV/Metode/
ties theory and knst. 1979;22(3):459gement: a stakeho
ommenting proce11-23; cited 2012-dex.507.en.htmlrup M, Cecchetti Aa policy for stakealth Technology A
sess Health Care.
M. La participatiorevue de la littéra
belges. Bruxelles: 2-526-5 Available
ments_and_externfWouters A. Les psoins de santé: faoi Baudouin; 2008bs-frb.be/uploadedments_and_externnante%281%29.pdable: pour une po
KCE Reports
e and policy in six common challeng
y Assessment ealth Technology (978-87-7676-64
/HTA_Handbook_
nowledge utilizatio9-70. older approach.
edure (Hearing) [K-01-13]. Available
A, Turk E. Involvineholder involvemeAssessment, . 2009;25(Suppl 2
on des patients daature et aperçu deFondation Roi
e from: http://www
nal_sites/09%29_
patients, partie ire entendre d’aut
8. 978-2-87212-54dFiles/KBS-nal_sites/09%29_df litique du care. Pa
s 174
es.
9-8)
_net_
on.
Koeln: e
ng ent in
2):84-
ans la es
.kbs-
_Publ
tres 43-2
_Publ
aris:
KCE Reports 17
18. Léonard paradigméconomiq
19. Léonard responsaMangez innovatiocrise ? Rl'Associa2010. p.
20. Léonard responsa2010;30(
21. Mamboupratique (GCP). B(KCE); 20from: http://kceeline_low
22. Albertijn Jonckheeen langagfemmes ede santé based Adhttp://kceconseils-inte%CC
23. Patient aExcellenchttp://wwchome.js
24. NICE. Ca132. 201http://ww
74
C. La responsabime pour refonder lque/Ethics and EcC. Un nouveau p
abilisation capacitE, Nyssens M, Re
ons économiques Regards interdiscip
tion d’Economie S113-31. C. Fixer les priori
abilisation "capaci(322):622-4. rg F, Gailly J, Wepour l’accouchem
Bruxelles: Centre f010. KCE Reports
e.fgov.be/sites/defw_risk_birth_0.pdfM, Baeke A-M, Jaer P. Accouchemege courant à intégenceintes. Bruxel(KCE); 2011. KC
dvices and Recome.fgov.be/fr/publica-scientifiques-trad
C%81gr and public home [Nce;2011 [cited 20
ww.nice.org.uk/getsp aesarean section:1. Understanding
ww.nice.org.uk/nice
lisation capacitan'Etat-providence?conomics. 2012;Aaradigme pour lesante. In: Degavereman P, editors. Tet sociales en Eu
plinaires - Actes dSociale: Presse U
tés en soins de satante" des citoyen
ei-Hong Z. Recomment à bas risque.fédéral d’expertises 139B (D/2010/1
fault/files/page_dof anssens S, Bielenent normal : consegrer aux documenles: Centre fédéra
CE CLEAR (Commmmendations) 1B ation/report/accouuits-en-langage-c
National Institute f12-01-11]. Availabtinvolved/patientsa
Information abou NICE guidance Aemedia/live/13620
S
nte: un nouveau ? Éthique et Accepted for publics assurances socre F, Desmette D, Transformations europe : quelles sordu Colloque de Universitaires de L
anté: pour une ns. Prescrire.
mandation de bon Good Clinical Prae des soins de sa0.273/63) Availa
ocuments/kce_13
n F, Mambourg F, eils scientifiques tnts d’information dal d'expertise des
mon Language EvAvailable from:
uchement-normal-courant-a%CC%8
for Health and Clible from: andpublic/patienta
ut NICE clinical guAvailable from: 0/57166/57166.pd
Stakeholder Involv
cation. iales: la
et rties de
Louvain;
nne actice nté ble
9c_guid
traduits des soins idence-
-0-
nical
andpubli
uideline
df
2
2
2
2
2
3
3
3
3
vement
25. NICE. Treatadults: InforUnderstandfrom: http://w
26. NICE. DoneAlzheimer's guidance 21534-5) Avahttp://www.n
27. Thomas V. involvementles soins deFondation R
28. Caldon LJMConsumers Health ServStudies. 201
29. Milne RG. Revaluation oand SMS reof Consume
30. Ward LJ, Rhsocial care eJournal of C
31. Barr J, Ogdwhat patientinvolvementof Consume
32. Avard D, Stin health geJournal of C
33. Thomas P, Whealth serviinform future2010;34(5):
ting generalised armation about NICing NICE guidancwww.nice.org.uk/nepezil, galantamin
disease: Informa17. 2011. Understilable from: nice.org.uk/nicem"We were the grot at NICE. In: Pro
e santé et la politiqRoi Baudouin. M, Marshall-Cork H
as researchers –vice Research. Int10;34(5):547-50.
Reducing non-atteof the effectiveneseminders at a Scoer Studies. 2010;3hodes CA. Embededucation – a univ
Consumer Studiesen K, Rooney K. Vt partnership in clt of educationally er Studies. 2010;3anton Jean M, Grnomics: the reality
Consumer StudiesWilson C, Jones Pce users and caree policy. Internatio525-31.
anxiety disorder anCE clinical guidelince 197 (978-1-849nicemedia/live/13
ne, rivastigmine anation about NICE ttanding NICE guid
edia/live/13419/5oup's conscience"-oceedings of 'Patieque de santé'; 201
H, Speed G, Reed– innovative experernational Journa
endance at speciass and cost of patittish health board
34(5):570-80. dding consumer cversity office's pe
s. 2010;34(5):596-Viewpoint: let's teinical practice canengaged patients
34(5):610-2. régoire G, Page My behind the polic
s. 2010;34(5):508-P. Strengthening ters in Wales: a foconal Journal of Co
nd panic disorder ne 113. 2011. 936-463-8) Availa314/52602/52602nd memantine for technology apparadance (978-1-849
3624/53624.pdf - patient and publents, partenaires 11; Bruxelles:
d MWR, Collins KAriences in UK Natial of Consumer
alist clinics: an ient-focussed boo
d. International Jou
culture in health anrspective. Interna-602.
each medical studn be, with the s. International Jo
M. Public involvemcies. International -24. the voice of mentcus group study toonsumer Studies.
71
in
able 2.pdf
aisal 936-
ic dans
A. ional
oking urnal
nd tional
ents
urnal
ment
al o
72
34. Tetroe J.Research
35. NICE. Thhttp://ww7.pdf
36. IQWiG | cited 201procedur
37. Venturiniactor-net2010;19(
38. Venturinidigital mebefore pr
39. MommerN, AnnemadolesceinternatioKCE = Fefédéral dKnowledgAvailablehttp://kceldren_me
40. De GendMélard FBelgium]Health CAvailableand-chiro
41. Van MeeBuyse B,AssessmCentre (KAvailabletherapy
Knowledge Tranh: A Primer. Focuhe guidelines man
ww.nice.org.uk/me
Basic principles [K2-01-12]. Availab
res.926.en.html i T. Diving in magtwork theory. Pub(3):258-73. i T. Building on faethods. Public Unrint(Dec.5). rency G, Van Denmans L, Schoentjeent mental health conal overview. Heederaal Kennisce'expertise des soige Centre; 2011.
e from: e.fgov.be/sites/defental_health.pdf dt T, Desomer A, GF, et al. [Ostheopa
. Health Services are Knowledge C
e from: http://kce.fopractic-state-of-aerhaeghe A, Anne, Benoit K, et al. H
ment (HTA). BrussKCE); 2011. KCE e from: http://kce.f
slation at the Cans Technical brief.
nual. London: NICdia/FA1/59/Guide
Koeln: IQWIG;201ble from: https://ww
ma: How to explolic Understanding
ults: how to reprenderstanding of Sc
n Heede K, Verhaees E, et al. Organcare : study of thealth Services Res
entrum voor de geins de santé = BeKCE Reports 170
fault/files/page_do
Goossens M, Hanathy and chiroprac
Research (HSR)Centre (KCE); 201fgov.be/publicationaffairs-in-belgiummans L, Haentjen
Home Oxygen Theels: Belgian HealtReports 156C (D
fgov.be/publication
S
nadian Institutes o2007;18.
CE; 2007. AvailabelinesManualChap
11 [updated 2010-ww.iqwig.de/meth
ore controversies wg of Science.
sent controversiecience. 2010;onlin
eghe N, Swartenbisation of child an
e literature and ansearch (HSR). Bruzondheidsdzorg =lgian Health Care
0C (D/2011/10.27
ocuments/KCE_1
nquet G, Léonard ctic: state of affairs. Brussels: Belgia0. KCE Reports 1n/report/ostheopa
ns P, San Miguel Lerapy. Health Tecth Care Knowledg
D/2011/10.273/25)n/report/home-oxy
Stakeholder Involv
of Health
le from: pters200
-11-24; hods-
with
s with ne
broekx nd n ussels: = Centre e 3/78)
70c_Chi
C, s in
an 48
athy-
L, chnology ge ygen-
4
4
4
4
4
4
4
455
vement
42. Jonckheer PDe Troyer Mand manageBelgian Hea165 Availabamong-gene
43. Moret-Hartmassessmentthe drug meAssessmen
44. Amblard H, traduction. Lorganisation
45. Cargo M, Mresearch: st2008;29:325
46. HAS. Practi2010. Availsante.fr/port06/guideline531.pdf
47. NICE. Guide2008. (1-84http://www.ne2008.pdf
48. Ilott I, MeadInvolvementthe experienmidwifery an
49. AGREE. AG50. ADAPTE. M51. Elliott J, Hee
Methods ToFoundation;
P, Stordeur S, LebM, et al. [Burnout aement]. Health Sealth Care Knowledble from: http://kceeral-practitioners-
man M, van der Wt and ill-structured
ebeverine. Internat in Health Care. 2Bernoux P, Herre
Les nouvelles appns. Paris: Seuil; 19
Mercer SL. The valtrenghtening its pr5-50. ce guidelines “Folable from: http://wtail/upload/docs/ae_by_formal_cons
e to the methods 4629-741-9) Avainice.org.uk/media
d J, Roberts J, Hamt in NICE Guidelinnce of the allied hnd nursing. Octob
GREE Instrument Manual for Guidelinesterbeek S, Luke
oolkit: a practitione; 2005. 90-5130-5
beer G, Roland Mamong general prervices Research dge Centre (KCE)e.fgov.be/publicati-prevention-and-m
Wilt GJ, Grin J. Head problems: a casational Journal of T2007;23(3):316–2eros G, Livian Y-Fproches sociologiq996. lue and challengeractice. Annu Rev
rmal consensus” www.has-application/pdf/201sensus_quick_me
of technology applable from:
a/B52/A7/TAMetho
mmond R. Enhanne Development: ealth professions
ber 2004. training manual. 2ne Adaptation v1.ensmeyer CJ, Sloer’s manual. Bruss506-0 Available fro
KCE Reports
M, De Schampheleractitioners: preve(HSR). Brussels:
); 2011. KCE Repon/report/burnout
management alth technology e study concerninTechnology 23. F. Une sociologie dques des
es of participatory v Public Health.
method. Paris: HA
11-ethodology_guide_
praisal. London: N
odsGuideUpdated
ncing Stakeholder Learning lessons , health visiting,
2003. .0. 2007.
ocum N. Participatsels: King Baudouom: http://www.kb
s 174
eire J, ention
orts t-
ng
de la
AS;
_110
NICE;
dJun
from
tory uin
bs-
KCE Reports 17
frb.be/upFRB/File
52. NICE. Hostakehold(1-84629http://ww_are_dev
53. IQWIG. Greports: shttps://wwsion_of_c
54. IQWiG | 09-23; cithttps://ww
55. VIKC. Drevaluerefrom: http://wwe/Draaibo%20aang
56. Morestin Synthesiz2010. Avhttp://ww
57. AlexandeRole RelaChronic Iprint(Nov
58. Bos V, VZonMw; 2
59. CHSRF. achieve mServices Available
74
ploadedFiles/KBS-s/EN/PUB_1540_ow NICE clinical gders, the public an9-850-4) Availableww.nice.org.uk/meveloped_4th_edn_Guideline on the ssample. Koeln: IQww.iqwig.de/downcomments_on_prProduct preparatted 2012-01-13]. Aww.iqwig.de/produraaiboek richtlijnenn van richtlijnen. U
ww.oncoline.nl/uplooek%20openbaregepast%20voor%F, Gauvin F-P, Hzing Knowledge available from:
ww.ncchpp.ca/docser JA, Hearld LR, ationships and PaIllness. Health Sev. 18).
Van Kammen J. Kn2007. Public engagememeaningful publicResearch Found
e from:
-_Participatoty_tooguidelines are devnd the NHS. londoe from: dia/62F/36/How__FIANL_LR.pdf submission of com
QWIG; 2010. Avainload/Sample_Gureliminary_reportstion [Koeln: IQWIGAvailable from: uct-preparation.92n. Onwikkelen, imUtrecht: VIKC; No
oaded/docs/Algeme%20versie%202%20drukwerk_moi.
Hogue M-C, Benoiabout Public Polic
s/MethodPP_EN.Mittler JN, Harvey
atient Activation arvices Research.
nowledge synthes
ent (part II): how dc engagement? Otation; 2009 Feb.
S
olkit_New_edition.veloped: an overvon: NICE; 2009 Ja
NICE_clinical_gu
mments on prelimiilable from: uidelines_on_the_s.pdf G;2011 [updated 2
29.en.html mplementeren en ovember 2009. Av
meenOncolineEnP%20november%2pdf t F. Method for
cies. Montréal: NC
pdf y J. Patient–Physmong Individuals 2011;Epub ahead
sis : a guide. The
do deliberative prottawa: Canadian HInsight and action
Stakeholder Involv
pdf iew for anuary
idelines
inary
_submis
2011-
vailable
Pallialin02009
CCHPP;
ician with
d of
Hague:
ocesses Health n 49
6
6
6
6
6
6
vement
http://www.c49_e.pdf
60. AccountAbilLondon: Acchttp://www.a202010%20
61. CCOHTA. SConsultationHealth Techhttp://64.26.pdf
62. Griffiths J, MBackgroundpreventing NWHO; 2008http://www.winvolvement
63. E.P P, Vijfvithe impact oZoetermeerhttp://www.k55_hsr_imp
64. Hulstaert F,L, Vernelen Technologyknowledge CAvailable frodiagnostics-
65. Eyssen M, KMonitoring oSyndrome]. Belgian HeaKCE reportshttp://kce.fgprevention-o
chsrf.ca/Migrated/
lity. AA1000 StakecountAbility; 2008accountability.org/0PRINT.pdf Summary Report on Sessions. Ottawhnology Assessme.163.205/media/p
Maggs H, George d paper prepared fNoncommunicable8. Available from: who.int/dietphysict.pdf inkel D, Vennekenof the Belgian Hear: 2010. Availablekce.fgov.be/sites/dpact_study%20kce Huybrechts M, VK, et al. [Molecul
y Assessment (HTCentre (KCE); 20om: http://kce.fgov-in-belgium Kohn L, Lambert Mof Infants in PreveHealth Technolog
alth Care Knowleds 46 Available fromov.be/publication/of-sudden-infant-d
/PDF/insightAction
eholder Engagem8. Available from:/images/content/5
of 2004 COMPUSwa: Canadian Cooent (CCOHTA); 2df/compus_stake
E. Stakeholder Infor the WHO/WEFe Diseases in the
calactivity/griffiths-
ns A, van Hoesel alth Care Knowlede from: default/files/page_e.pdf
Van Den Bruel A, Car Diagnostics in
TA). Brussels: Belg05 22/11/2005. Kv.be/publication/re
M-L, Van Den Steention of Sudden gy Assessment (Hdge Centre (KCE)m: /report/home-mondeath-syndrome
n/Insight_and_Ac
ment Standard.
5/4/542/AA1000S
S Stakeholder ordinating Office fo004. Available froholder_report_ap
nvolvement: F Joint Event on Workplace. Gene
-stakeholder-
P, Daue F. Studydge Centre.
_documents/2008
Cleemput I, BonneBelgium]. Health gian Health Care CE reports 20 eport/molecular-
een D. [Home Infant Death HTA). Brussels: ); 2006 21/12/200
nitoring-of-infants-
73
ction_
ES%
or om: ril05.
eva:
y into
8-
eux
6.
-in-
74
66. Mathieu al. [QualiClinical PCentre (Khttp://kcecare-for-d
67. GuillaumL, et al. OSolidarityHealth Creports 1http://kceprocesse
68. Van den Sande SProjectioBrusselsreports 1http://kcepersons-
69. BorgermaFallon C,physicianHealth Creports 1http://kceon-prima
C, Nobels F, Peetty and organizatio
Practice (GCP). BKCE); 2006 12/05e.fgov.be/publicatidiabetes-2
me P, Moldenaers Optimisation of they Fund. Health Seare Knowledge C33C (D/2010/10.2
e.fgov.be/publicaties-of-the-special-sBosch K, Willemé, et al. [Residentians 2011 – 2025 ].: Belgian Health C67 Available from
e.fgov.be/publicatiin-belgium-projecans L, Dubois C, , et al. Impact of ans. Health Serviceare Knowledge C25C (D/2010/10.2
e.fgov.be/publicatiary-care-physician
ters G, Van Royeon of the care for russels: Belgian H
5/2006. KCE reporion/report/quality-
I, Bulté S, Debruye operational procervices Research Centre (KCE); 201273/46) Availableion/report/optimisasolidarity-fund é P, Geerts J, Breal care for older p. Health Services Care Knowledge C
m: ion/report/residenctions-2011-%E2%Rieppi S, Vanhae
academic detailinges Research (HSR
Centre (KCE); 201273/16) Availableion/report/impact-
ns
S
n P, Dirven K, Wediabetes 2]. Good
Health Care Knowrts 27 Available frand-organization-
yne H, Devriese Scesses of the Spe(HSR). Brussels: 0 28/07/2010. KC
e from: ation-of-the-opera
eda J, Peeters S, Versons in BelgiumResearch (HSR).
Centre (KCE); 201
tial-care-for-older%80%93-2025 eren S, Geukens Ng on primary care R). Brussels: Belg0 29/03/2010. KC
e from: of-academic-deta
Stakeholder Involv
ens J, et d
wledge om: -of-the-
S, Kohn ecial Belgian
CE
ational-
Van de m :
11. KCE
r-
N,
gian CE
ailing-
vement
KCE Reportss 174