outline of cerqual approach - trusted evidence. informed
TRANSCRIPT
TheGRADE-CERQual approach:Assessingconfidenceinfindingsfromsynthesesofqualitativeevidence
ConfidenceintheEvidencefromReviewsofQualitativeresearch
WhydidwedevelopGRADED-CERQual?
• Systematicreviewsofqualitativeresearch(alsocalledqualitativeevidencesyntheses)becomingincreasinglycommon
• Alsoincreasinglybeingusedinguidelineorpolicydevelopmentprocesses
• Usersneedmethodstoassesshowmuchconfidencetoplaceinfindingsfromthesereviews
• Userslikelytomakethesejudgementsanyway– itishelpfultoprovideasystematicandtransparent wayfordoingthis
HowwasGRADE-CERQual developed?• Researcherswithbackgroundsinqualitativeresearchandsystematicreviews• Broadconsultationwithwidegroupofstakeholders
Neededanapproachthat:• Couldbeappliedtotypicaltypesofqualitativestudyapproachesanddata• Waseasytouse• Allowedjudgementstobereportedtransparently• Allowedthejudgementstobeunderstood
RelationshiptoGRADE
• CERQualispartoftheGRADEWorkingGroup
• CERQualsharesthesameaimastheGRADEtoolusedtoassessthecertaintyofevidenceofeffectiveness
• However,CERQualisgroundedintheprinciplesofqualitativeresearch
CERQualisnotatoolfor:
•Assessinghowwellanindividualqualitativestudywasconducted
•Assessinghowwellasystematicreviewofqualitativestudieswasconducted
•Assessingquantitativestudiesofqualityofcare
•Assessingconfidencein‘narrative’or‘qualitative’summariesoftheeffectivenessofanintervention,wheremeta-analysisisnotpossible
Whatdoes theCERQualapproachdo?•CERQualaimstotransparentlyassessanddescribehowmuchconfidencetoplaceinfindingsfromqualitativeevidencesyntheses
CERQualisappliedtoindividualsynthesisfindings
• Inthecontextofaqualitativeevidencesynthesis,afindingis…:
• Findingsfromqualitativeevidencesynthesescanbepresentedas:• themes,categoriesortheories• Asbothdescriptiveormoreinterpretivefindings
…ananalyticoutputthatdescribesaphenomenonoranaspectofaphenomenon
Whatdowemeanby’confidenceintheevidence’?
Anassessmentoftheextenttowhichareviewfindingisareasonablerepresentationofthephenomenonofinterest
• i.e.thephenomenonofinterestisunlikelytobesubstantiallydifferentfromthereviewfinding
TheCERQualapproach
What skillsdoyou need toapplyCERQual?
• Anunderstanding of systematic review methodology• Anunderstanding of the principles of qualitative research
CERQualmadeeasy
Scenario:
Decisionmakersareconsideringanewintervention- Buthowarefemalepatientslikelytoexperienceit?
Reviewofqualitativeresearchiscarriedout–findingsdescribewomen’sexperiencesoftheintervention
Afterassessingallfourcomponentsanoverallassessmentismade,expressedaseither:
- Highconfidence- Moderateconfidence- Lowconfidence- Verylowconfidence
Component1:Methodologicallimitations
Theextenttowhichthereareproblemsinthedesignorconductoftheprimarystudiessupportingareviewfinding
Concernsaboutmethodologicallimitations
theprimarystudiesunderlyingareviewfindingareshowntohaveproblemsinthewaytheyweredesignedorconducted
� Wearelessconfidentthatthefindingreflectsthephenomenonofinterestwhen:
� Acritical appraisaltoolforqualitativestudiesshouldbeused tomakethisassessment� Typicallyincludesappraisalsofhowtheparticipants
andsettingswereselected,howdatawascollectedandanalysed,researcherreflexivityetc
� Currentlynowidespreadagreementaboutthebesttool– researchagendainplace
Component2:Relevance
Theextenttowhichthebodyofevidencefromtheprimarystudiessupportingareviewfindingisapplicabletothecontextspecifiedinthereviewquestion
Concernsaboutrelevance
thecontextsoftheprimarystudiesunderlyingareviewfindingaresubstantivelydifferentfromthecontextofthereviewquestion
Wearelessconfidentthatthefindingreflectsthephenomenonofinterestwhen:
Assessing relevance - examples
• Exampleofindirectrelevance:Inasynthesisfocusingonchildren,oneincludedstudyfocusedonchildrenfrom3-5yearswhilethesynthesiswasinterestedintheagegroup10-18years
• Exampleofpartialrelevance:Inthesamesynthesis,severaloftheincludedstudiesfocusedongirlsonly,whilethesynthesiswasinterestedinallchildren
• Exampleofuncertainrelevance:Inthesamesynthesis,theagesofthechildreninsomeofthestudieswasunclear
Component3:Coherence
Anassessmentofhowclearandcogentthefitisbetweenthedatafromtheprimarystudiesandthereviewfinding*
*HasbeenupdatedsincePLOSarticle
Concernsaboutcoherence
Wearelessconfidentthatthefindingreflectsthephenomenonofinterestwhen:- Someofthedatacontradictthefinding- Someofthedataareambiguous
Assessingcoherence–transformationofthedata
Degree of transformation
Component4:Adequacyofdata
Thedegreeofrichness andquantityofdatasupportingareviewfinding
Concernsaboutadequacyofdata
thedataunderlyingareviewfindingarenotsufficientlyrichoronlycomefromasmallnumberofstudiesorparticipants
• Reviewauthorsneedtomakeajudgementonwhatconstitutesdatathatarenotsufficientlyrichortoosmallanumberinthecontextofaspecificreviewfinding
Wearelessconfidentthatthefindingreflectsthephenomenonofinterestwhen:
Making anoverallassessment
METHODO-LOGICAL
LIMITATIONS
COHERENCE
RELEVANCE
ADEQUACY OF DATA
Confidence
Afterassessingeachoftheseparatecomponents,wemakeanoveralljudgementoftheconfidenceineachreviewfinding
Confidencecanbeassessedashigh,moderate,loworverylow
• Highconfidence:Itishighlylikelythatthereviewfindingisareasonablerepresentationofthephenomenonofinterest
•Moderateconfidence:Itislikelythatthereviewfindingisareasonablerepresentationofthephenomenonofinterest
• Lowconfidence:Itispossiblethatthereviewfindingisareasonablerepresentationofthephenomenonofinterest
• Verylowconfidence:Itisnotclearwhetherthereviewfindingisareasonablerepresentationofthephenomenonofinterest
CERQual qualitativeevidenceprofile
� Transparencyoftheprocessofassessingeachcomponentisenhancedthroughtheuseofaqualitativeevidenceprofilelikethisone
� Inthisprofile,reviewauthorsshouldindicate:¡ Whethertheyhaveno,minor,moderateorsubstantialconcernsregardingeachCERQual componentandwhatthese
concernsare¡ TheiroverallCERQual assessmentandtheirexplanationofthis
� TheQualitativeEvidenceProfilecanbeincludedasanappendixtoaqualitativeevidencesynthesis
Summary of Qualitative Findings tables
February17,2017 38
ASoQF tableisthefinaloutputoftheprocessofmakingaCERQualassessment,andincludesfour elements:1. Asummaryofeachreviewfinding2. AnoverallCERQual assessmentforeachreviewfinding3. Anexplanationoftheoverallassessment(couldbeincludedasfootnotes)4. Referencetothestudiescontributingdatatothereviewfinding,including
clarificationofthecontextsinwhichthosestudieswereconducted
Operationalising theCERQual approach• OurrecentlypublishedPLOSMedicinepaperprovidessomeguidanceonoperationalizingeachCERQual component– pleasehavealookatthispaperbeforeyoutrytoapplyCERQual!
•WearenotrecommendinganyspecifictoolstoassesseachCERQual component.Rather,wehaveidentifiedarangeofissuesthatshouldbeconsideredforeachcomponent
•Moredetailedguidanceandpromptsarebeingdevelopedthisyearandwillbepublishedasaseriesofpapers
•MembersoftheCERQual ProjectGroupareavailabletomentornewusers
Insummary:CERQualhasthepotentialto:• Facilitatereflectiononfindingsfromqualitativeevidencesyntheses•Makemoreexplicit:• wheretherearegapsorinsufficientevidenceinrelationtoareviewquestion• whereonlypoorqualitystudiesareavailable
• Promptthegenerationofnewexplanationsorconceptstoexplainpatternsinfindings
ApplyingCERQual involvesjudgements.CERQualattemptstomakethesejudgementstransparentandincreasethecontributionofqualitativeresearchtodecisionmaking
GettinginvolvedinGRADE-CERQual• YouareinvitedtojoinourmailinglistandtheGRADE-CERQualProjectGroupviaourwebsite:www.cerqual.org
• You can makeanimportant contribution by:• Piloting CERQual inyour qualitative evidence synthesis• Recording what worked well andwhat was moredifficult inthe process• Sharing this feedbackwith us,sothat we can use this toimprove ourguidance forundertakingCERQual assessments
• Practical resources (e.g.templates)are available andwe can alsolinknew CERQual users with amentor.Please contact us torequest access tothese
• TheGRADE-CERQual ProjectGroupwill holdmeetings andtrainingsessionsduringthe year – information on these will besentoutviaour mailing list
TolearnmoreaboutCERQual
• Jointhemailinglist• Jointheprojectgroup• Joinawebinarorfurtherface-to-facetraining• Forthcomingseriesofpapers,probablyinImplementationScience