development of a methodological approach to evaluate the · pdf file ·...
TRANSCRIPT
• Nine stakeholders were invited and all took part.• Interviews from Apr-Aug/16; duration 32-77 min.• Identified themes: context, the logic model,
development and implementation, views on DCEand its components, what to evaluate, andoutcome evaluation challenges (Fig. 3).
Results (cont.)• The overall aim of this study is to understand the role of health system
level initiatives in promoting the earlier diagnosis of cancer (Fig. 1).
• As part of this study we are evaluating the Detect Cancer Early(DCE) Programme in Scotland, assessing its processes and outcomes.
• Here we describe the methodology adopted to inform thedevelopment of this evaluation (Stage 1 of evaluation).
• Stage 1’s aims were to involve stakeholders, to enhance understandingof DCE and to help define which key aspects should be evaluated.
Background
Systematic review In-depth interviews• Characteristics and outcomes of initiatives• Stakeholders’ views and experiences• Use of a narrative synthesis approach
• Interviews with international cancer experts• Purposive sampling• Framework analysis
Developing and refining the evaluation• Review of programme documents and development of a logic model• Semi-structured interviews with key stakeholders (framework analysis) and refinement of the logic model
Stage 1
Process evaluation(implementation and service delivery)
Outcome evaluation(effectiveness and outcomes)
• Informed by the MRC guidance onprocess evaluation of complexinterventions1
• Interviews and questionnaires withstakeholders
• Review of programme documents
• Critically summarise evidence from ISD Scotland datasets –to what extent were key objectives met?
• Additional analyses with a focus on socio-economic factorsand rurality, and selected additional or unintended outcomes
• Nested case studies: focus on awareness, primary care andscreening (TBC)
Understanding the international landscape
DCE evaluation
Figure 1: Full study diagram
• Stage 1 consisted of: 1) analysis of policy documents; 2) creation of alogic model; and 3) semi-structured interviews with key stakeholders.
• Criterion sampling and maximum variation sampling were adoptedto ensure inclusion of different stakeholders from a range of regions.
• Interviews were face-to-face or over the telephone, recorded,anonymised, transcribed and analysed using framework analysis.
Methods
• Seventy-seven policy documents were reviewed and summarised; a logicmodel (Fig. 2) was developed and feedback sought from DCE managers.
• Reviewed documents also aided the development of an interview topicguide and the creation of a list of DCE stakeholders to be interviewed.
Results
Resources/Inputs Outputs
Funding• Government funding• Sponsorship from charities for specific projects
Evidence base• NAEDI strategies• Available literature
• Behavioural change theories?
Scottish Government• NHSScotland
• Healthcare Improvement Scotland• Regional NHS Boards• Special NHS Boards
• NHS Health Scotland• NHS National Services Scotland
• NSS ISD; Medical payments (sGMS, SQoF); National Services Division
• Scottish Pathology Network• Scottish Clinical Imaging Network
• Health and Social Care Management Boards• Scottish Cancer Taskforce
• DCE Programme Board• DCE Operational Subgroup• CWT Data and Definitions Group• National Screening Comms Group• Scottish Primary Care Cancer Group• Colonoscopy Short Life Working Group
Cancer Charities• Scottish Cancer Coalition• Other Cancer Charities• Volunteers• Community
Scottish Cancer Prevention NetworkMarket Research companyAdvertising agencyMedia AgencyPR AgencyDigital AgencyLarge commercial organisations/key Scottish employers
Health structure, physical and human resources• Primary care services (general practices and community
pharmacies)• Secondary care services (hospitals, hospital pharmacies,
specialist consultations, radiology, pathology and otherdiagnostics)
• Scottish Breast Screening Centres• Scottish Bowel Screening Centre• Staff time (clinical, administrative, laboratory, etc.)
Additional partners/collaborationsProfessionals’ timeProfessionals’ resources
Activities
Small funded, local projects• Young people as influencers• FIT• Referral rates, detection rates and
staging before and after DCE• Other projects
Performance management and monitoring• Cancer Waiting Times (CWT) targets• Heat target• sQoF rewards (bowel screening)• Practice profiles
• Campaigns on mass awareness channels (TV,radio, printed press, outdoors) and bus stops
• Scottish public surveyed; attitudinal tracking reportspre and post campaigns
• National and regional roadshows and events, localhealth promotion activities and pilot studies
• Online activity using DCE (getchecked and weec) and charity websites and social media (Facebook,Twitter and YouTube)
• Celebrity support and use of case studies• DCE information materials given in field marketing
activities, and distributed to general practices andcommunity pharmacies
• Assets (e.g. hideaway books) given to local teams• Targeted strategies for those at risk (lung), men
(bowel), deprived women (breast), screening non-responders and key influencers (e.g. children)
Social marketing campaigns • Insight gathering (public attitudes and
knowledge)• Cancer specific campaigns
• Bowel cancer (screening)• Breast cancer (screening)• Breast cancer (symptomatic)• Lung cancer
• Overarching campaigns• Priming campaign• wee c• #getchecked
Stakeholder engagementPR and partnershipField marketing and partnership
• Reports about engagement and facilitatorprogrammes
• Updated guidelines published, printed anddisseminated online
• Primary care professionals recruited
• Published official reports on CWT, HEAT, sQoF, breast and bowel screening uptake
• PTI data checked to track overall activity andconsultations due to cancer specific symptoms
• Performance monitoring by the government andfollow up when NHS Boards do not meet targets
• Practices sign up to sQoF• Practices develop an action plan to improve bowel
screening uptake• Health Boards are provided with non-responders
lists from the bowel screening centre• DCE provided with bowel screening centre data on
FOBT kits requested and returned• Provision of data on referrals to secondary breast
clinics
Symptom management and referral• Primary Care Engagement Programme
• 7 sessions (bowel and lung cancer)• 4 pharmacy education sessions• Other local training
• Primary Care Facilitator Programme• Bowel screening awareness
training, sharing information aroundcancer resources
• Review of referral guidelines betweenprimary and secondary care
Outcomes
• Findings reported to DCE or published in peer-reviewed journals
Public• Increased awareness and knowledge about
cancer signs/symptoms and benefits ofscreening and early detection
• Public is empowered to check for symptoms and seek help
• Increase in bowel and breast screening uptake
• Increase in the number of patients consulting due to cancer related symptoms
Primary care professionals• Increased knowledge and awareness of
screening programmes• Increased knowledge and awareness of
alarm symptoms and referral guidelines• Increased awareness of the importance of
early investigation• Increased use of information resources on
screening, referral and symptom appraisal
Initial (1-3 years) Intermediate (4-6 years) Longer term (7-10 years)
Assumptions:Role of pathways to treatment in survival• Early presentation in primary care and screening participation lead to early detection and early detection results in improved survival
• “Early diagnosis can improve survival, manage the expected increase in cancer incidence and improve patient and carer experience”• Early referral and early treatment result in improved survivalDCE and the public• Public’s knowledge and awareness (of symptoms and cancer) will increase and fear of cancer will decrease with the social marketing campaigns• Increased knowledge/awareness/reduced fear will lead to symptom appraisal and help-seeking behaviour• Targeted campaigns will successfully reach the most deprived populations, men, and non-responders to screening• The public will engage with DCE during the campaigns• Messages from campaigns will have a sustained effectDCE and primary/secondary care• Practices will get involved in the DCE social marketing campaigns to help with dissemination• Financial incentives will lead to the development of successful strategies to increase bowel screening uptake• Professionals will agree with and follow updated cancer referral guidelines• Primary Care Engagement Programme will increase professionals’ knowledge about cancer and this will translate into better care• Increase in demand will be managed with capacity increase in screening, diagnosis and treatmentDCE overall impact• Cancers diagnosed at Stage 1 are a good proxy of patient survival, despite acknowledged limitations• It will be possible to separate changes in staging due to improvements in data recording from actual increase of cancers diagnosed at Stage 1• Reporting all cancers together is better than reporting cancer types separately, despite recognised limitations• It will be possible to measure the impact of DCE using administrative data, survey data and official statistics
Primary care professionals• More patients with alarm symptoms
receive timely referrals• Increased primary care role in
promoting informed screeningparticipation
Secondary care• More diagnostic/test procedures
carried out
Overall health system• Improvement in overall five-year
cancer survival (no target set)• Referrals are promoted at their
earliest opportunity for patients witha suspicion of cancer
• Increasingly faster access todiagnostics in secondary care (notarget set)
• Automated systems to identify bowel screening non-responders
• Practice profiles are made available• Increase in bowel screening uptake• Increase in breast screening uptake
amongst the most deprived
Context/external factors:Age, deprivation and other individual/biological factors• Older age is associated with higher cancer incidence and the population in Scotland is getting older• Older people and the most deprived are more affected by late diagnosis, treatment delays and survival deficit• Most deprived have higher cancer mortality rates, take part in screening less often and may have poorer access to health services• Multimorbidities (more common with older age and amongst the most deprived) are associated with poor survival• More deprived populations can have problems with literacy and targeted campaigns are needed• Different cultures/ethnicities are associated with different challenges regarding help-seeking behaviour and screening uptake• Men take part in bowel screening less often; more deprived women take part in breast screening less often• Tumour size, growth rate and location have an impact on survivalChallenges in diagnosis• Challenges diagnosing cancer with less distinctive or common symptoms, or cancers which present without alarm symptomsHealth care context• Current scenario in primary, secondary, tertiary care and the NHS in general is of financial strain, with limited resources/capacity and lack of time to
engage in different projects• Increasing role of primary care in cancer control• Increased awareness that 4/10 cancers can be prevented by lifestyle changes – change of individual behaviour a key concern but hard to achieve• Increased importance of genetic testing and personalised cancer care• Quality of care at treatment and immediately after treatment influences survivalOverall context• Scotland has remote areas (rural areas, islands) where access to care and recruiting professionals is more complex• Programme is funded by the government and its continuity is dependent on the government
Detect Cancer Early Programme - Logic Model
Problem: Scotland has poor cancer survival rates compared to other European countriesProgramme’s aim: improve overall five year survival for people in Scotland diagnosed with cancer
Public• More people consult in primary care
when noticing symptoms (no targetset)
• More people make an informed decision to take part in screening programmes
Primary care practices• Local initiatives (e.g. screening pilots, local
audits of screening) implemented toimprove bowel screening uptake
• Practices are rewarded for meeting targets• More practices receive SQoF rewardsRegional Health Boards• More boards meet CWT targets• More boards meet HEAT targetsHealth systems• Increased proportion of cancers
diagnosed at stage 1 (initial target 25%increase)
• Less cancer cases coded with anunknown stage
• More screening tests carried out• Timely production of data and better data
collection and reporting of cancer stages
• Young people acting as influencers(symptom checking and screening)
• Other outcomes
Overall health system• Maintained improved cancer
survival• Improved survival for more
cancer types• Reduced variation in the
number of referrals andpositive cancer diagnosesacross general practices
• Reduction in emergency presentation diagnoses (notarget set)
• Bowel screening rates similarto other screeningprogrammes
• Reduction of socioeconomicgradient of uptake in breastand bowel screening
• Reduction of gender disparity of uptake in bowel screening
• Optimum data quality oncancer staging
Secondary care and diagnostic capacity“Ensure screening capacity; imaging, diagnostics departments and treatment centres are prepared for the increase”
Local initiatives• To improve work practices and reduce DNA rates
for colonoscopy and mammography• To provide a formal 7-day CT service• Surgeons recruited• Creation of new cancer pathways• Acquisition of additional diagnostic equipment• Additional consultant surgeon and consultant
radiology sessionsCreation of new pathway for direct access to CTFunding to roll out FIT screening nationally
Secondary care• Increase in the number of diagnostic tests• Improved diagnostic capacity
Figure 2: DCE logic model
Development of a methodological approach to evaluate the Detect Cancer Early Programme in Scotland
Natalia Calanzani, David Weller, Christine Campbell
©d-
map
s.co
m
60 km
40 mi
Figure 3: Key themes
“I know it was a three-year programme but it is bigger than that, it's trying to promote
one influencer of cancer survival and improve patient experience at the same time, and be efficient in the way that we
use money, and provide improved cancer diagnostics faster.” (ID 7)
• Results are informing the design of the DCE evaluation in terms ofcausal assumptions to explore, implementation issues to investigateand outcomes to assess – these include:→ Whether increase in referrals led to more cancer diagnoses and
additional investment resulted in improved diagnostic capacity.→ Investigating any workload impact of implementing the
programme in primary and secondary care.→ Assessing whether DCE was delivered as planned across
different Health Boards.→ Exploring ways to investigate system-level outcomes in
addition to hard data on survival and its surrogates.→ Assessing whether there was variation in outcomes across
different population groups.• Importantly, challenges collecting, assessing and interpreting data
should not be underestimated.
Discussion
1. Moore G, Audrey S, Barker M, et al. Process evaluation of complex interventions. 2014.https://www.mrc.ac.uk/documents/pdf/mrc-phsrn-process-evaluation-guidance-final/
References
Overarching contextual issues
Development and implementation• A political decision with a political imperative• Aims, drivers and influencers• An evidence-based initiative• Resource allocation• Stakeholder involvement• Voiced concerns and barriers• Developing relationships, systems and roles• Delays and adaptations
Views on DCE and its components• A driver of change and other aspirations• Belief in DCE and feedback on its activities• Perceived impact of activities• Sustainability concerns• The role of targets
Logic model• Comprehensiveness and
usefulness• Initiatives: scope, importance,
links and trajectory• Proposed outcomes• Underlying assumptions
Outcome evaluation challenges• Accessing and understanding data• Bias and confounding• Results take time• Unmeasurable, poorly defined or outcomes
with different definitions
What to evaluate• Early diagnosis, mortality, survival or
surrogates• Impact on primary/secondary care• Improvement in awareness and knowledge• Seeing the bigger picture• Unanticipated outcomes
• Evidence and international comparisons• Scottish context and pioneering• NHS constraints and regional variation
• Cultural change or shifts• Tackling cancer• Role of the media
Implementation and mechanisms of impact
Outcomes
Description of the intervention and its causal assumptions
Context
Themes derived from interviews and organised according to the MRC framework of process evaluation of complex interventions1
• Deprivation, age and gender• Individual behaviour and change• Government characteristics
“Primary care's quite defined, marketing's quite defined, then we
come to secondary care and it's like what did you do? Well I suppose we invested in that and we gave
boards money and then kinda left them to get on with it.” (ID 2)
“There was a distinct anxiety that you could just increase activity to a rate that would effectively just overwhelm the capacity in some of the clinical services.” (ID 4)
“I think the main challenge in Scotland is around equity and there's no doubt, you know, Scotland's got a very sort of wide socioeconomic gap
across its society.” (ID 9)
This study is funded by the Scottish Government and supported by scholarships from the University of Edinburgh as part of NC’s PhD studies.