development of a methodological approach to evaluate the · pdf file ·...

• Nine stakeholders were invited and all took part.• Interviews from Apr-Aug/16; duration 32-77 min.• Identified themes: context, the logic model,

development and implementation, views on DCEand its components, what to evaluate, andoutcome evaluation challenges (Fig. 3).

Results (cont.)• The overall aim of this study is to understand the role of health system

level initiatives in promoting the earlier diagnosis of cancer (Fig. 1).

• As part of this study we are evaluating the Detect Cancer Early(DCE) Programme in Scotland, assessing its processes and outcomes.

• Here we describe the methodology adopted to inform thedevelopment of this evaluation (Stage 1 of evaluation).

• Stage 1’s aims were to involve stakeholders, to enhance understandingof DCE and to help define which key aspects should be evaluated.

Background

Systematic review In-depth interviews• Characteristics and outcomes of initiatives• Stakeholders’ views and experiences• Use of a narrative synthesis approach

• Interviews with international cancer experts• Purposive sampling• Framework analysis

Developing and refining the evaluation• Review of programme documents and development of a logic model• Semi-structured interviews with key stakeholders (framework analysis) and refinement of the logic model

Stage 1

Process evaluation(implementation and service delivery)

Outcome evaluation(effectiveness and outcomes)

• Informed by the MRC guidance onprocess evaluation of complexinterventions1

• Interviews and questionnaires withstakeholders

• Review of programme documents

• Critically summarise evidence from ISD Scotland datasets –to what extent were key objectives met?

• Additional analyses with a focus on socio-economic factorsand rurality, and selected additional or unintended outcomes

• Nested case studies: focus on awareness, primary care andscreening (TBC)

Understanding the international landscape

DCE evaluation

Figure 1: Full study diagram

• Stage 1 consisted of: 1) analysis of policy documents; 2) creation of alogic model; and 3) semi-structured interviews with key stakeholders.

• Criterion sampling and maximum variation sampling were adoptedto ensure inclusion of different stakeholders from a range of regions.

• Interviews were face-to-face or over the telephone, recorded,anonymised, transcribed and analysed using framework analysis.

Methods

• Seventy-seven policy documents were reviewed and summarised; a logicmodel (Fig. 2) was developed and feedback sought from DCE managers.

• Reviewed documents also aided the development of an interview topicguide and the creation of a list of DCE stakeholders to be interviewed.

Results

Resources/Inputs Outputs

Funding• Government funding• Sponsorship from charities for specific projects

Evidence base• NAEDI strategies• Available literature

• Behavioural change theories?

Scottish Government• NHSScotland

• Healthcare Improvement Scotland• Regional NHS Boards• Special NHS Boards

• NHS Health Scotland• NHS National Services Scotland

• NSS ISD; Medical payments (sGMS, SQoF); National Services Division

• Scottish Pathology Network• Scottish Clinical Imaging Network

• Health and Social Care Management Boards• Scottish Cancer Taskforce

• DCE Programme Board• DCE Operational Subgroup• CWT Data and Definitions Group• National Screening Comms Group• Scottish Primary Care Cancer Group• Colonoscopy Short Life Working Group

Cancer Charities• Scottish Cancer Coalition• Other Cancer Charities• Volunteers• Community

Scottish Cancer Prevention NetworkMarket Research companyAdvertising agencyMedia AgencyPR AgencyDigital AgencyLarge commercial organisations/key Scottish employers

Health structure, physical and human resources• Primary care services (general practices and community

pharmacies)• Secondary care services (hospitals, hospital pharmacies,

specialist consultations, radiology, pathology and otherdiagnostics)

• Scottish Breast Screening Centres• Scottish Bowel Screening Centre• Staff time (clinical, administrative, laboratory, etc.)

Additional partners/collaborationsProfessionals’ timeProfessionals’ resources

Activities

Small funded, local projects• Young people as influencers• FIT• Referral rates, detection rates and

staging before and after DCE• Other projects

Performance management and monitoring• Cancer Waiting Times (CWT) targets• Heat target• sQoF rewards (bowel screening)• Practice profiles

• Campaigns on mass awareness channels (TV,radio, printed press, outdoors) and bus stops

• Scottish public surveyed; attitudinal tracking reportspre and post campaigns

• National and regional roadshows and events, localhealth promotion activities and pilot studies

• Online activity using DCE (getchecked and weec) and charity websites and social media (Facebook,Twitter and YouTube)

• Celebrity support and use of case studies• DCE information materials given in field marketing

activities, and distributed to general practices andcommunity pharmacies

• Assets (e.g. hideaway books) given to local teams• Targeted strategies for those at risk (lung), men

(bowel), deprived women (breast), screening non-responders and key influencers (e.g. children)

Social marketing campaigns • Insight gathering (public attitudes and

knowledge)• Cancer specific campaigns

• Bowel cancer (screening)• Breast cancer (screening)• Breast cancer (symptomatic)• Lung cancer

• Overarching campaigns• Priming campaign• wee c• #getchecked

Stakeholder engagementPR and partnershipField marketing and partnership

• Reports about engagement and facilitatorprogrammes

• Updated guidelines published, printed anddisseminated online

• Primary care professionals recruited

• Published official reports on CWT, HEAT, sQoF, breast and bowel screening uptake

• PTI data checked to track overall activity andconsultations due to cancer specific symptoms

• Performance monitoring by the government andfollow up when NHS Boards do not meet targets

• Practices sign up to sQoF• Practices develop an action plan to improve bowel

screening uptake• Health Boards are provided with non-responders

lists from the bowel screening centre• DCE provided with bowel screening centre data on

FOBT kits requested and returned• Provision of data on referrals to secondary breast

clinics

Symptom management and referral• Primary Care Engagement Programme

• 7 sessions (bowel and lung cancer)• 4 pharmacy education sessions• Other local training

• Primary Care Facilitator Programme• Bowel screening awareness

training, sharing information aroundcancer resources

• Review of referral guidelines betweenprimary and secondary care

Outcomes

• Findings reported to DCE or published in peer-reviewed journals

Public• Increased awareness and knowledge about

cancer signs/symptoms and benefits ofscreening and early detection

• Public is empowered to check for symptoms and seek help

• Increase in bowel and breast screening uptake

• Increase in the number of patients consulting due to cancer related symptoms

Primary care professionals• Increased knowledge and awareness of

screening programmes• Increased knowledge and awareness of

alarm symptoms and referral guidelines• Increased awareness of the importance of

early investigation• Increased use of information resources on

screening, referral and symptom appraisal

Initial (1-3 years) Intermediate (4-6 years) Longer term (7-10 years)

Assumptions:Role of pathways to treatment in survival• Early presentation in primary care and screening participation lead to early detection and early detection results in improved survival

• “Early diagnosis can improve survival, manage the expected increase in cancer incidence and improve patient and carer experience”• Early referral and early treatment result in improved survivalDCE and the public• Public’s knowledge and awareness (of symptoms and cancer) will increase and fear of cancer will decrease with the social marketing campaigns• Increased knowledge/awareness/reduced fear will lead to symptom appraisal and help-seeking behaviour• Targeted campaigns will successfully reach the most deprived populations, men, and non-responders to screening• The public will engage with DCE during the campaigns• Messages from campaigns will have a sustained effectDCE and primary/secondary care• Practices will get involved in the DCE social marketing campaigns to help with dissemination• Financial incentives will lead to the development of successful strategies to increase bowel screening uptake• Professionals will agree with and follow updated cancer referral guidelines• Primary Care Engagement Programme will increase professionals’ knowledge about cancer and this will translate into better care• Increase in demand will be managed with capacity increase in screening, diagnosis and treatmentDCE overall impact• Cancers diagnosed at Stage 1 are a good proxy of patient survival, despite acknowledged limitations• It will be possible to separate changes in staging due to improvements in data recording from actual increase of cancers diagnosed at Stage 1• Reporting all cancers together is better than reporting cancer types separately, despite recognised limitations• It will be possible to measure the impact of DCE using administrative data, survey data and official statistics

Primary care professionals• More patients with alarm symptoms

receive timely referrals• Increased primary care role in

promoting informed screeningparticipation

Secondary care• More diagnostic/test procedures

carried out

Overall health system• Improvement in overall five-year

cancer survival (no target set)• Referrals are promoted at their

earliest opportunity for patients witha suspicion of cancer

• Increasingly faster access todiagnostics in secondary care (notarget set)

• Automated systems to identify bowel screening non-responders

• Practice profiles are made available• Increase in bowel screening uptake• Increase in breast screening uptake

amongst the most deprived

Context/external factors:Age, deprivation and other individual/biological factors• Older age is associated with higher cancer incidence and the population in Scotland is getting older• Older people and the most deprived are more affected by late diagnosis, treatment delays and survival deficit• Most deprived have higher cancer mortality rates, take part in screening less often and may have poorer access to health services• Multimorbidities (more common with older age and amongst the most deprived) are associated with poor survival• More deprived populations can have problems with literacy and targeted campaigns are needed• Different cultures/ethnicities are associated with different challenges regarding help-seeking behaviour and screening uptake• Men take part in bowel screening less often; more deprived women take part in breast screening less often• Tumour size, growth rate and location have an impact on survivalChallenges in diagnosis• Challenges diagnosing cancer with less distinctive or common symptoms, or cancers which present without alarm symptomsHealth care context• Current scenario in primary, secondary, tertiary care and the NHS in general is of financial strain, with limited resources/capacity and lack of time to

engage in different projects• Increasing role of primary care in cancer control• Increased awareness that 4/10 cancers can be prevented by lifestyle changes – change of individual behaviour a key concern but hard to achieve• Increased importance of genetic testing and personalised cancer care• Quality of care at treatment and immediately after treatment influences survivalOverall context• Scotland has remote areas (rural areas, islands) where access to care and recruiting professionals is more complex• Programme is funded by the government and its continuity is dependent on the government

Detect Cancer Early Programme - Logic Model

Problem: Scotland has poor cancer survival rates compared to other European countriesProgramme’s aim: improve overall five year survival for people in Scotland diagnosed with cancer

Public• More people consult in primary care

when noticing symptoms (no targetset)

• More people make an informed decision to take part in screening programmes

Primary care practices• Local initiatives (e.g. screening pilots, local

audits of screening) implemented toimprove bowel screening uptake

• Practices are rewarded for meeting targets• More practices receive SQoF rewardsRegional Health Boards• More boards meet CWT targets• More boards meet HEAT targetsHealth systems• Increased proportion of cancers

diagnosed at stage 1 (initial target 25%increase)

• Less cancer cases coded with anunknown stage

• More screening tests carried out• Timely production of data and better data

collection and reporting of cancer stages

• Young people acting as influencers(symptom checking and screening)

• Other outcomes

Overall health system• Maintained improved cancer

survival• Improved survival for more

cancer types• Reduced variation in the

number of referrals andpositive cancer diagnosesacross general practices

• Reduction in emergency presentation diagnoses (notarget set)

• Bowel screening rates similarto other screeningprogrammes

• Reduction of socioeconomicgradient of uptake in breastand bowel screening

• Reduction of gender disparity of uptake in bowel screening

• Optimum data quality oncancer staging

Secondary care and diagnostic capacity“Ensure screening capacity; imaging, diagnostics departments and treatment centres are prepared for the increase”

Local initiatives• To improve work practices and reduce DNA rates

for colonoscopy and mammography• To provide a formal 7-day CT service• Surgeons recruited• Creation of new cancer pathways• Acquisition of additional diagnostic equipment• Additional consultant surgeon and consultant

radiology sessionsCreation of new pathway for direct access to CTFunding to roll out FIT screening nationally

Secondary care• Increase in the number of diagnostic tests• Improved diagnostic capacity

Figure 2: DCE logic model

Development of a methodological approach to evaluate the Detect Cancer Early Programme in Scotland

Natalia Calanzani, David Weller, Christine Campbell

©d-

map

s.co

m

60 km

40 mi

Figure 3: Key themes

“I know it was a three-year programme but it is bigger than that, it's trying to promote

one influencer of cancer survival and improve patient experience at the same time, and be efficient in the way that we

use money, and provide improved cancer diagnostics faster.” (ID 7)

• Results are informing the design of the DCE evaluation in terms ofcausal assumptions to explore, implementation issues to investigateand outcomes to assess – these include:→ Whether increase in referrals led to more cancer diagnoses and

additional investment resulted in improved diagnostic capacity.→ Investigating any workload impact of implementing the

programme in primary and secondary care.→ Assessing whether DCE was delivered as planned across

different Health Boards.→ Exploring ways to investigate system-level outcomes in

addition to hard data on survival and its surrogates.→ Assessing whether there was variation in outcomes across

different population groups.• Importantly, challenges collecting, assessing and interpreting data

should not be underestimated.

Discussion

1. Moore G, Audrey S, Barker M, et al. Process evaluation of complex interventions. 2014.https://www.mrc.ac.uk/documents/pdf/mrc-phsrn-process-evaluation-guidance-final/

References

Overarching contextual issues

Development and implementation• A political decision with a political imperative• Aims, drivers and influencers• An evidence-based initiative• Resource allocation• Stakeholder involvement• Voiced concerns and barriers• Developing relationships, systems and roles• Delays and adaptations

Views on DCE and its components• A driver of change and other aspirations• Belief in DCE and feedback on its activities• Perceived impact of activities• Sustainability concerns• The role of targets

Logic model• Comprehensiveness and

usefulness• Initiatives: scope, importance,

links and trajectory• Proposed outcomes• Underlying assumptions

Outcome evaluation challenges• Accessing and understanding data• Bias and confounding• Results take time• Unmeasurable, poorly defined or outcomes

with different definitions

What to evaluate• Early diagnosis, mortality, survival or

surrogates• Impact on primary/secondary care• Improvement in awareness and knowledge• Seeing the bigger picture• Unanticipated outcomes

• Evidence and international comparisons• Scottish context and pioneering• NHS constraints and regional variation

• Cultural change or shifts• Tackling cancer• Role of the media

Implementation and mechanisms of impact

Outcomes

Description of the intervention and its causal assumptions

Context

Themes derived from interviews and organised according to the MRC framework of process evaluation of complex interventions1

• Deprivation, age and gender• Individual behaviour and change• Government characteristics

“Primary care's quite defined, marketing's quite defined, then we

come to secondary care and it's like what did you do? Well I suppose we invested in that and we gave

boards money and then kinda left them to get on with it.” (ID 2)

“There was a distinct anxiety that you could just increase activity to a rate that would effectively just overwhelm the capacity in some of the clinical services.” (ID 4)

“I think the main challenge in Scotland is around equity and there's no doubt, you know, Scotland's got a very sort of wide socioeconomic gap

across its society.” (ID 9)

This study is funded by the Scottish Government and supported by scholarships from the University of Edinburgh as part of NC’s PhD studies.