social exclusion task force ‘what works?’: guidance on evidence informed commissioning april 7...
TRANSCRIPT
Social Exclusion Task Force
‘What works?’: Guidance on evidence informed commissioning
April 7th 2008
Social Exclusion Task Force
Based in the Cabinet Office
Ministers Successor to the Social Exclusion Unit
Working across government departments
Champion the voice of vulnerable groups
Evidence-based policy
Mission of the SETF:To extend the opportunities enjoyed by the vast majority of people in the UK today to those whose lives have been characterised by deprivation and exclusion
Five principles for tackling social exclusion
1. Better identification and early intervention
2. Identifying what works
3. Multi-agency working
4. Personalisation, rights and responsibilities
5. Supporting achievement and managing underperformance
Current work programme
Adults New Public Service Agreement on socially excluded adults Adults facing Chronic Exclusion (ACE) pilots
Children and families Families at Risk Review and Pathfinders Family Nurse Partnership (FNP) pilots Multi-Systemic Therapy (MST) pilots
Research New Analysis of social exclusion over the life course Guidance on ‘what works’ – evidence-informed
commissioning
‘What works?’: Guidance on evidence informed commissioning –
A tool for commissioners and providers of services for excluded groups
What is the guidance on ‘what works?’
A tool which sets out guiding principles in the use of evidence to inform and monitor services/programmes for excluded groups, by drawing upon evidence to:
Help assess whether a programme is likely to be promising;
Set objectives to evaluate a programme or service;
Review progress along the way.
Designed to support and sit alongside current commissioning frameworks.
Target audience – local authority commissioners and service providers focusing on delivery for vulnerable groups.
Aim and value of the guidance
Strengthen the capacity of commissioners and providers of services to evaluate ‘what works’ by looking at the evidence.
Improve the outcomes for vulnerable individuals by ensuring services designed to support them are based on research evidence.
Build up an evidence base on what works locally and nationally.
Ensure a desired direction of travel e.g. monitoring.
User-friendly document.
Socially Excluded Adults PSA and FARR.
What the guidance will coverSection Key questions
1. Introduction What is the guidance and who is they for?
2. Using research evidence in service planning
What are the benefits of using research? How do we use research-evidence in service planning? How do we ask a research question?
3. Locating relevant research evidence
How can we find the evidence we need? Reviews - primary and secondary research; What databases can we use?
4. Assessing and appraising research evidence
How can we know whether evidence is reliable? What is qualitative and quantitative research? How do we know what is ‘good’ research? Includes a grading tool and checklist on appraising research
5. Outcome focused evaluation
How do we know whether we are achieving service outcomes? How do we plan for an evaluation? Reviews SMART outcomes – Specific, Measurable, Achievable and Realistic and Time-limited;
6. Appendices – web based What does the jargon mean?
Checklist for appraising research evidence
How can we ensure our work is ethical
What do the numbers mean
Case studies
GRADE
MINIMUM SOURCES OF EVIDENCE
TYPE OF EVIDENCE DESCRIPTION STRENGTHS WEAKNESSES COMMENTS IMPACT ON SERVICE USERS
1 Positive reports from service users and practitioners at follow up
Personal testimonyPractice experience
Account by a person intimately acquainted with or affected by a particular issueAccumulated wisdom from repeated exposure to similar situations or problems
Powerful and immediate; may give vivid insights into events concealed from much of the populationEnables practitioners to identify anomalous situations where counter-intuitive decisions can and should be made
Difficult to verify; may lead to inflation of prevalence; accounts of distressing personal trauma may inhibit critical appraisalErrors can be consistently repeated unless tested by other sources of evidence; hard for lay people to challenge
Sufficient basis for pilot intervention where plausible link between activities and outcomes exists
Unknown effectiveness
2 Several positive pre- post studies comparing performance at baseline to follow up
Client opinion studySingle case design
Narrative or survey accounts of user views or reported needsRepeated standardised measurement of a client’s situation or problems over time
Valuable insights from those at the receiving end; compels professionals to stay focused on the client’s prioritiesEasy and practical; can be used by practitioners with minimal training; clients are able to collaborate and contribute
Correlation between satisfaction and actual outcomes is low; needs described may not translate into actual service useMore difficult to apply with non-behavioural interventions; absence of controls weakens attribution of cause and effect
Sufficient basis for pilot intervention at multiple sites where plausible link between activities and outcomes exists
Limited evidence of effectiveness
3 Positive evaluations by several studies featuring comparison groups
Quasi-experimental studyCross sectional studyCohort study
No random allocation to groups; use of natural populations.
Powerful method of exploring the impact of an intervention when randomisation is impossible; can also be applied to communitiesSurvey can be repeated at intervals, illustrating changing trends over timeBest source of evidence on association between childhood experience and adult outcomes.
Difficulty in ensuring equivalence of groups and natural changes in group composition over time can result in less reliable findingHard to detect why changes have occurred; difficult to observe trends in minority populationsData often emerge too late for effective policy making; may be drop out over time.
Basis for investment where need is urgent and more robust evaluation results are not immediately available
May be beneficial
4 Positive evaluations by several randomised controlled trials
Randomised controlled trial (RCT)
One group receives an intervention, another receives none or an intervention of another type; the chance of being allocated to either group is identical
Changes in the client’s situation can be attributed to the intervention with a high degree of confidence; best approach for controlling bias; transparent methodology means results can be independently checked and replicated
Mistakes in randomisation can invalidate results; cannot be applied to many important social problems; without equipoise – an equal chance that benefits will occur in both groups – ethical difficulties may arise; high level of statistical skill may be required for analysis.
Highly desirable for large investment in single service site
Likely to be beneficial
5 Intervention positively evaluated by at least one systematic review or meta-analysis
Systematic reviewMeta-analysis
Aggregation of results from eligible studies; eligibility criteria defined in advance (usually but not necessarily RCTs); review methodology is replicable
Best source of reassurance that an intervention works (or doesn’t); meta-analysis pools numerical results; large reviews carry great statistical power; pooling of results gives more accurate indication of actual effect size by moderating outliers
Requires a substantial number of robust primary studies in a given area; methodology less well developed for synthesising qualitative data; strict inclusion criteria may results in few or no eligible studies; reviews are technically demanding
Highly desirable for large investment in multiple service sites
Highly likely to be beneficial
Key players
Barnardo’s
Research in Practice
National Foundation for Educational Research
SETF
Local authorities
Third sector
Academics
Government departments
Dissemination and publication in Spring 2008
5
Workshops – key findings Method/Consultation 4 workshops; variety of LAs across England; linked into LARC.
Findings Commissioning appears to be very varied across LAs - despite numerous
frameworks;
There is a need for a culture change in the use of research evidence;
There needs to be greater innovativeness in the selection of programmes;
There needs to be greater transparency in the selection of programmes;
Guidance needs to be accessible, user-friendly, not burdensome!
Guidance should be a tool which helps not only in commissioning but decisions to decommission.
Questions for discussion
For commissioners/ serviceproviders What are your initial
thoughts on the proposed contents of the guidance?
Would this be helpful for you – how would you use it? Anything missing / to be added?
Does this resonate with any guidance or support materials you already use?
For researchers, policymakers and academics What are your initial
thoughts on the proposed contents of the guidance?
Is there anything missing which needs to be considered?
How can we bridge the gap between ‘what works’ and practice?
Next Steps Peer Review
Academics Local authority commissioners Third sector Government departments Specialist organisations
Commissioning courses
Home?
Publication (Spring 08)
www.cabinetoffice.gov.uk/social_exclusion_task_force