headstart kent knowledge seminar 2 - kelsi · time agenda who 9.30 – 9.40 introductions (around...
TRANSCRIPT
HeadStart Kent Knowledge Seminar 2
21st January 2015
Measuring Outcomes
Time Agenda Who
9.30 – 9.40 Introductions (around the table and presenters)
Florence/Angela
9.40 – 9.45 Recap of Learning from Seminar 1
Key Messages: Summary
Broader Strategic Issues For Headstart Project
Alex Hassett
9.45 – 9.55 Feedback on the Board and Shadow Youth Board
Angela Ford
9.55 – 10.05 Broader Evaluation Programme
Ugochi Nwulu
10.05 – 10.30 Activity: Challenges faced measuring resilience outcomes. Individual and
group exercise – feedback to the wider group
Alex Hassett
10.30 – 11.00 • Challenges and practical issues in measuring resilience
• Measures of resilience – some ideas
• A domains approach to measuring resilience
Mark Kerr
11.00 – 11.20 Coffee Break
11.20 – 12.00 Activity: Mapping where your service fits and what you measure Mark Kerr and Alex
Hassett
12.00 – 12.15 Feedback on learning
Alex Hassett
12.15 – 12.30 Way Forward
Florence / Angela
Resilience is not a trait but an interaction between risk and protective factors
We need to ensure we take an ecological and developmental view of resilience
It is useful to focus on resilience in terms of the areas or ‘domains’ of a person’s life that can be changed.
Negotiation and navigation
We need to consider what resources are available and how accessible are they
Key Messages: Summary
The following are the long term issues that need to be addressed:
Developing an overarching framework of resilience that the range of agencies can sign up to
Encouraging a long term interdependency between individuals, services, agencies on providing an holistic approach to young people
Providing a coherent system for evidence based evaluation ensuring that each element of the system is clear on how they evidence outcomes and impact
Broader Strategic Issues for Headstart Project
Reminder of Kent Activity C
ante
rbu
ry Penn State Resilience in
secondary schools
Penn State Resilience in primary schools.
Penn State Resilience in community and target workers
Family approach TBC
No
rth
We
st K
en
t Safe Spaces in schools
Safe Spaces in community hubs
Coping packs
Family Focus KS2 ACP
Peer mentors
Active listening mentors
Youth MH First Aid
Online counselling
Than
et Restorative approaches
in schools
Restorative approaches in the community
Target Restorative approaches in schools
Restorative Ambassadors
Restorative approaches families
Resilience Mentors: evidence based model of intensive support. FRIENDS
Digital World: full services directory , volunteering & mentoring opportunities, self -referral form and sign posting to social media
Social Marketing: skills roadshows, coping packs, phubbing.
Pa r t n e rs h i p P ro g ra m m e B o a rd , S h ad ow B o a rd , Kn ow l e d ge S e m i n ars
Coproduction throughout Young people and Families
Feedback from Young People
Young people found the domains resilience approach useful Identified areas of HeadStart Kent they felt would have most
impact Resilience mentors Coproduction social marketing Family resilience Safe Spaces Peer support Online directory
Ensure Kent’s Emotional Wellbeing Strategy is central to developments.
Contributing to service redesign
Connecting to the system and enabling change
Wider stakeholders already mapping and exploring system redesign
Outcomes
Early Help: improved emotional resilience and receive early support
Access: Receive timely, assessing and effective support
Whole Family: Recognises and strengthens and wider family relationships.
Recovery and Transition: Prepared for and experience positive transitions
Kent’s Emotional Wellbeing Strategy for Children, Young People and Young
Adults
Every one needs to consider how they contribute to building resilience, and what they could do enhance it further.
If our outcomes frameworks are to be guided by the domains (risk and protective factors), we need strategic cohesion across Kent including:
Workforce being prepared to work systemically.
Shared language
Less duplication
Easy moving and less transitions.
Workshop
There is a greater awareness of activity locally and countywide and a lot has happened over a short period of time
People fed back that the knowledge seminars have been useful and thought provoking.
Some of the challenges include: How to build coherence and ecological links when more than one
intervention is working in the same area.
How to involve more young people of greater diversity
How to get passion and buy in from professionals
How to increase the understanding and scale of social marketing
Workshop Messages
HeadStart Kent Knowledge Seminar
21 January 2015
For HeadStart information http://kelsi.org.uk/pupil_support_and_wellbeing/targeted_support/inclusion/kiass/headstart.aspx HeadStart Kent Twitter is: @HeadStartKent #headstartmatters
HeadStart Kent Knowledge Seminar
21st January 2015
Evaluation progress
Ugochi Nwulu [email protected]
Ugochi Nwulu - KCC / University of Kent
Rob Comber - Education and YP’s Services, KCC
(Eileen McKibbin - Research and Evaluation, KCC)
Gabriela Sette - CHSS, University of Kent
(Prof Patricia Wilson - CHSS, University of Kent)
Evaluation team
Key evaluation questions:
1. What are the HeadStart interventions?
2. What is the theory of change across the programme?
3. How does each intervention contribute to the theory of change?
4. What is working well and not so well in the implementation and the delivery of HeadStart Kent?
5. What are the critical and effective elements of the programme which now need to be scaled up fora Kent wide approach to building emotional health and resilience
Process evaluation of HeadStart Kent
Kent area Intervention 1 Intervention 2 Intervention 3
Data sources:
HeadStart operational team / Community
practitioners / School staff / Resilience mentors
Young people
Evaluation methods
Case study
Focus group Questionnaires
Learning from participants and stakeholders
How we will measure the impact:
HeadStart Schools data
- demographics, risk factor profiles
- pupil absences, exclusions
- numbers accessing targeted support
- CYP who participate in the HeadStart programme
Baseline data collection
January to March:
National HeadStart conference
Synergies with the national evaluation - field work, surveys
Informal interviews and refinement of plans
March to May:
Focused data collection period
Data analysis and write up
Evaluation report
Will include plans for an impact evaluation of fully scaled up projects
Next steps
Take a few minutes to think about the challenges you face in measuring resilience outcomes in the work that you do or the work that you commission. Please note down your challenges and concerns
Now spend a few minutes discussing with your group what those challenges are. Please can each group decide on the 2 main challenges or concerns they face when considering measuring resilience outcomes?
Exercise One
Example outcome model
Educational Development
Results In
Increased Protective Factors
Decreased Risk
Factors
Producing
Personal Development
SocialDevelopment
Intrinsic Outcomes (individual well-being)
Extrinsic Outcomes (wider social good)
Source: Young Foundation, 2012
Can we measure resilience
The ‘what’ are we measuring – what is our Dependent Variable (DV)
Whether to use a global resilience scale or domain based measure
Self report versus third party rating
Age appropriateness of measure
Validity of measure
Requirements of analysis
Challenges in measuring
‘The development of a measurement instrument capable of assessing a range of protective mechanisms within multiple domains provides an approach to operationalising resilience as a dynamic process of adaptation to adversity (Olsson et al., 2003)
Ideally, measures of resilience should be able to reflect the complexity of the concept and the temporal dimension. Adapting to change is a dynamic process (Donoghue and Sturtevant, 2007)
Can we measure resilience?
Assessments of resilience need to consider:
I. a) the risk or adversity
II. b) assets/resources that might offset the effect of the risk
III.c) the outcome
Quantitative direct measurement – using a resilience measurement scale as an outcome measure
Quantitative indirect measurement – modelling a range of data with multivariate statistics
Qualitative – understanding individual experiences
Measuring Resilience
Several scales developed but not widely adopted and no clear preferred option
Definitional clarity needed which influences how we tackle this
Virtually no valid measures or children
Would need to measure availability of resources at all ecological levels to understand those that demonstrate
Only potential measure: California Healthy Kids Survey – The Resilience Scale of the Student Survey (Sun and Stuart, 2007)
Measuring Resilience cont’
Remember ‘resilience’ is the ability to overcome adversity due to the interaction of risk and protective factors
Cannot assess resilience until the child experiences adversity
Current Headstart project requires us to focus on the antecedents of resilience i.e. protective factors
The ‘DV’ problem
Headstart programme aims more suitable for a domain approach
The time limited (currently) nature of your work means for many children (hopefully) resilience will not be experienced due to the absence of significant adversity
The number of partner organisations, some with specific domains of focus, mean individually you are unlikely to provide all protective factors needed. However collectively you will
Resilience or Protective Factors?
Key domains What are you working on?
Secure Base
Education
Friendships
Talents and interests
Sociable Behaviour
Positive Values
Resilience Domains (Daniel & Wassell)
One of many decisions that must be taken alongside considerations such as age, ability, domain of interest and ecological level
Can be a combination – both self and others e.g. teacher, parent, carer or other professional and results combined
Research has indicated problems if incongruity between child, parent, and teacher reports
Self Report or Third Party Rating
Variable focused Link among measures of degree of risk/or adversity,
outcome, potential quality of individual or environment (to compensate/protect)
Person focused Compare people with different profiles (within or across
time) on sets of criteria to ascertain what differentiates resilient from non resilient children
Models / Approach to Resilience
Source: Masten, (2001)
Variable focused studies
Compensatory effects
Enough positive assets could offset the burden in child’s life from one or many risk influences
Three models
Main effect
Indirect
Interaction
Variable focused studies
Main effect models
Asset
Bipolar predictor
Risk
Outcome
-
+
- +
Variable focused studies
Indirect models (example)
Asset
Effective parenting
Risk
Outcome
-
+
+
Variable focused studies
Interaction models
Moderator
Risk activated moderator
Risk
Outcome
– Not found very often (difficult to detect)
Pros and Cons
Variable focused + Max. stats power, suitable for searching specific links
between predictor and outcome - Fail to capture striking patterns in lives of real people, risk
of losing sense of the whole
Person focused + variables assembled in naturally occurring configuration,
well suited for search for common and uncommon patterns in lives
- Can obscure specific linkages
Resilience-Based Practice
Three Main Principles: Inclusive, respectful and engaged
practice
Strengths-based practice
Solution focused approaches
Also, Fostering community and social
connectedness
Attachment theory and Circle of Security
Differences in ecological emphases: Australia vs. UK
Community
Family
Individual
Resilience in Practice - UK
Intervention Themes:
Improvement of self-esteem / to like self more
Improvement of peer relationships
Improvement in school experience / behaviour
Control of anger / managing disagreements
Naming feelings / emotional literacy
Target
Anger, aggression
Strategies
Anger control, emotional intelligence
Intended Outcomes
Increased self-esteem, improved peer relationships and school experience
Resilience in Practice- Australia Intervention Themes Subtypes Occurrence
1. Health a) Physical/Medical 11%
b) Mental/Behavioural 4%
2. Emotions & Attachment a) Parent-child dyads 14%
b) Broader family r/ships 3%
3. Parenting Skills & Confidence a) Expectations/boundaries 22%
b) Support: Peer/Playgroups 7%
4. Legal Issues 5%
5. Employment & Education or Training
8%
6. Finances & Housing 5%
7. External Supports for CHILDREN a) From the School 7%
b) From the Community 4%
8. Reduce Social Isolation 10%
Australian Practice… Chains of Support
NSWb1:
Address mother’s social isolation link mother with community supportive playgroup mother-child bonding and attachment is facilitated new social networks and connections with the community are created
NSWa1:
Address uncontrolled behaviour, aggression in children/poor attachment evident assist father in putting strong boundaries, routines and expectations in place at home children seen as having greatly improved emotional regulation, able to cope in new spaces or with new people father more competent and relaxed
Mapping interventions to Literature
1. Health
2. Emotions & attachment
3. Parenting confidence and skills
4. Legal issues
5.Employment/E&T/edn/training
6. Finances/Housing
7. External supports for children
8. Reduce Social Isolation
Res. Domains
Secure Base
Education
Social Competencies
Literature
Build Caring relationships
Establish and maintain Self-Efficacy
Mobilizing protective resources
Creating Opportunities
Foster Resilience Strings
Ecological
Resilience-based Practice – Implications…
Practice should aim to target all ecological levels to align to a comprehensive view of resilience
Policy: Ecological supports, especially community level investment
Building a common language around resilience to promote strategic change as disciplines and agencies endeavour to work in concert
Flexibility for assessment and practice frameworks – creativity and individualisation
Resilience-based Practice – Implications… cont’
Comparative research: Underlying processes vs. behaviours; impacts of various ecological levels
Impact and unique predictive value of attachment as central to intervention efforts
Family definitions of adversity and their priorities for help
Evaluate, evaluate, evaluate! Assessment of outcomes to see if resilience-based interventions work!!
Essentially, resilience-based practice needs to aim for consistency
in scope and application + flexibility
Outcome evaluation as critical for determining best practice
‘Absence of a conceptually sound and psychometrically robust measure of resilience for children under 12’ (Windle et al., 2011)
Reading ability
Problems with cognitive processing of Likert Scales
Distinguishing between how they feel now, and how they typically feel – ‘situational effect’
Developmental age versus chronological age
Issues with Age (Self Report)
Measuring Resilience - Outcomes
OUTCOMES:
Measured through informal channels and processes:
Feedback from parents and other agencies, observations, anecdotal evidence, children’s art
Positive (presence) and negative (absence) indicators of resilience:
Improved social skills, decreased anxiety, increased emotional regulation, better school performance, engagement in community activities
Measuring Resilience – Implications…
Consistency in outcome measurement to aid evaluation
Policy: Focused drive to incorporate sound outcome measurement
Research to examine breadth of assessment and measurement tools – recommend utility and relevance in the context of actual outcomes for clients
Again, the emphasis lies on consistency of usage: Assessment and outcome measurement procedures need to align to a resilience-based framework if this is the approach being explicitly espoused by the organization
To be a valid measure it must have been subject to a number of validation checks including: Content validity
Internal consistency
Criterion validity
Construct validity
Reproducibility
And others…
Should also go through peer review
So always try and use and existing measure!
Validity of Measure
Selection of Measures
What are you going to do with the data?
What questions do you want the data to answer?
What type of analysis do you need to do?
Do you want to be able to generalise your findings?
The type of data you collect influences the analysis you can do
Considering Analysis
Within group or between groups?
The need for ‘controls’
Score cut off points – defining expected levels, what does success look like?
Sample size and the impact on power and effect
Technical Considerations
Who are the focus of your activities?
What is their adversity?
Why do you think creative activities might build resilience? (your ‘theory of change’)
How might they benefit from a resilience promoting activity?
How will you know they have benefited?
How do you currently demonstrate this?
Your Work
Creative Activities
New skills/learning
Social connections/networks
Efficacy
Cognitive Health
You have 40 minutes for this exercise. As a group we would like you to think about each of your services or interventions that you or your service provides or you commission. Using the domains framework, and the chart provided, think about which domains you have an impact on and how you measure or could measure outcomes within that domain.
Once each group member has mapped their service and interventions look at the gaps and think about how they could be filled.
Spend the last 5 minutes of the exercise discussing what you feel your key learning has been from this exercise.
Activity