deb james (ncvo), measuring what matters
TRANSCRIPT
“measuring what matters”
Deb James, Early Years Evaluation Team
• Share some of what we learnt from our evaluation – from areas
with plans in place
• Introduce an approach to “measuring what matters” and some
materials (all optional!) that you might find helpful
• Do a group exercise looking at how you could start to measure
the difference you’re making and tell a stronger story about your
successes
In today’s workshop we will …
Using information to make a judgement about how you’re doing;
assessing the value or worth of what you’re doing.
What IS evaluation?
• Accountability
• Learning and improvement
• Motivation and engagement
Why it isn’t always a priority …
“It comes at the end”
“It’s done to you by an
expert”
“It’s paperwork … actually
doing stuff is more important
than form filling”
“Not sure where to start”
… and why it matters?
What did we find in our
own recent evaluation?
Ten messages from our work with
areas that have plans in place.
Let’s see if they ring true for you?
1. Plans vary but many areas share similar goals and priorities –
People-Place-Community. Local context is key to understanding
what areas plan to do – how you tackle isolation in one area can be
very different to how you might do it in another.
2. Some great stories about changes happening – some small, some
very significant – areas themselves are best placed to say what’s
small and what’s significant. The distance travelled matters.
3. Partnerships worry about things being slower than anticipated
(though there is no national pressure on deadlines). Causes include
sometimes having over-ambitious goals or being a bit let down by
partners.
Learning about plans and progress
and reviewing …
4. For many areas in the first year delivering on their plan, “behind the
scenes” work had taken more energy and time than anticipated and
a lot of achievements were in setting up/maintaining partnerships
rather than “out front” (making changes in the community).
5. Following review activities, a common realisation has been that the
Big Local journey is up and down, even round and round in circles,
rather than in a straight line of growth and improvement – and that’s
OK.
6. Some anxiety about evaluation … It’s too soon? Where do we
start? Don’t we need special skills and knowledge? We need to
focus on doing the work not measuring the results?
…
7. Some things seem too hard to measure – eg, ambitious or long-
term changes, or “fluffy” changes.
8. Areas are realising that not everything works – it isn’t always
comfortable, but it’s helpful to acknowledge this and Local Trust is
not expecting everything to work or run smoothly all the time.
9. A pause to reflect and review was helpful … it can be hard to find
the time but worth it
10. Because things sometimes feel slow, there’s a danger that
reviewing can leave areas feeling demoralised. It’s good to think
more broadly about success and to value more than just how
many things have been ticked off a ‘to do’ list.
…
*
PLAN - Planning and identifying what matters most
- what you need to assess or measure and how you
might do it
DO - Collecting the information you need and
analysing it – thinking about what it means
USE - Using your findings – to make judgements,
to learn/make changes/improve, to report to others
What does evaluation involve?
1. PLAN
2. DO
3. USE
A common view is that evaluation is simply about collecting
information (eg, doing a survey) and then writing a report about what
you find. In fact it involves three main stages:
Stage 1:
Evaluation planning
Five tips to help you plan how you’ll assess your
progress – help you be clearer about what you want to
measure and how you might measure it.
1. PLAN
2. DO
3. USE
1. Define success
If you don’t have a clear, shared
view of what success looks like,
how will you know you’ve
achieved it?
1. Define success
Example – an area’s success criteria
Our success criteria …
When considering how successful we have been, we’ll see if we’ve done what we
said we would, we’ll see if we’ve made a difference, and we will also consider how
true we have been to these values:
• we will have worked in an inclusive and respectful way
• we will have kept talking and listening (ongoing community engagement)
• we will have been bold and imaginative, not afraid to try new things
• we will have linked projects and activities together where it makes good sense
• we will have done the best we can
• we will leave a legacy behind
*f
Residents will report an increased confidence in accessing job
readiness and employment support services and a more positive
outlook on improving their financial situation through work, having
been assisted through a new local high-street service that is friendly,
informal and community-led
What would you measure? It helps to get to the heart of the
change, and to express changes as single, simple statements.
2. Describe clear changes
Members of the community
will help us run the job club,
sharing their time and skills
with others
Out-of-work residents who
use the club will feel more
confident about seeking
support to find
training/employment
Out-of-work residents who
use the club will feel more
positive about their chances
of finding work.
We will set up an informal,
friendly job club in our Big
Local Hub to support long-
term unemployed residents
Activity Activity Change Change
We will offer youth nights at the community
centre
Young people will be more employable
We will offer group work
sessions during youth nights at the community
centre
Young people will improve
their social and communication
skills
Young people will be more employable
3. Have stories that work
Would a story map help?
The things we do or
fund to make a
difference (activities)
The differences we want
to make
(changes/results)
End result
1. Activities that preserve or
improve the appearance
of our green spaces and
open spaces
2. Activities that promote
community access/
enjoyment of our open
spaces e.g. growing
projects, allotments
Our open spaces are (and
look) better cared for
More people are involved in
activities that protect or
improve our local environment
Our open spaces are more
accessible and welcoming
More use of our parks and
open spaces (that is, they are
better used by all parts of the
community)
An
improved
environment
that is cared
for and
enjoyed by
our
community
Pause
Suggestions so far. You might want to take time out to …
(1) consider what success means to your partnership. What else, what
other criteria - besides what you’ve done and the difference you make -
would you like to assess yourself by …
(2) make sure you are clear about the changes you want to make and
how you think you’re going to make them, is it clear what the changes
are and who they are for …
(3) check you are clear about how what you do will help bring about the
difference you want to make. Check your ‘story’ works … is it worth
trying to map/summarise your ‘change stories’? This can have a
number of benefits – making things seem more measurable, helping
you focus on what kinds of activities to try/fund, helping you tell your
story.
4. Be specific
(break ‘big’ changes down)
• Breaking big changes down and being more specific about them
can make them easier to measure – that is, spend some time just
clarifying what you mean by ‘big’ changes like “community
cohesion” or “social inclusion”, “quality of life”, or “improved
environment”
• Some changes are long-term and lots of other factors come into
play … try to focus on what’s realistically in your control
Example – break down big changes
Our big change
Our specific changes
(what an improved local environment
means to us)
Improved local
environment for
the whole
community
Cleaner open spaces
Safer open spaces
Our open spaces are better used by local
people
The community take better care of the space
What does the change look
like?
How will we know? What
signs or indicators would be
good evidence?
(things you can count,
things people tell you, things
you can see are all valid)
5. Ask ‘how will we know’ questions
Improved local environment for the whole community
Our specific changes
(what an improved local environment
means to us)
What we will count, measure or
observe
(signs of the change)
Cleaner open spaces Level of littering
Level of graffiti
Level of complaints
Level of use of dog litter bins
How clean people feel the space is
Safer open spaces ?
Spaces better used by local people ?
The community take better care of the
space ?
*
Exercise - how will you know? (what evidence?)
*
Our specific changes
(what an improved local
environment means to us)
What we will count, measure or observe
(signs of the change)
Cleaner open spaces • Level of littering
• Level of graffiti
• Level of complaints
• Level of use of dog litter bins
• How clean people feel space is
Safer open spaces • Level of reported crime
• Number of accidents in play area
• People’s perceptions of crime
• How safe people feel spaces are
More visited/used by local
people
• Level of use by families
• Proportions/profile of those using (eg, age, families)
• Level of bookings for community events and attendance
at these
The community take better
care of the space
• Number of volunteers helping with litter picks and events
• Level of littering/graffiti (see above)
Our specific
changes
(what an improved
local environment
means to us)
What we will count, measure
or observe
(signs of the change)
How we will
count it
(methods)
Cleaner open spaces • Level of littering
• Level of graffiti
• Level of complaints
• Level of use of dog litter bins
• How clean people feel space is
Annual litter pick
Council records
Survey
Safer open spaces • Level of reported crime
• Number of accidents in play area
• People’s perceptions of crime
• How safe people feel spaces are
Police records
Council records
Survey
More visited/used by
local people
• Level of use by families
• Level of bookings for community
events and attendance at these
Question in schools
parent survey
Council records
The community take
better care of the space
• Number of volunteers helping with
litter picks and events
• Level of littering/graffiti (see above)
Monitoring records of
volunteers
(as above)
The last part of the plan (collecting evidence)
How will you know exercise
Choose a change that you’d like to measure progress with –
eg, older people will have a better quality of life, young people
will be reaching their full potential, or one of your “fuzzy” ones!
(1) How will you know you’re making a difference? In
your group come up with three or more signs you could look
for or measure to help you know if the change is happening.
(2) What methods could you use to collect evidence?
(3) Who could help and how?
Stage 2:
collecting and analysing
information
1. PLAN
2. DO
3. USE
Some of the options open to you and some
good practice tips
Depending on your questions and the type of information you need (eg, numbers
or stories), you could use:
1. Survey questionnaires and feedback forms (your own)
2. Standardised questionnaires or scales (others)
3. Available data (info that others have already collected)
4. Interviews or focus groups (what people think, how things are perceived)
5. Participatory, creative and arts methods (great for young people)
6. Case studies (often based on interviews)
7. Documents and records - your own (minutes, event reports) or others’
(news articles, correspondence)
8. Visual records (photographs, video)
Other? Is anyone using any other methods? (Check out the Local Trust
“checking your progress toolkit” for ideas and tips on what to use and when.)
Options for collecting information
1. Prioritise – you can’t measure everything or collect
information from everyone. What steps or changes matter
most to telling your story?
2. Be realistic – and proportionate – what you do to measure
your success shouldn’t get in the way of actually doing stuff.
3. Be lawyers not scientists? - think about ‘the best
possible evidence’ not ‘scientific proof’
4. Be a recycler! – what else is already being
measured/counted, what info is already out there that you
could use for your purposes?
5. Be flexible and open - this kind of work isn’t ‘neat’ and we
are all trying something new, taking risks and trying to learn
more about what works. Be open to using different
approaches and methods – counting numbers and capturing
stories are both valid.
Top tips for collecting information
6. Consider different perspectives – it can help to
include different groups and different viewpoints
7. Try to capture learning as well as success – try
and create the kind of environment where you can learn
from what doesn’t work as well as what does!
8. Think about how you’ll use information BEFORE
you collect it
9. Don’t collect information you won’t really use
10. ALWAYS do a test run first of any questionnaire or form
you use
…
Stage 3:
Using your findings
Reflect. Learn. Improve. Share. Celebrate.
1. PLAN
2. DO
3. USE
• Remember that “the purpose of evaluation is to improve, not
prove.” (J Stufflebeam)
• There are lots of good reasons for reviewing regularly,
building in space/time to review as you go along, not least
because it helps you see if something’s not working and
means you can change direction more quickly if you need to.
• Your report to Local Trust will ask you to reflect on your
learning … should be a group activity, try to avoid one person
writing on behalf of the group as you can lose useful insights
and assumptions can be made.
• Aim for honesty! “All my successes have been built on my
failures.” (Benjamin Disraeli)
• Ask the right questions … including what worked and what
didn’t and what did we learn?
Reflect, learn, improve
Asking the right questions is key
• The achievement we’re most
proud of? What’s gone well?
• What do we feel more
confident about? What have
we got better at doing?
• What difference have we
made or contributed to in our
area? How do we know
we’ve made this difference?
• What’s not gone well? What
are the main lessons we’ve
learnt from things that
haven’t gone so well? What
could we do better?
Don’t stop with your report to Local Trust – consider using:
• Meetings, events or community forums
• Websites, facebook timelines, blogs or twitter
• Newsletter articles and case studies
• Photo galleries and posters
Here’s some examples of sharing and celebrating that we liked
using visuals, facebook, posters, and using story telling and
posting impact stories online …
Share and celebrate