rmbc cmf theory of change web version · title: microsoft powerpoint -...

Rotherham Building Stronger Communities

Draft ‘Theory of Change’ for the evaluation Steve Connelly

University of Sheffield1st February 2018

Introduction

This slide pack sets out a draft ‘theory of change’ for the CMF-funded projects within RBSC. It is intended to • be the basis for a structured approach to the

evaluation of the programme• prompt discussion about what can and should be

evaluated, and how• lead to better understanding of how the projects might

work, where there might be synergies and overlaps and potential conflicts, and identify previously unpredicted outcomes (good and bad)

• identify areas of lack of clarity for further discussion and development

Structure

• key points and outstanding questions and issues

• logic models (LM), theories of change (ToC), and the context/mechanism/outcome approach

• BSC’s 9 types of mechanism, and relationship to projects and outcomes

• an example of a ToC for one mechanism• the full ToC• an example of individual project logic models

key points• it’s very complex – even ‘simplified’ down to 9 linear ways

of having effects (‘mechanisms’)• most mechanisms have multiple outcomes, so we have an

‘attribution problem’ – which projects caused what?• the project deliverables are focused very much on

activities, less on the processes by which they will have effects (capacity change (8), behaviour change (10), direct benefits (7))

• of the 16 deliverables related to wider benefits, only 5 are attributed to projects

• the (understandable) project focus on ‘doing stuff’ with communities leaves out tackling contextual factors in both communities and state/3rd sector bodies: how do we ensure support for enduring change both ‘culturally’ and in material terms?

outstanding questions/issues

• how can the complexity of the project be presented in ways which enable informed partner input?

• we need more measures of if/how activities are having effects, and how these lead to benefits

• what (if any) evidence do we currently have to support the causal links and assumptions?

• what differences might there be between groups in terms of how relevant/accurate the assumptions are? and so of how effective the intervention is?

• what risks are associated with the assumptions? what measures can be taken to mitigate them?

• how do we examine each activity for its sustainability?

Logic models

logic models set out the logic of an intervention: how it is supposed to have effects

Adapted from Mayne, J (2015) ‘Useful Theory of Change Models’, Canadian Journal of Program Evaluation, 30 (2) 119-142

Activities/ outputs

Capacity change

Behaviour change

Direct benefits

Wider benefits

Logic models + assumptions become theories of change

• a full ‘theory of change’ (ToC) includes the ways in which the logic model stages are assumed to connect.

• these assumptions also point to conditions which have to be present if the LM is to be made real (i.e. if the intervention is to work) and suggest alternative outcomes

Assumptions about reach, how capacity is changed,

and conditions required

Activities/ outputs

Capacity change

Behaviour change

Direct benefits

Wider benefits

Assumptions about how

capacities lead to behaviour change, and conditions required

Assumptions about how behaviour

change benefits the

individual

Assumptions about how

direct benefits lead

to wider social

benefits

Assumptions about the

basic purpose and function of the activity

mechanism + context → outcome

• evaluation theorists also use a language of ‘mechanism’ for the process which leads from activity to benefits, and the idea that mechanisms act in contexts to lead to outcomes: the same mechanism in a different context might lead to different outcomes

• here ‘mechanism’ means the first three stages of the LM...

...and ‘outcomes’ are the benefits

Activities/ outputs

Capacity change

Behaviour change

Direct benefits

Wider benefits

9 types of mechanism identifiable across the projects

Working with key individuals

Providing advice

Training/education

Getting individuals together across communities

Environmental projects

Financial support for community groups

Targeted state support to families and young people

Increasing enforcement activity

Working with governance organisations (state and VCF)

9 types of mechanism, 8 partners, 24 projects


Providing advice

Training/education







RMBC: ACE

RMBC: CYPS

RMBC: RE

Clifton Learning Partnership

REMA

RUCST

Premier Learning

Kimberworth Park Community P’ship

2

8

3

19

5

10

1

3

9

20

6

11

4

7

14

15

16

22

23

3 12

13

14

15

12

14

15

16

18

21

mechanisms in detail (1)

Activities/ outputs

Capacity change

Behaviour change

Working with key

individuals

Providing advice

Training/education

Getting individuals together

across communities

Street champions

Community navigators

Good neighbour project

Increased engagement skills, confidence,

knowledge to - provide help to others

- engage with the state/VCF

- mediate disputes

Engagement with others in the community

Reporting problems to relevant governance organisations

Drawing up a ‘Residents’ Charter’

Advice services Increased knowledge of rights, services available

More, faster, earlier, appropriate accessing of services

ESOL, and other skills/knowledge taught

along with ESOL

Better knowledge of English

Better understanding of rights/responsibilities

Other knowledge/skills (with vocational etc.

relevance)

More engagement with service providers

More exercising of democratic rights

More socially responsible action within communities

Move into employment

ESOL, Citizenship Project, BSC Forum,

festivals/events, sport, Love is Louder events

Better understanding, trust, respect for different Others

Less antagonistic behavioursEnforcement of norms of

tolerance within communities (e.g. stopping/reporting hate

crime)Self-organised inter-community

collaborative working Better school attendance

mechanisms in detail (2)


Financial support for community

groups


Increasing enforcement

activity

Working with governance

organisations

Activities/ outputs

Capacity change

Behaviour change

CLP environmental projects

Provided with skills and opportunity to deliver projects More respect amongst participants and others for the immediate environment

Direct improvements: litter cleared, allotments, plantingLess littering in futureMore food growing Better school attendance

Giving small grants to community

groups/organisations Providing support to

groups to bid

miscellaneous miscellaneous

Outreach/street working with young people

Targeted bespoke family support

YP more confident and self-aware as citizensBetter (state) understanding of YPs’ issues

Reduced ASB (including drugs)Collective activities Participation in youth forumIncrease in education participation ratesReduction in family abuse

Reducing CTax fraudEnforcing housing

standardsEnforcing environmental regulations - fly tipping

Understanding of risks and penalties for non-compliance/lawbreakingStaff and IT (surveillance, reporting) increased

Cessation of illegal activityCompliance with rules

BSC Forum?

Better understanding of communitiesTrust, understanding, skills to work in partnership Better knowledge of possible funding/resource streams

More appropriate service provisionMore effective partnership working: reduced duplication, signposting between organisations, new joint actionsMore external resources successfully bid for

mechanisms and outcomes


Providing advice

Training/education







Direct benefits

Quicker settling in

More effective resolution of service problems

More harmonious community life (less involvement in racial abuse etc., disputes resolved)

Satisfaction/wellbeing from social engagement/participation

More confident and able to exercise rights/responsibilities

(Better) employed/higher income

Living in better physical environment (greenery, litter, fly tipping…)

More secure tenancy in better housing

More harmonious personal lives (within families, at school, less friction with authorities)

Mechanisms

direct and wider benefits

Wider benefitsDirect benefits

Quicker settling in


More harmonious community life (less involvement in racial abuse etc., disputes resolved)

Satisfaction/wellbeing from social engagement/participation

More confident and able to exercise rights/responsibilities

(Better) employed/higher income

Living in better physical environment (greenery, litter, fly tipping…)

More secure tenancy in better housing

More harmonious personal lives (within families, at school, less friction with authorities)

More appropriate service provision

More appropriate calls on services

More participation in ‘governance’ broadly

Less hate crime and other ASB

More meaningful interactions between different groups

Better landlord/tenants relationships

Better neighbourhood environment

Better housing (physical, security)

Wealthier community

Higher (mental/physical/ emotional) well-being

Positive changes in state (etc.)/resident

behaviour

Positive changes in resident/resident

behaviour and relationships

Improved conditions for (deprived) communities

theory of change for one mechanism

• Getting individuals together across communities

Antagonism between communities, and

towards the state for perceived

biases/failings, comes from a lack of

understanding of, and empathy with,

different Others, and direct contact will

address this (rather than worsen it)

Activities: ESOL, Citizenship Project,

BSC Forum, festivals/events,

sport, Love is Louder events

Capacity changes: Better

understanding, trust, respect for different Others

Behaviour changes: Less antagonistic behavioursEnforcement of norms of tolerance within communities (e.g. stopping/reporting hate crime)Self-organised inter-community collaborative working Better school attendance

Direct benefits

Wider benefits

People who engage are those whose positions need changing Contact provides opportunities for effective, meaningful dialogue, and learning about the Other – and not for increased antagonism/reinforcing of prejudices.Sufficient, competent staff are involved, who can (at minimum) effectively deal with tension/friction, and are able to facilitate effective (potentially very difficult) interactions

Individuals are sufficiently confident to act on new understandings within their cultural context, and robust enough for this to be sustained Opportunities and support are available for further engagement Local and national policy, the media and wider societal discourses, do not provide overriding counter-messages

Behaviour changes are enduringSkills are successfully deployed (personal and mediation etc.) Communities respond positively to changed behaviours

Mechanisms Activities Capacity change Behaviour changeWorking with key individuals Street champions

Community navigatorsGood neighbour project

Increased engagement skills, confidence, knowledge to - provide help to others- engage with the state/VCF- mediate disputes

Engagement with others in the community Reporting problems to relevant governance organisations Drawing up a ‘Residents’ Charter’

Assumptions Tacking problems requires communication (at minimum) between state and residents, and key individuals can provide routes into the community which the state (and VCF?) cannot achieve themselves, due to problems of trust, legitimacy, language, resource capacity etc.

Suitable individuals can be identified, and will keep with the programmeProvider organisations are trusted enough for relevant people to engage with them Programmes will deliver the required skills and knowledge

Participants will be willing and able to actParticipants will share the programme goals and act appropriately Participants will have the networks and legitimacy to actCultural norms support increasing engagement with the state Service providers will respond and reinforce behaviour

Providing advice Advice services (REMA and CLP) Increased knowledge of rights, services available More, faster, earlier, appropriate accessing of servicesAssumptions ignorance is a problem - people would

value services/rights but fail to take them up or act on their rights because they don’ know what is available, how to access them etc. or what their rights are.

Those in need will know about the services- and have the confidence and ability (mobility, language) to access the serviceAppropriate and accurate advice will be available, from people with the skills/attitudes/knowledge to engage with the target population

People will have the confidence and resources to act on the knowledgeCultural norms support increasing engagement with the state Appropriate services will in fact be available and accessible and able to meet any increased demand

Training/education ESOL, and other skills/knowledge taught along with ESOL

Better knowledge of EnglishBetter understanding of rights/responsibilities Other knowledge/skills (with vocational etc. Relevance)

More engagement with service providersMore exercising of democratic rightsMore socially responsible action within communitiesMove into employment

Assumptions People lack knowledge and skills needed to play a full part in society -and they see education/training as a way of addressing this

Those in need will know about what is available- and have the confidence and ability (mobility, language) to access the servicePeople will be able to attend regularly enough to learnSufficient high quality training, appropriate to users’ needs, is available

Education/knowledge will empower peopleCultural norms support increasing engagement with the state Service providers are willing/able to engage and meet increased demandGovernance organisations are willing/able to engage with democratic expression


ESOL, Citizenship Project, BSC Forum, festivals/events, sport, Love is Louder events

Better understanding, trust, respect for different Others Less antagonistic behavioursEnforcement of norms of tolerance within communities (e.g. stopping/reporting hate crime)Self-organised inter-community collaborative working Better school attendance

Assumptions Antagonism between communities, and towards the state for perceived biases/failings, comes from a lack of understanding of, and empathy with, different Others, and direct contact will address this (rather than worsen it)

People who engage are those whose positions need changing Contact provides opportunities for effective, meaningful dialogue, and learning about the Other –and not for increased antagonism/reinforcing of prejudicesSufficient, competent staff are involved, who can (at minimum) effectively deal with tension/friction, and are able to facilitate effective (potentially very difficult) interactions

Individuals are sufficiently confident to act on new understandings within their cultural context, and robust enough for this to be sustained Opportunities and support are available for further engagement Local and national policy, the media and wider societal discourses, do not provide overriding counter-messages

the full ToC: mechanisms and assumptions

Mechanisms Activities Capacity change Behaviour changeEnvironmental projects CLP environmental projects Provided with skills and opportunity to

deliver projects More respect amongst participants and others for the immediate environment

Direct improvements: litter cleared, allotments, plantingLess littering in futureMore food growing Better school attendance

Assumptions Antagonism and dissatisfaction with neighbourhoods related to poor environmental condition, and social and physical improvements can be achieved if people improve the environment themselves

People with enthusiasm can be recruited and retained

Project outputs can be maintained (and not vandalised) – either by the state or by the community Improved environment encourages more environment-friendly behaviours


BSC Forum? Better understanding of communitiesTrust, understanding, skills to work in partnership Better knowledge of possible funding/resource streams

More appropriate service provisionsMore effective partnership working: reduced duplication, signposting between organisations, new joint actionsMore external resources successfully bid for

Assumptions Service provision is currently weakened by governance issues, in particular around integrated/partnership working, and (lack of ) understanding of relevant communities (in particularly Roma)

Service providers – as organisations and individuals – are able and willing to learnGovernance organisations are willing to work together

Service providers are adequately resourced to act on new understanding Governance organisations are able to work together (adequate resources/support available)


Reducing CTax fraudEnforcing housing standardsEnforcing environmental regulations - fly tipping

Understanding of risks and penalties for non-compliance/lawbreakingStaff and IT (surveillance, reporting) increased

Cessation of illegal activityCompliance with rules

Assumptions Violations of environment/housing regulations play a significant role in creating antagonism between communities, and stricter enforcement can be done effectively without antagonising communities/undermining trust in the state

People beyond those directly affected know about enforcement action and the risk of itPeople ‘read’ enforcement in terms of upholding law, not as oppression Intelligence – from IT and staff/community reporting – can be effectively and swiftly acted upon

Calculations of costs/benefits of illegal/no-compliant activity favour cessation and compliance (may be affected by perceptions of risksProvision of services to enable compliance (e.g. of waste disposal, support for housing repairs)


Outreach/street working with young peopleTargeted bespoke family support

Better (state) understanding of YPs’ issues Reduced ASBCollective activities Participation in youth forumIncrease in education participation ratesReduction in family abuse(Reduction in drug use?)

AssumptionsFinancial support for community organisations/groups

Giving small grants to community groups/organisations Providing support to groups to bid

Assumptions Appropriate groups can be reachedSmall financial barriers are important in preventing groups achieving aimsLack of capacity/knowledge/confidence in bidding is a significant barrier

the full ToC: outcomes and assumptionsDirect benefits(Better) employed/higher income

Assumptions: Language skills and confidence are barriers to employmentRelevant and accessible jobs exist

Better physical environment (greenery, litter, fly tipping…)Assumptions: Sufficient numbers of people participate, and continue to participate/act

Other members of community and service providers support actions


Assumptions: Service providers are willing/able to respond to demandMore harmonious community life (less involvement in racial abuse etc., disputes resolved)

Assumptions: Behaviour changes are enduringSkills are successfully deployed (personal and mediation etc.)

Communities respond positively to changed behaviours

More secure tenancy in better housingAssumptions: Landlords are able/willing to make investments

Tenants are able/willing to be ‘good tenants’ – longer lets, less damage etc.

Quicker settling inAssumptions: New arrivals are reached, and are willing/able to respond

Service providers are willing/able to respond to demand

Satisfaction/wellbeing from social engagement/participationAssumptions: Engagement is in fact positive (and not stressful etc.)

Benefits are significant in relation to other causes of ill-being

More confident and able to exercise rights/responsibilitiesAssumptions: Participation and learning are sufficient to enable active engagement

Service providers, governance organisations, other members of community are receptiveOpportunities for engagement exist and are accessible

More harmonious personal lives (within families, at school, less friction with authorities)Assumptions: Service providers, governance organisations, other members of community are receptive

and supportive.

rmbc cmf theory of change web version · title: microsoft powerpoint -...

Documents