This project has been funded with support from the European Union Programme for Employment and Social Solidarity –

PROGRESS (2007-2013). For more information see http://ec.europa.eu/progress

The information contained in this publication does not necessarily reflect the position or opinion of the European Commission.

Evaluation of the Mentoring

for Excluded Groups and

Networks (MEGAN) European

Project

Final Report

1

Contents

Executive Summary ................................................................................................ 2

Introduction.......................................................................................................... 16

Project findings .................................................................................................... 29

Pilot findings: Hungary ......................................................................................... 44

Pilot findings: Portugal ......................................................................................... 68

Pilot findings: United Kingdom ............................................................................ 81

Randomised control trial feasibility study ........................................................... 97

Conclusions and Recommendations ................................................................... 104

Technical Appendix.............................................................................................108

2

Executive Summary

Key points:

Project impact

The MEGAN project tested the feasibility of using a Randomised Control Trial (RCT)

methodology to determine the impact of mentoring on a range of soft and hard

outcomes, supported by qualitative research. The study was conducted in three

countries: Hungary, Portugal and the UK.

Using an RCT approach we have, for the first time, been able to successfully isolate the

impact of mentoring on employment and employment related outcomes in Portugal.

Qualitative research also highlighted improvements in confidence, resilience,

communication and social integration amongst mentees.

Mentors can support mentees at the initial stages of engagement with professional

support, acting as a ‘gateway’ and encouraging engagement. Once initial complex

barriers have been addressed, mentoring can assist mentees in developing their soft

skills and moving towards social integration. Mentoring can also help to support mentees

to achieve hard outcomes such as employment.

Mentoring also has an important impact on mentors. Non-peer mentors built on existing

skills in new areas, while peer mentors achieved new practical and personal skills that

improved their employability. A number of mentors moved into employment.

Project design

Mentoring is likely to be most effective if goals are embedded in the project design and

delivery. Mentoring is flexible enough to achieve a range of hard and soft outcomes in

any number of settings, as long as objectives are articulated and understood by mentors

and mentees from the start.

Mentoring is often delivered as part of a wider, holistic package of support rather than as

a standalone intervention; this proved to be an effective model of delivery on the MEGAN

project. The additional value of mentoring, over and above what the delivery

organisation could deliver normally, is the intensity of support a mentor can provide.

Mentoring and professional support can successfully coexist. Within the MEGAN project

mentors played the role of identifying mentees’ needs and supporting them to engage

with professional services productively.

3

Experimental evaluation design: RCT feasibility

Smaller community organisations can successfully administer a challenging RCT

evaluation design. Key factors in success are a flexible randomisation approach (i.e.

continuous or by cohort - depending on delivery partners’ needs), upfront training in

randomisation rationale and procedure, and commitment from delivery partners.

RCTs can be successfully administered within close-knit communities. However it may be

more appropriate to randomise communities as opposed to individuals within them,

particularly where communities are disadvantaged and all members need support.

RCTs are dependent on robust data collection, which can be challenging when

interventions are working with hard to engage populations and during periods of

organisational change.

Introduction

The Mentoring for Excluded Groups and Networks (MEGAN) project sought to test

mentoring as a means of facilitating the social inclusion and employment of vulnerable

groups including Roma people, foreign nationals, young people, women, long-term

unemployed people and offenders. The project builds on learning from a previous EU-

funded PROGRESS project, Models of Mentoring for Inclusion and Employment (MOMIE).

The MOMIE project focused on the main variable of peer vs. non peer mentoring as a tool

for supporting vulnerable groups.

The MEGAN project was delivered in three EU member states (UK, Hungary and Portugal)

and focused on marginalised communities and their descendents who face social exclusion

and isolation and multiple and complex barriers to the labour market. The project was

designed to assess how mentoring could assist with social integration for different groups

of disadvantaged people.

MEGAN was funded by the European Commission DG Employment, Social Affairs and

Inclusion. It supports the Europe 2020 flagship initiatives of the 'Innovation Union' and the

'Platform against Poverty and Social Exclusion'. It also complements the EU Youth Strategy,

the Roma Youth Action Plan, the Action Plan on Improving the Situation of Roma within the

OSCE area, and a high number of directives on the equal treatment and integration of

migrants within the EU.

The specific objectives of this project were to:

Enhance the social inclusion of migrant communities through mentoring;

Promote effective and cost efficient strategies to support the social inclusion and

employment of vulnerable groups;

4

Measure the impact of mentoring on a range of hard (e.g. employment and reoffending)

and soft (e.g. confidence and motivation) outcomes for marginalised communities;

Test the feasibility of conducting a randomised control trial of mentoring to identify

impacts.

Evaluation

The final two project objectives are the specific focus of the evaluation and of this report,

delivered by the Centre for Economic and Social Inclusion. The evaluation consisted of the

following elements:

Impact analysis

MEGAN used a Randomised Control Trial (RCT) methodology to measure the impact of

mentoring on a range of outcomes such as employment and employment related activities,

educational attainment, offending, literacy and health. This involved measuring outcomes

of a treatment group (in receipt of mentoring) and a control group (not in receipt of

mentoring) in order to determine the role of mentoring in achieving the stated outcomes.

Outcomes for project participants (treatment and control group) were measured every

three months, during the six months of the project and for six months following the end of

the project. At each stage, researchers monitored key outcomes for both mentees and

control group participants. Outcomes for both groups have been compared at the end of

the project to measure the impact of mentoring.

This methodology supports the Europe 2020 Innovation Union strategy through a robust

and rigorous evaluation method. However, the substantial challenge of conducting a robust

impact analysis of mentoring projects is evident in the lack of existing studies. Therefore,

this project seeks to test the feasibility of conducting an RCT to test the impact of

mentoring with disadvantaged groups communities. By adopting the methodology, the

project aims to contribute to the ongoing work to develop and implement research and

innovation to address societal challenges.

Qualitative research

To gather detailed information on the project delivery and experience, in-depth interviews

were conducted with mentees and control group members, mentors and project

coordinators. Interviews explored the design and delivery of each pilot and participants’

experiences; asking detailed questions about participants’ lives, support they received, their

reasons for wanting to receive mentoring and the impact of their participation in the

project.

5

Interviews were conducted every three months during the six months of the project and for

six months following the end of the project, alongside the hard outcome measures.

Project delivery

Key points:

Mentoring is often delivered as part of a wider, holistic package of support rather than as

a standalone intervention; this proved to be an effective model of delivery on the MEGAN

project.

The additional value of mentoring, over and above what the delivery organisation could

deliver normally, is the intensity of support a mentor can provide.

Mentoring and professional support can successfully coexist. Within the MEGAN project

mentors played the role of identifying mentees’ needs and supporting them to engage

with professional services productively.

In each pilot, volunteers were recruited or referred to receive mentoring support.

Volunteers were then randomly divided into treatment and control groups by the evaluation

team at Inclusion. The treatment group received mentoring for a period of 6 months, in

addition to mainstream support from the delivery partner. The control group received

mainstream support from the delivery partner only. Coordinating organisations were given

the freedom to design their pilots to meet the needs of their target groups and to decide

whether peer or non-peer mentoring was most appropriate. However, some elements were

mandatory:

Weekly contact between mentors and mentees;

Regular support and supervision for mentors;

Mixed forms of contact between mentor and mentees (e.g. 1-2-1 meetings, group

sessions, telephone and email contact).

Hungary - BAGázs: Roma community

The target group for the Hungarian pilot was the Roma community living in Bag, a village

40km outside of Budapest. Thirty-three mentees received mentoring support as part of

MEGAN from a total of 24 peer mentors. The pilot was delivered by BAGázs, a not-for-profit

organisation.

The Hungarian pilot has three strands aimed at different groups: men, women and young

boys. Pilot strands were strongly aligned with other activities that BAGázs was undertaking

in the community: house building for the men, literacy programmes for the women and

6

football for the young boys. The rationale for this was both to attract volunteers and

maximise participation, and to achieve BAGázs’ wider goals of bringing the community

together and supporting Roma integration. The mentoring meetings were the only element

of support that was additional to activities and services available to the control group.

In all three strands of the Hungarian pilot, mentors and mentees met formally on a weekly

basis for one hour, 1-2-1 sessions, at a time determined by the mentor. However, given

their proximity and existing relationships, many mentors and mentees met informally much

more frequently than this.

Portugal - Aproximar CRL and Santa Casa Misericórdia da Amadora

(Pendulum Consortium): families on social benefits

The target group for the Portuguese pilot was local residents in receipt of the Minimum

Income, a benefit for those living in extreme poverty. There were 38 mentees involved in

MEGAN, split into an ‘Entrepreneurial’ and a ‘Disadvantaged’ group at baseline depending

on their initial skills.

The pilot was delivered by the Pendulum Consortium, a partnership of two organisations,

Aproximar CRL and Santa Casa Misericórdia da Amadora (SCMA), formed following

commissioning and the official start of the MEGAN project. Aproximar played a coordinating

role and SCMA was responsible for delivery.

The Portuguese mentoring model involved peer mentoring from 15 peer mentors, primarily

drawn from a base of previous MOMIE project clients. The Pendulum model had a defined

structure and a specific work focus. Mentees engaged in 24 sessions over the same number

of weeks. Each session was divided in two sections; the first with a particular focus on

work-related skills and a second where mentor and mentee could engage on themes of

their choice.

UK - London Probation Trust and Praxis: Foreign National offenders

The target group for the UK pilot was Foreign National offenders who had been released

from prison and were on a minimum of a six month supervision order with their probation

officer. There were 35 UK mentees involved in MEGAN.

The pilot was run through a partnership between the London Probation Trust (LPT) and

Praxis Community Projects (Praxis). LPT was responsible for referring mentees onto the

pilot. Praxis was responsible for recruiting and training mentors, matching mentors to

mentees, and managing the mentoring relationships.

The UK mentoring model involved 24 non-peer mentors, i.e. mentors were not also foreign

national offenders, although many had experience of immigration issues either personally

or professionally. Mentees attended an initial meeting with their mentor and probation

7

officer to draw up an action plan setting out what they hoped to achieve from mentoring.

Mentors and mentees then met once a week on average over six months. The mentors also

met as a group to discuss challenges and best practice.

The UK pilot proved challenging to deliver. The target group were transient and dispersed

all over London and poor attendance by mentees was an issue that affected the delivery of

mentoring as well as the evaluation.

Models of mentoring

Key points:

Mentoring is likely to be most effective in achieving its goals if these goals are embedded

in the project design and delivery. Hard outcomes were more detectable where pilots

took a ‘goal-oriented’ approach, whereas soft outcomes were more evident where pilots

took an ‘open-ended’ approach.

Both goal-oriented and open-ended approaches to mentoring can be successful in

achieving outcomes. Organisations should be realistic about what they are aiming to

achieve and how they will demonstrate impact when designing their projects.

Mentoring is flexible enough to achieve a range of hard and soft outcomes in any

number of settings, as long as objectives are articulated and understood by mentors and

mentees from the start.

The evaluation found two prevalent models of mentoring, which we have categorised as

‘goal-oriented’ and ‘open-ended’ mentoring. The former is most associated with the

Portuguese model and the Hungarian men’s pilot strand. The other Hungarian pilot strands

were delivered as open-ended mentoring. The UK pilot incorporated both models.

Goal-oriented mentoring

Pilots deemed as ‘goal-oriented’ have a specified target that is built into the pilot design. In

some cases, it was the mentees who approached mentoring as a ‘goal-oriented’ activity,

and entered the process with a need they wanted to address. Mentees engaged in ‘goal-

oriented’ mentoring described mentoring as support to help them achieve a singular goal

connected to the focus of the project or their own need.

The evaluation found that having a clear goal could increase the chances that mentoring

will address the specified need (and potentially in a shorter timeframe). Our findings

suggest that mentees may see less value in goal-oriented mentoring if the specified ‘need’

is not addressed, even if this is due to factors outside of the mentor’s control (such as the

economic environment).

8

Mentees in ‘goal-oriented’ schemes may be less likely to perceive soft outcomes such as

increases confidence and motivation. It may be that mentees do in fact improve in these

areas without being consciously aware of it but this may be difficult to capture. The

Portuguese pilot was goal-orientated but mentees were able to identify soft outcomes. This

is likely to result from the structure of the pilot which allocated 50 percent of mentoring

time to themes relating to social and professional skills and 50 percent to wider discussion.

Open-ended mentoring

As opposed to ‘goal-oriented’ mentoring, ‘open ended’ mentoring incorporates a range of

intermediate goals rather than one specific end goal. In these cases, mentoring is soft

outcome focussed (e.g. improving communication skills and relationships), with a view to

achieving hard outcomes in the longer term. Mentees may be ‘working towards’ hard

outcomes, but the mentoring relationship is not specifically geared towards achieving these

outcomes, and mentees may not even be aware of the ultimate outcome as it is sometimes

implied rather than openly stated.

Open-ended mentoring can be centred on specific activities; however, there is not a set

target attached to the activity as with ‘goal-oriented’ mentoring. In this model, mentees

may find they gain a wider range of outcomes and can access support that is more flexible

to respond to changing needs. However, mentees in open-ended projects tend to find it

more difficult to identify the skills that they have gained and may also be less likely to

achieve ‘hard’ outcomes. Hard outcomes are often those which funders seek to measure.

Project impact

Key points:

Using an RCT approach we have, for the first time, been able to successfully isolate the

impact of mentoring on employment and employment-related outcomes in Portugal.

Mentors can help to support mentees to achieve a range of hard and soft outcomes

including employment, confidence, resilience, communication and life skills.

Improvements in soft skills may make mentees more employable.

Where mentoring support is intensive, outcomes may be delayed, therefore outcomes

may not be able identified until after mentoring projects have been completed.

Mentors can support mentees at the initial stages of engagement with professional

support, acting as a ‘gateway’ and encouraging engagement. Once initial complex

barriers have been addressed, mentors can assist mentees in developing their soft skills

and moving towards social integration.

9

Mentoring also has an important impact on mentors. Non-peer mentors were able to

build existing skills in new directions, while peer mentors were equipped with new

practical and personal skills that improved their employability. A number of mentors

moved into employment.

The evaluation ran over a twelve month period in an attempt to measure the impacts of six

months’ mentoring, as well as whether these impacts they were sustained for six months

post project.

Soft outcomes

Approximately 250 qualitative in-depth interviews were conducted with mentees and

control group participants quarterly over the evaluation period, and the soft outcomes of

mentoring have been ascertained from the analysis of these interviews.

Mentees gained the skills, confidence and resilience to cope on their own and manage their

lives more effectively as a result of mentoring. Mentees described feeling more positive

about their futures, better able to cope with rejection (for example, in job applications) and

better able to manage emotions and relationships. From an evaluation point of view, we

found that interviewees’ confidence and communication skills improved over the duration of

the project.

Mentoring empowered mentees by giving them a forum in which to discuss the problems

they were facing and to receive emotional and practical support to tackle their problems.

Mentoring also provided a gateway to support and services that could improve mentees’

circumstances and wellbeing, and helped to equip them with the skills they needed to

better engage with support (e.g. communication skills, language, literacy).

Mentoring created social networks amongst those who were socially isolated, thereby

improving their community and wider social links. In situations where individuals were

isolated from wider society, mentees formed relationships with individuals and services they

would likely have never have otherwise come into contact. This helped them to become

more engaged in mainstream society. When mentoring was delivered within closed

communities, there is evidence that mentoring has played a role in empowering and

preparing the community to engage with wider society, for example through public

speaking at conferences, engaging with services outside of the immediate community,

socialising in areas outside of their immediate location, and producing media to highlight

their circumstances to a broader audience.

Hard outcomes

We analysed the impact of mentoring on a range of hard outcomes. Our analysis took into

account the effects of differing personal characteristics and starting points (e.g. gender,

10

age, prior qualifications and employment status) of both treatment and control participants

on these outcomes. This ensures that our analysis isolates the impact of the mentoring

treatment specifically.

Outcomes data was collected by the projects themselves in the Portuguese and in

Hungarian pilots and we were able to conduct an analysis of the impact of the MEGAN

project at the 6 and 12-month points (i.e. project end and six months post project) on a

range of outcomes. The UK analysis has been based on existing administrative information

and includes more detail on migrant status and offending as the study group are largely

migrant offenders. Information on employment and recent offending at the 12-month point

was collected (i.e. six months post project). However, information on other outcomes, such

as confidence in budgeting and financial management, were not available for the UK.

At the six-month point, i.e. the end of mentoring relationships, there were no statistically

significant differences between the treatment and control groups in either Hungary or

Portugal. However, there were observable differences in employment outcomes in Hungary

between treatment and control group mentees at this point although they were not

statistically significant. Furthermore, those that were in work reported that this work was

better paid than those in the control group.

At the twelve-month point, i.e. six months after the end of mentoring relationships, the

analysis results showed that the mentoring intervention in Portugal produced statistically

significant positive effects on two outcome variables - paid employment in the last three

months and budgeting and management of household finances. The employment outcomes

were largely found to have occurred amongst the entrepreneurial group, which is

unsurprising given that they were objectively more employable i.e. they had higher skill

levels and better Portuguese language ability.

The fact that this difference was not observed at the end of the project may suggest a

‘lock-in’ effect of the Portuguese model which means that the intensive nature of the

project prevented or delayed mentees from seeking work whilst mentoring was ongoing. In

terms of budgeting outcomes, we can speculate that the effect may be lagged because

mentees only perceived changes in their budgeting ability once they had sufficient time to

put those skills into practice.

The 12-month analysis for Hungary showed no significant change on any outcome measure

between control and treatment groups. The 12-month UK analysis showed no significant

impact on either of the two outcome measures that were tested (employment and

offending).

11

Impacts on mentors

All mentors appeared to gain personal satisfaction and a sense of worth from supporting

mentees. This was found to particularly be the case for peer mentors who benefitted as

much, if not more, than the mentees. Mentors in the Portugal and Hungary pilots spoke of

a reciprocal relationship where they gained as much from the experience as their mentee,

in terms of confidence and self-belief.

Non-peer mentors often had pre-defined ideas about what skills they could bring to the

mentoring relationship and how they would use these skills; they could therefore become

disillusioned if their experience of supporting a mentee was not as they expected. For this

reason, non-peer mentors may require additional in-project support and networking

opportunities.

In Portugal, seven of the peer mentors who were involved in either the MOMIE or the

MEGAN project went on to find employment. In Hungary, one male mentor went on to

secure employment. Others were working to set up a Business Club in the settlement to

help the residents start and run their own businesses. The UK pilot involved non-peer

mentors who were largely already employed. However, we do know that following their

involvement in mentoring at Praxis two mentors secured full-time jobs in the public sector

and one secured a temporary job.

Feasibility study

Key points:

Organisations should aim to use the most robust evaluation methodology to determine

the impact of their projects. Smaller community organisations can successfully

administer a challenging RCT evaluation design. Key factors in success are a flexible

randomisation approach (i.e. continuous or by cohort - depending on delivery partners’

needs), upfront training in randomisation rationale and procedure, and commitment from

delivery partners.

RCTs can be successfully administered within close-knit communities. However it may be

more appropriate to randomise entire communities as opposed to individuals within

them, particularly where communities are disadvantaged and all members need support.

RCTs are dependent on robust data collection which can be challenging when

interventions are working with hard to engage populations and during periods of

organisational change. The conditions in which the project is being delivered, and the

potential impact on outcomes should be considered during the design phase.

12

Larger sample sizes are required where the impact of the intervention is expected to be

small. Therefore future evaluations may wish to focus on large-scale mentoring projects.

The MEGAN evaluation adopted a rigorous randomised controlled trial (RCT) approach to

test the impact of mentoring on outcomes such as employment, educational attainment,

offending, literacy and health measures. The randomised trial is considered the ‘gold

standard’ for evaluation because the random assignment into treatment and control means

that differences in outcomes between the two groups can be attributed to the project itself,

rather than to any personal characteristics or environmental factors.

However, RCTs are not always used in practice because of practical difficulties in

implementation. The MEGAN project sought to test the feasibility of conducting an RCT to

test the impact of mentoring with disadvantaged groups. By adopting the methodology,

the project aimed to contribute to the ongoing work to develop and implement research

and innovation to address societal challenges.

Implementation

Countering the advantages of RCTs are the difficulties (and sometimes impossibilities) in

allocating people randomly within a particular area to a treatment or control group. For

voluntary programmes, there can be an ethical difficulty in advertising projects that some

people will be denied entry to if they apply. In the UK the concept of a randomised control

trial, where by definition not all referred mentees can receive the mentoring ‘treatment’, led

to disengagement by some probation officers resulting in lower than expected referrals to

the pilot.

Although the results from randomised trials have a credibility that other evaluation designs

do not, the estimate of the programme effect may be biased if any of the following occur:

if some of those control group do actually participate in the programme. Pilots were

rigorous in implementing the RCT design and we do not believe that control group

members were mentored in the same way as the treatment group. However in two pilots

the close nature of the communities involved meant that the distinction between

treatment and control could be blurred.

if those randomised to the programme affect the behaviour or attitudes of those in the

control group (which might happen if members of the two groups know one another).

The UK pilot had the clearest distinction between treatment and control groups. As

opposed to the other two pilots, where there may have been contamination issues

between members of small, tightly-knit communities; in the London pilot the mentees in

the treatment and control groups were unlikely to mix. The Hungarian pilot was

conducted in a small village with a large proportion of the residents involved, and thus

the support BAGázs provided has a huge impact on the settlement as a whole. From an

13

evaluation point of view this means that the intervention had effects on the wider

population and the control group as well as the treatment group. In some cases both

mentees and control group members participated in a pilot ‘activity’ (e.g.

building/repairing houses, playing football) that other residents were not involved in –

again diminishing the advantage the treatment group experience. The parents of some

young people in the control group may have been mentored, meaning that the impact of

mentoring on adults would be experienced by their children.

if surveys to measure outcomes get differential non-response in one or other of the

groups. In the UK pilot, we were only able to conduct qualitative interviews with those

still engaged with the pilot and thus were more likely to speak to mentees whose

relationships had been successful.

if being denied access to a programme makes those in the control group act differently

to how they would act in the total absence of the project. We do not believe this to have

been an issue. However this was clearly a concern for the Hungarian pilot and is why

control group members had good access to the services of BAGázs.

External practical factors are often overlooked when designing programmes but they can

present challenges to conducting a robust evaluation. An external factor which affected the

impact assessment were the changes to UK probation services. The UK data collection

should have been the most rigorous as it was independent of the delivery partner and

already being collected for other purposes. However data collection was reliant on regular

updates from London Probation Trust. With major structural reform to probation services

ongoing it was very difficult to formalise the data collection and transfer process, in spite of

significant goodwill on the part of LPT. As a result data was not collected as regularly as

was envisaged at project inception and we have limited UK data for outcomes twelve

months and none at six months.

Recommendations

The evaluation provides some useful learning for organisations considering implementing a

mentoring programme in terms of some practical considerations surrounding the design

and delivery of a successful programme:

Consider the timing of the mentoring intervention within the mentees’ life

journey. The benefit of mentoring can be maximised once initial barriers to making

positive life changes, such as language difficulties and housing issues, have been

overcome. Although mentoring can support mentees to address these initial barriers,

professional support is likely to be required. In order to be most effective, mentoring

programmes should be implemented once these barriers have been addressed.

14

An effective programme must have a sufficient lead-in time built in before the

start of delivery. This is particularly critical for mentoring projects where intensive

mentor training and careful relationship matching will be required upfront.

Establishing a good match between mentor and mentee is critical. A poor match

can lead to mentee and mentor disengagement. Mentors and mentees do not necessarily

need to share personal characteristics and experiences (although this can help) and the

matching should be based on the needs of the mentee and the support that the mentor

can offer in this respect.

Before starting to work, mentors and mentees should clarify the aim and

purposes of the partnership. This would help mentees to have realistic expectations

and feel involved in the design of the intervention and give the mentors a clear goal and

direction.

Ongoing engagement with referral and support agencies is key. This is a

particularly important point for organisations receiving mentoring through a third party.

Where there is an association with a statutory organisation, either as referrer or

deliverer, additional efforts may be required to ensure that mentees fully understand the

voluntary nature of participation.

Maintaining mentee engagement can be challenging. Useful mechanisms to

maintain engagement include: addressing practical issues such as removing barriers to

attendance (i.e. by covering travel expenses) and ensuring that mentees understand

what to expect from the programme and the potential benefits (i.e. by creating

information materials for prospective mentors and referral organisations).

Mentors must be supported by delivery organisations throughout the

programme. Supporting mentors throughout helps to ensure that the experience is

positive and beneficial.1 Some issues that mentees experience can be distressing and

difficult for a mentor to deal with. Delivery organisations should offer opportunities to

mentors to share experiences and to seek help in areas of uncertainty.

Provide scope for progression in mentoring programmes. Being a mentee can

also be the starting point of development in that mentees can go on to become mentors

and progress from there. It is therefore important that mentoring programmes are

sufficiently embedded to allow for progression to occur.

The evaluation also indicates the following lessons for (a) implementation of randomised

control trials and (b) conducting research with disadvantaged groups:

1 For example: Praxis has been successful in securing the Approved Provider Standard (APS) from the

Mentoring and Befriending Foundation for MEGAN and future mentoring projects. The APS logo is a national quality standard mark of safe and competent practice.

15

Smaller community organisations can successfully administer a challenging

RCT evaluation design. Key factors underpinning success are: a flexible randomisation

approach (in line with the needs of the delivery organisation); upfront training (to ensure

understanding of the rationale for RCT evaluation, the randomisation process and the

ongoing need for separation of treatment and control); and strong commitment from

delivery partners.

Where target communities are small/close-knit it may be more appropriate to

randomise whole communities as opposed to individuals within them. This is

particularly relevant where all members are disadvantaged and need support.

When evaluating programmes that target dispersed or hard to engage groups, additional

efforts will be required to implement an evaluation. It is necessary to be flexible on

interview times, dates and locations and to allocate additional fieldwork resources to this

group.

Future evaluations may wish to focus on large-scale mentoring programmes.

As part of a holistic package of support, the impact of mentoring specifically is likely to

be small. Therefore larger numbers of participants would make it more likely that a

statistically significant impact can be identified through evaluation.

16

1 Introduction

Background to MEGAN

1.1 The Mentoring for Excluded Groups and Networks (MEGAN) project sought to test

mentoring as a means of facilitating the social inclusion and employment of

vulnerable groups including Roma people, foreign nationals, young people, women,

long-term unemployed people and offenders.

1.2 The project built on learning from a previous EU-funded PROGRESS project, Models

of Mentoring for Inclusion and Employment (MOMIE). The MOMIE project focused

on the main variable of peer vs. non peer mentoring as a tool for supporting

vulnerable groups. An understanding of the nature of the varieties and

appropriateness of mentoring was gained through the project.2

1.3 Using the learning from MOMIE about what worked well in mentoring relationships,

the MEGAN project sought to test the transferability of the mentoring model and its

impact on a range of aspects of social inclusion. This further social experimentation

will test theories on what forms of mentoring is most effective and appropriate for

different groups.

1.4 The MEGAN project was delivered in three EU member states (UK, Hungary and

Portugal) and focused on migrant communities and their descendents facing multiple

and complex barriers to the labour market, social exclusion and isolation. The

project was designed to address the issues faced by different groups of

disadvantaged people, with a range of social needs.

1.5 MEGAN was funded by the European Commission DG Employment, Social Affairs and

Inclusion and it supports the Europe 2020 flagship initiatives of the 'Innovation

Union' and the 'Platform against Poverty and Social Exclusion'. It also compliments

the EU Youth Strategy; the Roma Youth Action Plan; the Action Plan on Improving

the Situation of Roma within the OSCE area; and a high number of directives on the

equal treatment and integration of migrants within the EU.

1.6 The project sought to address the key challenges of employment, education, social

inclusion and innovation by developing and testing a socially innovative approach to

policy priorities.

2 See Finnegan, L et al (2012) Models of Mentoring for Inclusion and Employment (MOMIE): Final

project report

17

Aims and objectives

1.7 The main objective of the MEGAN project was to pilot and evaluate mentoring as a

method of intervention for socially excluded individuals and networks. The specific

objectives of this project were to:

Test the impact of mentoring on a range of hard and soft outcomes for

vulnerable migrant communities;

Test the feasibility of conducting a randomised control trial of mentoring;

Promote effective and cost efficient strategies to support the social inclusion and

employment of vulnerable groups;

Enhance the social inclusion of migrant communities through mentoring.

1.8 The project tested not only the impact of mentoring interventions; but also the

feasibility of conducting a randomised control trial to measure any impacts.

1.9 It is important to be clear upfront that, whilst this project is aimed at evaluating

mentoring as a method of intervention for all socially excluded individuals and

networks, we are in fact only able to draw conclusions about within the contexts

they are delivered. As we will see, the three MEGAN pilots were very different in the

design and emphasis. Therefore what this evaluation aims to do is to learn lessons

about the delivery and impact of mentoring across a range of settings. This will

enable us to understand the transferability of mentoring and to maximise its

effectiveness in a range of contexts.

Defining mentoring

1.10 In order to explore the impact of mentoring, we first need to define mentoring itself.

Tolan et al (2008)3 describe mentoring as having four characteristics:

interaction between two individuals over an extended period of time

inequality of experience, knowledge, or power between the mentor and mentee

(recipient), with the mentor possessing the greater share

the mentee is in a position to imitate and benefit from the knowledge, skill, ability,

or experience of the mentor

3 Tolan et al. (2008) Mentoring Interventions to Affect Juvenile Delinquency and Associated Problems,

The Campbell Collaboration

18

the absence of the role inequality that typifies other helping relationships and is

marked by professional training, certification, or predetermined status differences

such as parent-child or teacher-student relationships.

1.11 'Peer mentoring' can be said to incorporate all aspects of the definition of mentoring

outlined above. However, the 'peer' element of the intervention is open to

interpretation. The United Nations Office on Drugs and Crime (2002) uses this

definition to describe 'peer education':

'The use of same age or same background educators to convey educational

messages to a target group. Peer educators work by endorsing “healthy”

norms, beliefs and behaviours within their own peer group or community and

challenging those who are “unhealthy”.'

1.12 The difference between mentoring and peer mentoring can therefore be described

as the mentors being of 'the same age or same background' as their mentees. One

feature of peer mentoring would be for the mentor to have been in a similar

situation to their mentee, which corresponds to the 'same background' element of

the United Nations Office on Drugs and Crime's definition.

1.13 Of the three MEGAN pilots, two are using peer mentoring models: Portugal and

Hungary; and one, the United Kingdom, is using a non-peer mentoring model. In

defining peer mentoring this project considered the definition above and determined

'peer mentoring' to be 'mentoring' as described by Tolan et al (2008) by mentors

who are of the same age; and/or who have been in a similar situation; and/or come

from a similar background as their mentee.

MEGAN partnership

1.14 The MEGAN project was delivered in partnership by a range of public and voluntary

and community sector organisations and social enterprises across the UK, Hungary

and Portugal.

Lead partner

1.15 The project was led by the UK National Offender Management Service (NOMS), an

agency of the Ministry of Justice. As lead partner, NOMS had overall responsibility for

the project’s management and administration and for supporting all projects in

transnational and dissemination activity. NOMS had been actively exploring the

potential of mentoring in a criminal justice context for some time. NOMS took a

strategic approach, pro-actively assessing the potential of mentoring, developing the

evidence and testing new approaches. It actively supported the project from the

outset both financially and by providing professional support and commitment.

19

Evaluation partner

1.16 The Centre for Economic and Social Inclusion acted as evaluation partner on the

MEGAN project. Inclusion is a leading centre for research, analysis and public policy,

particularly in the fields of welfare to work, criminal justice, employment and skills.

Inclusion has a dedicated research team, with expertise in qualitative and

quantitative research, evidence reviews and impact assessment. Inclusion has a

broad range of knowledge across the criminal justice field and in support services for

disadvantaged groups, including offenders.

Delivery partners

1.17 Each MEGAN pilot was administered by a coordinating organisation which was

responsible for the recruitment and training of mentors, recruitment of volunteers to

be potential beneficiaries and ongoing coordination of the project including

supervising and supporting mentors and mentees.

UK Partner - London Probation Trust and Praxis: Foreign national offenders

1.18 The UK pilot took place in London and was run in partnership between two

organisations: Praxis and the London Probation Trust.

1.19 The London Probation Trust (LPT) ceased operation in May 2014. It was responsible

for offenders who had received community sentences or those released from prison

early to serve the remainder of their sentence on license in the community. In the

MEGAN project, LPT were responsible for referring mentees onto the pilot from the

cohort of foreign national offenders they managed. LPT Probation Officers were

responsible for referring their clients on to Praxis. In June 2014 London Probation

Trust became the London Community Rehabilitation Company.

1.20 Praxis is a charitable organisation working with vulnerable migrants who find it

difficult to settle in the UK. They work with refugees, asylum seekers, those with no

recourse to public funds due to their immigration status and foreign national

offenders. The latter group were the focus of the MEGAN project. Praxis was

responsible for recruiting and training the mentors; matching mentors to referred

mentees; and managing the mentoring relationship

Hungarian Partner - BAGázs: Roma community

1.21 The Hungarian pilot was delivered by BAGázs, a not-for-profit organisation (‘Public

Benefit Organisation’) founded in 2010 by then-members of the Hungarian Probation

Service who were delivering the forerunner MOMIE project in the village of Bag. Bag

is a village of about 4,500 inhabitants around 40km outside of Budapest, with a

20

segregated settlement of about 500 Roma people. This settlement has very high

levels of criminal behaviour, drug and alcohol dependency, unemployment, lack of

qualifications and poverty.

1.22 BAGázs exists to support those living in the Roma part of Bag to achieve their goals;

obtain new skills and knowledge; (re)engage with employment and other activities;

and develop new methods of conflict resolution. It does this through a mixture of

mentoring programmes; the organisation of non-formal education through

workshops, summer camps and cultural activities; and by representing the needs

and values of the Roma people in the wider Hungarian context.

Portuguese Partner- Aproximar CRL and Santa Casa Misericórdia da Amadora (Pendulum Consortium): Families on social benefits

1.23 The Portuguese pilot was delivered by the Pendulum Consortium, a partnership of

two organisations, Aproximar CRL and Santa Casa Misericórdia da Amadora (SCMA).

1.24 Aproximar, CRL is a social cooperative founded in 2006. It is a communitarian, not-

for-profit organisation based in the Amadora district, near Lisbon, aiming to

empower individuals, groups and organisations to answer with autonomy to their

needs and environment demands by managing personal, social and professional

pathways and delivering innovative projects. Aproximar delivers social innovation

consultancy to other organisations in the third sector, public and private sectors.

Aproximar coordinated MEGAN implementation in Portugal.

1.25 SCMA is a not for profit organisation, founded in 1987 that pursues the Catholic

Church Social Doctrine and is based in the Amadora District close to Lisbon. SCMA

provides social services support according to Amadora community needs, including

health; geriatric; educational and social needs. It also provides social support for

families receiving Minimum Income benefits.

1.26 The MEGAN project is delivered in the Amadora district near Lisbon, an area of high

deprivation that includes neighbourhoods that resemble favelas, with a population of

immigrants from Cape Verde, Sao Tome e Principe, Guinea Bissau, Romania and

Angola, often living in very poor conditions.

Expert partners

1.27 The project was also supported by the following expert partners:

St Giles’ Trust - a ‘train the trainers’ programme was delivered by UK expert

partner the St Giles’ Trust, an organisation with extensive experience in running

mentoring projects. St Giles developed mentor training materials and also

21

provided on-going support and advice to each of the projects following the

training.

Dr. Ioan Durnescu – a peer review of the evaluation study has been conducted by

Dr Durnescu from Bucharest University. A peer review is evaluation of scientific,

academic, or professional work by others working in the same field. Ioan

Durnescu is associate professor at the Faculty of Sociology and Social Work and

co-editor of the European Journal of Probation.

Project delivery

1.28 In each pilot, volunteers were recruited or referred to receive mentoring support.

Volunteers were then divided into treatment and control groups by the evaluation

team at Inclusion. The treatment group received mentoring for a period of six

months, in addition to mainstream support from the delivery partner. The control

group received mainstream support from the delivery partner only.

1.29 Delivery partners were given the freedom to design their projects to meet the needs

of their target groups. However, some elements were stipulated in the overall

project design, based on learning from the MOMIE project, and with the aim of

achieving the MEGAN project aims and objectives described above. The mandatory

elements of project delivery were:

Weekly contact between mentors and mentees;

Regular support and supervision for mentors;

Mixed forms of contact between mentor and mentees i.e. 1-2-1 meetings, group

sessions, telephone and email contact.

1.30 Figure 1.1 describes the overall model of project delivery within which delivery

partners designed their own mentoring pilots. More details of pilot delivery can be

found in the following chapters.

Mentor recruitment and training

1.31 In total, 63 mentors were recruited to the project (24 in the UK, 24 in Hungary and

15 in Portugal). Mentor recruitment was relatively unproblematic across the partner

countries, although slower in the UK as mentors were new to the project. In Portugal

and Hungary several mentors had previously been mentors or mentees during the

previous MONIE project.

1.32 The Portuguese pilot recruited peer mentors from their service users. Mentors were

identified based on project coordinators’ previous knowledge of individuals who were

22

committed and had good communication skills as well as those who had experience

of mentoring from the previous MOMIE project.

1.33 In Hungary, peer mentors were recruited from the local community following an

initial community engagement event4 to raise the profile of the pilot in the village.

Mentors were selected on the basis of their motivation and trustworthiness.

Figure 1.1: Overall model of project delivery

1.34 In the UK, a range of targeted approaches were used for mentor recruitment, such

as advertising on volunteering websites and in the local press. In addition, a number

of mentors had volunteered with Praxis in the past and were looking to be involved

in a different capacity. Volunteers were interviewed by the project coordinator prior

to being recruited to the project. Interviews covered the prospective mentors’

motivation for being involved with the project, as well as criminal records checks.

1.35 The non-peer mentoring model was selected to try to maximise the potential pool of

volunteers, as enthusiasm and commitment to the project were deemed the most

important factors for participation. In practice, many of the mentors who

volunteered were from migrant backgrounds and could perhaps also be classed as

peer mentors.

4 A home renovation programme run by Bagazs in conjunction with Prezi.com, a software provider based in Hungary.

23

1.36 The initial two-day, intensive ‘train the trainers’ event was delivered by St Giles Trust

in February 2013. The main aim of the course was “to enable delivery partners to

develop a mentoring project in their own settings by developing their skills and

knowledge and to provide learning and working materials which could be used and

adapted”.

1.37 A key element of the initial training course was creating interaction and developing

support networks between the MEGAN partners and create channels of

communication so that partners could freely share ideas and training resources.

1.38 The training programme for MEGAN built on and developed the course previously

delivered as part of the MOMIE project. As some participants were now experienced

practitioners they were actively involved in the process to create an interactive

learning environment with everyone contributing fully.

1.39 Again building on the foundations laid in the MOMIE project St Giles adapted and

further developed the “Mentoring Pack” originally created by SOVA, Clinks and St

Giles Trust. This meant that every participant took a comprehensive mentoring

guidance pack back to their own partner agencies for use when recruiting and

training their own mentors. Key areas covered in the training and consolidated in the

Mentoring Pack were:

Recruitment & Selection criteria and process (incl. advertising & marketing)

Referral procedures and “matching”.

Support and supervision, sustaining motivation, incentives, roles, rights and

responsibilities, boundaries and risk assessment.

Equality and Diversity – ensuring your service caters to everyone equally.

Health & Safety – Creating a safe environment for Mentors & Mentees

Communication Skills – Effective verbal and non-verbal communication.

Dealing with difficult situations and difficult clients.

Exit Strategies for Mentors & Mentees.

1.40 Delivery partners reported that they found the training event useful and appreciated

being actively involved in the event as many had been delivering mentoring for a

number of years. The content and pace of the course was deemed appropriate and

there was opportunity to review the delivery of the training programme at a later

stage when the partner organisations visited a prison project in HMP High Down and

met peer mentors there in 2014.

24

1.41 Each of the projects trained their mentors using materials customised from that

initial event and used a variety of techniques including team-building activities, role

play, training on setting boundaries and specific training for non-peer mentors on the

issues faced by the target group.

Evaluation methodology

1.42 The key objectives of the evaluation were to:

Measure the impact of mentoring on a range of hard (e.g. employment and

reoffending) and soft (e.g. confidence and motivation) outcomes for marginalised

communities;

Test the feasibility of conducting a randomised control trial of mentoring to

identify impacts.

1.43 The evaluation consisted of primary and secondary research to ensure learning was

taken from existing evidence and applied in this project. An overview of the

evaluation methodology is provided below. Full details can be found in the technical

appendix to this report. The methodology comprised the following strands :

Thematic review of existing evidence

Impact analysis relating to hard outcomes

Distance travelled measurement of soft outcomes

Qualitative interviews on project delivery and participant experience.

Thematic evidence review

1.44 Inclusion conducted a review of existing evidence relating to previous mentoring

schemes5. The purpose of the review was to systematically collect and assess recent

quantitative and qualitative findings on the impact of mentoring and peer-mentoring

schemes in order to:

identify the changes that peer mentoring and non-peer mentoring bring about,

examine the research methods used to assess mentoring schemes and how

success and failure are defined and measured

5 Centre for Economic and Social Inclusion (2010) Models of mentoring for inclusion and employment:

Thematic review of existing evidence on mentoring and peer mentoring.

http://stats.cesi.org.uk/MOMIE/Models%20of%20Mentoring%20for%20Inclusion%20and%20Employment_%20A%20review%20of%20exisitng%20evidence.pdf

25

inform the development of the peer mentoring schemes being established by the

transnational partners through MOMIE

contribute to the development of evidence-based success criteria against which

the interventions can be measured

contribute to the broader debate on the effectiveness of peer mentoring schemes.

1.45 Full findings can be found in a separate report (Finnegan and Foster, 2014) but

overall the refresh of the evidence has found that the conclusions and

recommendations made in MOMIE review (Finnegan et al, 2010) stand. Challenges

with the research on the effectiveness and outcomes of mentoring remain, but the

available evidence base does suggest that there is value in continuing to support

mentoring programmes for vulnerable groups.

1.46 The review made the following recommendations:

Ensure that mentoring programmes are designed to take account of research

findings on effective mentoring programme features, including the need to match

mentors and mentees on the basis of background and interests, the need for

mentoring to last for more than 12 months and involve regular contact, and the

need to integrate mentoring with other support.

Aim to identify and monitor a suite of intermediate outcomes alongside the key

outcome(s) and consider soft outcomes alongside hard outcomes.

Use quality standards to ensure all structured mentoring is quality mentoring.

Ensure that there is a positive benefit to participation in mentoring programmes

for the mentor as well as the mentee, for example, through offering skill

development opportunities.

Explore innovations, including group mentoring and technology, which may have

the potential to dramatically increase the supply of mentors and close the

mentoring gap.

Facilitate more connections between mentoring research and practice and utilise

evaluation methods that will address current research gaps, such as longitudinal

studies to assess long-term outcomes from mentoring.

26

Impact Analysis

1.47 MEGAN used a Randomised Control Trial (RCT) methodology to measure the impact

of mentoring on a range of outcomes such as employment and employment related

activities, educational attainment, offending, literacy and health. This involved

randomly assigning the group of potential mentees into either a ‘treatment’ (in

receipt of mentoring) or ‘control’ (not in receipt of mentoring) group. The impact

analysis involved measuring outcomes of the treatment group and a control group in

order to determine the role of mentoring in achieving the stated outcomes. An RCT

is the most robust form of impact analysis possible.

1.48 Outcomes for project participants (treatment and control group) were measured

every three months, during the six months of the project and for six months

following the end of the project. At each stage, researchers monitored key outcomes

for both mentees and control group participants. Outcomes for both groups have

been compared at the end of the project to measure the impact of mentoring.

1.49 This methodology supports the Europe 2020 Innovation Union strategy through a

robust and rigorous evaluation method. However, the substantial challenge of

conducting a robust impact analysis of mentoring projects is evident in the lack of

existing studies. Therefore, this project seeks to test the feasibility of conducting an

RCT to test the impact of mentoring with disadvantaged groups communities. By

adopting the methodology, the project aims to contribute to the ongoing work to

develop and implement research and innovation to address societal challenges.

1.50 To maximise statistical validity, each pilot was requested to recruit and support a

minimum cohort of 30 treatment and 30 control group members; a total of 180

participants. The delivery partners were successful in recruiting a total of 197

participants across the three projects (106 treatment and 91 control group).

1.51 In addition, delivery partners were tasked with recruiting 15 mentors per project. In

total 63 mentors were recruited. A full breakdown of mentors and mentees for each

project is detailed in Figure 1.2 below.

27

Figure 1.2: Mentors and mentees by project

Distance travelled measure

1.52 The evaluation used a distance travelled model in order to measure ‘soft outcomes’.

‘Distance travelled’ is about the changes that occur as people progress along a

pathway, for example towards employment or secure housing. Relevant outcomes

may concern changes in personal or interpersonal skills, often termed soft outcomes.

Soft outcomes such as mental well-being, teamwork and relationships were

measured in the MEGAN project evaluation using a psychometric test. However for a

range of issues the psychometric test results were not robust enough to report (see

Technical Appendix for further details).

Qualitative research

1.53 In order to gather detailed information on the project, depth interviews were also

conducted with mentees and control group members, mentors and project

coordinators.

1.54 Interviews explored the design and delivery of each project and participants’

experiences; asking detailed questions about participants’ lives, support they

received to address any issues they are facing, their reasons for wanting to receive

mentoring and the impact of their participation in the project. Mentors were asked

Partner Cohort Treatment Control Total

Men 12 10

Women 10 9

Young People 11 10

Mentors 24

Entrepreneurs 22 22

Disadvantaged 16 16

Mentors 15

Foreign nationals 35 24 59

Mentors 24

Mentees 106 91 197

Mentors 63

Hungary –

Roma

62

Portugal –

Benefit

claimants

76

UK –

Foreign

national

offenders

Total

28

about their motivation for joining the project, and any benefits they believe have

accrued during the process.

1.55 Depth interviews were conducted in Hungary and Portugal through interpreters over

the course of two-days, at three month intervals for 12 months. During each

evaluation visit, the team conducted up to 30 interviews. In the United Kingdom, the

evaluation team had a sample of contact details for mentors and mentees from

which to recruit. Approximately 250 qualitative in-depth interviews were conducted

with mentees and control group participants quarterly over the evaluation period.

29

2 Project findings

2.1 This chapter brings together findings on the impacts of mentoring across the three

projects and draws on both the qualitative and quantitative elements of the

evaluation. This chapter explores learning from MEGAN in terms of the design of

mentoring programmes, delivering mentoring programmes and how mentoring

affects those that participate – both as mentors and mentees. Detailed findings on

individual pilots are contained in subsequent chapters.

Design of mentoring programmes

2.2 One of the key findings from the MEGAN literature review was that the success of a

mentoring programme depended in large part on the quality of the match between

what the mentor can offer and what the mentee needs in terms of support.

The mentor ‘offer’

2.3 As described above, the definition of mentoring is necessarily a broad one, and thus

exactly what the mentor can offer the mentee is determined by the specific

environment a project operates in, as well as the individuals involved. Each project

had to decide on an individual basis, therefore, what support a mentor could, and

indeed, should offer.

2.4 Overall, the support a mentor is able to offer their mentee is dependent upon a

number of factors:

The mentors’ background, skills and abilities. The mentor offer is heavily

influenced by whether they are a peer, and therefore currently or previously in

similar circumstances to the mentees, or non-peer and usually therefore a

‘professional’ or ‘accomplished’ person who may have skills or insight a ‘peer’

lacks.

The mentors’ motivation for joining the programme. Mentors themselves

will often have a predetermined idea about the skills they can (and want) to offer

to a mentee. This is often closely linked to their reason for volunteering in the first

place

Mentor training and resources. The skills mentors acquire in training and the

access they have to referral to other support will affect what they can do for

mentees.

30

Programme design and objectives. The mentor offer will also be tailored to

the way in which the programme is set up and the outcomes it aims to achieve.

For example in Portugal mentors were working within a set structure towards a

common goal (employment). In the UK, the mentor offer was arguably more fluid

depending on the mentors’ skills and the mentees’ needs.

2.5 Given the above, it is clear that the recruitment of mentors should be a selective

process in which mentors’ characteristics and motivations are discussed to ensure

their suitability for providing support to the target group. Our findings suggest that

mentors should be given as much information and preparation prior to delivery as

mentees. Mentors must be clear about what will be required of them from the start,

both to ensure that the support fits with the objectives and ethos of the programme

but also to give them the opportunity to withdraw should the objectives and/or

responsibilities not fit with their original expectations.

2.6 Mentors in all three pilots were interviewed and/or attended a briefing meeting prior

to joining the MEGAN project. However, there is some evidence that not all were full

prepared for what the mentoring relationship would entail (see paragraph 2.52).

Mentees’ needs

2.7 What the mentor can offer should be taken into consideration during the matching

process. As far as possible the mentor ‘offer’ should be matched to the mentee need

i.e. what the mentee is looking for or needs from the mentoring process.

2.8 Mentees’ needs can vary substantially even when mentees are considered to be from

the same group, for example ‘offenders’ or ‘unemployed people’. In both the

Hungarian and Portuguese pilots the mentee group was further sub-divided (by age

and gender in Hungary and by skills levels in Portugal). However, evidence in later

chapters will show that, even within these groups, mentees needed different types

of support, depending on their personal circumstances and situation in life. For

example in Hungary women with drug or alcohol issues in the family required

different types of support to those who did not.

2.9 Similarly, the reasons that mentees participate in a programme and their motivations

for volunteering may also vary widely. Some mentees are seeking help to resolve a

particular issue (such as employment or housing) whilst others may have a wider

range of ‘unspecified’ needs. For example within the UK programme, some mentees

just wanted someone to talk to as they were isolated without friends and family,

others were seeking help with completing paperwork.

2.10 This implies that, whilst a mentoring programme may be designed to meet a

particular need or targeted at a group with a specific common need, there is likely to

31

be significant variation within the group of mentees and that the programme must

have sufficient flexibility to account for that variability within its overarching purpose.

Models of mentoring

2.11 The evaluation has also identified a ‘typology’ of mentoring as delivered within the

MEGAN project. There appeared to be two prevalent models, which we have

categorised as ‘goal-oriented’ and ‘open-ended’ mentoring. The former is most

associated with the Portuguese model and the Hungarian men’s project. The other

Hungarian pilots were delivered as open-ended mentoring. The UK pilot incorporated

both models.

Goal-oriented mentoring

2.12 Projects deemed to be ‘goal-oriented’ have a specified target that is built into the

project design. For example the Portuguese pilot and Hungarian men’s group project

were both designed around practical issues (work and home improvements

respectively). In some cases it was the mentees who approached mentoring as a

‘goal-oriented’ activity, and enter the process with a need they wanted to address.

For example, in the UK some mentees had a specific issue they wish to address

relating to their immigration status.

2.13 Mentees engaged in ‘goal-oriented’ mentoring described mentoring as support to

help them achieve a singular goal connected to the focus of the programme or their

own need. This can be viewed positively as the mentee is actively working towards

achieving a particular outcome. Having a clear goal can increase the chances that

mentoring will address the specified need (and potentially in a shorter timeframe).

As will be discussed later on, this focus may be the driving factor behind the success

of the Portuguese pilot in achieving employment outcomes.

2.14 However, our findings also suggest that mentees may see less value in goal-oriented

mentoring if the specified ‘need’ is not addressed, even if this is due to factors

outside of the mentor’s control. For example the economic climate or discrimination

against Roma or offenders. In addition, once this need has been met, they may no

longer wish to engage in mentoring. Mentoring relationships may therefore be

shorter in these instances which may be satisfactory, provided that the mentee is

happy with the outcome and the benefits of mentoring are sustained.

2.15 Mentees in ‘goal-oriented’ schemes may be less likely to perceive wider outcomes

outside of that targeted by the programme. It may be that mentees do in fact

improve in these areas without being consciously aware of it but this may be difficult

to capture. Mentees may also be less likely to achieve wider skills development if

mentoring relationships are cut short due to the mentee feeling they have achieved

32

their specific goal. The Portuguese pilot was goal-orientated but mentees were still

able to identify soft outcomes. This is likely to result from the structure of the pilot

which allocated half of mentoring time to themes relating to social and professional

skills and half to wider discussion.

‘Open-ended’ mentoring

2.16 As opposed to ‘goal-oriented’ mentoring, instances of ‘open ended’ mentoring are

those where there are a range of intermediate goals rather than one specific end

goal. In these cases, mentoring tends to focussed on a range of intermediate

outcomes, for example improved communication skills and relationships, with a view

to achieving hard outcomes such as employment in the longer term. Mentees may

be ‘working towards’ hard outcomes, but the mentoring relationship is not

specifically geared towards achieving these outcomes, and mentees may not even be

aware of the ultimate outcome as it is sometimes implied rather than openly stated.

2.17 For example, the young people’s and women’s mentoring projects in Hungary had a

general focus of improvement in life skills with a longer-term (outside of this project)

focus on improving hard outcomes such as educational attainment, literacy and

health. Thus, whilst the MEGAN evaluation aims to measure the success of the

Hungarian pilot in achieving such outcomes, it was not a stated aim of the project

itself.

2.18 Open-ended mentoring can be centred on specific activities; however, there is not a

set target attached to the activity as with ‘goal-oriented’ mentoring. Sessions also

tend to be less structured and may be involve more open discussion between mentor

and mentee about the mentees’ life and any issues that might be concerning them.

In this model, mentees may therefore find they gain a wider range of outcomes and

can access support that is more flexible to respond to changing needs.

2.19 However, mentees in open-ended programmes tend to find it more difficult to

identify the skills that they have gained and may also be less likely to achieve ‘hard’

outcomes. This can be problematic if, as was identified with the UK pilot, mentees

come to an open-ended programme with a specific goal in mind such as achieving

Leave to Remain. In this example, mentees’ expectations were misaligned with the

mentoring model, which may explain some of the difficulties in engaging and

retaining UK mentees. In addition, it is the achievement of hard outcomes that

projects are often required to demonstrate as a condition of current and future

funding. Open-ended mentoring programmes may find it more difficult to

demonstrate their impact and outcomes and may struggle to attract funding as a

result.

33

Mentoring ‘plus’

2.20 Mentoring is often delivered as part of a wider, holistic package of support rather

than as a standalone intervention; this proved to be an effective model of delivery

on the MEGAN project.

2.21 In the UK, mentors worked with mentees who were also accessing support from

Praxis’ professional advisers in relation to immigration and other legal issues. In

Portugal, mentors worked with mentees alongside employment advisers from Santa

Casa Misericordia da Amadora (SCMA) who supported them to find work. In

Hungary, mentoring took place in addition to a number of other small projects such

as a home renovation project for men, reading and writing support for women and a

football team and after school classes for boys.

2.22 Evaluation findings suggest that mentoring and professional support can and should

coexist. Within the MEGAN project mentors played the role of identifying mentees’

needs and supporting them to engage with professional services productively. In this

way mentoring acted as a ‘bridge’ between the mentor and professional services. In

the previous MOMIE study it was found that peer mentors in particular could help to

smooth and quicken the process of engaging with professional services by giving

mentees the confidence to seek and access support.6 The MOMIE study also

emphasised that mentoring could not be a replacement for professional services and

that, to be most effective, delivery organisations must carefully consider how and

where mentoring fits within their model of support.

2.23 Figure 2.1 shows a typical trajectory of the milestones achieved by mentees during

their mentoring ‘journey’ and the role of mentors and professional services within

this journey. Our evaluation findings indicate that a mentor’s role is particularly

important in early engagement and the development of mentees’ attitudes and soft

outcomes but that support from professional services may take a greater precedence

in moving the mentee towards hard outcomes. However, the balance between

mentoring and professional support is fluid and the support offer should reflect the

mentees’ needs.

6 Finnegan, L et al (2012) Models of Mentoring for Inclusion and Employment: Final Report

34

Figure 2.1: The mentoring trajectory

Attitude and behaviour

Soft skills (e.g.

communication)

Basic /hard skills (e.g. Literacy/ numeracy)

Hard outcomes (e.g. employment)

INFORMAL MENTORING

PROFESSIONAL SERVICES

2.24 The evaluation did find something distinct about mentoring compared to other types

of formal and informal relationships. In Portugal, mentees were often able to

articulate quite clearly how mentoring had an impact on them as distinct to, or in

addition to, the general support provided by SCMA to find work. One mentee

described the mentor relationship as more ‘intensive’ than the relationship with

professional advisers. This was similar in the UK. In Hungary the distinction was less

clear but in the women’s scheme mentees were perhaps confiding in mentors about

issues that they would have been reluctant to share with other members of the

community.

2.25 The relationships between the mentor and the delivery organisation and/or referral

organisation can have both a positive and negative impact on outcomes. The MOMIE

project found that community organisations tended to have better engagement and

sustainment rates than statutory organisations, based on more trust-based and

informal relationships. This is important as close links with the delivery organisation

can motivate mentees to participate in mentoring and improve outcomes. It was a

challenge for some MEGAN projects to ensure that mentees understood that

mentoring was voluntary and not connected to any negative consequences.

35

Impact of mentoring on mentees

2.26 The findings of the evaluation indicate that mentoring can have a positive effect on

both soft and hard outcomes. The predecessor MOMIE project identified positive

effects on soft outcomes, such as improvements in mentee attitude and behaviour

and improvements in interpersonal relationships and integration into the community.

2.27 As discussed above, mentoring is commonly delivered in addition to other support.

The nature of mentoring as an integrated part of a package of services has, to date,

made it difficult to isolate the impact of mentoring specifically. This is why a

randomised control trial approach was judged to be appropriate to remove the ‘white

noise’ caused by other forms of support, which make it very difficult to attribute

outcomes to mentoring specifically. In this evaluation, control group members also

had access to the additional support and activities offered by the partner

organisations. Therefore it should be possible to identify the additional impact of

mentoring.

2.28 However, it is also important to be realistic about what mentoring can achieve,

particularly in the context of wider economic difficulties, structural change and

widespread discrimination; as was present in the different forms in the three partner

countries. This is discussed in more detail in the country specific chapters.

Soft outcomes

2.29 One of the main aims of the MEGAN pilot projects was to address social exclusion

amongst “excluded groups and networks”. The qualitative findings of the evaluation

demonstrate that the three pilots were successful in addressing issues of social

isolation amongst their target groups through tackling complex barriers and

improving mentees’ soft skills. This was achieved through a combination of both

mentoring and professional support but the role of the mentor was to act as an

outlet and many mentees benefitted from simply having someone to talk to for the

first time. Mentees often did not have friends, family or others that they could

discuss matters with in a constructive manner. The introduction of a mentor, whose

role was to serve this very function, had a marked impact on many mentees.

2.30 The specific role of mentoring in achieving soft outcomes was as follows:

Mentees gained the skills, confidence and resilience to cope on their own and

manage their lives more effectively. Mentees described feeling more positive

about their futures, better able to cope with rejection (for example, in job

applications) and better able to manage emotions and relationships. From an

evaluation point of view, we found that interviewees’ confidence and

communication skills improved over the duration of the qualitative research.

36

Mentoring empowered mentees by giving them a forum in which to discuss the

problems they were facing and to receive emotional and practical support to

tackle their problems.

Mentoring provided a gateway to professional support and services which could

improve mentees’ material circumstances and wellbeing, either by signposting

mentees to services/support or making contact on mentees’ behalf. Mentoring

also helped to equip them with the skills they needed to get the most out of that

involvement with support/services by improving their communication skills.

Mentoring created social networks amongst those who were socially isolated,

thereby improving their community and wider social links. This is evidenced by the

fact that once mentoring relationship officially ended mentors and mentees

continued to see each other informally and many reported that they would start

seeing each other again once the imposed evaluation embargo ended7.

In situations where individuals were isolated from wider society, mentees formed

relationships with individuals and services they would likely have never have

otherwise come into contact. This helped them to become more engaged in

mainstream society. This was particularly the case in the UK. When was delivered

within closed communities, as in Hungary and Portugal, there is evidence that

mentoring has played a role in empowering and preparing the community to

engage with wider society, for example through public speaking at conferences,

engaging with services outside of the immediate community, socialising in areas

outside of their immediate location, and producing media to highlight their

circumstances to a broader audience8.

2.31 The evaluation found that the vast majority of mentees involved in the project

enjoyed having a mentor. Conversely, the vast majority of those in the control group

wanted a mentor, even if they did not clearly understand what the benefits might

be. Whilst this is not a quantifiable impact of the project, it does speak as to the

wellbeing of individuals and the merits of these three pilots.

2.32 Qualitative research with mentees in the six months after the mentoring relationship

ended indicated that the soft outcomes achieved through mentoring were sustained

and several mentees had used the skills they had developed to go on to a) achieve

hard outcomes b) empower and support others.

7 Mentoring relationships were stopped for six months in order to measure post-programme outcomes

for the purposes of the evaluation. 8 The Hungarian mentors and mentees produced a video series and blog to highlight the plight of Roma people. This is available at http://nalunkatelepen.blog.hu/.

37

Hard outcomes

2.33 We have analysed the impact of the mentoring treatment on a range of hard

outcomes, controlling for the personal characteristics and starting points of both

treated and control participants. This method isolates the impact of the mentoring

treatment. The details of the analysis are contained in the technical appendix.

2.34 We have sufficient information on both the pilots in Portugal and in Hungary to

conduct an analysis of the impact of the MEGAN project at the 6 and 12-month

points (i.e. project end and six months after project) end on a range of outcomes.

For the UK information on two indicators; employment and recent offending; was

provided for the 12-month point (i.e. six months after project end).

2.35 After discussion, we agreed that the mentoring interventions were significantly

different in the UK, Portugal and Hungary, and therefore a pooled analysis would be

less informative than separate analyses. We have analysed and reported the data

separately by pilot; the results are quite different.

At project end

2.36 At the six-month point, i.e. the end of mentoring relationships, there were no

statistically significant differences between the treatment and control groups in

either Hungary or Portugal. In Portugal there were early signs that the treatment

group may do better than the control group in terms of employment with slightly

higher number in work. However the difference was minimal and the majority of

mentees were not in work at that stage.

2.37 In Hungary more of the treatment group were in employment than the control

group. This was largely to due the introduction of a government project aimed at the

integration of the Roma community through employment. Whilst open to both

treatment and control groups, mentees appeared to be better able to benefit from

these opportunities. Although these differences in outcomes were observed, small

sample sizes meant that when the results were not statistically significant.

Six months post project

2.38 At the twelve-month point, i.e. six months after the end of mentoring relationships,

the analysis results showed that the MEGAN mentoring intervention in Portugal

produced statistically significant positive effects on two outcome variables - paid

employment in the last three months and budgeting and management of household

finances. This means that we can be confident that higher rates of employment and

budgeting skills observed amongst mentees, when compared to those in the control

group who did not receive mentoring, are a result of the MEGAN mentoring project.

38

For the first time we are able to The employment outcomes were largely found to

have occurred amongst the entrepreneurial group which is unsurprising given that

they were objectively more employable i.e. higher skill levels and better Portuguese

language ability.

2.39 The fact that this difference was not observed at the end of the project may suggest

a ‘lock-in’ effect of the Portuguese model which means that the intensive nature of

the pilot prevented or delayed mentees from seeking work whilst mentoring was

ongoing9.

2.40 The 12-month analysis for Hungary showed no significant change on any outcome

measure between control and treatment groups. The previous differences in

employment between mentees and the control group had disappeared, to coincide

with the ending of the previously mentioned government project. Mentees did not

appear to have been able to capitalise on that experience to find other employment

and project coordinators reported a general lack of motivation amongst those who

had lost their jobs.

2.41 The 12-month UK analysis showed no significant impact on either of the two

outcome measures that were tested i.e. employment and offending.

Impacts of mentoring models

2.42 These findings suggest that the most measurable impacts of mentoring were

observed where the pilot had clearly defined objectives i.e. the goal-oriented model.

The employment objective of the Portuguese pilot was well articulated, delivery was

structured around that objective and mentoring relationships were primarily focused

on meeting that objective. The Portuguese pilot was also well aligned with the

evaluation (and funding criteria) which sought to measure the impact of mentoring

on employment.

2.43 We were unable to identify statistically significant impacts of mentoring for the two

pilots most closely associated with the open-ended model. We know from qualitative

research that mentees in the Hungarian and UK pilots identified positive outcomes in

terms of enhanced community engagement, more positive attitudes/behaviour and

improved parenting skills. Over a longer time period these may have been translated

into measurable hard outcomes such as employment, as illustrated by the trajectory

of mentoring (paragraph 2.23). The Ministry of Justice is actively looking at how

distance travelled or intermediate outcomes can and should be demonstrated in

9 Wilson, T. (2013). Youth Unemployment: Review of Training for Young People with Low

Qualifications. Department for Business, Innovation and Skills.

39

mentoring programmes.10 It was hoped that the MEGAN distance travelled measure

would pick up some of these outcomes but difficulties with data collection has

obviated the use of these results.

2.44 These findings do not suggest that one model of mentoring is better or preferable to

another; rather they support the hypothesis that having a clear goal can increase the

chances that mentoring will address the specified need within a relatively short

timeframe.

Longitudinal outcomes – MOMIE to MEGAN

2.45 The Portuguese and Hungarian delivery partners were also involved in the preceding

MOMIE project and many of the project team and mentors/mentees took part in

both projects. The case studies below take a longitudinal look at the experiences of

some people involved in both projects, their journeys and progress. Names have

been changed to protect their anonymity.

Case Study 1, Adela, Portugal

At the time that Adela became a mentee in the MOMIE project, she was not working. She

lived with her family: son, mother and brother. Her mother was ill, suffering from a

malignant tumour and had lost a kidney. Adela had friends with whom she could openly

talk but she always tried not to worry them with her problems.

During the project, Adela did some temporary work and started volunteering. At the end of

the project she was still unemployed but being a MOMIE mentee made her that she was

not alone. She felt more confident because the pilot team believed in her and had not given

up on her.

Adela found a job after MOMIE ended but as MEGAN started her job contract was coming

to an end. She was worried about her financial situation and felt and isolated because her

mother and brother had returned to their home. She knew that she could count on her

friends but it was not the same as support from her mother.

Adela became a Mentor on the MEGAN project. She felt proud that the pilot team believed

she had the skills to help others. She was also grateful because she felt that the invitation

was a sign not to give up and to stay positive, which reinforced her confidence. Acting as a

mentor in MEGAN helped Adela to maintain a positive attitude. She found that time that

she spent preparing the mentoring sessions helped her to think about her own skills and

what she wanted to achieve. By the end of MEGAN, Adela was working as a cook in a

restaurant and attending entrepreneurship training to learn how to start her own business.

10 See NOMS Analytical Summary 2013 “Intermediate outcomes of mentoring interventions: a rapid evidence assessment” for more information.

40

Case Study 2, Bernardo, Portugal

When the MOMIE project started Bernardo was volunteering at a gardening company and

was unemployed. He had some health problems but was feeling quite well. As a MOMIE

mentee Bernardo gained confidence. He began volunteering with older people at the SCMA

which he had always wanted to do.

Bernardo was randomised into the control group on the MEGAN project. He was still

volunteering and completed his basic education. By the time the MEGAN project finished

Bernardo was doing a temporary job in the construction area. However he felt frustrated

because he still did not have the life he wanted and he commented that he felt less

confident than he had done during the MOMIE project.

Case Study 3, Carlota, Portugal

When the MOMIE project started Carlota was unemployed. She was undergoing treatment

for Non-Hodgkin Lymphoma and caring for a young cousin. She was feeling depressed

because her doctors would not allow her to work. As a MOMIE mentee Carlota received

emotional and moral support. She remained unemployed but was managing her condition

and received authorization to work.

When MEGAN began Carlota was volunteering at a nursing home. She was determined to

find a job and felt motivated but with some feelings of anxiety and lacking in confidence.

Being a MEGAN mentee helped Carlota to increase her confidence and to meet new people.

She began working in a restaurant and is now living in another city, in the southern region

of Portugal.

Impact of mentoring on mentors

Role of training, support and supervision

2.46 A key learning point for the design and delivery of successful future mentoring

programmes is that mentors require in-depth training prior to the start of mentoring

relationships. This will equip them with an understanding of some of the issues that

may arise through working with the target group and with the skills to tackle them.

Mentors also require ongoing support throughout the mentoring process to discuss

issues and share solutions, and to ensure continuous skills development. Training

and support was provided by all three pilots in MEGAN.

2.47 Important areas for training and support is likely to include:

Training on target groups and specialist issues (for non-peer mentors);

41

Training in practical skills (particularly for peer mentors);

Training in communication and listening skills;

Support to address mentees’ emotional and practical needs;

Support to access referral routes and additional support where needed.

Outcomes for mentors

2.48 Qualitative findings indicate that mentoring can have a substantial impact on

mentors’ outcomes as well as on the mentee. Mentors developed a wide range of

skills through the training they received at the start of the project. By actively

applying these skills whilst engaging with mentees, mentors were continuously

developing themselves at the same time as mentees.

2.49 Mentors on all three pilots received support from the delivery organisation. This

support was usually focused around problem solving and overcoming challenges,

which are skills, which in turn helped mentors to find opportunities for themselves.

Mentor support meetings tended to be conducted as group sessions with other

mentors, which enabled sharing and learning amongst the group. It also helped to

build networks for peer mentors who may previously have been quite isolated.

Mentors were also supported to achieve personal goals by the project coordinators.

2.50 Pilots also collected data on the employment outcomes for peer mentors. In

Portugal, seven of the peer mentors who were involved in either the MOMIE or the

MEGAN project went on to find employment. In Hungary, one male mentor went on

to secure employment at Habitat for Humanity, the company that helps run the

home renovation programme. Others were working to set up a Business Club in the

settlement to help the residents start and run their own businesses. The UK pilot

involved non-peer mentors who were largely already employed. However, we do

know that following their involvement in mentoring at Praxis: two mentors secured

full-time jobs in the public sector, one secured a temporary job and a further two

secured voluntary work placements.

Peer vs. non peer mentors

2.51 As discussed, all mentors appear to gain personal satisfaction and a sense of worth

from supporting mentees. This was found to particularly be the case for peer

mentors who appeared to benefit as much, if not more, than mentees. Mentors in

the Portugal and Hungary pilots spoke of a reciprocal relationship where they gained

as much from the experience as their mentee, in terms of confidence and self-belief.

In the Hungarian pilot some mentors were felt to have gained more than their

mentees from their involvement in MEGAN, becoming role models in the community.

42

2.52 The MEGAN and MOMIE evaluations both identified that non-peer mentors often had

pre-defined ideas about what skills they could bring to the mentoring relationship

and how they would use them. In the event that this situation did not materialise as

expected non-peer mentors could become disillusioned. The fluid nature of

relationships and the need for flexibility is therefore important to emphasise in pre-

programme training. Non-peer mentors may also require additional in-programme

support and networking opportunities. Non-peer mentors often volunteered to gain

experience of working with the target group as a route to a future job/career. This is

likely to be reflected in the employment outcomes identified for UK mentors.

Conclusions

2.53 The key conclusions from this evaluation fall into two distinct areas: firstly findings

about the impact of the MEGAN project on the mentors and mentees involved and,

secondly, the learning that can be drawn from the MEGAN project in relation to

design and delivery of mentoring programmes more widely. Conclusions relating the

randomised control trial feasibility study are discussed separately in chapter six.

2.54 In terms of the impact of mentoring on mentees, the MEGAN project is significant in

that we have been able to successfully demonstrate that mentoring can have a

statistically significant impact on employment and employment related outcomes.

This is a step forward in building the evidence base and the business case for

mentoring because, as the MEGAN literature review concluded; “During the past

decade, mentoring has become increasingly common as a social intervention aimed

at improving the life chances of disadvantaged people in the absence of positive role

models. There is evidence to suggest that mentoring can benefit people in some

contexts in different ways. However, the evidence to support how far it can achieve

hard outcomes such as gaining employment and reducing recidivism remains

inconclusive.”

2.55 These impacts are achieved through a mentee journey which sees mentors

supporting mentees at the initial stages of engagement with professional support,

acting as a ‘gateway’ and encouraging engagement. Once initial complex barriers

have been addressed, mentoring can assist mentees in developing their soft skills

and moving towards social integration; qualitative research in this evaluation

highlighted improvements in mentees’ confidence, resilience, communication and

social relationships. Mentoring can then go on to support mentees to achieve hard

outcomes such as employment.

2.56 This project also gathered further evidence of the impact that mentoring can have

on the mentors themselves. Non-peer mentors were found to have built on existing

skills in new areas, while peer mentors achieved new practical and personal skills

43

that improved their employability. A number of mentors moved into employment and

others achieved volunteering and training outcomes. As previously identified in the

MOMIE project, mentoring can have a particular impact on peer mentors who often

spoke of having benefitted as much as mentees in terms of their self belief and

social integration.

2.57 However it is important to be realistic about what mentoring can/ should aim to

achieve. The evaluation has demonstrated that mentoring can provide the additional

support required to achieve both soft and hard outcomes but it is clear that this was

within the context of wider barriers (such as the economy and social attitudes)

which perhaps hindered the scale of the outcomes that mentoring was able to

achieve.

2.58 The MEGAN project experience also offers a variety of useful learning opportunities

on which funders and delivery organisations can build future programmes. Key

amongst these is that mentoring is likely to be most effective if goals are embedded

in the project design and delivery. Mentoring is flexible enough to achieve a range of

hard and soft outcomes in any number of settings, as long as objectives are

articulated and understood by mentors and mentees from the start.

2.59 What the mentor can offer should be taken into consideration during the matching

process. As far as possible the mentor ‘offer’ should be matched to the mentee need

i.e. what the mentee is looking for or needs from the mentoring process. Mentors

and mentees need not be necessarily peers but some shared interests and

experiences can help to build the initial rapport.

2.60 Mentoring is often delivered as part of a wider, holistic package of support rather

than as a standalone intervention; this proved to be an effective model of delivery

on the MEGAN project. The additional value of mentoring, over and above what the

delivery organisation could normally offer is the intensity of support a mentor can

provide. Delivery organisations were not able to provide the same intensity and

regularity of support to all their clients to due to limited resources and mentors filled

this gap.

2.61 However, mentors must not be seen as a ‘free’ substitute for delivery staff. Rather,

mentoring and professional support can and should coexist. Within the MEGAN

project mentors played the role of identifying mentees’ needs and supporting them

to engage with professional services productively. Nor is mentoring necessarily a

low-cost option. As identified in the MEGAN literature review, to be successful

upfront investment in mentor training is required as are ongoing reporting, support

and supervision opportunities.

44

3 Pilot findings: Hungary

Pilot design

3.1 The Hungarian pilot was delivered by BAGázs, a not-for-profit organisation (‘Public

Benefit Organisation’) founded in 2010 by then-members of the Hungarian Probation

Service who were delivering the forerunner MOMIE mentoring project in the village

of Bag11. Bag is a village of about 4,500 inhabitants around 40km outside of

Budapest, with a segregated settlement of about 400 Roma people. At the start of

the pilot, the settlement had very high levels of criminal behaviour, drug and alcohol

dependency, unemployment, lack of qualifications and poverty.

3.2 The MEGAN project got underway in Bag in April 2013.

3.3 BAGázs supports those living in the Roma part of Bag to achieve their goals; obtain

new skills and knowledge; (re-)engage with employment and other activities; and

develop new methods of conflict resolution. It does this through a mixture of

mentoring programmes; the organisation of non-formal education through

workshops, summer camps and cultural activities; and by representing the needs

and values of the Roma people in the wider Hungarian context.

3.4 BAGázs was established during the MOMIE project with a single member of staff,

supported by volunteer non-peer mentors (mostly local university students). At the

time of reporting BAGázs had four full-time members of staff and a number of

volunteers to support its various activities in Bag. Mentoring was the biggest part of

the organisation’s activities. However, it also ran a project training high school

students to teach children from the settlement (as part of the students' volunteering

requirements) and another project connected to improving community resources in

Bag.

Pilot delivery

3.5 The Hungarian MEGAN project was delivered in a challenging political and economic

context. The economy had been stagnant since the 2008/09 financial crisis and then

slipped back into recession during 2012, closing off employment opportunities for all.

This is a particular challenge for Hungary’s Roma inhabitants, who have significantly

poorer educational outcomes than the wider population.

11 http://www.bagazs.org/en/

45

3.6 During the mentoring project, a government initiative was introduced whereby

residents had to sign up to an education and employment scheme in exchange for a

higher level of state minimum income. Through this scheme many Roma residents

found ‘employment’ in occupations such as gardening and litter picking in areas

surrounding the village. However despite an economic recovery in 2013, this scheme

was discontinued – leaving many of Bag’s Roma residents without any work. This

change coincided with the end of official mentoring relationships and thus it is

important to acknowledge that it might have had a greater impact on employment

rates amongst mentors/mentees than mentoring relationships ending.

3.7 As a result of the lack of educational resources and employment opportunities

available to them, Roma people in Hungary are more likely to live in poverty, have

poor health and to engage in criminal behaviour. They also face widespread

discrimination and even hatred in Hungarian society, with many living in segregated

communities that resemble shanty towns and educated separately in state schools.12

This further closes off opportunities for Roma people to improve their living

standards.

3.8 Culturally, the Hungarian Roma live in relatively atomised communities that regard

outsiders with deep suspicion.13 In recent decades, European Roma have

increasingly engaged with evangelical religious movements, with many creating their

own churches or engaging with missionary organisations.14

3.9 The EU Roma strategy for Hungary15 is “not aimed specifically at Roma but at all

vulnerable groups” however it does acknowledge that Roma do face poverty and

social exclusion. The work BAGázs do has a particular focus on Roma as they believe

they face widespread discrimination in Hungarian society. In keeping with the EU

strategy they place a focus on involving the Roma themselves in their work as well

as promoting greater engagement between the Roma settlement and the village of

Bag as a whole.

Delivery challenges

3.10 The main challenge facing the delivery organisation in Hungary was to motivate the

local people, particularly adults, in the pilot. There was a certain distrust of outsiders

from adults in the community to begin with, as the MOMIE project had only worked

12 The Economist (2013) Roma in Hungary: A terrible waste of human potential: http://www.economist.com/blogs/easternapproaches/2013/08/roma-hungary 13 Ibid. 14 Hughes, D. (2009) Growing marginalisation of Hungary's Roma, BBC Radio 4:

http://news.bbc.co.uk/1/hi/8227099.stm 15 National strategy for Roma integration, Hungary. http://ec.europa.eu/justice/discrimination/roma-integration/hungary/national-strategy/national_en.htm

46

with youngsters, and thus mentoring was somewhat unproven amongst older

individuals.

“The biggest challenge was to make them believe change is possible… a lot of

people felt hopeless in the settlement” (Pilot Co-ordinator)

3.11 To combat this, pilot co-ordinators invested a substantial amount of time in the

settlement; visibly showing their commitment and support to the community. BAGázs

staff were in the settlement almost every day of the week and over time, they

established the trust and buy-in of the community to the mentoring project.

3.12 Another decision informed by the findings from MOMIE was to concentrate more on

the wider community, to secure as large an impact as possible. During the MOMIE

pilot the co-ordinators felt that mentors gained as much if not more than mentees,

and so MEGAN focussed on projects that benefit the entire settlement. For example,

pilot co-ordinators reported that embedding mentoring alongside the home

renovation programme, which produced visible changes in the community, helped

engage adults in the settlement with the mentoring project.

Target group

3.13 Participants in the Hungarian pilot were split into three groups: men, women and

young boys. Each of these groups had a defined focus:

The men’s mentoring scheme was aligned with a home improvements project

that BAGázs ran in partnership with Habitat for Humanity16. Mentors and mentees

(alongside participants who are members of the control group) worked in a team

on home improvement projects, to earn resource credits or discounted access to

renovation materials in return for their participation. The men were aged between

18 and 52 and were likely to have a number of children; approximately a quarter

of them were in work at the start of the pilot.

The women’s mentoring scheme focused on helping mentees to support their

children’s education and behaviour, and develop their own reading and writing

skills and confidence and motivation. This activity complemented ‘women’s clubs’

that BAGázs ran with Roma women in the settlement. Women’s clubs focused on

reading and writing skills and other challenges that the women were facing; and

were open to all female residents in the settlement, including control group

mentees. The women were aged between 21 and 53 and had between one and

16 Habitat for Humanity is an international development charity that aims to break the cycle of poverty by

eliminating poverty housing and homelessness. They started working with the Roma people of Bag in 2011 to improve the standard of housing in the settlement.

47

ten children at the start of the pilot. Very few of the women worked at baseline as

most stayed at home to look after the house and children.

The young people’s mentoring scheme, targeted boys aged 9-15, and

focused on behaving well, doing well at school and staying away from crime and

drug-use. It was closely aligned with the BAGázs youth football team: mentees

and members of the control group were asked to volunteer for the MEGAN project

as a condition of signing up for the football team; and most mentors were

members of the adult football team and run the youth team’s training sessions.

3.14 The rationale for expanding the target groups in comparison to the MOMIE project

(which only worked with teenage boys) was that BAGázs wanted to increase the

number of people participating in mentoring activities and extend their reach into the

community. The rationale for working with an even younger group of boys was to

engage with them early, as young Roma often slip into drugs and crime at an early

age.

3.15 The Hungarian pilot was successful in recruiting 62 volunteer mentees, 29 of whom

were randomised into the control group, and 33 into the treatment group (i.e. those

in receipt of mentoring support). The 62 are split into the various experimental

groups as shown below:

Figure 3.1: Participants by experimental group

Experimental group Control Treatment Grand Total

Men 10 12 22

Women 9 10 19

Young people 10 11 21

Grand Total 29 33 62

Data on all research participants by experimental group. Base=62.

3.16 Demographic information collected at the beginning of the pilot suggests that adults

randomised into the control group were broadly comparable to the treatment

group:17 they are similar in age, and the adults have comparable levels of

unemployment (shown on the chart below) and offending histories.

17 The interpretation of quantitative information should be treated with a degree of caution throughout, as the number of people participating in each part of the Hungarian MEGAN project is low.

48

Figure 3.1: Length of time since last worked, Hungarian women’s and men’s scheme mentees and control group

0% 20% 40% 60% 80% 100%

Control

Treatment

Curently in work Fewer than 3 months Between 3 and 6 months

Between 6 and 12 months Between 1 and 2 years Between 2 and 5 years

Between 5 and 10 years More than 10 years Never worked

Source: Data on Hungarian adult mentees: control and treatment group, base=41

3.17 For young people, some differences between treatment and control groups could be

observed however as there were only 21 young people on the pilot; this would be

expected in a randomisation process.

3.18 For example, similar numbers had previously truanted from school (4 in the

treatment group and 6 in the control group) yet as a percentage, the gap seems

larger: 60% versus 40%18. Conversely, the treatment group were more likely to

have drunk alcohol in the past although again the differences are not large.

Mentors

3.19 The Hungarian pilot ran a mix of peer and non-peer mentoring models. Based on the

experience of MOMIE, the young mentees were supported mainly by non-peers –

older boys, many of whom are university students from outside of Budapest. Both of

the adult groups were supported by peer mentors. BAGázs project coordinators

reported that in their experience Roma people had difficulty trusting outsiders, so it

was felt that a peer-support model would have a greater impact for the adults.

3.20 To recruit the adult mentors, BAGázs made use of a project they ran with Prezi.com,

which brought its 120 employees to the settlement in Bag for an intensive

renovation weekend in January 2013. Through this activity, BAGázs engaged a

number of households in the settlement and was able to learn which adults were

trustworthy and which were responsible enough to be mentors. These were the

individuals that BAGázs approached to participate in MEGAN as mentors.

18 Of those for whom data was collected

49

3.21 To recruit the mentors for the young boys, BAGázs approached former mentors and

mentees from the MOMIE project and members of the football team.

3.22 In total, 24 mentors were recruited: 8 mentors for the women’s group, 9 mentors for

the men’s group and 7 mentors for the young boys.

Mentoring model

3.23 The Hungarian mentoring activities were strongly aligned with other activities that

BAGázs was undertaking in the community: house building for the men, literacy

schemes for the women and football for the young boys. The rationale for this was

both to attract volunteers and maximise participation, and to achieve BAGázs’ wider

goals of bringing the community together and supporting Roma integration.

3.24 In the case of the men’s and young people’s schemes, mentoring itself was quite

focused around the activities of the renovation project and football team

respectively, with those participants who were randomised to the control group and

not assigned mentors also fully participating in these activities. The mentoring

meetings were the only element of support that was additional to activities and

services available to the control group. However, there was some qualitative

evidence in the men’s scheme, that being a treatment group member may have

increased the likelihood of benefitting from the home renovation programme. In this

way mentoring acted as a gateway to other forms of support.

"If you are not part of the project, you cannot have opportunities to learn,

save money, renovate your house." (Mentee)

3.25 In all three strands of the Hungarian pilot, mentors and mentees met formally on a

weekly basis for one hour, 1-2-1 sessions, at a time determined by the mentor.

However, given their proximity and existing relationships, many mentors and

mentees met informally much more frequently than this.

3.26 In formal sessions the discussion was relatively open-ended and tailored around the

mentees needs, concerns and interests – often with a strong focus on practical daily

challenges.

3.27 As well as the mentoring relationships and aligned activities through the renovation

programme, women’s clubs and football team, BAGázs arranged other structured

activities for mentors and mentees. These included a fishing trip for the men and a

cultural excursion for the women. As well as keeping people engaged, this fits with

BAGázs’ overall objective to engage the whole community and represent the

interests of the Roma people outside of the settlement:

50

"We want to have more visibility in the village...it's part of a wider integration

model...people have to know the gypsies better to be more tolerant." (Project

coordinator)

Impact on mentees

Quantitative results

3.28 At this stage it is important to point out that none of the differences outlined

below were statistically significant. Whilst they are suggestive of differences

between the treatment and control groups, we cannot be sure that the differences

are not a result of a random distribution.

3.29 The qualitative research also indicated that the overall package of support BAGázs

offered was more important than the mentoring package for many mentees. This

means we would not expect large differences between the outcomes for those in the

control and treatment group. This is due to the fact that the additional support the

treatment group received, mentoring, may have had a relatively small impact

compared to the overall impact on the community as a result of BAGázs’ wider

programmes. To summarise, the success of BAGázs’ work overall may have

undermined the evaluation’s chances of finding the benefits of mentoring.

3.30 Despite the above, quantitative findings suggest that:

Higher employment amongst mentees in the treatment group, particularly males,

although this is dependent on wider labour market conditions and no difference

was found after 12 months;

Higher average earnings for those in employment in the treatment group;

Improved budgeting skills amongst all residents, both the treatment and control

group possibly a result of the wider work BAGázs does in the settlement.

Very little difference between outcomes for young people in the control and

treatment groups, and treatment group mentees may have even experienced

worse outcomes.

Pilot end

3.31 Here we are looking at mentee outcomes after six months, the point where

mentoring relationships were planned to finish and thus we are looking at the effects

of undertaking six months mentoring support. We are concentrating on the

differences between those who received mentoring, the ‘treatment group’, and those

51

who did not receive mentoring, the ‘control group’ to provide an indication of the

impact of mentoring support in this context.

Adults

3.32 The headline finding is that those in the treatment group were more likely to be in

employment and less likely to be in receipt of state benefits. Although these

differences are not statistically significant. This means that the covariates used in the

regression were more powerful at explaining the difference in employment rates

than whether or not an individual had received mentoring.

Figure 3.2: Any paid employment in the past three months?

3.33 0 2 4 6 8 10 12 14 16

Treatment

Control

No Yes

Source: Quantitative data provided by partners at 6 month collection point. Base=39.

Figure 3.2: Currently claiming state benefit? Experimental group

Yes No Grand Total

Treatment 10 12 22

Control 13 4 17

Source: Quantitative data provided by partners at 6 month collection point. Base=39.

3.34 When looking at differences between experimental groups: both men and women in

the treatment group were more likely to be in employment than their control group

participants. However, in general Roma men are much more likely to work than

Roma women and this is reflected in the figures.

3.35 Not only was the treatment group more likely to be in employment, the jobs they

were working in were also likely to be higher paying than those in the control group.

52

Figure 3.3: Average gross weekly earnings of those in employment Experimental group

(HUF)

Treatment 20,000 Control 17,100 Total 19,033

Source: Quantitative data provided by partners at 6 month collection point. Base=21 in work.

3.36 In terms of non-employment outcomes, six months of mentoring appeared to have

less of an effect on drugs and crime outcomes. For crime, this was because no

respondents reported that they had received any criminal offences in the past three

months, whereas similar numbers in the control and treatment groups reported

using drugs in the past three months.

Figure 3.4: Any drug-taking or alcohol abuse in past three months? Experimental group

No Yes Grand Total

Treatment 14 8 22

Control 10 7 17

Grand Total 24 15 39

Source: Quantitative data provided by partners at 6 month collection point. Base=39.

3.37 There was also some evidence that budgeting skills, the condition of houses and

children’s attendance at school had improved over the period. However, these skills

had also improved for the control group which highlights another key feature of the

pilot. BAGázs’ work did not just help those with a mentor but also affected the entire

settlement as it was incorporated into their wider programmes. This was a key

finding from qualitative observations. There is evidence from the qualitative work,

for example, that the home renovation programme, which taught the men to think

more about the future, also helped them budget more effectively.

53

Figure 3.3: In the past three months has your budgeting and management of household finances improved?

0 5 10 15 20 25

Treatment

Control

Frequency

Improved Stayed the same Got worse

Source: Quantitative data provided by partners at 6 month collection point. Base=39.

3.38 After six months of mentoring, over half of all those involved in the mentoring

project, both treatment and control, had improved their budgeting skills over the

previous quarter. The treatment group did not significantly outperform those in the

control group.

Young People

3.39 On most measures, the treatment group partook in more ‘negative’ activities than

the control group. For example, when looking at the numbers who have truanted

from school in the past three months – those in the treatment group are more likely

to report having done so. Nobody reported having been excluded from school.

Figure 3.5: Have you truanted from school in the past three months? Experimental group Yes No Grand Total Control 1 9 10 Treatment 6 5 11 Grand Total 7 14 21

Source: Quantitative data provided by partners at 6 month collection point. Base=39.

3.40 Similarly, looking at the numbers who had drunk alcohol in the past three months,

those who were in the treatment group were more likely to have done so. However,

those with a mentor were less likely to have been in trouble or a fight at school in

the past three months.

3.41 There are a number of possible explanations for these results, however due to the

small sample sizes, these results are not statistically significant and are not

necessarily due to mentoring they could just be random distributions. One possible

explanation is that mentoring had a negative impact on mentees, although there was

54

no qualitative evidence to support this. Another is that a readiness to admit to

truanting is in fact a positive impact, as accepting a problem can be the first step to

solving it. However, we also found no qualitative evidence to support this

hypothesis.

Six months after project end

3.42 This section of the report sets out the impact of the mentoring project after 12

months, six months after the mentoring relationships finished, to investigate whether

or not mentoring had any sustained impacts on mentees. In the Hungarian pilot we

believe, based on our qualitative research, that most mentoring contact stopped

after six months. This was made easier by the fact this coincided with winter in Bag

when activity in the village slows, and many residents stay inside and concentrate on

heating their homes and feeding their families. However mentors and mentees are

still likely to see each other on an almost daily basis, although not an intensive

mentoring session and we should bear this in mind when interpreting the results.

Adults

3.43 Analysis was undertaken to assess the impact of the mentoring treatment on a range

of outcomes (employment, work experience, education or training) controlling for

the personal characteristics (age, gender, education, employment status at baseline)

of both treated and control participants. This method isolates the impact of the

mentoring treatment.

3.44 The analysis for Hungary showed no significant change between control and

treatment groups on any outcome measure. The coefficients showed a negative

association between treatment and outcome on three measures, but this was

sufficiently close to zero effect that no conclusion should be drawn. For the other

three measures, the analysis proved statistically impossible.

Figure 3.4: Impacts of mentoring on key outcomes: Hungary

55

Source: Management information provided by partners and Inclusion analysis

3.45 Furthermore, looking at overall employment outcomes, there is no difference

between the control and treatment groups six months after mentoring stopped.

Figure 3.5: Have you had any paid employment in the past three months?

0% 20% 40% 60% 80% 100%

Treatment

Control

Yes No

Source: Quantitative data provided by partners at 12 months. Base =20 for the treatment group and 19 for

the control group.

3.46 However, when looking at the gross wage of those in employment – we can see that

those in the treatment group are more likely to be in higher wage employment.

Figure 3.7: Average weekly gross earnings of those in employment

Control Treatment Grand Total

(HUF) 16,050.00 20,318.18 18,285.71

56

Source: Quantitative data provided by partners at 12 months. Base =21 of all those in employment.

3.47 Looking at other positive outcomes such as whether a mentee has entered education

or training, or whether they have taken up any voluntary positions, the evidence is

less clear. Very few mentees, whether in the treatment or control group, have done

any work experience or volunteering or been involved in education or training over

the past few months. As mentioned previously, this pattern is likely to be caused by

government reforms in Hungary at the time. During the six month reporting period,

many residents were involved in a mandatory education scheme which allowed them

access to a higher state benefit, a scheme which had ended by the 12 month

reporting period.

3.48 As our findings above show, there was some evidence that mentees’ budgeting skills

had improved in the six months for which they were being mentored compared to

the control group. Looking at the data for 12 months, we can see that those

improvements did not continue once mentoring relationships had ended (i.e.

budgeting skills had not improved over and above the control group in the past three

months) but they were sustained (i.e. budgeting skills stayed the same and had

not got worse in the past three months for the treatment group).

Figure 3.6: What has happened to your budgeting skills over the past three months?

0% 20% 40% 60% 80% 100%

Control

Treatment

Improved Stayed the same Got worse Data not available

Source: Source: Quantitative data provided by partners at 12 months. Base =22 for the treatment group and

19 for the control group.

3.49 No mentees reported that at the 12 month point their reading or writing had

improved in the past three months. This is a reflection of the priorities of various

projects: in Hungary the emphasis was on home renovation, parental support and

reduced drug and alcohol use. Thus as minimal reading or writing support was

provided, we would not expect improvements in this field. Mentoring may well be an

57

appropriate method to boost reading and writing skills but it was not in this

context19.

Young People

3.50 At 12 months, the quantitative data for young people on the mentoring project

showed very little difference between the treatment and control groups.

3.51 For some indicators, this was because no mentees reported having taken part in

‘negative’ activities: either truanting, being excluded from school or taking drugs in

the past three months.

3.52 Compared to the start of the pilot, and the six month data collection point, this was

however a significant improvement amongst the young people of Bag. Whilst there

was not a difference between the control and the treatment group, there was a large

difference between the start and end of the pilot. This suggests that while mentoring

alone did not have a significant impact on truanting rates of young mentees, the

whole package of support offered by BAGázs may have affected truanting rates in

the settlement.

3.53 For example, a mother may be a mentee in the women’s project but her child in the

control group and thus any positive impacts on the mother’s parenting skills,

encouraging her children to attend school, would appear as an improvement

amongst the control group. This is known in the literature as contamination and

means we are less likely to capture evidence of a mentoring impact. However, this is

not necessarily a negative result because it means that there have been benefits to

all young Roma people in Bag, not just mentees.

3.54 For other indicators, the treatment group actually presented a higher incidence of

‘negative’ activity than the control group.

Figure 3.9: In the past three months have you been in any fights or in trouble at school?

Control Treatment Grand Total

Yes 2 4 6

No 6 3 9

Data not available 2 4 6

Grand Total 10 11 21

19 This was the case in the Portuguese pilot because of the specific focus the project had on developing these skills.

58

Source: Quantitative data provided by partners at 12 months. Base =11 for the treatment group and 10 for

the control group

3.55 As explained with the six month data, this finding could either be an anomalous

result from the data due to small sample sizes or evidence that mentoring has a

negative impact or that admitting to a negative activity is actually a positive impact.

There was no qualitative evidence to support either of these hypotheses.

Qualitative findings

The value of mentoring

3.56 Mentees identified various needs that they thought mentoring could help to address,

and generally viewed their mentors as responsive to these needs. There were

different values placed on mentoring for each of the three experimental groups:

men, women and young boys. Participant in the women’s project were most likely to

comment on the ‘soft’ benefits of mentoring and highly valued these benefits.

3.57 Female mentees reported that they wanted help with their children’s development

and often their own skills in terms of communication skills and confidence building.

Female mentees were particularly positive about the contribution that a mentor

could make in these areas, particularly as many of them felt unable to talk about

their problems previously.

“We can talk about everything...she’s the only one I trust.” (Female mentee)

3.58 Male mentees tended to identify getting the most out of the renovation programme

as their main or singular goal, although some were also focused on getting a job.

They were keen to learn new skills, around carpentry and plumbing, as well as

maximising the number of ‘credits’ they could earn to purchase subsidised building

materials. Project coordinators highlighted that there were large cultural barriers

amongst the men in Bag around sharing personal or family problems and none of

the male mentees reported this as a reason for joining the project.

3.59 Young mentees were quite focused on developing their footballing abilities, but also

acknowledged that a positive older role model could have an influence on their

studies and their behaviour at home and at school. A particular challenge with this

group is their age, with many of the younger ones struggling to understand exactly

what mentoring is and what they are likely to get out of it.

"They understand that it's a good thing to have a mentor, but they don't

really know what mentoring is, because they are very small." (Project

coordinator)

59

3.60 In terms of those placed in the control group, the majority did not demonstrate a

clear understanding of why this was this case or what exactly it meant for them

personally. Many did not express disappointment upon finding out that they would

not have a mentor as they did not fully understand what mentoring was but were

happy to get involved with the wider work that BAGázs does in the settlement.

3.61 Some of the young people commented that they were disappointed to have ‘missed

out’, but were not sure what the benefit of having a mentor would have been, apart

from a vague awareness that mentors could help with schoolwork. This is a message

that may have been communicated by pilot co-ordinators. A few also mentioned that

they would have liked to have one of the older boys to talk to and play football with.

The mentoring relationship

3.62 Most mentoring relationships had been successfully initiated and developed positively

from the start. The majority of mentors and mentees already knew each other

before the project began (some very well), and reported that they had a lot in

common. Both mentors and mentees viewed this as important, particularly as it

allowed them to quickly develop trust, which both viewed as the most essential

aspect of a mentoring relationship.

"I trust him 100 per cent." (Male mentee)

"I only have to say half what I am saying, and he knows what I am likely to

say." (Young mentee, N.B. this is a Hungarian expression that roughly

translates as 'we finish each other's sentences’.)

"Many times they are relatives, and I think that is very important in this kind

of community. Not too close relatives, but already they have some common

things." (Project coordinator)

3.63 However, some mentoring relationships had broken down after a period of time and

others had never started. This tended to be because of a lack of interest from the

mentee in having someone to talk to in a mentoring capacity. Some mentees also

stated that the mentor was not really in a position to help them due to struggling

with their own issues.

3.64 During the project, mentors and mentees tended to meet once a week for an hourly

one-to-one session, in general this was judged to be sufficient contact. For the men

these conversations were based around very practical matters: their upcoming plans

for their houses or employment opportunities in the area. For the boys: these

conversations centred on how they were performing at school, as well as how they

could improve at football.

60

3.65 On the women’s scheme, these conversations were more likely to be personal:

discussing problems at home with their partners, or how their children were getting

on at school and how these issues could be resolved. They were much more likely to

open up than either of the other groups.

"We can talk about everything."(Female mentee)

“I trust her; I tell her things I cannot tell anyone else” (Female mentee)

3.66 Women also commented that having a peer mentor allowed them to establish trust

more quickly, as the mentor was likely to understand their experiences. It was

deemed particularly important for a mentor to have experience of raising children if

they were advising on childcare issues.

The impacts of mentoring

3.67 As much of the evidence we and others have collected, the impacts of mentoring are

closely linked to how a programme is designed and which objectives they hope to

achieve. To this end, the three different mentoring schemes conducted in Hungary

achieved different outcomes amongst their target groups.

Women

3.68 The women’s scheme was the group that most embraced the ‘soft’ benefits of

mentoring, as opposed to harder, more practical benefits. Many women told us how

much they valued having someone to talk to, and that they had no-one else in their

lives who they could discuss their problems with like their mentor. As a result, many

mentees missed seeing their mentor after six months when the mentoring

relationship stopped. In this way, mentoring was able to help them widen their social

networks and develop support structures they did not have previously.

3.69 The control group seemed not to appreciate the value of this relationship as much –

they did not know what they were missing out on. However, a number did report

feeling ‘isolated’ and ‘overwhelmed’ by their problems. The continuing presence of

women’s clubs run by BAGázs, which got the women of the settlement together as

one group helped to provide some support to the women of the control group and

mentees once the mentoring relationship ended. Whilst these did not provide the

same intensive one-to-one support mentoring did, it did offer an arena for women to

share their problems. Again, participants reported that this was hugely beneficial to

the settlement as a whole but from an evaluation point of view, would make the

additional benefit of mentoring difficult to distinguish.

"She pays attention to my problems and she tries to give advice…[whereas]

my family doesn’t." (Female mentee)

61

3.70 Women also commented that they valued the indirect effect mentoring had on their

children. Indeed, the women’s project focused on improving parenting skills and

mentees in particular reported improvements in their children’s attitudes, behaviour

and academic achievement. Female mentees hugely benefitted the support their

mentors gave them in dealing with problems their children may have been

experiencing around attendance at school, or drug or alcohol abuse.

"She is the mentor of my kids as well" (Female mentee)

3.71 Mentoring also appeared to have a positive impact on women’s budgeting skills. This

was particularly important in empowering women to take control of the household

finances for the first time, as Roma men traditionally manage the household budget.

Whilst this was not a particular focus of the mentoring relationship, mentees were

more likely to acknowledge their budgeting skills had improved than those in the

control group.

Men

3.72 Men identified the practical benefits of participating in the housing repair project

(open to treatment and control group) but were less able to distinguish on the

additional benefits of mentoring (that would only be open to the treatment group).

For example, they were grateful for the new practical skills they had learnt about

home renovation and how, for example, to fix their roofs or insulate their houses.

They also valued having someone available to discuss their plans with, especially if

their mentor was particularly experienced.

3.73 As the project progressed, mentees were able to identify some subtler benefits.

Some reported that the project had made them more motivated to spend their time

usefully. Whilst they often focussed on the home renovation programme as opposed

to mentoring, mentees were more likely to be actively involved in this programme

than men in the control group. As with the women’s group, mentoring helped

mentees to widen their social networks which allowed them to benefit from other

programmes BAGázs ran as they were more likely to be actively involved with the

organisation. Furthermore, some mentees were more likely to report that they had

learnt to plan for the future and budget accordingly (although again it is not clear if

this was a benefit of mentoring or the home renovation programme overall).

"I have learnt not to follow my head but to make a list with how much money

I have and what I will need." (Male mentee)

3.74 Following the conclusion of the mentoring project the project coordinator noted that

male mentees were better able to reflect on the skills they had gained through

participating in mentoring. Male mentors have subsequently gone on to become

62

community leaders, representing the Roma community to local government officials

and helping to lead on new projects run by BAGázs.

Young people

3.75 The outcomes for mentees in the boys’ project were the least evident. This may be

due to the fact that mentees and control group members were involved in communal

activities (e.g. playing football) and had access to peer support, so did not

distinguish this from mentoring. Having said this, mentees did comment that they

appreciated having someone who offered help and advice. Furthermore all the

mentees were happy that they had a mentor, and enjoyed spending time with their

mentor even though they could not describe how this had affected other parts of

their lives.

"He [mentor] makes me happy...there is nothing else he could help me with because I

don't have any other problems." (Young mentee)

Relationship with the co-coordinating organisation

3.76 Mentors, mentees and control group participants reported very positive relationships

with BAGázs staff and an appreciation of the work they have done throughout the

settlement. Initial scepticism appeared to have been overcome by the organisation’s

efforts to introduce a range of activities, including mentoring, to the settlement.

BAGázs staff had built trust between themselves and community members by

working closely with and even employing some residents.

"I am really happy that they are here…they are like us." (Male mentee)

3.77 There was also great optimism amongst the community about the programmes

planned for the future: a programme to encourage residents to grow their own food,

the continuation of the home renovation programme and education programmes, a

business start-up programme, and the expansion of the football teams amongst

others.

3.78 There was still evidence of disenchantment amongst some of those not participating

in BAGázs activities. However, this was limited and overall there had been a wide

ranging positive impact on the community.

3.79 As discussed previously, the wider package of support BAGázs offered appeared to

be more valuable to the men and young people than the mentoring offer itself. The

overwhelming majority of men and boys participating in the qualitative research

stated that they were extremely grateful for organising the home renovation

63

programme and football teams in the settlement, but were less able to reflect on the

specific benefits of mentoring.

3.80 Coordinators felt some groups more difficult to work with. For example, men were

more difficult to engage but ultimately saw the benefits of the project.

Impacts on Mentors

3.81 There were numerous examples of mentors in the project making significant gains:

often to a greater degree than the mentees in their charge. These benefits were

closely linked to the other programmes BAGázs run for each cohort.

3.82 Two female mentors, inspired by the literacy programme ran in the settlement,

helped to promote and engage a voluntary education programme for young people

in Bag. Local schoolteachers, and social workers have also asked for their help in

engaging children from more chaotic families in the settlement, and they have done

a lot of work to improve relationships between the Roma and non-Roma residents

overall. The women have greatly benefitted from this new responsibility and sense of

purpose, and one has taken it upon herself to become a local leader for the women

in the settlement. She also helps organise the women to work around the village, as

part of a local government scheme.

"I feel that I get energy and strength out of the relationship. It is a very good

feeling that I can help somebody who needs some support and makes me feel

empowered… I say the truth and what I feel directly, before I would not be

able to do that." (Mentor, 6 months into the mentoring relationship)

3.83 In the men’s scheme, one mentor has gone on to secure employment at Habitat for

Humanity, the company that helps run the home renovation programme. Others are

working to set up a Business Club in the settlement to help the residents start and

run their own businesses.

3.84 The young boys’ mentors have gained valuable, practical experience on football

coaching (which has allowed one to secure a coaching position in Budapest); as well

as a growing sense of maturity that comes with being a role model for others. They

believe that as an individual the younger boys they look up to, they have an

opportunity to influence their lives.

"I came out from the jail and did the mentor programme. Here in the

settlement it’s like this - if someone comes out from jail, the children think ah,

he is God – it’s a very big thing being in jail. My strategy was that once I

realised that I can influence (my mentee), I feel it is a very big power because

in his eyes I am a big person, I thought that I can use this power, and I told

64

him that jail is not a good place, not safe. So that was my strategy" (Mentor

interview)

3.85 Many of the young boys’ mentors in the MEGAN project had previously been

mentees in the forerunner MOMIE project. This supports findings from the MOMIE

evaluation that soft skills and confidence gained through mentoring helped young

people become role models, as many have gone on to become mentors. There was

limited evidence of this from the MEGAN evaluation.

Up scaling and future plans

Plans for mentoring

3.86 BAGázs intends to continue to run a number of mentoring programmes in Bag,

taking the lessons learnt from both MOMIE and MEGAN and continuing their

outreach to the entire Roma community. As this evaluation has shown, they have

found it most effective to carry out separate schemes for different groups of

residents and they will continue with this approach. Another highlight form the

evaluation is that mentoring is most valuable when offered as one part of the

overall support offer BAGázs provides. They will, therefore, continue to run

mentoring amongst all of the other programmes they currently run, as well as a

number of new programmes currently starting.

3.87 For the women’s group, BAGázs intends to develop a more individualised mentoring

programme for each mentee. The pilot co-ordinator intended to meet with each

mentor at the start of the next programme and discuss their mentee’s needs and

design a programme of support accordingly. The fact that delivery staff now have an

intimate knowledge of the settlement and the residents makes this possible, and

they want to work with mentors to develop individual plans for turning mentee’s lives

around.

3.88 They also hope to extend the mentoring project to young women and girls in Bag.

They had already help establish a girl’s football team and hope to use their learning

from the women’s and young boys’ mentoring schemes to design a mentoring

scheme for the girls.

3.89 The men’s mentoring project is being re-designed to include, or attempt to include,

all men in the settlement. The project will focus on the set-up of allotments in the

settlement and will again focus on very practical benefits to the community, as

project coordinators have found this is the best way to engage male members of the

community. Project coordinators were unsure if this project would involve mentoring

as the cultural barriers discouraging men from discussing personal problems are very

large. More information on this project is provided in the section below.

65

3.90 In terms of work with young people in the settlement, there have been a number of

developments. During the course of the evaluation the number of football teams in

Bag (a key component in engaging young people with their work) has grown from

one to four, including a girl’s team.

3.91 BAGázs also intend to increase the level of mentoring offered to the children,

although this time without the presence of a control group. They are proposing a

three-tier mentoring model, with an older group of mentors responsible for the

younger teenagers who in turn will mentor very young children. This is in response

to a recognition that mentors gain as much, if not more, than their mentees and

they hope the young teenage group can make huge gains under this system.

Plans for the organisation

3.92 BAGázs continues to do sterling work helping the residents of Bag to help

themselves and tackle stereotypes about Roma people throughout Hungary. They

intend to produce a short series of videos, to be published online to challenge

prejudices about the Roma community, promoting what the residents of Bag do in

conjunction with BAGázs. They are published at: http://nalunkatelepen.blog.hu/

3.93 However, life is still difficult for residents of the village and in particular has been

affected by wider political movements across Hungary and recent government

decisions. During the course of the evaluation, many of those in the settlement (both

mentees and control group members) lost the part-time work they had been given in

exchange for higher social income payments. Very few residents work in the wider

economy and thus the lack of these public sector ‘jobs’ had a detrimental impact on

the community. BAGázs was working to counteract these effects and a number of

programmes have been planned for the future, in addition to the mentoring plans

outlined above.

3.94 The first will involve BAGázs, in partnership with Prezi20 providing subsidised

gardening materials and offering advice and support to help households grow their

own food. They hope to help the community become more self-sufficient, allowing

them to spend less money on food. However, the benefits are wider than this as

they feel that it can help “develop personality as well as the community.”

3.95 One of the main potential benefits to the community of encouraging gardening will

be to encourage individuals to plan for the long-term. As their resources are scarce,

many are living from one day to the next and not planning for the future but the

very nature of planting seeds that will take months to grow, forces them to do so.

They hope that by encouraging the community to take a longer term outlook they

20 Prezi are a Hungarian software company that has worked with BAGázs in the past.

66

can highlight the importance of immunising their children and ensuring they attend,

and succeed at, school.

3.96 Furthermore, as a project that actively involves the non-Roma residents, they are

simultaneously hoping to encourage greater integration in the village. In the past,

Roma and non-Roma residents in the village hardly ever interacted and there was a

lack of trust and desire to engage by both parties. BAGázs are hoping that this new

project could play a key role in breaking down these barriers and helping the village

as a whole to engage more effectively.

3.97 The second project planned for the future, and targeted at the male residents is to

set up an ‘enterprise club’. This will help to demonstrate how residents can set up

and run their own businesses to help make a living. The first such project has been

set-up between a mentee and a mentor to rear pigs in the settlement.

3.98 Two mentors, who have become team leaders amongst the men’s and women’s

schemes, have also taken positions of responsibility as unofficial mediators between

the settlement and the local government. In partnership with the work of the policy

reference group21 they are helping to promote Roma causes in the wider

environment and integration between the settlement and the other residents of Bag.

Conclusions

3.99 The key overarching finding is that the impacts achieved by mentoring are

dependent on how the programme is designed. The impacts we have found for each

of the three strands in the Hungarian pilot are closely linked to the objectives and

design of each project initially. Furthermore, we have found that the overall package

of support provided by BAGázs could be more beneficial to mentees than the

mentoring relationship itself.

3.100 From a methodological point of view, this means we have not been able to find

statistically significant differences between outcomes for the treatment and control

groups. The fact that the work BAGázs do is so holistic and that mentoring only

works as one element of this means that randomising entire communities or

settlements would be more likely to provide robust evidence. In some ways, the

success of BAGázs’ work overall may have undermined the evaluation’s chances of

finding significant impacts of mentoring.

3.101 However, we have seen that mentoring can have a positive impact on a variety of

outcomes.

21 See LPW’s report

67

3.102 When jobs are available, mentees were more likely to be able to take advantage of

this situation as found by their increased employment after six months – but this is

dependent on jobs being available. Roma people still face significant barriers to

accessing mainstream employment and thus when public sector jobs are withdrawn,

very few are able to secure stable employment. This is the case for mentees as well

who may be closer to the labour market than those in the control group but are still

reliant on wider economic conditions to secure work.

3.103 In terms of softer outcomes, there was clear evidence that the work BAGázs has

done has increased the budgeting skills of residents, and indications that mentoring

may be leading this change. Over the course of the evaluation, budgeting skills

increased for all residents but most quickly for those with a mentor. This is a key aim

of the work BAGázs does and appears to be something they are able to achieve. It

also highlights the value of locating mentoring as part of a wider package of

interventions when working with this target cohort.

3.104 For other soft outcomes, it appeared that male mentees found it more difficult to

discuss their problems with mentors and thus did not make progress with personal

issues. Because of this, it is uncertain whether mentoring will continue for adult

males in Bag. For female mentees, there was substantial qualitative evidence that

mentoring could make a difference and attempting to track, and measure, any

potential effects on female mentees’ families could be an interesting piece of work to

contribute to the literature.

3.105 We were not able to see any ‘hard’ outcomes for mentees on the young person’s

project, however all young mentees were happy that they had a mentor and enjoyed

the time spent in their company. We believe that this is because the alternative

programmes BAGázs run are likely to have more of an impact on young people than

mentoring, and thus the differences between the treatment and control groups are

small.

3.106 The plan to extend the young boy’s project to a ‘three-tier’ system will be interesting

to observe. The evaluations of both the MOMIE and MEGAN projects found that

outcomes for mentors from mentoring are significant. However, we do not know

how the middle group in this three tier system, those who are both mentees and

mentors are likely to benefit from support. Evaluating and tracking their progress

should provide some key evidence on how successful such an intervention can be.

3.107 Overall, the Hungarian pilot has taught us that the objectives of a mentoring

programme need to be explicitly considered at the start, and that mentoring can and

indeed should only work as one part of a holistic package of support.

68

4 Pilot findings: Portugal

Pilot design

Delivery partners

4.1 The Portuguese pilot began delivery in June 2013. The pilot was delivered by the

Pendulum Consortium, a partnership of two organisations, Aproximar CRL and Santa

Casa Misericórdia da Amadora (SCMA).

4.2 Aproximar, CRL is a Social Cooperative founded in 2006. It is a not for profit

Communitarian Organisation based on Amadora, aiming to empower individuals,

groups and organisations by answering their needs and environmental demands with

autonomy. Aproximar manages personal, social and professional pathways and

delivers innovative projects and social innovation consultancy to other organisations

in the third Sector, Public Sector and Private Sector. Its main activities are Project

Management, mentoring development and coordinating the implementation of the

MEGAN project in Portugal.

4.3 SCMA is a not for profit organisation, founded in 1987 that pursues the Catholic

Church Social Doctrine and is based in the Amadora District close to Lisbon. SCMA

provides social services support according to Amadora community needs, including

health; geriatric; educational and social needs. It also provides social support for

families receiving Minimum Income benefits.

4.4 The MEGAN pilot was delivered in the Amadora district near Lisbon, an area of high

deprivation that includes neighbourhoods that resemble favelas, with a population of

immigrants from Cape Verde, Sao Tome e Principe, Guinea Bissau, Romania and

Angola, often living in very poor conditions.

Target group

4.5 Pilot participants volunteering for mentoring were local residents in receipt of the

Minimum Income, a benefit for those living in extreme poverty. The majority were

immigrants who had been in Portugal for a number of years. The majority had also

previously worked in Portugal but had been unemployed for several years. For the

purposes of the pilot volunteers were divided into two groups: (1) Disadvantaged

who tended to have low skills and often low levels of spoken Portuguese and (2)

Entrepreneurial who tended to have higher skills and qualifications and who were

originally thought to be suitable to be supported into self employment. In practice

the support given to the two groups was the same and paid employment tended to

be the main focus.

69

4.6 We have undertaken analysis of the characteristics of the two mentee groups

(disadvantaged and entrepreneurial). Initial analysis found that mentees in the

entrepreneurial group tended to be slightly older, with an average age of 44 versus

39 in the disadvantaged group. They were also more likely to be male. As Figure 5.1

shows, the entrepreneurial group were more highly qualified than the disadvantaged

group.

Figure 5.1 Skills levels of mentees

4.7 We also undertook analysis of the characteristics of the treatment and control

groups to check the validity of the randomisation process. Randomisation took place

on a cohort basis at the start of the pilot, with a further, smaller randomisation

taking place to ‘top up’ the sample following drop outs.

4.8 Analysis found that the treatment and control groups were near-identical which

indicates that the randomisation was good.22 For example figure 5.2 shows that

there was no marked difference between the treatment and control groups in terms

of length of time out of the labour market. However there was a slightly higher

incidence of long-term health conditions or disabilities amongst the treatment group

(26 percent compared to 18 percent for the control group).

22 If the randomisation is good then the characteristics of the treatment and control groups should be identical as they have an equal chance of being selected for either group.

70

Figure 5.2 Labour market history, treatment and control groups

0% 20% 40% 60% 80% 100%

Control

Treatment

Fewer than 3 months Between 6 and 12 months

Between 1 and 2 years Between 2 and 5 years

Between 5 and 10 years More than 10 years

Currently in work

Mentoring model

4.9 The pilot used peer mentors who were also in receipt of the Minimum Income

benefit. The decision to do this was based on the Pendulum vision of empowering

local people. Mentors were selected based on their skills and ability to support others

and were trained in all of the themes of the pilot prior to being matched with their

mentees. They were supported throughout the pilot through regular supervision

sessions with SCMA and Aproximar.

"We wanted to take the risk and have peer mentors, to have the possibility to

incorporate the presence of the peers on our daily interventions and

spreading impact more quickly.” (Project coordinator)

"We think it's important to have people who has crossed or is crossing the

same difficulties as the mentees. We think this will be more efficient." (Project

coordinator)

4.10 There were 15 peer mentors involved with the pilot. The primary source of peer

mentors was a pool of people who had already passed through SCMA’s services, and

stayed in contact. Project coordinators also approached peer mentors from the

preceding MOMIE project23 to see whether they would be willing to participate again.

23 MEGAN builds on learning from a previous EU-funded PROGRESS project, Models of Mentoring for

Inclusion and Employment (MOMIE). The MOMIE project focused on the main variable of peer vs. non

peer mentoring as a tool for supporting vulnerable groups. An understanding of the nature of the varieties and appropriateness and mentoring was gained through the project

71

4.11 Pendulum coordinators undertook the process of matching mentors to mentees,

focusing on entrepreneurial vs. disadvantaged groups, similar levels of education and

looking to match working mentors with short-term unemployed mentees. Project

coordinators commented that ideally they would have liked more employed mentors

to inspire unemployed mentees, but levels of unemployment in Portugal were high,

making this more difficult than expected. Instead they tried to match on aspiration

and past experience, as well as current situation.

4.12 The mentoring model had a defined structure. Mentees engaged in 24 sessions over

the same number weeks. Mentors and mentees met once and week either 1-2-1 or

with one mentor and their two mentees. Meetings were 1½ hrs in which half of the

time of the discussion centred on a topic and the other half of the session was open

for free discussion on the subjects of the mentees’ choice. Topics included

communication, emotional intelligence, cooperation, resilience, volunteering,

entrepreneurship, job search techniques and professionalism. Each topic was

covered over two sessions.

4.13 Mentors used their own individual mechanisms to facilitate sessions with their

mentees, for example one mentor used ‘life stories’ to encourage mentees to explore

the issues and barriers they faced. There was also some informal contact outside of

the pilot.

Pilot delivery

4.14 The MEGAN project was delivered during a very difficult time economic in Portugal.

Unemployment was high and it appeared that the informal economies that

immigrants previously relied upon for work, for example construction, were in

decline. Many of the MEGAN participants lacked the basic skills to obtain work

through more formal routes, such as the ability to write in Portuguese. Mentors were

tasked with equipping mentees with these skills in order to help to broaden their

work search and employment opportunities. Pendulum decided not to have any

particular structure of supervision for the mentees because they wanted the mentors

to play this role and to be the central relationship for mentees.

Relationships between the delivery partner and participants

4.15 Pilot participants (including mentors, mentees and the control group) were very

positive about the support provided to them by SCMA. MEGAN participants receive

substantial support from SCMA to find work, as well as with a range of other needs.

Participants appeared to have limited social networks, and relied heavily on the

support they receive from SCMA. This relationship with SCMA meant participants

may have been more likely to engage with the MEGAN project.

72

"They always tell me that when I have problems I can come here, any time."

(Control group)

4.16 Control group participants spoke positively of the employability support provided by

SCMA which comprised regular attendance at job clubs, access to phone and

internet, training courses and volunteering opportunities both at SCMA offices and in

the community. As peer mentors, mentors were also receiving support from SCMA to

find work and reported similarly positive experiences. Many had been encouraged to

pursue self-employment and were at the initial stages of setting up their own

businesses in areas such as catering and cleaning, however; most were still to find

paid employment.

“I think with the help of Santa Casa it will be possible to arrange a job”

(Control group)

4.17 Both project coordinators and mentees were (in general) able to articulate a

difference between the ‘general’ support provided by SCMA to all MEGAN participants

and the more personalised support offered through mentoring. This is discussed

further below in relation to the impact of mentoring but it is important to note that in

Portugal, mentoring was intended to be additional and integral to the wider package

of support but with the same focus on an eventual employment outcome.

“Mentoring works and is an interesting, complementary intervention for

vulnerable people regarding the intensity of the intervention it can bring, and

the gains of scale that can be brought by formalising the role of peers.”

(Project coordinator)

4.18 In contrast to the MOMIE project, the nature of the relationship between mentees

and SCMA staff during MEGAN was a more formal relationship due to the fact that

participants were reliant on SCMA for their social support regarding state benefits.

Although the relationship between SCMA practitioners and MEGAN participants was a

lot less intensive than it had been in MOMIE. The arrangements under MEGAN may

have some impact on sustainment rates, which is important because, as previously

reported in the MOMIE evaluation, participants’ relationship with the piloting

organisation can have a big impact on willingness to engage and levels of

sustainment.

Delivery challenges

4.19 A number of delivery challenges were identified by the project team but these were

able to be overcome during the course of the project. In relation to mentors there

was an initial issue with the balance between volunteering and the professional

dimension for mentors. This was addressed by scheduling sessions more flexibly and

providing support according to needs. In addition it was found that mentors had very

73

difference levels of knowledge about certain elements of the project – in particular in

relation to the themes that they were asked to cover with mentees (see paragraph

4.12). In response to this the project team provided networking opportunities for

mentors to share knowledge and good practice. Mentors were also provided with

training and supervision to ensure that they were equipped with the skills they

needed to support mentees.

4.20 Pendulum also had less control over mentoring relationships than in previous project

(e.g. mentees were not incentivised or actively encouraged to engage and mentees

were less well known to organisation). This created some issues with the

commitment of mentees, resulting in some non-attendance and drop outs. The

project team responded to this by positively reinforcing the expected benefits of the

project and by supervising mentor/mentee meetings to ensure quality. To increase

the numbers of participants some new mentees were brought into the project – this

was an attempt to ensure sufficient numbers for a robust evaluation.

Mentee outcomes

4.21 This section combines qualitative and quantitative data on the impact of mentoring

on mentees ‘hard’ outcomes such as participation in employment and likelihood of

offending. It also includes data on ‘soft’ outcomes such as managing finances,

building relationships and confidence. Hard outcome data is provided by the

Pendulum project team and soft outcome data is self reported.

Quantitative findings

4.22 This analysis explores the differences between those who received mentoring, the

‘treatment group’, and those who did not receive mentoring, the ‘control group’ to

provide an indication of the impact of mentoring support in this context. Due to

small sample sizes in the Portuguese mentoring project we are not able to analyse

outcomes for the entrepreneurial and disadvantaged groups separately. However,

where relevant, we have indicated how outcomes differ between the two groups.

At project end

4.23 Here we are looking at mentee outcomes after six months, the point where

mentoring relationships were planned to finish and thus we are looking at the effects

of undertaking six months mentoring support. Due to small sample sizes the

differences between treatment and control group outcomes at project end are

not statistically significant. Whilst they are suggestive of differences between the

treatment and control groups, we cannot be sure that the differences are not a

result of a random distribution.

74

4.24 Figure 5.3 suggests that those in the treatment group were more likely to have

worked at some point during the previous three months. Indeed, 32% of those, for

whom we have data, in the treatment group had found employment compared to

19% in the control group. Those in the entrepreneurial group were slightly more

likely to have worked than the disadvantaged group.

Figure 5.3 Participation in paid employment over the 3 months prior to project end

Any paid employment in past three months?

Control Treatment Grand Total

No 30 26 56

Yes 7 12 19

(Data not available) 1

1

Grand Total 38 38 76

4.25 There was no difference between the treatment and control groups regarding the

likelihood of having volunteered or took part in work experience schemes. Qualitative

work suggested that some mentees had found volunteering opportunities through

SCMA or the national employment service as this was a service that these two

organisations could provide, but that mentors were unable to help with directly.

4.26 Mentees were slightly more likely to have undertaken education or training over the

three months prior to the end of the project than those in the control group. Whilst

mentors did not have the power to sign up mentees for courses, they could provide

the motivation for the mentee to identify such an opportunity for themselves and

signpost them to potential opportunities. Qualitative findings indicated that some

mentors researched opportunities and passed these on to their mentees.

4.27 No participants, whether in the treatment or control group, reported having

committed a criminal offence in the past three months or having taken drugs or

abused alcohol in the past three months. There were budgeting improvements seen

amongst the group but these occurred in both the treatment and the control group

although at a slightly higher incidence in the treatment group.

Six months after project end

4.28 Here we are looking at impacts after 12 months, six months after the mentoring

relationships finished, to investigate whether or not mentoring had any sustained

impacts. The analysis showed that the MEGAN intervention in Portugal produced

statistically significant positive effects of the mentoring treatment on two

75

outcome variables - paid employment in the last three months and budgeting and

management of household finances.

4.29 The impacts of the mentoring treatment on the outcome variables is shown in Figure

5.5. The green bars indicate that differences between treatment and control

outcomes are statistically significant at the 5% level.

Figure 5.5: Impacts of mentoring on key outcomes: Portugal

4.30 Six months after the mentoring relationship had ended, the treatment group

were more likely to have been in work over the past 3 months. As may be

expected the majority of those in employment were from the entrepreneurial group

as opposed to the disadvantaged group. This is unsurprising given that they were

objectively more employable i.e. higher skill levels and better Portuguese language

ability.

4.31 Data also suggests that those who were in work in the treatment group, were

earning more with average gross weekly earnings of €107 compared to €92 amongst

the control group. In addition, the analysis also found that the treatment group

were more likely to have improved their budgeting and financial

management skills.

4.32 The fact that this difference was not observed at the end of the project may suggest

a ‘lock-in’ effect of the Portuguese model which means that the intensive nature of

the project prevented or delayed mentees from seeking work whilst mentoring was

76

ongoing. Lock in would not explain why budgeting outcomes would be lagged as we

may expect participants to benefit from better financial management skills with

immediate effect. However it may be that it took individuals some time to put these

skills into practice and to perceive improvements in their behaviour.

4.33 There were no significant differences between the extent of participation in

volunteering or work experience or in education or training in the previous three

months. Reading and writing skills had continued to improve amongst all participants

but particularly those in the treatment group, again this was not a statistically

significant difference.

Qualitative findings

4.34 Mentees were very positive about the mentoring experience overall. At the end of

the project they reported benefits in terms of their confidence, self esteem and

motivation. These were directly attributed to the experience of having been

mentored. Some control group participants also reported positive feelings of

confidence and motivation at the end of the project which they attributed to the help

they had received from SCMA. Control group participants did not tend to feel that

they would have benefitted from a mentor, partly because they were unclear about

the concept and purpose of mentoring but also because they could not see how a

mentor would help them to find work – the overarching goal for the Portuguese

pilot.

4.35 However mentees were often able to articulate quite clearly how mentoring had

impacted them as distinct to, or in addition to, the general support provided by

SCMA to find work. One mentee described the mentor relationship as more

‘intensive’ than the group sessions at SCMA and there were a number of key areas

within which mentoring was reported to have increased mentees’ confidence in their

ability to find work, over and above the control. These were:

An enhanced understanding of the behaviours required to be successful

in applying for jobs: A number of mentees described how mentors had

discussed how to present themselves in the best light; how to dress and speak,

and how to approach an employer or application. One mentee explained that,

although she had found a job by volunteering at SCMA, mentoring had helped her

to understand how to behave to get that job. Similarly another mentee described

how he had learned where to look for a job through the collective sessions at

SCMA but that the mentoring sessions taught him how to behave when he gets

there. As such mentoring has increased mentees’ social capital and employability.

77

“Through the collective sessions at SantaCasa I learn where to go and where

to look for a job but [mentoring] has taught me what to do when I get there.”

(Mentee)

Higher levels of self efficacy and control amongst the treatment group:

Six months after the project ended the two groups were in similar positions, most

still unemployed, doing job search, volunteering and training but the treatment

group appeared to be more positive about their future prospects. They tended to

exhibit a greater sense that there were steps they could take to find work, rather

than it being a matter of pure luck. One mentee described how, despite all the

non-responses to jobs she has applied to, she has the courage not to become de-

motivated. In this sense mentoring appeared to have increased levels of resilience

amongst mentees:

“I’ve got what I need, I have courage, I have strength but I need a job. I

know a job will appear and now I’m ready.” (Mentee)

“Before the mentoring I was feeling very dull and would start crying for no

reason. I stayed in the house and didn't do anything. When I started [being

mentored], because my mentor is very educated and very dynamic, it made

me realise there is always a way to overcome my problems.” (Mentee)

Ability to manage emotions: Although less commonly discussed than the

previous responses to mentoring, a small number of the female mentees

mentioned that mentoring discussions had covered personal issues that were

affecting them and that as a result their relationships with family had improved.

Male mentees appeared to be less likely to discuss wider personal issues with their

mentors but one or two male mentees also alluded to help from their mentors to

‘control their emotions’. As such mentoring appeared to also help individuals to

address wider issues that may be affecting their ability to find work.

'”Before I met [my mentor] I was more worried, more tense but now I am

relaxed” (Mentee)

4.36 The peer mentoring model was well received by the majority of mentors and

mentees. A very small number of mentees commented that there may have been

more value in being mentored by someone in employment. However, the majority of

mentees found substantial value in communicating with someone who had similar

issues to them. Several mentees commented that there was a large amount of

shared learning in their relationship with their mentor due to the similarities in their

experiences. Mentees also felt that they were gaining new knowledge from their

mentor.

78

"With things in common we can talk more directly about things that concern

both of us." (Mentee)

4.37 In terms of those that had found work many had done so through social networks,

which seem to be key in Portugal. None specifically mentioned that they had found a

job through networks that their mentors had introduced them to, however given the

levels of isolation of some of the most disadvantaged MEGAN participants we may

expect that mentoring can also help in this way.

Mentor outcomes

4.38 It is clear the mentors also gained skills and knowledge from the project, particularly

from participating in the MEGAN training and from regular supervision. Mentors were

highly complementary about the support they received and felt that this helped them

to support their mentees. This support was particularly important for discussing any

issues they were having with their mentees. Mentors also found value in talking with

other mentors and finding solutions together. Project coordinators echoed the value

of this and felt it helped to empower the mentors.

"They know that during the supervision process they will have more support

from us. Namely the strategies to overcome challenges and the sharing of

good practice. We are planning to grow together." (Project Coordinator)

4.39 Some mentors felt that their experiences would increase their own chances of

finding work in future because of the additional confidence and courage, as well as

practical skills, they had gained. A number described positive feelings about doing

something useful and ‘giving something back’ to SCMA for the support they had

received. Others had become friends with their mentees and/or other mentors.

“I gave these sessions but I also learned from them.” (Mentor)

4.40 Of the 15 peer mentors involved with the project, seven have started volunteering

since the project ended and three have started on training courses, run in

conjunction with either SCMA or the employment centre. In terms of employment

outcomes, seven of those peer mentors who were involved in either the MOMIE or

the MEGAN project went on to find employment.

Up scaling and future plans

4.41 Although MEGAN has now officially ‘closed’ Pendulum is taking mentoring forward in

its ongoing work. There is another project commencing in autumn 2014 which will

involve four cross-cutting groups: single parent families, young people, the

unemployed, and ethnic minority groups. Mentoring is now embedded as part of

79

their everyday service provision and peer mentoring will be available to these four

groups. The project will be delivered through 2 waves of last four months each and

involve weekly, five-hour interventions. It is hoped that MEGAN mentees will become

mentors within the new project in the same way that MOMIE mentees became part

of MEGAN.

4.42 In addition to their own projects Pendulum is also working to roll out mentoring in

other contexts. This would operate through ‘train the trainer’ and “train mentors”

events to spread mentoring as an approach. This training is about challenging

organisations and empowering them to try mentoring for themselves. Pendulum is in

process of delivering Trainers courses to the Red Cross, the Department of Social

Security and the General Directorate of Prison Services. An online platform delivers

the following features: organisations can speak to trainers, watch videos about

mentoring, and undertake simulations.

4.43 Pendulum has also developed a direct, mentor training package. This training for

mentors is quite intensive, it focuses on empowering them with strategies to support

mentees’ needs, reinforcing communications skills, session management, planning

the journey and attending supervisions. The Department of Social Security and the

General Directorate of Prison Services have commissioned Pendulum to deliver

mentor training. Pendulum also hopes to make training for mentors available via e-

learning.

4.44 Pendulum has been working closely with its MEGAN Policy Reference Group (PRG).

The PRG has provided methodological oversight and input, with a paper about

mentoring presented at the ‘Second World Congress on Resilience’ in Timisoara in

May 2014. The PRG has also taken responsibility for dissemination of good practice

across sectors. Further details of PRG activity can be found in the report of MEGAN’s

PRG activities.

Conclusions

4.45 The Portuguese mentoring pilot was able to achieve employment outcomes for a

vulnerable group of participants in a challenging economic climate. The implications

of this success for the future funding and delivery of mentoring support is significant.

Where mentoring may traditionally have been considered to generate soft outcomes,

this Portuguese model has been found to have achieved both soft and hard

outcomes for its participants.

4.46 In terms of soft outcomes, these have been articulated to include increased self

confidence and resilience in handling work and life situations. Empowering an

individual to help themselves is very important in, achieving a lasting impact from a

time-bound project.

80

4.47 The peer mentoring approach adopted in Portugal is interesting in terms of the

personal journey that some of the peer mentors have experienced from participation

in the predecessor MOMIE project as mentors, to acting as mentors in MEGAN.

Future mentoring programmes may wish to consider the scope to explicitly build in

opportunity for mentee-mentor progression, potentially as an indicator of distance

travelled.

4.48 The learning from the MEGAN pilot is already being used by the Portuguese partners

to develop a mentoring model than can be transferable into other contexts and to

delivery with other groups. This both demonstrates the flexibility of mentoring in

general as a form of support and suggests that the structure of the Portuguese

mentoring model is fluid enough to adapt successfully to different environments.

81

5 Pilot findings: United Kingdom

Pilot design

5.1 The MEGAN project in London started was run through a partnership between two

organisations: Praxis and the London Probation Trust (LPT). The project started in

June 2013 although referrals continued to be accepted onto the project for a further

12 months due to lower than expected referrals. Initially limited to four target

boroughs of London, the project was eventually widened to all of London to combat

this.

5.2 At the time of delivery, LPT was the largest of the 35 probation trusts operating

across England and Wales, and employed more than 3,000 staff supervising more

than 45,000 offenders at any one time. LPT was responsible for offenders who

received community sentences or those released from prison early to serve the

remainder of their sentence on license in the community. For the MEGAN project,

LPT was responsible for referring mentees onto the project from a cohort of Foreign

National offenders (FNOs) managed though probation services in each London

borough. Individual Probation Officers (POs) were responsible for referring their

clients on to Praxis, managing the bulk of the mentoring project.

5.3 Praxis is a charitable organisation working with vulnerable migrants who find it

difficult to settle in the UK. This includes refugees, asylum seekers, those with no

recourse to public funds due to their immigration status and Foreign National

Offenders – the focus of this project. For the MEGAN project, Praxis was responsible

for recruiting and training the mentors; matching mentors to referred mentees; and

managing the mentoring relationships.

5.4 Praxis was chosen by LPT to deliver the MEGAN project because of their experience

with the target cohort. Praxis project coordinators indicated that the FNOs they were

asked to work with on this project were not particularly different from the vulnerable

migrants they worked with on a number of other projects in that they have many of

the same issues, and thus Praxis’ experience was directly relevant.

Pilot delivery

5.5 The main external factor influencing the delivery of the MEGAN project in the UK was

the introduction of the ‘Transforming Rehabilitation’ programme, which has seen

wholesale changes to the delivery of probation services in England and Wales.

5.6 Transforming Rehabilitation is the largest reform to probation services for a number

of decades. Under the reformed system, probation services have been divided into

82

two sections: those offenders who pose the highest risk of serious harm to the public

will continue to be managed by a single National Probation Service, which will form

approximately 30% of all probation services; while the supervision of low-medium

risk offenders will be the responsibility of Community Rehabilitation Companies

(CRCs). CRCs will form the remaining 70% of probation services and will operate

across 21 contract package areas in England and Wales. They will retain current

probation staff but will be managed by market providers24.

5.7 Probation Officers (POs), placed in CRCs, faced an uncertain future operating under

the new providers. This uncertainty led to limited engagement from some POs with

the MEGAN project, increased turnover of probation officers, and reassignment of

offenders on probation. As the data collection procedure in London was reliant on

regular updates from London Probation Trust (LPT), these changes severely affected

the collection of quantitative data for the project. Praxis, however, were keen to

point out that the support they received from LPT staff on the MEGAN project was

above and beyond what they often experience on similar projects. In fact, the

partnership working between Praxis and the probation service was vital for the

successful running of the project. Despite this, the changes to the probation service

undoubtedly affected the project to some extent.

5.8 Another external shift in the policy environment in the UK that has affected the

project is the scaling back of legal aid. Legal aid is used in the UK to help those who

need financial help to pay for legal advice. The legal aid budget has been cut back,

directly affecting a number of mentees on the project who found it extremely

difficult to fight their immigration cases for the right to remain in the UK. This

affected mentoring relationships as the support mentors were able to offer once a

mentee had been refused the right to stay in the country was limited.

Pilot-specific challenges

5.9 From Praxis coordinators’ point of view, one of the principal challenges faced:

difficulties recruiting mentees, was a result of the evaluation methodology itself. The

concept of a randomised control trial, where by definition not all referred mentees

can receive the mentoring ‘treatment’, led to disengagement by some probation

officers resulting in lower than expected referrals.

"Some probation officers could not understand the control group and the

treatment group, they were a bit hostile towards me because I couldn't take

(all) their mentees." (Project co-ordinator)

24 For more details see: https://www.justice.gov.uk/transforming-rehabilitation

83

5.10 This issue was compounded by the wider context the project was delivered in, as

described above, as POs were often working under severe strain. Some became

frustrated when one of the few support offers available to their clients was denied

because they were placed in a ‘control group’. In some cases, this left them unwilling

to refer other clients in the future.

5.11 The fact that the target cohort were transient and dispersed all over London did, on

the other hand, cause a number of issues for both the project and the evaluation. In

direct contrast to the settled populations in the Hungarian and Portuguese studies,

mentees in the UK pilot were harder for the delivery organisation to keep in contact

with if they did not attend mentoring sessions. Poor attendance by mentees was an

issue that affected the delivery of mentoring as well as the evaluation team when

trying to interview participants.

5.12 For this reason there was a higher drop-out rate in the London project than in the

other two projects. In fact, some mentees ‘disappeared’ altogether and were

unreachable to either Praxis or probation staff. The evaluation, therefore, was only

able to speak to those individuals who were continuing to actively engage with their

mentors. When interpreting qualitative findings, it is important to take into account

the fact that the evaluation only spoke to individuals for whom mentoring had been

relatively successful, there were others who dropped-out of the project that were

unreachable. This supports one of our main findings from the evaluation that

mentoring appears to be most effective for those in settled circumstances. Those

mentees living more chaotic, with no permanent place of residence for example,

were more likely to drop out of the project.

Target group

5.13 The target group in this project was FNOs who had been released from prison and

were on a minimum of a six month supervision order with their probation officer.

This group was one of a wider cohort LPT felt could benefit from mentoring support.

The key aims of the project in London were to “evaluate mentoring as a tool for

excluded groups”, to help reduce reoffending amongst vulnerable groups and help

them find employment opportunities. LPT project managers therefore felt that FNOs

were a suitable target group for the MEGAN project. The project aimed to help

mentees appreciate that they have alternative choices to reoffending and offered

support to help them explore those choices.

5.14 MEGAN also viewed by LPT staff as a chance to develop and learn lessons about

joint working between the voluntary, community and social enterprise (VCSE) sector

and probation services. The aim here was to demonstrate the value the VCSE sector

could provide in delivering services for offenders in the future, particularly in the

wake of changes to the structure of probation service.

84

5.15 The mentees referred to Praxis from LPT were an extremely diverse group. For

example, the first 32 mentees referred to the project represented over 20

nationalities from Chile to Congo and Portugal to Palestine.

5.16 The mentees referred to the project all had their own unique immigration positions

and the mentee cohort stretched the entire spectrum of potential statuses. Some

had the right to remain in the country such as European Economic Area (EEA)25

nationals, visa holders, those granted indefinite leave to remain and successful

asylum seekers. However, others were appealing failed asylum cases, or had been

judged illegal immigrants and were facing deportation.

5.17 Looking at the order requirement26, there was a mix between those on community

orders, suspended sentence orders and those released from imprisonment. The

average length of time on an order was around 2 years for both groups: 30 months

for the control group and 23 months for the treatment group – this difference is not

statistically significant.

5.18 Length of time on order is also an indication of risk of harm (classified as either high,

medium or low) which the chart below shows did not differ between the control and

treatment groups.

25 The European Economic Area (EEA) provides for the free movement of goods, persons, services and capital through the European Union (EU) so EEA nationals are free to live and work in the UK. 26 Once judgement has been passed, individuals may be asked to serve community orders, aimed to combine punishment with changing behaviour and making amends through, for example, unpaid work; suspended

sentence orders, which allows defendant’s to serve a period of probation in the community for the length of

their sentence providing they do not break the law for the duration of their order; or face imprisonment when, upon release, they may be required to attend meetings with a supervisor.

85

Figure 5.1: Proportion of treatment and control cohort assigned to each risk level

Source: Data provided by London Probation Trust at baseline. Base= 32

Mentors

5.19 Praxis coordinators made the decision to recruit non-peer mentors for the UK pilot.

This means that the mentors were not drawn from the same population of Foreign

National offenders that mentees were recruited from and thus faced different

circumstances. Despite this, Praxis recruited mentors through a number of their own

existing pathways and, as an organisation focussed on helping vulnerable migrants,

a lot of the people recruited had experience of immigration issues either personally

or professionally. Due to the recruitment pathways, mentors were quite likely to

share certain characteristics with their mentees, thus potentially undermining the

intention that mentors should be non-peers. In particular, mentors and mentees

were likely to share experience of the immigration process in the UK and settling and

finding work as a Foreign National.

5.20 Mentors included those who had volunteered on other Praxis programmes and some

who were undertaking information, advice and guidance training with Praxis, for

whom the practical experience of mentoring a Foreign National offender was

particularly useful.

5.21 In total, Praxis recruited 24 mentors on to the project. The mentors received an

intensive one-day training programme that covered:

The rationale for the project and an introduction to the workings of the probation

service and the mentoring process.

0% 20% 40% 60% 80% 100%

Control

Treatment

High Low Medium N/A (blank)

86

Basic listening and communication skills and role plays of potential conversations

they were likely to have with a mentee.

What to expect in the first meeting and techniques to use if they think things

started to go wrong.

When they should seek external support from Praxis if they felt the mentee had

problems they were not able to resolve. This was particularly important for

working with the client group who were likely to have issues with immigration

hearings, for example. As only qualified personnel are allowed to give

immigration advice in the UK, mentors were given advice on how and when to

refer their mentees on to an assigned external Praxis adviser for further support.

5.22 Praxis reported that the training programme had been particularly successful, and

the mentors concurred with this view. All mentors indicated that they felt equipped

to deal with the problems they were likely to face and felt confident about what

support they were able to offer and when they should refer clients on to other

support organisations.

Mentoring model

5.23 Praxis undertook intense engagement work with Probation Officers (POs) across

London (having started off in three boroughs, the project was eventually extended

across London) throughout the course of the MEGAN project to explain its aims,

objectives and the process involved in referring their clients for mentoring. POs were

given information about who the project was aimed at (Foreign National offenders)

and how it could benefit their clients. There were concerns that mentees were not

sufficiently informed about the project before volunteering and crucially failed to

understand that the project was voluntary. As they were referred by their POs, they

may have felt inclined to join the pilot and unable to refuse an offer their PO made.

This was an issue that arose in the forerunner MOMIE project27.

5.24 Those in the treatment group were then matched to a mentor by Praxis. The

matching process took into account mentor preferences for wanting to share certain

commonalities with mentees (e.g. age, gender, language) as well as mentee reports

provided by POs.

5.25 In their initial meeting, which POs also attended, the mentors and mentees drew up

an action plan setting out what the mentee hoped to achieve. Following this initial

meeting, mentors and mentees went on to meet once a week on average over the

27 Finnegan, L et al (2012) Models of Mentoring for Inclusion and Employment: Final Report

87

course of six months. The mentors also met as a group to discuss the challenges

they were facing as well as to share best practice and techniques that they and their

mentees had found particularly useful.

5.26 It was expected that ‘soft outcomes’ would form the bulk of action plans agreed in

initial meetings but qualitative interviews highlighted that mentees were also

interested in ‘hard’ (and hard to achieve) outcomes from the project such as

overcoming immigration problems and winning the legal right to reside in the UK. All

clients referred to the project, both treatment and control group were given

information about support that was available from Praxis professional advisers to

help with these issues, as mentors are not qualified to offer immigration advice.

5.27 This mismatch between mentees’ needs and the support mentors were able to offer

may have been responsible for the higher drop-out rate in the London project. One

possible explanation for this was that the referral organisation and the delivery

organisation were separate in the London project, which was not the case

elsewhere. Furthermore as both other projects took part in the forerunner MOMIE

project, mentoring was more embedded in the other two projects and its objectives

more widely understood. As we have found elsewhere in this evaluation, the impact

that any mentoring programme can have is strongly linked to how clearly the

objectives are set out at the start of the project. For the reasons outlined above, in

the London project, the objectives were not as clearly defined as elsewhere.

Impact on mentees

Quantitative results

5.28 This analysis explores the differences between those who received mentoring, the

‘treatment group’, and those who did not receive mentoring, the ‘control group’ to

provide an indication of the impact of mentoring support in this context.

Six months after project end

5.29 This section of the report sets out the impact of mentoring after 12 months, six

months after the mentoring relationships finished, to investigate whether or not

mentoring had any sustained impacts on mentees. This is the only data set we have

for the UK. The differences between treatment and control group outcomes at

project end are not statistically significant. Whilst they are suggestive of

differences between the treatment and control groups, we cannot be sure that the

differences are not a result of a random distribution.

5.30 Figure 5.2 shows that there was an apparent small difference between the

employment rates of the treatment group when compared to the control group.

88

However Figure 5.3 shows the results of statistical tests which indicated that UK

analysis showed no significant impact on either of the two outcome measures that

were tested (employment and offending).

Figure 5.2 Any paid employment in the past 3 months?

0% 20% 40% 60% 80% 100%

Yes

No

Control

Treatment

5.31 The analysis on offending outcomes shows a similar picture with some very small

observed differences between treatment and control but when subjected to

statistical tests these differences do not hold up and could have occurred by chance.

Only information on employment and recent offending at the 12-month point was

collected for the UK. Data on other outcomes such as confidence in budgeting was

not available for the UK. This may be where we would expect to see an impact from

an open-ended mentoring programme.

Figure 5.3: Impacts of mentoring on key outcomes: UK

89

Qualitative findings

The value of mentoring

5.32 The value of mentoring in the UK pilot was relative to the needs of mentees with

chaotic lives and extremely difficult circumstances. An initial concern for project

coordinators was how little mentees knew about what to expect or what they could

hope to achieve from mentoring after being referred by their Probation Officer (PO),

thus several mentees entered the pilot with very high expectations of the support

they could receive. The POs were keen to secure as much support as possible for

their client, and the mentees were keen to take advantage of any support on offer.

This combination meant that mentees were not always certain what to expect from

the Praxis project. Whether this was due to a lack of communication between POs

and their clients, or Praxis coordinators and POs is unclear.

5.33 When interviewing mentees, many of them were keen to get any help they could as

they were often in dire situations with no recourse to public funds or right to work.

This left them reliant on friends, family (which many did not have) or faith

organisations and charities to provide food and shelter for them, and for many this

support was not coordinated successfully all of the time. Many were struggling to

avoid reoffending as a result of these circumstances.

"My situation is worse than an animal's, I don't have anywhere to sleep, I

can't work or claim benefit, there is no finance coming to me." (Control group

mentee)

"Life is very stressful at the moment, at least in prison you have your food

and you have your bed." (Control group mentee)

5.34 Furthermore, given the complex nature of many of the problems the mentees were

experiencing, mentor support was not always sufficiently able to help them navigate

the issues they faced. In fact, on immigration matters the Praxis advisers were the

only people who could offer effective support. This support was available to both the

control and treatment groups. Mentors could however add value, signposting

mentees to alternative support.

5.35 Whilst mentors could not give professional advice, for some mentees (especially

those who had difficulties reading English) help understanding and completing

paperwork for an often bureaucratic system was the support they wanted. Given the

target group Praxis were working with, this was an issue for a large number of

mentees.

5.36 For others, simply ”having someone to talk to” was their motivation for signing up

for mentoring. However, this was only useful for those who did not have support

90

networks established in London, for those that did, such support was not an aspect

of mentoring they expressly valued.

5.37 Many mentees were also experiencing problems with outstanding debts they held

from a number of sources: immigration lawyers, landlords, loan sharks, and

borrowing from friends to cover their outgoings. Several wanted support from a

mentor to both help source emergency finance and to consolidate these debts or

negotiate payment holidays with various debtors. As well as struggling to attain

money in the first place, some mentees also had problems managing the money

they were able to attain and needed support to do so.

The mentoring relationship

5.38 Each mentor took a different approach to the exact model of support they provided,

in comparison to, for example, the Portuguese pilot where each session was

structure around a certain topic (see chapter 4). In general, mentors would try to

help solve housing, employment and migration issues mainly through signposting to

other forms of provision and helping mentees to navigate often highly confusing and

bureaucratic systems.

5.39 The extent to which the mentee led discussions and how much mentors decided on

the direction for each session, was decided on a relationship-by-relationship basis.

An advantage of this method was that it allowed mentees to dictate the pace of

what they were learning and may have contributed to building resilience amongst

the mentees.

"For me it depends on the mentee, it depends on where the mentee is.

Sometimes to do an action plan or to write a CV is the best thing to do for the

person, other times [it is] just being here at the regular time, creating a space

to listen to the person. The two cases require a different approach." (Mentor)

5.40 Mentors appeared to benefit from the process and enjoyed the opportunity to

regularly give something back to a vulnerable group of individuals. However, some

mentors reported their frustration at poor attendance from mentees on the project,

especially if this happened on a number of occasions. Praxis coordinators tried to

organise meetings at their offices at the same time every week to try to make these

meetings as easy as possible to attend for the mentees, but mentees were not

always able to attend.

5.41 Mentees were generally positive about the meetings but both they, and project co-

ordinators highlighted the cost of travel as a major barrier to attending meetings.

Praxis was able to reimburse travel expenses but many did not have the finances

available to pay for travel up-front. Mentees were also happy to attend sessions even

91

when they were not ostensibly making any progress towards their goals as they

appreciated that the process of securing work, or the right to work, was likely to be

a lengthy one.

Impacts of mentoring

5.42 The main impacts of mentoring seen in the London project were achieved through

mentors acting as a gateway to other support and signposting mentees to different

organisations that were able to offer them help. Mentors, in conjunction with Praxis,

had a wealth of informal knowledge about how certain legal, housing and

employment systems worked in the UK that they were able to pass on to their

mentees. The mentees, on the other hand, as Foreign Nationals often did not have

this knowledge and thus the support proved very valuable. For example, a number

of mentors were able to call around on behalf of their mentees and find lawyers who

could represent them in immigration cases either through Legal Aid support or for a

fee the mentee was able to pay.

5.43 In one case, a mentor was able to negotiate a payment schedule to pay fees

overdue to their mentee’s lawyer, thereby persuading the lawyer to continue to

represent the mentee and helping them secure their right to work in the UK. Mentors

also advised on housing issues and some negotiated with landlords on behalf of their

mentees when, for example, language may have been a barrier.

“I was able to speak to his landlord and agree that they would pay off the

outstanding amount over a number of months, so that [mentee] and his

family could continue to live in the property.” (Mentor)

5.44 However, it was not always clear how much of this support was additional to

what Praxis was able to offer anyway, to control group participants for example.

Mentors were certainly able to offer more intensive one-to-one support but mentees

often heard about and had greater access to a wider range of Praxis’ services as a

result of attending mentoring sessions, than those in the control group. In this way,

mentors acted as a gateway to accessing other support both internally, provided by

Praxis, and externally, provided by another organisation.

5.45 The extent to which mentors could offer further support was highly dependent on

the result of their mentee’s immigration case. For mentees who were granted the

right to work in the UK, the mentoring relationship could progress to looking at

employment opportunities and progression in work. Mentors helped their mentees

with job applications, CV writing and general interview skills, to help prepare them

for moving into employment.

92

5.46 However, this was only possible once their legal right to work in the country,

something a mentor was largely unable to affect, had been granted. Mentoring

appeared to be most effective once this initial barrier had been overcome,

thereby demonstrating a key finding of the evaluation; that mentoring relationships

appear to be most valuable for those living in settled conditions but facing difficult

circumstances (e.g. social and labour market exclusion). Mentees’ immigration cases

were often highly complex and mentors were not qualified to provide advice on and

thus influence, so they could only offer the full benefit of their services once

immigration cases had been settled.

5.47 Initial obstacles that mentees needed to overcome were not limited to immigration

issues; those with severe drug or alcohol problems also struggled to see the full

benefit of a mentoring relationship, according to project co-ordinators. These

individuals were more likely to live chaotic lives and were often unable to attend

mentoring sessions due to external crises or a lack of organisation. It was suggested

that, similar to immigration issues, mentors did not have long enough to deal with

such complex barriers in weekly sessions.

"I think with alcohol and drugs, six months is a very short time to work on

these things." (Project co-ordinator)

5.48 Mentors and project co-ordinators suggested that an additional form of support to

deal with drug or alcohol issues was needed prior to, or alongside, mentoring.

Relationship with coordinating organisation

5.49 Mentees were happy with the support provided by Praxis both through mentoring

and wider support. Praxis was often said to be one of very few places the mentees

could go for support: through services such as drop-in advice sessions. Some of the

mentees we spoke to were extremely grateful for this.

“They [Praxis and the mentor] were like my two guardian angels." (Mentee)

5.50 Mentors also stated that the support the Praxis offered to them had been useful in

helping them support their mentees. Praxis staff researched topics on behalf of

mentors such as the immigration system or English classes for Foreign Nationals. In

the same way that mentors had greater exposure to these processes than mentees,

delivery staff were more knowledgeable than mentors and were able to advise them

accordingly. However, some mentors reported that it would have been useful to

meet as a mentor group more often than happened on the project. Praxis had

planned to arrange for the mentoring group to meet once a month but they only met

twice over the duration of the project.

93

5.51 As mentioned previously, there did not appear to be a clear distinction between

support that came from the mentors and support from Praxis. However, the added

value of mentoring appeared to be that mentors were able to explain and run

through processes in detail with the mentees in a manner that delivery staff were

not able to do. Furthermore, those in the treatment group had easier access to

Praxis’ other services when they attended mentoring sessions, than any of those in

the control group.

Case study, Mentoring Dofi28, UK

I have met with Dofi on several occasions in the last two months, during which time he has

been awaiting news of his ‘Leave To Remain’ application and has been unable to work

following his arrest for working under a different name.

Dofi has been in the UK for nine years, having overstayed a work visa and recently got

married to a Ghanaian National who has ‘Limited Leave To Remain’ with no recourse to

public funds. His wife works as a chef 30 hours a week, but they struggle to provide for

her two young children on her small salary of £800 per month. They have been relying on

the kindness of their church community, but are deeply in debt and facing destitution. Her

income doesn’t even cover rent and so they don’t have enough money for the family’s basic

needs, such as food, clothes, and toiletries.

I have introduced Dofi to his local food bank (which he has used six times) and have raised

£550 for him from various charitable foundations. This has gone towards his rent, bills and

legal fees. We have also discussed cheaper housing possibilities and I am helping the family

with researching a move out of London.

I have been in touch with Dofi’s lawyers to check his application and have helped his wife

write a letter of support for his application. He received the good news this week that his

‘Limited Leave to Remain’ application has been approved. Now that he is allowed to work in

the UK, I am supporting him with CV-writing and job-hunting. Dofi has started looking for

work contacting friends and his previous employer. He already has two interviews lined up

next week, one is with a previous employer so he is confident that a job will be confirmed

soon.

Dofi has struggled over the last few months financially, and the family are now in arrears to

their landlord, services and their bank. They have been facing destitution and we will

signpost him to various places for debt advice.

28 Names have been changed to protect anonymity.

94

Mentor outcomes

5.52 Mentors on the pilot achieved both soft and hard outcomes from their involvement

as mentors. In terms of soft outcomes, the main benefit identified was the feeling of

having helped someone who was in desperate need of support. Many of the

mentors, themselves Foreign Nationals or from migrant backgrounds, wanted to help

others who were in a similar situation to themselves or their relatives at one point in

time. Volunteering at Praxis, and specifically mentoring on the MEGAN project,

offered them the perfect opportunity to do this.

“Certainly I’ve learned a lot…I realised I could contribute even though we

could not achieve a huge transformation for the mentee.” (Mentor)

5.53 We cannot be sure that we have captured all of the positive hard outcomes achieved

by mentors, and the numbers reported are therefore the minimum outcomes for

mentors. However, we do know that following their involvement mentoring at Praxis:

Two mentors secured full-time jobs in the public sector, with references provided

by Praxis helping them to secure this work;

1 mentor secure a temporary job;

2 mentors secured voluntary work placements , with references again provided by

Praxis to secure these positions.

5.54 This highlights another key finding of the evaluation; that mentoring can have a

profound impact on mentors, in some cases over and above that achieved by

mentees.

Up scaling and future plans

5.55 At the close of the evaluation, Praxis was intending to continue their mentoring

programme. The organisation had secured two streams of funding to do this: (1)

from London Probation Trust and (2) from the UK Cabinet Office’s Rehabilitation

Social Action Fund; a grant fund given to organisations to provide them with

resources to demonstrate what works and deliver social action programmes at a

greater scale, for example in new prisons and communities. The MEGAN project was

deemed a good example of such a programme.

5.56 The roll-out of the mentoring programme will also place an emphasis on evaluation,

in an attempt to collect further evidence of the effectiveness of mentoring. However,

the planned projects will not use a randomised control trial technique as Praxis

coordinators felt this methodology created ethical issues, such as resentment

amongst referral organisations when mentees were placed in the control group. In

95

lieu of this, a propensity score matching method will be used to identify a similar

group of individuals through criminal justice records, and track their progress over

time.

5.57 When discussing the extension of mentoring to Foreign National Offenders, the

project co-ordinators reported that including mentoring in an action plan mentees’

created with their Probation Officer might have improved the project. However,

mentees would have to agree to its inclusion on their action plan, which would then

make mentoring meetings almost mandatory to attend. Co-ordinators felt his would

have increased mentees’ motivation to attend meetings and make the most of the

mentoring relationship. However, it is uncertain how introducing a mandatory

element would affect mentees’ participation in mentoring relationships.

Conclusions

5.58 Overall, the UK pilot was successful in achieving outcomes for both mentors and

mentees as well as helping the delivery organisation some valuable lessons for

rolling mentoring out in the future.

5.59 Findings from the UK pilot support key findings of the evaluation as a whole. Namely

that:

Mentoring works well for those in settled conditions but still facing social and

labour market exclusion

Mentoring is most effective at securing outcomes when these are clearly

understood and agreed by both mentors and mentees from the start of a project

Mentors can benefit as much, if not more, from mentoring relationships as

mentees.

5.60 Regarding the mentoring relationships, it appears that mentoring is most effective

when mentees are motivated to actively engage with the project. A key learning

from the UK pilot is that increased engagement can be achieved through either

removing barriers to attendance, such as Praxis was able to do by covering travel

expenses, or ensuring mentees understand the value of mentoring.

5.61 In addition, UK findings support previous research which highlight that it is vital that

when mentees volunteer for such a scheme they understand what to expect29. The

separation between the referral process (LPT) and the delivery of mentoring (Praxis)

may have caused a mismatch between mentees’ and mentors’ expectations. Placing

29 Finnegan, L et al (2012) Models of Mentoring for Inclusion and Employment: Final Report

96

mentoring on a rehabilitation action plan was one suggested way of ensuring

mentees appreciate what they can expect from a project when they commit to it.

However as the UK pilot was the least established of the three projects, it should be

expected that the objectives of mentoring will become better understood over time.

The mentoring projects in Hungary and Portugal have both established themselves

since the MOMIE project and the purpose of mentoring in each environment become

better understood.

5.62 Another important lesson from the UK, is that mentoring was most effective once

initial external barriers had been overcome. This was highlighted in the project as

many mentees were involved in complex immigration cases. Much of the support

mentors were able to offer, such as help to find work or settle in an area, was only

relevant once mentees had the right to live and work in the UK. There were also

difficulties for those mentees with the most chaotic lives and severe drug and/or

alcohol issues. Mentors and project co-ordinators felt that mentoring was not the

right method for dealing with these issues and alternative programmes of support

were necessary in these circumstances.

5.63 These reasons may help explain why the drop-out rate in the UK was higher than in

other projects as some mentees were not able, due to external factors in their lives,

or willing, as they did not understand the benefit, to attend meetings with their

mentors. As the target cohort in the UK were one of the most challenging and

vulnerable groups in society to work with we believe that external factors as

opposed to a lack of motivation were the causes of most drop-outs.

5.64 However these drop-outs and the impact external factors can have on mentees

highlight the fact that mentoring in this form needs to sit within a holistic

package of support. It may be appropriate to conduct a ‘pre-filtering process’ to

determine who is most likely to benefit from mentoring support and position

mentoring further along the support journey.

5.65 For those who did overcome these initial barriers, mentoring was an effective tool for

helping them to stabilise their lives and find work. For mentors, they benefitted from

being able to help others as well as using the experience to help them secure hard

outcomes such as voluntary or paid employment.

97

6 Randomised control trial

feasibility study

6.1 This chapter describes the work undertaken to deliver the evaluation objective to;

“test the feasibility of conducting a randomised control trial (RCT) of mentoring to

identify impacts”. The chapter covers an explanation of why an RCT approach was

selected for this project and why this necessitated a feasibility study, and draws

conclusions about the feasibility of conducting an RCT within the context of the

MEGAN pilot.

Randomised control trials

6.2 The randomised trial is considered the ‘gold standard’ for evaluation because the

random assignment into treatment and control means that differences in outcomes

between the two groups can be attributed to the project itself, rather than to any

personal characteristics or environmental factors.

6.3 Randomised controlled trials (RCTs) are the best way of determining whether a

policy is working. What makes RCTs different from other types of evaluation is the

introduction of a randomly assigned control group, which enables you to compare

the effectiveness of a new intervention against what would have happened if you

had changed nothing. The introduction of a control group eliminates a whole host of

biases that normally complicate the evaluation process – for example, if you

introduce a new scheme, how will you know whether those receiving the extra

support might not have achieved a positive outcome anyway?30

6.4 Countering the advantages of RCTs are the difficulties (and sometimes

impossibilities) in allocating people randomly within a particular area to a treatment

or control group. For voluntary programmes, there can be an ethical difficulty in

advertising projects that some people will be denied entry to if they apply31. In the

UK the concept of a randomised control trial, where by definition not all referred

mentees can receive the mentoring ‘treatment’, led to disengagement by some

probation officers resulting in lower than expected referrals to the pilot.

30 Haynes, L et al (2011) Test, Learn, Adapt: Developing Public Policy with Randomised Controlled

Trials, Cabinet Office 31 Purdon, S et al (2001) Research methods for Policy Evaluation, DWP Research Working Paper No 2.

98

Feasibility study

6.5 The objective of an RCT is to test the impact of a particular intervention on one or

more outcomes by tracking these outcomes both for the group that receives the

intervention (the ‘treatment group’) and a comparable group that doesn’t (the

‘control group’). The impact is the difference in outcomes between the treatment

and control groups. In order to say that any differences between the treatment and

control groups are not just down to chance, we check whether impacts are

statistically significant. Finding significant impacts requires large numbers of people

in both groups.

6.6 We asked partners to aim for 30 mentees in each cohort (which meant attracting 60

participants to the project within each cohort, as half of these will be placed in the

control group), but to see whether they could achieve higher numbers than that. We

conducted preliminary calculations – called power calculations – which suggested

that projects of this size are unlikely to produce statistically significant findings.

Therefore, we positioned this research as a ‘feasibility study’, designed to move the

evidence closer towards an answer to the question ‘does mentoring work, and what

kind of impact does it have on certain outcomes?’, and to assess the practicalities

and challenges of conducting a full-scale RCT of mentoring interventions for

disadvantaged or vulnerable groups.

6.7 It was agreed that, whilst it would be ideal for the feasibility study to find significant

impacts, even if no statistically significant result was derived the study would still be

extremely valuable in furthering the debate around mentoring and supporting a

larger study in the future.

Randomisation

6.8 The recruitment and randomisation process was conducted slightly differently for

each project. In Hungary and Portugal, a cohort recruitment method was used,

whereby volunteers are recruited to the project over a period of time then all

randomised at the same time by Inclusion into either the treatment or control group.

In Portugal, there were some amendments to this as extra volunteers were recruited

to cover project drop-outs. In the United Kingdom, volunteers were recruited, and

randomised continuously, to minimise the risk of drop-out between recruitment and

the start of the project.

6.9 The randomisation itself was conducted using a random number generator at

Inclusion.

99

Data collection

6.10 To measure impact, we recorded information on all project participants, both

treatment and control group, throughout the project. We collected data at baseline,

at three months, at six months when the mentoring relationship ends and at nine

and twelve months to assess whether any impacts of mentoring are sustained.

6.11 Information we measured at baseline included:

Age

Gender

Ethnic group

Whether the participant has a long-term health condition or disability

Whether the participant is employed, unemployed or economically inactive

The length of time since the participant last worked

Whether the participant is currently in full- or part-time education

The level of the participant’s highest qualification

Whether the participant is currently claiming state benefits

Whether the participant has a criminal record or history of offending

The length of time since the participant’s last offence.

6.12 Outcome measures collected at each further data collection point were:

Whether participants have received mentoring in the past three months [we want

to check whether those in the treatment group are actually being mentored, and

whether those in the control group are getting mentoring support elsewhere]

The number of weeks participants have had a mentor for in the past three months

Whether participants have done any paid employment in the past three months

The number of full weeks spent in paid employment in the past three months

Whether participants have done any work experience or volunteering in the past

three months

100

Whether participants have been involved in any education or training in the past

three months

Whether participants have committed any criminal offences in the past three

months

The number of criminal offences committed in the past three months.

6.13 These outcomes were collected differently across the three partners from the start of

the projects. In Hungary and Portugal, data was collected by project partners and

provided to the evaluation team as they had unrivalled access to the control and

treatment group in both cases. In Hungary, BAGázs was the only organisation able

to collect robust data for those living in the settlement. In Portugal, Santa Casa

(under the Pendulum umbrella) had access to both the participants and information

in their role as the management of the state benefit. In the United Kingdom, the

data was provided by London Probation Trust which routinely collected data on

project participants: foreign national offenders, as a matter of course.

Implementation

6.14 Given the innovative and experimental design, it was essential that all partners were

aware of their role in the evaluation and were bought in to delivering a rigorous

methodology. To facilitate the process and to support delivery partners Inclusion

held a number of information sessions and distributed documents outlining the

research and evaluation process to partners.

6.15 The first challenge was to establish robust ‘treatment’ and ‘control groups’. This was

done by ensuring that when marketing the intervention, partners made clear to

volunteers that even if they did put themselves forward, there was a chance they

wouldn’t be assigned a mentor.

6.16 We aimed to ensure that all potential participants were given the same information

about the project before they decided whether or not they wanted to take part. This

was particularly important where more than one delivery team member was working

on participant recruitment. We asked partners to put together a recruitment ‘script’

to be used whenever potential participants are being told about the project. Each

partner tailored their recruitment script to the specifics of their project, and we

worked with partners individually to make sure that it is appropriate for the research

and contained all the necessary information.

6.17 Once volunteers were confirmed onto the project, the randomisation was computed

by Inclusion, to avoid any potential bias by in allocation. The randomisation had to

be performed in this way, i.e. including all those who had volunteered to the project,

101

because there are likely to be some fundamental differences between those who are

willing to participate and those who are not.

6.18 However, in the United Kingdom we experienced some issues around the ethics of

denying volunteers access to a programme from which they could potentially benefit.

Although the coordinating organisation was one board and understood the

importance of the RCT approach, the involvement of a further delivery organisation

complicated the experiment slightly. The coordinating organisation, Praxis, had to

rely on Probation Officers, who may know little about the project and were often

concerned about the experimental design, to refer to the project. These Probation

Officers were keen to secure support for their clients, and thus finding out a referee

had been placed in the control group may have been demoralising. This could have

had an effect on future referrals: for example officers may have been less likely to

refer someone if previous referees had been placed in the control group or may

attempt to game the system. No evidence of this was found in practice. However it

does represent a key challenge with the randomised control trial methodology, in

that ensuring referral organisations both understand and engage with the process is

crucially important.

6.19 At the time of MEGAN the Probation Service in the UK was also undergoing radical

change with 70% of the Service due to be privatised from 2015. The process of

preparing for changes had already begun at the time of the trial and with such

upheaval pending it was difficult for Probation Officers to focus on the project. As

described previously, in the UK we relied on the Probation Trust to provide us with

the data on control and treatment group participants, as opposed to the delivery

organisation elsewhere. With this major structural reform, it was very difficult to

formalise this process and as a result data was not collected as regularly as was

envisaged at project inception.

6.20 Once randomised, it was crucially important to constrain project partners from

offering mentoring, or any other services that would not be available if this project

didn’t exist, to the control group. Whilst those in the control group were able to

access other programmes of support (as long as they were also available to the

treatment group) partners were discouraged from going to great lengths to

encourage participants in the control group to seek out other services. This is

because actively pushing control group participants towards support is an

intervention itself and may have contaminated our results.

102

Experimental validity

6.21 Although the results from randomised trials have a credibility that other evaluation

designs do not, the estimate of the programme effect may be biased if any of the

following occur:

if some of those control group do actually participate in the programme.

Pilots were rigorous in implementing the RCT design and we do not believe that

control group members were mentored in the same way as the treatment group.

However in two pilots the close nature of the communities involved meant that

the distinction between treatment and control could be blurred.

if those randomised to the programme affect the behaviour or attitudes

of those in the control group (which might happen if members of the two

groups know one another). The UK pilot had the clearest distinction between

treatment and control groups. As opposed to the other two pilots, where there

may have been contamination issues between members of small, tightly-knit

communities; in the London pilot the mentees in the treatment and control groups

were unlikely to mix. The Hungarian pilot was conducted in a small village with a

large proportion of the residents involved, and thus the support BAGázs provided

has a huge impact on the settlement as a whole. From an evaluation point of view

this means that the intervention had effects on the wider population and the

control group as well as the treatment group. In some cases both mentees and

control group members participated in a pilot ‘activity’ (e.g. building/repairing

houses, playing football) that other residents were not involved in – again

diminishing the advantage the treatment group experience. The parents of some

young people in the control group may have been mentored, meaning that the

impact of mentoring on adults would be experienced by their children.

if surveys to measure outcomes get differential non-response in one or

other of the groups. In the UK pilot, we were only able to conduct qualitative

interviews with those still engaged with the pilot and thus were more likely to

speak to mentees whose relationships had been successful.

if being denied access to a programme makes those in the control group

act differently to how they would act in the total absence of the project.

We do not believe this to have been an issue. However this was clearly a concern

for the Hungarian pilot and is why control group members had good access to the

services of BAGázs.

103

Conclusions

6.22 Organisations should aim to use the most robust evaluation methodology to

determine the impact of their projects. Smaller community organisations can

successfully administer a challenging RCT evaluation design. Key factors in success

are a flexible randomisation approach (i.e. continuous or by cohort - depending on

delivery partners’ needs), upfront training in randomisation rationale and procedure,

and commitment from delivery partners.

6.23 RCTs can be successfully administered within close-knit communities. However it

may be more appropriate to randomise entire communities as opposed to individuals

within them, particularly where communities are disadvantaged and all members

need support.

6.24 RCTs are dependent on robust data collection which can be challenging when

interventions are working with hard to engage populations and during periods of

organisational change. The conditions in which the project is being delivered, and

the potential impact on outcomes should be considered during the design phase.

6.25 Larger sample sizes are required where the impact of the intervention is expected to

be small. Therefore future evaluations may wish to focus on large-scale mentoring

projects.

104

7 Conclusions and

Recommendations

7.1 This chapter brings together the conclusions drawn throughout the report and from

the MEGAN literature review to deliver a set of recommendations for the design and

delivery of mentoring programmes.

Design of mentoring programmes

Programmes should be designed in line with the growing body of evidence on

the key features of effective mentoring.

7.2 Key features include: the presence of structured activities deployed flexibly according

the mentee need, applying a quality standard to ensure all structured mentoring is

quality mentoring. Praxis has been successful in securing the Approved Provider

Standard (APS) from the Mentoring and Befriending Foundation for MEGAN and

future mentoring projects. The APS logo is a national quality standard mark of safe

and competent practice. Evidence varies on the desired duration of a mentoring

relationship but suggests that a relationship lasting for more than 12 months and

involving regular contact is preferable.

Mentoring programmes are not a replacement for professional support.

7.3 Mentoring is often delivered as part of a wider, holistic package of support rather

than as a standalone intervention; this proved to be an effective model of delivery

on the MEGAN project. The additional value of mentoring, over and above what the

delivery organisation could normally offer is the intensity of support a mentor can

provide. Delivery organisations were not able to provide the same intensity and

regularity of support to all their clients to due to limited resources and mentors filled

this gap.

Consideration should be given to the most appropriate timing of the mentoring

intervention.

7.4 There were indications across all three pilots, that the benefits of mentoring were

maximised once initial barriers had been overcome. In the UK, these barriers related

to a mentee’s immigration status and their right to live and work in the country. In

Portugal, these barriers related to language and literacy difficulties with mentors

reporting that they find it extremely difficult to work with mentees who had limited

grasp of Portuguese, either written or spoken. In Hungary, the main barrier was a

reluctance to engage with outsiders or even to discuss personal issues with peers in

105

the men’s scheme. Evaluation findings indicated that mentors could support mentees

in addressing these initial barriers, for example by signposting and liaising with

professionals on the mentees’ behalf, however mentors were able to provide more

intensive support once these barriers had been resolved which often occurred with

the help of professional services.

Providing scope for mentee progression supports achievement of outcomes

beyond the lifetime of a single project.

7.5 In the Portuguese and Hungarian pilots a number of MEGAN mentors were

previously mentees within the MOMIE project. Being a mentee may therefore be the

starting point of a long-term development process in which mentees go on to

become mentors and progress from there. It would be advantageous for mentoring

programmes to have sufficient longevity and to be embedded in wider support in

such a way that allows this type of progression to occur.

Delivery of mentoring programmes

An effective programme should have a sufficient lead-in period prior to the start

of delivery.

7.6 Although true of many projects, it is particularly critical for mentoring projects to

have a sufficient lead in time to given that intensive mentor training is likely to be

required upfront. In addition, the importance of the mentor-mentee matching

process implies the need for a sufficient period time to assess mentees’ needs and

mentors’ skills. MOMIE findings suggest that a new mentoring programme is likely to

need three months lead-in time and up to six months after implementation for the

programme to become fully operational.

It is critical to establish a good match between mentor and mentee to maintain

mentee engagement and maximise positive outcomes.

7.7 Mentors and mentees do not necessarily need to share personal characteristics and

experiences (although this can help) and the matching should be based on the needs

of the mentee and the support that the mentor can offer in this respect. The MOMIE

evaluation emphasised the importance of the flexibility to change mentors in event

that mentees’ needs and the mentor’s skills do not or are no longer a good match. In

MEGAN there were some instances were mentees were given the flexibility to

change to another mentor better suited to their needs, for example in Portugal.

Before starting to work together, mentors and mentees should clarify the aim

and purposes of the partnership.

106

7.8 Findings from the MEGAN and MOMIE projects, and wider evidence, suggest that an

upfront discussion about what they hope to achieve would help mentees to have

realistic expectations and feel involved in the design of the intervention. It would

also give the mentors a clear goal and direction, as well as an understanding of how

their skills may be used. Maintaining engagement and motivation on both sides is

likely to be enhanced under these circumstances.

Maintaining mentee engagement can be challenging and can require flexibility

and innovation to achieve.

7.9 Project coordinators in MEGAN offered learning on how to maintain mentee

engagement which includes: addressing practical issues such as removing barriers to

attendance (i.e. by covering travel expenses) and ensuring that mentees understand

what to expect from mentoring and the potential benefits (i.e. by creating

information materials for prospective mentors and referral organisations). In open-

ended mentoring potential benefits may be less tangible therefore a ‘hook’ or

incentive can be helpful as in Hungary where the youth scheme engaged participants

through football and the men’s scheme through home improvement.

Close working with referral and delivery agencies supports mentee recruitment

and engagement, as well as enabling signposting.

7.10 Working with referral partners was found to be particularly critical in the UK pilot

where referral numbers were initially low. Project coordinators conducted regular

visits to referral agencies to ensure that the pilot maintained visibility and that

Probation Officers understood what the pilot could off their clients. A sufficient lead-

in time can be helpful in enabling relationships with referrers to be built before the

programme begins. However, ongoing engagement is likely to be required. This is a

particularly important point for organisations receiving mentoring through a third

party (e.g. a statutory body). This was not an issue in Hungary and Portugal as the

delivery organisations were self-referring. However all organisations delivering

mentoring should have strong links to other service providers to enable

Supporting mentors throughout the programme works to ensure that benefits

are reciprocal.

7.11 Within the MEGAN project, and in many mentoring programmes, mentors are giving

their time on a voluntary basis and should be supported throughout to ensure that

the experience is positive and beneficial. Some issues that mentees experience can

be distressing and difficult for a mentor to deal with, including financial

disadvantage, domestic violence and drug and alcohol abuse, even if they are not

directly supporting mentees to deal with these challenges. Delivery organisations

must offer opportunities to mentors to share experiences and to seek help in areas

107

of uncertainty. As well as support from the project coordinators, MEGAN mentors

found networking group sessions helpful in this sense.

108

8 Technical Appendix

Randomised Control Trial Impact Analysis

8.1 Inclusion conducted a Randomised Control Trial (RCT) with a treatment group of

mentees and a control group of individuals who volunteered for mentoring but were

randomly assigned to the control group.

Analysis

8.2 The source data for this analysis is based on the management information supplied

by the projects in Hungary and Portugal on participants and control cases at the 12

month point. This gives some chance for the service to have an effect on outcomes.

8.3 We transform the data so that information that is in character format becomes more

amenable for analysis. The qualification variables are recoded to a common basis

using the international classification of education 'isced'. This has been done based

on information linking the Hungarian and Portuguese qualification information given

to the international standards.

8.4 The analysis used is to undertake a multivariate logistic regression comparing

outcome variables with (a) whether the people are in the control or treatment

groups and (b) a set of variables showing the characteristics of the individual at the

start. These include sex, age (included in the conventional manner as age and age-

squared), qualification level, whether they were employed at the start of provision.

In Portugal, ethnicity is added, and in Hungary, whether people had a criminal

record.

8.5 For each analysis, the analysis used using the R language is of the form:

glm(emp_12 ~ treat +poly(age,2) + sex + isced + employ_baseline + ethnic,

data=portugal, family=binomial()).

8.6 This produces an analysis model in which the effect of each variable (all categorical,

except age) is assessed against a baseline. The baseline for the Portuguese analysis

is:

treat Control group

sex Female

isced No qualifications (0)

109

employ_baseline Economically inactive (not looking or available for work)

ethnic Black

8.7 For Hungary, there are no ethnic distinctions, but there is an offender variable

recording possession of a criminal record (or not). This would be expected to impact

on the chances of starting work, and is therefore included in the analysis.

8.8 The 'Estimate' gives the regression coefficient of the effect of the variable (or, with

categorical variables, the category of the variable, on the odds of a person being in

employment (or other outcome measure).

8.9 The figures in the main report include error bars including the standard errors for the

estimates for the treatment. These have been omitted from the tables below for

space reasons.

Portugal analysis

Employment outcomes

8.10 This shows a positive and significant (at the 5% level) impact of 'treatment' i.e.

mentoring on employment outcomes measured at the 12 month point. Statistical

significance at the 5% level is represented by (*).

Estimate Pr(>|z|) Signif.

(Intercept) 2.8248 0.5082

Treat Treatment 3.3578 0.0136 *

poly(age, 2)1 4.0150 0.4239

poly(age, 2)2 -4.9745 0.3655

Sex Male -0.6214 0.5992

isced2 -3.8293 0.0914

isced3 -1.6112 0.4184

isced5 -17.9091 0.9978

isced6 -22.7960 0.9972

employ_baseline: In paid

employment -21.7927 0.9973

employ_baseline: Unemployed

(looking for work and available to start)

-1.8923 0.6297

Ethnic: Other ethnic group -18.1955 0.9968

Ethnic: White -1.7928 0.1156

110

8.11 No factor other than treatment (mentoring) is significant, which may indicate that

the remainder of the support offered by the projects made much the same impact

regardless of the respondent's initial conditions.

Benefits outcomes

8.12 The same analysis was undertaken for benefit receipt at the 12 month stage.

Estimate Pr(>|z|)

(Intercept) 18.7143 0.9960

Treat Treatment 1.5148 0.0858

age -0.1041 0.7331

I(age^2) 0.0015 0.6631

Sex Male -0.7121 0.3636

isced2 -18.0953 0.9961

isced3 -18.7106 0.9960

isced5 0.2223 1.0000

isced6 -0.8636 0.9999

employ_baseline In paid employment 17.8449 0.9978

employ_baseline Unemployed (looking for work

and available to start) 1.6187 0.3302

Ethnic Other ethnic group 16.9356 0.9970

Ethnic White 0.3293 0.6841

8.13 This shows no significant difference in benefit receipt (yes/no) at the 12 month

point.

Voluntary work outcomes

8.14 As an intermediate outcome measure, whether or not people were undertaking

voluntary work or other unpaid work experience was tested.

Estimate Pr(>|z|)

(Intercept) -68.0576 0.9941

Treat Treatment 1.1910 0.3768

age 1.2764 0.1234

I(age^2) -0.0120 0.1682

Sex Male -1.1799 0.3800

isced2 14.4186 0.9979

isced3 15.7755 0.9977

isced5 36.5758 0.9976

isced6 37.1416 0.9975

111

employ_baseline In paid employment -0.8002 1.0000

employ_baseline Unemployed (looking for work

and available to start) 18.6480 0.9980

Ethnic Other ethnic group 5.8472 0.0573

Ethnic White 0.6901 0.5867

8.15 This shows no significant difference in voluntary work (yes/no) at the 12 month

point.

Training outcomes

8.16 Also, as an intermediate outcome measure, whether people were undertaking

training at the 12 month point was tested.

Estimate Pr(>|z|)

(Intercept) 0.3698 0.9473

Treat Treatment 0.9269 0.2241

age 0.0437 0.8641

I(age^2) -0.0008 0.7853

Sex Male 0.0773 0.9141

isced2 -0.9371 0.5058

isced3 -1.8255 0.2116

isced5 -18.1505 0.9963

isced6 -19.0383 0.9962

employ_baseline In paid employment 17.0470 0.9966

employ_baseline Unemployed (looking for work

and available to start) -0.3289 0.8384

Ethnic Other ethnic group 16.9651 0.9952

Ethnic White 0.1415 0.8480

8.17 This shows no significant difference in training activity (yes/no) at the 12 month

point.

Improvement in reading or literacy outcomes

8.18 Respondents were asked at the 12 month point whether they felt their reading or

literacy had improved. As with other measures, reading and literacy support was

provided to both control and treatment groups, with the difference being that the

treatment group had additional mentoring support.

Estimate Pr(>|z|)

112

(Intercept) 0.3698 0.9473

Treat Treatment 0.9269 0.2241

age 0.0437 0.8641

I(age^2) -0.0008 0.7853

Sex Male 0.0773 0.9141

isced2 -0.9371 0.5058

isced3 -1.8255 0.2116

isced5 -18.1505 0.9963

isced6 -19.0383 0.9962

employ_baseline In paid employment 17.0470 0.9966

employ_baseline Unemployed (looking for work and available to start)

-0.3289 0.8384

Ethnic Other ethnic group 16.9651 0.9952

Ethnic White 0.1415 0.8480

8.19 This shows no significant difference in reading or literacy outcomes (yes/no) at the

12 month point.

Improvement in budgeting outcomes

8.20 Respondents were asked at the 12 month point whether they felt their budgeting

had improved. As with other measures, budgeting support was provided to both

control and treatment groups, with the difference being that the treatment group

had additional mentoring support.

Estimate Pr(>|z|)

(Intercept) 0.2750 0.9710

Treat Treatment 3.3958 0.0102 *

age -0.0563 0.8681

I(age^2) 0.0023 0.5898

Sex Male -1.0456 0.3342

isced2 -3.9653 0.0711

isced3 -1.9993 0.3039

isced5 -17.6123 0.9978

isced6 -22.1814 0.9973

employ_baseline In paid employment

-20.9641 0.9974

employ_baseline Unemployed

(looking for work and available to start)

-0.3906 0.8693

Ethnic Other ethnic group -17.9436 0.9968

Ethnic White -1.7109 0.1104

113

8.21 This shows a positive and significant difference in budgeting outcomes (yes/no) at

the 12 month point. It is likely that the improvement in employment outcomes and

budgeting outcomes are linked, possibly due to increased financial resources and

also that as some of the employment outcomes were self-employed, budgeting skills

are required to be self-employed successfully.

Hungary analysis

8.22 Similar analyses were conducted for the Hungarian project. However, in this project,

there were no significant results (either positive or negative). In this project, as with

the others, support was provided to both control and treatment groups, with the

difference between them being that the treatment group had mentoring additionally

to the support provided equally to both groups.

8.23 In several of the analyses, responses were so close between control and treatment

groups that the analysis showed no difference.

Employment outcomes

Estimate Pr(>|z|)

(Intercept) -8.8224 0.9993

Treat Treatment -1.0538 0.4258

age 0.3637 0.3853

I(age^2) -0.0054 0.3537

Sex Women -19.1912 0.9968

Offender Yes 1.2840 0.3362

isced1 20.7382 0.9980

isced2 20.3347 0.9981

isced3 19.9802 0.9981

employ_baseline In paid employment 21.3455 0.9966

employ_baseline Unemployed (looking for work

and available to start) -16.2866 0.9973

8.24 This shows that there was no statistically significant effect of the mentoring

treatment on the employment outcomes at the 12 month point in the Hungarian

trial.

114

Benefits outcomes

Estimate Pr(>|z|)

(Intercept) 22331.6809 0.9822

Treat Treatment 2349.9349 0.9820

age -1538.5713 0.9820

I(age^2) 24.8224 0.9820

Sex Women -865.0878 0.9959

isced1 7269.9613 0.9829

isced2 4820.8900 0.9830

isced3 4286.7831 0.9833

employ_baseline In paid employment -2885.5864 0.9892

employ_baseline Unemployed (looking for work

and available to start) -3312.4032 0.9882

Offender Yes -2448.0420 0.9846

8.25 This shows no significant difference in benefit receipt (yes/no) at the 12 month

point.

Voluntary work outcomes

Estimate Pr(>|z|)

(Intercept) 14.1061 0.9976

Treat Treatment -0.5443 0.6853

age 0.2849 0.5190

I(age^2) -0.0048 0.4663

Sex Women -20.4091 0.9965

isced1 -18.6702 0.9954

isced2 0.6036 0.7956

isced3 -0.1316 0.9576

employ_baseline In paid employment -17.2628 0.9970

employ_baseline Unemployed (looking for work

and available to start) -16.3174 0.9972

Offender Yes -1.5994 0.2872

8.26 This shows no significant difference in voluntary work (yes/no) at the 12 month

point.

115

Training outcomes

Estimate Pr(>|z|)

(Intercept) -46.2175 0.9968

Treat Treatment -0.1143 0.9369

age 3.1758 0.2121

I(age^2) -0.0550 0.2240

Sex Women -19.2890 0.9977

isced1 0.3897 1.0000

isced2 20.0921 0.9983

isced3 18.9611 0.9984

employ_baseline In paid employment -18.3131 0.9978

employ_baseline Unemployed (looking for work

and available to start) -18.1659 0.9978

Offender Yes -0.5836 0.7199

8.27 This shows no significant difference in training activity (yes/no) at the 12 month

point.

Improvement in reading or literacy outcomes

Estimate Pr(>|z|)

(Intercept) -25.5661 1.0000

Treat Treatment -0.0000 1.0000

age 0.0000 1.0000

I(age^2) -0.0000 1.0000

Sex Women 0.0000 1.0000

isced1 -0.0000 1.0000

isced2 0.0000 1.0000

isced3 0.0000 1.0000

employ_baseline In paid employment 0.0000 1.0000

employ_baseline Unemployed (looking for work

and available to start) -0.0000 1.0000

Offender Yes -0.0000 1.0000

8.28 This shows no significant difference in reading or literacy outcomes (yes/no) at the

12 month point.

Improvement in budgeting outcomes

Estimate Pr(>|z|)

116

(Intercept) 2523.2078 0.9987

Treat Treatment 15.8206 0.9999

age -136.4115 0.9986

I(age^2) 1.8637 0.9987

Sex Women -103.0983 0.9998

isced1 80.0010 0.9998

isced2 43.3194 0.9993

isced3 -70.7195 0.9997

employ_baseline In paid employment -130.7714 0.9998

employ_baseline Unemployed (looking for work and available to start)

-127.8115 0.9997

Offender Yes 45.2650 0.9995

8.29 This shows no significant difference in budgeting outcomes (yes/no) at the 12 month

point.

UK analysis

8.30 The source data for this analysis is based on the management information supplied

by the project in the UK on participants and control cases at the 12 month point.

This gives some chance for the service to have an effect on outcomes.

8.31 The UK has no qualification data. In the UK, all had a criminal record so this did not

discriminate between cases. An additional variable was added relating to

participants' migration status. A number of respondents were asylum seekers at the

baseline and so did not have the legal right to work. This was included so that any

interaction between this status and employment changes could be controlled.

8.32 As the data was administrative, the only outcome data recorded was employment

and offending data. The intermediate outcomes of voluntary work, training, and

improvements in literacy and budgeting, were not recorded.

8.33 In the UK, as with the other projects, other forms of support were provided to both

control and treatment, but mentoring was only provided to the treatment group.

UK analysis

Employment outcomes

Estimate Pr(>|z|)

(Intercept) -26.5661 0.9999

Treat Treatment 0.0000 1.0000

117

Age, represented as numeric age and: -0.0000 1.0000

The square of numeric age 0.0000 1.0000

Sex Male 0.0000 1.0000

Migrant EEA 0.0000 1.0000

Migrant N/A -0.0000 1.0000

Migrant Right to work -0.0000 1.0000

Employment at baseline In paid employment 53.1321 0.9998

Employment at baseline N/A / not collected 0.0000 1.0000

Employment at baseline Unemployed (looking for

work and available to start) 0.0000 1.0000

Ethnicity Black - Other African 0.0000 1.0000

Ethnicity N/A 0.0000 1.0000

Ethnicity Other -0.0000 1.0000

Ethnicity White - Other White 0.0000 1.0000

8.34 This shows no significant impact of 'treatment' i.e. mentoring on employment

outcomes measured at the 12 month point. There was no change in employment

between baseline and the 12 month point. Those in employment at the baseline

were also in employment at the 12 month point, and there were no employment

changes for the others.

Offending outcomes

Estimate Pr(>|z|)

(Intercept) -33.8300 0.9989

Treat Treatment 0.7363 0.6550

Age, represented as numeric age and: 9.0051 0.3895

The square of numeric age 5.1881 0.7091

Sex Male 12.3560 0.9996

Migrant EEA 26.0586 0.9988

Migrant N/A 19.9489 0.9981

Migrant Right to work 19.9258 0.9981

Employment at baseline In paid employment -19.3553 0.9987

Employment at baseline N/A / not collected -21.0767 0.9994

Employment at baseline Unemployed (looking for

work and available to start) -5.7797 0.9998

Ethnicity Black - Other African 0.4300 0.8017

Ethnicity N/A -20.7998 0.9991

118

Ethnicity Other -17.7563 0.9985

8.35 This shows no significant difference in benefit receipt (yes/no) at the 12 month

point.

Recoding qualifications

8.36 For the Portuguese and Hungarian analysis, the qualifications data was recoded to

international (ISCED) standards from national descriptions. This was checked with

local partners and used standard descriptions through EUROSTAT. The R code for

recoding is shown below.

library(car)

hungary$isced <- recode(hungary$qual, "0='0'; c(1,2,3,4)='1'; c(5,6,7,8)='2';

c(9,10,11,12)='3'")

portugal$isced <- recode(portugal$qual, "'Iliteracy'='0'; 'Basic'='2'; 'High School'='3';

Distance Travelled Measure

8.37 Inclusion sought to measure progress against ‘soft’ outcomes via a set of

psychometric tests. The tests were designed to be undertaken by both the treatment

group of mentees and the control group at the start to baseline soft skills and again

at the end of the pilot to measure progression in soft skills. This was a refinement on

the outcomes star approach used to measure distance travelled within the MOMIE

project.

8.38 The psychometric tests were designed by the evaluation team at Inclusion based on

other best practice models. Participants were asked to select which of four

statements they agreed most with, under a series of headings. These statements

were then scored and a total for all indicators calculated which would indicate that

the individual was either:

Score of 9: Disengaged – this is when a mentee is not willing to talk about

their problems or get any help

Score of 18: Cooperating – this is when your mentee accepts that they need

help and goes along with help that is being provided but doesn’t actively try to

help themselves

Score of 27: Gaining control – this is when your mentee feels like they are in

control of their own lives and can manage with only occasional support

119

Score of 36: Independent – this is when your mentee does not or no longer

has an issue with a particular area of their life and can manage on their own

8.39 Different tests were developed for adults, for young people and for children. Tests

were translated into Hungarian and Portuguese. English versions of the tests can be

found below.

Data collection

8.40 Psychometric tests were conducted every three months by the evaluation team or

the project coordinator and soft outcome results are recorded alongside with hard

outcome measures. Outcomes measures included:

Money management

Relationships

Life skills

Mental wellbeing

Use of drugs and alcohol

Work, education and training

Offending

Planning

Teamwork

Measurement issues

8.41 In practice we experienced some difficulties in administering the psychometric tests.

The first of these was that tests were administered by the evaluation team when

they made their visits to the pilots. However not all participants were available at

each visit. In Hungary adults were sometimes at work or unavailable and children

could be at school. In Portugal some adults had stopped attending the support

(because they had entered work) or were not attending on the dates of the visits. In

the UK there were difficulties in arranging to see participants on one occasion, let

alone repeatedly. Therefore psychometric tests were obtained from those available

to be interviewed rather than all participants. Pilot coordinators administered tests

and returned them to the evaluation team where possible. This issue could be

overcome in future studies. However in the MEGAN project, because of the

120

importance of the RCT feasibility study, the collection and reporting of hard

outcomes data took precedence over the distance travelled measures.

8.42 In addition, although there were known issues with drugs and alcohol amongst some

of the participants (based on reports from project teams) we found that no

interviewees were willing to admit to such issues in psychometric tests. This is likely

to be because these are issues that participants are sensitive to revealing to the

evaluation team and translators. However, due to issues with literacy it was not

judged to be feasible to ask participants to self-complete the tests.

8.43 In addition to these issues of administration, we also identified a potential issue with

the design of the psychometric tests. At the end of the evaluation we undertook

aggregate analysis of the level of change between all participants with data recorded

at baseline and all those with data recorded at six months after project end. Results

were not significant. One explanation for this may have been that there was no

change. Alternatively it may be that a 4-point scale was too small to enable progress

to be identified. We would recommend a redesign of the scales if these tests were to

be used again.

8.44 As a result of these problems, we have not reported the results from the distance

travelled work as we do not believe it to be robust.