Terms of Reference

Formative evaluation of “Opportunities for Adolescents” modelling project, Romania, 2014-2015

1. Context

In the past years Romania has showed a consistent and strong economic growth with rates above the European Union (EU) average between 2.8% (in 2014) and 3.5% growth rate (in 2013 and 2015) 1. Unemployment rate was 6.8% in 20142, being low compared to EU average (over 10%), but youth unemployment (15-24 years old) is high (24% in 2014)3 and it registers values above the EU average (22.2% in 2014). Reducing poverty and social exclusion remains a major challenge for Romania. Although decreasing, the rate of people at risk of poverty or social exclusion remains high at 40.4 % in 20134, far above the EU average. With a rate of 48.5%, children are at greater risk of poverty or social exclusion than the rest of the population. Romania accounts for one of the highest rates in the EU and for the largest gap between children and the total population. There are also more vulnerable groups of children, such as Roma children who deserve particular attention. According to the European Commission’s (EC) 2012 report on employment and social developments5, half of the adolescents in Romania are at risk of social exclusion6, which is associated with an individual’s difficulty to concurrently belong to the welfare system that promotes social integration, the democratic and legal system, which refers to integration as a citizen, the labour market, which ensures economic integration, and the family and community system, which is related to interpersonal integration.

Romania is subject to great disparities in wealth, opportunities, education, skills, health and these have intensified in many areas in the past decade. There is a profound territorial character to disparities, with pronounced variations between regions, counties and between urban and rural areas, which requires tailored and strategic interventions in order to fight against poverty, social exclusion, improving access to education. Large cities as Bucharest, Cluj, Constanta and Iasi have significant budgets, human resources, collect local taxes, which gives them flexibility in terms of developing services compared to rural communities that manage small budgets and have small revenues from local taxes.

There are around 2 million adolescents (defined as children from 10 to 18 years old) living in Romania, out of which 47% are living in urban areas, while the gender distribution is slightly in favour of boys, with a percentage of 52%. Following the international trend, adolescents in Romania were historically targeted by HIV/AIDS programmes and policies and more recently through youth

1 http://ec.europa.eu/economy_finance/eu/countries/romania_en.htm2 Idem.3 http://ec.europa.eu/eurostat/statistics-explained/index.php/Unemployment_statistics#Youth_unemployment_trends4 http://ec.europa.eu/eurostat/statistics-explained/index.php/People_at_risk_of_poverty_or_social_exclusion5 Employment and Social Developments in Europe, 2012 http://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=7315 6 European Commission, Directorate-General for Employment, Social Affairs and Inclusion: Employment and Social Developments in Europe, 2012

1

employment measures7. Adolescents’ needs and problems are not clearly identified and therefore policies, programmes and projects cannot provide an adequate response to these needs. Previous research8 conducted by UNICEF Romania shows that adolescents face major problems like poverty, social exclusion, and challenges related to education, health and employment.

The Determinants Analysis on Adolescents9 study shows that the most vulnerable adolescents are from Roma families or poor communities, the ones with disabilities, who come from special protection institutions, pregnant girls and adolescents using drugs. According to the findings of the above study, Roma children and adolescents from poor families have low school participation rates and higher drop-out rates. Moreover, looking at children with disabilities, the findings show that they “shoulder the greatest impact of determinants as regards the enjoyment of child rights” 10. In the case of youth leaving the child care system, it is noted that “institutionalisation has a series of negative consequences which reflect on social and professional integration. Often, these young people don’t know how to lead an independent life, are not used to making decisions and sometimes lack the skills needed for practicing a trade11.”

The same study revealed that there are several risky behaviours that adolescents are most vulnerable to and which affect their social, health and education performance. These behaviours include: the use of alcohol, tobacco and of illegal drugs.

42% of adolescents drank an alcoholic beverage at least once during their lifetime. Having drunk alcohol at least once is more common among older adolescents (more than half having drunk alcohol, compared to 21% of under 14-year-olds), in boys (57%, compared to 27% in girls) and in urban areas (half, compared to 32% in rural).

Almost a quarter (23%) of adolescents smoked at least one cigarette in their lifetime; the proportion is higher in adolescents aged 14-18 (33% vs. 6% in the younger group), in boys (32%, compared to 16% in girls) and in urban areas (28%, compared to 19% in rural).

Drugs’ use is not very common among adolescents. However, 5.4% of the adolescents above 14 years of age have used some kind of drugs during their lifetime. Many of them still believe that the occasional use of synthetic drugs is harmless.

The adolescents’ vulnerabilities in urban areas, especially adolescents aged 14 to 18 years old, are mainly due to the exposure to the above mentioned risky behaviours, aggravated by the lack of youth friendly services and of interventions for adolescents using drugs and alcohol. Where such specialized services exist, adolescents are reluctant in accessing them, especially if offered by public institutions, due to excessive bureaucracy, long lists of documents required, associated costs, and lack of confidentiality, etc.

The same research indicated that health, social protection and education systems have limited capacity to identify and provide adapted services to vulnerable adolescents and their families to help them avoid risks, access basic services and fully enjoy their rights. Community-based services are not 7 Youth Guarantee Programme – Under the Youth Guarantee, EU Member States should ensure that, within four months of leaving school or losing a job, young people under 25 can either find a good-quality job suited to their education, skills and experience; or acquire the education, skills and experience required to find a job in the future through an apprenticeship, traineeship or continued education.8 Determinants Analysis on Adolescents, UNICEF Romania, 2013: http://www.unicef.ro/wp-content/uploads/Situatia-adolescentilor-din-Romania-2013.pdf9 Idem10 Determinants Analysis on Adolescents, UNICEF Romania, 2013: http://www.unicef.ro/wp-content/uploads/Situatia-adolescentilor-din-Romania-2013.pdf11 Determinants Analysis on Adolescents, UNICEF Romania, 2013: http://www.unicef.ro/wp-content/uploads/Situatia-adolescentilor-din-Romania-2013.pdf

2

youth-friendly and specialized services for vulnerable adolescents are scarce. Overreliance on the education system to provide life skills and information, limited parenting skills, social norms associated with reproductive health and substance abuse are among the other determinants of the poor prioritization of adolescents.

The latest research12 highlights the importance of non-cognitive skills13 and resilience in the development of adolescents. The concept of “non-cognitive skills” does not appear explicitly in the Romanian policies in education, health, social protection and assistance. However, in the last years, increased efforts to develop coherent sets of national sectorial and inter-sectorial strategies14 explicitly promoted the idea of developing competences. Some of these strategies also include, as specific objectives or measures for specific intervention, several references to non-cognitive skills development under various names (socio-emotional competences, independent living competences, etc.).

A 2014 report on social norms in adolescents15 indicated the need to develop alternative strategies of communication with adolescents. The information-education-communication campaigns against smoking, alcohol consumption and illegal drugs use carried out at school level (at least) have no impact on the teenagers and one of the most important vectors leading to the adoption of a leading behaviour is the image of the teenager within the group: in the groups where smoking, drinking and drug use are seen as cool, teenagers tend to adopt these behaviours to be included in the group; on the contrary, in groups where these behaviours are regarded as not desirable, teenagers tend not to adopt them.

The Government of Romania and UNICEF demonstrated commitment to review and adjust policies promoting adolescents’ development. The category of deprived adolescents were included through the priority objectives set in the National Strategy for the Protection and Promotion of Children’s Rights 2014-202016 and the National Strategy for Youth 2015-202017. The National Strategy for the Protection and Promotion of Children’s Rights 2014-2020 targets the decrease in the number of adolescents using of substances, adolescents who live in institutions and the number of pregnancies in adolescents girl, while the National Youth Strategy for Youth tackles the situation of and policies for the young people defined as “citizens aged 14 to 35 yeas”18. The overall objective of the Youth Strategy is to support active youth participation in the country’s economic, social, educational, cultural and political life, while providing equal opportunities for access to education, employment and decent living conditions, with a special focus on the adolescents and young people who, for various reasons, may have fewer opportunities.

12 Non-cognitive skills in adolescence (final report is under editing), UNICEF Romania, 2015.13 Non-cognitive skills are learnable skills which are scientifically proven to be equally as important as cognitive skills in helping young people build constructive and positive lives and avoid risky behaviours.14 National Strategy for Child Rights Protection and Adoption 2014-2020: http://www.copii.ro/Files/HG%20strategia%20copil.pdfhttp://www.copii.ro/Files/Anexa%201%20Strategia%20nationala%20aprobata%20de%20Guvern.pdfhttp://www.copii.ro/Files/anexa%202%20Strategia%20nationala%20aprobata%20Guvern.pdf15 Social norms analysis in adolescents, UNICEF Romania, 2015: http://www.unicef.ro/wp-content/uploads/Norme_sociale_adolescenti.pdf16 Approved through the Government Decision no. 1113/2014. The National Strategy for the Promotion and Protection of Children’s Rights 2014-2020 targets adolescents, especially through lenses of prevention of risky behaviours, such as substance abuse and early initiation of sexual activity and lenses of mental health. 17 Approved through the Government Decision no. 24/2015. The Strategy sets down a series of specific objectives and lines of action targeting adolescents and youth with increased social vulnerability, as well as others aimed to promote social inclusion and combat youth poverty.18 Youth Law no. 350/21.07.2006.

3

The Romanian legislation does not set down specific provisions for adolescents or adolescence and relevant issues are regulated under laws that concern children and youth. Law no. 272/2004 on protection and promotion of child rights (amended and republished in Official Monitor no. 159/05.03.2014) contains special provisions according to which authorities have to take into consideration the adolescents’ opinions in different situations, such as when applying special protection measures, respect for religious beliefs and schooling choice. Thus, upon reaching the age of 10, 14 or 16 years, increases the diversity of situations in which adolescents are consulted. Youth Law No. 350/2006 sets forth the legal framework “required to ensure adequate conditions for the social and professional integration of young people according to their needs and aspirations”. This regulatory act defines youth as “citizens aged 14 to 35”. Therefore, the target group of this study19 falls within these two categories and under the laws that cover them.

Working with and for adolescents is an emerging area of the UNICEF Country Programme 2013-2017 in Romania. Based on the in-depth determinant analysis of the situation of adolescents 20 focusing on marginalized ones, the Country Programme Action Plan has a specific intermediate result related to adolescents under the Programme Component 1 Social inclusion and disparity reduction that has the following Programme Component Result 1: By the end of 2017, national and local authorities progressively reduce social exclusion and disparities for marginalized children21 in their access to education, early childhood development, public health and protection/prevention services by addressing key institutional and societal bottlenecks at decentralized levels. The result will be produced by developing at local level models of integrated basic services for adolescents aiming at diversifying and increasing the access of adolescent boys and girls, especially vulnerable ones, to basic (education, health, social protection) and specialized services, empower vulnerable adolescents to practice safer behaviours and benefit from interventions and services to reduce their risks and vulnerability to alcohol, tobacco, drugs, unprotected sex and HIV/AIDS. Partnerships are the key element in order to raise awareness and promote positive inter-generational dialogue, social inclusion of vulnerable adolescents and change, and to create an enabling environment for adolescent development. Opportunities are created for various forms of involvement and participation of adolescents in their local communities to voice concerns and propose solutions on issues affecting their lives. Evidence generated through research and by local models will be used for policy advice to inform and shape policies and strategies at national level, and evidence-based advocacy will be used to support the rights of all adolescents, especially the vulnerable ones.

As part of the horizontal cooperation, UNICEF Country Offices in Romania and Bulgaria facilitated an exchange of experience between the two countries in 2014-15. As a result, a National NGO Coalition on adolescents and youth Not in Education, Employment or Training (NEET) has become active in Romania.

To increase the capacity of professionals working with adolescents, UNICEF and its partners organized training sessions, taking into consideration the need assessment of professionals and the latest research in the area of adolescents’ development. This innovative training package includes topics such as: adjustment of services and attitudes towards adolescents, especially vulnerable ones; psychological development of adolescents; non-cognitive skills; role of referral in working with adolescents. The training was delivered to over 100 professionals working with adolescents either in the Resource Centres for Adolescents or in the education, health and social protection systems. The

19 Determinants Analysis on Adolescents in Romania, UNICEF, 2013: http://www.unicef.ro/wp-content/uploads/Situatia-adolescentilor-din-Romania-2013.pdf20 Determinants Analysis on Adolescents, UNICEF Romania, 2013: http://www.unicef.ro/wp-content/uploads/Situatia-adolescentilor-din-Romania-2013.pdf21 Children living in marginalized communities, economically deprived areas, from ethnic minorities, especially Roma, from families living in or at-risk of poverty or dysfunction.

4

training was developed in partnership with specialists from the Cluj University, Faculty of Psychology, and independent experts in adolescents’ development.The evidence generated will also be used to mobilize resources from the national and local budgets, as well as for leveraging funds from the new EU Structural Funds 2014-2020 to ensure the sustainability and replicate the model developed in other areas.

2. Object of the evaluation

The object of the formative evaluation is the modelling project “Opportunities for Adolescents” implemented in five metropolitan areas during the period June 2014 - June 2017: Bucharest (4rd sector), Bacău, Cluj-Napoca, Constanţa, and Iaşi. The formative evaluation will cover the period June 2014 – December 2015. In 2014, the first year of implementation, a Theory of Change was developed – see Annex 1. The model addresses the lack of access to adolescents and their families living in urban communities, especially the vulnerable ones, to information, social, education and health services at three levels: 1/ adolescents and their families, 2/ city/community, and 3/ national/central.The budget allocated for this modelling projects was of approximately 250,000 US$ / year.

2.1. Description of the modelling project

Following the research recommendations to increase the availability and capacity of public authorities/services to support the realisation of adolescents’ rights and to increase cooperation with existing NGOs from urban areas22, UNICEF Romania partnered with: i) the Ministry of Youth and Sports (further referred as MoYS); ii) mayoralties (Department for Social Assistance) in 5 cities; iii) General Directorates for Social Assistance and Child Protection (GDSACP) in Bucharest, 4rd district; iv) NGOs and adolescents in 5 cities and put the basis of an innovative model of integrated services for adolescents, aged 10-18, boys and girls, from urban areas. The model supports adolescents’ access to existing services in health, education and social/child protection, informing them about their rights and opportunities, and enabling their participation in information and capacity building group activities conducted by specialists.

The purpose of the model is to generate evidence for policy development for adolescents, accompanied by resource leveraging. The “Opportunities for All Adolescents” model of intervention, dedicated to the adolescents aged 10-18, boys and girls from five cities and metropolitan areas aims to:

Influence the mind-sets of adolescents and increase their knowledge and awareness regarding the risks of alcohol, tobacco and illegal drugs use, and

Broaden the adolescents’ social capital, especially to increase their skills (including non-cognitive skills) and competencies.

The modelling project was designed and it is implemented in partnership with: MoYS and National Antidrug Agency (NAA) at national level, with a role and contributions in

designing the modelling intervention at city level and analysis of accumulated evidence for adjustments regarding the legislation on adolescents and youth;

Mayoralties in 5 cities and Social Assistance Departments (SAD)/General Directorates for Social Assistance and Child Protection (GDSACP) with a specific role in provision of technical and methodological support to professionals operating at the level of the adolescents model; contributing as well to documentation of progress, participating in evaluations and feedback for key recommendations, as well as for the key adjustments of the modelling project;

22 Idem.

5

NGOs with a role in development and testing of activities at city level mainly through a team of social workers and psychologists: Alianta pentru Lupta Impotriva Alcoolismului si Toxicomaniilor (Alliance for Fighting Against Alcoholism and Drug Abuse- ALIAT, Asociatia Romana AntiSIDA (Romanian Association Against AIDS ) - ARAS, PARADA, Alaturi de Voi (Close to You) - ADV, Forumul Tinerilor din Romania (Romanian Youth Forum) – FTR;

Adolescents with role of inputting the action plans of the services designed for them and providing feedback.

Mayoralties in Bacău, Bucharest and Cluj-Napoca and ADV NGO in Iaşi and Constanţa provided the spaces for the Resource Centres for Adolescents. Multidisciplinary teams of professionals working within the model are hired by ALIAT and ADV and provide services to adolescents. The methodological guidance of the professionals is provided by UNICEF and local partners. At the level of Social Assistance Department in Bacău and Cluj-Napoca cities and GDSAPC in Bucharest, Iaşi and Constanţa, representatives of these institutions facilitate the access of the teams to adolescents in vulnerable situations and facilitate cooperation with other relevant institutions (School Inspectorate, etc.) and NGOs in order to reach adolescents in need of integrated services.

In Bucharest, the partnership at the model level was extended to include adolescents in vulnerable situations and partnerships were recently signed with ARAS and PARADA, NGOs that provide services to adolescents using drugs and adolescents living in the streets. ARAS and PARADA identify and provide services (counselling, referral) to adolescents in vulnerable situations and refer them to the Resource Centre for Adolescents in Bucharest based on a cooperation agreement. This partnership will be strengthened in 2016.

A multidisciplinary team of professionals (psychologists, social workers) trained in adolescents’ development and risky behaviours deliver information to adolescents and

6

parents in schools in five cities, in the area of health, education, social protection and invite adolescents and parents to join the development workshops and support group activities taking place in the Resource Centre for Adolescents and/or organized within the schools;

Social workers from the GDSACP/SAP23 department target the families that are beneficiaries of different schemes of cash benefits (vulnerable families and at risk of poverty) and refer the adolescents and parents from those families to the specialists and activities in the Resource Centres for Adolescents;

Workshops and support group activities are organized in the Resource Centres for Adolescents on a daily basis for all adolescents and parents who access the services of the modelling project. The workshops are adjusted to respond to adolescents needs (e.g. reproductive health, prevention of risky behaviours, development of non-cognitive skills24) and are adjusted to respond to age and gender issues;

For vulnerable adolescents, professionals in the Resource Centre for Adolescents perform a rapid assessment of their needs and, on this basis, refer and accompany them to other services available at the local level. In urgent situations, the professionals apply a special protocol providing adolescents with immediate counselling and accompaniment to the most relevant specialized service available.

For most vulnerable adolescents in Bucharest, professionals from PARADA and ARAS provide specialized services to street children and adolescents using drugs and facilitate their access to the interventions offered by the Resource Centres for Adolescents (e.g. support groups, counselling sessions). The results of this cooperation will be visible after piloting the methodologies for the assistance of those children and adolescents (mid-2016).

In order to ensure adolescents participation in the co-design of the modelling project at the level of each city, with support from Romanian Youth Forum (NGO), a group of adolescents, boys and girls, are actively participating in the development of activities. Adolescents are participating at the staff meetings and are making recommendations on the types of activities organized in schools and in the Resource Centres for Adolescents.

An internal monitoring system was put in place and it is used by the five centres to collect quantitative data on beneficiaries (a database of adolescents, girls and boys, services received, including referral to other services).

As part of the model, UNICEF developed an online platform for adolescents: www.adolescenteen.ro, which includes:

information about the physical, emotional and intellectual development of teenagers throughout adolescence;

information about safe and risky behaviours; a forum where teenagers can ask specialists for solutions to age-specific issues; a database of teen-friendly education, health and protection services available in the five

cities where the model is implemented (Bucharest, Iasi, Bacau, Constanta, Cluj-Napoca).

A Facebook page for teenagers called “Comunitatea de idei” (Ideas Community) is currently running: https://www.facebook.com/comunitateadeidei?fref=ts. The online platform contains information about the services provided by the Resource Centres for Adolescents in 5 cities, including information about other service providers at local level. Professionals from the modelling project

23 General Directorate for Social Assistance and Child Protection/Social Assistance Department.24 Non-cognitive skills are learnable skills which are scientifically proven to be equally important as cognitive skills in helping young people build constructive and positive lives and avoid risky behaviours. Non-cognitive skills are: optimism, motivation, self-control, perseverance, integrity and curiosity etc. Another non-cognitive skills is the resilience which is defined as the capacity of an adolescent to “bounce back” and overcome difficult circumstances and adversity in life. Supporting adolescents to build their resilience involves the “7Cs”: confidence, competences, connections, character, control, coping ability and contribution.

7

interact with adolescents in the online environment and provide counselling and information to adolescents in 5 cities.

As part of the modelling project, UNICEF is implementing a communication for social change campaign. Based on the findings and recommendations of a research25, UNICEF Romania has directed the campaign on preventing adoption of risky behaviours by adolescents, with a focus on alcohol, by building non-cognitive skills in adolescents, namely a self-esteem which allows them to resist peer pressure and a self-image that does not include these specific risk behaviours. This campaign is complemented by targeted interpersonal communication activities with adolescents from the five cities and metropolitan areas.

When it started effectively on the ground in June 2014, the project was implemented in a total number of 5 cities assessed based on specific criteria (no. of adolescents, boys and girls; % of adolescents with risky behaviours; availability of services for adolescents26, relevance of the intervention in the specific context).

The modelling project relies on the principles of equity: all rights, for all adolescents and everywhere, by prioritizing the most deprived. The main categories targeted by the project are adolescents from families affected by poverty, adolescents at risk of initiating risky behaviours, with migrant parents and in single parent families, Roma adolescents, living on the streets and using drugs. For all these categories the modelling project is striving to provide the appropriate support in order to address vulnerabilities by building non-cognitive skills and facilitate access to friendly services.

The formative evaluation will assess to what extent the CPAP27 indicators on adolescents were achieved. The model should reach by the end (December 2017), the following targets:

Table - CPAP indicators

Indicator 2.1. Status of model of integrated basic services for adolescents. Baseline: No model implemented (2012).Status: one model being testedTarget: At least one model of integrated basic services for adolescents evaluated and costed (2017).

Indicator 2.2 Use of new evidence to support the development of policies for adolescents. Baseline: None (2012)Status: 2 studies on adolescents used for policy development Target: Studies/new evidence/evaluations generated and used to support policy development (2017)Means of verification: desk review.

Indicator 2.3. % of adolescents from communities targeted in models28 supported by UNICEF reached by the campaign on social norms who recall campaign messages.Baseline: 0 (2015)Target: 10% (2017)Means of verification: evaluationIndicator 2.4. Number of inter-institutional procedures, guidelines, methodologies for professionals and laws related to provision of services for adolescents29 (incl. vulnerable adolescents) developed in the

25 Social norms on adolescents, UNICEF Romania http://www.unicef.ro/wp-content/uploads/Norme_sociale_adolescenti.pdf26 Mapping of social services for adolescents, UNICEF, 2014. 27 Country Programme Action Plan 2013-2017.28 45 selected communities in Bacau county model and 5 cities (Bucharest – 4th district, Bacau, Cluj, Constanta and Iasi) in adolescents model.

8

adolescent model endorsed by the concerned ministries/agencies.Baseline: 0 (2014)Target: 4 (2017)Means of verification: Desk review.

Indicator 2.5. Number of adolescents evaluated, provided with information and counselling and/or referred to other services in the adolescent model in 5 urban/metropolitan areas.Baseline: 0 (2014)Target: 8,400 adolescents Means of verification: databases of the adolescents centres

It will also look at the performance indicators set for the whole duration of the model implementation (by the end of 2017) and assess to what extent targets have been reached:

At the level of adolescents: o facilitate the access for at least 10,000 adolescents to adolescents and youth-friendly

and other specialised services (out of which 600 adolescents using drugs and alcohol will receive support; 1.000 street children and Roma adolescents included in the assistance programme);

o 40 000 adolescents adolescent girls and boys informed on the availability of adolescent - friendly services and healthy life style in Resource Centres for Adolescents;

o 8 400 adolescents evaluated, provided with information and counselling and/or referred to other services;

o 40 000 adolescents targeted by the communication for social change campaign in 5 cities;

o 50 adolescents involved in the network of adolescents advocates; o 20 000 persons accessing the online platform;

At the level of the authorities: 5 local authorities with capacity increased to address adolescents’ issues, with focus on most vulnerable adolescents;

100 staff from adolescent-friendly services and other specialised services (for adolescents using drugs) in selected counties and cities trained to work with vulnerable adolescents;

At least 5 new adolescents-friendly services developed in selected cities Policies and normative framework respond to adolescents needs (National Youth Strategy and

plans revised); Models of cost-effective interventions on adolescents and most vulnerable adolescents

developed, evaluated and proposed for scale-up. Better policies for adolescents30.

The evaluation of the model will be done based United Nations Evaluation Group norms: relevance, efficiency, effectiveness, and sustainability. However, considering that the modelling project is still in the first phase of implementation, the impact will not be assessed in the context of this formative evaluation. The evaluation will take into consideration if objectives of project were achieved.

29 Methodology for assistance of adolescents who use drugs; inter-institutional procedures for immediate intervention in cases of adolescents living in the streets; youth law amendments; guidelines for professionals (school counsellors, psychologists and social workers) on how to work with adolescents.30 Methodology for assistance of adolescents who use drugs - draft; inter-institutional procedures for immediate intervention in cases of adolescents living in the streets - draft; youth law amendments; guidelines for professionals (school counsellors, psychologists and social workers) on how to work with adolescents – incipient work.

9

3. Rationale for the formative evaluation

Implementation of the modelling project in five cities started in June 2014, after an analysis of situation of adolescents in Romania. The evaluation comes at a moment when the model has to develop/adjust its implementation strategy for the second phase of the model implementation and thus, it will provide key elements for strategic future development in the second phase.

The model of intervention is in the second year of implementation and the moment of evaluation has been chosen because so far the data and evidence generated would allow to evaluate its effectiveness, efficiency and relevance, as well as elements of sustainability (e.g. replicability at national level). The summative evaluation is planned to take place at the end of 2017, when the model is supposed to be taken over by national and local authorities.

Table. 1 Modelling project Workplan 2013-2017

2nd quarter 2013

1st quarter 2014

2nd quarter 2014

1st quarter 2016

4th quarter 2016

2nd quarter 2017

Determinants analysis on adolescents

Mapping of social services

for adolescents in

5 cities

Start of the modelling project on

adolescents in 5 cities

Beginning of formative evaluation

Adjustment of the model

on adolescents

Summative evaluation

* identification of key issues

affecting adolescents

*targeted analysis of

adolescents issues in 5

cities where the model will

be implemented

and analysis of services

* design of minimum package

* the formative

evaluation is expected to

input the modelling on adolescents

* the state co-pilots the

model

* impact of the

modelling project

The primary audience for the evaluation are UNICEF, national and local authorities, and implementing partners. The evaluation findings will be used by UNICEF to inform the second phase of modelling project implementation in 5 cities. UNICEF will use the evidence to advocate for and provide technical assistance to the Ministry of Youth and Sports (MoYS), Ministry of Education (MoE), Ministry of Labour, Family, Social Protection and Elderly (MLFSPE), National Authority for Child Rights Protection and Adoption (NAPCRA) and Ministry of Health (MoH) for developing and amending policies related to development of friendly integrated services for adolescents, prevention of risky behaviours and development of non-cognitive skills in adolescents. All partners will benefit from the conclusions and recommendations to further their own research, interventions and advocacy work. Local authorities will use the evaluation results to further develop and sustain integrated friendly services for adolescents. A summary of the evaluation will be shared with the adolescents.

UNICEF and its partners will be responsible for dissemination of the evaluation findings and ensuring an equitable participation of all interested stakeholders in the process. The formative evaluation is expected to contribute to national and regional knowledge on adolescents “Second Decade, Second

10

Chance”. UNICEF Romania working closely with UNICEF Regional Office for the Central and Eastern Europe and Commonwealth of Independent States (CEE/CIS) and Government of Romania are expected to use the evaluation as a knowledge management tool, as well as a priority topic of interest within future horizontal cooperation initiatives in our region and beyond.

4. Purpose of evaluation

The purpose of the formative evaluation is to analyse the status of implementation and determine if the modelling project needs any adjustments and collect evidence related to the modelling project’s effectiveness and efficiency in reducing risky behaviours in adolescents. Also the evaluation provide information on model’s relevance and in identifying the lessons learned and good practices.

The evidence generated by the interventions at local level and the lessons learned will be used for advocacy and policy advice with authorities and stakeholders at local and national levels to ensure that legislation and policies incorporate an adequate focus on adolescents in health, education and social protection. The cross-sectorial approach at local level is reflected in key strategies, such as the one for child rights promotion and the youth strategy. The evidence generated will also be used to mobilize resources from the national and local budgets, as well as for leveraging funds from the new EU Structural Funds 2014-2020 to ensure the sustainability at local level and replicability of the model developed at national level.

In addition, it will generate necessary evidence to advocate for the need of addressing risky behaviours at adolescents, while developing their strengths and abilities, as well on the need of integrated friendly services for adolescents, especially for the most vulnerable ones.

5. Objectives of the evaluation

There are two main objectives to be attained based on this evaluation:1. Measure the relevance, effectiveness, efficiency and elements of sustainability of the modelling

project in reducing risky behaviours in adolescents in 5 urban areas;

2. Identify the lessons learned and good practices and make recommendations for modelling project adjustment, as well as recommendations on how to promote legislation/policy development related to adolescents (information related to legislation, institutional building, and monitoring and evaluation mechanisms) with a particular focus on the definition of adolescents’ friendly services.

6. Scope and focus

The overall scope of the evaluation is to assess the performance of the current modelling project “Opportunities for Adolescents” addressing risky behaviours in adolescents, boys and girls, especially for most vulnerable adolescents (e.g. Roma adolescents, adolescents living in the streets and using drugs, adolescents from families affected by poverty) and make proposals for improvement and/or adjustment in terms of design, what has worked and what has not worked, internal and external factors that affected in a positive or negative way the modelling project implementation. The evaluation will cover the period of implementation June 2014 – December 2015.

The focus will be on the five cities: Bucharest (4th sector), Bacău, Cluj-Napoca, Iaşi, Constanţa, where the modelling project is implemented and the policy development at national level.

11

This evaluation will consider all stakeholders involved who contributed to current results and will include: adolescents and their families, local stakeholders – professionals working in the modelling project (social workers, psychologists), local authorities, professionals from public authorities in GDSACP/SAPS, local authorities, NGOs, and as well national stakeholders, institutions mayoralties – MoYS, MoE, MLFSPE and NAPCRA, MoH. Other stakeholders should be involved as well, such as: other ministries – European Funding, international organizations - World Bank, relevant NGOs, academia, mass-media, donors, etc.

6.1. Evaluation questions

Considering OECD-DAC criteria31, the evaluation will specifically address the following categories of question which are expected to provide accurate insights related to the objective of the evaluation, scope and focus. The questions below should complemented by specific ones taken into account specific threats, obstacles and bottlenecks if this is considered of strategic importance regarding objectives of the formative evaluation.

Relevance:- To what extent does the modelling project respond to the needs of adolescents, including the

most vulnerable adolescents?- Were the context (political/social/economic/cultural) and latest research taken into account in

the design / implementation of the modelling project? - To what extent is the modelling project relevant to national policies and programmes and cross-

sectoral strategies32?- To what extent is the modelling project relevant in relation with CEE/CIS Regional Key

Leadership Area (RKLA) 1033?

Effectiveness: – Has the modelling project achieved satisfactory results in relation to stated objective?– What is the degree of achievement of the modelling project’s expected results in accordance

with the targets set for the performance indicators?– Are the services provided effectively in order to reach adolescents, including most vulnerable

adolescents? Are all processes based on a child rights approach? Are any age and gender issues considered in providing the services?

– What are the main constraints on supply and on demand? Which parts of project are most and least effective? What factors explain success?

Efficiency: – Does the modelling project use resources in the most economical/efficient manner to achieve

expected results? What is the value added (in terms of financial value) of the modelling project? Are any other economical alternatives feasible? How cost-effective is the project for reaching most vulnerable adolescents?

– How does project costs compare to other similar programmes or standards?34 – How do costs for reaching adolescents, including most vulnerable ones, compare with public

services costs?

31 http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm 32 National Strategies on Health and Health Services, on Youth, on the Protection and Promotion of Child Rights, on Social Inclusion and Poverty Reduction, on Antidrug, etc.33 CEE/CIS RKLA 10 on adolescents – Second Decade, Second Chance – concept notes to be made available for the evaluation team.34 Such as the ones defined by the Government Decision no. 23/ 6 January 2010, Standard cost in social services.

12

Sustainability: – To what extent is the current context more or less favourable to continue such approaches in

the near future?– Are the interventions modelled likely to continue when external support is withdrawn?– Is the modelling project replicable? At local or national level?

Impact: N/A in the context of this formative evaluation

Lessons learned and unplanned outcomes: – What are the lessons learned at each level of intervention that should be taken into account for

implementation and further replication of the modelling project? – Are there any unplanned outcomes worth considering for filling in capacity gaps and/or

addressing remaining bottlenecks?

7. Methodology

The approach followed from the outset of the evaluation will be as participatory as possible. Stakeholders at all levels, including vulnerable adolescents and their families, will participate in the evaluation through discussions, consultations, provision of comments on draft deliverables and some will reply to the recommendations made by the evaluation in the management response. In gathering data and views from stakeholders, the evaluation team will ensure that it considers a cross-section of stakeholders with potentially diverse views to ensure the evaluation findings are as impartial and as representative as possible.

The evaluation will apply the UNEG norms and standards35, including evaluation criteria of relevance, efficiency, effectiveness, and sustainability, and the UNEG ethical guideline36, including UNICEF Procedure for Ethical Standards in Research, Evaluation, and Data Collection and Analysis (effective as of 1st April 2015), in order to ensure quality of evaluation process. Moreover, the evaluation should mainstream gender and human rights considerations throughout. Concerning gender, the formative evaluation will carefully analyse aspects related how the model responds to the individual needs of adolescent boys and girls. The report should use gender-sensitive, child-sensitive and human rights-based language throughout, and whenever possible, disaggregation of data by gender, age, ethnicity and social origin, should be made.

The evaluation team will propose the methodology design which should demonstrate impartiality and lack of bias by relying on a cross-section of information sources (e.g. stakeholder groups, including beneficiaries, etc.) and using a mixed methodological approach (e.g. quantitative, qualitative, participatory) to ensure triangulation of information through a variety of means.

As concerns evaluability, the theory of change and data available allow for the assessment of the progress achieved and evaluation of the project results. The evaluation will consider the baselines and the targets set for indicators, which are considered reliable, and the evaluation team will have access to all information related to the modelling project at the level of each involved institution. The evaluation will rely as well on relevant available data resulted from project monitoring or other means of data generation and interpretation, such as: reporting materials from partners; monitoring reports, including UNICEF monitoring field trips and experience exchanges; databases of the Resource Centres for Adolescents, forms, fiches, etc.; minutes and lists of participants from information sessions. All these data sources are assessed as highly reliable, including data collected

35 UNEG Norms for Evaluation in the UN System. 2005, http://www.uneval.org/document/download/562 36 UNEG Ethical Guidelines for Evaluation. 2008, http://www.uneval.org/document/download/548.

13

through the databases, since control mechanisms have been in place at all data collection levels (Resource Centre for Adolescents, city, and at model level).

The evaluation will use mixed methods in order to assure data triangulation and it could integrate:- Analysis of the quantitative data collected by the Resource Centres for Adolescents. The

evaluation will seek to collect disaggregated data based on the following criteria: geographical – at city level; gender – boys/girls, male/female; ethnicity; age groups.

– Qualitative data, obtained through in-depth interviews and focus groups with key informants in the government, public authorities at city level, partner organizations (civil society and intergovernmental organizations), staff working in the modelling project, service users, and others.

Most of existing sources of information assessed as the most reliable, such as reports and studies already referred to, are referenced by web links. Additionally, the following data sources with a contact list of all for the project relevant stakeholders, project implementing partners and consultants will be made available to the evaluation team once a contractual agreement has been made:

– UNICEF programme materials such as country programme documents, strategies, project proposals and reports to the donors.

– Modelling project documents such as monthly and annual reports of partners and professionals working in the modelling project, etc.

The quality assurance process will consist in the following steps: review of research tools prior to collecting the data, review of all deliverables and corrective actions recommended. All evaluation tools and deliverables will be reviewed by the Adolescent Development Specialist and Child Rights Systems Monitoring (M&E) Specialist.

7.1. Limitations of the evaluationPotential limitations and risks may be linked to the staff fluctuation within the involved institutions during the implementation of the model. It is possible that staff initially part of the modelling is no longer working for the partner of the modelling. Other risks might be related to the identification of adolescents who were part of the initiative due to change in their locations (many of them may have graduated high-schools targeted by the model) and guardians /parents can be reluctant in allowing adolescents to participate in the evaluation (informed consent). Other risks may refer to the political changes in the Government that may interfere with the implementation of data collection and consultations. As both local and national elections are scheduled this year, political changes in the local and national administration may happen.

8. Work Plan and tentative time frame

The evaluation process will include an inception phase during which a detailed evaluation framework and an inception report will be prepared. The evaluation framework will build on the Theory of Change. It will provide details on how to respond to the evaluation questions, which indicators to use, sources of verification.

This phase may include and/or be developed in parallel with the comprehensive analysis of available information – desk review: including national laws, policies, action plans etc., county and local strategies, reporting materials from partners; monitoring reports, including UNICEF monitoring field trips and experience exchanges. This phase may also include initial interviews with key

14

stakeholders at national level (i.e. MLFSPE, NAPCRA, MoH, MoY, MoE), at city level (GDSACP/SPAP) and NGOs partners.

It is expected that data collection will start after submission and approval of inception report, including proposed methodology and instruments.

By end of data analysis a draft evaluation report is to be submitted according to UNICEF standards and Global Evaluation Report Oversight System (GEROS)37. After submission of comments from UNICEF and key stakeholders, a final evaluation report, including an executive summary38 and a bibliography annex, are to be submitted for review.

In this context, phases and tentative time frame are proposed in the table below:

Phases and time frame Expected activitiesInception phaseBeginning of April 2016 Inception meetingEnd of April 2016 Submission of the inception reportMid-May 2016 Comments for the inception reportMid-June 2016 Approval of inception reportData collectionMid-June 2016 Submission of data collection instrumentsEnd of June 2016 Comments on proposed instrumentsBeginning of July 2016Mid- September 2016

Finalization of instruments and pre-testingField data collection

ReportingEnd of October 2016 Submission of 1st draft reportEnd of November 2016 A PowerPoint presentation of the preliminary findings for the meeting

with major stakeholders to present findings and preliminary conclusions, discuss and finalize the recommendations to be organized by UNICEF.

Mid-December 2016 Submission of final evaluation report.DisseminationMid-December 2016 Development of a brief advocacy note (around 2,000 words)

summarizing key findings of evaluation, relevant policy issues and recommendations and a PowerPoint presentation of key finding and recommendations

Mid-January 2017 Launch of the evaluation reportPost-evaluationEnd of January 2017 Development of management response

9. Deliverables

– Inception report including the evaluation methodology approved by Mid-May 2016 (in English)– Evaluation instruments finalized and pre-tested by end of May 2016 (in Romanian)– Field data collection completed by the end of June 2016– Draft Evaluation Report according to UNICEF standards and GEROS by end of August 2016 (in

English and also translated into Romanian)– A PowerPoint presentation of the preliminary findings by end of September 2016 (in Romanian)

37 http://www.unicef.org/evaluation/files/UNEG_UNICEF_Eval_Report_Standards.pdf 38 Recommendations for ‘Writing a good Executive Summary’ are attached in Annex 2.

15

– Final Evaluation Report (including an executive summary and a bibliography annex), complying with UNICEF Evaluation Report Standards and GEROS by end of October 2016 (in English and also translated into Romanian)

– A brief advocacy note (around 2,000 words) summarizing key findings of evaluation, relevant policy issues and recommendations by end of November 2016 (in Romanian)

Evaluation team will participate in the meeting with major stakeholders to present findings and preliminary conclusions, discuss and finalise the recommendations and in the launch of the final evaluation report.

10.Evaluation team, required experience and credentialsThe independent evaluation team, institution/organization and/or consortium should be built of national experts. The team should be led by an experienced evaluator to be supported by at least one or two experts on social/child protection/adolescents development. To strengthen their capacity for performing the task, applicants may establish cross-sector forms of association, such as between experts and/or organizations/institutions in various fields of practice. The evaluation team will have to comply with the Ethical Code of Conduct for Evaluation on the UN System (UNEG/FN/CoC[2008])39 and UNEG ethical guideline.

Competencies required by the team to carry out the evaluation are a combination of a number of years of experience in the subject area and of evaluation methods as per below:– Advanced university degree in social sciences, law, political science or public policy;– Proven knowledge in adolescents development, including most vulnerable ones, child rights,

child/social protection and health systems and on reforms and policy debates in these areas;– Familiarity with rights-based approaches and with principles of gender mainstreaming;– Excellent knowledge and expertise in designing and conducting evaluations, knowledge

management and research;– Proven experience in conducting data collection for various research, incl. participatory

approaches and methods; proven ability to conduct interviews, focus group discussions and writing reports for publication; proven experience in conducting desk reviews and field visits;

– Strong analytical and conceptual thinking;– Excellent oral and written English language skills, demonstrable with samples of publications

(evaluation reports, relevant research, etc.); ability to synthesize complex information into key messages;

– Ability to work in a multi-disciplinary team and establish harmonious and effective working relationships;

– Familiarity with the work of the United Nations an asset;– Availability for work within the proposed time frame;– Ability to communicate and expertise in cooperation with different stakeholders, professionals,

communities, families and children.

Successful applicants will provide samples of evaluations conducted; those should include, but not be limited to, programme & policies evaluations.

39 Ethical Code of Conduct for Evaluation on the UN System (UNEG/FN/CoC[2008]), http://www.unevaluation.org/document/detail/100

16

11. Roles and responsibilities of stakeholders in the evaluationIn order to provide adequate support for performing the formative evaluation, the following roles and responsibilities of stakeholders are suggested below:

External stakeholders– At local level: adolescents and their families, social workers and school counsellors at city level,

professionals working in the modelling project, public local authorities, NGOs, are already informed about the process of evaluation and are expected to contribute during data collection process and some to provide feedback on draft report.

– At national level – NAPCRA, MoE, MoY, MoH, NAA will participate in the evaluation through discussions, consultations, provision of comments on draft documents and some will reply to the recommendations made by the evaluation in the management response.

UNICEF Country Office – The UNICEF focal point for the evaluation is the Child Rights Systems Monitoring Specialist

(M&E) ensures that the evaluation process is carried out as per UNICEF policies and provide technical support to the evaluation throughout the process.

– The Adolescents Development Specialist is the key informant throughout the evaluation process: prepares the TORs for the evaluation exercise in consultation with the Child Rights Systems Monitoring Specialist (M&E) and suggests the best proposal for the evaluation; liaises with evaluation team and provides initial briefing to evaluators on the framework and expectations of the evaluation; provides feedback on evaluation design and research tools and all reports and deliverables; facilitates contact with county and local stakeholders included in the evaluation exercise; facilitates access to complementary background documents to be included in the desk review and to all necessary documents throughout the evaluation process;

Evaluation team– Has the overall responsibility for successful completion of all phases of the summative

evaluation including inception, evaluation tools and methodology, data collection and reporting;– Manages and carries out all consultations, meetings, focus groups and interviews with key

informants, including logistics related to travel, financial and other arrangements that are related to the implementation of the evaluation;

– Submits deliverables and invoices (if applicable) in a timely manner.

12.Evaluation budget and sources of fundingA detailed budget for the evaluation will be part of the financial proposal from the evaluation teams when they express their interest for the evaluation.

The estimated budget for the formative evaluation is around 40,000 USD and the source of funding is SC 150335, Velux. This amount does not include organization of consultation meetings with stakeholders and launch of the report which will be covered by UNICEF separately.

13.General conditionsReporting. The contractors will report to UNICEF Child Rights Systems Monitoring (M&E) Specialist and will work closely as well with UNICEF Adolescents Development Specialist.

Payment calendar. Taking into account the tasks and timeframe mentioned above, fees will be paid in three instalments after submission of deliverables and upon approval by supervisor, as follows:– 30% of the contract total will be released upon acceptance by UNICEF of the inception report;

17

– 30% of the contract total will be paid after approval by UNICEF of the draft report;– 40% of the contract will be paid after submission and approval by UNICEF of final evaluation

report and all requested deliverables.

Ownership. UNICEF will have sole ownership of all final deliverables; no parts of the methodology will be reproduced without the permission of UNICEF.

18

Annex 1. Theory of Change for the modelling project ‘Opportunities for Adolescents’ 2014 – 2015

Initial Activities Expected outputs of OPA Expected outcomes of OPA Impact

Ado

lesc

ents

and

thei

r fam

ilies

, esp

ecia

lly v

ulne

rabl

e on

es, l

ivin

g in

ur

ban

com

mun

ities

lack

acc

ess

to in

form

atio

n, s

ocia

l, ed

ucat

ion

and

- Adolescents, including vulnerable ones, access friendly cross-sectoral services and information (face-to-face and online);

- Parents of adolescents access parenting classes, counselling and information about teens’ development to build their capacity to care for their children

- Adolescents are more informed about their rights, development and increase their capacity to make healthy choices

- Vulnerable adolescents are identified and referred to specialized social and medical services and schools.

- Parents are more informed and have increased capacity for better parenting

- Adolescents40 practice healthier behaviours;- Adolescents have increased access to life skills

counselling and have better access to health and social services

- Parents provide a more supportive family environment for adolescents;

Adolescents’ needs are m

et by their families and com

munities,

resulting in a reduction of these adolescents being in vulnerable

City

/Com

mun

ity

10041 professionals hired and trained to:- carry out outreach activities and provide information

to adolescents in schools and communities;- organize workshops on adolescents development,

non-cognitive skills and healthy behaviours for adolescents and their parents in the Resource Centers;

- liaise with and refer cases of very vulnerable adolescents to professionals within local public structures (schools, social workers, health services)

- organize capacity building workshops for municipalities representatives on adolescents development;

- develop online platform in order to facilitate adolescents participation;

Integrated services are developed in close partnership with local authorities and service providers based on needs assessment

- 5 urban centres accessible and adapted to adolescents, including vulnerable ones, with professionals able to work in a friendly manner with adolescents, develop their non-cognitive skills and strengthen their capacity to adopt healthy behaviours.

- Health and social services, and schools are sensitised to adolescents’ needs

- Municipalities are sensitised and increase their capacity to support services for adolescents development

- 5 functional and sustainable Resource Centres provide quality, age-appropriate, cross-sectoral, preventive and referral services and information to adolescents and their parents, including vulnerable ones;

- 5 functional local networks of resources centres, health and social services, and schools work together to identify and support that act in the benefit of adolescents;

- Municipalities support adolescents’ centres and other services for adolescents with financial and human resources

40 - Approximately 10 000 adolescents out of which 8.400 adolescents from general population; 1.000 street and Roma adolescents and 600 adolescents using drugs41 Professionals include personnel of the Resource Centres for Adolescents, from schools, General Direction for Child Rights Protection and Social Assistance; Public Services for Social Assistance; Counselling Centers for the Evaluation, Prevention, Counselling and Treatment of Drug Abuse.

19

heal

th s

ervi

ces,

he

nce

are

at ri

sk o

f so

cial

exc

lusi

on

Nat

iona

l/ ce

ntra

l

- Development of new methodologies/ procedures to identify and support adolescents and their parents, especially vulnerable ones (incl. adolescents using drugs and those living in the street)

- Guidelines for school counsellors on adolescents’ development and non-cognitive skills.

- Costing of Resource centres- Development of amendments and appropriate budget to

youth law- Advocacy for national replication

- Methodologies, procedures and guidelines adopted

- Youth law amendments and budget adopted;

- Better quality, more age-appropriate, cross-sectoral, preventive and referral services and information are available to adolescents, especially vulnerable ones, at the national level

20

Annex 2. Writing a good Executive Summary

Primarily for key decision-makers who do not have time to read the full evaluation report, an Executive Summary should provide an overview of the essential parts of a report: a summary of the project/programme evaluated, the purpose of the evaluation, the methods used, the major findings and the recommendations. It should be very short — ideally two to three pages — and should “stand alone” (without requiring reference to the rest of the report).The Executive Summary should be clear, simple and comprehensible to those not familiar with the programme, allowing readers to quickly grasp the important findings and recommendations. The Executive Summary format below has two parts. The first part is standard bibliographical information, name of report, name of author, etc. The second part is the Executive Summary itself: background, purpose, methodology, findings, and recommendations. Terms of Reference Page 2

Section 1 – Bibliographical Information

Title: Usually includes the type of report and name of the projectExample: Evaluation of Early Childhood Development Programme

Author(s): Names of all the authors of the report in the following format: Last Name, First Initial; Last Name,First InitialExample: Macom, X.; Pickett, W.

Institutions: Name of the institution contracted to implement the study, survey or evaluation. If a consultant is hired independently of an institution, this may be left blankExample: London School of Hygiene and Tropical Medicine

Date: Year of publication

Region: Abbreviated name of UNICEF region. CEE/CIS, EAPRO, ESARO, MENA, ROSA, TACRO or WCARO

Country: Country name

Type: Survey, study or evaluationSurvey: An assessment of the conditions of specified population group/s (children, women, adolescents) or public goods (health services, school, water system) at a point in time, e.g. MICS, KAP surveys; survey of quality of health services; Hygiene Practices surveyStudy: An investigation designed to improve knowledge about something (a problem or phenomenon) and understand its direct and underlying causes as well as its consequences on people or environment, e.g. A study of vulnerability of young people to illicit substance abuseEvaluation: A process to determine as systematically and objectively as possible the value or significance of a programme, project, policy or strategy, e.g. Evaluation of effectiveness and sustainability of UCI programme. Judgement on the value or significance is based on criteria such as relevance, effectiveness, efficiency, impact and sustainability. In rights-based programming, coverage (to what extent all social groups benefit) is another important evaluation criterion. Evaluations can use data from surveys and studies. Further, in evaluation of humanitarian assistance programmes, four additional criteria are recommended: coverage, coordination, coherence and protection.

Theme: The theme of an evaluation or study is the same as the theme of the related project or programme.Page 3 Evaluation Technical Note SeriesPartners: Sponsoring organizations (funders and /or initiators) UNICEF in collaboration with a donor, another international organization, a government ministry, a local research institution, etc.

Languages: Languages that the report is available in

Section 2 — Summary

21

Background:Brief information about the programme or project that the evaluation is related to, including the major stakeholders, partners and implementing organizations/agencies involved and their respective roles, as well as a brief description on how the evaluation came about.

Purpose/Objective:A statement of why the assessment is needed, how it will benefit the program/project. List the objectives, specifically, what you hoped to learn from the evaluation, questions to be answered by the evaluation.

Methodology:A short description of the type of research methods used: sampling (describe the composition, location and number of people surveyed/interviewed/observed and how they were selected), data collection (survey, interviews, observation, desk review, etc.) Actual questions and surveys do not need to be included. Please include limitations if there are any significant drawbacks the audience should be aware of.

Key Findings and Conclusions:Summary of significant findings and conclusions of the evaluation. All results do not need to be reported in full. Include important data and relevant, succinct conclusions drawn from findings. (If you have difficulty deciding what to include, a good rule of thumb is to look at those findings that led to your conclusions/recommendations; also key conclusions should be short answers to questions raised in the Objectives section.)

Lessons Learned:Recommendations that can be generalised beyond the specific case to apply to programs globally.

Recommendations:Overall suggestions of how the project/program can be improved based on the findings.

22