module i: evaluation-basics, theory of change and logic of ......module i: evaluation-basics, theory...

Module I: Evaluation-Basics, Theory of change and logic of intervention, Research methods

Objectives:

Gain insight on the basic evaluation concepts, purpose and standards, introduces the terminology, critical issues and current debates in the field of evaluation.

Gain practical experience through a series of exercises involving the design of a conceptual framework and the theory of change

Explore and learn qualitative and quantitative research skills for project planning, monitoring and evaluation activities.

The program content:

Evaluation-Basics : Definition of Evaluation, Why is Evaluation important, Terminology, Evaluation and the project cycle management, Type of evaluations, differences and similarities between evaluation and other organizational functions: audit, monitoring, reviews, social research, knowledge management systems

Theory of change and logic of intervention: Causal chain, Logical framework, Theory of change, Developing a theory of change, Components: Activities, outputs, outcomes, impacts;

Research methods: Data Collection and Analysis- Quantitative data; Methods for collecting quantitative data; Challenges and limitations of quantitative methods; Qualitative data; Methods for collecting qualitative data; Challenges and limitations of quantitative methods; Analysis;

Literature: Reference literature: Reading materials include textbook chapters and current literature. Assignments will be posted on the course website

EvalCommunity (2017) – E-Learning tools: http://www.evalcommunity.com/resources/e-learning/

PATTON, M.Q. (2001), “Evaluation, knowledge management, best practices, and high quality lessons learned”, American Journal of Evaluation, 22(3), 329-336.

PATTON, M.Q. (2002) Qualitative Research and Evaluation Methods, Thousand Oaks, Sage Publications;

RIDDE Valery et DAGENAIS Christian, Les Presses de l’Université de Montréal (2012). Approches et pratiques en évaluation de programmes, Nouvelle Edition;

United Nations Evaluation Group (2016). Norms and Standards for Evaluation. New York: UNEG. Electronic version of this publication is available at: http://www.unevaluation.org/2016-Norms-and-Standards

http://www.evalcommunity.com/resources/e-learning/

http://www.unevaluation.org/2016-Norms-and-Standards

Word Bank - Ray C. RIST and Jody Zall Kusek “Ten Steps Ten Steps to a Results Based Monitoring and Evaluation System” (2004) Electronic version of this publication is available at: https://www.oecd.org/dac/peer-reviews/World%20bank%202004%2010_Steps_to_a_Results_Based_ME_System.pdf

https://www.oecd.org/dac/peer-reviews/World%20bank%202004%2010_Steps_to_a_Results_Based_ME_System.pdf


1

Module II: Process of evaluation, planning evaluations, managing evaluations

Objectives:

To gain an understanding of evaluation process and the important elements that makes up the design of an evaluation process, methods and techniques to achieve this successfully, learn how to apply them;

To develop successfully and deliver a planning evaluation with all tools and techniques as the first step of evaluation process;

to understand the concept of effective managing evaluation and develop awareness of how to manage evaluation


Process of Evaluation: Determine the purpose and users, planning of evaluation, develop measurable objectives, develop evaluation questions, collect credible evidence, analyze information and develop conclusions, report finding – draft and final report, use and follow up: management response, dissemination of the evaluation products, and follow up to the implementation of the management response.

Planning evaluations: what is an evaluation plan and important consideration, why it is needed and what does it include, who is required to develop it, when should the evaluation plan be undertaken and carried out, can the evaluation plans be amended? It also provides step-by-step guidance on how to develop an evaluation plan and provides tools for evaluation planning.

Managing evaluations: Evaluation management overview, logistic management, calendar, budget management, organizing the evaluation’s human resources, role, responsibilities and management of stakeholders, risk management, quality management, resistance and conflict,

Reference literature: Reading materials include textbook chapters and current literature. Assignments will be posted on the course website


PATTON, M.Q. (2001), “Evaluation, knowledge management, best practices, and high quality lessons learned”, American Journal of Evaluation, 22(3), 329-336.

RIDDE Valery et DAGENAIS Christian, Les Presses de l’Université de Montréal (2012). Approches et pratiques en évaluation de programmes, Nouvelle Edition;


2


Word Bank - Ray C. RIST and Jody Zall Kusek “Ten Steps Ten Steps to a Results Based Monitoring and Evaluation System” (2004) Electronic version of this publication is available at: https://www.oecd.org/dac/peer-reviews/World%20bank%202004%2010_Steps_to_a_Results_Based_ME_System.pdf




Table 2. List of predicting professors for teaching

Academic staff (full-time or part-time) Title or degrees Institution Department

Fation Luli Master in Evaluation

ASPE – Albanian Society of Program Evaluation

President Coaching and Training

Suela Sefa Master in Public Policies

ASPE – Albanian Society of Program Evaluation Coaching and Training

Virgil Pamfil Master in Management

RoSE – Romanian Society of Evaluators Coaching and Training

Mirela Mustata PhD in Communication


Course Program: Module III: Cost-Benefit Analysis Method

(developed in the study program: the MP-Evaluation) Lecturer: TO BE DECIDED Course: Cost-Benefit Analysis Method – a tool to assess Investment projects Semester: I Hours load (3 lectures / seminars 2) Number of credits: 5 Objectives:

To increase understanding of theoretical foundations in economics for analysis of costs and benefits,

To develop skills to apply these economic principles towards an improved understanding of how to assess all relevant costs and benefits of a project, action, policy or program.


Introduction to the course. Introduction to cost-benefit analysis. Aim of cost-benefit analysis. Importance to policy-makers and link to Impact Assessment. Ex ante and ex post cost-benefit analysis. Guidance documents

Basic issues in cost-benefit analysis. Defining the baseline. Rationale for Cost Benefits Analysis. Identification of project for funding. Description of various EU funding schemes. Discussion of Projection objectives and relevance for funding. Project Identification. Option Analysis. Common Errors in CBAs regarding these points

Financial Analysis. Background. Principles of Financial Analysis. Basic Elements of Financial Analysis Financial Return on Investment. Sources of Financing. Financial Sustainability. Determining Total Costs and Revenues. Financial Analysis Indicators

Economic Analysis. Background. Principles of Economic Analysis. Conversion of market to accounting prices. Monetisation of non-market impacts. Inclusion of indirect effects. Social Discount Rate. Economic Indicators

Risk Assessment. Background. Sensitivity Analysis. Calculation of the Distribution of the Performance Indicator. Major Downfalls of Risk assessments in CBA analysis. Discussion of the Results and of the Ways to Mitigate Risks

CBA Step-by-Step. Case studies. Overview of project appraisal. Context analysis and Project objectives. Project Identification. Project Selection/Option analysis. Financial Analysis. Economic Analysis. Risk Assessment. Overview of Project Appraisal. CBA Check list

Literature: Literature: Lecture prepared by the teacher Reference literature:

Reading materials include textbook chapters and current literature. Assignments will be posted on the course website

Guide to Cost-Benefit Analysis of Investment Projects - Economic appraisal tool for Cohesion Policy 2014-2020 http://ec.europa.eu/regional_policy/sources/docgener/studies/pdf/cba_guide.pdf .

Project Cost Estimating Manual https://www.tmr.qld.gov.au/-/media/busind/techstdpubs/Project-delivery-and-maintenance/Project-Cost-Estimating-Manual/Nov-2015/ProjectCostEstimatingManual.pdf

http://ec.europa.eu/regional_policy/sources/docgener/studies/pdf/cba_guide.pdf

https://www.tmr.qld.gov.au/-/media/busind/techstdpubs/Project-delivery-and-maintenance/Project-Cost-Estimating-Manual/Nov-2015/ProjectCostEstimatingManual.pdf














Course Program: Module IV: Evaluation of policies – specific features

(developed in the study program: the MP-Evaluation) Lecturer: TO BE DECIDED Course: Module IV: Evaluation of policies – specific features Semester: I Hours load (3 lectures / seminars 1) Number of credits: 5 Objectives:

To increase understanding whether a specific policy is achieving its objectives by tracking implementation quality and by ascertaining whether there was a causal impact, either intended or unintended

To develop research and investigative skills such as problem framing and solving and the ability to assemble and evaluate complex evidence and arguments.

To develop communication skills to critique, create and communicate understanding and to collaborate with and relate to others


Policy basics. What is the meaning of policy? Features of public policies. Why is policy evaluation important? Evaluation within the policy process. Political decision making process. Policy evaluation challenges Policy cycle.

Policy implementation instrument. Criteria for selecting instruments to implement a policy. From a policy proposal to a Government decision. Steps in policy development. Problem

definition. Defining policy options – comparative analysis. Stakeholders analysis. Consultation process. Impact assessment of options

Policy impact assessment (PIA). What is impact assessment? Benefits of evaluation. Summary of key analytical steps. Defining the scope and level of analysis of the PIA. Gathering information and consulting stakeholders. Key analytical steps in impact assessment. Assessing administrative burdens. Identifying potential obstacles and incentives to compliance. The evaluation criteria. Ranking the options. Arrangements for future monitoring and evaluation

Methods and techniques for policy evaluation. Case studies. Cost benefit analysis. Cost effectiveness analysis. Expert panels. Focus Groups. Interviews. Models. Multi-criteria analysis. Observation techniques. Priority Evaluation method. Regression analysis. SWOT Analysis.



The Magenta Book -Guidance for evaluation, 2011, ISBN 978-1-84532-879-5

EC - Better regulation for better results - An EU agenda, COM(2015) 215 final http://ec.europa.eu/smart-regulation/guidelines/docs/swd_br_guidelines_en.pdf

EC- Better Regulations Toolbox 2015, http://ec.europa.eu/smart-regulation/guidelines/docs/br_toolbox_en.pdf

http://ec.europa.eu/smart-regulation/guidelines/docs/swd_br_guidelines_en.pdf

http://ec.europa.eu/smart-regulation/guidelines/docs/br_toolbox_en.pdf

http://ec.europa.eu/smart-regulation/guidelines/docs/br_toolbox_en.pdf












Course Program: Module V: Evaluation of programs - specific features

(developed in the study program: the MP-Evaluation) Lecturer: TO BE DECIDED Course: Module V: Evaluation of programs - specific features Semester: I Hours load (3 lectures / seminars 1) Number of credits: 5 Objectives:

To provide students with the theoretical and practical bases for program evaluation

To develop basic skills in a variety of approaches to evaluation, including techniques that are particularly suitable for evaluating of various programmes

To develop research and investigative skills such as critically assess evaluations and interpret evaluation findings, design and prepare an evaluation plan, explain how ethical principles apply to evaluation, incorporate principles of cultural competency into evaluation design, methods, and language used, etc.

To develop communication skills to collaborate with program stakeholders about their programs and about evaluation and integrate input from program stakeholders in the development of an evaluation plan


Program evaluation basics. What is a program evaluation? Terms linked to programme evaluation. Inter-relationships between Program Evaluation and other evaluative activities. Program features. Purpose of project and programme evaluations. Program Evaluation

Principles and standards. Governance of Program Evaluations. Roles and responsibilities of Program Evaluation team

Steps for implementing Program Evaluations. Processes in managing program evaluations including set-up, implementation, governance and reporting. Types of programme evaluation – internal vs. external evaluation. Ex-ante, mid-term and end-of-phase, ex-post programme evaluation.

Programme Evaluation criteria. And examination questions. Relevance. Effectiveness. Efficiency. Impact. Sustainability. Specific criteria used in Programme evaluation for Humanitarian Aid - Relevance/Appropriateness, Coherence, Coordination, Efficiency, Connectedness, Coverage.

Programme evaluation process. Who Does What. Evaluation process – phases, functions, steps. How evaluations are scheduled. Focus on results – results based management. Challenges and expectations. Integrating Gender Equality. Writing TORs. - checklist. Selecting an external evaluator vs. engaging local professionals. Qualifications required – self-evaluation. Preparing Evaluation workplans. Evaluation design and methodology. Setting sources of information and methods for collecting relevant information. Setting accountabilities and reporting. Workplan checklist.

Methods and techniques for program evaluation. Setting evaluation criteria and relevant questions. Integrating Gender Equality. Information collection and analysis. Identifying results. Preparing and writing evaluation reports. Evaluation reports -framework and roles. Lessons learned. Assessing evaluation reports – evaluator’s self-assessment. Getting approval. Optimizing benefits – building value. Key success factors. Participatory evaluations. Tips and Case Studies



EC - Europe-aid Co-operation Office, Joint Evaluation Unit - Guidelines for Project and Programme Evaluation, volume 3, 2006 ISBN: 92-79-00681-9 https://www.oecd.org/dac/evaluation/dcdndep/47469201.pdf

https://www.oecd.org/dac/evaluation/dcdndep/47469201.pdf







Course Program: Module VI: Evaluation of projects – specific features (developed in the study program: the MP-Evaluation) Lecturer: TO BE DECIDED Course: Module VI: Evaluation of projects – specific features Semester: I Hours load (3 lectures / seminars 1) Number of credits: 5 Objectives:

To enable a clear, understanding of the vocabulary and standards of project evaluation and to provide information on different EC funding opportunities for projects

To provide students with the theoretical and practical bases for evaluation of projects as part of Project Cycle Management

To develop basic skills in a variety of approaches to evaluation, including the developing of a project logic model that summarizes a project’s promises

To develop research and investigative skills such as critically assess evaluations and interpret evaluation findings, design and prepare an evaluation plan, explain how ethical principles apply to evaluation, incorporate principles of cultural competency into evaluation design, methods, and language used, etc.

To develop self-assessment skills and knowledge to comply with best international practice.

To develop communication skills to collaborate with program stakeholders and to present the results of evaluation in the evaluation report


Project management basics. The three levels of evaluation. The Project approach. Types of projects. EC Policy, programmes and projects. What Is Project Management? Understanding the Project Life Cycle. The project cycle, main documents and responsibilities. The Underlying Principles of Project Management. Project lifecycle. Evaluation throughout the Project Management Cycle. Quality support and assessment system

Initiate your own project. The Purpose of Initiating. The Process of Initiating. Implications for EC support to projects. Identifying the Opportunities. PCM and managing ‘Calls for Proposals’. Ranking and Roughing out Opportunities and Benefits. The Project Initiation Document. Identifying problems. Setting SMART objectives. Estimating timelines. Essential Steps for a Successful initiation.

PCM principles and the Logical Framework Approach. What is LFA? Why to use it? What does the jargon mean? What is needed to produce a Logical Framework? The three stages of success in developing LFA. Initiate a project by using LFA. Self-assessment. Tips.

Project appraisal. Funding opportunities for projects. Principles of Programme assistance EC Programmes eligible for Albania. Types of assessment procedures. Outline on the evaluation and selection process. Aim and steps. Actors involved, roles and responsibilities. General principles applying to the evaluation and selection process. Main steps for assessment. Preparatory activities. administrative check. Verification of eligibility Evaluation of the full Application Forms. Evaluation Grids. How to score and write comments and recommendations. Outcomes of the evaluation of full Application Forms

Mid-term and ex-post evaluation of projects. Evaluation vs. Monitoring Vs Review. Project evaluation features. Purpose of project evaluations. Project evaluation by project partners (internal evaluation). Outsourcing the evaluation. Evaluation criteria. Project evaluation process. Who Does What. Evaluation process – phases, functions, steps. How evaluations are scheduled. Focus on results – results based management. Preparing Evaluation workplans. Evaluation design and methodology. Setting sources of information and methods for collecting relevant information. Setting accountabilities and reporting. Workplan checklist. Key success factors. Participatory evaluations. Tips and Case Studies



Project Application and Assessment in European Territorial Cooperation, ISBN 978-80-

971865-6-2, INTERACT Programme, August 2011

EC - Europe-aid Co-operation Office, Joint Evaluation Unit - Guidelines for Project and Programme Evaluation, volume 3, 2006 ISBN: 92-79-00681-9 https://www.oecd.org/dac/evaluation/dcdndep/47469201.pdf

World Bank (2000). The Log frame Handbook, World Bank: http://wbln1023/OCS/Quality.nsf/Main/MELFHandBook/$File/LFhandbook.pdf

https://www.oecd.org/dac/evaluation/dcdndep/47469201.pdf

http://wbln1023/OCS/Quality.nsf/Main/MELFHandBook/$File/LFhandbook.pdf












Course Program: Module VII: Standards in Evaluation

(developed in the study program: the MP-Evaluation) Lecturer: TO BE DECIDED Course: Module VII: Standards in Evaluation - key pillars for a quality development evaluation Semester: I Hours load (3 lectures / seminars 1) Number of credits: 5 Objectives:

To increase understanding of the content of Principles for the Evaluation of Development Assistance

To facilitate benchmarking of evaluations across countries and to enable development of occupational competences according to standards

To develop skills to apply the evaluation standards sensibly and adapted to local and national contexts and objectives of each evaluation of a project, action, policy or program.


Introduction to the course. Standards – basics. What are the standards – definition and meaning. Why are they important? The power and authority of standards: International Standards, Regional/European Standards and National Standards. Benefits of Reference Standards. Benchmarking. Standards Bodies. Standards in Evaluation.

International evaluation principles. Objectivity. Independence of evaluators, Transparency and focus. Participation of all parties in the evaluation process. Reliability. Completeness and clarity of reports. Fairness and protection of interests of involved parties. Utility.

Occupational standards in evaluation. International vs. National occupational standards. Occupations concerned with evaluation and relevant standards. Occupational standards in evaluation and Evaluators’ professional development framework. Example of best international practice in using occupational standards in evaluation - Country study case – Romania. Evaluators’ self-assessment according to occupational standards.

International evaluation quality standards (DAC evaluation standards). Rationale, purpose and objectives of an evaluation. Evaluation scope. Context. Evaluation methodology. Information sources. Independence. Evaluation ethics. Quality assurance. Relevance of the evaluation results. Completeness.

Programme Evaluation standards (AEA - JCSEE - standards for the evaluation of educational programs). Utility Standards. Feasibility Standards. Propriety Standards. Accuracy Standards. Evaluation Accountability Standards.



Yarbrough, D. B., Shulha, L. M., Hopson, R. K., and Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.



Table - List of predicting professors for teaching











Course Program: Module VIII: Competencies in Evaluation (Developed in the study program: the MP-Evaluation) Lecturer: TO BE DECIDED Course: Competencies in Evaluation

Semester: I Hours load: TO BE DECIDED Number of credits: TO BE DECIDED Objectives:

Gain insight on five domains of competencies of evaluator for evaluation work in Albania and in the International Development Context.

Explore and learn the divers approaches for competencies and professionalization of evaluation

The program content: Domains are high level categories under which the competencies for evaluators are organized. Reflective Practice These competencies are focus on the fundamental norms and values underlying evaluation practice and awareness of one’s evaluation expertise and needs for growth. Technical Practice These competencies are focus on the specialized aspects of evaluation, such as design, data collection, analysis, interpretation and reporting. Situational Practice These competencies are focus on the application of evaluative thinking in analyzing and attending to the unique interests, issues, and contextual circumstances in which evaluation skills are being applied.

Management Practice These competencies are focus on the process of managing a project/evaluation, such as budgeting, coordinating resources and supervising. Interpersonal Practice These competencies are focus on people skills, such as communication, negotiation, conflict resolution, collaboration, and diversity. Other approaches: Other approaches exist in the field of evaluation, and we will explore with real examples, the main practices. Literature: Literature: Lecture prepared by the teacher Reference literature: Reading materials include textbook chapters and current literature. Assignments will be posted on the course website



Laurie Stevahn (2015) Evaluation Competencies and Professionalization Electronic version of this publication is available at http://www.samea.org.za/index.php?module=Pagesetter&type=file&func=get&tid=17&fid=file1&pid=35

Canadian Evaluation Society (2014), Electronic version of this publication is available at https://evaluationcanada.ca/txt/2_competencies_cdn_evaluation_practice.pdf

American Evaluation Society (2014) American Evaluation Association Guiding Principles For Evaluators Electronic version of this publication is available at http://www.eval.org/p/cm/ld/fid=51



http://www.samea.org.za/index.php?module=Pagesetter&type=file&func=get&tid=17&fid=file1&pid=35

http://www.samea.org.za/index.php?module=Pagesetter&type=file&func=get&tid=17&fid=file1&pid=35

https://evaluationcanada.ca/txt/2_competencies_cdn_evaluation_practice.pdf

http://www.eval.org/p/cm/ld/fid=51

module i: evaluation-basics, theory of change and logic of ......module i: evaluation-basics, theory...

Documents