evaluation in the government of canada robert lahey 23 november 2004 part one: an overview of the...
TRANSCRIPT
![Page 1: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/1.jpg)
Evaluation in the Government of Canada
Robert Lahey23 November 2004
PART ONE: An Overview of the Evaluation System in Canada
![Page 2: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/2.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 2
1. Background on the Canadian System2. Historical Perspective on how Evaluation
has evolved3. Government Accountability Structure4. Global View on how Evaluation Supports
Government’s Vision
OutlinePART ONE: An Overview of the Evaluation System in Canada
![Page 3: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/3.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 3
Population 31 million
Sovereign parliamentary democracy
Federation of 10 provinces and 3 territories
A federal Parliament and 13 provincial / territorial legislatures
Areas of federal responsibility include: national defence, criminal law, postal service, census, copyrights, trade, external relations, finance, transportation, citizenship, Indian affairs, fisheries and agriculture
Provincial / territorial governments: authority for education, hospitals, property and local government
Federal Government
Over 100 departments and agencies
Each has a Minister who reports to Parliament
Budgets approved annually
Formal Evaluation Policy in Canada for more than 30 years
Background on Canadian System of Government
![Page 4: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/4.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 4
Growing demand for information to support increasingly complex and costly programs
New management systems created for financial administration and planning, programming and budgeting
Evaluation as a practice not yet formalized
Historical Perspective: The 1960s
![Page 5: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/5.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 5
Building of key infrastructure elements in departments
Treasury Board Evaluation Policy (1977) was the first formalized evaluation policy in Canada
Evaluation Policy centre created within the new Office of the Comptroller General (1978)
• Considerable focus on building evaluation capacity in federal departments
Historical Perspective: The 1970s
![Page 6: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/6.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 6
High expectations for evaluations not always met Government-wide review led by Parliament
Historical Perspective: The 1980s
![Page 7: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/7.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 7
Evaluation capacity affected by general government downsizing
AG report (1993) indicated renewal of evaluation capacity needed
New Review Policy (1994) linked evaluation closer to internal audit
New emphasis on performance measurement / monitoring: Departmental Performance Reports to Parliament
Historical Perspective: The 1990s
![Page 8: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/8.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 8
‘Capacity gap’ for Evaluation (and Internal Audit) identified – need to reinvest in evaluation capacity within departments – both $ and skilled evaluators
New Evaluation Policy (2001): Commitment to link Evaluation to broader accountability and reporting requirements (strategic) as well as management decision-making (focused)
Emphasis on Results based Management Measuring Results Results based Management and Accountability Framework (RMAF)
Re-establishment of Policy Centre: Oversight of Evaluation practice in departments capacity building role; quality control role; a champion
Move in direction of ‘participatory’ evaluation Objectivity, not independence, emphasized Greater link to management and performance measurement / monitoring
Historical Perspective: The 2000s
![Page 9: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/9.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 9
Need for formal evaluation increases as resources become scarcer and the identification of priorities becomes more important
Necessary, but not sufficient, conditions for an effective evaluation system are: government investment and support; formalized policies and standards; and, leadership for capacity building
Evaluation quality depends on an approach that balances: timeliness, usefulness, methodological purity, client requirements and cost
Critical mass in capacity is required to ensure evaluation remains credible, relevant and strategic
Performance measurement / monitoring will typically not address ongoing issues of ‘cause and effect’; for that, ad-hoc evaluation is generally required
The priority given Evaluation is greatly enhanced if there is a perspective on modern management shared by the political level and the Public Service and evaluation is seen as integral to that vision
Historical Perspective: Lessons Learned
![Page 10: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/10.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 10
Canadian Government Accountability Structure
Parliament of CanadaHouse of Commons
Prime MinisterPrime Minister
Cabinet (Privy Council)Cabinet (Privy Council)
Parliamentary Committees Parliamentary Committees
TB Minister
Treasury Board Secretariat (TBS)(Secretary appointed)
Government’s management board (financial, management and human resources) Comptrollership function Centre of Excellence for Evaluation Policies and standards Capacity building Links evaluation and performance measurement
Public Servants
All Other Ministers
Federal Depts. /Agencies(DMs appointed)
Deputy Ministers (DMs) accountable for the application of Evaluation Policy
within their departments Heads of Evaluation implement policy as per TBS standards and guidelines Internal accountability and reporting to DMs External accountability and reporting to TBS and Parliament
Public Servants
Auditor General (AG)(appointed)
Conducts independent audits of government operations Produces periodic oversight reports on the conduct of evaluation Promotes accountability and best practices Reports directly to Parliament
![Page 11: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/11.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 11
Global Views on Public Sector Management in Canada
Vision: “Results for Canadians” – Improve government services and respond
to demands for better value and transparency
Tools and Initiatives: Applied through a wide-reaching series of initiatives Commitment to link evaluation and performance measurement /
monitoring to broader accountability and reporting requirements
Incentives and Use: Perspective on modern management shared by the political level and
the Public Service
![Page 12: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/12.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 12
Generate and use reliable information on results for the purposes of: Sound management, resource allocation and
decision making Accountability to Parliament and Canadians Continuous process of cultural change with no
static end state
Global Views: Goals to Achieve
![Page 13: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/13.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 13
4 Government Commitments
Roles & Responsibilities
Agenda for Change
ResultsResultsResponsible Responsible
SpendingSpendingValuesValues
Citizen Citizen FocusFocus
The Results for Canadians Management Framework
![Page 14: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/14.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 14
Learn & Adjust
StrategicAnalysis
Plan forResults
ImplementMonitor, Measure, Evaluate
Report onResults
Analysis of the current environment, past performance, emerging priorities and significant risks to achievement of desired results
Developing Strategic and Operational Plans – identify and plan to achieve key results and mitigate risk
Action taken to achieve results
Ongoing performance measurement and periodic evaluation to determine progress and allow for corrective action
Providing integrated Financial and non-financial information on: results and accountability for internal and external use
Evaluation: A Critical Link to Results-Based Management
![Page 15: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/15.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 15
Gain broad support for the “Vision”
Create the necessary ‘enablers’ (policies, guidelines, standards)
Invest in capacity building across the system Human capital
Tools necessary to support implementation
Provide the necessary incentives for implementation
Monitor and evaluate implementation
Learn and adjust, as necessary
Approach to Implementing the RFC Vision
![Page 16: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/16.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 16
Tools and Initiatives to Support the Vision
Modern Comptrollership
Broad direction for management reforms
Strengthened Evaluation and Internal Audit functions
Investment in capacity building
Functional specialists in departments (Evaluators; Auditors)
Learning for managers
Results-based Management and Accountability Frameworks (RMAFs)
Improved Performance Reporting to Parliament
Departmental Performance Reports
Canada’s Performance
Monitoring implementation
Modern Comptrollership “Capacity Check”
Managing for Results Self-Assessment Tool
Evaluation and Active Monitoring of Policy implementation
New government-wide initiatives
Expenditure and Management Review
Management Accountability Framework
![Page 17: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/17.jpg)
Evaluation in the Government of Canada
Robert Lahey23 November 2004
PART TWO: A Detailed Examination of the Process and Conduct of
Evaluation in Federal Departments
![Page 18: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/18.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 18
1. Module 1: Purpose of Evaluation2. Module 2: Developing the Capacity to
‘Do’ Evaluation3. Module 3: Developing Capacity to ‘Use’
Evaluation4. Module 4: Evaluation vs. Performance
Measurement/Monitoring5. Module 5: Results based Management
and Accountability Frameworks
OutlinePART TWO: A Detailed Examination of the Process and Conduct of
Evaluation in Federal Departments
![Page 19: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/19.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 19
Outline of Module Definitions and Terminology Key Uses and Users of Evaluation A Logic Model for the Use of Evaluation in a
Federal Department• Performance Measures for the Evaluation
Function
MODULE ONE: Purpose of Evaluation.. Why is it important? And, to whom?
![Page 20: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/20.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 20
Evaluation Summative Evaluation Formative Evaluation
Performance Measurement
Performance Monitoring
Results
Results Based management
DEFINITIONS AND TERMINOLOGY
![Page 21: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/21.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 21
How and where do results of Evaluation get used?
Within federal departments—several users
Deputy Ministers Assistant Deputy Ministers (ADMs) Program Managers Input to Corporate functions (policy, planning)
Treasury Bard of Canada Secretariat (TBS)
Funding renewal for individual programs Expenditure and Management Reviews Management Accountability Frameworks (MAF)
Elected Officials
Departmental Performance Reports Canada’s Performance Parliamentary Committees
General Public
Access to Evaluation Reports (ATIP) Web-based increases transparency
KEY USES AND USERS OF EVALUATION
![Page 22: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/22.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 22
Defined four broad ways Evaluation supports senior management
Four activity areas where skill set of Evaluators and Evaluation information is used
Traditional role (conducting program and policy evaluation) represents major use of Evaluation resources (some 75%)
A LOGIC MODEL FOR THE USE OF EVALUATION
IN A FEDERAL DEPARTMENT
![Page 23: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/23.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 23
Informed TBS decision-making
on program funding and
renewal
Program & PolicyEvaluation
LOGIC MODEL for HRSDC EVALUATION FUNCTION
ACTIVITIES
OUTPUTS
IMMEDIATE OUTCOMES
INTERMEDIATE OUTCOMES
ULTIMATE OUTCOMES
PerformanceMeasurement
Planning & Priorities
Evaluation Reportson program/policy• relevance• success • alternative
• Briefing material• Special studies• Corporate
intelligence
• RMAF advice and sign-off
• Tools & advice on performance measurement
• Evaluation plans and priorities linked to corporate priorities
• Objective and timely evidence-based information
• Lessons learned• Action plans
• Implementation of improved performance measurement system
• OperationalizedResults-Based Measurement (RBM)
• Improved measurement and reporting of results
• Better data collection
• Integrated evaluation and corporate plans
• Information for corporate priority-setting
Better design and resourcing of programs and
policies
Improved program
management and accountability
Results-based departmental
decision-making
RBM culture in HRSD
Parliamentary reporting
• Strengthened governance & stewardship of public funds• Better accountability to Parliament
• Better results for Canadians• Effective use of resources
Support & Advice toCorporate InitiativesACTIVITIES
OUTPUTS
IMMEDIATEOUTCOMES
INTERMEDIATEOUTCOMES
ULTIMATEOUTCOMES
A LOGIC MODEL FOR THE USE OF EVALUATION
IN A FEDERAL DEPARTMENT
![Page 24: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/24.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 24
How well are we doing in terms of expectations?
Derive measures from Logic Model
Caution how information is used, particularly during capacity building period
PERFORMANCE MEASURES FOR THE EVALATION FUNCTION
![Page 25: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/25.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 25
Outline of Module Policies and Standards Champion for the Function Investment in Evaluation Capacity Recruitment, training and development of
Evaluators Vehicles to Train and Develop Educating the ‘Users’ of Evaluation Organizing an Evaluation Unit in a Federal
Department
MODULE TWO: Developing the Capacity ‘To Do’ Evaluation. A Key Role for the Central Agency
![Page 26: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/26.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 26
Identifies expectations, roles and responsibilities
Federal Departments given flexibility in their application
Deputy Minister (DM) accountable
Treasury Board of Canada Secretariat (TBS) oversight of individual departments…through the Centre of Excellence for Evaluation (CEE)
Periodic review by the Auditor General of Canada (AG) of how/how well Evaluation Policy is being implemented …provides system-wide oversight
POLICIES AND STANDARDS
![Page 27: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/27.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 27
Important to have a champion/advocate for the function
Several roles:
Setting and adjusting policy and standards for the function Developing and executing human resource (HR) strategies to train and develop Evaluators Advisory role (practical advice) to individual departments Oversight and monitoring quality of evaluation practices and products Corporate resource on Evaluation and accountability
for government officials
Establish a network for Evaluators (continuous learning)
CHAMPION FOR THE FUNCTION: THE CENTRE OF EXCELLENCE FOR EVALUATION (CEE)
![Page 28: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/28.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 28
Need for both $ and trained personnel
Policy not prescriptive on size of departmental Evaluation units…a decision of the DM
Accountabilities of the DM (several) serve to leverage resources for the function
In recent years, focus of TBS/CEE on building a ‘sustainable’ Evaluation function Infrastructure Product Use
With flexibility in resourcing, important that DMs understand importance and use of Evaluation
Canadian experience....periods of general downsizing (fewer resources for the function) and capacity building (new investment)
Danger of fluctuating resource levels….losing skilled/experienced Evaluators. A long-term investment to replace.
INVESTMENT IN BUILDING EVALUATION CAPACITY
![Page 29: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/29.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 29
Recruitment University graduates generally not ‘Evaluation-specific’ Entry into the function from many disciplines…social sciences
(Economics, Psychology, Sociology); Education; Mathematics/Statistics; Public Administration Both a strength and a weakness
Training Need to distinguish between training for ‘entry level’, ‘mid level’
and ‘senior’ Evaluators Link to the competencies required for Evaluators (Competency Profile) Various vehicles for formal training
Development Importance of ‘on the job’ training Continuous learning…link to changing/evolving role of Evaluation Communication skills particularly important for senior level Evaluators (Departmental Heads of Evaluation) Several vehicles and occasions for development No formal ‘certification’ to work as an ‘Evaluator’ in federal government
RECRUITMENT, TRAINING AND DEVELOPMENT OF EVALUATORS
![Page 30: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/30.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 30
Formal curriculum developed by CEE for ‘entry level’ Evaluators (Evaluator Internship Program)…not mandatory
Training needs identified for mid-level Evaluator
CEE partnering with others for training delivery…e.g. National Statistical agency (Statistics Canada); Canadian Centre for Management Development; Professional Associations (Canadian Evaluation Society (CES); department-led training; private sector trainers/course deliverers
Network of Evaluators encourages informal ‘communities of practice’ and sharing of ‘lessons learned’ across like evaluations
Link to other professional groups (e.g. Internal Auditors; Comprehensive Audit)
VEHICLES TO TRAIN AND DEVELOP EVALUATORS
![Page 31: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/31.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 31
‘Users’ of evaluation (in departments, central agencies, politicians)
Need to be educated on how, where, and why evaluation can/should be used/resourced
Importance of aligning ‘user’ expectations with reality of what can/will be delivered by Evaluators
Useful to use a variety of vehicles: briefing notes; presentation decks; training modules/workshops aimed at senior managers
Avoid technical jargon and deliver at appropriate level of understanding
Education an ongoing activity; not a ‘one-time’ event
Basic distinctions need to be made…e.g. difference between Internal Audit and Evaluation, an ongoing area of confusion for managers: Internal Audit: Are we doing things right? (efficiency issues) Evaluation: Are we doing the right things? (effectiveness and
performance issues)
EDUCATING THE ‘USERS’ OF EVALUATION
![Page 32: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/32.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 32
In Canada, federal departments determine their own arrangements
Much variation in terms of number of full-time staff hired vs use of consultants
Generally, most departments use some consultants…e.g. for surveys, specialized tasks, etc.
Important for Unit to have minimum core of full-time Evaluators
Wide range in size of Units dedicated to Evaluation…e.g. 1 person (small agency) to 25-30 f/t staff (largest department)
Typical Evaluation Unit: 10 or fewer f/t staff, plus resources to hire consultants for Evaluation
ORGANIZING AN EVALUATION UNIT
![Page 33: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/33.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 33
Outline of Module Central Agency-driven demand for Evaluation Steering the Evaluation function in departments Credibility building for Evaluators Developing a Comprehensive Evaluation Plan Advisory Committees Educating ‘Users’ of Evaluation Participatory Evaluation and Accountability for
Follow-up Evaluations in ‘full public view’
MODULE THREE: Developing Capacity ‘To Use’ Evaluation: Overcoming Resistance and Making Effective Use of Evaluation
![Page 34: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/34.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 34
CENTRAL AGENCY-DRIVEN DEMAND FOR EVALUATION
Deputy Minister in department faces variety of accountabilities requiring performance/results information
Required for good management practices and internal and external reporting
Program-specific funding renewal decisions Horizontal and vertical expenditure reviews Accountability ‘report card’ of the DM (Management
Accountability Framework (MAF) Departmental Performance Report to Parliament Ability to respond to queries of Parliamentary
Committees; issues raised by the Auditor General
![Page 35: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/35.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 35
Steering the Evaluation Function in Departments
Evaluation and Audit Committee, chaired by Deputy Minister Requirement of TBS for every government department Senior level committee; formal Terms of Reference; meet
minimum four times a year Review and approve annual and multi-year plan for Evaluation Senior managers identify new/changing priorities Table completed Evaluation reports and management
response Forum to alert senior managers of areas of risk to department Vehicle to inform/educate senior managers (senior ‘users’ of
evaluation
![Page 36: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/36.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 36
Credibility Building for Evaluators
How Evaluators see themselves. How others see Evaluation.
Two critical factors for credibility: Understanding the needs/requirements of senior
decision makers Delivering quality product on time ‘Answering the right questions at the right time’ Critical factors for success: Head of Evaluation well linked to senior management Appropriate balance between methodological rigour
and delivery in a timely fashion Responsive to changing priorities
![Page 37: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/37.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 37
Developing a Comprehensive Evaluation Plan
Three key questions:
i. What are the guiding principles for evaluation planning?
ii. What is the relevant ‘Evaluation Universe’?iii. How best to determine priorities and schedule
evaluations over a multi-year period?
![Page 38: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/38.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 38
Advisory Committees: Building trust and avoiding surprises
Useful vehicle to assist conduct of an Evaluation study
Representation from broad set of stakeholders Forum to clarify study objectives; table early
findings; review conclusions and recommendations
Creates more ‘open’ evaluation process Avoid surprises at the end Helps ensure follow-through on study
recommendations
![Page 39: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/39.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 39
‘Users’ of evaluation (in departments, central agencies, politicians)
Need to be educated on how, where, and why evaluation can/should be used/resourced
Importance of aligning ‘user’ expectations with reality of what can/will be delivered by Evaluators
Avoid technical jargon and deliver at appropriate level
of understanding Education an ongoing activity; not a ‘one-time’ event
EDUCATING THE ‘USERS’ OF EVALUATION
![Page 40: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/40.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 40
Conduct of evaluation not ‘independent’ of program managers New Policy (2001) removed ‘independence’; stressed ‘objectivity’ and ‘transparency’ Attempt to bring evaluation closer to managers…not a threat, but a tool to assist good management practices Puts onus on Managers for ongoing performance
monitoring as well as accountability for implementing evaluation recommendations
Participatory Evaluation and Accountability for Follow-up
![Page 41: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/41.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 41
Broad demand for greater accountability in government
Evaluation reports accessible to wide audience (via ATIP)
Web posting increases accessibility to general public
Evaluation Policy stresses ‘transparency’
Requires a culture allowing for: full disclosure of information unbiased analysis and reporting openness in government
Evaluation ‘In Full Public View’
![Page 42: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/42.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 42
Outline of Module Historical Context Definitions and Distinctions Departmental Performance Framework Using Logic Models to Derive Performance
Measures Results Measurement Continuum Model Roles and Responsibilities for Delivery General Strategy for Implementing M&E Systems
MODULE FOUR: Evaluation versus Performance Measurement/Monitoring: A Critical Distinction
![Page 43: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/43.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 43
Mid-to-late 90’s onward: Increasing government emphasis on ‘performance measurement’,
‘monitoring’ and ‘results’ Drivers: Results based management; Greater
accountability to Parliament and to citizens Put onus on managers to measure and report on
performance/results Required clarification of terminology, roles and
responsibilities in the process
Historical Context
![Page 44: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/44.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 44
How is Evaluation different from Performance Monitoring? How does one link to the other? Are they both necessary tools to support good
management practices and accountability?
Definitions and Distinctions
![Page 45: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/45.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 45
Planning Reporting and Accountability Structure (PRAS) provides broad departmental framework
Policy being updated (Fall 2004) to Management Resources and Results Structure (MRRS)
Business line structure: architecture and basis for cascading set of performance measures through organization
Brings greater focus to managers on measuring ‘results’
Evaluation specialist has the skill set to facilitate this process
Departmental Performance Framework: Basis for Ongoing Performance Monitoring
![Page 46: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/46.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 46
Performance indicators derive from logic model
What needs to be measured and reported?
What are the key questions that need to be answered?
How will it be used? By whom? How often? ‘Monitoring’ versus ad hoc ‘Evaluation’ Performance measurement development/implementation
Technical cost considerations An iterative process
Using Logic Models to Derive Meaningful Performance Measurement
![Page 47: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/47.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 47
Results Measurement Continuum Model
![Page 48: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/48.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 48
Managers ultimately accountable for performance measurement and reporting
Evaluation specialists support to the development and implementation of the performance measurement system through: creating PM understanding---learning events, seminars,
information packages facilitating PM development applying needed rigour and discipline---‘challenge’ guidelines Evaluation study as one option for measuring
performance, where: ad hoc study more cost-effective ‘cause and effect’ (attribution) issue important IM/IT specialist---advice re systems and data capture issues
Roles and Responsibilities for Delivery
![Page 49: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/49.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 49
Challenges: political; organizational; technical; resourcing
Useful to do a ‘readiness assessment’: Is there an enabling environment: political will; commitment; a champion Technical capacity, including HR skills Necessary infrastructure to produce, collect, analyze and report relevant information Necessary infrastructure/institutions to ‘use’ evaluation and monitoring information Priority needs to be established from the top Resourcing: a corporate responsibility. Needs to be linked to accountability and modern management practices
General Strategy for Implementing Evaluation and Performance Monitoring Systems
![Page 50: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/50.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 50
Outline of Module Why have we introduced the RMAF? RMAF Components Using the RMAF to Greatest Gain Helpful Hints for RMAF
Development/Implementation
MODULE FIVE: Results Based Management and Accountability Framework (RMAF): An Important Vehicle
![Page 51: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/51.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 51
Tool designed to assist the move to results measurement/reporting Broader government commitment on results reporting (Results for Canadians) Formal requirement for certain programs Facilitates development to performance measurement and establishes priority and timeframe for evaluation
Why have we introduced the RMAF?
![Page 52: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/52.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 52
All RMAFs have five major sections: A: Profile B: Logic Model C: Ongoing Performance Measurement Strategy D: Evaluation Strategy E: Reporting Strategy Together, describes the rationale, resources and
performance logic of program or initiative Presents the measurement, evaluation and reporting strategies that will be implemented to track progress
RMAF Components
![Page 53: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/53.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 53
A collaborative effort of key program stakeholders to clearly articulate:
Program objectives in terms that allow for measuring/monitoring performance/success Linkages of program activities to program goals Brings technical (Evaluator) and non-technical (Manager) to
common agreement on meaningful measures of performance and a measurement strategy Up-front investment in RMAF pays off in Clarifying program role and resourcing needs Establishment of meaningful measurement systems More systematic and disciplined monitoring of program performance Improved management information, supporting RBM Improved accountability, both to internal and external stakeholders
(senior managers; elected officials; citizens)
Using the RMAF to Greatest Gain
![Page 54: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/54.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 54
Accountability with RMAF development; performance monitoring; reporting…lies with Manager Carry out development and implementation as
collaborative effort (Manger; Evaluation specialist; IM/IT specialist) Flexibility, particularly in early stages:
‘acceptable’ RMAF versus perfection
performance measurement as an ‘iterative’ process Select realistic performance indicators based on what will provide the best information, not the easiest to implement
Helpful Hints for Implementing RMAFs
![Page 55: Evaluation in the Government of Canada Robert Lahey 23 November 2004 PART ONE: An Overview of the Evaluation System in Canada](https://reader036.vdocuments.us/reader036/viewer/2022070306/551a02a255034619378b4993/html5/thumbnails/55.jpg)
REL Solutions Inc. Brasilia, November 23, 2004 55
Evaluation and performance measurement: Important link to effective pubic sector governance Broad application across:
individual programs sectors or business lines whole organization
Not an end in itself, a means to an end Evaluation capacity building to a self-sustaining state requires:
Commitment/political will Resources (both $ and HR) Time Energy
Final Thoughts