Social Sciences and HumanitiesResearch Council of Canada

Conseil de recherches ensciences humaines du Canada

Evaluation Follow-up:Monitoring management responses and action plans and capturing evaluation evidence

Mom Yem, Social Sciences and Humanities Research CouncilShannon Clark Larkin, Social Sciences and Humanities Research Council

Canadian Evaluation Society Conference May 2 – May 5, 2010

2

Presentation Outline

Context

Developing the Management Response Action Plan (MRAP) database

Overview of Tool & Process

Lessons Learned

Forward Looking

3

Context

Government prioritiesAccountability for resultsEvidence-based decision-making

SSHRC’s corporate prioritiesImprove reporting on the results and impacts of SSHRC funded research and trainingIntegrate all performance, evaluation and audit activities into program delivery and decision-making

4

Context

SSHRC’s evaluation governance

“SSHRC’s Performance and Evaluation Committee (PEC) ensures the integration of evidence-based decision-making into SSHRC’s management structure and practices through guidance and oversight of the organization’s performance and evaluation functions.”

PEC is responsible for reviewing management responses to evaluations and related action plans, and tracking their implementation

Head of Evaluation (Director, Corporate Performance and Evaluation) is responsible for providing PEC with the information required to effectively carry out this mandate

5

Developing the MRAP database

6

Objectives

Track management responses and action plans in response to evaluation recommendations

Monitor the status of implementation of actions

Provide evidence of evaluation use and impact on decision-making

7

Key Considerations

Management buy-in

Explore existing tools/IT supportSimple tracking database; or Web-based tracking database

Learn from other organizations Environment Canada (adapted internal Audit tool)National Research Council (word document template)

8

Key Considerations

Data Requirements

Identify data requirements and develop a blueprint for the database structure

Map the requirements on the basis of a relational database and develop a data dictionary

9

Mapping of data requirements

10

Key Considerations

Reporting Requirements

Design and customize sample reports e.g., number of action items implemented, number of high priority action items pending, etc.

Reporting accurately relies on completeness of information and quality of data

11

Overall Considerations

Development phaseProper planning ensures the efficiency of the database Ensure flexibility in development to accommodate changes in table structureDevelop and test the database

Implementation phaseFlat database provided to programs requesting for status updates on the implementation of actionsPrograms completed the spreadsheets, with guidance and support

12

Overview of Tool & Process

13

Management Response Matrix - Basis of Development

14

Management Response Matrix - ExampleSummary Table of Management Response to the Joint Initiative Evaluation

Summary Table of Management Response to the Joint Initiative Evaluation – modified from its original version for the purpose of providing an example, implementing the suggested management response guidelines, particularly for the Responsibility and Timeline fields.

15

Guidance on the MRAP

Process for management responses and actions resulting from evaluation recommendations

Guide on “Developing, approving and tracking management responses to evaluation recommendations”

Process for following-up on the status of implementation of actions

Guide on evaluation follow-up process

16

Management Response Matrix (Follow-up Criteria)

Implementation TrackingStatus: Identify the status of implementation of action itemsComment: Provide comments and explanations relating to the implementation status (e.g. deferred/cancelled action items)Time-frame: Annual basis

17

Management Response Matrix - Implementation Tracking

18

Lessons Learned

19

Lessons Learned - Completing the response matrix

Management buy-in and support Draft guidance document circulated to programs/directors and senior staff for feedback Draft guidance document endorsed by PEC, 2008

Simplified guidance documents for better communicationHigh-level summary (2pager)Detailed guidelines and proceduresCommon and standardized templates

Encourage proper completion of the Management Response Matrix

Potential non-response and non-compliancemissing information for ‘Responsibility’ and ‘Priority/Timeline’emphasize actionable responses to recommendationsinconsistencies in responses and structure

Foundation for building a database

20

Lessons Learned - Status Update

1st round of updates on the implementation statusEstablish clearly defined roles and responsibilities Availability of support to programs

e.g., demo and training of the MRAP database

Limited resources and time constraintsReasons for requesting status updates

What is the purpose? Why are we doing this?How will it be used?

Reports submitted to PEC and TBS (e.g. MAF Assessment)

21

Forward Looking

Refine the MRAP guidelines and procedures for completion and follow-up

Continue to work with programs to successfully achieve full implementation of the MRAP processes over the long-term

Develop an integrated database toolModule 1: MRAP (developed/implemented)Module 2: Evaluation Evidence (under-development)

22

Forward Looking

Considerations for developing andtesting Module 2:

Evaluation evidence matrixcreated according to TBS’core evaluation issues (2009)Not all core issues are addressed in an evaluationLink to the MRAP fields: program name, recommendation, etc.Adapted for summativeevaluation more thanformative evaluation

23

Thank you.

Questions? Comments?

Contact Information:Mom Yem: mom.yem@sshrc-crsh.gc.caShannon Clark Larkin: shannon.clark-larkin@sshrc-crsh.gc.ca