the evaluation checklist

The Evaluation Checklist

Wes Martz, Ph.D.

2

A mnemonic device that consists of a list of activities, items, and

criteria used to perform a certain task.

3

Guidance for the collection of relevant evidence used to

determine the merit, worth, or significance of an evaluand.

4

Existing Evaluation Checklists

• Evaluation management • Evaluation models • Evaluation values and criteria • Metaevaluation • Evaluation capacity building • Checklist creation

http://www.wmich.edu/evalctr/checklists

5

Checklist Organizer

•Party planner lists

•Grocery lists

•Travel lists

Taxonomic

•Pre-flight checklist

•Product assembly instructions

•Medical procedures

Procedural

•Personnel selection qualities

•Rating sheets in gymnastics competition

•Sections of evaluation checklists

Evaluative

6

Good Checklists

Practical

Parsimonious

Precise

7

Criteria to Evaluate Checklists

• Applicability to full range of intended uses• Clarity• Comprehensiveness• Concreteness• Ease of use• Fairness• Parsimony• Pertinence to the content area

Stufflebeam, D. L. (2000) Guidelines for developing evaluation checklists: The checklist development checklist. Available at http://www.wmich.edu/evalctr/checklists/guidelines_cdc.pdf

8

Checklist Advantages

• Consolidate vast knowledge in a parsimonious manner• Improve task performance• Reduce influence of halo and Rorschach effects• Reduce resource use• Improve memory recall• Set out minimum necessary steps in a process

Scriven, M. (2005). Checklists. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 53-59). Thousand Oaks, CA: Sage.

9

Checklist Disadvantages

• Evaluation myopia• Inappropriate use• Fatigue resulting from overuse• Unnecessary complexity decreases reliability• Burdensome process delays completing evaluation

Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009). doi: 10.1016/j.evalprogplan.2009.10.005

10

Organizational Effectiveness Evaluation Checklist (OEC)

Developing and Validating an Evaluation Checklist: A Case Example

11

OEC Overview

• Organizational evaluation process framework• Iterative, explicit, weakly sequential• Six steps, 29 checkpoints• Criteria of merit checklist

– 12 universal criteria of merit– 84 suggested measures

12

OEC Overview

Step 1

Establish the boundaries of the evaluation

Step 2

Conduct a performance needs assessment

Step 3

Define the criteria to be used for the evaluation

Step 4

Plan and implement evaluation

Step 5

Synthesize performance data with values

Step 6

Communicate and report evaluation

activities

13

Criteria to Evaluate the OEC

• Applicability to full range of intended uses• Clarity• Comprehensiveness• Concreteness• Ease of use• Fairness• Parsimony• Pertinence to the content area

14

OEC Validation Process

• Phase 1: Expert panel review– Critical feedback survey– Written comments made on checklist

• Phase 2: Field test– Single-case study– Semi-structured interview

15

Expert Panel Overview

• Study participants– Subject matter experts (organizational and evaluation theorists)– Targeted users (professional evaluators, organizational

consultants, managers)

• Review OEC for providing critical feedback• Identify strengths and weaknesses• Complete the critical feedback survey• Write comments directly on the checklist

16

Expert Panel Data Analyses

• Critical feedback survey– Descriptive statistics– Parametric and nonparametric analysis of variance

• Written comments on checklist– Hermeneutic interpretation– Thematic analysis to cluster and impose meaning– Triangulation across participants to corroborate or falsify the

imposed categories

17

Key Findings

• Pertinent to content area• Clear• Fair• Sound theory

• Parsimony and ease of use were identified as areas to address

Content relevance Representativeness Substantive validity

18

Field Test Overview

• Conducted evaluation using revised OEC• Strictly followed the OEC to ensure fidelity• Post-eval semi-structured client interview• A formative metaevaluation to detect and

correct deficiencies in the process

19

Observations from Field Test

• Structured format minimized “scope-creep”• Identified several areas to clarify in OEC• Reinforced need for multiple measures,

transparency in standards• Minimal disruption to the organization

20

Validity Study Assessment

• Strengths– Relatively quick validation process– Based on relevant evaluative criteria– Features a real-world application

• Weaknesses – Single-case field study– Selection of the case study– Selection of the expert panel members

Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009). doi: 10.1016/j.evalprogplan.2009.10.005

21

Lessons Learned

• Checklist development should address unique attributes of the evaluand

• Sampling frame is critical• Checklist validation should be grounded in theory,

practice, and use• Mixed method approach provides increased confidence

in validation conclusions• All checklists are a “work-in-process”

22

Additional Checklist Resources

• Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009).

• Scriven, M. (2007). The logic and methodology of checklists. Available at http://www.wmich.edu/evalctr/checklists/papers/logic&methodology_dec07.pdf.

• Stufflebeam, D. L. (2000) Guidelines for developing evaluation checklists: The checklist development checklist. Available at http://www.wmich.edu/evalctr/checklists/guidelines_cdc.pdf.

• Stufflebeam, D. L. (2001). Evaluation checklists: Practical tools for guiding and judging evaluations. American Journal of Evaluation, 22, 71–79.

http://www.wmich.edu/evalctr/checklists

The Evaluation Checklist

martz@EvaluativeOrganization.com

@wesmartz