Planning How to ConductPlanning How to Conductthe Evaluationthe Evaluation

Dr. Suzan Ayers Dr. Suzan Ayers

Western Michigan UniversityWestern Michigan University(courtesy of Dr. Mary Schutten)(courtesy of Dr. Mary Schutten)

Relationship of focus and Relationship of focus and planningplanning

• The structure of evaluation consists of:The structure of evaluation consists of:– focusingfocusing the evaluation the evaluation– collecting, organizing, analyzing & collecting, organizing, analyzing &

reportingreporting info info– administeringadministering the evaluation (see CIPP the evaluation (see CIPP

notes)notes)

• Evaluation should be conducted flexiblyEvaluation should be conducted flexibly

• Evaluator needs clear understanding of Evaluator needs clear understanding of evaluation’s purpose and roleevaluation’s purpose and role

• FocusingFocusing the evaluation determines what the evaluation determines what information is needed information is needed – Origin and context of the proposed Origin and context of the proposed

evaluationevaluation– Identifying/selecting questions, criteria, Identifying/selecting questions, criteria,

standardsstandards

• Once the evaluation questions are Once the evaluation questions are known, the next step is to determine known, the next step is to determine what information is needed to answer what information is needed to answer each questioneach question

• PlanningPlanning the evaluation consists of the evaluation consists of collectingcollecting info, info, organizingorganizing info, info, analyzinganalyzing info, info, reportingreporting info, and info, and administeringadministering the the evaluationevaluation

SAMPLESAMPLE• Evaluation Question:Have the critical Evaluation Question:Have the critical

program activities occurred on time and program activities occurred on time and within budget?within budget?

• Things evaluator would need to know:Things evaluator would need to know:– Which activities were viewed as Which activities were viewed as criticalcritical??– Program time frames budgetProgram time frames budget– Program time frames budget by activityProgram time frames budget by activity– When each activity began/endedWhen each activity began/ended– Total cost of each critical activityTotal cost of each critical activity

Identifying Design andIdentifying Design andData Collection MethodsData Collection Methods

• Involve client and stakeholders in deciding Involve client and stakeholders in deciding necessary info to best answer each questionnecessary info to best answer each question

• Designs specify organization/structure for data Designs specify organization/structure for data collectioncollection

• Causal designsCausal designs: (quasi)experimental designs: (quasi)experimental designs– Descriptive case studies, multiple regression or other stats Descriptive case studies, multiple regression or other stats

methods to answer evaluation Qsmethods to answer evaluation Qs

• Descriptive designsDescriptive designs: describe: describe (case study)(case study), , analyze the analyze the program, show a trendprogram, show a trend (time series)(time series), , assess public assess public opinionsopinions (cross-sectional)(cross-sectional), , illustrate a processillustrate a process (thick description)(thick description)

– Commonly used in needs assessment and process Commonly used in needs assessment and process studiesstudies

• Evaluator and stakeholders examine Evaluator and stakeholders examine each question carefully to identify any each question carefully to identify any important research design issuesimportant research design issues

• Most evaluations are multiple research Most evaluations are multiple research designs or combinationsdesigns or combinations

• Important to discuss early to see if:Important to discuss early to see if:– groups are available, appropriateness of groups are available, appropriateness of

random assignment, time collecting data, random assignment, time collecting data, etc… etc…

– Is the design “do-able?”Is the design “do-able?”

Identifying AppropriateIdentifying AppropriateInformation Sources Information Sources

• Once needed information is agreed upon, Once needed information is agreed upon, the source(s) of that information must be the source(s) of that information must be specifiedspecified– ““SourceSource”: group of individuals or ”: group of individuals or

location of existing information that location of existing information that answers each Qanswers each Q

– Who will have information or access to Who will have information or access to it?it?

– Who will be able to collect those data?Who will be able to collect those data?

• Using existing data as information Using existing data as information sourcesource– Does necessary information already Does necessary information already

exist in a readily available form?exist in a readily available form?• Commonly used information sourcesCommonly used information sources

– Program recipients, deliverers, persons Program recipients, deliverers, persons who have knowledge of the program who have knowledge of the program recipients, public documents/databasesrecipients, public documents/databases

• Policies that restrict information sourcesPolicies that restrict information sources– Do policies exist concerning collecting Do policies exist concerning collecting

data from clients or existing files?data from clients or existing files?•Confidentiality, anonymity, privacy, IRB Confidentiality, anonymity, privacy, IRB

protocolsprotocols

• Client involvement in identifying sourcesClient involvement in identifying sources– Evaluator, by training and experience, often Evaluator, by training and experience, often

can identify key sources of informationcan identify key sources of information

– Client will be able to identify sources of Client will be able to identify sources of information that may be missed by the information that may be missed by the evaluatorevaluator

– Collaboration yields helpful answers and Collaboration yields helpful answers and enhances sense of shared ownershipenhances sense of shared ownership

Identifying Data Collection Identifying Data Collection Methods and InstrumentsMethods and Instruments

• Data collected Data collected directly from individualsdirectly from individuals identified as sources of informationidentified as sources of information– Self reportsSelf reports

• interviews, surveys, rating scales, focus interviews, surveys, rating scales, focus groups, logs/journalsgroups, logs/journals

– Personal Products:Personal Products:•tests tests (objective, essay)(objective, essay), performances , performances (simulations, role-(simulations, role-

play, competency testing)play, competency testing), work samples , work samples (portfolios)(portfolios)

• Data collected byData collected by independent observer independent observer– Narrative accountsNarrative accounts– Observation forms (rating scales, Observation forms (rating scales,

checklists)checklists)

• Data collected withData collected with technological device technological device– AudiotapeAudiotape– VideotapeVideotape– Time-lapse photosTime-lapse photos– OthersOthers

•BP cuffs, speed, graphic recordings of BP cuffs, speed, graphic recordings of performance skills, computer collation of performance skills, computer collation of participant responsesparticipant responses

• Data collection fromData collection from unobtrusive unobtrusive measuresmeasures

• Data collected from Data collected from existing informationexisting information – Public documentsPublic documents

•federal, state, local, databases etc..federal, state, local, databases etc..

– Review of organizational documentsReview of organizational documents•client files, notes of employees/directors, client files, notes of employees/directors,

audits, minutes, publicationsaudits, minutes, publications

– personal filespersonal files•Correspondence, e-mails, etc.Correspondence, e-mails, etc.

• After identifying for use some of the After identifying for use some of the methods on the next slides, it is important methods on the next slides, it is important to review adequacy of techniquesto review adequacy of techniques– Will the info collected provide a Will the info collected provide a

comprehensive picture?comprehensive picture?– Are the methods legal and ethical?Are the methods legal and ethical?– Is cost of data collection be worthwhile?Is cost of data collection be worthwhile?– Can they be collected w/o undo Can they be collected w/o undo

disruption?disruption?– Can data be collected w/in time Can data be collected w/in time

constraints?constraints?– Will the information be reliable and valid Will the information be reliable and valid

for the purposes of the evaluation?for the purposes of the evaluation?

Determining Appropriate Determining Appropriate Conditions for Collecting Conditions for Collecting

InformationInformation• 3 key issues around data collection:3 key issues around data collection:

– Will sampling be used?Will sampling be used?– How will data actually be collected?How will data actually be collected?– When will data be collected?When will data be collected?

• Specifying sampling procedures to be Specifying sampling procedures to be employedemployed– Sampling helps researcher draw inferences Sampling helps researcher draw inferences

about the population in the studyabout the population in the study– Sampling is a tool to use whenever resources Sampling is a tool to use whenever resources

and time are limited and time are limited – Sampling useful when it will not diminish the Sampling useful when it will not diminish the

confidence of resultsconfidence of results– Sample size must be appropriate; too small a Sample size must be appropriate; too small a

sample is of limited valuesample is of limited value

• Specifying how the information will be Specifying how the information will be collectedcollected– Who will collect data? Who will collect data? – For interviews, focus groups etc… how will For interviews, focus groups etc… how will

characteristics of the evaluator influence data characteristics of the evaluator influence data collection?collection?

– What training should be given to people What training should be given to people collecting the data?collecting the data?

– In what setting should data collection take In what setting should data collection take place?place?

– Confidentiality protected?Confidentiality protected?– Special equipment, materials needed?Special equipment, materials needed?– Evaluators need a bigger ‘bag of tools’ needed Evaluators need a bigger ‘bag of tools’ needed

than most researchers to examine a broad than most researchers to examine a broad array of phenomenaarray of phenomena

• Specifying when the information will be Specifying when the information will be collectedcollected– When will the information be needed?When will the information be needed?– When will the information be available?When will the information be available?– When can the information conveniently When can the information conveniently

be collected?be collected?

Determining Appropriate Determining Appropriate Methods to Organize, Analyze, Methods to Organize, Analyze,

Interpret InformationInterpret Information• Develop a system to code, organize, store, and Develop a system to code, organize, store, and retrieve dataretrieve data

• For each evaluation question, specify how For each evaluation question, specify how collected information will be analyzedcollected information will be analyzed– Identify statistical/summarizing techniquesIdentify statistical/summarizing techniques– Designate some means for conducting the analysisDesignate some means for conducting the analysis

• Interpreting results Interpreting results (Statistical reports do (Statistical reports do not not speak for the speak for the themselves)themselves)

– Share information with clients to gain perspective on Share information with clients to gain perspective on potential interpretations of the datapotential interpretations of the data

– Criteria/Standards guide interpretation of some QsCriteria/Standards guide interpretation of some Qs– Eval plan should allow recording of multiple or Eval plan should allow recording of multiple or

conflicting interpretationsconflicting interpretations– Interpretations should consider multiple perspectivesInterpretations should consider multiple perspectives

Determining Appropriate Ways Determining Appropriate Ways toto

Report Evaluation FindingsReport Evaluation Findings• Using a matrix appropriate way to plan Using a matrix appropriate way to plan

reporting reporting (Fig 13.1)(Fig 13.1)

– Audience, content, format, date, context of Audience, content, format, date, context of presentationpresentation

• Suggested Qs Suggested Qs (Brknerhoff, Brethower, Hluchyj, & Nowakowski, (Brknerhoff, Brethower, Hluchyj, & Nowakowski, 1983)1983)

– Are reporting audiences defined?Are reporting audiences defined?– Are report formats and content appropriate for Are report formats and content appropriate for

audience needs?audience needs?– Will the evaluation report balanced information?Will the evaluation report balanced information?– Will reports be timely and efficient?Will reports be timely and efficient?– Is the report plan responsive to the rights to info Is the report plan responsive to the rights to info

of the audiences?of the audiences?

Appropriate Ways to ReportAppropriate Ways to ReportEvaluation FindingsEvaluation Findings

Worksheet approach; summarize: 8 topics for Worksheet approach; summarize: 8 topics for each Qeach Q

1. Information required to answer the 1. Information required to answer the questionquestion

2. Design(s) to be used to collect information2. Design(s) to be used to collect information

3. Sources(s) of that information3. Sources(s) of that information

4. Method(s) for information collection4. Method(s) for information collection

5. Information-collecting arrangements5. Information-collecting arrangements-sampling procedure-sampling procedure-collection procedure-collection procedure-schedule for collection-schedule for collection

6. Analysis procedures6. Analysis procedures

7. Interpretation procedures (including 7. Interpretation procedures (including standards)standards)

8. Reporting procedures8. Reporting procedures-audiences-audiences-content-content-format-format-schedule-schedule-context-context

The Management PlanThe Management Plan• Final task in planning the evaluation Final task in planning the evaluation

study is describing how it will be carried study is describing how it will be carried outout

• Management plan essential to help Management plan essential to help oversee the projectoversee the project– Who will do it?Who will do it?– How much will it cost?How much will it cost?– Will it be within budget?Will it be within budget?

• Evaluation management is multifacetedEvaluation management is multifaceted– supervise staffsupervise staff– serve as liaison to evaluation clients, serve as liaison to evaluation clients,

participants, and stakeholdersparticipants, and stakeholders– identify and cope with political influencesidentify and cope with political influences– communicate, communicate, communicate, communicate,

communicatecommunicate• Evaluation, whether by one person or a team, Evaluation, whether by one person or a team,

cannot afford to be disorganized or haphazardcannot afford to be disorganized or haphazard• Management plan needed to structure and Management plan needed to structure and

control resourcescontrol resources• Good management plans specify for ea eval Q Good management plans specify for ea eval Q

(Fig 13.4)(Fig 13.4) : :– Tasks & timelines, personnel/resources for each Tasks & timelines, personnel/resources for each

task, costtask, cost

• Managing and estimating time for Managing and estimating time for conducting evaluationconducting evaluation– PERT PERT (program eval & review technique)(program eval & review technique) & Gantt charts are & Gantt charts are

commonly used to estimate time on taskscommonly used to estimate time on tasks– Gantt charts Gantt charts (Fig 13.6)(Fig 13.6): simple displays that : simple displays that

include chronologically scaled time frames include chronologically scaled time frames for each evaluation taskfor each evaluation task•Y Y (vertical)(vertical) axis: tasks axis: tasks•X X (horizontal)(horizontal) axis: time scale axis: time scale•Horizontal line drawn for each task show time Horizontal line drawn for each task show time

neededneeded– Help highlight interim deadlines or milestones Help highlight interim deadlines or milestones

that must be met to stay on the time in the that must be met to stay on the time in the studystudy

– Well-specified milestones essential monitoringWell-specified milestones essential monitoring tooltool

• Analyzing personnel needs and Analyzing personnel needs and assignmentsassignments– Quality of evaluation depends heavily on Quality of evaluation depends heavily on

those who carry it outthose who carry it out

– Are qualified individuals available to carry Are qualified individuals available to carry out the tasks?out the tasks?• ““Personnel role specifications” for all tasks Personnel role specifications” for all tasks

(Suarez, 1981)(Suarez, 1981)

• Specify who would manage the study, complete Specify who would manage the study, complete the eval design, select or develop instruments, the eval design, select or develop instruments, collect data, analyze data, write summary reports, collect data, analyze data, write summary reports, etc…etc…

• Estimating costs and developing Estimating costs and developing budgetsbudgets– Staff salary and benefitsStaff salary and benefits– ConsultantsConsultants– Travel and per diem ( for staff/consultants)Travel and per diem ( for staff/consultants)– Communications (postage, phone, etc.)Communications (postage, phone, etc.)– Printing and duplicationPrinting and duplication– Data processingData processing– Printed materialsPrinted materials– Supplies and equipmentSupplies and equipment– SubcontractsSubcontracts– Overhead (facilities, utilities)Overhead (facilities, utilities)

If initial estimate exceeds expectations, review If initial estimate exceeds expectations, review each line itemeach line item

• Agreements and contractsAgreements and contracts• Potential problems that arise during the Potential problems that arise during the

evaluation can be more easily resolved if evaluation can be more easily resolved if client and evaluator share a firm client and evaluator share a firm understandingunderstanding

• A well-documented agreement A well-documented agreement prior to prior to launching the evaluation study launching the evaluation study concerning important procedures is very concerning important procedures is very helpfulhelpful

• http://http://www.wmich.edu/evalctr/checklists/contractwww.wmich.edu/evalctr/checklists/contracts.pdfs.pdf