assessing impact evaluating staff development second edition
Post on 24-Feb-2016
43 Views
Preview:
DESCRIPTION
TRANSCRIPT
ASSESSINGIMPACT
Evaluating Staff DevelopmentSecond EditionJoellen Killion
Corwin Press, 2008A joint publication with the National Staff Development Council
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 2
As you get settled . . .
Write any questions you have about evaluating staff development on index cards.
One question per card, please.
Intended outcome: think like evaluators and build staff development evaluation into the planning process for the TSSRRP grant.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 3
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 4
Describe the eight-step evaluation process and the components of each step.
Define evaluation in relationship to staff development. Understand how a staff development program’s theory
of change and logic model influence the success of a staff development program and its evaluation.
Construct an evaluation framework for a staff development program.
Acquire strategies, tools, and resources to assist in evaluating staff development.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 5
Norms
Share experiences to enrich others.
Ask questions. Learn by doing. Set aside any preconceived
notions about evaluating staff development.
Apply to your own work.
THINK S – A – L – S – A !
A systematic, purposeful process of
studying, reviewing, and analyzing data
gathered from multiple sources in order to
make informed decisions about a
program.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 6
Evaluation is:
Page 8IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 7
Key ElementsSystematic How rigorous is the process?
Is it conducted in accordance with standards and guidelines?
Standards Does it have merit and/or worth?Does it meet predetermined standards of success?
Audience Who will use the evaluation?For whom is the evaluation being done?
Intended Uses How will the evaluation be used?What decisions will be made as a result of the evaluation?
Pages 8-9IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 8
Scientifically Based Research vs. Evaluation
Scientifically based research identifies interventions that have been “proven” to be successful under conditions that allow generalizations that these interventions will produce similar “positive” results in other situations.
Evaluation determines whether the selected interventions (based on scientific research or not) have merit (achieved its intended results), worth (perceived as valuable), and/or impact under these conditions.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 9
Purposes of Evaluation
Determine merit and/or
worth
Assess impact
Identify improvements
Provide accountability
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 10
Means goals vs. End goals
Provide training Increase student achievement
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 11
Evaluation to Assess Impact
Requires the program’s goal to be about increasing student achievement.
Requires a theory of change to explain how the program will produce intended results.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 12
Program vs. Event A program is a set of
purposeful, planned actions and the support system necessary to achieve the identified goals. Effective staff development programs are ongoing, coherent, and linked to student achievement. Page 11-13
IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 13
Staff Development vs.
Staff Development Program
Talk it over.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 14
Planning1. Assess
Evaluability
2. Formulate Evaluation Questions
3. Construct Evaluation Framework
Conducting4. Collect Data
5. Organize, Analyze, & Display Data
6. Interpret Data
Reporting7. Disseminate
and Use Findings
8. Evaluate the Evaluation
Evaluating Staff Development
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 15
Self-Assessment
+ understand
— not sure
? don’t understand
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 16
Types of Evaluation
Planning
Formative
Summative
Pages 13-17IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 17
Black Box Evaluations
Input Output
Actions Results
Glass Box Evaluations
?
Pages 26-28IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 18
ProfessionalDevelopment
Action
Student Achievement
Results
Black Box
?
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 19
Black Box
Student Achievement
ProfessionalDevelopment
CurriculumDevelopment
Nonacademicfactors
?
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 20
Student Achievement
Results
ProfessionalDevelopment
Actions
Glass Box
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 21
Student Achievement
ProfessionalDevelopment
Coaching/Follow-upInstructional ResourcesImplementation MonitoringStudent Assessment
Glass Box
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 22
Black Box vs. Glass Box
What is the difference?
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council23
Assumptions1. The staff development program is data-driven,
research-based, and well-defined (p. 21).
2. The school, district, or regional agency has the capacity, including fiscal and human capital, to implement both the program and evaluation with fidelity to their designs (p. 22).
3. Key stakeholders in the school, district, or agency intend to use the evaluation results to make decisions about the program (p. 23).
24
Step 1:Assess
Evaluability
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 25
The degree to which a proposed plan for staff development clearly articulates what the staff development program is and how it is expected to produce change.
Step 1: Assess EvaluabilityPlanning Phase
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 26
Read and Talk1. Turn to p. 33.2. Read last paragraph3. Turn to p. 344. Read first 2 paragraphs
Can you think of an example?
What was the issue?
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 27
Four Components of
Assessing Evaluability
Examine program goals, objectives, standards for success
Examine program’s theory of change
Examine program’s logic model
Adjust program based on assessment
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 28
Clarifying Your ProgramWhat are the program’s goals?
Goal is desired state. Improve physical well-being.
Objectives describe the changes expected. Lose 15 pounds in six months.
Activities explain how to achieve the goals and objectives. Walk daily. Eat healthful foods.
Pages 35-36IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 29
Goal, Objective, or Activity? Develop responsible citizens. Increase student proficiency in applying the
scientific process in a laboratory setting as measured by a performance task.
Provide opportunities for professional learning. Establish professional learning communities. Increase student performance 15% on state
assessment in mathematical problem solving.
G
GA
A
O
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 30
SMART Goals
Specific change Measurable change Attainable, realistic outcome Results driven (student achievement) Timebound
—Garmston, JSD, Summer 1997, pp. 64-65
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council
Sample SMART Goals
By 2010, student achievement in mathematics at all grade levels 3–8 will increase by 15% as measured on the state assessment.
By 2010, 90% of the 10th-grade students will demonstrate proficiency in writing on the state writing assessment.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 32
SHORT-TERMKnowledge
Skills
Teachers gain knowledge and skills.
Increase teachers’ content knowledge.
MEDIUMAttitudes
AspirationsBehaviors
Teachers gain aspirations, attitudes, and behaviors.
Change teachers’ instructional practices.
LONG-TERMIntended Results
Students gain aspirations, attitudes, behaviors, knowledge, skills.
Increase student achievement.
ResultsWhat results occur from professional learning for
individuals, schools, communities …
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 33
Setting Standards for Acceptable Performance
Specifies how good is good enough
Specifies “success” in advance
Provides a benchmark/baseline for comparison before and after staff development
Page 39IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 34
Sample Standards for Success Increased student achievement
8% overall increase on state assessment
7% more students in the advanced category and 20% fewer students in the not-proficient category
25% more underrepresented students in the proficient category
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 35
Types of Change
Knowledge Attitude Skill Aspiration Behavior
Page 38IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 36
KASABKnowledge Conceptual understanding of information,
theories, principles, and research
Attitude Beliefs about the value of particular information or strategies
Skill Strategies and processes for applying knowledge
Aspiration Desires, or internal motivation, to engage in a particular practice
Behavior Consistent application of knowledge and skills
Page 38IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council37
Student Teacher Principal
Knowledge Students know problem-solving strategies.
Teachers know multiple problem-solving strategies and when to use them.
Principals know what to look for in a problem-solving lesson.
Attitude Students enjoy problem solving.
Teachers believe students’ competence in math is important for success both in and out of school.
Principals believe teachers make a difference in students’ academic success.
Skill Students know how to use their problem solving strategies.
Teachers know how to assess students’ problem-solving ability.
Principals know how to observe and give feedback.
Aspiration Students want to participate in higher- level math classes.
Teachers want all students to learn math.
Principals want all teachers and students to succeed.
Behavior Students apply mathematical reasoning to solve problems in and outside of math.
Teachers provide students with daily practice in problem solving.
Principals monitor student problem-solving scores on benchmark tests and identify students who need extra help.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 38
Theory of Change Identifies the components/actions of a
program (what the program does)
Specifies the relationship among the components to explain how the change occurs (sequence of actions)
Delineates the underlying assumptions upon which the program is based
Pages 40-46IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 39
Source: University of Wisconsin Extension Service
Everyday Theory of Change
Headache Get pills Take pills Headache gone; return to work; contribute to economic well-being of community
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 40
Theory of Change
Components/Actions of the program
Sequence of components to show how results will occur
Assumptions underlying the program
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 41
Teachers participate in collaborative learning experiences.
Teachers implement new learning in their instruction.
Student performance increases.
Page 41IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 42
Activity
Form groups of two to three people.
In each group, make five cards.
S Asp BAttK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 43
Possible Theories of Change
K + S B + Att + Asp
Asp + Att K B
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 44
Activity Using the index cards you made,
create two to three possible theories of change. You might begin with a familiar one.
For each theory of change you create, identify its underlying assumptions.
For each theory of change, also identify the potential implications for key decisions about professional development.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 45
Sample Theory of Change
Review the theory of change on page 42. Start at 12 o’clock.
Read the assumptions on which this program is based on the first paragraph on page 43.
State in your own words what a theory of change is. Page 42
IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 46
Write the goals, objectives, standards of success, and indicators of success for your case.
Create a theory of change for the staff development program you want to evaluate. When you are satisfied, map it on the large chart paper. Write your assumptions below the diagram.
Post your chart.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 47
Students
Teachers
Coaches
Layering Theories of Change for Systemic Reform
Principals
Central Office
Organization
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 48
What is the value of having a theory
of change for a staff development
program?
Talk it over.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 49
Logic Model Specific kind of action plan of the program
specifying the flow of changes (results) that will occur Inputs/resources Activities/processes Initial outcomes Intermediate outcomes Results
Results (Goals)
Inputs Activities IntermediateOutcomes
InitialOutcomes
Pages 46-50IN THE BOOK
Homework Assignment Read pages 46-51 Create a logic model for at least 2 actions
or components of your staff development program’s theory of change. (Worksheet #4)
Assess your progress using the Evaluability Assessment Rubric on page 51.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 50
Whew! My brain is tired!51
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 52
Review of Session Outcomes
Describe the eight-step evaluation process and the components of each step. √
Define evaluation in relationship to staff development. √ Understand how a staff development program’s theory
of change and logic model influence the success of a staff development program and its evaluation. √─
Construct an evaluation framework for a staff development program.
Acquire strategies, tools, and resources to assist in evaluating staff development.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 53
Final Activity
What have you learned
about evaluating staff
development that will
help you as you develop
the process for
evaluating your own staff
development program?
ASSESSINGIMPACT
Evaluating Staff DevelopmentSecond EditionJoellen Killion
Corwin Press, 2008A joint publication with the National Staff Development Council
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 55
Logic Model Specific kind of action plan of the program
specifying the flow of changes (results) that will occur Inputs/resources Activities/processes Initial outcomes Intermediate outcomes Results
Results (Goals)
Inputs Activities IntermediateOutcomes
InitialOutcomes
Pages 46-50IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 56
Inputs/Resources
Outputs/Activities
Initial Outcomes
K-Knowledge S-Skills
IntermediateOutcomesA-Attitudes
A-AspirationsB-Behaviors
Intended Results
Results
Formative Questions Summative Question
Program Goals/Objectives
Implementation Impact
Logic Model: Planning and Evaluating
Theory of Change
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 57
What’s the value of having both a theory of change
and a logic model
for a staff development
program?
Talk it over.
58
Step 2: Formulate Evaluation
Questions
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 59
Step 2: FormulateEvaluation Questions
Planning PhaseCarefully and thoughtfully crafted evaluation questions give structure and focus to the evaluation framework.
Page 53IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 60
A Good Question . . .
Focuses your evaluation. Guides the design of the evaluation
framework. Clarifies what the intended users
want to know. Facilitates both design of evaluation
and use of findings.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 61
Types of Evaluation Questions Need
What is the problem we want to solve? Design/Planning
How do we prepare the staff? Process
Is the program being implemented as planned?
ImpactWere the intended results achieved?
Pages 58-59IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 62
Different Questions Require Different Kinds of Evaluation Processes and Evidence
Did student achievement increase by 3% as measured by the state test?
What components of the program do teachers perceive to be most beneficial in changing their instruction?
What changes in teachers’ instructional practices occurred?
To what degree was the program implemented?
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 63
Characteristics of a Good Question
Reasonable Appropriate Answerable Specific in terms of
program success Specific in terms of
measure of performance Page 60
IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 64
ActivityRead the
examples of evaluation
questions and the
commentary on pages 60–63.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 65
Sample Evaluation Questions Improve student writing performance by 8% as
measured on the district grade-level writing rubric: Did students’ classroom writing performance (as measured by the district’s grade-level rubric) increase from last year to this year by 8% as a result of teachers’ use of process writing?
Increase student performance in math problem solving by 5% on the state math assessment: Did students’ scores on the state math assessment in the area of problem solving increase by 8% in classrooms where teachers were using the new instructional strategies to teach students how to solve problems?
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 66
Evaluation Question
What is/are your evaluation question(s)? Be sure that the questions align with the
program’s goals. Review your theory of change to see if it is possible to answer your questions given the program’s design.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 67
Share your evaluation questions with a partner and ask for his/her commentary
about whether your question is:
Reasonable. Appropriate. Answerable. Specific in terms of program’s
success. Specific in terms of measurement.
68
Step 3:
Construct the Evaluation Framework
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 69
Step 3: Construct the Evaluation Framework
Planning Phase
Decisions about the evaluation Who will conduct the evaluation? What do you need to know (questions)? How will you know (evidence needed)? Who or what will tell you (data sources)? How, when, and where will you collect evidence (data-
collection methods)? How will you plan to analyze the data? When will the evaluation be completed?
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 70
Beginning Steps of an Evaluation Framework
I need to know . . . By collecting the following kinds of
information . . . Information will be most useful if it
comes from . . . The data can best be collected
through . . .
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 71
Mini-Lessons Impact Means . . . Measuring Change Experimental vs.
Quasi-Experimental Design
Data Sources Data-Collection
Methods
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 72
Mini-Lesson 1: Assessing Impact
Assessing impact means measuring change.
Change in what?
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 73
Mini-Lesson 2:Data Sources
Sources are the people, places, documents, etc., that provide the information you need.
What and who are possible data sources for each of your evaluation questions?
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 74
How effectively are teachers implementing inquiry-based instruction in their science lesson?
Possible data sources: Teachers Students Principals/administrators Student test scores Science kit checkout Lesson plans
Test question analysis Observational report Standards Pre- & post-teacher
tests Student work Teacher reflection
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 75
Mini-Lesson 2: Triangulation of Data Sources
Multiple sources
of information for the same kind
of information
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 76
Types of Data Sources
Authentic Approximate
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 77
Determining Data Sources
Goal Standard of Success
Indicator of Success
Data Source
Increase student achievement in mathematics
By 10% in three years
Fair and reliable measures of mathematics achievement
• State assessment
• Level tests• Schoolwide
common assessments
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 78
Mini-Lesson 3:Data Collection Methods
Specific tools or processes used to collect the desired data Surveys Interviews Logs Observations Tests Focus Groups Documents/Artifacts Expert Judgments Etc.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 79
Sample Data Collection Instruments
Review the sample data collection tools on pages 159–194.
Consider how these tools might be adapted for your evaluation.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 80
Determine Appropriateness of Data-Collection Methodologies
Financially feasible Reasonable in terms of human
capacity Appropriate for type of evidence
desired Authentic vs. approximate
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 81
TaskComplete the evaluation framework for an evaluation you want to do. Program objectives Evaluation questions Type of design Comparison group Data sources Data-collection methods Timeline Page 70
IN THE BOOK
ASSESSINGIMPACT
Evaluating Staff DevelopmentSecond EditionJoellen Killion
Corwin Press, 2008A joint publication with the National Staff Development Council
83
Step 4:Collect
Data
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 84
Step 4: Collect DataConducting Phase
Systematic collection of data
Page 97IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 85
Possible Ways to Score Lesson Plans
Is complete: yes or no Includes basic features, such as a standard,
instructional strategy, and a method of assessment (present or not present)
Meets standards of quality (rubric), e.g., appropriateness of the instructional strategy and assessment process for the standard
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 86
Collect Data
The key in data collection is to be methodical and systematic.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 87
Collect Data In your data
collection process, what do you want to be particularly thoughtful about?
88
Step 5:Organize, Analyze, and Display Data
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 89
Step 5: Organize, Analyze, and Display Data
Conducting Phase
Page 101IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 90
Analyzing Data
Describing Counting Factoring Clustering Comparing Seeking
trends/patterns
Examining outliers Finding covariation Eliminating rival
explanations Modeling Univariate analysis
Pages 103-105IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 91
Ways to Analyze Data Review pages 103–105. Which one(s) will be most appropriate for
your data and evaluation question?
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 92
Ways to Analyze
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 93
Sample AnalysisLevel of Teacher Use Student Achievement
Routine Use
Mechanical Use
Nonuse Use
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 94
What are the best ways to display your data for the target audiences of your evaluation?
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 95
Step 6:Interpret Data
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 96
Step 6: Interpret DataConducting Phase
Making meaning from data Forming and supporting claims of merit, worth,
and/or impact Clarifying attribution and contribution Making recommendations
Page 109IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 97
Evidence and Claim
Evidence is the data.
Claim is the statement of value or worth, the conclusion drawn based on the data.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 98
Evidence or Claim
Read the statements on the handout.
Determine which are claims and
which are evidence.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 99
Types of Claims
Merit Worth Impact Attribution vs. Contribution
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 100
Rigor of Claims
Are the claims strong or weak?
Are the claims major or minor?
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 101
Attribution vs. Contribution
Attribution: Program is solely responsible for producing results.
Contribution: Program was involved in producing results.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 102
Forming Recommendations
Revisions to make Actions to take Decisions to make Possibilities to consider
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 103
Interpreting Data
Involve stakeholders Form and support claims Clarify attribution and contribution Make recommendations
104
Step 7:Disseminate and
Use Findings
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 105
Step 7: Disseminateand Use Findings
Reporting Phase
Consider audiences Determine types of reports
Interim versus final Oral versus written
Choose dissemination strategies
Page 119IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 106
“Simplicity as a virtue means that we seek clarity, not complexity”
(Patton, 1997 p. 310)
“To write so that you could not be misunderstood”
(Personal communication, 2001)
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 107
Disseminate Findings
Who wants to know what you learned? In other words, who is/are the audience(s) for your evaluation report? To whom are you obligated to report? To whom do you want to report?
What is the best format in which to present your findings so that they will be clear and easily understood?
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 108
Disseminate Findings
Look at the possible components of an evaluation report on page 123.
Consider one audience for your results.
Which components will be appropriate for that audience?
109
Step 8:Evaluate
the Evaluation
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 110
Step 8: Evaluate the Evaluation Reporting Phase
Resources Design Analysis Interpretation Findings Reporting Evaluator
Page 129IN THE BOOK
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 111
Was the evaluation . . .
EffectiveWorth doingWell done Efficient
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 112
Planning1. Assess
Evaluability
2. Formulate Evaluation Questions
3. Construct Evaluation Framework
Conducting4. Collect Data
5. Organize, Analyze, & Display Data
6. Interpret Data
Reporting7. Disseminate
and Use Findings
8. Evaluate the Evaluation
Evaluating Staff Development
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 113
Review of Session Outcomes
Describe the eight-step evaluation process and the components of each step.
Define evaluation in relationship to staff development. Understand how a staff development program’s theory
of change and logic model influence the success of a staff development program and its evaluation.
Construct an evaluation framework for a staff development program.
Acquire strategies, tools, and resources to assist in evaluating staff development.
Intended outcome: think like evaluators and build staff development evaluation into the planning process for the TSSRRP grant.
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 114
Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 115
Final Activity
What have you learned
about evaluating staff
development that will
help you as you develop
the process for
evaluating your own staff
development program?
top related