evaluating the effectiveness of professional learning
DESCRIPTION
Access Handouts at: http://tinyurl.com/catxtsh . Evaluating the effectiveness OF PROFESSIONAL LEARNING. Jacqueline Kennedy Associate Director of Strategic Initiatives Learning Forward Raleigh, NC December 12-13, 2012. Our Agenda . Our Outcomes . - PowerPoint PPT PresentationTRANSCRIPT
Access Handouts at: http://tinyurl.com/catxtsh
EVALUATING THE effectiveness OFPROFESSIONAL LEARNING
Jacqueline Kennedy Associate Director of Strategic Initiatives
Learning ForwardRaleigh, NC
December 12-13, 2012
Our Agenda
• Evaluating Professional Learning Essentials
#1
• Revisiting the Process of Educator Change
#2 • Delving Deeper into Evaluation Components
#3
Our Outcomes
• Define evaluation in relationship to professional learning;
• Examine the process of teacher change and its impact on learning; and
• Acquire strategies, tools, and resources to assist in evaluating professional learning.
Our Essential Question
• How can evaluating professional learning leverage school, school system, and state improvement effort?
• How will I align professional learning objectives to measurable short, medium and long-term results for educators and students?
• How will I collaborate with others to construct a framework that outlines a detailed plan for evaluation?
• How do I incorporate evaluation into my work and normative practice?
Our Norms
• Share experiences to enrich others.• Ask questions.• Learn by doing.• Set aside any preconceived notions about
evaluating professional learning.• Apply to your own work.
THINK S – A – L – S – A !
Where are we now?
• Independently, read the case study. • At your table discuss components of your
professional learning evaluation.• Chart your process, considering actions, steps, and
information needed to address questions posed. • Identify questions that emerge about evaluating
professional learning. • Be prepared to discuss your work.
Burning Questions
• Select one question shared at your table that might have implications for the whole group.
• Be prepared to share it.• Post all other questions
(one per sticky note) on the chart.
The Standard
Professional learning that increases educator effectiveness and results for all students uses a variety of sources and types of student, educator, and system data to plan, assess, and evaluate professional learning.-Standards for Professional Learning, 2011
Evaluation is:
A systematic, purposeful process of studying, reviewing, and analyzing data gathered from multiple sources in order to make informed decisions about a program. – Killion, 2008
The systematic investigation of merit and worth. – Thomas Guskey, 2000
Link to Student Results
1. Standards-based
professional learning
2. Changes in educator
knowledge, skills, and
dispositions
4. Changes in studentresults
3. Changes in educator practice
4 Key ComponentsSystematic(Established process)
How rigorous is the process?Is it conducted in accordance with standards and guidelines?
Standards(Predetermined criteria)
Does it have merit and/or worth?Does it meet predetermined standards of success?
Audience Who will use the evaluation?For whom is the evaluation being done?
Intended Uses
How will the evaluation be used?What decisions will be made as a result of the evaluation?
Which best describes your experience?A B
Externally driven and designed Internally driven and designed
Summative evaluations only Planning, formative, and summative evaluation
Event-based Program-based Looking for answers/solutions form others
Discovering or creating solutions and alternatives with others
Feared Embraced
Filed/shelved Used
Done as an afterthought Planned from the beginning
Documentation Evaluation
Process-focused Results-driven
Presentation of results Reflective dialogue
Shifting PerspectivesFrom To
Externally driven and designed Internally driven and designed
Summative evaluations only Planning, formative, and summative evaluation
Event-based Program-based Looking for answers/solutions form others
Discovering or creating solutions and alternatives with others
Feared Embraced
Filed/shelved Used
Done as an afterthought Planned from the beginning
Documentation Evaluation
Process-focused Results-driven
Presentation of results Reflective dialogue
Group Think
1. How do identified shifts influence you and your work in North Carolina?
2. What aspect of evaluating professional learning do you find essential?
4 Purposes
1. Determine merit and/or worth – determine overall value of program
2. Assess impact – determine overall effectiveness of a program
3. Identify improvements– determine program strengths and weaknesses
4. Provide accountability – determine evidence of compliance and resources
Worth
The extent to which a program(s) or activity(ies) is/are essential to a school’s, district’s, agency’s, or individual’s mission. Worth is an indication of the program’s or activity’s perceived value to constituents or to a single individual.
– Do participants value this program?
MeritThe value of the program is judged by comparing its performance against established standards of excellence in the profession.• Did the program
achieve its goals and meet its standard(s) of success?
Means goals vs. End goalsMeans End
Provide professional learning (Activities or strategies employed to achieve a result)
e.g. Train 100% of the teachers on differentiation.
Increase student achievement results(Results or behavioral changes expected)
e.g. Student achievement increase by 10% on a reading inventory.
Programs Demonstrating Merit
May or may not assess the impact of the program on student achievement results.
– Will — if the goal is to increase student achievement (results)
– Will not — if the goal is to provide service
Evaluation to Assess Impact
• Requires program goals to be about increasing student achievement results.
• Requires a theory of change to explain how the program will produce intended results.
Program vs. Event Evaluation
A program is a set of purposeful, planned actions and the support system necessary to achieve the identified goals. Effective professional learning programs are ongoing, coherent, and linked to student achievement.
Example
1. A workshop on Common Core standards.2. A workshop on Common Core standards
coupled with planned planned classroom-support, ongoing coaching and feedback, demonstrations, and observations in other classrooms incorporated into a comprehensive system, ongoing plan designed to develop educator knowledge, attitudes, skills, aspirations, and behaviors.
3 Types of Evaluation
1. Planning – before program design to provide information on conditions or needs to address
2. Formative – during implementation to provide information on whether the program is working as designed
3. Summative – after completion to provide information on outcomes or overall impact
Note to Self
• How are key components, purposes, and types essential to understanding evaluation?
• What about evaluating professional learning interests your most?
Black Box Evaluations
Input Output
Actions Results
Glass Box Evaluations
?A simplistic approach to professional learning evaluation that fail to amplify the
underlying theory and operation of the professional learning program.
A comprehensive approach to professional learning evaluation that illuminates how professional learning program components interact to produce results.
Black Box
ProfessionalLearningAction
Student Achievement
Results?
Focus on outputs rather than what
occurs in the program or what is presumed to be causing those outcomes and why.
Student Achievement
ResultsProfessional
Learning
CurriculumDevelopment
Nonacademicfactors
?
Black Box Focus on inputs and fail to shed light on HOW a program’s
activities and resources interact to
produce results.
Student Achievement
Results
ProfessionalLearning
Actions
Glass Box Focus on what occurs and how it occurs
within the program.
Glass Box
Student Achievement
ResultsProfessional
Learning
Coaching/Follow-upInstructional ResourcesImplementation MonitoringStudent Assessment
Focus on illuminating factors contributing to transformation
process.
Black Box vs. Glass Box
What is the difference?
BREAKSee you back in 15 minutes.
Your Evaluation Process
Write 4-5 sentences that describe the evaluation process/steps you currently use as a leader member responsible for assisting others with evaluating professional learning. • Be specific.• Be frank. • Write in complete sentences.
8 Smooth Evaluation Steps
Planning1. Assess
Evaluability
2. Formulate Evaluation Questions
3. Construct Evaluation Framework
Conducting4. Collect Data
5. Organize, Analyze, & Display Data
6. Interpret Data
Reporting7. Disseminate
and Use Findings
8. Evaluate the Evaluation
Deconstructing the Steps
1. Distribute evaluation step cards at your table.2. Independently, read about your assigned step.3. Regroup with others with the same step.
– Describe how the step contributes to evaluation.– Identify concepts embedded in the step. – Recommend strategies for addressing this step
to increase the effectiveness of professional learning evaluations.
Group Discussion Protocol
• Return to your original table group. • In round robin fashion, beginning with step one,
share key points, evidence, and strategies. • Consider how steps are independently and
collectively important.• Be prepared to discuss insights gained with the
whole group.
Note to Self
• Revisit your process. • Based on what you now
know about the eight evaluation steps, which steps were present? Missing?
• What is one actions you will take to address missing steps in your current process?
Summer Institute Academy Vision
Maria Pitre MartinLynne Johnson
LUNCHSee you back in 45 minutes.
Assumptions about Change
• Independently, read the article by Michael Fullan. • With a partner, respond to one or more of the
questions below. • Jot down notes, and identify common themes that
emerged. 1. What aligned with your thinking? Share why.2. What challenged your thinking? Share why.3. Describe something you want to explore. Be specific. 4. What was the most important takeaway for you?
Change is Learning
Concerns-Based Adoption Model : Developed by Bill Rutherford, Gene Hall, Shirley Hord, and Susan Loucks-Horsley
4 Components:1. Stage of Concern – 7 stages of responses2. Levels of Use – eight ranges of intervention use3. Innovation Configuration – described actions4. Change facilitators – leaders of learning
Change Learning Exchange• Distribute numbered cards (#1-4) at your table.• Read and the corresponding article on change.• Prepare a two-minute talk about your article. Use
any of the following to prepare.– Why your focus area is important.– Implications of the change process.– Ways to facilitate your area of change. – Challenges you anticipate when helping
others understand this area of change.
Change Spotlight• Find a partner that read a different article.• Take two minutes each, discuss your article.
Focus on any of the following: – Why your focus area is important.– Implications of the change process.– Ways to facilitate your area of change. – Challenges you anticipate when helping
others understand this area of change. • Listen for the timer to repeat the process.
Note to Self
• What was one benefit of revisiting the change process?
• What new insights did you gain as a result of your reading and discussion with others?
Change Self-Assessment
• Use the self-assessment to determine the level of implementation in your system.
• Note the evidence that informs your score.• Identify your area of strength and area for
improvement. • Be prepared to meet with others about your
results, and evidence.
Strengths
• Determine which area is your strongest.• Meet with colleagues who share that same area
as a strength. • Share the evidence of your strength, i.e., what is
it you are doing in relationship to this area?• Record strategies you receive.
Areas for Improvement
• Determine which area is your greatest area for improvement.
• Meet with colleagues who share that same area for improvement.
• Identify what happens when this area is a gap.• Use strengths collected from others to identify
possible strategies you might consider to address this gap.
Group Reflection1. What does this mean
for your work?2. What do you need to
share and with whom?3. How will you continue
to develop strengths?4. What is one strategy
you will take away to develop an area for improvement?
BREAKSee you back in 15 minutes.
Evaluation of ill-conceived professional learning programs waste limited human and fiscal resources. In these cases, evaluations sanction programs with gaps in their conceptual frameworks by appearing to condone mediocrity. Wasted evaluations rob students and educators of powerful opportunities to learn at high levels. And when evaluations attempt to assess programs with fuzzy goals, faulty conceptual frameworks, or illogical, poorly structured activities, the evaluation may be difficult to design and of questionable integrity.
-Killion, J. (2008). Assessing Impact. p. 33.
Thorough planning facilitates sound
evaluation.
Evaluating Professional LearningPlanning
1. Assess Evaluability
2. Formulate Evaluation Questions
3. Construct Evaluation Framework
Conducting4. Collect Data
5. Organize, Analyze, & Display Data
6. Interpret Data
Reporting7. Disseminate
and Use Findings
8. Evaluate the Evaluation
Step 1: Assess EvaluabilityPlanning Phase
The degree to which a proposed plan for staff development clearly articulates what the staff development program is and how it is expected to produce change.
Four Components of Assessing Evaluability
1. Examine program goals, objectives, standards for success
2. Examine program’s theory of change3. Examine program’s logic model4. Adjust program based on assessment
Clarifying Your Program
• What are the program’s goals?– Goal is desired state.
• Improve physical well-being.– Objectives describe the changes expected.
• Lose 15 pounds in six months.– Activities explain how to achieve the goals and
objectives.• Walk daily.• Eat healthful foods.
Goal, Objective, or Activity?
1. Produce responsible citizens.2. Increase student accuracy in applying the
scientific process in a laboratory setting.3. Provide opportunities for professional learning.4. Create professional learning communities.5. Increase student performance 15% on state
assessment in mathematical problem solving.
SMART Goals: Combine Goals and Objectives
• Specific change• Measurable change• Attainable, realistic outcome• Results driven (student achievement)• Timebound
- Garmston, JSD, Summer 1997, pp. 64-65
Creating Goals & Objectives
• In table groups, work collaboratively to determine program goals and objectives.
• Refer to the Teacher Quality Project simulation to guide table group discussions, and focus applications.
• Jot down any notes on the process.
SHORT-TERMKnowledge
Skills
Teachers gain knowledge and skills.
Increase teachers’ content knowledge.
MEDIUMAttitudes
AspirationsBehaviors
Teachers gain aspirations, attitudes, and behaviors.
Change teachers’ instructional practices.
LONG-TERMIntended Results
Students gain aspirations, attitudes, behaviors, knowledge, skills.
Increase student achievement.
Goals What results occur from professional learning for
individuals, schools, communities …
Setting Standards for Acceptable Performance
• Specifies how good is good enough• Specifies “success” in advance• Provides a benchmark/baseline for
comparison before and after professional learning
Sample Standards for Success
Increased student achievement– 8% overall increase on state assessment– 7% more students in the advanced category and
20% fewer students in the not-proficient category– 25% more underrepresented students in the
proficient category
Types of ChangeWhat is changing?
• Knowledge• Attitude• Skill• Aspiration• Behavior
KASABsKnowledge Conceptual understanding of information,
theories, principles, and research
Attitude Beliefs about the value of particular information or strategies
Skill Strategies and processes to apply knowledge
Aspiration Desires, or internal motivation, to engage in a particular practice
Behavior Consistent application of knowledge and skills
Student Teacher Principal
Knowledge Students know problem-solving strategies.
Teachers know multiple problem-solving strategies and when to use them.
Principals know what to look for in a problem-solving lesson.
Attitude Students enjoy problem solving.
Teachers believe students’ competence in math is important for success both in and out of school.
Principals believe teachers make a difference in students’ academic success.
Skill Students know how to use their problem solving strategies.
Teachers know how to assess students’ problem-solving ability.
Principals know how to observe and give feedback.
Aspiration Students want to participate in higher- level math classes.
Teachers want all students to learn math.
Principals want all teachers and students to succeed.
Behavior Students apply mathematical reasoning to solve problems in and outside of math.
Teachers provide students with daily practice in problem solving.
Principals monitor student problem-solving scores on benchmark tests and identify students who need extra help.
Types of Change
• In table groups, work collaboratively to align types of change (KASAB) to each objective.
• Refer to the Teacher Quality Project simulation to guide table group discussions, and focus applications.
• Jot down any notes on the process.
2/3 : 5/6 = 4/6 : 5/6 =
4 : 5
GOAL: Improve student achievement in rational numbers through professional development focused on increasing teachers’ content knowledge and content-specific pedagogy.
Sample KASAB WorksheetType of Change ExampleKnowledge Understands rational numbers and content-
specific vocabulary
Attitude Believes that developing students’ understanding of rational numbers contributes to their success in higher-level math, high school graduation, and college and career readiness
Skill Uses higher-order thinking skills to elicit students’ understanding of rational numbers and thinking during problem solving
Aspiration Desires to teach all students to succeed in higher-level math
Behavior Applies effective question skills in math instruction to elicit student thinking during problem solving
• Increase teachers’ content knowledge and precision in content vocabulary related to rational numbers as evident in lesson plans and instructional materials KNOWLEDGE
• Increase teachers’ accuracy and frequency of use of questioning strategies in math instruction by at least two levels as described in the IC maps SKILLS, ATTITUDE, ASPIRATION, & BEHAVIOR
• Increase students’ ability to explain their thinking as they solve rational number problems by two levels as described on the IC maps KNOWLEDGE & SKILLS
• Increase students’ achievement on benchmark assessments on rational number problems KNOWLEDGE & SKILLS
Objectives
Objectives Evaluation Questions EvidenceIncrease teachers’ content knowledge and precise vocabulary in rational numbers as evident in lesson plans and instructional materials KNOWLEDGE
Did teachers’ understanding of rational numbers increase by at least 20% as measured an analysis of lesson plans and instructional materials?
Pre and post-lesson plan and instructional material analysis
Increase teachers’ accuracy and frequency of use of questioning strategies in math instruction by at least two levels as described in the IC maps SKILLS, ATTITUDE, ASPIRATION, & BEHAVIOR
Are teachers demonstrating accurate and frequent use of questioning techniques in their daily instruction as described in the IC Maps?
IC MapWalk through guideTeacher observation guide
GOAL: Improve student achievement in rational numbers through professional development focused on increasing teachers’ content knowledge and content-specific pedagogy
Objectives Evaluation Questions EvidenceIncrease students’ ability to explain their thinking as they solve rational number problems by two levels as described on the IC maps KNOWLEDGE & SKILLS
Are students explaining their thinking as they solve problems involving rational numbers as described in the IC Map for student thinking in math?
IC MapStudent thinking guideStudent observation guide
Increase students’ achievement on benchmark assessments on rational number problems KNOWLEDGE & SKILLS
Did student performance on the six-week benchmark assessments increase by 10% from previous assessment over the school year on rational number problems
Benchmark assessments
GOAL: Improve student achievement in rational numbers through professional development focused on increasing teachers’ content knowledge and content-specific pedagogy
Theory of Change
• Identifies the components/actions of a program (what the program does)
• Specifies the relationship among the components to explain how the change occurs (sequence of actions)
• Delineates the underlying assumptions upon which the program is based
Source: University of Wisconsin Extension Service
Everyday Theory of Change
Headache Get pills Take pills Headache gone; return to work; contribute to economic well-being of community
Theory of Change
• Components/Actions of the program• Sequence of components to show how results
will occur• Assumptions underlying the program
Teachers participate in collaborative learning experiences.
Teachers implement new learning in their instruction.
Student performance increases.
Professional development core features:•Content focus
•Active learning
•Coherence•Duration•Collective participation
Increased teacher knowledge and skills; changes in attitude and beliefs
Change in instruction
Change in student learning
Core Conceptual Framework for Studying the Effects of Professional Development on Teachers and Students Desimone, L. (2009)
Professional Learning
Opportunities
Teacher interpretation and
utilization of available
understandings and skills
Teacher outcomes:•Change in practice•No change in practice
Student outcomes:
•Positive changes•No changes•Negative changes
Students’ interpretation and
utilization of available understandings and
skills
Student Learning
Opportunities
Timperley, H. & Alton-Lee, A. (2008). Reframing teacher professional learning: An alternative policy approach to strengthening valued outcomes for diverse learners. Review of Research in Education, 32, pp. 328-369.
Sample Theory of Change
• Review the theory of change. • Generate a list of assumptions depicted in the
theory of change.• State in your own words what a theory of
change is.• Be prepared to share your definition.
Assumptions • Teachers’ understanding of various instructional
strategies provides their use of them. • In order to use new instructional strategies, teachers have
the appropriate material, human, and time resources. • Success builds with an increase in practice and support.• Consistency and accuracy in implementation of new
practices come over time. • Student achievement comes from consistently applied,
research-based instructional practices delivered by teachers who have in-depth understanding of content and content-specific instructional strategies.
Creating Theories of Change
• In table groups, work collaboratively to develop a theory of change and identify guiding assumptions aligned to the simulation.
• Refer to the Teacher Quality Project simulation to guide table group discussions, and focus applications.
• Jot down any notes on the process.
Partner Chat
• What is the value of having a theory of change for a staff development program?
Logic ModelSpecific kind of action plan of the program specifying the flow of changes (results) that will occur • Inputs/resources• Activities/processes• Initial outcomes• Intermediate outcomes• Results
– Results (Goals)
Inputs Activities IntermediateOutcomes
InitialOutcomes
Sample Logic Model
• Review the sample logic model. • Notice how the inputs, activities, initial
outcomes, and intermediate outcomes align in a LOGICAL way to demonstrate how the program moves toward the intended results.
Creating a Logic ModelAt your table group, create a logic model for at least two actions or components of your professional learning program’s theory of change.
The Power of What We Think
I can only change how I act if I stay aware of my beliefs and assumptions. - Margaret Wheatley
Running alongside our visible and expressed values is a competing set of values we’re unaware of. - Robert Kegan
Begin challenging your own assumptions. Your assumptions are your windows on the world. Scrub them off every once in awhile, or the light won't come in. - Alan Alda
Assumptions About Professional Learning
1. Read each assumption statement. 2. Code each with your level of agreement.3. Identify potential implications for the role of
teacher evaluation.4. Share your thoughts and answer the questions
with others at your table.
Evaluation Assumptions
• The staff development program is data-driven, research-based, and well-defined.
• The school, district, or regional agency has the capacity, including fiscal and human capital, to implement both the program and evaluation with fidelity to their designs.
• Key stakeholders in the school, district, or agency intend to use the evaluation results to make decisions about the program.
Note to Self
• How does stating one’s beliefs, and unmasking assumptions help reveal barriers to change?
• How do competing values form obstacles to change?
Moving to Step 2 Means …
We believe we have: 1. A planned, coherent,
in-depth professional learning program that is– Feasible.– Clear.– Powerful.– Worth doing.
2. KASABs.3. Goals.4. Standards for success.5. Theory of change.6. Logic model.
Debriefing – Implications for NC
1. Reflect on the learning for the day using the the following three areas to guide your thinking.– What are the benefits?– What are the barriers?– What are the implications?
2. Consider beginning an evaluation framework (see sample) or theory of change for a current professional learning program.
End-of Day Reflections
• How are the ideas presented today CONNECTED to what you already knew?
• What new ideas did you get that EXTENDED or pushed your thinking in new directions?
• What is still CHALLENGING or confusing for you to get your mind around? What questions or wonderings do you now have?
Problems of Practice Exchange
• Consider your current challenges, dilemmas, and frustrations to strengthen professional learning.
• Outline what you believe to be the most critical areas you seek solutions, strategies, and answers.
• Bring your notes with you tomorrow.