You need

• Internet connection• “Parking Lot” • Materials– Agenda and articles Evidence Analysis Sheet for

Elements coveredFor demo– Copies of the Teacher Evaluation rubrics – GREEN BOOK

Welcome Session 1• Everyone needs: – 1.) a green book Instructional Practice– 2.) a three-hole punched Notebook packet for a binder– 3.) the “Connection Document” in the Notebook– 3.) a teacher performance rubric (optional)– 4.) a purple book (just to take a quick tour)

• DO NOW: Please skim/read the 2 articles on the changes in evaluation (pages 1-3). We will discuss them today.

• And, if you have time tonight, read the article on difficult conversations in this handout.

Please Read the 2 Brief Articles in the Handout about

Evaluation ChangesDO NOW

Please skim/read the 2 articles on the changes in evaluation (pages 1-3). We will discuss them today.

And, if you have time tonight, read the article on difficult conversations in this handout.

Introductions• Name• Position• School (if not a district-wide administrator)• What you hope a calibration session will do for

you and the teachers you evaluate. (Just a sentence or 2.)

Agenda for Two Days

IntroductionsContent:1. Calibrating the Feedback on the 4 Standards

for Teachers and Specialists2. Written Feedback: SJEIR Format for standard,

judgment, evidence, impact statement, and recommendation

3. Effective Face-to-Face conferencing and feedback during the year and for Formative and Summative Evaluations

Essential Questions

How do we implement the educator evaluation system in a way that:

Improves teaching, and therefore--Raises student achievement

Has high inter-rater reliability across the districtDoesn’t overwhelm us with the work load?Has integrity and reflects the district’s and school’s

value system

Instructional Practices Book for Data Gathering and Calibrating

• Most current and comprehensive book in the country

• Based on 6,000 pages of research and practice• ELL and special education essays• 40 educators worked on the book along with an

M.D./Ph.D. in neurology who assisted with the brain based learning components

• Read the table of contents

Quick Materials TourGreen Book,

Instructional Practices for Teachers

Packet:Connection Document: Part 6.21

Handout withAgendaArticlesNote-taking form

WHAT ARE WE LOOKING FOR IN EVALUATION?WHAT IS PROFICIENCY IN TEACHING?

Setting Standards as a District

Heifetz: Change Technical versus Adaptive Change

https://www.youtube.com/watch?v=UwWylIUIvmo

Technical versus Adaptive ChangeTechnical

1. Easy to identify

2. Often lend themselves to quick and easy (cut-and-

dried) solutions

3. Often can be solved by an authority or expert

4. Require change in just one or a few places; often

contained within organizational boundaries

5. People are generally receptive to technical solutions

6. Solutions can often be implemented quickly—even by

edict

EXAMPLES

Take medication to lower blood pressure � Increase penalty for drunk driving� State: Developed new processes and procedures for

evaluating teachers

Adaptive1. Difficult to identify (easy to deny)

2. Require changes in values, beliefs, roles, relationships, &

approaches to work

3. People with the problem do the work of solving it

4. Require change in numerous places; usually cross

organizational boundaries

5. People often resist even acknowledging adaptive

challenges.

6. “Solutions” require experiments and new discoveries;

they can take a long time to implement and cannot be

implemented by edict

EXAMPLES

Change lifestyle to eat healthy, get more exercise and lower �stress

Raise public awareness of the dangers and effects of drunk �driving, targeting teenagers in particular

Districts: Change the way in which teachers are evaluated

Adaptive Changes in EvaluationThe SHIFTs: Processing Partners

Blurring the distinctions between supervision and evaluation (coaching and rating)

Instead of focusing on teaching, focusing on learning when evaluating teachers

Expanding our definitions of data and data from an observation to include artifacts

Increasing the frequency of feedback and face-to-face conferences Expecting teachers to be partners in assessing their own effectiveness

Although the process changes and the documents, this change is deep change and takes time to become effective.

Think and then Share: What is noteworthy to you? Share with a partner.

Learning-Focused; Student-Focused“As a result, students….”

“The task predicts performance.”

Elmore says, in Instructional Rounds, “What predicts performance is what

students are actually doing…the instructional task is the actual work that students are asked to do during the process of instruction-not what

teachers think they are asking students to do or what the official

curriculum says that that student are asked to do...”

Why is this shift to the student and the student’s responsibility important in supervision and evaluation?

Impact of Teacher Performance on Student Performance

Stronge, J. and Tucker, P. Teacher Evaluation and Student Achievement. National Education

Association. 2000 p.2 (see Sanders) gb I-4

When children, beginning in third grade, were placed with three high performing teachers in a row, they scored, on average at the 96th percentile on Tennessee’s statewide mathematics assessment at the end of fifth grade.

Sanders, W. and Rivers, J. Cumulative and residual effects of teachers on future student academic achievement (Research Progress Report) Knoxville TN. U of TN Value-Added Research and Assessment 1996 gb I-4

When children with comparable achievement histories starting in third grade were placed with three low performing teachers in a row, their average score on the same mathematics assessment was at the 44th percentile.

Teacher Impact Can be Significant

Teachers performing in the top 20% = student scores at

96%

Teachers performing in the bottom 20% = student scores at

44%

Differential 52%

Hershberg,T., Simon,V. Lea-Kruger,B. (2004, December) The Relevance of

Value Added. The School Administratoranb 130

While family income remains the best predictor of absolute achievement, good instruction is 10 to 20 times more powerful in predicting student growth over the year than family background/family income/race/ gender. (p.10).

Teacher Achievement Gap 130

Higher performing teachers implement more of what they learn in workshops than lower performing teachers.

As a result, the gap in student achievement between the higher performers and the lower performers is constantly increasing.

Supervision and evaluation and professional development is the only way to reduce the teacher achievement gap.

Gb 427

Workshops and courses result in

10% to 15% classroom

implementation unless aligned with teacher evaluation

andfeedback/coaching

Re-teaching

ProfessionalDevelopment

Coaching

Assessment

10-15%Mastery

THE DEPARTMENT OF EDUCATION’S GUIDELINES

Processes and Procedures

Evidence Observation-based, Artifacts, Beyond Classroom

• Three categories of evidence must be collected for each educator:1. Multiple measures of student learning, growth, and

achievement 2. Judgments based on observations and artifacts of

professional practice3. Additional evidence relevant to standardsoThis includes evidence collected by the educator and

shared with the evaluator relating to fulfilling Standard III: Family and Community Engagement and Standard IV: Professional Culture from the Model System Teacher Rubric

21Massachusetts Department of Elementary and Secondary Education

Implementation ResponsibilitiesEducators and Evaluators

• Educator responsibilities:– Identifying, collecting, and organizing artifacts/evidence related to

goal progress– Documenting action steps (from their plans) completed– Collecting and submitting common artifacts requested by the

evaluator– Collecting and submitting evidence related to Standards III and IV

• Evaluator responsibilities:– Making resources and supports available for success implementing

standards III and IV– Identifying common artifacts/evidence related to standards III and IV– Observing practice and artifacts and providing regular and specific

feedback on performance as related to Standards I and II– Monitoring progress—including midpoint check-ins– Organizing and analyzing data over time 22

WHY DO WE NEED TO CALIBRATE?Defining Proficiency for All Teachers and Evaluators

Calibration of Behaviors, Data and Feedback

1. The same observable behaviors and data to rate teachers on elements

2. Consistent quality feedback enables an evaluator to expect higher levels of teacher performance---which leads to higher student achievement

3. Let’s try an observation!

Clearly defined and observable behaviors

InAll EvaluationsSMART Goals

Feedback

The 4 Standards (See 6.41 in the Notebook)

Standards and Indicators for Effective Teaching PracticeI. Curriculum, Planning, and Assessment

II. Teaching All Students

III. Family and Community Engagement

IV. Professional Culture

I-A Curriculum and Planning

I-A-4 Well-Structured Lessons

I-B Assessment I-C Analysis

II-A InstructionII-A-2. Student Engagement

II-B Learning Environment

II-C Cultural Proficiency

II-D Expectations

III-A Engagement

III-B Collaboration

III-C Communication

IV-A Reflection IV-B Professional

Growth IV-C

Collaboration IV-D Decision

Making V-E Shared

Responsibility V-F Professional

Responsibilities

I Curriculum and Planning (Packet:6.23)

I-A-4: Well-Structured LessonsUnpacking the Standard

Exemplary: Develops well-structured and highly engaging lessons with challenging, measurable objectives and appropriate student engagement strategies, pacing, sequence, activities, materials, resources, technologies, and grouping to attend to every student’s needs. Is able to model this element.

Define Precisely in terms of Observable BehaviorsElement I-A-4 The Well-Structured Lesson

• Count off by 4’s to form groups. • Read the exemplary description for I-A-4 in the “connection

document” on 6.23. • Then, read the assigned pages from the green book ( group 1,

2, 3, or 4)• Share the main points from your reading with the group once

finished. Be prepared to share with the whole group.

1. Mastery objectives 11-13, 39-402. Activators 43-503. Summarizers 62-684. 11 components of Effective Group work 91-9

Observable Data Points for I-A-1The Well-Structured Lesson

• Read and report out on your section to your group

• What are you looking for?• Could there be more evidence that you want

to collect from the classroom or from the teacher?

• Add additional “evidence” that you might collect or observe

Group OrganizationTravel farthest place during past 12 months

Leader

Gopher

Reporter

Scribe

Oldest car

Scribe

Leader

Gopher

Reporter

Newest car

Reporter

Scribe

Leader

Gopher

Most years employed PP Schools

Gopher

Reporter

Scribe

Leader Gb 93-94

Note Taking(a skill that develops over time)

AVOID JUDGMENT (“I like”) OR JUDGMENTAL (“boring”) WORDS

Element And Rating based on Evidence

Evidence Observed

Teacher

Evidence Observed

Student

Other evidence you can obtain

IA-4 Well Structured Lesson

QuoteDescribe[your comments, questions]

Avoid: “Too noisy” (judgment)Instead say: Almost all students were talking (observable behavior)

Rating a Teacher’s Performance

NOTE:• A video is de-contextualized. (You would know

the teacher and his/her status: new, experienced, on an improvement plan)

• Single, brief moment• This video provides context (but not a very long

lesson) • https://

www.teachingchannel.org/videos/preparing-for-formal-observations

Share Observations with your Group

First: Share ObservationsThen rank the element as:

–Exemplary–Proficient–Needs Improvement–Unsatisfactory

What do the ratings mean? • Exemplary: This rating is rare (2-5% in most districts) and given

when a teacher surpasses the standards. This teacher is a model for others in a particular area.

• Proficient: This rating is a high standard. It indicates that a teacher is an effective educator.

• The dividing line between the top two and bottom two is based upon if students are being taught effectively and if students are learning

• Needs Improvement: This rating indicates that in an area a teacher’s techniques are not effective and that as a result students are not effectively being taught.

• Unsatisfactory: This rating indicates serious inadequacies in a teacher’s work that indicates either that these areas are completely missing or mainly ineffective.

Defining the Levels In Everyday Terms

ExemplaryProficient

__________________________Needs Improvement

Unsatisfactory

Proficient-----------Needs Improvement

WHAT IS EVIDENCE?Going deeper: Information versus Evidence

Sources of Evidence for Formative and Summative Ratings for Standards 1 and 2

• Three categories of evidence must be collected for each educator:1.Multiple measures of student learning,

growth, and achievement2.Judgments based on observations and

artifacts of professional practice3.Additional evidence relevant to standards

37Massachusetts Department of Elementary and Secondary Education

Evidence vs. Information 35

Data is everything the S and E sees, tastes, hears, smells or has reported

Information has insufficient verification to make a judgment. It leads the S and E to gather sufficient data to make a judgment

Evidence has sufficient verification to make a judgment

Parents come to you on the soccer field and in the hallways to tell you their child’s teacher is “the best teacher their child has ever had”- Information not evidence-

1. Classroom teacher2. Observations 3. Post conferences4. Walk throughs5. Student work6. Room tour

JudgmentEvidence has sufficient verification to make a judgment

The music teacher reports to you that the teacher continues to not pick up his/her students on time at the end of the period

What might be used to make this evidence?

Judgment

Direct and Indirect Data pb 236

• DIRECT: what you see, hear, touch, smell, taste

• INDIRECT: what others see, hear, touch, smell, taste and report to you

Information Evidence

Information is raised to evidence by verification through volume accumulation or triangulation

Assessment Data for a Teacher Who Has Low Growth Scores (Turn and Talk)

Judgment

What additional data will you collect to determine the reason for the low growth scores before you make a judgment?

Time for another video?

• Video (10-15 minutes) 2nd grade Science Primary https://vimeo.com/75200970 (31:00 14 1:00 5) Balance

• Experienced Teacher ELA https://www.teachingchannel.org/videos/evidence-arguments-lesson-planning

• Use the teacher rubric levels to rate the teacher on those elements individually

• Be prepared to compare and discuss ratings with others in the room

• How calibrated are the ratings?

Standard II: Teaching All StudentsII-A-2. Student Engagement

II-A-2.Student Engagement

Uses instructional practices that leave most students uninvolved and/or passive participants.

Uses instructional practices that motivate and engage some students but leave others uninvolved and/or passive participants.

Consistently uses instructional practices that are likely to motivate and engage most students during the lesson.

Consistently uses instructional practices that typically motivate and engage most students both during the lesson and during independent work and home work. Is able to model this element.

Student Engagement: Connection Document

1. Student to student interactions Effective turn and talk (embedded and monitored) Effective processing partners gb. 209-211, 230 (embedded and monitored) All 11 components of effective group work gb. 91-96 Questioning strategies that lead to students interacting with one and other during Q and A sessions. (processing partners, wait time gb 222-225) Students are respectful of other students comments and ideas gb 81-82

2. Effective praise gb 226-227 3. Establishing a homework routines gb 96-105 4. Communicating agenda gb 34-35

Agenda is posted Has enough specific steps Is communicated orally at the outset, Is tied to the mastery objective Is referred to at each step of the lesson

5. Issues directions one step at a time gb 59-60 6. Strategies for assigning relevant homework and for assisting student with successfully completing homework gb 42, 99107,246 7. Manages space and proximity to monitor engagement and facilitate student to student contact gb 82-85 8. Uses questioning strategies that engage students and maintain their attention gb. 212-213

Note TakingRank each element

4=Exemplary; 3=Proficient; 2=Needs improvement 1=Unsatisfactory

Element And Rating based on Evidence

Evidence Observed

Teacher

Evidence Observed

Student

Other evidence you can obtain

Student Engagement

How Calibrated Are You?• Compare in your group:– What are the areas of discussion/difference?– Be prepared to share

– Proficient versus Needs Improvement – Go back to the descriptors. – With practice, this becomes a surer process

FORMALIZING THE WRITING ABOUT THE STANDARDS AND ELEMENTSCALIBRATING YOUR FEEDBACK

Standard, Judgment, Evidence, Impact, (R)(optional Recommendations/Commendations/Suggestions)

• Standard/benchmark: Identifies the area of performance that the paragraph will discuss.

• Judgment: Gives the teacher a clear indication of his/her level of success (or lack there of) in using the area of performance indicated in the claim.

• Evidence: Data of sufficient quality or quantity to support the judgment

• Impact statement: Indicates the impact the teaching had on student learning and development

• Recommendations: Clearly stated areas for a teacher that you direct to address. This can be contrasted with a suggestion which is only an area that a teacher could look into.

Internationally Accepted Structure for Written Feedback

Sample SJEIR Paragraphs

• Also, find the sample SJEIR paragraphs following the agenda (Planning)

• These paragraphs correspond with Chapter 1 in the Green Book.

• I searched under “planning”

Use your Teacher/Student Evidence to Write an SJEI(R) paragraph

Also, there are samples on line from Ribas AssociatesClient Login: Dbrady Saturday6OR: Handout—only Standard I—Well-Planned Lesson

RibasAssociates.com Client Login dbrady Saturday6

SJEI(R)

Chapter 1/ key word- planning/:

Mr. C planned (Standard) in a way that anticipated areas of potential difficulty connected to this lesson. (Judgment)

Evidence##The concepts taught are a part of the fourth grade unit on electricity. He created all the activity sheets connected to this lesson: Wiring Diagram A & B and Individual Apprentice Packets. Each activity sheet had clear and concise directions that highlighted the steps that students needed to complete. Mr. C modified the commercial materials to prevent student frustration. The lesson originally required the batteries being placed in a paper towel roll but he recognized this may be difficult for some of the students. He made battery holders out of wood so the students’ focus would be on the objective of the lesson and not trying to keep the batteries in the roll. ##

Impact:As a result of his planning, students were able to work with a high level of independence.

• Bill needs to (Judgment)

• incorporate more higher order thinking questions into his question and answer periods. (Standard/benchmark)

• Bill asked a total of 16 questions. Ten of those questions were at the recall level, five were at the comprehension level and only one was at the higher order thinking skills level (Evidence).

• Asking higher order thinking questions results in developing students’ higher order thinking skills (Impact).

• Bill should read the section on levels of questions found on pages 214 to 217 in Instructional Practices That Maximize Student Achievement (2010) to learn about the levels of questions and their impact on students and should use the Bloom’s taxonomy chart on page 288 to see lists of verbs that are used when asking higher order thinking questions (Recommendation). (SJEIR)

The person who has lived outside the U.S.

Leader

Gopher

Reporter

Scribe

The person who has most recently been on a beach

Scribe

Leader

Gopher Reporter

The person who was born closest to today’s date

Reporter

Scribe

Leader Gopher

The person whose last name is closest to the end of the alphabet

Gopher

Reporter

Scribe Leader Gb 93-94

The 4 Standards (See 6.41.)

Standards and Indicators for Effective Teaching Practice I. Curriculum,

Planning, and Assessment

II. Teaching All Students

III. Family and Community Engagement

IV. Professional Culture

I-A Curriculum and Planning

I-B Assessment I-C Analysis

II-A Instruction (II-A-2 page 6.30 Student Engagement)

II-B Learning Environment

II-C Cultural Proficiency

II-D Expectations

III-A Engagement III-B

Collaboration III-C

Communication

IV-A Reflection IV-B Professional

Growth IV-C

Collaboration IV-D Decision

Making V-E Shared

Responsibility V-F Professional

Responsibilities

Element II-A-2Student Engagement

• Read the exemplary description and observable behaviors for element II-A-2.

• Read the pages from the green book assigned to you and your partner

• Give your table mates a 2 minute summary of your area

1,2 Effective praise 226-227

3,4 Issues directions one step at a time 59-60

5,6 space and proximity 82-85

The person who has lived outside the U.S.

Leader

Gopher

Reporter

Scribe

The person who has most recently been on a beach

Scribe

Leader

Gopher Reporter

The person who was born closest to today’s date

Reporter

Scribe

Leader Gopher

The person whose last name is closest to the end of the alphabet

Gopher

Reporter

Scribe Leader Gb 93-94

Observable Data Points for II-A-2Student Engagement

• Brainstorm a list of data points you would want to observe (in classroom, outside classroom, artifact, assessment data, teacher interview, other) that would help you evaluate a teacher’s performance on this element

• If you finish before the other groups go back and decide whether each data point is evidence or information

• Note on your connection document any data points that are not already noted

Use your notes and draft an SJEIR paragraph that indicates a rating

Element Observed Teacher

Observed Students

Other possible

I A 4 Well-Structured LessonElement II-A-2Student Engagement

StandardJudgment

Ms. J made effective use did not use planning effectivelyTo a degree, planned, but did not anticipate….Evidence

2 or 3 examplesImpact

As a result, students(Recommendation)

Examples: Planning • Exemplary: Students’ digital portfolios for American History demonstrated that all

levels of students clearly attained the high standards set for them. Noteworthy was the performance of the at risk students in these classrooms. In visits to your classroom it was clear that all students were engaged and responsible for each lesson. In the Salem trial and the final presentations, each student demonstrated their deep understandings about the major themes emphasized by the History department. Their performance on the departmental final exam and departmental rubric for primary source analysis and historical analysis showed that all students developed new analytical and writing skills as well as new knowledge. The curriculum and lesson plans supported this growth from training all students as peer assessors to the carefully planned presentations.

• Proficient: Students gained solid knowledge in your American History classes. In each visit to the classrooms, it was clear that students understood and were generally focused on their task. Their work indicated improvement in historical reading and writing skills. Their collaborative work supported their gains. Their final portfolios were generally well done.

Needs Improvement and Unsatisfactory

• NI: As recorded in observations, students’ behavior and participation in class and their performance in departmental assessments indicate that your planning was not effective for many students. At times groups were completely unproductive and many students did not complete major assignments. Student results in their primary source analysis and on the final exams were below the standards that would have been expected for these students. Your daily and unit lesson plans were often too general and did not provide scaffolding or supports for students as they developed their portfolio, wrote skits, or critiqued their work.

• Unsatisfactory: As recorded in observations, your lack of planning resulted in most of your students not fulfilling departmental curricular expectations including the digital portfolio, the Salem trials, and the primary source critiques. Their performance on both the content area and the historical analysis area of the exams was low.

Calibration Video

• Video (10-15 minutes) https://vimeo.com/album/2589004 (17:00 minilesson geom/alg Dan)Solve for X: 2X + 6(x+5) -10x +2 =8

• Use the teacher rubric to rate the teacher on those elements individually

• Be prepared to share• https://

www.teachingchannel.org/videos/evidence-arguments-lesson-planning

• Teaching Channel Example https://www.teachingchannel.org/videos/reading-like-a-historian-curriculum

How Calibrated Are You?• Compare at your table• What are the areas of discussion/difference?• Be prepared to share

• Proficient versus Needs Improvement • Go back to the descriptors. • With practice, this becomes a surer process

Exit Slip

Use note card to answer:– What would you like more of?– What would you like less of?– What insight or new understanding did you gain

today in working on calibration with colleagues?