teacher and leader effectiveness/ · pdf filepresentation on ntgs strategies . 10:00-11:15 ....
TRANSCRIPT
TEACHER AND LEADER EFFECTIVENESS/ STANDARDS AND ASSESSMENT
COMMUNITY OF PRACTICE:
SEMINAR ON NON-TESTED GRADES AND
SUBJECTS
AUGUST 28, 2012
WELCOME AND OVERVIEW / INTRODUCTION AND FRAMING
Brad Jupp, Senior Program Advisor on Teacher Initiatives, U.S. Department of Education
2
INTRODUCTIONS
Name, role, state
Brief description of where your State is in implementing NTGS strategies, highlighting immediate and long-term challenges the state is facing in implementing NTGS strategies
Comment on what you hope to learn or gain from today’s meeting
3
AGENDA
9:00-9:30 Welcome and Overview
9:30-10:00 Presentation on NTGS Strategies
10:00-11:15 Lessons from Delaware’s Non-Tested Grades and Subject Strategy
11:15-12:15 Lessons Learned from North Carolina’s Non-Tested Grades and Subjects Strategy
12:15-1:00 Lunch
1:00-2:00 Lessons Learned from Tennessee’s Non-Tested Grades and Subjects Strategy
2:00-2:30 Three State Reflection
2:30-3:30 Systems for Engagement and Iteration
3:30-4:00 Closing
4
MEETING OBJECTIVES States will learn about and reflect upon important recent
developments in measuring student growth and improving rigor
and accuracy of measurement tools in non-tested grades and
subjects (NTGS) by looking at promising practices and lessons
learned from the implementation of NTGS strategies in three
leading states.
States will discuss past efforts and upcoming opportunities to
engage educators in developing measures of student learning in
NTGS in order to enhance educator engagement in NTGS
initiatives.
State will capture knowledge and reflections to further the
collective knowledge on NTGS within the community of practice.
5
PRESENTATION ON NTGS STRATEGY
Objective: States will consider key decisions in developing an effective NTGS strategy.
Guiding Questions:
What NTGS work is currently underway by states in the
CoP and in the RSN?
How much responsibility for NTGS strategy-setting and
execution will states release to districts and or schools?
What options do states have to define or change what is
considered “non-tested grades and subjects”?
7
DEFINING KEY TERMS
Race to the Top made the distinction between “tested grades and subjects” and “non-tested grades and subjects,” and set a definition for student growth:
Tested grades and subjects – Defined as those covered by “the state’s assessment under the ESEA”
Non-tested grades and subjects – Defined as those without such data
Student growth – Requires “individual student achievement data from two or more points in time”
When considered together, these
definitions typically limit the
tested grades and subjects to
Grade 4-10 in the subjects of
English language arts and
mathematics
Source: Reform Support Network, Measuring Student Growth for Teachers in Non-Tested Grades and Subjects: A Primer
9
DEFINING EDUCATORS IN NON-TESTED GRADES AND SUBJECTS
Most states have adequate assessment data for approximately 25-35 percent of teachers, leaving 65-75 percent of all teachers without
adequate information to calculate a value-added score (Goe, 2010)
Source: National Comprehensive Center for Teacher Quality, Measuring Teachers’ Contributions to Student Learning Growth for Nontested Grades and Subjects
10
DEFINING EDUCATORS IN NON-TESTED GRADES AND SUBJECTS: TENNESSEE
Tested 36% Non-
Tested 64%
Source: Tennessee Department of Education, First to the Top, Teacher
Evaluation in Tennessee: A Report on Year 1 Implementation
Non-Tested Area Percent # of Teachers
Early Grades (PK-3) 27.05% 14,814
Special Education 10.25% 5,616
High School Core 6.03% 3,303
Fine Arts 5.44% 2,982
Career & Technical Education 5.07% 2,777
Health-Wellness/PE 4.89% 2,677
Library Media Specialists 2.40% 1,312
World Languages 1.49% 817
English Language Learners (ELL)
1.07% 588
11
DEFINING EDUCATORS IN NON-TESTED GRADES AND SUBJECTS: LOUISIANA
37%
63% Tested Teachers
Non-tested Teachers
With the current value-added assessment model
Source: Louisiana Department of Education, Non-tested Grades and Subjects: An Overview of National Efforts to Quantify Teacher Effectiveness
12
DEFINING EDUCATORS IN NON-TESTED GRADES AND SUBJECTS: LOUISIANA
Current Assessments
• LEAP and iLEAP
• 3rd through 8th grade core
• Math
• English / Reading
• Science
• Social Studies
• High School EOC
Teachers with Tested Subjects
37%
Source: Louisiana Department of Education, Non-tested Grades and Subjects: An Overview of National Efforts to Quantify Teacher Effectiveness
13
DEFINING EDUCATORS IN NON-TESTED GRADES AND SUBJECTS: LOUISIANA
Elementary Core, Grades 1 and 2
•Expand LEAP and iLEAP coverage •Math •English / Reading •Science •Social Studies
Teachers with Tested Subjects
16%
Total Coverage 53% Source: Louisiana Department of Education, Non-tested Grades and Subjects: An Overview of National Efforts to Quantify Teacher Effectiveness
14
DEFINING EDUCATORS IN NON-TESTED GRADES AND SUBJECTS: LOUISIANA
General Electives, Vo-tech, and ROTC
• Vocational Technology may require more performance assessments
Teachers with Tested Subjects
14%
Total Coverage 66% Source: Louisiana Department of Education, Non-tested Grades and Subjects: An Overview of National Efforts to Quantify Teacher Effectiveness
15
DEFINING EDUCATORS IN NON-TESTED GRADES AND SUBJECTS: LOUISIANA
Health and Physical Education
•Includes all grades, K-12 •Current assessments would likely include a combination of performance and written tests
Teachers with Tested Subjects
Source: Louisiana Department of Education, Non-tested Grades and Subjects: An Overview of National Efforts to Quantify Teacher Effectiveness
8%
Total Coverage 75%
16
DEFINING EDUCATORS IN NON-TESTED GRADES AND SUBJECTS: LOUISIANA
Art, Band, Choir, and Dance classes
•Includes all grades, K-12 •Current assessments would likely include a combination of performance and written tests
Teachers with Tested Subjects
6%
Total Coverage 81% Source: Louisiana Department of Education, Non-tested Grades and Subjects: An Overview of National Efforts to Quantify Teacher Effectiveness
17
DEFINING EDUCATORS IN NON-TESTED GRADES AND SUBJECTS: LOUISIANA
Pre-K and Kindergarten
• Age and literacy of Pre-K students may preclude use of written or computer based assessments
Teachers with Tested Subjects
4%
Total Coverage 85% Source: Louisiana Department of Education, Non-tested Grades and Subjects: An Overview of National Efforts to Quantify Teacher Effectiveness
18
DEFINING EDUCATORS IN NON-TESTED GRADES AND SUBJECTS: LOUISIANA
With the addition of 240 new assessments
Other Misc. Courses
• Mostly advanced high school electives in Math, Science, and Social Studies • Spanish, French, and other Romance languages • ESL assessments
Teachers with Tested Subjects
12%
Total Coverage 97%
With an additional 240 summative assessments, nearly 100% of
teachers are covered. But what are the trade-offs?
Source: Louisiana Department of Education, Non-tested Grades and Subjects: An Overview of National Efforts to Quantify Teacher Effectiveness 19
DEFINING EDUCATORS IN NON-TESTED GRADES AND SUBJECTS: LOUISIANA
Remaining Untested even after the addition of 240 new assessments
577 Courses
• 325 less common vocational courses
• 37 less common performance based
• 22 less common foreign languages (e.g. Hungarian, Japanese, Russian)
3% of Teachers
Even with the 240 additional assessments, LA still has 3% of its teachers in NTGS. Should they could
consider more assessments? Alternative measures? Other?
Source: Louisiana Department of Education, Non-tested Grades and Subjects: An Overview of National Efforts to Quantify Teacher Effectiveness
2%
20
MEASURING GROWTH IN NON-TESTED GRADES AND SUBJECTS: FEDERAL AND STATE PRIORITIES
Rigorous
• Measures require that students achieve high expectations for student progress toward college and career readiness
Between two or more points in time
• Approach allows for the measurement of student progress over two or more points in time
Comparable across classrooms
• Measures are comparable insofar as they all predict progress toward standard in the subject being assessed
• The measures used in non-tested subjects and grades are as rigorous as those in tested subjects and grades
Sources: Reform Support Network, Measuring Student Growth for Teachers in Non-Tested Grades and Subjects: A Primer; National Comprehensive Center for Teacher Quality, Measuring Teachers’ Contributions to Student Learning Growth for Nontested Grades and Subjects
22
APPROACHES TO NON-TESTED GRADES AND SUBJECTS
Measures of Collective
Performance
The use of measures required by ESEA
and/or other standardized
assessments used to measure the collective
performance of groups of teachers
Student Learning Objectives (SLOs)
A participatory method of setting measurable goals, or objectives,
based on specific assignment or class, such
as the students taught, the subject matter
taught, the baseline performance of students, and the measurable gain in student performance
during the course of instruction
Other Measures of Student
Growth
The development and or adaption of other measures of student
growth for non-tested grades and subjects
used across schools or districts
Source: Reform Support Network, Measuring Student Growth for Teachers in Non-Tested Grades and Subjects: A Primer 23
MEASURES OF COLLECTIVE PERFORMANCE
• Measures may assess the performance of the school, grade level, instructional department, teams or other groups of teachers
• Measures can take a variety of forms including:
School-wide student growth measures
Team-based collaborative achievement projects
Shared value-added scores for co-teaching situations
State and District Examples
Tennessee* District of Columbia Charlotte-
Mecklenburg, NC Maryland
*TN is using school-wide TVAAS, but is phasing in additional measures for educators in NTGS
Sources: Reform Support Network, Measuring Student Growth for Teachers in Non-Tested Grades and Subjects: A Primer; National Comprehensive Center for Teacher Quality, Measuring Teachers’ Contributions to Student Learning Growth for Nontested Grades and Subjects; RSN analysis
24
MEASURES OF COLLECTIVE PERFORMANCE
Advantages
• Address the variety of teaching assignments by using agreed upon measures for which schools or groups of teachers share responsibility
• Build collective school-wide or team-based effort around student achievement
Implementation Challenges
• Measures of collective performance mask high and low performers in the group and give little information about how individual teachers are doing with their classrooms
• May be perceived as unfair because teachers are held to a measure which they may have had limited ability to impact
Are there other advantages or implementation
challenges associated with this approach?
25
STUDENT LEARNING OBJECTIVES (SLOS)
• Can be based on state summative assessments, but they may also be based on teacher-developed or other classroom assessments if they are “rigorous and comparable across classrooms”
• General method draws on both effective pedagogical practices and approaches to goal setting, and evaluation and task motivation found in multiple professions
• In some instances, SLOs are shared by a team of job-alike teachers
State and District Examples
Charlotte-Mecklenburg,
NC* Denver, CO Louisiana
Rhode Island Maryland* New York
*Mentioned previously; using multiple approaches to measure student growth for teachers in NTGS
Sources: Reform Support Network, Measuring Student Growth for Teachers in Non-Tested Grades and Subjects: A Primer; National Comprehensive Center for Teacher Quality, Measuring Teachers’ Contributions to Student Learning Growth for Nontested Grades and Subjects; RSN analysis
26
STUDENT LEARNING OBJECTIVES (SLOS)
Advantages • Adaptable to the wide variety of
teaching assessments
• Can be adapted to new assessment structures as they are developed
• Can have credibility with current educators because they are immediately relevant to setting and measuring classroom expectations
• Can have face validity as teacher often tasked with developing the objectives
• Permit individual incentives, but can also be used in conjunction with measures of school or group performance to create collective incentives
• Permit high degrees of specialization for teachers and students
Implementation Challenges • Difficult to create comparability and
rigor without common assessments, or common requirements for assessment
• Predictive validation of SLOs in alignment with growth measured by a value-added or student growth measure has been completed only on a limited scale
• Requires significant time and attention from administrators and evaluators
Are there other advantages or implementation
challenges associated with this approach?
27
OTHER MEASURES OF STUDENT GROWTH
• May be developed at either the SEA or LEA level
• Also possible to include teacher-developed assessments of student learning or growth falling into this category when those assessments meet expectations for rigor and comparability across classrooms in a district or across classrooms statewide
State and District Examples
Delaware Hillsborough, FL
Florida New York*
North Carolina Illinois
Harrison County, CO
*Mentioned previously; using multiple approaches to measure student growth for teachers in NTGS
Sources: Reform Support Network, Measuring Student Growth for Teachers in Non-Tested Grades and Subjects: A Primer; National Comprehensive Center for Teacher Quality, Measuring Teachers’ Contributions to Student Learning Growth for Nontested Grades and Subjects; RSN analysis 28
OTHER MEASURES OF STUDENT GROWTH Advantages • Create comparability within tested fields
of study, and can create similar rigor through multiple classrooms and schools
• A relatively small number can be developed for most commonly taken courses (e.g., graduation required courses)
• Permit individual incentives but can also be used in conjunction with measures of school or group performance to create collective incentives
• May be ready to be adapted in some fields, such as current interim or benchmark assessments, and AP or IB assessments
• Can increase teacher buy-in and professional growth as they can play a critical role in developing tests
Implementation Challenges
• Assessments will not cover all teaching assignments or courses taken by students
• Current assessments – such as current benchmark or interim assessments, AP or IB assessments – may be designed for purposes other than assessing student growth and/or as a measure of teacher effectiveness
• Assessments may require time for teachers to work together to develop consistent scoring patterns
• Requires attention to ensuring comparability across classrooms
Are there other advantages or implementation challenges
associated with this approach? 29
QUICK TIPS FOR SELECTING MEASURES BASED ON AN EXAMINATION OF STRATEGIES USED STATES AND DISTRICTS ACROSS THE COMMUNITY OF PRACTICE
Make sure that the state has grade level and subject CCR standards for all subjects
Consider using existing tests that are already available and appropriate for this purpose
Explore alternative measures (e.g., portfolios, performances, products, and projects)
Consider using multiple approaches to measure student growth (e.g., school-wide VAM and SLOs)
Determine whether they provide useful information to differentiate teacher effectiveness
Focus on measures that meet federal and state requirements and priorities (e.g., shows growth between two points in time, comparable across classrooms, rigorous)
Involve teachers and administrators in decision-making processes 30
COMMON CHALLENGES Common Challenges Examples
Selecting appropriate measures
Quality, rigor, consistency across classrooms in a district or classrooms statewide
Data quality Attribution and student-teacher links; roster verification
Financial resources Costs of developing and administering additional assessments
Human capacity Staff needed to develop and implement assessments now and over time; sufficient to ensure fidelity of implementation
Professional development
Training everyone involved in using rubrics (e.g., for grading student portfolios or performances) to ensure reliability
System capacity System to support standardized collection of data
State and district priorities
Statutes, regulations or other factors that would affect the design or implementation of assessments
What other challenges has your state faced while implementing your NTGS strategy?
32
CONSIDERATIONS FOR STATES: MOVING FORWARD
Collaborative effort: Encourage districts to join forces with other districts or regional groups to determine appropriate measures for non-tested grades and subjects. This approach also contributes to greater comparability because teachers will be using the same measures across schools, districts and regions.
Example: Throughout the year, the TDOE has worked with educator groups across the state in non-tested grades to identify and develop additional growth measures directly tied to the students of these educators.
Example: In Georgia, districts select assessments and set SLOs for each grade/subject, which helps increase comparability across schools within districts. However, the state has developed a SharePoint site where districts can share SLOs with other districts so that they can collaborate and ensure their SLOs are as rigorous as other districts.
33
CONSIDERATIONS FOR STATES: MOVING FORWARD Stakeholder engagement: Work with peers and technical experts to design a plan for engaging stakeholders (e.g., teachers, administrators, school board members, union representatives, business leaders) in the discussions about measures. Be transparent by documenting every step toward decisions related to selection of measures so that decisions can be tracked, explained and communicated publicly.
Example: The Illinois Department of reached out to two unions – the Illinois Education Association and the Illinois Federation of Teachers – to involve leaders in early discussions around the evaluation system, well before decisions had been made. Working in collaboration with the state board, the Illinois Education Association chose to present the new evaluation system in terms of student learning. They talked with teachers about how the new system would help them identify what was going on in their classrooms and see whether student learning was occurring at the levels it should be.
34
CONSIDERATIONS FOR STATES: MOVING FORWARD
Educator Engagement in Non-Tested Grades and Subjects
I know I apply I
participate I lead
NTGS development is also an opportunity to engage and deeply involve educators in the evaluation reform process by seeking their input and
expertise in the design of new assessments
How are educators being engaged in your state’s NTGS strategy?
35
CONSIDERATIONS FOR STATES: MOVING FORWARD Feedback loops: States need multiple feedback loops across a large and complex field. Feedback loops are strategies for evaluating the effectiveness of educator engagement approaches (e.g., surveys, focus groups, systematically looking for changes in practice). System leaders should use information from feedback loops to inform the continuous improvement of their engagement activities in the same way that we expect teachers to use feedback and student performance information to make adjustments to their classroom practice.
Example: Several states in the RSN have developed or are in the process of developing different types of feedback loops ranging including analysis of NTGS data to drive process enhancements.
36
CONSIDERATIONS FOR STATES: MOVING FORWARD Communications: Use multiple vehicles to communicate effectively and consistently with key audiences. Peer-to-peer communications is most effective channel for educators, but also hardest to do well and at scale.
• Example: Posting “frequently asked questions” pages on websites (LA) and providing training to educators statewide (DE) as well as creating a hotline for educator concerns (TN).
37
Tools
Outreach Efforts
Messaging and Opinion
Research
Newsletters and Updates
Outreach and Issue
Briefs
Social Media
Websites
CONSIDERATIONS FOR STATES: MOVING FORWARD Quality control: Some states are requiring districts to submit their plans and methodology for developing growth measures for NTGS. By having a vetting process or state audit in place, states can help ensure that districts make good faith efforts to measure teacher performance in NTGS in a fair and reliable manner.
Example: Some states (RI) use exemplars in their training process to establish a clear standard of quality. State and districts are also establishing audit processes for NTGS measures to ensure consistent quality and rigor.
38
DISCUSSION
• How much responsibility for NTGS strategy-setting and execution will/should states release to districts and or schools?
39
RSN TA-RELATED WORK
40
SLO Guide
• Targeting Growth: Using Student Learning Objectives as a Measure of Educator Effectiveness
Educator Engagement Guide
• Engaging Educators: A Reform Support Network Guide for States and Districts
SLO Work Group
• Work group for states implementing SLOs as a part of their evaluation systems, special focus on quality and on improving instructional practice
Quality Evaluation Rollout Work Group
• Work group for states fully implementing evaluation systems by 2012-13
LESSONS FROM DELAWARE’S NON-TESTED GRADES AND SUBJECT
STRATEGY
Facilitator: Elizabeth Shaw, RSN
Presenters: Dr. Linda Rogers, Associate Secretary, and
Diane Donohue, Special Assistant for Educator Effectiveness
41
LESSONS FROM DELAWARE’S NON-TESTED GRADES AND SUBJECT STRATEGY Objective: Participants will learn from Delaware’s rigorous methods to assess student performance measures being used for teacher evaluation decisions and draw applicable lessons to systematically increase rigor of assessments in NTGS.
Guiding Questions:
How can states build internal systems to assess and increase rigor of
measurements?
How can states and districts engage educators in these feedback
loops, both through communication and active involvement of
educators?
What are the greatest challenges in building systems of
measurement in non-tested grades and subjects that maintain high
levels of rigor and comparability across schools and districts?
42
BACKGROUND
• DPAS II – Charlotte Danielson’s Framework for Teaching
Senate Bill 260 – required a fifth component solely dedicated to “Student Improvement”
• DPAS II(R) – January 2010
Race to the Top
New Regulation required:
Multiple Measures
Rigorous and Comparable across classrooms, schools, district or state
Component Five acts as a gate-keeper
43
EDUCATOR INVOLVEMENT Cohort 1 Cohort 2 Cohort 3 Cohort 4
ELA/Reading ESL Family & Consumer Sciences
Counselors
Math Health Education Business, Finance & Marketing
Librarians
Science Physical Education Technology Education Educational Diagnosticians
Social Studies Visual Arts Health Sciences Physical & Occupational Therapists
World Languages Performing Arts Agriscience Psychologists
Skilled & Technical Sciences
Speech/Language Pathologists
Driver’s Education Social Workers
Visiting Teachers
Special Education – DCAS Alt
Nurses
Early Childhood
44
DPAS II(R) COMPONENT FIVE
• What’s New? Three Educator Groups Group I – DCAS educators who teacher reading
and/or math in grades three (3) through ten (10)
Group II – Any other educator who teaches at any grade level or subject other than DCAS reading and/or math
Group III – Any educator who does not meet the criteria for Group I or Group II will defer to Group III
45
DPAS II(R) COMPONENT FIVE
• What’s New? Three Measures
Measure A – Based upon DCAS instructional scale scores for reading and/or math in grades three (3) through ten (10)
Measure B – External and Internal Assessments
External – DDOE approved, standardized assessments (DIBELS, STAR Reading, STAR Math)
Internal – DDOE approved, educator developed pre/post student assessments specific to subject and grade level
Measure C – Growth Goals
DDOE approved, educator developed goals specific to content area and/or job assignment
46
HOW DO WE MATCH THE EDUCATOR GROUPS TO THE MEASURES?
Measure A Measure B Measure C
Group I Must use a minimum of two (2) Measures
50% Must use Measure A
50% Must use at least (1) Measure B
Group II Must use a minimum of four (4) Measures
50% Must use at least (1) Measure B
50% Must use at least (1) Measure C
Group III Must use a minimum of four (4) Measures
100% Must use at least four (4) Measure Cs
47
HOW WILL MEASURE A BE RATED?
Exceeds Satisfactory Unsatisfactory (with administrator discretion)
Unsatisfactory
65% or more of a teacher’s DCAS student growth targets are met
50-64% of a teacher’s DCAS student growth targets are met
35-49% of a teacher’s DCAS student growth targets are met (conference between administrator and teacher could provide option to upgrade to a “Satisfactory” rating
Less than 35% of a teacher’s DCAS student growth targets are met
48
HOW WILL MEASURE B AND C BE RATED?
Exceeds Satisfactory Unsatisfactory
The agreed upon “exceeds” target is met or surpassed.
The agreed upon “satisfactory” target is met or surpassed, but the “exceeds” target is not met.
The agreed upon “satisfactory” target is not met.
49
NEXT STEPS
• All training materials from August sessions posted to the DOE website
• Additional county-wide “team” training scheduled for September
• Online training modules being finalized
• Updated teacher, specialist and administrator guides being finalized
• SunGard – online training of Performance Plus
50
CHALLENGES
• Process to audit documents uploaded to Performance Plus to ensure accuracy
• Process to provide a feedback loop for districts/schools with implementation for the 2012-13 school year
• Process to handle questions/concerns related to implementation
• TIME…
51
LESSONS LEARNED FROM NORTH CAROLINA’S NON-TESTED GRADES AND
SUBJECTS STRATEGY
Facilitator: Elizabeth Shaw, RSN
Presenter: Jenn Preston, Race to the Top Coordinator for Teacher and Leader Effectiveness, NC Department of Public
Instruction
52
LESSONS LEARNED FROM NORTH CAROLINA’S NON-TESTED GRADES AND SUBJECTS STRATEGY Objective: Participants will understand how NC has approached NTGS through a statewide assessment design strategy, and will also draw applicable lessons from NC’s work to engage educators in that process.
Guiding Questions:
What prerequisite capacity, expertise, educator investment or
political will is necessary to design statewide measures of student
learning?
How can states draw from the capacity of educators in the field to
increase the quality of their measures?
What is the ongoing cost of this strategy? How has NC planned for
this cost?
What types of processes and systems can be used to improve the
quality of measurements over time?
53
PRESENTERS
Dr. Rebecca Garland Chief Academic Officer [email protected] (919) 807-3305
Jennifer Preston Race to the Top Coordinator for Educator Effectiveness [email protected] (919) 807-4187
[email protected] http://www.ncpublicschools.org/educatoreffect/
55
MEASURES OF STUDENT LEARNING
• Common summative assessments used across LEAs and Race to the Top-funded charter schools
• Design process includes over 800 educators in two steps:
Blueprint creation
Item reviews
• Thirty MSLs implemented during 2012 – 2013 in grades 4 – 12 social studies, science, ELA, and mathematics
56
PROMISING PRACTICES
• Deep engagement of educators, not only as part of the design process but as key communicators
• LEA flexibility and ownership of administration
• Partnership with higher education and “trendsetter” LEAs
• Use of regional infrastructure for support and information
57
MAJOR CHALLENGES
• Staffing and funding constraints (state-and LEA-level)
• Shifting to new standards at the same time as new assessments
• Perception of expanded “bubble tests” in all content areas and grades
• Data quality and integration
• Grounding work in better outcomes for students
58
EDUCATOR ENGAGEMENT
I know
Design group members received training on the Common Core State Standards and NC Essential Standards, as well as the overall educator effectiveness model
I apply
Design group members provided feedback on how measure student learning in their grades and content areas
I participate
Design group members used assessment design principles and understanding of standards to create blueprints for MSLs
I lead
Design group members are training colleagues on the MSLs and how they fit into educator effectiveness model
59
CONTINUOUS IMPROVEMENT
• Formative feedback from Race to the Top evaluation team
• Meetings with LEA “Educator Effectiveness” teams
• Statistical analysis of scores
Distribution studies
Comparison of scores from different sections
Value-added modeling
60
LESSONS LEARNED FROM TENNESSEE’S NON-TESTED GRADES AND SUBJECTS
STRATEGY Facilitator: Elizabeth Shaw, RSN
Presenters: Sara Heyburn, Assistant Commissioner, Teachers and Leaders
and Dru Davidson, Chair of Arts Education, Memphis City Schools
61
LESSONS LEARNED FROM TENNESSEE’S NON-TESTED GRADES AND SUBJECTS STRATEGY
Objective: Participants will understand and draw applicable lessons from TN’s portfolio-based assessment pilot in fine arts, as well as from TN’s teacher involvement in assessment design.
Guiding Questions:
How did the portfolio-based assessment in fine and performing arts develop through
state and district collaboration? What conditions allowed this to happen?
How is TN grappling with the issue of scale as the portfolio-based assessments are
implemented statewide this year? What has the state learned about working with
partners to access talent and feedback during scaling?
How does the state balance the need to maintain high level of quality for NTGS
implementation while allowing for varying levels of interest and commitment at
individual districts?
What data has TN collected on its measures and how has that informed their strategy?
What strategies has TN used to engage educators in assessment development?
Where has the state been more or less successful at engaging educators?
62
SUMMARY OF PROGRAM EFFORTS TO DATE
• Vision/Framing for work:
Focus on measures that will help students and teachers
Desire to keep tight on criteria, but loose on district usage decisions
Engage non-tested educators throughout the process
For some subjects outside the three R’s, utilizing a portfolio approach
• Process: TDOE determined timeline and steps; several phases of work
63
SUMMARY OF PROGRAM EFFORTS TO DATE
• Promising Practices:
Fine Arts Portfolio
Educator Engagement
• Challenges:
Ensuring portfolio models truly measure growth (vs. achievement)
Ensuring the right people are at the table
Resources for implementation
64
HOW ARE EDUCATORS ENGAGED IN DEVELOPING NTGS MEASURES?
I know
•Educators can articulate which non-tested group they fall into and the measures available to them.
•Ex: A high school P.E. teacher can articulate that he/she will receive a school-wide value added score.
I apply
• Educators can use developed non-tested measures to drive student learning and professional development.
• Ex: Music teachers will use the fine arts portfolio to identify areas of reinforcement and refinement in their teaching.
I participate
•Educators work on educator group teams to develop NTGS measures.
•Ex: Educators in pre-K and K have gathered to develop a portfolio model to measure growth in the early grades.
I lead
• Educators drive the creation of NTGS measures in their subject area or field.
• Ex: Dru Davison driving forward the creation of the Fine Arts portfolio.
65
BUILDING SYSTEMS TO IMPROVE MEASUREMENT OVER TIME
• In approving measures, standards for evaluating teachers on portfolios need to be aligned to college and career ready standards. Educator groups will continue to meet periodically once a
measure has been approved to ensure alignment.
• Data will be collected for all measures that are approved to assess predictive power and psychometric robustness. Any approved measure that yields a value-added score
will go through the same validity and reliability analysis as state assessments.
66
CURRENT STATE OF NTGS MEASURES IN TN NTGS Group 2012-13 Status
Fine Arts Portfolio measure approved for use
Grades 1-3 Use of Stanford 10 to create value-added; RFP for state assessment
Pre-K and K Portfolio measure in development
CTE Inclusion of CTE-concentrator school-wide score
P.E./Health Portfolio measure in development
HS Social Studies RFP for government assessment; assessing validity of portfolio aligned to CCSS
Special Education Pursuing the removal of statute prohibiting use of test scores from students with disabilities for individual teacher effect scores
HS Science Assessing validity of portfolio aligned to CCSS
World Languages Assessing validity of using STAMP to create value-added scores
67
FOCUS ON FINE ARTS • 7 step process for approving a growth measure:
1. Educator group is formed.
2. Educator group identifies appropriate college and career ready standards in subject area.
3. Educator group identifies and/or develops the measure according to state criteria.
4. Measure is reviewed by various stakeholders.
5. State reviews with input from technical advisors.
6. If conditionally approved, measure goes through a pilot.
7. Measure is approved for statewide use.
68
LESSONS LEARNED • Engaging educators is a must.
• Growth doesn’t have to mean value-added. However,
It is important to hold a high bar for what are approved as alternate measures for growth.
It is important to be as consistent as possible within and across content areas and grades.
• Don’t underestimate the effort needed to identify, develop and implement high quality measures.
It is easy to deprioritize NTGS, but it takes time, expertise and capacity to develop and implement high quality measures.
69
THREE STATE REFLECTION
Objective: Participants will discuss commonalities and differences in the three states’ strategies and reflect on implied common design principles for implementing measures in NTGS.
Guiding Questions:
What are the commonalities between the NTGS strategy
of each of the three states?
What are the differences between the NTGS strategies of
each of the three states?
What common or different design principles are emerging
based on the varied experiences of the states?
71
INDIVIDUAL REFLECTION Please take 5 minutes to engage in individual reflection and answer these two questions:
• What do states’ NTGS strategies have in common? Where do they differ?
• What are the emerging trends in these state’s approaches to NTGS? What is most notable in the differences between their approaches?
72
SYSTEMS FOR ENGAGEMENT AND ITERATION
73
Facilitator: Brad Jupp, Senior Program Advisor on Teacher Initiatives, U.S. Department of Education
SYSTEMS FOR ENGAGEMENT AND ITERATION
Objective: Participants will reflect on the central ideas from the seminar (engaging educators in NTGS strategies and building systems to improve measurement over time) and will discuss how they can use these lessons to inform their practice in their home state.
Guiding Questions:
How has the content of the seminar shaped your plans for
implementing NTGS strategies?
How does the state plan to build systems for continuous
improvement of measurement tools? What concerns or challenges
does the state face in building systems for iteration?
How does the state plan to engage educators in developing NTGS
measures? What particular concerns or challenges does the state face
in working to ensure that educators are actively engaged in system
design? 74
INDIVIDUAL REFLECTION Please take 5 minutes to engage in individual reflection on how the content of the day will shifted your state’s NTGS strategy by answering the questions below:
Based on your pre-work: What has the state learned today that will change your NTGS educator engagement strategy? Please fill out changes on the template
What specific ideas or strategies have been identified that you will take back with you? How has the content of the seminar shaped your plans for implementing NTGS strategies?
How do you plan to build systems for continuous improvement of measurement tools? What concerns or challenges do you face in building systems for iteration?
75