feasibility of using rate with teacher candidates
TRANSCRIPT
2016
ANN BULLOCK, KRISTEN CUTHRELL, ELIZABETH FOGARTY, JOY STAPLETON
EAST CAROLINA UNIVERSITYC O L L E G E O F E D U C AT I O N
D E PA R T M E N T O F E L E M E N TA RY E D U C AT I O N A N D M I D D L E G R A D E S E D U C AT I O N
Feasibility of Using RATE with Teacher Candidates
• Recommends approximately 700 initial license candidates a year
• 54% ELEM • Largest
producer of teachers in NC
• Highest employment rate in NC
Context
Importance of assessment system with multiple measures of effectiveness Our home-grown assessments found to be invalid
or unreliable measures of performance (Henry et al., 2013) Portfolio assessment Dispositions Observation Instrument
Rapid Assessment of Teacher Effectiveness
RATE is a validated observation instrument in use with practicing teachers
Draws on existing teacher effectiveness constructs
Observe classroom lesson videos; Results in less expensive teacher observation
Focuses on direct instruction RATE design has 6 items, each item is assessed
on a 3 point scale (Gargani and Strong,2014)
Methods
Proof of concept study
Research Questions
RQ 1: Does the RATE observation instrument’s reliability and validity hold when used with pre-service teachers?
RQ2: Is the RATE observation instrument a feasible instrument to be used by large teacher education programs?
RQ3: Do you have to use two raters in order for the rater scores to remain reliable and valid?
Procedures - Videos
Preservice Teachers: ELEM candidates submit full lesson videos to an
electronic portfolio system during final semester of internship
identify their subject, topic, and 20 consecutive minutes of direct instruction to be viewed
179 videos were used for scoring in this study
Procedures- Training
March 2015 Strong and Gargani held an orientation session with
38 faculty, university supervisors, instructional coaches and COE administrators
Focus was on introducing faculty to the RATE instrument
May 2015 Day 1: 38 participants received training on the RATE
instrument Day 2: participants scored videos
Procedures - Scoring
8 30 minute sessions were held on Day 2 Within each 30 minute session a participant would
score the 20 minute video individually then meet with the other person who scored the video.
Scores were compared and reconciled150/179 videos were scored twice using both
rater pairs and solo ratersAt a later date- RATE personnel evaluated all
videos for comparison purposes.
Comparing ECU Faculty independent scores to RATE independent scores
Paired Raters Arriving at Same Scores
Independent VS Reconciled Sum Scores
Conclusions
RQ 1: Discrepancies in paired scores exist but rater reliability was higher than in initial RATE validation study Biggest discrepancy: clarity and flow
RQ 2: Instrument is shorter and less complex than existing instrument However, having two individuals rate one video
seems prohibitive in a large program.
RQ 3: To maintain reliability and validity two raters are needed.
Next Steps
Pilot RATE instrument with preservice teachers in live observations (spring 2016)
Determine predictive nature of RATE with preservice teacher observation Explore relationship between RATE and CLASS
instruments Explore relationship and possible predictive nature of
RATE with edTPA
References
Cuthrell, K., Stapleton, J., Bullock, A., Lys, D., Smith, J., and Fogarty, E. (2014). Mapping the journey of reform and assessment for an elementary education teacher preparation program. Journal of Curriculum and Instruction, 8(1).
Gargani, J., & Strong, M. A. (2014). Can we identify a successful teacher better, faster, and cheaper? Evidence for innovating teacher observation systems. Journal of Teacher Education. Pre-print copies available online.
Good, T., & Lavigne, A. (2015). Response to "Rating Teachers Cheaper, Faster, and Better: Not So Fast": It's about evidence. Journal of Teacher Education, 0: 0022487115587110v1-22487115587110
Henry, G. T., Campbell, S. L., Thompson, C. L., Patriarca, L. A., Luterbach, K .J., Lys, D. B., & Covington, V. M. (2013). The predictive validity of measures of teacher candidate programs and performance: Toward an evidence-based approach to teacher preparation. Journal of Teacher Education 64, 439-453.
Sawchuk, S. (2011). University of Michigan project scales up "high leverage" teaching practices.Education Week. Retrieved from the www on August 31, 2012 at http://blogs.edweek.org/edweek/teacherbeat/2011/10/high_leverage_teaching_practices.html
Strong, M., Gargani, J., & Hacifazlioğlu, Ö. (2011). Do we know a successful teacher when we see one? Experiments in the identification of effective teacher. Journal of Teacher Education, 62(4), 367-382.
Strong, M. (2011). The Highly Qualified Teacher: What is Teacher Quality and How Do We Measure It? New York: Teachers College Press.
KRISTEN CUTHRELL: [email protected]
JOY STAPLETON: [email protected]
AERA 2016EAST CAROLINA UNIVERSITY
Feasibility of Using RATE with Teacher Candidates