TQP: A Longitudinal Study of Ohio

Teacher Education

Spring 2007 Update

OCTEOCrowne Plaza Dublin

April 5, 2007

• Higher Education Act of 1998– Title II Report on Teacher Quality

• Ohio Report Card Task Force (1999-2001)– OCTEO Focus Groups

• Political will formed between Ohio Department of Education, Ohio Board of Regents, Governor’s Office, Teacher Education Institutions (Private and Public), and School Districts

• Factors in Ohio– Standards Based Curriculum– Value-Added Measurement (VAM)– SB311 signed into law January 3, 2007

Factors and Forces

Am. Sub. S. B. No. 311 126th General Assembly

Passed by the Senate December 6, 2006 Passed by the House December 19, 2006 and Signed by Gov. Taft January 3, 2006

Senators Gardner, Padgett, Harris, Clancy, Mumper, Jacobson, Cates, Goodman, Niehaus

Representatives Webster, Blessing, Combs, Evans, D., Martin, Schlichter, Setzer

Sec. 3319.234. The teacher quality partnership, a consortium of teacher preparation programs that have been approved by the state board of education under section 3319.23 of the Revised Code, shall study the relationship of teacher performance on educator licensure assessments, as adopted by the state board under section 3319.22 of the Revised Code, to teacher effectiveness in the classroom. Not later than September 1, 2008, the partnership shall begin submitting annual data reports along with any other data on teacher effectiveness the partnership determines appropriate to the governor, the president and minority leader of the senate, the speaker and minority leader of the house of representatives, the chairpersons and ranking minority members of the standing committees of the senate and the house of representatives that consider education legislation, the superintendent of public instruction, the state board of education, the Ohio board of regents, and the partnership for continued learning.

• Confidentiality Promise Promotes Collaboration– Participants (all data coded)– Colleges and Universities

• Preservice, Demographics, Inservice data• Ohio aggregated data reported publicly• Institutions get their own data to use in program review and

accreditation reports

– Districts• Inservice data• Ohio aggregated data reported publicly• District gets their own data to use in EY, PD review

– FERPA ruling from OSU attorney

Factors and Forces

• All 50 Ohio Teacher Preparation Institutions

– Institutional Representatives (50)– Principal Investigators (UC, OSU, UD)– Research Design Teams (13 IHEs)– Field Researchers (10 IHEs – changes as needed)

• District Partners– Entry Year Coordinators– Administrators in field study districts

• Ohio Advisory Board

• External Audit Panel• Multiple federal, state, corporate, and foundation funders

Stakeholders and Participants

TQP Research Questions

1. Do variables of teacher background, initial preparation, and on-going professional learning relate to teacher practices, student learning and achievement?

2. How do specific elements of teacher preparation and aspects of school contexts impact novice teachers’ development during their first three years of teaching?

3. Do high value-adding teachers (HVATs) have characteristics, instructional practices, and understandings that differ from other teachers along the value-added continuum?

4. What specific school contexts are associated with HVA novice and experienced teachers?

Benefit to InstitutionsBenefit to Institutions• If 5 student teachers participate, a customized report is

given to each institution. It is not made public by TQP.• Both pre-service and in-service surveys are done at TQP

expense using valid and reliable instruments based on national studies.

• Patterns identified on annual reports have been useful for internal review and reform.

• An NCATE/TQP cross-walk has simplified using survey data in accreditation reports.

• Comparison of local and state, sometimes national, data.• Ultimately intend to use “best practices” findings to improve

preparation program.

A Public University’s Story• Facilitated discussion within the teacher

education unit by presenting questions to consider and a common language for the discussion.

• Assisted in responding to a need to cultivate a climate of assessment.

• Unit developed a systematic approach to the organization and display of data

• Identify trends within the teacher education unit and follow over time.

Story (con’t.)• Compare teacher candidates’ demographics and

program perceptions with those of other Ohio programs.

• Compare perceptions and demographics among and between our main and regional campuses.

• Trace and quantify changes in perceptions with regard to specific subsets within the data.

• Access to longitudinal data about candidate perceptions of individual licensure programs which inform programmatic decisions.

Challenges to InstitutionsChallenges to Institutions• Getting surveys administered• Collecting and submitting demographic datafile• Maintaining student confidentiality• Securing and meeting IRB requirements• Using the data to inform:

– Program and policy decisions– Supply/demand issues

• Tension between external regulatory bodies and internal program implementation and resources

Benefit to DistrictsBenefit to Districts

• If 5 teachers participate, a customized report is given to the district. It is not made public by TQP.

• The only available source of Praxis III data.

• Comparison of local, state and national data on induction and mentorship.

A Large Urban District’s StoryA Large Urban District’s Story• TQP Partnership agreement:

– Agreed to and signed by Superintendent and Union President, March 2005.

– Administer TQP Inservice Teacher Survey annually to all novice teachers hired Fall 2005 – Spring 2009.

– Help recruit novice teachers for three year follow-up case studies.

– Help recruit experienced teachers for one year case studies.

– Work with TQP to get teacher-level value-added scores into the TQP data set.

ExpectationsExpectations• Learn what the most effective teachers do

to promote academic progress of students in their district.

• Improve hiring practices by knowing what an effective district teacher looks like.

• Improve teacher retention through enhanced induction and professional development programs.

• Ultimately improve student learning in the district.

2005 – 2006 School Year2005 – 2006 School Year

• Informational meetings early September

• Lots of communication from administration and union encouraging teachers to attend meetings

• All novice teachers encouraged to complete survey.

• A subset of novice and experienced teachers participated in the pilot.

Challenges in Large Urban DistrictsChallenges in Large Urban Districts

• Late summer/early fall teacher hires and placements

• Concern about Praxis III and other beginning teacher issues

• Entry Year Teachers (EYTs) are first to get RIF notice. They seek jobs in other districts rather than risk not being called back.

• If re-hired, may not be in same grade/subject.• Pressure to meet assessment standards.

• Survey student teachers ‘03-’04 through ’08-’09– Anticipate 30,000+ participants– Printed or on-line surveys administered by

institutional liaisons– Expect to add value-added scores of graduates

• Follow novice teachers each year through ’08-’09– Anticipate 5,000+ participants– Printed or on-line surveys

Longitudinal GraduateLongitudinal Graduate Follow-up Study

FY 2004 FY 2005 FY 2006 FY 2007 FY 2008 FY2009 ’03-‘04 Cohort I (Pilot)

Pre-service Hard copy

N (S) =1544 N (D) = 3460

In-service HC/Web N=1051

In-service Web

N=550

In-service Web N=61

In-service Web

In-service Web

’04-‘05 Cohort II

Pre-service HC

N (S) = 4941 N (D) = 6980

In-service HC/Web N=1437

In-service Web N=48

In-service Web

In-service Web

’05-‘06 Cohort III

Pre-service HC/Web

N (S) =5229 N (D) = 6440

In-service HC/Web N=204

In-service Web

In-service Web

’06-‘07 Cohort IV

Pre-service HC/Web

N (S) = 1372 N (D) =?

In-service HC/Web

In-service Web

’07-‘08 Cohort V

Pre-service HC/Web

In-service Web

’08-‘09 Cohort VI

Pre-service HC/Web

S = Surveys Collected D = Demographic Data Submitted

Pre-service Subscales Report Example

• Sample mid-point findings from:– Demographic Database (submitted by

Institutional Representatives at each IHE)– TQP Preservice Teacher Survey (completed

during student teaching, usually in a seminar)– TQP Inservice Teacher Survey (completed

each spring 2004 – 2009 hard copy or on-line)

TQP Ohio Cohort I (2004), Cohort II (2005), and Cohort III (2006) ACT Scores

2121.4

22.53

0

5

10

15

20

25

National State Teacher Completers Total

AC

T S

core

s

Average GPA of 15,248 teacher completers = 3.48

* As High School Seniors** National Percentiles N = 17,127 teacher completers in Ohio from Academic Years 2004, 2005, and 2006.

N for Ohio TQP Teacher Completers = 17,127

TQP Ohio Academic Years 2004-2006 Demographic Data: Percentage by Ethnicity

0.322.45

0.52 0.91

93.22

0 1 20 0.2 0.4

92.6

1

7

25

85

77

17

6.7

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00

American Indian/Alaskan Black/African American Asian American Hispanic Caucasian

Per

cen

tag

e

Ohio TQP Completer Profile Ohio Student Profile (K-12) Ohio Teacher Profile National Teacher Profile

Scale: 1=Very Negative; 7=Very PositiveN=2,805-2,812 depending on the item

TQP Ohio 2005 and 2006 Entry Year Teacher's Satisfaction Level: Mean Score by Item

4.975.21

5.01

6.18

5.595.82

4.73

1.00

2.00

3.00

4.00

5.00

6.00

7.00

Salary

/fring

e be

nefits

Oppor

tunitie

s for

pro

fess

ional a

dvanc

emen

t

Leve

l of p

erso

nal/pr

ofes

siona

l cha

lleng

e

Leve

l of p

rofe

ssion

al aut

onomy/d

ecisi

on m

aking

auth

ority

Gener

al wor

k con

ditions (

hour

s, cla

ss si

ze, w

ork l

oad, e

tc.)

Inte

racti

ons w

ith co

lleag

ues

Inte

racti

ons w

ith st

udents

Me

an

34.77%

47.93%

11.34%1.16% 0.99%

0200400600800

100012001400

None Less Than 1 Hour

More than1 hour butLess or

Equal to 3hours

More than3 hours but

less orEqual to 5

Hours

More than5 hours

N=2,305

New Ohio teachers report being observed little by their mentors in a typical week

7.72

7.42

7.83

7.51

6.98

7.61

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

Efficacy for classroom management Efficacy for student engagement Efficacy for instructional strategies

Su

bsc

ale

Mea

n

Preservice Inservice

Efficacy for Preservice (Cohorts I, II, III) and Inservice (Cohorts I, II)Efficacy for Preservice (Cohorts I, II, III) and Inservice (Cohorts I, II)

3.98 4.02

4.654.58

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

5.50

6.00

Preservice Inservice

Su

bsc

ale

Mea

n

General teacher efficacy Personal teacher efficacy

Efficacy for Preservice (Cohorts I, II, III) and Efficacy for Preservice (Cohorts I, II, III) and Inservice (Cohorts I, II)Inservice (Cohorts I, II)

TQP Preservice Cohorts I, II, III: Concerns Subscale Means (N=11,200)

2.62

2.31

3.00

1.00

1.50

2.00

2.50

3.00

3.50

4.00

Self Concerns Task Concerns Impact Concerns

Co

nce

rn S

ub

scal

e M

ean

Notes on the Data– Self-report, exams, interviews, documents, VAM– Reported for items and sub-scales– Surveys are too long, but…

• What to cut?• Sub-scale reliability consistent throughout

– Interpretation:• Implicit standard (4 on 1-5 scale)• Relative score compared to other items, sub-scales• Confidence intervals

– Ultimately, findings will be based on Structural Equation Modeling related to VAM

• Challenges of Large-Scale Collaboration– Ownership, Turf, and Trust– Data Collection on Main and Regional Campuses– Multiple Data Bases that Don’t Connect Well– Communication Both Internal and External– Resources vs. Costs– Setting Priorities– Follow-through on Tasks– Recruitment of Participants– Maintaining Political Will

Sustaining the Partnerships

– Ability to share and use data– Common language used statewide– Some issues transcend local context; other issues

vary according to resources and other local decisions– Respond to emerging needs

• Revise protocol to collect needed data• Work with ODE/OBR/ESB to develop legislated reports• Enable on-line surveys, submission and retrival of data

– Communication through email, meetings, newsletters

What questions or comments do you have ?What questions or comments do you have ?

For Additional Information:For Additional Information:http://www.tqpohio.org