jack b. monpas-huber, ph.d. director of assessment and program evaluation spokane public schools

High School Teachers’ Instructional Use of WASL Data:

Exploring the Role of School Culture and Motivation

Jack B. Monpas-Huber, Ph.D.Director of Assessment and Program Evaluation

Spokane Public Schools

2

Who I Am

Master of Science, Sociology, 1997

Ph.D., Educational Psychology

Director of Assessment and Program Evaluation

3

Acknowledgments

My Committee Catherine Taylor, Michael Knapp, Susan Nolen, Paul LePore

WERA Board

Northshore School District

Bob Hamilton, Dennis Milliken, Carolyn O’Keeffe, Michele Williams, Diane Baerwald

Assessment Directors

Pat Cummings, Bob Silverman, Nancy Katims, Dan Phelan, Michael Power, Don Schmitz, Dennis Milliken, Peter Hendrickson, Linda Elman

Spokane Public Schools

Brian Benzel, Nancy Stowell, Irene Gonzales, Lorna Spear, Tammy Campbell, Emmett Arndt, Susie Lynch, Dary Van Dusen, Anne Berman

My Family Kimberlee and John Henry Monpas-Huber

4

Background of the Project

Work experience: Assessment department of large school district Providing data to schools to support DBDM Dealing with school cultures, politics, leadershipResearch interests: Sociology of education / school organization Motivation Measurement, statistics & research design Validating large-scale accountability systems

5

Organization of this Presentation

Framing the Problem Teachers’ Use of State Assessment Data Research Methods Results of the Study Discussion

6

Framing the Problem Rise of state accountability programs High stakes attached to student performance Data fed back to schools for “data-based

decisionmaking” “Theory of Action” research (Fuhrman, 2004)

Two functions: Accountability Instructional/feedback

How do the two forces shape teachers’ instructional use of data?

7

Research Questions

1. Considering how much data the state provides to educators, how much are high school teachers using state assessment data as a resource to improve instruction? How useful do they find it?

2. Considering the mounting policy pressures to improve performance on the state assessment, what motivates teachers to use state assessment data? What is the influence of policy pressure specifically, and aspects of school context in general?

8

Limits in Focus State Assessment data Instructional decisions Certificated teachers High schools

9



10

Teachers’ Use of State Assessment DataRelevant Literatures

Accountability Systems Data-based Decision-making Teacher Motivation Accountability and High Schools

11

Teachers’ Use of State Assessment DataCapacity for Teacher Data Use

Technical Skills Technical skills for working with data Databases, software Analysis and interpretation of systematically

collected data Capacity building efforts in Washington Hypothesis:

Exposure to training in assessment or in WASL item development should be a strong predictor of use of WASL data

12

Teachers’ Use of State Assessment DataCapacity for Teacher Data Use

Access to Data Advances in computer technology Assessment personnel Teachers may vary in their perception of access to

data Hypothesis:

Teachers who perceive more access to data should be more likely to use such data than teachers who perceive less access to data

Access as necessary but not sufficient condition

13

Teachers’ Use of State Assessment DataTeacher Motivation and Data Use

The Policy Perspective Some instructional changes are difficult Teachers need consequences Behavioral perspective on motivation Research on high stakes testing Research issue: perceived pressure as both

outcome and predictor Hypothesis:

Teachers who perceive higher levels of pressure will be more likely to use assessment data than teachers who perceive lower levels of pressure

14


Alternative Perspectives Cognitive perspectives on motivation Motivation stems from

mind/thought/interpretation Social context and cognition

15


Expectancy “Teacher’s perceived probability that the teacher’s effort will

result in the attainment of the goals” (Kelley, Heneman, & Milanowski, 2002, p. 378)

“Will do” of motivation; efforts will result in positive outcomes

Kentucky and North Carolina research Hypothesis:

Teachers who report higher levels of expectancy (that working with assessment data will actually help them improve instruction for their students) will be more likely to use assessment data than those who expect less to result from it

16


Efficacy “Teacher beliefs about their span of influence and

performance capacity” (Kelley & Finnigan, 2003, p. 604) “Can do” of motivation The role of performance feedback in motivation

research Hypothesis:

Teachers who feel more efficacious working with assessment data will be more likely to use assessment data than teachers who feel less efficacious

17


Goals Motivation as product of intentions or goals people have

for engaging in a behavior People pursue variety of goals Goals may conflict with each other Teachers and “perceived policy intentions” of

accountability policies (Leithwood, Steinbach, & Jantzi, 2002) Ingram, Louis, & Schroeder (2004) study Hypothesis:

Teachers will be more likely to use state assessment data if they perceive its underlying purpose as consistent with their own goal of helping students learn

18

Teachers’ Use of State Assessment DataSummary of Motivation Research

Pressure, expectancy, efficacy, goals Filtered through school context Some aspects of context (collaboration, feedback

data) influence motivations These motivations vary:

Among teachers within one school Possibly by groups of teachers between schools

Research issues Motivations as predictors of data use Motivational effects may be different in different

schools

19

Teachers’ Use of State Assessment DataBuilding a Model of Teacher Data Use

Quantity of Teacher Data Use =β0 (mean)+ β1(state assessment training)+ β2(perceived access to data)+ β3(perceived pressure)+ β4(expectancy)+ β5(efficacy)+ β6(goal alignment)+ e (unmodeled variation)

20

Teachers’ Use of State Assessment DataContextual Influences on Teacher Motivation and Data Use

To the extent that motivations are shared by teachers in one school, what influences these motivations?

Are some schools more “motivating” than others, in this case, in regard to using and learning from state assessment data?

Sociological perspectives on school culture and other contextual influence

21

Teachers’ Use of State Assessment DataContextual Influences: Culture

Focus on shared attitudes and behavior is focus on culture

Two perspectives on culture (Swidler, 1995): “Inside out” – internalized attitudes (motivations?)

predict behavior “Outside in” – shared practice, norms, codes

regulate behavior irrespective of internal beliefs

22

Teachers’ Use of State Assessment DataCultural Perspectives

The Loose Coupling Perspective (Weick, 1976; Firestone, 1985) Educational organizations are multi-layered Classrooms disconnected from administration Because teaching and learning is not precise, schools do not evaluate

technical quality of instruction High schools especially loosely coupled Challenges bureaucratic models of schools which emphasize centrality

of leadership and formal rational procedures Also helps explain why reform movements have historically failed to

change instruction in schools Loose coupling and assessment data

“Stick them in a drawer”

23

Teachers’ Use of State Assessment DataCultural Perspectives

Professional Accountability Abelmann & Elmore (1999) Strong and weak internal accountability systems O’Day (2004)

Professional Collaboration Student learning data as centerpiece of

collaborative work Recurrent predictor in past research

24

Teachers’ Use of State Assessment DataLeadership

Leaders filter and frame accountability policy (Spillane)

Transformational leadership has positive effects on teacher motivation

Trust, collaboration, shared accountability Principals may vary in how they frame assessment

results School-level variable that influences motivations

and assessment data

25

Teachers’ Use of State Assessment DataMethodological Observations

Lots of qualitative case studies No quantitative studies of use as a criterion or

dependent variable Lack basic descriptive data about levels or

frequencies of use

26

Teachers’ Use of State Assessment DataA Tentative Model

Level 1: Teacher

Teacher WASL Data Use = β0 + βj(Training in State Assessment) + β2(Perceived Access) + β3(Perceived Pressure) + β4(Expectancy) + β5(Efficacy) + β6(Goal Alignment) + R

Level 2: School

β0 = γ00 + γ01(Demographics) + γ02(Professional Accountability) + γ03(Professional Collaboration) + γ04(Principal Leadership) + U0

β3 = γ10 + γ11(Principal Leadership) + U1

27



28

Research MethodsDesign Issues

Teacher survey Study population: certificated teachers in high

schools in western Washington school districts that employ a full-time assessment director

Instrument: 4-page questionnaire Matrix sampling

Three forms Each contained common and unique items

29

Research MethodsSchool Sample Characteristics

Percent Mean SD

Overall response rate 47.9%Enrollment 1564 229N classroom teachers 82 10Ethnic composition (% white) 78 10SES (% receiving FRL) 17 10

Final sample size for analysis:376 “WASL teachers” (teach 10th grade AND math, English, science, special education, or ELL)

30

CONTENT

Frequency Percent Valid Percent Cumulative

Percent ELL 12 3.2 3.2 3.2 ENG 103 27.3 27.3 30.5 MATH 113 30.0 30.0 60.5 SCI 81 21.5 21.5 82.0 SPED 68 18.0 18.0 100.0

Valid

Total 377 100.0 100.0

Years Since Preservice Traning


Percent 0-5 96 25.5 25.5 25.5 6-10 70 18.6 18.6 44.0 11-15 58 15.4 15.4 59.4 16-20 47 12.5 12.5 71.9 21-25 23 6.1 6.1 78.0 26-30 27 7.2 7.2 85.1 30+ 27 7.2 7.2 92.3 9 29 7.7 7.7 100.0

Valid

Total 377 100.0 100.0

Years Teaching at This School


Percent 0-5 207 54.9 54.9 54.9 6-10 78 20.7 20.7 75.6 11-15 36 9.5 9.5 85.1 16-20 26 6.9 6.9 92.0 21-25 15 4.0 4.0 96.0 26-30 7 1.9 1.9 97.9 30+ 5 1.3 1.3 99.2 9 3 .8 .8 100.0

Valid

Total 377 100.0 100.0

Research MethodsSample Characteristics – Teachers

31

Research MethodsScale Development

Classical Test Theory Internal consistency reliability (Cronbach’s

coefficient alpha (α)) Item-total correlations Exploratory factor analyses (EFA)

Item Response Theory Rating Scale Model (Wright & Masters, 1982) Item difficulty, fit statistics

32

Research MethodsOutcome Measures

Frequency of WASL Data Use Utility of WASL Data Use

33

Research MethodsPredictor Measures Perceived Access to Data Training in State Assessment Training in WASL Item Construction Pressure to Increase WASL Performance WASL Goal Alignment Efficacy with WASL Data Principal WASL Commitment Principal Trust Departmental Professional Collaboration Professional Accountability

34



35

I review the past WASL scores of my current students to see who has scored below standards


Percent Never 67 17.8 17.8 17.8 Once a Year 138 36.6 36.6 54.4 A Few Times a Year 101 26.8 26.8 81.2 Every Semester 34 9.0 9.0 90.2 Monthly 33 8.8 8.8 98.9 Weekly 3 .8 .8 99.7 9 1 .3 .3 100.0

Valid

Total 377 100.0 100.0

ResultsHow much are teachers using data?

36

I use the past WASL scores of my current students to make decisions about how best to instruct my students


Percent Never 107 28.4 28.4 28.4 Once a Year 78 20.7 20.7 49.1 A Few Times a Year 92 24.4 24.4 73.5 Every Semester 51 13.5 13.5 87.0 Monthly 33 8.8 8.8 95.8 Weekly 15 4.0 4.0 99.7 9 1 .3 .3 100.0

Valid

Total 377 100.0 100.0


37

I review WASL scores and/or released item data with other teachers in my content area


Percent Never 65 17.2 17.2 17.2 Once a Year 105 27.9 27.9 45.1 A Few Times a Year 125 33.2 33.2 78.2 Every Semester 53 14.1 14.1 92.3 Monthly 22 5.8 5.8 98.1 Weekly 3 .8 .8 98.9 9 4 1.1 1.1 100.0

Valid

Total 377 100.0 100.0


38

I review our schools WASL results to determine what professional development I need in a particular area


Percent Never 146 38.7 38.7 38.7 Once a Year 118 31.3 31.3 70.0 A Few Times a Year 65 17.2 17.2 87.3 Every Semester 31 8.2 8.2 95.5 Monthly 12 3.2 3.2 98.7 Weekly 1 .3 .3 98.9 9 4 1.1 1.1 100.0

Valid

Total 377 100.0 100.0


39

I use WASL released items as assessments and score student responses


Percent Never 85 22.5 22.5 22.5 Once a Year 51 13.5 13.5 36.1 A Few Times a Year 112 29.7 29.7 65.8 Every Semester 63 16.7 16.7 82.5 Monthly 42 11.1 11.1 93.6 Weekly 20 5.3 5.3 98.9 9 4 1.1 1.1 100.0

Valid

Total 377 100.0 100.0


40


0% 20% 40% 60% 80% 100%

Review WASL scores of current students

Review WASL data w/other teachers

Review WASL data to see what PD I need

Use released items as assessments

Use WASL scores to make instructionaldecisions for current students

Never Once/yr Few times/yr Each semester Monthly Weekly

41

ResultsHow much are teachers using WASL data?

Range: 0-100Mean: 37.5SD = 13.9

What does this scale mean in practical terms?

42

The WASL assessments generally measure the knowledge and skills I am teaching in my classroom


Percent Strongly disagree 35 9.3 9.3 9.3 Disagree 90 23.9 23.9 33.2 Neutral 50 13.3 13.3 46.4 Agree 170 45.1 45.1 91.5 Strongly agree 24 6.4 6.4 97.9 9 8 2.1 2.1 100.0

Valid

Total 377 100.0 100.0

ResultsHow much do teachers benefit from WASL data?

43

The assessments I use in my classroom are similar to those used on the WASL



Valid

Total 377 100.0 100.0


44

WASL data help me decide what content to teach my students



Valid

Total 377 100.0 100.0


45

WASL data help me understand my students strengths and weaknesses



Valid

Total 377 100.0 100.0


46


WASL data are of [very little] use to me in my instruction (reversed)

31 8.2 8.2 8.288 23.3 23.3 31.661 16.2 16.2 47.7

140 37.1 37.1 84.949 13.0 13.0 97.98 2.1 2.1 100.0

377 100.0 100.0

Strongly disagreeDisagreeNeutralAgreeStrongly agree9Total

ValidFrequency Percent Valid Percent

CumulativePercent

WASL data are of use to me in my instruction

47


WASL data help me decide which instructional strategies to use with my students

50 13.3 13.3 13.397 25.7 25.7 39.090 23.9 23.9 62.9

117 31.0 31.0 93.917 4.5 4.5 98.46 1.6 1.6 100.0

377 100.0 100.0

Strongly disagreeDisagreeNeutralAgreeStrongly agree9Total

ValidFrequency Percent Valid Percent

CumulativePercent

48


0% 20% 40% 60% 80% 100%

WASL generally measures knowledge/skillsI'm teaching

My assessments are similar to those on WASL

WASL data help me decide what content toteach

WASL data help me understand mystrengths/weaknesses

WASL data are of use to my instruction

WASL data help me decide which instructionalstrategies

Strongly disagree Disagree Neutral Agree Strongly Agree

49


Range: 0-100Mean: 46.4SD = 15.1

50

ResultsWhat motivates teachers to use WASL data?

Level 1: Teacher

Teacher WASL Data Use = β0 + βj(Training in State Assessment) + β2(Perceived Access) + β3(Perceived Pressure) + β4(Expectancy) + β5(Efficacy) + β6(Goal Alignment) + R

Level 2: School

β0 = γ00 + γ01(Demographics) + γ02(Professional Accountability) + γ03(Professional Collaboration) + γ04(Principal Leadership) + U0

β3 = γ10 + γ11(Principal Leadership) + U1

51

ResultsVisualizing Hierarchical Linear Modeling, 1/3

r = 0.52

52


53Pressure to Increase WASL Performance (0-100)

Freq

uenc

y of

WA

SL D

ata

Use

(0-1

00)


54

ResultsFrequency of WASL Data Use – HLM Results

Null Level 1 Level 2

Fixed Part Coef. SE T Coef. SE T Coef. SE T

For Intercept B0jIntercept Y00PctWhite

37.57 1.22 37.51 0.77 48.9*** 37.57-0.25

0.600.07

62.4***-3.78***

For Utility B1j 0.31 0.05 7.34*** 0.31 0.04 7.36***

For WASL Training B2j

0.05 0.02 2.60** 0.05 0.02 2.36**

For Pressure B3j 0.11 0.04 2.84** 0.11 0.04 2.90**

For Principal WASL B4j

0.09 0.04 2.08* 0.09 0.04 2.14*

For Efficacy B5j 0.11 0.03 4.20*** 0.12 0.03 4.35***

Random Part SD Var. Chi SD Var. Chi SD Var. Chi

Intercept u0j 4.72 22.28 64.58*** 2.44 5.97 39.38** 1.09 1.20 23.32

Level 1 effect rij 13.13 172.30 10.73 115.09 10.70 114.59

Model Fit Deviance Deviance Chi Deviance Chi

3023.2 2883.2 145** 2872.8 12**

Intraclass correlation coefficient: uoj / uoj + rij = .11 = 11%

55

ResultsFinal Model of Frequency of Data Use – HLM Results

Frequency of WASL Data Use =β0 (mean)+ β1(utility of WASL data use)+ β2(training in WASL item writing)+ β3(perceived pressure to increase WASL scores)+ β4(principal commitment to WASL improvement)+ β5(efficacy with WASL data)+ r (unmodeled variation)

β0 = γ00 + γ01(school ethnic composition) + u0

56

ResultsModeling Frequency of WASL Data Use

Coefficientsa

-.283 3.829 -.074 .9412.050 .772 .129 2.655 .008

-2.448 .628 -.177 -3.897 .000

1.104 .576 .082 1.916 .056

2.442 .610 .192 4.001 .000

.105 .038 .114 2.743 .006

.260 .044 .282 5.935 .000

.108 .026 .189 4.132 .000

.055 .020 .123 2.695 .007

(Constant)The principal at this school --- encourages teachers to make decisions based on dataThe principal at this school --- takes primary responsibility for presenting and interpretingWASL results for teachersThe principal at this school --- places too much emphasis on the WASL (reverse)The principal at this school --- commits resources to training teachers how to helpstudents develop the skills necessary to succeed on the WASLPerceived Pressure to Increase WASLUtility of WASL DataWASL EfficacyTraining in WASL

Model1

B Std. Error

UnstandardizedCoefficients

Beta

StandardizedCoefficients

t Sig.

Dependent Variable: Frequency of WASL Data Usea.

Model Summary

.657a .431 .419 10.56126Model1

R R SquareAdjustedR Square

Std. Error ofthe Estimate

Predictors: (Constant), Training in WASL, PerceivedPressure to Increase WASL, The principal at thisschool --- takes primary responsibility for presentingand interpreting WASL results for teachers, Theprincipal at this school --- places too much emphasison the WASL (reverse), WASL Efficacy, The principal atthis school --- commits resources to training teachershow to help students develop the skills necessary tosucceed on the WASL, Utility of WASL Data, Theprincipal at this school --- encourages teachers tomake decisions based on data

a.

57

ResultsUtility of WASL Data Use – HLM Results

Null Level 1

Fixed Part Coef. SE T Coef. SE T

For Intercept B0jIntercept Y00 46.45 0.84 55.29*** 46.41 0.49 94.53***

For Access B1j 0.09 0.03 3.38***

For Frequency B2j 0.28 0.04 6.86***

For Accountability B3j -0.11 0.03 -3.81***

For Collaboration B4j 0.12 0.04 3.13**

For WASL Goal Alignment B5j

0.63 0.04 14.20***

For WASL Efficacy B6j 0.06 0.02 2.66**

Random Part SD Var. Chi SD Var. Chi

Intercept u0j 1.49 2.21 24.57 0.11 0.01 9.01

Level 1 effect rij 15.00 225.12 9.51 90.36

Model Fit Deviance Deviance Chi

3102.78 2782.0 320.78**

Intraclass correlation coefficient: uoj / uoj + rij = .008 = 0.8%

58

ResultsFinal Model of Utility of Data Use – HLM Results

Utility of WASL Data Use =β0 (mean)+ β1(perceived access to WASL data)+ β2(frequency of WASL data use)+ β3(departmental professional accountability)+ β4(departmenal professional collaboration)+ β5(WASL goal alignment)+ β6(WASL efficacy)+ r (unmodeled variation)

59



60

Conclusions

High School Teachers Use of Data Teachers are using data with moderate frequency and

gaining some value from it This aspect of WASL program is working

Motivation and School Context Multiple motivations are at work (pressure and efficacy) Principal leadership provides incentive to use Sensemaking of data is social / collaborative Data as feature of more tightly coupled schools

61

Thank you!

To contact me:

Jack B. Monpas-Huber, PhDDirector of Assessment and Program EvaluationSpokane Public [email protected](509) 354-7396 Office(206) 947-9926 Cell

mailto:[email protected]

jack b. monpas-huber, ph.d. director of assessment and program evaluation spokane public schools

Documents

teacher data useaccess

strong predictor of

school cultures

aspects of school context

role of school culture

influence of policy

research questionsconsidering

perceived pressure