Lessons learned from adapting U.S. developed MKT measures for use in Norway

Presentation at the University of Huelva, SpainMay 27th, 2011

Arne JakobsenDepartment of Education

University of Stavanger, Norway

Disposition

• History and background (Norway)– Students (TIMSS, PISA,…)– Teachers’ knowledge is important for students

achievement– Government decide to invest in Professional

development – Professional development programs to meet teachers’

needs– Mathematical knowledge for teaching (MKT)

Our project research question: What are the possibilities and challenges of using the MKT measures in Norway?

• The project so far– Translation of items – Pilot: 142 (150) teachers measured and 7 focus group

interviews

MKT

• Classroom studies in the U.S.• Work of teaching• Tasks of teaching• Areas of MKT• MKT items• MQI - back to the classroom

Tasks of teaching with items. • Giving and evaluating explanations

– Explaining a common procedure– Explain a concept or idea– Evaluating student explanations for evidence of understanding– Characterizing the quality of an explanation–  

• Interpreting and evaluating non-standard methods/ideas and multiple solutions– Evaluating a particular non-standard method– Evaluating multiple possible solutions for a particular problem

  • Choosing examples and problems

– Choosing examples to introduce a concept– Choosing examples that use or illustrate a particular idea– Choosing examples that lend themselves to particular strategies– Changing the context/numbers of a problem without changing

the mathematical content– Designing a mathematically similar problem– Designing a simpler version of a problem– Designing a sequence of problems to teach an idea

 

Tasks with items (cont.)

• Choosing and using representations– Representing a particular idea in multiple ways– Interpreting a particular representation in multiple ways– Choosing a diagram/story/model to represent and

idea/concept/expression– Selecting a representation to match an instructional

purpose

• Analyzing student errors– Identifying errors of the same type

• Evaluating difficulty

• Choosing and using definitions

(+ more.) 

MKT - the «egg»

Ball, D. L., Thames, M. H., & Phelps, G. (2008, p. 403).

Example item

Translation and adaption of measures

• Delaney's studies (Irland)• Documentation of changes• Aspects of equivalence

Aspects of equivalence

Translation and adaption

Translation and adaptation

Mosvold, R., Fauskanger, J., Jakobsen,

A., & Melhus, K. (2009). 

MKT, released items

The multiple-choice format

Focus group interviews.

How can Norwegian teachers’ reflections about the MKT items contribute to our understanding of challenges related to the multiple-choice format?

Item Responce Theory (IRT)

• How can information about items psychometric properties give new insight when adapting U.S. developed MKT measure for use in Norway?

• An item performance is described by an item characteristic function (with different difficulty and slopes).

• Monotonically increasing function – as the MKT increase the probability of correct answer increase.

• Slope and difficulty is of importance.

Item performance

-3

-2.9

-2.8

-2.7

-2.6

-2.5

-2.4

-2.3

-2.2

-2.1

-2

-1.9

-1.8

-1.7

-1.6

-1.5

-1.4

-1.3

-1.2

-1.1

-1

-0.9

-0.8

-0.700000000000001

-0.600000000000001

-0.5

-0.4

-0.3

-0.2

-0.1

0 0.1

0.2

0.3

0.4

0.5

0.600000000000001

0.700000000000001

0.8

0.9

1 1.1

1.2

1.3

1.4

1.50000000000001

1.6

1.7

1.80000000000001

1.90000000000001

2.00000000000001

2.1

2.20000000000001

2.30000000000001

2.40000000000001

2.50000000000001

2.60000000000001

2.70000000000001

2.80000000000001

2.90000000000001

3.00000000000001

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

2-Parameter Logistic Model

Item 1

Item 2

Item 3

Theta

P(u

=1)

From IRT analyses of data

IRT findings

We found that items can be divided into three groups:

1) Items that do not seem to function in Norway 2) Items that function well, but have relatively

high difference in item characteristics in Norway compared to the U.S., and

3) Items that seem to function well and that have item characteristics close to what is reported in the U.S.

Quantitative + qualitative analyses!

From IRT analyses of the data

• Snippets from the results• Quantitative + qualitative analyses• Aspects of equivalence

Teachers’ MKT

Focus group interviews:

We expected that teachers with strong MKT in one content area would be strong in other content areas relevant for their level of teaching.

The teachers argued that some of the content areas were not relevant and more difficult compared to others

- our motivation to study correlations

Correlation between NCOP and GEOMETRY is 0.701 (p-value < 0.0005).

Correlations (Pearson, p-value<0.005)

• Teachers’ MKT scores in different content areas are correlated

• All correlations are significant – weakest correlation between PFA and NCOP

Correlations

NCOP PFA

GEOM 0.701 0.673

PFA 0.580

Conclusion

• We find that teachers’ MKT scores in the three content areas are correlated

– Teachers with high MKT in one content area have high MKT in other content areas and vice versa

– Despite comments from teachers that not finding PFA items relevant for their teaching, their PFA MKT is related to MKT in the other content areas

Final comments

• Psychometrically items seems to function well in Norway

• Want to establish link between Norwegian teachers’ MKT and student achievements (UiO-TIMSS 2011)

• Discussing items as part of professional development to learn more about teachers MKT?

• Framework and other theories?

• Possibilities and limitations?