context assessment in international large scale studies ... · 2.designing questionnaires as a data...

Context assessment in international Large Scale Studies:

From student background information to the quality of schooling

Eckhard Klieme

German Institute for International Educational Research (DIPF)Frankfurt am Main, Germany

”Center for Research on Educational Testing

Tokyo, February 4, 2013

Klieme: Was ist guter (Mathematik-) Unterricht? GDM 2011, Freiburg

Overview

1. Main goal of Large Scale Studies: Providing indicators for educational monitoring

2. Designing questionnaires as a data source for indicators:Evolution of the questionnaire design for PISA

3. Example of policy area studied in LSA: Covering assessment & evaluation policies in PISA 2015

4. Explanatory power of Large Scale Studies5. Conclusions

Klieme: Learning from Large Scale Assessment and Research Studies Klieme: Context assessment in international Large Scale Studies (LSA)

Klieme: Was ist guter (Mathematik-) Unterricht? GDM 2011, FreiburgKlieme: Learning from Large Scale Assessment and Research Studies

Indicators: 1) Achievement

Klieme: Context assessment in international Large Scale Studies (LSA)


Pressure to perform Support Quality of relation

Uni

ted

Sta

tes

New

Zea

land

Uni

ted

Kin

gdom

Aust

ralia

Can

ada

Sw

eden

Finl

and

Nor

way

Japa

n

Luxe

mbu

rg

Bel

gium

Net

herla

nds

Fran

ce

Sw

itzer

land

Aust

ria

Ger

man

y

Pol

and

Hun

gary

Rus

s. F

eder

atio

n

Kor

ea

0,50,40,30,20,1-0,0-0,1-0,2-0,3-0,4-0,5

Indicators: 2) School climate



Indicators: 3) School policies

School autonomy Academic School activities reg. currriculum & selectivity to promoteassessment science learning

others Japan others Japan others Japan


Klieme: Was ist guter (Mathematik-) Unterricht? GDM 2011, FreiburgKlieme: Context assessment in international Large Scale Studies (LSA)

School autonomyregarding

curriculum and assessment


Academicselectivity


School activitiesto promote

science learning


Overview






Seite 11Ort, Datum | Name Referent | Name der Veranstaltung |Titel der PowerPoint-PräsentationPGB, April 2012 | Core 6

OutcomesAchieve-

ment

Science

Reading

Maths


Background OutcomesFamily Education Achieve-

ment

Out-of-school science experience

Science

Student SES & family

Ethnicity & migration

Educational pathways in early child-hood

Reading

Maths


Background OutcomesFamily Education Achieve-

ment


Science




Reading

Maths

Problem Solving &Collab. PS


Background OutcomesFamily Education Non/meta-

cognitiveAchieve-

mentFurther


Science-related outcomes: interest…

Science Science career




General behaviour and attitudes

Dispositions for Coll. PS

Reading

Maths


Seite 15Ort, Datum | Name Referent | Name der Veranstaltung |Titel der PowerPoint-Präsentation

Background Schooling OutcomesFamily Education Actors Core

processesRessource allocation

Non/meta-cognitive

Achieve-ment

Further


Teacher qualification &professional

knowledge

Science teaching practices

Learning time and curriculum



School-level learning environment for science




Parental involvement

Leadership and school

management

School climate: Interpersonal relations, trust, expectations

RessourcesGeneral behaviour and attitudes


Reading

Maths


Locus of control within the school

system

Assessment, evaluation and accountability

Allocation, selection and choice

Teaching & Learning

School policies

Governance

Seite 16Ort, Datum | Name Referent | Name der Veranstaltung |Titel der PowerPoint-Präsentation

Background Schooling OutcomesFamily Education Actors Core

processesRessource allocation

Non/meta-cognitive

Achieve-ment

Further


Teacher qualification &professional

knowledge

Science teaching practices

Learning time and curriculum



School-level learning environment for science




Parental involvement

Leadership and school

management

School climate: Interpersonal relations, trust, expectations

RessourcesGeneral behaviour and attitudes


Reading

Maths


Locus of control within the school

system

Assessment, evaluation and accountability

Allocation, selection and choice

Teaching & Learning

School policies

Governance


Overview







PISA 2015 – Field Trial: Questions on assessment, evaluation, and accountability

PGB, October 2011 | Core 6 |

General assessment practice (ScQ)Purpose of assessment results (ScQ)

Evaluation policies (ScQ)

Classroom assessment(TQ, StQ)

External evaluation (ScQ)

Teacher evaluation

(ScQ,TQG)

Internal valuation

(ScQ,TQG)

Foci Grading (TQG)

See „evaluation policies“

Teacher evaluation methods

Processes Classroom assessment practices (TQG/TALIS)

Use of achievement data for

accountabilityUse of external

evaluation

Teacher incentives Consequences Feedback: student perception (StQ).use by students (StQ)

Adaptation of instruction (StQ, TQS)



PGB, October 2011 | Core 6 |


General assessment practices (ScQ)

Teacher-made

Standardized


QIn your school, are standardised* and/or teacher-made assessments of students in <national modal grade for 15-year-olds> used for any of the following purposes?

Standardisedassessment

Teacher-made

assessment

Neither ofthem isused

a)To provide individual feedback to students to guidetheir learning 1 2 3

b) To inform parents about their child’s progress 1 2 3

c)To make decisions about students’ retention orpromotion 1 2 3

d) To group students for instructional purposes1 2 3

e)To compare the school to <district or national> performance 1 2

3

f) To monitor the school’s progress from year to year1 2 3

g) To make judgements about teachers’ effectiveness1 2 3

h)To identify aspects of instruction or the curriculum thatcould be improved 1 2

3

i) To adapt teaching to the students’ needs 1 2 3

j) To compare the school with other schools1 2 3

Purpose of assessment (ScQ)Klieme: Context assessment in international Large Scale Studies (LSA)

Formative use of assessment:

„Assessment for learning“


Classroom assessment practices (TQ)Klieme: Context assessment in international Large Scale Studies (LSA) Klieme: Context assessment in international Large Scale Studies (LSA)

Formativeassessment in the

classroom


Perceived feedback (StQ)Klieme: Context assessment in international Large Scale Studies (LSA)

Process-oriented feedback


Adaptation of instruction (StQ/TQ)Klieme: Context assessment in international Large Scale Studies (LSA)

Adaptive instruction


Process-orientedfeedback

Classroom: StudentFormative Assessment achievement Practices and interest

Adaptiveinstruction


Explanatory model


School: Internal evaluation

Process-orientedfeedback

Classroom: StudentFormative Assessment achievementPractices and interest

AdaptiveSchool: instructionformative use ofAssessment


Explanatory model


Overview







Predictor Predicting school level

AchievementR²=.61

Interest R² = .33

OTL: Concept FamiliarityConcepts: Overclaiming

OTL: Math TasksApplied Reasoning ExamplePure Math Reasoning Example

Challenge (Complex Problems)Structure (Disciplinary climate)Teacher Support

Instructional Practices: teacher-directedInstructional Practices: student orientedInstructional Practices: formative

,496-,389,101,080-,086

,109,124-,111

-,070-,327,092

,121,078-,025-,007-,042

,183,065,307

,004,228-,060

27

Klieme: Learning from Large Scale Assessment and Research Studies

Explaining outcomes on the school level: PISA 201２ Field trial



Internal Evaluation Change in

Achievement

Extracurrricular Change in

Activities School Climate

(All-day-schools)

Klieme: The Role of LSA in Research on Educational Effectiveness and School DevelopmentKlieme et al.: Measuring efficacy, efficiency, and equity: The case of PISA Klieme: Learning from Large Scale Assessment and Research Studies

Explaining change at the school level: PISA 2000-2009 Germany - school panel



Overview







Conclusions

-Questionnaires provide important data on the learning context, such as student and school background, and non-cognitive outcomes.-The new (2012, 2015) PISA design incorporates information on teaching & learning, school policies, and governance.-Contextual data, based on questionnaires, can help to understand and monitor the (formative) use of assessment in classrooms.-Longitudinal designs (follow-up, school panel) significantly enhance the explanatory power of LSA studies.

Klieme: The Role of LSA in Research on Educational Effectiveness and School DevelopmentKlieme: Learning from Large Scale Assessment and Research Studies Klieme: Context assessment in international Large Scale Studies (LSA)


Thank you for your attention!

[email protected]


context assessment in international large scale studies ... · 2.designing questionnaires as a data...

Documents