log file analyses, transversal skills & learning analytics: a … · 2019. 6. 12. · learning...

Transversal Skills & Log-File Analyses

ERC & ETS, Dublin, May 16th, 2019Samuel Greiff

Log File Analyses, Transversal Skills & Learning Analytics:

A Good Match, But Not a Perfect One

Samuel GreiffUniversity of Luxembourg



• TIMSS: 60+ countries• PIRLS: 57 countries• ICCS: 38 countries • PISA: 70+ countries• PIAAC: 33 countries• TALIS: 34 countries• NAEP (national)• Epstan (national)• Ländervergleich (national)

Large-Scale Assessments



Autor, Levy, & Murnane (2003)

1960 20151970 1980 2000

50

52

48

46

44

42

40

38

54

56

58

60

62

1990

Manual &routine tasks

Complex non-routine tasks

Collaborative tasks

Increase in demands

Decrease in demands

Trans-versalskills

6



Problem Solving in LSA

• Complex Problem Solvingà PISA 2012

• Collaborative Problem Solvingà PISA 2015

• Adaptive Problem Solvingà PIAAC Cycle 2

7



PISA 2012• ~510 000 participants from 65 countries• Focus

- Mathematics- Reading- Science- Problem Solving

Ø Domain: Complex Problem Solving

8



Complex Problem Solving in PISA

• Definition

- Capacity to engage in, to

understand, and to resolve

problem situations

• Relevance

- Addressing dynamics,

complexity and intransparency

of work-life and societal tasks

- Non-routine Problem Solving

• Assessment

- Computerized tasks

- Elaborate and subsequently

solve dynamic problems

9

Retrieved from

https://www.oecd.org/pisa/test/testquestions/question3/



PISA Creative Problem Solving Tasks


ERC & ETS, Dublin, May 16th, 2019Samuel Greiff 12

PISA 2012 Problem Solving dimensions

Exploring & Understanding

Representing & Formulating

Planning & Executing

Monitoring & Reflecting

OECD (2012)



Problem Solving & the other domains

Mean R2 of Complex Problem Solving with other domains: 61% overlap

Mean R2 of other domains: 77% overlap

Copyright OECD (2014)



PISA 2015• ~540 000 participants from 72 countries• Focus

- Mathematics- Reading- Science- Problem Solving

Ø Domain: Collaborative Problem Solving

15



Collaborative Problem Solving in PISA

• Definition- Solving a given problem relying on individual

capacities and knowledge; interacting and including others’ capacities and knowledge

• Assessment- Computerized tasks- Required interaction with computer-simulated

agents to successfully solve a problem

16



Benefits of CBA• Standardization• Automatic scoring• Tailored testing (CAT)• Facilitating complex designs• New item types• Collecting additional information


ERC & ETS, Dublin, May 16th, 2019Samuel Greiff !21



1ASSESSING PROBLEM-SOLVING SKILLS IN PISA 2012

38 © OECD 2014 CREATIVE PROBLEM SOLVING: STUDENTS’ SKILLS IN TACKLING REAL-LIFE PROBLEMS – VOLUME V

In the unit CLIMATE CONTROL, students are told that they have a new air conditioner but no instructions for it. Students can use three controls (sliders) to vary temperature and humidity levels, but first they need to understand which control does what. A measure of temperature and humidity in the room appears in the top-right part of the screen, both in numeric and in graphical form. All items in this unit present an interactive problem situation, with context classified as personal and technological.

The unit CLIMATE CONTROL is a typical MicroDYN unit, with a first “knowledge-generation” task and a second “knowledge-application” task. Knowledge generation in the MicroDYN environment requires students to carefully monitor the effects of their interventions. The increase in the level of an input variable leads either to an increase, a decrease, a mixed effect (increase and decrease for different variables), or to no effect in one or more output variables.

CLIMATE CONTROL: Item 1

• Figure V.1.10 •CLIMATE CONTROL: Item 1

Question 1: CLIMATE CONTROL CP025Q01 Find whether each control influences temperature and humidity by changing the sliders. You can start again by clicking RESET.

Draw lines in the diagram on the right to show what each control influences.

To draw a line, click on a control and then click on either Temperature or Humidity. You can remove any line by clicking on it.

Top Control Temperature

Central Control Humidity

Bottom Control

In the first item in the unit, students are invited to change the sliders to find out whether each control influences the temperature or the humidity level. The problem-solving process for this item is representing and formulating: the student must experiment to determine which controls have an impact on temperature and which on humidity, then represent the causal relations by drawing arrows between the three controls and the two outputs (temperature and humidity). There is no restriction on the number of rounds of exploration that the student is allowed. Full credit for this question requires that the causal diagram is correctly completed. Partial credit for this question is given if the student explores the relationships among variables efficiently, by varying only one input at a time, but fails to correctly represent them in a diagram.



Question 2: CLIMATE CONTROL CP025Q02 The correct relationship between the three controls, Temperature and Humidity is shown on the right.

Use the controls to set the temperature and humidity to the target levels. Do this in a maximum of four steps. The target levels are shown by the red bands across the Temperature and Humidity graphs. The range of values for each target level is 18-20 and is shown to the left of each red band. You can only click APPLY four times and there is no RESET button.



Bottom Control



Axis 2: Assessment instruments VI



How to (even start to) explore log files?1) How do students behave during assessment?

àHow long do students interact with a task?àDoes time-on-task vary across countries and subgroups?

2) Can we identify “good” behavior?àAre students applying optimal strategies?àCan we explain final performance through behavior?

3) Can we identify different levels of low performance?àWhere do unsuccessful students fail?àWhere should interventions be heading?

24



1) How do students behave during assessment?

àHow long do students interact with a task?àDoes time-on-task vary across countries and subgroups?

25



• No large difference of time on task by item score!

26

correct

incorrect



How do students behave during assessment?

- Broad variation among studentsàProcess data highlight heterogeneous behavior- BUT: No clear relation to performanceàNo easy solution for explanations of performance on an

individual level

27



2) Can we identify “good” behavior?

àAre students applying optimal strategies?àCan we explain final performance through behavior?

28



• Application of VOTAT? Strong connection to performance!

30

Climate Control Q1Total

incorrect correctnoVOTAT 42.2% 9.7% 51.9%

VOTAT 6.9% 41.1% 48.1%

Total 49.1% 50.9% 100.0%



Relation between VOTAT application and PISA score

31



3) Can we explain low performance?

àWhere do unsuccessful students fail?àWhere should interventions be heading?

32



1ASSESSING PROBLEM-SOLVING SKILLS IN PISA 2012

38 © OECD 2014 CREATIVE PROBLEM SOLVING: STUDENTS’ SKILLS IN TACKLING REAL-LIFE PROBLEMS – VOLUME V

In the unit CLIMATE CONTROL, students are told that they have a new air conditioner but no instructions for it. Students can use three controls (sliders) to vary temperature and humidity levels, but first they need to understand which control does what. A measure of temperature and humidity in the room appears in the top-right part of the screen, both in numeric and in graphical form. All items in this unit present an interactive problem situation, with context classified as personal and technological.

The unit CLIMATE CONTROL is a typical MicroDYN unit, with a first “knowledge-generation” task and a second “knowledge-application” task. Knowledge generation in the MicroDYN environment requires students to carefully monitor the effects of their interventions. The increase in the level of an input variable leads either to an increase, a decrease, a mixed effect (increase and decrease for different variables), or to no effect in one or more output variables.



Question 1: CLIMATE CONTROL CP025Q01 Find whether each control influences temperature and humidity by changing the sliders. You can start again by clicking RESET.

Draw lines in the diagram on the right to show what each control influences.

To draw a line, click on a control and then click on either Temperature or Humidity. You can remove any line by clicking on it.



Bottom Control

In the first item in the unit, students are invited to change the sliders to find out whether each control influences the temperature or the humidity level. The problem-solving process for this item is representing and formulating: the student must experiment to determine which controls have an impact on temperature and which on humidity, then represent the causal relations by drawing arrows between the three controls and the two outputs (temperature and humidity). There is no restriction on the number of rounds of exploration that the student is allowed. Full credit for this question requires that the causal diagram is correctly completed. Partial credit for this question is given if the student explores the relationships among variables efficiently, by varying only one input at a time, but fails to correctly represent them in a diagram.



Question 2: CLIMATE CONTROL CP025Q02 The correct relationship between the three controls, Temperature and Humidity is shown on the right.

Use the controls to set the temperature and humidity to the target levels. Do this in a maximum of four steps. The target levels are shown by the red bands across the Temperature and Humidity graphs. The range of values for each target level is 18-20 and is shown to the left of each red band. You can only click APPLY four times and there is no RESET button.



Bottom Control



• How consistently did students apply VOTAT?

34

All countries

26,9

10,4

4,37,0

4,1 2,6 3,1

41,6

0

10

20

30

40

50

no isolated variation isolated variation of 1input variable

isolated variation of 2input variables

isolated variation of allinput variables (VOTAT)

Percen

t

incorrect

correct


ERC & ETS, Dublin, May 16th, 2019Samuel Greiff 35



A brief summary• What has happened?

- Shift to technology-based assessment

- “New” innovative domains

37

• What will / might happen?- Non-cognitive skills- Real collaboration- Formative assessment- Classroom relevance



System monitoring

Teacherevaluation

Formative Assessment for learning

Externalevaluation

Summativeassessment of

learning

Internal evaluation

Monitoring ofteacher

performance

System Level

School Level

Classroom Level

(Klieme, 2017)



• Many behaviors and many countries…

39



Greiff et al. in CHB (2016)



Greiff et al. in C&E (2018)



Assessment for learning (“stealth assessment”)

Shute et al. (in press)



Learning Analytics, Machine Learning, EDM...• Can personalize and adapt the learner’s learning

environment- Where did students struggle?- When were wrong paths chosen?- How carefully did students work?



D’Mello & Graesser (2012)



Appendix

Crystal Island



Scientific impact of LSA

Research areas: Educational Research (229), Economics (59), Assessment and Testing (54), Science Research (54)

Hopfenbeck et al., 2018

log file analyses, transversal skills & learning analytics: a … · 2019. 6. 12. · learning...

Documents