hr lsa trends
DESCRIPTION
trensdsTRANSCRIPT
Advancements in Large-‐scale Assessment
Heiko Rölke & Krisz:na Toth DIPF -‐ Deutsches Ins:tut für Interna:onale
Pädagogische Forschung Frankfurt am Main
TAO Days | 10.09.2012 Seite 1
Overview
• Trends in LSA • Closer Look at Trends • Example: Making Use of Data • Challenges in LSA
TAO Days| 10.09.2012 Seite 2
Trends
Main trend: CBA
Sub-‐trends/effects of CBA: • CAT
• towards forma:ve assessment • Complex items
• Simula:on, interconnec:on • Interweaving
• different tests, ques:onnaires, etc. • Big data
TAO Days | 10.09.2012 Seite 3
Trend: CBA
Computer-‐based Assessment has to serve a purpose • Costs? • Time! • Validity!
TAO Days | 10.09.2012 Seite 4
CAT
• Also not prac:ced for its own sake • Not as important in pure summa:ve assessment • But:
• Strict :me limits (e.g. PIAAC) • Growing demand for forma:ve aspects in summa:ve assessment
TAO Days | 10.09.2012 Seite 5
Complex Items
• Closer at reality • Simula:on Most important: • Ability to assess new domains
TAO Days | 10.09.2012 Seite 6
Examples of Complex Items § Simula+on of an “Email / Web Scenario”
The test person receives an email and should book cinema :ckets online.
§ Dynamic Model (MicroDYN)
The test person should explore and master a dynamic system with input (exogenous) variables influencing output (endogenous) variables.
§ Automaton with Finite State Machine (MicroFIN) The test person should interact with a mobile phone and set :me to summer:me.
Interweaving of Tests and Ques:onnaires • Framing
• test only if precondi:ons are fulfilled • Double-‐check
• findings of ques:onnaire
TAO Days | 10.09.2012 Seite 8
Big Data
• Log data, not only results • Find out what is going on • Make use of data
• E.g. par:al scoring
• -‐> Examples from PIAAC PS-‐TRE
TAO Days | 10.09.2012 Seite 9
Simula:on-‐based assessment
• Modern assessments: • real life situa:ons: e.g. web environments • complexity of instrument – various ways
• Monitoring students‘ ac:ons -‐ log files • Analysis of test-‐taking paths is a rela:vely new field in educa:on
(reason: printed vs. computer-‐based data collec:on) • Process data requires to consider and evaluate methods
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 10
Aim • to inves:gate how log data can be integrated into the process
of educa+onal assessment and evalua:on to support researchers and prac::oners in making use of the data assembled in computer-‐based test delivery
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 11
Aim • to inves:gate how log data can be integrated into the process
of educa+onal assessment and evalua:on to support researchers and prac++oners in making use of the data assembled in computer-‐based test delivery
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 12
Hypertext Item
13
14
15
829709 15 {"sender":"15","type":"loading","data":""} 833463 15 {"sender":"15","type":"loaded","data":""} 848153 15 {"sender":"15","type":"user_interac+on","":"<?xml version=\"1.0\">
\u000d\u000a<cbaloggingmodel:EmbeddedLinkLogEntry xmlns:cbaloggingmodel=\"htp://www.soucon.de/cba/cbaloggingmodel\" id=\„-‐ b576759f1:-‐7fed\" sourcePageId=\"Item15_linklist\" targetPageId=\"Item15_website1\" textFieldId=\"cbaTextField_71"/>\u000d\u000a"} 849094 15 {"sender":"15","type":"variable_change","data":{"name":
"snapshot_url„,"value":"htp://localhost:8101/cba-‐run:me/itemjsessionid= LB1?custom_servicehandler=downloadService&file= C:\\...snapshot5537437125227002523.xml"}}
855852 15 {"sender":"15","type":"user_interac+on","data":"<?xmlversion= \"1.0\“encoding=\"UTF8\"?>\u000d\u000a <cbaloggingmodel:BuRonLogEntryxmlns:cbaloggingmodel=\"htp://www.sou con.de/cba/cbaloggingmodel\id=\"cbaBackBuRon_14_13454937565947\"/> \u000d\u000a„}":"15","type":"unloaded","data":""}
Example log
Promising methods • Sta:s:cs and visualisa:on • Clustering • Classifica:on
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 16
Process Measures
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 17
1. Number of page visits 2. Number of different page visits 3. Visit of relevant page 4. Time spent on the relevant page 5. Ra:o of :me spent on the relevant page 6. Ra:o of :me spent on the opening screen 7. Comple:on :me
Sta:s:cs
Process measures Mean Standard Devia+on
Number of page visits 5.18 3.13 Number of different page visits 2.38 1.38 Time spent on the relevant page 10.87 9.77 Ra:o of :me spent on the relevant page .14 .11 Ra:o of :me spent on the opening screen .64 .18 Comple:on :me 67.74 27.49
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 18
Distribu:on of different page visits
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 19
Distribu:on of different page visits
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 20
Visualisa:on
21 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-‐Scale Assessments
Visualisa:on
22 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-‐Scale Assessments
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits
Nr. of different page visits
Comple:on :me
Relevant page visited (Y/N)
Ra:o of :me on relevant page
Ra:o of :me on star:ng page
Distribu:on of sequences (%) 23% 16% 34% 27%
Ra:o of correct responses
23 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits
Nr. of different page visits
Comple:on :me
Relevant page visited (Y/N)
Ra:o of :me on relevant page
Ra:o of :me on star:ng page
Distribu:on of sequences (%) 23% 16% 34% 27%
Ra:o of correct responses 25.0% 1.7% 92.6% 84.4%
24 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits .72 5.41
Nr. of different page visits .36 2.54
Comple:on :me 28.38 62.08
Relevant page visited (Y/N) No No
Ra:o of :me on relevant page 0 0
Ra:o of :me on star:ng page .93 .60
Distribu:on of sequences (%) 23% 16%
Ra:o of correct responses 25.0% 1.7%
25 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits 4.73 9.57
Nr. of different page visits 2.29 4.53
Comple:on :me 59.60 91.60
Relevant page visited (Y/N) Yes Yes
Ra:o of :me on relevant page .20 .17
Ra:o of :me on star:ng page .62 .45
Distribu:on of sequences (%) 34% 27%
Ra:o of correct responses 92.6% 84.4%
26 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits 5.41 4.73
Nr. of different page visits 2.54 2.29
Comple:on :me 62.08 59.60
Relevant page visited (Y/N) No Yes
Ra:o of :me on relevant page 0 .20
Ra:o of :me on star:ng page .60 .62
Distribu:on of sequences (%) 16% 34%
Ra:o of correct responses 1.7% 92.6%
27 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits .72 5.41 4.73 9.57
Nr. of different page visits .36 2.54 2.29 4.53
Comple:on :me 28.38 62.08 59.60 91.60
Relevant page visited (Y/N) No No Yes Yes
Ra:o of :me on relevant page 0 0 .20 .17
Ra:o of :me on star:ng page .93 .60 .62 .45
Distribu:on of sequences (%) 23% 16% 34% 27%
Ra:o of correct responses 25.0% 1.7% 92.6% 84.4%
28 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
Summary and Conclusions
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 29
Summary and Conclusions
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits .72 5.41 4.73 9.57
Nr. of different page visits .36 2.54 2.29 4.53
Comple:on :me 28.38 62.08 59.60 91.60
Relevant page visited (Y/N) No No Yes Yes
Ra:o of :me on relevant page 0 0 .20 .17
Ra:o of :me on star:ng page .93 .60 .62 .45
Distribu:on of sequences (%) 23% 16% 34% 27%
Ra:o of correct responses 25.0% 1.7% 92.6% 84.4%
30 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
Future work • Pilot study – sample size • Valida:on
• Other types of items require new process measures
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 31
32
Future work • Pilot study – sample size • Valida:on
• Other types of items require new process measures • We have a lot of work to do:
• Souware developer • Test developer • Psychometricans
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 33
(Selected) Challenges
• Authoring and Management • Delivery • Re-‐use and exchange
TAO Days | 10.09.2012 Seite 34
(Selected) Challenges
• Authoring and Management • Delivery • Re-‐use and exchange
TAO Days | 10.09.2012 Seite 35
(Selected) Challenges
• Authoring and Management • Delivery • Re-‐use and exchange
TAO Days | 10.09.2012 Seite 36
(Selected) Challenges
• Authoring and Management • Delivery • Re-‐use and exchange
TAO Days | 10.09.2012 Seite 37