advancements in large-scale assessment heiko rölke & krisztina toth dipf - deutsches institut...
TRANSCRIPT
![Page 1: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/1.jpg)
Advancements in Large-scale Assessment
Heiko Rölke & Krisztina TothDIPF - Deutsches Institut für Internationale
Pädagogische ForschungFrankfurt am Main
TAO Days | 10.09.2012Seite
1
![Page 2: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/2.jpg)
Overview
• Trends in LSA• Closer Look at Trends• Example: Making Use of Data• Challenges in LSA
TAO Days| 10.09.2012 Seite 2
![Page 3: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/3.jpg)
Trends
Main trend: CBA
Sub-trends/effects of CBA:•CAT• towards formative assessment
•Complex items• Simulation, interconnection
•Interweaving• different tests, questionnaires, etc.
•Big data
TAO Days | 10.09.2012 Seite 3
![Page 4: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/4.jpg)
Trend: CBA
Computer-based Assessment has to serve a purpose•Costs?•Time!•Validity!
TAO Days | 10.09.2012 Seite 4
![Page 5: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/5.jpg)
CAT
• Also not practiced for its own sake• Not as important in pure summative assessment• But: • Strict time limits (e.g. PIAAC)• Growing demand for formative aspects in summative
assessment
TAO Days | 10.09.2012 Seite 5
![Page 6: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/6.jpg)
Complex Items
• Closer at reality• SimulationMost important:• Ability to assess new domains
TAO Days | 10.09.2012 Seite 6
![Page 7: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/7.jpg)
Examples of Complex Items Simulation of an “Email / Web Scenario”
The test person receives an email and should bookcinema tickets online.
Dynamic Model (MicroDYN)The test person should explore and master a dynamicsystem with input (exogenous) variables influencingoutput (endogenous) variables.
Automaton with Finite State Machine (MicroFIN)The test person should interact with a mobile phone and set time tosummertime.
![Page 8: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/8.jpg)
Interweaving of Tests and Questionnaires• Framing• test only if preconditions are fulfilled
• Double-check • findings of questionnaire
TAO Days | 10.09.2012 Seite 8
![Page 9: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/9.jpg)
Big Data
• Log data, not only results• Find out what is going on• Make use of data• E.g. partial scoring
• -> Examples from PIAAC PS-TRE
TAO Days | 10.09.2012 Seite 9
![Page 10: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/10.jpg)
Simulation-based assessment
• Modern assessments:• real life situations: e.g. web environments• complexity of instrument – various ways
• Monitoring students‘ actions - log files• Analysis of test-taking paths is a relatively new field in education
(reason: printed vs. computer-based data collection)• Process data requires to consider and evaluate methods
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
10
![Page 11: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/11.jpg)
Aim
• to investigate how log data can be integrated into the process of educational assessment and evaluation to support researchers and practitioners in making use of the data assembled in computer-based test delivery
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
11
![Page 12: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/12.jpg)
Aim
• to investigate how log data can be integrated into the process of educational assessment and evaluation to support researchers and practitioners in making use of the data assembled in computer-based test delivery
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
12
![Page 13: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/13.jpg)
Hypertext Item
13
![Page 14: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/14.jpg)
14
![Page 15: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/15.jpg)
15
829709 15 {"sender":"15","type":"loading","data":""}833463 15 {"sender":"15","type":"loaded","data":""}848153 15 {"sender":"15","type":"user_interaction","":"<?xml version=\"1.0\">
\u000d\u000a<cbaloggingmodel:EmbeddedLinkLogEntryxmlns:cbaloggingmodel=\"http://www.softcon.de/cba/cbaloggingmodel\" id=\„-b576759f1:-7fed\" sourcePageId=\"Item15_linklist\" targetPageId=\"Item15_website1\" textFieldId=\"cbaTextField_71"/>\u000d\u000a"}849094 15 {"sender":"15","type":"variable_change","data":{"name":
"snapshot_url„,"value":"http://localhost:8101/cba-runtime/itemjsessionid=LB1?custom_servicehandler=downloadService&file=C:\\...snapshot5537437125227002523.xml"}}
855852 15 {"sender":"15","type":"user_interaction","data":"<?xmlversion=\"1.0\“encoding=\"UTF8\"?>\u000d\u000a<cbaloggingmodel:ButtonLogEntryxmlns:cbaloggingmodel=\"http://www.soft
con.de/cba/cbaloggingmodel\id=\"cbaBackButton_14_13454937565947\"/>\u000d\u000a„}":"15","type":"unloaded","data":""}
Example log
![Page 16: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/16.jpg)
Promising methods
• Statistics and visualisation• Clustering• Classification
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
16
![Page 17: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/17.jpg)
Process Measures
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
17
1. Number of page visits2. Number of different page visits3. Visit of relevant page4. Time spent on the relevant page5. Ratio of time spent on the relevant page6. Ratio of time spent on the opening screen7. Completion time
![Page 18: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/18.jpg)
Statistics
Process measures MeanStandard Deviation
Number of page visits 5.18 3.13Number of different page visits 2.38 1.38Time spent on the relevant page 10.87 9.77Ratio of time spent on the relevant page .14 .11Ratio of time spent on the opening screen .64 .18Completion time 67.74 27.49
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
18
![Page 19: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/19.jpg)
Distribution of different page visits
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
19
![Page 20: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/20.jpg)
Distribution of different page visits
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
20
![Page 21: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/21.jpg)
Visualisation
21Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 22: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/22.jpg)
Visualisation
22Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 23: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/23.jpg)
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits
Nr. of different page visits
Completion time
Relevant page visited (Y/N)
Ratio of time on relevant page
Ratio of time on starting page
Distribution of sequences (%) 23% 16% 34% 27%
Ratio of correct responses
23
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 24: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/24.jpg)
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits
Nr. of different page visits
Completion time
Relevant page visited (Y/N)
Ratio of time on relevant page
Ratio of time on starting page
Distribution of sequences (%) 23% 16% 34% 27%
Ratio of correct responses 25.0% 1.7% 92.6% 84.4%
24
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 25: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/25.jpg)
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits .72 5.41
Nr. of different page visits .36 2.54
Completion time 28.38 62.08
Relevant page visited (Y/N) No No
Ratio of time on relevant page 0 0
Ratio of time on starting page .93 .60
Distribution of sequences (%) 23% 16%
Ratio of correct responses 25.0% 1.7%
25
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 26: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/26.jpg)
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits 4.73 9.57
Nr. of different page visits 2.29 4.53
Completion time 59.60 91.60
Relevant page visited (Y/N) Yes Yes
Ratio of time on relevant page .20 .17
Ratio of time on starting page .62 .45
Distribution of sequences (%) 34% 27%
Ratio of correct responses 92.6% 84.4%
26
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 27: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/27.jpg)
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits 5.41 4.73
Nr. of different page visits 2.54 2.29
Completion time 62.08 59.60
Relevant page visited (Y/N) No Yes
Ratio of time on relevant page 0 .20
Ratio of time on starting page .60 .62
Distribution of sequences (%) 16% 34%
Ratio of correct responses 1.7% 92.6%
27
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 28: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/28.jpg)
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits .72 5.41 4.73 9.57
Nr. of different page visits .36 2.54 2.29 4.53
Completion time 28.38 62.08 59.60 91.60
Relevant page visited (Y/N) No No Yes Yes
Ratio of time on relevant page 0 0 .20 .17
Ratio of time on starting page .93 .60 .62 .45
Distribution of sequences (%) 23% 16% 34% 27%
Ratio of correct responses 25.0% 1.7% 92.6% 84.4%
28
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 29: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/29.jpg)
Summary and Conclusions
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
29
![Page 30: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/30.jpg)
Summary and Conclusions
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits .72 5.41 4.73 9.57
Nr. of different page visits .36 2.54 2.29 4.53
Completion time 28.38 62.08 59.60 91.60
Relevant page visited (Y/N) No No Yes Yes
Ratio of time on relevant page 0 0 .20 .17
Ratio of time on starting page .93 .60 .62 .45
Distribution of sequences (%) 23% 16% 34% 27%
Ratio of correct responses 25.0% 1.7% 92.6% 84.4%
30
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 31: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/31.jpg)
Future work
• Pilot study – sample size• Validation
• Other types of items require new process measures
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
31
![Page 32: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/32.jpg)
32
![Page 33: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/33.jpg)
Future work
• Pilot study – sample size• Validation
• Other types of items require new process measures• We have a lot of work to do:• Software developer• Test developer• Psychometricans
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
33
![Page 34: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/34.jpg)
(Selected) Challenges
• Authoring and Management• Delivery• Re-use and exchange
TAO Days | 10.09.2012 Seite 34
![Page 35: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/35.jpg)
(Selected) Challenges
• Authoring and Management• Delivery• Re-use and exchange
TAO Days | 10.09.2012 Seite 35
![Page 36: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/36.jpg)
(Selected) Challenges
• Authoring and Management• Delivery• Re-use and exchange
TAO Days | 10.09.2012 Seite 36
![Page 37: Advancements in Large-scale Assessment Heiko Rölke & Krisztina Toth DIPF - Deutsches Institut für Internationale Pädagogische Forschung Frankfurt am Main](https://reader033.vdocuments.us/reader033/viewer/2022051515/551436435503466d1a8b52ba/html5/thumbnails/37.jpg)
(Selected) Challenges
• Authoring and Management• Delivery• Re-use and exchange
TAO Days | 10.09.2012 Seite 37