eat: an r package for automation of data preparation and irt...
TRANSCRIPT
![Page 1: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/1.jpg)
eat: An R Package for Automation of Data Preparationand IRT Modeling
Karoline Sachse, Martin Hecht, Sebastian Weirich,Nicole Haag, Malte Jansen, Sebastian Wurster,
Christiane Penk, Anna Lenski, Thilo Siegle
Institute for Educational Quality ImprovementHumboldt-University, Berlin
February 10, 2012
Psychoco, Innsbruck February 10, 2012 1 / 26
![Page 2: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/2.jpg)
Overview
1 eat - HistoryThe Institute for Educational Quality ImprovementACER ConQuestThe Idea
2 eat - ConceptOverviewData
3 eat - ExamplesData PreparationUnidimensional 1PL model with automateModelsGrouping options in automateModelsautomateModels
4 DiscussionOutlook
Psychoco, Innsbruck February 10, 2012 2 / 26
![Page 3: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/3.jpg)
eat - History The Institute for Educational Quality Improvement
The Institute
Independent research and test institute founded by the 16 federalstates in 2004
Nationwide Educational Standards Assessments in German, the firstforeign language, Mathematics and Science which allow comparisonof federal states (N ≈ 30, 000)
Assessment tests in German, Mathematics and the first foreignlanguage in the 8th grade at secondary school (once a year)
Assessment tests in German and Mathematics in the 3rd grade atprimary school (once a year)
Psychoco, Innsbruck February 10, 2012 3 / 26
![Page 4: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/4.jpg)
eat - History ACER ConQuest
ConQuest
Commercial Software developed by ACER (Wu, Adams & Wilson,1997)
Major scaling tool of the Organisation for Economic Co-operation andDevelopment’s Programme for International Student Assessment(PISA)
Fits a large number of different item response models
Rasch, partial credit, rating scale, facets, ...Latent regressionMultidimensionality
Estimation
Marginal Maximum LikelihoodGaussian quadrature/ Monte Carlo approximationsPerson parameter estimation: EAP, MLE, WLE, Plausible values
Psychoco, Innsbruck February 10, 2012 4 / 26
![Page 5: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/5.jpg)
eat - History The Idea
Automation of Data Preparation and Analysis
automate data preparation1 read in & check SPSS-files2 merge data frames (booklets)3 recode & dichotomize data
automate IRT calibration1 write ConQuest syntax, generate appropriate data input2 execute ConQuest3 read in ConQuest output
facilitate reporting1 write out results (graphics, tables, ...)
⇒ ”eat” (”Educational Assessment Tools”)
Psychoco, Innsbruck February 10, 2012 5 / 26
![Page 6: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/6.jpg)
eat - Concept Overview
Implemented Modules
Psychoco, Innsbruck February 10, 2012 6 / 26
![Page 7: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/7.jpg)
eat - Concept Overview
Wrapping ConQuest
Psychoco, Innsbruck February 10, 2012 7 / 26
![Page 8: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/8.jpg)
eat - Concept Data
Typical Items
Psychoco, Innsbruck February 10, 2012 8 / 26
![Page 9: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/9.jpg)
eat - Concept Data
Typical Items - Scores
Psychoco, Innsbruck February 10, 2012 9 / 26
![Page 10: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/10.jpg)
eat - Concept Data
Data Structure
Psychoco, Innsbruck February 10, 2012 10 / 26
![Page 11: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/11.jpg)
eat - Examples Data Preparation
automateDataPreparation
dataset <- automateDataPreparation ( inputList = inputList, path = path,
loadSav = TRUE,
checkData = TRUE,
mergeData = TRUE,
recodeData = TRUE,
aggregateData = TRUE,
scoreData = TRUE,
writeSpss = TRUE )
Psychoco, Innsbruck February 10, 2012 11 / 26
![Page 12: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/12.jpg)
eat - Examples Data Preparation
Data Preparation Input
Psychoco, Innsbruck February 10, 2012 12 / 26
![Page 13: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/13.jpg)
eat - Examples Data Preparation
Data Preparation Logfile
Psychoco, Innsbruck February 10, 2012 13 / 26
![Page 14: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/14.jpg)
eat - Examples Unidimensional 1PL model with automateModels
Simple Unidimensional 1PL Model
results01 <- automateModels( dataset = dataset , folder = folder )
Psychoco, Innsbruck February 10, 2012 14 / 26
![Page 15: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/15.jpg)
eat - Examples Unidimensional 1PL model with automateModels
ConQuest Dataset & Label File Creation
Psychoco, Innsbruck February 10, 2012 15 / 26
![Page 16: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/16.jpg)
eat - Examples Unidimensional 1PL model with automateModels
ConQuest Syntax Creation
Psychoco, Innsbruck February 10, 2012 16 / 26
![Page 17: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/17.jpg)
eat - Examples Unidimensional 1PL model with automateModels
ConQuest Run
ConQuest runs due to automatic creation and execution of batch files
Psychoco, Innsbruck February 10, 2012 17 / 26
![Page 18: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/18.jpg)
eat - Examples Unidimensional 1PL model with automateModels
ConQuest Output Files
Item parameter estimates
.shw, .itn, ...
Person parameter estimates
.wle, .mle, .eap, .pvl, ...
⇒ Many different output files
Psychoco, Innsbruck February 10, 2012 18 / 26
![Page 19: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/19.jpg)
eat - Examples Unidimensional 1PL model with automateModels
ConQuest Item Parameter Output
Psychoco, Innsbruck February 10, 2012 19 / 26
![Page 20: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/20.jpg)
eat - Examples Unidimensional 1PL model with automateModels
ConQuest Person Parameter Output
Psychoco, Innsbruck February 10, 2012 20 / 26
![Page 21: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/21.jpg)
eat - Examples Unidimensional 1PL model with automateModels
eat Reporting
Item parameter estimates
Person parameter estimates
Psychoco, Innsbruck February 10, 2012 21 / 26
![Page 22: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/22.jpg)
eat - Examples Unidimensional 1PL model with automateModels
eat Log
all objects (dataset, item.grouping, ...) will be archived into an.RData file
an INFO file will be created
Psychoco, Innsbruck February 10, 2012 22 / 26
![Page 23: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/23.jpg)
eat - Examples Grouping options in automateModels
Multidimensional vs. Unidimensional Analysis
results02 <- automateModels( dataset = dataset , id = "id" , folder = folder ,
item.grouping = item.grouping ,
select.item.group = c ( "ER" , "EL" ) )
results03 <- automateModels( dataset = dataset , id = "id" , folder = folder ,
item.grouping = item.grouping ,
select.item.group = c ( "ER" , "EL" ) , cross="all")
Psychoco, Innsbruck February 10, 2012 23 / 26
![Page 24: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/24.jpg)
eat - Examples Grouping options in automateModels
Person groups & weights
dataset <- cbind ( dataset , "weight1" = as.character(sample(c(0.8, 1, 1.2),
nrow(dataset), replace=TRUE)), "weight2" = as.character(sample(c(1),
nrow(dataset), replace=TRUE)), stringsAsFactors = FALSE )
results04 <- automateModels( dataset = dataset, folder = folder
context.vars = c ( "weight1" , "weight2" ) ,
item.grouping = item.grouping ,
select.item.group = "ER" ,
person.grouping = person.grouping ,
select.person.group = list ( "gr.9" , "gr.10" ) ,
weight = list ( "weight1" , "weight2" ) )
Psychoco, Innsbruck February 10, 2012 24 / 26
![Page 25: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/25.jpg)
eat - Examples automateModels
automateModels – Overview
automateModels(dataset, id = NULL, context.vars = NULL, items = NULL,
item.grouping = NULL, select.item.group = NULL,
person.grouping.vars = NULL, person.grouping.vars.include.all = FALSE,
person.grouping = NULL, select.person.group = NULL,
additional.item.props = NULL, folder, overwrite.folder = TRUE,
analyse.name.prefix = NULL, analyse.name = NULL,
analyse.name.elements = NULL, data.name = NULL, m.model = NULL,
software = NULL, dif = NULL, weight = NULL, anchor = NULL,
regression = NULL, adjust.for.regression = FALSE, q3 = FALSE,
missing.rule = NULL, cross = NULL, subfolder.order = NULL,
subfolder.mode = NULL, additionalSubFolder = NULL, run.mode = NULL,
n.batches = NULL, run.timeout = 1440, run.status.refresh = 0.2,
email = NULL, smtpServer = NULL, write.txt.dataset = FALSE,
delete.folder.countdown = 5, conquestParameters = NULL )
Psychoco, Innsbruck February 10, 2012 25 / 26
![Page 26: eat: An R Package for Automation of Data Preparation and IRT …eat.r-forge.r-project.org/other/TalkEat2012.pdf · 2012-02-14 · eat: An R Package for Automation of Data Preparation](https://reader031.vdocuments.us/reader031/viewer/2022040600/5e8c1fdd40abd42d243631b6/html5/thumbnails/26.jpg)
Discussion Outlook
Thank you
Thank you for your attention!
http://r-forge.r-project.org/eat
Special thanks to
Alexander Robitzsch (Measurement Statistician, bifie)Martin Mechtel (IT Director, IQB)
Psychoco, Innsbruck February 10, 2012 26 / 26