a framework and approaches to develop an in-house cat with freeware and open sources
DESCRIPTION
Symposium at IACAT 2012 SydneyTRANSCRIPT
Symposium 1A framework and approaches to develop an in-house CAT with freeware and open sources.
Tetsuo Kimura (Niigata Seiryo University)
Kyung (Chris) T. Han (Graduate Management Admission Council)
Michal Kosinski (University of Cambridge)
Kojiro Shojima (The National Center for University Entrance Examinations in Japan
CAT is greedy! CAT likes big pool!
The outline of the symposiumFramework to develop a CAT
(Thompson & Weiss, 2011)
Introduction of freeware and open sources for CAT
development
Approaches to develop an in-house CAT with freeware and
open sources
ExametrikaR package ltm
Moodle UCATConcerto
SimulCATR package catR
The framework to develop a CAT
Step Stage Primary work1 Feasibility, applicability,
And planning studiesMonte Carlo simulation; business case evaluation
2 Develop item bank content or utilize existing bank
Item writing and review
3 Pretest and calibrate item bank
Pretesting; item analysis
4 Determine specifications for final CAT
Post-hoc or hybrid simulations
5 Publish live CAT Publishing and distribution; software development
Framework Proposed by Thompson & Weiss (2011)
The three stages of CAT development
Pretesting & Item Analysis: Construction of Item Bank
Simulating CAT with Existing Item Bank: Determine specifications
Implementing CAT: Publishing a CAT on a software
ExametrikaR package ltm
Moodle UCATConcerto
SimulCATR package catR
Pretesting & Item Analysis: Construction of Item Bank
Pretesting
Item analysis: calibration, elimination of misfit &
equating
More pretests with new items and anchored items
Item bank
Calibrated items
Anchored items
Simulating CAT with Existing Item Bank: Determine specifications
Simulating CAT
Examining: Item selection rules,
Item exposure,Stopping rules, etc.
Determine CAT specifications
Item bank
Calibrated items
Implementing CAT: Publishing a CAT on a software
Specify CAT AlgorithmOn a CAT Software
Implementing CAT
Examine CAT Results
Item bank
Calibrated items
The outline of the symposiumFramework to develop a CAT
(Thompson & Weiss, 2011)
Introduction of freewares and open sources for CAT
development
Approaches to develop an in-house CAT with freewares and
open sources
ExametrikaR package ltm
Moodle UCATConcerto
SimulCATR package catR
The three stages of CAT development
Pretesting & Item Analysis: Construction of Item Bank
Simulating CAT with Existing Item Bank: Determine specifications
Implementing CAT: Publishing a CAT on a software
ExametrikaR package ltm
Moodle UCATConcerto
SimulCATR package catR
Exametrika
The three stages of CAT development
Pretesting & Item Analysis: Construction of Item Bank
Simulating CAT with Existing Item Bank: Determine specifications
Implementing CAT: Publishing a CAT on a software
ExametrikaR package ltm
Moodle UCATConcerto
SimulCATR package catR
R package: ltm
• ltm: Latent Trait Models under IRT– Dimitris Rizopoulos
• This R package provides a flexible framework for IRT analyses for dichotomous and polytomous data under a Marginal Maximum Likelihood approach. The fitting algorithms provide valid inferences under Missing At Random missing data mechanisms.http://rwiki.sciviews.org/doku.php?id=packages:cran:ltm
• ltm: An R Package for Latent Variable Modeling and Item Response Theory Analyses. 2006, Journal of Statistical Software, 17(5), 1-25. http://www.jstatsoft.org/v17/i05/
ltm: Available Features• Descriptives:
– samples proportions, missing values information, biserial correlation of items with total score, pairwise associations between items, Cronbach’s α, unidimensionality check using modified parallel analysis, nonparametric correlation coefficient, plotting.
• Dichotomous data: – Rasch Model, Two Parameter Logistic Model, Birnbaum’s
Three Parameter Model, and Latent Trait Model up to two latent variables (allowing also for nonlinear terms between the latent traits).
ltm: Available Features• Test Equating:
– Alternate Form Equating (where common and unique items are analyzed simultaneously) and Across Sample Equating (where different sets of unique items are analyzed separately based on previously calibrated anchor items).
• Plotting: – Item Characteristic Curves, Item Information Curves, Test
Information Functions, Standard Error of Measurement, Standardized Loadings Scatterplot (for the two-factor latent trait model), Item Operation Characteristic Curves (for ordinal polytomous data), Item Person Maps.
ltm: Available Features• Polytomous data:
– Graded Response Model and Generalized Partial Credit Model.
• Goodness-of-Fit: – Bootstrap Pearson χ2 for Rasch and Generalized Partial
Credit models, fit on the two- and three-way margins for all models, likelihood ratio tests between nested models (including AIC and BIC criteria values), and item- and person-fit statistics.
• Factor Scoring: – Empirical Bayes (i.e., posterior modes), Expected a Posteriori
(i.e., posterior means), Multiple Imputed Empirical Bayes, and Component Scores for dichotomous data.
ltm:examples
The outline of the symposium
Pretesting & Item Analysis: Construction of Item Bank
Simulating CAT with Existing Item Bank: Determine specifications
Implementing CAT: Publishing a CAT on a software
ExametrikaR package ltm
Moodle UCATConcerto
SimulCATR package catR
SimulCAT
The outline of the symposium
Pretesting & Item Analysis: Construction of Item Bank
Simulating CAT with Existing Item Bank: Determine specifications
Implementing CAT: Publishing a CAT on a software
ExametrikaR package ltm
Moodle UCATConcerto
SimulCATR package catR
R package: catR
• catR : Latent Trait Models under IRT– David Magis & Gilles Raîche
• This R package catR was developed to perform adaptive testing with as much flexibility as possible, in an attempt to provide a developmental and testing platform to the interested user.
• Random Generation of Response Patterns under Computerized Adaptive Testing with the R Package catR. Journal of Statistical Software, 48(8), 1-31. http://www.jstatsoft.org/v48/i08/.
catR: Available Features• The item bank can be provided by the user previously
calibrated according to the 4PL model or any simpler logistic model, or randomly generated from parent distributions of item parameters.
• The package proposes– several methods to select the early test items, several methods
for next item selection– different estimators of ability (maximum likelihood, Bayes modal,
expected a posteriori, weighted likelihood), – three stopping rules (based on the test length, the precision of
ability estimates or the classification of the examinee).
• The output can be graphically displayed.
catR:example
The outline of the symposium
Pretesting & Item Analysis: Construction of Item Bank
Simulating CAT with Existing Item Bank: Determine specifications
Implementing CAT: Publishing a CAT on a software
ExametrikaR package ltm
Moodle UCATConcerto
SimulCATR package catR
Moodle UCAT
UCAT: Rasch-based CAT program written in BASIC (Linacre, 1987)
http://www.rasch.org/memo69.pdf
Moodle UCAT: converted into PHP so that CATs can be administered on a major open source LMS, Moodle
(Kimura, Ohnishi & Nagaoka, 2012)
26
Development StatusCAT setting window
• Ending conditions• Logit to unit conversion• Logit bias
CAT administration window• Set item difficulty individually or category by category• Set student’s ability individually or as a whole
Administer CAT and provide result individuallyRetrieve CAT processes and results
Recalibration of item difficulty & estimate ability
Unit = Logit×10 + 100
Moodle UCAT beta ver.
Under Development for Ver.1 to be released in late August 2012
CAT Algorithm: Initial Ability Estimation
27
UCAT Moodle UCAT
Lower Limit (LL) =
AVG(D) - (0.5+0.5*RND)
Upper Limit (UL) = LL + 1
B0 = AVG(D) - 0.5*RND AVG(D): average item difficulty RND: random value between 0 & 1 B0 : initial ability
Assign each student’s initial ability in the CAT administration window based on other test results or intelligently one by one, or as a whole.
CAT Algorithm: Ability (B) Estimation
28
UCAT / Moodle UCAT
the number of successes
probability of success of a student of ability Bm on the i-th dministered item of difficulty Di
)(
)(
1
11
1
)1(
DiBm
DiBm
mi
m
imimi
m
imim
mm
e
ep
PP
PRBB
:mR
:miP
CAT Algorithm: Standard Error (SE) Estimation
UCAT / Moodle UCAT
m
imimi
m
PPSE
1
1
)1(
1
CAT Algorithm: Item Selection
30
UCAT / Moodle UCATNext item will be selected randomly between LL and UL
score when he next (m-th) answer will be wrong
If no item found between LL & UL , use the closest.
m
imimi
m
imimi
m
imim
m
PpLLUL
Pp
PRBLL
1
1
11
)1(
1
)1(
:1mR
Ability estimate when the next answer will be wrong
Ability estimate when the next answer will be correct
CAT Algorithm: Ending ConditionUCAT / Moodle UCAT
Prescribed number of itemPrescribed SEBoth number of item and SEAll item
CAT Algorithm: Item Selection (logit bias)
32
Moodle UCATLL and UL can be adjusted by adding logit value to the Logit bias box in the CAT setting window
BiasULULBiased
BiasLLLLBiased
_
_
Positve logit value decrease the chance of answer correct
Negative logit value increase the chance of answer correct
Moodle UCAT demo
The outline of the symposium
Pretesting & Item Analysis: Construction of Item Bank
Simulating CAT with Existing Item Bank: Determine specifications
Implementing CAT: Publishing a CAT on a software
ExametrikaR package ltm
Moodle UCATConcerto
SimulCATR package catR
Concerto
Questions & Answers
• Tetsuo Kimura (Niigata Seiryo University)tetsuo.kmr<AT>gmail.com
• Kyung (Chris) T. Han (Graduate Management Admission Council)
khan<AT>gmac.com• Michal Kosinski (University of Cambridge)
mk583<AT>cam.ac.uk• Kojiro Shojima (The National Center for University
Entrance Examinations in Japan)shojima<AT>rd.dnc.ac.jp