a framework and approaches to develop an in-house cat with freeware and open sources

Symposium 1A framework and approaches to develop an in-house CAT with freeware and open sources.

Tetsuo Kimura (Niigata Seiryo University)

Kyung (Chris) T. Han (Graduate Management Admission Council)

Michal Kosinski (University of Cambridge)

Kojiro Shojima (The National Center for University Entrance Examinations in Japan

CAT is greedy! CAT likes big pool!

The outline of the symposiumFramework to develop a CAT

(Thompson & Weiss, 2011)

Introduction of freeware and open sources for CAT

development

Approaches to develop an in-house CAT with freeware and

open sources

ExametrikaR package ltm

Moodle UCATConcerto

SimulCATR package catR

The framework to develop a CAT

Step Stage Primary work1 Feasibility, applicability,

And planning studiesMonte Carlo simulation; business case evaluation

2 Develop item bank content or utilize existing bank

Item writing and review

3 Pretest and calibrate item bank

Pretesting; item analysis

4 Determine specifications for final CAT

Post-hoc or hybrid simulations

5 Publish live CAT Publishing and distribution; software development

Framework Proposed by Thompson & Weiss (2011)

The three stages of CAT development

Pretesting & Item Analysis: Construction of Item Bank

Simulating CAT with Existing Item Bank: Determine specifications

Implementing CAT: Publishing a CAT on a software

ExametrikaR package ltm

Moodle UCATConcerto

SimulCATR package catR

Pretesting & Item Analysis: Construction of Item Bank

Pretesting

Item analysis: calibration, elimination of misfit &

equating

More pretests with new items and anchored items

Item bank

Calibrated items

Anchored items

Simulating CAT with Existing Item Bank: Determine specifications

Simulating CAT

Examining: Item selection rules,

Item exposure,Stopping rules, etc.

Determine CAT specifications

Item bank

Calibrated items

Implementing CAT: Publishing a CAT on a software

Specify CAT AlgorithmOn a CAT Software

Implementing CAT

Examine CAT Results

Item bank

Calibrated items

The outline of the symposiumFramework to develop a CAT

(Thompson & Weiss, 2011)

Introduction of freewares and open sources for CAT

development

Approaches to develop an in-house CAT with freewares and

open sources

ExametrikaR package ltm

Moodle UCATConcerto

SimulCATR package catR

The three stages of CAT development

Pretesting & Item Analysis: Construction of Item Bank

Simulating CAT with Existing Item Bank: Determine specifications

Implementing CAT: Publishing a CAT on a software

ExametrikaR package ltm

Moodle UCATConcerto

SimulCATR package catR

Exametrika

The three stages of CAT development

Pretesting & Item Analysis: Construction of Item Bank

Simulating CAT with Existing Item Bank: Determine specifications

Implementing CAT: Publishing a CAT on a software

ExametrikaR package ltm

Moodle UCATConcerto

SimulCATR package catR

R package: ltm

• ltm: Latent Trait Models under IRT– Dimitris Rizopoulos 　

• This R package provides a flexible framework for IRT analyses for dichotomous and polytomous data under a Marginal Maximum Likelihood approach. The fitting algorithms provide valid inferences under Missing At Random missing data mechanisms.http://rwiki.sciviews.org/doku.php?id=packages:cran:ltm

• ltm: An R Package for Latent Variable Modeling and Item Response Theory Analyses. 2006, Journal of Statistical Software, 17(5), 1-25. http://www.jstatsoft.org/v17/i05/

ltm: Available Features• Descriptives:

– samples proportions, missing values information, biserial correlation of items with total score, pairwise associations between items, Cronbach’s α, unidimensionality check using modified parallel analysis, nonparametric correlation coefficient, plotting.

• Dichotomous data: – Rasch Model, Two Parameter Logistic Model, Birnbaum’s

Three Parameter Model, and Latent Trait Model up to two latent variables (allowing also for nonlinear terms between the latent traits).

ltm: Available Features• Test Equating:

– Alternate Form Equating (where common and unique items are analyzed simultaneously) and Across Sample Equating (where different sets of unique items are analyzed separately based on previously calibrated anchor items).

• Plotting: – Item Characteristic Curves, Item Information Curves, Test

Information Functions, Standard Error of Measurement, Standardized Loadings Scatterplot (for the two-factor latent trait model), Item Operation Characteristic Curves (for ordinal polytomous data), Item Person Maps.

ltm: Available Features• Polytomous data:

– Graded Response Model and Generalized Partial Credit Model.

• Goodness-of-Fit: – Bootstrap Pearson χ2 for Rasch and Generalized Partial

Credit models, fit on the two- and three-way margins for all models, likelihood ratio tests between nested models (including AIC and BIC criteria values), and item- and person-fit statistics.

• Factor Scoring: – Empirical Bayes (i.e., posterior modes), Expected a Posteriori

(i.e., posterior means), Multiple Imputed Empirical Bayes, and Component Scores for dichotomous data.

ltm:examples

The outline of the symposium

Pretesting & Item Analysis: Construction of Item Bank

Simulating CAT with Existing Item Bank: Determine specifications

Implementing CAT: Publishing a CAT on a software

ExametrikaR package ltm

Moodle UCATConcerto

SimulCATR package catR

SimulCAT

The outline of the symposium

Pretesting & Item Analysis: Construction of Item Bank

Simulating CAT with Existing Item Bank: Determine specifications

Implementing CAT: Publishing a CAT on a software

ExametrikaR package ltm

Moodle UCATConcerto

SimulCATR package catR

R package: catR

• catR : Latent Trait Models under IRT– David Magis & Gilles Raîche

• This R package catR was developed to perform adaptive testing with as much flexibility as possible, in an attempt to provide a developmental and testing platform to the interested user.

• Random Generation of Response Patterns under Computerized Adaptive Testing with the R Package catR. Journal of Statistical Software, 48(8), 1-31. http://www.jstatsoft.org/v48/i08/.

catR: Available Features• The item bank can be provided by the user previously

calibrated according to the 4PL model or any simpler logistic model, or randomly generated from parent distributions of item parameters.

• The package proposes– several methods to select the early test items, several methods

for next item selection– different estimators of ability (maximum likelihood, Bayes modal,

expected a posteriori, weighted likelihood), – three stopping rules (based on the test length, the precision of

ability estimates or the classification of the examinee).

• The output can be graphically displayed.

catR:example

The outline of the symposium

Pretesting & Item Analysis: Construction of Item Bank

Simulating CAT with Existing Item Bank: Determine specifications

Implementing CAT: Publishing a CAT on a software

ExametrikaR package ltm

Moodle UCATConcerto

SimulCATR package catR

Moodle UCAT

UCAT: Rasch-based CAT program written in BASIC (Linacre, 1987)

http://www.rasch.org/memo69.pdf

Moodle UCAT: converted into PHP so that CATs can be administered on a major open source LMS, Moodle

(Kimura, Ohnishi & Nagaoka, 2012)

26

Development StatusCAT setting window

• Ending conditions• Logit to unit conversion• Logit bias

CAT administration window• Set item difficulty individually or category by category• Set student’s ability individually or as a whole

Administer CAT and provide result individuallyRetrieve CAT processes and results

Recalibration of item difficulty & estimate ability

Unit = Logit×10 + 100

Moodle UCAT beta ver.

Under Development for Ver.1 to be released in late August 2012

CAT Algorithm: Initial Ability Estimation

27

UCAT Moodle UCAT

Lower Limit (LL) =

　 AVG(D) － (0.5+0.5*RND)

Upper Limit (UL) = LL ＋ 1

B0 = AVG(D) － 0.5*RND AVG(D): average item difficulty RND: random value between 0 & 1 B0 : initial ability

Assign each student’s initial ability in the CAT administration window based on other test results or intelligently one by one, or as a whole.

CAT Algorithm: Ability (B) Estimation

28

UCAT / Moodle UCAT

the number of successes

probability of success of a student of ability Bm on the i-th dministered item of difficulty Di

)(

)(

1

11

1

)1(

DiBm

DiBm

mi

m

imimi

m

imim

mm

e

ep

PP

PRBB

:mR

:miP

CAT Algorithm: Standard Error (SE) Estimation

UCAT / Moodle UCAT

m

imimi

m

PPSE

1

1

)1(

1

CAT Algorithm: Item Selection

30

UCAT / Moodle UCATNext item will be selected randomly between LL and UL

score when he next (m-th) answer will be wrong

If no item found between LL & UL , use the closest.

m

imimi

m

imimi

m

imim

m

PpLLUL

Pp

PRBLL

1

1

11

)1(

1

)1(

:1mR

Ability estimate when the next answer will be wrong

Ability estimate when the next answer will be correct

CAT Algorithm: Ending ConditionUCAT / Moodle UCAT

Prescribed number of itemPrescribed SEBoth number of item and SEAll item

CAT Algorithm: Item Selection (logit bias)

32

Moodle UCATLL and UL can be adjusted by adding logit value to the Logit bias box in the CAT setting window

BiasULULBiased

BiasLLLLBiased

_

_

Positve logit value decrease the chance of answer correct

Negative logit value increase the chance of answer correct

Moodle UCAT demo

The outline of the symposium

Pretesting & Item Analysis: Construction of Item Bank

Simulating CAT with Existing Item Bank: Determine specifications

Implementing CAT: Publishing a CAT on a software

ExametrikaR package ltm

Moodle UCATConcerto

SimulCATR package catR

Concerto

Questions & Answers

• Tetsuo Kimura (Niigata Seiryo University)tetsuo.kmr<AT>gmail.com

• Kyung (Chris) T. Han (Graduate Management Admission Council)

khan<AT>gmac.com• Michal Kosinski (University of Cambridge)

mk583<AT>cam.ac.uk• Kojiro Shojima (The National Center for University

Entrance Examinations in Japan)shojima<AT>rd.dnc.ac.jp