A comparison of exposure control procedures in CATs using the 3PL model

Andrey Leroux Myriam LopezIan HembryBarbara Dodd

• The purpose of the study• Compare the progressive-restricted standard error, the randomesque, the Sympson-Hetter and no exposure control methods in computerized adaptive testing• Manipulated conditions:

• Item pool size• Stopping rules

• Criteria• Bias• RMSE• Item utility• Item overlap

• The application of computerized adaptive tests and computer-based tests has increased.• Reduce item exposure rates and increase item pool usage.

• Refers to constraining the administration of more popular items that would otherwise become compromised due to repeated administrations (Georgiadou, Triantafillow, & Economides, 2007)• Guarantees more variety in the items the examinees receive.

Variables related to the control of item exposure• Precision of measurement

• The degree that the CAT system with exposure controls estimates examinees’ abilities when compared to the examinees’ known abilities.• Exposure rate

• The number of times an item is administered to the total number of CATs administered.• Pool utilization

• The percentage of items not administered throughout any of the CAT administrations.• Test overlap

• The number of common items amongst the examinees.

Randomization strategies

• Randomly select an item for administration from a group of several items near the optimal level of maximum information.• 5-4-3-2-1 strategy (McBride & Martin, 1983)• Randomesque strategy (Kinsbury & Zara, 1989)

• Repeated selects the same number of the most informative items from which one is randomly selected for administration through testing.• Decrease the overlap in items seen by examinees of similar abilities.

Conditional strategies

• Specify a desired maximum value and use the exposure control parameters to control whether or not the item can be administered. • The values of these parameters have to be set through an iterative adjustment process in which at each step the effects of the previous adjustments are estimated using computer simulations of adaptive test administrations.

Sympson-Hetter (SH)

• Let t denote the iteration steps; P (t)(Ai|Si), the value of the control parameter for item i at step t; and P (t)(Si) and P (t)(Ai), the probabilities of selecting and administering item i at step t. If the simulation at step t is completed, P (t)(Si) and P (t)(Ai), are estimated, and for items for which the estimates of P(t)(Ai)do not meet the requirement, the values of the control parameters are adjusted.

Stratified strategies• When using maximum information, items with large a are more likely to be selected than those with small a values.• Stratify the item pool and are constrained to be administered from a given strata.• a-stratified procedure (Chang & Ying, 1999)

• Group items with similar a values and select within a group at each stage.• a-stratified with b-blocking (Chang, Qian, & Ying, 2001)

• Group items into M blocks in ascending order of b-parameter values. Then, each of the M blocks is stratified into K strata according to their a parameters.

Combined strategies

• Two or more than two procedures are combined.• Progressive-restricted procedure (PR) (Revuelta & Ponsoda, 1998)

• Progressive--Decide the maximum exposure rate (100k)%• Restricted—

s=h/m• Progressive-restricted standard error procedure (PR-SE)

• s: the ratio of stopping rule SE over the current SE.• Dichotomous 3PL model: Administer fewer items than PR, but yield similar correlations between estimated and known theta and low item overlap as PR.• Polytomous partial credit model: use most of the item pool.

Method• Study design

• Four exposure methods• Progressive-restricted standard error procedure (exposure rate =.3)• Sympton-Hetter (exposure rate =.3)• Randomesque (five items)• No exposure control

• Dichotomous 3PL model• 2 item pool sizes (300, 540 items)• Two stopping rules

• fixed-length with 50 items• Variable-length with .3 of the SE or a maximum of 50 items

• Item pools• Nine different test forms

• Each contained 60 items, so the large item pool has 540 items• Randomly selected 5 of the 9 forms for the small item pool

• Each test form contained 6 content areas• 24%, 16%, 15%, 15%, 23%, 7%

• Data generation• SAS macro IRTGEN• 200 datasets

• CAT simulations• Content balancing

• Kingsbury and Zara (1989) : the every next item is chosen from the item content group with the largest difference in percentage between desired and current.• EAP for ability estimation.

• Criteria• Bias, RMSE, item exposure rates, pool utilization and item overlap across test administrations.

Results

Since EAP was estimated, theta are shrink-aged, and the negative theta will be positively biased.

Discussion

• The PR-SE method yielded good precision, low non-administration rates, and reduction in item overlap.• Suggestions for use:

• Randomesque: minimal exposure control with high precision and test efficiency• SH: for more balance between precision and item exposure control • PR-SE: higher item exposure control

Comments

• Stratified strategies have been used in CAT. Why the authors stated these method are seldom used?• By definition, bias is the difference between the expected estimate and the parameter, and RMSE is the root of averaged squared difference between the estimate and the parameter. Therefore, Equation 2 should be called averaged residual, and Equation 3 called the root of squared averaged residual.• Barrada, Olea, Ponsoda, & Abad (2008) incorporated a quadratic form to have a nonlinear function. The purpose of nonlinear s is to control the contribution from a random component and to make information component more flexible.

Future studies

• Investigate the impact of different distributions of ability• Compare to other exposure control procedures• The PR combined with the different stopping rule• The PR-SE method should be compared with contemporary ones like on-the-fly SH method, omega method proposed by Chen (2011).• It is possible to incorporate a nonlinear s to the Progressive-restricted standard error procedure (PR-SE). The s parameter in PR-SE method is defined as the ratio of stopping rule SE over the current SE when administrating an item.

• Another possible idea is to apply PR method and PR-SE method to CDM-CAT.