computerized adaptive testing for the singapore employability
Post on 12-Sep-2021
2 Views
Preview:
TRANSCRIPT
Computerized Adaptive Testing for the
Singapore Employability Skills System
Patricia Rickard – CASAS
Linda Taylor - CASAS
Richard Ackermann – Team Code
Jim Olsen – Alpine Testing Solutions
Background
• The Singapore Workforce Development
Agency plays a key role in implementation
of Employability Skills System (ESS)
“to enhance the employability and
competitiveness of employees and job
seekers, building a workforce to meet
changing needs of Singapore’s economy.”
The Singapore Employability Skills System
(ESS) aims to …• equip the workforce with generic and portable skills that
will enable workers to better adapt to new job demands,
work challenges and changing work environment
• help lower skilled and low waged workers (LWWs) gain
and improve their jobs through higher skills
• enhance recognition of ESS as an alternate entry level
qualification for the 504,800 Singaporean workers
without “N” and “O” level qualifications
Objectives
Understand how the ESS system
– meets the time constraints of workers
– uses CAT and CBT together with human scoring
for a battery of tests
– serves walk-ins, businesses, remote testing
locations and prisons
– offers assessment in multiple deployments
Objectives
• Understand challenges in customizing
computer delivered assessments for
adults with low to high literacy skills
CASAS developed and
customized for Singapore • CAT tests for
– math
– reading
– listening
• Computer-based tests for
– writing
– speaking
7
Level 1 Pre-Beginning
Level 2 Beginning
Level 3 High Beginning
Level 4 Low Intermediate
Level 5 High Intermediate
Level 6 Advanced
Level 7 High Advanced
Level 8 Proficient
Nil
SOA 1
SOA 2
SOA 3
SOA 4
SOA 5
SOA 6
SOA 7
SOA 8Test Score
250
245
235
220
210
200
190
180
150
WPLN
Assessment
Statement of
Attainment
WPLN Training
N
R L
WS
WPLN training modules are designed to raise
a learner’s proficiency gradually
Test Administration Environment
• Secure proctored locations
– Assessment Center
– Business Sites
– Prisons
• Test Security
• Multiple proctors
Demonstration
• Sample test items
– Multiple choice
– Performance-based
• Test delivery and administration system
• Scoring
– Computer
– Human
A to teach young people
how to stay healthy
C to provide nutritious meals for
young people
B to identify schools with
unhealthy conditions
D to interest teenagers in health
care careers
2. What is the purpose of the new project?
CAT Psychometric Procedures
• Selection of initial item
• Rasch model calibration and proficiency
estimation
• Stopping rule based on multiple criteria
• Review of test score results and
psychometric analyses
CAT Mechanism
00.511.522.5
2121.52222.52323.524
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
SE
Ab
ilit
y
Items
18 Items
Item correct
Item incorrect
SE
Ability
Score Distribution-Reading N=12,625
0
50
100
150
200
250
300
13
6
15
9
16
7
17
0
17
4
17
7
18
0
18
3
18
6
18
9
19
2
19
5
19
8
20
1
20
4
20
7
21
0
21
3
21
6
21
9
22
2
22
5
22
8
23
1
23
4
23
7
24
0
24
3
24
6
24
9
25
2
25
5
25
8
26
1
26
4
27
1
Fre
qu
en
cy
Score
SOA = 2 SOA = 3 SOA = 4 SOA = 5 SOA = 6 SOA = 7 SOA = 8SOA = 1SOA = 0
Score Distribution-Math N=10,095
SOA = 0 SOA = 2 SOA = 3 SOA = 4 SOA = 5 SOA = 6 SOA = 7 SOA = 8SOA = 1
Descriptive Statistics
Descriptive Statistics N Minimum Maximum Mean Std. Error Std. Deviation
Scaled Score 12,625 139 270 220.0 0.15 17.03
Starting Ability 12,625 195 205 199.48 0.01 19.91
Final Ability (at exit) 12,625 139.39 269.72 219.97 0.02 17.03
Starting SE 12,625 2.00 2.66 2.08 0.00 0.08
Final SE 12,625 0.48 3.57 0.51 0.00 0.07
Descriptive Statistics N Minimum Maximum Mean Std. Error Std. Deviation
Scaled Score 10,095 136 274 218.9 0.19 19.52
Starting Ability 10,095 195 205 198.03 0.01 6.82
Final Ability 10,095 135.69 273.74 218.88 0.02 19.51
Starting SE 10,095 2.00 3.03 2.08 0.00 0.08
Final SE 10,095 0.48 3.25 0.52 0.00 0.08
Reading
Math
Mean Reading Ability Estimate N=12,625
185
190
195
200
205
210
215
220
225
230
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Ab
ility
Sco
re
Items
Mean
Reading Standard Error (SE) N=12,625
0
0.5
1
1.5
2
2.5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
SE
Items
SE
Mean Math Ability Estimate N=10,095
180
190
200
210
220
230
240
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Ab
ilit
y Sc
ore
Items
Mean
Math Standard Error (SE) N=10,095
0
0.5
1
1.5
2
2.5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
SE
Items
SE
Reading RIT Value and Measure
Correlation is .943 between Singapore Reading Item Rasch
Calibration RITS and the Rasch Item Measure computed from the
concurrent calibration of the CAT item vectors.
-6
-4
-2
0
2
4
6
180 190 200 210 220 230 240 250 260
Measu
re
RIT Value
Math RIT Value and Measure
-8.00
-6.00
-4.00
-2.00
0.00
2.00
4.00
6.00
8.00
10.00
170 180 190 200 210 220 230 240 250 260 270
Mea
sure
RIT Value
Correlation is .968 between Singapore Math Item Rasch Calibration RITS
and the Rasch Item Measure computed from the concurrent calibration of
the CAT item vectors.
Stopping Rule – Reading N=12,625
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
120.0%
139
162
171
175
179
183
187
191
195
199
203
207
211
215
219
223
227
231
235
239
243
247
251
255
259
263
Score
20 Items max.
SE <0.5
Stopping Rule – Math N=10,095
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
120.0%136
165
169
174
178
182
186
190
194
198
202
206
210
214
218
222
226
230
234
238
242
246
250
254
258
262
267
Score
20 Items max.
SE <0.5
top related