changing issues, changing questions, changing approaches: an overview of research in the alternative...
TRANSCRIPT
Changing issues, changing questions, changing approaches
An overview of research in the Alternative Admissions Research Project
The late 1980s to about 2001
The main questions were about:
• Access• Finding ways to identify talented students
from poor schools (focus on the DET)
(extract: Placement Test in English for Education Purposes construct)
• aim to predict the performance of candidates in future situations in which language will be an important, but not sole, variable (language-as-vehicle rather than language-as-target);
• acknowledge the effects on cognitive functioning of the quantity and quality of prior mediated learning opportunities experienced by an individual (i.e. take seriously the impact of educational disadvantage by building in mediated learning opportunities).
• be based on a notion of knowing and learning which views learners as actively involved as individuals and in collaboration with others, in creating and negotiating meaning in a wide variety of settings. This process of conceptual development is seen as highly dependent on specific areas of expertise involving knowledge and information, and on the connections between these.
• be based on a componential model of language ability, which comprises topical knowledge and language knowledge, mediated by strategic competence (metacognitive strategy use) and affective schemata.
The main assessment related challenges were:
• producing a greater range of scores (spread), producing a greater range of scores (spread), so that capable students could be more so that capable students could be more clearly differentiated from weaker studentsclearly differentiated from weaker students
• improving (raising) the level of stronger improving (raising) the level of stronger students’ scoresstudents’ scores
• increasing the predictive validity of the test increasing the predictive validity of the test (that is, did the test correctly distinguish (that is, did the test correctly distinguish between weaker and stronger students)between weaker and stronger students)
Towards ‘dynamic’ testing:the old ELPT and the new PTEEP
•The same three major writing tasks (summary, description and contrast/comparison)
•The same additional text types (graph and diagram)
The PTEEP included structured and sequenced tasks designed to act as mediation for the writing assignments. The ELPT did not include these ‘scaffolding’ tasks.
•The same prose texts
DifferencesSimilarities
ASSESSING THE IMPACT OF THE SCAFFOLDING APPROACH
Questionnaire evidence from pilot groups suggests that candidates found the new test to be more ‘user-friendly’.
Correlations between first-year academic (UCT) performance and the selection tests strengthened.
Stronger candidates use preparatory tasks whereas weaker candidates do not see the connections between tasks (such tasks therefore help to widen the gap).
Based on scores from the same tasks (summary, short description and one page essay), the range of scores increased.
Candidate perceptions
Predictive validity
Task preparation
study
Range of scores
ELPT and PTEEP Scores
0
100
200
300
400
500
600
No
. o
f C
an
did
ate
s
10 20 30 40 50 60 70 80 90 100
Test Results (%)
ELPT
PTEEP
Quantitative approaches used at this stage and in early 2000s
• Problems with correlational analysis• A move to ‘survival analysis’• Classical item analysis• Psychometrically ‘naïve’ tests relying on
ranking – limitations for score use• Reliance on group performance for
‘standards’
The early 2000s
• Expansion of “AARP” testing• Introduction of Quantitative Literacy test• Major innovations in the area of test
delivery (use of innovative technology etc)• Pressure on tests reduced ‘potential’
aspect• Development of Health Science
Consortium group of tests
2005 →the main questions …
• Concerns largely shifted from access to throughput
• Placement function emphasised – implications for curriculum (focus mainly on extended progs)
• Concerns about the new school curriculum and qualification – target now all applicants (mass test)
• Increasing tensions with Department of Education (with stronger FET curriculum links)
Issues and challenges
• Academic literacy/ies construct widened to include greater focus on quantitative literacy (and reading not writing the focus)
• For the first time direct testing in the school curriculum (Mathematics & Maths Literacy)
• Radically different development trajectories, research traditions for the new NBTs (achievement tests) and TAPs (re-designing the old PTEEPs)
Research approaches:quantitative
• From Classical Test Theory to Item Response Theory
• Psychometric test development approaches
CTT vs IRT (Hambleton and Jones, Comparison of CTT and IRT and their applications to test
development)
Item and Test Characteristic Curves; Test Information Functions
and Equating
0
2
4
6
8
10
12
14
16
18
20
-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3
Information
Theta
Test Information Functions for Forms QL1 and QL2
Form QL1 TIF
Form QL2 TIF
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
-3.0 -2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0
P(theta)
Theta
Test Characteristic Curves for Forms QL1 and QL2
Form QL2 TCC
Form QL1 TCC
0.00
0.20
0.40
0.60
0.80
1.00
1.20
-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3
P(U=1|THETA)
THETA
ICC Plots for 3-PL Form QL1 Series2
Series3
Series6
Series19
Series20
Series23
Series24
Series26
Series28
Series50
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3
SUM of P(u=1 | THETA)
THETA
Test Characteristic Curve for 3-PL Form QL1
TCC form QL1
THE NATIONAL BENCHMARK
TESTS PROJECT
Why did HESA commission the NBTP?
2. Demonstrable inefficiencies in Higher Education itself (low throughput etc.)
• Difficulties in identifying students’ educational needs• Lack of appropriate curriculum flexibility at entry to meet these needs
2. Concerns about how to interpret the new NSC
In summary ….
• The NBTP is about higher education getting its own act in order – it is not about pointing fingers at the school system
• It sets out to do this by providing information about the competence of entering students in terms of 3 core domains of knowledge / skills
• it is important to note that higher education’s ‘take’ on what these core sets are, and at what level they should be mastered, will in all probability differ somewhat from those deemed most salient by the school-leaving system.
What do the NBTs aim to do?
– Provide additional information about performance in core, underlying areas (additional to NSC information)
– The core (domain) areas are:
• Academic literacy one 3-hr test • Quantitative literacy• Mathematics one 3-hr test
Performance in domain areas suggests that academic performance will not be adversely affected.
If admitted, students should be placed on regular programmes of study.
Challenges in domain areas identified such that it is predicted that academic progress will be affected.
If admitted, students’ educational needs should be met in a way deemed appropriate by the institution (eg extended or augmented programmes, special skills provision).
Serious learning challenges identified: it is predicted that students will not cope with degree level study without extensive & long-term support, perhaps best provided through bridging programmes or FET. Institutions registering students performing at this level would need to provide such support.
0%
100%
Proficient
Basic
Intermediate
Example - Competency Specification Proportions NBT Quantitative Literacy
Competence area
KnowingApplying routine
procedures in familiar contexts
Applying multistep procedures in a
variety of contexts
Reasoning and
reflecting
1+ 1- 2+ 2- 3+ 3- 4+ 4- n %
Comprehending: identifying or locating
Vocabulary5 0 7 7 2 4 1 5 31
19.0
15-20
Representations of numbers and operations
1 0 1 2 1 1 1 0 7 4.3 5-10
Conventions for visual representations
4 0 10 6 1 5 1 5 3219.6
20-25
Acting, interpreting, communicating
Using representations of data
2 0 6 3 1 4 1 4 2112.9
20-25
Computing (and estimating?)
0 0 4 10 2 4 0 2 2213.5
15-20
Conjecturing 0 0 0 0 0 0 0 1 1 0.6 0-5
Interpreting5 0 10 5 3 5 2 5 35
21.5
10-15
Reasoning 0 0 1 0 0 0 1 4 6 3.7 5-10
Representing quantitative information
0 0 3 1 0 0 2 0 6 3.7 5-10
Describing quantitative relationships
0 0 1 0 1 0 0 0 2 1.2 0-5
n 17 0 43 34 11 23 9 26 163
% 10.4 0.0 26.4 20.9 6.7 14.1 5.5 16.0
% 10.4 47.2 20.9 21.5
Mathematical and statistical ideas
Quantity, number and operations (LO1) 1 0 6 9 3 4 1 2 26
31.0
25-30
Shape, dimension and space (LO3) 3 0 4 3 0 2 1 1 14
16.7
10-15
Relationships, pattern, permutation (LO2a) 0 0 5 2 1 0 2 1 11
13.1
10-15
Change and rates (LO2b)0 0 4 2 1 0 1 1 9
10.7
10-15
Data representation and analysis (LO4a) 2 0 5 3 1 4 0 6 21
25.0
20-25
Chance and uncertainty (LO4b) 0 0 1 0 0 2 0 0 3 3.6 5-10
n
6 0 25 19 6 12 5 11 84
%
7.1 0.0 29.8 22.6 7.1 14.3 6.013.1
% 7.1 52.4 21.4 19.0
15-20 25-30 25-30 25-30
How are the benchmark (cut-off points) derived?
• The process is fundamentally different to the examination paper design procedures, and the norm referenced standardising and resulting processes of the NSC
• All items need to have been through rigorous review (fairness, content etc) and be statistically robust
• All items need to have been piloted
• The benchmark setting process is NOT about whether students can pass an item or not - the process is based on a set of probability assessments made by first year lecturers, with the core questions being:
“if a student can’t pass this item / do this, will s/he experience academic difficulties – and if so, how severe?”
NBT information
INDIVIDUAL LEVEL• Benchmark level (Basic, Intermediate, Proficient)• Description of what this means for each domain (ie what does being in the
‘Basic’ category mean a student knows and can do in Mathematics)• Clear recommendations about the type and extent of support needed
GROUP LEVEL• At the level of a faculty, or qualification, or institution ….• Give clear indication of the needs and strengths of entering cohorts, either
before entry, or at registration: useful for placement into existing courses, and/or with course design or modification.
DATA BASED ON
FEB 2009 PILOTS
ACADEMIC LITERACY (overall)
N = 12,202
Participating institutions: Mangosuthu, Rhodes, Stellenbosch, UCT, UKZN, UWC, Wits.
Serious learning challenges – long term, pre-tertiary intervention needed.
Challenges identified such that it is predicted that academic progress will be adversely affected. If admitted, students’ educational needs should be met in a way deemed appropriate by the institution (eg extended or augmented programmes, special skills provision)
Performance such that academic performance will not be affected.
If admitted, students should be placed on regular programmes of study.
851
55715780
0
1000
2000
3000
4000
5000
6000
7000
Total
ACADEMIC LITERACYNBT Benchmark Levels, February 2009
Basic
Intermediate
Proficient
ACADEMIC LITERACY by Faculty
250
63
289
22141
31
317
1256
119
1067
187
668
210
1179
2289
35
1393
340
575
142
1007
0
500
1000
1500
2000
2500
Commerce Education Engineering Health Humanities Law Science
ACADEMIC LITERACYNBT Benchmark Levels, February 2009
Basic
Intermediate
Proficient
Quantitative LiteracyN = 12,202
3055
6125
3022
0
1000
2000
3000
4000
5000
6000
7000
Total
QUANTITATIVE LITERACYNBT Benchmark Levels, February 2009
Basic
Intermediate
Proficient
Quantitative Literacy by Faculty
596
136
563
78
590
142
599
1807
77
1265
292
673
212
1235
1392
4
921
179121
29
669
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Commerce Education Engineering Health Humanities Law Science
QUANTITATIVE LITERACYNBT Benchmark Levels, February 2009
Basic
Intermediate
Proficient
Mathematics (overall)
1644
7390
737
0
1000
2000
3000
4000
5000
6000
7000
8000
Total
MATHEMATICSNBT Benchmark Levels, February 2009
Basic
Intermediate
Proficient
Mathematics overall[Intermediate ‘Top’ and ‘Bottom’]
5233
2157
0
1000
2000
3000
4000
5000
6000
Total
MATHEMATICSNBT Intermediate Benchmark Level, February 2009
Intermediate Bottom
Intermediate Top
Mathematics by Faculty
910
23
289
43 50 30
455
2765
58
1956
411
68 80
1658
372 387
821
189
0
500
1000
1500
2000
2500
3000
Commerce Education Engineering Health Humanities Law Science
MATHEMATICSNBT Benchmark Levels, February 2009
Basic
Intermediate
Proficient