‘at the coal face’ experiences of computer-based exams john winkley, btl paul humphries, edexcel...
Post on 11-Jan-2016
213 Views
Preview:
TRANSCRIPT
‘At the Coal Face’
Experiences of Computer-based Exams
John Winkley, BTL
Paul Humphries, Edexcel
Dorit Reppert, CCEA
July 8th 2003
Evaluating Computer-based Exams
Key Stakeholders:
• candidates• test centres• examiners• Awarding Bodies
Key Stakeholders:
• candidates• test centres• examiners• Awarding Bodies
Key Success Factors:
• engagement• accessibility• technical usability• system reliability• security• test validity, reliability• data analysis
Key Success Factors:
• engagement• accessibility• technical usability• system reliability• security• test validity, reliability• data analysis
Project Background
PEP 2 Project
Schools:
• 19 schools• GCSE style exams• over 1,000 candidates• approx. 1,400 exams
learndirect & Army Test Centres:
• 13 test centres• real National Tests• over 300 candidates• approx. 300 tests
The ExamBase System
ExamBase Client
ExamBase Server
MicroBoard website
Test Centre
Assessment Producer
exam
exam
question
marked question
data
ExamBase Client
ExamBase Client
marked ‘papers’
Standard PCsNot limit use of PCs at other timesSecure, ReliableEasy-to-use, Engaging, Accessible
Standard PCsNot limit use of PCs at other timesSecure, ReliableEasy-to-use, Engaging, Accessible
On-screen marking
MicroBoard
Test Centre
MarkerBase Server
exam
Examiner
Marking Centre
partially marked ‘papers’
partially marked ‘papers’
fully marked ‘papers’
Examiner
Exam Security
Web services
MicroBoard
Web interface
Web service consumer
ExamBase Server
Test Centre
ExamBase Client
Secure internet (https/ssl) Encrypted file transfer
Secure internet (https/ssl) Encrypted file transfer
Encrypted file transfer (htps/ssl/proprietary)
Encrypted file transfer (htps/ssl/proprietary)
MicroBoard
MicroBoard
MicroBoard
ExamBase Server
(red) (red) Downloaded but lockedDownloaded but lockedThis exam has been downloaded but has not been scheduled to run
(orange) (orange) Downloaded and ScheduledDownloaded and ScheduledThis exam has been scheduled but is not accessible by candidates until the scheduled start time.
(green) (green) Unlocked and ready to runUnlocked and ready to runThe four hour scheduling window has begun and the exam is ready to be accessed by candidates.
(yellow) (yellow) PausedPausedThe exam is currently paused by the administrator.
(blue) (blue) FinishedFinishedThe exam has been scheduled, should have been used and the four hour window for access has now finished or the exam has been closed (see 3.5 Finish exam).
(red) (red) Downloaded but lockedDownloaded but lockedThis exam has been downloaded but has not been scheduled to run
(orange) (orange) Downloaded and ScheduledDownloaded and ScheduledThis exam has been scheduled but is not accessible by candidates until the scheduled start time.
(green) (green) Unlocked and ready to runUnlocked and ready to runThe four hour scheduling window has begun and the exam is ready to be accessed by candidates.
(yellow) (yellow) PausedPausedThe exam is currently paused by the administrator.
(blue) (blue) FinishedFinishedThe exam has been scheduled, should have been used and the four hour window for access has now finished or the exam has been closed (see 3.5 Finish exam).
ExamBase Server
Example On-ScreenExample On-ScreenAssessment QuestionsAssessment Questions
The Test Centre Experience
Priority 1 – a reliable technical infrastructure:
includes trained support staff unreliability may interrupt an exam funding, management processes are key factors
Priority 1 – a reliable technical infrastructure:
includes trained support staff unreliability may interrupt an exam funding, management processes are key factors
Priority 2 – an appropriate exam environment:
setting-up takes longer can be difficult to provide adequate privacy
Priority 2 – an appropriate exam environment:
setting-up takes longer can be difficult to provide adequate privacy
The Test Centre Experience
Schools:
computer-based exams entirely new to schools
transfer of responsibility from administrators to technical staff
48% found software installation easy or very easy
54% had no problems with candidate registration
all schools ran exams successfully
Schools:
computer-based exams entirely new to schools
transfer of responsibility from administrators to technical staff
48% found software installation easy or very easy
54% had no problems with candidate registration
all schools ran exams successfully
The Test Centre Experience
learndirect centres :
technical infrastructure already exists for e-learning and formative assessment
used to maintaining ‘business critical’ IT systems
95% of 29 Skills for Life project centres found the installation easy or very easy
all centres found registration of candidates using MicroBoard easy
learndirect centres :
technical infrastructure already exists for e-learning and formative assessment
used to maintaining ‘business critical’ IT systems
95% of 29 Skills for Life project centres found the installation easy or very easy
all centres found registration of candidates using MicroBoard easy
The Candidate Experience
School Pupils:
‘digital natives’ at ease with IT over 94% use a computer at home at least once a week exams are mandatory
School Pupils:
‘digital natives’ at ease with IT over 94% use a computer at home at least once a week exams are mandatory
Adult Learners:
great disparity of IT experience some with no IT experience have opted for National Tests
Adult Learners:
great disparity of IT experience some with no IT experience have opted for National Tests
The Candidate Experience
School Pupils:
Overwhelmingly positive
92% said they enjoyed the tests
School Pupils:
Overwhelmingly positive
92% said they enjoyed the tests
Adult Learners:
computer-based exams seen as new and positive
“almost unanimously they reported preferring to take an online test rather than a paper test” - Ufi
Adult Learners:
computer-based exams seen as new and positive
“almost unanimously they reported preferring to take an online test rather than a paper test” - Ufi
“It didn’t feel like an exam”
“more enjoyable
than writing”
The Candidate Experience
Common concerns:
technology failures:
• rare but disconcerting• mainly due to PC system problems• candidates can continue on a spare PC without losing work
the exam environment
• keyboard noise in particular
Common concerns:
technology failures:
• rare but disconcerting• mainly due to PC system problems• candidates can continue on a spare PC without losing work
the exam environment
• keyboard noise in particular
MarkerBase
The Examiner Experience
General responses:
9 out of the 10 examiners found the marking either very enjoyable or enjoyable
all found the software either very easy or easy to use
most would prefer to work at home
liked choice of being able to mark ‘by candidate’, ‘by question’ or both
General responses:
9 out of the 10 examiners found the marking either very enjoyable or enjoyable
all found the software either very easy or easy to use
most would prefer to work at home
liked choice of being able to mark ‘by candidate’, ‘by question’ or both
Potential for more rapid marking :
marking speed increased with experience
marking ‘by question’ increases efficiency
Potential for more rapid marking :
marking speed increased with experience
marking ‘by question’ increases efficiency
The Examiner Experience
More accurate tallying:
automatic tallying of marks
error-prone task removed
More accurate tallying:
automatic tallying of marks
error-prone task removed
The Awarding Body Experience
Engaging learners:
closer to day-to-day experiences for many
exam stress reduced
positive impact on self-esteem
more engaging than paper-based exams
Engaging learners:
closer to day-to-day experiences for many
exam stress reduced
positive impact on self-esteem
more engaging than paper-based exams
Accessibility:
Much greater flexibility:• can register candidates up to the exam start• ‘on demand testing’ will be possible
‘on screen’ approach allows tests to be taken:• in mobile situations (e.g. touring bus)• in remote locations (e.g. using laptops)
Accessibility:
Much greater flexibility:• can register candidates up to the exam start• ‘on demand testing’ will be possible
‘on screen’ approach allows tests to be taken:• in mobile situations (e.g. touring bus)• in remote locations (e.g. using laptops)
The Awarding Body Experience
The Awarding Body Experience
Quality assurance:
Less opportunity for human error
Quality checks have to be refocused
Quality assurance:
Less opportunity for human error
Quality checks have to be refocused
Security:
less vulnerable to lost or mishandled papers
encryption technologies increase security
Security:
less vulnerable to lost or mishandled papers
encryption technologies increase security
The Awarding Body Experience
Exam reliability:
100% reliability for ‘closed’ question types
‘open’ question types:• marking remains subjective• MarkerBase helps examiners to be more consistent
Exam reliability:
100% reliability for ‘closed’ question types
‘open’ question types:• marking remains subjective• MarkerBase helps examiners to be more consistent
The Awarding Body Experience
Exam validity:
Good correlation with paper-based National Tests
GCSEs, using wider range of question types, will require additional validation effort
Exam validity:
Good correlation with paper-based National Tests
GCSEs, using wider range of question types, will require additional validation effort
The Awarding Body Experience
Results generation:
possibility of more rapid feedback to candidates
provide results to test centre rather than to candidates
Results generation:
possibility of more rapid feedback to candidates
provide results to test centre rather than to candidates
Data analysis:
richer feedback to candidates and test centres
easier to detect anomalies in candidate responses
Data analysis:
richer feedback to candidates and test centres
easier to detect anomalies in candidate responses
Conclusions
very popular with candidates - more engaging and interactive, but additional validation needed
greater security and reduced risk of human error
improved accessibility and flexibility – highly desirable for Awarding Bodies
quality of technical infrastructure is vital
may initially place a burden on test centres, but with the promise of efficiency gains over time
very popular with candidates - more engaging and interactive, but additional validation needed
greater security and reduced risk of human error
improved accessibility and flexibility – highly desirable for Awarding Bodies
quality of technical infrastructure is vital
may initially place a burden on test centres, but with the promise of efficiency gains over time
top related