understanding the work of the hkeaa: quality assurance
DESCRIPTION
Understanding the work of the HKEAA: Quality Assurance. C. S. Tong Secretary General 7 June 2013. About HKEAA. Established in 1977 (independent statutory body) 400 staff in 7 office locations 8,000 exam personnel (seasonal). - PowerPoint PPT PresentationTRANSCRIPT
1
Understanding the work of Understanding the work of the HKEAA: Quality Assurancethe HKEAA: Quality Assurance
C. S. Tong C. S. Tong Secretary GeneralSecretary General
7 June 20137 June 2013
About HKEAA Established in 1977
(independent statutory body) 400 staff in 7 office
locations 8,000 exam personnel
(seasonal)
Mission: To provide valid, reliable and equitable examinations and assessment services in a professional, innovative, efficient and effective manner.
How are examination papers set?
Curriculum and Assessment (C&A) Guides Subject Committees to oversee the
continual development of each subject Moderation committees to set papers with
rigorous security and quality control mechanism
Moderation committee members are appointed on an annual basis subject to various security requirements
Who are Moderation Committee members?
Subject Expert/
University Academics
HKEAA Subject
Managers
Open nomination for membership from schools as well as from the relevant Subject Committee (current S5 or S6 teachers not eligible)
Secondary School
Teachers
6
How are scripts handled? All scripts are scanned and then archived for
safe-keeping: over 10,000,000 pages scanned for Onscreen Marking
How are Written Papers marked?
Marking schemeMarking scheme Appointment of Markers
Appointment of Markers
Sample scripts selection
Sample scripts selection
Standardisation meeting
Standardisation meeting
Markers’ meeting
Markers’ meeting
Allocation of scripts to markers
Allocation of scripts to markers
MarkingMarking CheckmarkingCheckmarking Statistical analysis of marks
Statistical analysis of marks
8
How are Written Papers marked?
OSM facilitates real time, continuous monitoring of the quality of the marking:
Marking by questions to enhance consistency in applying the marking criteria
Double marking for open-ended questions Use of control scripts to validate marking
consistency Scripts are selected for checkmarking by
experienced Chief/Assistant Examiners
Where are the Assessment Centres?Where are the Assessment Centres?
Sheung Shui:185Sheung
Shui:185
Tsuen Wan:163Tsuen Wan:163
Lai King:376Lai King:376
Sha Tin Wai:92Sha Tin Wai:92
San Po Kong:73San Po Kong:73
Fortress Hill:382
Fortress Hill:382
Tai Hing:100Tai Hing:100
Tseung Kwan O:21Tseung Kwan O:21
About 1,400
workstations
Facilities at the Assessment Centres
10Lounge/Pantry Conference Room
WorkstationsReception Area Control Room
OSM Procedures
1st checkmarking
2nd checkmarking
Markers’ meeting
satisfactorymarking
satisfactorymarking
Complete
Training scripts
Qualifying scriptsReference scriptsNormal scripts Control scripts
Markers’ meeting to clarify and finalize marking scheme
Provision of Reference Scripts to illustrate the marking requirements and standard
Training Scripts & Qualifying Scripts to ensure quality of marking before markers start marking ‘real’ scripts
How does the Authority monitor Quality of Marking?
Monitoring Quality of Marking
Selected scripts (including control scripts etc) have been discussed by a team including the Chief Examiner and awarded standardized scores
Control scripts are randomly distributed to markers to ensure consistency of marking standards
Monitoring Quality of Marking Marker’s score can deviate from the
standardized score to within a specified tolerance
When the deviation exceeds the tolerance level, the marker will be contacted for clarification of marking scheme/ further training and qualifying/ or stopped from further marking and problematic scripts can be voided and redistributed to other markers for marking as necessary
Standardized Score
Marker’s Score
Marker’s score agrees with Standardized Score
Tolerance level
Illustration
Standardized Score
Marker’s Score
Marker’s score much higher
than standardized
score
Marker’s score much lower than
standardized score
Marker’s score agrees with Standardized Score
Tolerance level
Illustration
Standardized Score
Marker’s Score
Marker’s score much higher
than standardized
score
Marker’s score much lower than
standardized score
Marker’s score agrees with Standardized Score
Tolerance level
Illustration
Monitoring Quality of Marking
Chief / Assistant Examiners will check each marker’s marking statistics
(including marking speed, mark distribution, score deviation from control scripts etc)
scrutinize markers’ marking records and select scripts for checkmarking
contact markers and further clarify the marking scheme if necessary
19
Why Double Marking? To address the issue of inter-rater
variability for open ended questions, double marking may be adopted
Feasibility study carried out to assess the cost benefit between such factors as Marking reliability Markers’ standardization Increased marking time Marking cost
Marker 3Chief Examiner
or Assistant Examiner
Discrepancy within range
The discrepancy among marks is still out of range
Candidate’s mark
Discrepancy within range
Big discrepancy between the two marks
Candidate’s mark
Candidate’s mark
Marker 1
Marker 2
Double Marking: how?Double Marking: how?
The Candidate’s final mark will be the sum of the two highest closest marks
Post examination reviewExternal vetting of examination papersPost examination reviewInternal audit & risk monitoringExamination Paper Quality Audit (EPQA)Internal post-mortem meetings
SBAMarking processMechanisms for handling irregularities, complaints and queriesStandards setting and maintenanceGrading proceduresQuality check of examination dataFeedback to schools
Setting of examination papersCurriculum and Assessment Guide & Assessment FrameworkExamination paper development mechanismConfidentiality and security measures
Conduct of examinationsExamination regulationsPublic Examinations Information Centre (PEIC)Public Examinations Communication and Support System (PECSS)Oral Recording System (ORS)On-site checking & auditsExamination logistics & security measures
HKDSE Recognition
Safeguarding of Standards
Compliance & Quality of
Delivery
Continuous Improvement
Processing & reporting of results
How to ensure overall Quality?
Vetting of Examination papers
Different subjects are selected annually for vetting and peer review
Vetting reports provide useful feedback for subject managers and Subject Committees
Reports and responses from the vetting exercise as well as reports on examination irregularities are submitted to the Public Examinations Board for discussion
24
Discrepancy CheckingScripts are sampled for cross checking at various stages of marking and grading as part of our quality assurance framework:
Randomly selected candidates and all their scripts extracted for manual checking
Scripts of candidates with large discrepancies in the levels obtained in different subjects
Scripts of candidates with large discrepancies in component levels (eg over 1000 scripts from Chinese Language checked in 2012)
25
Discrepancy CheckingPerformance at the school level are cross-checked across different core subjects: eg LS vs Chin 圖表標題 y = 0.9663x + 9.008
R2 = 0.9337
0
100
200
300
400
500
600
0 100 200 300 400 500 600
L
(L)線性
Outlier schools to be manually checked
26
Discrepancy Checking
Math vs English圖表標題 y = 0.9434x + 15.1
R2 = 0.8901
0
100
200
300
400
500
600
0 100 200 300 400 500 600
M
(M)線性
Assessment Quality-Assurance Platform
Based on Rasch Model Item-level and paper-level analysis to
ensure quality: Item difficulties Discrimination power Item reliability Paper reliability (internal consistency)
36
International Recognition
Over 150 overseas institutions responded to our survey proforma
70 institutions from the Mainland accept students from HK on the basis of HKDSE
149 institutions from Taiwan accept students from HK on the basis of HKDSE
37
International Recognition ComparisonInternational Recognition ComparisonInternational Recognition ComparisonInternational Recognition Comparison
UCAS Tariff HKDSE Advanced Placement
(AP)GCE Advanced Level
(GCE AL)International
Baccalaureate (IB)5**
140 A* 130 5* 7120 5 5 A 110 6100 B 90 4 80 4 C 570 60 3 D 50 440 3 E 30 20 3
145