report on the dbe nsc november 2018 examinations select...
TRANSCRIPT
Report on the DBE NSC November 2018 Examinations
Select Committee on Education and Recreation
13 February 2019
Dr Mafu Rakometsi
1
1. Umalusi mandate and regulatory framework
2. Framework for Quality Assurance of Assessment
3. The quality assurance processes undertaken in 2018
3.1 Scope of the 2018 quality assurance of assessment
3.2 Areas of good practice/compliance
3.3 Tracking of directives for compliance 2016-2018
4. Standardisation and Resulting
5. Conclusion
Presentation Outline
2
Umalusi mandate and regulatory framework
Umalusi derives its mandate for quality assurance of assessment from the:
• National Qualifications Framework (NQF) Act No. 67 of 2008
Sections 27 (h) and 27 (i) and
• The General Further Education and Training Quality Assurance Act asamended in 2008. Section 17 (A) which state that:
(3) The Council must, perform the external moderation of assessment of all assessment bodies and education institutions
(4) The Council may adjust raw marks during the standardisation process and the
(5) The Council must, with the concurrence of the Director-General and after consultation with the relevant assessment body or education institution, approve the publication of the results of learners if the Council is satisfied that the assessment body or education institution has satisfied the conditions that warrant such an approval.
4
Framework for Quality Assurance of Assessment
Quality assurance of assessment is conducted to ensure that assessment leading to the award of certificates in schools, adult education centres and technical and vocational education and training colleges is of the required standard. This is in order to ensure that the certificates issued by Umalusi are credible.
This is achieved through:
• Moderation of examination question papers, Practical Assessment
Tasks (PAT) and Common Assessment Tasks in the case of Life
Orientation
• Monitoring and moderation of School Based Assessment (SBA)
• Monitoring of the conduct, administration and management of
assessment and examination processes
• Monitoring and Moderation of marking
• Management of concessions and examination irregularities
• Standardisation of assessment outcomes
• Approval of release of results
6
National senior Certificate
The National Senior Certificate examinations are administered by three Assessment bodies (Department of Basic Education (DBE), Independent Examinations Board (IEB) and South African Comprehensive Assessment Institute (SACAI), all 3 assessment bodies are quality assured and certificated by Umalusi:
Overview of candidates that wrote the NSC examinations
Assessment Body 2016 2017 2018
DBE FT 664 600 622 802 619 133
DBE PT 1 237 2 689 1 150
TOTAL 665 837 625 491 620 283
IEB FT 11 101 11 518 11 587
IEB PT 479 474 628
TOTAL 11 580 11 992 12 215
SACAI PT 1 207 1 438 1 608
TOTAL 1 207 1 438 1 608
7
Overview of the Quality Assurance of Assessment and Examination Processes
• Umalusi moderated and approved 116 DBE NSCNovember 2018 question papers with their markingguidelines which went through different moderation levelsbetween February and August 2018.
• Between July and October 2018, Umalusi conductedverification and moderation of SBA of 15 subjects thatwere selected from across the nine provinces.
• The state of readiness to conduct examinations wasconducted across all the nine (9) PEDs.
• Umalusi monitored a sample of 261 examination centresand 28 marking centres across the nine (9) PEDs.
9
Overview of the Quality Assurance of Assessment and Examination Processes
• Umalusi attended 122 marking guideline discussions.
• Twenty-eight (28) subjects were sampled for theverification of marking in 94 marking centres.
• DBE presented a total of 67 subjects for thestandardisation of the November 2018 National SeniorCertificate (NSC) examinations.
• All irregularities that were identified were managed in accordance with the Regulations pertaining to the Conduct, Administration and Management of the National Senior Certificate Examinations by the Provincial Examination Irregularities Committees (PEICs), challenges experienced were captured per Provincial Education Department.
10
Moderation of Question Paper
In 2018 Umalusi moderated and approved 116 question papers with their marking guidelines for the 201811 examinations.
The question papers went through different levels of moderation and approved at different levels as indicated in the table below:
Number of
moderations
November
2015
November
2016
November
2017
November
2018
One 11,5% 22,6% 17,5% 32,8%
Two 67,7% 68,4% 79,3% 64,6%
Three 17,7% 8,3% 3,2% 2,6%
Four 2,3% 0.8% 0% 0%
Five 0,8% 0% 0% 0%
12
Moderation of Question Papers
Areas of Good Practice
The DBE is commended for the improvement in the percentageof question papers that were approved at first and secondmoderation. The analysis of the question paper moderationreports revealed that 97.4% of the November 2018 questionpapers met all external moderation criteria during the first andsecond moderations, compared to 96.8% in November 2017.
Equally commendable was the fact that 38 question paperswere approved at first moderation.
Umalusi noted an improvement of more than 3% in compliancewith the following criteria at first moderation in comparison withthe November 2017 examination :
Language and bias (from 56% to 59%); and
Marking guidelines (from 37% to 41%).
13
Moderation of Question PapersDirectivesDBE must
Nov 2016 Nov 2017 Nov 2018 Progress in 2018
Reduce the number of
Question Papers requiring
more than two external
moderation sessions due
to non - compliance
91% of question papers complied
96.8% of
question
papers
complied
97.4% of
question
papers
complied
A gradual reduction of question papersrequiring more than two moderations forapproval was noted. Only three questionpapers went up to third moderation in2018.
Retrain the examiners
and internal moderators
on the following aspects;
(technical details (TD),
quality of questions (QQ),
quality of marking
guidelines (MG))
Question
papers that
complied
with criteria:
TD: 53%
QQ: 40%
MG: 35%
Question papers that complied with criteria:
TD: 48%QQ: 29%MG: 37%
Question papers that complied with criteria:
TD: 41%QQ: 28%MG: 41%
Less than 50% of the question paperscomplied fully with these three criteria(technical details, quality of questionsand quality of marking guidelines) – in2018 the compliance to technicaldetails and quality of question paperscriteria declined further, to a state worsestate than in 2016 and 2017. The Markingguideline even though it did not meetthe 50% mark, it is steadily improving.
Improve examiners and
internal moderators ability
to set higher order
questions and balance
distribution of cognitive
levels
59% of
question
papers
complied
with the
cognitive skills
requirements
60% of
question
papers
complied
59% of
question
papers
complied
The compliance level is stagnatingbetween 59% and 60% - hence noimprovement or decline. Thecompliance level is still very low – moreshould be done to improve.
14
Moderation of Question PapersDirectives
DBE must
Nov 2016 Nov 2017 Nov 2018 Progress in 2018
Improve on the development of
marking guidelines at the time of
setting the question paper rather
than to rely on post-examination
refining of the marking
guidelines
83%
compliant
80%
compliant
A decline of 3% has
been noted. Though
compliance is high,
there is room for
improvement.
Conduct workshops to
capacitate examiners and
internal moderators in the setting
of question papers, placing
more emphasis on the criteria
with lower levels of compliance,
i.e
Technical details (Compliance
to this criterion has shown a
decline since 2016);
Internal moderation;
Quality of questions; and
Cognitive skills.
Partially
compliant
Internal
moderation
75%
compliant
Internal
moderation
71%
compliant
The decline of 4%
noted in the internal
moderation criterion,
indicate that question
papers submitted for
external moderation
were in an
unacceptable state
and could not be
approved.
15
Scope of Moderation of SBA
Subjects moderated during July and October 2018
* Fitting and Machining; Welding and Metal work
Accounting Mathematical Literacy
Business Studies Mathematics
Consumer Studies Mechanical Technology*
Economics Music
Geography Physical Sciences
History SASL HL
Life Orientation Tourism
Life Sciences
17
Moderation of SBA
Areas of Good Practice
Areas of good practice were observed in different provinces and the good practices differed from one subject to another. Some of the good practices noted are as follows:
Inclusion of the subject improvement plans in some Western Cape teacher files;
The use of district/provincial common assessment tasks in most provinces was also commendable, since it helped to maintain quality and standards;
Gauteng and Western Cape are commended for the improved quality of internal moderation in Life Sciences.
18
Moderation of SBADirectives
DBE must ensure:
Nov
2016
Nov 2017 Nov 2018 Progress in 2018
That the quality assurance
of assessment in subjects
with a practical
component for both the
Practical Assessment Task
(PAT) and the practical
examinations is attended
to
Not-
compliant
compliant Done. A generic guideline to
standardise the moderation of
PATs was developed.
The guidelines have been
customized in 6 subjects ( to
make it subject specific) and the
other subjects will follow in 2019
That adherence to subject
assessment policies in all
subjects in all provinces is
strengthened;
Not-
compliant
New directive this will be
observed in 2019.
Discourage the use of
recycled tasks and
shadow marking in SBA
through effective internal
moderation processes.
Not-
complia
nt
Not-
compliant
Not
compliant
Shadow marking is still evident in
some subjects.
19
Moderation of SBADirectives
DBE must :
Nov
2016
Nov
2017
Nov 2018 Progress in 2018
Ensure that internal
moderation is conducted
efficiently and effectively
at all levels of the system
Not-
compliant
New directive, this will be
observed in 2019.
Ensure that sufficient
focused support is given
to SASL HL SBA and the
moderation thereof.
Not-
compliant
New directive, this will be
observed in 2019.
20
Monitoring the State of Readiness
The state of readiness (SOR) was conducted across all PEDs between 5 September 2018 and 2 October 2018
Areas of Good Practice
Umalusi noted the following areas of good practice:
• Development of clear Standard Operating Procedure across qualityassurance processes as a measure to standardise operations and toimprove on standards in systems used in the delivery of qualityexaminations and assessments.
• An increase in the monitoring of high risk examination centresthrough the deployment of resident monitors across the PEDs.
• Online monitoring report by monitors to expedite daily reports inMpumalanga Province.
22
Monitoring the State of ReadinessDirectives
DBE must:
Nov 2016 Nov 2017 NOV 2018 Progress in 2018
Effectively implement the
policy on the registration of
immigrant candidates;
Not-
compliant
Partially
compliant
Complia
nt
PEDs strengthened the
implementation of policy,
and there were only few
incidents where candidates
did not submit relevant
documentation;
Ensure that all districts
conduct an audit of
examination centres to
verify their readiness to
administer examinations;
Not-
compliant
Partially
compliant
Complia
nt
Compared to the 2017 audit,
Umalusi noted a significant
improvement in the efforts of
the DBE and the PEDs in
auditing the examination
centres, across all nine
provinces.
Ensure that security features
at districts/nodal points are
evaluated and improved.
Storage facilities should
have features such as
double locking systems,
alarms in working condition,
and surveillance cameras;
Not-
compliant
Partially
compliant
Complia
nt
Umalusi noted substantial
improvements on security
features at the nodal
points/districts and storage
facilities during the state of
readiness across the PEDs
23
Monitoring the State of Readiness
Directives
DBE must ensure that
Nov 2016 Nov 2017 Nov 2018 Progress in 2018
The North West and Free State
PEDs strengthen or improve
printing facilities to avoid
manual handling of question
papers;
Not
compliant
Not
compliant
Manual packaging of
question papers in the
Free State and North
West was still not
addressed.
Proper surveillance systems are
installed at all printing facilities.
Not
compliant
Partially
compliant
North west did not
adequately address
norms and standards
requirements for
security at the printing
site.
24
Monitoring of the writing phase
EC FS GP KZN LP MP NC NW WC Total
Number
of centres
35 24 34 36
(includes
1 in
Eswatini)
34 19 20 21 38 261
Areas of Good Practice
Umalusi noted the following areas of good practice:
• Introduction of web-cameras as a resource to facilitate
efficiency in SASL HL examinations across centres offering the
subject.
• Implementation of a three tier monitoring approach across PEDs.
26
Monitoring the Writing of Examinations
Directives
DBE must:
Nov 2016 Nov 2017 Nov 2018 Progress in 2018
Ensure that invigilators read
examination rules to
candidates, check question
papers for technical errors
with the candidates and
give candidates 10 minutes
reading time during which
no writing or scribbling should
be allowed;
Not
Compliant
Not
Compliant
Compliant This was
addressed and
is no longer a
directive.
Ensure that all candidates
who leave the examination
room during writing are
accompanied by invigilators,
and it is advisable to keep a
record of such candidates.
Not
Compliant
Compliant Compliant This was
addressed and
is no longer a
directive.
27
Monitoring the Writing of ExaminationsDirectives
DBE must:
Nov 2016 Nov 2017 Nov 2018 Progress in 2018
Ensure that the
examination centres verify
the candidates at the
entry point for relevant
documentation to avoid
impersonation;
Not
compliant
Not
Compliant
Partially
compliant
Forty-one centres
out of the 261
centres monitored
admitted
candidates, without
verifying their
identity.
Ensure that the Chief
Invigilators prepare daily
situational report and file
copies of dispatch form in
the examination file for
reference.
Not
compliant
Not
compliant
Partially
compliant
Approximately
25.4% of the
examination
centres monitored
did not comply with
the requirement of
preparing the daily
situational reports as
well as filing of
dispatch forms.
28
Monitoring the Writing of Examinations
Directives
DBE must:
Nov 2016 Nov 2017 Nov 2018 Progress in 2018
Ensure that the key to the
facility that stores
examination material is kept
by the Chief invigilator
before the start of the
examination session;
Not
compliant
Compliant Compliant Full compliance was
observed from all the
examination centres in 2018.
Ensure that all examination
sessions have a seating
plan drawn, and available
for verification;
Not
compliant
Not
compliant
Compliant Full compliance was
observed from all the
examination centres in 2018.
Ensure that principals are
appointed as chief
invigilators as per regulation,
and letter of delegation
must be issued in cases
where they are not able to
administer the sessions;
Not
compliant
Compliant Compliant Full compliance was
observed from all the
examination centres
monitored.
29
Audit of Appointed Markers
An audit of appointed markers was conducted in all the nine (9) PEDs and the following were observed:
• PEDs used PAM as the basis for selection of markers,
• Enhanced criteria for selection of markers was observed in some PEDs;
• In certain PEDs the marker evaluation outcomes of the previous examination were used as one of the selection criteria,
Areas of Good Practice
Some PEDs were innovative in their marker appointment
processes and used provincially determined criteria to
enhance the PAM criteria.
31
Audit of Appointed Markers
Directives
DBE must:
Nov 2016 Nov 2017 Nov 2018 Progress in 2018
Ensure that marker
selection panels consider
recommendation made
by principals and/or
district officials in the
appointment of markers.
Not-compliant
A few cases were observed in two PEDs. A new directive will be observed in 2019
Ensure that PEDs verify
qualifications of markers
at all levels before
appointments are made.
Not-
compliant
Partially
compliant
Partially
compliant
A few discrepancies in the selection process were still observed by Umalusi during the 2018 audit.Five out of nine provinces did not adhere to 2016 & 2017 directive of verifying qualifications before making appointments, in most of the verified subjects.
32
Marking Guideline Discussions
Area of Good Practice
The PED chief markers and internal moderators markedthe authorisation scripts separately and without anydiscussion. Their performance declared them competentand they were duly authorised to train markers at theprovincial marking centres.
34
Marking Guideline DiscussionsDirectives
DBE must ensure that
Nov
2016
Nov 2017 Nov 2018 Progress in 2018
All chief markers and internal
moderators come to the
meetings having marked the
required sample of scripts.
In 41
question
papers only
2 provinces
pre-marked
less than 20
scripts
In 26
question
papers
only 2
provinces
pre-
marked
less than
20 scripts
In 21
question
papers, 8
provinces
pre-
marked
less than
20 scripts.
A reduction in non
compliance has been
noted (from 41 question
papers in 2016 to 21
question papers in 2018).
The directive is partially
addressed.
Sufficient time is allowed
between the writing of a
question paper and its
marking guideline discussion
meeting. (This allows time for
chief markers and internal
moderators to mark the
sampled number of scripts in
preparation for the marking
guideline discussion
meetings).
An average
of 7 days
between
examinatio
n day and
marking
guideline
meeting
An
average
of 9 days
between
examinati
on day
and
marking
guideline
meeting
Compliant In 2018 an average of 9
days between
examination day and
marking guideline meeting
was maintained.
35
Marking Guideline DiscussionsDirectives
DBE must ensure that
Nov
2016
Nov 2017 Nov 2018 Progress in 2018
All provinces offering a
subject are represented at
the marking guideline
discussion, irrespective of
scripts marked at their
province or not.
12 subjects
involved
Same
subjects as
in 2016
Compliant Only subjects that are
marked centrally did not
have representatives
from all nine provinces.
The print quality of all
diagrams/ illustrations in all
question papers are
thoroughly checked
before delivered to
marking centres.
Math
Literacy
Paper 1 in
three
provinces
(NC, MP &
GP)
Partially
compliant
The print quality of the
time zone map for the
Tourism question papers
in all provinces was poor.
Different people are
appointed for different
levels of the same question
papers.
IsiNdebele,
Tshivenda
and
Xitsonga
FAL
Tshivenda
and
Xitsonga
FAL
Partially
compliant
Improvement in
IsiNdebele HL and FAL:
meetings were
organised on different
dates.
36
Marking Guideline Discussions
Directives
DBE must ensure that
Nov
2016
Nov 2017 Nov 2018 Progress in 2019
Interpreters are available
throughout the SASL HL
marking guideline
discussion sessions.
Not
compliant
A new directive will be
observed in 2019
Strategies are developed
on how to conduct
marking guideline
standardisation meetings
for subjects with a practical
components such as Visual
Arts
Not compliant
Complaint Marking Guideline
meetings were
conducted for Virtual
Arts and Design.
37
Areas of Good Practice
The following areas of good practice were noted:• The discussed and approved marking guidelines were
adhered to for most subjects.• Approval of changes and additions to marking guidelines
followed due process.• The quality of marking was found to be good. Where novice
markers were appointed, they were appropriately assisted bysenior markers.
• The determination of and adherence to a marking tolerancerange for examination scripts made marking more reliable.Variances in marks allocated were mostly within the agreedtolerance range.
• Positive reporting regarding the fairness, reliability and validityof marking was commendable.
• Pairing of inexperienced and experienced markers and deafand non-deaf markers for SASL HL was commendable.
Verification of Marking
39
Verification of Marking
Directives
DBE must ensure that:
Nov 2016 Nov 2017 Nov 2018 Progress in 2018
The quality of marking in
Mathematical Literacy needs
attention, especially in Gauteng
and Kwa-Zulu Natal.
(Visual Arts in EC, Sesotho in GP
and FS)
Not compliant
Not compliant
Complaint The moderators had no
complaints.
Marking in each subject is
synchronised across PEDs, and a
more effective way needs to be
found to guarantee that any
approved changes to the final
Umalusi approved MG arrive
timeously in the PEDs
Not compliant
Not compliant
Not Compliant
Gauteng and Limpopo
started marking earlier
than all the other PEDs.
40
Monitoring of Marking
Areas of Good Practice
It is pleasing to note that the monitors reported positively about
their respective monitoring sessions, and the following positive
remarks regarding areas of good practice were made:
The use of a CD by two PEDs with all the necessary templates
and a manual document to be kept as back-up was well
received;
The compilation of a comprehensive marking manual and a
manual containing most marking forms used at venues were
of great help;
Some centres had back up plans (like the use of generators)
for the unforeseen load shedding occurrences.
42
Monitoring of MarkingDirectives
DBE must:
Nov 2016 Nov 2017 Nov 2018 Progress in 2018
Ensure that the security at the
main entrances into the
marking centres is tight, and
security guards must be
trained by the assessment
body in order for them to be
effective and efficient;
Not
compliant
Not
compliant
Compliant The level of
improvement was
observed. Strict access
control was
maintained at the
main entrance of each
centre.
Ensure that all training material
are delivered on time in order
to allow the determined norm
time to be achieved without
pressure;
Not
compliant
Not
compliant
Partially
compliant
Late delivery of
marking guidelines was
observed at some
marking centres.
Ensure that monitoring of
marking centres by the
assessment body takes place
and evidence of such visits is
availed to the centre.
Not
compliant
Compliant Compliant The directive was fully
adhered to.
43
What is Standardisation?
A process used to eliminate the effect of
factors other than the learners’
knowledge, abilities and
aptitude on their performance.
45
Why Standardise Results?
• To ensure that learners are
not advantaged or
disadvantaged by factors
other than their knowledge
of the subject, abilities and
aptitude.
• To achieve comparability
and consistency of the results
from one year to the next.
46
Standardisation Principles
No adjustment
should exceed 10%
of the historical
average in either
direction (upward or downward)
If the distribution of
the raw marks is
below or above the historical
average, the marks may
be adjusted either way subject to limitations
In the case of an individual candidate,
the adjustment
effected should not
exceed half of the raw
mark obtained by
the candidate
After considering qualitative
and quantitative
reports, Umalusi
formulates positions on
each subject
47
Results Standardised by Umalusi Per Qualification
Department of Higher Education and
Training (DHET)
• National Certificate (Vocational)
• General Education and Training
Certificate
• N1-N3
Department of Basic Education (DBE) • National Senior Certificate
• Senior Certificate (amended)
Independent Examinations Board
(IEB)
• National Senior Certificate
• General Education and Training
Certificate
South African Comprehensive
Assessment Institute (SACAI)
• National Senior Certificate
Benchmark Assessment Agency
(BAA)
• General Education and Training
Certificate
48
Standardisation Process
(1) Question papers are set by examiners
(2) Examiners submit question
papers to internal
moderators
(3) Internal moderators
submit QPs to external
moderators
(4) Monitoring exams
(5) Monitoring exams’ state of
readiness
(6) Question papers are
approved by external
moderators
(7) Verification of marking and capturing of
marks
(8) Standardisationand approval of
results by the ASC
(9) Announcement of the approval of results by Council
18 months
49
Standardisation Decisions DBE NSC
Description November
2016
November
2017
November
2018
Number of instructional
offerings presented
58 58 67
Raw marks 26 38 39
Adjusted (mainly upwards) 28 16 17
Adjusted (mainly downwards) 4 4 11
Number of instructional
offerings standardised
58 58 67
50
Concluding remarks
The conduct, administration and management of the
National Senior Certificate is regulated and it is imperative
that all the processes undertaken are premised on the
“Regulations Pertaining to the Conduct, Administration
and Management of the National Senior Certificate”; and
Non adherence to the Regulations and Umalusi policies
may lead to instances which can compromise the
credibility and integrity of the assessments and
examinations.
51
Approval of the results
The evidence presented to Umalusi suggests that the conduct,
administration and management of the DBE November 2018 NSC
examinations was satisfactory.
The rigor with which the DBE monitored the examinations both in
the preparatory phase for the conduct of the examinations and
the actual conduct of the examinations is an indication of the
level of commitment to the delivery of a credible examination.
Umalusi appreciates the efforts put into the whole process.
The DBE is also commended on the swift response in dealing with
the different cases of alleged irregularities reported during the
conduct of the examinations.
The DBE is still urged to address the directives for compliance
issued in the 2018 quality assurance report.
52
Approval of the results
Having studied all the evidence at hand on the
management and conduct of the National Senior
Certificate examinations administered by the
Department of Basic Education (DBE) Umalusi is
satisfied that, apart from instances of irregularities,
there were no systemic irregularities reported that
could have compromised the overall integrity and
credibility of the November 2018 NSC examination,
and as a result the Executive Committee of Council
approved the release of the DBE November 2018
NSC results.
53