THE STATE OF READINESS national senior certificate (NSC) examination and annual national assessment (ANA)

31 AUGUST 2010

Portfolio Committee on Basic Education

Presented by: Nkosinathi Sishi, PhDChief Director: Educational Measurement, Assessment and Public ExaminationsRSA

PURPOSE OF THE SOR

The state of readiness visit in 2010 focused on the following key areas:

1. The readiness of the PEDs to administer the 2010 NSC examination.

2. The status of the monitoring and moderation systems relating to School Based Assessment (SBA) at the Grade 10, 11 and 12 levels.

3. The State of Readiness of the PEDs to administer the ANA in Grade 3, 6 & 9.

4. Readiness of learners to take the assessment

International comparability of QUALITY, STANDARDS, Systems, and practices

Strategic goal:

1.Credible educational measurement, assessment run with integrity2.Quality practices

Strategic objectives1.National assessment and measurement of high international esteem2.Valid and fair educational judgments 3.National historical learner records4.Integrity of Site-based assessment5.Recognition of prior learning6.Credible qualifications, certification and accreditation7.Educational excellence 8.NQF Principles

PrinciplesFair Assessment

Developmental Assessment

1. Assessment should be developmental rather than judgemental,2. Credibility of site-based assessment remains a big challenge.

Fit for Purpose Assessment

1. Valid Assessment, Reliable Assessment , Integration, Relevance 2. Credibility and Standards3. Flexibility and Recognition of Prior Learning,4. Legitimacy, Access and Redress5. Articulation and Portability.

3

Methodology

1. Hypothesis2. Self-evaluation3. Meetings and interviews 4. Site visits: Provincial offices, printing and distribution

centres, districts5. Documents analysis6. Sampling7. Plenary 8. Shortcomings identified in the 2008 and 2009

monitoring process

Specific Aim

1. Establishing system preparation for the 2010 NSC examination

2. Combat readiness for the Annual National Assessment

3. Credibility and integrity of Site-based Assessment4. Preparation of learners for the 2010 national

assessment and public examinations

6

MAKING VALID EDUCATION ASESSMENT JUDGEMENTS

Operational ObjectivesOperational Objectives1. High quality question papers are set and moderated2. Candidates are accurately registered & resulted on the

examination computer system 3. Analytical reports are generated4. Question papers are accurately printed, securely packed and

distributed to examination venues5. All candidates including special needs catered for6. Answer scripts accurately marked and moderated7. All examination processes are conducted with integrity8. Prioritisation of security and accountability in the distribution of

scripts and the collection of answer books

Operational Objectives

1. SBA is completed in accordance with policy prescripts and moderated.

2. All examination and SBA marks are accurately captured.3. All examination irregularities are appropriately managed.4. All candidates are accurately and timeously resulted.5. Effective feedback provided to schools.6. All eligible candidates receive certificates.

National Senior Certificate Examinations 2010

• School-based assessment is ongoing throughout the year.• Practical examinations in Visual Art and Design began on 16 August

2010.• Practical examinations in Computer Applications Technology and

Information Technology to be conducted from 13 to 15 October 2010• The first written examinations on 25 October 2010:– Information Technology Theory, and – Computer Application Technology Theory paper.

• The last examinations to be written on 3 December 2010:– Religion Studies Paper 2– Civil Technology, and – Hospitality Studies

Province Full Time Enrolments 2009 Full Time Enrolments 2010

Eastern Cape70 622 70 852

Free State30 462 29 094

Gauteng101 268 94 546

KwaZulu Natal139 965 131 379

Limpopo93 287 93 806

Mpumalanga55 735 55 885

North West31 395 29 585

Northern Cape10 765 10 399

Western Cape46 861 46 927

Total580 360 562 473

Province Part Time Enrolments 2009

Part Time Enrolments 2010

Eastern Cape3 516 11 141

Free State919 2 022

Gauteng8 278 22 852

KwaZulu Natal12 577 19 990

Limpopo7 834 9 529

Mpumalanga2 989 4 684

North West1 029 2 672

Northern Cape494 1 281

Western Cape1 715 6 047

Total39 351 80 218

2010: No of candidates

Fulltime 562 473

Part-time 80 218

Total 642 691

2009: No of candidates

Fulltime 580 360

Part-time 39 351

Total 619 711

Integrated Examination Computer System

1. The examination computer system is the backbone of the administration of the NSC - the system is still being enhanced.

2. All candidates have been registered on the Integrated Examination Computer System (IECS).

3. Registration data has been audited by the DBE. 4. A third schedule of Entries has been sent to all schools to ensure

accurate registration of all candidates.5. At the end of August 2010, mark sheets will be printed.6. Letters of admission to the examination will be sent to schools in

September 2010.7. In 2010 a full parallel run on the old system will be conducted to

improve confidence in the processing of results8. Both systems are maintained and supported by the State

Information Technology Agency (SITA)

13

Question Papers1. Security surrounding all processes relating to preparation of

question papers is satisfactory.2. All question papers are set by the Department of Basic

Education to ensure a national standard.3. 260 question papers have been set by panels of examiners and

moderated subject specialists.4. The question papers have been moderated by Umalusi to ensure

appropriate standard5. The first batches of question papers have been handed to

PEDs for printing and packing6. Final handover of question papers is 10 September 2010.7. Question papers are being brailled for blind candidates and

adapted for deaf candidates.

Printing, Packing and Distribution

1. All staff working in the printing and packing sections have security clearance and have signed confidentiality agreements.

2. PEDs have received the first two sets of question papers printing.3. Question papers will be distributed to examination centres on the

morning of the examination, except where extreme distances are involved.

15

Preparation for the examinationPreparation for the examination

1. Invigilators who supervise the writing of exams will be trained across all PEDs in September.

2. The principal is usually the Chief Invigilator.3. Detailed regulations govern the conduct of exams and the

managing of irregularities – DBE has issued a Guideline document to assist this process

4. Markers and chief markers have been appointed/ will be appointed in September 2010.

5. Marking centres have been finalized and markers will be trained before marking commences.

16

Answer scripts

1. Each answer script is the product of a candidate’s 12 years at school and his/her performance is encapsulated in the script.

2. In 2010, emphasis is being placed on the return of answer scripts, and management of scripts during marking process:– All scripts to be returned to the district office the same day– Scripts to enjoy the same attention and security as the question

paper. – All scripts correctly labelled with exam number and subject.

• Scanning of scripts to be introduced by all PEDs as soon as possible. – All scripts to be checked at each stage of the process.– Implementation of sanctions where Chief Invigilators do not

manage the return of scripts correctly.

17

Examination Centres

1. All examination centres are audited to ensure adequate resourcing.

2. All examination centres are monitored, with a special focus on examination centres with a history of irregularities.

3. Centres that do not satisfy national criteria are de-registered and alternative centres are arranged. – This includes private centres.

18

MarkingMarking

1. The quality of marking will be the focus in 2010.2. Marking centres have been established across all PEDs based on

the prescripts contained in the Policy.3. Markers have been appointed based on the PAM criteria.4. Training of markers is ongoing and specialised training sessions

will be organised prior to the marking sessions.5. National Marking Guideline discussions will be hosted in Pretoria to

ensure standardisation of marking across all PEDs 6. Internal moderators to be appointed for each subject at each

marking centre.7. DBE will appoint external moderators to evaluate the marking at

each centre.

19

School Based Assessment (SBA)School Based Assessment (SBA)

1. School based assessment constitutes 25% of the final promotion mark.2. SBA is moderated at school, district and provincial level.3. PEDs monitor the collection of marks from schools quarterly.4. DBE issued a Guideline document to assist PEDs with the collection,

control and capture of SBA marks to address shortcomings experienced in previous examinations, such as outstanding marks.

5. SBA marks are currently being moderated by the PEDs and capture will be completed by 15 November.

20

Processing and Release 2010

1. Marking will be completed by 16 December across all PEDs.2. Mark Capture will be completed by 18 December .3. A technical team will prepare proposals in respect of adjustments

for the pre-standardization meeting – 21 December. 4. Pre-standardization meetings will be held on 22 & 23 December. 5. Umalusi standardisation meeting to be held on 24 December. 6. Results will be thoroughly checked by DBE, PEDs and Umalusi on

28 – 30 December.7. Umalusi approval meeting to be held on 31 December 2010.8. Results will be released on 6 January 2011.

21

Analysis and FeedbackAnalysis and Feedback

1. Detailed analysis of results per school, per district, per province, per subject will be made available.

2. Reports from chief markers/internal moderators will be consolidated for distribution to schools.

3. Outcomes of the Umalusi standardisation meetings will be made available to curriculum specialists.

4. Workshops will be conducted with teachers and subject advisors.

5. Under-performing schools will be brought to account and improvement plans will be designed.

6. Curriculum specialists to develop subject improvement plans.

Overview of State of Readiness1. The monitoring exercise was severely impacted on by the public service

strike - not possible to visit district offices and schools in North West & Northern Cape.

2. It was established that schools are filtering candidates in respect of:1. progression to Grade 12 and2. the subjects that they may offer.

3. School Assessment Irregularities Committees are not yet functional. 4. The collection and reconciliation of answer books from examination centres

the exams needs attention. 5. At several district and circuit offices storage facilities were inadequate.6. In the electronic age, security of examinations now also means securing

electronic copies of the question papers – this needs attention.7. Varying SBA moderation practices within and across the provinces do not

give confidence that all candidates treated in a fair & equitable manner. 8. DBE is engaged in the process of developing guidelines for moderation

which should resolve the current plethora of practices.

23

Progress in Mpumalanga

1. DBE is managing the NSC, SC and ABET examinations in the province and working collaboratively with the regional staff.

2. DBE staff are deployed to the province.3. Certain functions have been decentralised to regions while DBE will

manage centralised functions. 4. Focus is on establishing improved systems and processes and

building staff capacity.5. Supplementary, Senior Certificate and June ABET Level 4

examinations have been successfully administered.6. All question papers to be printed and stored in Pretoria, and will

delivered on a weekly basis to the regional offices or nodal areas and daily to the examination centres.

7. Scripts will be collected from schools daily

24

the black box

1. McKinsey report: quality of an education system cannot go beyond the effectiveness of class room interaction

2. Low impact of teacher human capital means our measures are inadequate

3. School organization

NSC CHALLENGES NSC CHALLENGES

1. Five week world cup recess may have impacted on the teaching and learning momentum.

2. The teacher strike has resulted in a school shut down.3. NSC policy is still not fully understood by learners, parents and

educators.4. Security of question papers.5. Introduction of a new examination computer system.6. Moderation of the quality and standard of SBA needs attention.7. Conduct of examinations at private centres is not always

satisfactory.

2010 ANA STATE OF READINESS2010 ANA STATE OF READINESS

Awareness:1. On the whole provinces ,districts and schools visited are aware of the

continued implementation of ANA in 2010.2. Plans are afoot to strengthen advocacy around awareness especially in

districts and schools.

Communication strategy:1. Advocacy around ANA needs to be strengthened and monitored

1. its purpose2. administration 3. capturing, analysis, reporting, and 4. post assessment improvement plans and support

2. Communication and flow of information needs to be improved to ensure a consistent message communicated at all levels.

2010 ANA STATE OF READINESS2010 ANA STATE OF READINESS

Statistics:

Learner statistics must be verified before sending to DBE– provinces will be requested to deal with shortages.

Storage of test papers in districts:

Inadequate space to store the large quantities of Gr 1 – 6 test papers in districts.

PEDs will need to rent warehousing space in some districts.

RecommendationsFacilitate and maintain rigorous moderation systems for SBA.

1. Develop a policy for the moderation of SBA2. Monitor provincial moderation systems.3. Moderate sample assessment tasks and learner evidence

in the case of Life Orientation – July and October 2010.4. Moderate SBA tasks and learner evidence.

Recommendations

Provide diagnostic feedback to schools.

1. Develop comprehensive analysis (qualitative and quantitative) of data.

2. Detailed reports made available to provinces, districts and schools.

3. Conduct post test analysis of ANA and public examinations.

Overview1. filtering candidates in respect of progression to Grade 12. It

was also evident in visits to schools that restrictions were placed on certain learners in respect of the subjects that they may offer.

2. the establishment and functionality of the School Assessment Irregularities Committee is neglected

Overview

• PEDs need to focus on the collection and reconciliation of answer books from examination centres to marking centres and storage as adequate systems are not in place in all provinces to ensure that all scripts are accounted for.

• Generally storage facilities at Provincial offices are excellent but at several district/regional offices and circuit offices storage facilities were well below the expected norms and standards for a secure storage facility and provinces must improve storage facilities as a matter of urgency.

Overview

• Security is always a concern in examinations but in the electronic age, security of examinations is fast moving away from merely securing physical copies of the question papers. There is now a need to secure the electronic copies of the papers.

• Moderation practices vary across the provinces, and even within provinces. The Department of Basic Education is engaged in the process of developing guidelines for moderation which should ensure that each candidate is being treated in a fair manner.

Overview

• One province has been requested to visit their printing and packing procedures to improve security in this area.

• The Annual National Assessments will be printed nationally this year. Provinces indicated that this would alleviate many problems. The key area of concern at the moment is that schools provide accurate statistics in respect of learner numbers and the Language of Learning and Teaching.

CONCLUSION

• Based on the information that was made available to the monitoring team, it can be concluded that the Provincial Departments of Education have made significant strides towards readiness for the NSC examinations

• However in respect to the annual national assessment, learner preparation has been disrupted by the strikes and it is recommended that these are shifted to February 2011. There are educational reasons to support that such tests are in future conducted earlier than later in the year

ConclusionConclusion1. Learner preparation: Registration, irregularities, subject changes,

item analysis2. Stratification of the learner attainment strategy3. System administration4. Validation of standards5. Credibility of Site-Based Assessment6. Logistics regarding the ANA7. Analysis of ANA results8. Item writing, banking and discrimination9. Promotion requirements10. Release of results11. Top candidates