5th NATIONAL

STAKEHOLDERS

CONFERENCE

Dissemination of

National Achievement Test (NAT)

Findings 2016-17

National Education Assessment System

Ministry of Federal

Education & Professional

Training Islamabad

(Pakistan)

January 24, 2018

NAT 2016 – Today’s presentation contents

NAT-2016 at a glance

Grade 4 - Key highlights

Grade 8 - Key highlights

Physical facilities vs Student score

Research questions

HLM analysis findings at school & student level

Way forward.

………………………………….

Short briefing on TIMSS study

Short briefing on UNESCO ‘Participation study’

NAT 2016 at a glance – Grades & sample size

Assessment Grades: 4 and 8

PPS sample : 30,000 students

No. of schools planned: 1500 (20 students per school)

Subjects assessed:

Grade 4: Mathematics, Urdu reading & Urdu Writing

Grade 8: Science, English Reading & English Writing

Test language: Urdu, Sindhi and English

Schools covered: Public, Private, NCHD, BECS & Community model schools

NAT 2016 –Highlights - Assessment instruments

1. Standardised test (2 booklet ‘A & B’ in each subject)

2. Background questionnaires:

a) Students and their parents

b) Subject teachers (Mathematics, Science, Urdu/Sindhi, English)

c) Head teacher

3. Test administrator manual

4. Provision of stationary items for 30,000 sample students

NAT 2016 – Highlights - Stationary items

Stationary items with NEAS logo: Clip board, one pouch

containing: ballpoint, pencil, eraser, sharpener, scale, protractor

NAT 2016 at a glance – Grade 4 sample

Source: NEMIS data base 2015

750 750100.00

15000

13197

87.980

2000

4000

6000

8000

10000

12000

14000

16000

Planned No. Response No. % Response

Schools Students

NAT 2016 at a glance – Grade 8 sample

750 74999.87

1500014287

95.250

2000

4000

6000

8000

10000

12000

14000

16000

Planned No. Response No. % Response

Schools Students

NAT 2016 at a glance – District covered

Source: NEMIS data base 2015

155

10

31

13 101

2536

29

139

90

10 20 13 7 124

3529

142

92

10

24

11 81

23

3629

0

20

40

60

80

100

120

140

160

180

Total District/Regions District Covered Grade 4 District Covered Grade 8

Province/ Regions Total District/RegionsDistrict Covered

Grade 4 Grade 8

National 155 139 142

% Response 100% 90 % 92%

AJK 10 10 10

Balochistan 31 20 24

FATA 13 13 11

G.B 10 7 8

ICT 1 1 1

KPK 25 24 23

Punjab 36 35 36

Sindh 29 29 29

NAT 2016 at a glance – Characteristics

Assessed Grade 4 & 8 students before annual examination;

Winter and summer zones students were assessed separately;

New questions included in BGQs like nutrition, SES, health;

Policy questions were framed according to international standard

and shared with provinces/areas before NAT 2016;

National Steering Committee constituted under Chairmanship of

Federal Secretary, M/o FE&PT; and,

Conducted without the financial and technical assistance of the

international partners.

NAT 2016 at a glance –Partner institutions

Punjab Examination Commission (PEC) Lahore Punjab;

Balochistan Examination & Assessment Commission

(BEAC), Quetta;

Provincial Education Assessment Centre (PEACE)

Jamshoro Sindh;

Provincial Education Assessment Centre (PEACE)

Abbottabad, Khyber Pakhtunkhwa;

Kashmir Education Assessment Centre (KEACE),

Muzaffarabad AJK;

G.B Education Assessment Centre Gilgit; and,

FATA Education Assessment Centre Peshawar.

NAT 2016 at a glance – Services utilised

For Printing of assessment tools: Printing Corporation of

Pakistan (PCP), Islamabad

Sending & receiving assessment bags: Pakistan Post Office

(PPO) Islamabad

For data entry: National Institute of Psychology (NIP)

Quaid -e- Azam University for 125,000 tools

NEAS 5 Experts Trained120 LMTS at national level

Trained 3300 Test Administrators (TAs) at provincial

level Assessed 27484 students + 27484 parents +

1499 Head Teaches + 6000 Teachers at school level

110 monitors deputed by M/o FEPT and provinces/regions

NAT 2016 at a glance – Software used for analysis

1. Microsoft Excel

2. SPSS Statistical Package for the Social Sciences

3. ConQuest (Generalised Item Response Modelling-IRT analysis)

4. HLM (Hierarchical linear modelling)

5. AM (AM is a statistical software package for analysing data from

complex)

6. Iteman (provide detailed item and test analysis reports using

classical test theory (CTT)

7. StatPlanet for visualization

Grade 4 - Key highlights

(i) Mathematics (ii) Urdu Reading (iii) Urdu Writing

Understanding NEAS scale (mean 500)

Mean

Scale

0 100 200 300 400 500 600 700 800 900 1000

-500 -400 -300 -200 -100 0 +100 +200 +300 +400 +500

Standard Deviation

The scale used in NEAS is the same as in TIMSS, PISA, SACMEQ, TOEFL, Vietnam, LOAS and many other develop & developing countries. The range of scaled scores is between 0-1000.

The achievement scales are constructed so that a student achieving 50% correct marks receives a scaled score of 500 and the Standard Deviation of the scale is set at 100.

Grade 4 – Maths score by nation & regions (mean = 500)

462

459

465

469

479

491

500

548

484

400 420 440 460 480 500 520 540 560

AJK

GB

KPK

FATA

Balochistan

ICT

Sindh

Punjab

National Score

Mathematics Score

Score

Grade 4 – Urdu result by nation & regions (mean = 500 & combined mean = 1000)

464 469 452 473 485 492 513 536 485

431 453 475 478 490 525 514 543489

895 922 927 951 9751,016 1,027

1,079

974

0

200

400

600

800

1,000

1,200

Urdu Reading, Writing & Combined Score

Urdu Reading Urdu Writing Combined Score

17

481

493

497

502

508

523

Grade 4 – Subject results by gender (mean 500)

18

518

489

494

511

508

495

Grade 4 – Subject results by location (mean 500)

Winter Zone

Summer Zone

19

510

453

462

508 508

463

Grade 4 – Subject results by school zone (mean 500)

20

519

515

471

489

504

508 507

501

490

Grade 4 – Subject results by administrative gender (mean 500)

21

511

508

487

483

511511

514

499

482

Grade 4 – Subject results by medium of instruction (mean 500)

22

547

521

519

484

493

506

510

527

537

514

509

490

Grade 4 – Subject results by school level (mean 500)

23

553

508

496

472

494

499

503

511 511

499

491

448

Grade 4 – Subject results by school type (mean 500)

Student score

Teacher score

24

489 485 484

Grade 4 – Subject results teachers & students (mean 500)

Each 'X'

represents 27

students

Items which were below

average difficulty

Item which were above

average difficulty

0

Item-Person Map: Grade 4 Math

Each 'X'

represents 27

students

Items which were below

average difficulty

Item which were above

average difficulty

0

Item-Person Map: Grade 4 Math

26

Grade 4 – Mathematics (mean = 500)

China

Iran

Arabian sea

Grade 4 - Urdu/Sindhi – Reading (mean = 500)

China

Iran

Arabian sea

Grade 4 - Urdu/Sindhi – Writing (mean = 500)

China

Iran

Arabian sea

Grade 4 – Combined Score (mean = 1500)

China

Iran

Arabian sea

Mathematics score

Writing s

core

Grade 4 – Maths vs Writing Score (mean 500)

Average in writing but very low in maths

Very high in maths but very low in writing

NAT 2016 Grade 8 – Subject results Grade 8 – Key highlights(i) Science (ii) English Reading (iii) English Writing

Grade 8 – Science result by national & region (Mean=500)

439

452

463

473

474

476

477

566

478

0 100 200 300 400 500 600

AJK

GB

KPK

Sindh

ICT

Balochistan

FATA

Punjab

National Score

Science Score

Grade 8 –English result by nation & region (mean=500

& combined mean = 1000)

459 464 466 496 466 475 521 555 488

448 454 461 473518 512

557 549496

907 918 927969 985 986

1,078 1,104

984

0

200

400

600

800

1,000

1,200English Reading, Writing & Combined Score

English Reading English Writing Combined Score

35

488

520

526

484

492

512

Grade 8 – Subject results by gender (mean 500)

36

488

519

484

495

506

515

Grade 8 – Subject results by location (mean 500)

Winter zone

Summer zone

37

471

505505

467465

506

Grade 8 – Subject results by school zone (mean 500)

Urdu

English

Sindhi

38

463

494

519

533

486

453

477

502

504

Grade 8 – Subject results by medium of instruction (mean 500)

Grade 8 – Subject results by level of school

39494

500

540

579

632

535

499497495

500

527528

40

495

556

493

498

519

541

Grade 8 – Subject results by type of school (mean=500)

Student score

Teacher score

41

488

620621

496478

661

Grade 8 – Subject results teachers & students (mean 500)

Each 'X'

represents 24

students

Items which were below

average difficulty

Item which were above

average difficulty

0

Item-Person Map: Grade 8 Science

Each 'X'

represents 27

students

Items which were below

average difficulty

Item which were above

average difficulty

0

Item-Person Map: Grade 8 Science

43

Grade 8 – Science (mean = 500)

China

Iran

Arabian sea

Grade 8 – English Reading (mean = 500)

China

Iran

Arabian sea

Grade 8 – English Writing (mean = 500)

China

Iran

Arabian sea

Grade 8 – Combined (mean = 500)

China

Iran

Arabian sea

NAT 2016 Grade 8 – Subject results Physical facilities Vs students score

Grade 4 - Combined score vs school facilities

Grade 4 – School facilities composite index

China

Iran

Arabian sea

Grade 8 - Combined score vs School facilities

Grade 8 – School facilities composite index

China

Iran

Arabian sea

Average percentage of school facilities

3136 36

4245 48

57

64

48%45% 46% 48%

51% 49%

58%

67%

0

10

20

30

40

50

60

70

80

Grade 4 Grade 8

Research questions & HLM analysis findings

Research questions

At Student level – Level -1

1. To what extent are student characteristics associated with Grade 4 and

Grade 8 learning achievement in Pakistan?

2. To what extent are instructional support factors associated with Grade 4

and Grade 8 students learning achievement in Pakistan?

3. To what extent are student family SES associated with Grade 4 and

Grade 8 student learning achievement in Pakistan?

At School level –Level -2

4. To what extent are teacher characteristics and instruction-related

(classroom environment) factors associated with Grade 4 and Grade 8

student learning achievement in Pakistan?

5. To what extent are head teacher characteristics and school-related

(school environment) factors associated with Grade 4 and Grade 8

student learning achievement in Pakistan?

Two-level hierarchical linear modeling for student achievement

Student Achievement

Student SES

Time Factor Student Characteristics

Home Environment

H/Teacher Characteristics

Teacher Characteristics

Management Factors

Classroom Environment

School Environment

Satisfaction

Factor

Learning Support

At school level -2

At student level-1

Findings at school level

Hierarchical linear modelling (HLM)

NAT 2016 – Urban-rural differences

Student attending school located in urban areas did better in all the subjects

.

As compared to the student attending schools located in rural areas.

NAT 2016 – Students Absenteeism

Students in schools with less student absenteeism did better in all subjects

than those in schools where absenteeism is common among students.

NAT 2016 – Experienced school leadership

Students of such schools which have head teachers with higher years of experience performed better

Than students in schools with head teachers with fewer years of experience.

NAT 2016 – Student involvement in co-curricular

activitiesStudents in schools taking part in

co-curricular activities significantly better

Than those students who have

not in taking part in co-curricular activities.

NAT 2016 – Course completion by teachers

Students who reported that they

had completed all of the course

book did better

As compared to those students who

said they had not completed the

course book.

NAT 2016 – Use of local language in classroom teaching

Students scored worse when

teachers used local language as

medium of instruction/teaching at

grade 8 level

As compared to those whose

teachers did not use local

language.

NAT 2016 – School with boundary wall

Students in schools with boundary

wall did better in all the subjects

than their counterparts in schools with

non-functional wall or without

boundary wall.

Findings at student level

Hierarchical linear modelling (HLM)

NAT 2016 – Student age factor

Younger students performed better in all subjects

As compared to older students of the class.

NAT 2016 – Books at home

Students who had books for use at home did better in all subjects

As compared to those who had no such books at home.

NAT 2016 – Getting homework

Students in schools where

teachers were given homework

frequently, scored better

As compared to those who were

not given homework frequently

NAT 2016 – Checking homework by the teacher

Student who had their homework

checked by their teachers more

frequently did better in all the

subjects

As compared to those who had

homework rarely or never checked

by their teachers.

NAT 2016 – Feedback (correction) on homework

Students in schools where homework mistakes were pointed out by teachers scored better

As compared to those schools where such facility was not provided.

NAT 2016 – Home possession

Students from homes with more home

possession items (i.e. calculator,

computer, internet, dictionary, radio,

mobile phone and landline phone) did

significantly better in all subjects,

When compared to student from homes with fewer home possession items.

NAT 2016 – Educated parents

Students whose parents were

highly educated did better in all

the subjects

As compared to their counterparts whose parents were less or not educated.

NAT 2016 – Paid tuition

Students who did not take paid

tuition performed significantly better

As compared to those students

who took paid tuition.

NAT 2016 – Helping parents

Student who said that household chores affected their school work did worse in all Students

As compared to their counterparts

who said they were not involved

in household work.

NAT 2016 – Distance travel to school

Students with less travel distance to

school outperformed their

counterparts

Who travelled long distances to

school.

Way forward

# Description – Expenditure on Rs. In Million

1 Printing cost of assessment tools through PCPP. 4.505

2 Training of 120 Lead Master Trainers. 4.050

3 Stationary for 33,000 students 2.5

4 4.1 Test Administration Training in Provinces 4.2 Conduct of Test Administration in 1500 schools. 3.3 Monitoring of NAT

12.781

5 Total Expenditure 23.836

6 Expenditure paid till date. 19.358

7 Pending claims of provinces / areas assessment centers 4.478

8 Per student cost = Rs.722.00

NAT 2016 - Expenditure

Conceptual framework model: Factors influencing

student learning achievement

Outside School Factors

1.0 SUPPORTING INPUTS

1.1 Strong parent and

community support

1.2 Effective support from the

education system

1.3 Adequate material support

1.3.1 Frequent and

appropriate teacher

development activities

1.3.2 Sufficient textbooks and

other resource materials

1.3.3 Adequate physical

facilities

CHILDREN’S

CHARACTERISTICS

3.0 SCHOOL CLIMATE

3.1 High expectations of

students

3.2 Positive teacher attitudes

3.3 Oder and discipline

3.4 Organised curriculum

3.5 Rewards and Incentives

Outside School Factors

2.0 ENABLING CONDITIONS

2.1 Effective leadership

2.2 A capable teaching force

2.3 Flexibility and autonomy

2.4 High amount of time-in-

school

5.0 STUDENTS

OUTCOMES

5.1 Student participation

(attendance,

promotion,

graduation,

continuation)

5.2 Academic

achievement

5.3 Social skills

5.4 Economic success

after school

4.0 TEACHING/LEARNING PROCESS

4.1 High amount of learning time

4.2 Variety in teaching strategies

4.3 Frequent homework , checking and

feedback

4.4 Frequent formative assessment and

feedback

Contextual Factors

Socio-Cultural

Socio- Economic

Political

Way forward

Constitution of National/Provincial Task Force for implementation &

monitoring of NEAS findings;

Clear guidelines as medium of instruction in schools before TIMSS

assessment;

Conducting National / Provincial / Area level seminars for awareness

purposes for sharing the NAT 2016 findings accurately;

Conducting regional levels studies in districts where interesting findings are

noticed;

Alignment of Assessment Studies at National and Provincial level;

(Cont…)

79

Way forward

80

Consultation process with provinces to ensure the correct

transformation of National Curriculum SLO’s into new Text Book

contents;

Inclusion of agenda point in forthcoming IPEMC regarding TIMSS

For sustainability purposes the Restructuring of NEAS is important

immediately; and,

Rationalization of NEAS HR and technical capacity.

neaspakistan@gmail.com

www.neas.gov.pk

www.facebook.com/neaspakistan

Thank you for your patience