2 ASSET is a registered trademark and Mindspark is a trademark of EI

WHY THIS IDEA HAS LARGER RELEVANCE

‘Student Learning Study (SLS)’ is a benchmarking study of student learning conducted by Educational Initiatives (EI), with the financial support of Google.org, USA. The study has been carried out in 48 districts in 18 states and 1 Union territory of India. About 101643 students from 2399 selected government schools studying in Classes 4, 6 and 8 were tested in language and maths. The test papers for each class were developed commonly in 13 languages. This working paper is a sequel to the last 2 papers on the Student Learning Study and explores the performance of the different states that participated in the study. Although comparisons of performance are provided in this paper for the purpose of creating cross-learning and remedial opportunities, it is to be noted that the overall learning level of even the best performing states is low and renders much scope for learning improvement.

COMPARISON OF OVERALL PERFORMANCE Based on the relative performance/rank of the states in different classes and subjects, an attempt was made to consolidate the performance of 17 states. This showed that there are significant state-wise differences in student performance across states. As could be seen in Graph 1, Kerala, Maharashtra, Orissa and Karnataka were clearly performing overall better than the national average. Jammu and Kashmir, Madhya Pradesh and Rajasthan were among the states that ranked among the bottom three overall. Bihar performed the same as the national average.

The overall percentage of students scoring zero across all states tested in India was 0.8% for all classes and subjects. However, the percentages of zero scorers were much higher for some states (especially Madhya Pradesh, Jammu and Kashmir, Rajasthan and Gujarat), clearly showing that these states had students falling behind in learning. Further, a comparison of information in Graph 1 with that in Graph 2 reveals that the extent of students scoring zero and the overall performance of a state showed differences. Andhra Pradesh, Assam, Chhattisgarh and Jharkhand, although below the national average, have fewer students scoring zero in the test compared to better performing states such as Haryana and Karnataka. This indicates that while the efforts taken by the states’ performing below national average could be addressing the lowest ability students, they do not provide adequate support for overall improvement of all students. Similarly Haryana’s and Karnataka’s efforts are not addressing the low ability students who are scoring zero. Jammu and Kashmir, Madhya Pradesh and Rajasthan which ranked among the bottom 3 in overall performance also had the highest number of students scoring zero indicating that these students are being left behind in these states.

SLS MAIN FINDINGS Learning levels are extremely low. The learning that happens is not ‘Learning with understanding’ and a number of misconceptions exist among students on the concepts learnt. Learning gains, seen across classes, is slightly incremental and not a large jump. Students find it difficult to express their thoughts in their own words in writing. In all the states tested, fewer students were found to comprehend what they read. There are significant state-wise differences in student performance across states. The extent of students scoring zero and the overall performance of a state showed differences. The levels of learning of Indian students in government schools in class 4 and class 8 tested is much lower than the international average as

represented by studies like Trends in International Maths and Science Study (TIMSS) and Progress in International Reading and Literacy Study (PIRLS).

Detailed report of the study could be accessed at http://www.ei-india.com/research-student-learning-study/

3 © Educational Initiatives Pvt. Ltd.

CAN DIFFERENT STUDIES REPORT DIFFERENT PERFORMANCE RANKINGS Measuring student learning is a complex exercise and the performance reported is influenced by a number of factors. Assuming that the student samples are selected by robust procedures in all studies, test scores would further depend on – a. What is measured: Test papers used in different studies may not measure identical things. For example, questions could measure rote

learning or questions could measure deeper understanding of concepts; and there may be differences in the length of the paper, for a longer paper may mean that more information has been obtained to arrive at conclusions.

b. How a test is administered: The way the test is administered is very important to come to conclusions about student learning. For

example, different tests may be administered in a classroom or outside a school environment; testing may be carried out by the teacher who is close to the class or evaluators who have been specifically trained for standardised administration; teams in different states may be synchronised to follow identical processes or different state teams may follow processes that have inherent differences while administering the tests, and so on, all of which will throw up changes in the test scores.

Could some states do better due to the districts that were selected for the study?

In SLS, the districts were selected by sorting and dividing them into groups based on the Human Development Index (HDI) or Literacy rate where HDI was not available. (HDI is a well accepted standard measure devised by UN to measure well being, especially child welfare. HDI uses education and hours of schooling as one of its key components while building the index). As HDI reflects the well being and literacy of the districts, this method of selection ensures a better representation of the state by taking into account the variability of districts in terms of education. This method also optimises representation even if the number of districts that could be tested is not large.

The districts were distributed into four groups (High Tail, Above Median, Below Median and Low Tail) if four districts were to be sampled and into two groups (Above Median and Below Median) if two districts were to be sampled. One district per group was randomly selected. An equal number of students were tested in each district. For example, in Orissa, while one of the districts was Khurda (Bhubaneshwar) a better district in HDI, the other district was Koraput – third from last in the state’s HDI. Similarly, in Madhya Pradesh, Bhopal was selected from ‘High Tail’, Hoshangabad from ‘Above Median’, Betul from ‘Below Median’ and Panna from ‘Low Tail’. The districts thus sampled were well distributed along the HDI scale to ensure variability and appropriate representation of the state.

In order to ensure (a) the sample size is adequate, (b) that the sample allowed for design / cluster effect caused due to intra class correlation (as all students are tested in a class), and (c) the districts selected are representative of the student learning in the state, the following steps were taken:

The number of students tested per state was proportional to the size of the state. The sample size was checked for 95% confidence level and +5 % precision. The sample size was adjusted for design effect to allow for intra class correlation. The number of districts per state was decided by keeping the minimum students to be tested per class at 750 students per

district ( i.e., either two districts or more than two to be selected) The districts were sorted based on HDI and then divided in two or four groups and one selected from each. The schools were stratified (for e.g., urban and rural schools, category of schools as primary, upper primary schools, high

schools, etc) and the sampled based on probability proportional to size technique.

4 ASSET is a registered trademark and Mindspark is a trademark of EI

OVERALL PERFORMANCE IN DIFFERENT CLASSES AND SUBJECTS

A high correlation of 0.85 and above was observed in the performance across the classes and subjects tested in each state. This means that if a state does well in a class or subject compared to other states, then it more or less tends to do well compared to other states in other classes and subjects too.

Kerala, Maharashtra, Orissa and Karnataka are clearly doing well in almost all classes and subjects tested. However, Maharashtra has

to focus on Language in class 8 and Karnataka has to do so in class 4. Punjab starts to catch up from class 6 in language and needs attention in maths overall. Haryana starts to catch up in class 8 in both

language and maths but needs to focus on lower classes. Although Uttarakhand is above the national average in most classes tested, the observed difference is practically not substantive

enough. Tamil Nadu does well in language in class 4, but starts performing lower than the national average in class 8. Similarly, Chhattisgarh is similar to national average in language in class 4, but is substantively lower than the national average in all other classes and subjects.

Bihar and Jharkhand show a substantively low performance compared to national average in class 4 language and are below national average in other classes and subjects. Gujarat is substantively low in maths in class 4 and 6. Andhra Pradesh and Assam are below the national average in all classes and subjects with Andhra Pradesh substantively lower from class 8 in language and class 6 in maths, while Assam is substantively lower in all classes except in class 8 language. Rajasthan is substantively much lower in class 4 and 6 in language and maths, while Madhya Pradesh and Jammu Kashmir are substantively low in all classes and subjects.

5 © Educational Initiatives Pvt. Ltd.

COMPARISON WITH INTERNATIONAL STUDIES

The overall finding of the study reported that the level of learning of Indian students in government schools in class 4 and class 8 tested is much lower than the international average as represented by studies like Trends in International Maths and Science Study (TIMSS) and Progress in International Reading and Literacy Study (PIRLS). The same trend of performance lower than the international average is seen in all the Indian states that participated in the study. This indicates that the prevalence of low learning levels compared to the international average remains consistent across states in this category of schools. It is not that some states perform exceptionally low compared to others.

Which of these has the same value as 342?

A. 3000 + 400 +2 B. 300 + 40 + 2 C. 30 + 4 + 2 D. 3 + 4 + 2

Sample Question: Internationally 87.0% of students answered this correctly, while 43.7% of students from Indian public schools could get this correct.

MathsClass 4

Subtract: 6000 -2369 A. 4369 B. 3742 C. 3631 D. 3531

Performance of Class 8 Students

Group %

correct

India 59.1

International students (TIMSS) 88.0

Sample Question: As per national curriculum framework the concept of 4 digit subtraction with borrowing is introduced at class 5 level. When a question based on this concept was asked in Class 8, Indian students performed lower than international students.

MathsClass 8

Language Class 4

Passage Excerpt: When Lakhan discovered that he had mice in his house, it did not bother him much at first. But the mice multiplied. They began to bother him. They kept on multiplying and finally there came a time when even he could stand it no longer.

Sample Question: Internationally, 79.0% of students answered this correctly, while 41.8% of students from Indian public schools could get this correct.

Why did Lakhan want to get rid of the mice?

A. He had always hated mice. B. There were too many of them. C. They laughed too loudly. D. They ate all his cheese.

Question involves retrieving stated information in the text and making straight forward inferences from it.

6 ASSET is a registered trademark and Mindspark is a trademark of EI

PATTERNS IN MISCONCEPTION ANALYSIS Students develop a cognitive understanding of the world around them through interactions based on their daily experiences. Teachers and schools help build this understanding. ‘Misconceptions’ are concepts that students acquire which are not in line with or do not match the conventional expert view in that topic. Generally, student misconceptions persist until students recognise that their understanding is flawed. The misconceptions identified in the student learning study for language and maths in classes 4, 6 and 8 are listed in the main report. As given in the example below from Class 6 maths and Class 4 language, the misconceptions were found to hold true for all states, although the degree of misconception varied from state to state. In which of these numbers below does 3 have the greatest place value?A. 136 B. 239 C. 301 D. 743

x:Total score in the paper; y: Percentage of students

Optio

ns

Andh

ra P

rade

sh

Assa

m

Biha

r

Chan

digar

h *

Chatt

isgar

h

Gujar

at

Harya

na

Jamm

u and

Kas

hmir

Jhar

khan

d

Karn

ataka

Kera

la

Madh

ya P

rade

sh

Maha

rash

tra

Oriss

a

Punja

b

Rajas

than

Tami

l Nad

u

Uttar

akha

nd

Natio

nal

A 4.8 5.8 3.4 4.6 5.1 1.6 5.9 8.1 6.4 1.4 0.1 6.2 2.7 2.9 2.3 7.9 3.4 5.5 4.2

B 6.5 10.7 4.8 2.4 5.3 5.9 7.0 10.0 7.3 1.6 1.4 8.1 5.5 8.1 3.7 11.8 7.9 8.2 6.5

C 36.8 42.7 41.4 32.0 43.1 40.4 29.8 20.6 28.9 41.5 1.6 28.9 51.4 61.2 27.7 31.6 30.4 34.6 35.8

D 33.7 18.4 28.7 49.6 28.3 36.4 40.6 27.3 41.4 47.1 94.9 28.8 31.5 18.2 55.8 26.3 51.8 32.0 37.4

Nationally about 35.8% of the students think that greater the number, the greater is the place value of all its digits. Some students may also have chosen option C because in the number 301, 3 is the greatest digit. The state level data reveals that most students from all states choose the same common wrong answer, however the extent of students choosing this is found varying, for example, while in Kerala 94.9% are choosing the incorrect option D, in Orissa 18.2% students are choosing the same. Passage excerpt: ….. But the babies cannot dig their way out of the sand above them. They peep out and make some sound while hatching out of their shells. At that time, the mother, who has been guarding them hears them. She then digs them free.

Why does the mother crocodile dig her babies out of the sand? A. The babies cannot come out by

themselves. B. She is playing a game with them. C. She is saving them from enemies. D. The babies start fighting with each other. x:Total score in the paper; y: Percentage of

students

Optio

ns

Andh

ra P

rade

sh

Assa

m

Biha

r

Chan

digar

h *

Chha

ttisga

rh

Delhi

Gujar

at

Harya

na

Jamm

u and

Kas

hmir

Jhar

khan

d

Karn

ataka

Kera

la

Madh

ya P

rade

sh

Maha

rash

tra

Oriss

a

Punja

b

Rajas

than

Tami

l Nad

u

Uttar

akha

nd

Natio

nal

A 51.9 32.0 36.9 38.5 41.0 35.4 45.2 32.1 25.3 32.2 43.6 46.5 31.3 40.2 39.0 41.9 37.7 59.4 30.8 41.5

B 6.4 9.2 4.3 4.0 7.9 4.4 7.0 4.3 16.1 7.8 3.8 4.3 6.5 5.5 2.7 4.7 6.2 7.5 8.2 6.0

C 20.8 35.4 26.2 34.8 27.4 50.3 17.0 24.7 13.7 32.1 29.7 40.1 22.5 41.0 45.9 26.0 21.1 22.3 41.6 29.3

D 3.8 2.9 3.1 6.3 3.6 2.4 5.5 3.3 8.4 4.7 3.4 2.0 4.9 3.3 1.3 10.6 6.3 1.7 2.5 3.9

In the passage, the reason as to why the mother crocodile dig the babies out of the sand precedes the action of the mother crocodile digging them out. There is also an intervening sentence between the sentences giving the reason and the actual action. This could also have made it difficult for the students to connect the two pieces of information. However, about 40% of the students have managed to do this. Students of most of the states are clearly making a common error by choosing option C as the answer, especially more students in Delhi (50.3%), Orissa (45.9%), Uttaranchal (41.6)% and Maharashtra (41.0%) are choosing the wrong answer more than the correct answer.

x

Maths Class 6

Language Class 4

y

y

x

7 © Educational Initiatives Pvt. Ltd.

TRENDS IN SKILL-WISE PERFORMANCE Analysis of performance in different skills often revealed that individual states that had relatively lower performance compared to other states in one skill also had relatively lower performance in other skills, as could be seen from the graph below.

Similar trends were also observed across the states, among the strong and weak skills. For example, in maths Class 4, most of the states showed that ‘Number concepts’ was a strong skill, while for some states like Assam, Bihar, Chandigarh and Jammu & Kashmir ‘Operations on whole numbers’ was a strong skill. The weak skill across the states was found to be ‘Measurement and applications’ except for states like Jammu & Kashmir and Maharashtra, where the weak skill was ‘Problem solving’ and ‘Fraction concepts’ respectively. In Language, as could be seen from the table below, while vocabulary and its usage were found to be strong skills, the weak skill across classes was reading comprehension.

S. No Subject Class Strong Skill Weak Skill 1 Language 4 Knows names of objects, birds and animals not seen in

daily life and words denoting actions and feelings Reads short stories independently, Retrieves stated information, Makes straightforward inferences, Interprets and integrates ideas and information, Examines and evaluates content, language, and textual elements

2 Language 6 Knows a wider range of names of objects, birds and animals not seen in daily life and words denoting actions and feelings

3 Language 8 Uses words appropriate to the context 4 Maths 4 Number Concepts Measurement and applications 5 Maths 6 Number Concepts Area and Perimeter 6 Maths 8 Number sense, related concepts and basic number

competency Mensuration - Area and Perimeter, Volume and Surface Area

The performance in Language Oral Reading Test showed that about 60 or more percent of students were able to do all the questions in the test in Delhi, Orissa, Tamil Nadu and Punjab. In all the states tested, fewer students comprehended what they were reading. For e.g., in Gujarat while more than 87.1% of students could read a simple word, only 40.3% could read a short passage and only 22.6% could comprehend the information implicit in the passage they read.

STATE SPECIFIC RECOMMENDATIONS Student Learning Study shares overall recommendations in its report. State specific findings and recommendations for each state are also made available online at the following link http://www.ei-india.com/research-student-learning-study/. States need to disseminate these findings to schools, SCERT, DIET, etc. so that appropriate action can be taken.

8 ASSET is a registered trademark and Mindspark is a trademark of EI

ABOUT EI

As a part of the SLS project, a student interviews video study was conducted in 36 classrooms across 3 states – Rajasthan, Gujarat and Andhra Pradesh to understand why students answer certain questions in the ways they do. The interviews were carried out by trained EI education specialists who probed the students keeping in mind not to prompt or show approval or disapproval while exploring the answers given by the students. During the interviews the atmosphere established was that of a casual interaction in which students were allowed to express freely their ideas and thoughts. The response we often get from students in such a non-judgmental interview atmosphere shows us that students usually like to participate and enjoy the same. The interviews are a means to identify the possible reasons for students selecting a particular answer for a given question. They also demonstrate to the teacher to learn to focus on what students learn and think rather than just on what they teach. Sample videos could be accessed in http://www.ei-india.com/sls/

A Class 4 student explaining how he

understood the question. We are an educational research organization that focuses on learning research through assessments. EI has been started by a group of IIM Ahmedabad alumni with first-hand experience of setting up and running educational institutions. It has been formed with a mission to work towards qualitative improvement in India’s educational system and our vision is “A world where children everywhere are ‘Learning with Understanding’”. PROJECTS OF EI: Andhra Pradesh Randomised Evaluation Study (2004 onwards): Done in partnership with Harvard University, Azim Premji Foundation, World Bank and the Government of Andhra Pradesh, this is a longitudinal study across 8-9 years and covers currently 100,000 elementary school kids and measures the impact of various inputs (e.g., block grants, additional teachers) with outcome-based teacher incentives. Assessment of Student Learning in Sarva Shiksha Abhiyan – RGSM, Chhattisgarh (2008 onwards): The test was developed in Hindi and administered to approximately 3 lakh students in about 1900 schools in 16 districts in Chhattisgarh states. The tests have already been conducted for students of class 3 to 8 for Language and Maths and the report submitted to RGSM. Municipal School Benchmarking Study (2004-2007): Supported by ICICI Bank, this study assessed learning in 35,000 municipal school students from class 2, 4 and 6 across the 6 biggest towns in five states- Gujarat, Andhra Pradesh, Rajasthan, Chhattisgarh and Uttarakhand. UNICEF Learning Assessment Study for Quality Education (2005-2006): assessed mathematics and language acquisition among primary school children in the UNICEF quality package schools in 13 states of India. The tests were standardised across 9 languages and involved very intricate development cycle involving language experts from all over India. Teacher Needs Assessment (2008 onwards): is a census study that has been initiated by the Royal Education Council, Government of Bhutan. In this project all teachers of Bhutan are assessed for their general ability, competence in subject knowledge and pedagogical practices. Bhutan Annual Status Student Learning Study (2008-2009): This study was done in partnership with Royal Education Council and Ministry of Education Bhutan. Nearly 34000 students in classes 4, 6 and 8 are tested for learning in Language, Maths and Science in 424 schools. EI’S PRODUCTS AND LEARNING SOLUTIONS: ASSET: is an objective-type, multiple-choice test for students of Classes 3 to 10. It is a scientifically designed, skill based assessment developed in India for Indian schools. It assesses students’ level of proficiency in the skills and concepts underlying the school syllabus and provides them feedback about their strengths and weaknesses. Know more about ASSET at www.ei-india.com Mindspark: is a computer based self-learning programme that helps the child improve her skills. It allows each student to follow a learning path that is based on her need. Mindspark is currently available for Maths for classes 1-10 in English version. Mindspark can be accessed at www.mindspark.in. Rural Mindspark: Hindi version is currently available on demand for some Maths modules. Contact EI to know more about other language versions and modules.

Some Partners / Clients Google Inc. Government of Andhra Pradesh Michael and Susan Dell Foundation Rajiv Gandhi Shiksha Mission, Chhattisgarh Royal Government of Bhutan Suzlon Foundation WIPRO Applying Thought World Bank UNICEF, India CONTACT DETAILS: Educational Initiatives Pvt. Ltd

Head Office Ahmedabad

Bangalore Office

Delhi Office

Hyderabad Office

Mumbai Office

613-615 JB Towers, Opp. Doordarshan Tower, Drive-in Road, Ahmedabad 380054, INDIA Tel: 079 – 40269696, 40269625 Fax: 079-26841400

No.97, 2nd Floor, Robertson Road, Frazer Town, Bangalore-560005 INDIA Tel: 080 – 41657715, 64567049, 41237162

3rd Floor, Lakshey Deep Plaza, A-252A, Sant Nagar, East of Kailash, New Delhi-110065 INDIA Tel: 011-26462264, 40584952

Flat No 401/A, Shri S.K.Vihar, H.No:3-4-578/1, Beside Sri Chaitanya College, Narayanaguda, Hyderabad-500029, INDIA Tel: 040 – 32419534

Row House Number 1,Sea Coast 2, (Near Cidco Guest House), Kille Gaothan, Belapur, Navi Mumbai-400614 INDIA Tel: 022 - 32686329, 27561430

E-mail us at: assessment@ei-india.com Website: www.ei-india.com August 2010