Assessing the Educational Consequences of Teacher Deficits in Content Matter and

Language of Instruction (L2)

Steve Walter - GIAL

Some conventional wisdom: Schools are only as good as their teachers.

Research Context

Location (Cameroon, Philippines) Experimental program testing L1 instruction Early exit program 12 experimental schools; 12 comparison schools Testing began in 2008 Testing in Reading, Math, English All teachers speak L1

Key observation about Kom schools

Standard Schools Mean Mean Experimental Schools

GS Mentang 10.3 31.7 CBC Fujua CS Kindoh 14.1 31.8 GS Ngwah GS Yuwi 14.5 33.9 GS Laikom GS Meli 15 42.1 GS Kitchu CBC Fundeng 20.4 46.9 GS Bolem GS Baichu 23.2 47.9 GS Muteff GS Atondum 24.6 50.5 GS Ameng GS Fundong Village 24.8 50.5 GS Mboh CBC Abuh/Wainchia 26.6 54.6 CBC Kikfuini GS Njinikijem 28.3 64.7 GS Wombong PS Ngwah 36.3 65.6 GS Ilung CS Wombong 37.6 72.6 CBC Belo

Sources of Variation – some obvious and some less so

Some “obvious” hypotheses Motivation (avocation) Commitment to teaching Training and Education Ability and creativity Teacher-centered instruction Low compensation

Some “less obvious” hypotheses Low content mastery on the part of teachers Low mastery of the language of instruction by teachers Low time on task Crowded curriculum requirements Poorly constructed curricula Erratic attendance; automatic promotion Clumsy screening and placement practices

The Investigation Venue:

Government Teacher Training Center in Fundong

Participants: 185 students tested (139 female, 44 male) 150 in Form 3 (the last year of training) 29 in Form 2 6 in Form 1

Content of Tests: English (including reading comprehension) Basic math

Distribution of Participants by Age

3230282624222018

20

15

10

5

0

age

Pe

rce

nt

Histogram of age

The Test Instruments

The English/reading Assessment

Two texts for reading comprehension “Tenali and the Thieves” Grade 3 level 668 words 13.36 words per sentence 10 comprehension questions “Living an Adventurous Life” Grade 9 level 468 words 19.5 words per sentence 10 comprehension questions

English knowledge and vocabulary Word meanings and associations – 10 items ex. 6. crush a. sweet b. cheap c. change d. smash

Synonyms and antonyms – 10 items ex. 4. Which of the following words is closest in meaning to the word resolve? a. faint b. require c. weird d. determination

Words in context (naturalness) – 10 items ex. 5. The teacher praised the girl because her ideas were so __________. a. unique b. odd c. purple d. foolish

Nature of the Comprehension Questions

General comprehension (vocabulary, textual content, etc.) – 7 items 4 items from Tenali 3 items from Adventure Understanding the plot structure – 2 items 2 items from Tenali Simple factual comprehension – 4 items 1 items from Tenali 3 items from Adventure Drawing inferences and conclusions – 7 items 3 items from Tenali 4 items from Adventure

Nature of the Vocabulary Section

30 vocabulary items (taken from the Cameroon textbook series for teaching English) Level 4 – 10 items ex. oil, respect, good, open Level 6 – 9 items ex. ambition, estimate, acquire, pursuit Level 8 – 6 items ex. cunning, resolve, prior, endorse Levels 10 and 12 – 5 items ex. mutual, crush, unique, append Categories of vocabulary items General use vocabulary – 23 items ex. shelf, trick, chance, dead Academic vocabulary – 7 items ex. structure, resolve, mutual, append

The Math Test

Classification by response method Free response (supply a math fact or do an actual computation) 12 items Multiple choice 30 items Classification by manner of presentation Direct (simple computation or directed solution) 22 items

Story problems (an abstract scenario is created from which information must be gleaned in order to solve a problem)

20 items

The Math Test – Classification by Content

Basic operations – 4 items Math facts (basic and advanced) – 4 items Operations with Fractions – 3 items Basic algebra – 6 items Conversions and equivalences – 2 items Probability and statistics – 2 items Logical and applied analysis – 7 items Symbolic representation – 6 items Problem solving with percentages – 2 items Area and perimeter – 3 items (two of which were discarded) Graphical representation – 1 item Ratios – 2 items

A Look at the Results

Distribution of Overall Performance

70605040302010

25

20

15

10

5

0

OVERALL_PER

Pe

rce

nt

Histogram of OVERALL_PER

Distribution of Performance on the English Assessment

907560453015

20

15

10

5

0

eng_PER

Pe

rce

nt

Histogram of eng_PER

Distribution of Performance in Math

70605040302010

25

20

15

10

5

0

Math_PER

Pe

rce

nt

Histogram of Math_PER

Summary Statistics of Performance

Mean Stan. Deviation

Math 37.77 10.51

English/reading 65.18 13.79

Overall 52.41 9.42

The Results for English and Reading

Suggested Standards for Reading Comprehension

Barr, Blachowitz, Katz, and Kaufman (2002) Proficiency Level Level of Reading Comprehension

Independent 90 – 100 percent Instructional 75 – 79 percent Borderline 50 – 74 percent Frustration below 50 percent

Reading Comprehension – a closer look

Comprehension vs. Textual Difficulty

Vocabulary Knowledge vs. Textual Difficulty

Predictive probabilities .92 for Tenali .37 for Adventure

Vocabulary – a closer look

Relationship between Academic and General Vocabulary among GTTC trainees

100806040200

100

80

60

40

20

0

Vocab_no_ac_PER

Vo

ca

b_

ac_

PER

S 18.2455R-Sq 29.3%R-Sq(adj) 28.8%

Fitted Line PlotVocab_ac_PER = 9.333 + 0.5839 Vocab_no_ac_PER

Mean AWL = 48.0 Mean GWL = 66.2

Best Predictors of Reading Comprehension

For the Level 3 Text Level 4 vocabulary (F = 18.68; p = 0.000; r-sq. = 10.61%) For the Level 9 Text Academic vocabulary (F = 23.33; p = 0.000) Form (years in GTTC) (F = 8.05; p = 0.000; r-sq. = 17.35%)

Vocabulary and Text Comprehension

Comparison to Grade 5 Readers Level 3 Text

Distribution of Reading Comprehension among GTTC trainees relative to the Barr et al. Standards

Barr, Blachowitz, Katz, and Kaufman (2002) When reading a… Proficiency Level Percent Comprehension Level 3 Text Level 9 Text

Independent 90 – 100 percent 77.6% 2.4% Instructional 75 – 79 percent 9.1% 2.4% Borderline 50 – 74 percent 11.5% 61.2% Frustration below 50 percent 1.8% 33.9%

Implications and Questions raised by the Data

Levels of general proficiency in English are obviously low. The level of reading comprehension is quite limited. How effectively can such teachers teach via English? Is there a self-perpetuating negative spiral in place in terms of

learning and teaching English? What is the probability of educational success for non English-

speaking children when their teachers have low proficiency in the language of instruction?

How much improvement in teachers’ proficiency is needed to raise students’ performance?

What steps could be taken to improve teachers’ proficiency in English?

The Math Results

Distribution of Performance in Math

1009080706050403020100

25

20

15

10

5

0

Math_PER

Pe

rce

nt

Histogram of Math_PER

Mean = 37.77 SD = 10.51

Performance by Content Area

No. of items

Mean Score

Standard Dev.

Number who answered zero items correctly in

this category

Basic operations 4 49.4 22.2 3

Math facts 4 46.8 24.6 11

Fractions 3 33.8 26.9 44

Basic algebra 6 25.3 23.1 47

Percentages 2 23.4 28.6 90

Conversions 2 60.4 34.4 24

Probability and statistics 2 32.9 32.3 69

Logical analysis 7 42.0 16.5 3

Symbolic representation 6 30.9 20.2 17

Geometry (area and perimeter) 3* 24.7 43.3 119

Graphical representation 1 9.5 29.4 143

Ratios 2 62.7 35.2 24

GTTC trainees vs. Class 5 students

Class 5 Students GTTC

Skill or knowledge domain Mean SD Mean SD

Basic operations 10.9 17.7 49.4 22.2 The number system 19.9 14.7 52.1 21.6 Fractions 12.6 13.3 22.8 34.1 Algebra -- -- 24.3 20.7 Word Problems Basic operations 19.5 19.8 31.7 28.4 Analysis -- -- 44.8 14.9 Geometric figures 23.8 23.1 22.2 30.1 Graphs and tables 23.3 21.7 22.2 27.4 Percent, interest, commissions -- -- 30.1 34.3

Basic statistics -- -- 32.9 32.3 Math Overall 16.6 7.7 37.8 10.5

Some Inferences for Math Learning and Instruction

Mastery of basic operations is not strong. It is reported that many have taken no math courses

beyond primary (Grade 6) and the data appear to support this claim.

Complex numbers including decimals are troublesome. Mathematical abstractions (models, variables,

symbols) are especially difficult. If teachers lack knowledge and skill, we must assume

that students will as well.

Data from the Maguindanaon MLE Experience in the Philippines:

Comparing L1 vs. L2 speakers of the

Language of Instruction

At the Kindergarten Level

At the Grade 1 Level

At the Grade 2 Level

(Negative) Impact on Instructional Efficiency

A Simple Probabilistic Model of Learning

S = student T = teacher Prob. X Prob. X Prob. = Prob. T knows T can explain S will understand S learns

A Simple Probabilistic Model of Learning

Suggested probabilities Teachers’ performance on primary-level math items = 42.24% Teachers’ performance on English assessment (Grade 3 text plus

vocabulary) = 75.9% Class 5 students’ performance on English assessment (easiest story plus

vocabulary) = 38.6% * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * Prob. X Prob. X Prob. = Prob. T knows T can explain S will understand S learns .4224 X .759 X .386 = .124 (actual mean math score = 17.0%)

Conclusions and Challenges

Teacher mastery of curricular content, while frequently considered suspect, has not been widely measured.

Limitations in linguistic competence are widely recognized in fact, but little understood in terms of instructional impact.

The weakness of educational systems in linguistically diverse developing countries involves teachers as well as students.

How much competence in a language is required to enable quality instruction?

What kinds of language learning/teaching interventions are needed to developed the needed competence?

Thank you.