The Impact of Evaluation on Thai ELT

Richard Watson ToddKing Mongkut’s University of Technology Thonburi

Impacts of evaluation

Most countries: most influential evaluation is University Entrance Exams– The Chinese experience in 2007

• Parents giving children Ritalin

• Parents trying to close airport

The Thai experience

Survey of secondary school teachers’ problems

156 respondents Most serious problem = the influence of the

university entrance exam

A brief history of Thai university entrance 1967 - 1997: multiple-choice exams as sole

criterion 1998: 10% of marks from school scores

(GPA and percentile ranking) By 2006, school scores increased to 30%

(GPA only)

A brief history of Thai university entrance 2006: name change to O-NET and A-NET

– Inclusion of essay question– Link to school curriculum– Norm-referencing through T-scores– Marking fiasco

2007: multiple-choice only

Future plans for Thai university entrance 2008: Add 3 more core subjects (health,

technology, arts) 2010: A-NET replaced by aptitude test 2010: GPA system simplified 2010: Scores for ethics

Stated reasons for changes

Change away from exams only– validity problems– student stress– growth of tutorial schools– lack of attention to secondary schooling– move to learner-centredness

Keeping school scores to 30%– reliability problems

Stated reasons for changes National Education Act of 1999

– Section 26– "Educational institutions shall assess learners' performance

through observation of their development; personal conduct; learning behaviour; participation in activities and results of the tests accompanying the teaching-learning process commensurate with the different levels and types of education. Educational institutions shall use a variety of methods for providing opportunities for further education and shall also take into consideration results of the assessment of the learners' performance."

Reasons for changes

Conflict between educational ideals and assessment inertia

Conflict between Ministry of Education and Council of University Rectors– MoE: motivate students at secondary schools– CUR: emphasise reliability

Current situation

Constant flip-flopping Heavy emphasis on multiple-choice exams

– 70% from multiple-choice entrance exams– 30% from school GPA, but 60% of school

scores from exams with heavy multiple-choice bias

Emphasis on multiple-choice exams unlikely to change in future

Impact of university entrance

Impact on teaching and learning– teach vocab and grammar– no higher-level thinking– rote learning– ephemeral memory of facts– receptive skills emphasised

Impact of university entrance

Impact on other exams– School exams use entrance exams as model– The case in 2006

• April 2006: O-NET including essay

• June 2006: new semester

• July 2006: mid-term school exams include essay

• August 2006: CUR announces ‘no more essays’

• September 2006: final school exams have no essay

Impact of university entrance

Impacts on students and society– students focus on grades, not learning– more cheating and selfishness at school– reliance on tutorial schools– inequalities in access to higher education

Validity and reliability

Multiple-choice is practical– but 2006 marking fiasco not caused by essays

Multiple-choice is reliable– only if questions are well-designed

Validity and reliability

Poorly designed questions from secondary schools– ______ sports bag do you like?

• a. What

• b. Who

• c. Which

• d. Whose

Validity and reliability

Poorly designed questions from secondary schools– People swimming here do so his own risk

• a. Everybody swim here

• b. People swim here because he like risk

• c. People who swim here will have accident

• d. People did not swim here

Validity and reliability

Content validity: Test what you teach– O-NET based on school curriculum– Use of school GPA scores– Reliance on multiple-choice means prominent

objectives ignored– Multiple-choice may be more a measure of test-

taking strategies than of language ability

Validity and reliability

Predictive validity– Research into predictive validity

• GPA better predictor than exams

• Tutorial schools influence scores

– University entrance exams are not particularly valid for predicting candidates’ future performance

How good is university entrance?

Detrimental effects on teaching and learning Detrimental effects on school evaluations Detrimental effects on students and society Good practicality Reasonable reliability Poor content validity Weak predictive validity

Directions for change

Previously– inconsistent– directionless– knee-jerk reactions– driven by personal agendas

How to decide on and implement changes

Implementing changes

Gatekeeping is only one among many considerations

Use National Education Act as a guide Emphasise validity over reliability

Change university entrance from a problem to an agent for beneficial development