DEVELOPMENT OF AN ONLINE DEVELOPMENT OF AN ONLINE READING TESTREADING TEST

Associate Professor Dr. Alisa VanijdeeAssociate Professor Dr. Alisa Vanijdee

Dean, School of Liberal ArtsDean, School of Liberal ArtsSukhothai Thammathirat Open University, Thailand Sukhothai Thammathirat Open University, Thailand

Alisa.Van@stou.ac.th;avanijdee@hotmail.com Alisa.Van@stou.ac.th;avanijdee@hotmail.com

OBJECTIVESOBJECTIVES

(1) develop an online English reading test; (2) validate the quality of the test;

(3) study the reading performance-level of the test-takers;

(4) analyse common weaknesses found in the test results, and

(5) propose guidelines to improve the reading performance of test-takers.

INFORMANTSINFORMANTS

31 academics and graduate students

RESEARCH INSTRUMENTSRESEARCH INSTRUMENTS

(1) 3 sets of English reading tests; (2) test evaluation forms, and (3) a test-specifications crosscheck form.

RESULTSRESULTS

(1) 3 sets of online reading tests posted on the website of Sukhothai Thammathirat

Open University;

RESULTSRESULTS

(2) the tests are qualified as specified:

the tests were designed based on the reading process and the reading evaluation

process, with the reading performance level based on CEFR level;

the tests were modified according to the IOC scores of lower than .50 and the

experts and pilot-test takers suggestions together with the trial test scores of the

pilot-test takers

the reliability of the 3 sets was analysed by Kuder Richardson 20 formula, which yielded .86, .88, and .61 respectively; the difficulty and discrimination was

accepted at .20 < = p <= .80 — the rest were modified;

RESULTSRESULTS

(3) the reading performance level of the 3 sets of tests: marks level Number of test takers

Version 1 Version 2 Version 3

76-100 C 1 (effective operational proficiency) 3 3 2

51-75 B 2 (independent user: vantage) 12 14 11

26-50 B 1(independent user: threshold)16 14 18

0-25 Needs improvement 0 0 0

Total number

31 31 31

RESULTSRESULTS

(4) the test-takers weaknesses included insufficient understanding of complex structure and lack of vocabulary;

(5) the guidelines for reading improvement focus on reading strategies practice at the cognitive, metacognitive, and socio-affective level.

DEVELOPMENT PROCESSDEVELOPMENT PROCESS

Theoretical Framework

- Reading process framework

(Khalifa & Weir, 2009: 62)

- designed based on the mikroskills

from Alderson (2000) and

- test usefulness by Bachman & Palmer

(1996)

Test specifications for ReadingTest specifications for Reading

• CEFR• Unaldi’s (2009) comprehensive framework • Text difficulty -vocabulary profile of ‘lexitutor’ -Flesch-Kincaid Grade Level

Test specifications for grammarTest specifications for grammar

• selected from Grammar City & Guilds ESOL

Test specifications for VocabularyTest specifications for Vocabulary

• AWL list

TEST SPECIFICATIONSTEST SPECIFICATIONS

• designed based on the mikroskills from Alderson (2000) and test usefulness by Bachman & Palmer (1996).

• Unaldi (2009) detailed specification

Unaldi’s detailed specificationsUnaldi’s detailed specifications

• CONTEXTUAL• • Response Method: Gap filling • • Text length: up to 500 words

Genre:Genre: everyday materials such as letters and emails(e.g., enquiries, orders, letters of confirmation etc.), public information leaflets brochures, short official documentsstraightforward instructions for equipmentexpository and informative newspaper/magazine articles on familiar subjects personal letters with description of events, feelings and wishessimple informational sources (e.g., junior encyclopaedias, leaflets and brochures)

Rhetorical task: narrative, descriptive, instructive, expositoryPattern of exposition: define, describe, elaborate, illustrate, compare and contrast, classifyExplicitness of text structure: explicit

Structural resourcesWords/sentence: Ave Flesch-Kincaid Grade Level: Ave: 8The complexity of sentence structure:

mostly simple sentences (but some use of subordinate clauses in PET)

Cohesion: explicit

Lexical resources95% K1-20 words = K1 84.7 %

K2 8.7% K3 2.3%AWL words: 2.5%

Nature of information (abstract/concrete): concreteContent knowledge: not required: everyday situations encountered in work, school, leisure etc., (personal feelings, opinions and experiences, hobbies and leisure…familiar topics in expository texts

COGNITIVE Type of reading:- understand the main points and/or relevant points though not necessarily in detail (description of events, feelings and wishes, significant and clearly signalled reasoning, and argumentation)-identify unfamiliar words from the context -extrapolate the meaning of occasional unknown words from the context and deduce sentence meaning

Text level: word, sentence and across sentences

SAMPLES OF TEST SPECIFICATIONS ANALYSISSAMPLES OF TEST SPECIFICATIONS ANALYSIS

See word file

Components of the TestComponents of the Test

Grammar Vocabulary AWL words Reading

Test on CD

• processed through lexitutor software (http://www.lextutor.ca/vp/). The analysis revealed frequencies of vocabulary, types and levels (K1, K2, K3, AWL); types, tokens, families, tokens per family, types per family).

• Flesch-Kincaid Grade Level was also taken into consideration when adapting reading texts to correspond the specifications in B1 B2 C1.

The level of difficulty of theThe level of difficulty of theVocabularyVocabulary

Thank you.