Meeting the Challenge: the Development of TETET-

Test of English for Thai Engineers and Technologists

Asst.Prof. Natjiree Jaturapitakkul, Ph.D.

School of Liberal Arts, KMUTT, Thailand

Outline of Presentation

• Rationale for TETET development

• Details of TETET

• Development of TETET

• Q&A

RATIONALE for TETET Development

Rationale

- A wish to build up English test identity

Rationale

To construct an English test that is

-suitable for KMUTT students

(engineering and technology fields)

-able to assess student’s actual ability in

communication in their fields before

graduation (not TOEIC)

- particularly for THAI engineers and

technologists

Details of TETET

What is TETET?

A computer-based standardized test designed to

measure engineers and technologists’ English skills

needed in workplace environments in Thailand

Who should take TETET?

• TETET is ideal for:

– University students in

engineering and technology

faculties

– People working or wanting

to work in engineering and

technology positions

What does TETET involve?

• A computer-based test covering the 4 language skills

• Lasting 2.5 – 3 hours

• Engineering- oriented content but not biased towards any

particular discipline

• Covering the main work requirements of Thai engineers and

technologists

What is the content of TETET?

• Reading

– Survival reading, Internet, technical manuals,

e-mail

• Writing

– Reports, memorandum, e-mail

• Listening

– Meetings, conversations, telephones

• Speaking

– Questions and answers, longer business communication

Reading in TETET

– Survival reading

• To measure basic reading abilities

– Internet

• To measure reading comprehension and evaluation

– Technical manuals

• To measure reading of instructions

– E-mail

• To measure reading of communications

INSERT SNAPSHOTS

Survival Reading

Reading from Internet

Reading Technical Manual

E-mail Reading

Writing in TETET

• Memorandum

- To measure ability to use common phrases

• Reports

– To measure grammatical accuracy

• E-mail

– To measure ability to write extended texts

INSERT SNAPSHOTS

Writing Memorandum

Writing Reports

Writing E-mail

Listening in TETET

• Meetings

– To measure ability to identify both gist and specific meanings with

multiple speakers

• Conversations

– To measure the ability to draw inferences from conversations

• Telephones (AVRS)

– To measure the ability to understand specific instructions

INSERT SNAPSHOTS

Listening in Meetings

Listening to Informal Conversations

Listening to Telephone (AVRS)

Speaking in TETET

• Questions and answers

– To measure the ability to engage in interactional speaking

• Longer business communication

– To measure the ability to produce comprehensible longer

discourse

Speaking Interface

Score reporting on TETET

• Scores are determined by the number of correct

answers.

• TETET candidates receive 5 grades:

– A grade for overall proficiency

– A grade for proficiency in each skill

• Grades range from 0 (poor) to 7 (excellent)

Example of TETET Score Report

Listening 3 Has an adequate understanding of predictable listening situations such as general instructions, but comprehension is severely reduced in more complex situations.

Reading 3 Has an adequate understanding of simple texts for immediate survival needs, such as finding information in telephone directories, and is able to find specific facts from texts.

Writing 4 Copes well with familiar written situations by producing both short and long texts, and performs adequately in less predictable contexts such as writing a report where there may be some fluency and accuracy problems.

Speaking 2 Copes adequately with familiar speech situations, but speech may be slow, full of pauses and difficult to comprehend at times.

Overall Band 3 Has partial command of the language in simple workplace situations, but Score likely to make many mistakes, and has great problems in dealing with

complicated or unfamiliar situations.

Sample Certificate

How does TETET compare to

TOEIC and TOEFL?

• TETET bandscale

– 0

– 1

– 2

– 3

– 4

– 5

– 6

– 7

• TOEFL score

– 0-200

– 340

– 400

– 450

– 500

– 550

– 600

– 640-677

• TOEIC score

– 0-140

– 145-250

– 255-360

– 365-470

– 475-580

– 585-690

– 695-875

– 880-990

DETAILS of TETET Development

8 Steps in developing TETET

Defining the TLU domain & Outline of test

Constructing the table of TETET specifications

Developing the draft of the TETET

Going through a priori validation process by experts

Revising the test and trying out the test in

a pilot study (paper-based)

Doing item analysis and item revise

Obtaining a final version of TETET (CBT) + Band scores description

Conducting concurrent validation processes

Figure1: A summary flowchart of TETET development

Step 1: Defining the TLU Domain and Outline of Test

4 methods employed in defining TLU and test construct

Referring to purposes of the study

Studying theoretical background i.e. Strategies used in

different skills, Bloom’s Taxonomy, ESP theory

Studying previous research studies/ existing standardized

tests i.e. TOEIC, TOEFL, IELTS,

Doing need analysis by interviewing the stakeholders involved i.e. senior university administrators, human

resource managers, engineers, engineering (both under & graduate) students

(Interview questions concern experts’ expectations, test components, test content, test format, test taking time, test length and recommendations).

Outline of Test

See MS word file – TETET Outline of Test

Step 2: Constructing the table of TETET specification

Step 3: Developing the draft of TETET (paper-based)

Concerns

1. Topic and text do not bias towards any disciplines

2. Level of text specificity (mild-semi level of ESP)

3. Copyright concern

4. Authentic texts/ materials

5. Text length

Step 4: Going through a priori validation process by experts

3 lecturers and 2 engineering experts validated content by

using an IOC Index form.

H, M and L were used to identify the degree to which the

item (question) measured the ability indicated in the objectives.

H = High degree of congruence with the objective

M = Moderate degree of congruence with the objective

L = Low degree of congruence with the objective

* Items which obtained the high degree of congruence with the objectives less than 75% were revised or discarded.

Step 5: Revising the test and trying out (a pilot study with

paper-based)

Data and comments from all experts were

used to modify the test and used for the pilot

study.

43 engineering undergraduate students at

KMUTT participated in the pilot study

(2 versions of TETET).

Step 6: Doing item analysis and item review

Some items were revised and discarded

according to the following criteria:

Item Difficulty index (IF) should be between 0.2-

0.8

Item Discrimination index (ID) should be higher

than 0.2.

Step 7: Obtaining a final version of TETET (CBT) + creating band scores description

See MS word – TETET Full descriptors

Step 8: Conducting concurrent validation process

Conducting a main study with 42 students who also took

TOEIC test.

TETET has a very strong correlation with TOEIC (r = 0.888).

TETET provides coverage of all aspects of English used by engineers and technologists.

TETET also has very high internal consistency (KR-21

scores of 0.915, higher than TOEIC).

There are strong correlations between the different

sections of TETET and between different versions of the test.

Correlations between different sections and versions of TETET

Reading Listening Writing Speaking

Total TETET

0.788 0.931 0.928 0.850

Speaking 0.474 0.753 0.719

Writing 0.677 0.849

Listening 0.643

Useful websites

More details about KMUTT Testing Unit

http://arts.kmutt.ac.th/crs/testing.htm

Preview of TETET

http://sola.kmutt.ac.th/tetet/index.php?q=vdo

Thank you for your kind attention

natjiree.jat@kmutt.ac.th