test methods in language testing

Test Methods

(Five facets affecting

performance on language

tests)

Chapter 5 - Test Methods

FOCUS

Test method facets (features, aspects) are the primary focus of this chapter.

Factors affecting test performance:

Example:

- ‘Live’ versus “recorded” presentation of aural material,

-Personality of examiner,

-Filling in the blanks in isolated sentences as opposed to completing blanks in a text

-The amount and type of text to be processed


*What is known: the effects of test methods on test performance

Example:

*The amount and type of preparation or prior experience with a

given test

*TOEFL (Test of English as a Foreign Language) : may not have any

prior preparation (proficiency test)

*CPE (Certificate of Proficiency in English): taking a course whose

syllabus is based upon the CPE (achievement test)

* In order to more fully understand variation in language test

performance, it is necessary to develop a framework for

describing the specific features (facets) of test method.


*There have been a number of descriptions of the

characteristics of language testing methods over the years

(Carroll, 1968; Ciark, 1972; Cohen, 1980, 1979; Hawkey,

1982; Mackey, 1968; Morrow, 1977; Savard, 1968; Weir,

1983).

*In the current book: the framework of test method facets

is an extension and recasting of Caroll’s, Clark’s and

Weir’s taxonomies, incorporating more recent views of the

nature of language.


Five Facets Affecting Performance on Language Tests

(Bachman, 1990)

1. Testing environment

2. Test rubric

3. The nature of the input the test taker receives

4. The nature of the expected response to that input

5. The relationship between input and response


1. Testing Environment

Test takers might be expected to perform differently under

differing environmental conditions.

Testing environment includes 4 facets (aspects):

1.a. Familiarity of the place and equipment used in

administering the test (familiar place: less threatening –

equipment: pencil versus computer)

1.b. The personnel involved in the test: a superior, a peer, or

a subordinate)

1.c. The time of testing: early in the day, just after a heavy

noon meal

1.d. Physical conditions: noise, temperature, humidity, seating

arrangement, lighting)

2. Test Rubric

Test rubric specifies how test takers are expected to

proceed in taking the test.

Test rubric includes 3 main facets:

2.a. Test organization (i.e. salience of parts - sequence of

parts - relative importance of parts)

2.b. Time allocation

2.c. Instructions (i.e. language – channel - specification of

procedures & tasks - explicitness of criteria for correctness)


2. Test Rubric

2.a. Test organization: Tests consists of different parts such as items, questions, or sub-tests. So,

- The salience of (these) parts : whether parts are explicitly marked and identified or not…

Ex. Test A – A test of how well you can recognize a well organized English paragraph

“Listening Comprehension” “Reading Comprehension” etc.

- The sequence of (these) parts: easy to difficult – power tests; random or same difficulty – speeded tests; standard tests – no opportunity, fixed order of subtests

- Their relative importance: The parts of a test are not always weighted equally

Ex. A part in which nearly everyone receives the same score, and which is highly correlated with the other parts of the test will contribute relatively little to the total test score.

eff

ect

test

perf

orm

ance


2. Test Rubric

2.b. Time allocation: The amount of time allocated for the test or its parts is likely to affect test performance.

- Speeded tests: Speed is a variable – Not all the test takers can complete in the allocated time.

- Power tests: Sufficient time is allocated for almost all.

2.c. Instructions: Performance depends on how well the test conditions are understood. Facets of instruction includes:

- Language: Instructions in test-takers’ native or tested language

- Channel: Aural or visual channel, or both

- Specification of procedures & tasks: procedure - whether circling or checking the alternative; task – type (selection, construction) and form of the response (native/target language)

- Explicitness of criteria for correctness: multiple choice-the criteria for correctness is clear; writing tests - vague

Test

takers

should

be

aw

are

of

this


3. Input: the information contained in a given test

task, to which the test taker is expected to respond.

Input includes 2 main facets:

3.a. Input format

3.b. Nature of language input


3. Input: the information contained in a given test task, to which the test taker is expected to respond. Two facets:

3.a. Input format

- Channel of presentation: aural, visual (e.g. listening comprehension test)

- Mode of presentation: Input will be in receptive mode

- Form of presentation: language (e.g. reading passage), nonlanguage (e.g. pictures), both (e.g. a passage with tables)

- Vehicle of presentation: live vs. canned human input (e.g. tape recording)

- Language of presentation: native, target, both

- Identification of the problem: specific (e.g. correcting errors underlined) – general (e.g. identifiying and correcting an unspecified number of errors)

- Degree of speededness: Input may be perceived as speeded if test taker perceives speed as a factor in performance. The perception of speededness vary from one test-taker to another.


3. Input

3.b. Nature of language input: When the form of the input

is language, that language can be characterized by its

- Length: Not a critical facet, but longer language samples

include more potential effects in terms of…

- Propositional content: Vocabulary (frequency, specialization),

degree of contextualization (embedded/ reduced), distribution

of new information (compact/diffuse), type of information

(concrete/abstract, positive/negative, factual/counterfactual),

topic, genre

- Organizational characteristics: grammar, cohesion,

rhetorical organization

- Pragmatic characteristic: illocutionary force, sociolinguistic

characteristics (dialect or variety)

These

chara

cte

rist

ics

decid

e t

he c

om

pre

hensi

bilit

yof

input.


4. Expected Response

Expected response includes 3 main facets:

4.a. Format of expected response

4.b. Nature of language

4.c. Restrictions on response



4.a. Format of expected response

- Channel: aural, visual (Listening comprehension test: marking

the correct choices on multiple-choice questions / responding

speaking

- Mode: productive

- Type of response: selected (multiple choice tests),

constructed (production of a language sample – interview tests

of communicative proficiency)

- Form of expected response: language, non-language

(making a mark, drawing a picture), both

- Language of response: native, target, both



4.b. Nature of language


- Length: Not a critical facet, but longer language samples include

more potential effects in terms of…

- Propositional content: Vocabulary (frequency, specialization:

technical registers, slangs), degree of contextualization

(embedded/ reduced), distribution of new information

(compact/diffuse), type of information (concrete/abstract,

positive/negative, factual/counter-factual), topic, genre

- Organizational characteristics: grammar, cohesion, rhetorical

organization

- Pragmatic characteristic: illocutionary force, sociolinguistic

characteristics (dialect or variety, register)


4.c. Restrictions on response

- Channel: variety of conditions (face-to-face, in quiet rooms, phone

conversations)

- Format: restricted: selection/identification response types;

unrestricted: composition test

- Organizational characteristics: grammatical forms (multiple

choice vocabulary item: test-taker deals with the meanings of specific

words), organization of discourse (following a specific rhetorical pattern:

comparing & contrasting, arguing for & against)

- Propositional and illocutionary characteristics: Language tests

restrict illocutionary force of responses: Picture description tests: to

describe what is given in a picture

- Time or length of response: In testing, situations, administrative

considerations place additional restrictions on time and length.


5. The Relationship between Input and Response

The relationship between input and response

includes 3 main facets:

5.a. Reciprocal input and response

5.b. Nonreciprocal input and response

5.c. Adaptive input and response



5.a. Reciprocal input and response

can be defined as the use of language by one individual to produce an effect in another individual through the reduction of uncertainty with knowledge of results.

*one individual - another individual: a sender and a receiver need to be involved.

*to produce an effect: indicates that the language use has a communicative goal, or illocutionary intention.

*reduction of uncertainty: characterizes the means by which the communicative goal is effected, that is, by means of a change in information at the disposal of the receiver.

*There is an interaction and a feedback: Ex. a well conducted oral interview



5.b. Nonreciprocal

*Nonreciprocal language use: there is no interaction

between language users.

*There is no feedback: Ex. reading a book.

*Ex. A cloze or dictation test



5.c. Adaptive input and response

*The relationship between input and response is adaptive if the input is influenced by the response, but without the feedback that characterizes a “reciprocal relationship”.

*An individual taking an adaptively administered multiple-choice test, for example, typically begins with an item that is of medium difficulty.

*If she answers this item correctly, the next item presented will be slightly more difficult.

*If she misses that item, the next item presented will be slightly easier…



Applications of this framework to language testing

The applications of this framework to language testing are

potentially in 4 areas:

1. The description of language tests

2. Language test design

3. The validation of language tests

4. The formulation of hypotheses for language testing

research



1. The description of language tests

*What is my purpose?: Test users are frequently faced with the need to compare and evaluate different tests with respect to their possible utility for a given testing need.

*A common framework could be very useful for describing and comparing the types of tasks included in two similar tests or in a test and a language program.

*Ex. Using subjective ratings of experienced language testers, Bachman, Davidson, and Lynch (1988) found considerable differences between the structure sections on one form of the TOEFL and one form of the First Certificate in English (FCE) in several facets of rubric:

*Relative importance of parts - specification of procedures and tasks - explicitness of criteria for correctness.



2. Language test design

*An essential step in the development of language tests is

operationalizing the constructs that we want to measure.

*The framework of test facets (we have talked about) may

provide language testers

- a detailed basis for the description of different tests for

purposes of selection, and for the design of specific

language tests

- an appropriate means of codifying and describing the tests

they are developing, using, or researching.



3. The validation of language tests

*A detailed description of the aspects of the test method is

as essential to the validation process as is that of the

abilities to be measured.

*The validation of language tests must therefore be based

on a detailed description of both the abilities to be

measured and the facets of the test methods.

*That’s the reason why the facets of test methods are

applicable to the validation process.



4. The formulation of hypotheses for language testing research

*As stated in Chapter 1, one of the goals of language testing research is to investigate the factors that affect performance on language tests.

*Considerable research has already demonstrated that test methods affect performance on language tests, and much of this has examined the effects of specific facets, either by themselves, or as they interact with other facets.

*As these facets have a role in performance, test-developer should make use of them effectively while preparing tests.

test methods in language testing

Education

test methods test methods

test rubric test rubric

language test performance

test organization

test methodsthere

test performanceexample

test methodswhat

test methodsfive facets