stages of test development and common test techniques (1)

STAGES OF TEST DEVELOPMENT

&COMMON TEST TECHNIQUES

DANIEL F. CALI. O.

STAGES OF TEST

DEVELOPMENT

STATING THE PROBLEM

SPECIFICATIONS

WRITING AND MODERATING

ITEMS

ITEMS ON NATIVE

SPEAKERS

ON NON-NATIVE

SPEAKERS

ANALYSIS OF RESULTS

CALIBRATION OF SCALESVALIDATION

HANDBOOKS FOR TEST TAKERS

TRAINING STAFF

OUTLINESTAGES OF TEST DEVELOPMENT

• SHOULD BE A TEST DESIGNED BY A SINGLE PERSON OR A TEAM?

TEAM!

• IT IS DIFFICULT TO DESIGN A TEST BECAUSE OF…

…OBJECTIVITY

…CRITICISMS

…NATIVE COMMAND OF THE LANGUAGE

STEPS…

STATING THE PROBLEM

QUESTIONS

KIND?

PURPOSE?

ABILITIES?

HOW DETAILED?

HOW ACCURATE?

WASHBACK?

CONSTRAINTS?

FINAL

PROFICIENCY

PROGRESS

DIAGNOSTICPLACEMENT

RESULTS

EXPERTISE

FACILITIES

TIME

• ONCE THE PROBLEM IS CLEAR, IT IS ALSO IMPORTANT TO:

GATHER INFORMATION ON ALREADY EXISITING TEST DESIGNED FOR SIMILAR SITUATIONS!

STATING THE PROBLEM

WRITING SPECIFICATIONS FOR THE TEST

• A) CONTENT

• B) STRUCTURE, TIMING, MEDIUM/CHANNEL AND TECHNIQUES

• C) CRITERIAL LEVELS OF PERFORMANCE

• D) SCORING PROCEDURES

A) CONTENT

• INCLUDE CLEAR SPECIFICATIONS REGARDING:

• SKILLS (SUB-SKILLS)

• TYPE OF TEXTS

• ADDRESSEES

• LENGTH OF TEXTS

• TOPICS

• (READABILITY)

• STRUCTURAL RANGE

• VOCABULARY RANGE

• DIALECT, ACCENT AND STYLE

• SPEED OF PROCESSING

B) STRUCTURE, TIMING, MEDIUM/CHANNEL AND TECHNIQUES

• TEST STRUCTURE

• NUMBER OF ITEMS

• NUMBER OF PASSAGES

• TIMING

• MEDIUM

• KIND OF TEST TECHNIQUE(S)

C) CRITERIAL LEVELS OF PERFORMANCE

• ACCURACY

• APPROPRIACY

• RANGE

• FLEXIBILITY

• SIZE

D) SCORING PROCEDURES

• WHAT RATING SCALE WILL BE USED?

• HOW MANY PEOPLE WILL RATE EACH PEACE OF WORK?

• WHAT HAPPENS IF TWO OR MORE RATERS DISAGREE ABOUT A PIECE OF WORK?

WRITING AND MODERATING ITEMS

• A) SAMPLING

• B) WRITING ITEMS

• C) MODERATING ITEMS

A) SAMPLING

HOW THE TEXTS ARE GOING TO BE CHOSEN?

B) WRITING ITEMS

• TRY TO LOOK AT THE TEST THROUGH THE EYES OF TEST TAKERS!

• AN ITEM WITHOUT A KEY IS INCOMPLETE!

• “THE BEST WAY TO IDENTIFY ITEMS THAT HAVE TO BE IMPROVED OR ABANDONED IS THROUGH THE PROCESS OF MODERATION”.

C) MODERATING ITEMS• INTERVATION OF TWO COLLEAGUES:

INFORMAL TRIALLING OF ITEMS ON NATIVE SPEAKERS

• 20 OR MORE…

• THEY SHOULD BE SIMILAR TO THE GROUP BEING TESTED IN TERMS OF:

AGE

EDUCATION

GENERAL BACKGROUND

TRIALLING OF THE TEST ON A GROUP OF NON-NATIVE SPEAKERS

SIMILAR TO THOSE FOR WHOM THE TEST IS INTENDED

• PROBLEMS IN ADMINISTRATION AND SCORING CAN BE NOTED.

ANALYSIS OF RESULTS OF THE TRIAL; MAKING OF ANY NECESSARY CHANGES

• STATISTICAL AND QUALITATIVE ANALYSIS.

HOW DIFFICULT ARE THE ITEMS?

DISCOVER MISINTERPRETATIONS (TO BE MODIFIED OR DROPPED)

CALIBRATION OF SCALES

• IT IS IMPORTANT TO COLLECT SAMPLES OF PERFORMANCE AND ASSIGN EACH OF THEM TO A POINT ON THE RELEVANT SCALE.

VALIDATION

LOW STAKES

TEST

HIGH STAKES

TEST

TOEFL

POP QUIZE

S

WRITING HANDBOOKS FOR TEST TAKERS, TEST USERS AND STAFF

RATIONALE DEVELOPMENT DESCRIPTION SAMPLE ITEMS

ADVICES ON PREPARING FOR

TAKING THE TEST

TEST SCORES INTERPRETATIO

NS

TRAINING MATERIALS

TEST ADMINISTRATIO

N

TRAINING STAFF

NEED TO BE TRAINE

D

INTERVIEWERS

RATERS

SCORERSCOMPUTER OPERATORS

PROCTORS

COMMON TEST

TECHNIQUES

OUTLINE

COMMON TEST

TECHNIQUES

GAP FILLING

MULTIPLE CHOICE

TRUE OR

FALSE

SHORT-ANSWERS

DEFINITION

WHAT ARE TEST TECHNIQUES?

• MEANS OF ELICITING BEHAVIOR

RELIABLE AND VALIDBEHAVIOR

BEHAVIOR WHICH CAN BE RELIABLY

SCORED

LESS TIME AND

EFFORT

BENEFITIAL WASHBACK

TECHNIQUES

MULTIPLE CHOICE ITEMS

• STEM

• DISTRACTOR

MULTIPLE CHOICE ITEMS

ADVANTAGES DISADVANTAGES

WASHBACK

LACK OF DISTRACTORS

GUESSING

PERFORMANCE

TESTING OF RECEPTIVE

SKILLS

RELIABLE SCORING

DIFFICULT TO WRITE ITEMS

GREAT DEMAND ON TIME AND EXPERTISE

CHEATING

YES/NO AND TRUE/FALSE ITEMS

MULTIPLE CHOICE (2 ITEMS)

TOO INFORMAL

REASONS?Reliability and validity

SHORT-ANSWER ITEMS

READING TEST

S

LISTENIN

G TEST

S

GAP FILLING ITEMS

• IT IS A VALUABLE TECHNIQUE AS LONG AS THE CONTEXT IS PROVIDED!

• A: WHAT WILL HE DO?

• B: I THINK HE ______ RESIGN.

WILL, MIGHT, COULD MAY, ETC.

BUT:

A: I WONDER WHO THAT IS.B: IT ______ BE THE DOCTOR.A: HOW CAN YOU BE SO CERTAIN?

• Now you know what a teacher should do when preparing tests, what advices would you give?

• (Each one should only mention only 1 advice)

Names Advices

Carolina Think of the applicability to real situations ar real life situations.

Paola Put in your students´ shoes.

Mauricio Consider multiple intelligences.

THANK

YOU!

REFERENCES

• Huges, Arthur. Testing for Language Teachers. New York: Cambridge UP, 2003.