experimental research

1Second Language Research Methods by Seliger and Shohamy

What is experimental research?What is experimental research?

An attempt by the researcher to maintain control

over all factors that may affect the result of an

experiment.

What is the role of the researcher?What is the role of the researcher?In doing this, the researcher attempts to

determine or predict what may occur.

2

Experimental Research and Research Experimental Research and Research ParametersParameters

3

1- Synthetic/ Analytic/ Holistic Constituent

2- Heuristic A Priori

3- Level of control and manipulation: Low High

4- Level of explicitness of data collection: Low High EXPERIMENTAL RESEARCH DESIGN

According to the research parameters,

experimental research is analytic and deductive.

The major components of experimental The major components of experimental researchresearch

All experimental approaches involve the control

or manipulation of three basic components:

the populationthe treatmentthe measurement of the treatment

4

The Type and Number of GroupsThe Type and Number of Groups

Experimental research is concerned with studying

the effects of specified and controlled treatments

given to subjects usually formed into groups.

Groups can be formed according to the purpose of

the experiment or they can be natural as they

appear prior to the research. (e.g. Spanish

speakers, males, students in 3-A)

5

It is up to researcher to use pre-existing groups

or to set up a new group for the experiment.

To use already existing group or the one which is

formed for the experiment is one of the

distinguishing points between experimental

designs and quasi-experimental designs.

6

The TreatmentThe Treatment

This refers to anything done to groups in order to

measure its effect.

Treatments are the independent variable in research.

The treatment is a controlled and intentioned

experience. (e.g. a language teaching method

constructed for the experiment)

7

Measurement or ObservationMeasurement or ObservationIt is the evaluation or observation of the effects of the

treatment.

In second language research, the effects of the

treatment(s) can be evaluated by

a language test

a judgment or oral test

a communicative task and etc.

8

Campbell and Stanley’s conventions for the Campbell and Stanley’s conventions for the components of the experimental researchcomponents of the experimental research

XX stands for an experimental treatment.an experimental treatment.

e.g. a teaching method

materials used in a language lab

OO stands for observation or measurement of the observation or measurement of the

effectseffects of the treatment.

RR stands for randomizationrandomization, or the random the random

assignment of subjects to groupsassignment of subjects to groups in order to control

for extraneous variables.9

Experimental Research DesignsExperimental Research Designs One-Shot Design

Single Group Design One Group pre-test+ post-test

Time-Sampling Designs

Static Group or pre-experimental design

Control Group Design

Pre-Post test with randomization of groups

Factorial Design

Quasi-Experimental Design

Separate Sample Design10

Single Group DesignsSingle Group Designs

1.1. One-Shot Design: X OOne-Shot Design: X O (Pilot Study Design)

A single treatment is given to a single group or

individual, then they are observed or measured.

It is more appropriate to pilot studies in which

the researcher wishes to try out instruments or

treatments before running a full-scale

experiment.

12

Example of One-Shot Design:

A second language teacher uses a new method

for teaching the communicative use of the target

language. After three weeks of using this method,

a test is administreted to the class. The class

performs well on the test.

X : the new method for communicative use of TL

O : the test applied to the class13


Advantages and Disadvantages of One-Shot Advantages and Disadvantages of One-Shot DesignsDesigns

The disadvantage is that there is no way of

knowing the characteristics of the group or the

individual before the treatment.

The advantage is that it correctly identifies what

to avoid in experimental research. Another one is

that it is used for first testing of instruments of

experimental procedures. 14


2- One Group pre-test + post-test: O1 X O22- One Group pre-test + post-test: O1 X O2

Subjects are observed or measured twice on the

dependent variable. This design eliminates the

need for a control group design as it uses the

subjects as their own controls.

15


Example of One Group pre-test + post-test Design

In a study on the effects of second language

instruction, the researcher wishes to establish that

the group has no knowledge of a grammatical

structure which will be taught in the lesson. In order

to establish this, two tests are constructed which

specifically test for the target structure in a discrete

point test. One test will be used as the pre-test and

the other will become the post-test.

16


17

Single Group DesignsSingle Group Designs Advantages and Disadvantages of Advantages and Disadvantages of

One Group Pre-test & Post-test DesignsOne Group Pre-test & Post-test Designs

The disadvantage is that there is no certainty that

the possible differences that appear in O2 are the

result of treatment. The result may be affected by

attrition or maturation.

The other disadvantage is that the pre-test may

sensitize the subjects to specific aspects of the

treatment and thus confound what is measured by

the post-test.

18

Advantages and Disadvantages of One Group Advantages and Disadvantages of One Group Pre-test &Post-test DesignsPre-test &Post-test Designs

This design is useful as it controls a number of

extraneous variables which can affect the

homogeneity of subjects when more than one

group is involved.

As the same group is used for both pre and post

tests, there is no need to match to another group.


19

Single Group DesignsSingle Group Designs3- Time Sampling Designs: O1, O2, O3, On…X, 3- Time Sampling Designs: O1, O2, O3, On…X,

On+1 On+1

In this design, a number of samples or

observations are taken over a period of time.

That’s why, in other words, this design refers to

“time-series”.

The treatment is controlled, added after a

number of observations or measurements.

This design can be used to avoid the problems of

One Group pre-test+post test.

Example of Time-Sampling Design

A study wishes to investigate the effectiveness of teaching students the use of relative clauses in English. For the purpose of the research, a treatment is conducted consisted of a lesson which gives the formal features of relative clause formation in English along with practice in using relative clauses in speech and writing. The treatment is preceded by several in-class writing assignments spaced over several weeks. For each writing assignment the number of relative clauses in learners’ writing is tallied and categorized. After the treatment, several similar writing assignments are given and the number of relative clauses are counted and categorized.

20

Advantages and Disadvantages of Time-Sampling Advantages and Disadvantages of Time-Sampling DesignDesign

This design let us ascribe any changes in the subjects’

performance to the treatment with greater assurance.

It discounts the effect of history as in this design the

researcher develops a norm for the population.

In this design, there is no control group, therefore, the

results are obtained from one group.

21


To sum up, collecting data over an extended

period before and after the treatment allows us to

establish a normal pattern of performance for the

language or acquisition behavior under

investigation. This allows the researcher to

exclude the possible interaction of incidental

exposure to language material outside the

classroom, or any natural development change

which may take place regardless of instruction.22


In single group designs, the experimental group acts

as it own control; the comparison is between its

performance with treatment and without treatment.

In designs using control group, there are two or

more groups and the comparison is between them.

The multi-group designs are concerned with

measures to standardize the groups being compared

so that maintain external and internal validity.

24

Control Group DesignsControl Group Designs

1- Static Group or “pre-experimental” Design 1- Static Group or “pre-experimental” Design X O1X O1

O1O1

In this design, a treatment is developed for a

group and its performance is compared with

equivalent group, which does not receive any

treatment.

But, there is no clue if the groups are really equal

to each other before the treatment. 25


The difference in performance on the dependent

variable may be due to intrinsic group differences

such as:

the first language background

sex

exposure to second language

time of the day which instruction takes place

the level of motivation of groups

the effects of different teachers on groups 26


There is no need to form groups which is

different from the previous one, to reassign the

teachers again or to rearrange the timetables.

These features can be considered as advantage,

but, if the variables are not controlled, they can

cause invalidity.

27


When using a static design, to avoid the problems

related to invalidity, the researcher can match subjects

into two groups for various characteristics such as

placement test scores, sex, first language and teacher

rankings to make the groups more comparable.

Matching is a way of increasing the comparability of

the groups when subjects cannot be randomly assigned.

Group A: X O1 Group B: -- O1

28


A Variation on the Control Group Matching A Variation on the Control Group Matching DesignsDesigns

ESL cloze (o1) acceptability (02)

Film (X)Film (X) ComparisonComparison

NS cloze (01) acceptability (02)

29


The film is the treatment. ESL refers to experimental subjects while NS refers to control subjects. Following the treatment, the groups take a modified cloze test in which target concepts were left blank. It was followed by an acceptability task in which the same test was presented with the blanks filled in, some with acceptable cognates and others with unacceptable ones. The acceptability performance of ESL learners was compared with that of the native speakers. The cloze test is the pre-test to establish that the ESL learners did have the concepts with which to judge acceptability.Lightbown and Libben (1984)

2- 2- Pre-test/Post-test with Randomization of Pre-test/Post-test with Randomization of GroupsGroups

Group A’ : O1 X O2Group A’ : O1 X O2

Group A R Group A R

Group A’’ : O1 – O2Group A’’ : O1 – O2

Randomization is a procedure with which to

reduce the amount of systematic error that might

result from biases in the assignment of subjects to

groups. 30


It also provides better control of variables that could affect internal validity.

By randomizing, we are claiming that any effects of extraneous variables occur by chance and that chance is equally distributed between both groups.

An advantage of randomization is that it provides the option of not using pre-test as a method of making groups more comparable. When the subjects are assigned to groups randomly, the effect of subject variables is spread throughout the two groups.

31


32

Control Group DesignsControl Group DesignsThe larger the number of subjects in each group, the

more likely it is that the subject variables will negate

each other through the randomization process.

Internal Validity and RandomizationInternal Validity and Randomization

Volunteers are solicited for a study on the effects of

computer-assisted instruction on SL achievement.

From the population of volunteers, subjects are

assigned to computer-assisted group first and to the

control group receiving no computer-assisted

instruction second.

Although assigning of subjects to the treatment

or the control group would seem to be random,

the first volunteers may be different from those

who may have been cajoled into participating in

the study. The order of assignment may be a

reflection of motivation at some level and it may

affect the results of the study.

33


Factorial designs are similar to the true experimental

designs and include all of the elements found in those

designs, such as grouping by randomization, pre and

post-testing and treatments.

The difference is that the effects of several

independent variables may be tested at the same time.

35

Factorial DesignsFactorial Designs

In simple experimental designs, the additional

variables are referred to as extraneous variables.

These variables must be controlled if an effect is to be

attributed to a specific independent variable.

However, in factorial designs, more than one variable

may be treated as an independent variable.

36


Example:

In a study to test for the effect of language

laboratory training on pronunciation, it is also

decided to measure for the effect of such training

on learners of different proficiency levels, as well

as the effect of different kinds of language

laboratory practice.

37


The effect of practice vs. no practice in language lab. can

be a function of both the proficiency level of the learner

and the type of practice administered in the laboratory.

38


Proficiency Level Factor: Proficiency Level Factor: Advanced learners will

focus on their pronunciation performance as they are

aware of their problems. Beginners are not aware of

their pronunciation problems as they are dealing with

more basic issues in language learning.

Language Laboratory Factor: Language Laboratory Factor: The type of practice

in the laboratory is another factor. The practice can

be contextualized -practice in communicative

settings- or it can be decontextualized –practice in

little or no communicative settings.

39


Gb1 O1 Xd O2

R Gb2 O1 Xc O2

Gb3 O1 - O2 No laboratory practice

Ga1 O1 Xd O2

R Ga2 O1 Xc O2 Ga3 O1 - O2 No laboratory

practice

pre-test post-test

Ga: Advanced Proficiency Xd: Decontextualized PracticeGb: Beginning Proficiency Xc: Contextualized Practice

40


Variations on Factorial Designs in Second Variations on Factorial Designs in Second Language ResearchLanguage Research

A variation of the factorial design can be found in

Flynn (1984) in a study of the ability of Japanese

and Spanish speakers to correctly imitate

sentences with right or left branching adverbial

phrases. Her hypothesis was that the primary

branching direction of the first language would

affect the ability of the learner to imitate such

sentences in English.

41


In this study, subjects were not randomized as

would be the case in true experimental designs.

Learners were given a pre-test to establish

proficiency level and were then divided into three

levels: beginning, intermediate and advanced. They

were further grouped according to first language.

There was no experimental treatment, but an

experimental task was administered, in which

learners were asked to imitate a set of stimulus

sentences. 42


In this kind of studies, the treatment took place

before the experiment and consisted of the

subjects’ exposure to the second language. The

purpose of the pre-test in these studies is to

establish levels of ability before the task is

administered. In addition, there is no

randomization in assigning subjects to groups,

because the experimental groups consist of

natural first language groups.

43


Quasi-experimental designs (Campbell and Stanley 1963) are constructed from situations which already exist in the real world, and are probably more representative of the conditions found in educational contexts.

Quasi-experimental research is likely to have external validity as it is conducted under conditions closer to those normally found in educational contexts.

As this designs are less disruptive than others, it is easier to gain access to subject populations and thus easier to conduct such research.

45

Quasi-experimental DesignsQuasi-experimental Designs

For these reasons, quasi-experimental designs are

ideal for teacher-conducted research and for pilot

studies, in which the exploration of a research

idea is the primary goal. But, the greatest problem

is controlling for sources of internal validity.

46

Quasi-experimental DesignsQuasi-experimental Designs

Separate Sample pre-test/post-test DesignSeparate Sample pre-test/post-test Design

This design is useful for those situations in which the researcher has access to only one group of subjects at a time.

Example:

As in the previous example, the aim of the

study is to measure the effects of language

laboratory training on the pronunciation of

advanced level class entering every three weeks.

48

Separate Sample DesignsSeparate Sample Designs

In this design, the same experiment is replicated

each time with a different class, the population

characteristics are assumed to be the same.

Group 1: (Week 1) O1 X O2



49


The Advantages of Using This DesignThe Advantages of Using This Design

It allows for a larger population to be treated,

and overcomes the problem of lack of access to

large enough groups of subjects at any one time.

It allows us to control for changes due to history.

The other way to control for history using this

design, is to conduct inter-group comparisons

between post and pre tests.

50


experimental research

Documents