research methods in english language learning and teaching (educ 2101) 1 st semester (2015-2016)...

Research Methods

in English Language Learning and

Teaching (EDUC 2101)

1st Semester (2015-2016)

Instructor: Dr. Sadek Firwana

What is research?

Research is a systematic process of inquiry consisting of three elements or components: (1) Problem, question or hypothesis (2) Data(3) Analysis and interpretation.

Why should teachers do research?

Benefits of doing research for teachers

For teachers, a primary reason for doing research is to

become more effective teachers. Research

contributes to more effective teaching, not by

offering definitive answers to pedagogical questions,

but rather by providing new insights into the

teaching and learning` process.

Reasons for not doing research

In spite of these benefits, there are a variety of reasons why teachers may be hesitant or unable to undertake research.

Various meanings and purposes of research

1. Exploring the Ideas In order to address some of the obstacles you

face in doing research, begin by identifying a specific area of research you are interested in researching. Describe in as much detail as possible what you would like to investigate and in what context you would undertake this study. Then list the personal and professional obstacles you face in undertaking this research.

Research: What?• Strictly speaking anything in language teaching.• General areas:

– Language policy– Syllabus, curriculum– Materials, textbooks, etc. – Technology– Methodology– Specific skills– Teaching of grammar, vocabulary, etc.– Language for specific purposes– Culture– Motivation, affective factors– Etc.

1. Problem/Question2. Formulate a Hypothesis3. Experiment4. Collect and Analyze Results5. Conclusion6. Communicate the Results

1.1. Problem/QuestionProblem/Question:

Develop a question or problem that can be solved through investigation.

2. Formulate a HypothesisFormulate a Hypothesis: Predict a possible answer to the problem

or question.

Example: If students are exposed more to English, then their fluency will increase.

3. Experiment3. Experiment:

• Develop and follow a procedure.The outcome must be measurable

(quantifiable).

4. Collect and Analyze Results4. Collect and Analyze Results:

• Modify the procedure if needed.• Confirm the results by retesting.• Include tables, graphs, and

photographs.

5. ConclusionConclusion:

• Include a statement that accepts or rejects the hypothesis.

• Make recommendations for further study and possible improvements to the procedure.

6. Communicate the ResultsCommunicate the Results: • Be prepared to present the project to

an audience.

What are the major types of research and how do they differ?

What are the major types of research and how do they differ?

Basic vs. applied research

Basic research Applied research Its purpose is to acquire

knowledge for the sake of knowledge.

It deals with human and societal problems in the hopes of finding solutions to real-world problems.

Its main goal is to contribute to a fuller understanding of the world.

It is more limited in its questions and conclusions.

What are the major types of research and how do they differ?

Basic vs. applied research

Basic research Applied research It is often undertaken in

disciplines like biology, physics, astronomy, and geology.

It does not attempt to define a theory of language learning that accounts for all language learners, rather it sets forth findings that apply to a particular time, place, and context.

It is used by researchers who want to verify theories of their discipline.

Types of data used by basic and applied research

Primary vs. Secondary Data

Primary Data Secondary Data In using primary data,

researchers gather original data to answer a particular research question. When researchers gather first hand data, “the outcome is knowledge nobody had before”

In using secondary data, researchers examine what others have discovered about a particular topic. For example, if teachers want to know about the advantages and disadvantages of using peer review in a writing class, they can investigate what others have written on the topic.

Literature reviews are an example of secondary data.

Fundamentals in research

• Validity =>

• Reliability =>

The "truthfulness" of a measure; a valid measure is one that measures what it claims to measure.

A measurement is reliable when it is consistent; when it gives the same results when repeated

Research should be able to meet the tests of:

Exercises on Research Methods

1. To be considered true research, it must:

Exercises on Research Methods

2. A primary function of the methodology is to:

Exercises on Research Methods

3. A well-designed and well-conducted study:

Exercises on Research Methods

4. Reading the professional literature in the field of interest _________:

Exercises on Research Methods

5. A ________ is a specific mechanism or strategy the researcher uses to collect, manipulate, or interpret data.

Exercises on Research Methods

5. A ________ is a specific mechanism or strategy the researcher uses to collect, manipulate, or interpret data.

• research tool

Exercises on Research Methods

6. Research methodology refers to __________:

Exercises on Research Methods

7. Research problems should ________:

Exercises on Research Methods

8. The researcher analyzed data and found that explicit teaching of reading strategies improves students' performance on standardized tests. In this study, explicit teaching of reading strategies is __________ .

Exercises on Research Methods

9. The researcher's study suggested that elementary students who are exposed to English for 2 hours a day are more likely to develop their speaking skill than students who are exposed less to English. In this study student‘s speaking skill is ________ .

Exercises on Research Methods

10. A good research plan should include _______:

Exercises on Research Methods

11. A teacher researcher has been facing some problem teaching one specific class English. In his attempt to solve this problem he has been observing and gathering data that may help him overcome the problem. For him these data constitute __________ .

Exercises on Research Methods12. A teacher researcher wrote an article about a research project concerning

discipline problems in English classes and ways of overcoming them. This article was widely read by other teachers who have been facing similar problems. For those teachers, the teacher researcher's article constitutes _____:

Exercises on Research Methods

13. Disadvantages of questionnaires include ____________:

Exercises on Research Methods14. In descriptive research, the researcher typically studies a______

and then draws conclusions about the______.

Exercises on Research Methods15. Experimental research differs from descriptive research in that it:

Exercises on Research Methods16. In experimental design, the variable that is manipulated is the

_____________:

Designs in educational experimentation

There are several different kinds of experimentaldesign, for example:• the controlled experiment in laboratory conditions (the ‘true’ experiment): two

or more groups

Designs in educational experimentation

• the field or quasi-experiment in the natural setting rather than the laboratory, but where variables are isolated, controlled

and manipulated.• the natural experiment in which it is not possible to isolate and control variables.

Checklist for conducting research

Questions to guide the design of a research project: Checklist: 1. Is the question worth investigating? Is the question feasible? 2. Does the research question imply a strong causal relationship between two or more variables?

Checklist for conducting research

Introduction: • What is the purpose of the study? • What are the research questions and hypotheses posed by the

study?

Checklist for conducting research

Design: • Does the question suggest an experimental or a non-

experimental design?

Checklist for conducting research

Method: a. What methods are available for investigating the question? b. Which of these are feasible given available resources and

expertise?

Checklist for conducting research

Method:

c. Is it possible to utilize more than one data collection method?

Checklist for conducting research

Participants:

a. Is the description of the participants adequate? b. Is the method of selection clear?

Checklist for conducting research

Materials: a. Is there a description of the tests, questionnaires, rating scales etc? c. Is there a description of any equipment?

Checklist for conducting research

Results: a. Do they answer the study questions? b. What are the findings of similar studies? c. Are there any contradictory findings? d. How can these be accounted for? e. What additional questions and suggestions for further research are thrown up by the

research?

Research methods

• Quantitative (why? Is there a relation between...?) - use numbers and statistics to see if something you say about the object is true or false (experiments, quasi-experiments, surveys)

• Qualitative (how? What kind of?) - analyze words and images to see what the object is like (interviews, focus groups, observations, case studies)

• Mixed - combine both approaches to get the full picture

Research methods in use(examples)

• Quantitative – how many students who believed in their success got an A for a math test?

• Qualitative – what do students think is important for doing well in math? What are the students’ beliefs?

• Mixed – do students’ beliefs influence their grades? What are the students’ beliefs about their English performance?

Sampling

Sampling is the way in which you choose participants for your experiments, interviews, surveys etc.

Probability Convenience

you carefully search for you invite those representatives of different participants who are groups of population available(time-consuming and costly) (the findings can be biased)

Your research question should be

•Up-to-date•Interesting•Manageable

How to choose an interesting question

Research questionsQuantitative research method1) Is there a (positive/negative) relationship between X and Y?2) Does X influence Y? 3) What is the prevalence of X among the population?4) To what extent does X influence Y?5) Does X affect Y?6) Does X have effects on Y?7) Does X have impact on Y?8) Does X cause Y?9) Is X the effect of Y?10)Does X determine Y?

Research questions (continued)

Qualitative research method1) What is the perception of X in Y population?2) How does X population describe Y?3) What are the main characteristics of X, as characterized by Y?4) In what ways is X influenced by Y?5) What are the most popular means of X?6) How do the participants perceive their roles in X?7) What are the main differences between X and Y?8) What kind of difficulties might influence X?9) Are the participants satisfied with X?10)What are the perceived effects of X on Y?

Major Steps in Conducting Research

52

Conclusion / Report Writing

Problem formulation

Research design development

Data analysis

Selection of data sources

Datacollection

Data collection:

• Observation• Asking primarily open-

ended questions• Scalar items seldom used• Unobtrusive data collection• Data: words, pictures,

behavior• Greater ethical issues

• Asking primarily closed-ended questions

• Scalar items primarily used• Statistical databases• Data: numbers• Fewer ethical issues

Qualitative Research Quantitative Research

Focus Groups

Qualitative Research Designs

ObservationObservation

Qualitative Data

Qualitative Data

Case Studies

Action Research Grounded Theory

In-depthInterviews

Different Approaches for Data Collection/Data Analysis

• Experiment• Survey • Quasi-Experiment • Secondary Data Analysis• Content Analysis• Methods (Archival

Analysis )

• Participant Observation • Individual Qualitative

Intervie• Focus Group Discussion

Qualitative Research Methods

• Examples: • Participant Observation • Ethnography• Individual Qualitative

Interview• Textual Analysis • Focus Group Discussion

Key features of Qualitative Research

1. Collection primarily of qualitative rather than quantitative data Qualitative methods emphasize observations about natural behavior and

artifacts that capture social life as it is experienced by the participants rather than the numerical representations of the categories predetermined by the researcher.

2. Exploratory research question.

Qualitative researchers typically begin their projects seeking to discover what people think and how they act, and why, in some social setting.

3. Inductive reasoning (Reasoning that moves from more specific kinds

of statement to more general ones) Only after immersing themselves to many observations, do qualitative

researchers try to develop general principles to account their observations.

Ways to collect qualitative data

1. Participant Observation 2. Ethnography3. Individual Interview

a) Semi-structured interviewb) Unstructured interview

4. Textual Analysis 5. Focus Group Discussion

Individual Interviews

Semi-structured interview• Interviews with an interview guide containing

primarily open-ended questions that can be modified for each interview.

Unstructured interview • The interviewer starts with only a general

sense of the topics to be discussed and creates questions as the interaction proceeds.

Interview Guide

• A list of topics to cover and the order in which to cover them that can be used to guide less structured interviews.

• The questions must be judged its appropriateness for each interview, and may be re-ordered, and re-worded if necessary.

Types of Questions

Open-ended questions vs. Close-ended question Open-ended • “How would you describe your performance in English classes”? Closed-ended • “Do you think the new your performance in English classes good?”

Probes Questions that allow the interviewee to further expand on

their earlier comments. • “Can you elaborate on what you mean by . . .” • “Why is that so?”

Focus Groups

• A type of group interview.• Focuses on group interaction on a topic selected

by the researcher. • Ideally 4- 12 participants. • The interaction is directed by a moderator who

asks questions and keeps the discussion on the topic.

Different Purposes for Research • Pure Research : Discovery of knowledge for the sake of

knowledge. • Applied Research: Conducted with some practical

purposes in mind. • Exploratory Research: Conducted in order to

explore the phenomena. • Explanatory Research : Conducted in order to

explain the phenomena.

• Evaluation Research: Conducted in order to develop and “evaluate” programs.

Evaluation research

• Research specifically designed to assess the impact of a specific program, policy or legal change.

• Often the focus of an evaluation is whether the program, policy, or law has succeeded in effecting intentional or planned change.

Collecting Data From People

• Opinions, perceptions, stories– Use interviews, focus groups, surveys

• Only people can tell you what they think or believe– But what people say they do is not necessarily

what they actually do

Collecting Data From People

• Risk of bias: social desirability response

– Say what they think researcher wants to hear or what they believe is socially appropriate

– Especially true if asking about socially stigmatized or illegal behaviors

• Remember: hypothetical questions are not necessarily a reliable predictor of actual behavior

General Guidelines

• When gathering data from people:

– Keep it simple, clear, easy, short

– Respect respondents time and intelligence

– Tell them how they were selected and why their participation is important

Option: Closed-ended Questions

• To what extent, if at all, has this workshop been useful in helping you to learn how to enhance your students’ speaking skills?

Little or no extent

Some extent

Moderate extent

Great extent

Very great extent

No opinion

Advantages of Closed-ended Questions

• Best when very specific data is required

• Results are reliable: everyone was asked the same question and answers

• Easy to analyze

Option: Semi-structured Question

• In what ways, if at any, has this workshop been useful to you?

– They then answer in their own words

• Sometimes it is set by first asking a Yes or No question and then followed up with a probe: “Can you tell me more” kind of probe.– Did you find the workshop useful?

• If yes, “what was most useful?”• If no: “what would have made it useful to you?”

Advantages of Open-Ended Questions:

• Capture participants’ views in their own words

• Enable clarification of questions if participant does not understand

• Used in focus groups

• Often useful in the early phases of a research project when researchers are not sure what it is they want to know and/or how to ask the questions

Face-to-face Options

• Interviews– Dynamic process of gathering information– Personal interaction between people– Can be structured, semi-structured or unstructured

• Focus groups– Dynamic process of gathering information– Small group interaction (6-12 people)– Semi-structured

In-Person Interviews

• Interviews are most useful when:

– Other methods, such as questionnaires, do not make sense

– When literacy is an issue

– When a personal interaction is desired

– When questions are not too personal

74

In-Person Interviews

• Interviews are most useful when:

– the survey is too complicated to ensure it can be completed correctly as a self-administered questionnaire

– At the beginning of a project when the issues and questions are still being determined

In-person Interviews

• Can last one hour or more– Allow in-depth exploration of a topic– Can cover more complex issues than a survey– Interviewers will need to adjust the questions to fit

the time available• Some questions will have to be dropped• It is possible to ask for additional time

76

Interviewer Skills

• Clarity of purpose

• Good communication skills

• The ability to establish rapport

• Accept whatever they say with empathy and without judgment

Interviewer Skills

• Be aware of cultural norms: eye contact, direct questions, gender issues

• Be aware of body language or verbal cues:

– You might think you are signaling that you are listening but interviewee might see them as clues to the “correct answer”

Developing the Interview Guide

• Define purpose

– Link to your research objectives

– Do not ask irrelevant questions

– Time constraints force choices

• Essential versus nice to know

• Decide degree of structure required

– Interviews can be highly structured, semi-structured or a combination

Developing the Interview Guide

• Draft interview questions– Sequence questions so they flow smoothly from

question to question– Start with easy questions, move to the heart of the

interview and end with demographic questions– Balance: if you ask about what they think are the

major supports, follow with what they think are the major barriers

Developing the Interview Guide

• Avoid asking “why” questions, if that might be seen as aggressive or critical

• Pre-test the interview guide– Conduct interview in as close to real setting

– But ask tester for feedback at the end of the pre-test

• What worked, what did not, suggestions?

Conducting Interviews

• Let interviewees know:– Why they are being interviewed

– How they were selected

– How the data will be used

– Whether it is confidential

– How long the interview will take

– Whether you might want to talk to them again

Conducting Interviews

• Try to pick a time and place that is quiet and free of distractions.

• Ideally, have a second person help take notes.

• Tape recording: get permission.

– Might want to ask when setting the appointment.

– Test equipment before the interview

– Have a plan B in case of technology glitch

Conducting Interviews

• Stick to your interview guide

– If asking close-ended questions, ask exactly the way written

– Have a prepared answer for likely questions and requests for clarifications

• Pre-tests can help identify likely questions

Conducting Interviews

• If asking open-ended questions, go with the flow of the interview guide, not too directive– Sometimes people do not want to answer a

question; respect that and move to next question– Sometimes people may answer questions before

you ask– If you think you’ve gotten the answer, you can

confirm or skip the question

Conducting Interviews

• Take good notes without distracting from the conversation.– Write while maintaining eye contact.– Write key words or phrases, not verbatim.– But if someone is saying something you want to

capture, it is OK to ask them to repeat it or to finish what you are writing before asking the next question—especially if you are not taping the interview.

Conducting Interviews

• Share interview questions ahead of time.– No surprises

• Offer to share a summary of what you understand from the interview.– This might be especially useful to give the

interviewee a greater feeling of control.– It might also provide more information

• Thank you note afterwards.

Writing Up the Interview

• Every word is valuable.• Take time to write up your notes as carefully

and in-depth as possible.• It is best to do at least a brief clean-up of notes

immediately afterwards (leave an hour between interviews).

• Write up full notes within a day of the interview: memory decay sets in quickly.

Exercise

• Develop a short interview guide to find out about the people in the class: who they are and why they are taking this class?

• Individually, write up 5 open-ended questions

• Form pairs and interview each other

• Write up your notes and turn them at the next class

• Discussion: what questions were the same? What questions were different?

Focus Groups

• A small group of people are brought together to discuss specific topics of interest to the researchers.

• The group process tends to elicit more information than individual interviews because there is cross-conversation and discussion.

– Different views can be explored.

Elements of Focus Groups

• Small groups (6-12 people)

• Comfortable environment

• Good facilitation essential

• Facilitator and note-taker

• Tape record if possible

• Ask few open-ended questions

• How many? Until no new themes emerge

Selection of Participants

• Overall, they should reflect diverse groups and views.• Unbiased, choices should make sense• But individual focus groups may have to be

homogeneous.

How Many Focus Groups?

• No rules

• Generally, do until no new issues emerge or common themes are consistent (usually occurs between 3-6 focus groups).

• However, researchers may want to include more groups to ensure that all groups or areas were included.

Process:

• Introduction:– Purpose of focus groups– Why they were selected– How the information will be used– The ground rules– Overview of the process– If tape recording, explain why

Ground Rules

• “What is said here, stays here.”

• Everyone is encouraged to participate but not everyone has to answer every question.

• Respect different viewpoints.

• There are no right or wrong answers.

• Only one person speaks at a time.

Example of Focus Group Questions

What did you learn from the English course that is most helpful to you outside the classroom?

What were the greatest challenges in studying this English course?

What should the English language program teach so that graduates are prepared well to be good users of English?

Facilitator/moderator

• Know the script so focus group appears conversational.

– Do not read the script.

• Make sure everyone is heard.

– Ask: “What do other people think?”

– Ask: “We have heard from a few people, do others have the same views or different views?”

Skilled Facilitation

• Active listening.

• Accepting all views while managing differences of opinion.

– “So we have different perspectives.”

• Probing for elaboration.

– “Tell me more.”

Skilled Facilitation

• Managing time.

– Closing off discussion and moving to next topic.

• Invisible: say as little as possible

– Let conversation flow across the table with minimal direction.

• Keep your personal views outside the room.– Leave ego at the door.

Process:

• Have participants introduce themselves.

• First question: easy, ice-breaker.

• Ask main questions: the heart of the resesarch.

• Last questions:– Summary question: “What is the most important thing that

was said here that we should take with us?”

– Some facilitators put answers on flip charts

– Ask: “Any last comments?”

Focus Groups: Initial Write-ups

• Do not over schedule: 2 focus groups per day is plenty for one moderator

• Build in time to do the initial write-up• Write up your impressions immediately after focus

group: major issues and major points of discussion.• You may also want to capture anything unusual that

happened during the focus group.• Compare with your partner.

Working With Audio-Tapes

• Ideally, the focus group tape will be transcribed verbatim.

– These transcripts are the datasets

• However, if verbatim transcription is not possible, listen to the tape afterwards while writing in-depth notes.

– You will be surprised how much you did not hear during the actual focus group.

Exercise: Focus Group

• You have been asked to design a focus group to learn what your students think are important characteristics that they should learn and what is important to them in terms of the enhancement of their language learning.

• Working in small groups, develop 5 questions that would be appropriate for a focus group.

• Share with class.

Focus Groups

• Advantages:– Relatively quick

– May be less expensive (staff time) than in-depth interviews

– Provides flexibility to make changes in process

– Can examine different perspectives

– Fun!!

How: Methods (Tools?)Means of gathering data with certain procedures:• Observation (sound, video, note-taking, material handed out)• Survey/Feedback:

– questionnaire, – interview (formal or informal)

• Conversation analysis• Tests• Verbal reports• Learner diaries• Ethnographic methods e.g. observation (multiple sources of data) and

participation by researcher• Case study• Linguistic survey• Corpus-based research• Text-based research• Etc.

Research: Who?• Researchers:

– Teachers– Learners (why not?)

• Being researched:– Learners (individuals, grades, groups: age, proficiency,

etc.)– Teachers– All teaching material– Teaching situation– Texts---> combinations possible or necessary

Research: Steps of research• Determine general research area/topic• Do survey of literature, i.e. research the topic• Formulate research question or hypothesis• Determine aim of research• Describe theoretical basis• Determine research design:

– Methodology– Tools

• Collect data• Describe process and results• Discussion and Conclusion

Pilot study or prototype

• Useful before a large research project to test feasibility of hypothesis, planned methods, use of technology, etc.

• It is much smaller study than eventual project.

Tip on collecting sources

• Any photocopy you make or any good statement you write down, ALWAYS write the source down, i.e. where you found it.

• That means:1. author,2. date of publication,3. title of book/journal article/website/…, place of

publication and publisher (for book) OR title of journal, edition and number, page(s) (for journal) OR URL, date viewed (for website)

In closing …

As researcher be aware of:• your own assumptions on language learning and

teaching• the exact aim of research• boundaries of research (or put differently: what

you are NOT researching)• choosing appropriate methods for the research• the fact that research of language teaching and

learning is about humans! “Protect” them and let them “benefit”. (McKay 2006, viii)

Research Design Descriptive Design

A descriptive research design describes some phenomenon at a specific point in time. A descriptive study needs a clearly defined phenomenon of interest (e.g., teachers' knowledge of literacy development) that is systematically and precisely measured.

Research DesignCausal Comparative Design

Causal comparative designs examine the effect of an independent variable that cannot be manipulated by the researcher (i.e., gender, socioeconomic status, age, etc.) on a dependent variable.