Practice of Quantitative ResearchDr Eric Jensen (e.jensen@warwick.ac.uk)

1

Research Design: Combining Methods?

Lecture Overview

• Research Design Process and Choices

• Qualitative, quantitative, then integrated or ‘mixed’ methods research paradigms and designs.

Definition: Research Design

Process of choosing a way to answer your research question, which requires knowing both what your options are and how to evaluate their relative strengths and weaknesses.

Research Design = Choices• Research design all about making choices.

• To make a good choice, you need to know (1) what your options are and (2) how to evaluate those options.

• Sometimes, your best choice will be a single research method

• Other times, an integrated combination of methods will best serve your purposes.

• It is helpful to think of methods as tools that offer a set of strengths that can be used to accomplish range of goals.

Research Design: Getting Started

Qualitative Research Design

• Qualitative Research typically starts with observations – i.e. it is INDUCTIVE.

• These observations are then used to create theory or generate hypotheses.

• This process leads to research goals such as discovery and exploration.

Qualitative Research Design• Inductive research purposes aimed at

theory-generation and discovery support an “emergent” approach to research design.

• Can lead to shifts in data collection and analysis strategies.

• This approach often calls for a flexible merger of data collection and analysis.

Interpretivism • The social world cannot be described without

investigating how people use language and symbols to construct social practices and understand their experiences.

• No social explanation is complete unless it can adequately describe the role of meanings in human actions

• Human actions not governed by discrete, objective patterns of cause/effect (as in positivism), but by social actors situated interpretations and meanings.

• Quantitative MethodsUsed to answer any counting related question: How many? What proportion?

Quantitative Design

TheoryDeduction

HypothesisOperationalization(research design)

Data CollectionData-organization

Data Analysis Interpretation

ResultsInduction

From Corbetta (2003: 59)

PHASES

PROCESSES

Hypothetico-Deductive Model

From Theory to Hypotheses• Theory involves wide-ranging statements about the

world. These are located at a high level of abstraction and generalization.

• Quantitative analysis is usually involved in empirically testing particular hypotheses that are derived from theory.

• Hypotheses are general statements at a lower level of abstraction. They involve particular relationships (and directionality) between two (or more) concepts.

• Sometimes different theories will give rise to competing hypotheses. These can be empirically arbitrated.

Deductive Approach: Underlying assumptions

• Fundamental premise for mainline quantitative social science: there are truths that exist (independently of human opinions about them) to be discovered through observation / measurement

• Research approaches in this paradigm require evidence gathered through observation and standardised, transparent measurement systems that could be replicated by others.

• Emphasis on objectivity (although researchers’ interests, opinions and theoretical commitments may influence interpretations of results!)

Model building and testing

• To establish generalizable knowledge about a social phenomenon, deductive reasoning is used.

• This can involve deriving ‘hypotheses’ based on theory or prior research, etc. (or initial inductive observations)

• Once we have a hypothesis, we can develop models of what we would expect to find from particular dependent variables if the hypothesis is correct.

• We then collect ‘observed data’ and to test the ‘fit’ with the model. Inferring significance from this comparison of observed to hypothesised values depends on quality of ‘fit’.

Different Paradigms – So why combine methods?

QUALITATIVE RESEARCH

•Typical Strengths: Emergent Research Design, Exploration, In-depth Understanding.

•Typical Weaknesses: Lack of Breadth, Time-Consuming, Limited Reliability, Limited Ability to Generalize to Population.

Different Paradigms – So why combine methods?

QUANTITATIVE RESEARCH

•Typical Strengths: Easy to Generalize Results, Large-Scale, Statistical Analysis, Visualisations.

•Typical Weaknesses: Loss of Depth and Context, Limitations in Accuracy / Validity, Disconnected from Real World.

Mixed Methods Research Approach:

A mixed methods approach to research design “accepts that quantitative, qualitative,

and mixed research are all superior under different circumstances and it is the

researcher’s task to make the decision about which research approach […] should be used

in a specific study” (Johnson and Onwuegbuzie 2004: 22-23).

• Remember that more methods do not necessarily improve a research project & to choose this method, you must have a clear rationale.

• Therefore, the decision to combine qualitative and quantitative methods should be driven by the value for effectively addressing your research question

Choosing your approach

Mixed Methods Research Design

• Methodological Eclecticism

• Motivations for Combining Research Methods

• Pragmatic Epistemology vs. Epistemological Fidelity

Methodological Eclecticism

• More methods = not necessarily better

• Hammersley and Atkinson (1995): methodological eclecticism describes “anything goes” approach to mixed methods.

• Alternative to mere eclecticism is having clear purposes for using multiple methods

• Requires understanding reasons why researchers choose to integrate methods.

Motivations for Combining Methods

Motivations for Combining Methods

• Convergent Findings, which uses qualitative and quantitative methods to address the same research question.

• Most likely to rely on a Convergent Findings motivation when greater certainty needed.

• Certainty comes from showing methods with different strengths yield similar results.

• This motivation also known as triangulation or cross validation

Motivations for Combining Methods

• Additional Coverage motivation assigns different strengths of multiple methods to different goals within overall project.

• This approach relies on division of labour, matching each method’s strengths to separate goal within overall research project.

Motivations for Combining Methods

• Connected Contributions motivation links methods together so one method enhances effectiveness of another. (Morgan also calls this ‘Complementary Assistance’)

• Aim of linking is to use what you learn from one method to inform how you will use another method. (e.g. pilot focus groups used to design large-scale survey)

Combining Research Methods

• Decision to combine methods must start with consideration of how you can combine qualitative and quantitative methods to serve your research purposes.

• Epistemology is one of first potential issues

Epistemology: Consistency Optional?• Mixed methods research typically follows

overarching epistemological approach or ‘paradigm’ based on Pragmatism.

• Pragmatic paradigm seeks to minimize questions about nature of reality by concentrating on social action as basis for knowing.

• Seeks to replace questions about what is true with questions about what is useful.

• With regard to Research Design, Pragmatism concentrates on extent to which your research procedures serve your research purposes.

Pragmatic epistemology• Pragmatism leads to what Patton (2002: 257) has

called a “paradigm of choices” where you justify choices about research procedures based on their ability to meet overall project goals.

• Essentially, this approach to research design:

“accepts that quantitative, qualitative, and mixed research are all superior under different circumstances and it is the researcher’s task to make the decision about which research approach […] should be used in a specific study” (Johnson and Onwuegbuzie 2004: 22-23).

Sequential Approach (Morgan)

Seminar Discussion

• Have you decided whether you are taking a quantitative, qualitative or mixed methods approach in your project?

• If yes, why? (in detail)

• In no, what are you hoping to achieve? How does this connect to the strengths and weaknesses of different methodological options?

Practice of Quantitative ResearchDr Eric Jensen (e.jensen@warwick.ac.uk)

31

Research Design: Combining Methods?

Example: Quantifying Qualitative

Survey Data

Key Question of Educational Impact

Are zoo educators achieving anything of value?

Specific Research QuestionsHow and why is learning taking place within the zoo?

-Answering these research questions required mixed methods social scientific approach (quantitative/qualitative)

ZSL wanted to find out specifically

Are there any differences in the impact of a zoo visit on a pupil if they have a tailored educational presentation by zoo staff compared to not having an education session (self-guided)?

Mixed quantitative and qualitative approach

• Pre- and post-visit surveys completed the day before the zoo visit and the day after the zoo visit.

• Sample size: n = 3018 respondents

• Multiple measures of learning about animals, habitats and conservation.

1. Thought-listing measure:

‘what do you think of when you think of a zoo?’ (x5)

Pre- and Post – visit forms - 3 Components:

2. Annotated drawing of ‘favourite wildlife habitat’ with all the plants and animals which live there

Thought-listing*What do you think of when you think of ‘the zoo’? (write below):

1.______________________________________

2.______________________________________

3.______________________________________

4.______________________________________

5.______________________________________

Annotated Drawings

  

*What did you draw above?

*Please draw your favourite wildlife habitat

and all the plants and animals that live there.(Please put names or labels on everything)

Thought-listing Measure (*Manifest Content*)

What do you think of when you visit a zoo?Top ranked words both pre- and post- visit: ‘fun’ and ‘animals’

‘Habitats’ – 9th most used word in pre-visit data - 4th most used word post-visit

‘Cages’ – 5th most used word pre-visit

- 13th most used word post-visit

‘Learning’ – 11th most used word pre-visit

- 3rd most used word post-visit

Annotated Diagrams of favourite habitat

Analysis of all paired forms – scoring on basis of 1-3

1= negative change in accuracy of representation (animals/habitat)

2= no change in accuracy

3 = positive change

Pre-session

Post- session

3

Annotated Diagrams of favourite habitat

Pre-session Post- session

Visits without education sessions showed a statistically significant positive impact on this measure of learning, with increasing accuracy of habitat and animal representations.On same measure, visits including education sessions showed almost double the increase in learning on this measure.

Other trends included:More appreciation for traditionally ‘less-charismatic’

animals Thought listing measure – post visit, increased mention of ‘bugs’,‘birds’ and ‘fish’ compared to pre-visit Annotated drawing of animal in favourite habitat, increase in post-visit of animals such as cockroaches and fewer mammals (than pre-)

Pre-visit Post-visit

Satisfaction• 57% of those who had an education session

expressed highest level of satisfaction with their visit (‘5’ out of 5 on scale).– compared to 49% for self-guided.

• Overall, pupils very happy with their zoo visit:– Zoo Educator–led: 84% satisfied, 5% dissatisfied– Self-guided: 77% satisfied, 5% dissatisfied

• This shows that attending an additional educational intervention (tailored to the particular zoo context) resulted in increased satisfaction compared to self-guided.

Inter-Coder Reliability

Inter-coder reliability

• Intercoder reliability = widely used term for extent to which independent coders evaluate message or other content characteristics reach the same conclusion.

• Some prefer a more specific term for the type of consistency required in content analysis is inter-coder (or inter-rater) agreement.

Inter-coder reliability• “Given that a goal of content analysis is to identify and

record relatively objective (or at least intersubjective) characteristics of messages, reliability is paramount. Without the establishment of reliability, content analysis measures are useless" (Neuendorf, 2002, p. 141).

• “Interjudge reliability is often perceived as the standard measure of research quality. High levels of disagreement among judges suggest weaknesses in research methods, including the possibility of poor operational definitions, categories, and judge training" (Kolbe and Burnett,1991, p. 248).

Content Analysis: Inter-coder Reliability

Calculating, reporting inter-coder reliability

•All content analysis projects should be designed to include the assessment and reporting of inter-coder reliability(or at min., intra-coder reliability) .

•Inter-coder Reliability is a necessary (although not sufficient) criterion for validity in the study and without it all results and conclusions in the research project may justifiably be doubted.

Inter-coder Reliability (entering in SPSS)

Unitvar1cod

er1var1cod

er2var2cod

er1var2cod

er2var3cod

er1var3cod

er2

 case 1                

 case 2              

 case 3                

 case 4                

 case 5                

 case 6                

 .                

 .                

Strengths of Content Analysis (Secondary analysis)

• Economy of time and money (in terms of data collection if secondary analysis).

• Simple to repeat portion of study if necessary.

• Permits study of processes over time.

• Researcher seldom has any effect on the subject being studied.

Strengths of Content Analysis (General)

• Reliability – it is possible to measure particular theoretically-relevant variables in a way that is in principle inter-subjective.

• Level of inter-subjectivity achieved is known (through inter-coder reliability).

• Clear, well-established method offers transparency and procedural clarity for research reader / users.

Weaknesses of Content Analysis• Limited to examination of recorded communications.

• Problems of validity are commonplace with e.g. media content data. – E.g. Are the data sources meaningful measures of what

we want to measure?– for example: is what appears in the media a good

representation of public opinion? (no) – Are blogs a good representation of the population’s

opinions? (probably not)

• Latent measures inevitably involve subjective interpretation (although this can be mitigated through inter-coder reliability measures).