Moving Beyond the Qual-Quant Divide

Peter DavisSocial Development Research Initiative

Agnes QuisumbingIFPRI

A4NH Gender-Nutrition Methods Workshop II, December 2-4, 2014Bioversity International - Rome, Italy

Introduction

• Qualitative and quantitative research in the social sciences are too often conducted separately

• There are a number of reasons for the qual-quant divide:– Different philosophical underpinnings– Different disciplinary traditions– Different research concerns and subjects– Different research contexts– Different gauges and status markers for judging research

CG researchers may not make the most of qualand quant approaches

• From the gender inventory, CGIAR researchers feel that gender research is the monopoly of qualitative researchers, but at the same time, is given lower priority or viewed as less legitimate:

“Data analysis is deemed a particularly weak point since analyzing the gender data is not always prioritized, especially not from surveys. Another challenge is that the results from the case studies are not always taken as seriously as survey results but rather considered anecdotal.”

But quant methods are often viewed as extractive

“There is an emphasis on deductive, quantitative methods – especially in the baselines – even if researchers sometimes mix methods…Some gender data is even completely neglected during the analysis phase.

The GFPs try add qualitative components to projects, but the need for more FGDs, unstructured interviews and participatory methods was noted even if they may be time consuming; both to facilitate the interpretation of quantitative data and to allow the participants to analyze their own realities.

The latter was seen as important to mitigate power asymmetries between the researchers and communities, for achieving bottom-up change, and to address the problem in quantitative research of not always feeding back the results to the communities.”

Moving beyond the divide makes research better

What are the essential differences between qualitative and quantitative research approaches?

For five minutes – discuss with your neighbor the key differences between qualitative and quantitative research

Inherent ambiguity in the subjective-objective distinction(adapted from Searle 2010)

Ontological (about what exists) Epistemological (about claims to knowledge)

Subjective • Individual observer dependent (pains, aches, tickles)

• Collective observer dependence for the most important aspects of their existence (e.g. money, marriage, property boundaries, declarations of war). Much of the subject matter of the social sciences.

Non-scientific claims that are observer dependent. (e.g. aesthetic claims or judgmentsabout art and literature - such as -that Rembrandt was a better painter than Rubens). Susceptible to observer bias and idiosyncratic tastes.

Objective Observer independent – things that exist independently of the minds of observers (e.g. mountains, molecules, tectonic plates). Much of the subject matter of the natural sciences.

Natural and social scienceknowledge claims. Attempt to be observer-independent and can be based on either quantified or non-quantified evidence, or both.

Implications

• There is no reason that objectively verifiable claims cannot be made about ontologically subjective objects or states of affairs in the world

• But subjective claims are not scientific claims (social or otherwise). They are opinions or tastes

• Claims based on qualitative or quantitative data can be objective• The natural sciences use both without question• The social world is challenging because much of it is ontologically

subjective and because states of affairs are causally and contextually complex

• To restrict ourselves to either quantitative or qualitative data unnecessarily hinders our ability to understand the social world

Page 8

Example of a Q-squared study:

The CPRC-DATA-IFPRI Longitudinal Study

The CPRC-DATA-IFPRI Bangladesh longitudinal study

• The study combined three IFPRI evaluations which started in 1994, 1996 and 2000/03, and used a mixture of quantitative and qualitative methods

• In 2006-7 we resurveyed the entire set of these households (plus new households created due to household division) in three phases (qual-quant-qual)

The 2006-7 Study’s 3 Phases

3 phases of data collection:

• Summer 2006: focus group discussions investigating causes of decline and improvement and the long term impact of 3 interventions (116 FGDs in 11 districts)

• Winter 2006-7: quantitative resurvey of panel households (1787 core + 365 splits in 14 districts)

• Spring-Summer 2007: life-history interviews and village histories in 8 districts (161 households –293 individuals)

Map of the Study Sites

Nilphamari (38)

Kurigram (39)

Tangail (39)

Kishoreganj (19)

Mymensingh (18)

Manikganj (72)

Jessore (36)

Cox’s Bazar (32)

Arrows show life-history

districts (number of

interviews)

What does qual research bring to quant?

• Challenges in researching the social world– dealing with complexity– intangibility– ontological subjectivity– morality and ethics

• Exploration of context (in space and time)• A better qualitative understanding of social realities so that good

variables are chosen for quantitative measurement• Better identification of complex processes and causal relationships –

which can then be more systematically investigated quantitatively (qualwork to inform formulation of hypotheses that can be tested quantitatively)

What does quant research bring to qual?

• Larger numbers of randomly selected cases control bias and increase representativeness of cases and therefore generalizability of findings

• The ability to compare the effects of interventions without being led astray by random variation or confounding causes

• Reduction in observer dependence due to larger numbers of cases and more formal methods

• Systematic, transparent, repeatable methods (replicability)

Practical issues in combining qualitative and quantitative research

1) Planning

• clarifying research questions

• piloting and field testing

• using flexible qual approaches in exploratory research

• identifying variables – drawing from qualitative research to help design quant research instruments

• consider how to avoid qual findings being seen as anecdotal or with questionable representativeness?

• hypothesis testing and comparing interventions

2) Sampling

• What is the population?

• Are samples representative of the population?

• What is a case?

• Will qual cases be a subsample of quant sample? If yes, should be linked using same IDs

3) Fieldwork

Will the methods be sequential?• quantqualquant• qualquantqual

Parallel?• quant and qual occurring at the same time but separately

Concurrent?• quant and qual occurring at the same time in the same place

(integrated fieldwork)

Personnel

• Training, expertise, supervision, rapport

4) Data analysis decisions• Do analysts have the necessary skills to analyse both sets of data?• Will quant data arise from qual interviews? Quantifying qual data – what

is lost in reduction to numbers too early?• Can quant data be used in qual analysis?

5) Presentation of findings and user engagement• Are findings presented separately?• What happens when qual and quant findings disagree? Using

disagreements as opportunities for learning

6) Ethical concerns

• Protection of research participants

– Is there more risk of harming participants?

– Maintenance of anonymity when ‘thick descriptions’ are part of the dataset

– Release of data to other users

• Is there greater potential to influence policy in a positive direction? And less risk of misrepresenting social, economic, political reality?

Useful publications and websites:

Journals• Journal of mixed methods research http://mmr.sagepub.com• International Journal of Multiple Research Approaches http://mra.e-contentmanagement.com

Books, papers and book chapters• Bryman. A. Social research methods. Chapters 21 and 22• Mayoux, L. “Quantitative, Qualitative or Participatory? Which Method, for What and When?”

Chapter 13 in Desai and Potter, Doing Development Research. Sage: London.• Tashakkori, A. Handbook of mixed methods in social & behavioural research• R. Kanbur (Ed.), Q-squared: Qualitative and quantitative methods of poverty appraisal. New

Delhi, India: Permanent Black– Available from www.q-squared.ca/pdf/Q2_WP1_Kanbur.pdf

• McGee, R. Constructing Poverty Trends in Uganda: A Multidisciplinary Perspective. Development and Change 35(3): 499–523 (2004).

Useful websites:• www.q-squared.ca a research project looking at mixed methods approaches in poverty

research• www.compasss.org linked to Charles Ragin’s Qualitative Comparative Analysis approach (QCA)