“is my research good?” quality assessment of qualitative research
TRANSCRIPT
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Eva Dobozy, PhD – WAIER Forum 8 August 2015
“Is my research good?” Quality assessment of qualitative research
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Organisation of presentation: What is research methodology? What is quality research? Who uses qualitative research? Why use qualitative research? Ontological concerns Epistemological concerns Research paradigms and methodological
concerns Standards for quality assurance in
quantitative/qualitative research
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
What is research methodology and how is it different from research methods?
Researchers (particularly adhering to positivist research conventions) may use the term methodology and method interchangeably.
“This practice is unfortunate because they are not the same. The former refers to philosophy and the latter refers to technical procedures applied to conduct research…. Simply put, methodology refers to how each of
logic, reality, values and what counts as knowledge inform research. On the other hand, methods are the techniques and procedures followed to
conduct research, and are determined by the methodology.”McGregor & Murnane (2010)
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
What is quality research?
Methodology
Makes overt (explains) the philosophical underpinnings of the research
Method
Makes overt (explains) the data gathering and data analysis techniques used to arrive at the findings. These are clearly aligned with
(a) the methodology chosen (b) the research problem
under investigation
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
I am the chair of the newly formed Human Research Ethics Advisory Committee (Humanities)
Role:• to provide scientific review of the ethics application to promote
scientific integrity.
Process:• ethics applications (non-low risk) be first reviewed by an Advisory
Committee for methodological rigor before going to the HREC for ethics approval
Anticipated impact:• help minimise the time the ethics committee spends on scientific/non-
ethics review • ensure the science is reviewed by someone with knowledge in the
appropriate area
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
What is qualitative research?
Qualitative research means different things to different people working in different paradigms, locations and areas
Qualitative research is a form of social inquiry that is person-centred
Qualitative research emerged from a number of different research traditions or disciplines
Great variation of approaches
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
What is qualitative research? (cont.)
Generic definition by Denzin & Lincoln (2011):• “Qualitative research is a situated activity that locates the observer in the
world. • Qualitative research consists of a set of interpretive, material practices
that make the world visible. • These practices transform the world. • They turn the world into a series of representations, including
fieldnotes, interviews, conversations, photographs, recordings, and memos to the self.
• At this level, qualitative research involves an interpretive, naturalistic approach to the world.
• This means that qualitative researchers study things in their natural settings, attempting to make sense of or interpret phenomena in terms of the meanings people bring to them.”
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Who uses qualitative research?
“Once seen as the realm of outlaws and bohemians, qualitative research is now an accepted and prominent feature of the academic landscape.”
(Orne & Bell, 2015)
Developed in the social and human sciences (sociology, psychology, philosophy, education), qualitative research is now also increasingly accepted as a legitimate methodology in criminal law and medicine.
Scholars in ALL academic disciplines use qualitative research
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Why use qualitative research?
Two main reasons:
(1) – the pluralisation of life worlds
(2) – the problem (or fiction) of objectivity of quantitative methods
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Why use qualitative research? (cont.)
(1) The pluralisation of life worldsSocial diversity and rapid social, cultural and technological changes result in a magnitude of different life experiences and realities. No more ‘grand narratives’:
• The student• The teacher• The business person
Context-dependency is required
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Why use qualitative research? (cont.)
(2) The fiction of objectivity of quantitative methods“Value freedom as an unachievable goal, at all stages of a [research] project”.
(Payne & Payne, 2004)
“Bias can occur in the planning, data collection, analysis, and publication phases of research.”
(Pannucci & Wilkins, 2010)
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
What is the purpose of qualitative research?
The aim of qualitative research is to investigate and better understand the social reality of:
• Individuals• Groups• Cultures
• Qualitative researchers explore people’s:• behaviours• perspectives• feelings• experiences
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Ontological concerns of qualitative research
Ontology is concerned with the following question:
What constitutes reality?
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Ontological concerns of qualitative research (Realist ontology)
Realist Ontology assumes that there are real world objects and behaviours (distinct from the human knower) that can be observed and measured
Positivist/Objectivist beliefs:
There is an objective and universal reality
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Ontological concerns of qualitative research (Relativist ontology)
Relativist Ontology assumes that reality is different for different people (depending on ethnicity, culture, gender, age etc). Reality is constructed intersubjectively through the meanings and understandings developed socially and experientially.
Interpretivist/Subjectivist beliefs:
Reality is socially constructed.
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
What is your ontology?
Realist Relativist
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Epistemological concerns of qualitative research
Epistemology is concerned with the following questions:
What is knowledge?How do we come to know?
What makes justified beliefs justified?
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Epistemological concerns of qualitative research (Realist epistemology)
Realist epistemology is related to scientific knowledge production and objective, independent knowledge.
Truth-value = absolute Characteristics:
• Objectivist• Mind and viewpoint independent• Universal• Value-neutral - The truth-value of the research is believed to be strong
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Epistemological concerns of qualitative research (Relativist epistemology)
Relativist epistemology is related to scientific knowledge production and subjective, viewpoint dependent knowledge.
Truth-value = relative Characteristics:
• Subjectivist• Person-centred• Relative – mind and viewpoint dependent• Pluralist• Value-laden
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
What is your epistemology?
Truth-value of scientific research:
Absolute Relative
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Philosophical assumptions governing qualitative research (research paradigm) Positivist qualitative research
• Objective and universal reality• Research can (as) objectively (as possible) measure and report phenomena using
validated instruments = Researchers are invisible• Theory-driven and theory-confirming research, assigning dependent/independent
variables
Interpretivist qualitative research• Reality is socially constructed• Research reports on member’s understand and experience of phenomena• Researcher values are declared and written into the script
Critical qualitative research• Reality is socially constructed and reproduced (class struggles)• Research as social critique and emancipation
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
What is your research paradigm?
Positivist research
Interpretive research Critical research
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Methodological concerns of qualitative research
EthnographyBased on social and cultural anthropology, ethnographic researchers immerse themselves in the lives of people, social and cultural groups
Grounded TheoryThe primary goal is to develop theory that is grounded in data, which is systematically collected and analysed
PhenomenologyThe primary goal is to document ‘lived experience’ of people, social and cultural groups
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Methodological concerns of qualitative research (cont.) Case Study
The primary goal is to investigate and compare different cases as phenomena with blurred boundaries between concept and context
Action ResearchThe primary goal of action research is to improve practice through collaboration and documentation of action using scientific frameworks and methods
Design-based ResearchSimilar to action research, the primary goal of design-based research is the collaborative and iterative development of solutions to praxis related problems, but mainly in technology-enhanced learning environments
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
What is your preferred research methodology?
Ethnography
Phenomenology Grounded TheoryCase Study
Action ResearchDesign-based Research
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Methods used in qualitative research
Most common research strategies: Field research
• Interviews• Individual and Focus Group• Unstructured, Semi-Structured, Structured
• Observation • Non-participant, Participant, mass observation
• Library research• Document Analysis
• Policy and curriculum analysis and mapping
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
What are your preferred research methods?
F2F Individual Interviews
Focus Groups Participatory Observation Non-Participatory Observation
Document Analysis Questionnaires
E-Interviews
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Standards for quality assurance in quantitative/qualitative research
Contemporary quantitative/qualitative research accepts the impossibility of ‘absolute objectivity’ Researchers are advised to ‘constrain’ their personal
prejudices Quality assurance requires:
• The following of a set of protocols and accepted procedures • Transparency of procedures that were followed
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Standards for quality assessment of qualitative research
Language Strategic alignment between research paradigm and
language used in the reporting of findings
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Standards for quality assessment of qualitative research - Language
“The conventional methodological criteria of quantitative research — validity, reliability and empirical generalisability — are generally not directly applied to qualitative research because of the different frameworks, sampling approaches, size of sample and goals of qualitative research.”
(Kitto et al., 2008)
Remember: Most qualitative research in grounded in relativist philosophy (but not all)!
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Parallel quality criteria Guba & Lincoln (1989)
Realist/Positivist termInternal validityGeneralisability ReliabilityObjectivity
Relativist termCredibilityTrustworthinessDependabilityConfirmability
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
What is your research language?
Realist / Positivist Relativist / InterpretivistMostly associate with Mostly associated with
Quantitative Research Qualitative ResearchDo you prefer to talk about:
Reliability Rigour Validity Credibility Generalisability Trustworthiness Objectivity Confirmability
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Quality standards in positivist research
ObjectivityThis term refers to a principle drawn from positivism that researchers are able to remain detached from and neutral about their object of investigation, so as not to ‘contaminate’ the study based on personality, beliefs and values of the researcher.
GeneralisabilityThis term refers to an act of reasoning that involves drawing broad inferences from particular observations
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Quality standards in positivist research (cont.)
ReliabilityThis term refers to the consistency of a measure of a concept – replicability of the research
ValidityThis term refers to the issue of whether an indicator really measures a particular concept
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Quality standards in interpretivist research
Confirmability (not objectivity)The term refers to steps taken by the researcher to reduce the effects of bias. Declaring values positions and triangulation of theories, data and methods help ensure, as far as possible, that the research findings reported are the results of investigation and accurate representations of the experiences and ideas of research participants.
Transferability (not generalisability)This term refers to the need for sufficient thick description of the phenomena and context under investigation to enable the reader to make transfers and comparisons to other sites and circumstances, making possible the construction of general policy.
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Quality standards in interpretivist research (cont.)
Dependability (not reliability)This term refers to the dependability of findings. This may be achieved through detailed description of research design, data gathering procedures, such as triangulation of methods and interpretive rigor and reflexivity (showing the close link between credibility and dependability).
Credibility (not internal validity)This term refers to a principle congruency of findings with reality and the adoption of appropriate and well recognised research strategies and methods.
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Questions to ask when assessing the quality of qualitative research
What is the research paradigm (ontology & epistemology)? What is the research methodology? What are the research methods? Is there strategic alignment between research paradigm,
research methodology and research methods? (check language!)
Are the reported findings credible, dependable and transferable?
Are the limitations of the study recognised?
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Thank you.
Questions
???