Validity of Qualitative Data

Carolyn SeamanUMBC

Baltimore, USA

Presented to the

International Software Engineering Network (ISERN)

21 August 2001

Glasgow, Scotland

What is Qualitative Data?

• Data in the form of words and pictures, not numbers

• Data is richer and carries more information, but is harder to analyze than quantitative data

• Can be either objective or subjective• Methods designed to elicit perceptions, feelings,

and opinions

Advantages

• Richer, more explanatory results

• Terminology of results closer to “customer”

• More participatory research process

• Avoid errors in interpretation

• Opportunity to clarify and explain findings

• Many important questions simply can’t be answered quantitatively

Overview of Techniques

Data Collection

• Prior Ethnography• Participant

Observation• Interviewing• Surveys• Document Analysis

Data Analysis

• Coding• Constant Comparison

Method• Cross-case analysis• Member checking• Auditing

Are Qualitative Studies Valid?

• Stupid question

• Validity concerns are no more (and no less) present in qualitative studies, but they are different

• It’s not harder to do a valid qualitative study, it’s just harder to show validity convincingly

Ensuring Validity

• Validity of data– Is the data accurate?– Triangulation– Recording (audio, video, scribing)– Instrument bias– Political motivations

Ensuring Validity (cont.)

• Validity of analysis– “Weight of evidence” - how much is enough?– Variety as well as quantity of evidence– Contradictory evidence

• emerging theory must be modified in the presence of evidence that contradicts the current version of the theory

• all contradictory evidence must be accounted for

– Outliers, extreme cases, and surprises are good!

Showing Validity

• Documentation– all steps in process must be documented - keep

a log– auditing– transparency– keep track of sources for each data item– document exploration of all rival explanations

Showing Validity (cont.)

• Dissemination (papers, talks, etc.)– make validity concerns explicit– spell out all procedures to ensure validity

(triangulation, recording, etc.)– use quotes from data to illustrate conclusions– explain how contradictory evidence was handled– quantify where possible and appropriate– rival explanations

Miles & Huberman:

Standards for the Quality of Conclusions

• Objectivity - treatment of contradictory evidence and rival explanations; documentation of methods

• Reliability - competence of researchers; quality of methods

• Internal validity - logical sense, triangulation, rival explanations

• External validity - representativeness; documentation of context

• Utilization - relevance

Appropriateness

When are qualitative methods appropriate?– Subject of study involves human behavior – No concrete or easily stated hypotheses– Variables hard to define or quantify– Little previous work – Quantitative results may be hard to interpret– When there’s probably not a yes/no answer

Conclusions

• Qualitative data is highly useful in many types of studies

• The validity of qualitative results is as variable as that of quantitative results

• One can take steps to ensure the validity of qualitative data and analysis methods

• Harder to show that one’s qualitative work is valid