qualmethods minicourse

7/29/2019 QualMethods minicourse

http://slidepdf.com/reader/full/qualmethods-minicourse 1/50

Using Qualitative MethodsUsing Qualitative Methodsin Empirical Studies of in Empirical Studies of Software EngineeringSoftware Engineering

University of Maryland Baltimore County University of Maryland Baltimore County Fraunhofer USA Center for Empirical Software Fraunhofer USA Center for Empirical Software

Engineering Maryland Engineering Maryland

ESELAW 2009 ESELAW 2009

12 November 2009 12 November 2009

São Carlos São Carlos- -SP, Brazil SP, Brazil



DefinitionsDefinitions

Qualitative data - data in the form oftext and pictures, not numbersQualitative analysis – analysis of

© Carolyn Seaman, 2009

trends, patterns, and generalizationsGrounded theory – theory formedbottom-up from the (usuallyqualitative) dataRich data – data that includes a lot of

explanatory and context information



Why Qualitative Methods?Why Qualitative Methods?

Problem : Difficult to answer complexSE questions with a purelyquantitative approach because


Typically have small sample sizesExperiments are expensive to runNeed some support for a hypothesis

before investing effort in full experimentSolution : Use a qualitative approachthat includes a quantitative aspect



Types of resultsTypes of results

A qualitative study will result in:Propositions tied to a trail of“evidence”


-Complex findings that incorporatethe messiness of the phenomenon

under studyExplanationsAreas for future study



Types of ResearchTypes of ResearchQuestionsQuestions

Qualitative methods are most appropriatewhen:Subject of study involves human


e av orNo concrete hypothesesVariables hard to define or quantify

Little previous workQuantitative results may be hard tointerpret



Advantages to ResearchersAdvantages to Researchers

Richer resultsResults more explanatory


Avoid errors in interpretation



Advantages to PractitionersAdvantages to Practitioners

Richer , more relevant resultsTerminology of results


Opportunity to clarify and explainfindings



Overview of TechniquesOverview of Techniques

Data Collection

Prior Ethnography

Data Analysis

Codin


ParticipantObservationInterviewing

SurveysDocument Analysis

ConstantComparison MethodCross-case analysis

Member checkingAuditing



Participant ObservationParticipant Observation

Definition: non-covert directobservation of phenomenon


inspection meetingscollected both qualitative and quantitative

datadid not participate in the inspectionused data forms as well as field notes



Observation Data FormObservation Data FormInspection Data Form

Class(es) inspected Inspection date: Time:

Author:Moderator:Reviewers:

Name Responsibility Preparation time Present


Amount of code inspected:Complexity of classes:Discussion codes:D = Defects Q = Questions C = Classgen defect U = Unresolved issues G/D = Globaldefects G/Q = Global questions P = Process issues A = Administrative issues

M = Miscellaneous discussion

Time logged (in minutes):D______ Q_____ C_____ U_____ G/D______ G/Q______ P______ A______ M______



Field Notes ExampleField Notes Example

The "step" function is a very important but complicated function.[Reviewer1] did not have time to review it in detail, but[Author] said he really wanted someone to go over it carefully,so [Reviewer1] said she would later.

-


values. This is a problem because often the code is such thatthere is no way to tell what a particular variable was initializedto. [Reviewer2] said "I have no way to see initial value". Thiswas a global discussion, relevant to many classes, including

[Reviewer2]’s evidently.



InterviewingInterviewing

Interviews are good for gettingopinionsfeelin s

© Carolyn Seaman, 2009 12

goalsprocedures (both formal and informal)

not facts



Standard Interview FormatsStandard Interview Formats

Structured (standardized)Tightly scripted, almost verbalquestionnaire


,Analyze like questionnaire“How many times a day do you access the internet? [0, 1-5, 5-10, 10-15, 15+]”




Unstructured(Open/Informal/Conversational)Guided by a very scant script.Rich but not re licable.


Difficult to be systematic, problem ofcoverage.Minimize interviewer effects, preservesinterviewee point of view.Interviewee led, interviewer probes.“Please, tell me about your internet usage...”




Semi-structuredGuided by a script ( interview guide ), butinteresting issues can be explored in


.Good balance between richness andreplicability.Mixed analysis techniques.

“In a typical day, how often do you use the internet?”



Interview questionsInterview questionsClosed

Predetermined answer format (e.g. Yes/No)Easier to analyze

OpenNo predetermined answer format


More complete responseCombinationClosed, with opportunity to elaborate

Probes

Pitfalls :leading questionsdouble-barreled questions

judgmental questions



Interview GuideInterview Guide

A script for use by interviewer only“Wish list” vs. structuredFlow/direction to interview


equ re top csTransitions between topic areasImportant for replicabilityWording and sequence are critical



Interview DesignInterview DesignConsiderationsConsiderations

Context switchingFlow between open and closed


“Shape ” of interviewMost important stuff first

Wording



Interview ShapesInterview Shapes

FunnelBegin with open, gradually become more closedGood if you’re not sure what you’re going to get

Pyramid


,Good with nervous interviewees

Hour glassBegin with open, gradually become more closed,then open up again at end to pick up things youmight have missedGood if you know what you want, but suspectthere are important things you don’t know aboutyet



Interview Guide 2a : In-depth project interviewsWho : Developers on [Project1], [Project2], [Project3]Subjects covered : general opinions of GSS processes and productsDuration : 60-90 minutes

What do you like about the current process using GSS?

Example Interview GuideExample Interview Guide


at o you s e a out t e current process us ng

Do you depend on any other groups, either for information or help withGSS, or for work to be done related to GSS?

What do you like about the applications resulting from using GSS?

What do you dislike about the applications resulting from using GSS?

Have there been any problems with the interface between GSS and otherCOTS products?

What do you see as the top risks associated with the use of GSS? Howwould you mitigate these risks?



Interviewing PointersInterviewing Pointersgive clues about the level of detail you want

establish rapport , but be subject neutralavoid jargon , esp. academesedispel any notion of the “ right ” answer

la the novice when a ro riate


probe , but do not leadalways be aware of your biasesbe sensitive to their work (environment/schedule)no more than 60 minuteslet interviewee know next stepsend with “ anything else I should know? ”say Thank you !



Recording of interviewsRecording of interviews

Audiorecording


Scribing



AudiorecordingAudiorecording

Best memory mechanismFull transcription or just verbatim quotesStill take notes

Tapes fail, digital files are deleted


Does not record all aspects (esp. context / facial expressions)Required consent

Always ask first.

Do NOT hide recorder, keep it visible at alltimes.Give the option to turn it off at any point.



NotetakingNotetaking

Very hard to take notes andinterview at the same timeThere are some super-


researchers who can do itInevitably results in incompletenotesSlows down the interviewSometimes inevitable



ScribingScribing

Partner-based interviewingAdvantages of a single contact vs. trading-offCan share roles (interviewer/scribe)


ta e notes, t oug to erentdegree

Group debrief: what did you get/miss?Synchronize notes: overlap and emphasis

Clarify while it is still in your head



Writing up the interviewWriting up the interview




Interview NotesInterview Notes

Write it up immediatelyDescriptive vs. reflective notesUse Observer’s Comments

Impressions, state of mind, assumptions, notes to


How detailed?Verbatim transcript

only possible with audiorecordingExtremely labor-intensive

Summaries with major points quotedOK, but use LOTS of quotesStart closer to verbatim at the beginning of a study



Interviewing ExerciseInterviewing Exercise

Background :The National Federation of Makers of Feijoada(FNFF ) is concerned that the nationalconsumption of feijoada is declining due to


ecreas ng qua ty o e oa a .So they have asked us to interview the topfeijoada chefs in the country (as determined byregional competitions)The goal is to find out the secrets to masterfeijoada making, so that it can start to be taught inelementary schools .

28




Three versions of the interviewguide


You will be the interviewees

So take a moment to think of your

favorite feijoada chef

29




RecapFirst interview: pyramid

Started with easy, closed questionsEnded with open-ended questions

1. What do you think makes your feijoada the best?2. What is special about your ingredients?3. What are the basic steps to making feijoada?4. Who taught you to make feijoada?5. How long does it take you to make a feijoada?

1. How often do you make feijoada and how long does ittake you?

2. What do you think makes your feijoada the best?3. Of course, you always wash your hands thoroughly

before you start, right?


econ nterv ew: funnelStarted very broadly, with open questionsFollowed up with narrower, closed questions

Third interview: just bad

Leading, judgmental questionsDouble-barreled questionsSwitching from topic to topicSwitching between open and closed

30

1. What is your name, please?2. How often do you make feijoada?3. How long does it take to make feijoada?

4. What are the ingredients you use?5. What do you think makes your feijoada the best?

. the end of the cooking?

5. What kind of pot do you use?



Constant ComparisonConstant ComparisonMethodMethod

Qualitative analysis method

Meant to generate grounded theoryOperates on a set of field notesB sic rocess:


codinggroupingwriting field memo

forming hypothesesRepeated periodically in parallel withdata collection



What’s a Code?What’s a Code?

A labelA concept


A categoryA relationship

A theme



What’sWhat’s Coding?Coding?

Open coding - assigning codes topieces of textual dataCoded “chunks” can overlap


One chunk can have several codesAxial coding - grouping, categorizing,combining coded chunks

Selective coding - making sense of it



What’s here? What are the pieces?

Identification/discovery of conceptsClassification labelin of henomena

Open CodingOpen Coding


Abstraction (this is part of that)Comparative analysis (this is different fromthat)

Categorization (organization, grouping)Value-neutral, at least initially“complexity” not “high complexity” or “lowcomplexity”



Preparing for codingRead the dataRead background material and research designCreate pre-formed codes, if applicable

Coding by hand

Open CodingOpen Coding ProcessProcess


, .

Photocopy, scissors, and envelopesMS Word commentsExcel

Coding tools – NVivo, Atlas TI

Coding schemePre formed or post formed codesConstant iterationStructure develops over time



• Background :

• Study of the role of documentation insoftware maintenance

• Interviews with experienced software

OpenOpen Coding ExerciseCoding Exercise


ma nta ners n severa organ zat ons• Process :

• I’ll show you an example• Then you’ll try it – code one excerpt with

one code• Find a partner – compare your codings• I’ll show you my coding of the excerpt



Respondent BackgroundInformation Gathering

Transition to maintenanceTypes of documentationCharacteristics of DocumentationQuality of documentation

Coding SchemeCoding Scheme


Properties of documentationMissing documentationCreating documentationLocation of documentationImportance of documentationHuman sources of informationQuality of ProcessGreat Quotes

Human Sources of Information



Open Coding andOpen Coding andQuantificationQuantification

One form of coding

Objective is to derive quantitative data from qualitative datafor future statistical analysisUsually involves counting

How many subjects said…?


…How many times did …?

Or timingHow long did subjects spend doing…?How long did it take to …?

Inevitably loses richnessOften seems a little like missing the point

What’s the point of collecting rich data when you’re just goingto condense it down to numbers?

But often is an effective and necessary way to reduce thesize of the data



How are things related?

Initial process of reassemblingRelationships among categories and codes

Axial CodingAxial Coding


Structure (why?)Process (how?)Explanations not causal prediction



How does it all fit together?

Also called sense makingRelationships among relationships

Selective CodingSelective Coding


Theory constructionThe central categoryStoryline memosRole of literatureWrite, write, write!!!Field Memos



Field MemosField Memos

The “ single most powerful analytical tool ” forqualitative researchersSimply, a piece of writingMaybe will later become part of a report,


Summarizes and synthesizes:

A propositionAn open questionA chain of evidence and logic

The complexity of a conceptRich descriptionVersion control and organization



Judging ValidityJudging Validity

Validity of methodsTriangulation


Contradictory evidenceWeight of evidence

How much is enough?Variety as well as quantity ofevidence



Using Qualitative andUsing Qualitative andQuantitative MethodsQuantitative Methods

Together Together

Qualitative and quantitative methodsbest used in combination


an s mp y e use n para e oaddress the same researchquestions

There are other strategies to betterexploit the strengths andweaknesses of the methods



Example Design 1Example Design 1 :: StatisticalStatisticalHypothesis Testing withHypothesis Testing withFollowFollow- -up Interviewsup Interviews

Classic design – often done withoutfully exploiting the interview data


Example scenario:Blocked subject-project experiment toevaluate a new testing techniqueStatistical results show that technique ismore effective on some applications thanon othersQualitative results show why



Want to evaluate a new technique, butnot sure what the evaluation criteriashould be

Example Design 2 : Using GroundedTheory to Identify Variables


Evaluating a collaborative design processUse participant observation of designmeetings to generate hypotheses aboutproperties of the resulting designsGrounded hypotheses are used to design aquantitative evaluation of the resultingdesigns



Example Design 3Example Design 3 :: Using Prior Using Prior Investigation toInvestigation to

OperationalizeOperationalize VariablesVariables

Relevant variables are known, but therange and types of values is difficult tospecify


Example scenario:Want to study the relationship betweendeveloper experience and types of defectsFirst use interviews to identify the range of

developer experience (in its complexity) anda taxonomy of defect typesQuantitative study then is much moreeffective when using this operationalization



ConclusionsConclusionsEmpirical software engineering researchers areaddressing more and more complex research

questions that have increasingly humanelementsQualitative methods, usually in combination with


,

this complexityQualitative methods are both flexible andrigorousQualitative analysis provides richer, morerelevant, and more explanatory resultsThe most effective research designs combinequalitative and quantitative methods



BibliographyBibliographyDay, I. Qualitative data analysis: A user-friendly guide for social scientists . New York: Routledge. 1993.Eisenhardt, Kathleen M. "Building theories from case studyresearch." Academy of Management Review, 14:4, 1989.Gilgun, J.F., "Definitions, Methodologies, and Methods inQualitative Family Research." Qualitative Methods in Family

Research . Sage, 1992. pp. 22-29Glaser, Barney G. and Anselm L.Strauss. The Discovery of Grounded Theory: Strategies for Qualitative Research. de


, .

Jick, Todd D. "Mixing qualitative and quantitative methods:Triangulation in action." Administrative Science Quarterly,24:4, 1979.Judd, Charles M., Eliot R. Smith, and Louise H. Kidder.Research Methods in Social Relations. Harcourt, Brace,Jovanovich, 1991.Miles, Matthew B. and A. Michael Huberman. Qualitative DataAnalysis: An Expanded Sourcebook. Sage Publications, 1994.

qualmethods minicourse

Documents