case studies for software engineers - northeastern university · ªwhat is a case study? ªwhat is...

1

1

NASA SW Engineering Workshop 2005 Tutorial

© 2004, Perry, Sim, and Easterbrook

Case Studies for Software Engineers

Dewayne E. PerryThe University of Texas at Austin

Co-authors: Susan Elliott Sim, University of California, Irvine

and Steve Easterbrook, University of Toronto

2



Overview

1. Recognizing Case Studies

2. Designing Case Studies

3. Publishing Case Studies

4. Reviewing Case Studies

Authorized licensed use limited to: IEEE Xplore. Downloaded on January 20, 2009 at 14:53 from IEEE Xplore. Restrictions apply.

2

3



1. Recognizing Case Studies

4



Overview of this Section

IntroductionWhat is a case study?What is not a case study?Why conduct a case study?

IdentificationHow can I tell it’s a case study?

AnatomyWhat are the parts of a case study?

CritiqueHow can I evaluate a case study?


3

5



What is a case study?

A case study is an empirical research method.It is not a subset or variant of other methods, such as experiments, surveys or historical study.

Best suited to applied problems that need to be studied in context.

Phenomena under study cannot be separated from context. Effects can be wide-ranging.How and why questions

Settings where researcher has little control over variables, e.g. field sites.

Effects take time to appear.Days, weeks, months, or years rather than minutes or hours.

6



What is not a case study?

Not an exemplar or case historyIn medicine and law, patients or clients are “cases.” A review of interesting instance(s) is called a case study.Not a report of something interesting that was tried on a toy problem

Not an experience reportRetrospective report on an experience (typically, industrial) with lessons learned

Not a quasi-experiment with small nWeaker form of experiment with a small sample sizeUses a different logic for designing the study and for generalizing from results


4

7



Why conduct a case study?

To gain a deep understanding of a phenomenonExample: To understand the capability of a new toolExample: To identify factors affecting communication in code inspectionsExample: To characterize the process of coming up to speed on a project

Objective of InvestigationExploration- To find what’s out thereCharacterization- To more fully describeValidation- To find out whether a theory/hypothesis is true

Subject of InvestigationAn intervention, e.g. tool, technique, method, approach to design, implementation, or organizational structureAn existing thing or process, e.g. a team, releases, defects

8



When to use case studies

YesNoHow, why?CaseStudy

NoNoHow, why?History

Yes/NoNoWho, what where, how many, how much?

ArchivalAnalysis

YesNoWho, what, where, how many, how much?

Survey

YesYesHow, why?Experi-ment

Focuses on contemporary

events?

Requires Control of Behavioral

Events?

Form of Research Question

Strategy


5

9








10



How can I tell it’s a case study?

Has research questions set out from the beginning of the study

Data is collected in a planned and consistent manner

Inferences are made from the data to answer the research questions

Produces an explanation, description, or causal analysis of a phenomenon

Can also be exploratory


6

11








12



What is a case study?

A case study is an empirical inquiry thatInvestigates a contemporary phenomenon within its real-life context, especially whenThe boundaries between phenomenon and context are not clearly evident.

The case study inquiryCopes with the technically distinctive situation in which there will be many more variables of interest that data points, and as one resultRelies on multiple sources of evidence, with data needing to converge in a triangulating fashion, and as another resultBenefits from the prior development of theoretical propositions to guide data collection and analysis.


7

13



Parts of a Case Study Research Design

A research design is a “blueprint” for a studyDeals more with the logic of the study than the logisticsPlan for moving from questions to answersEnsures that the data is collected and analyzed to produce an answer to the initial research questionStrong similarities between a research design and a system design

Five parts of a case study research design1. Research questions2. Propositions (if any)3. Unit(s) of analysis4. Logic linking the data to the propositions5. Criteria for interpreting the findings

14



Part 1: Study Questions

Case studies are most appropriate for research questions that are of the “how” and “why” variety

The initial task is to clarify precisely the nature of the study questions (i.e. make sure they are actually “how” or “why” questions)

Examples:“Why do 2 organizations have a collaborative relationship?”"Why do developers prefer this tool/model/notation?" "How are inspections carried out in practice?“"How does agile development work in practice?" "Why do programmers fail to document their code?“"How does software evolve over time?“"Why have formal methods not been adopted widely for safety critical applications?“"How does a company identify which software development projects to start?"


8

15



Types of Case Studies

ExplanatoryAdjudicates between competing explanationsExample: How important is implementation bias in requirements engineering?

Rival theories: existing architectures are useful for anchoring, vs. existing architectures are over-constraining during RE

DescriptiveDescribes sequence of events and underlying mechanismsExample: How does pair programming actually work? Example: How do software immigrants naturalize?

CausalLooks for causal relationship between conceptsExample: Requirements errors are more likely to cause safety-related defects than programming errors are

See study by Robyn Lutz on the Voyager and Galileo spacecraft

ExploratoryCriteria or parameters instead of purposeExample: Christopher Columbus’ voyage to the new worldExample: What do CMM level 3 organizations have in common?

16



Part 2: Study Propositions

Propositions are statements that help direct attention to something that should be examined in the case study, i.e. point to what should be studied

Example: “Organizations collaborate because they derive mutual benefits”

Propositions will tell you where to look for relevant evidence

Example: Define and ascertain the specific benefits to each organization

Some studies may not have propositions – this implies a topic of “exploration”

Note: Even exploratory studies should have both clearly-stated purposes and clearly-stated criteria for success


9

17



Part 3: Unit of Analysis

The unit of analysis defines what a “case” is in a case study

Example: a unit of analysis (case) may be an individual, and the case study may be the life history of that person

Other units of analysis include decisions, social programs, processes, changes

Note: It is important to clarify the definition of these cases as they may be subjective, e.g. the beginning and end points of a process

What unit of analysis to use generally depends on the primary research questionsOnce defined, the unit of analysis can still be changed if desired, e.g. as a result of discoveries based on dataTo compare results with previous studies (or allow others to compare results with yours), try to select a unit of analysis that is or can be used by others

18



Examples of Units of Analysis

For a study of how software immigrants naturalizeIndividualsDevelopment teamOrganization

For a study of pair programmingProgramming episodePairs of programmersDevelopment teamOrganization

For a study of software evolutionModification reportFileSystemReleaseStable release


10

19



Part 4: Linking Logic

Logic or reasoning to link data to propositions

One of the least well developed components in case studies

Many ways to perform this, but none as precisely defined as the treatment/subject approach used in experiments

One possibility is pattern matchingDescribe several potential patterns, then compare the case study data to the patterns and see which one is closer

20



Part 5: Interpretation Criteria

Need criteria for interpreting a study’s findings

Also a relatively undeveloped component in case studies

Statistical tests not possible when only single data points are captured (as is the case with single-case studies)

Currently there is no precise way of setting the criteria for interpreting these types of findings


11

21



Generalizing from Case Study to Theory

“The appropriately developed theory is also at the level at which generalization of the case study results will occur”

Theory for case studies is characterized as analytic generalization and is contrasted with another way of generalizing results known as statistical generalization

Understanding the difference between these two types of generalization is important

22



Analytical and Statistical Generalization

Figure 2.2 Making Inferences: Two Levels


12

23



Statistical Generalization

Making an inference about a population on the basis of empirical data collected about a sampleThis method of generalization is commonly recognized because research investigators have quantitative formulas characterizing generalizations that can be made

Examples: significance, confidence, size of the effect, power of test

Using this as a method of generalizing the results of a case study is a “fatal flaw”, since cases are not sampling units, nor should they be chosen for this reasonStatistical generalizations are considered a Level One Inference

24



Analytical Generalization

Previously developed theory is used as a template with which to compare the empirical results of the case studyIf 2 or more cases support the same theory, replication may be claimedResults may be considered more “potent” if 2 or more cases support the same theory but don’t support the same rival theoryAnalytical generalizations are considered a Level 2 InferenceAim toward analytical generalization in doing case studies

Avoid thinking in terms of samples when doing case studies


13

25








26



How can I evaluate a case study?

Using the same criteria for other empirical research

Construct ValidityConcepts being studied are operationalized and measured correctly

Internal ValidityEstablish a causal relationship and distinguish spurious relationships

External ValidityEstablish the domain to which a study’s findings can be generalized

Experimental ReliabilityDemonstrate that the study can be repeated with the same results


14

27



2. Designing a Case Study

28




BackgroundScientific MethodRole of TheoryEmpirical ApproachesConcepts and Terminology

Designing a Case StudyPlanningData CollectionData Analysis

EvaluationValidity and Threats to Validity


15

29



Scientific Method

No single “official” scientific methodhttp://www.sit.wisc.edu/~crusbult/methods/science.htm

However, there are commonalities

WorldTheory

Observation

Validation

30



High School Science Version

1. Observe some aspect of the universe.

2. Invent a tentative description, called a hypothesis,that is consistent with what you have observed.

3. Use the hypothesis to make predictions.

4. Test those predictions by experiments or further observations and modify the hypothesis in the light of your results.

5. Repeat steps 3 and 4 until there are no discrepancies between theory and experiment and/or observation.


16

31



Some Characteristics of Science

Explanations are based on observationsA way of thinkingRelationships are perceptible in a way that has to make sense given accepted truths

Creativity is as important as in artHypotheses, experimental designsSearch for elegance, simplicity

32



Some Definitions

A model is an abstract representation of a phenomenon or set of related phenomena

Some details included, others excluded

A theory is a set of statements that provides an explanation of a set of phenomena

A hypothesis is a testable statement that is derived from a theory

A hypothesis is not a theory!

In software engineering, there are few capital-T theories

Many small-t theories, philosophers call these folk theories


17

33



Science and Theory

Science seeks to improve our understanding of the world.

Theories lie at the heart of what it means to do science.

Production of generalizable knowledgeScientific method <-> Research Methodology <-> Proper Contributions for a Discipline

Note to self: theory provides orientation for data collection

34



Definition of Theory

A set of statements that provide a causal explanation of a set of phenomena

Logically completeInternally consistentFalsifiable

Components: concepts, relations, causal inferences

More than straight description

Example: Conway’s Law- structure of software reflects the structure of the team that builds it


18

35



Empirical Approach

Solution Creation

ValidationQuestionFormulation

Research Methodology

36



Empirical Approaches

Three approachesDescriptiveRelationalExperimental


19

37




DescriptiveGoal: careful mapping out a situation in order to describe what is happeningNecessary first step in any research

Provides the basis or cornerstoneProvides the what

Rarely sufficient – often want to know why or howBut often provides the broad working hypothesis

38




RelationalNeed at least two sets of observations so that some phenomenon can be related to each otherTwo or more variables are measured and related to each otherCoordinated observations -> quantitative degree of correlation Not sufficient to explain why there is a correlation


20

39




ExperimentalFocus on identification of causes, what leads to whatWant “X is responsible for Y”, not “X is related to Y”Experimental group versus control groupWatch out for problems

40



Concepts and Terminology

Predictor and criterion variablesExample: assessment predicting performance

Construct – an abstract idea that is used as an explanatory concept

Example: need for social approval

Reliable measurement – consistency

Validity in various forms


21

41



Concepts and Terminology

Aspects of empirical reasoningEmpirical principles: accepted truths justified on the basis of observationsDeductive-statistical reasoning – universal lawsInductive-statistical reasoning – probabilistic assertions

They deal with uncertaintyThey are not absolute, invariant rules of nature

Behavioral sciences are not sufficient to determine exactitude

Human values and individual states of mindUnique nature of the situation which is usually not staticHistorical and social factors

42



Validity

In software engineering, we worry about various issues:E-Type systems:

Usefulness – is it doing what is neededEmbodying important required characteristics – is it doing it in an acceptable or appropriate way

S-Type programs:correctness of functionality – is it doing what it is supposed to doEmbodying important required characteristics – are the structures consistent with the way it should perform

In empirical work, worried about similar kinds of thingsAre we testing what we mean to testAre the results due solely to our manipulationsAre our conclusions justifiedWhat are the results applicable to

The questions correspond to different validity concernsThe logic of demonstrating causal connectionsThe logic of evidence


22

43







44



How Many Cases?

Number of literal replications Its a discretionary, judgmental choice that depends on the certainty you want to have about your multiple-case resultsAs with statistical significance measures, there is greater certainty with a larger number of cases2 or 3 may be sufficient if they all have very different rival theories and the degree of certainty required is not high5, 6, or more may be needed for higher degree of certainty

Number of theoretical replicationsConsider the complexity of the realm of external validityIf you are uncertain about effects of external conditions on your case study results, you may want to include more cases to address the impacts of these conditions in your studyIf external conditions are not thought to produce much variation in the phenomenon being studied, a smaller number of theoretical replications may be used


23

45



Case Study Designs

4 types of designs based on a 2x2 matrix

Type 1 - single-case (holistic) designs Type 2 – single-case (embedded) designsType 3 – multiple-case (holistic) designsType 4 – multiple-case (embedded) designs

Figure 2.4 Basic Types of Designs for Case Studies (page 40)

46



Rationale for Single-Case Designs

As you might guess, a single-case design uses a single case study to address the research questions5 reasons to use a single-case design

The single case represents the critical case in testing a well-formulated theory

Example: the case meets all of the conditions for testing the theory thoroughly

The single case represents an extreme or unique caseExample: a case with a rare disorder

The single case is the representative or typical case, i.e. informs about common situations/experiencesThe single case is revelatory – it is a unique opportunity to study something that was previously inaccessible to observationThe single case is longitudinal – it studies the same single case at two or more different points in time


24

47



Holistic vs. Embedded Case Studies

The same case study can involve more than one unit of analysis if attention is given to subunit(s) within the case – this is called an embedded case study

Example: a case study about a single organization may have conclusions about the people (subunits) within the organization

If the case study examines only the global nature of one unit of analysis (not any subunits), it is a holisticdesign

Example: a case study about an organization

48



Holistic Designs

StrengthsAdvantageous when no logical subunits can be defined Good when the relevant theory underlying the case study is holistic in nature

WeaknessesCan lead to abstract studies with no clear measures or dataHarder to detect when the case study is shifting focus away from initial research questions


25

49



Embedded Designs

StrengthsIntroduces higher sensitivity to “slippage” from the original research questions

WeaknessesCan lead to focusing only on the subunit (i.e. a multiple-case study of the subunits) and failure to return to the larger unit of analysis

50



Multiple-Case Designs

If the same study contains more than a single case, it is a multiple-case design

AdvantagesEvidence is considered more compellingOverall study is therefore regarded as more robust

DisadvantagesRationale for single-case designs usually cannot be satisfied by multiple casesCan require extensive resources and time


26

51



Replication in Multiple-Case Studies

When using multiple-case studies, each case must be carefully selected so that it either:

Predicts similar results (literal replication)Predicts contrasting results but for predictable reasons (theoretical replication)

If all cases turn out as predicted, there is compelling support for the initial propositionsOtherwise the propositions must be revised and retested with another set of casesWith replication procedures, a theoretical framework must be developed that states the conditions under which a particular phenomenon is likely to be found (a literal replication) and the conditions when it is not likely to be found (a theoretical replication)

This framework is used to generalize to new cases

52



Replication Logic vs. Sampling Logic

Consider multiple-cases analogous to multiple experiments (NOT analogous to multiple subjects within an experiment or multiple respondents in a survey)This replication logic used in multiple-case studies must be distinguished from the sampling logic commonly used in surveys

Sampling logic requires defining a pool of potential respondents, then selecting a subset from that pool using a statistical procedureResponses from the subset are supposed to accurately reflect theresponses of the entire poolThis procedure is used to determine the prevalence or frequency of a particular phenomenon

Sampling logic is not for use with case studiesCase studies are not the best method for assessing the prevalence of phenomenonCase studies would have to cover both the phenomenon of interestand its context, yielding a larger number of potential variables, and thus requiring an impossible number of cases Sampling logic simply cannot be used for all types of empirical investigations


27

53



Replication Approach for Multiple-Case Studies

Figure 2.5 Case Study Method (page 50)

54



Rationale for Multiple-Case Designs

Multiple-case designs are useful when literal or theoretical replications would provide valuable information for the study

More results that back your theory typically adds more credibility to your case study


28

55



Multiple-Case Designs: Holistic or Embedded

A multiple-case study can consist of multiple holistic cases or multiple embedded cases, depending on the type of phenomenon being studied and the research questionsNote there is no mixing of embedded and holistic cases in the same multiple-case studyIt is also important to note that for embedded studies, subunit data is NOT pooled across the subunits, but is used to draw conclusions for the subunit’s case only

56



Selecting Case Study Designs – Single/Multiple?

If you have a choice and the resources, multiple-case designs are preferred

Analytic conclusions independently arising from two cases will be more powerful than from a single caseThe differences in context of multiple cases that have common conclusions provide for expanded generalizability of findingsIf two deliberately contrasting cases are selected and findings support the hypothesized contrast, the results represent theoretical replication and strengthen external validity

Single-case studies are often criticized due to fears about uniqueness surrounding the case

Criticisms may turn to skepticism about your ability to do empirical work beyond a single-case study If you choose single-case design, be prepared to make an extremely strong argument justifying your choice for the case


29

57



Selecting Case Study Designs – Closed/Flexible?

A case study’s design can be modified by new information or discovery during data collection

If you modify your design, be careful to understand the nature of the alteration:

Are you merely selecting different cases, or are you also changing the original theoretical concerns and objectives?Flexibility in design does not allow for lack of rigor in design

58



Collecting the Evidence

Six Sources of Evidence

Three Principles of Data Collection


30

59



Six Sources of Evidence

Documentation

Archival Records

Interviews

Direct Observation

Participant-observation

Physical Artifacts

60



Documentation

Letters, memos, and other written communication

Agendas, announcements, meeting minutes, reports of events

Administrative documents – proposals, progress reports, summaries and records

Formal studies or evaluations of the same site

Newspaper clippings, articles in media or newsletters

Example: Classifying modification reports as adaptive, perfective or corrective based on documentation

Audris Mockus, Lawrence G. Votta: Identifying Reasons for Software Changes using Historic Databases. ICSM2000: pp. 120-130


31

61



Archival Records

Service records- clients served over a period of time

Organizational records- org. charts and budgets

Maps and charts- layouts

Lists of names and relevant articles

Survey data – census records

Personal records – diaries, calendars, telephone lists

Example: Study of parallel changes to source code was based on revision control logs

Dewayne E. Perry, Harvey P. Siy, Lawrence G. Votta: Parallel changes in large-scale software development: an observational case study. ACM Trans. Softw. Eng. Methodol. 10(3): 308-337 (2001)

62



Interviews

Open-ended interviews- address facts of a matter, opinions about an event

Focused interview- short period of time typically an hour, same approach as open-ended.

Formal survey- produces quantifiable data

Example: Used semi-structured interviews to understand the effect of distance on coordination in teams

Rebecca E. Grinter, James D. Herbsleb, Dewayne E. Perry: The geography of coordination: dealing with distance in R&D work. GROUP 1999: pp. 306-315


32

63



Direct Observation

Field visits- creates opportunity for direct observation

Photographs of site Need permission in order to proceed

Can be used to calibrate self-reportsExample: Informal, impromptu interactions

Example: Followed software developers around to characterize how they spend their time

Dewayne E. Perry, Nancy A. Staudenmayer, Lawrence G. Votta: People, Organizations, and Process Improvement. IEEE Software 11(4): 36-45 (1994)

64



Participant-observation

Not a passive observer, in contrast to direct observation, but actually participate in setting

Useful for large organizations or small groups

Observe participant bias “inside” when actively participating during the case study

Example: Seaman participated in 23 code inspections over period of five months at NASA/Goddard Space Flight Center’s Flight Dynamics Division

Carolyn B. Seaman, Victor R. Basili: Communication and Organization: An Empirical Study of Discussion in Inspection Meetings. IEEE Trans. Software Eng. 24(7): 559-572 (1998)


33

65



Physical Artifacts

Technological tool, instrument, or device

Artifacts can be collected or observed as part a field visit

Works of art or types of physical evidence

Example: Diachronic study of art records to determine whether right-handedness was a recent or old trait

Two rival hypotheses: Physiological predisposition vsSocial/environmental pressureTested by counting unimanual tool usage in art representations1200 examples from 1500 BC to 1950, world sources92.6% used right handGeo/historical distribution uniformly highSeems to support physiological interpretation that right-handedness is an age-old trait

66



Six Sources of Evidence: Strengths and Weaknesses

Yin, R.K. (2002). Case Study Research: Design and Methods (3rd ed.). Thousands Oaks, CA: Sage Publications, p. 86.

Source of E vidence S trengths W eaknesses D ocum entation Stable – can be reviewed

repeatedly Unobtrusive – not created as a result of the case study Exact – contains exact nam es, references, and details of an event Broad coverage – long span of tim e, m any events, and m any settings

Retrievability – can be low B iased selectivity, if collection is incom plete Reporting bias – reflects (unknown) bias of author Access – m ay be deliberately blocked

A rchival R ecords {sam e as above for docum entation}

Precise and quantitative

{sam e as above for docum entation}

Accessibility due to privacy reasons

In terview s Targeted – focuses directly on case study topic Insightful – provides perceived causal inferences

B ias due to poorly constructed questions Response bias Inaccuracies due to poor recall Reflexivity – interviewee gives what interview wants to hear


34

67



Strengths and Weaknesses Cont’d

Yin, R.K. (2002). Case Study Research: Design and Methods (3rd ed.). Thousands Oaks, CA: Sage Publications, p. 86.

Source of Evidence Strengths Weaknesses Direct Observations

Reality – covers events in real time Contextual – covers content of event

Time consuming Selectivity – unless broad coverage Reflexivity – event may proceed differently because it is being observed Cost- hours needed by human observers

Participant Observations

{same as above for direct observation}

Insightful into interpersonal behavior and motives

{same as above for direct observation}

Bias due to investigator’s manipulation of events

Physical Artifacts Insightful into cultural features Insightful into technical operations

Selectivity Availability

68



Principles of Data Collection

Use Multiple Sources of Evidence

Create a Case Study Database

Maintain a Chain of Evidence

These principles can be applied to all six data collection methods


35

69



Multiple Sources of Evidence

Triangulation (four types for evaluation)Data sources (data triangulation)Different evaluators (investigator triangulation)Perspective to same data (theory triangulation)Methods (methodological triangulation)

Encourages evidence collection from more then one source in order to show same facts and/or findings

Example: Different approaches were used collect data about how developers spend their time.

Dewayne E. Perry, Nancy A. Staudenmayer, Lawrence G. Votta: People, Organizations, and Process Improvement. IEEE Software 11(4): 36-45 (1994)

Collected cross-sectional and direct observation dataMarc G. Bradac, Dewayne E. Perry, Lawrence G. Votta: Prototyping a Process Monitoring Experiment. IEEE Trans. Software Eng. 20(10): 774-784 (1994)

Collected longitudinal data

70



Multiple Sources of Evidence Convergence of Evidence (Figure 4.2)

FACT

Documents Archival Records

Open-endedInterviews

Focus InterviewsStructured Interviewsand Surveys

Observations(direct and participant)


36

71



Case Study Database

Case study notesFrom interviews, documents, etc.Categorized, complete

Case study documentsAnnotated bibliography of the documents- facilitates storage, retrieval, help later investigators share the database

Tabular materials Collected or created and can be stored

NarrativesSupported answers to the questionsConnect pertinent issues

The database performs a formal assembly of evidence

72



Chain of Evidence

Forms explicit links betweenQuestions askedData collected Conclusion drawn


37

73



Chain of Evidence Maintaining Chain of Evidence (Figure 4.3)

Citations to Specific Evidentiary Sources in the Case Study D.B.

Case Study Report

Case Study Database

Case Study Protocol (linking questions to protocol topics)

Case Study Question

74



Data Analysis

Analytic Strategies

3 general strategies

5 specific analytic techniques

Criteria for high quality analysis


38

75



Characteristics of Case Study Analysis

Data analysis consists of examining, categorizing, tabulating, testing and recombining both quantitative and qualitative evidence to address the initial propositions of a study

Analyzing case study evidence is difficult because strategies and techniques have not been well defined

Every case study should have a general analytic strategy to define priorities for what to analyze and why

76



Criteria for High Quality Analysis

Present all the evidence

Develop rival hypotheses

Address all major rival interpretations

Address most significant aspect of the case study

Use prior or expert knowledge


39

77



Objectives of Analytical Study

Produce high quality analyses

Present all evidence and separate them from any interpretation

Explore alternative interpretations

78



Needs for Analytic Strategies

Investigations on how the evidence is to be analyzed easily become stalled

Analytic tools can only be helpful if the investigators know what to look for

Analytic strategies are needed to address the entire case study since verbatim and documentary texts are usually the initial phase


40

79



Benefits of Analytic Strategies

Put the evidence in preliminary order and treat the evidence fairly

Prevent false starts

Save time

Produce compelling analytic conclusions

Rule out alternative interpretations

Help investigators use tools and make manipulations effectively

80



Three General Strategies

1. Relying on Theoretical Propositions

2. Thinking about Rival Explanations

3. Developing a Case Description


41

81



GS 1 - Relying on Theoretical Propositions

Shapes the data collection plan and gives priorities to the relevant analytic strategies

Helps to focus attention on certain data and to ignore other useless data

Helps to organize the entire case study and define alternative explanations to be examined

82



GS 2 - Thinking About Rival Explanations

Defines and tests rival explanations

Relates to theoretical propositions, which contain rival hypotheses

Attempts to collect evidence about other possible influences

The more rivals the analysis addresses and rejects, the more confidence can be placed in the findings


42

83



GS 3 - Developing a Case Description

Serves as an alternative when theoretical proposition and rival explanation are not applicable

Identifiesan embedded unit of analysis an overall pattern of complexity to explain why implementation had failed

84



Five Specific Analytic Techniques

1. Pattern Matching

2. Explanation Building

3. Time-Series Analysis

4. Logic Models

5. Cross-Case Synthesis

Note: They are intended to deal with problems of developing internal and external validity in doing case studies


43

85



AT 1 - Pattern Matching

Pattern matching compares an empirically based pattern with a predicted one

If the patterns coincide, the results can strengthen the internal validity of the case study

Types of pattern matching:1. Nonequivalent dependent variables as a pattern2. Rival explanations as patterns3. Simpler patterns

86



PM 1 - Nonequivalent dependent variables

Quasi-experiment may have multiple dependent variables (variety of outcomes)

If, for each outcome, the initially predicted values have been found, and at the same time alternative “patterns” of predicted values (including those deriving from methodological artifacts or threats to validity) have not been found, strong causal inferences can be made


44

87



PM 2 - Rival explanations as patterns

Each case has certain type of outcome, and the investigation has to be focused on how and why this outcome occurred

This analysis requires the development of rival theoretical propositions, articulated in operational terms

Each rival explanation involves a pattern of independent variables that is mutually exclusive: If one explanation is to be valid, the others cannot be

88



PM 3 - Simpler Patterns

There may be only 2 different dependent (or independent) variables, pattern matching is possible as long as a different pattern has been stipulated for these 2 variables

The fewer the variables, the more dramatic the different patterns will have to allow any comparisons of their differences


45

89



AT 2 - Explanation Building

Analyzes the case study data by building an explanation about the case

Stipulates a presumed set of causal links, which are similar to the independent variables in the use of rival explanations

Has mostly occurred in narrative form

May lead to starting a cross-case analysis, not just an analysis of each individual case

Disadvantage: may drift away from original focus

90



Explanation Building

Series of iterations in building explanation1.Making initial theoretical statement2.Comparing the findings of the initial case against such a

statement3.Revising the statement4.Comparing other details of the case against the revision5.Comparing the revisions to the facts of 2nd, 3rd or more

cases6.Repeating the process if needed


46

91



AT 3 - Time Series Analysis

The objective of time series analysis is to examine relevant “how” and “why” questions about the relationship of events over timeTime series analysis can follow intricate patternsThe more intricate the pattern, the firmer the foundation for conclusions of the case study

Three types of Time Series Analyses:Simple Time SeriesComplex Time SeriesChronologies

92



TS 1 - Simple Time Series

Trace changes over time

Single variable only, so statistical analysis of data is possible

Match between a trend of data points compared tosignificant trend specified before investigationrival trend specified earlierany other trend based on some artifact or threat to internal validity


47

93



TS 2 - Complex Time Series

Contain multiple set of variables (mixed patterns) which are relevant to the case study

Each variable is predicted to have different pattern over time

Create greater problems for data collection, but lead to elaborate trend that strengthens the analysis

Any match of a predicted with an actual time series will produce strong evidence for an initial theoretical proposition

94



TS 3 - Chronologies

Trace events over time

Sequence of a cause and effect cannot be inverted

Some events must be followed by other events on a contingency basis after an interval of time

Cover many different types of variables

Goal is to compare chronology with that predicted by the explanatory theory


48

95



AT 4 - Logic Models

Stipulate a complex chain of events over time

Events are staged in repeated cause-effect-cause-effect patterns

Match empirically observed events to theoretically predicted events

Four types of logic models:Individual-Level Logic ModelFirm or Organizational-Level Logic ModelAn alternative configuration for an Organizational-Level Logic ModelProgram-Level Logic Model

96



AT 4 - Logic Models

A) Individual-level logic modelAssumes the case study is about an individual person

B) Firm or organizational-level logic modelTraces events taking place in an individual organization

C) An alternative configuration for an organizational-level logic model

Encounters dynamic events that are not progressing linearlyChanges may reverse course and not just progress in one direction (Transformation and reforming)

D) Program-level logic modelAnalyzes data from different case studies by collecting data on rival explanations


49

97



AT 5 - Cross-Case Synthesis

Case study consists of at least 2 cases

Using multiple case studies will Treat each individual case study as a separate studyHave to create word tables that display data from individual cases according to some uniform frameworkExamine word tables for cross-case patternsRely strongly on argumentative interpretation, not numeric propertiesBe directly analogous to cross-experiment interpretations

98








50

99



Validity

4 primary types of validityConstruct ValidityInternal ValidityExternal Validity

100



Construct Validity

Are we measuring what we intend to measureAkin to the requirements problem: are we building the right systemIf we don’t get this right, the rest doesn’t matter

Constructs: abstract conceptsTheoretical constructionsMust be operationalized in the experiment

Necessary condition for successful experiment

Divide construct validity into three parts:Intentional ValidityRepresentation ValidityObservation Validity


51

101



Construct Validity

Intentional ValidityDo the constructs we chose adequately represent what we intend to studyAkin to the requirements problem where our intent is fair scheduling but our requirement is FIFOAre our constructs specific enough?Do they focus in the right direction?Eg, is it intelligence or cunningness

102



Construct Validity

Representation ValidityHow well do the constructs or abstractions translate into observable measuresTwo primary questions:

Do the sub-constructs properly define the constructsDo the observations properly interpret, measure or test the constructs

2 ways to argue for representation validityFace validity

Claim: on the face of it, seems like a good translationVery weak argumentStrengthened by consensus of experts

Content validityCheck the operationalization against the domain for the constructThe extent to which the tests measure the content of the domain being tested - ie, cover the domainThe more it covers the relevant areas, the more content valid

Both are nonquantitative judgments


52

103



Construct Validity

Observation ValidityHow good are the measures themselvesDifferent aspects illuminated by

Predictive validityCriterion validityConcurrent validityConvergent validityDiscriminant validity

Predictive ValidityObserved measure predicts what it should predict and nothing elseE.g., college aptitude tests are assessed for their ability to predict success in college

Criterion ValidityDegree to which the results of a measure agree with those of an independent standardEg, for college aptitude, GPA or successful first year

104



Construct Validity

Observation Validity – cont.

Concurrent ValidityThe observed measure correlates highly with an established set of measuresEg, shorter forms of tests against longer forms

Convergent ValidityObserved measure correlates highly with other observable measures for the same constructUtility is not that it duplicates a measure but is a new way of distinguishing a particular trait while correlating with similar measures

Discriminant ValidityThe observable measure distinguishes between two groups that differ on the trait in questionLack of divergence argues for poor discriminant validity


53

105



Internal Validity

Are the values of the dependent variables solely the result of the manipulations of the independent variables?

Have we ruled out rival hypotheses?

Have we eliminated confounding variables?Participant variablesExperimenter variablesStimulus, procedural and situational variablesInstrumentationNuisance variables

Confounding sources of internal invalidityH: History

events happen during the study (eg, 9/11)M: Maturation

older/wiser/better between during studyI: Instrumentation

change due to observation/measurement instrumentsS: Selection

differing nature of participantseffects of choosing participants

106



Internal Validity

Demonstrating that certain conditions are in fact the cause of other conditions, i.e. conditions not mentioned or studied are not the actual cause

Example: if a study concludes that there is a causal relationship between X and Y without knowing some third factor Z may have actually caused Y, the research design has failed to deal with threats to internal validity

Internal validity applies to explanatory and causal studies only, not to descriptive or exploratory studiesIt is important to challenge any inferences you make during your study as any incorrect inferences may detract from internal validity


54

107



External Validity

Two positionsThe generalizability of the causal relationship beyond that studied/observed

Eg, do studies of very large reliable real-time systems generalize to small .COM companies

The extent to which the results support the claims of generalizability

Eg, do the studies of 5ESS support the claim that they are representative of real-time ultra reliable systems

108



External Validity

Establishing the domain to which a study’s findings can be generalized

i.e. are the study’s findings generalizable beyond the immediate case study?

Case studies have been criticized for offering a poor basis for generalization since only single cases are studied

This is contrasting case studies (which rely on analytical generalization) with survey research (which relies on statistical generalization), which is an invalid comparison

Generalization of the theory must be tested by replicating the findings over several different cases.


55

109



Reliability

Demonstrating that the operations of a study can be repeated with the same results

Note: the repetition of the study should occur on the same case, not “replicating” the results on a different case

“The goal of reliability is to minimize the errors and biases in a study”

A prerequisite for reliability testing is documented procedures for the case study

A good guideline is to perform research so that an auditor could follow the documented procedures and arrive at the same results

110



Tactics to Address Quality in Case Study Design

Figure 2.2 Case Study Tactics for the Four Design Tests (page 34)


56

111



3. Publishing Case Studies

112




Targeting Case Study Reports

Composition Styles for Case Studies

General Guidelines from Software Engineering

Sample Paper Outlines


57

113



Targeting Case Study Reports

Case studies tends to have a more diverse set of potential audiences than other research methods

Each audience has different needs, no single report will serve all audiences simultaneouslyTherefore, may need more than one version of a case study report

Case study report itself is a significant communication deviceCase study can communicate research-based information about a phenomenon to a variety of non-specialistsPractitioners like case studies, so context is important

Orient the case study report to an audienceThe preferences of the potential audience should dictate the form of your case study reportGreatest error is to compose a report from an egocentric perspectiveTherefore, one must identify the audience before writing a case study reportRecommendation: examine previous case study reports that have successfully communicated with the identified audience

114



Formats for Written Case Study Reports

Classic single-case study- a single narrative is used to describe and analyze the case

Multiple-case version of this classic single case- individual cases are presented in separate chapters- also contain chapters that contain cross-case analysis

Non-traditional narrative (single or multiple)- use question-and-answer format

Cross-case analysis (multiple-case studies only)- entire report consist of cross-case analysis- each chapter would be devoted to a separate cross-case issue

Note: Format should be identified during the design phase of case study


58

115



Sequences of Studies

The best empirical studies are performed as part of a sequence

Each going deeper or shedding light on a different aspect of a problemCan deploy different tactics, strategies, methods

Rationales to use case study as part of a larger, multimethod study1. To determine whether converging evidence might be

obtained even though different methods has been used2. After analyzing data collected by other methods, case

study might be able to illustrate an individual case in greater depth

3. Case study may be used to elucidate some underlying process which another method is used to define the prevalence or frequency of such processes

116



Composition Structures for Case Studies

1. Linear-Analytic StructuresStandard approach

2. Comparative StructuresEntire study is repeated two or more times

3. Chronological StructuresEvidence are presented in chronological order

4. Theory building StructuresEach chapter reveal a new part of a theoretical argument

5. “Suspense” StructuresThe outcome of the case study is presented in the initial chapter, then followed by the “suspenseful” explanation of the outcome

6. Unsequenced StructuresThe sequence of sections or chapters assumes no particular importanceWhen using this structure, the investigator should make sure that a complete description of the case is presented. Otherwise, he/she may be accused of being biased


59

117



Issues in Reporting

1. When and How to Start Composing?Start composing early in the analytic processBibliography, methodological and descriptive data about the cases being studied are the sections which could be written early in the process

2. Case Identities: Real or Anonymous?Anonymity issue can be at two levels: entire case and individualperson within a caseMost desirable option is to disclose the identities of both the case and individualsAnonymity is necessary when:

Using the real name will cause harm to the participantsThe case report may affect the subsequent action of those that are studied

Compromises that can be made:Hide individual identity but identify the caseName the individuals but avoid attributing any particular point of view or comment to a single individualThe publish report is limited to the aggregated evidence

Only if these compromises are impossible then the investigator should consider making the entire case study and the informants anonymous

118



Issues in Reporting

3. The Review of the Draft Case Study: A Validating Procedure

There should be a draft report and it should be reviewed not just by peers, but also by the participants and informants in the caseThe reviewers may disagree with the conclusion and interpretations, but not the actual facts of the caseThis process increases the construct validity of the study and reduced the likelihood of falsely reporting an event


60

119



General Guidelines from SEBarbara A. Kitchenham, Shari Lawrence Pfleeger, Lesley M. Pickard, Peter W. Jones, David C. Hoaglin, Khaled El

Emam, and Jarrett Rosenberg, “Preliminary Guidelines for Empirical Research in Software Engineering,” IEEE Transactions on Software Engineering, Vol. 28, No 8, August 2002.

Experimental Context

Experimental Design

Conducing the Experiment and Data Collection

Analysis

Presentation of Results

Interpretation of Results

120



Sample Paper Outline 1

1. Introduction

2. The Traditional Inspection Approach2.1 Software Inspection Basics2.2 Inspection Challenges

2.2.1 The Defect Detection Activity…

2.2.2 The Defect Collection Activity…

3. Changes to the Inspection Implementation at DaimlerChrysler3.1 Case Study Environment3.2 Defect Detection Approach3.3 Defect Collection Approach

4. Analysis Approach4.1 Some Misconceptions in

Inspection Data Analysis4.2 A Model for Explaining the

Number of Defects Detected4.3 Measurement4.4 Analysis Approach

5. Results5.1 Descriptive Statistics5.2 Correlation and Regression

Analysis5.3 Path Analysis Results

6. Threats to Validity6.1 Threats to Internal Validity6.2 Threats to External Validity

7. Conclusion

Oliver Laitenberger, Thomas Beil, and Thilo Schwinn, “An Industrial Case Study to Examine a Non-Traditional Inspection Implementation for Requirements Specifications,” Empirical Software Engineering: An International Journal, vol. 7, no. 4, pp. 345-374, 2002.


61

121




1. Introduction

2. Method2.1 Research Setting2.2 Data Collection2.3 Data Analysis

3. Results3.1 Mentoring

3.1.1 Evidence3.1.2 Implications

3.2 Difficulties Outside of the Software System3.2.1 Evidence3.2.2 Implications

3.3 First Assignments3.3.1 Evidence3.3.2 Implications

3.4 Predictors of Job Fit3.4.1 Evidence3.4.2 Implications

4. Applications of the Patterns

5. Conclusions

Appendix A: Question Sets

Appendix B: Variables Used in Analysis

Susan Elliott Sim and Richard C. Holt, “The Ramp-Up Problem in Software Projects: A Case Study of How Software Immigrants Naturalize,” presented at Twentieth International Conference on Software Engineering, Kyoto, Japan, pp. 361-370, 19-25 April, 1998.

122




1. Introduction

2. Related Work2.1 Configuration Management2.2 Program Analysis2.3 Build Coordination2.4 Empirical Evaluation

3. Study Context3.1 The System Under Study3.2 The 5ESS Change

Management Process

4. Data and Analysis4.1 Levels of Parallel

Development4.2 Effects of Parallel

Development on a File4.3 Interfering Changes4.4 Multilevel Analysis of

Parallel Development4.5 Parallel Versions

5. Validity

6. Summary and Evaluation6.1 Study Summary6.2 Evaluation of Current

Support6.3 Future Directions

Dewayne E. Perry, Harvey P. Siy, and Lawrence G. Votta, “Parallel Changes in Large Scale Software Development: An Observational Case Study,” presented at Twentieth International Conference on Software Engineering, pp. 251-260, 19-25 April 1998.


62

123



4. Reviewing Case Studies

124




Characteristics of Exemplary Case Studies


63

125



What Makes an Exemplary Case Study?

The exemplary case study goes beyond the methodological procedures

Mastering the techniques does not guarantee an exemplary case study

126



Characteristics of an Exemplary Case Study

1. The Case Study Must Be SignificantThe case should be unusual and of general public interestThe issue are nationally important, either in theory or practical termsPrior to selecting a case study, the contribution should be described in detail assuming that the intended case study were to be completed successfully

2. The Case Study Must be “Complete”Completeness can be characterized in at least three ways:

The boundaries of the case are given explicit attentionExhaustive effort is spent on collecting all the relevant evidenceThe case study was not ended because of nonresearchconstraints


64

127



Characteristics of an Exemplary Case Study

3. The Case Study Must Consider Alternative PerspectivesThe case study should include consideration of rival propositions and the analysis of the evidence in terms of such rivalsThis can avoid the appearance of a one-sided case

4. The Case Study Must Display Sufficient EvidenceThe report should include the most relevant evidence so the reader can reach an independent judgment regarding the merits of the analysisThe evidence should be able to convince the reader that the investigator “knows” his or her subjectThe investigator should also show the validity of the evidence being presented

5. The Case Study Must Be Composed in an Engaging MannerA written case study report should be able to entices the reader to continue reading

128



Case Study as a Research Method

The case study is a distinct research method with its own research designs

It is not a subset or variant of research designs used for other strategies (such as experiments)

ScientificSynergistic relationship between theory and dataStarting a case study requires a theoretical orientation, which drives data collection

Useful for answering “how” and “why” questionsIn contrast to who, what, when, how many, how muchHow, why = explanatory, descriptive

Does not require control over eventsMore observational

Focus on contemporary eventsLess historical
