theories in empirical software engineering

TheoriesinEmpiricalSoftwareEngineering

RoelWieringa

Sidekicks:DanielMéndezLutzPrechelt

21October2015 IASESE 1

Whoarewe?Roel WieringaUniversityofTwente,Germany

http://wwwhome.ewi.utwente.nl/~roelw/


LutzPrechelt

FUBerlinhttp://www.mi.fu-berlin.de/w/Main/LutzPrechelt

DanielMéndez

TUMünchenhttp://www4.in.tum.de/~mendezfe/

Whoareyou?

Quickround• Whoare you?• What is your experience inconductingempirical studies?

• What are your expectations?

3

Whatdoyouthink?

Whydoweneedscientifictheoriesinsoftwareengineering?

4

4. Methodology(thestudyofresearchmethods)a. Notionofconceptualframework;statementsaboutthemb. Notionofgeneralization;statementsaboutthem

3. Theory(statementaboutmanyresearchresults)a. Conceptualframeworkb. Generalization

2. Researchquestions(what,how,whenwhere,….,why)aimedatgeneralizableknowledge,researchmethod,andresearchresult

1. Practicedomain:SW,methods,tools,processes(asis/tobe)


Looking atresearchfrom thesky

Generalknowledge isthegoldweareafter

Hardwork to growknowledge

Grassroots

• Everything onthe slidesinthis talk,except the examples,isatlevel4.• Theexamples ontheseslidescontain explicitlevelindications.

• Theseparateexample slidesreportabout researchthat contains 2and 3.• Thereported researchstudiessome aspectof1.

Agenda

Time Topic

09:00– 10:30 OpeningandIntroduction

10:30 – 11:00 Coffeebreak

11:00– 12:30 InferringTheoriesfromData

12:30– 13:30 Lunch

13:30– 15:00 Designing ResearchbasedonTheories

15:00– 15:30 Coffeebreak

15:30– 16:30 Hands-onWorkingSession andQ&A

16:30– 17:00 Wrapup(all)

6

WhatisaScientificTheory


Scientific theories• Atheory isabeliefthat there isapattern inphenomena• Ascientific theory isatheory that– Hassurvived testsagainst experience

• Observation,measurement• Possibly experiment,simulation,trials

– Hassurvived criticism by critical peers• Anonymous peerreview• Publication• Replication


Examples (level3)• Theory ofcognitive dissonance• Theory ofelectromagnetism• TheBalancetheorem insocial networks• Theories X,Y,Z,and Wof(project)management• TechnologyAcceptance Model

• Hannayetal.ASystematic“ReviewofTheoryUseinSoftwareEngineeringExperiments”.IEEETOSEM33(2),February2007

• Limetal.“TheoriesUsedinInformationSystemsResearch:IdentifyingTheoryNetworksinLeadingISJournals”./ICIC2009,paper91.

• Non-examples– Speculations based onimagination rather than fact:Conspiracy theories

about who killed JohnKennedy– Opinions that cannot be refuted:TheDutchlostthe WorldChampionship

because they play likeprimadonnas


Designtheories

• Adesigntheory isascientific theory about anartifact inacontext

• Vriezekolk:What isatheory• Méndez:What isatheory

21October2015 IASESE4 10

TheStructureofTheories


Thestructure ofscientific theories

1. Conceptual framework– Constructs used to express beliefs about patterns inphenomena– E.g.Theconcepts ofbeamforming,ofmulti-agentplanning,ofdata

location compliance.(level3)

2. Generalizations– stated interms oftheseconcepts,that express beliefs about

patterns inphenomena.– E.g.relationbetween angle ofincidence and phase difference,– Statementabout delayreduction onairports.(level3)

• Generalizations haveascope,a.k.a.targetofgeneralization


Thestructure ofdesign theories1. Conceptual framework2. Generalizations

– Artifact specification XContextassumptions→Effects– Effects satisfy arequirement to someextent


1. Architectural structures:Classofsystems,componentswithcapabilities,interactions– E.g.entities,(de)composition, taxonomies,cardinality,events,

processes,procedures,constraints,…(level4)– Useful for case-basedresearch(observational casestudies,case

experiments,simulations,technical actionresearch)– Typically qualitative

2. Statisticalstructures:Population,variableswith probabilitydistributions,relationsamong variables– Useful for sample-based research(surveys,statisticaldifference-

makingexperiments)– Typically quantitative

Two kindsofconceptual structures

21October2015 IASESE14

• Prechelt:What isatheory,the structure oftheories

• Vriezekolk:Thestructure oftheories• Méndez:Thestructure oftheories


TheUseofTheories


Uses ofaconceptual framework• Framing aproblemorartifact:choosingwhich concepts to

use– Usingthe theory ofinfectuous diseases to understand apatient’s

symptoms– Usingconcepts offorce&energyto understand behavior ofamachine– Usingconceptofacoordination gatekeeperto understand a

distributedSEproject(all three examples atlevel1)

• Describe aproblemorspecify an artifact:using theconcepts• Generalize about theproblemorartifact• Analyze aproblemorartifact (i.e.analyze theframework)


Functions ofgeneralizations

• Functions ofgeneralizations– Explanation:explain phenomenaby identifyingcauses,mechanisms orreasons

– Prediction:statewhat will happeninthefuture• Design:use generalizations to justify adesignchoice


• Prechelt:the use oftheories• Vriezekolk:the use oftheories• Méndez:the use oftheories


Usability oftheories• When isadesigntheory

Contextassumptions XArtifact design→Effectsusable by apractitioner?1. He/she iscapable to recognize Contextassumptions2. and to acquire/buildArtifact under constraints ofpractice,3. effects will indeed occur,and4. He/she can observe this,and5. Theywill contribute to stakeholdergoals/satisfy

requirements

• Practitionerhasto asses theriskthat each ofthesefails


• Prechelt:the usability oftheories• Vriezekolk:the usability oftheories• Méndez:the usability oftheories


Agenda

Time Topic




12:30– 13:30 Lunch




16:30– 17:00 Wrapup(all)

22

ScientificInference


Case-basedinference

• Descriptiveinference:Describingobservations• Abductive inference:Providinganexplanation• Analogicinference:Generalizetosimilarcases


Data

Explanations

Observations

Generalizations

Abduction

AnalogyDescription

Proposition(s) to generalize

Scopeofgeneralization

• Architectural explanation mustbe thebasisoftheanalogic generalization;

• Otherwise,weengage inwishful/magical thinking– You haveobserved that some smallcompaniesdid not putacustomerrepresentative on-siteofan agileproject;

– you explain this asaresult oftight resources (level3);– you generalize by analogy that this will happen in(almost)all smallcompanies(level3).


Data

Explanations

Observations

Generalizations

Abduction

AnalogyDescription

Architectural

Architectural

Sample-based inference

• Descriptiveinference:Describesamplestatistics• Statisticalinference:Generalizetopopulationparameters• Abductive inference:Provideanexplanation• Analogicinference:Expandthescopeofatheorybasedonsimilarity


Explanations

Observations

GeneralizationsStatistical inference

AbductionAnalogyDataDescription

• Causal explanations can be supported by sample-baseddesigns(treatmentgroup/controlgroup)

• Generalization from apopulation,to similar populationsmustbe based onarchitectural explanation– Inan experimentwithasampleofstudents you observe adifference between

treatmentgroup and controlgroup;– By randomness you generalize topopulation ofstudents– Your explanation:this difference iscaused by the treatment(level3);– Inturnexplainedby cognitive processes ofstudents (level3);– generalizedby analogy to novicesoftwareengineers(level3).


Explanations

Observations

Generalizations

AbductionAnalogyDataDescription

Statistical inference

Architectural

Causal &Architectural

• Vriezekolk:Inferring theories from data• Méndez:inferring theories from data• Prechelt:Applying/inferring theories to/fromdata


Agenda

Time Topic




12:30– 13:30 Lunch




16:30– 17:00 Wrapup(all)

29

ResearchDesign


Theresearchsetup

• Inexperiments weareinterested inthe effectofthetreatmentonthe OoS– Requires capability to apply treatmentand control

• Inobservational studiesweareinterested inthe structure anddynamics ofthe OoS itself– Only weak supportfor causality


PopulationSample of Objects of

Study

Representsone or more

populationelements

Treatment instruments

Measure-ment

instruments

• Case-baseddesigns– provide architecturalexplanations– generalizebyarchitecturalanalogy

– Nondeterminism across cases is not quantified

• Sample-based designs– Collectsamplestatistics– Infer properties ofdistributionoverpopulation

– May be purely descriptive!– Possibly a causal explanation– To generalize further, need architectural explanation too– Nondeterminsim within the population is quantified, but not

across analogous populations


Fieldversuslab


• If aphenomenoncannot be (re)produced inthe lab,it canonly be investigated inthe field

• Which ofthe followingdesignscan be done inalab?

Case-basedinference Sample-based inferenceNo treatment(observational study)

Observational casestudy Survey

Treatment(experimental study)

Single-case mechanismexperiment,Technicalactionresearch

Statisticaldifference-makingexperiment

E.g. simulation, test of individual OoS Treatment group /

control group designsE.g. test with client, pilot project

• VriezekolkTheresearchsetup• Méndez:Theresearchsetup• Prechelt:Theresearchsetup


Agenda

Time Topic




12:30– 13:30 Lunch




16:30– 17:00 Wrapup(all)

35

Hands-onWorkingSession


Hands-onWorking Session1. What isyour researchquestion?2. Describe aresearchsetupto answer it3. What inferences doyou planto baseonthis setup?

Groups of3• 15:30Each personfirstdrafts aflipchartwith his/heranswers for

own research• 15:45Each groupmembercomments onthe two flipcharts of

others inhis/hergroup,inparticular on:– Arethe answers clear?– Arethe answers defensible?

• 16:30Each personfinalizes (for now)his/herflipchart• 16:31Pasteto the wall.Seewhat you can learn fromother designs.• 16:45Plenary wrap-up


Q&A


Youprobablycan’task anyway,soaskus!

“Namingthepaininrequirementsengineering:AdesignforaglobalfamilyofsurveysandfirstresultsfromGermany”

Méndez&WagnerInformation&Softwaretechnology2015

“TowardsBuildingKnowledgeonCausesofCriticalRequirementsEngineeringProblems”

Kalinowski etalTwenty-SeventhInternationalConferenceonSoftwareEngineeringand

KnowledgeEngineering(SEKE2015)pp.1-6

40

• Internationalon-line surveyofrequirements engineeringprofessionals’opinionabout causes and effects ofREproblems

• Researchquestions– RQ1WhataretheexpectationsonagoodRE?– RQ2HowisREdefined,applied,andcontrolled?– RQ3HowisREcontinuouslyimproved?– RQ4WhichcontemporaryproblemsexistinRE,andwhatimplications

dotheyhave?– RQ5Arethereobservablepatternsofexpectations,statusquo,and

problemsinRE?

• Observational research

41

Whatisatheory

• Theresearchersformulated34hypothesesabout– REimprovement

• Isbeneficial• Ischallenging

– REstandardization• Hamperscreativity• Improvesquality• ….

– Company-specificstandards• ….

42

• Thistheory(consistingof34proposedgeneralizations)istestedagainst– Opinions ofprofessionals,basedontheirexperience– Criticalpeerreviewinthepublicationprocess

• Theopinionsofprofessionalsarethemselvestheoriesbasedonexperience,– butnotsubjectedtosystematictests– nortocriticalpeerreviews

43

Thestructureoftheories

1. Conceptualframework– Requirements,needs,goals,specification,RE

skill,etc.2. Generalizations– Alliftheclaimsaboutsocialmechanismson

previousslides

44

45

customerProjectteam

Requirementsengineer

Product Requirementsspecification

Nosolution approachAgileapproachNoexperienceREconsidered unimportantNoREqualificationNotimeTeamtoo smallDifferentinterestsNodomainknowledge

NotemplatePoor techniquesNocompleteness check

REconsidered unimportantNoREskillsUnclear needsUnrealistic expectationsNoengagementUnclear requirements

Frequentchanges

Poorly defined

Brazilian theory ofsocial mechanisms that leadto incompleterequirements

Artifact:Requirements engineeringprojectContext:softwaredevelopment

46

customerProjectteam

Requirementsengineer

Product Requirementsspecification

Nosolution approachAgileapproachNoexperienceREconsidered unimportantNoREqualificationNotimeTeamtoo smallDifferentinterestsNodomainknowledgeNocontactpersonSolutionorientation

NotemplatePoor techniquesNocompleteness checkNocompanystandard

REconsidered unimportantNoREskillsUnclear needsUnrealistic expectationsNoengagementUnclear requirementsNocontactpersonSolutionorientationDomaincomplexity

Frequentchanges

Poorly defined

Businessdept

conflict

German theory ofsocial mechanisms that leadto incompleterequirements

• Theconceptual structure ofsocial mechanisms inthe previous two slidesisarchitectural:– Components– Interactions

• Conceptual structure ofthe causal theories onthe nextslidesisstatistical:– Variables– Distributionoverpopulation

47

48

• Brazilian respondents’theory about causes and effects ofincompleterequirements

• German respondents’theory about causes and effects ofincompleterequirements

49

Theuseoftheories• “Requirementsareincompletebecausecustomershave

unclearneedsandhasnoREskills”– Frameaphenomenon:requirementscanbecompletelyspecified– Describeit:describeallmechanismsthatareresponsiblefor

incompleterequirements– Specifyatreatment:trainthecustomerinREskills(??)– Analyzeit:—– Generalizeaboutit:claimthatthisisresponsibleforincomplete

requirementsmoreoften/always– Predictaneffect:predictthatitwillhappeninthenextproject– Explainaneffect:explainthatincompletenessisduestounclearneeds

andabsenceofREskillsincustomer

50

Usability oftheories• Thetheory of34hypothesesisnot intended to be used by

professionalsto improve their practice.Consider the theory``improvingREskillsreduces requirements incompleteness’’

1. Professionaliscapable to recognize Contextassumptions– Yes:recognizablewhen there isrequirements engineering

2. Capable to acquire/buildArtifact under constraints ofpractice– That depends onthe available budget(time,money)for REtraining

3. Theeffects will indeed occur– That depends onthe training;and onother factorscausingREincompleteness

4. He/she can observe this– Hardto saywhether requirements aremorecomplete

5. Theywill contribute to stakeholdergoals/satisfy requirements– Hardto saywhether REcompletenesswill contribute to stakeholdergoals

51

Inferringtheoriesfromdata– Description

• Interpretationoftheanswersoftherespondents• Descriptivestatistics

– Statisticalinference• Nostatisticalinference

– Abductive inference• Theassumedexplanationoftherespondent’sanswersisthat

theybasethemonexperience

– Analogicinference• Otherprofessionalswillanswersimilarly;butpossiblydifferent

acrosscountries/cultures

52

Theresearchsetup


Study


populationelements


Measure-ment

instruments

53

All REprofessionals

SampleofREprofessionalsNotreatment

On-line surveytool,questionnaire

“WhySoftwareRepositoriesAreNotUsedForDefect-InsertionCircumstanceAnalysis

MoreOften:ACaseStudy”Lutz Prechelt,AlexanderPepper

Informationand SoftwareTechnology

55

“WhySoftwareRepositoriesAreNotUsedForDefect-InsertionCircumstanceAnalysisMoreOften:ACaseStudy”

Lutz Prechelt,AlexanderPepperInformationand SoftwareTechnology

• Pepper tried to minesoftwarerepositories ofthe contentmanagementsystemFiona,produced by Infopark,inordertoidentify correlates ofdefectinsertion,hoping that they can beused to improve the softwareprocess.– Engineeringcycle ofthe client

• Pepper and Prechelt observed this.– Casestudy

• Validationofacommunity-wide developmentofMSRtechniques for DICA.– Engineeringcycle ofresearchcommunity

• Researchquestionthat emerged from the case:why areMSRtechniques for DICAnot used moreoften? 56

Whatisatheory• Theory1,heldbythecommunity:

– MSRcanprovideinformationaboutimprovementopportunitiesofthesoftwareprocess(p.3rightcolumn)

• Artifact:MSR• Context:anysoftwaredevelopmentprocess

57

Descriptivegeneralization

• Theory2,proposedbyPrechelt andPepperbasedonthecasestudy:– R1:…– …– R5:Thereisnoaffordablemethodtoassessthe

reliabilityoftheresultsofMSRinDICA– R6:ThereliabilityofMSRresultsinDICAislow– R5andR6arethemajorreasonswhyMSRisnotused

forDICA

• Artifact:MSR• Context:organizationsthatdevelopweb

applicationsforalongperiodoftime,confusedefectswithissues,andhavenodedicatedstafftomaintainbugtracks(sect8.1)

58

Descriptivegeneralizations

Rationalexplanation of a phenomenon.

(= architecturalexplanation, where somecomponents are actors that have goals and mayhave reasons foractions)

Thestructureoftheories

• Conceptualframework– Definitionsofchange,defect,rework,issue,bug,bugfix,defectinsertion,defectcorrection

– Difficulty,cost,utility,reliabilityofatechnique• NB1conceptssharedwiththeOoS• NB2architecturalframework

• Generalizations– Previousslide

• NBtheyareabouttheeffectsofaclassofartifactsinaclassofcontexts

59

Theuseoftheories• “MSRcanprovideinformationaboutimprovement

opportunitiesofthesoftwareprocess”– Frameaphenomenon:softwareimprovementisaproblemoflackof

dataaboutthesoftwareprocess– Describeit:describesoftwarerepositories– Specifyatreatment:specifyMSRtechniques,toolsandsteps– Analyzeit:analyzethemeaningoftheoutputofMSR– Generalizeaboutit:claimthattheoutcomewillbeobtainedinall

softwareprocesses– Predictaneffect:predictthatitwillhappeninthenextproject– Explainaneffect:explainthatanimprovementhasoccurredbecause

ofremovalofaweakspotintheprocess

60

Usability oftheories1. Professionaliscapable to recognize Contextassumptions

– yes

2. Capable to acquire/buildArtifact under constraints ofpractice– Prechelt &Pepper:considerable effortintheir case

3. Theeffects will indeed occur– Noevidence that reliable informationabout processeswill be

produced

4. He/she can observe this– No:considerable uncertaintywhether effects haveoccured

5. Theywill contribute to stakeholdergoals/satisfyrequirements– Noevidence that process improvementswill occur

61

Applying existingtheoriestodataandInferring neworupdatedtheoriesfromdata

• Description– Casedescriptionsofeverystep– InterpretationofeverystepintermsofR1– R6

• Statisticalinference– Notpossiblefromacase– (butthereisoneinsidethiscasetoinvestigatethe

relationbetweendefectdescriptionsandissuedescriptions)

• Abductive inference– Explanationofnon-useintermsofR1– R6– Rationalexplanationintermsofreasonsofactors

• Analogicinference– Descriptionsandexplanationgeneralizedbyanalogy– Discussionofexternalvalidity

62

How did it happen? • Existing theory 1

assumed, and falsified• New theory 2 emerged

from the data and fromopinions of actors in theOoS. Or were thepropositions R1-6 specified before the case study was started?

Theresearchsetup


Study


populationelements


Measure-ment

instruments

63

Sources of evidence p. 5:Context information, raw data of version archive andbugtracker, analysis steps taken and not taken, issues and arguments of those steps, data provided by MSR tools,Infopark’s interpretation of the outcomes of the steps

MSR tools providing data;Peppers work notes;Pepper’s memory(sect 8.3)

MSR tools

One complex Object of Study: Infopark and its software repositories

Other software development organizations and their repositories

Treatment is the 4–step procedure listed in sect 2.3 performed by Pepper at Infopark

“ExperimentalValidationofaRiskAssessmentMethod”

Vriezekolk,Etalle&Wieringa

21stWorking ConferenceonRequirements Engineering:

Foundationsfor SoftwareQuality(REFSQ)2015

65

• Labexperimentto testreliability ofamethod,RASTER,to assess riskoftelecomavailability– Researchquestion:Howreliable isRASTER?– Researchsetup:Sixgroupsofthree students eachhadto estimate likelihood and impactofalistofnon-availabilityrisks for an emailservice,usingthe RASTERmethod

66

Whatisatheory

• Designtheory– RASTERxprofessionalsprovidingservicesduringincidentsanddisasters→availabilityriskassessments

• Theoryoftheexperiment– Sourcesofvariabilityinassessmentare

• Ambiguityorincompletenessofthemethoddescription• Misunderstandingofthemethod,• Lackofexperience• Lackofmotivation• Casecomplexity• Disturbancefromtheenvironment

67

Empirical test,Peer review?

Empirical test,Peer review?

Artefact, context

Artefact, context

ThestructureoftheoriesDesigntheory1. Conceptualframework

– Rasterconcepts(infrastructurecomponent,vulnerability,risk,impact,likelihood,…)

2. Thedesigngeneralization

Theoryoftheexperiment1. Conceptualframework

– Riskassessor,team,targetofassessment,asse4ssmentenvironment

2. Generalizations– Claimsaboutmechanismsthatproducevariability

68

Theuseoftheories• “RasterxProfessionals→riskassessments”

– Frameaphenomenon:riskassessmentsaremadebyprofessionals– Describeit:describetelcoinfrastructurearchitectureandits

vulnerabilities– Specifyatreatment:useRASTERtoassessrisks– Analyzeit:Traceriskstoarchitecturecomponents– Generalizeaboutit:claimthatotherprofessionalswouldfindthe

samerisksofsimilartelcoarchitectures– Predictaneffect:predictthatthiswillhappeninthenextproject– Explainaneffect:ExplainassessmentsintermsofRASTERmethodand

ToA

69

Usability oftheories1. Professionaliscapable to recognize Contextassumptions

– Yes

2. Capable to acquire/buildArtifact under constraints ofpractice– RASTERrequires relatively little training;RAisexpensive,butnot due to

RASTER

3. Theeffects will indeed occur– Hasbeenshown inexperiments and pilots

4. He/she can observe this– Plain for all to see

5. Theywill contribute to stakeholdergoals/satisfy requirements– Goalisto obtain accurateand reliable assessments

70

Inferringtheoriesfromdata– Description

• OutcomeofRA’sonpaper• Krippendorf’s alphatomeasureinterrateragreement• Outcomeofexitquestionnairestoassessourcesofvariability

– Statisticalinference• Samplenon-random,andtoosmall.

– Abductive inferenceObservedvariabilityexplainedby1. lackofexpertknowledge,2. differencesinassumptions,3. difficultytochoosebetweenadjacentordinalvaluesforlikelihood

– Analogicinference• 1and 2absent/reduced inthe field,so less variability there• 3motivates improvement ofthemethod to reduce this phenomenon

71

Theresearchsetup


Study


populationelements


Measure-ment

instruments

72

RAprofessionals intelcoDoing RAinaquiet room

Self-selected sampleofstudentsInaquiet room

ApplicationofRASTERto asmallcase

Personalobservation,Exitquestionnaire,RASTERforms

Oralinstruction,written casedescription and RASTERhelp

Similarities and dissimilarities!Used both to reason from sampleto population

1. Theory ofvariability formulated;2. Designed aresearchsetupthat minimized the impactofthesesources;3. Explained observed variation interms ofthis theory4. Used this to generalize to population and to improve RASTER

theories in empirical software engineering

Science