surveys in software engineering

SurveysinSoftwareEngineering

Rafael Maiani de Mello

Marco Torchiano

Daniel Méndez Guilherme H. TravassosFurther coaches

Ground rules for today

1. Whenever you have a question/remark: share them with the group!

2. Stay flexible

3. There are no further rules J

Feelfreeto…

copy,share,andalter,

filmandphotograph(aslongaswelookgreat)

tweetandliveblog

today’spresentationsgiventhatyouattributeittoitsauthor(s)andrespectitsrightandlicensesofitsparts.

Whoare we?

MarcoTorchianoPolitecnico diTorinohttp://softeng.polito.it/torchiano/

RafaelMaiani deMelloPontificalCatholicUniversityofRiodeJaneirohttp://inf.puc-rio.br/~rmaiani

GuilhermeHorta TravassosCOPPE,FederalUniversityofRiodeJaneirohttp://www.cos.ufrj.br/~ght

DanielMéndezTechnicalUniversityofMunichhttp://www4.in.tum.de/~mendezfe/

Whoareyou?

Quick round:

• Who are you?

• What is your experience in conducting survey research?

• What are your expectations?

What doyou think?

Whydoweneedsurveyresearchinsoftwareengineering?

AgendaTime Topic

09:00– 11:00 SessionI- IntroductiontosurveysWherewewillprovidethebasictheoreticalconceptsofpopulationsurveys:generalmethod,sourceoferrors,sampling,instrumentdesign.

11:00– 11:30 Morningbreak

11:30– 13:00 SessionII- BestpracticesWherewewillfocusonthekeyaspectsofdesigningandconductingsoftwareengineeringsurveysandpresentobservedissuesandevidencebasedlessons.

13:00– 14:30 Lunch14:30– 16:30 SessionIII- Hands-on(BYOL)

Wheretheparticipantsareexpectedtodesignandimplementasimplesurveyonarealonlinetool.BringYourOwnLaptop,ortabletatleast.

16:30– 17:00 Afternoonbreak

17:00– 18:30 SessionIV– Q&AWherewewilldiscusswiththeparticipantsaboutthemostimportantissuesandcomeupwithsomegeneralrecommendation.

19:30– 22:00 TourCiudadRealReceptionattheTownHallofCiudadRealatCasa-Museo López-Villaseñor

SessionI

IntroductiontoSurveyResearch

Session I

Introduction to Survey research

BigPicture…1st layer

9

Philosophy of science

Principle ways of working

Methods and Strategies

Fundamental tools

Epistemology

Empirical methods

Statistics

Hypothesis testingCase studies

Logic

Examples

Theories

ESEreliesoneverylayer!

10

Philosophy of science

Principle ways of working

Methods and Strategies

Fundamental tools

Setting of Empirical Software Engineering:

§ Theory building and evaluation

are supported by

§ Methods and Strategies

Analogy: Theoretical and Experimental Physics

BigPicture…2nd layer

11

Theory/System of theories

(Tentative)Hypotheses

Observations / Evaluations

Study Population

Induction

PatternBuilding

Deduction

Falsification /Support

TheoryBuilding

BigPicture3rd layer:Methods

12

• EachmethodIcanapply…• Hasaspecificpurpose

• Reliesonaspecificdatatype

Purposes

• Exploratory• Descriptive• Explanatory

• Improving

DataTypes

• Qualitative• Quantitative

Example: Grounded Theory


Study Population

Qualitative Data

DescriptiveExploratory, or

Explanatory

Further reading: Vessey et alA unified classification system for research in the computing disciplines

Theory/System oftheories


Observations / Evaluations

Study Population

PatternBuilding

Falsification /Support

TheoryBuilding

BigPicture3rd layer:MethodsFormal /

ConceptualAnalysis

Grounded Theory

Confirmatory• Case & Field

Studies• Experiments • Simulations

Survey andInterview Research

• EthnographicStudies

• Folklore Gathering

Exploratory• Case & Field

Studies• Data Analysis

ObservationalStudies

Survey(Cross-Sectional) Case study Case-Control

Survey

Systematicobservationalmethodtogatherqualitativeand/orquantitativedatafrom(asampleof)entitiesto

characterizeinformation,attitudesand/orbehaviorsfromdifferentgroupsofsubjectsregardinganobjectofstudy

15

Descriptivestatistics+Analyticstatistics

Planning

Surveyprocess

16

Define research objectives

Chose collection mode Chose sampling frame

Questionnaireconstruction and pretest

Design and select sample

Recruit and measure

Data coding and editing

AnalysisPost-survey adjustments

Surveyprocess

17





Recruit and measure



Research question +Characterization of constructs and population

What is the productivity ofJava software developers?

Parameter/ Construct

Target population

Surveyprocess

18





Recruit and measure



How many lines of Java code have you written in the last week?

Measurement

Surveyprocess

19





Recruit and measure



A couple hundreds

Response

Surveyprocess

20





Recruit and measure



200 LOCs

Edited response

Surveyprocess

21





Recruit and measure



Research question +Characterization of constructs and population

What is the productivity ofJava software developers?

Parameter/ Construct

Target population

Surveyprocess

22





Recruit and measure



Developers working for companies in the region having NACE activity code J

62.0.x

Frame population

Surveyprocess

23



Questionnaireconstruction and pretest Design and select sample

Recruit and measure



2 developers in each of 100 randomly selected companies

Sample

Surveyprocess

24





Recruit and measure



• [email protected]• [email protected]…

Respondents

Measurementvs Representation

25




Recruit and measure


AnalysisMake adjustments

Construct Target population

Measurement

Response

Edited response

Frame population

Sample

Respondents

Post-survey adjustments

Instrument

Measurementperspective

• Construct

• Measurement

• Response

• Editedresponse

26




Recruit and measure



μi

Yi

yiyip

Construct

• Elementofinformationsoughtbyresearchers• Examples

• Howmanynewjobscreated• Howmanyincidentsofcrimewithvictims

• Whichdevelopmentstoolsused• Formulation

• Easytounderstand

• Imprecise• Abstract

27

μi

Construct

• LevelofAbstraction/Measurementperspective

• Directlyobservable

• E.g.Staffforaproject

• Afewdefinedwaystomeasure

• Nondirectlyobservable

• Intentiontoadoptatechnology

• Nosinglewell-definedmeasure

28

Measurement

• Howtogatherinformationaboutconstructs• Objectivemeasures

• Electronic• Physical

• Answerstoquestions• Visual(formalquestionnaires)• Oral(structuredorsemi-structuredinterviews)

29

Yi

Validity

• Gapbetweenconstructsandmeasurement• Ideallythemeasureistheresultofjustoneamongseveral

possibletrials• Inpracticethemeasurementmayintroduceanerror

• Eachtrialintroducesadifferenterror

• Validity:=correlationbetweenYandμ

30

Yit = µi + ✏it

Response

• Theactualdatacollectedthroughthesurvey• Aquestionmayrequire

• Searchownmemory• Accessrecords

• Askotherpersons• Closedquestionsalreadycontainpossibleanswers• Sometimesaresponseisnotprovided

31

yi

• Gapbetweentheidealmeasurementoutcomeandresponseobtained

• Responsebias• Systematicmisreporting

• Reliability

• Variabilityoverseveraltrials

Measurementerror

32

yi � Yi

Editedresponse

• Reviewprocessbeforeusingdata• Rangechecks

• Consistencychecks• Illegibleanswersdetection

• Skippedquestions• Outlierdetection

33

yip

Processingerror

• Gapbetweenvariablesusedinanalysisandthoseprovidedbytherespondent

• Erroneousoutlieridentification• Codingerror

34

Errors– measurementpov

35

Construct

Measurement

Response

EditedResponseProcessing

Error

Measurement Error

Validity

Representationperspective

36

• Targetpopulation

• Framepopulation

• Sample

• Respondents

• Post-surveyadjustments




Recruit and measure



Y

YC

ya

yr

Representationperspective

37

Target

Frame

Sample

Respondents

Targetpopulation

• Thesetofunitstobestudied• Abstractpopulationdefinition

• E.g.softwareprojects• Time?

• Insoftwarecompaniesonly?• Italiancompaniesonly?• Completedorjuststartedprojects?

38

Framepopulation

• Allunitsinthesamplingframeconstitutetheframepopulation• Intheory

• Thesubsetoftargetpopulationthathasachancetobeselected

• Inpractice• asetofunitsimperfectlylinkedtothetargetpopulation

members

• E.g.telephonenumbers

39

Framinginstrument• The(conceptual)instrumentusedtoidentifytheunitsofstudy

• Householdphonenumberstogetpersons

• Companyrecordstogetemployees• CustomerIDstogetcustomers

• SocialNetworkIDstogetmembers• Warning:oftentheframeelementsareindirectlylinkedtothe

unitsofanalysis,throughrespondents.E.g.

• UoA:softwareprojects(UoA)• Respondents:developers

• Frameelements:softwarecompanies

40

Target

Frame

CoverageError

41

Undercoverage

Ineligible units

Covered population

Realitycheck

• During2012USAPresidentialElectionsCampaignbecauseofaneffectofFederalregulationspollingcellphoneswasmoreexpensive

• Asaresult,manypublicpollsleavecellphoneusersoutoftheirsamples

• Duetothegrowingpopularityofcellphonesastheonlypointofcontactforyoungvotersandminorities,poolersleftkeyconstituenciesforObamaoutofthepollsandskewedthenumbersforRomneyinsomesamples

42http://www.politico.com/news/stories/1112/84103.html?hp=l1

“That’swhysomepolls lookedsodifficultforthepresident,becausetheywereunder-polling theelectorateforthepresident”

J.Messina(CampaignManagerforObama)

Coveragebias

• Twofactors1. Differencebetweencoveredandnotcoveredpopulation

• Y:meanoftarget• YC:meanofcovered YU:meanofuncovered

2. Proportionofnoncoveredpopulation• C:#coveredunitsU:#uncoveredunits

43

Y C � Y =U

C(Y C � Y U )

Sample

• Unitsselectedfromtheframepopulation• Timeandcostopportunity

• Sampling:==Deliberatenon-observation• Mayintroducedeviationbetween

• Samplestatistic• Fullframestatistic

44

SamplingDesign

• Strategyfollowedtoestablishthesamplefromtheframepopulation

• Non-probabilisticsamplingdesignsinclude:1. Accidentalsampling(simplyuseconvenience)

2. Judgementsampling(applysometechnicalcriteriatosample)

3. Snowballing(sharethesamplingdecisionwithpartofthesubjects)

4. QuotaSampling(establishfixedquotabygroups)

45

SamplingDesign

• Probabilisticsamplingdesignsinclude:1. Simplerandomsampling(randomselect"n"unitsfromthe

framepopulation)2. StratifiedSampling(simplerandomsamplingfromeach

stratumestablishedintheframepopulation)• Example:Tosamplejavadevelopersbycountry

46

SampleSizeFormula

• Recommendedwhenworkingwithprobabilisticsamplingdesigns

• SS:samplesize

• Z: Z-value,establishedthroughaspecifictable(Z=2.58for99%ofconfidencelevel,Z=1.96for95%ofconfidencelevel

• p: percentage selecting a choice, expressed as decimal (0.5 used as default forcalculating sample size, since it represents the worst case).

• c:desiredconfidenceInterval,expressedindecimalpoints(Ex.:0.04).

47

SampleSizeFormula

• Correctionformulabasedonafinitepopulationwithapopsize

48

Population Confidence LevelConfidence

IntervalSample Size

10,000 95% 0.01 4,899

10,000 95% 0.05 370

500 95% 0.01 475

500 95% 0.05 217

Samplingerror

• Samplingbias• Systematicexclusionofsomemembers

• Orsignificantlyreducedchanceofselection• Samplingvariance

• Idealsetofsamplesalldrawnfromthesameframe

49

Vs =

SX

s=1

�ys � Y C

�2

S

Samplingerrorreduction

• Probabilisticsamplingü Allunitshavenonzeroselectionprobability

• Stratifiedsamplingü Representationofkeysub-populationsiscontrolled

• Elementsamplesü Asopposedtoclustersamples

• Samplesize

50

Respondents

• Thesubsetofsampleforwhichameasurementcouldbecollected

ü Itemmissingdata:incompletemeasures• Fullparticipation(i.e.100%responserate)realisticallypossible

onlyforinanimateunits

51

Respondents

52

000000000000000000000000000000000000000000

111111x1111111111111111111111111111x11111111111111x111111111x111111111111111111111111111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Framedata Interview data

Data items

Samplecases

Respondents

Nonrespondents

Item missing data

Non-responseerror

• Non-responsebiasü Non-responserate:mS /nSü Differencebetweenrespondentsandnon-respondents

53

yr � ys =ms

ns(yr � ym)

Post-surveyadjustment

• Weightingü Compensateunder-representationdueto

ü Nonresponsepatternsü Mismatchbetweenframeandtargetpopulation

• Imputationü Itemmissingdataarereplacedbyestimations

54

Session II

Best Practices on Planning Surveys

DisclaimerThere is no universal silver bullet!

BestPractices

Defining research objectives

Sampling

Questionnaire Design

Recruiting

Characterizing the Target Population

Designresearchobjectives

Challenge:

Knowthelimitationsofsurveyresearch

à Surveyresearchoptsforanswersthatrelyonexperiences,opinions,andobservations(folklore)oftherespondents

• Developinternalquestionstohelpyoudepictingtheresearchobjective

• Optfordescriptivequestions(“whatishappening?”)orexplanatoryquestions(“whyisthishappening?”)ratherthannormativequestions(“whatshouldwedo?”)

Designresearchobjectives

Challenge:

Identifytherealtargetpopulation

à Avoidtorestrictthetargetpopulationbasedonfactorssuchasitssizeoritsavailability.

• Basedontheresearchobjectives,answerthefollowingquestion:“Whocanbestprovideyouwiththeinformationyouneed?”,insteadofanswering“Whoareprobablyavailabletoparticipate?"

BestPractices


Sampling


Recruiting


CharacterizingtheTargetPopulation

Challenge:

Identifyingthesurveyunitofanalysis

à Thesurveysubjects(individuals)aretypicallythesurveyunitofanalysis.However,insomecasesitmaybeagroupofindividuals,suchashouseholdsororganizationalunits/projectteamsinSEresearch

• Basedontheresearchobjective,identifywhichentityshouldbeusedtoguidesamplinganddataanalysing• Forinstance,investigating Javadevelopers programmingpracticeisaresearch

objectivedifferentfrominvestigatingjavaprogrammingpracticeinsoftwarehouses

Unit of analysis


Image source: http://www.telegraph.co.uk/finance/personalfinance/8080294/Poorest-households-hit-15-times-harder-by-Government-cuts.html

How do developers perform code debugging?

How code debugging have been performed by developers from software houses?

List of software developers

List ofsoftwarehouses

Challenge:

CharacterizingthesubjectsandunitsofanalysisinSEsurveys

à Differentresearchobjectivesmaydemanddifferentattributestocharacterizingindividuals/groupsofindividualsinvolvedinthesurveys

à Whatattributesarenecessarytoidentifya“representative”population?

• Standardscanbeespeciallyhelpfultoprovidescalesandevennominalvalues• Forinstance,CMMI-DEVmaturitylevelcanbeusedtocharacterize

organizationalunitsregardingtheirmaturityinsoftwareprocess.RUProlescanbeusedtocharacterizesubjects’currentposition



Challenge:

CharacterizingthesubjectsandunitsofanalysisinSEsurveys

• Individuals canbecharacterizedthroughattributessuchas:experienceintheresearchcontext,experienceinSE,currentprofessionalrole,locationandhigheracademicdegree]

• Organizations canbecharacterizedthroughattributessuchas:size(scaletypicallybasedinthenumberofemployees),industrysegment (softwarefactory,avionics,finance,health, telecommunications, etc.),locationandorganization type(government, privatecompany,university,etc.)

• Projectteams canbecharacterizedthroughattributessuchasprojectsize;teamsize,client/productdomain(avionics, finance,health,telecommunications, etc.)andphysicaldistribution

BestPractices


Sampling


Recruiting


Sampling

Challenge:LookingfortheFramePopulation

à SuitablesamplingframesarerarelyavailableinSEresearch.Weoftenneedtoperform“indirectsampling”(forinstance,thereisnoyellowpagesforsoftwareprojectsinacountry).

• Firstofall,youshouldsearchforcandidatesofsourcesofpopulation.Avoidtheconvenienceonsearchingcandidates,tryingtoanswer:“Wherearepresentativepopulationfromthesurveytargetpopulationorevenalltargetpopulationisavailable?”

Image source: http://www.bryan-allen.com/Photography/General/i-fsVh3h2

Sampling

Anuniverseofimperfectalternatives!

SamplingAgoodsourceofpopulation...

• ...should not intentionally represent a segregated subset from the targetpopulation, i.e., for a target population audience “X”, it is not adequate tosearch for units from a source intentionally designed to compose a specificsubset of “X”

• ...should not present any bias on including on its database preferentially onlysubsets from the target population. Unequal criteria for including search unitsmean unequal sampling opportunities

• … allow identifying all source of population’ units by a distinct logical ornumerical id

• ...should allow accessing all its units. If there are hidden elements, it is notpossible to contextualize the population

Sampling

Challenge:

LookingfortheFramePopulation

• InthecaseofsurveyshavingSEresearchersastargetpopulation,youcanuseresultsfrompreviouslyconductedsystematicliteraturereviews(SLR)regardingyourresearchtheme• SocialnetworksaddressedtointegrateacademicssuchasResearchGate and

Academia.edu canbealsousefulinthiscontext.

SamplingChallenge:


• Lookforcataloguesprovidedbyrecognizedinstitutes/associations/governmentstoretrieverelevantsetofSEprofessionals/organizations.Someexamples:

• SEI(www.sei.cmu.edu) instituteprovidesanopenlistoforganizationsandorganizationalunitscertifiedineachCMMI-DEVlevel.

• FIPA(www.tivia.fi/in-english)providesinformationregardingFinlandITorganizationsanditsprofessionals.

• CAPES(www.capes.gov.br/) providesatoolforaccessinginformationregardingBrazilianresearchgroups.

Sampling

Challenge:


• Sourcesavailable inthewebsuchasdiscussion groups,projectsrepositories andworldwideprofessional socialnetworks canbehelpfultoidentifyrepresentativepopulationscomposedbySEprofessionals

Such sources can restrict at any moment the access to the

content available!

Sampling

Challenge:


• Howtofindtheframepopulationinthesourceofpopulation?• Onceyouhaveidentifiedasourceofpopulation,youneedtoestablish

steps/procedurestosystematicallydepictingthesurveysamplingframe.• Suchpracticeisimportanttoassessthesamplesrepresentativeness,

alsosupportingfuturere-executions

Sampling

Challenge:

Establishingthesurveysamplesize

• ParticipationratesinvoluntarysurveysinSEperformedoverrandomsamplestendtobesmall(lowerthan10%).Whataretheimplicationsonresponserates,butalsoonrepresentativeness?

• Takepreferencetoprobabilisticsamplingdesigns.• Independentfromtheamountofrespondents,itwillbepossibletocalculate

theresultsconfidence

• Involuntarysurveyswithpractitioners,establishsignificantlyhighersamplesizes,consideringtheexpectationofaverylowparticipationrate

Image source:http://www.playbuzz.com/viralpx/a-what-do-people-really-think-about-you

BestPractices


Sampling


Recruiting


QuestionnaireDesign

Challenge:Todesignaclear,simpleandconsistentsurveyquestionnaire

Remember:

Bad questionnaires can led subjects initially willing to participate to give up!

QuestionnaireDesign

Challenge:

Todesignaclear,simpleandconsistentsurveyquestionnaire

• Usesimpleandappropriatewordingforthesurveyquestions• Avoidtechnicaltermsasmuchaspossibleordefinetheminthe

questionnaire,accordingtothesurveytargetpopulation

• Takepreferencetodesignshortquestionsregardingasingleconcept• Avoiddoublebarreledquestions

• Avoidvaguesentenceswhilewritingsurveyquestions

Image source: http://www.fooj.it/wp-content/uploads/2015/07/boring-office.jpg

QuestionnaireDesign

Inyouropinion,doyouagreeordisagreethatcoderefactoringisaneed?Andwhataboutcodesmelldetection?

a) Istronglyagreeb) Ipartiallyagreec) Iagreed) Idisagree

QuestionnaireDesign

Coderefactoringisanessentialpracticeforimprovingtheunderstandingofobject-orientedcode.

a) Totallyagree

b) Partiallyagree

c) Neitheragreenordisagree

d) Partiallydisagree

e) Totallydisagree

QuestionnaireDesign

Challenge:


• Avoidbiased questions,whichcanbedonebycarefullyphrasingthequestionsthatdonotsuggestlikelyanswersorresponses

• Avoidingsensitive questions• InSEcontext,thesensitivequestionscanbeaboutrespondentsincome,

opinionaboutorganizationormanagement,etc.

• Avoidtoaskaboutfarpastevents

QuestionnaireDesign

Doyoupreferworkinginprojectsfollowingagilemethodsorthosefollowingusualnon-agile approaches?

Considering themaincharacteristicsofthelast10softwareprojectsyouhaveworkedon,pleaseanswerthefollowingquestions:

Askingage,gender,maritalstatusforcharacterizingrequirementsengineers

QuestionnaireDesign

Challenge:


• Itisimportanttoavoiddemandingquestions(requiringtoomucheffortfromrespondentstoanswer)

• Avoiddoublenegatives

QuestionnaireDesign

Afterreadingtheattachedpapersregardingnonfunctionalrequirements(NFR),pleaseanswerthefollowingquestions:

1. WhichofthefollowingNFRdoyoudisagreearenotrelevantinthecontextofreal-timesystems?

…

QuestionnaireDesign

Challenge:


• Becarefulonselectingthe ResponseFormat!• Wrongchoicesofresponseformatmayleadyouto:

• Losepreciousdata• Losetheopportunityofapplyingrelevantstatisticaltests

• Significantly(andunnecessarily)increasedataanalysisefforts

QuestionnaireDesign

Free-text

Numeric values

• Open questions• Allow coding• Content analysis• High effort on data

analysis

• Open questions• Allow a wide range

of statistical analysisInterval

Scale

• Closed questions• Not necessarily equally

distributed intervals• Significantly restricts

statistical analysis

Ordinal/ Likert scale

• Closed questions• Intervals are

considered equally distributed

• Statistical analysis is less restrictive than Interval Scale

Nominal• Closed questions• Statistical analysis

based on frequency

QuestionnaireDesign

HowmuchexperiencedoyouhaveinJavaprogramming?

a) VeryHighexperienceb) HighExperiencec) FewExperienced) VeryFewexperience

HowmuchexperiencedoyouhaveinJavaProgramming?

a) Lessthanoneyearb) 1yearto3yearsc) 3yearsto5yearsd) Morethan5years


__5__years


I have been working with Java programming atcompanies since 2011. Before, I got my firstJava certification in 2009, when I startedworking in personal projects. But I havedifficult with object-oriented parts…_________

DoyouhaveexperienceinJavaprogramming?

()Yes()No

BestPractices


Sampling


Recruiting


RecruitingChallenge:

Controllingrecruitmentandparticipation

• Sendindividualbutstandardinvitationmessages• Itisexpectedthatgreatmostoftheindividualmessagessentwillberead

• Avoid"spreadingspree":mailinglists,foruminvitationmessages,crowdsourcingtools(suchasAmazonMechanicalTurk)• Youwillhavefewornocontrolonwhoreadtheinvitation.So,whowas

effectivelyrecruited?

• Neverallowforwarding(whichisdifferentfromsnowballing)!• Itwillviolatethesample

• Sendaquestionnaire’sindividualtokentoeachsubject

RecruitingChallenge:

Stimulatingparticipation

• Remindersshouldbeusedwithcare.• Avoidremindingwhoalreadyhadparticipated

• Avoidremindingmorethanonce

• Theinvitationmessageshouldclearlycharacterizetheinvolvedresearchers,theresearchcontextandpresenttherecruitmentparameters

• Includeintheinvitationmessageacomplimentandanobservationregardingtherelevanceofsubjectparticipation

Image source: http://quotesgram.com/you-are-important-to-me-quotes/

Challenge:

Stimulatingparticipation

• Establishafiniteandnotlongperiodtoanswerthesurvey• One-twoweeks

• Offerrewards(raffles,donations,payments,sharingresults)• Takeintoaccountthelocalpolicies

Image source: http://quotesgram.com/you-are-important-to-me-quotes/

Recruiting

BestPractices


Sampling


Recruiting


PilotingtheSurveyChallenge:

Youhaveonlyoneshot!Onceyoustartedthesurvey,thereisusuallynowayback

• Pilotthepopulationandsamplingactivitiesü Usea(smaller)sampleofthesamplingframe,reproducingallplannedstepsü Willallowyoutochecktheadequacyoftheframepopulationtoyoursurvey.

• Pilotthequestionnaireü Isitclear,unambiguous,didyoumaybemisssomequestions?

• Pilottherecruitmentü Doesitisworkingeffectively?

• Pilotthedataanalysisü Doyouhaveplannedfortheproperdataanalysistechniques?Whatisthenecessary

dataquantityand quality?

SessionIII

Hands-On

Session III

Hands-On!

The“Plan”1. Defineupto4researchobjectives2. Teamassignment

<lunch>

1. ShortintroductionintothetoolFor each team:2. (Online)Surveydesignand implementation3. Piloting /Testing4. Wrap-up5. (Optional:Running survey with ESEMparticipants)

1. Your research objective2. Your (coarse) research questions3. Survey target population4. Survey unit of analysis5. Possible sources of population

Definitionof...

à Everyone sketcheshis/heridea for asurveyonasticky note

à Wemakea(plenary)selection and teamassingment

94

15-20 minutes

5 minutes

Back-UpQuestions• Are functional and non-functional requirements really

distinct?• Can current software testing techniques support the test

of context awareness systems?• What development practices can contribute more to

decrease the software technical debt?• What could be the software engineering gap when

developing scientific (e-science) software?• What is the impact of Continuous Software Engineering

on software productivity and quality?• What is software?


<lunch>

1. ShortintroductionintothetoolFor each team:2. (Online)Surveydesignand implementation3. Piloting4. Wrap-up5. (Optional:Running survey with ESEMparticipants)

Login

http://ww2.unipark.de/www/

Login


Team1User:iasese_1Password:A2C04aq.

Team2User:iasese_2Password:A2C04sw.

Team3User:iasese_3Password:A2C04de.

Team4User:iasese_4Password:A2C04fr.

Selectyourproject

• Pleasesticktoyourownproject(watchtheteamnumber)

• Pleasetakecarewhatyoudo,youhaveadminrightsinthesystem

Startediting

Participationlink(page-based)questionnaireeditor

Dataexport

Don’ttouchJ

Questionnaire

ExportingdataDataexport(takesabit)

Codebook

BeforeyoustartTestandreset

Acclimatizationphase

Team1User:iasese_1Password:A2C04aq.

Team2User:iasese_2Password:A2C04sw.

à Everyteamgets to know the toolà Incaseofquestions,askJ

Team3User:iasese_3Password:A2C04de.

Team4User:iasese_4Password:A2C04fr.

10 minutes



<lunch>

1. ShortintroductionintothetoolFor each team:2. (Online)Surveydesignand implementation3. Piloting4. Wrap-up5. (Optional:Running survey with ESEMparticipants)

Wrap-Up

Eachgroup:Brieflyintroducethesurvey(researchobjective,targetpopulation,questionnaire)

à Askquestionsà Seewhatyoucanlearnfromotherdesigns

Alltogether• ShallwerunoneofthesurveysduringESEM?

15-20 minutes

Session IV

Q&A

Furtherreading…

• Groves, Fowler, Couper, Lepkowski, Singer and Torangeau, 2009. “Survey Methodology – 2nd edition” John Wiley and Sons

• Conradi R., Li J., Slyngstad O. P. N., Kampenes V. B., Bunse C., Morisio M., Torchiano M. “Reflections on conducting an international survey of CBSE in ICT industry” IEEE 4th International Symposium on Empirical Software Engineering November, 2005

• Linåker, J. Sulaman, S. M., de Mello, R. M. and Höst, M. 2015. Guidelines for Conducting Surveys in Software Engineering. TR 5366801, Lund University Publications. http://lup.lub.lu.se/record/5366801/file/5366839.pdf

• de Mello, R. M. and Travassos, G. H. 2016. Surveys in Software Engineering: Identifying Representative Samples. Proc. of 10th ACM/IEEE ESEM, Ciudad Real.

• de Mello, R. M. 2016. Conceptual Framework for Supporting the Identification of Representative Samples for Surveys in Software Engineering. Doctoral Thesis, COPPE/UFRJ, 138p. http://www.cos.ufrj.br/uploadfile/publicacao/2611.pdf