surveys in software engineering

SurveysinSoftwareEngineering

Rafael Maiani de Mello

Marco Torchiano

Daniel Méndez Guilherme H. TravassosFurther coaches

Ground rules for today

1. Whenever you have a question/remark: share them with the group!

2. Stay flexible

3. There are no further rules J

Feelfreeto…

copy,share,andalter,

filmandphotograph(aslongaswelookgreat)

tweetandliveblog

today’spresentationsgiventhatyouattributeittoitsauthor(s)andrespectitsrightandlicensesofitsparts.

Whoare we?

MarcoTorchianoPolitecnico diTorinohttp://softeng.polito.it/torchiano/

RafaelMaiani deMelloPontificalCatholicUniversityofRiodeJaneirohttp://inf.puc-rio.br/~rmaiani

GuilhermeHorta TravassosCOPPE,FederalUniversityofRiodeJaneirohttp://www.cos.ufrj.br/~ght

DanielMéndezTechnicalUniversityofMunichhttp://www4.in.tum.de/~mendezfe/

Whoareyou?

Quick round:

• Who are you?

• What is your experience in conducting survey research?

• What are your expectations?

What doyou think?

Whydoweneedsurveyresearchinsoftwareengineering?

AgendaTime Topic

09:00– 11:00 SessionI- IntroductiontosurveysWherewewillprovidethebasictheoreticalconceptsofpopulationsurveys:generalmethod,sourceoferrors,sampling,instrumentdesign.

11:00– 11:30 Morningbreak

11:30– 13:00 SessionII- BestpracticesWherewewillfocusonthekeyaspectsofdesigningandconductingsoftwareengineeringsurveysandpresentobservedissuesandevidencebasedlessons.

13:00– 14:30 Lunch14:30– 16:30 SessionIII- Hands-on(BYOL)

Wheretheparticipantsareexpectedtodesignandimplementasimplesurveyonarealonlinetool.BringYourOwnLaptop,ortabletatleast.

16:30– 17:00 Afternoonbreak

17:00– 18:30 SessionIV– Q&AWherewewilldiscusswiththeparticipantsaboutthemostimportantissuesandcomeupwithsomegeneralrecommendation.

19:30– 22:00 TourCiudadRealReceptionattheTownHallofCiudadRealatCasa-Museo López-Villaseñor

SessionI

IntroductiontoSurveyResearch

Session I

Introduction to Survey research

BigPicture…1st layer

Philosophy of science

Principle ways of working

Methods and Strategies

Fundamental tools

Epistemology

Empirical methods

Statistics

Hypothesis testingCase studies

Examples

Theories

ESEreliesoneverylayer!

Philosophy of science

Principle ways of working

Methods and Strategies

Fundamental tools

Setting of Empirical Software Engineering:

§ Theory building and evaluation

are supported by

§ Methods and Strategies

Analogy: Theoretical and Experimental Physics

BigPicture…2nd layer

Theory/System of theories

(Tentative)Hypotheses

Observations / Evaluations

Study Population

Induction

PatternBuilding

Deduction

Falsification /Support

TheoryBuilding

BigPicture3rd layer:Methods

• EachmethodIcanapply…• Hasaspecificpurpose

• Reliesonaspecificdatatype

Purposes

• Exploratory• Descriptive• Explanatory

• Improving

DataTypes

• Qualitative• Quantitative

Example: Grounded Theory

Study Population

Qualitative Data

DescriptiveExploratory, or

Explanatory

Further reading: Vessey et alA unified classification system for research in the computing disciplines

Theory/System oftheories

Observations / Evaluations

Study Population

PatternBuilding

Falsification /Support

TheoryBuilding

BigPicture3rd layer:MethodsFormal /

ConceptualAnalysis

Grounded Theory

Confirmatory• Case & Field

Studies• Experiments • Simulations

Survey andInterview Research

• EthnographicStudies

• Folklore Gathering

Exploratory• Case & Field

Studies• Data Analysis

ObservationalStudies

Survey(Cross-Sectional) Case study Case-Control

Survey

Systematicobservationalmethodtogatherqualitativeand/orquantitativedatafrom(asampleof)entitiesto

characterizeinformation,attitudesand/orbehaviorsfromdifferentgroupsofsubjectsregardinganobjectofstudy

Descriptivestatistics+Analyticstatistics

Planning

Surveyprocess

Define research objectives

Chose collection mode Chose sampling frame

Questionnaireconstruction and pretest

Design and select sample

Recruit and measure

Data coding and editing

AnalysisPost-survey adjustments

Surveyprocess

Recruit and measure

Research question +Characterization of constructs and population

What is the productivity ofJava software developers?

Parameter/ Construct

Target population

Surveyprocess

Recruit and measure

How many lines of Java code have you written in the last week?

Measurement

Surveyprocess

Recruit and measure

A couple hundreds

Response

Surveyprocess

Recruit and measure

200 LOCs

Edited response

Surveyprocess

Recruit and measure

Research question +Characterization of constructs and population

What is the productivity ofJava software developers?

Parameter/ Construct

Target population

Surveyprocess

Recruit and measure

Developers working for companies in the region having NACE activity code J

62.0.x

Frame population

Surveyprocess

Questionnaireconstruction and pretest Design and select sample

Recruit and measure

2 developers in each of 100 randomly selected companies

Sample

Surveyprocess

Recruit and measure

• Ben.gates@google.com• Dan.Schmidt@microsoft.com…

Respondents

Measurementvs Representation

Recruit and measure

AnalysisMake adjustments

Construct Target population

Measurement

Response

Edited response

Frame population

Sample

Respondents

Post-survey adjustments

Instrument

Measurementperspective

• Construct

• Measurement

• Response

• Editedresponse

Recruit and measure

Construct

• Elementofinformationsoughtbyresearchers• Examples

• Howmanynewjobscreated• Howmanyincidentsofcrimewithvictims

• Whichdevelopmentstoolsused• Formulation

• Easytounderstand

• Imprecise• Abstract

Construct

• LevelofAbstraction/Measurementperspective

• Directlyobservable

• E.g.Staffforaproject

• Afewdefinedwaystomeasure

• Nondirectlyobservable

• Intentiontoadoptatechnology

• Nosinglewell-definedmeasure

Measurement

• Howtogatherinformationaboutconstructs• Objectivemeasures

• Electronic• Physical

• Answerstoquestions• Visual(formalquestionnaires)• Oral(structuredorsemi-structuredinterviews)

Validity

• Gapbetweenconstructsandmeasurement• Ideallythemeasureistheresultofjustoneamongseveral

possibletrials• Inpracticethemeasurementmayintroduceanerror

• Eachtrialintroducesadifferenterror

• Validity:=correlationbetweenYandμ

Yit = µi + ✏it

Response

• Theactualdatacollectedthroughthesurvey• Aquestionmayrequire

• Searchownmemory• Accessrecords

• Askotherpersons• Closedquestionsalreadycontainpossibleanswers• Sometimesaresponseisnotprovided

• Gapbetweentheidealmeasurementoutcomeandresponseobtained

• Responsebias• Systematicmisreporting

• Reliability

• Variabilityoverseveraltrials

Measurementerror

yi � Yi

Editedresponse

• Reviewprocessbeforeusingdata• Rangechecks

• Consistencychecks• Illegibleanswersdetection

• Skippedquestions• Outlierdetection

Processingerror

• Gapbetweenvariablesusedinanalysisandthoseprovidedbytherespondent

• Erroneousoutlieridentification• Codingerror

Errors– measurementpov

Construct

Measurement

Response

EditedResponseProcessing

Measurement Error

Validity

Representationperspective

• Targetpopulation

• Framepopulation

• Sample

• Respondents

• Post-surveyadjustments

Recruit and measure

Representationperspective

Target

Sample

Respondents

Targetpopulation

• Thesetofunitstobestudied• Abstractpopulationdefinition

• E.g.softwareprojects• Time?

• Insoftwarecompaniesonly?• Italiancompaniesonly?• Completedorjuststartedprojects?

Framepopulation

• Allunitsinthesamplingframeconstitutetheframepopulation• Intheory

• Thesubsetoftargetpopulationthathasachancetobeselected

• Inpractice• asetofunitsimperfectlylinkedtothetargetpopulation

members

• E.g.telephonenumbers

Framinginstrument• The(conceptual)instrumentusedtoidentifytheunitsofstudy

• Householdphonenumberstogetpersons

• Companyrecordstogetemployees• CustomerIDstogetcustomers

• SocialNetworkIDstogetmembers• Warning:oftentheframeelementsareindirectlylinkedtothe

unitsofanalysis,throughrespondents.E.g.

• UoA:softwareprojects(UoA)• Respondents:developers

• Frameelements:softwarecompanies

Target

CoverageError

Undercoverage

Ineligible units

Covered population

Realitycheck

• During2012USAPresidentialElectionsCampaignbecauseofaneffectofFederalregulationspollingcellphoneswasmoreexpensive

• Asaresult,manypublicpollsleavecellphoneusersoutoftheirsamples

• Duetothegrowingpopularityofcellphonesastheonlypointofcontactforyoungvotersandminorities,poolersleftkeyconstituenciesforObamaoutofthepollsandskewedthenumbersforRomneyinsomesamples

42http://www.politico.com/news/stories/1112/84103.html?hp=l1

“That’swhysomepolls lookedsodifficultforthepresident,becausetheywereunder-polling theelectorateforthepresident”

J.Messina(CampaignManagerforObama)

Coveragebias

• Twofactors1. Differencebetweencoveredandnotcoveredpopulation

• Y:meanoftarget• YC:meanofcovered YU:meanofuncovered

2. Proportionofnoncoveredpopulation• C:#coveredunitsU:#uncoveredunits

Y C � Y =U

C(Y C � Y U )

Sample

• Unitsselectedfromtheframepopulation• Timeandcostopportunity

• Sampling:==Deliberatenon-observation• Mayintroducedeviationbetween

• Samplestatistic• Fullframestatistic

SamplingDesign

• Strategyfollowedtoestablishthesamplefromtheframepopulation

• Non-probabilisticsamplingdesignsinclude:1. Accidentalsampling(simplyuseconvenience)

2. Judgementsampling(applysometechnicalcriteriatosample)

3. Snowballing(sharethesamplingdecisionwithpartofthesubjects)

4. QuotaSampling(establishfixedquotabygroups)

SamplingDesign

• Probabilisticsamplingdesignsinclude:1. Simplerandomsampling(randomselect"n"unitsfromthe

framepopulation)2. StratifiedSampling(simplerandomsamplingfromeach

stratumestablishedintheframepopulation)• Example:Tosamplejavadevelopersbycountry

SampleSizeFormula

• Recommendedwhenworkingwithprobabilisticsamplingdesigns

• SS:samplesize

• Z: Z-value,establishedthroughaspecifictable(Z=2.58for99%ofconfidencelevel,Z=1.96for95%ofconfidencelevel

• p: percentage selecting a choice, expressed as decimal (0.5 used as default forcalculating sample size, since it represents the worst case).

• c:desiredconfidenceInterval,expressedindecimalpoints(Ex.:0.04).

SampleSizeFormula

• Correctionformulabasedonafinitepopulationwithapopsize

Population Confidence LevelConfidence

IntervalSample Size

10,000 95% 0.01 4,899

10,000 95% 0.05 370

500 95% 0.01 475

500 95% 0.05 217

Samplingerror

• Samplingbias• Systematicexclusionofsomemembers

• Orsignificantlyreducedchanceofselection• Samplingvariance

• Idealsetofsamplesalldrawnfromthesameframe

�ys � Y C

Samplingerrorreduction

• Probabilisticsamplingü Allunitshavenonzeroselectionprobability

• Stratifiedsamplingü Representationofkeysub-populationsiscontrolled

• Elementsamplesü Asopposedtoclustersamples

• Samplesize

Respondents

• Thesubsetofsampleforwhichameasurementcouldbecollected

ü Itemmissingdata:incompletemeasures• Fullparticipation(i.e.100%responserate)realisticallypossible

onlyforinanimateunits

Respondents

000000000000000000000000000000000000000000

111111x1111111111111111111111111111x11111111111111x111111111x111111111111111111111111111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Framedata Interview data

Data items

Samplecases

Respondents

Nonrespondents

Item missing data

Non-responseerror

• Non-responsebiasü Non-responserate:mS /nSü Differencebetweenrespondentsandnon-respondents

yr � ys =ms

ns(yr � ym)

Post-surveyadjustment

• Weightingü Compensateunder-representationdueto

ü Nonresponsepatternsü Mismatchbetweenframeandtargetpopulation

• Imputationü Itemmissingdataarereplacedbyestimations

Session II

Best Practices on Planning Surveys

DisclaimerThere is no universal silver bullet!

BestPractices

Defining research objectives

Sampling

Questionnaire Design

Recruiting

Characterizing the Target Population

Designresearchobjectives

Challenge:

Knowthelimitationsofsurveyresearch

à Surveyresearchoptsforanswersthatrelyonexperiences,opinions,andobservations(folklore)oftherespondents

• Developinternalquestionstohelpyoudepictingtheresearchobjective

• Optfordescriptivequestions(“whatishappening?”)orexplanatoryquestions(“whyisthishappening?”)ratherthannormativequestions(“whatshouldwedo?”)

Designresearchobjectives

Challenge:

Identifytherealtargetpopulation

à Avoidtorestrictthetargetpopulationbasedonfactorssuchasitssizeoritsavailability.

• Basedontheresearchobjectives,answerthefollowingquestion:“Whocanbestprovideyouwiththeinformationyouneed?”,insteadofanswering“Whoareprobablyavailabletoparticipate?"

BestPractices

Sampling

Recruiting

CharacterizingtheTargetPopulation

Challenge:

Identifyingthesurveyunitofanalysis

à Thesurveysubjects(individuals)aretypicallythesurveyunitofanalysis.However,insomecasesitmaybeagroupofindividuals,suchashouseholdsororganizationalunits/projectteamsinSEresearch

• Basedontheresearchobjective,identifywhichentityshouldbeusedtoguidesamplinganddataanalysing• Forinstance,investigating Javadevelopers programmingpracticeisaresearch

objectivedifferentfrominvestigatingjavaprogrammingpracticeinsoftwarehouses

Unit of analysis

Image source: http://www.telegraph.co.uk/finance/personalfinance/8080294/Poorest-households-hit-15-times-harder-by-Government-cuts.html

How do developers perform code debugging?

How code debugging have been performed by developers from software houses?

List of software developers

List ofsoftwarehouses

Challenge:

CharacterizingthesubjectsandunitsofanalysisinSEsurveys

à Differentresearchobjectivesmaydemanddifferentattributestocharacterizingindividuals/groupsofindividualsinvolvedinthesurveys

à Whatattributesarenecessarytoidentifya“representative”population?

• Standardscanbeespeciallyhelpfultoprovidescalesandevennominalvalues• Forinstance,CMMI-DEVmaturitylevelcanbeusedtocharacterize

organizationalunitsregardingtheirmaturityinsoftwareprocess.RUProlescanbeusedtocharacterizesubjects’currentposition

Challenge:

CharacterizingthesubjectsandunitsofanalysisinSEsurveys

• Individuals canbecharacterizedthroughattributessuchas:experienceintheresearchcontext,experienceinSE,currentprofessionalrole,locationandhigheracademicdegree]

• Organizations canbecharacterizedthroughattributessuchas:size(scaletypicallybasedinthenumberofemployees),industrysegment (softwarefactory,avionics,finance,health, telecommunications, etc.),locationandorganization type(government, privatecompany,university,etc.)

• Projectteams canbecharacterizedthroughattributessuchasprojectsize;teamsize,client/productdomain(avionics, finance,health,telecommunications, etc.)andphysicaldistribution

BestPractices

Sampling

Recruiting

Sampling

Challenge:LookingfortheFramePopulation

à SuitablesamplingframesarerarelyavailableinSEresearch.Weoftenneedtoperform“indirectsampling”(forinstance,thereisnoyellowpagesforsoftwareprojectsinacountry).

• Firstofall,youshouldsearchforcandidatesofsourcesofpopulation.Avoidtheconvenienceonsearchingcandidates,tryingtoanswer:“Wherearepresentativepopulationfromthesurveytargetpopulationorevenalltargetpopulationisavailable?”

Image source: http://www.bryan-allen.com/Photography/General/i-fsVh3h2

Sampling

Anuniverseofimperfectalternatives!

SamplingAgoodsourceofpopulation...

• ...should not intentionally represent a segregated subset from the targetpopulation, i.e., for a target population audience “X”, it is not adequate tosearch for units from a source intentionally designed to compose a specificsubset of “X”

• ...should not present any bias on including on its database preferentially onlysubsets from the target population. Unequal criteria for including search unitsmean unequal sampling opportunities

• … allow identifying all source of population’ units by a distinct logical ornumerical id

• ...should allow accessing all its units. If there are hidden elements, it is notpossible to contextualize the population

Sampling

Challenge:

LookingfortheFramePopulation

• InthecaseofsurveyshavingSEresearchersastargetpopulation,youcanuseresultsfrompreviouslyconductedsystematicliteraturereviews(SLR)regardingyourresearchtheme• SocialnetworksaddressedtointegrateacademicssuchasResearchGate and

Academia.edu canbealsousefulinthiscontext.

SamplingChallenge:

• Lookforcataloguesprovidedbyrecognizedinstitutes/associations/governmentstoretrieverelevantsetofSEprofessionals/organizations.Someexamples:

• SEI(www.sei.cmu.edu) instituteprovidesanopenlistoforganizationsandorganizationalunitscertifiedineachCMMI-DEVlevel.

• FIPA(www.tivia.fi/in-english)providesinformationregardingFinlandITorganizationsanditsprofessionals.

• CAPES(www.capes.gov.br/) providesatoolforaccessinginformationregardingBrazilianresearchgroups.

Sampling

Challenge:

• Sourcesavailable inthewebsuchasdiscussion groups,projectsrepositories andworldwideprofessional socialnetworks canbehelpfultoidentifyrepresentativepopulationscomposedbySEprofessionals

Such sources can restrict at any moment the access to the

content available!

Sampling

Challenge:

• Howtofindtheframepopulationinthesourceofpopulation?• Onceyouhaveidentifiedasourceofpopulation,youneedtoestablish

steps/procedurestosystematicallydepictingthesurveysamplingframe.• Suchpracticeisimportanttoassessthesamplesrepresentativeness,

alsosupportingfuturere-executions

Sampling

Challenge:

Establishingthesurveysamplesize

• ParticipationratesinvoluntarysurveysinSEperformedoverrandomsamplestendtobesmall(lowerthan10%).Whataretheimplicationsonresponserates,butalsoonrepresentativeness?

• Takepreferencetoprobabilisticsamplingdesigns.• Independentfromtheamountofrespondents,itwillbepossibletocalculate

theresultsconfidence

• Involuntarysurveyswithpractitioners,establishsignificantlyhighersamplesizes,consideringtheexpectationofaverylowparticipationrate

Image source:http://www.playbuzz.com/viralpx/a-what-do-people-really-think-about-you

BestPractices

Sampling

Recruiting

QuestionnaireDesign

Challenge:Todesignaclear,simpleandconsistentsurveyquestionnaire

Remember:

Bad questionnaires can led subjects initially willing to participate to give up!

QuestionnaireDesign

Challenge:

Todesignaclear,simpleandconsistentsurveyquestionnaire

• Usesimpleandappropriatewordingforthesurveyquestions• Avoidtechnicaltermsasmuchaspossibleordefinetheminthe

questionnaire,accordingtothesurveytargetpopulation

• Takepreferencetodesignshortquestionsregardingasingleconcept• Avoiddoublebarreledquestions

• Avoidvaguesentenceswhilewritingsurveyquestions

Image source: http://www.fooj.it/wp-content/uploads/2015/07/boring-office.jpg

QuestionnaireDesign

Inyouropinion,doyouagreeordisagreethatcoderefactoringisaneed?Andwhataboutcodesmelldetection?

a) Istronglyagreeb) Ipartiallyagreec) Iagreed) Idisagree

QuestionnaireDesign

Coderefactoringisanessentialpracticeforimprovingtheunderstandingofobject-orientedcode.

a) Totallyagree

b) Partiallyagree

c) Neitheragreenordisagree

d) Partiallydisagree

e) Totallydisagree

QuestionnaireDesign

Challenge:

• Avoidbiased questions,whichcanbedonebycarefullyphrasingthequestionsthatdonotsuggestlikelyanswersorresponses

• Avoidingsensitive questions• InSEcontext,thesensitivequestionscanbeaboutrespondentsincome,

opinionaboutorganizationormanagement,etc.

• Avoidtoaskaboutfarpastevents

QuestionnaireDesign

Doyoupreferworkinginprojectsfollowingagilemethodsorthosefollowingusualnon-agile approaches?

Considering themaincharacteristicsofthelast10softwareprojectsyouhaveworkedon,pleaseanswerthefollowingquestions:

Askingage,gender,maritalstatusforcharacterizingrequirementsengineers

QuestionnaireDesign

Challenge:

• Itisimportanttoavoiddemandingquestions(requiringtoomucheffortfromrespondentstoanswer)

• Avoiddoublenegatives

QuestionnaireDesign

Afterreadingtheattachedpapersregardingnonfunctionalrequirements(NFR),pleaseanswerthefollowingquestions:

1. WhichofthefollowingNFRdoyoudisagreearenotrelevantinthecontextofreal-timesystems?

QuestionnaireDesign

Challenge:

• Becarefulonselectingthe ResponseFormat!• Wrongchoicesofresponseformatmayleadyouto:

• Losepreciousdata• Losetheopportunityofapplyingrelevantstatisticaltests

• Significantly(andunnecessarily)increasedataanalysisefforts

QuestionnaireDesign

Free-text

Numeric values

• Open questions• Allow coding• Content analysis• High effort on data

analysis

• Open questions• Allow a wide range

of statistical analysisInterval

• Closed questions• Not necessarily equally

distributed intervals• Significantly restricts

statistical analysis

Ordinal/ Likert scale

• Closed questions• Intervals are

considered equally distributed

• Statistical analysis is less restrictive than Interval Scale

Nominal• Closed questions• Statistical analysis

based on frequency

QuestionnaireDesign

HowmuchexperiencedoyouhaveinJavaprogramming?

a) VeryHighexperienceb) HighExperiencec) FewExperienced) VeryFewexperience

HowmuchexperiencedoyouhaveinJavaProgramming?

a) Lessthanoneyearb) 1yearto3yearsc) 3yearsto5yearsd) Morethan5years

__5__years

I have been working with Java programming atcompanies since 2011. Before, I got my firstJava certification in 2009, when I startedworking in personal projects. But I havedifficult with object-oriented parts…_________

DoyouhaveexperienceinJavaprogramming?

()Yes()No

BestPractices

Sampling

Recruiting

RecruitingChallenge:

Controllingrecruitmentandparticipation

• Sendindividualbutstandardinvitationmessages• Itisexpectedthatgreatmostoftheindividualmessagessentwillberead

• Avoid"spreadingspree":mailinglists,foruminvitationmessages,crowdsourcingtools(suchasAmazonMechanicalTurk)• Youwillhavefewornocontrolonwhoreadtheinvitation.So,whowas

effectivelyrecruited?

• Neverallowforwarding(whichisdifferentfromsnowballing)!• Itwillviolatethesample

• Sendaquestionnaire’sindividualtokentoeachsubject

RecruitingChallenge:

Stimulatingparticipation

• Remindersshouldbeusedwithcare.• Avoidremindingwhoalreadyhadparticipated

• Avoidremindingmorethanonce

• Theinvitationmessageshouldclearlycharacterizetheinvolvedresearchers,theresearchcontextandpresenttherecruitmentparameters

• Includeintheinvitationmessageacomplimentandanobservationregardingtherelevanceofsubjectparticipation

Image source: http://quotesgram.com/you-are-important-to-me-quotes/

Challenge:

Stimulatingparticipation

• Establishafiniteandnotlongperiodtoanswerthesurvey• One-twoweeks

• Offerrewards(raffles,donations,payments,sharingresults)• Takeintoaccountthelocalpolicies

Image source: http://quotesgram.com/you-are-important-to-me-quotes/

Recruiting

BestPractices

Sampling

Recruiting

PilotingtheSurveyChallenge:

Youhaveonlyoneshot!Onceyoustartedthesurvey,thereisusuallynowayback

• Pilotthepopulationandsamplingactivitiesü Usea(smaller)sampleofthesamplingframe,reproducingallplannedstepsü Willallowyoutochecktheadequacyoftheframepopulationtoyoursurvey.

• Pilotthequestionnaireü Isitclear,unambiguous,didyoumaybemisssomequestions?

• Pilottherecruitmentü Doesitisworkingeffectively?

• Pilotthedataanalysisü Doyouhaveplannedfortheproperdataanalysistechniques?Whatisthenecessary

dataquantityand quality?

SessionIII

Hands-On

Session III

Hands-On!

The“Plan”1. Defineupto4researchobjectives2. Teamassignment

<lunch>

1. ShortintroductionintothetoolFor each team:2. (Online)Surveydesignand implementation3. Piloting /Testing4. Wrap-up5. (Optional:Running survey with ESEMparticipants)

1. Your research objective2. Your (coarse) research questions3. Survey target population4. Survey unit of analysis5. Possible sources of population

Definitionof...

à Everyone sketcheshis/heridea for asurveyonasticky note

à Wemakea(plenary)selection and teamassingment

15-20 minutes

5 minutes

Back-UpQuestions• Are functional and non-functional requirements really

distinct?• Can current software testing techniques support the test

of context awareness systems?• What development practices can contribute more to

decrease the software technical debt?• What could be the software engineering gap when

developing scientific (e-science) software?• What is the impact of Continuous Software Engineering

on software productivity and quality?• What is software?

<lunch>

1. ShortintroductionintothetoolFor each team:2. (Online)Surveydesignand implementation3. Piloting4. Wrap-up5. (Optional:Running survey with ESEMparticipants)

http://ww2.unipark.de/www/

Team1User:iasese_1Password:A2C04aq.

Team2User:iasese_2Password:A2C04sw.

Team3User:iasese_3Password:A2C04de.

Team4User:iasese_4Password:A2C04fr.

Selectyourproject

• Pleasesticktoyourownproject(watchtheteamnumber)

• Pleasetakecarewhatyoudo,youhaveadminrightsinthesystem

Startediting

Participationlink(page-based)questionnaireeditor

Dataexport

Don’ttouchJ

Questionnaire

ExportingdataDataexport(takesabit)

Codebook

BeforeyoustartTestandreset

Acclimatizationphase

Team1User:iasese_1Password:A2C04aq.

Team2User:iasese_2Password:A2C04sw.

à Everyteamgets to know the toolà Incaseofquestions,askJ

Team3User:iasese_3Password:A2C04de.

Team4User:iasese_4Password:A2C04fr.

10 minutes

http://ww2.unipark.de/www/

<lunch>

Wrap-Up

Eachgroup:Brieflyintroducethesurvey(researchobjective,targetpopulation,questionnaire)

à Askquestionsà Seewhatyoucanlearnfromotherdesigns

Alltogether• ShallwerunoneofthesurveysduringESEM?

15-20 minutes

Session IV

Furtherreading…

• Groves, Fowler, Couper, Lepkowski, Singer and Torangeau, 2009. “Survey Methodology – 2nd edition” John Wiley and Sons

• Conradi R., Li J., Slyngstad O. P. N., Kampenes V. B., Bunse C., Morisio M., Torchiano M. “Reflections on conducting an international survey of CBSE in ICT industry” IEEE 4th International Symposium on Empirical Software Engineering November, 2005

• Linåker, J. Sulaman, S. M., de Mello, R. M. and Höst, M. 2015. Guidelines for Conducting Surveys in Software Engineering. TR 5366801, Lund University Publications. http://lup.lub.lu.se/record/5366801/file/5366839.pdf

• de Mello, R. M. and Travassos, G. H. 2016. Surveys in Software Engineering: Identifying Representative Samples. Proc. of 10th ACM/IEEE ESEM, Ciudad Real.

• de Mello, R. M. 2016. Conceptual Framework for Supporting the Identification of Representative Samples for Surveys in Software Engineering. Doctoral Thesis, COPPE/UFRJ, 138p. http://www.cos.ufrj.br/uploadfile/publicacao/2611.pdf

surveys in software engineering

Engineering

getting started with snap surveys - survey software | online

what is software engineering? software engineering ·...

31031testing1 testingtesting 310313 software engineering...

assoc. prof. dr. ahmet turan Özcerİt. system and software...

cse 308 software engineering software engineering strategies

cs3400 - principles of software engineering - software...

ece450 - software engineering ii1 ece450 – software...

310313 software quality assurance 1 310313 software...

safety critical software engineering safety critical...

310414 software development 1 software development process...

cc20o7n software engineering 1 cc2007n software engineering...

borehole shear-wave surveys for engineering site...

cseb233: fundamentals of software engineering introduction...

310414 software quality assurance 1 310414 software...

requesting engineering surveys final - durban...

engineering surveys and services - missouri department...

software engineering & software performance engineering ·...

software engineering lecture 1 introduction to software...

engineering surveys and services - missouri department...

chapter 18 coastal engineering surveys