change impact analysisand change-based testing for...
TRANSCRIPT
©VectorSoftware
MichalJablonski3/5/2017
Changeimpactanalysis andchange-basedtestingforembeddedsoftware
©VectorSoftware
AboutVectorSoftware
©VectorSoftware
Organizationsworld-wideuseourtestautomationplatformtoreducethedevelopmenttime,risk,andcostofdeliveringreliable,secure,andcompliantsoftware.
VectorSoftwareMission
©VectorSoftware
Whatdevelopmentchallengesdoesyourorganizationface?
§ Timepressurestoreleasenewproducts/upgradestomarket§ Resistancetochangeinestablisheddevelopmentprocesses§ Prioritizingfeatures/developmentinitiatives§ Lackofinformationsharingwithinthedevelopmentorganization§ Lackofbudget
25% 27% 34% 35%
72%
0% 20% 40% 60% 80% 100%
Budget
Information flow
Proritizing
Resistance to change
Time pressure
Source:2016SoftwareTestingTechnologyReport(VectorSoftware,224responders)
©VectorSoftware
Whatisthemostpressingsoftwaredevelopmentconcernyourorganizationiscurrentlyfacing?
39% 27%
18% 10%
6%
0% 10% 20% 30% 40% 50%
SoftwareQuality
TimetoMarket
Cost
TechnicalDebt
Security
Whatisthemostpressingsoftwaredevelopmentconcernyourorganizationiscurrentlyfacing?
Source:2016SoftwareTestingTechnologyReport(Vector Software,224responders)
©VectorSoftware
CanweallowsamenumberofdefectsperSLOC?
§ „Developingsoftwareforembeddedsystemsisnotgettinganyeasierorlessexpensive.Sincesystemsoffermorefunctions,codesizegrows…checkingthatcodebecomesachallenge.”
Gartner(HypeCycleforEmbeddedSoftwareandSystems)
§ Does#bugsgrowwith#SLOCs?
§ Someotherexamples:• 787Dreamliner– 7MSLOC• AirbusA380– 100MSLOC• FordF150– 150MSLOC
©VectorSoftware
HowtoimproveSoftwareQuality§ CantestingimproveSoftwareQuality?
• Testingdoesnotimprovequalitydirectly• Testingcansetthebenchmarkforyourstateofquality
§ Testing togetherwithgoodmetrics,KPIs andqualitygatescanpreventbugslippage
RequirementsSpecification
Architecture
SystemDesign
Coding
SystemTesting
IntegrationTesting
UnitTesting
©VectorSoftware
Choice:qualityortimetomarket?
©VectorSoftware
Howcanchangebasetestinghelpus?
©VectorSoftware
Howoftentest software?
§ Assoonandasoftenaspossible!
§ Findanybuginearlystage=reducethecostoffixingthebug
§ Howoftenisoftenandcanwedoitevenmoreoften?
*DevOps:Shiftleftwithcontinuoustestingbyusingautomationandvirtualization(IBM;Dibbe Edwards)**HowGoogleTestsSoftware(JamesA.Whittaker,JasonArbon,andJeffCarollo)
TestingPhase ~Cost/Bug**
System Test $5000
IntegrationTest $500
FullBuild $50
UnitTest $5
PhaseoftheS/W Dev Relativecost tofixabug*
Maintenance 100x
Testing 15x
Implementation 6.5x
Design 1
©VectorSoftware
HowVectorCAST testsVectorCAST?
§ Manydifferenttestcampaigns• UnitTesting• IntegrationTesting• SystemTesting• RequirementTesting• ...
§ ThreedifferentOSsupported• Windows,Linux,Solaris
§ 300+compliersconfigurationsupported• IAR,Keil,CodeComposer,Renesas,VisualStudio,....• Runningonhost,simulatorandtargets
§ Resultinginover36Ktestcases.
©VectorSoftware
Software- ChangeProcess§ Softwaredeveloperchangescode
• Bugfixes• Newfeatures• Requirementchanged
§ CodeChangesneedtobetested• Unit,integration,system– testing
§ Executionofalltestcasesinallconfigurationsisimpossible• Codechangeissystemtestedinoneconfiguration• Maybeunittestedinoneconfiguration
§ DuetoruntimeIntegrationtestsareexecutedonlyonceperweekormonth• IssuesfoundinintegrationtestingneedtobetracedbacktotheresponsibleSWdeveloper• ResponsibleSWdeveloperneedstounderstandtheproblemandfixthebug
§ Ifnotallconfigurationsaretestedornointegrationtestisestablishedthecustomerwillfindthebugafterproductrelease
©VectorSoftware
Whatischangebasedtesting?
§ Executetestcasesbasedonsourcecodechanges• Thetestmanagementsystemneedstobeawarewhichcodechanged
• Thiscanbearchieved e.g.usingatimestamporachecksum• Testcaseneedstobeassociatedwithsourcelineofcode
• Thiscanbearchieved e.g.usingcodecoverageanalysis
§ Onlyre-buildtestenvironmentswherethesourcecodechanged• Manyunit-,orintegration– testenvironmentsarenotaffectedbythecodechange
§ Onlyexecutetestcasesthatareaffectedbythecodechanges• Eveninsideonefileonlysomefunctionshavechanged• Inmostcasesyoudon‘thavetoexecutealltestcasesonthatfile
§ Requirementchanged• Failallthetestcaseslinkedwiththechangedrequirement• Requiretheuserto„visit”affectedtestcases.
©VectorSoftware
ChangeBaseTestingsimulation
Sourcecodechange
©VectorSoftware
WhatTestEnvironmentsneedtoberebuilt?
UnitTestsEnvironments
IntegrationTestEnvironments
SystemTestEnvironmentsSourcecode
Files(.c,.cpp,.h)
©VectorSoftware
WhatTestCasesneedtobeexecuted
UnitTestEnvironments
Integration TestEnvironments
SystemTestEnvironmentsSourcecode
Files(.c,.cpp,.h)
©VectorSoftware
WhatTestCasesneedtobeexecuted
UnitTestEnvironments
Integration TestEnvironments
SystemTestEnvironmentsSourcecode
Files(.c,.cpp,.h)
©VectorSoftware
RegressionTestinginallVariants...
§ ...isnowpossible§ Reducednumberofenvironmentsthathavetobebuilt
§ Reducednumberoftestcases
§ Changebasedtestselectionhastobedoneineachconfiguration
Sourcecode
Files(.c,.cpp,.h)
©VectorSoftware
ChangeBasedTestingbenefits
§ Reducedtestexecutiontime§ Allowsthedevelopertoexecuteall!(necessary)testcasesbefore
check-in§ Nointegrationtesterrorsthatshowupafterweeks§ Improvementofinfrastructureutilization§ Improvementoftimetomarket§ Qualityimprovements
Time
Cost
©VectorSoftware
QuestionsandAnswers
!?
©VectorSoftware
ConnectWithUs
Americas Europe, MiddleEast,andAfrica Asia-PacificandRussia1351SouthCountyTrailEastGreenwich,RI02818UnitedStatesofAmerica
8DuncannonStreetLondonWC2N4JFUnitedKingdom
St.Töniser Str.2a47906KempenGermany
222-0033WiseNext3F2-5-14Shin-Yokohama,Kouhoku-kuYokohama,Kanagawa,Japan
[email protected]+14013987185
[email protected]+442036030120
[email protected]+4921528088808
[email protected]:+861084184600India:+918026583300Japan:+81452859387Singapore:+6531582718