software testing in a distributed environment
TRANSCRIPT
![Page 1: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/1.jpg)
Software Testing in a Distributed EnvironmentNeil Langmead
Unrestricted
![Page 2: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/2.jpg)
2
The Agile DevOps World
Many stakeholders Distributed across teams Huge potential for
misunderstanding Lost In Translation…• And this is just English!
![Page 3: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/3.jpg)
3
The Distributed Agile World
Requirement to: Merge in parallel Stop bugs proliferating Testing Early and Often Fix Before the Merge! Time Zero Fixing Fast Desktop Feedback
/main branch
BL130Released to customer
devel: john
/main/task115New loading form
devel: pat
devel: pablo
/main/task114Typo in about form
/main/task116GDI resource leak
This check-in introduces a bug, but only “john” will get affected by it until he merges! He has a chance to fix it before affecting anyone
![Page 4: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/4.jpg)
4
The Importance of Scalable SCM
DSCM systems require as a minimum: Scalability, to 80+ MLOC (e.g. PLM) Robustness, security Efficient branching and merging Automated Workflow, API Rapid Release Cadence
![Page 5: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/5.jpg)
5
Process Transformation- Siemens CT
Strong correlation in net new coding violations ofgood practice (blue)along normalizedcode churn (orange).
Post Perforce, newviolations flatten to zero- why?
![Page 6: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/6.jpg)
6
The Problem of Complexity
Complexity Manager is now an official role!
LOC estimated to increase by 20% in 2016
Multiple build targets, where only a few source files may differ, complicate the testing process
![Page 7: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/7.jpg)
7
The Problem of Technical Debt
![Page 8: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/8.jpg)
8
The Problem of Technical Debt
Like financial debt, difficult to control/ pay back
Accumulates even when we do nothing (e.g. lib versioning)
! The importance of the NULL
Release• How long does it take for your
organisation, if 1 LOC is changed, to up issue to a new version? • Can be weeks, or even
months…
![Page 9: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/9.jpg)
9
How Do We Test?
Need Total Testing approach• Consider FOSS implications• Logic Flow/ Syntax / Semantic Analysis• Architecture• NFR (reliability ,security…)• Unit Testing / Coverage• BVA, Equivalence Partitioning• Fuzzing• Concurrency/ Threading issues• Proliferation through branching
![Page 10: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/10.jpg)
10
How Do We Test?
“Connected Desktop” Developer IDE’s connected
to build server AND SCM Time Zero Approach• Don’t let the bug grow!
On the Fly Analysis• Eclipse, Visual Studio, etc
![Page 11: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/11.jpg)
11
How Do We Test?
![Page 12: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/12.jpg)
12
How Do We Test?
![Page 13: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/13.jpg)
13
Change Based Testing (CBT)
How about we only test what’s changed?
BUT how do we identify changed components?
Tight integration between SCM, Jenkins, JIRA
High levels of Automation Continuous Testing
![Page 14: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/14.jpg)
14
CBT & Invasiveness
• Define & Measure Invasiveness of a defect
Compute transitive closure of a change, to assess impact on the system
Can be done on feature, defect or enhancement level from the SCM repo
![Page 15: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/15.jpg)
15
CBT & Invasiveness
![Page 16: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/16.jpg)
16
CBT – Issue Invasiveness
• Set of Scripts to interrogate the Helix repository
• Selects on IssueID, Author• Cross Referencing Matrix
yields the required change metrics, which is then passed to the unit testing tools
![Page 17: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/17.jpg)
17
CBT – Least Common Ancestor
![Page 18: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/18.jpg)
18
CBT – Least Common Ancestor
![Page 19: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/19.jpg)
19
Insights from Repository Analytics (RA)
Other insights become apparent from Helix RA:Comparison measures contractor productivity (violations of good practice, contributions, change impact) by project & location.
![Page 20: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/20.jpg)
20
Insights from Repository Analytics
Other insights become apparent from Helix RA:Comparison measures contractor productivity (violations of good practice, contributions, change impact) by project & location.
![Page 21: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/21.jpg)
21
Test Driven Development Analysis
Outlying project reveals one developer (ID #1040) who is contributing greatly (i.e. skilled and experienced) but adds few new tests. “1040” needs to on boarded regarding the TDD standard for for greater compliance and project quality.
![Page 22: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/22.jpg)
22
Do we now need a Dynamic Architecture?
Architecture is the hardest part of a SW system to change
Often set early, not very adaptable
Expensive, technically challenging
“Elastic Architecture” methods required• Component based• Flexible API, “Plug and Play”
![Page 23: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/23.jpg)
23
Conclusions
Complexity is here to stay Need advanced, efficient methods to reduce test effort Use Branch Per Task/ Test Per Task Change Based Testing Approach to reduce technical debt Time Zero Testing before the Merge Advanced Repository Analysis to gain additional insights Agility in the Architecture is the Next Frontier (Neil’s opinion…) “Continuous Testing” is here to stay
![Page 24: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/24.jpg)
24
Demonstration
demonstration of the Total Testing Platform, in use at Siemens
Thank you for your attention!
![Page 25: Software Testing in a Distributed Environment](https://reader036.vdocuments.us/reader036/viewer/2022070523/58ed4baa1a28ab27378b46db/html5/thumbnails/25.jpg)