automated testing presentation to epics collaboration meeting 2007 @ desy paul gibbons phd diamond...
TRANSCRIPT
![Page 1: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/1.jpg)
Automated Testing
Presentation to EPICS Collaboration Meeting 2007 @ DESY
Paul Gibbons PhDDiamond Light Source Ltd.
![Page 2: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/2.jpg)
Automated Testing – Why?
• Code Guru– Edit– Compile– Link– Publish
• No testing needed.
(Also turn off all compiler warnings as I never write incorrect code.)
![Page 3: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/3.jpg)
Automated Testing – Why?
• We live in the real world
![Page 4: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/4.jpg)
Automated Testing – Why?
• Others will need to change it.
• Can you trust them not to mess up?
![Page 5: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/5.jpg)
Automated Testing – Why?
Build 0.1 Test 0.1
Develop
Build 0.2
Test 0.2Release 0.2
New Requirement
InitialRequirements
Develop
Build 0.3Test 0.3 Release 0.3
Time
![Page 6: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/6.jpg)
Automated Testing – Why?
• Manual testing is:– Expensive– Time consuming– Error prone.
![Page 7: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/7.jpg)
Automated Testing – Why?
• Automated Testing:– Repeatable. – Running the test requires little human intervention.– Can encapsulate the requirements better than a
requirements document – is always up to date.– Tests not only the code you know you have changed
but also the code you think you have not.– Expensive up front investment.
![Page 8: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/8.jpg)
Automated Testing – Why?
Late night call out to modify Fred’s module. Make modification. Automated tests show no problems. Manually test new feature. Go home confident .
Next day review change with Fred and add test to automated tests.
![Page 9: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/9.jpg)
Automated Testing – Why?
• New requirement calls for re-factoring– Before adding any new feature you gradually
refactor the code. – At each point run automated test.– At end of refactoring you have confidence that
the software still behaves correctly and you are ready to add new features.
![Page 10: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/10.jpg)
Automated Testing – Why?
• Testing on its own is not a panacea.– Different defect-detection techniques
• Reviews, formal/informal design/code inspections, personal desk checking, prototyping, unit test, integration test, system test, beta tests
– Defect Detection Rates:• None above 75%• Testing gives 30-40%
– A combination is required to get > 95%.
![Page 11: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/11.jpg)
Automated Testing – Why?
• Many other advantages of thinking about testing up front.
• See books on Test Driven Development.
![Page 12: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/12.jpg)
Automated Testing – How
Module to Be tested
Test Script
Interface being tested.
Report. OK or Failure?
Environment in which moduleExecutes.
![Page 13: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/13.jpg)
Automated Testing – EPICSTypes of IOC software at Diamond.
Serial Device InterfaceVME Register Interface
Device Independent Records – e.g. Motor
Groups of ‘standard’ records thattogether provide functionality – aperture slits, diffractometer table.
Gensub records with logic built into C function
Frameworks: asyn, stream
Sequences
![Page 14: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/14.jpg)
Automated Testing – EPICS
Current testing is 99% manual.
Normally requires a complete system with hardware.
![Page 15: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/15.jpg)
Automated Testing - EPICS
Option 1.Integration Test.
• Create a complete test system that includes hardware.– The expense precludes this for all software.– Testing is only possible once hardware is
available
Option 2. Isolation – Requires simulation of environment.
![Page 16: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/16.jpg)
Automated Testing - EPICS
You want to test the logic of code.
Use the language of the interface.
A test environment should :– not impose any constraints on the interface– be simple and quick to construct.
![Page 17: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/17.jpg)
Automated Testing – IOC
Logical Units that interface via records:
•Accompanying Test IOC•Channel access script to perform the tests and generate a report.
What about the environment it requires?
Simulation mode to be used in Test IOCs of other modules.
![Page 18: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/18.jpg)
Automated Testing – IOC
C functions that can be built for Linux e.g. genSub systems:
Use CPPUnit
![Page 19: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/19.jpg)
Automated Testing – Device Simulation
Module to Be tested
Device Simulation
Test Script
Interface being tested.
Back doorReport. OK or Failure?
![Page 20: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/20.jpg)
Automated Testing – Device Simulation
Serial( RS232/IP) -
VME Register – simulation device drivers.
• Hardware loopback
• asyn IP client
• Java – COSYLAB
• asynPort that runs Python
Record Simulation Mode
![Page 21: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/21.jpg)
Automated Testing – Plan• Many options. Need to agreed on standards for:
– Test language – Should match interface being tested. For Channel Access use Python or Jython.
– How to build the test.– How to run the test.– Reporting success and failure. xUnit XML format
Provide parsers to convert from other formats
Current work:
PyUnit + serial device simulation + use of record simulation mode
![Page 22: Automated Testing Presentation to EPICS Collaboration Meeting 2007 @ DESY Paul Gibbons PhD Diamond Light Source Ltd](https://reader035.vdocuments.us/reader035/viewer/2022070400/56649f155503460f94c2a3a0/html5/thumbnails/22.jpg)
Automated Testing - References
• Code Complete, Second Edition by Steve McConnell. ISBN-10: 0-7356-1967-0
• Various work by Kent Beck a founder of the Agile Manifesto.