Using Docker/Software Containers for Automated Testing
Pini Reznik
Software Testing Checklist
1 Create System and Acceptance Tests [ ] 2 Start Acceptance test Creation [ ] 3 Identify test team [ ] 4 Create Workplan [ ] 5 Create test Approach [ ] 6 Link Acceptance Criteria and Requirements to form the basis of acceptance test [ ]
7 Use subset of system test cases to form requirements portion of acceptance test [ ] 8 Create scripts for use by the customer to demonstrate that the system meets requirements [ ] 9 Create test schedule. Include people and all other resources. [ ] 10 Conduct Acceptance Test [ ] 11 Start System Test Creation [ ] 12 Identify test team members [ ] 13 Create Workplan [ ] 14 Determine resource requirements [ ] 15 Identify productivity tools for testing [ ] 16 Determine data requirements [ ] 17 Reach agreement with data center [ ] 18 Create test Approach [ ] 19 Identify any facilities that are needed [ ] 20 Obtain and review existing test material [ ] 21 Create inventory of test items [ ] 22 Identify Design states, conditions, processes, and procedures [ ] 23 Determine the need for Code based (white box) testing. Identify conditions. [ ] 24 Identify all functional requirements [ ] 25 End inventory creation [ ] 26 Start test case creation [ ] 27 Create test cases based on inventory of test items [ ] 28 Identify logical groups of business function for new sysyem [ ] 29 Divide test cases into functional groups traced to test item inventory [ ] 1.30 Design data sets to correspond to test cases [ ] 31 End test case creation [ ] 32 Review business functions, test cases, and data sets with users [ ] 33 Get signoff on test design from Project leader and QA [ ] 34 End Test Design [ ] 35 Begin test Preparation [ ] 36 Obtain test support resources [ ] 37 Outline expected results for each test case [ ] 38 Obtain test data. Validate and trace to test cases [ ] 39 Prepare detailed test scripts for each test case [ ] 40 Prepare & document environmental set up procedures. Include back up and recovery plans [ ] 41 End Test Preparation phase [ ] 42 Conduct System Test [ ] 43 Execute test scripts [ ] 44 Compare actual result to expected [ ] 45 Document discrepancies and create problem report [ ] 46 Prepare maintenance phase input [ ]
47 Re-execute test group after problem repairs [ ] 48 Create final test report, include known bugs list [ ] 49 Obtain formal signoff [ ]
From http://www.softwaretestinghelp.com/testing-checklist/
Software Testing Checklist
1 Create System and Acceptance Tests [ ] 2 Start Acceptance test Creation [ ] 3 Identify test team [ ] 4 Create Workplan [ ] 5 Create test Approach [ ] 6 Link Acceptance Criteria and Requirements to form the basis of acceptance test [ ]
7 Use subset of system test cases to form requirements portion of acceptance test [ ] 8 Create scripts for use by the customer to demonstrate that the system meets requirements [ ] 9 Create test schedule. Include people and all other resources. [ ] 10 Conduct Acceptance Test [ ] 11 Start System Test Creation [ ] 12 Identify test team members [ ] 13 Create Workplan [ ] 14 Determine resource requirements [ ] 15 Identify productivity tools for testing [ ] 16 Determine data requirements [ ] 17 Reach agreement with data center [ ] 18 Create test Approach [ ] 19 Identify any facilities that are needed [ ] 20 Obtain and review existing test material [ ] 21 Create inventory of test items [ ] 22 Identify Design states, conditions, processes, and procedures [ ] 23 Determine the need for Code based (white box) testing. Identify conditions. [ ] 24 Identify all functional requirements [ ] 25 End inventory creation [ ] 26 Start test case creation [ ] 27 Create test cases based on inventory of test items [ ] 28 Identify logical groups of business function for new sysyem [ ] 29 Divide test cases into functional groups traced to test item inventory [ ] 1.30 Design data sets to correspond to test cases [ ] 31 End test case creation [ ] 32 Review business functions, test cases, and data sets with users [ ] 33 Get signoff on test design from Project leader and QA [ ] 34 End Test Design [ ] 35 Begin test Preparation [ ] 36 Obtain test support resources [ ] 37 Outline expected results for each test case [ ] 38 Obtain test data. Validate and trace to test cases [ ] 39 Prepare detailed test scripts for each test case [ ] 40 Prepare & document environmental set up procedures. Include back up and recovery plans [ ] 41 End Test Preparation phase [ ] 42 Conduct System Test [ ] 43 Execute test scripts [ ] 44 Compare actual result to expected [ ] 45 Document discrepancies and create problem report [ ] 46 Prepare maintenance phase input [ ]
47 Re-execute test group after problem repairs [ ] 48 Create final test report, include known bugs list [ ] 49 Obtain formal signoff [ ]
From http://www.softwaretestinghelp.com/testing-checklist/
Waves of Innovation
Testing Complex Applications
6
Cargo Delivery Pipeline
Shipping Goods
8
Shipping with containers
9
Scalability with Containers
Start a “VM” in milliseconds by removing OS.
VM Container
What is Docker?
Batteries Included but Replaceable
• Docker Machine• Docker Compose• Docker Swarm• Notary• Kitematic• Docker plugins• runC and Open Container Project• Docker Trusted Registry (Enterprise Hub)
Containers Eco-system
• Orchestration• Networking• Persistent Storage• Minimal OS• Containers State• More ...
Orchestration - ApacheMesos
13
Orchestration - Kubernetes
Networking for Containers
15
Persistent Storage
Minimal OS
State of Containers
18
Software Testing Challenges
• Creating clean environments• Multi-machine Setup• Test data injections• Environment tear down• Environment freezing• Testing deployments/upgrades• More ...
Creating Clean Environments
$ time docker run busybox echo hellohello
real 0m0.279suser 0m0.012ssys0m0.004s
Multi-machine Setups
docker-compose.yml
$ docker run -d -v /var/voume_with_small_test_dataset:/data my_image
$ docker run -d -v /var/voume_with_large_test_dataset:/data my_image
$ docker run -d -v /var/voume_with_prod_dataset:/data my_image
Test Data Injection
Dependencies Injection
zookeeper:image: jplock/zookeeper:3.4.5
master:image: redjack/mesos-
master:0.21.0links:- zookeeper:zookeeper
zookeeper:image:
jplock/zookeeper:4.0.snapshot
master:image: redjack/mesos-
master:0.21.0links:- zookeeper:zookeeper
Tear Down Environments
$ time docker kill 78239d7854ed78239d7854ed
real 0m0.377suser 0m0.008ssys0m0.008s
Freezing Test Environments
- Multiple test environments- Hand over from testers to developers- Sharing environments- store broken containers as images- store state with CRIU
Testing Deployments/Upgrades
Bottom Line - Benefits for Testing
• Unittest– New development environment in a container for each
build with tests in milliseconds
• System tests– Fully functional runtime environment in a second
• End-to-end tests– Multi-machine complex systems in couple of seconds
The Future? Cloud Native Applications.
Pre-DevOps
DevOps
Post-DevOps
Future - Cloud Native Applications
Shipped
Mantl
Future - Software Supply Chains
Demo
Full CD Setup in 30 seconds
Questions?