Lessons Learned on Automation
Liang Gao ([email protected])
Lesson -1 Automation Cost Money
You need people to write scripts You need testbed to run those scripts
(lots of equipments) You need people to monitor those test
bed runs, and debug running failures. You need people to maintain those
scripts to reflect product change Can you justify all those to your upper
management?
You might if you
Show some proof in the front. Can go to your boss and say: I can
reduce major release cycle from 8 month to 5 month if I do this.
Can go to your boss and say: I can reduce customer found bugs down by 20% if I do this.
Lesson 2: Automated Testing is Dangerous
Once it is automated, chances are, it will not be manually executed anymore.
No exploratory testing can be done If your script has problem, and can not
catch bugs (output is pass even it should be fail), it will be going into darkness for ever
You may lose your chance to catch bugs for ever and you don’t know.
Lesson 3: No Framework, No Automation
If you have one couple of hundred scripts in hand, you might be fine without a framework.
You need one if you have thousands of scripts to manage. How to know which test case passed/failed? Where to get a decent running report. How to group test cases easily. How to debug
You need one if there are many people develop scripts in parallel.
Lesson 4: Use Standard Script Language
VB/TCL/Perl/Python/Ruby Customized scripts need learning
period Customized script language – Who
maintain it? No community support Hard to communicate with others –
developers, other test outside the company
Lesson 5: Separate writer and runner
Engineer should not develop script and then execute script. Script execution is a dedicated job.
Debug takes time Test bed problem? Script problem? Image regression bug?
Script integration takes time. Script execution should be a 24/7
factory, should be a machine. Script is just a by-product, it is the full version report that you want
Lesson 6: Test Bed Independent is very Important
Separate the writer and runner requires the script should be testbed independent.
Script should be able to run from testbed to testbed with minimum change to the test framework configuration, not to script itself.
Hard coded router/switch names, IP addresses, interface names are not good when switch testbeds
A handover process is needed between the writer and runner.
Develop a “script checker” tool to check the hard coded values in the script as an acceptance criteria.
Lesson 7: Manage Your Scripts the Same as Your Bugs
Script need to have states like bugs (S) – Submitted: Script is submitted to the
regression team (A) – Assigned: Script is assigned to a
regression engineer (I) – Integration: regression engineer is
putting it on the regression test bed. (P) – Production: Script is in the regression
testbed and will be run on regression testing against release
(F) - Feedback needed: Script has errors, more feedback from writer needed.
Lesson 8: Design You Scripts As Data Driven.
Script need to be data driven Different data means different test cases.
Test_case.tcl {router1 eth0} {router2 eth0} {router3 eth0 eth1} {router4 eth0 eth1 eth2} {mode 1} {phase 1} {traffic 1 speed}Can test all combinations of Mode, Phase and Traffic with one single script.
Data generation can be automated too. Can catch more bugs when vary the data
Lesson 9: Log is More Important Than Script
When fail, most of the time we only look at the logs, not scripts for debugging.
Read log like read a book More debugging info dumped when
fail.
Lesson 10: test case designer and automator separate?
Don’t use automator who doesn’t respect testing
Don’t use automator who doesn’t understand testing
C company use same tester to design, manual execute and automate the test cases. And so is J company
Lesson 11: User Standard Testbed
DUT2 DUT3 DUT4
VLAN1
VLAN2
VLAN3
VLAN4
Four Ethernet on each of the Device Under Test
1 2 3 4 1 2 3 4 1 2 3 4
被测设备 2
DUT11 2 3 4
Lesson 11: User Standard Testbed
DUT1 DUT2 DUT3 DUT4
Ethernet1 Open Open Shutdown Shutdown
Ethernet 2 Open Shutdown Open Shutdown
Ethernet 3 Shutdown Open Open Shutdown
Ethernet 4 Shutdown Shutdown Shutdown Shutdown
Logical Topology:
DUT2
VLAN1VLAN2
VLAN3
DUT1
DUT3
Lesson 12: Dynamic script allocation
Different release has different feature set
Each regression run need to pick different scripts to run
Tricks for dynamic script allocation Static file for version matching Database
Lesson 13: Use Database to keep automation result history
Database for full version report Database for regression bug
debugging
Lesson 14: Script maintenance – version per branch
Certify scripts for each new release Keep them in a separate code branch
Version control
Version 1.0, Dec 24th, 2007, Liang Gao Version 1.1 March 2008, Liang Gao