automation from start to finish
DESCRIPTION
a step to step guide to do software test automation on large projects.TRANSCRIPT
Testing Automation From Start to Finish
Liang Gao
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
2
Testing Automation Attribute:
贵expensive
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
3
Testing Automation Attribute:
冰山iceberg
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management
4
Agenda
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Frequently functionality regression testing Sample based performance testing Repetitive work for system and solution
testing
5
Automation is useful for….
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Faster turnaround Reduce human errors Cost saving
6
Frequently functionality regression testing
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
7
Sample based performance testing
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Semi-automation Configure 1000 VPN tunnels Create 10000 routing table entries …..
8
system and solution testing
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management
9
Agenda
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
No matter manual or auto
10
The Goal
Accurate
Testing Report
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management
11
Agenda
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
12
Remote Control Your Products
Refrigerator
Scanner
Tablet
Telecommuter House with Router
Television
Television - Closed-circuit
Truck
Video Camera
Video Phone
Tunnel
Set Top Box (STB)
Printer
Personal Computer Web Browser
Weather Station
Web Server
Woman - End User
Woman - Sitting
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
2 ways of software manual testing (or Device Control) Mouse and Eye (expectation) Keyboard and Eye (expectation)
Automation is to repeat Mouse and Eye – GUI automation Keyboard and Eye – CLI automation
13
Remote Control Your Testing
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Windows based GUI automation AutoIT
Browser based GUI automation QTP WinRunner Silktest AutoIT Selenium
14
GUI Automation
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Expect TCL Expect Perl Expect Python Expect
15
CLI Automation
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Most network devices (network equipment) Management interface: Browser or console
Mobile devices Keyboard simulation Serial port, USB port or Bluetooth
16
Embedded devices State of the Union
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
17
A Sigma Approach (in progress)
Replay keyboard simulation, Screen capture and Image
comparison
Result Verification (manual, only once)
Keyboard simulation, predefined key sequence (XML), render engine and
Screen capture
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
18
Pick the Tool is just beginning万里长征第一步
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management
19
Agenda
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Lessons learned: API (Library) design is key for Productivity Modularity Maintainability Expansion
Don’t start with programming, start with API design
磨刀不误砍柴工 – It is true…
20
API Design
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Design pattern or other OO technique apply Need senior software architect blessing More design brain storming sessions, more
reviews, the better. Spend more time on the API design, it will
pay off in the long run
21
API Design
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
22
API Principle
Control Library
API
Scripts
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Script only contains testing actions, no software independent info
23
API Design - Principle
•Connect to Device
•Issue XXX command to send SMS messages
•Verify messages have been sent successfully
•Verify messages saved in the SMS sent folder
•Delete message
•Verify message has been deleted
Device_id = connect_to_device();
Status = Send_SMS(device_id, $string);
Assert($status);
Status = Get_SMS(sent, $string)
Assert($status)
Status = delete_SMS(sent,$string);
Assert($status)
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
API User Manual
Most important document for script development
API 名称
描述
输入参数 参数 1参数 2参数 3
返回值
使用举例
24
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management
25
Agenda
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
26
Script Development Process
×Ô¶¯»¯½Å±¾¿ª·¢Á÷³Ì²Î¿¼
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
All common development coding standard applies Version control Standard template Standard header 20% of the code is comments Well indented Use API and functions Enough error checking
27
Define Script Quality
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management
28
Agenda
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Testing script has it own quality too Exactly match the manual testing steps Just by reading logs, we know exactly what
scripts are trying to do Device and hardware/software environment
independent Enough debugging message when test case
failed Clean out environment before exit or before
execution 29
Define Script Quality
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
30
Script Quality Checklist
Others Script Quality ChecklistScript Quality
ChecklistSigma
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management
31
Agenda
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Always remember our goal You need at least an execution engine for
Error resuming batch run Report generation
32
Automation Framework ?
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Want to design and implement an Automation Framework?
Checkout
<<Automation Framework: Design and Implementation>>
33
Automation Framework
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management
34
Agenda
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
35
Execution Process
Always remember the goal
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Image upgrade process Parallel execution Total run time Debugging time Report collection and generation
36
Execution Process
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Where to get the image Can you automatically get the image and
upgrade to your product Call this a “clean” procedure
37
Image upgrade process
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
To save total run time – sometimes even automation is too long
This is where the script need to be environment independent
Be aware on the equipment cost issue How many parallel scripts can an engineer
run and manage (also equipments)
38
Parallel execution
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Total run time per engineer Total Debugging time per engineer Total reporting time per engineer
39
Run Time
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
If it is parallel, need a process to parallel report collection and generation
Testing Information Database? Web Interface?
40
Report collection and generation
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Realize when you need automation Always remember the goal Find the way to control your product API design Define the development process Script Quality Do I need an automation framework? Define the execution process Team management Equipment management
41
Agenda
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Who write the scripts? Who write the framework Who execute the scripts Are they happy
42
Team Management
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Whoever write the test plan, write the test scripts Knows test case the best Not professional coder Hard for project management
Professional scripter Need to understand the test case first High productivity Good Quality
43
Who Write Scripts
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Or any other testing related tools If a tester develop a great testing tool, who
maintains it? Who to coordinate the common share library
development Who to develop testing tools if there is a
request Most of the time developer does this. If have resources, think about a Tools Team
44
Who Write Framework
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Whoever write the scripts, execute the scripts Low productivity Bad script portability Unpredictable schedule (bad for scheduled release)
Dedicated resources (professional executer) Streamline resources High productivity Fixed total run time Most modern companies do it this way
45
Who Execute the Scripts
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Test cases writer and professional scripter Scripter and executer
46
Interaction process
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Repeatable test cases
47
Case Writer and Scripter
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Script integration process Newly develop scripts submit to runner Use “State” to manage
Script maintenance process New image change Use “State” to manage
48
Scripter and Runner
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Manual tester, scripter are doing creative things
Script runner may be board Role rotation Clear role for a clear career path
49
Team Building
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
50
Streamlining the team
脚本开发
脚本执行
脚本开发 脚本开发 脚本开发
脚本执行 脚本执行
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
No matter manual or auto
51
The Goal
Accurate
Testing Report