ready to automate? - prismnet.comwazmo/papers/ready_to... · nis it really vb (or vba)? ... ready...

Ready to Automate? 10/31/2001

1Copyright © 2001 Bret Pettichord

1

Ready to Automate?

Bret [email protected]

Copyright © 2001 Bret Pettichord

2

Ready to Automate?

1. Automation isn’t a label for other kinds of problems.



3

Ready to Automate?

1. Automation isn’t a label for other kinds of problems.2. Testers aren’t trying to use automation to

prove their prowess.

4

Monitoring and Logging

Diagnostic features can allow you ton View history and details of transactionsn View the state of running methods, threads or

processes n Attach a debugger and get a stack tracen Get reports of unusual behavior

Learn about the logging available from web servers, databases and other standard system components



5

Diagnostic FeaturesThese make the software behavior more visible.n Assertions. These logical statements in the code make

assumptions explicit. If false, there must be a bug. Typically assertion checking is only done during testing and debugging.

n Event triggers. Provide identifying signals when tasks begin and end.

n Database Integrity Checks. Ongoing tests for database corruption, making corruption quickly visible to the tester.

n Code Integrity Checks. Quick check to see whether code has been overwritten.

n Memory Integrity Checks. Check for wild pointers, other corruption.

n Access methods. Special access methods allow viewing component state.

6

Ready to Automate?

1. Automation isn’t a label for other kinds of problems.2. Testers aren’t trying to use automation to prove

their prowess.3. Testability features can be added to the

product code.



7

Varieties of Vendorscript

C-like languagesn Look like C, except without pointersn Standard idioms aren’t supportedVisual Basic-like languagesn Is it really VB (or VBA)?Hybrid languagesn How will you learn it?n How can you tell if it’s a language bug?

8

Ready to Automate?


their prowess.3. Testability features can be added to the product

code.4. Testers and developers work cooperatively

and with mutual respect.



9

Ready to Automate?



code.4. Testers and developers work cooperatively and with

mutual respect.5. Automation is developed on an iterative

basis.

10

Essential Capabilities

Maintainabilityn Will the tests still run after product design changes?n Can tests be easily updated for next release?

Integrityn Can your test results be trusted?n Do you get lots of false alarms?n Are you sure that failed tests always appear in the test

results?

Reliabilitityn Can you rely on the test suite to actually do some testing

when you really need it?n Will it run on all the platforms and configurations you need

to test?



11

Ready to Automate?



code.4. Testers and developers work cooperatively and with

mutual respect.5. Automation is developed on an iterative basis.6. You understand your requirements for

automation.

12

Test automation approaches

Unit integration testingn Test small units in the native programming

language

API testingn Test the product via an application programming

interface

GUI test automationn Use a GUI tool to test via the product user

interface



13

Unit integration testingWhat is a unit?n A class, function or proceduren Typically the work product of a single developer

Allow calls to classes, functions and components that the unit requires

Create stubs or simulators for units being depended on

Typically cheaper and easier

Unit integration testingTest units in context

Expensive and often difficult

Unit isolation testingTest each unit in isolation

14

API testing

Does your product have an API?n Askn Might be undocumentedn It may be cheaper to create or expose one

for testing than to build GUI test automation infrastructure



15

Test Interfaces

Interfaces may be provided specifically for testing.n Exceln Xconq

Existing interfaces may be able to support significant testing.n InstallShieldn Autocadn Interleafn Tivolin Any defined client/server

interfaceAny interface is easier to automate than a GUI.

16

Test Interfaces

Which interface will be easier to test?

Product Under Test

DomainCode

PresentationCode

Pro

gram

min

g In

terf

ace

Use

r In

terf

ace

Pro

gram

min

g In

terf

ace

TestAutomation

Libraries



17

Comparison of Approaches

Must have tool license to run

Anyone can runAnyone can runFlexibility

Requires tool training

Need to understand API

Tests are in same language as product

Training required

GUI test tools (purchase)

Scripting test harnesses (public domain)

Unit test harnesses (public domain)

Tool support required

Testers + automation experts

Programmer/ users

DevelopersWho can do it?

GUIAPIUnit

18

Dimensions of Automation

Interfacesn Unit, API, GUI…

Risksn Regression, Load, Concurrency, Robustness…

Test Creationn By tester/programmers, by non-programmers, by programs

and test models…

Evaluationn Compare with prior results, automated oracles, assertions

and flight recorders…

Coveragen Instrumenting the code to understand what’s been tested.



19

Ready to Automate?

1. Automation isn’t a label for other kinds of problems.2. Testers aren’t trying to use automation to prove their prowess.3. Testability features can be added to the product code.4. Testers and developers work cooperatively and with mutual

respect.5. Automation is developed on an iterative basis.6. You understand your requirements for automation.7. You are open to different concepts of what test

automation can mean.

20

Automation Architecture

Staff Skills System

UnderTest

TestRequirements

AutomationArchitecture



21

Opportunities for Automation

Platform Setup and ResetPre-load Database with Testbed DataSmoke TestsRegression TestingConfiguration and Multi-platform TestingLoad TestingRandomized Testing

Code Coverage MeasurementMemory Leak and other specialized testingTest compliance with interface standardsCollect performance metricsConfirm pre-defined release criteria

22

Staffing Models

Domain Experts + AutomatorsTester/ProgrammersJunior ProgrammersCentral Automation TeamSpare-time AutomationManual Testers with ToolsTest Expert + Warm Bodies



23

Ready to Automate?

1. Automation isn’t a label for other kinds of problems.2. Testers aren’t trying to use automation to prove their prowess.3. Testability features can be added to the product code.4. Testers and developers work cooperatively and with mutual

respect.5. Automation is developed on an iterative basis.6. You understand your requirements for automation.7. You are open to different concepts of what test automation

can mean.8. Test automation is lead by someone with an

understanding of programming and testing.

24

Questions?

For more informationn Software Testing Hotlist

www.testinghotlist.comn Lessons Learned in Software Testing

Cem Kaner, James Bach, Bret Pettichordwww.testinglessons.com

n Bret [email protected]

ready to automate? - prismnet.comwazmo/papers/ready_to... · nis it really vb (or vba)? ... ready...

Documents