foxpro user group world tour testing computer software mike stewart microsoft mike stewart microsoft

FoxPro User Group World TourFoxPro User Group World Tour

Testing Computer SoftwareTesting Computer Software

Mike StewartMicrosoft

Mike StewartMicrosoft

FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002

Who Am I and Why Should You Care?Who Am I and Why Should You Care?

Software Test Lead for the Visual FoxPro team Numerous Knowledge Base articles and white

papers for Microsoft Consultant developing custom applications

using VFP, C++, Delphi and VB Unix sysadmin and systems programmer

using C++


What this Session is NotWhat this Session is Not

A lot of code examples – no example can fit every situation

Testing the “Microsoft way” – this is all based on software QA industry methodologies

One size fits all – everyone’s testing needs are different

For people with aversions to blue Power Point slides


Why Test?Why Test?

Customer satisfaction – testers are customer advocates

Cost – the later a bug is found, the more it costs to fix

Quality – the earlier a bug is found, the more likely it is to be fixed


What is the goal of testing?What is the goal of testing?

Find bugs – more specifically, get bugs fixed Verify that it works Prevent errors before they get into the product Provide information on risk – how solid is the

product?


Important Things to RememberImportant Things to Remember

“A test that reveals a problem is a success. A test that did not reveal a problem was a waste of time.” - Kaner

You will not find all of the bugs Myers’ 20 line program: 100 trillion paths Program takes two inputs of four-digit

number: almost 400 million possibilities Test something 10,000 iterations: what if it

fails at 10K + 1?


Exhaustive TestingExhaustive Testing

“The only exhaustive testingthere is, is so much testing that the tester is exhausted!” — Hetzel


“White Box” vs. “Black Box” Testing“White Box” vs. “Black Box” Testing

White box testing treats the software as a box that you can see into.

You are looking into the box to see how it works

Assumes you have source code Black box testing treats the software as a box you

cannot see into. Put in data (input) and see what comes out of

the box (output). No source code needed.

Black box testing on finds about 55% of the bugs


Why White Box Testing isn’t CoveredWhy White Box Testing isn’t Covered

Doesn’t test the software in the manner in which it will be used

It doesn’t do spec testing – it has no way of knowing if it is functioning as designed or not

It does not find missing code – some estimate this accounts for 30% of bugs

Not enough time! It’s a discipline of its own, and we’d be here all day.


Steps in testingSteps in testing

Test planning Write test plans Spec and code reviews

Build verification – is it even stable enough to test?

Function and system testing – run the full suite of test cases

Media verification No viruses Bits on the media are the release bits The product can be installed from the media


Essential Testing IssuesEssential Testing Issues

Establish a bug tracking system ATS Web – free at http://msdn.microsoft.com

/vstudio No sticky notes

Establish processes Use cases Test plans Test cases Bug entry and resolution

Automate, automate, automate!


Bug TrackingBug Tracking

Accountability Assures that the bug does not fall through the

cracks When closed, we are assured that it is fixed.

Metrics How great is the severity of bugs found? How fast are we closing bugs? Are we ready to ship?

Liability CYA – cover yourself against legal liability (I

am not a lawyer, however)


Test PlansTest Plans

A test plan is a general “plan of attack”, not a detailed list of test cases.

List areas to test, and how they will be tested. Notepad File/Open example

“Notepad’s File/Open functionality will be tested for various file types, file size limits, file name limits and regional encoding. Various media need to be considered, such as removable media and networks.”


Test CasesTest Cases

Each task can be compartmentalized into separate automation programs, or conglomerated into one big program.

Keep test cases atomized: test one particular feature at a time.

Create a matrix, and build test cases from that Could be something as simple as an Excel

spreadsheet

Example: the Notepad File/Open dialog


Test Cases for File/Open/FilenameTest Cases for File/Open/Filename

Long file names – boundary testing Single letter file names – boundary testing Long extensions – boundary testing Edit box text length – boundary testing Invalid characters in file names – functional Valid file names – functional Wild card characters – functional Edit keys (Ctrl-C, etc.) – functional or spec Paste in binary from clipboard - pathological MRU dropdown – functional Path completion – functional Large fonts – functional Gain and lose focus – functional High ASCII chars - functional


Bug Entry and ResolutionBug Entry and Resolution

Minimum to track: Severity Priority The tester who found it The dev who fixed it

Enter good bugs: The easier to reproduce, the more likely it is to

get fixed Check on a second machine – could be a

configuration/OS issue If it only happens on one OS, say so


Bug Entry and ResolutionBug Entry and Resolution

If you’re org is tracking metrics, close resolved bugs as quickly as possible

Set severity appropriately, and leave the priority to the project manager(s)

Your org will have to decide how to set severity and priority – everyone has different needs


Testing MethodologiesTesting Methodologies

Smoke testing – basic functionality Exploratory testing

Testing while learning the product Misses large areas of the product

Boundary conditions Cannot test every value, so test for the most

important values Hex boundaries and the boundary of field

length Boundary -1, boundary, and boundary +1 Also test largest value allowed, if it is not a

hex value



Stress What happens when a large number of users

hit it? Low memory, low disk space, large amounts

of data, repetitive Bugs can often not be reproduced Web Application Stress Tool is great for web

apps.



Functional Does it do what the spec says it should do? How does it handle error conditions?

Performance A discipline unto itself, worthy of an entire

session Regression testing

Make sure nothing got broken Does not find a lot of bugs



Code Coverage – for VFP, use the Coverage Profiler

Code coverage testing tells you what code has not been hit, and therefore needs to have test cases that hit this code.

You are unlikely to get 100% coverage The VFP Coverage Profiler can also reveal

bottlenecks.



Race conditions Two pieces of code are colliding Especially important in multi-threaded and

multi-user apps Error conditions - does it fail gracefully? UI

Tab order is easy to miss if you are a mouse-addicted developer

When testing as a dev, do not use the mouse, because you know the mouse will work.



Ad Hoc testing – “guerrilla testing” Great for checking basic functionality Can give broader testing than more formal

means Hard to verify coverage


AutomationAutomation

The Active Accessibility test harness included with VFP 7.0

Other automation products: Rational – Visual Test Mercury Interactive – Testing suite with

various products for web and thick client applications


Automation TipsAutomation Tips

The test should be self-documenting Do not use “pass” or “fail” as the test result Test should say what it expected and what it

received, and whether it passed or failed. Write “drivers” or hooks into your software if

needed – for example, VFP’s Active Accessibility or Project Manager hooks

Tests should be self-sustaining – create your own data, and clean up. Expect nothing.


When Do You Ship?When Do You Ship?

You will ship with bugs if your testing is adequate.

Define show-stoppers Will customers find the bug? Does it crash your product? Will only pathologic testers find it?

Define risk Fixing a bug has a 60% risk of introducing new

bugs When the show-stoppers are fixed, ship


Selling Testing to ClientSelling Testing to Client

Bulk contracts – sell the job, not the hours to develop it, and the hours to test it.

Once the software is in the customer’s hands, bugs are much more expensive to fix

How much does it cost to have 30 users find a bug vs. a qualified, dedicated test effort at $x/hour, that assures quality?

Other ideas?


“Things They Should Teach in the Testing Stuff Class, but Don’t”

“Things They Should Teach in the Testing Stuff Class, but Don’t”

Run a high-color scheme like Marine High-Color

Use large fonts (Display/Settings/Advanced) Throw away your mouse


More informationMore information

Testing Computer Software – Kaner The Art of Software Testing - Myer Software Quality Testing Engineering

magazine: http://www.stickyminds.com STAR Conference – the DevCon for

professional software testers (www.stqe.com)


ADVISOR DEVCON Web Update PageADVISOR DEVCON Web Update Page

http://www.Advisor.com/CMF0109update

This session WILL have updates.


Thank you!

Please remember to fill out your evaluation.

foxpro user group world tour testing computer software mike stewart microsoft mike stewart microsoft

Documents

testing test planning

system testing

exhaustive testing

goal of testing

white box testing isnt

media slide

source code black box

c slide