upload ppt by browse button

30
Automated Testing with Team Test Unit, Web, Performance, Load, Manual, and Ordered Tests Code Coverage and Code Analysis

Upload: techweb08

Post on 07-Jul-2015

316 views

Category:

Business


0 download

TRANSCRIPT

Page 1: upload ppt by browse button

Automated Testing with Team Test

Unit, Web, Performance, Load, Manual, and Ordered Tests Code Coverage and Code Analysis

Page 2: upload ppt by browse button

2 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Quality Challenges

““Software bugs, or errors, are so Software bugs, or errors, are so prevalent and so detrimental that they prevalent and so detrimental that they cost the U.S. economy an estimated cost the U.S. economy an estimated $59.5 billion annually, or about 0.6 $59.5 billion annually, or about 0.6 percent of the gross domestic percent of the gross domestic product…an estimated $22.2 billion, product…an estimated $22.2 billion, could be eliminated by an improved could be eliminated by an improved testing infrastructure that enables testing infrastructure that enables earlier and more effective earlier and more effective identification and removal of software identification and removal of software defects.”defects.”

(Source: NIST 2002)(Source: NIST 2002)

Satisfyingusers

Businessinterruption

risks

Ongoingmaintenancecosts rising

Less avail.resources

Page 3: upload ppt by browse button

3 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

How Does VSTS/TFS Help?

Increased Communication and Integration…

Code Analysis Tools

Code Profiling Tools

Unit Testing and Code Coverage

Load Testing

Other Testing Tools

Test Case Management

Page 4: upload ppt by browse button

4 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Unit Testing

Each unit is tested independently

A large percentage of defects are identified during unit testing

Automatic and repeatable code testing

Simplifies integration

Self documenting

Auto-generate

Isolate small portion of code & determine whether it works correctlyIsolate small portion of code & determine whether it works correctly

Page 5: upload ppt by browse button

5 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Benefits of Automated Unit Testing

Repeatable unit that verifies code still works

Confirm code is tested

Build a set of regression tests

Make changes with confidence

Aid in understanding code

Limits: integration, web and Windows UI testing, load, performance testing

Page 6: upload ppt by browse button

6 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Unit Test Types

Class (standard)– Test the properties and methods of a class

Data-driven– Bind unit test parameters to a datasource

ASP.NET– Test classes (or business logic) inside an ASP.NET application – Run in the context of the web server (ASP objects available)

Web Services– Define a web reference to the service

Page 7: upload ppt by browse button

7 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Extensions of Unit Testing

Regression Testing– Modifications are validated with regression tests– Write new code and verify system continues to function

correctly

Integration Testing– Helps validate how tested components interact with one

another– Can identify problems that occur when units are

combined

Scenario Testing– Combine sequence of unit tests to cover a scenario

Page 8: upload ppt by browse button

8 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Writing Effective Unit Tests

Atomic– Should not require others tests to be run first

Cover all cases – One test for each sceneario– Cover all conditions, exceptions, nulls, etc.

Able to re-run without configuration– Database create/read/update/delete without having to

modify the database before or after the test is run

Test a common application state

Unit tests should be …

Page 9: upload ppt by browse button

9 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Unit Test Conditions

Success baseline– Common, successful, most-likely call to your code

Parameter mix– Pass a varied set of parameters– Data-driven

Bounds checking– Stress the upper and lower limits of your parameters

Null values– Determine outcome of passing nulls to parameters

Error conditions– Trigger error conditions and validate expected outcomes

Code coverage scenarios– Test all conditions in your code

Page 10: upload ppt by browse button

10 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Creating Unit Tests (1 of 2)

Attributes are used to denote which methods and classes should be loaded for tests– TestClass – denotes a class for testing– TestMethod – denotes a method for testing

• must return void (Sub for VB.NET) and have no parameters

Certain attributes are associated with initialization– TestInitialize – run before each test– TestCleanup – run after each test– ClassInitialize – run once before running any tests in the class

– ClassCleanup – run once after running all tests in the class

Assert – static type used for asserting test values

ExpectedException – attribute used for determining the exception that should be thrown from the test

Page 11: upload ppt by browse button

11 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Creating Unit Tests (2 of 2)

Methods on the Assert type– AreSame / AreNotSame– AreEqual / AreNotEqual– IsNull / IsNotNull– IsInstanceOfType / IsNotInstanceOfType– IsTrue / IsFalse

Other Assert types– CollectionAssert – collection equivalency

– StringAssert – string comparison/regex matching

Web test types– HtmlDocument – grants access to tags in the html document– Validate*/Extract* - types used for reading field values from the

request and response

Page 12: upload ppt by browse button

Unit TestsUnit Tests

Page 13: upload ppt by browse button

13 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Code Coverage

Improves effectiveness of tests

Show a measurable indication of code that was covered by unit tests– Strive for 75% and higher

Shows the ratio of executed logic to the total logic

Can be used in conjunction with unit and load testing

Helps to find unused code

Determine code that is exercised by tests

Page 14: upload ppt by browse button

Code CoverageCode Coverage

Page 15: upload ppt by browse button

15 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Static Code Analysis

Provides a means to enforce coding standards

Configurable

Reduction in code reviews for small issues

Provides guidance on how to fix

Define a check-in policy to enforce code analysis pass

Coverage includes issues with design, globalization, interoperability, maintainability, naming, performance, reliability, security, and usage

Warning: developers can supress messages with attirbute [SuppressMessage("AdventureWorks.Rules", "AW14441")]

MORE INFO see, “Writing Quality Code” in TFS documentation

Page 16: upload ppt by browse button

Static Code AnalysisStatic Code Analysis

Page 17: upload ppt by browse button

17 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Web Testing

Record page navigation and user interactions with web pages

Run tests back to determine errors

Seed tests from a database

Run tests from a group of users

Validation rules

Extractions

Page 18: upload ppt by browse button

Web TestingWeb Testing

Page 19: upload ppt by browse button

19 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Load Testing

Load Testing: ensure application works under expected, concurrent user load

Create test cases that simulate real user conditions

Distribute tests across cases based on user behavior

Assume standard think times

Distribute across connection types

Simulate an application performance in a production environment

Provide repository to look at a performance trend over time, to see if changes are helping or hurting

Simulate multiple users against an application simultaneouslySimulate multiple users against an application simultaneously

Page 20: upload ppt by browse button

20 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Stress Testing

Stress Testing: determine breaking points in your application

Find scalability issues before the application is deployed

Step (increase) user load over time, monitor

Find breaking point and use to monitor application– Change software

– Add hardware

Page 21: upload ppt by browse button

21 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Performance Testing

Test at single user

Test at normal load

Test at peak load (1.5x normal load)

Measure against expected performance– Peak load should allow for a 20-30% increase in metric

Set thresholds on the output to avoid digging through data

Page 22: upload ppt by browse button

Load, Stress, and Load, Stress, and

Performance TestingPerformance Testing

Page 23: upload ppt by browse button

23 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Manual Test

Tracked like other tests when executing test groups

Presented to the tester to enter and confirm results

Defined as text file or Word document

Page 24: upload ppt by browse button

24 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Generic Test

Wrap existing code and have that code executed as part of the testing process

Centralized results of all tests

Page 25: upload ppt by browse button

25 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Ordered Test

Test to group (and order) other tests

Results of ordered tests either succeed or fail as a group

Used for scenario and module-level testing

Add any test except a load test to an ordered test

Page 26: upload ppt by browse button

Manual TestManual Test

Page 27: upload ppt by browse button

27 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

Test Case Management

Organize tests into lists

Run tests

Filter and group the display of the tests

Import additional tests

Export tests

Organize and manage test casesOrganize and manage test cases

Page 28: upload ppt by browse button

28 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

The Test Lab

Clients– Tester machine– Used to create and edit tests / view results

Servers– Deployed test environment (web servers, load balancer, database server,

etc.)

Agents– Computers running tests (multiple-threads, each a user)– Listen for commands from controller– Execute tests and pass results back to controller

Controller– Central computer that administers agents and collects results

– Distributes load by applying weights to agents

Page 29: upload ppt by browse button

29 For Discussion Purposes Only | © 2007 Computer Enterprises, Inc.

The Test Lab Database

Run against a test version of the database

Test data should be same nature and size as production

Test a steady state of the database

Automate the initialization of the database

Extract data using rules to obfuscate personal information– Visual Studio Database Developer

Page 30: upload ppt by browse button

Test Lab Database InitializationTest Lab Database Initialization