enabling better software through better testing jason anderson, microsoft jasand@microsoft.com
Post on 23-Dec-2015
220 Views
Preview:
TRANSCRIPT
Enabling Better Software through Better Testing
Jason Anderson, Microsoftjasand@microsoft.com
Agenda
• Team System Overview• Test Authoring & Execution
● Manual Testing● Web Testing● Automated \ Unit Testing● Load Testing
• Test Case Management• Code Coverage• Test Extensibility• Team Tester and Team Foundation Server• Questions
Development Teams
InfrastructureInfrastructureArchitectArchitect
SolutionSolutionArchitectArchitect
Project ManagerProject Manager
DeveloperDeveloper
TesterTester
End UserEnd User
Visual Studio Team SystemPro
cess
an
d A
rch
itect
ure
Pro
cess
an
d A
rch
itect
ure
G
uid
an
ceG
uid
an
ce
Vis
ual S
tud
io In
du
stry
V
isu
al S
tud
io In
du
stry
Part
ners
Part
ners
Dynamic Code Analyzer
Visual Studio
Team Architect
Static Code Analyzer
Code Profiler
Unit Testing
Code Coverage
Visio and UML Modeling
Team Foundation Client (includes CAL)
Visual Studio Professional Edition
Load Testing
Manual Testing
Test Case Management
Application Designer
Logical Infra. Designer
Deployment Designer
Visual Studio
Team DeveloperVisual Studio
Team TestLoad Testing
Manual Testing
Test Case Management
Change Management
Work Item Tracking
Reporting
Project Site
Visual Studio
Team Foundation Integration Services
Project ManagementBig Build
Class Designer
Test AuthoringTest Authoring
QA SKU UI: Authoring Tests
• Design Principal: Enable authoring features specific to testing without impeding standard development experience
• Primary UI Elements● Solution Explorer● Test View
• Largest Real-estate reserved for code● Result details can share space via tab selection
with explicit user gesture
Types Explained
Visual Visual Studio Studio IDEIDE
EDT QA ToolsEDT QA Tools
ManualManual
UnitUnit
GenericGeneric
WebWeb
LoadLoad
ASPASP
Ordered Ordered 33rdrd Party Party
Test ExecutionTest Execution
QA SKU UI: Execution
• Design Principal: Provide rich environment for management and control of test cases
• Primary UI Elements● Test Manager● Command Line Parser● Results Window
• Primary user scenario is not test author, rather to manage and execute tests
• Local or Remote Execution● Distributed Across Machines● With Machine Constraints
Writing and Writing and Executing TestsExecuting Tests
Test Case ManagementTest Case Management
Test Case Management
• Fully Integrated with Visual Studio• Uses a local file that can be checked into source code control
and shared● Can be used without the project system● Versioning
• Test Lists to managed Test Cases● Tests in one or more Test Lists● Test Lists can be hierarchical● Can be used from the Command Line
Test CaseTest CaseManagementManagement
Code CoverageCode Coverage
Code Coverage
• Test Authoring isn’t about quantity, it’s about quality
• Code coverage helps you monitor your tests effectiveness● Team Members can analyze results at a
high-level● Can also analyze source for specific
missed methods and branches
Code Coverage
void PurchaseItem(int itemID)
{
if (itemID == 0) {
throw new Exception();
} else
{ ProcessOrder(itemID);
}
}
UnitTest()
{
PurchaseItem(1);
}
Code Coverage
void PurchaseItem(int itemID)
{
if (itemID == 0) {
throw new Exception();
} else
{ ProcessOrder(itemID);
}
}
UnitTest()
{
PurchaseItem(1);
}
Code Coverage
void PurchaseItem(int itemID)
{
if (itemID == 0) {
throw new Exception();
} else
{ ProcessOrder(itemID);
}
}
UnitTest()
{
PurchaseItem(1);
}
Code Coverage
void PurchaseItem(int itemID)
{
if (itemID == 0) {
throw new Exception();
} else
{ ProcessOrder(itemID);
}
}
UnitTest()
{
PurchaseItem(1);
}
Code Coverage
void PurchaseItem(int itemID)
{
if (itemID == 0) {
throw new Exception();
} else
{ ProcessOrder(itemID);
}
}
UnitTest()
{
PurchaseItem(1);
}
Code Coverage
void PurchaseItem(int itemID)
{
if (itemID == 0) {
throw new Exception();
} else
{ ProcessOrder(itemID);
}
}
UnitTest()
{
PurchaseItem(1);
}
Code CoverageCode Coverage
Test ExtensibilityTest Extensibility
Why Test Extensibility
• Must not lose investment in current infrastructure
• Testing requirements are diverse• Integration beyond Microsoft
Extensibility Story
• Generic Tests● Requires minimal integration work● Limitations to Integration with VS
• Extended Test Types● Custom type for full Test Type extensibility● Means of adding authoring, execution, and
reporting extensibility● Custom types are 1st class test citizens
Generic TestsHow execution works
GenericGenericTestTestTypeType Existing TestExisting Test
MetadataMetadata
Visual Visual Studio Studio IDEIDE
ExecutionExecutionProcessProcess
YourYourTestTestProcessProcess
GenericGenericTestTestTypeType Existing TestExisting Test
MetadataMetadata
Generic TestsHow execution works
Visual Visual Studio Studio IDEIDE
ExecutionExecutionProcessProcess
YourYourTestTestProcessProcess
YourYourTestTest
YourYourTestTest
YourYourTestTest
Test Type ExtensibilityExtended Types
• Test Environment can be extended to include your test tools
• Authoring, Execution, Results plug-ins• Offers first class test type• Used internally to create test types
Test Type Extensibility FrameworkTest Type Extensibility Framework
MS Test TypesMS Test Types VSIP Test TypesVSIP Test Types
Test Type Extensibility How execution works
Visual Visual Studio Studio IDEIDE
ExecutionExecutionProcessProcess
Test AdaptorTest Adaptor
TestTest
Test UpdatesTest Updates
ResultsResults
Details ViewerDetails Viewer
Run ConfigRun ConfigSettingsSettings
Test View
Your Test Your Test SourceSource
Your Test Your Test SourceSource
Your Test Your Test CaseCase
Your Test Your Test CaseCase
Your Test Your Test Result Result
SummarySummary
Your Test Your Test Result Result
SummarySummary
Your Your DetailsDetailsYour Your
DetailsDetails
Your Test Your Test CaseCase
Your Test Your Test CaseCase
Test Explorer
Custom Custom Editor / WizardEditor / Wizard
Test ItemsTest ItemsManagement UIManagement UI
GeneralGeneralResults ViewResults View
Custom DetailedCustom DetailedResults ViewResults View
GenericsGenerics Test TypeTest Type
Hook into RunHook into Run Config SettingsConfig Settings
Context-sensitive Context-sensitive Right-click CommandsRight-click Commands
Complete control ofComplete control ofTest item executionTest item execution
Team Tester and Team Tester and Team Foundation ServerTeam Foundation Server
Visual Studio Team SystemPro
cess
an
d A
rch
itect
ure
Pro
cess
an
d A
rch
itect
ure
G
uid
an
ceG
uid
an
ce
Vis
ual S
tud
io In
du
stry
V
isu
al S
tud
io In
du
stry
Part
ners
Part
ners
Dynamic Code Analyzer
Visual Studio
Team Architect
Static Code Analyzer
Code Profiler
Unit Testing
Code Coverage
Visio and UML Modeling
Team Foundation Client (includes CAL)
Visual Studio Professional Edition
Load Testing
Manual Testing
Test Case Management
Application Designer
Logical Infra. Designer
Deployment Designer
Visual Studio
Team DeveloperVisual Studio
Team TestLoad Testing
Manual Testing
Test Case Management
Change Management
Work Item Tracking
Reporting
Project Site
Visual Studio
Team Foundation Integration Services
Project ManagementBig Build
Class Designer
Visual Studio Team SystemPro
cess
an
d A
rch
itect
ure
Pro
cess
an
d A
rch
itect
ure
G
uid
an
ceG
uid
an
ce
Vis
ual S
tud
io In
du
stry
V
isu
al S
tud
io In
du
stry
Part
ners
Part
ners
Change Management
Work Item Tracking
Reporting
Project Site
Integration Services
Project Management
Dynamic Code Analyzer
Visual Studio
Team Architect
Static Code Analyzer
Code Profiler
Unit Testing
Code Coverage
Visio and UML Modeling
Team Foundation Client (includes CAL)
Visual Studio Professional Edition
Load Testing
Manual Testing
Test Case Management
Application Designer
Logical Infra. Designer
Deployment Designer
Visual Studio
Team DeveloperVisual Studio
Team Test
Change Management
Work Item Tracking
ReportingVisual Studio
Team Foundation
Class Designer
Big Build
Roles Addressed by Team System
Project Manager
Architect Developer Tester
Change Management
Work Item Tracking
Reporting
Project Portal
Visual StudioTeam Foundation
Integration Services
Project Management
Visual Visual Studio Studio IDEIDE
Work Item Integration
• Project Management can assign work items and track across lifecycle of the project
• Testers and Developers can assign defects and these integrate with PM work items
• Artifacts (tests, product modules, source code) can be associated with Work Items
Reporting Integration
• Information aggregated from Team System services used in reporting
• Stored in central database for trending• Drill-down into operational stores for
more detail
Reviews Find / Fix Rates…
..and Daily Trends
…and Bugs Against Code Churn and Testing Activity
Visual Studio Team System Team Test• Write your Tests with or without Code• Manage and Execute Test Cases• Extend the Visual Studio Test Types• Integrated with TFS
● Source Code Control● Project Management● Bug Tracking● Reports
• All fully functional from the Visual Studio Team System 2005
© 2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
top related