technical meeting automated testing with vs2010
DESCRIPTION
automated testing with vs2010 ... test case specification, execution, automation and test infrastructure with VS2010 CodedUI and Lab managementTRANSCRIPT
Technical Meeting Automated Testing with
VS2010
Suite of Application Lifecycle Management tools
http://msdn.microsoft.com/en-us/library/fda2bad5.aspx
application lifecycle management
http://www.clemensreijnen.nl/post/2008/02/18/ALM-Definitions.aspx
work together
__with work packages __on alm artifacts
Testing Lifecycle
See also on www.tmap.net: - The Test Lifecycle Model - The 4 essentials
http://msdn.microsoft.com/en-us/library/bb385832.aspx
Microsoft Test Manager 2010
Test Case Management
See also on www.tmap.net: Control Phase
Test Plans Test Suites Test Configurations
Organize the testing effort in a Test Execution Plan Create dynamic test suites for changing selection criteria Define various test configurations and share automation across those
Test (Execution) Plans
Test (Execution) Plans
Test Plan Organization Naming of test plans for agile environments: Iteration number and name and one extra test plan for end to end tests.
Test Configurations
Test Configurations
Test Suites
Test Suites Organization Naming of test suites for agile environments: Suite per user story (requirement based) and one additional test suite for regression tests.(query based?) (test cases are copied to the regression test suite)
Proposed organization
TCM.exe
TCM plan /list
TCM suites /lists planid:16
Share MTM Artifacts
mtm://clemenes-pc:8080/tfs/defaultcollection/p:MSF/Testing/testplan/connect?id=19
mtm://server:8080/tfs/ProjectCollection/p:Project/testing/testcase/open?id=67
Open Test results Open Test Cases (or any work item) Open Test Runs Open Test Plans Connect to test plans
mtm://<server name>:<port>/<tfs vdir>/<Collection name>/p:<project name>/<center group>/<group specific>
http://blogs.msdn.com/b/vstsqualitytools/archive/2010/02/09/sharing-content-in-microsoft-test-manager.aspx
Test Case Specification
See also on www.tmap.net: Specification Phase
Test Case Shared Steps Test Data Iterations
Link test cases and requirements to provide requirement traceability Data drive test cases with different parameters Refactor common shared steps in test cases
Test Case
Test step types
Test steps Make them descriptive Not every step is a validation step Use the step attachment
Test steps copy-past
Test Data Iterations Data drive test cases with different parameters
Test Data Iterations
Test data copy-past
Shared Steps Refactor common shared steps in test cases Reduce test maintenance by sharing test steps across test cases
Shared Steps Shared Step candidates?
• REUSE – Many steps, reused for many test cases . Benefit: time, retyping, boring work.
• EXPECTED CHANGES – for example environment variables change [URL]. Benefit: Time, test cases
which need this step are updated automatically, also action recording.
• PRE- CONDITIONS – bring the system under test to a certain state and execute the different
verifications. Benefit: beside time, you are certain the application under test is in the same state for every test case
• BACK TO INITIAL STATE - after test execution bring the environment back to the initial state for
new test cases. Benefit: beside time, you are certain the application under test is always in a clean state for new test cases.
Test Execution
Test Runner
A test that you will want to run multiple times. Different manual tests that contain common steps. Verifying a bug is fixed by using a manual test.
Test Case Execution
See also on www.tmap.net: Execution Phase
Test Case Execution
Action recording
TCM run /create
From TCM run /export id:239 /resultsfile:”c:\runresult”
Share with mtm://
mtm://clemens-pc:8080/tfs/defaultcollection/p:MSF/Testing/testrun/open?id=292
Controlling the testing effort BUG’s
File a BUG
Diagnostic Data Adapters File a Bug File an Exploratory Bug Verify a Bug
Diagnostic Data Adapters
File a Bug Submit a Bug using Test Runner
Ways to file a Bug
Submit a Bug from the Test Runner Submit a Bug from the Verify Bugs View Submit a Bug from Test Results for a Test
Verify a Bug Verify a Bug Fix Using Microsoft Test Manager
The Bug Flow
T
E
S
T
D
E
V
Link build 2 test plan. Configure Data collectors.
Setup build and deploy
Build definition is
the intergration
Execute Manual Tests
File Bug
Resolve Bug
Checkin Build Deploy
Verify fix by using playback
Close or reopen Bug
Test Impact
T
E
S
T
Execute ‘passed’ Tests
Execute ‘passed’ Tests
Test Impact
D
E
V
Assign build 2 test plan. Configure Data collectors.
Setup build and deploy
Build definition is
the intergration
Execute ‘passed’ Tests
Make code changes
Checkin Build Deploy
Assign ‘NEW’ build 2 Test Plan
Evaluated and reset tests impacted by code churn
Create test from Bug
File an Exploratory Bug Submit an Exploratory Bug using Test Runner
Record scenario by business user with MTM on application under test. Action recording is captured.
1
Create test steps (with parameters) in MTM by using the action recording.
2
Save test case and execute it by a tester on application under test, by using MTM.
3
CON: - Manual actions and tests. - Test steps text are technical names - Action recording (step 1) must be clean. - Unknown what is validated, during step 2, need
additional knowledge. - The business user must think of the important
scenarios The action recording during step 1 can have unnecessary steps, these can be removed during the execution (1) or during the specification (2).
PRO: - Reuse scenario knowledge business user - No typing detailed test steps - Easy to parameterize - Writing standards for test steps - MTM capabilities in step 3. (bug, codedUI,
recommended test etc …) Most benefit during step 2 (test case writing), benefit in step 1: run once in common situation. Benefit in step 3: common MTM benefits (record playback, fast forward, parameters etc) and benefit of well written steps.
Test Activities Planning…
RECAP
Test Case Management RECAP
Test Case Specification
RECAP
Test Case Execution
RECAP
Test Case Management RECAP
Intellitrace
F5
1 Open the employee application Set the intellitrace options Set a breakpoint at …
2 When on the breakpoint, save the trace file on your desktop
3 Close Visual Studio and double click the intellitrace file
4 In the intellitrace summary, open the main threat …
5 … and search for The values that where entered during the previous run.
6 And now from Microsoft Test Manager
6 Set the diagnostic adapter
7 Configure the build… it needs a drop location for the symbols (pdb files)
8 Run the test case and see it crash… file a bug.
9 See the itrace file in the bug report...
10
Close MTM, open VS (with or without the employee application). Got to the bug and click the itrace file… See the exception and start debugging…
11
Go back in time
11
Go back in time … and find the entered values
Regression Test Candidates
See also on www.tmap.net: Completion Phase
Test Automation
Test automation Where? When? How? Who? Why? What?
met Visual Studio 2010
Where? When? How? Who? Why? What?
Where? When? How? Who? Why? What?
Where? When? How? Who? Why? What?
VS2010 Test Tools UI Test Automation Capabilities
Fast Forward CodedUI
Test Infrastructure
1
2
3
No investment in any kind of automation. Just click and test.
Clean up of the Action log maybe re-execute, for more smooth Fast Forward.
Basic CodedUI. Only use the default generation, add your own assertion.
Advanced CodedUI. Customize the CodedUI and UIMap for optimization.
0
Test Automation Investment Levels
1 Clean up of the Action
2
VS2010 ALM MTLM usages patterns for Windows Azure hosted application development
Basic CodedUI
3 Advanced CodedUI
Fast Forward
Cleaning the action recording When: executing the same test case on multiple environments or when using test data iteration or … and don’t forget the shared steps
CodedUI
CodedUI from existing action recording from Manual Tester
CodedUI from existing action recording from Manual Tester
CodedUI from existing action recording from Manual Tester
1
3
4
2
CodedUI Create a new action recording by Developer
UIMap
UIMap
Data-Driven Coded UI Test
Test Infrastructure
Where and how to run the automated test?
Test Environments
See also on www.TMap.net: - Checklist "Intake Test Environment“ - Checklist "Test Environment“ - Infrastructure Phase
Test lab Visual Studio 2010 Lab Management
physical test environments virtualized test environments
Where and how to run the automated test?
TFS 2010
Physical Test Client
MTLM Client
Physical Test Client
Hyper-V Host
VM
System Center Virtual Machine Manager (SCVMM)
TFS 2010
VM
Library Shares
share
share
MTLM Client
physical test environments
virtualized test environments
Build application
Provision ‘clean’ env
Deploy application
Run tests
physical test environments versus virtualized test environments
Visual Studio
Test Manager
Team Foundation Server
Build Server
Test Agent
Test Agent
Test Agent
Lab for Physical Environments, the different flavors…
A
Flavor A: Execution from VS2010…
Configurations for automated test execution:
Real developer test / test automation dry run (Triggered by: right mouse click – run tests, in test project) http://msdn.microsoft.com/en-us/library/dd286580.aspx No additional configuration needed* * Data Diagnostic Settings in the Local.Testsettings file are configurable.
Pro: Easy setup
Debug-able test automation
Con: No collection of test results in TFS Run on developer environment
B
Flavor B: Execution during build with Build controller…
Configurations for automated test execution:
Part of Build Verification Tests (triggered by: build) Set the Build Service to run in interactive mode http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx Configure the build to run the tests http://msdn.microsoft.com/en-us/library/ms182465.aspx#CreateBuildType * Data Diagnostic Settings in the *.Testsettings file are configurable.
Pro: Easy setup
Test results in build result
Con: No collection of test results in TFS Build Controller needs to run in interactive mode Tests are executed on build environment Run on build environment
not recommended, strange to run UI tests on build
C
Flavor C: Execution during build with Test controller…
Configurations for automated test execution:
Part of Build Verification Tests on multiple test agents (triggered by: build) Configure Test Controller (don’t register it with a project collection ) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers Configure Test Agents on clients (interactive mode) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents Configure *.Testsettings file in solution to use Test Controller and Test Agents Configure the build to run the tests (see B)
Pro: Test run on test environments
Tests run on multiple environments Test Results in Build result
Con: No collection of test results in TFS Harder to configure Need for specific test client environments Test Settings from VS
Preferred configuration above flavor B
D
Flavor D: Execution from Microsoft Test Manager…
Configurations for automated test execution:
Part of Regression Tests (triggered by: MTM user, right mouse click on test case, run) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. http://msdn.microsoft.com/en-us/library/ee390842.aspx http://msdn.microsoft.com/en-us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html
Pro: Test run on test environments
Tests run on multiple environments Test Result in MTM and TFS
Test Settings from MTM
Con: Manually triggered by Tester Hard to configure Need for specific test client (same as MTM?)
Other type of test than BVT
E
Flavor E: Execution from MTM during Build…
Configurations for automated test execution:
Part of BVT (triggered by: Build) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run TCM or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx
Pro: Test run on test environments
Tests run on multiple environments Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con: Hard to configure Need for specific test client (same as MTM?)
Preferred configuration above flavor C Flavor D and E can be configured together
Test Controller
Test Agent
New physical environment
Execute CodedUI During the build with Visual Studio 2010
C
Flavor C: Execution during build with Test controller…
Configurations for automated test execution:
Part of Build Verification Tests on multiple test agents (triggered by: build) Configure Test Controller (don’t register it with a project collection ) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers Configure Test Agents on clients (interactive mode) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents Configure *.Testsettings file in solution to use Test Controller and Test Agents Configure the build to run the tests (see B)
Pro: Test run on test environments
Tests run on multiple environments Test Results in Build result
Con: No collection of test results in TFS Harder to configure Need for specific test client environments Test Settings from VS
Preferred configuration above flavor B
1 Configure Test Controller.
2 Add new Test Settings file Set remote execution and test controller.
3 Create a new build definition Set the Automated Test Properties
4 Queue a new Build Watch the Test Agent and Test Environment
Execute CodedUI Within a test case as an associated automation manualy
D
Flavor D: Execution from Microsoft Test Manager…
Configurations for automated test execution:
Part of Regression Tests (triggered by: MTM user, right mouse click on test case, run) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. http://msdn.microsoft.com/en-us/library/ee390842.aspx http://msdn.microsoft.com/en-us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html
Pro: Test run on test environments
Tests run on multiple environments Test Result in MTM and TFS
Test Settings from MTM
Con: Manually triggered by Tester Hard to configure Need for specific test client (same as MTM?)
Other type of test than BVT
1 Configure Test Controller.
2 Associate the CodedUI with a Test Case.
2 Create Test Environment
3 Set test Settings
4 Associate the CodedUI with a Test Case.
5 Run Automated Tests from Test Manager
Execute CodedUI Within a test case as an associated automation from the build
E
Flavor E: Execution from MTM during Build…
Configurations for automated test execution:
Part of BVT (triggered by: Build) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run MTC or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx
Pro: Test run on test environments
Tests run on multiple environments Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con: Hard to configure Need for specific test client (same as MTM?)
Preferred configuration above flavor C Flavor D and E can be configured together
Execute automated test cases on different environments
0 Same steps as previous.
TFS 2010
Physical Test Client
MTLM Client
Physical Test Client
Win 7 IE8 configuration
Win 7 IE9 configuration
1 Configure additional environments in Test Plan properties.
Win 7 IE8 configuration
Win 7 IE9 configuration
TC
1 Get Test Plan ID Get Test Suite Get Test Configurations with TCM.exe
2 Create command line for Test Run for the different Test Configurations
"run /create /title:""Run on Host for specific Test Point"" /planid:16 /querytext:""SELECT * FROM TestPoint WHERE ConfigurationId=13"" /settingsname:""Automated Test on Windows 7 Environment IE9"" /testenvironment:""Win 7 IE9"" /collection:http://clemenes-pc:8080/tfs/defaultcollection /teamproject:EmployeeApplication"
"run /create /title:""Run on test env for specific Test Point"" /planid:16 /querytext:""SELECT * FROM TestPoint WHERE ConfigurationId=12"" /settingsname:""Automated Test on Windows 7 Environment IE8"" /testenvironment:""Win 7 IE8"" /collection:http://clemenes-pc:8080/tfs/defaultcollection /teamproject:EmployeeApplication"
3 Add build commands
6 Queue a new Build Watch the Test Agent and Test Environment
Build application
Provision ‘clean’ env
Deploy application
Run tests
• Reproduce the exact conditions of a bug or other development issue.
• Build, deploy, and test applications automatically in a clean environment.
• Reduce the time required to create and configure machines for testing an application.
• Run multiple copies of a test or development at the same time.
• Enable members of a team to create and manage virtual environments without requiring system administrator privileges.
VS2010 Lab Management
System Center Virtual Machine Manager (SCVMM)
TFS 2010
Hyper-V Host
VM
VM
Library Shares
share
share
MTLM Client
VM
Lab Agent
TFS 2010
staat voor resultaat