dev303. product 1st year cost (per user) renewal cost (per user) hp quality center$8,000$1,800...
TRANSCRIPT
Climbing the Agile Testing Ladderwith VS 2013, TFS 2013, MTM and Lab Manager
Adam CoganDamian Brady
DEV303
History #1• 1990 – We were building .exes and sending
emails• 2000 – First version of AutomatedQA TestComplete
• 2001 – First version of Mercury Interactive QuickTest Professional (now HP Unified Functional Tester)
• 2002 – First version of IBM RobotJ (now IBM Rational Functional Tester)
• 2005 – Visual Studio with Team Edition for Testers
History #2• 2006 – HP acquires Mercury for $4.5bn
HP Quality Center and QTP (now HP Unified Functional Tester)Big fat horrible ActiveX controls and VBScript
• 2007 – SmartBear acquires AutomatedQA (with TestComplete)
• 2008 – Visual Studio improvements – record a WebTest from Fiddler
• 2008 – Art of Test commences• 2009 – Telerik acquire Art of Test – Telerik Test Studio• 2010 – First release of Microsoft Test Manager• 2011 – Microsoft moves into the Gartner Magic Quadrant for
Software Quality Suites
History #3• 2012 – Telerik release Load and Performance Testing
Telerik release Test Studio for iOS (in the app store)• 2012 – MTM Exploratory Testing, UI improvements• 2013 – Telerik move into the Gartner Magic Quadrant
No sign of Telerik Load & Performance in the cloud• 2013 – A good chunk of MTM moves to the web
TFS releases a preview of their Load and Performance Testing in the Cloud
• 2013 – Microsoft moves into #3 in the Gartner Magic Quadrant for Software Quality Suites
• HP QC is still #1. Still ActiveX, still VBScript, and still $$
Test Product Comparisons
Product1st Year Cost
(per user)Renewal Cost
(per user)
HP Quality Center $8,000 $1,800SmartBear
TestComplete $4,999 -
VS Test Professional $2,169 $899
Gartner Magic Quadrant for Integrated Software Quality Suites
The Tools
SSW – What do we do
Sydney, Brisbane, Melbourne
Adam CoganChief Architect at SSW
Microsoft Gold PartnerMicrosoft Regional DirectorALM MVPCertified Scrum Trainer
@AdamCoganhttp://www.adamcogan.com
Damian BradySolution Architect and QLD State Manager at SSW
Loves:TFSScrumASP.NET MVCHTML5 + CSS + JS
@Damovisahttp://www.damianbrady.com.au
AgendaWhat is Agile Testing?Exploratory Testing (MTM)UI Testing (Coded UI + Selenium)Test Management (Web)Stakeholder FeedbackLab Management
Agile TestingWhat’s the difference?
Flickr: photo by plastanka
Agile TestingWhat does Scrum say?
https://www.scrum.org/Scrum-Guides
What does this mean for testing?• Make team communication easy• Make testing easy• Test that the feature works
Not what’s in section 31.4.2a of version 29.5-c of the Technical Specification Document
• Important: Acceptance Criteriahttp://rules.ssw.com.au/SoftwareDevelopment/RulesToBetterUserAcceptanceTests
• Important: Give stakeholders feedback opportunities
Different types of testing• ?• Unit Tests• Integration Tests
CodedCodedUISelenium (web only)
• Performance Tests• Load Tests…
Different types of testing #2• Smoke Tests• Manual Tests• Organized Manual Tests
From Acceptance CriteriaUser Acceptance Tests – Test Cases
• Automated Tests
UAT (Black Box to White Box)
Manual testingOrganized
Manual Testing
Automated Testing Code Tests
70-80%
Exploratory Testing Test Cases andTest Steps
Coded UI TestsSelenium Tests
Unit Tests and Integration Tests
10-20% dreaming 5 - 10%
Introducing the professional testing tool
Using MTM• Verdict?• Create a Test Plan• Create a Test Suite• Create Tests• Create Test Steps• Run Tests
So many steps before doing any actual testing
Exploratory TestingThe ultimate agile test
Flickr: photo by suckamc
UAT (Black Box to White Box)
80-100%
Exploratory Testing Test Cases andTest Steps
Coded UI TestsSelenium Tests
Unit Tests and Integration Tests
Manual testingOrganized
Manual Testing
Automated Testing Code Tests
Why do Exploratory Testing?• Testers are smart• But how do you force them to give you good bug
reports?• ?
Repro: Try to remember
Dev: “Works on my machine”
Tester: “It’s working now, what did you do?”
MTM Exploratory Testinghttp://rules.ssw.com.au/SoftwareDevelopment/RulesToBetterUserAcceptanceTes
ts/Pages/Do-you-use-Exploratory-Testing.aspx
*
Exploratory Testing (#1)• Acceptance Criteria• Test one PBI• Find a bug• Capture the repro steps• Create a test case (so this one doesn’t
happen again)
http://www.change.org/en-AU/petitions/canberra-airport-corporation-name-the-upcoming-airport-after-david-warren-the-inventor-of-the-black-box
Summary of Exploratory Testing• Verdict?• No need for detailed repro test steps until
there’s a problem• Create bugs with awesome screenshots• Create test cases for repeatability with
awesome steps• If you do it this way, you’re at a level of
maturity above 80% of all teams• …
Exploratory Testing• ? Using it each sprint?
I predict by the end of next year, more than 50% of you will be using the Exploratory Testing tool in each sprint
- Adam Cogan
Exploratory Testing Automated Tests!• Enter one thing• Check one thing
Exploratory Testing Automated Tests!• Create a Coded UI Test from a recording• Not Exploratory Test but a Test Case
• Add an assertion to our test• Trying to have devs and testers work
together
• Note: Automation saves time for the testers• Note: But increases the load on developers
Coded UI Test[TestMethod]public void AssertGoogleShowsSswTestPageIsFirst() { this.UIMap.OpenBrowserAndGoogleSearchForSSWTesting();}
Action: DoneAssertion: …
Coded UI Test – With Assertion[TestMethod]public void AssertGoogleShowsSswTestPageIsFirst() { this.UIMap.OpenBrowserAndGoogleSearchForSSWTesting(); this.UIMap.AssertFirstLinkIsSSWRules();}
Action: DoneAssertion: Done
Coded UI – Finding a Search Boxthis.mUISearchEdit = new HtmlEdit(this);#region Search Criteriathis.mUISearchEdit.SearchProperties[HtmlEdit.PropertyNames.Id] = "lst-ib";this.mUISearchEdit.SearchProperties[HtmlEdit.PropertyNames.Name] = "q";this.mUISearchEdit.FilterProperties[HtmlEdit.PropertyNames.LabeledBy] = null;this.mUISearchEdit.FilterProperties[HtmlEdit.PropertyNames.Type] = "SINGLELINE";this.mUISearchEdit.FilterProperties[HtmlEdit.PropertyNames.Title] = "Search";this.mUISearchEdit.FilterProperties[HtmlEdit.PropertyNames.Class] = "lst lst-tbb";this.mUISearchEdit.FilterProperties[HtmlEdit.PropertyNames.ControlDefinition] = "name=\"q\" title=\"Search\" class=\"lst lst-t";this.mUISearchEdit.FilterProperties[HtmlEdit.PropertyNames.TagInstance] = "3";this.mUISearchEdit.WindowTitles.Add("Google");#endregion
115 lines
400 lines!
Tip: Rename these so they sort in order
Summary of Coded UI Tests• Verdict?
Get devs and testers working together
Automate to save the testers’ time
Your test can:• be part of your build• run in lab environments
Coded UI Tests
Verbose
Brittle
Hard to maintain
Can give you nightmares
More painful than an ingrown toenail
UI TestingYou can get Coded UI Tests rightFor Windows Apps:• Use Coded UI• Refactor: Marcel De Vries• Code First: codeduicodefirst.codeplex.comFor Web:• Use Selenium…
Selenium Tests (#2)
• Do one thing (search for SSW in Bing)
• Check one thing (Make sure we’re in the top 3
links)
• Add a reference to Selenium
• Write all the code (less than 400 lines)
Selenium - .Net Bindings[TestMethod]public void TestWebsiteHasImportantThingsFirst(){ // Arrange var browser = new OpenQA.Selenium.IE.InternetExplorerDriver(@"C:\"); browser.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(10)); // Act browser.Navigate().GoToUrl("http://bit.ly/CoolTurk"); var paragraphs = browser.FindElementsByCssSelector("p") .Select(p => p.Text); // Assert paragraphs.Take(20).Should().Contain( p => p.Equals("I like sex", StringComparison.InvariantCulture));
browser.Close();}
new OpenQA.Selenium.Chrome.ChromeDriver(); new OpenQA.Selenium.Firefox.FirefoxDriver();
Selenium Tests• Verdict?• Fall in love with WebDriver!
Concise
Cross-browser
Can’t use an Action Recording*
*
Selenium components for Coded UI Cross Browser Testin
g
What I want• Use MTM to record test runs
Including Exploratory Testing!MTM | Test Case | Run | [x] Create Action Recording
• It gives me 10 lines of Selenium code
Behaviour Driven DevelopmentLevel 1:• Behaviour-focused method namesLevel 2:• + Fluent AssertionsLevel 3:• + SpecFlow (VS Extension) or• + BDDfy
Making Manual Testing EasyCurrent requirements of MTM:• Kind of a large installation• Windows only• UI could be better – ribbon
Testing in the WebWeb Access for running Functional tests
Flickr: photo by 8724931@N07
Testing in Web Access (#3)• Open “Test” (aka Web Test Case
Management)• Read a test case in Web Access• Run the test case from the web• Log a bug
Testing in Web Access• “Web Test Case Management” or “Testing
Hub”• New in TFS 2012.2• Manage test cases in Web Access• Substantially better in 2013• Create/Modify Test Plans through Web UI• Open MTM from web• Open Microsoft Test Runner from web (deep link)• Paste images into work items in the web
• Cool – No need to install MTM!
Summary of testing in Web Access• Verdict?• Test case management from the web• Run tests from the browser• Any* browser or operating system• No need to install MTM
No recording (unless you “Run using
client”)
Stakeholder FeedbackThe earlier the better
Flickr: photo by drongo3
ScenarioWe want informal feedback from someone not normally on the Scrum team? How?
• Meeting/Demo
• Over-the-shoulder
Stakeholder Feedback (#4)
#1 – Invite to Scrum meeting
#2 – Recordings with Jing
#3 – Feedback Tool
Summary of Feedback Tool• Verdict?• Ask anyone* for feedback!• Rich feedback – text, videos, screenshots,
markup• Automatically added to TFS (a step to go
from Feedback to PBI)• vs MTM?
One environment is not enough• Many devices and browsers• BYOD (Bring your own device) becoming
more common• In the past:• Complicated Hyper-V environments for lab testing• Expensive infrastructure and hardware• Maintenance
Lab ManagementTest in multiple environments
Flickr: photo by gosheshe
Lab Management• Standard Environments (new in 2012)• Azure VMs
(no longer charged for VMs that are turned off)
Summary of Lab Management• Verdict?• Great for Windows apps in particular• Environments can be set up for any
machine you have• Tests can be run on these environments• Build definitions to run tests automatically• Use Azure for VMs you can start and stop as
required
RecapWhat is Agile Testing?Exploratory Testing (MTM)UI Testing (Coded UI + Selenium)Test Management (Web)Stakeholder FeedbackLab Management
Evaluate this session and you could win instantly!
Head to...aka.ms/te
© 2013 Microsoft Corporation. All rights reserved.Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.