dev303. product 1st year cost (per user) renewal cost (per user) hp quality center$8,000$1,800...

Climbing the Agile Testing Ladderwith VS 2013, TFS 2013, MTM and Lab Manager

Adam CoganDamian Brady

DEV303

History #1• 1990 – We were building .exes and sending

emails• 2000 – First version of AutomatedQA TestComplete

• 2001 – First version of Mercury Interactive QuickTest Professional (now HP Unified Functional Tester)

• 2002 – First version of IBM RobotJ (now IBM Rational Functional Tester)

• 2005 – Visual Studio with Team Edition for Testers

History #2• 2006 – HP acquires Mercury for $4.5bn

HP Quality Center and QTP (now HP Unified Functional Tester)Big fat horrible ActiveX controls and VBScript

• 2007 – SmartBear acquires AutomatedQA (with TestComplete)

• 2008 – Visual Studio improvements – record a WebTest from Fiddler

• 2008 – Art of Test commences• 2009 – Telerik acquire Art of Test – Telerik Test Studio• 2010 – First release of Microsoft Test Manager• 2011 – Microsoft moves into the Gartner Magic Quadrant for

Software Quality Suites

History #3• 2012 – Telerik release Load and Performance Testing

Telerik release Test Studio for iOS (in the app store)• 2012 – MTM Exploratory Testing, UI improvements• 2013 – Telerik move into the Gartner Magic Quadrant

No sign of Telerik Load & Performance in the cloud• 2013 – A good chunk of MTM moves to the web

TFS releases a preview of their Load and Performance Testing in the Cloud

• 2013 – Microsoft moves into #3 in the Gartner Magic Quadrant for Software Quality Suites

• HP QC is still #1. Still ActiveX, still VBScript, and still $$

Test Product Comparisons

Product1st Year Cost

(per user)Renewal Cost

(per user)

HP Quality Center $8,000 $1,800SmartBear

TestComplete $4,999 -

VS Test Professional $2,169 $899

Gartner Magic Quadrant for Integrated Software Quality Suites

The Tools

SSW – What do we do

Sydney, Brisbane, Melbourne

Adam CoganChief Architect at SSW

Microsoft Gold PartnerMicrosoft Regional DirectorALM MVPCertified Scrum Trainer

@AdamCoganhttp://www.adamcogan.com

Damian BradySolution Architect and QLD State Manager at SSW

Loves:TFSScrumASP.NET MVCHTML5 + CSS + JS

@Damovisahttp://www.damianbrady.com.au

AgendaWhat is Agile Testing?Exploratory Testing (MTM)UI Testing (Coded UI + Selenium)Test Management (Web)Stakeholder FeedbackLab Management

Agile TestingWhat’s the difference?

Flickr: photo by plastanka

Agile TestingWhat does Scrum say?

https://www.scrum.org/Scrum-Guides

What do we care about in Agile?

http://www.agilemanifesto.org

What does this mean for testing?• Make team communication easy• Make testing easy• Test that the feature works

Not what’s in section 31.4.2a of version 29.5-c of the Technical Specification Document

• Important: Acceptance Criteriahttp://rules.ssw.com.au/SoftwareDevelopment/RulesToBetterUserAcceptanceTests

• Important: Give stakeholders feedback opportunities

Different types of testing• ?• Unit Tests• Integration Tests

CodedCodedUISelenium (web only)

• Performance Tests• Load Tests…

Different types of testing #2• Smoke Tests• Manual Tests• Organized Manual Tests

From Acceptance CriteriaUser Acceptance Tests – Test Cases

• Automated Tests

UAT (Black Box to White Box)

Manual testingOrganized

Manual Testing

Automated Testing Code Tests

70-80%

Exploratory Testing Test Cases andTest Steps

Coded UI TestsSelenium Tests

Unit Tests and Integration Tests

10-20% dreaming 5 - 10%

Introducing the professional testing tool

Using MTM• Verdict?• Create a Test Plan• Create a Test Suite• Create Tests• Create Test Steps• Run Tests

So many steps before doing any actual testing

Exploratory TestingThe ultimate agile test

Flickr: photo by suckamc

UAT (Black Box to White Box)

80-100%

Exploratory Testing Test Cases andTest Steps

Coded UI TestsSelenium Tests

Unit Tests and Integration Tests

Manual testingOrganized

Manual Testing

Automated Testing Code Tests

Why do Exploratory Testing?• Testers are smart• But how do you force them to give you good bug

reports?• ?

Repro: Try to remember

Dev: “Works on my machine”

Tester: “It’s working now, what did you do?”

MTM Exploratory Testinghttp://rules.ssw.com.au/SoftwareDevelopment/RulesToBetterUserAcceptanceTes

ts/Pages/Do-you-use-Exploratory-Testing.aspx

*

Exploratory Testing (#1)• Acceptance Criteria• Test one PBI• Find a bug• Capture the repro steps• Create a test case (so this one doesn’t

happen again)

http://www.change.org/en-AU/petitions/canberra-airport-corporation-name-the-upcoming-airport-after-david-warren-the-inventor-of-the-black-box

Summary of Exploratory Testing• Verdict?• No need for detailed repro test steps until

there’s a problem• Create bugs with awesome screenshots• Create test cases for repeatability with

awesome steps• If you do it this way, you’re at a level of

maturity above 80% of all teams• …

Exploratory Testing• ? Using it each sprint?

I predict by the end of next year, more than 50% of you will be using the Exploratory Testing tool in each sprint

- Adam Cogan

Exploratory Testing Automated Tests!• Enter one thing• Check one thing

Exploratory Testing Automated Tests!• Create a Coded UI Test from a recording• Not Exploratory Test but a Test Case

• Add an assertion to our test• Trying to have devs and testers work

together

• Note: Automation saves time for the testers• Note: But increases the load on developers

Coded UI Test[TestMethod]public void AssertGoogleShowsSswTestPageIsFirst() {     this.UIMap.OpenBrowserAndGoogleSearchForSSWTesting();}

Action: DoneAssertion: …

Coded UI Test – With Assertion[TestMethod]public void AssertGoogleShowsSswTestPageIsFirst() {     this.UIMap.OpenBrowserAndGoogleSearchForSSWTesting();     this.UIMap.AssertFirstLinkIsSSWRules();}

Action: DoneAssertion: Done

Coded UI – Finding a Search Boxthis.mUISearchEdit = new HtmlEdit(this);#region Search Criteriathis.mUISearchEdit.SearchProperties[HtmlEdit.PropertyNames.Id] = "lst-ib";this.mUISearchEdit.SearchProperties[HtmlEdit.PropertyNames.Name] = "q";this.mUISearchEdit.FilterProperties[HtmlEdit.PropertyNames.LabeledBy] = null;this.mUISearchEdit.FilterProperties[HtmlEdit.PropertyNames.Type] = "SINGLELINE";this.mUISearchEdit.FilterProperties[HtmlEdit.PropertyNames.Title] = "Search";this.mUISearchEdit.FilterProperties[HtmlEdit.PropertyNames.Class] = "lst lst-tbb";this.mUISearchEdit.FilterProperties[HtmlEdit.PropertyNames.ControlDefinition] = "name=\"q\" title=\"Search\" class=\"lst lst-t";this.mUISearchEdit.FilterProperties[HtmlEdit.PropertyNames.TagInstance] = "3";this.mUISearchEdit.WindowTitles.Add("Google");#endregion

115 lines

400 lines!

Tip: Rename these so they sort in order

Summary of Coded UI Tests• Verdict?

Get devs and testers working together

Automate to save the testers’ time

Your test can:• be part of your build• run in lab environments

Coded UI Tests

Verbose

Brittle

Hard to maintain

Can give you nightmares

More painful than an ingrown toenail

UI TestingYou can get Coded UI Tests rightFor Windows Apps:• Use Coded UI• Refactor: Marcel De Vries• Code First: codeduicodefirst.codeplex.comFor Web:• Use Selenium…

Selenium Tests (#2)

• Do one thing (search for SSW in Bing)

• Check one thing (Make sure we’re in the top 3

links)

• Add a reference to Selenium

• Write all the code (less than 400 lines)

Selenium - .Net Bindings[TestMethod]public void TestWebsiteHasImportantThingsFirst(){    // Arrange    var browser = new OpenQA.Selenium.IE.InternetExplorerDriver(@"C:\");    browser.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(10));     // Act    browser.Navigate().GoToUrl("http://bit.ly/CoolTurk");    var paragraphs = browser.FindElementsByCssSelector("p")                            .Select(p => p.Text);     // Assert paragraphs.Take(20).Should().Contain( p => p.Equals("I like sex", StringComparison.InvariantCulture));

    browser.Close();}

new OpenQA.Selenium.Chrome.ChromeDriver(); new OpenQA.Selenium.Firefox.FirefoxDriver();

Selenium Tests• Verdict?• Fall in love with WebDriver!

Concise

Cross-browser

Can’t use an Action Recording*

*

Selenium components for Coded UI Cross Browser Testin

g

What I want• Use MTM to record test runs

Including Exploratory Testing!MTM | Test Case | Run | [x] Create Action Recording

• It gives me 10 lines of Selenium code

Behaviour Driven DevelopmentLevel 1:• Behaviour-focused method namesLevel 2:• + Fluent AssertionsLevel 3:• + SpecFlow (VS Extension) or• + BDDfy

Making Manual Testing EasyCurrent requirements of MTM:• Kind of a large installation• Windows only• UI could be better – ribbon

Testing in the WebWeb Access for running Functional tests

Flickr: photo by 8724931@N07

Testing in Web Access (#3)• Open “Test” (aka Web Test Case

Management)• Read a test case in Web Access• Run the test case from the web• Log a bug

Testing in Web Access• “Web Test Case Management” or “Testing

Hub”• New in TFS 2012.2• Manage test cases in Web Access• Substantially better in 2013• Create/Modify Test Plans through Web UI• Open MTM from web• Open Microsoft Test Runner from web (deep link)• Paste images into work items in the web

• Cool – No need to install MTM!

Summary of testing in Web Access• Verdict?• Test case management from the web• Run tests from the browser• Any* browser or operating system• No need to install MTM

No recording (unless you “Run using

client”)

Stakeholder FeedbackThe earlier the better

Flickr: photo by drongo3

ScenarioWe want informal feedback from someone not normally on the Scrum team? How?

• Email

• Meeting/Demo

• Over-the-shoulder

Stakeholder Feedback (#4)

#1 – Invite to Scrum meeting

#2 – Recordings with Jing

#3 – Feedback Tool

Summary of Feedback Tool• Verdict?• Ask anyone* for feedback!• Rich feedback – text, videos, screenshots,

markup• Automatically added to TFS (a step to go

from Feedback to PBI)• vs MTM?

One environment is not enough• Many devices and browsers• BYOD (Bring your own device) becoming

more common• In the past:• Complicated Hyper-V environments for lab testing• Expensive infrastructure and hardware• Maintenance

Lab ManagementTest in multiple environments

Flickr: photo by gosheshe

Lab Management• Standard Environments (new in 2012)• Azure VMs

(no longer charged for VMs that are turned off)

Summary of Lab Management• Verdict?• Great for Windows apps in particular• Environments can be set up for any

machine you have• Tests can be run on these environments• Build definitions to run tests automatically• Use Azure for VMs you can start and stop as

required

RecapWhat is Agile Testing?Exploratory Testing (MTM)UI Testing (Coded UI + Selenium)Test Management (Web)Stakeholder FeedbackLab Management

Contact…AdamCogan@ssw.com.au

DamianBrady@ssw.com.au

@AdamCogan

@Damovisa

Thank You!

SSWSydney | Melbourne | Brisbane | Adelaide

info@ssw.com.auwww.ssw.com.au

Evaluate this session and you could win instantly!

Head to...aka.ms/te

© 2013 Microsoft Corporation. All rights reserved.Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.