Download - uTest STPCon 2011 Presentation
|
In-The-Wild Testing:Your Survival May Depend On It
Presented by Matt Johnston | @matjohnstonSTPCon, Fall 2011
| 2
Agenda
• Current State of Affairs• SoLoMo Boom• Surviving In-The Wild• Crowdsourcing 101• Secret Sauce• Key Takeaways
The Challenge
|| 3
How We Arrived AtThe Status Quo
| 4
Stuff Gets Built; Then It Breaks
1. Marketing identifies a need
2. Product specifies a solution
3. Engineering builds the product
4. QA tests the application
5. Help desk listens to complaints
6. Sales & execs blame everyone
7. Then we do it all over again and
hope for better results… Vicious Cycle
| 5
Testing Innovation
Agile Testing
Manual vs. Test Automation Managing
QA Teams
Exploratory vs.Test Cases
Partnering Testers & Engineers
In-House vs. Outsource
Games, Tours& Quick Attacks
In T
he L
ab
| 6
Testing Innovation
In T
he L
ab
InThe
Wild
| 7
To Make Matters Worse
• Users are less tolerant
• Social media & app stores give everyone a megaphone
• So their problem… is your problem
The Challenge
|| 8
Why It Matters More:The SoLoMo Boom
| 9
Enterprise Consumerization
• The Consumerization of Enterprise Computing– Technological disruption is pervasive– NO industry is exempt– B2B is B2C– Everyone’s a retailer
• Not just for games and personal use– Docs– Data– Publishing
• Not just your public site– CRM systems– BI & analytics tools– Productivity suites
| 10
Social: Real-Time Communication
• Enterprise social revolution isn’t coming… it’s here– Nothing is disconnected– No industry is immune
• The stats on social media adoption in Fortune 100 alone:– 77% leverage Twitter– 69% utilize LinkedIn– 61% enable Facebook– 57% incorporate YouTube
• Gartner: By 2016, social integrated w/ nearly all B2B apps
| 11
Social: Impact On Testing
• Enterprise adoption of social presents security & privacy hurdles – Internally, a channel for spam, malware and data breaches– Confidential data may be posted and trigger regulatory penalties– Externally, increasingly used for user authentication on your products
• And functional testing challenges– Testing code & integrations that aren’t yours– That constantly changes– And you will get the blame
| 12
Local: Can You Hear Me Now?
• Businesses use location for more than just marketing• Numerous applications for location-based technologies
– Asset tracking– Maps & routing– Location finders– Check-in services– Geo-based personalization
• Web and mobile have gone local– 1 of 5 searches has local intent– 1 of 3 mobile searches has local intent
| 13
Local: Impact On Testing
• Not just a mobile problem– Localization testing– Geo-based personalization
• But it is a mobile problem too– Apps used outside the confines of the
QA lab, under in-the-wild conditions
• When apps & users are distributed around the country or globe, a portion of testing should be too
| 14
• ABI Research anticipates worldwide enterprise mobile data revenues to reach $133 billion by 2014
• Enterprise mobility no longer just for email– Business apps: CRM, ERP, HR systems– Productivity apps: docs, spreadsheets, presentations– Collaboration apps: email, IM, publishing – Medical apps: health records, patient education
• Tablet-mania, both inside and out– Aflac– Mercedes-Benz– Wells Fargo– SAP
Mobile: Enterprise Impact
| 15
• Mobile web vs. native apps• Form factors matter again
– Feature phones– Smart phones– Tablets
• Ever-expanding HW & SW costs• Back to the future
– A return to the late 90’s web- Lack of automation or load tools- Lack of security or usability standards- Lack of dev kits or tools- Extremely fluid landscape
Mobile: Impact On Testing
| 16
Web Testing Matrix Is Tough
OS & Browser
Features
1. Coverage
| 17
Mobile Testing Matrix Is Insane
OS & Browser
Features
Handset Makers& Models
1. Coverage
|
Mobile Testing Matrix Is Insane
18
OS & Browser
Features
Handset Makers& Models
WirelessCarriers
1. Coverage
|
Mobile Testing Matrix Is Insane
19
OS & Browser
Features
Handset Makers& Models
WirelessCarriers
Location
1. Coverage
|
2. Managing Resources & Costs
• Multiple products, roadmaps & launch schedules• Managing peaks & valleys; unexpected twists & turns
20
|
3. Time-to-Market
• Never-ending race for each new app and version• Can exhaust the best test managers and teams
21
|
SoLoMo: The Bottom Line
• Becoming impossible for QA leaders to assure quality
22
The Challenge
|| 23
Argh!!What Do We Do?
| 24
• Keep testing in-the lab– Keep training– Keep automating– Keep innovating
• Recognize that it will never again be sufficient• Identify the right way for you to test in-the-wild
– Real-world conditions– Mirror your user base
- Technologically: OS, browser, anti-virus, device, carrier- Geographically: Continent, country, city, language- Demographically: Age, gender, education, employment, industry, hobby
– Maximize relevant testing coverage– Manage signal-to-noise ratio
From Surviving To Thriving
| 25
• Two ways to get there: Beta programs and crowdsourced testing
Testing In The Wild
| 26
• Pros:– No direct costs– Lives outside the lab
• Cons:– Users ≠ testers
- Rarely diagnostic- Poor signal-to-noise ratio
– Puts unfinished product in front of customers
• Many firms discontinued beta as a core part of QA– Hidden costs– Takes too long– Not actionable results– Didn’t yield higher quality
Beta “Testing” Programs
| 27
• Pros:– Lives outside the lab– Profiled, professional testers– Mirrors user base– Protects company IP– Much higher signal-to-noise ratio
• Cons:– Carries direct costs– Requires careful partner vetting– Requires buy-in from QA leadership
• Increasingly popular complement to in-the-lab testing
Crowdsourced Testing
The Challenge
|| 28
A Word AboutCrowdsourcing
| 29
Crowdsourcing Across Industries
| 30
Mob Mentality
BUT crowds often look (and act) like unruly mobs
| 31
From Mob to Community
And mobs don’t work in every category
Example: delivering a skilled service at an enterprise level of predictability and efficiency requires an orderly “community” capable of consistently producing the desired results
| 32
So how do you turn this –
| 33
Into this?
The Challenge
|| 34
SecretSauce
| 35
Tenet 1: Know Thy Task
| 36
Tenet 2: Know Thy Partner
• Selection criteria are vital:– Referenceable customer successes
– By company size– By industry
– Ability to adapt to your legacy systems and processes– Ability to satisfy legal requirements
– IP protection– NDA
– Ability to do the job– Consistent– Predictable– Professional
| 37
1. Community profiling– Technical– Geographic– Demographic
2. Community ratings & micro-ratings– By testing type– By app type– By industry
3. Highly precise matching– Between each project & each tester
Tenet 3: Know Thy Community
| 38
Tenet 4: Know Thyself
• Is your culture highly cautious & risk-averse?• Are you in a highly regulated industry?
– Defense industry– PCI, PII or PHI
• Do you have an appetite for innovation?• Are you centrally organized or decentralized?
– Sourcing– IT & IS– Engineering– QA & QE
| 39
Tenet 5: Know Thy Trends(Enterprises That Already Leverage The Crowd)
The Challenge
|| 40
EnterprisePerspective
|
Global power Google taps into the crowd to augment its in-house testing resources– Challenge:
- Tech execs sought ways to scale testing and achieve in-the-wild coverage- Wanted to ensure testing had real-world relevance to end-user experience
– Strategy:- Have invested heavily in outsourcing, in-house resources and test automation- Google was eager to find a better, more scalable approach to app testing- Google explored a wide variety of alternatives and found uTest
– Results:- After extensive 2009 pilot project, Google has since expanded its use to 18+ apps
– Web, desktop and mobile- Now leverages in-the-wild testing broadly & frequently, as part of core QA playbook- Relationship is pioneering in-the-wild testing as complement to enterprise’s in-the-lab
41
|
Security Essentials – a free anti-virus app– Challenge:
- v1.0 product in entirely new category, so IP preservation was vital- Needed to test functionality under real-world conditions on vast matrix of hardware,
browsers, Windows OSes, and third-party applications- Needed testing coverage in strategic geo-locations (China, India, South America) to
mirror expected user-base and fill blind spots
– Strategy:- Recognized that it couldn’t solve this solely in the lab- Assemble a targeted team of testers to perform a series of exploratory and test case
execution to cover stated testing criteria
– Results:- After two months of continuous testing in ten countries on four continents – hundreds of
detailed bug reports, executed test cases, and user reviews had been prioritized- The company continues to run regression tests on the software to coincide with major
upgrades or new releases
42
Microsoft Security Essentials
The Challenge
|| 43
Bottom LineSummary
| 44
Key Takeaways
• Apps Universe Has Forever Changed Testing– Exponentially more diverse user environments
– Devices– Software configurations– Locations– User demographics
– Users less tolerant– Quality issues no longer private
• QA Must Play Catch-Up– By improving inside the QA lab– By moving beyond it, where your users live, work & play
– Beta programs and/or crowdsourced testing
– App quality winners are combining: in-the-lab testing + in-the-wild testing
The Challenge
|| 45
The End. Well, actually it’s just beginning…
(more at InTheWildTesting.com)
Matt Johnston | CMO @ [email protected] | @matjohnston