sauce labs test coverage report august 2016 · pdf fileperhaps it’s not a big surprise...
TRANSCRIPT
S A U C E L A B S T E S T C O V E R A G E R E P O R T A U G U S T 2 0 1 6
Customers often ask us what browser and OS combinations they
should test against. There really is no single right answer as it depends
on your app and your users. However, with over 640 million tests
completed to date, we have a lot of data that can help you in forming
a test coverage strategy.
This coverage report summarizes anonymous data from tests run on
Sauce Labs in the last 90 days. It is useful for identifying testing trends
and understanding what browsers / OS combinations might best apply
to your organization.
3 Top OSes
3 Top Browsers
4 Unique Number of OSes
4 Mac, Windows and Linux by Industry
5 Top Browsers by Industry
5 Chrome Browser Versions
6 FireFox Browser Versions
6 Other Browsers
6 Summary
TABLE OF CONTENTS
TOP OSes
This is the percentage of tests performed by OS. Windows continues to be the
top platform tested, followed by Mac OS.
TOP OSes USED ON SAUCE L ABS (BY TEST VOLUME)
8.3%
19.4%
72.3%
Windows
Mac OS
Linux
TOP BROWSER
This is the percentage of testing performed on browsers (regardless of
platform). Chrome has recently topped Internet Explorer as the most used
browser world wide, so it’s no surprise that this is the most tested browser on
Sauce. Firefox is #2 followed by Internet Explorer. The Edge browser is very
small now, but expected to grow as Windows 10 becomes more mainstream.
According to Mozilla, 57% of browser use is Chrome. Many companies dictate
which browsers employees can use and many consumers use the default
browser that comes with their system. The following table shows general
browser preferences by region.
TOP BROWSERS USED ON SAUCE L ABS (BY TEST VOLUME)
6.7%
10%
45%
37.8%
Google Chrome
Firefox
Internet Explorer
Safari
Learn more at saucelabs.com
3
Country Chrome Firefox Internet Explorer
USA 46% 12% 19%
Germany 33% 34% 12%
France 47% 21% 11%
Japan 38% 15% 27%
Indonesia 49% 36% 2%
Source: Mozilla - http://tinyurl.com/h7ywx3n
UNIQUE NUMBER OF OSES
This is a measure of how many different OSes are used by industry.
The more platforms an app is tested against, the smaller the chance
your users will experience poor quality. Perhaps it’s not a big surprise
that the software industry tests across the widest variety of browser / OS
combinations. Rounding out the top 3 is Finance and Media where the stakes
for quality are high. It might surprise you to learn the average number of
browsers that our customers test against is 114.
UNIQUE OSes TESTED (BY INDUSTRY )
Technolo
gy
Finan
ce
Media
/Adve
rtisin
g
Retail
Non-P
rofit
E-Com
merc
e
Consulti
ng
Consum
er Pro
ducts
Tele
com /
Comm
Serv
ices
Educatio
n
Hosp
itals
& Heal
thcar
e
Ave
rag
e
240
180
120
60
0
201
175.5
156143.5 143
107 102
86
59
42.5 42.5
MAC, WINDOWS AND LINUX BY INDUSTRY
This is a measure of the mix of OSes tested by industry. E-commerce tends to
test heavily against Macs, for all other industries, Windows dominates.
Windows Mac Linux
OS TESTS BY INDUSTRY
Technolo
gy
Retail
Consulti
ng
Finan
ce
Tele
com /
Comm
Serv
ices
E-Com
merc
e
Educatio
n
Consum
er Pro
ducts
Hosp
itals
& Heal
thcar
e
Media
/Adve
rtisin
g
Non-P
rofit
100.00%
75.00%
50.00%
25.00%
0.00%
Industry
Learn more at saucelabs.com
4
TOP BROWSERS BY INDUSTRY
This is a measure of total number of browsers tested by industry. Firefox and
Chrome dominate across most industries. In media and non-profit FireFox
is the leading browser. Again, in ecommerce we see significant use of Safari
where it is assumed that Macs are a significant factor.
Chrome Firefox IE Safari
BROWSER TESTS BY INDUSTRY
Technolo
gy
Retail
Consulti
ng
Finan
ce
Tele
com /
Comm
Serv
ices
E-Com
merc
e
Educatio
n
Consum
er Pro
ducts
Hosp
itals
& Heal
thcar
e
Media
/Adve
rtisin
g
Non-P
rofit
100.00%
75.00%
50.00%
25.00%
0.00%
Industry
CHROME BROWSER VERSIONS
Chrome is an evergreen browser which means it updates automatically. So it is
expected that testing follows a regular upgrade cadence and we generally see
testers upgrading to the latest versions. This was true in late 2015, however,
in early 2016 version 48 dominated for half the year.
L ATEST GOOGLE CHROME VERSION ADOPTION ( V. 44 - 51 )
10/1/15 1/1/16 4/1/16 7/1/16
5150494847464544
Learn more at saucelabs.com
5
FIREFOX BROWSER VERSIONS
FireFox finds its heaviest use outside North America. With regular updates, we
find that users tend to test on the latest versions to ensure their apps continue
to function properly.
L ATEST F IREFOX VERSION ADOPTION ( V. 40 - 47)
4746454443424140
10/1/15 1/1/16 4/1/16 7/1/16
OTHER BROWSERS
For IE and Safari, people generally upgrade upon new OS versions so
organizations generally test on the major versions. As a result, we don’t see
much variation in testing and have not included them in this report.
SUMMARY
As you can see, the browser and OS combinations that matter to your
organization will differ based on your industry and the demographics of your
users. Regardless, the more test coverage you have, means a higher quality
app and fewer bugs that your users will experience. For a complete list of the
700+ browser and OS combinations provided by Sauce Labs, please go to
www.saucelabs.com/platforms.
Learn more at saucelabs.com
6
Sauce Labs provides the world’s largest cloud-based platform for the
automated testing of web and mobile applications. Its award-winning
service eliminates the time and expense of maintaining an in-house testing
infrastructure, freeing development teams of any size to innovate and
release better software, faster.
Sauce Labs is a privately held company funded by Toba Capital, Salesforce
Ventures, Triage Ventures and the Contrarian Group. For more information,
please visit saucelabs.com.
SAUCE LABS INC. 539 BRYANT STREET #303 SAN FRANCISCO, CA 94107 USA
ABOUT SAUCE L ABS