testing mobile apps - cdn.ttgtmedia.comcdn.ttgtmedia.com/rms/pdf/mobile-application-testing.pdf ·...

E-guide

Testing Mobile Apps Criteria for quality assurance, tracking changes, and a review of 5 great tools for developers

http://searchcrm.techtarget.com/




of 14

In this e-guide

App Quality Alliance compiles mobile testing criteria p.2

Five mobile testing tools for developers to consider p.6

How to track configuration changes in mobile app testing p.12

E-guide

In this e-guide:

Developers and service providers with a myriad of innovative ideas are excited by the capabilities made possible by mobile applications. Testing is important before and after product release, as information is in a constant state of flux, and customers expect the best from mobile apps.

Ahead, App Quality Alliance (AQuA) shares testing guidelines that will give developers confidence that they are testing mobile apps thoroughly. Next, we review five mobile testing tools (Appium, Selendroid, ios-driver, Sauce Labs and Perfecto Mobile), to kick-start your research in a growing software market. Finally, software quality and testing veteran Amy Reichert reviews tips on how to track configuration changes in mobile app testing.






of 14

In this e-guide




E-guide

App Quality Alliance compiles mobile testing criteria

Gerie Owen, Test Architect/Business Solutions Analyst at Eversource

For development teams just entering the wonderful world of mobile application development, mobile application testing can be full of unknowns. Developers and quality engineers may find themselves wondering what they've forgotten to test against, or worse -- missing bugs because they didn't know to perform certain tests. The App Quality Alliance is pulling together data from its members to create testing guidelines that will give developers confidence that they are testing mobile apps thoroughly.

The App Quality Alliance (AQuA) is a non-profit group working with the industry to improve the quality of mobile apps. Its members include AT&T, LGE, Microsoft, Motorola, Oracle, Samsung and Sony Mobile. SearchSoftwareQuality caught up with Martin Wrigley, executive director of AQuA, for an interview at the Mobile App Europe conference in Potsdam, Germany.

SearchSoftwareQuality: In your presentation, you spoke of 10 poor practices still in use by many app developers. Can you describe a couple of the most insidious?






http://www.appqualityalliance.org/

http://mobileappeurope.com/

of 14

In this e-guide




E-guide

Martin Wrigley: Apps need to work well in a real environment, in customers' hands. But many developers use ideal environments for the functional testing of their [apps]. This may mean that they have the fastest, newest phone, or that they only use Wi-Fi connectivity, or that they forget that real users fill up their phones' device memory with other apps, photos, music and messages.

Developers don't always remember that a device can lose connectivity -- and their app doesn't necessarily behave in a way that makes sense to the user when that happens. It isn't good for an app to freeze when the device is set on airplane mode -- but that still happens all too frequently.

There is no substitute for having a separate team (apart from the development team) test an app on real devices in a real situation before giving the app to users.

Tell us about AQuA's app testing criteria. How were these developed?

Wrigley: AQuA has criteria for Java, Android and Apple iOS apps. We are working on Microsoft criteria as well. The criteria are designed to catch the majority of common errors in apps. These criteria have been developed over time, mostly in response to errors that our members have found in apps. As we find new trends, such

The testing criteria are reviewed on an ongoing basis, and new versions are released usually two or three times a year, as necessary.






http://searchsoftwarequality.techtarget.com/answer/Functional-testing-Unit-testing-integration-testing-and-beyond

http://searchsoftwarequality.techtarget.com/answer/Functional-testing-Unit-testing-integration-testing-and-beyond


of 14

In this e-guide




E-guide

as one recently where apps were neglecting to release the camera on devices, we add to or amend the criteria to catch these trends.

Our member representatives are involved in apps on a day-to-day basis, either testing apps on devices, looking at the impact on the network, or delivering apps to paying customers and having a reputation for quality.

The testing criteria are reviewed on an ongoing basis, and new versions are released usually two or three times a year, as necessary. We receive a good amount of feedback from the thousands of developers who have signed up to receive information and updates from us, and we ask for new versions of the criteria to be reviewed by them before we make updates public, in much the same way that you should have someone test your app externally.

AQuA members debate each new addition to the testing criteria. We are keen to ensure that they remain "right-sized" and relevant for apps that are downloaded today. We consider the criteria to be a complete but minimum set of tests needed to ensure a good customer experience.

How will following these criteria improve app quality?

Wrigley: If a developer follows the test criteria, he will avoid the top 10 errors and more. The testing criteria give developers a QA system in a box. Since the launch of the Apple App Store, we've seen the introduction of tools that could be used by designers as well as developers. We are seeing






http://searchsoftwarequality.techtarget.com/answer/Clarifying-software-requirements

http://searchsoftwarequality.techtarget.com/definition/quality-assurance

of 14

In this e-guide




E-guide

apps that are much more attractive to look at, but the teams behind them may not necessarily have the software engineering skills that the traditional development teams may have had. As a consequence, they may not have the testing expertise to look beyond the immediate functional testing that people tend to do.

The AQuA testing criteria gives developers a cost-effective way of performing a structured set of tests -- a checklist -- to avoid easily overlooked errors that otherwise keep causing issues in the apps.

What testing advice do you have for teams building their first mobile apps?

Wrigley: In the planning of your app, think about the types of tests you need to do and don't skimp on them. It's always tempting to put in another feature and miss out on some of the testing, but if that ends up with your app crashing, that extra feature will not be much compensation.

Next article






of 14

In this e-guide




E-guide

Five mobile testing tools for developers to consider

Matthew Heusser, Consultant, Writer and Change Agent focusing on Quality Software

Delivery and Justin Rohrman, Consulting Software Tester and Writer

Mobile testing tools are experiencing a growth spurt. New products and services emerge nearly every day, and the time and energy needed to evaluate them may make all the free trials in the world seem not so free. This article looks at several mobile testing tools, listing their benefits, features and tradeoffs to help testers and IT managers make smarter investment decisions.

The first major challenge with mobile testing is the host -- that is, the device the tests will run on. Unlike a laptop or desktop, where the tests are developed and run on the same device, automated tests are typically planned and programmed on a laptop and run on some mobile device that may be plugged into the laptop and emulated. Or the mobile device might run somewhere else and be connected to the test driver over the Internet. Most large testing programs require testing on a handful, if not dozens, of machines, making physical plug-and-play untenable. These challenges should receive special attention when reviewing tools.






of 14

In this e-guide




E-guide

Appium

Appium is a small server written in node.js. Appium allows users to access their Android or iOS environment where they have the server running.

Since Appium uses the WebDriver framework, it can run tests in any language that WebDriver supports. Webdriver is a popular Web API that developers can program in many languages, including Ruby, Python, JavaScript and many more.







of 14

In this e-guide




E-guide

Appium interacting with iOs

On the plus side, Appium is free and based on open source components. On the downside, that means setup can be a hassle; you have to install open source prerequisites and make sure the versions are compatible.

Selendroid and ios-driver

Two other options fairly similar to Appium are Selendroid (which recently replaced AndroidDriver) and ios-driver. They are both device-specific implementations of Selenium WebDriver for mobile devices. Selendroid is a







of 14

In this e-guide




E-guide

framework for testing software on Android devices; ios-driver applies specifically to iOS devices.

With Selendroid, testers can run tests on an emulator or on an actual device. As of version 0.6.5, ios-driver does not support running tests on real devices. But that feature may be added in the next release.

Testers using Selenium products can benefit from the massive community of people that are also Selenium customers. Several websites offer free community-based support and testers can rest assured that most problems they experience will have been covered by someone else.

Sauce Labs

Sauce Labs offers a mobile testing product based on the Appium framework. Strictly speaking, Sauce Labs uses the Appium tool to host the tests, providing the environment for running digital tests. This approach provides the same benefits of freedom of programming language and dual (iOS, Android) platform support while removing the need for physical hardware.

In other words, Sauce Labs uses the Appium framework, but gives testers the device to run the tests on a virtual mobile device, then bills the device as a metered service by the CPU hour. This service is built in the same way as






of 14

In this e-guide




E-guide

other Sauce Lab products, so testers get similar features, such as the potential to watch videos of tests as they run, or to link those errors reported in a test to a particular point in the video, making debugging much easier.

However, unlike the previously mentioned tools, mobile testing supported by the Sauce Labs platform is not free. The manual subscription plan is $12.00 a month, the individual plan is $49.00 a month and the small team plan is $149.00 a month. Enterprises should contact Sauce Labs for further pricing information.

Perfecto Mobile

Perfecto Mobile is not based on WebDriver. It has a native UI based on a proprietary TCL-based programming language. This approach lets testers who are not programmers select behaviors from a simple, no-coding development environment and write expected results in simple English. Tests designed with these proprietary tools may be more costly to move away from than tests written with open-source tools.

In addition to its native UI, Perfecto Mobile provides a RESTful API for integration with other interfaces including WebDriver and JUnit. Like Sauce Labs, Perfecto Mobile can provide either simulators or cloud-based real devices to run tests on. Perfecto Mobile recommends running tests on real






of 14

In this e-guide




E-guide

devices. Perfecto Mobile provides live scheduled demos and prerecorded videos, but less extensive written documentation than the other products listed here.

The choice is yours

Driving the user interface with mobile testing tools is massively popular in the software world, and automation of mobile websites is quickly becoming common practice. This growth has caused an explosion in companies creating mobile testing tools. Now that testers have so many tools to choose from, the choice is more complicated, often made worse by folklore surrounding test automation and marketing material. Hopefully, this article provides a starting point for testers' research.

Next article






of 14

In this e-guide




E-guide

How to track configuration changes in mobile app testing

Amy Reichert, Software quality and testing veteran

Configuration changes are updated constantly in mobile application development as development companies try to keep up with demand. Mobile app testing, on the other hand, is already complicated by the number of paths needed to ensure a quality product. The process is burdened by testing requirements around screen size, platforms, connectivity and configuration.

With an already full plate, how does a mobile app testing team add in testing configuration changes in a release? Typically, configuration settings are not the highest priority changes, so testing them falls behind other feature testing commitments. You can try a couple of options to see what works for the mobile app testing team's schedule.

The first is random selection. For example, if there are 20 configuration changes pick the top 10 that are most important to clients, or that occur in the most-used features. Or list each by priority or importance and by the popularity of the associated feature and test your way down the list.






of 14

In this e-guide




E-guide

Testers also can divide the list between releases. Granted, not all features will be tested in the initial release, but a group can be tested each release until testing is completed. Create rotating test suites and mix them into your regression testing executions. Once each has been thoroughly tested, go back through and mix them at random or by feature and add them to the regression test along with their matching feature.

Another logical option is to test them with the feature they control. If the testing numbers are too high rotate the feature and create small subsets of test suites that cover a piece of each feature. Switch out testing suites each regression cycle. Each testing suite contains a portion of every feature so in this way, full testing occurs over a series of regression testing events but all features are covered to some extent.

Next Article






of 14

In this e-guide




E-guide

Getting more PRO+ exclusive content This e-guide is made available to you, our member, through PRO+ Offers—a collection of free publications, training and special opportunities specifically gathered from our partners and across our network of sites.

PRO+ Offers is a free benefit only available to members of the TechTarget network of sites.

Take full advantage of your membership by visiting http://pro.techtarget.com/ProLP/ Images; Fotalia

© 2015 TechTarget. No part of this publication may be transmitted or reproduced in any form or by any means without written permission from the publisher.






http://pro.techtarget.com/ProLP/

testing mobile apps - cdn.ttgtmedia.comcdn.ttgtmedia.com/rms/pdf/mobile-application-testing.pdf ·...

Documents