mobile applications testing: from concepts to practice
DESCRIPTION
As applications for smartphones and tablets become incredibly popular, organizations encounter increasing pressure to quickly and successfully deliver testing for these devices. When faced with a mobile testing project, many testers find it tempting to apply the same methods and techniques used for desktop applications. Although some of these concepts transfer directly, testing mobile applications presents its own special challenges. Max Saperstone says if you follow the same practices and techniques as you have before, you will miss critical defects. Learn how to effectively test mobile applications, and how to add more structure and organization to generate effective test ideas to exploit the capabilities and weaknesses of mobile devices. Max shares first-hand experiences with testing mobile applications and discusses how to address various challenges. Work on real problems on your own device and learn firsthand how to be productive while testing mobile applications.TRANSCRIPT
rial
Presented by:
Max Saperstone
rought to you by:
340 Corporate Way, Suite Orange Park, FL 32073 888‐2
MB AM Tuto4/7/2014 8:30 AM
“Mobile Applications Testing: From Concepts to Practice”
Coveros, Inc.
B
300,68‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
or almost a decade, Max Saperstone has been a test engineer focusing on test automation
ng
Max Saperstone Coveros, Inc.
Fand the continuous integration/continuous delivery process. Max specializes in open source tools—Selenium, JMeter, AutoIT, Cucumber, and Chef. He has led several testing automationefforts, including developing an automated suite focused on web-based software to operate over several applications. Max also headed a major project developing an automated testing structure to run Cucumber tests over multiple test interfaces and environments, while developing a system to keep test data “ageless.” He is currently developing a new testiarchitecture for SecureCI to allow testing of multiple interfaces, custom reporting, and minimal test upkeep.
Mobile Application Testing Tutorial
Student Handbook
February 2014
Copyright 2014 Coveros, Inc. All rights reserved.
Copyright © 2014 by Coveros, Inc. All rights reserved. No part or parts of this Coveros, Inc.
documentation may be reproduced, translated, stored in any electronic retrieval system,
transmitted in any form or by any means, electronic, mechanical, photocopying, recording or
otherwise, without prior written permission of the copyright owner.
Coveros, Inc.
4000 Legato Rd
Suite 1100
Fairfax, VA 22033
Phone: 703-349-6109
www.coveros.com
Mobile Application Testing Tutorial Welcome
Copyright 2014 Coveros, Inc. All rights reserved. Page 1
Welcome
Slide 1
Welcome to the Mobile Testing for Test Professionals training course. In this course, you will learn all about how to apply Mobile Testing to your Software Testing efforts.
1© Copyright 2013 Coveros, Inc.. All rights reserved.
Mobile Application Testing
Tutorial
Mobile Application Testing Tutorial Course Trainers
Copyright 2014 Coveros, Inc. All rights reserved. Page 2
Course Trainers
Slide 2
Slide 3
2© Copyright 2013 Coveros, Inc.. All rights reserved.
Trainer
Jeffery Payne
Jeffery Payne is CEO and founder of Coveros, Inc., a software company that helps organizations accelerate the delivery of secure, reliable software. Coveros uses agile development methods and a proven software assurance framework to build security and quality into software from the ground up. Prior to founding Coveros, Jeffery was Chairman of the Board, CEO, and co-founder of Cigital, Inc. Under his direction, Cigital became a leader in software security and software quality solutions, helping clients mitigate the risk of software failure. Jeffery is a recognized software expert and popular speaker at both business and technology conferences on a variety of software quality, security, and agile development topics. He has also testified before Congress on issues of national importance, including intellectual property rights, cyber-terrorism, Software research funding, and software quality.
3© Copyright 2013 Coveros, Inc.. All rights reserved.
Trainer
Alan Crouch
Alan Crouch is a Senior Software Security Specialist with Coveros, Inc. a Virginia-based firm focusing on agile, software quality, and application security. Alan has worked closely with federal agencies and private companies to advise, audit, and support IT security and governance teams. In addition to his cybersecurity experience, he has a strong background in software engineering, test analysis, test automation, and security testing. With a passion for software and security, Alan has focused his career on building secure software and developing better software security practices.
Mobile Application Testing Tutorial Course Trainers
Copyright 2014 Coveros, Inc. All rights reserved. Page 3
Course Trainers
Slide 4
Slide 5
4© Copyright 2013 Coveros, Inc.. All rights reserved.
Trainer
Max Saperstone
Max Saperstone has been working as a Software and Test Engineer for almost a decade, with a focus on Test Automation and the CI/CD process. He specializes in open source tools, including the Selenium Tool Suite, JMeter, AutoIT, Cucumber, and Chef. Max has lead several testing automation efforts, including developing an automated suite focusing on web-based software to operate over several applications for Kronos Federal. He also headed a project with Delta Dental, developing an automated testing structure to run Cucumber tests over multiple test interfaces and environments, while also developing a system to keep test data 'ageless.' He is currently developing a new testing architecture for SecureCI™ to allow testing of multiple interfaces, custom reporting, and minimal test upkeep.
5© Copyright 2013 Coveros, Inc.. All rights reserved.
Trainer
Richard Mills
Richard Mills has more than 20 years of experience in software engineering specializing in software development with a concentration on pragmatic software process and tools. Rich has a specific focus in Agile development methods, static and dynamic software analysis tools, configuration management, engineering, automated software build management, and continuous integration. He currently works as a Senior consultant with Coveros, Inc. Helping his customers build software better, faster and more securely. As a consultant, Rich helps customers improve software delivery by coaching and mentoring in the Agile development methodologies, automating software build, test, deployment and employing secure development techniques. Prior to joining Coveros, Rich spent 16 years with Cigital, Inc. fulfilling a wide range of roles including consulting, training, and software engineering.
Mobile Application Testing Tutorial Course Trainers
Copyright 2014 Coveros, Inc. All rights reserved. Page 4
Course Trainers
Slide 6
6© Copyright 2013 Coveros, Inc.. All rights reserved.
Trainer
Jeffrey Pierce
Jeff Pierce is a Managing Consultant with Coveros, Inc., a software company that helps organizations accelerate the delivery of secure, reliable software. Coveros uses agile development methods and a proven software assurance framework to build security and quality into software from the ground up. Jeff is a leader with over twenty years experience in information technology (IT) program/project management and client management in all phases of business investment and system development lifecycle. He has proven technical excellence in development and deployment of strategic web-based enterprise information systems utilizing cloud computing and continuous integration technologies. Jeff also has experience with strategy and roadmap development, policy and procedure development, program management office implementation, IT and business management, AGILE SCRUM and XP management, systems engineering, technical team building and leading large and small on-shore and off-shore full-time and contract staff with an outstanding record of accomplishment.
Mobile Application Testing Tutorial About Coveros
Copyright 2014 Coveros, Inc. All rights reserved. Page 5
About Coveros
Slide 7
Coveros is a consulting company that helps organizations build better software. We provide software development, application security, QA/testing, and software process improvement services. Coveros focuses on organizations that must build and deploy software within the constraints of significant regulatory or compliance requirements. The primary markets we serve include: DoD, Homeland Security & associated critical infrastructure companies, Healthcare providers, and Financial services institutions
7© Copyright 2013 Coveros, Inc.. All rights reserved.
Coveros helps organizations accelerate the delivery of secure, reliable software
Our consulting services:– Agile software development
– Mobile application development
– Application security
– Software quality assurance
– Software process improvement
Our key markets:– Financial services
– Healthcare
– Defense
– Critical Infrastructure
About Coveros
Corporate Partners
Mobile Application Testing Tutorial Agenda
Copyright 2014 Coveros, Inc. All rights reserved. Page 6
Agenda
Slide 8
8© Copyright 2013 Coveros, Inc.. All rights reserved.
Agenda
Introduction to Mobile Testing
Mobile Web Applications
Native Mobile Applications
Unique Mobile Testing Considerations
Mobile Testing Platforms
Mobile Testing Techniques
Tools for Mobile Testing
Wrap-Up
Mobile Application Testing Tutorial Introduction to Mobile Testing
Copyright 2014 Coveros, Inc. All rights reserved. Page 7
Introduction to Mobile Testing
Slide 9
Slide 10
9© Copyright 2013 Coveros, Inc.. All rights reserved.
Introduction to Mobile Testing
10© Copyright 2013 Coveros, Inc.. All rights reserved.
Agenda
What is Mobile Application Testing?
Why is Mobile Application Testing Important?
What Do You Need to Know to Be a Good Mobile Tester?
Types of Mobile Applications
Mobile Application SDLC
Mobile Application Testing Tutorial Definition of Mobile Application Testing
Copyright 2014 Coveros, Inc. All rights reserved. Page 8
Definition of Mobile Application Testing
Slide 11
Companies everywhere are rapidly building and deploying mobile applications. Testing these applications must be accurate, reliable, usable and rapid – keeping up with the pace of development and changing mobile platforms. Additionally MAT will look at testing areas that have typically only been tested by hardware engineers and will need to testing focus in order for mobile applications to be built, delivered, and executed. Title Month Year
11© Copyright 2013 Coveros, Inc.. All rights reserved.
Definition of Mobile Application Testing
Mobile Application Testing is a process by which application software developed for mobile devices is tested for functionality, security, usability and consistency.
Although Mobile Application Testing is a relatively new aspect of software testing, all existing standard testing practices and techniques can be applied.
What is Mobile Application Testing?
Mobile Application Testing Tutorial Mobile Application Testing Importance
Copyright 2014 Coveros, Inc. All rights reserved. Page 9
Mobile Application Testing Importance
Slide 12
App stores can reduce the time in the feedback loop, and customers can be utilized for early and frequently feedback. Without it, you may be developing an application your customers don’t want
As with all software, code quality verification is important. Users are quicker than ever to abandon an application that doesn’t work perfectly the first and every time.
Users have high expectations and are App stores also make it extremely easy to get software out to the public at a rapid pace. You want to ensure your company is putting their best foot forward to maintain a good reputation Having a consistent, predictable release process will ensure good software is released every time
The extensive choice of browsers, operating systems, devices and applications make it possible for enterprises to develop new, interactive and highly personal ways for customers and employees to access products, services and applications. Mobile devices are at the same time sources of rich information, powerful entertainment and purchasing platforms, and important work tools.
Companies that successfully implement mobile strategies engage both customers and employees in new ways, and create an environment of personal interaction that enhances sales and productivity. What makes mobile so attractive to consumers—instant access to a multitude of apps; a wide range of rapidly evolving devices and functions; a choice of operating systems; new, cloud-based services that require no storage on the device—also makes it tricky for companies to deploy effectively.
The fourth annual World Quality Report—from Cap Gemini, Sogeti and HP—reveals that only 31% of organizations surveyed currently have formal processes in place to test their mobile applications, while just 18% of those have a focus on security
12© Copyright 2013 Coveros, Inc.. All rights reserved.
App stores reduce the user feedback loop.
User feedback is both early and frequent.
Usability and quality is king, when users are quicker than ever to abandon an application that doesn’t work the first time.
Users can get updates rapidly and expect seamless updates and integration.
Mobile devices are changing quickly and constantly getting new capabilities and features.
Why Is Mobile Application Testing Important?
Mobile Application Testing Tutorial Mobile App Testing Difference
Copyright 2014 Coveros, Inc. All rights reserved. Page 10
Mobile App Testing Difference
What Do You Need To Be a Good Mobile Tester?
13© Copyright 2013 Coveros, Inc.. All rights reserved.
Mobile Application Testing Tools aren’t Mature or Available– Many tools are created for specific device/OS combinations
– Cross-platform tool support is limited
– No one tool will do
Mobile Application Testing Requires Specialized Skills– UI and usability focused
– Strong testing background
– Understand networking
– Engineering/hardware focus
– Constantly learning
Cloud Components
Cost of Entry
Automation Immaturity– Automation image based and time consuming
– Automation difficult to leverage outside the framework
What Makes Mobile App Testing Different?
14© Copyright 2013 Coveros, Inc.. All rights reserved.
What Do You Need To Be a Good Mobile Tester?
History of Mobile Devices and Emerging Trends
Basic Software Testing Skills
Basic Understanding of the Mobile Software Lifecycle
Basic Understanding of Telecommunications
Awareness of Mobile Operating Systems/Platforms– Experience in virtualization for Linux, or VMWare, and data center
operations
Experience with cloud infrastructure and SaaS
Knowledge of Mobile testing tools and frameworks
Knowledge and experience with test automation tools
Mobile Application Testing Tutorial Understand the Basics of Software Testing
Copyright 2014 Coveros, Inc. All rights reserved. Page 11
Understand the Basics of Software Testing
Slide 15
This course is not meant to teach you the basics of software testing, it assumes you know them and can apply them.
15© Copyright 2013 Coveros, Inc.. All rights reserved.
Understand the Basics of Software Testing
Knowledge of Software Testing basics is essential to testing any platform. Even if you just starting your career, you need to know:
– Software Testing Goals and Objectives
– Software Testing Principles
– Software Testing processes, methodologies, and techniques
If you don’t have a strong foundation in software testing, we suggest you start simple and enroll in some basic training programs.
What You Need to Know
Mobile Application Testing Tutorial Understand the Basics of Telecommunications
Copyright 2014 Coveros, Inc. All rights reserved. Page 12
Understand the Basics of Telecommunications
Slide 16
Understanding the basics of Telecommunications will always give you advantages when doing testing
mobile applications. Title Month Year
16© Copyright 2013 Coveros, Inc.. All rights reserved.
Understand the Basics of Telecommunications
Mobile App Testing involves networks, not just the app or device itself.
Can you identify and understand the differences in these technologies?
– 2G, 3G, 4G, Edge, LTE, CDMA, GPRS, GSM, HSCSD, SIM, SMS, WAP, TCP, UDP, Wi-Fi
Telecom is global – …therefore, so is your app.
What You Need to Know
Mobile Application Testing Tutorial Mobile Operating Systems/Platforms
Copyright 2014 Coveros, Inc. All rights reserved. Page 13
Mobile Operating Systems/Platforms
Slide 17
Slide 18
17© Copyright 2013 Coveros, Inc.. All rights reserved.
Awareness of Mobile Operating Systems/Platforms
OS Design
– Integration Points– Device Capability Usage– Security
Features/Vulnerabilities
Device Operation
– Standalone
– Offline
– Wi-Fi
– Interruptions
What You Need to Know
18© Copyright 2013 Coveros, Inc.. All rights reserved.
Awareness of Mobile Operating Systems/Platforms
Device Design
– Variable Screen Size/Resolution
– Device specific Browser Limitations/Behaviors
– Variable input/navigation methodologies Touch Screen
Keyboard/virtual keyboard
Ball/Pointer
Voice
Menu Structures
– CPU Constraints/Variability
– Memory Constraints/Variability
– Power utilization constraints
– Keypad lock
What You Need to Know
Mobile Application Testing Tutorial Emerging Trends
Copyright 2014 Coveros, Inc. All rights reserved. Page 14
Emerging Trends
Slide 19
19© Copyright 2013 Coveros, Inc.. All rights reserved.
It’s important to stay on top of emerging trends!
Staying on top of emerging trends will better help you
understand what you may need to test for when your
business wants to add or integrate a new piece of
technology into your mobile application. In addition, it
will make your products more competitive in the
marketplace.
Emerging Trends
Mobile Application Testing Tutorial Types of Mobile Applications
Copyright 2014 Coveros, Inc. All rights reserved. Page 15
Types of Mobile Applications
Slide 20
Slide 21
20© Copyright 2013 Coveros, Inc.. All rights reserved.
Types of Mobile Applications
Mobile Web Apps– Non-Responsive
– Responsive
Native Apps– Device-Specific
– Responsive
– Hybrid Apps
21© Copyright 2013 Coveros, Inc.. All rights reserved.
Mobile Web Applications
Web apps are really websites that, in many ways, look and feel like native applications. They are run by a browser and typically written in HTML5.
Users access them as they would access any web page: they navigate to a special URL.
Types of Mobile Applications
Mobile Application Testing Tutorial Native Applications
Copyright 2014 Coveros, Inc. All rights reserved. Page 16
Native Applications
Hybrid Applications
22© Copyright 2013 Coveros, Inc.. All rights reserved.
Native Applications
Native apps live on the device and are accessed through icons on the device home screen.
Native apps are installed through an application store and are developed specifically for one platform.
They can take full advantage of all the device capabilities and can work offline.
Types of Mobile Applications
23© Copyright 2013 Coveros, Inc.. All rights reserved.
Hybrid Applications
Hybrid apps are application that combines elements of both native and web applications.
Hybrid apps are installed through an application store but integrate with a large amount of content from Web-based services.
They can often function without an Internet connection, but may have limited capabilities when disconnected
The majority of apps fall into this category.
Types of Mobile Applications
Mobile Application Testing Tutorial Mobile Application SDLC
Copyright 2014 Coveros, Inc. All rights reserved. Page 17
Mobile Application SDLC
Slide 24 - [Source: Harty, Julian. “Mobile Apps”]
Title
Slide 25
24© Copyright 2013 Coveros, Inc.. All rights reserved.
The Waterfall Mobile Application SDLC
Requirements
Implementation
• Design
• Code
Verification
• Unit Tests
• Internal Installation
• System Tests
Launch/Release
• Pre-Publication
• Publication
Engagement
• Search
• Trust
• Download
• Installation
• Upgrade
• Downgrade
Validation
• Payment
• Use
• Feedback
25© Copyright 2013 Coveros, Inc.. All rights reserved.
The Agile Mobile Application SDLC
La
un
ch
Rele
ase
s 1
-n
Sprint 1Sprint 0 Sprint 2
End Game Engagement
Requirements
Design
Implementation
Testing
Launch/Release
Engagement/Validation
Mobile Application Testing Tutorial Mobile Web Applications
Copyright 2014 Coveros, Inc. All rights reserved. Page 18
Mobile Web Applications
Agenda
26© Copyright 2013 Coveros, Inc.. All rights reserved.
Mobile Web Applications
27© Copyright 2013 Coveros, Inc.. All rights reserved.
Agenda
– Overview
– Types of Mobile Web Apps Non-Responsive Applications
Responsive Applications
– Testing
– Technologies
Mobile Application Testing Tutorial What is a Mobile Web Application?
Copyright 2014 Coveros, Inc. All rights reserved. Page 19
What is a Mobile Web Application?
Exercise
28© Copyright 2013 Coveros, Inc.. All rights reserved.
Overview
What is a Mobile Web Application?
A mobile web application is an application that is accessed by users over a network such as the Internet or an intranet on a mobile device.
Mobile Web applications are generalized for multiple platforms and not installed locally but made available over the Internet through a browser.
e.g. Online Banking Services
Mobile Web Apps can be written with HTML, CSS and JavaScript
29© Copyright 2013 Coveros, Inc.. All rights reserved.
Check your HTML 5 capabilities
With your Mobile Device, go to: – http://webdirections.org/deviceTest/
Exercise
Mobile Application Testing Tutorial Advantages
Copyright 2014 Coveros, Inc. All rights reserved. Page 20
Advantages
Non-Responsive Apps
30© Copyright 2013 Coveros, Inc.. All rights reserved.
Overview
Advantages
You can use simple HTML, CSS, JS for most mobile applications
You can maintain and fix bugs in real time.
Quick upgrade and migration
Disadvantages
You can’t sell your application
Must be responsive to multiple resolutions, operating systems and browsers
Struggle with performance and testability
31© Copyright 2013 Coveros, Inc.. All rights reserved.
Types of Mobile Web Apps
Non-Responsive Apps
A non-responsive web application that responds the same way on a laptop’s web browser as it would on a mobile device.
More and more website’s are moving away from non-responsive apps, but there are many reasons why someone would choose to have a non-responsive app.
Mobile Application Testing Tutorial Responsive Apps
Copyright 2014 Coveros, Inc. All rights reserved. Page 21
Responsive Apps
Responsive Web Apps - Example
32© Copyright 2013 Coveros, Inc.. All rights reserved.
Types of Mobile Web Apps
Responsive Apps
Responsive Web design is an approach that suggests that an app should respond to the user’s behavior and environment based on screen size, platform and orientation.
The practice often consists of flexible screens, grids, and layouts, images and an intelligent use of CSS media queries, automatically switching to accommodate for resolution, image size and scripting abilities.
33© Copyright 2013 Coveros, Inc.. All rights reserved.
Types of Mobile Web Apps
Responsive Web Apps - Example
1920x1080 Screen Resolution – Windows 7
www.coveros.com
Mobile Application Testing Tutorial Responsive Web Apps - Example
Copyright 2014 Coveros, Inc. All rights reserved. Page 22
Responsive Web Apps - Example
Responsive Web Apps - Example
34© Copyright 2013 Coveros, Inc.. All rights reserved.
Types of Mobile Web Apps
Responsive Web Apps - Example
1920x1080 Screen Resolution – Windows 7
www.coveros.com
35© Copyright 2013 Coveros, Inc.. All rights reserved.
Types of Mobile Web Apps
Responsive Web Apps - Example
1136 x 640 Screen Resolution – iPhone5
www.coveros.com
Mobile Application Testing Tutorial Testing a Mobile Web App
Copyright 2014 Coveros, Inc. All rights reserved. Page 23
Testing a Mobile Web App
Slide 36
36© Copyright 2013 Coveros, Inc.. All rights reserved.
Testing
Testing a Mobile Web App
Testing a mobile web app is fairly similar to how you would test any other web application. In fact, you can often use the same tools.
Using emulators and user-agents you can often replicate resolutions and device types on your laptop.
However, there are some things you won’t be able to replicate (gestures, shakes, etc.), so don’t think that you can avoid testing on an actual mobile device.
For responsive web apps, you can often reuse tests suites for various resolutions, but no good test plan for a responsive app will have one set of identical tests for each resolution.
Mobile Application Testing Tutorial
Copyright 2014 Coveros, Inc. All rights reserved. Page 24
This Page Intentionally left blank
Mobile Application Testing Tutorial Native Mobile Applications
Copyright 2014 Coveros, Inc. All rights reserved. Page 25
Native Mobile Applications
Agenda
37© Copyright 2013 Coveros, Inc.. All rights reserved.
Native Mobile Applications
38© Copyright 2013 Coveros, Inc.. All rights reserved.
Agenda
Overview
Types of Native Apps
– Device Specific Applications– Responsive Applications– Hybrid Applications
Testing
Technologies
– OS API– APP Store– Device Capabilities
Mobile Application Testing Tutorial What is a Native Application?
Copyright 2014 Coveros, Inc. All rights reserved. Page 26
What is a Native Application?
Device-Specific Applications
39© Copyright 2013 Coveros, Inc.. All rights reserved.
Overview
What is a Native Application?
Native applications are developed for a specific platform and installed on a mobile device.
Native applications can be pre-installed or installed through an app store on demand.
e.g. Phone app vs. Instagram.
Native applications are built in a language for a specific mobile device
40© Copyright 2013 Coveros, Inc.. All rights reserved.
Types of Native Apps
Device-Specific Applications
Device-Specific Applications are native apps built for not only one platform (e.g. iOS) but also a device (e.g. iPhone).
Responsive Applications
Responsive Applications are platform-specific native apps that are built to respond to various screen size, resolutions and other built-in device capabilities based on the mobile device the application is installed upon.
Mobile Application Testing Tutorial Hybrid Applications
Copyright 2014 Coveros, Inc. All rights reserved. Page 27
Hybrid Applications
Testing a Native App
41© Copyright 2013 Coveros, Inc.. All rights reserved.
Types of Native Apps
Hybrid Applications
By a strict definition, Hybrid apps are technically native apps. However, they are part native apps, part web apps; they live in an app store and can take advantage of the many device features available but rely on HTML being rendered in a browser embedded within the app itself.
Hybrid apps allow an organization to have consistency between different device platforms.
They can also provide presence in the app store without having to build much of a native app.
They can provide some cross-platform development, because HTML components can be reused to reduce costs.
42© Copyright 2013 Coveros, Inc.. All rights reserved.
Testing
Testing a Native App
Testing a native app is a mix of software and hardware testing. A tester has to test the app in all the ways a user may use the app (not just standard user behavior).
Using emulators you can often replicate resolutions, gestures and device platforms on your laptop.
However, there are some things you won’t be able to replicate, so don’t think that you can avoid testing on an actual mobile device.
Since each of the selection points are determined by the developer, test that there is enough space to perform actions and that you are properly applying OS Gestures.
Mobile Application Testing Tutorial Testing a Hybrid App
Copyright 2014 Coveros, Inc. All rights reserved. Page 28
Testing a Hybrid App
Slide 43
43© Copyright 2013 Coveros, Inc.. All rights reserved.
Testing
Testing a Hybrid App
Arguably, testing a hybrid app is the most complicated to plan and strategize. A good tester must know how to test both native app and mobile web app components to test a hybrid app successfully.
You can test many components like you would a mobile web app (even the same tools).
Testing the components built with the native code, should be performed just like a typical native app.
However, don’t forget about your integration points!
There are native mobile browser functionality that can have features that can cause unintended behavior in the embedded browser.
Mobile Application Testing Tutorial Unique Mobile Testing Considerations
Copyright 2014 Coveros, Inc. All rights reserved. Page 29
Unique Mobile Testing Considerations
Agenda
44© Copyright 2013 Coveros, Inc.. All rights reserved.
Unique Mobile Testing Considerations
45© Copyright 2013 Coveros, Inc.. All rights reserved.
Agenda
User Expectations
Network
Data Storage
Operating System
Jail Breaking
Hardware/Device Capabilities
Security
Mobile Application Testing Tutorial User Expectations Challenges
Copyright 2014 Coveros, Inc. All rights reserved. Page 30
User Expectations Challenges
Network Challenges
46© Copyright 2013 Coveros, Inc.. All rights reserved.
User Expectations Challenges
Global Users– Global User Community
Internationalization/Localization requirements
Instant Feedback (online ratings)
Expectations – Intuitive UI
– Performance – 60% of mobile app users will abandon app or site if load time > 3 seconds
– Seamless across multiple devices
– Seamless across various connectivity options (3g, 4g, Wi-Fi)
– Security
– Data utilization
– Power utilization
– Easy Installation and Launch
Testing Mobile Applications
47© Copyright 2013 Coveros, Inc.. All rights reserved.
Network Challenges
Networks– More than 400 mobile networks globally
Numerous Global Standards– NMT (Nordic Mobile Telephony)
FDMA (Frequency Division Multiple Access)
– GSM (Global System for Mobile Communications) TDMA (Time Division Multiple Access)
FDMA (Frequency Division Multiple Access)
– UMTS (3GSM) (Universal Mobile Telecommunications System) W-CDMA (Wideband Code-Division Multiple Access)
– IS-95 (CDMA one) / IS-2000 (CDMA 2000) CDMA (Code-Division Multiple Access)
– WIMAX / LTE (Long Term Evolution) OFDMA (Orthogonal Frequency Division Multiple Access)
Unique Mobile Testing Considerations
Mobile Application Testing Tutorial Network Challenges
Copyright 2014 Coveros, Inc. All rights reserved. Page 31
Network Challenges
48© Copyright 2013 Coveros, Inc.. All rights reserved.
Network Challenges
Multiple Carriers– Verizon, AT&T, Sprint, T-Mobile, Clearwire, US Cellular, Leap
Wireless, nTelos, etc.
– Not all 4G Networks are created equal. 4G for Verizon is not the same as 4G for AT&T
Network Hopping– Cell to Cell Hopping
– Roaming
– Cell to Wi-F
– 3G to 4G, Edge to 3G, etc.
– International Networks
Near Field Communication
Bluetooth
Unique Mobile Testing Considerations
49© Copyright 2013 Coveros, Inc.. All rights reserved.
Network Challenges
Implementation variability and issues– Tunneling
– Web Proxies limiting site connectivity
Transcoding
http header information filtering
Several Bandwidth/Connection Modes– 1G, 2G, 3G, 4G
– Wi-Fi, Wi-Fi/3G, Wi-Fi/4G)
Unique Mobile Testing Considerations
Mobile Application Testing Tutorial Data Storage Challenges
Copyright 2014 Coveros, Inc. All rights reserved. Page 32
Data Storage Challenges
Operating System Challenges
50© Copyright 2013 Coveros, Inc.. All rights reserved.
Small Capacity
Data Privacy / Shared Storage
Data Encryption (or lack there of)
Cloud Data Storage
Shared Data Resources – Photos
– Calendar
– Video
– Messages
– Notifications
Data Storage Challenges
Unique Mobile Testing Considerations
51© Copyright 2013 Coveros, Inc.. All rights reserved.
Operating System Challenges
App Store Integration
OS API
Fragmentation
Backwards Compatibility
No Industry-wide standards
Capability, Capacity and Performance
Unique Mobile Testing Considerations
Mobile Application Testing Tutorial Operating System Challenges
Copyright 2014 Coveros, Inc. All rights reserved. Page 33
Operating System Challenges
52© Copyright 2013 Coveros, Inc.. All rights reserved.
Operating System Challenges
Interruptions are events that would interrupt normal operations
There are two types of Interruptions– Application Interruptions:
Installation, Launch, Run, Exit, Upgrade, Uninstall/Delete
– Device/OS Interruptions Incoming Call
Receiving Incoming Call
Receiving Message
Device Shutdown
Device Restore
Remove Battery
Camera Activated
Loss of Connectivity
Regain Connectivity
Unique Mobile Testing Considerations
Mobile Application Testing Tutorial About Jailbreaking
Copyright 2014 Coveros, Inc. All rights reserved. Page 34
About Jailbreaking
Jail Breaking Challenges
53© Copyright 2013 Coveros, Inc.. All rights reserved.
About Jailbreaking
Jailbreaking (iOS) or Rooting (Android) is the process of removing the limitations on mobile devices through the user of software or hardware exploits.
Jailbreaking permits root access to the operating system, allowing download of non-approved apps, extensions and themes not available otherwise.
Large populations of users utilize Jailbroken/rooted mobile devices.
Corporate policy may permit devices from being Jailbroken
Unique Mobile Testing Considerations
54© Copyright 2013 Coveros, Inc.. All rights reserved.
Jail Breaking Challenges
Providing escalated privileges by overcoming limitations in a computer system or device by deliberating overriding security, administrative and marketing restrictions by the creator
Non-Approved Applications
Malware, Worms– First iOS Worm, named iKree, was released to affect Jailbroken
iPhones in 2009
Unique Mobile Testing Considerations
Mobile Application Testing Tutorial Hardware Challenges
Copyright 2014 Coveros, Inc. All rights reserved. Page 35
Hardware Challenges
Slide 55
Slide 56
55© Copyright 2013 Coveros, Inc.. All rights reserved.
Hardware Challenges
Screen Resolution– Size
– Clarity
– Portrait vs. Landscape
Bluetooth / NFC
Camera
Processing Power
Speaker/Microphone
Flashlight
Accelerometer
Magnetometer
Unique Mobile Testing Considerations
56© Copyright 2013 Coveros, Inc.. All rights reserved.
Hardware Challenges
Touch Screen– Single Touch Inputs
– Multi-Touch Inputs
– Virtual Keyboard
– Multiple Inputs Simultaneously
Physical Inputs– Keypads
– Track Ball
GPS / Location Services
Power Consumption
Memory Usage– Launch, Background, Foreground, Exit
– Simultaneous Applications
– Memory Sweeping
Unique Mobile Testing Considerations
Mobile Application Testing Tutorial Security Challenges
Copyright 2014 Coveros, Inc. All rights reserved. Page 36
Security Challenges
Slide 57
Slide 58
57© Copyright 2013 Coveros, Inc.. All rights reserved.
Security Challenges
Encryption– Data at Rest
On Device
On Middle-Tier Servers
On Databases
Cached Content
– Data in Transit
Privacy– Application Privileges
Features
Data
Capabilities
– 3rd Party Application Integration
User Configurable Trust Settings
Unique Mobile Testing Considerations
58© Copyright 2013 Coveros, Inc.. All rights reserved.
Security Challenges
Operating System– Malware
– Critical Bugs
– Out of Date (Upgrading)
– Support ends after two years typically
Network– No Firewalls
– Few devices limit Internet connections
– Unsecure Communication Channels Bluetooth Discovery Mode
Unsecure Networks
Unauthorized Modification, Jail Breaking, Rooting
Unique Mobile Testing Considerations
Mobile Application Testing Tutorial Mobile Testing Platforms
Copyright 2014 Coveros, Inc. All rights reserved. Page 37
Mobile Testing Platforms
Agenda
59© Copyright 2013 Coveros, Inc.. All rights reserved.
Mobile Testing Platforms
60© Copyright 2013 Coveros, Inc.. All rights reserved.
Agenda
Introduction
Desktop Browsers
Mobile Emulation
Cloud Testing
Physical Device Testing
Remote-Device Testing
Recommendations
Mobile Application Testing Tutorial Choosing the Right Platform
Copyright 2014 Coveros, Inc. All rights reserved. Page 38
Choosing the Right Platform
What is possible with Desktop Browsers?
61© Copyright 2013 Coveros, Inc.. All rights reserved.
Introduction
Choosing the Right Platform
There are many different device platforms, that are often fragmented and grow at different paces
iOS, Android, Blackberry, Windows phone, Symbian, etc.
Multiple form factors and screen resolutions
Users get new devices every 24 months
There are often carrier restrictions on devices (beyond network considerations)
62© Copyright 2013 Coveros, Inc.. All rights reserved.
Desktop Browsers
What is possible with Desktop Browsers?
Desktop browsers can be extremely useful when developing and testing mobile web apps.
Desktop browsers like Chrome and Firefox have many tools to effectively use user-agents to simulate multiple mobile browser types with little or no cost.
Leveraging desktop browsers does not replace device testing but they can provide robust debugging options and can be used to simulate multiple screen sizes.
Mobile Application Testing Tutorial User-Agent Switcher for Chrome
Copyright 2014 Coveros, Inc. All rights reserved. Page 39
User-Agent Switcher for Chrome
Slide 63
https://chrome.google.com/webstore/detail/user-agent-switcher-for-c/djflhoibgkdhkhhcedjiklpkjnoahfmg/reviews?hl=en-US
You can also add user-agent strings as necessary (highly recommended) Title Month Year
63© Copyright 2013 Coveros, Inc.. All rights reserved.
Desktop Browsers
User-Agent Switcher for Chrome
Example: iPhone Agent on Google Chrome in Incognito Mode
Mobile Application Testing Tutorial What is Mobile Emulation?
Copyright 2014 Coveros, Inc. All rights reserved. Page 40
What is Mobile Emulation?
iPhone Emulator
64© Copyright 2013 Coveros, Inc.. All rights reserved.
Mobile Emulation
What is Mobile Emulation?
Emulators offer the less expensive testing option, but they have many limitations for mobile testing and may not give an accurate depiction of the mobile user experience.
However, as part of the mobile testing process the use of emulation has an important role. There is certain testing that you can conduct with emulators that provides value. For example, testing for screen size or visual form factor.
However, emulation is often an approximation of the rendering and can only get you so far.
Leveraging emulators is a positive step towards real device testing, however, not sufficient for comprehensive testing needs.
65© Copyright 2013 Coveros, Inc.. All rights reserved.
Emulator
iPhone Emulator
Mobile Application Testing Tutorial iPad Emulator
Copyright 2014 Coveros, Inc. All rights reserved. Page 41
iPad Emulator
Android Emulator
66© Copyright 2013 Coveros, Inc.. All rights reserved.
Emulator
iPad Emulator
67© Copyright 2013 Coveros, Inc.. All rights reserved.
Emulator
Android Emulator
Mobile Application Testing Tutorial What is Cloud Testing?
Copyright 2014 Coveros, Inc. All rights reserved. Page 42
What is Cloud Testing?
Sauce Labs
68© Copyright 2013 Coveros, Inc.. All rights reserved.
Cloud Testing
What is Cloud Testing?
Cloud Testing platforms allow testers and developers to run their applications on cloud-based emulators and simulators in virtual environments.
This eliminates the need to maintain emulator, simulators for all your testing/development teams.
You can easily test on all types of OS versions and device models.
Some cloud-based tools allow mobile teams to receive real-time alerts against any measurement criteria, allowing them to address issues before they impact end-users
There are often APIs that allow automated testing through these platforms.
69© Copyright 2013 Coveros, Inc.. All rights reserved.
Cloud Testing
Sauce Labs
Mobile Application Testing Tutorial What is Device Testing?
Copyright 2014 Coveros, Inc. All rights reserved. Page 43
What is Device Testing?
Crowdsourcing
70© Copyright 2013 Coveros, Inc.. All rights reserved.
Physical Device Testing
What is Device Testing?
Testing on a real, physical device with full functionality to the tester.
Physical device testing will always provide the most accurate testing results, but is also the most costly.
Without access to the real processor and hardware to understand the quirks that some real device have, testers may not discover some real life issues that do not present themselves in emulators
71© Copyright 2013 Coveros, Inc.. All rights reserved.
Crowdsourcing is the practice of obtaining testing services by soliciting contributions from a large group of people, and especially from an online community, rather than from traditional employees, outsourcing or suppliers.
It combines the efforts of numerous self-identified volunteers or part-time workers, where each tester of their own initiative adds a small portion to the greater testing efforts.
Crowdsourcing is different from outsourcing in that the work comes from an undefined public rather than being commissioned from a specific, named group.
Crowdsourcing
Mobile Application Testing Tutorial What are Remote Device Access (RDA) Service?
Copyright 2014 Coveros, Inc. All rights reserved. Page 44
What are Remote Device Access (RDA) Service?
Which Platform is Best for My Mobile Testing?
72© Copyright 2013 Coveros, Inc.. All rights reserved.
Remote Device Access
What are Remote Device Access (RDA) Service?
RDA services enable access to a live device over the Internet. As a Mobile Apps Tester, you should be aware of such services and should suggest your managers about the capabilities of such services.
Due to the large number of devices available in the market, it is not feasible to buy a new device every time. That makes RDA a much more cost effective solution.
73© Copyright 2013 Coveros, Inc.. All rights reserved.
Recommendations
Which Platform is Best for My Mobile Testing?
Type of Testing Browser Emulator Device RDA Cloud Emulator
Debugging x x x
Feature Testing x x x
Usability Testing x x x
Exploratory
Testing
x x x x x
Performance
Testing
x x x x x
Automated Testing x x x
UAT x x
Security Testing x x x x x
Pre-Prod Testing x x
Mobile Application Testing Tutorial Mobile Testing Techniques
Copyright 2014 Coveros, Inc. All rights reserved. Page 45
Mobile Testing Techniques
Agenda
74© Copyright 2013 Coveros, Inc.. All rights reserved.
Mobile Testing Techniques
75© Copyright 2013 Coveros, Inc.. All rights reserved.
Agenda
– Exploratory Testing
– Device Access
– Security Testing
– Performance Testing
– Compatibility Testing
– Submission Guideline Testing
Mobile Application Testing Tutorial Mobile Testing Techniques
Copyright 2014 Coveros, Inc. All rights reserved. Page 46
Mobile Testing Techniques
Exploratory Testing
76© Copyright 2013 Coveros, Inc.. All rights reserved.
Mobile Testing Techniques
Test Management
Exploratory Testing
Service/API Testing
Device Access
Security Testing
Back-End Testing
Mobile UI Automation
Performance Testing
Submission Guidelines
Testing
Compatibility Testing
77© Copyright 2013 Coveros, Inc.. All rights reserved.
Exploratory Testing
Exploratory Testing is still the most widely used mobile testing technique
Necessary for some validation
Frequent key in outsourcing and crowdsourcing scenarios
Advantages– Tester can assess performance of application under actual
operating conditions Determination/Evaluation of Performance
Determination/Evaluation of Ease of Use
Identification of unexpected behaviors and errors
Does it feel right?
– Test-As-You-Fly A Goddard Space Flight Center Technical Standard which essentially means
that you test the system under test as it is intended to be used.
Mobile Testing Techniques
Mobile Application Testing Tutorial Standard Mobile App States
Copyright 2014 Coveros, Inc. All rights reserved. Page 47
Standard Mobile App States
Interruption Testing
78© Copyright 2013 Coveros, Inc.. All rights reserved.
App Store– Upload– Download– Update Broadcasting– Monetization scenarios
Installation and Removal– Multiple Modes– Verification of App Logo and Splash
Operation– Start/Stop– Pause/resume– Credentials– Auto-Start
Operational Modes– Standalone– Offline– Wi-Fi– Cabled– Airplane Mode
Standard Mobile App States
Exploratory Testing
79© Copyright 2013 Coveros, Inc.. All rights reserved.
Incoming Call
SMS Received
Voicemail
Social media notifications
Alarms
Calendar reminders
Battery low
Battery Critical
Power Sleep Cycle
Battery Removed
Battery Charging
Low Memory
Device Switched Off
Device Reboot
Network outage and Recovery
Bluetooth Connection request
Phone/System Updates
Data Transfer Cable Insertion
Camera Activation
Home Button
Search Button
Volume Control Changes
Screen Lock
Interruption Testing
Exploratory Testing
Mobile Application Testing Tutorial User Interface Testing
Copyright 2014 Coveros, Inc. All rights reserved. Page 48
User Interface Testing
User Inputs
80© Copyright 2013 Coveros, Inc.. All rights reserved.
Layouts, alignment, colors, fonts
– Views/Tables
– buttons, Menus, Links, Pickers, Toolbars
– Alerts/Messages
– Themes, Color Schemes, Styles
– Brightness Settings
UI Elements– Position
– Size
– Inputs
– Actions
Images and Symbols– Clarity
– Alignment
– Density (load performance)
Error Message Displays
Consistency– Across the application
– Across various devices
User Interface Testing
Exploratory Testing
81© Copyright 2013 Coveros, Inc.. All rights reserved.
Gestures– Tap
– Press/Hold
– Double Tap
– Drag
– Pinch
– Spread
– Swipe/Flick
– Rotation
– Shake
– Tilt
User Inputs
Exploratory Testing
http://uxdesign.smashingmagazine.com/2011/10/06/not-your-parents-mobile-phone-ux-design-guidelines-smartphones/
Mobile Application Testing Tutorial Accessibility and Usability Testing
Copyright 2014 Coveros, Inc. All rights reserved. Page 49
Accessibility and Usability Testing
Slide 82
Slide 83
82© Copyright 2013 Coveros, Inc.. All rights reserved.
Usability– Efficiency
– Accuracy
– Recall
– Emotional Response
Accessibility– Screen Size
– Lighting Conditions and brightness settings
– Multi-screen capability and compatibility
– Touch Interface size
Accessibility and Usability Testing
Exploratory Testing
83© Copyright 2013 Coveros, Inc.. All rights reserved.
Workflows– Information Presentation
– Ease of Task Completion
– Collaborative Task Completion (Sequential and Parallel)
Application Integration and Interactions– Native Application/Device
– Web Application/Device
– Hybrid Application/Device/Web App
Accessibility and Usability Testing
Exploratory Testing
Mobile Application Testing Tutorial Mobility
Copyright 2014 Coveros, Inc. All rights reserved. Page 50
Mobility
Connectivity
84© Copyright 2013 Coveros, Inc.. All rights reserved.
Geo aware (GPS, Tower Triangulation)
Accelerometer (Motion, Gestures)
Magnetometer (Compass)
Traversing zones/cells/networks– GPS Signal fade/interference
– WiFi Signal fade/interference
– Cellular Signal fade/interference
Mobility
Exploratory Testing
85© Copyright 2013 Coveros, Inc.. All rights reserved.
Near Field Communication– Android Beam
– Bluetooth
– Wi-Fi
Networks– Variable Networks Data throughput support
– Variable network connectivity protocol types 4g/3g/analogue
1G, 2G, 3G, 4G
Wi-Fi, Wi-Fi/3G, Wi-Fi/4G)
– Signal Strength No Network
Intermittent connectivity
Strong Signal
Varying signal strength
– Trust level settings (Security/Data Exposure)
Connectivity
Exploratory Testing
Mobile Application Testing Tutorial Usability: Deadly Application Failures
Copyright 2014 Coveros, Inc. All rights reserved. Page 51
Usability: Deadly Application Failures
Testing Device Sensors
86© Copyright 2013 Coveros, Inc.. All rights reserved.
Exploratory Testing
Usability: Deadly Application Failures
• Insufficient features and functionality
• Consumption of resources (Battery, CPU, Storage)
• Failure to operate well under sub-optimal connectivity situations
• Slow Performance
• Interoperability• doesn’t integrate with native device features
• causes errors in other existing applications
• Difficult to use
87© Copyright 2013 Coveros, Inc.. All rights reserved.
Rotation/Tilt– Not supported (user expectations problem)
– Could cause screen res/layout issues (switching from Portrait to Landscape mode)
– Other unexpected results
Movement/Shake Inputs– Not supported (user expectations problem)
– Unexpected results Does jogging on a treadmill cause song over-shuffling on iPod?
Geo-Location Capabilities– Verify location mapping correct
– Verify location capabilities while in motion
– Verify lack of freeze-ups while in motion
– Verify re-location upon loss/re-gain of signal
Magnetometer/Rotation– Verify Compass direction correct
Light– Automatic brightness adjustments
Heat– Device function shutdown, reduced performance
Testing Device Sensors
Device Access Testing
Mobile Application Testing Tutorial Security Testing
Copyright 2014 Coveros, Inc. All rights reserved. Page 52
Security Testing
Performance Testing
88© Copyright 2013 Coveros, Inc.. All rights reserved.
Security Testing
Network Attacks– Traffic Analyzers
Server Side Attacks– API/Service Attacks
– Back-end Server
– Database Attacks
Client-Side Attacks– Cross Site Scripting
– Data Privacy Exploitation
– Buffer Overflows
Injection and Data Flow Analysis
Jail-breaking/Rooting
Penetration Testing
Mobile Testing Techniques
89© Copyright 2013 Coveros, Inc.. All rights reserved.
Performance Testing
Mobile apps are often supposed to scale to massive volumes. We have to delve into lots of different realistic scenarios for performance over more than just WI-FI.
Performance testing should cover real world mobile networks (Not all networks are created equal).
– 4G LTE, 3G, 2G, EDGE, GPRS
Don’t forget to test middle-tier and back-end for performance
If you’re not testing your mobile app out in the real world like a client would experience, your performance testing is not complete
Mobile Testing Techniques
Mobile Application Testing Tutorial Performance and Stress
Copyright 2014 Coveros, Inc. All rights reserved. Page 53
Performance and Stress
Compatibility Testing
90© Copyright 2013 Coveros, Inc.. All rights reserved.
Low resources– Optimization Modes
– Power saving Modes
– Variable Battery Levels (Full, Low, Critical, Charging)
– Memory/CPU utilization Launch, Background,
Foreground, Exit
Simultaneous Applications utilization
Continuous Keypad entry (across various keypad controls/events)
Memory Sweeping
– Storage Space Installation
Application Utilization
– Battery Consumption
Server Load– Stress testing
Multi-user
Multi-functional
– Backups
Crowd-Source Testing
Network Testing – Internetwork
– Network Roaming
– Wife Cell
Performance and Stress
Mobile Testing Techniques
91© Copyright 2013 Coveros, Inc.. All rights reserved.
Testing for Compatibility:-Testing the compatibility of your application with native device features (i.e. To make sure your application is not hampering native device functionality)
Platform
Device
Other Applications
Application backwards compatibility
Browsers
Networks
Compatibility Testing
Mobile Testing Techniques
Mobile Application Testing Tutorial Submission Guideline Testing
Copyright 2014 Coveros, Inc. All rights reserved. Page 54
Submission Guideline Testing
Slide 92
Just an excerpt from the Apple Guidelines:
2. Functionality 1. Apps that crash will be rejected
2. Apps that exhibit bugs will be rejected
3. Apps that do not perform as advertised by the developer will be rejected
4. Apps that include undocumented or hidden features inconsistent with the description of the app
5. will be rejected
6. Apps that use non-public APIs will be rejected
7. Apps that read or write data outside its designated container area will be rejected
8. Apps that download code in any way or form will be rejected
9. Apps that install or launch other executable code will be rejected
10. Apps that are "beta", "demo", "trial", or "test" versions will be rejected
11. iPhone apps must also run on iPad without modification, at iPhone resolution, and at 2X iPhone
12. 3GS resolution
13. Apps that duplicate apps already in the App Store may be rejected, particularly if there are many
14. of them
15. Apps that are not very useful or do not provide any lasting entertainment value may be rejected
16. Apps that are primarily marketing materials or advertisements will be rejected
92© Copyright 2013 Coveros, Inc.. All rights reserved.
Submission Guideline Testing
Google Play Store– Quick review
– Simple standards to meet
– http://play.google.com/about/developer-content-policy.html
Apple App Store– Notoriously difficult standards
– Long review time
– Stringent Requirements
– https://developer.apple.com/appstore/guidelines.html
Mobile Testing Techniques
Mobile Application Testing Tutorial Submission Guideline Testing
Copyright 2014 Coveros, Inc. All rights reserved. Page 55
17. Apps that are intended to provide trick or fake functionality that are not clearly marked as such
18. will be rejected
19. Apps larger than 20MB in size will not download over cellular networks (this is automatically
20. prohibited by the App Store)
21. Multitasking apps may only use background services for their intended purposes: VoIP, audio
22. playback, location, task completion, local notifications, etc.
23. Apps that browse the web must use the iOS WebKit framework and WebKit JavaScript
24. Apps that encourage excessive consumption of alcohol or illegal substances, or encourage
25. minors to consume alcohol or smoke cigarettes, will be rejected
26. Apps that provide incorrect diagnostic or other inaccurate device data will be rejected
27. Developers "spamming" the App Store with many versions of similar apps will be removed from
28. the iOS Developer Program Title Month Year
Mobile Application Testing Tutorial Android Core Quality Requirements Categories
Copyright 2014 Coveros, Inc. All rights reserved. Page 56
Android Core Quality Requirements Categories
Apple App Store Requirements Categories
93© Copyright 2013 Coveros, Inc.. All rights reserved.
Functionality– Permissions– Installation– Audio– UI and Graphics– User / App State
Performance and Stability– Stability– Performance– Media– Visual Quality
Google Play– Policies– App details page– User Support
Android Core Quality Requirements Categories
Submission Guideline Testing
94© Copyright 2013 Coveros, Inc.. All rights reserved.
Functionality
Metadata
Locations
Push Notifications
Game Center
iAds
Trademarks and trade dress
Media Content
User Interface
Purchasing and Currencies
Scraping and Aggregation
Device Damage
Personal Attacks
Violence
Objectionable Content
Privacy
Pornography
Religion, Culture, and ethnicity
Contests, sweepstakes, lotteries, and raffles
Charities and Contributions
Legal Requirements
Apple App Store Requirements Categories
Submission Guideline Testing
Mobile Application Testing Tutorial Tools
Copyright 2014 Coveros, Inc. All rights reserved. Page 57
Tools
Agenda
95© Copyright 2013 Coveros, Inc.. All rights reserved.
Tools
96© Copyright 2013 Coveros, Inc.. All rights reserved.
Agenda
Introduction
Multi-platform Testing Tools
Android Testing Tools
IOS Testing Tools
Blackberry testing tools
Windows Mobile Testing Tools
Remote Device Access Services
Outsource/Crowd Source Testing Services
Mobile Application Testing Tutorial Multi-platform Test Tools
Copyright 2014 Coveros, Inc. All rights reserved. Page 58
Multi-platform Test Tools
Multi-Platform Test Tools
97© Copyright 2013 Coveros, Inc.. All rights reserved.
Selenium 2 (WebDriver)
Can run on emulators and devices
Native support exists for Android, iPhone and iPad but is deprecated
Third party drivers have been developed to fill in these gaps– iOS-Driver
– Selendroid
– Appium
Support for all major browsers– Browsers downloaded and installed on devices to supplement
native browsers can be fully tested with WebDriver.
Multi-platform Test Tools
Testing Tools
98© Copyright 2013 Coveros, Inc.. All rights reserved.
Gomez, mobiReady, and W3C mobileOK Checker
Tests web based apps and evaluates their mobile performance.
Examines mobile best-practices and usability, and provides steps to correct
Web-based testing, no download or install needed
Multi-Platform Test Tools
Testing Tools
Mobile Application Testing Tutorial Android Tools
Copyright 2014 Coveros, Inc. All rights reserved. Page 59
Android Tools
Android Debugging Tools
99© Copyright 2013 Coveros, Inc.. All rights reserved.
Monkey Runner – Runs on simulator or device
– Generates pseudo-random streams of user events such as clicks, touches, or gestures, as well as a number of system-level events.
– Can run an automated start-to-finish test of an Android application. You provide input values with keystrokes or touch events, and view the results as screenshots.
Robotium– Support for native and hybrid applications
– Can write function, system and acceptance test scenarios, spanning multiple Android activities
– Integrates smoothly with Maven or Ant to run tests as part of continuous integration.
Android Tools
Testing Tools
100© Copyright 2013 Coveros, Inc.. All rights reserved.
Dalvik Debug Monitor Server (DDMS)– Port-forwarding services
– Screen capture on the device
– Thread and heap information on the device
Logcat – Process, and radio state information
– Incoming call and SMS spoofing
– Location data spoofing and more
TaskKiller– A utility application used to kill process
– Useful to skip device reboot.
Android Debugging Tools
Testing Tools
Mobile Application Testing Tutorial iOS Tools
Copyright 2014 Coveros, Inc. All rights reserved. Page 60
iOS Tools
Blackberry Tools
101© Copyright 2013 Coveros, Inc.. All rights reserved.
UIAutomation– Lets you run an automated set of tests against an application– Can be used to perform automated tests on real devices and on the
iPhone Simulator– It enables you to quickly track regressions and performance issues
NUnit– Unit level testing for iOS apps
Test Studio for iOS– Lets developers test native iOS apps as well as hybrid and Web apps– The tool can test common iOS controls and record test scripts for
gestures such as taps, multiple finger gestures such as zooms, swipes and taps.
iPhone test automation– Fonemonkey– UISpec– Bromine (not recommended)
iOS Tools
Testing Tools
102© Copyright 2013 Coveros, Inc.. All rights reserved.
Testing Tools
BlackBerry device simulator – “Fledge”
Jamo Solutions – “M-eux Test”
Borland Silk Mobile
Blackberry Tools
Max1
Mobile Application Testing Tutorial Windows Tools
Copyright 2014 Coveros, Inc. All rights reserved. Page 61
Windows Tools
Remote Device Access Services
103© Copyright 2013 Coveros, Inc.. All rights reserved.
MSDN Hopper Test Tool– Simulates input stress on Windows Mobile devices
– Hopper will stress all applications that are available through the menu system by rapidly sending keystrokes and screen taps in a random fashion
MSDN Device Emulator– Mimics the behavior of Windows Mobile
– Provides a virtual hardware platform that you can use to test applications on multiple virtual devices
– Runs code compiled for ARM microprocessors
MSDN Cellular Emulator
Windows Tools
Testing Tools
104© Copyright 2013 Coveros, Inc.. All rights reserved.
DeviceAnywhere
Perfecto Mobile
Keynote
Paca Mobile Center
pCloudy
Remote Device Access Services
Testing Tools
Mobile Application Testing Tutorial Security Testing Tools
Copyright 2014 Coveros, Inc. All rights reserved. Page 62
Security Testing Tools
Performance Testing Tools
105© Copyright 2013 Coveros, Inc.. All rights reserved.
Kryptowire (Native App Testing Tool)
OWASP ZAP (Mobile App Testing Tool)
For more information and training on Security Testing Tools, check out the Security Testing Course from SQE.
Additional resources at owasp.org
Security Testing Tools
Testing Tools
106© Copyright 2013 Coveros, Inc.. All rights reserved.
NeoLoad
BlazeMeter
SOASTA
Mobitest
Performance Testing Tools
Testing Tools
Mobile Application Testing Tutorial Crowd Source Testing Services
Copyright 2014 Coveros, Inc. All rights reserved. Page 63
Crowd Source Testing Services
Outsourced Mobile App Testing Services
107© Copyright 2013 Coveros, Inc.. All rights reserved.
GetJar
Mob4Hire
uTest
Ubertesters
App Lover
Apphance
Beta Bait
Bugcrowd
Crowd Source Testing Services
Testing Tools
108© Copyright 2013 Coveros, Inc.. All rights reserved.
Coveros
Mobica
Intertek
RapidSoft
Belatrix
Ranorex
HP
TestQuest
Utopia Solutions
Wipro
Outsourced Mobile App Testing Services
Testing Tools
Mobile Application Testing Tutorial Wrap-Up
Copyright 2014 Coveros, Inc. All rights reserved. Page 64
Wrap-Up
Agenda
109© Copyright 2013 Coveros, Inc.. All rights reserved.
Wrap-Up
110© Copyright 2013 Coveros, Inc.. All rights reserved.
Agenda
Wrap-Up
Recommendations
Questions
Mobile Application Testing Tutorial Mobile Testing Books we recommend
Copyright 2014 Coveros, Inc. All rights reserved. Page 65
Mobile Testing Books we recommend
Questions?
111© Copyright 2013 Coveros, Inc.. All rights reserved.
Mobile Testing Books we recommend
“Tap Into Mobile Application Testing” Kohl, Jonathan
“A Practical Guide to Testing Wireless Smartphone Applications” Harty, Julian
“Mobile Application Security” Dwivedi, Himanshu
“Mobile Developer’s Guide To The Galaxy”
http://www.enough.de/fileadmin/uploads/dev_guide_pdfs/Mobile_DevGuide_13.pdf
112© Copyright 2013 Coveros, Inc.. All rights reserved.
Questions?
Contact Information:
Mobile Application Testing Tutorial Notes
Copyright 2014 Coveros, Inc. All rights reserved.
Notes