testing mobile applications using perfecto mobile mobilecloud v2
DESCRIPTION
fgdfhdfhfhdTRANSCRIPT
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 2 of 49
The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as
of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be
a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the
date of publication.
This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY,
AS TO THE INFORMATION IN THIS DOCUMENT.
Microsoft grants you a license to this document under the terms of the Creative Commons Attribution 3.0 License. All other
rights are reserved.
2013 Microsoft Corporation.
Microsoft, Active Directory, Excel, Internet Explorer, SQL Server, Visual Studio, and Windows are trademarks of the Microsoft
group of companies.
All other trademarks are property of their respective owners.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 3 of 49
Table of Contents Introduction ...................................................................................................................................................................................................................................... 4
Overview ....................................................................................................................................................................................................................................... 4
Visual Studio ALM Rangers ................................................................................................................................................................................................... 4
Contributors ................................................................................................................................................................................................................................ 4
Personas ........................................................................................................................................................................................................................................ 4
Related Resources..................................................................................................................................................................................................................... 4
Exercises ........................................................................................................................................................................................................................................ 5
Device testing approach for IT Applications ....................................................................................................................................................................... 6
Need for Device Testing ......................................................................................................................................................................................................... 6
Challenges in Device Testing ..................................................................................................................................................................................................... 7
Device Testing Lifecycle for IT Applications (DTL-IT) ................................................................................................................................................. 7
Envision Phase ............................................................................................................................................................................................................................ 7
Plan Phase .................................................................................................................................................................................................................................... 8
Build Phase................................................................................................................................................................................................................................ 11
Stabilize Phase ......................................................................................................................................................................................................................... 14
Native App Best Practices ................................................................................................................................................................................................... 14
Environmental preparation ...................................................................................................................................................................................................... 16
Installing the Perfecto Mobile Team Foundation Server (TFS) add-in ............................................................................................................. 16
Accessing the Perfecto Mobile Cloud Platform ........................................................................................................................................................ 17
Hands-On Lab ............................................................................................................................................................................................................................... 20
Exercise 1 Accessing the Perfecto Mobile Cloud and running an automated test (10 min) .............................................................. 20
Exercise 2 Run an automated test in the Perfecto Mobile Cloud within Visual Studio (15 min) ...................................................... 26
Exercise 3 Run an automated test in the Perfecto Mobile Cloud within Microsoft Test Manager (10 min) ................................ 31
Exercise 4 Continuous Integration - Run an automated test in the Perfecto Mobile Cloud within Team Build on TFS (10
min) .............................................................................................................................................................................................................................................. 38
Exercise 5 Continuous Integration - Run an automated test in the Perfecto Mobile Cloud within Team Build on Team
Foundation Service in the cloud (10 min) .................................................................................................................................................................... 41
Conclusion ...................................................................................................................................................................................................................................... 45
Appendix ......................................................................................................................................................................................................................................... 46
Automation Tool Comparison .......................................................................................................................................................................................... 46
FAQ .............................................................................................................................................................................................................................................. 48
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 4 of 49
Introduction
Overview How do you develop and test apps on iOS, Android, and Windows Phone devices? You leverage the Visual
Studio Industry Partner (VSIP) solutions and our Team Foundation Server (TFS) integrations. For example, for IOS
and Android development, use solutions from ITRMobility and Xamarin to develop within the Visual Studio IDE
using .NET. You can use Xcode, Eclipse, or IntelliJ and integrate with Team Foundation. Then you test your
mobile application using Perfectos Mobile Cloud solution.
Mobile applications can require complex testing on specific devices for which the acquisition and maintenance
can be expensive. Especially for devices based on Android as there are so many variations. Emulators are useful
but can be expensive and inaccurate. The best solution is to test your mobile applications on the actual device
using Visual Studio Application Lifecycle Management (VS ALM) and TFS tooling enabling an end-to-end
Application Lifecycle Management (ALM) experience. Deploying your application on the actual device, testing
from anywhere globally, and integrating with Visual Studio and TFS provides incredible business value.
The following guidance will describe how to install and setup Visual Studio and TFS to use the Perfecto Mobile
MobileCloud for TFS solution. The document also describes how to use the Perfecto Mobile solution within the
Visual Studio 2012 RTM ALM Virtual Machine or your own VM or local desktop. In addition, we will discuss the
architecture, the mobile cloud, testing of real devices, ScriptOnce, the integration with Visual Studio and
Microsoft Test Manager, and the end-to-end scenario.
Visual Studio ALM Rangers The Visual Studio ALM Rangers are a special group with members from the Visual Studio Product group,
Microsoft Services, Microsoft Most Valuable Professionals (MVP) and Visual Studio Community Leads. Their
mission is to provide out-of-band solutions to missing features and guidance. A growing Rangers Index is
available online1.
Contributors Anil Chandra Lingam, Anisha Pindoria, Brian Blackman, Dan Marzolini, Esteban Garcia, Hassan Fadili, Mathew Aniyan, Oliver
Hilgers, Bipin Puthiyaveetil, and Ranjit Gupta.
Refer to Visual Studio ALM Rangers Personas and Scenarios 2 for more information about the personas and
customers profiles referenced in this guide.
Related Resources Understanding the ALM Rangers http://aka.ms/vsarunderstand
Visual Studio ALM Rangers Home Page http://aka.ms/vsarmsdn
Visual Studio ALM Ranger Solutions http://aka.ms/vsarsolutions
Visual Studio ALM Ranger Solution vsarDeviceTest Project on Codeplex - http://aka.ms/vsardevicetest
1 http://aka.ms/vsarindex
2 http://go.microsoft.com/fwlink/?LinkID=230942
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 5 of 49
Exercises The Hands-on lab sections will show you how to use the Perfecto Mobile platform and Visual Studio to test
different mobile devices through automated scripts as a manual or general tester. The lab will take approximately
55 minutes to complete. The hands-on lab is structured for you to learn how to:
1. Run an automated test script on the perfecto mobile cloud and review the test results.
2. Create your own Visual Studio Perfecto Mobile Cloud project and run tests against real mobile devices in the
cloud.
3. Run an automated test script against real mobile devices in the cloud within Microsoft Test Manager 2012.
4. Run an automated test script within a Continuous Integration process on TFS Team Build.
5. Run an automated test script within a Continuous Integration process on Team Foundation Service Build in
the cloud.
This guide targets the Microsoft 200-300 level users of Visual Studio 2012. The target group is considered as
intermediate to advanced users of Visual Studio 2012 and has in-depth understanding of the product features in
a real-world environment. Parts of this guide may be useful to the Visual Studio novices and experts but users at
these skill levels are not the focus of this content.
WA
RN
ING
The mobile device environment might change.
Please read the next paragraph carefully.
The exercises run on real devices in the Perfecto Mobile Cloud. As real mobile devices are subjected to software
updates and other environmental changes over time, it is possible that tests could on occasion fail. Therefore, if a
test execution in the exercises fails the root cause can be different. Here are some examples:
Possible error scenario Possible root cause Solution
The URL
https://demo.perfectomo
bile.com is not available
anymore.
The demo environment is not
available anymore and the
production environment can be
used in a trial mode.
Use another URL in the exercises and Visual Studio
Code:
Use the production environment URL:
https://www.perfectomobile.com.
The test fails with an
error due to a device
error.
The referenced mobile device in
the exercise
- doesnt exist anymore.
- is disconnected.
- has a new device ID.
- is locked or blocked by
another user.
Validate the mobile device status:
Please check the portal https://demo.perfectomobile.com
in the area Interactive / Select Device and validate the
device status.
If the needed device has an issue use another available
device. For the exercise the device ID (DUT), device name
and OS name (os) are needed and must be changed. Then
retry the exercise / test execution.
The test fails with an
error due to an app error.
The ATM app is not installed on
the selected device.
Validate the app status on the mobile device:
If the ATM app doesnt exist on the mobile device please
contact the Perfecto Mobile support.
The test fails with an
error due to an OS
update pop up on the
device start screen.
A pop up message on the device
shows that an OS update is
available. So, the tests script
cannot select the ATM app icon
anymore.
Validate the mobile device screen:
Close the pop up message without executing the OS
device update and retry the exercise / test execution.
Table 1 Possible test execution error scenario.
For additional help please contact the Perfecto Mobile support: http://www.perfectomobile.com.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 6 of 49
Device testing approach for IT
Applications
Need for Device Testing The emergence of smartphones and tablets has changed the dynamics of internet usage. As per the latest market
research, over 40% of the users are accessing internet applications using devices other than PC. Success of the web
applications is now heavily dependent on the quality of service (QoS) on both PC and devices. We are now
observing similar patterns with regard to enterprise applications. The culture of BYOD (bring your own device) is
being welcomed by several major enterprises worldwide. This has led to interesting challenges and opportunities,
one of them is huge demand for Device testing, for ensuring consistent experience across devices.
Given the dynamics of the IT applications usage, testing approach must also undergo surgical changes. In this
paper, we highlight the changes needed to testing approach and present the approach for device testing.
Figure 1 - PC verses Devices Spread across verticals
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 7 of 49
Challenges in Device Testing There are now over 2000+ types of devices in the market and it is not feasible from both cost and time perspective
to test IT applications on every device type. This presents a significant challenge of identifying a minimum set of
devices which would represent the majority of the devices. Also, depending on the application usage, locale and
end-user persona, the minimum set for testing of devices can vary. For example, if a company in Australia wants
to adopt BYOD culture, then minimum set of devices on which an enterprise IT application needs to be device
tested would depend on the devices widely used by the employees in Australia. Hence, identifying the minimum
set of devices is key for successful device testing.
There will be a single testing strategy for PC, mobile and tablets, which optimizes the testing cycle from cost,
quality and time perspective. There is another school of thought that considers strategy for traditional testing on
PC and testing on devices should be separate. While there are some advantages of such approach, we have
observed that, a single testing strategy for both traditional testing on PC and testing on multiple devices is cost
efficacious.
Specially, we have observed following advantages:
Reducing the duplication of test cases
Reuse of test data
Effective sharing of test resources and test artefacts
Device Testing Lifecycle for IT Applications (DTL-IT)
Figure 2 - Device testing life cycle for it applications
Test Spread: Define the % effort in emulators, manual and cloud-based testing.
Envision Phase
How to select the minimum set of devices to be device tested?
Selecting the minimum set of devices that needs to be device tested is the most challenging aspect of device
testing. This would depend on factors such as platform to be supported, business usage of the application and
market data consideration for device market. For a typical web application that will be consumed on multiple
devices would have the following device priority matrix:
Envison
Obtain Customer requirements specific to usage of the application on different devices
Obtain customer requirements on the device spread
Obtain NFR
Plan
Define scope
Conduct Automation Assesement/ROI
Define test spread based on schedule
Create Device Testing Strategy
Define Test Sceanrios
Build
Create device automation framework
Create Test Cases
Stabilize
Execution of test cases on emulators/real devices based on the test spread
Run the Tests
Publish the Test Report
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 8 of 49
Priority
High Priority Platform (OS)
Browser (IE, Firefox, Chrome, Safari, etc.)
Form Factor (7'', 10'', 4'', etc.)
Medium Priority Input Mechanism (Stylus, Swipe, Camera, GPS, etc.)
Network Connection (GSM,CDMA,BES* -BB, Edge,2G,3G,4G,Wi-FI)
Low Priority Memory
CPU Type (ARM, Intel, etc.)
Table 2 Factors to be considered for device test mix
Depending on the usage of your application, define the device priority matrix and arrive at a set of devices on
which testing needs to be performed. Make sure that, you are choosing only those devices that are critical to your
business. This activity needs to be done during the planning phase. Also, given the dynamic nature of device
market, if time period between planning phase and SIT phase is significant, then re-evaluate the device priority
matrix on every iteration, so that you are including the new devices that have just arrived to the market in your
test mix.
Another strategy for testing the application on multiple devices is to do Beta Release of the application and do
dog-fooding.
Plan Phase
Figure 3 - Testing types across various layers for optimizing the efforts
Following are the key activities that needs to be performed during Plan phase:
Define User Scenarios for the IT application
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 9 of 49
Gather the cross platform / browser mix requirements of the scenarios and personas accessing those
scenarios which will mimic the real life usage profile mix of the application.
o Ex: 20 % of the total scenarios are accessed through mobile, 10 % through tablets and rest 70%
over desktop/laptop
o Mobile Spread is further split across Safari on IPhone 5 (60%) ,Android 4.0 /Chrome (35%),
Widows 8/ IE10 (15%)
o Desktop spread across Windows 8/Chrome, Windows 8 /Safari, Windows 8 /IE 10
Identify the scenarios that are critical to run on real devices like must /should /nice to run which will help,
propose the right mix to the customer.
Identify the device/browser spread across the customer user base if the customer does not have this
information, get the top N devices that are widely used in the region to help the customer to make the
right choice
Plan for any upcoming device/browser launches and make provision to accommodate these mix to
support agility
Scope the devices/browser mix for the project based on the schedule /availability of devices and budget
of the project.
Scope the test spread mix of devices /emulators/browsers ideal for the engagement based on the above
requirements
Calculate the ROI based on coverage (device spread) versus risk of not testing across all spread
Prepare the test strategy based on the above
Device support strategy
1) How many of the platform and devices should we support?
2) Business Considerations
3) Market data considerations
4)Failing gracefully when a device is not supported
Device Configuration Matrix
Device Model
Input mechanism (Eg:Touch,Key)
Screen Resolution /Orientation
Chipsets/Peripherals (GPS,Camera,Microphone)
Carrier
Locked /Un Locked
Upgrade/Install type (ROM.,OTA, App store, Tethered-kiosk scenario)
Network Type (GSM,CDMA,BES* -BB)
Network Connection (Edge,2G,3G,4G,Wi-FI)
Operating System (Windows/Android/Blackberry)
Browser
Recommendations
1) First Test on the major Desktop browsers to ensure that the system is functionally stable (VS 2012 Coded
UI Selenium support will help for Automation)
2) Run the tests on major platform/devices Emulators (Automation/ Manual based on ROI)
3) If the devices are available then test on real device. Set of Reference devices in-house shared by the team.
Test at least once all the scenarios in the reference devices manually
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 10 of 49
4) Breadth device testing done by external party where the efforts are more .Needs to identify partners with
this capability.
5) Cloud based mobile testing tools can be used in case the device spread is huge and one wants to
automate once and run the tests across multiple devices over the cloud on a real device. This eliminates the
maintenance overhead of managing the devices internally and optimizes costs by eliminating manual
execution of tests.
Real /Virtual Mobile Testing Decision Table
Table 3 Mobile testing decision table
Based on the combinations identified one can make use of combinatorial algorithms to decide the optimum
combinations of tests to be designed to ensure greater coverage and eliminate risks.
Device Testing Cost Pyramid
Figure 4 - Device testing Cost Pyramid
Emulators/Browser plug in based testing will be the most cost effective solution. The above pyramid helps make
the decision based on the cost factor. This also depends on the number of devices in our test spread and the
Virtual Mobile Testing Tool
Regression Test ,BVT
Sanity
Compatibility
Large set of Functional Testing scope
Wide Range of devices available for testing .Script once run on any device
Devices
UAT
Compatibility
Inter operability
Interrupt Test
Performance
End to end scenario
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 11 of 49
total number of tests to be executed. In case both variables are low then real device testing will work out to be
more economical compared to cloud based testing.
Test Automation Tools for Device testing
Key requirements for the Device Test Automation Tool:
Should be able to port the automated scripts of one platform to multiple platforms and multiple devices
.Script once run anywhere (Device anywhere/PerfectoMobile)
Speed up test execution
Eliminate manual efforts to reduce total cost
Easy to maintain
Should be extensible to support future platforms and devices with ease.
Should be able to run the tests on emulators as well as devices on demand (Nice to Have on Devices as
Running tests on real device takes lots of time and can be done only in serial fashion)
Resilient and smooth execution of automated tests: This is very important as no one would like to have
an automation that is not stable and needs frequent monitoring for failures. Automation framework
should be able to give consistent and predictable results for every run.
Automation guidelines
Automation on Virtual Device is recommended for the following areas
1) Any scenario that can be run on an emulator can be automated and can be tested without the real
device .Identify the regression suite for the functional test cases and we can automate those scenarios
for all the gestures to ensure that the major risks and coverage issues are addressed as part of early
iteration testing and regression tests are automated as part of iteration scope .Majority of the emulators
comes free.
2) Automation on Real device is recommended in early stages of the application life cycle to detect and fix
the bugs early.
3) Anything that can be run on a emulator for all the top 4 platform/devices
4) Performance and Stress testing on actual devices
5) Some Configuration testing on Emulators
6) Standard out of the box compliance check/design guidelines check
Automation is Not Recommended for the following areas
1) Configuration testing
2) UI look and feel verification testing
3) E2E Testing
4) Native App automation (Assuming not many complex functionality is delivered, Automation wont be
recommended)
5) Scenarios which are already addressed through automation for desktop access
6) Usability testing
Build Phase Following are the key activities that needs to be performed during Build Phase:
Identify the test scenarios based on functional and non-functional requirements
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 12 of 49
Make sure that, you are identifying User Scenarios that include gestures such as
Touch
Touch Zones
Swipe
Pinch Zoom
Identify the priority scenarios which can be automated based on the return on investment (ROI)
Useful tools for developers to catch bugs early
Tool Name Description Link
Device Test Planning
Helps plan the number of
devices and platforms to be
used in testing.
Planning tool is here
W3C mobileOK Checker
The W3C mobileOK Checker is
one of the best tools available
to you, to help you test the
efficacy of your Website on
mobile devices. This tool
performs several tests on a
Web page before it estimates
the level of compatibility of
your Website with the mobile
Web. W3C has developed a
mobileOK Basic Tests 1.0
specification, which works to
give you a clear idea of the
mobile-friendliness of your
Website.
W3c Mobile OK
Iphoney
A very accurate iPhone tester,
this is also available to you
completely free of cost.
Though iPhoney is not really a
simulator, it enables you to
build 320x480px Websites,
compatible with the iPhone
screen specifications. This
implies that you can test both
your code and your Website
images in the actual Apple
Safari type of environment,
including all the features of the
original iPhone, such as zoom,
plugins, landscape and portrait
modes and so on.
Iphoney
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 13 of 49
Gomez The Gomez mobile readiness
test analyses your Website on
the basis of over 30 well-
established, fool proof, mobile
Web development techniques.
It then rates your page on a
scale ranging between 1 and 5
points. This tool not only offers
you more-or-less accurate
results, but also offers advice
on how you can further
improve your site in order to
make it even more compatible
with mobile browsers. Make
note of the fact that this tool
first requires you to enter some
personal information, before
you can go ahead and use it.
Gomez
Table 4 Device testing tools
Device Usability Testing
Different elements of the form factor influence the way the web application runs on the device. For example, the
following factors are crucial when it comes to testing web application on modern devices:
a. Screen Orientation
The screen orientation Landscape / Portrait dictates how the application is shown on the browser.
Scenarios must be identified which test the application in different orientations and look out for issues
resulting from page overflowing from the screen, elements not being visible to the user etc. Also, if the
application uses change in orientation as a trigger, that also needs to be exercised.
b. Responsive design
Modern web applications are adopting Responsive Design practices to retain homogenous look and feel
across devices. The main objective of responsive design is to ensure that the same web application when
opened across different devices of multiple form factors, provides similar user experience on all the
devices, regardless of the screen size and resolution. This is accomplished by a combination of CSS and
JavaScript which render the width and layout of elements in such a fashion that all elements are visible
on the screen.
c. Control Replacements
There may be some controls on the web page which may be taking mouse or keyboard exclusive
controls, for example mouse over events, mouse wheel events, pressing escape button etc. When the
web application is run on devices, there should be alternative ways of triggering controls, for example - a
drop down triggered by mouse over could be displayed as a select triggered by touch.
All such controls need to be identified and be tested on devices.
Combinatorial Testing to generate effective number of scenarios
While testing for devices, we could potentially encounter large number of test combinations arising from
multiple device form factors, screen resolutions, screen sizes, OS and browser types etc. It may not be practical to
run all tests on all possible combinations. Hence, it is recommended to come up with a condensed subset of
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 14 of 49
scenarios which provides maximum test coverage using Combinatorial Testing. Combinatorial test generation
technique is based on the premises that defects occur in a system due to interaction between few numbers of
components (empirically less than 6). If we find all possible combination of components taken 6 at a time, then
we can uncover cover more than 90% of total number of bugs.
There are various tools available which allow users to generate combinatorial scenarios; PICT, ACTS to name a
few.
Security Testing on Devices
The following aspects of security needs to be tested on devices:
1. Authentication:
Apart from the standard authentication methods the web application exposes, there could be device
specific authentication methods like swipe gestures, Google account, Facebook account, device tokens,
device certificates etc.
2. Authorization:
A few aspects of authorization that could be sensitive from devices perspective could be elevation of
privileges, escaping License checks, insecure HTTP methods that could be tampered over network etc.
Depending on the OS, there could be issues with file permissions on the local system
3. Data Storage
Web applications may use local storage for cache management, temporary downloads etc. Appropriate
security testing needs to be done to ensure that this data does not contain any sensitive information and
cannot be accessed by unauthorized parties.
4. Generic Web application security
Security test cases should be executed on the devices which exercise the application vulnerabilities which
exploit the client side, for example:
XSS and HTML Injection
Command Injection (if the application utilizes a shell)
CSRF
SQL Injection
Cookies
HTML5
Stabilize Phase Following are the key activities that needs to be performed during the Stabilize Phase:
Execution of test cases on emulators and real devices as per the Test Spread
Run the Tests
Publish the Test Report
Native App Best Practices Native app testing and automation is no different than the webapp ,but we may have to script for each platform
separately .We have some best practices for native application to ensure that native application that one build
shines in the market place.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 15 of 49
1. Application start up Performance App should be responsive Indicate this using some progress bars
2. Application Resumption- Calls, Messages, Back button Click- Recovery handling needs to be good for
example in the middle of browsing if the user gets the call ,User should be shown the same content that
he was browsing before the call.
3. Consistent User Experience Back button- First page back button should exit the app, for subsequent
pages, it should show previous page. Dismissing the dialog should be supported by back button
4. Battery Drain Testing- Overall app should be benchmarked against some existing app in similar category
for Battery drain. E.g.: Social networking app should be benchmarked against a Social networking app.
Subsequent improvement can be done using battery drain tests.
5. Network handling Scenarios: 2G, 3G, Wi-Fi, Wi-Fi with proxies etc.
For example: App should not hang when the network coverage is lost and should provide the right
exception message back to the user.
6. Error Handling should be very meaningful. Not many mechanisms to troubleshoot issues. So, error
messages should be very meaningful.
7. For apps where huge content needs to be added, recommend fetching it after installation. This will help
the build to be smaller in size and help more market penetration as users wont download something
which is bit heavy.
8. Application should not intervene into the platform features functioning. It should not override the
platform capabilities and features. E.g. Location services setting of the app should not override the
location services setting in the device
9. Application should not compromise on the security of data and other security features like XSS scripting
10. Landscape and Portrait mode should be supported
11. Application should work fine in different themes and background colors
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 16 of 49
Environmental preparation In this section, we will cover the pre-requisites and installation of the MobileCloud for TFS template for Visual
Studio. Your options are to install the MobileCloud for TFS to your desktop where you have Visual Studio
installed or to a Virtual Machine with Visual Studio. MobileCloud for TFS integration with unit testing does not
require TFS. MobileCloud for TFS integration with Microsoft Test Manager does require TFS. The following
guidance will at times seem like a hands on lab, however, some detail is needed as we explain what is needed to
use the MobileCloud for TFS with Visual Studio and TFS.
In order to run Perfecto Mobile scripts from your development environment, you first install the Perfecto Mobile
Cloud for TFS Test Template. The installer creates a Project Template on C:\Program Files (x86)\Microsoft Visual
Studio 11.0\Common7\IDE\ProjectTemplates\CSharp\Test and places Perfecto Mobile assemblies on C:\Program
Files (x86)\PerfectoMobile\Visual Studio. This location is configurable during installation. Test projects created
using the Mobile Cloud Project template use the installed TFS Adapter to make calls to the Perfecto Mobile
service.
Installing the Perfecto Mobile Team Foundation Server (TFS) add-
in
Pre-Requisites
Visual Studio 2012 + Update 2 or higher
Microsoft Test Manager 2012 (required for integration with Test Case)
TFS 2012 Update 2 (if on-premise deployment) or Team Foundation Service3 in the cloud
SQL Server 2008 / 2012 (if on-premise deployment)
Perfecto Mobile Account - https://www.perfectomobile.com/trial/sign_up
NO
TE A user account is required to use the MobileCloud for TFS. This guidance and the HOL exercises assume you have
your own account.
For additional information, you may access the following resources:
- Microsoft Test Manager 2012: http://msdn.microsoft.com/en-us/library/vstudio/dd380763.aspx
- Team Foundation Server 2012: http://msdn.microsoft.com/en-us/vstudio/ff637362.aspx
Performing an Local Installation
Retrieve the installation package perfectomobile-tfs-add-in-installer-1.0.exe from
https://www.perfectomobile.com/support/tfs and run the installer, confirming the User Account Dialog box if it is
displayed. You will have the option to change the installation location folder if you wish.
Once the Perfecto Mobile Installer completes detailed instructions on creating a test class & method to execute
against the Perfecto Mobile cloud can be viewed here: http://aka.ms/perfectomobiletesttemplate
3 http://tfs.visualstudio.com
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 17 of 49
Installing to a Hyper-V Virtual Machine
Similar, to a desktop deployment, the Perfecto Mobile TFS add-in template solution can be deployed to a virtual
machine running in Hyper-V as long as an active network connection is provided to enable access to the Perfecto
MobileCloud.
Installing to the Brian Keller Virtual Machine
If you are using Brian Kellers Visual Studio ALM Virtual Machine 4 to create and execute test scripts using the
Perfecto Mobile TFS Add-in, additional configuration steps are required to set the current date and time and
configure internet connectivity. All software pre-requisites are installed on the Brian Kellers Visual Studio ALM
Virtual Machine.
Enabling Internet Connectivity
Perfecto Mobile is a cloud-based service so test cases must be executed from an internet-enabled machine.
To configure internet connectivity on an instance of Brian Kellers Visual Studio ALM Virtual Machine, an external
virtual switch with internet connectivity should be present in the Hyper-V manager. The ALM Virtual Machine
should then have a network adapter associated with the external virtual switch added to it. Further information
on creating a virtual network and adding a network adapter can be found here: http://aka.ms/configurevm
Synchronizing the Date and Time
Brian Kellers Visual Studio ALM Virtual Machine is configured with Time-synchronization disabled in order for
the sprint backlogs and reports to work properly. This causes an SSL exception when connecting to the Perfecto
mobile service: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS
secure channel.
In order to fix this issue, the VM should be set to synchronize time with the host. Follow these steps to enable
time-synchronization:
1. On the Virtual Machine settings, select the Integration Services tab
2. Check the time synchronization checkbox.
Once these steps have been completed successfully, the Perfecto Mobile TFS Add-in can be installed following
the instructions detailed in the Local installation section.
Accessing the Perfecto Mobile Cloud Platform This section will address the process and steps to access the Perfecto Mobile Cloud platform for a new or an
existing Perfecto Mobile user. The Perfecto Mobile cloud platform is accessible via the URL
https://demo.perfectomobile.com.
Proper login credentials are required to work with Perfecto Mobile TFS add-in and gain access to the
aforementioned URL. You register for a user account at the Perfecto Mobile at the
https://demo.perfectomobile.com site.
To use the Perfecto Mobile service, you have two options:
As a new user
As an existing user
4 http://aka.ms/almvms
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 18 of 49
New User Access
New Users must register to gain access to the demo site. Access Registration by entering the URL specified
above in a browser, and then navigate to the Click Here link as shown below.
Figure 5 - New User Registration link.
This link takes you to Perfecto MobileCloud registration page where you supply your registration information.
NO
TE Make note that the web page screen shots of the partners web site is subject to change from what we have shown
you in this guidance.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 19 of 49
Figure 6 - Mobile Cloud Platform Registration.
Once the I Accept button has been clicked, the relevant login credentials will be sent to the supplied email
address. User credentials are activated and can be used immediately.
Existing User Access
If you have an existing Perfecto Mobile account then login via the Perfecto Mobile URL as detailed above. After
entering your credentials and clicking Sign In, the landing page will display.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 20 of 49
Hands-On Lab If you are intending to follow the Hands-on Lab (HOL) step by step, use these times for each exercise as a
guideline. If, however, you are intending to investigate each step in detail double the times as a bare minimum.
NO
TE
This hands-on lab includes the following exercises:
1. Accessing the Perfecto Mobile Cloud and running an automated test
2. Run an automated test in the Perfecto Mobile Cloud within Visual Studio
3. Run an automated test in the Perfecto Mobile Cloud within Microsoft Test Manager
4. Continuous Integration - Run an automated test in the Perfecto Mobile Cloud within Team Build on TFS
5. Continuous Integration - Run an automated test in the Perfecto Mobile Cloud within Build Service on Team
Foundation Service in the cloud
Estimated time to complete this lab: 55 minutes.
Exercise 1 Accessing the Perfecto Mobile Cloud and running an
automated test (10 min)
GO
AL
In this exercise you will use the perfecto mobile cloud, to run a predefined automated test against the Bank of
America app on a specific mobile device. This automated script will run a functional test.
Persona Mapping
This exercise is focused on Christine as a tester.
Exercise
If using the Brian Keller TFS 2012 Demo Virtual Machine5, you can log on as Julia Ilyiana with the standard
password P2ssw0rd.
1. Open Internet Explorer and browse to the URL https://demo.perfectomobile.com.
2. Log on with your username and password.
NO
TE If you dont have an account, please request your own demo account under:
https://www.perfectomobile.com/trial/sign_up.
3. Click on Automation in the menu bar.
NO
TE When an information window New in Automation! appears, click OK.
5 http://aka.ms/BrianKellerAlmVms
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 21 of 49
4. Click File and Open to open an existing automation script:
Figure 7 - Switch to the automation area.
5. Select folder Public Scripts\Microsoft\ and open the script search_for_atm.
6. The file is a script for test automation and contains different automated test steps:
Figure 8 - List of automated test steps.
7. Click Running View to switch to the test steps execution status view.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 22 of 49
8. Click on Run in the menu bar to start the automated test.
9. Select the device Apple iPhone-4 for the field DUT, use the value iOS in the field OS to run the test on
this specific device and click Run:
Figure 9 - Device selection popup window.
10. The test execution starts and the status for each test step :
Figure 10 - Running view with test script status list.
11. Click Watch in the menu bar while the test is running to switch to the Devices Live video view:
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 23 of 49
Figure 11 - Devices Live video test execution view.
NO
TE The test script runs through the sign in process of the Mobile Banking app and finds the ATM location nearest to a
specific zip code.
12. When the test script completes successfully a test report is available. Click Yes to review the report
results.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 24 of 49
13. Click on General to check the test run result summary. There are information available about status, test
duration, tested device, etc.:
Figure 12 - General view of the test run results.
14. Click on Execution Sequence to review the individual test steps and recorded screenshots. Scroll down
to the last Text checkpoint to verify, that the expected text values Expected text (needle): sign in was
found in the loaded app page and that the elapsed test step time didnt reach the set value for Timeout:
40 sec.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 25 of 49
Figure 13 - Execution sequence view for all test steps.
15. OPTIONAL: Click also on Errors to see if any errors occurred during test execution (should be empty in
our case).
16. OPTIONAL: Click also on Statistics to get an overview about the passed vs. failed Checkpoints and
Actions.
REV
IEW
This exercise you ran an automated functional test against the mobile banking app provided by the company Bank of
America. For testing you used the perfecto mobile cloud. You have learned how to open an existing test script and
run that automation scripts against a selected mobile device. The generated test run report enabled you to verify the
test run in detail.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 26 of 49
Exercise 2 Run an automated test in the Perfecto Mobile Cloud
within Visual Studio (15 min)
GO
AL
In this exercise you will run a predefined automated test against the Bank of America app on a specific mobile device
using Visual Studio. You will code an automated test that will access the perfecto mobile cloud and will run an
automated test script as a functional test.
Persona Mapping
This exercise is focused on Christine as a tester.
Exercise
If using the Brian Keller TFS 2012 Demo Virtual Machine6, you can log on as Julia Ilyiana with the standard
password P2ssw0rd.
1. From within Visual Studio, create a new project via Ctrl+Shift+N.
2. Select the Mobile Cloud Project template for Visual C# in the folder Installed -> Templates ->
Visual C# -> Test and click OK:
Figure 14 - Mobile Cloud Project template.
NO
TE If dont see the Mobile Cloud Project template, please install the perfectomobile-tfs-add-in-installer-1.0 extension
for Visual Studio 2012.
3. Open code file MobileCloudTest1.cs and overwrite the string values for mobileCloudServer, user
and password in the public class MobileCloudTest1 with the following values:
// URL of the perfecto mobile cloud static String mobileCloudServer = "https://demo.perfectomobile.com/"; // user name, used for logon static String user = ""; // password, used for logon static String password = "";
Code 1 Parameters to initialize a connection to the perfecto mobile cloud.
6 http://aka.ms/BrianKellerAlmVms
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 27 of 49
These parameters are necessary to connect successfully to the Perfecto Mobile Cloud during the test run.
N
OTE If you dont have an account, please request your own demo account under:
https://www.perfectomobile.com/trial/sign_up.
Overwrite the string values scriptKey and DUT and add a parameter os in the test method ExecuteScript1(): // path to the automate test script in XML format String scriptKey = @"PUBLIC:Microsoft/search_for_atm.xml"; // Device ID of the emulated mobile device String DUT = "C1C30569F52259AD3E081816C30E54457DA55400"; // expected OS of the emulated mobile device String os = "iOS"; Dictionary runTimeParameters = new Dictionary(); // if needed, add additional runtime parameters below the DUT parameter runTimeParameters.Add("DUT", DUT); runTimeParameters.Add("os", os); // execute script with mobile cloud adapter adapter.Execute(scriptKey, TestContext, runTimeParameters, MobileCloudAdapter.REPORT_FORMAT_HTML);
Code 2 Coded start parameters for the test method ExecuteScript1.
NO
TE The value for the parameter DUT means the Device ID. You can find this Device ID in the perfecto mobile cloud
portal in the list of devices as a separate column.
4. Build the solution with Ctrl + Shift + b and select TEST -> Windows -> Test Explorer in the menu
to open the Test Explorer. Now you can see the test method ExecuteScript1 in the Test Explorer.
5. To run the test method right click on ExecuteScript1 and select Run Selected Tests:
Figure 15 - Test Explorer: Run test method ExecuteScript1.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 28 of 49
6. OPTIONAL:
While the selected test is running it is possible to view the script execution on the real mobile device
in the Perfecto Mobile Cloud. To see the execution in real time open Internet Explorer and browse to
the URL https://demo.perfectomobile.com/nexperience/dashboard.jsp?autoWatch=true.
Figure 16 See test execution on the real device in the cloud dashboard.
7. After a while you will see the test result. The test method ExecuteScript1 passed. Click on the link
Output to open the test result in a separate view:
Figure 17 - Test Explorer: Passed test method ExecuteScript1.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 29 of 49
8. Click on the link in the area Attachments PUBLIC_Microsoft_search_for_atm
Figure 18 - Test Output: Test method ExecuteScript1.
and you will see a html based test execution report:
Figure 19 - Html-based test execution report summary.
9. Scroll down to see the execution time for each test step in the area Execution Sequence:
Figure 20 - Html-based test execution report with execution sequences.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 30 of 49
NO
TE If the test fails, the html based test execution report shows which execution sequences failed.
10. Scroll down to see the device information in the area Devices and click on the link Video ():
Figure 21 - Device information.
11. A video starts and shows the recorded test execution:
Figure 22 - Recorded test sequence video window.
REV
IEW
In this exercise you ran an automated functional test within Visual Studio against the mobile banking app provided
by the company Bank of America. For testing you used the perfecto mobile cloud. You have learned how to create a
new Visual Studio Mobile Cloud project. You modified the provided test code to execute an existing test script and
run that automation script against a selected mobile device. The generated html based test sequence report enabled
you to verify the test run in detail. The recorded test execution video was helpful to visually verify the test execution
on the mobile device.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 31 of 49
Exercise 3 Run an automated test in the Perfecto Mobile Cloud
within Microsoft Test Manager (10 min)
GO
AL
The Perfecto Mobile Cloud can also be accessed from within a Microsoft Test Manager (MTM) test case. In this
exercise, you will create a test case and enable automation using the unit test you created in exercise 2 so that you
can see the end to end ALM integration for mobile device testing.
NO
TE Microsoft Test Manager requires integration with TFS where unit testing does not. The test project and unit test file
created in Exercise 2 will need to be checked into your TFS repository so that the test plan you create can access the
unit test.
Persona Mapping
This exercise is focused on Christine as a tester.
Exercise
If using the Brian Keller TFS 2012 Demo Virtual Machine7, you can log on as Julia Ilyiana with the standard
password P2ssw0rd.
1. From within Microsoft Test Manager, under the Plan tab, create a new test case.
2. Enter the title of the Test Case. There is no need to add any Test Steps.
Figure 23 - Create a new test case.
NO
TE The automation status of the Test Case is set to Not Automated. You will need to use Visual Studio to add the
associated automation as listed in the following guided steps.
3. After saving your test case, open Visual Studio and query the test case previously created.
4. Navigate to the Team menu item in the menu bar within Visual Studio, click on Go to Work Item
and provide the id number for the test case that was created in Microsoft Test Manager.
7 http://aka.ms/BrianKellerAlmVms
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 32 of 49
5. Once the test case has opened, navigate to the ASSOCIATED AUTOMATION tab as shown below:
Figure 24 - Test Case associated automation.
6. Click on the ellipse button. The test name previously created should appear. Highlight the test name
and click OK as shown below.
Figure 25 MTM: Associate the test case with the test script.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 33 of 49
7. After saving the test case, the Unit Test will now be associated with the Test Case.
Figure 26 MTM: Test case details.
NO
TE You will now need an Environment to run the test case with automation. If you do not have an Environment, you will
need to create one. If you do not have an Environment configured, the following steps will guide you through the
process of creating a new environment to run the Test Case.
8. Within Microsoft Test Manager, navigate to the Lab Center, select the Lab tab, and click New:
Figure 27 MTM: Create a new environment.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 34 of 49
9. Select Standard Environment and provide a suitable name. Then click Next:
Figure 28 MTM: Set environment name.
10. In the Machines tab, select Add Machine.
11. Provide the name of the computer from where the tests will run.
12. Follow the steps and click Verify:
Figure 29 MTM: Add machine
13. Once verification is complete, a new Environment is created and initialized in a running state.
14. The Environment can now be used by associating it in the Test Settings tab in the Lab Center.
15. Within Lab Center, select the Test Settings tab.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 35 of 49
16. Click New, provide the Name and select the tests type for Automated as shown below:
Figure 30 - MTM: Configure Environment with Test Settings.
17. Click Next to navigate to the Roles tab and select the Environment created previously.
18. Click Finish to create a new Test Setting. The completed Test Setting is shown below:
Figure 31 MTM: Completed Test Settings.
19. Next, navigate back to the Test Center and click on the properties in the Plan tab.
20. Select the Test Settings and Test Environment created previously.
Figure 32 MTM: Plan properties.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 36 of 49
21. Now click on the Modify link under Builds: Build in use section of the properties, click the Assign to
plan button to associate the build triggered in the Visual Studio steps previously.
The completed properties tab should look as follows:
Figure 33 MTM: Completed Properties tab.
22. Navigate to the Test Lab in Test Center and under Run select Run with options. Pick the selection that
was created in the properties section and click Run as shown below.
Figure 34 MTM: Select run options.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 37 of 49
23. OPTIONAL:
While the selected test case is running it is possible to view the script execution on the real mobile
device in the Perfecto Mobile Cloud. To see the execution in real time open Internet Explorer and browse
to the URL https://demo.perfectomobile.com/nexperience/dashboard.jsp?autoWatch=true.
Figure 35 See test execution on the real device in the cloud dashboard.
Allow enough time for the test to pass depending on your network connection. A successful test run is shown
below.
Figure 36 - MTM: Successful test run.
The Perfecto Mobile Cloud for TFS add-in will attach the test results to the test case run. Once again, the results
are an html file showing the sequences as they were executed within the Perfecto Mobile Cloud. This is shown
below:
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 38 of 49
Figure 37 MTM: Html-based test execution report with execution sequences
REV
IEW
In this exercise you created a test case and enabled automation of the functional test within Visual Studio against the
mobile banking app provided by the company Bank of America. For testing you used the perfecto mobile cloud. You
have learned how to integrate a test created in a Visual Studio Mobile Cloud project with Microsoft Test Manager.
The generated html based test sequence report enabled you to verify the test run in detail.
Exercise 4 Continuous Integration - Run an automated test in
the Perfecto Mobile Cloud within Team Build on TFS (10 min)
GO
AL
The Perfecto Mobile Cloud can also be accessed from within a Team Build. In this exercise, you will create a new
build definition and enable testing within Continuous Integration using the unit test you created in exercise 2 so that
you can see the end to end ALM integration for mobile device testing during the build process.
NO
TE
Continuous Integration requires integration with TFS. The test project and unit test file created in Exercise 2 will need
to be checked into your TFS repository so that the Team Build can access the unit test. Follow the steps from the
Exercise 2 on the Hands-on Lab first to create a Mobile Cloud Project in Visual Studio and start with this Exercise.
Persona Mapping
This exercise is focused on Christine as a tester.
Exercise
If using the Brian Keller TFS 2012 Demo Virtual Machine8, you can log on as Julia Ilyiana with the standard
password P2ssw0rd.
Before running Perfecto Mobile test cases as part of a Continuous Integration build with Team Foundation
Server, the solution from Exercise 2 must be checked-in to TFS Source Control by following these steps:
1. From Solution Explorer, Right-click on the solution.
8 http://aka.ms/BrianKellerAlmVms
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 39 of 49
2. Select Add Solution to Source Control:
Figure 38 Adding the Solution to Source Control.
3. On the Add Solution to Source Control dialog, select a Source Control Path and click OK:
Figure 39 Selecting the Source Control path.
4. Right-click on the solution again and select Check In.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 40 of 49
5. In the Pending Changes tab of Team Explorer, enter comments and click on Check In:
Figure 40 Checking in the solution.
6. Once the solution is checked in, create a new build definition by going to the Builds tab in Team
Explorer and clicking on New Build Definition:
Figure 41 Creating a new Build Definition.
Provide a Name for the new Build Definition and select all the defaults. On the Process tab, verify that your
solution is set as the value for Projects to Build. Expand the Automated Tests section in the build definition,
then Test Source and make sure that the Test Source Spec argument includes the name of your test
assembly. By default, this value is **\*test*.dll. If you selected the default name when creating the Perfecto
solution, your assembly name will be Mobile Coud Project1, which would not be picked up by the build
process. You should update this value to **\**.dll.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 41 of 49
Figure 42 Setting Build Definition Arguments.
You may now queue the build and your Perfecto Mobile test case will run during the build.
Figure 43 Build Results, including completed test case.
Exercise 5 Continuous Integration - Run an automated test in
the Perfecto Mobile Cloud within Team Build on Team Foundation
Service in the cloud (10 min)
GO
AL
The Perfecto Mobile Cloud can also be accessed from within a Build based on Team Foundation Service in the cloud.
In this exercise, you will create a new build definition and enable testing within Continuous Integration using the unit
test you created in exercise 2 so that you can see the end to end ALM integration for mobile device testing during
the build process. The difference between this Exercise 5 and the Exercise 4 before is, that Continuous Integration
will now run in the cloud.
NO
TE Continuous Integration requires integration with TFS. The test project and unit test file created in Exercise 2 will need
to be checked into your TFS repository so that the Team Build can access the unit test. Follow the steps from the
Exercise 2 on the Hands-on Lab first to create a Mobile Cloud Project in Visual Studio and start with this Exercise.
Persona Mapping
This exercise is focused on Christine as a tester.
Exercise
If using the Brian Keller TFS 2012 Demo Virtual Machine9, you can log on as Julia Ilyiana with the standard
password P2ssw0rd.
When you install Perfecto Mobile on a development machine, a set of assemblies get copied to C:\Program
Files (x86)\PerfectoMobile\Visual Studio. Since Perfecto Mobile cannot be installed on TFS Service, those
9 http://aka.ms/BrianKellerAlmVms
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 42 of 49
assemblies should be included as part of our solution so that they get included with the build and are available
when the test runs during the build.
The assemblies needed are: PerfectoMobile.MobileCloudHttpClient.dll and PerfectoMobile.TFS.Adapter.dll.
Figure 44 Adding Perfecto Mobile assemblies to a solution.
Before running Perfecto Mobile test cases as part of a Continuous Integration build, the solution must be
checked-in to TFS Source Control by following these steps:
1. From Solution Explorer, Right-click on the solution and select Add Solution to Source Control:
Figure 45 Adding the Solution to Source Control.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 43 of 49
2. On the Add Solution to Source Control dialog, select a Source Control Path and click OK:
Figure 46 Selecting the Source Control path.
3. Right-click on the solution again and select Check In.
4. In the Pending Changes tab of Team Explorer, enter comments and click on Check In:
Figure 47 Checking in the solution.
5. Once the solution is checked in, create a new build definition by going to the Builds tab in Team Explorer
and clicking on New Build Definition:
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 44 of 49
Figure 48 Creating a new Build Definition
Provide a Name for the new Build Definition and select all the defaults. On the Process tab, verify that your
solution is set as the value for Projects to Build. Expand the Automated Tests section in the build definition,
then Test Source and make sure that the Test Source Spec argument includes the name of your test
assembly. By default, this value is **\*test*.dll. If you selected the default name when creating the Perfecto
solution, your assembly name will be Mobile Coud Project1, which would not be picked up by the build
process. You should update this value to **\**.dll.
Figure 49 Setting Build Definition Arguments.
You may now queue the build and your Perfecto Mobile test case will run during the build.
Figure 50 Build Results, including completed test case.
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 45 of 49
Conclusion We have shown you how easy it is to integrate the Perfecto Mobile Cloud for TFS with Visual Studio. Testing of
your mobile application running on real devices in the cloud is possible as strictly unit tests or as automation
tests associated to a test case in Microsoft Test Manager. These integration points allow these tests to be run as
part of a build definition for daily, manual, or continuous integration builds. This integration provides the test
status reporting features, like status of tests (pass/fail) along with the failure reason.
This document does not cover the setup details required to develop for iOS, Android, or Windows Phone. Team
Explorer Everywhere 10 is required to enable developing and building Android and iOS applications. For an
overview of the lifecycle with Mobile Cloud see the overview poster for this ALM Ranger project.
10 http://msdn.microsoft.com/en-us/library/gg413285.aspx
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 46 of 49
Appendix
Automation Tool Comparison
Feature SeeTest from Experitest PerfectoMobile
Supported Mobile OS Android
iOS
Blackberry
WindowsPhone8
All emulators are supported
Android
iOS
Blackberry
Remote Access See Test does not provide
devices.
Connect all mobile devices to
one central hub/server that
resides in your organization to
form private cloud
Enable remote access from
anywhere in the organization to
any device
Provides devices which are
accessed over cloud.
Test Coverage Object identification by: Native
ID, HTML5-DOM, Image
Recognition and Text
Recognition
Browsers testing
Reboot device
Awake device from sleep mode
Operate device power button
Operate device Back, Call/Hang
Call, Home, Menu buttons
Shell commands for internal
device counters (e.g. CPU,
memory)
Landscape model
Supports Visual Objects,
Native Objects, DOM
Objects
Get Device Info, state, log
Open Device, Reboot device,
rotate device
Keyboard display
Cloud call, SMS, email
Native Object
Identification
Object identification by: id, class,
text, accessibility Label,
accessibility Identifier.
Simple recorder
Object spy for easy identification
of object native properties
Easy instrumentation with no
code change
Supports Native Objects,
DOM Objects
Object Spy
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 47 of 49
Web & Hybrid Object
Identification
Object identification by: id,
name, class, text, css.
JavaScript execution.
Support for native browser
Operations like click, send text,
select, touch and others.
jQuery support.
Object Spy for easy identification
of object web-DOM / hybrid
properties
Easy instrumentation with no
code change
Supports Native Objects,
Web Objects
Object Spy
Install/Uninstall application
Text Object Identification OCR engine with support for all
languages
Text input
Supports OCR
Text input
Image/Icon Object
Identification
Standard icons/images (e.g.
buttons)
Proprietary icons/images
Auto-sense capabilities for one-
click image extraction
Sensitivity calibration to changes
in image (e.g. size, color,
background, transparency)
Image identification on changing
backgrounds (e.g. transparent,
gradient, moving elements)
Allows simple and intuitive
method for pressing virtual
locations on touch handsets
based on text or image
analysis. The system
automatically finds those
elements, even if their
location changes, without
any change to the script
itself
Gesture Support Multitouch
Swipe
Drag & Drop
Zoom
Scroll inc. conditional scrolling
Swipe
Touch
Trackball
Keyboard Support Device keyboard:
All configurations (alphabet,
punctuation and numbers, and
punctuation and symbols)
All languages
PC keyboard special characters
and capitalization
Regardless of the method
implemented by the
handset (physical or virtual
keyboard, different
languages, capitalization),
simple type text function
hides the complexity of any
string typing
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 48 of 49
Scripting Capabilities Recorder
Script debugger
Script Multiplier: Script Once, Run
on Multiple Devices
Can convert SeeTest to different
programming languages e.g.
C#,Java etc. and hence
extensibility is high
Recorder
Script Multiplier: Script
Once, Run on Multiple
Devices
Perfecto Script cannot be
converted to any
programming languages
Plug-in to Existing Testing
Frameworks
Plug-in to
QTP
MS Test
Test Complete
Perl
Python
JUnit3, JUnit4, TestNG
Full functionality supported:
Scripting, editing and reporting
(inc. screenshots) capabilities
directly from within test tools
Plug in to
QTP
MS Test
Continuous integration
support
Full and seamless integration to
QC,TFS,JUnit, NUnit, PyUnit,
TestNG, Ant
Full and seamless
integration to QC,TFS etc.
Comprehensive Reporting Capture the screen shots and
gives summary reports
Capture screenshots and
record videos Share test
reports with other
colleagues and partners,
Share devices with others to
demo new apps/features,
reproduce a bug.
Table 5 Leading Automation tools comparison
FAQ Which type of Mobile applications need to be tested on different mobile networks like 2G/3G/4G LTE Conditions?
For specialized applications where on location testing for specific cellular networks is to be considered like Free
usage allowance or two way SMS
Why to test such applications on different mobile networks?
Mobile networks have limited bandwidth and high latency (compared to Wi-Fi and broadband) which effects the
application performance by increasing the page load time on mobile device resulting in testing an application in
real world conditions
How to test an application on different mobile networks?
Network simulation tools helps in reproducing different network conditions by limiting bandwidth and
simulating latency as a result help you in testing whether all mobile users will be getting best user experience in
acceptable response times on mobile devices
-
ALM Rangers - Testing mobile applications with Perfecto MobileCloud
Page 49 of 49
Device anywhere and perfecto are also good solutions if it is required to test an application on a particular
networks like US Network, Canada network
Check list for network testing of mobile applications:--
Test behavior of application when there is no network or network problem is there.
Test the application on all supported network connections.
Test application does not hang whenever there is a change in the signal strength.
For a device it should switch to mobile network when there is no Wi-Fi and back to Wi-Fi when there is
both WI-FI and mobile network
Test the robustness of the application in low bandwidth/high latency scenario.
Is the application giving the correct error whenever is some network related issue rather than any
exception or false error.