automating hybrid apps- which tool and why_unicom

www.unicomlearning.com

Next Gen Testing Summit-2014

14th Nov, 2014 & Pune

Automating Hybrid Apps Testing: Which tools , Why & how does it affect what you already have ??Speaker Name : Indira Pai, Ayan Modak

Speaker Designation : Lead SQA Engineer, Sr. SQA Engineer

www.nextgentesting.org

Clarice Technologies

http://www.unicomlearning.com/

http://www.nextgentesting.org/


The Question

UNICOM PresentsNext Gen Testing Summit-2014


• You own and run the automation framework of a web based app which is widely used.

• To increase the reach of a widely used web app, the clients decide to

• “Go mobile” and launch a mobile version of the app. • Use “Hybrid App” technology to build the apps.

• As the automation testing owner, you are faced with the challenge of finding the right testing solution for the app in question. How do you go about it?




What are Hybrid Apps



Hybrid apps: • Are written with web technologies (HTML5, CSS

and JavaScript).

• Run inside a native container and leverage the device’s browser engine (but not the browser) to render the HTML and process the JavaScript locally.

• Use a web-to-native abstraction layer that enables access to device capabilities that are not accessible in Mobile Web applications, such as the accelerometer, camera and local storage.

• Are a union of web technology and native execution.




Points to Consider



• Automation Scope: Unit level or UI and Functional Level?

• Automation Support for WebView

• Support for Various Mobile App Gestures

• Mode of Locating Various Elements in the App

• Support for Multiple OS versions

• Cross Platform Support for Hybrid Apps

• Technological Considerations of the Automation Tool




Points to Consider



• Automation Scope: Unit level or UI and Functional Level?

• Unit Level: Testing the smallest possible unit which can be isolated in the system.

• GUI and Functional level:

• Not just testing certain apis or GUI flows,

• Testing a collection of related technologies and APIs that together form the new programmatic accessibility story.

• GUI and functional component -based flows which are user centric are automated.

• We have conducted our study for GUI and Functional level automation.




Points to Consider



Automation Support for WebView

• Hybrid apps use a WebView control (UIWebView on iOS, WebView on Android and others) to present the HTML and JavaScript files in a full-screen format, using the native browser rendering engine (not the browser itself).

• WebKit is the browser rendering engine that is used on iOS, Android, Blackberry and others.

• No constraint on using HTML and JavaScript for only in-browser implementations on mobile devices.




Points to Consider



• Support for Various Mobile App Gestures

• Touch and gesture interactions make mobile experiences easier and more fun to use.

• Core and commonly used gestures like Tap, Double Tap, Drag, Flick, Pinch, Spread, Press, Press and Tap, Press and Drag, Multi-finger Drag should be supported by the tool.




Points to Consider



Mode of Locating Elements in the App

• For automation scripts, the following considerations need to be taken into account.

• Using ID-based automation only may not be the most favourable approach as some popular hybrid app development frameworks may cause dynamic non pattern based ID generation for apps.

• In case there are restrictions as stated above for ID-based automation, XPath and CSS-Selectors are more preferred locators.

• The tool should be flexible in terms of locator support and support maximum locators such as ID, XPath and CSS-Selectors.




Points to Consider



Support for Multiple OS versions

• As the apps would be running on multiple devices with different OS versions, the tool should support the same.

• Limited support for certain OSs will limit benefits of automation.




Points to Consider



Cross Platform Support for Hybrid Apps

• The main advantage of an hybrid app is the cross platform support.

• The tool chosen for automation must therefore, support the same.

• Otherwise, the ROI for time and effort for automating the hybrid app is not optimal.




Points to Consider



Technological Considerations of the Automation Tool

Once the tool has met all of the considerations above, you can further evaluate it on the following parameters:

• Is the tool Open Source?

• Does it have Record and PlayBack support or is hands-on coding needed?

• Are there chances of false alarms (False Negative and False Positive)?

• Is re-usability of scripts on release candidate builds possible?

• Does it provide flexibility of technological know-how ?




How we did it @Clarice



• At Clarice, we develop and test numerous mobile apps.

• Based on the key parameters mentioned, we evaluated the capabilities of a couple of automation frameworks, like

• Appium

• Calabash

• MonkeyTalk




Tools Study



Appium

• An open source test automation framework

• Supports native and hybrid mobile apps automation

• It drives iOS and Android apps using the WebDriver JSON wire protocol.

• Technical “need to knows” for this framework:• Need a good understanding of WebDriver.

• For Details refer : http://appium.io/



http://docs.seleniumhq.org/projects/webdriver/

http://docs.seleniumhq.org/projects/webdriver/

http://appium.io/

http://appium.io/


Tools Study



Appium Pros:

• No compulsory source code access needed for automation (apart from getting XPaths, which can be done with access to www folder, or using Appium Inspector).

• Element selection using Selenium WebDriver (Selendroid for Android): Support for XPath and CSS selectors, not limited to ID-based automation.

• Final release candidate can be tested without any changes in the .apk/.ipa.

• Since it uses WebDriver, portability of scripts in multiple languages is also possible, which makes it very flexible.

• Supports maximum gestures like tap, flip, swipe, pinch, zoom, and tilt.




Tools Study



Appium Cons:

• For iOS app, Appium uses Instruments for initialization.

• Instruments itself is a little buggy during initialization, which would cause problems for Appium.

• Reinstalling Xcode, setting dev path correctly can solve this problem… Not exactly an Appium con, more of a configuration overhead

• Record and Playback support for Android:• Although “Record and Playback” support is there for iOS via Appium

Inspector, but

• For Android, it is still issue prone. Appium team encourages the use of uiautomatorviewer, which is an inspector tool provided by Google, that provides similar functionality to the Appium inspector tool.




Tools Study



Calabash:

• Enables Automated UI Acceptance Tests written in Cucumber to be run on iOS and Android applications.

• Tightly with a Xamarin.iOS and Xamarin.Android project

• The actual tests are written in Gherkin, backed with Ruby code, and run in the context of the Cucumber Framework.

• Initial support for C# and JVM-based languages is also there.

• Technical “need to knows” for this framework:• Elementary Ruby• Elementary Cucumber• BDD knowledge

• For details refer: http://calaba.sh/



http://calaba.sh/


Tools Study



Calabash Pros:

• Powerful WebView support (API support for helper functions like wait, double tap, css selectors query were very good).

• Support for XPath and CSS selectors, not limited to ID-based automation.

• Cucumber support helps to keep the script easy to write and read in terms of user scenarios.

• Supports gestures.




Tools Study



Calabash Cons:

• Compulsory source code (XCode project) access needed for iOS automation.

• No independent “Record and Playback” feature available (Can be done when integrated with UIAutomation Framework in Instruments for iOS for platforms under iOS7 and ADB screen record in Android).

• Final release candidate for iOS apps cannot be tested as the tests need to run on the build generated by the project (else certain events may not work), and then the final archived build is generated.




Tools Study



MonkeyTalk

• A cross-platform testing tool

• Records and plays back test scripts for• Native iOS • Native Android apps• Mobile web apps • Hybrid apps.

• Technical “need to knows” for this framework:• Elementary JS.• Respective framework knowledge in which App was developed

(elementary level).

• For details please refer : https://www.cloudmonkeymobile.com/monkeytalk



https://www.cloudmonkeymobile.com/monkeytalk


Tools Study



MonkeyTalk Pros

• No compulsory source code access needed for automation.

• Record and Playback feature.

• Can help in automating redundant steps.

• Can help in tracing back steps to an issue while exploratory testing.




Tools Study



MonkeyTalk Cons:

• Issues specific to the MonkeyTalk agent build are encountered:

• App when running with respective MonkeyTalk Agent can lead to issues like random crashes/ performance issues, which do not get encountered when the app is running without the MonkeyTalk Agent.

• Such issues are very difficult to debug and can lead to wastage of time and effort if the issue turns out to be MonkeyTalk build-specific.

• Final release build cannot be tested• The build has to be built with the respective platform’s agent,

• Hence the release candidate will have to be rebuilt without MonkeyTalk agents.




Appium



MonkeyTalk Cons (continued)• The framework is still very young and issue prone:

• Sometimes the Swipe action was interpreted as Tap, which has to be changed, else it gives error on playback.

• Does not record the correct classes when text was being typed in text fields (needs manual correction).

• Needs code-level knowledge as much as unit level test cases knowledge.

• More suitable for plain JQuery app, not much suited for Angular JS, and Sencha apps.

• Suitable for basic functions, but for adding more scenarios and logic, requires coding, which may not be high return of investment.

• ID based and initial Xpath based automation only:• Difficult to retrieve elements as CSS Selectors are not supported.• XPath support initial level, cannot support all cases in Webview. • Dynamic IDs cannot be automated if they are not pattern based.




Case Study



The Subject

• A hybrid app developed in Angular JS and Node JS, built using Apache Cordova for Android and iOS platforms.

• The app has basic screen navigations and mobile gestures like swipe, scroll, pinch, and tap.

• During app development, some UI elements have been assigned an ID, some have dynamic ids, which do not get generated in a pattern.

• Certain sanity flows, for example, login and navigation between screen 1 and screen 2 , and certain gestures like tap, swipe, scroll need to be automated.

• These flows needs to be run on the external release candidates for both Android and iOS builds.




Case Study



Automation Activity Observations and RemarksAppium Calabash MonkeyTalk

Was recording a script as initial automation PoC for the iOS app and playing it on the same app successful?

Yes Yes No

Was recording a script as initial automation PoC for the iOS app and playing it on the Android app successful?

Yes Yes No

Could all Sanity Flows be automated? Yes Yes No

Could we run our sanity scripts on the potential Release Candidate ?

Yes Android: YesiOS: No

No

Conclusion: Did the tool meet all requirements for automating the app in question?

Yes Almost No




Conclusion



The Conclusion

Although Appium and Calabash both are optimal candidates for hybrid app automation, we recommend to go ahead with Appium due to the following fine winning points:• Webdriver support which allows portability of the test script in multiple

frameworks.

• No dependability on project setup (apart from access to www folder helps in getting XPaths).

• Flexibility of running scripts on app: scripts can be run on the release candidate for the app.

Calabash is a promising alternative, if one is comfortable in Cucumber, Ruby-based test framework. C# and JVM-based languages are also supported.




You’ve chosen the tool, now what ?



Re-Using existing frameworks

After selecting the correct framework for your needs, you need to have the ability to re-use the existing framework you may have created earlier.

This may have been for a Web App or Desktop based app.

How can we make the best of this existing framework and still use the latest capabilities of a framework like Appium ?




Clarice Framework – Appium Integration






Questions



Questions




THANK YOU

Next Gen Testing Summit-2014Organized by

UNICOM Trainings & Seminars Pvt. Ltd. [email protected]

Speaker name:Email ID:


automating hybrid apps- which tool and why_unicom

Documents