mobile test automation perfecto star east
TRANSCRIPT
![Page 1: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/1.jpg)
101 – How To Deliver Winning Mobile Apps
Perfecto Mobile &
Toronto Association of System and Software
![Page 2: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/2.jpg)
Eran Kinsbruner is a director & mobile technical evangelist at PerfectoMobile, one of the leading mobile cloud and automation companies. Formerly CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at Qulicke & Soffa, Sun Microsystems, General Electric, and NeuStar. The co-inventor of a test exclusion automated mechanism for mobile J2ME testing at Sun Microsystems, Eran has experience in the mobile testing world. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and his professional mobile testing blog at ek121268.wordpress.com.
![Page 3: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/3.jpg)
What To Do?
Great News – Your first Mobile Project has arrived!
Now What?
![Page 4: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/4.jpg)
Presentation Objectives
• Mobile Market State
• Mobile QA/Dev Objectives
• Sustaining a Mobile QA Plan Based on CQL• Functional Testing
• Non Functional Testing
• Regression Testing/CI
• Pillars Of A Continuous Quality Lab
• Mobile QA Best Practices
![Page 5: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/5.jpg)
Mobile Market State
Digital experience drives innovation
• IOT
• Complex use cases
Fragmentation becomes more painful
• Devices
• OS’s
Tool mix as a key for agile adoption
• Commercial, Open Source, Proprietary
![Page 6: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/6.jpg)
Mobile QA & Dev Objectives
Source: OpenXcell
Qualit
y
Velocity
Increase Quality
Increase Velocity
Mobile QA & DevObjectives
![Page 7: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/7.jpg)
QA & Dev Testers Vision (aspiration)
Increase mobile app velocity without
compromising on quality
Keep up with mobile market changes and
meet end user requirements
Develop & Test against the real end
user profiles
Leverage unattended automation as part of
any nightly build through CI workflow
Receive actionable mobile app quality insights per build
![Page 8: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/8.jpg)
RBC – QA Objectives
v
Improved Quality Optimized Cost Faster Time to Market
Operational Effectiveness
Operational Efficiency
Business Value
• Increase Test Coverage• Test Effectiveness – reduce test
duplication between QA & business • Reduce defect leakage to production
• Reuse test automation during UAT test cycles
• Decrease QA test cycles – via test automation
• Near zero downtime/production failures
• Applications perform as expected
• Test automation cost savings• Manual and automation test
management savings 10% using BDD Approach
• Onsite to offshore ratio 40:60• QA Technical Solution Lead across all
projects• Leverage Perfecto Mobile cloud solution
to save on devices and data plans
• Reduction in Cost of Testing due to productivity improvements-Maintenance and Projects
• Reduce regression test cycle by 5-10 days per project
• Perfecto Mobile UFT
• Continuous improvement in test design & execution
• Maximize Test Automation coverage• Increase re-use - BDD
• Enable business to respond to market change
• Proactive feedback on market changes to existing production mobile applications
![Page 9: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/9.jpg)
SUSTAINING A MOBILE QA PLAN
![Page 10: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/10.jpg)
Sufficient Coverage
• Choose the right devices and OS platforms for your project
• Get Perfecto Mobile’s Q1 2015 Mobile Test Coverage Index
![Page 11: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/11.jpg)
Q1-2015 Mobile Test Coverage Index (Preview)
![Page 12: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/12.jpg)
What’s Coming? Why It’s Important?
![Page 13: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/13.jpg)
Manual & Interoperability
• Starting point for Pre-Automation phase (foundation)
• Candidates to keep manual:– Incoming events
• Calls
• Text Messages
• Notification pop ups
– Complex use cases to automate
![Page 14: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/14.jpg)
Functional Automation – That Works
• Reusable, Repeatable, Predictable Tests– Unattended
– Continuous
– Easy to maintain
• Object Support On Any Mobile App Style– Dom, Visual, Native
– Instrumentation (Yes/No)
– System Level Control
• Flexible Frameworks
100% accurate native object rec.
Much faster than OCR
Support all languages
Does not often detect GUI defects
Imperative for enduser’s experience
Imperative for GUI glitches
Implement theWYSIWYG
Slower then objectlevel analysis
Dependent on 3rd party
Does not support all languages
Object level analysis
Visual analysis (OCR)
May require Instrumentation
![Page 15: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/15.jpg)
Levels of Test Automation
Level 1
Level 2
Level 3
Level 4
Level 5
Navigation flows
Content validation
Error Validation
Data validation(if feasible)
UI validation(if feasible)
Check automation feasibility Identify scenarios for basic navigation & successful function flow
Navigation flows are automatedCheck automation feasibility of screen content
Level 2 Scenarios are readyCheck automation feasibility of error validation
Level 3 scenarios are readyCheck automation feasibility of data validation
Level 4 scenarios are readyCheck automation feasibility of UI validation and other complex features
Automated scenarios of basicNavigation % happy path are ready
Content validation scripts are ready and plugged in with exiting automated navigation BPT component
Error validation components are ready and plugged in with automated components of Level 2
Data validation scripts are written and plugged in with existing automated components of Level 3
Automated scripts for UI screen & other complex scenarios are ready and plugged with existing scripts
Data D
riven A
uto
matio
n Fram
ewo
rk
![Page 16: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/16.jpg)
Non Functional Testing
• Cover key business transactions response time & general mobile app availability
• Early actionable performancefeedback & insights
• Assures high end user experiencein various conditions (aka: Wind Tunnel)– Various network profiles – Various locations– Various load conditions– Incoming events – Per app insights
![Page 17: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/17.jpg)
Mobile App Monitoring
• The only Efficient Way to Monitor Mobile Apps
– Browsers and Emulators - irrelevant for mobile monitoring
• Capture Your True Mobile End User Experience
– Monitor mobile engagement scenarios, including interoperability on real devices on real networks
• Early Warnings and Fast Correction
– Real-time alerting triggered when transaction availability or response times are compromised
• Extend your existing quality assets for monitoring
– Automation test code
– Performance test code
![Page 18: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/18.jpg)
What Makes A True CQL
Quality Lab
![Page 19: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/19.jpg)
What Makes A Complete Continuous Quality Lab
Real Device Lab (Device as a Service)
Functional Manual Testing
Functional Automation
Performance Testing
Real environments & Profiles
Production Monitoring
Hybrid Cloud (Local, Public, Private)
Operations, Security & Support
3rd Party API’s & Integrations
![Page 20: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/20.jpg)
Mobile QA Best Practices
![Page 21: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/21.jpg)
Mobile Continuous Integration Recommended Work Flow
Dev Team
Source Control
Code Commit
CI Server
ReportingUnit Tests
Launch Perfecto Job
(Release Package)
Support today:
PM Native & Selenium
Automated App Deployment On Real Devices
Functional Testing
Integration Testing
Performance Testing
Continuous Feedback
Test Planning & Test Execution
Bug Reporting
![Page 22: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/22.jpg)
Build Your Mobile App Test Plan Properly
Functional
• Validation of Functionality
• Smoke / Regressions Testing
• Offline access testing
• Negative Testing
Non Functional
• Network Strength / Outage / Recovery
• Different NW Types
• Peripheral Testing
• Services
Interoperability (IOP)
• Voice / SMS interrupts
• Notifications
• Battery /Cable Removal
Memory Leak
• Memory Usage
• Memory Leaks
• Garbage Collection
Installation Testing
• New App Install
• Uninstall and Reinstall
• Upgrade testing
Language Testing
• Validation for Locales (French & English)
• Images and Text
• Currencies, time zones etc.
• Context
Performance Testing
• CPU Usage testing
• Network Usage
• Page Render time or activity Render time
Security Testing
• OWASP Vulnerabilities
• Sensitive data (App/Device)
• Dynamic Testing
• Static Code Analysis
• Data Encryption
Usability Testing
• User Experience
• Competitive Analysis
• Expert Review
![Page 23: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/23.jpg)
RBC Mobile Quality Lab
Team Size –
45 (Onshore/Offshore Mix)
>100 devices - Blackberry, Android & iOS platforms
Using Selenium
& UFT
![Page 24: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/24.jpg)
Continuous Automation Regression Model
• Executing automated test scripts 24*7 (consider offshore team)
– Private Perfecto Cloud – used for project development• Execute automated testing suite to reduce project
timelines
– Internal Perfecto Cloud –used for project and maintenance testing on supported devices
– Perfecto Public Cloud – test non supported devices (new market devices/ mimic customer behaviour). Leverage Perfecto public metrics i.e. what is currently in use by public (device and configuration)
Hybrid Cloud
![Page 25: Mobile test automation perfecto star east](https://reader034.vdocuments.us/reader034/viewer/2022050719/55a8698e1a28ab64028b46b9/html5/thumbnails/25.jpg)
Q&A