hassle-free continuous integration with real device testing
TRANSCRIPT
The Mobile DevOps Company
Ville-Veikko HelppiHead of Demand Generation / Bitbar
Hassle-Free Continuous Integration Combined with Real Device Testing
WEBINAR
Nikita AvvakumovBack-End Engineer / Greenhouse CI
Presenters
Agenda• The ‘Shift-Left’ Paradigm• Mobile DevOps & Agile • Delivery Pipeline for Mobile Apps• Absolute Requirements for Mobile DevOps• Hands-On Demonstrations
o Build with Greenhouse o Test with Bitbar
• Q&A
Hassle-Free Continuous Integration Combined with Real Device Testing
http://bitbar.com/testing/
The ‘Shift-Left’ – What People Think It Is
http://bitbar.com/testing/
Shift-Left Testing(happens earlier)
Traditional Testing(happens before release)
The ‘Shift-Left’ – What It Actually Means
http://bitbar.com/testing/
How To Get There: CBTPRCM BTDM
http://bitbar.com/testing/
CONTINUOUS TESTING
• Modern mobile app development is Agile, but development infrastructure does not support Agility well.
• DevOps practices are emerging in to the intersection of Development, QA and Operations.
• DevOps practice emphasizes the collaboration and communication of both software developers and operations while automating the process of software delivery and infrastructure changes.
• DevOps practice establishes a culture and environment where building, testing, and releasing software, can happen rapidly, frequently, and more reliably.
(Mobile) DevOps & Agile
Development
QA
Operations
Dev
Ops
http://bitbar.com/testing/
SPEED WINSIt’s a simple fact. The faster you can deliver new applications, features and upgrades to market, and collect feedback, the better your business performs. For that you need higher development pace. And a lot of automation.
THERE IS A DEMAND FOR MOBILE DEVOPSDevOps practices and tools have revolutionized the pace and quality of service delivery – but only for traditional server-based apps.
FRAGMENTATION WITH TOOLSIn mobile the delivery pipelines are more comple – in terms of tools and used methods. Cohesive and well-integrated development tool flow derives from Mobile DevOps practices.
MANUAL DOING IS NOT EFFECTIVEEvery stage requires a lot of manual work and silo’ed solutions from various vendors and/or open source tools.
Why Mobile DevOps
http://bitbar.com/testing/
Delivery Pipeline for Mobile Apps
Development
Source Code
Management
SCM
Build Test Deploy Production
Dev Tools:• IDEs• JIRA• Slack• etc.
SCM:• Git/
GitHub/GitLab
• Bitbucket• MSFT• etc.
Continuous Integration and Build
Automated Testing
Automation for app store deployment,beta testers& enterprise app stores
Production Monitoring and feedback to Dev
http://bitbar.com/testing/
Delivery Pipeline with Mobile DevOps
Development
Source Code
Management
SCM
Build Test Deploy Production
Dev Tools:• IDEs• JIRA• Slack• etc.
SCM:• Git/
GitHub/GitLab
• Bitbucket• MSFT• etc.
Continuous Integration and
Build
Automated Testing
Automation for app store
deployment,beta testers
& enterprise app stores
Production Monitoring and
feedback to Dev
The Scope of Mobile DevOps
http://bitbar.com/testing/
• Build and Continuous Integration must be tightly coupled when building mobile (Android and iOS) apps
• Seamless integration ’backwards’ to SCM and ’forward’ with testing on real devices
• Manual testing possible – but not recommended (due to efficiency of test automation)
Mobile DevOps: Build
http://bitbar.com/testing/
B
Continuous Integration & Build
http://bitbar.com/testing/
CONTINUOUSINTEGRATION
• Use of real mobile devices• Fully automated from Build to
Test (critical) via API• Debugging, problem fixing using
various test results (logs, videos, screenshots)
• Use of test automation
Mobile DevOps: Test
http://bitbar.com/testing/
T
Why Mobile App Tests Fail?
http://bitbar.com/testing/
The US & Global Coverage
US Market 45 Android devices
=~ 2 / 3 market
Global Market
75 Android devices=~ 1 / 4 market
~90% market coverage can be achieved
with 140 devices
~50% market coverage can be achieved
with 175 devices
http://bitbar.com/testing/
Local vs. Cloud-based Testing
• Must be fully integrated with and preferably online tool to automate tedious process of deploying new release for its targets
• Automation cuts the work effort of days to hours
• Must be seamlessly integrated with testing (test automation)
Mobile DevOps: Deploy
http://bitbar.com/testing/
D
• Monitoring mobile apps on real networks, using real devices and real back-end connection
• Synthetic monitoring vs. RUM• Must be seamlessly integrated with
testing (and can utilize various testing practices, frameworks, scripts, etc.)
Mobile DevOps: Monitoring
http://bitbar.com/testing/
M
Mobile DevOps - User ExperienceDevelopme
nt/IDE SCM
Build
Automated Testing
Deploy/ Release
“Production”
Automated buildsfor every code
change
Built binaries automatically tested
Tested applications, Screenshots for various devices
and languages, certificates, metadata ready to push to App stores
DeployTo Devices
Test apps in real world (Synthetic)
Apps Used by Users• Synthetic Monitoring• Real User Monitoring
Beta Testing (Humans)
http://bitbar.com/testing/
Test On Real Devices
Automation - Quickly and Efficiently
Use Existing Assets Now and Future Tools
Platform Designed For Agile Development
• Test real world scenarios on real, non-emulated devices with 100% certainty
• See exactly what the user experiences
• Reduce time and headache of device acquisition and management
The Key Things to Boost Mobile DevOps
• Use any automation framework or test exercisers (e.g. Testdroid App Crawler) to get instant reports and results
• Hit the ground running and get actionable results on a wide range of devices
• Run more tests across more devices in less time, with fewer resources
• Bring your own hardware, tools and frameworks and the Bitbar team will integrate them into the cloud
• Bitbar solutions will not lock you in to any tool or automation framework
• No vendor lock-in means that you have a long term and future proof platform for all of your mobile development and testing needs
• Bitbar integrates seamlessly to your development tools and CI process
• Instant results give immediate feedback to the development team
• Faster iterations mean more efficient development, faster time to market and more satisfied users
http://bitbar.com/testing/
• Standard hardware• Highly scalable
environments• Plug-and-Play of new
devices• Android & iOS • Extremely cost effective
Infrastructure
http://bitbar.com/testing/
Public Cloud
On-Premise• Device ‘Cloud’ built for internal testing use• Enterprise-grade testing infrastructure hosted
by the customer• Usually sits behind customer firewall and
connects to preproduction environments
• Private Device Cloud• Reserved and Dedicated devices• Hosted and Fully Managed by Bitbar• Devices chosen by and reserved exclusively for
customer
• Also known as Testdroid Cloud• On-demand devices (multi-tenant)• Mobile App testing on over 1000+ real
Android and iOS devices hosted by Bitbar
Private CloudBitbar TestingDeployment Options
http://bitbar.com/testing/
Demo
http://bitbar.com/testing/
THANK YOU!More information about mobile app testing,
mobile monitoring and mobile devops at http://bitbar.com