best practices in mobile ci (webinar)
TRANSCRIPT
Best Practices for Mobile CI
April 28, 2015
© Sauce Labs, Inc.
Presenters
Kevin Rohling
Co-founder / VP of Product at Emberlight
Founder/CEO of cisimple now [email protected]
Kristian Meier
Sales Engineer at Sauce Labs
WHAT IS CONTINUOUS INTEGRATION FOR MOBILE?
AUTOMATICALLY…
Build Test Deploy
EVERY COMMIT.
CONTINUOUS INTEGRATION
HOW DOES IT WORK
1. Developer commits code
CONTINUOUS INTEGRATION
HOW DOES IT WORK
2. CI Server picks up the change.
CONTINUOUS INTEGRATION
HOW DOES IT WORK
3. Code is automatically compiled
CONTINUOUS INTEGRATION
HOW DOES IT WORK4. Tests are run
CONTINUOUS INTEGRATION
HOW DOES IT WORK
5. Application is deployed
CONTINUOUS INTEGRATION
WHY?
• Reduce the need for human-based testing
• Faster Feedback
• Automate EVERYTHING
WHY DO I USE CI?
• No Touch Configuration
WHY DO I USE CI?
• No Touch Configuration
• Automated OTA Distribution
WHY DO I USE CI?
• No Touch Configuration
• Automated OTA Distribution
• Code Validation: Automated Builds and Tests
ONE SIZE DOES NOT FIT ALL
Business Drivers Engineering Challenges
Early StartupInitial Traction,
FundingStability, UX
Consulting FirmClient Status Updates,
DeadlinesAd Hoc Distribution, Unit Testing
Large Product TeamRetention, Growth,
RevenueAutomation, Communication
PR
OC
ES
S
MOBILE IS HARDER
• Infrastructure
MOBILE IS HARDER
• Infrastructure
• Compilation/Code Signing
MOBILE IS HARDER
• Infrastructure
• Compilation/Code Signing
• Testing
MOBILE IS HARDER
• Infrastructure
• Compilation/Code Signing
• Testing
• Deployment
SIMULATORS/EMULATORS vs DEVICES
SIMULATORS:
• Used by iOS, included w/ Xcode
• Execute i386 instruction set, not ARM
• Very fast compared to Emulators
• Do not have access to certain hardware functions
• (GPS, Bluetooth Radio, Accelerometers, etc)
SIMULATORS/EMULATORS vs DEVICES
EMULATORS:
• Used by Android, included w/ Android SDKs
• Execute ARM (or native device instruction set)
• Very slow compared to Simulators
• Do not have access to certain hardware functions
• (GPS, Bluetooth Radio, Accelerometers, etc)
SIMULATORS/EMULATORS vs DEVICES
DEVICES:
• Reproduces the actual performance experienced by your users
• The only way to catch manufacturer/OEM bugs
• Very expensive to configure and maintain compared
• Full access to hardware functions:
• (GPS, Bluetooth Radio, Accelerometers, etc)
WHEN TO USE DEVICES vs SIMULATORS/EMULATORS?
• Simulators and Emulators are an excellent solution for running automated tests
during development. They are inexpensive and will reliably catch most problems.
• Physical Devices can be used on a lower frequency (i.e. pre-release, weekly,
daily, etc.). They are the only way to catch performance problems, test certain
hardware features and find OEM issues. In the least, devices should be used
before every release.
UNIT TESTING
According to a recent survey:
-77% of developers said app quality is “very important or mission critical”
-80% of mobile developers test their apps manually
-Only 13% of mobile developers use automated testing
UNIT vs FUNCTIONAL TESTING
• Unit Testing - testing small pieces of code
• Functional Testing - testing button clicks and UI interaction
UNIT/FUNCTIONAL TESTING
BENEFITS:
• Instant Gratification (GREEN is good!)
• Repeatable
• Can automatically test every commit
UNIT/FUNCTIONAL TESTING
CHALLENGES:
• Unit Tests are not users
• LOTs of work to write and maintain them
• Which framework to use?
• What language/framework do your developers know?
• Open Source/Community Support
• 3rd Party Framework requirements
UNIT/FUNCTIONAL TESTING FRAMEWORKS
WHICH ONE TO USE?
CI TOOLS/SERVICES FOR MOBILE
Good: Open Source, Lots of plugins
Bad: Self-Hosted, DIY solution
Jenkins
CI TOOLS/SERVICES FOR MOBILE
Good: Hosted solution, OS X support, Lots of plugins
Bad: Tedious setup process
Travis CI
CI TOOLS/SERVICES FOR MOBILE
Good: Hosted solution, Designed specifically for mobile, Easy setup
Bad: Less flexible than other solutions
Ship.io
CI TOOLS/SERVICES FOR MOBILE
Good: Integrated w/ Xcode, Apple-Supported
Bad: Self-Hosted, iOS Only
Xcode CI
Using the Sauce Labs Cloud with CI
Sauce is the ideal Automated Testing Platform for CI
© Sauce Labs, Inc.
Appium
• “Selenium for mobile apps” - uses the
WebDriver spec to simulate user actions on
iOS and Android
• Leverage your existing Selenium test
architecture to help speed apps to market
• Maximize test code reuse across platforms
makes developers more productive
• Development led by Sauce Labs
Appium is an open
source framework
that automates
testing of native,
hybrid and mobile
web apps.
© Sauce Labs, Inc.
The Sauce Labs Platform
• Point your existing test
scripts to Sauce Labs
• We spin up a new,
pristine VM every time
• Run tests in parallel to
reduce test time
• Capture videos, logs and
test metadata
• Choose from over 500
browsers / OS
combinations
Demo
© Sauce Labs, Inc.
Learn more about Sauce Labs
• Sign up for a free trial of Sauce
Labshttps://saucelabs.com/signup/trial
• Watch a video
https://www.youtube.com/user/saucelabs