test butler: reliable android testing, at your service
TRANSCRIPT
![Page 1: Test Butler: Reliable Android Testing, at Your Service](https://reader034.vdocuments.us/reader034/viewer/2022052705/58e88bab1a28abf8658b5adb/html5/thumbnails/1.jpg)
Test ButlerReliable Android Testing, at Your Service
Drew Hannay (@drewhannay)
![Page 2: Test Butler: Reliable Android Testing, at Your Service](https://reader034.vdocuments.us/reader034/viewer/2022052705/58e88bab1a28abf8658b5adb/html5/thumbnails/2.jpg)
Android Testing @ LinkedIn● ~5000 instrumentation tests
● 2 runs per commit (pre-commit + post-commit)
● ~50 commits per day
● ~100 nightly runs to detect flaky tests
● 5000 * ((2 * 50) + 100) =
~1 million tests run per day
![Page 3: Test Butler: Reliable Android Testing, at Your Service](https://reader034.vdocuments.us/reader034/viewer/2022052705/58e88bab1a28abf8658b5adb/html5/thumbnails/3.jpg)
● Emulator flakiness
○ Animations can cause Espresso flakiness
○ CPU can randomly go to sleep
○ WiFi can randomly disconnect
○ Lock screen can appear and block test runs
○ Other apps can crash/ANR in the background and
block test runs
● Most of these happen < 1% of the time
○ But with 1 million tests run per day, that’s not
insignificant
● Tests shouldn’t fail due to issues with the
testing infrastructure
Testing Environment
![Page 4: Test Butler: Reliable Android Testing, at Your Service](https://reader034.vdocuments.us/reader034/viewer/2022052705/58e88bab1a28abf8658b5adb/html5/thumbnails/4.jpg)
Controlling the environment from tests● Turn WiFi on/off
● Change locales
● Set screen orientation
● Turn GPS on/off
![Page 5: Test Butler: Reliable Android Testing, at Your Service](https://reader034.vdocuments.us/reader034/viewer/2022052705/58e88bab1a28abf8658b5adb/html5/thumbnails/5.jpg)
Extra permissions● Many of these problems are fixable via reflection hacks
● Often requires special Android permissions
○ e.g. SET_ANIMATION_SCALE
● Painful at best, dangerous at worst
○ Add permissions and remember to remove before releasing (bad!)
○ Add permissions only to the debug manifest (better)
○ Either way gives your app different behavior in dev than in production (bad!)
● Some permissions are “development” permissions
○ These need to be granted with adb before they can be used
![Page 6: Test Butler: Reliable Android Testing, at Your Service](https://reader034.vdocuments.us/reader034/viewer/2022052705/58e88bab1a28abf8658b5adb/html5/thumbnails/6.jpg)
Introducing Test Butler● Two-part project: App & Library
● Library communicates with Test Butler app via RPC
● Test Butler app has permission to change
device settings
App apk Test apk
Test Butler Library
Test Butler apk Emulator
![Page 7: Test Butler: Reliable Android Testing, at Your Service](https://reader034.vdocuments.us/reader034/viewer/2022052705/58e88bab1a28abf8658b5adb/html5/thumbnails/7.jpg)
Test Butler: Permissions● Test Butler requests lots of permissions so your app doesn’t need to
● Signed with the system keystore for the AOSP emulators
● All signature-level permissions are granted automatically
○ No need to grant permissions with adb!
● This also means it can access permissions that can’t normally be used by
non-system apps
![Page 8: Test Butler: Reliable Android Testing, at Your Service](https://reader034.vdocuments.us/reader034/viewer/2022052705/58e88bab1a28abf8658b5adb/html5/thumbnails/8.jpg)
Test Butler: Custom IActivityController● Holds signature permission SET_ACTIVITY_WATCHER● Allows it to set a custom implementation of IActivityController
![Page 9: Test Butler: Reliable Android Testing, at Your Service](https://reader034.vdocuments.us/reader034/viewer/2022052705/58e88bab1a28abf8658b5adb/html5/thumbnails/9.jpg)
Test Butler: Custom IActivityController● No more system crash or ANR dialogs breaking tests!
● Caveat:
○ isUserAMonkey will always return true while using Test Butler
● Workaround:
○ Write a custom isUserAMonkey method that returns false
while running instrumentation tests
![Page 10: Test Butler: Reliable Android Testing, at Your Service](https://reader034.vdocuments.us/reader034/viewer/2022052705/58e88bab1a28abf8658b5adb/html5/thumbnails/10.jpg)
How do I use it?
![Page 11: Test Butler: Reliable Android Testing, at Your Service](https://reader034.vdocuments.us/reader034/viewer/2022052705/58e88bab1a28abf8658b5adb/html5/thumbnails/11.jpg)
How do I use it?
![Page 12: Test Butler: Reliable Android Testing, at Your Service](https://reader034.vdocuments.us/reader034/viewer/2022052705/58e88bab1a28abf8658b5adb/html5/thumbnails/12.jpg)
What’s next?● Add sample scripts to download & install the Test Butler app on emulators
● Add additional emulator controls (e.g. toggle airplane mode)
● Stabilize the rest of the Android test pipeline (stay tuned!)
![Page 13: Test Butler: Reliable Android Testing, at Your Service](https://reader034.vdocuments.us/reader034/viewer/2022052705/58e88bab1a28abf8658b5adb/html5/thumbnails/13.jpg)
Questions
![Page 14: Test Butler: Reliable Android Testing, at Your Service](https://reader034.vdocuments.us/reader034/viewer/2022052705/58e88bab1a28abf8658b5adb/html5/thumbnails/14.jpg)
References● Test Butler Github Repo
● Test Butler Announcement Blog
● Going Green: Cleaning up the Toxic Mobile Environment