don't fear our new robot overlords – a new way to test on mobile
DESCRIPTION
A talk I gave at europython 2014 about testing mobile web sites and apps on real devices using CV and a robotTRANSCRIPT
![Page 1: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/1.jpg)
Don’t fear our new Robot Overlords!A new way to test on mobile devices
!
!
Philip Brechler • europython 2014
![Page 2: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/2.jpg)
Agenda
![Page 3: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/3.jpg)
Agenda
• Problems when testing on mobile devices
![Page 4: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/4.jpg)
Agenda
• Problems when testing on mobile devices
• Idea
![Page 5: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/5.jpg)
Agenda
• Problems when testing on mobile devices
• Idea
• Concept
![Page 6: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/6.jpg)
Agenda
• Problems when testing on mobile devices
• Idea
• Concept
• Demo
![Page 7: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/7.jpg)
Agenda
• Problems when testing on mobile devices
• Idea
• Concept
• Demo
• Conclusion and Learnings
![Page 8: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/8.jpg)
Problems in general
CC-BY-SA: Javier Enjuto
![Page 9: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/9.jpg)
Problems in general• You need to check how the app
looks
CC-BY-SA: Javier Enjuto
![Page 10: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/10.jpg)
Problems in general• You need to check how the app
looks
• You can check coordinates, fonts etc.
CC-BY-SA: Javier Enjuto
![Page 11: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/11.jpg)
Problems in general• You need to check how the app
looks
• You can check coordinates, fonts etc.
• In the end you never know if it looks right
CC-BY-SA: Javier Enjuto
![Page 12: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/12.jpg)
Problems in general• You need to check how the app
looks
• You can check coordinates, fonts etc.
• In the end you never know if it looks right
• The human need to double check everything
CC-BY-SA: Javier Enjuto
![Page 13: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/13.jpg)
Problems with simulators
CC-BY: Steve Jurvetson
![Page 14: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/14.jpg)
Problems with simulators• To much computation power
CC-BY: Steve Jurvetson
![Page 15: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/15.jpg)
Problems with simulators• To much computation power
• No memory issues
CC-BY: Steve Jurvetson
![Page 16: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/16.jpg)
Problems with simulators• To much computation power
• No memory issues
• No touch interface, mouse pointers are to exact
CC-BY: Steve Jurvetson
![Page 17: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/17.jpg)
Problems with simulators• To much computation power
• No memory issues
• No touch interface, mouse pointers are to exact
• Different browser engines
CC-BY: Steve Jurvetson
![Page 18: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/18.jpg)
Problems with simulators• To much computation power
• No memory issues
• No touch interface, mouse pointers are to exact
• Different browser engines
• Some APIs not available
CC-BY: Steve Jurvetson
![Page 19: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/19.jpg)
Problems with emulators
CC-BY-SA: Marshall Astor
![Page 20: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/20.jpg)
Problems with emulators• Incredibly slow
CC-BY-SA: Marshall Astor
![Page 21: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/21.jpg)
Problems with emulators• Incredibly slow
• Hard to interact with
CC-BY-SA: Marshall Astor
![Page 22: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/22.jpg)
Problems with emulators• Incredibly slow
• Hard to interact with
• Android: Every manufacturer makes their own interface and browser
CC-BY-SA: Marshall Astor
![Page 23: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/23.jpg)
Problems with emulators• Incredibly slow
• Hard to interact with
• Android: Every manufacturer makes their own interface and browser
• Android: No Google Play Services
CC-BY-SA: Marshall Astor
![Page 24: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/24.jpg)
Best solution: Testing on real devices with real
inputs and a checking eye
![Page 25: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/25.jpg)
CC-BY: yeowatzup
![Page 26: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/26.jpg)
Idea
![Page 27: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/27.jpg)
Idea
• Saw a talk about Sikuli for automated testing of websites
![Page 28: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/28.jpg)
Idea
• Saw a talk about Sikuli for automated testing of websites
• Idea: Sikuli for mobile device testing but with a CI server
![Page 29: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/29.jpg)
![Page 30: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/30.jpg)
Sikuli
![Page 31: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/31.jpg)
Sikuli• Sikuli is a tool for automating
computer tasks with screenshots
![Page 32: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/32.jpg)
Sikuli• Sikuli is a tool for automating
computer tasks with screenshots
• A very python like script language
![Page 33: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/33.jpg)
Sikuli• Sikuli is a tool for automating
computer tasks with screenshots
• A very python like script language
• Invented at MIT, now developed at Sikuli Labs (University of Colorado)
![Page 34: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/34.jpg)
Sikuli• Sikuli is a tool for automating
computer tasks with screenshots
• A very python like script language
• Invented at MIT, now developed at Sikuli Labs (University of Colorado)
• Can be used for CI testing with Robot Framework or similar
![Page 35: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/35.jpg)
Idea
• Saw a talk about Sikuli for automated testing of websites
• Idea: Sikuli for mobile device testing but with a CI server
![Page 36: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/36.jpg)
Idea
• Saw a talk about Sikuli for automated testing of websites
• Idea: Sikuli for mobile device testing but with a CI server
• In short: Make a screenshot, use CV for detecting icon, tap icon
![Page 37: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/37.jpg)
Idea
• Saw a talk about Sikuli for automated testing of websites
• Idea: Sikuli for mobile device testing but with a CI server
• In short: Make a screenshot, use CV for detecting icon, tap icon
• With a robot
![Page 38: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/38.jpg)
Making a screenshot
![Page 39: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/39.jpg)
Making a screenshot
• Android: Use adb command and some bash magic
![Page 40: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/40.jpg)
Making a screenshot
• Android: Use adb command and some bash magic
• iOS: Making screenshots of devices is possible using Xcode but not with the terminal
![Page 41: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/41.jpg)
Making a screenshot
• Android: Use adb command and some bash magic
• iOS: Making screenshots of devices is possible using Xcode but not with the terminal
• But there is an open source project that reverse engineered some of iTunes’s and Xcode’s magic called libimobiledevice
![Page 42: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/42.jpg)
Detecting an icon
![Page 43: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/43.jpg)
Detecting an icon
• Use OpenCV to do a template recognition (finding an image in an image)
![Page 44: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/44.jpg)
Detecting an icon
• Use OpenCV to do a template recognition (finding an image in an image)
• OpenCV is written in C/C++ but has great python wrappers
![Page 45: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/45.jpg)
Detecting an icon
• Use OpenCV to do a template recognition (finding an image in an image)
• OpenCV is written in C/C++ but has great python wrappers
• Quite simple API for such a complex task
![Page 46: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/46.jpg)
Detecting an icon
• Use OpenCV to do a template recognition (finding an image in an image)
• OpenCV is written in C/C++ but has great python wrappers
• Quite simple API for such a complex task
• Get the coordinate in the screenshot of your searched icon
![Page 47: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/47.jpg)
The robot
![Page 48: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/48.jpg)
The robot• Tapsterbot
![Page 49: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/49.jpg)
The robot• Tapsterbot
• Open Source delta robot
![Page 50: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/50.jpg)
The robot• Tapsterbot
• Open Source delta robot
• Made with Arduino, 3 Servos and 3D printed parts
![Page 51: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/51.jpg)
The robot• Tapsterbot
• Open Source delta robot
• Made with Arduino, 3 Servos and 3D printed parts
• Costs: ~250€ (With printing)
![Page 52: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/52.jpg)
The robot• Tapsterbot
• Open Source delta robot
• Made with Arduino, 3 Servos and 3D printed parts
• Costs: ~250€ (With printing)
• Driver written in node.js
![Page 53: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/53.jpg)
Talking to a robot
![Page 54: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/54.jpg)
Talking to a robot
• The robot is connected via WebSocket
![Page 55: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/55.jpg)
Talking to a robot
• The robot is connected via WebSocket
• The robot is driven by a node.js „app“
![Page 56: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/56.jpg)
Talking to a robot
• The robot is connected via WebSocket
• The robot is driven by a node.js „app“
• Theoretically the robot could be exchanged quite easily
![Page 57: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/57.jpg)
Talking to a robot
• The robot is connected via WebSocket
• The robot is driven by a node.js „app“
• Theoretically the robot could be exchanged quite easily
• More robots?
![Page 58: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/58.jpg)
Tightening it all together
![Page 59: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/59.jpg)
Tightening it all together• Project Goldeneye
![Page 60: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/60.jpg)
Tightening it all together• Project Goldeneye
• To test an app or a mobile website just write a standard python unittest test
![Page 61: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/61.jpg)
Tightening it all together• Project Goldeneye
• To test an app or a mobile website just write a standard python unittest test
• All the heavy lifting is abstracted away from you
![Page 62: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/62.jpg)
Tightening it all together• Project Goldeneye
• To test an app or a mobile website just write a standard python unittest test
• All the heavy lifting is abstracted away from you
• Just tell the test to tap and to check
![Page 63: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/63.jpg)
Tightening it all together• Project Goldeneye
• To test an app or a mobile website just write a standard python unittest test
• All the heavy lifting is abstracted away from you
• Just tell the test to tap and to check
• Writing such tests is quite easy, even your QA can do this
![Page 64: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/64.jpg)
Demo
![Page 65: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/65.jpg)
Conclusion
![Page 66: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/66.jpg)
Conclusion
• Goldeneye could be useful for testing mobile apps and especially websites in a CI system
![Page 67: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/67.jpg)
Conclusion
• Goldeneye could be useful for testing mobile apps and especially websites in a CI system
• Testing of existing projects, especially large web projects could be easier
![Page 68: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/68.jpg)
Conclusion
• Goldeneye could be useful for testing mobile apps and especially websites in a CI system
• Testing of existing projects, especially large web projects could be easier
• Robots never get tired
![Page 69: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/69.jpg)
Conclusion
• Goldeneye could be useful for testing mobile apps and especially websites in a CI system
• Testing of existing projects, especially large web projects could be easier
• Robots never get tired
• The robot’s hardware is fine, the software isn’t
![Page 70: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/70.jpg)
Learnings
![Page 71: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/71.jpg)
Learnings
• A function in a function in a function isn’t normal, but in node.js it is
![Page 72: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/72.jpg)
Learnings
• A function in a function in a function isn’t normal, but in node.js it is
• Never trust a github project with one maintainer
![Page 73: Don't fear our new robot overlords – A new way to test on mobile](https://reader033.vdocuments.us/reader033/viewer/2022051611/54a02199ac79594b4c8b4a7b/html5/thumbnails/73.jpg)
Learnings
• A function in a function in a function isn’t normal, but in node.js it is
• Never trust a github project with one maintainer
• OpenCV is fun and not that hard to learn