Automated Testing for iOSRobert Collins @collinsrjIBM Watson Health

Please refer to speaker notes if reading this!Xcake Dublin - 12th October 2016

Note for people reading this: These slides are my own opinions and experiences. They do not reflect those of my employer! There are no guarantees about the information provided. It may be wrong or bad. Please take your own time to understand the material presented and form your own opinions. The included speaker notes were my thoughts at time of writing these slides. They are not exactly what I said - I forgot some stuff and included other stuff.

Hi my name is Robert Collins. Im a mobile architect at IBM Watson Health.Im going to talk today about a few testing topics were working through. These are based on our recent experience. Our group is pretty new. We only actively started developing for iOS in April this year.Our app is written in Swift, but I dont think anything I refer to here, excludes the Objective-C guys.

Battery LifeLocalisationMemory UsageBandwidth ConsumptionNetwork ConditionsSecurityStorageApp UpgradesFunction!AccessibleOS VersionsSize ClassesUsabilityDevice VersionsDevice OrientationCore Data

I noted down some of the things we need to test. Realistically, we barely have time to check all of these.

Its funny, at times its easy to forget the app has to function in addition to all of these things. Users have a really high expectation for the quality of mobile apps. The app has to look good as well as function.

In one of the first weeks I attended Xcake, Karl asked where we were from and I said IBM. Karl mentioned this could be quite different to some of the usual Xcake attendees who might have quite a different customer base.

I guess my own experience having worked in both enterprise and small shops is that traditional enterprise apps can be slow and clunky. Take for example IBM Lotus Notes. Its is an app that brings me daily frustration. (I can say that now given the release of IBM Verse.)Consumers are used to interacting with a small number of likely high quality apps. By quality I mean, stable, performant, great user experience. I think the enterprise is moving there too. Initiatives like IBM Design are really changing things for the better. Theyre bringing the great consumer app experiences to the enterprise.

To deliver the great experience, we have got to test.

We test not only to make our code more performant, but also to make our code more maintainable. To ensure the coupling of our classes was correct. Writing tests is about ensuring we really have thought about all the scenarios we can encounter.

Lets automate what we can. Were programmers, one of the premises of computing is to avoid us doing repetitive tasks. We do automated testing to allow us more time to work on the hard things.

Unit TestingXCTestCase

There isnt too much say about Unit Testing. I think its something all professional developers should be doing already. If youre not, you really need to get with it. I think the reasons should be obvious in most cases.Were aiming for about 70% unit test coverage. This number was suggested as a reasonable balance from another team. It might be a little low, it might be a little high. At the moment its our goal. Well refine.Dont ignore the performance piece of unit testing. Its not something our team has been using, but I think it will help you to understand how iOS will perform. One example I have of this is around JSON parsing. Parsing might not be linear. If you only test with a couple of items coming from a test server, how will you know what happens in the real world?

Protocols definitely make decoupling your components easier. I guess this is a bit of my Java background creeping through. You should be coding to an interface.

UI TestingXCTestCase

I wont go into how to UI test. Our goal here is test that the View and Controller code weve written functions as expected.Its not going to let you test that a button is blue. Dont try to do too much here. Consider this in a sort of testing pyramid way - lots of unit tests, less UI Tests, even less integration tests and then a small amount of manual testing.

The Network

Our app connects to a REST API. We have never managed to structure our projects so that the APIs are stable and complete before that feature is required. This has led to great frustration! If the dev server is unavailable youre unable to develop or test.

We looked at some solutions to this. One I proposed which kinda worked was to mocking up the REST APIs in simple Node.JS app. This started us off interacting with a real system quickly while we were waiting for the full API to be completed. At the start, this was really useful. It got the app team writing code, appreciating some of the challenges of networks. There were lots of headaches. We ended up changing the schema quite a few times! This would typically happen in such a way to cause a major headache for the app team. The mock server would go down as well.This approach did not let us test some of the more challenging aspects of networking e.g. how to manage slow downloads, how would our app work function if a very large data set was returned. How would the app deal with the network being unavailable.

Mock the Network for Tests

The approach were currently using is to mock in the app code using OHHTTPStubs. This library proved to be really useful for unit testing. We could achieve a high coverage level.Weve been able to test lots of states we were not able to run through before. What happens if connections die, what happens if slow, what messaging is presented to users if the app times out.

Mocking can cause real problems. Late integration is a real risk in any project. If you take the approach weve taken of mocking for UI Tests, you need to look at where youre doing your integration tests!!

Reuse

Reusability - tests require as much care an maintenance as your code. This becomes quite apparent as you start to write your UI tests.

Youve got to look at ways to simplify your tests. They will require care and refactoring. A nice step weve made recently is to move all mocked network JSON responses into a folder structure.

For the UI tests, this reuse will be really important. Youll have the same steps repeated over and over. If I take an app Im working on at the moment, we have a number of common setup steps e.g. login and search for a client. Youll reuse these in many of your tests. Factor them out.

Soak TestIf your users do stuff all day, maybe you should test that.

Text

This is something we havent yet done. Its definitely on our list though.

Our expectation is that our app will be used throughout a working day. Were building apps which must stay responsive throughout the day.There is some work to be done in the setup. Id recommend looking at a separate build target for Soak Tests. You want these to run, but not all the time. There is some tweaking to be done for config etc. here. Be conscious that youre testing a release versions here and not debug. Will the app be compiled with different settings to your release?

Physical Devices

Physical devices will always be a challenge. The simulator will set unrealistic expectations on performance. There will be issues you can only experience on real devices.You cant ignore testing on physical devices, but they can really help you expand your testing. - Again, this is a confidence thing!

Xcode Server

Well worth the 20.Its not entirely reliable. Take all the issues you have with Xcode in general and now make another system out of it.I havent found it to be the most performant system in the world. Im not sure if this is added overhead or if its just my build server is too slow.Its unclear to me if I can parallelise builds effectively. Something we code do with Jenkins. I really like sticking with the Apple way. In this instance I think it will be worth the pain. Just be prepared for some :-) It has been a big driver for us to move to Git.

Monkey Test

Text

Monkey testing is something I found from Android. The goal here is to simulate the in-pocket user. Theres a handy Github project which uses the Automation instrument to simulate random user interactions - swipe, tap, pinch these combined with some predefined user interactions allow you to simulate what would happen with annoying users who just start tapping everything.In my Android testing I got the odd crash out of it. Its hard to say its a must do. If youve got some spare cycles its definitely worth playing with. More so if youve an interesting user interface with many gestures.

Balance

Having really high code coverage and full coverage of all the possible scenarios in your user interface is nice. But if youre not shipping product its no good. Theres definitely a balance to be struck here. Where the balance lies is up to you and your team. If youre a super experienced team maybe you dont need tests (or maybe youve already written them).

You automate to:give you more timegive you the confidence to change featureskeep quality highhelp understand whats going on!

Thanks! Any questions?@collinsrj

Useful links:https://github.com/jonathanpenn/ui-auto-monkeyhttps://github.com/AliSoftware/OHHTTPStubs