PowerPoint Presentation

1

Prepared for:UXPA 2016Seattle, Washington

Tips & Tools: Remote Mobile Testing2Dana Douglas Tristan Wilson

User Experience specialists at UserWorks, a UX consulting firm in the DC area.

A little bit of background on how we got here.

This all started for us about a year ago.

Just started working on an in-person usability study of a mobile app related to diabetes management.

We wanted to be able to capture video of our participants mobile screens as they used them, both for archival purposes, and so that notetakers/observers could view the interactions live without being in the same room.

Unfortunately, the traditional camera-on-a-sled method of mobile capture wasnt ideal for us. We wanted users to be able to hold and rotate their devices naturally.

So we started looking into our options, and eventually came across an Android app called Mobizen. It allowed us to mirror a mobile devices screen wirelessly onto a computer in real time. That seemed like a perfect solution for our needs, so we gave it a shot.

For that project, we were just using Mobizen to mirror mobile devices locally. And it worked well for us.

But as we continued to work with it, we found that the tool also had some internet streaming functionality built in.

At first we were thinking, hm, maybe we could use this to allow clients to the view our sessions remotely. And that was true, but it was really just the tip of the iceburg.

Turns out, we had just stumbled on to something that might let us run mobile usability studies with remote participants as well.2

What We Set Out To DoTips & Tools: Remote Mobile Testing3Find a better way to conduct remote mobile user testsUnderstand the tools necessary to accomplish thisCompare available options, zero in on the bestTry new methods out in the real world

TWSo that set us on a journey which eventually led to us being here.

We wanted to figure out whether we could use new technologies to conduct better remote mobile research.

We wanted to get a better understanding of the tools that were out there..

.. And compare them to each other to find out what was best.

Our final goal was to get to a point at which we were comfortable enough with our new tools that we could put them to use in a real live study.3

What We Mean When We Say...Tips & Tools: Remote Mobile Testing4Remote TestingParticipant and researcher in different geographical locationsSynchronous research, live-moderated

Mobile Modern, portable smart devices Most commonly smartphones and tablets

TWBefore we get into it, we just wanted to clarify a couple of terms that well be using throughout the presentation.

As our title suggests, this talk is all about Tips and Tools for Remote Mobile Testing.

When we say Remote Testing, what were talking about is moderated user testing in which the participant and researcher are located in two different places.

When we say Mobile, were basically just talking about modern portable smart devices phones and tablets.4

Remote TestingA Bit of BackgroundTips & Tools: Remote Mobile Testing5

DDMost of you are probably very familiar with Remote Usability Testing, so you know there are many benefits, but also some drawbacks.

5

Benefits of Remote TestingTips & Tools: Remote Mobile Testing6More naturalistic environmentLower cost & increased efficiencyGreater convenienceExpanded recruitment capability

Enhanced Research Validity+ Participants can use their own device and their own setup+ More naturalistic environment and real-world use case

Lower Cost & Increased Efficiency+ Fewer travel-related expenses for researchers and participants+ Decreased need for fancy labs and/or equipment

Greater Convenience+ Ability to conduct global research from one location+ No participant travel to and from the lab

Expanded Recruitment Capability+ Increased access to diverse participant sample+ The money saved from reduced travel costs may allow for more participants to be included in the study6

Drawbacks of Remote TestingTips & Tools: Remote Mobile Testing7Potential reduction in quality of dataExpanded spectrum of technical issuesReduced scope of researchDiminished participant-researcher interaction

Potential Reduction in Quality of Data- Difficult to control testing environment, and particularly distractions in the participants environment

Expanded Spectrum of Technical Issues- More reliance on quality of internet connection- Dealing with a variety of hardware

Reduced Scope of Research- Typically limited to software testing - Shorter recommended session duration

Diminished Participant-Researcher Interaction- Restricted view of participant body language- Difficult to establish rapport

7

Drawbacks Specific to MobileTips & Tools: Remote Mobile Testing8Range of OS versions across devicesLack of mobile interaction tracking/capturing Touch, gestures, etc.Participant currently has to share their own screen Creates the potential for privacy concernsWireless nature leads to reduced speed and reliability when transferring data

When it comes to remotely testing mobile interactions, there are a few additional drawbacks to consider.

First, youre often dealing with a wider range of operating systems among the various mobile devices on the market.

Also, while on desktop, you can pretty easily track the participants mouse movements, that is not really possible with mobile devices. So it can be more difficult (or impossible) to see where participants are tapping or hovering when interacting with a mobile device.

With current offerings (which well get into more later), participants must share their own screen, rather than control the researchers screen. This has the potential to feel invasive on the part of the participant.

Finally, when relying on wifi, as mobile devices do, you are at the mercy of the bandwidth of the wireless network to transfer the data between participant and researcher. As many of us know, wifi can often be slower and less reliable than hard-wired networks.

8

Existing MethodsRemote Mobile Strategies of TodayTips & Tools: Remote Mobile Testing9

TWOne thing that we think is important to note, is that there ARE already a few ways of conducting remote mobile tests. They arent the most elegant solutions in the world, but you can make it work.9

The Browser ResizeTips & Tools: Remote Mobile Testing10Connect to researchers computer remotelyInteract with browser sized to mobile dimensionsUse laptop/desktop PC, not mobile device

Researcher resizes browser on their own computer to match dimensions/pixel density mobile deviceTesting conducted like traditional remote session on PC, but mimicking mobileParticipant interacts with the mobile-sized browser remotely with mouse/keyboard

One of these methods is what we call, The Browser Resize

The setup for this is essentially the same as a traditional desktop remote session. The researcher opens up a web page on their computer, and shares their screen with a remotely located participant. The difference here, is that while the participant is still interacting with the web page via their PC, the researcher has resized the window of the browser to mimic the dimensions of a mobile device.

The browser resize is great for situations in which youre just trying to test layout, or IA Its not so great if youre concerned about peoples interactions with your product on a real mobile device.10

The Standalone WebcamTips & Tools: Remote Mobile Testing11Participant uses an external webcam to take video of themself as they interact with mobile deviceVideo feed is live-streamed to researchers located remotelyNot an accurate depictionPosition external webcam to capture deviceStream interactions to remote researchers

11

The Laptop HugTips & Tools: Remote Mobile Testing12

Wrap arms around reverse-facing laptopInteract with device in view of laptop webcamLike standalone webcam method, participants stream a video feed of their mobile interactionsInstead of a separate webcam, participants use their laptop (or tablet) to take videoHinged laptop facilitates camera angle; easier on participant

The laptop can be used concurrently for communication with researchers and/or study material display

12

New OptionsEmerging Remote Mobile StrategiesTips & Tools: Remote Mobile Testing13

TW

While all of those other methods work, there are new methods emerging the are higher quality and higher fidelity and are easier for the participant and for the moderator. 13

Recipe for Remote MobileTips & Tools: Remote Mobile Testing14

Two major pieces of the remote mobile puzzle - Mirroring - StreamingMirroring is relatively commonplace in itself, but tricky when it comes to live streaming that feed over the internet

A majority of the remote mobile setups we found can be differentiated by the ways in which these two ingredients are combined.14

Mobile MirroringTips & Tools: Remote Mobile Testing15

Commonly via native wireless casting protocols: Google Cast(Android) AirPlay(iOS)

15

Mobile StreamingTips & Tools: Remote Mobile Testing16

Directly streaming from the mobile device itself- or -Re-capturing & streaming a locally-mirrored screen using a PC

16

Configuration A17Example Tools: Mirroring 360

DDThere are different combinations of these two ingredients (mirroring and streaming) that will accomplish what we need for remote mobile testing. These combinations or configurations vary in complexity.

The first, and most complicated, configuration includes two apps, or tools. First, the participant installs one tool on both their mobile device and computer. This enables them to mirror their mobile screen onto their PC. Then, both the participant and the researcher install one web conferencing tool on each of their PCs. This web conference tool then shares the participants PC screen which is mirroring their mobile device (equivalent of sharing your screen while playing a YouTube video). An example of a tool we found that would require this configuration is called Mirroring 360.

17

Configuration B18Example Tools: Reflector 2, Air Server, X-Mirage.

TW18

Configuration C19Example Tools: Zoom

DDIn this third configuration, it is simpler in that both the participant and the researcher install one web conferencing tool on each of their PCs. But that tool also works with the native screen mirroring technology on the participants mobile device (e.g., AirPlay), meaning the participant does not need to install an app on their phone. In addition, the web conferencing capability of this tool enables the researcher to see the participants mirrored mobile screen shared from the participants PC. An example of a tool that would require this configuration is Zoom.19

Configuration D20

Example Tools: Join.me, Mobizen, TeamViewer, GoToAssist

TW

Potentially the least complicated configuration that we found.

Notably, only on Android, as far as weve found. 20

Remote Connectivity Configurations21

PARTICIPANTRESEARCHER

TW21

The Tool of Our DreamsCharacteristics of an Ideal SolutionTips & Tools: Remote Mobile Testing22

DDAs we uncovered tool after tool that we thought had the potential to work in the remote mobile testing context,we quickly realized that we needed to come up with a way to assess the different tool options. So, we came up with a set of criteria that we considered the ideal characteristics of a tool that could serve the purpose of testing a mobile device remotely.

22

Characteristics of the Ideal ToolTips & Tools: Remote Mobile Testing23Ease (Participants and Moderators)Painless for participants to set up and use; not intimidatingQuick and simple to initiate screen sharing from both moderator and participant perspectiveAllows for remote mobile mirroring without a local computer as an intermediaryEasy for moderator to implement and controlCostThe cheaper the better if all else is comparable

First, the tool should be easy to use, both for the participant and for us, the researchers. This means easy to install and set up, as well as easy to use during the session. It also means that ideally the participant doesnt have to worry about setting up multiple devices, such a PC in addition to their mobile device, as you saw in a few of the configurations.

Secondly, of course, our ideal tool would be inexpensive. Knowing that technology changes rapidly, we knew we wouldnt want to invest a lot of money into a tool that would become outdated quickly.

23

Characteristics of the Ideal ToolTips & Tools: Remote Mobile Testing24PerformanceMinimal lag timeCan run alongside other applications without impacting experience/performanceAccurate representation of participants' actions and gesturesFeaturesAbility to carry out other vital aspects of research in addition to screen sharing (e.g., web conferencing, in-app communication, recording)Platform agnostic: fully functional on all major mobile platforms, particularly iOS and Android. Allows participants to make calls while mirroring screen

Third, we needed the tool to perform well. This includes minimal lag time between the participants interactions and what we saw on our end. We also wanted the tool to play nice with other applications that needed to be running on the device at the same time. And, if possible, we wanted to be able to see a clear representation of the participants interactions with the device (like taps and gestures) similar to mouse tracking on a desktop.

Fourth, we wanted the tool to be a one-stop shop in that it included things like web conferencing, chat, and recording directly within the tool. Also, we wanted it to at least work on both iOS and Android, but also other mobile platforms as well. And to avoid participants needing a landline or separate device or computer to communicate with us, we wanted the app to allow them to talk to us on the phone while also mirroring their screen.

24

Characteristics of the Ideal ToolTips & Tools: Remote Mobile Testing25PrivacyAllow participants to remotely control researchers mobile device via their ownOR, if participant must share their own screen:Considers participant privacy by clearly warning when mirroring beginsProvides participant with complete control over start and stop of screen sharingSnoozes device notifications while sharing screenShares only one application rather than mirroring the whole screen

Finally, we wanted to protect our participants privacy as much as possible and put them at ease. Therefore, the ideal tool would allow the participant to remotely control our mobile device from theirs. They would have to be able to control it from their mobile device (not their desktop) in order for the interaction to be as natural as possible.

But we knew that may not be an option, so we figured that if the participant had to share their own screen, we wanted to see a few features that would protect their privacy while they were sharing their screen.

First, the tool would clearly indicate to them when mirroring has begun. They should also have complete control over when the sharing would start and could stop it at any time. We thought it would be nice if the tool would snooze the participants notifications while the screen was being shared. (Technically, this is possible, but many participants do not know how to do it. Also, Do Not Disturb will silence the notifications, but they still appear on the screen and would therefore be mirrored to the researcher.)

Finally, as you can do with many desktop screen sharing tools, we thought it would be nice if the mobile tool would have the option to share only one application. Therefore, if the participant accidentally pulled up their messaging or photo app, it would not be shared with the researcher. 25

On the MarketTools Available NowTips & Tools: Remote Mobile Testing26

TW

So, over the last year, these are the options we found.

We just want to note that we are not affiliated with any of these tools, nor are we endorsing any of them. We evaluated them for our own purposes and simply wanted to share our findings with you. 26

Number of Viable Tools Found Per Platform27Tips & Tools: Remote Mobile Testing

An overview of the available tools across platforms.

Choices for Android are much more diverse.

iOS is more locked down so fewer developers are coming out with mirroring tools.

27

Mirroring Tools for iOS and Android28ZoomAirServerX-MirageLonelyScreenQuicktime 7Lookback

** Still in development for iOSJoin.meMobizenTeamviewerVNCGoToAssistVysorScreen Stream MirrorRemote CallMirrativ**YouTube Gaming**

Heres a current list of tools. This is changing quickly though, so tomorrow may be different.

Tools with ** are advertising that they are developing iOS versions.

28

Cross Compatible Tools - iOS & Android29Reflector 2Mirroring 360MirrorOp** Requires Jailbroken or Rooted Device

Only found three that claim to work on both Android and iOS. But spoiler alert, they dont work that well.

29

Windows MobileTips & Tools: Remote Mobile Testing30

Project My Screen

Tool RatingsCross ComparisonTips & Tools: Remote Mobile Testing31

DDNow that we had found a number of tools with potential to meet our needs, we wanted to evaluate and compare them in order to determine which tool or tools came closest to ideal.

31

Cream of the CropTips & Tools: Remote Mobile Testing32We tested and compared each of the tools we found

Six of them stood out as best-in-class: Mirroring360Reflector 2MobizenTeamViewerJoin.meZoom

Android iOS

Although there appear to be a number of tools available, we found that many of them had certain characteristics that made them either impractical or unsuitable for remote mobile research. For example, at least one required you to access the code of the app for it to work. Others required the participants phone to be jailbroken or rooted. And others had such severe privacy concerns that we didnt even consider asking a participant to use them.

After eliminating the obvious dealbreakers, we chose six that we considered best in class.

32

Our Subjective RatingsTips & Tools: Remote Mobile Testing33Next, we subjectively rated our top six tools across 5 of our ideal tool characteristics: Participant Ease of UseModerator ConvenienceAffordabilityPerformance & ReliabilityRange of Features

*None of the tools demonstrated sufficiently different privacy features and therefore privacy was not rated.

For those six tools, we conducted more in-depth in-house trial runs and then subjectively rated each tool across five categories so we could more easily compare and contrast their strengths and weaknesses. The five categories encompassed our ideal characteristics and included:

The rating scale was from 1 to 10, where 1 was the least favorable and 10 was the most favorable. Again, these were all our subjective ratings and were based on what we believed our needs would be for a remote mobile test.

33

Our Subjective Ratings34

TWRequires an app on the phone, and two apps on the computer.

Explain the graphs.

34

Our Subjective Ratings35

DDPrice is on $14.99 for a license. It works on both iOS and Android, but doesnt have web conferencing included so participants have to install a separate web conferencing app. It does offer an option to stream directly to YouTube, but that was not exactly useful to us for remote research.

35

Our Subjective Ratings36

TW

Free, so clearly affordability is the highest rating. Only works on Android. Only relevant feature for remote testing is screen sharing. Works over the internet, doesnt require participant to have computer. But it does require the participant to create an account. Its the tool that got us started. 36

Our Subjective Ratings37

DDThe strengths of TeamViewer were definitely in the ease of use for participants and moderators. The participant simply needs to open the app on their phone and read a 9 digit number to the researcher who can enter that into a desktop app and be automatically connected to the participants phone. The screen sharing currently only works on Android though. The main downfall of TeamViewer is the price: $809 for a one year business license (or free for personal use).

37

Our Subjective Ratings38

TW

No need for participant computer. Included web conferencing, chat, desktop sharing, and recording. Only works on Android. $20-$30/month on a Pro plan. 38

Our Subjective Ratings39

DDGenerally well rated overall. It is $14.99/month for the Pro license which was fairly middle of the road compared with the other tools. Includes web conferencing, chat, recording, and desktop and web cam sharing in addition to the mobile screen sharing. However, the mobile screen sharing currently only works on iOS.

39

Aggregate Tool Ratings40Tips & Tools: Remote Mobile Testing

DD

As you can see, Join.Me and Zoom were rated the best overall for our needs, but as you can see they werent necessarily significantly better than all of the others. All of the tools had their own strengths and may work better in different research scenarios. Team Viewer had notable strengths in ease of use and Mobizen, which is free, clearly had a strength in affordability. 40

Case Study

Tips & Tools: Remote Mobile Testing41

TW

After all of this in-house testing, it was finally time to put them to the test in the real world. 41

A Bit of BackgroundTips & Tools: Remote Mobile Testing42The Project: Redesign of a federal government websiteRemote usability test w/ nationwide sample of participantsTesting clickable wireframe for both desktop and mobileRecruited 7 participants for mobile group ( + 8 for desktop)

The Plan: Decided to use two tools: Zoom (iOS) Join.me (Android)

TW

Why we picked Zoom and Android because they allowed us to remote mobile screen sharing and also share a desktop. 42

The PrepTips & Tools: Remote Mobile Testing43Spent 5-10 minutes with each mobile participant in the days prior to data collection getting them set up with the tools and testing the mirroring and streaming technology. Required extra up-front time, but it: Was easier to walk participants through the process than send a long list of written instructions. Allowed for real time troubleshooting. Put the participants at ease.

TW

We wanted to make sure everything would go smoothly since this was our first time using these tools in a real study.

In a tightly packed schedule, it reduced the possibility of us running over time.

DD

So we wanted to show you what this setup looked like from the participants perspective and how easy it can be when everything works properly. First step: Participant clicks on join.me link in email.

43

44Tips & Tools: Remote Mobile Testing

Join.me Setup

DDJoin.me setup: Participant clicks on join.me link in email. Join.me app opens or downloads on their phone and joins the meeting. Moderator passes presenter role to participant. Participant accepts presenter role. Participant taps on icon to share screen and reads and accepts the privacy warnings. Screen is shared. 44

Zoom Setup45Tips & Tools: Remote Mobile Testing

TW

Zoom setupParticipant confirms their computer and phone are on same network. Participant selects option on Zoom desktop app to share iPhone screen. Participant swipes up on their iPhone to reveal AirPlay option. Participant selects AirPlay, then selects the computer (Zoom-Tristan).Participant turns mirroring toggle on. Screen is mirrored to participant computer and streamed to moderator computer.45

Bumps in the RoadTips & Tools: Remote Mobile Testing46Zoom participants needed to use a landline or their computer audio.Zoom participants needed to be on the same network on their phone and computer; we found this did not work on some enterprise networks.Sometimes the AirPlay option would not appear until the wifi was turned on and off a couple times. Join.me users have an icon that remains on the screen during mirroring.

DDThis project didnt succeed without some bumps along the way. But, due to the upfront work we did, we were able to address these issues and deal with them before the data collection even started.

46

More Bumps in the RoadTips & Tools: Remote Mobile Testing47For participants to see the tasks written out, we had to send them a PDF that they could have open on their computer screen.We also had to send the link to the prototype so participants could open it in their phones browser.Ensure consent form obtains permission to view and record participants phone screen. If a participant received a text while they were mirroring their screen, we were able to see the notification.

DDSince participants would be sharing their own screen instead of controlling ours, we had to adjust some of our tried and true methods to work within this new context.

47

It Worked!48

The lab setup for our remote mobile case study: Dana speaks with a participant via telephone. The users mobile display is streamed to our location live from California.The left computer displays the same view and shares it with observers.

TW48

The Proof: Join.meTips & Tools: Remote Mobile Testing49

TW

Here we have a brief clip from one of our Android sessions showing how smooth the transition was when we asked participants to begin sharing their screen. What youll see in the video is the moderators perspective. You will hear Dana passing the presenter role to the participants phone which they then accept and their screen with the wireframe we were testing will be shared via the Join.me meeting. 49

The Proof: ZoomTips & Tools: Remote Mobile Testing50

DD

I know that might not be the most exciting video clip, but let me tell you how exciting it actually was that it worked that smoothly during data collection when it really mattered.

Here we have a brief clip from one of our iOS sessions, again showing the smooth transition when we asked the participant to begin sharing their screen. What youll see/hear is me asking the participant to select the Share screen option on the Zoom desktop app, then swipe up on her phone, look for AirPlay, and select her computer from the list. Once she has turned the mirroring toggle on, her phone screen will be mirrored on her computer and streamed to mine.

Miraculously, it worked that smoothly is all seven of our sessions. 50

Lessons LearnedTips & Tools: Remote Mobile Testing51Planning ahead is keyPractice makes perfectAlways have a backupPut participants at easeTailor recruitment to make your life easier

DDBased on our experience in this first real world implementation of some of the tools we uncovered, we learned a few lessons about how to make a remote mobile usability test run as smoothly and successfully as possible. Some of these things we did and therefore thought they contributed to our success and other things we didnt do, but realized after the fact that they would have been helpful.

Planning ahead: Spending extra time up front making sure the setup worked for participants. Clearly thinking through the logistics of sending participants the information they needed (tasks, links, phone numbers) when they needed it.

Practice makes perfect. By becoming intimately familiar with the tools we were going to use during the session allowed us to more easily troubleshoot any issues that arose. In particular, we wanted to become familiar with what the participant sees on their end so we could give them accurate instructions.

Always have a backup. When technical issues arise, its always good to have a backup. We knew that if the phone screen sharing didnt work during the session, we could quickly relegate our testing method to one of the less optimal, but still valid mobile testing methods, such as re-sizing the browser to a mobile device-sized screen. If Zoom or Join.me didnt work at all, we knew we could revert to our more reliable and commonly used tool for sharing desktops remotely, GoToMeeting. Fortunately, we didnt need to use either of these options in our study.

Put participants at ease. We gave participants a verbal overview of the process and walked them through it on the phone, which allowed us to establish a rapport with them and gain their trust. Therefore, they were much more willing to stick with us through the bumps in the road.

Tailor recruitment. By limiting recruiting to either iOS OR Android (not both), you will only need to support one screen sharing tool. Obviously, this was something we did not do, but we feel it would have been easier if we had. However, one thing we did do, and we feel it made things go smoother, was that we recruited participants who already possessed basic mobile device interaction skills, such as being able to switch from one app to another. This made giving them instructions much easier.

51

The Future: Remote Mobile ToolsTips & Tools: Remote Mobile Testing52Our Ideal Tool DimensionsEaseCostPerformanceFeaturesPrivacy

DD

Thinking about the future of remote mobile testing, as a reminder, we were looking for tools with these certain characteristics, and the tools that we found generally fulfilled most of them. However, there are a couple areas with room for improvement. 52

The Future: Remote Mobile ToolsTips & Tools: Remote Mobile Testing53Our Ideal Tool DimensionsEaseCostPerformanceFeaturesPrivacy

DD

53

The Future: Remote Mobile ToolsTips & Tools: Remote Mobile Testing54Specific areas for improvementFeaturesAbility to work reliably with Android AND iOS

PrivacyScreen sharing of one particular app rather than the whole screenSharing control of the researchers mobile device screen

DDAs far as features, we have yet to find a tool that works reliably with both Android and iOS.

But the area where the most improvement is necessary is in participant privacy. We would love to see a tool that allows participants to share just one app rather than their whole screen (without having to change the apps code), or even better, a tool that allows participants to control the researchers mobile device from their own.

We believe the desire to test mobile interactions remotely will only continue to grow. So, as technology improves and the need for more robust tools is recognized, we anticipate more options and capabilities will emerge on the market in the near future. And by near future, we mean that these capabilities could be on the market in the next few months because these things change so rapidly.

54

Tristan WilsonUser Experience Specialisttwilson@userworks.comTips & Tools: Remote Mobile Testing55Dana DouglasUser Experience Specialistddouglas@userworks.comhttp://www.uxpa2016.org/sessionsurvey?sessionid=81