exploring design for multi-device, multi-environment and multimodal connected experiences

8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences

1/62


2/62

Research questions

In case of user having multiple devices:

1 Whether they would prefer to switch between multiple device to complete a particular task? IfYes or No then Why?

On what factors this switching depends:

• Type of tasks• Single or Multiple User• Environment

2 Whether they prefer to switch between modalities while completing a particular task? If Yes oNo then Why?

3 Is there any interaction between device and modality preference?


3/62

Plan of action

Single User Multiple (Two) Users

Home

Car

Single user inHome context

Two users inHome context

Two users inCar context

Single user inCar context


4/62

Plan of action

Single User Multiple (Two) Users

Home

Car

Single user inHome context

Two users inHome context

Two users inCar context

Single user inCar context

11

10

9

9

23

18

Completed

To Be Completed

1

1


5/62

Single user in Home context > Scenario and Tasks

You are watching a movie on TV and along with that you are following a Cricket match. Duringcommercial, you switch to Sports channel. You find that one of the player is leading the team to

win the match. You want to post about this on Facebook to show your happiness.

Subtasks :1. Changing the channel

2. Posting on Facebook

(i) Specify the action, (ii) Add text you want to post, (iii) Confirmation


6/62

Single user in Home context > Set of Devices

1) TV and smartphone 2) TV, smartphoneand smartwatch


7/62

Single user in Home context > Interaction Methods

Smartphone Smartwatch

Touch

Voice

Physical Motion/Gesture


8/62

Single user in Home context > Activity


9/62


1. First participant is introduced with scenario, subtasks and interaction methods along with anexample

2. He is asked to give first preference for each subtask and then, complete sequence of preferenfor each task.

3. These preferences are taken for both set of devices (TV, Phone; TV, Phone and Watch).

4. While having preferences they are also asked to think aloud to capture the reason behind the

preferences. Therefore, sessions are audio recorded.


10/62



11/62

You are driving your car to Home and receive a message from a friend saying that your close comfriend is in the town. In message (s)he also invites you at his/her place. You replies back that you reaching there in some time. You opens the navigator and set the destination to friend’s place.

User actions or point of decisions :

Message Notification

1. Your command to read it

System shows/tells you the message

2. Reply: Compose & Send

3. Open Navigator

4. You command system to change the destination to friend’s place

System changes it.

Single user in Car context > Scenario and Tasks


12/62

Car Dashboard Smartphone Smartwatch

Touch

Voice

Physical Mot

Single user in Car context > Interaction Methods


13/62

2) Car dashboard, Smartphoneand smartwatch

1) Car Dashboard

Single user in Car context > Set of Devices


14/62

Analysis


15/62

Analysis

1. Device Sets2. Tasks – Type and activity group3. Contexts- Home or Car4. No. of users- One or Two

1. Modality: Preference and Switchi2. Device: Preference and Switching

Effect of changing these On these


16/62

1. Tasks that involved changing the value under a

certain variable- Ta

2. Tasks that involved just one button touch/command/input- Tb

3. Tasks that involved composing - Tc

Analysis > Categorization of tasks


17/62

Results Summary


18/62

In Home context, Gear is preferred more than phone and in Car context, cwas most preferred, when an option among the devices is given

Voice was more preferred over any kind of device generally but touch and gesture wpreferred for particular type of task (ex. Navigation and changing channel).

For a particular task, while changing device sets, modality preference remained mosthe same implying that modality was more important w.r.t. the task than the

Result Summary - Preference


19/62

While going from one task to the next, modality switching is more frequentswitching (for both home and car) implying that users try to avoid device cha

than modality change.

Device and modality change is observed more while switching tasks of diffrather than tasks among the same activity.

There is more device and modality switching in case of home than car scen

Result Summary - Switching


20/62

Home ScenarioANALYSIS


21/62

1. Tasks that involved changing the value under

a certain variable- Ta



T1 > T2

Ta Tb

T1- Changing t

Posting on FacT2: (i)Specify t

T3: (ii) Add texT4: (iii) Confir



22/62

Combining the tasks w.r.t the activity they belong to

Here,T2, T3,T4 are tasks under the same activity of posting onFacebook.

T1 > T2

Ta Tb

T1- Changing t

Posting on FacT2: (i)Specify t

T3: (ii) Add texT4: (iii) Confir

Analysis > Categorization of sequence of tasks


23/62

Home ScenarioRESULTS


24/62

Results > Home- Phone

Table 2 PM PT PV

T1 25 20 22T2 14 19 32T3 12 24 30T4 13 24 29

25

1412

13

2019

24 2422

3230

29

0

5

10

15

20

25

30

35

T1 T2 T3 T4

PM

PT

PV


25/62

1. Task Wise (Home-Phone):

T1: Modality : Motion >Voice >Touch

T2: Modality : Voice >Touch >MotionT3: Modality : Voice >Touch >MotionT4: Modality : Voice >Touch >Motion

T1- Changing

Posting on FaT2: (i)Specify

T3: (ii) Add tT4: (iii) Conf

Results > Modality preference


26/62

TABLE 1

T1 T2 T3 T4PM 32 23 14 21PT 32 40 48 41PV 35 55 56 49GG 51 23 21 28GT 34 34 32 39GV 47 56 60 53

32

23

14

21

32

40

48

41

35

55 56

4951

2321

28

34 3432

3

47

5660

0

10

20

30

40

50

60

70

T1 T2 T3 T4

Results > Home- Phone, Watch


27/62

2.Task-wise (Home-Phone, watch):

T1: Device: Gear >Phone

Modality: Gesture >Voice >Touch

*preferences among modalities and devices was very close

T2: Device: Gear >PhoneModality: Voice >Touch >Gesture/Motion



T1- Changing



Results > Device and Mobile Preference

l f


28/62

3. Among tasks that were under the same activity (like T2, T3, T4):similar interactions (either in terms of devices or modalities) wereobserved.

Reasons:a) According to accessibility of the device and proximityb) Why was voice more preferred?

lesser chances of distraction from the primary task was involvedin this modality

4. Gear is preferred more than phone when an option between thedevices is given.Although the device is changed, modality preference in doing thetasks remained the same (among the device sets).

Results > Inferences

R l D i i M d li f


29/62

5. (a) Task Wise (Home-Phone, Watch ):

T1: Phone: Voice >Touch = Motion

Gear: Gesture > Voice > Touch

T2: Phone : Voice >Touch >MotionGear: Voice >Touch >Motion/Gesture

T3: Phone: Modality : Voice >Touch >MotionGear: Voice >Touch >Motion/Gesture

T4: Phone : Voice >Touch >MotionGear: Voice >Touch >Motion/Gesture

T1- Changing



Results > Device wise : Modality preference

R lt > D i i M d lit f


30/62

5. (b)Independent of tasks:

Phone: Voice >Touch > Motion

Gear: Voice >Touch > Motion/ Gesture* T1: Gesture > Voice > Touch T1- Changing


T3: (ii) Add t

T4: (iii) Conf


R lt > D i d M d lit S it hi g


31/62

6. (a) Home- Phone, Watch

(b) Home- Phone

Device and modality change is observed more while switchingtasks of different activities rather than tasks among the sameactivity.

Modality switching is more frequent than device change.

T1- Changing

Posting on FaT2: (i)SpecifyT3: (ii) Add tT4: (iii) Conf

Results > Device and Modality Switching

Device 4 1 0

Modality 8 3 3

Modality 7 4 4

T1 > T2 T2 > T3 T3 > T4



32/62

7. Phone > > Phone + Watch

T1: Modality Switch: 3/11Device Switch : 9/11




Modality preference is more affected by task type than the device.

Results > Device and Modality Switchingbetween device sets


33/62

Car ScenarioANALYSIS



34/62

1. Tasks that involved changing the value under

a certain variable- Ta



T1 > T2

Tb Tc

T1- Your comm

T2- Reply: Com

T3- Open Navi

T4- You comm

change the de

place




35/62

Combining the tasks w.r.t the activity they belong to

Here,T1 and T2 tasks are under the same activity ofmessaging (notification and composing)

T3 and T4 are tasks under the same activity ofnavigation.

T1 > T2

Tb Tc

T1- Your comm

T2- Reply: Com

T3- Open Navi

T4- You comm

change the

friend’s pl



36/62

Car ScenarioRESULTS

Results > Car- Car Dashboard


37/62

Results > Car Car Dashboard

21

11

16 16

12

19 1920

27

30

2524

0

5

10

15

20

25

30

35

T1 T2 T3 T4

CS

CT

CV

Results > Car- Car Dashboard Phone Watch


38/62

Results > Car Car Dashboard, Phone, Watch

28

3

17

5

18

14

28

38

41

47

43

39

01

0 0

7

13

6

27

16

32

20 20

6

24

1

6

2

7

2

28

36

25

0

5

10

15

20

25

30

35

40

45

50

T1 T2 T3 T4



39/62

1.Task-wise (Car-Car Dashboard, Phone, watch):

T1: Device: Car > Gear > PhoneModality: Voice >Touch > Gesture

*preferences among modalities and devices was very close

T2: Device: Car > Gear > PhoneModality: Voice >Touch >Gesture/Motion

T3: Device: Car > Gear > PhoneModality: Voice >Touch >Gesture/Motion

T4: Device: Car > Phone > GearModality: Touch > Voice > Gesture/Motion

T1- Your com

T2- Reply: Co

T3- Open Nav

T4- You comm

change th

friend’s

Results Inferences



40/62

2.Task-wise (Car-Car Dashboard):

T1: Modality: Voice > Steering > TouchT2: Modality: Voice > Touch > Steering

T3: Modality: Voice > Touch > SteeringT4: Modality: Voice > Touch > Steering T1- Your com

T2- Reply: Co

T3- Open Nav

T4- You comm

change th

friend’s

Results Inferences



41/62

3. (a) Task Wise (Car- Car Dashboard, Phone, Watch ):

T1: Car: Voice > Steering > TouchPhone: Voice >TouchGear: Voice > Touch = Gesture

T2: Car: Voice > Touch > SteeringPhone: Voice >TouchGear: Voice > Touch = Gesture

T3: Car: Voice > Touch > SteeringPhone: Voice >TouchGear: Voice > Touch > Gesture

T4: Car: Voice = Touch > SteeringPhone: Touch > VoiceGear: Voice > Touch > Gesture

y p

T1- Your com

T2- Reply: Co

T3- Open Nav

T4- You comm

change th

friend’s



42/62

3. (b)Independent of tasks:

Car: Voice > Touch > Steering*T1: Voice > Steering > Touch

Phone: Voice >Touch > Motion*T4: Touch > Voice > Motion

Gear: Voice >> Touch > Motion/ Gesture

* General: This information should be considered for any scenarioirrespective of the experiment done.

y p

T1- Your com

T2- Reply: Co

T3- Open Nav

T4- You comm

change th

friend’s



43/62

7. Car Dashboard > > Car Dashboard + Phone + Watch



T3: Modality Switch: 0/10

Device Switch : 2/10


There was NO modality switching observed between the device sets.

y gbetween device sets

T1- Your com

T2- Reply: Co

T3- Open Nav

T4- You comm

change th

friend’s



44/62

7. Car Dashboard > > Car Dashboard + Phone + Watch

y gbetween device sets

70% 20% 10%

No Modality Switching

No Device Switching


Device switching betweensets but continuity throughtasks


Device switch from T1 andT2 between sets and switchback to the same device.

T1 > T

Tb T

T1- Your com

T2- Reply: Co

T3- Open Nav

T4- You comm

change th

friend’s


45/62

Home and Car ScenarioRESULTS

Results > Home and car scenario


46/62

Modality and device switching

0

5

10

15

Home Car

Device

Modality

Home Car

Device 5 2

Modality 14 9.5

In the cadevice sC+P+W

There isand modin case ocar scen

In both tmodalityhigher tdevice.

Results > Home and car scenario


47/62

Modality Preference

Home (Phone)T1 Ta MotionT2 Tb VoiceT3 Tc VoiceT4 Tb Voice

Car (Car Dashboard)Tb VoiceTc VoiceTb Voice

Ta/c Voice


48/62

Themes and framework

Themes > Affinity


49/62

I n t u i t i v e n e s s D

i s t r a

c t i o

n

E a s e O F U s e E f

f o r t

Learnability

Familiarity

User Control

Continuity PhysicalFeatures

Feedback

Flexibility

Cause-Effect Framework


50/62

Home

Cause-Effect Framework


51/62

Car

Cause-Effect FrameworkC H


52/62

H

H

H

H

H

H

H

H

H

Car -> How to use

Comparison between featuresCar Travel booking

Cause-Effect FrameworkC H


53/62

M

L

L

M

M

M

M/L

L

Car -> How to use

Comparison between featuresCar Travel blogging

Cause-Effect FrameworkC > H t


54/62

Car -> How to use

Ideating on travelblogging

1. How to lower the informationrequired by the system/user?

2. How to Lower the frequency of interactions?3. How to use the environment?4. How to make the user feel controlled?5. How to increase the flexibility in the way of using a modality?6. How to incorporate the familiarity of a certain device in the feature?7. How to lower the distraction from the feedback?


55/62

Themes and inferences

Themes > Inferences


56/62

Distraction from the primary task came out to be major reason for voice to be themost preferred modality in both car and home scenario.

Most of participants wanted Facebook to be opened on TV so that they can avoidchanging devices whereas few others wanted it to be in phone because ofassociation of facebook with phone and occlusion etc. - Continuity

As confirmation is the final step , they always wanted to be sure about the abecause if here, they do something wrong, either they will go the initial step orsomething wrong will be posted.- User Control

Users feel more confident and in control while using touch than gesture becausefamiliarity and tangible feedback

Themes > Inferences


57/62

While composing text modality preference depends on type and size of text. If it’scomplex or big text they are likely to prefer touch whereas for small or simple textthey would choose voice- Flexibility

While switching channels, modality preference depends on distance b/w the twochannels-> modality preference e.g. just next channel-> motion, going to aparticular channel: voice, browsing: multiple channels- Flexibility

Few participants reported that choice of modality depends on type of feedreceived from system in previous task e.g. If system is communicating throughaudio, user tend to use voice.

Themes > Inferences


58/62

Participants found opening Facebook while watching TV too specific and lessfrequent than changing channel so they responded that it is likely to define andremember a gesture/motion for changing channel rather than opening facebookLearnability

Users always tend to avoid gestures and motion when multiple interactions have tobe done for completing same task e.g. browsing through channels. Touch is morepreferred in such cases.- Effort

Themes > Inferences


59/62

Varied Perception about voice:

Many people thought it to be quick and natural way of communication. But, fewothers told that they have to be conscious while giving commands, specially, whencommands are small or trivial.

For voice, I have to be more conscious, it won’t be the same way I talk. So it’s not the natural always.

I will use voice as it is natural and intuitive than touch .

Too much of voice inputs would look commanding.

TRIVIAL PART COMES IN PLAY. Voice is involving more effort and more energy, eventually my throatup . Saying same input to myself will sound mad.


60/62

Appendix

Home- Phone, WatchT1 PM PT PV GG GT GV


61/62

Phone, Watch

T1 PM PT PV GG GT GVPref 1 2 0 0 5 0 4Pref 2 2 1 4 2 0 2Pref 3 0 2 1 1 6 1Pref 4 1 5 2 1 1 1Pref 5 1 1 1 2 3 3Pref 6 5 2 3 0 1 0

32 32 35 51 34 47


23 40 55 23 34 56


14 48 56 21 32 60


21 41 49 28 39 53

Home- Phone


62/62

Phone

T1 PM PT PV

Pref 1 6 1 5

Pref 2 2 7 1

Pref 3 3 3 5

25 20 22

T2 PM PT PV

Pref 1 0 1 10

Pref 2 3 7 1

Pref 3 8 2 0

14 19 32

T3 PM PT PV

Pref 1 0 3 8

Pref 2 1 7 3

Pref 3 10 1 0

12 24 30

T4 PM PT PV

Pref 1 0 3 8

Pref 2 2 7 2

Pref 3 9 1 1

13 24 29

exploring design for multi-device, multi-environment and multimodal connected experiences

Documents