![Page 1: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/1.jpg)
. 1
22.06.2015
Sebastian Feuerstack, Állan C. M. de Oliveira, Mauro dos Santos Anjo, Regina B.Araujo, and Ednaldo B. Pizzolato
Model-based Design of Multimodal
Interaction for
Augmented Reality Web Applications
The Augmented “Drag-and-Drop”
![Page 2: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/2.jpg)
Overall Research Question
2
22.06.2015
How to model
interactions that
span Modes and
Media ?
http://www.multi-access.de
![Page 3: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/3.jpg)
Multimodal Interfaces
22.06.2015
Which Media ?
http://flic.kr/p/6eeLwg
Which Modes ?
http://flic.kr/p/6eeLwg http://flic.kr/p/7sZ6ho
3
![Page 4: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/4.jpg)
Interaction Resources
22.06.2015
How ? Interaction Resources
4
![Page 5: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/5.jpg)
Why modeling?
22.06.2015
Why modeling?
One specification that
is declarative and exact
discuss & execute & store & share
add new forms of interaction
no code but model changes
5
![Page 6: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/6.jpg)
Related Work 6
22.06.2015
![Page 7: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/7.jpg)
Use Case:
Augmented Drag and Drop
7
22.06.2015
Buying Furniture
Shopping Cart
Drag and Drop
Into my environment
![Page 8: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/8.jpg)
How to model?
22.06.2015
8
Cameleon Reference Framework
W3C MBUI
Task Model
Abstract UI
Custom
Interaction Resources
Mappings
![Page 9: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/9.jpg)
Abstract User Interface Model
9
22.06.2015
![Page 10: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/10.jpg)
Abstract User Interface Model
22. Juni 2015 The Augmented “Drag-and-Drop
10
![Page 11: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/11.jpg)
Abstract Behavior Model
22. Juni 2015
AUI:AIO:AIChoiceElement:
AISingleChoiceElement
initialized
listed
focus
suspended
organize
defocus
organized
suspend
focused
organize
Presenting
chosen
unchosen
drag
dragging
unchoose
drop
H
[in(focused)] choose
/ aios=find(parent.childs.chosen);
aios.all.unchoose
next||prev||parent
/aio=find(act);
aio.focus
Selectionpresent
11
![Page 12: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/12.jpg)
Multimodal Mapping: The Mouse-based Drag-and-Drop
22.06.2015
C
dst=SingleChoice.focusedC
Tw<0,3s
sc.dragdst.drop(aios)
sc=SingleChoiceElement.focused
Tw<0,3sLeftButton.pressed
LeftButton.released
sc.parent.dropfail
12
IR:IN:Pointer
stopped
move stop
moving
IR:IN:Button
released
press release
pressed
IR:IN:Mouse Mouse
ButtonWheel Pointer
LeftButton RightButton
- x,y- z
![Page 13: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/13.jpg)
Hand Gesture Resource Interactor
22.06.2015
IR:IN:HandGesture:PointingSelect
left_hand_appeared
thumb_down
Left Hand
left_hand_disappeared
selected
released
thumb_up
Command
Right Hand
NoLeftHandBothHands
NoRightHand
right_hand_disappearedright_hand_appeared
Pointing
move
moving
stopped
stop
move
NoHands
left_hand_appeared right_hand_appeared
left_hand_disappearedright_hand_disappeared
Cdst=SingleChoice.focused
C
Tw<0,3s
aois.dragdst.drop(aios)
sc=SingleChoice.focused
Tw<0,3sRightHand.cmd.selected
RightHand.cmd.released
![Page 14: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/14.jpg)
Video: Connecting the Web with AR
14
22.06.2015
Cam
Cam Pointing
Command
Monitor
Jessica
![Page 15: Model-based Design of Multimodal Interaction for Augmented …web3d2015.web3d.org/Presentations/Saturday/Paper_session... · 2015-07-02 · 1 . 22.06.2015 Sebastian Feuerstack, Állan](https://reader033.vdocuments.us/reader033/viewer/2022053010/5f0dbbb87e708231d43bd0d3/html5/thumbnails/15.jpg)
Conclusions & Future Work
22.06.2015
• Detailed, declarative modeling of multimodal interactions
• We can combine modes and media by mappings and
bridge realities
• All models are interpreted and can be changed while
running
• What`s still missing is a tool / editor
• We intend to contribute to
• W3C-MMI WG
• W3C Ubiquitous Application Design CG
15