Windows Touch Deep Dive

Reed TownsendProgram ManagerMicrosoft Corporation

CL13

Agenda> Introduction> Overview of Windows Touch API

surface> Real-world ISV questions and solutions> Demo projects

> 3D manipulations> Reimplementing Collage’s PhotoStrip

control> Wrap up

Touch - Framing the Opportunity> What is touch?

> It’s an input method> What does it mean to design for

touch?> When designing for inputs, we’re really

talking about user interface design (in the broad sense)

> UI is often synonymous with visuals and the input and interaction half is forgotten

Touch - Framing the Opportunity> 99% of design is done in the platform

> Controls and APIs are the language of UI> “The limits of my language mean the

limits of my world” – Ludwig Wittgenstein> Games (and sometimes web) follow a

different model> What are the common controls for

touch? ?

Windows Touch APIs> Good/Better/Best platform model

> Good:> What you get for free> Legacy, non-touch-aware apps

> Better> Minimal investment> Typically WM_GESTURE, mouse message

inspection> Best

> Enables touch-optimized experiences!> WM_TOUCH, manipulations & inertia, and

more

WM_GESTURE> Provides

notifications when the user performs gestures over your window

> Contains additional information like center of gesture and gesture-specific arguments

Typical sequence:WM_TABLET_QUERYSYSTEMGESTURESTAT

USWM_GESTURENOTIFYWM_GESTURE (GID_BEGIN)

WM_GESTURE (GID_PAN, GF_BEGIN)WM_GESTURE (GID_PAN, no flags)...WM_GESTURE (GID_PAN,

GF_INERTIA)...WM_GESTURE (GID_PAN, GF_END)

WM_GESTURE (GID_END)

Real-world Questions and AnswersWM_GESTUREQ: I registered for WM_TOUCH but I no longer get

WM_GESTURE. What gives?A: Yes, WM_GESTURE and WM_TOUCH are exclusive

in Win7.

Q: I’m implementing gestures in my app, but I stopped getting panning bounce feedback. Why?

A: Two possibilities: Panning messages are being accidentally eaten, or your app is providing custom handling of them but not calling the bounce feedback API.

GetSystemMetrics()> GetSystemMetrics():

> SM_DIGITIZER returns data about available digitizers on the system

> SM_TABLETPC returns data about Tablet functionality

> SM_MAXIMUMTOUCHES yields the largest number of contacts any of the available digitizers supports

Mouse Messages & GetMessageExtraInfo()

> GetMessageExtraInfo() returns the extra info associated with a message

> Mouse up and down messages are tagged with a special signature indicating they came from touch or pen:> Mask extra info against 0xFFFFFF80> 0xFF515780 for touch, 0xFF515700 for

pen

Richer Touch Experiences

demo WM_TOUCH Applications

WM_TOUCH> Provides raw touch

input data, conceptually similar to mouse messages

> Finger painting, custom gestures, feeding higher-level controls (e.g. Manipulations)

Typical sequence:WM_TOUCH (first finger down)WM_TOUCH (first finger moving)...WM_TOUCH (two fingers moving,

third comes down)WM_TOUCH (three fingers moving)...WM_TOUCH (one goes up, two

fingers moving)WM_TOUCH (two fingers moving)...WM_TOUCH (last finger goes up)

Real-world Questions and AnswersWM_TOUCHQ: How can I disable the press and hold gesture in my

window?A: WM_TABLET_QUERYSYSTEMGESTURESTATUS can be

used to disable P&H, flicks, and tap feedback. (WM_TOUCH windows already disable flicks by default. SetGestureConfig() controls gestures available via WM_GESTURE.)

Q: I’m building a UI framework. How can I add touch support?

A: Start with WM_TOUCH. If you want pan/zoom/rotate, use manipulations on top of WM_TOUCH.

Touch-Optimized Experiences

demo WM_TOUCH, Manipulations, and More

Manipulations & Inertia> Manipulations:

> Provides the full range of 2D affine transforms corresponding to user input

> Enables multiple concurrent interactions> Can be fed by any source of raw data> Are a superset of WM_GESTURE support

> E.g. Rotating and translating 3 photos simultaneously

Typical Flow

On contact down:> Hit test against

targets> (Add to map of

contact ids and MPs)

> MP->ProcessDown()

On contact moves:> (Look up mapping)

> MP->ProcessMove()

On contact up:> (Look up mapping)> MP->ProcessUp()> (Kick off inertia)

Get callbacks on event interface> Apply transforms

to targets

Advanced Manipulation Topics> Single-finger rotation

> PivotPoint, PivotRadius properties> Rotation around a point

> Generalization of single-finger rotation> Supporting mouse-based

manipulations

> All covered in the Platform SDK manipulations sample

Inertia> Grab initial state from manipulation

processor in completed event> Set up deceleration properties> Set up borders/margins> Call Process() on a timer

Demo Projects!> Demos:

> 3D manipulations – An approach to mapping 2D onto 3D.

> Reimplementing Collage’s PhotoStrip – Building custom touch controls

> Examples of new ways to use the platform and new user interface options

3D Manipulations> Set up 3D view,

axes, objects> Add WM_TOUCH> Add manipulations> Map manipulations

to 3D navigation> 2 finger camera

operations, zoom> Spherical panning

> Add inertia and other tweaks

3D Manipulations> Classes of note:

> D3DXDriver> CComTouchDriver> Camera

> Inherits from InertiaObj and _IManipulationEvents

> Example data flow:> WM_TOUCH message

arrives> CComTouchDriver::Process

InputEvent()> Camera::ProcessInputEven

t()> InertiaObj::ProcessInputEv

ent()> Manipulation Processor> Camera::ManipulationDelt

a()> Update the camera

Building a PhotoStrip Control> Part of a control

framework> Respond to touch

messages> Add constrained

manipulations for panning

> Add support for dragging in/out of control

> Add API surface> Flesh out overlay and

gallery

Building a PhotoStrip Control> Window hierarchy: > Example data flow:

> WM_TOUCH arrives> Hits overlay, routed to

appropriate control> Photostrip gets

message> Touch pans and

triggers photo removal

> Message to app window

> Message to gallery> Photo is displayed in

gallery

Overlay HWND

PhotoStrip1 HWND

PhotoStrip2 HWND

Gallery HWND

Implementation Notes> Smoothing input and manipulation

data> Mixed input layers

> 3D: manipulations + gestures> PhotoStrip: raw data + manipulations

> Supporting multiple interactions across windows> Overlay + forwarding approach> Capture> Mediating between inputs

Final Thoughts> What are the common controls for

touch?> We don’t know – they need to be defined> The existing controls were optimized for

mouse and keyboard over decades> Help shape the future of touch user

interfaces!

>>FUTURE

Surface HardwareWindows

VistaWindows 7

NativeWin32

Application

Touch Development Roadmap

WPF 3.5 SP1

Surface SDK1.0

Managed Wrapper and

Interop

WinForms Application

.NET 4.0 & Surface Toolkit (Q1 2010)

WPF 4.0 w/ Multi Touch APIs

Multi-Touch ControlsSurface

Multi-Touch Controls

&API

Multi-Touch API

Surface Toolkit for Windows 7

Multi-Touch Controls &

API

WPF Application

Surface Application

Related Content and ResourcesWorkshops and sessions:

> PDC09-WKSP08 – Windows 7 Developer Boot Camp

> PDC09-CL27 – Multitouch on Microsoft Surface and Windows 7 for .NET Developers

Contact us:> Forums:

http://social.msdn.microsoft.com/Forums/en-US/tabletandtouch/threads/

> Email: wintouch@microsoft.com

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Extra Slides

WM_GESTURE> Provides

notifications when the user performs gestures over your window

> Contains additional information like center of gesture and gesture-specific arguments

Typical sequence:WM_TABLET_QUERYSYSTEMGESTURESTAT

USWM_GESTURENOTIFYWM_GESTURE (GID_BEGIN)

WM_GESTURE (GID_PAN, GF_BEGIN)WM_GESTURE (GID_PAN, no flags)...WM_GESTURE (GID_PAN,

GF_INERTIA)...WM_GESTURE (GID_PAN, GF_END)

WM_GESTURE (GID_END)

Controlling Gesture Behavior> WM_GESTURENOTIFY – “Heads up, a

gesture is coming”

> SetGestureConfig()> Control how gestures work in your

window> Examples: 2D panning, explicitly setting

SFP flags> Can set in response to

WM_GESTURENOTIFY, at window creation, or other times

Single-Finger Panning (SFP)Enabling 2D panning

GESTURECONFIG gc = { GID_PAN, GC_PAN & GC_PAN_WITH_SINGLE_FINGER_VERTICALLY &

GC_PAN_WITH_SINGLE_FINGER_HORIZONTALLY & GC_PAN_WITH_INERTIA,

0 };

BOOL bResult = SetGestureConfig( hWnd, // window for which configuration is specified 0, // reserved, must be 0 1, // count of GESTURECONFIG structures &gc, // array of GESTURECONFIG structures

sizeof(GESTURECONFIG) // sizeof(GESTURECONFIG) );

Legacy Gesture Handling> If an app doesn’t handle

WM_GESTURE, it is treated as a gesture-unaware legacy app

> Panning and zooming fallback behaviors are provided

> If you want any default handling of gestures, make sure GID_BEGIN, GID_END, and the GID_ for the gesture in question are passed to DefWindowProc()

Real-world Questions and AnswersWM_GESTUREQ: I registered for WM_TOUCH but I no longer get

WM_GESTURE. What gives?A: Yes, WM_GESTURE and WM_TOUCH are exclusive

in Win7.

Q: I’m implementing gestures in my app, but I stopped getting panning bounce feedback. Why?

A: Two possibilities: Panning messages are being accidentally eaten, or your app is providing custom handling of them but not calling the bounce feedback API.

GetSystemMetrics()> GetSystemMetrics():

> SM_DIGITIZER returns data about available digitizers on the system

> SM_TABLETPC returns data about Tablet functionality

> SM_MAXIMUMTOUCHES yields the largest number of contacts any of the available digitizers supports

Mouse Messages & GetMessageExtraInfo()

> GetMessageExtraInfo() returns the extra info associated with a message

> Mouse up and down messages are tagged with a special signature indicating they came from touch or pen:> Mask extra info against 0xFFFFFF80> 0xFF515780 for touch, 0xFF515700 for

pen

Advanced WM_GESTURE Topics> Compound gestures:

> Zoom and Pan simultaneously: both messages contain information about both gestures

> Message bubbling> Unhandled gestures bubble up the parent

window chain> If they aren’t handled, OS legacy support kicks

in> Coalescing

> WM_GESTURE message are coalesced> Capture

> Similar to mouse, implicit capture to window on down

Advanced WM_TOUCH Topics> Primary contact

> First finger down is the “primary contact” and determines the target window for capture

> Capture> Similar to mouse, contacts are implicitly

captured on down and limited to the primary contact’s window

> Coalescing> WM_TOUCH is coalesced. Can disable

coalescing explicitly w/ a flag in RegisterTouchWindow().

Surface HardwareWindows

VistaWindows 7

NativeWin32

Application

Managed Wrapper and Interop

Touch Development Roadmap

WinForms Application

Windows 7 (2009)

Multi-Touch Controls

Multi-Touch API

WPF 3.5 SP1

WPF Application

Surface SDK1.0

Multi-Touch Controls &

API

Surface Application

>>FUTURE

Surface HardwareWindows

VistaWindows 7

NativeWin32

Application

Touch Development Roadmap

WPF 3.5 SP1

Surface SDK1.0

Managed Wrapper and

Interop

WinForms Application

.NET 4.0 & Surface Toolkit (Q1 2010)

WPF 4.0 w/ Multi Touch APIs

Multi-Touch ControlsSurface

Multi-Touch Controls

&API

Multi-Touch API

Surface Toolkit for Windows 7

Multi-Touch Controls &

API

WPF Application

Surface Application

>>FUTURE

Surface HardwareWindows 7

Windows 7

NativeWin32

Application

Touch Development Roadmap

Managed Wrapper and

Interop WPF 4.0 w/ Multi Touch APIs

Surface Toolkit

Surface SDKvNext

WinForms Application

Surface vNext (Future)

Multi-Touch ControlsSurface

Multi-Touch Controls

&API

Multi-Touch API

Surface-specific

Controls & API

WPF Application

Surface Application

YOUR FEEDBACK IS IMPORTANT TO US! Please fill out session evaluation

forms online atMicrosoftPDC.com

Learn More On Channel 9> Expand your PDC experience through

Channel 9

> Explore videos, hands-on labs, sample code and demos through the new Channel 9 training courses

channel9.msdn.com/learnBuilt by Developers for Developers….

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.