Developing ForMicrosoft Surface

Brad CarpenterGeneral ManagerSurface Platform

PC17

Agenda

Microsoft Surface and the future of computing

How you can be part of this revolution How the Surface SDK enables

rapid innovation Microsoft’s multi-touch roadmap Questions and Answer

CLI GUI NUI

Interface Paradigm Shift

Microsoft Surface computing uses sensing and display technology to

infuse everyday surfaces with digital content and is comprised of the

following unique attributes:

Microsoft Surface And NUI

Direct interaction Object recognitionMulti-touch Multi-user

Direct interaction Object recognitionMulti-touch Multi-user

How Surface Is Being Used Today

5 Verticals • Automotive

• Financial Services• Healthcare• Leisure and Entertainment• Retail

Microsoft Surface Hardware And SDK availability

announcing…

Developing ForMicrosoft Surface

Robert LevyProgram ManagerSurface SDK

Doug KramerLead DeveloperSurface SDK

Visualizing Surface Input

demo

Congratulations!

You already know the basics of building applications for Surface

Microsoft Surface Hardware & Drivers

Surface v1 Architecture

Windows Vista SP1

Vision System Windows Integration

Core API Shell UI & APIs

Surface Applications

WPF APIs

Surface And WPF

Expose Surface capabilities in a way consistent with WPF Infrastructure integration for consistent behavior

with mouse Automatic hit testing to find the control being touched Sets attached properties on controls

(IsContactOver, etc.) Raises routed events on controls (ContactDown, etc.)

Custom versions of most built-in WPF controls Custom controls for Surface-specific UX Base classes for building custom Surface controls

XAML <s:SurfaceWindow> <Canvas

s:Contacts.ContactDown=“OnContactDown“> <s:SurfaceButton Click=“OnButtonClick”/> </Canvas> </s:SurfaceWindow>

Code void OnContactDown(object sender,

ContactEventArgs e) { // ... } void OnButtonClick(object sender,

RoutedEventArgs e) { // ... }

Surface API Usage With WPF

XAML <Window> <Canvas Mouse.MouseDown=“OnMouseDown“> <Button Click=“OnButtonClick”/> </Canvas> </Window>

Code void OnMouseDown(object sender,

MouseEventArgs e) { // ... } void OnButtonClick(object sender,

RoutedEventArgs e) { // ... }

Surface Classes For WPF Input

MouseStylus

MouseDeviceStylusDevice

MouseEventArgsStylusEventArgs

Tablet

TabletDevice

Surface Versions Of Common Controls

SurfaceWindowFull screen on the SurfaceOriented towards the user

SurfaceMenuSurfaceContextMenu

SurfaceMenuItemMultiple menus can be used at

the same time

SurfaceButtonSurfaceRadioButton

SurfaceCheckBoxSurfaceToggleButton

Only “click” when all contacts are removed

Surface Versions Of Common ControlsSurfaceListBox

SurfaceScrollViewer• Determines scrolling vs. select

• Allows flicking• Provides “elastic” effect

SurfaceSliderSurfaceScrollBar• Allows flicking

• Track input results in smooth animation and capture

• Subtle UX when not in use

SurfaceTextBoxSurfacePasswordBox

• Invokes virtual keyboard oriented towards user when

touched

Neat, But Does It Justify Buying A Surface?

We’ve made it easy to port WPF and XNA apps to Surface

Great for developers Re-use code, skills, and tools Get started quickly

…but little value to users

Is this really just a big PC?

Reading upside down makes my

head hurt

Constantly reaching across the table is

not fun

I don’t like having to share and wait

my turn

Making Surface Shine

Great apps leverage all of what makes Microsoft Surface unique Multi-touch Multi-user 360° UI Gestures and manipulations Interaction with physical objects

WPF and the Surface SDK make this easy

ScatterView By Example

Introducing The ScatterView Control

Quickly enables 360°, multi-user apps Gives users direct, shared control of layout Standardizes manipulations for Resize,

Move, and Rotate Same familiar programming model

as ItemsControls, ListBox, TreeView, Menu, etc.

ScatterView In Action

demo

Recognizing Tagged Objects

Contact.Tag.Byte256 unique values

Contact.Tag.Identity>340,282,366,920,938,000,000,000,000,000,000,000,000

unique values

Byte Tags

Identity Tags

Introducing The TagVisualizer Control

Simplifies common scenarios involving tagged objects Displays interactive UI under, around, or next to objects Keeps UI in sync with movements Handles temporary lifting

TagVisualizer Control

Tag values to look for

UI for each object

Positioning of UI relative to tag

Max # of each object allowed

How to handle tag removal

TagVisualizerIn Action

demo

Is this really just a big PC?

Reading upside down makes my

head hurt

Constantly reaching across the table is

not fun

I don’t like having to share and wait

my turn

Controls For Making Surface Shine

ScatterView Multi-touch Multi-user 360° UI Natural interactions

with virtual objects

TagVisualizer Object recognition Natural

interactions with physical objects

Redesigning An App For Surface

demo

Surface Hardware

Windows 7

Surface Hardware

Windows VistaWindows 7

NativeWin32

Application

Touch Development Roadmap

WPF 3.5

Surface SDK 1.0

Managed Wrapper and Interop

WinForms Application

Windows 7 ReleaseNET 4.0 / Surface 2.0 Release

WPF 3.5 x WPF 4.0

Surface SDK 2.0

WPF Application

Multi-Touch API

Surface Application

Multi-Touch Controls

Multi-Touch API

Surface Multi-Touch

Controls & API

Multi-Touch API

Multi-Touch API and Controls

NET 3.5 / Surface 1.0 Release

Summary

Microsoft Surface is shaping the future Multi-touch, multi-user, 360° UI, object recognition, …

UI design requires creativity and innovation Coding builds upon what you already know Microsoft is creating a unified API for touch

Call To Action

Help us change the world Create apps using Surface’s unique capabilities

Get Microsoft Surface developer hardware Order forms on http://surface.com $1,500 discount until November 15th

Swipe your badge to receive an e-mail invitation to join http://community.surface.com Download the Microsoft Surface SDK & Simulator Participate in online forums and training

Related Content

Hands on Labs PCHOL19: Using the Microsoft Surface Simulator PCHOL20: A Lap Around the Microsoft Surface SDK Surface HOL: Exploring Vision Based Object

Recognition with Microsoft Surface Demo applications from partners Scavenger Hunt game PC03: Developing multi-touch applications

with Windows 7 Wednesday, 1:15

Questions?

Please submit session feedback!

Evals & Recordings

Please fill

out your

evaluation for

this session at:

This session will be available as a recording at:

www.microsoftpdc.com

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market

conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

APPENDIX

Simulator for Off-Table Dev & Test

Replicates input With multiple mice With one mouse

1. Click to place multiple contacts2. Click and drag to group them together3. Move, rotate, and stretch the group

Supports all types of contacts Finger, resizable blobs, and tags

Supports input automation for testing Recording and playback UI

Simulator APIs 37

Manipulation & Inertia Processor APIs Low level APIs for building controls with consistent

behavior Reduces the cost of building new controls Increases consistency for end users

Interpreting Surface input as 2D transformations Input: list of contacts to track Output: scale, rotation, movement, velocity data (~60 FPS) Used by: ScatterView, ScrollViewer, Slider, Concierge map

Two modes of inertia Initial velocity + deceleration rate

Realistic Used by ScatterView and Concierge map

Initial velocity + final position/orientation Goal-based Used by ScrollViewer and Slider

Ups & Downs

Mouse.MouseDown Mouse.MouseUp

39

Contacts.ContactDown Contacts.ContactUp

Up & Down Event Comparison

Similarities: Naming convention & meaning Routing strategy

Tunneling (“preview”) events & bubbling events Events stop once marked as handled Hit testing & capture logic

Differences: Need to find out which contact the event is for New Contact object is created with each Down

event

40

Movements

41

Mouse.MouseMove … Mouse.MouseMove …

Contacts.ContactChanged … Contacts.ContactChanged …

MouseMoved == ContactChanged?

ContactChanged includes: Position Orientation Recognized type Size Shape Area

42

Wait a sec… he said this was supposed to be the same as mouse. Why are these events named differently?

Movement Event Comparison

Similarities: Routing strategy

Tunneling (“preview”) events & bubbling events Events stop once marked as handled Hit testing & capture logic

Event for changes in position Differences:

ContactChanged is not just for movements Change events only happen between Down & Up Need to find out which contact the event is for

43

Position

Contact.GetPosition(UIElement)

Relative to specified UI element GetPosition(window) = {50, 100} GetPosition(button) = {10, 10}

To track position changes across multiplecontrols, get position relative to a common parent

44

Orientation

Contact.GetOrientation(UIElement)

Degrees clockwise from relative east

Range and usefulness varies Tagged objects: 360° Generic blobs: 180°

Fingers: 360° but…unreliable when straight down

45

0°

90°

180°

270°

Ellipse & Bounds

Rect Contact.BoundingRect

Ellipse Contact.GetEllipse(UIElement)

void Contact.UpdateEllipse(Ellipse, UIElement)

Ellipse is returned with transforms that counteract transforms on the specified element Adding the shape as a child element results in UI

matching the physical object Bounding rectangle is always screen-aligned UpdateEllipse method can improve performance

46