Microsoft Research Microsoft Research Faculty Summit 2003Faculty Summit 2003

Brad A. Myers & Jeffrey NicholsBrad A. Myers & Jeffrey NicholsThe Personal Universal Controller and .NET CFThe Personal Universal Controller and .NET CFPebbles Research ProjectPebbles Research ProjectHuman Computer Interaction InstituteHuman Computer Interaction InstituteCarnegie Mellon UniversityCarnegie Mellon University

http://www.pebbles.hcii.cmu.edu

Pebbles ProjectPebbles Project

Seen in the plenary demo yesterday!Seen in the plenary demo yesterday!

Use of Use of multiple devicesmultiple devices at the same timeat the same time Pocket PC Pocket PC and and desktop PCdesktop PC Mobile Phone Mobile Phone and “and “Smart Home”Smart Home” Multiple handhelds in a meetingMultiple handhelds in a meeting Pocket PC Pocket PC and and appliancesappliances

Multiple users with their devicesMultiple users with their devicesSingle user with multiple devicesSingle user with multiple devices

Early Pebbles WorkEarly Pebbles Work

For business meetings:For business meetings: For group workFor group work How laser pointers can be integratedHow laser pointers can be integrated

To augment desktop applicationsTo augment desktop applications For individualsFor individuals

For classroomsFor classrooms To make lectures more interactiveTo make lectures more interactive

For military command postsFor military command posts To facilitate communication and collaborationTo facilitate communication and collaboration

Recent Project:Recent Project: Personal Universal ControllerPersonal Universal Controller Using handhelds with appliances to improve the Using handhelds with appliances to improve the

user interfaceuser interface

Key FeaturesKey Features User interface-independent appliance specificationUser interface-independent appliance specification Automatic generation of GUI and speech interfacesAutomatic generation of GUI and speech interfaces

Specifications

Control

Feedback

Automatic Generation of UIsAutomatic Generation of UIs

BenefitsBenefits All interfaces consistent for the userAll interfaces consistent for the user

With conventions of handheldWith conventions of handheld Even from multiple manufacturersEven from multiple manufacturers

Multiple modalities (GUI + Speech UI)Multiple modalities (GUI + Speech UI)

Development HistoryDevelopment History

Visual C++ 5.0 Extensions for WinCEVisual C++ 5.0 Extensions for WinCE eMbedded Visual C++eMbedded Visual C++

Early Pebbles ApplicationsEarly Pebbles Applications

eMbedded Visual Basic 3.0 eMbedded Visual Basic 3.0 Early tests of Personal Universal Controller Early tests of Personal Universal Controller

(PUC) concept(PUC) concept

PersonalJava 1.1PersonalJava 1.1 Initial PUC implementationInitial PUC implementation

.NET Compact Framework.NET Compact Framework Current PUC implementationCurrent PUC implementation

OverviewOverview

IntroductionIntroduction

PUC ImplementationPUC Implementation

Advantages of .NET & .NET CFAdvantages of .NET & .NET CF

Limitations of .NET CFLimitations of .NET CF

Short DemoShort Demo

ResourcesResources

PUC ImplementationPUC Implementation

Implemented three interface Implemented three interface generatorsgenerators PocketPCPocketPC DesktopDesktop SmartphoneSmartphone

Implementation uses a large part of Implementation uses a large part of the .NET Compact Frameworkthe .NET Compact Framework UI ToolkitUI Toolkit GraphicsGraphics Networking & I/ONetworking & I/O XML ParsingXML Parsing

PUC System ArchitecturePUC System Architecture

CommunicationCommunication(802.11, Bluetooth, RF-Lite, etc.)(802.11, Bluetooth, RF-Lite, etc.)

PUC DevicesPUC Devices(automatic user (automatic user

interface generation)interface generation)

AdaptorsAdaptors(publishes description +(publishes description +

appliance state + controls appliance)appliance state + controls appliance)

XML-Based ProtocolXML-Based Protocol(two-way communication(two-way communicationof specification and state)of specification and state)

AppliancesAppliances(media players, cameras, etc.)(media players, cameras, etc.)

CommunicationCommunication(802.11, Bluetooth, RF-Lite, etc.)(802.11, Bluetooth, RF-Lite, etc.)

XML-Based ProtocolXML-Based Protocol(two-way communication(two-way communicationof specification and state)of specification and state)

Third-party componentsThird-party components Multi-platform componentsMulti-platform components .NET CF components.NET CF componentsLegend:Legend:

PUC Device ArchitecturePUC Device Architecture

ProtocolProtocolXML ParsersXML Parsers

<message><message> <state-change-notification><state-change-notification> <state>PlayMode</state><state>PlayMode</state> ......

NetworkNetworkTCP/IP SocketsTCP/IP Sockets

IPEndPoint end = new IPEndPoint(_ipAddress,_port);IPEndPoint end = new IPEndPoint(_ipAddress,_port);_socket.Connect(end);_socket.Connect(end);int len = _socket.Receive(num,4,SocketFlags.None);int len = _socket.Receive(num,4,SocketFlags.None);

Concrete InterfaceConcrete Interfaceglue to .NET CF Controlsglue to .NET CF Controls

Model Analysis &Model Analysis &Rule-Based DesignRule-Based Designanalysis of models (appliance & device),analysis of models (appliance & device),rule-based design of abstract interface,rule-based design of abstract interface,choose concrete controls & layoutchoose concrete controls & layout

Advantages of .NETAdvantages of .NET

Managed EnvironmentManaged Environment Garbage collectionGarbage collection

Modern Programming LanguagesModern Programming Languages Strong typesStrong types

Consistent, Well-Designed APIConsistent, Well-Designed API Object-orientedObject-oriented

Better documentationBetter documentation

Advantages of .NET CFAdvantages of .NET CF

For mobile application developmentFor mobile application development

Compatible with normal .NET Compatible with normal .NET FrameworkFramework Can develop for handheld and desktop Can develop for handheld and desktop

almost simultaneouslyalmost simultaneously

.NET CF being ported to multiple .NET CF being ported to multiple handheld platformshandheld platforms PocketPC, Smartphone 2003, WinCE 4.0?PocketPC, Smartphone 2003, WinCE 4.0?

Good performance on handheldsGood performance on handhelds

Advantages of .NET CFAdvantages of .NET CFCode PortabilityCode Portability

Portablility makes re-use very easyPortablility makes re-use very easy 90% from PocketPC to TabletPC*90% from PocketPC to TabletPC*

60% from PocketPC to Smartphone*60% from PocketPC to Smartphone* Different input model on Smartphone Different input model on Smartphone

requires different generator rules and requires different generator rules and several new widgetsseveral new widgets

* These numbers are estimates from development experience* These numbers are estimates from development experience

Advantages of .NET CF Advantages of .NET CF Versus JavaVersus Java

Solid, robust implementation on Solid, robust implementation on handheldshandhelds

Consistent programming interface Consistent programming interface across mobile platformsacross mobile platforms Java Micro Edition uses different Java Micro Edition uses different

APIs for Graphics and the UI toolkit APIs for Graphics and the UI toolkit from standard Javafrom standard Java

More features in the UI toolkitMore features in the UI toolkit More controls (vs. AWT)More controls (vs. AWT)

Limitations DisclaimerLimitations Disclaimer

I am not a Microsoft developer!I am not a Microsoft developer!

So, these limitations may have solutions I So, these limitations may have solutions I am not aware of:am not aware of:

I did not find the solutionsI did not find the solutions

They are fixed in more recent versions of They are fixed in more recent versions of the Compact Frameworkthe Compact Framework

Limitations of .NET CFLimitations of .NET CF

Compared to normal .NET Framework, CF has:Compared to normal .NET Framework, CF has: Fewer controlsFewer controls Fewer features in each controlFewer features in each control Can sub-class built-in controls, but…Can sub-class built-in controls, but…

No ability to owner drawNo ability to owner draw No ability to handle low-level eventsNo ability to handle low-level events

Must sub-class base Control class to make Must sub-class base Control class to make custom controlscustom controls

General Gripe: Controls do not know their own General Gripe: Controls do not know their own preferred or minimum size.preferred or minimum size.

Limitations of .NET CF Limitations of .NET CF continuedcontinued

Available CF controls are mostly adequateAvailable CF controls are mostly adequate Some custom controls neededSome custom controls needed

ImageButtonImageButton Time PickerTime Picker Scrolling Panel Scrolling Panel (Panels don’t have Auto Scroll)(Panels don’t have Auto Scroll)

Smartphone List ControlSmartphone List Control Smartphone Scrolling PanelSmartphone Scrolling Panel

Limitations of .NET CF Limitations of .NET CF continuedcontinued SmartphoneSmartphone

Still early in the design processStill early in the design process Navigation on screen based on Navigation on screen based on

Z-order and enabledness of controls.Z-order and enabledness of controls. Panels don’t automatically scroll, as Panels don’t automatically scroll, as

required by interface.required by interface. Back button can only be handled Back button can only be handled

through key-press events through key-press events (which can’t always be caught, and can’t (which can’t always be caught, and can’t be fully worked around)be fully worked around)

Scrollbars do not accept user inputScrollbars do not accept user input(we worked around this)(we worked around this)

Not many people working with phoneNot many people working with phone

Few resources when you have problems.Few resources when you have problems.

SummarySummaryAdvantages and LimitationsAdvantages and Limitations

Like any new set of APIs, there are bugsLike any new set of APIs, there are bugs

Overall, the good outweighs the badOverall, the good outweighs the bad

Rapid prototyping like Visual BasicRapid prototyping like Visual Basic

Power of modern language in C#Power of modern language in C# Easily able to share code across platforms Easily able to share code across platforms

(PocketPC, Desktop & Smartphone)(PocketPC, Desktop & Smartphone)

More than adequate performance on handheldsMore than adequate performance on handhelds

Little need to worry about memory or Little need to worry about memory or performance constraintsperformance constraints

PUC ImplementationPUC Implementation

Jeffrey NicholsJeffrey NicholsHuman Computer Interaction InstituteHuman Computer Interaction InstituteCarnegie Mellon UniversityCarnegie Mellon University

ResourcesResources

Usenet GroupsUsenet Groupsmicrosoft.public.dotnet.framework.compactframeworkmicrosoft.public.dotnet.framework.compactframework

www.opennetcf.orgwww.opennetcf.org

Books from Microsoft PressBooks from Microsoft Press.NET Compact Framework Core Reference*.NET Compact Framework Core Reference*

Visual C# .NET Step-By-StepVisual C# .NET Step-By-Step

Microsoft Visual C# .NET Language ReferenceMicrosoft Visual C# .NET Language Reference

Developer Labs (?)Developer Labs (?)Thanks Microsoft!Thanks Microsoft!

* Have not looked at this book.* Have not looked at this book.

PUC and the .NET CFPUC and the .NET CF

Brad A. Myers & Jeffrey NicholsBrad A. Myers & Jeffrey NicholsCarnegie Mellon UniversityCarnegie Mellon University

jeffreyn@cs.cmu.edujeffreyn@cs.cmu.eduhttp://www.pebbles.hcii.cmu.edu/puc/http://www.pebbles.hcii.cmu.edu/puc/

thanks!

Limitations of .NET CF Limitations of .NET CF continuedcontinued

Modifying any property of the UI from Modifying any property of the UI from outside the UI thread causes problems.outside the UI thread causes problems.

Example: Network thread receiving Example: Network thread receiving messages wants to set the value of a messages wants to set the value of a scrollbarscrollbar Application crashes 99% of the timeApplication crashes 99% of the time

Control.BeginInvoke method not Control.BeginInvoke method not implemented (yet?)implemented (yet?)

Must work-around by using Timers and Must work-around by using Timers and an event queue.an event queue.