running, working outrunning, late for a meetinghigh altitude bikingwalking, shopping in car,...

Lisa Ong Rinku SreedharPrincipal Software Engineer Senior Program ManagerSensors Sensors

Building Rich, Contextually Aware Applications Using Sensors

2-735

Promise of Context AwarenessWindows 10 Sensor APIs Demos

Agenda slide

Promise of Context Awareness

Set of facts about users

Reduce explicit interaction, more

responsive

Advanced inferences through machine

learning

Running, working out

Running, late for a meeting High altitude biking Walking, Shopping

In car, Driving, Stopped

Sleeping

How Sensors enable Context Awareness

Are you near your device?

Are you driving? Or walking or biking?

Are you trying to track to a fitness goal?

Did you forget where you parked?

Introducing the Contextual Awareness Sensor API

WinRT API to detect proximity state, distance and range

• Supports Short and Long Range Sensors

• ApplicationsAutomatic display off during a phone

callWake on approachIn pocket detection Gestures

Proximity API

Proximity DemoSurface Hub

Sensors API namespaceusing Windows.Devices.Sensors;

Getting the default sensor (most sensor APIs only return one sensor on the system )

Barometer barometer = Barometer.GetDefault();

Getting the sensor (when there are multiple sensors – e.g.: proximity and Custom Sensors)

DeviceWatcher watcher = DeviceInformation.CreateWatcher(ProximitySensor.GetDeviceSelector());watcher.Added += OnProximitySensorAdded;. . . void OnProximitySensorAdded(DeviceWatcher sender, DeviceInformation device){ ProximitySensor sensor= ProximitySensor.FromId(device.Id);}

Intro to Sensors API Pattern using Proximity

Intro to Sensors API Pattern using Proximity

using Windows.Devices.Sensors;

Getting the current readingProximitySensorReading reading = sensor.GetCurrentReading();bool isDetected = reading.IsDetected;

Subscribing to reading changessensor.ReadingChanged += ReadingChanged;

void ReadingChanged(ProximitySensor s, ProximitySensorReadingChangedEventArgs e){ ProximitySensorReading reading = e.Reading; bool isDetected = reading.isDetected}

API to detect user’s motion context

Run, Walk, On bicycle, In vehicle, Fidgeting, Stationary, Idle, Unknown

Most likely activity with confidence

30 day history

Typical ApplicationsHealth & fitness trackingNavigation, Maps, Safe

drivingPower Saving

Activity Detection API

DemoMS Research Predestination Appusing Simulator

Using Activity Sensors and Background Triggers

Getting the current activityvar reading = await activitySensor.GetCurrentReadingAsync();

Subscribing to activity changesactivitySensor.ReadingChanged += new TypedEventHandler<ActivitySensor, ActivitySensorReadingChangedEventArgs>(ReadingChanged);

Getting history of up to 30 daysDateTimeOffset yesterday = ...var history = await ActivitySensor.GetSystemHistoryAsync(yesterday);foreach (var entry in history) { ... }

Using a background taskvar trigger = new Windows.ApplicationModel.Background.ActivitySensorTrigger(reportIntervalMs);

trigger.SubscribedActivities.Add(ActivityType.InVehicle);// .. register the trigger, etc..

API to detect users steps Number of steps – walking and running Active time spent walking and running 30 day history

Typical ApplicationsHealth & Fitness Tracking – no fitness gadget requiredCombine wearables and phone sensor data

Pedometer API

Click icon to add picture

Using Pedometer APIGetting the pedometer reading changes

pedometer.ReadingChanged += new TypedEventHandler<Pedometer, PedometerReadingChangedEventArgs>(ReadingChanged);

void ReadingChanged( Pedometer sender, PedometerReadingChangedEventArgs args){ PedometerReading reading = args.Reading; if (reading.StepKind == PedometerStepKind.Walking) walkingSteps = reading.CumulativeSteps;}

Getting pedometer step count historyvar history = await Pedometer.GetSystemHistoryAsync(yesterday);

Demo Pedometer on Desktop and Mobile

Barometer API reports barometric station pressure

Altimeter API reports relative altitude/ elevation

Typical Applications Health & fitness Floor Sensing and Indoor Navigation Weather Forecasting

Barometer / Altimeter API

Using Barometer/Altimeter APISame Sensor API patterns….

Barometer barometer = Barometer.GetDefault();BarometerReading reading = barometer.GetCurrentReading();

double pressure = reading.StationPressureInHectopascals;barometer.ReadingChanged += ...

Altimeter altimeter = Altimeter.GetDefault();AltimeterReading altimeterReading = altimeter.GetCurrentReading();

double altitudeChange = altimeterReading.AltitudeChangeInMeters;altimeter.ReadingChanged += ...

Select a Report Interval

mySensor.ReportInterval = 500;

Add a completely new Sensor Examples: CO2 Sensor, UV Sensor, Heart Rate Sensor…

Hardware vendors can introduce new Sensor Types independent of Microsoft OS releases

Simple, familiar API for reading sensor data

Custom Sensor API

Click icon to add picture

Demo UV Custom sensors on IoT

Extensibility using Custom SensorsStep 1: Driver defines and implements sensor IDs and keys

// Driver-defined IDs: UV sensor and UV level readingGuid UVSensorID = new Guid("4025a865-638c-43aa-a688-98580961eeae");const String UVLevelKey = "{74879888-a3cc-45c6-9ea9-058838256433} 1";

Step 2: App finds that custom sensor

const selector = CustomSensor.GetDeviceSelector(UVSensorID);watcher = Windows.Devices.Enumeration.DeviceInformation.CreateWatcher(selector);

… and consumes its data

CustomSensorReading reading = sensor.GetCurrentReading();if (reading.Properties.ContainsKey(UVLevelKey))

float UVLevel = reading.Properties[UVLevelKey];

sensor.ReadingChanged += …

Buffers sensor samples to a batch Client specifies the report latency

accelerometer.ReportLatency = accelerometer.MaxBatchSize * desiredReportInterval;

Typical Applications:- Power saving features - Background scenarios like Sleep and

Activity monitoring

Sensor Batching

Click icon to add picture

Developer Proposition Reuse Existing Android/ iOS Code to Build

UWP Apps

Reusing Android Code – Sensors Supported Activity Detection Screen Rotation Accelerometer Gyroscope MagnetometerProximity 

 Reusing iOS Code – Sensors Supported Accelerometer Gyroscope Magnetometer

Reusing Android and iOS app/code

Click icon to add picture

Windows Bridges

‘Project Astoria’ (Java/C++)

‘Project Islandwood’ (Objective C/C++)

Lumia SensorCore APIs – released in 2014 with Activity Tracking features on Lumia Phones

Windows 10 now supports these features on devices ranging from mobile, desktop, IOT

All future enhancement for Activity Detection and Pedometer features will be on the UWP APIs

Guidelines to developers :For new app development – use UWP APIs To target Legacy Lumia phones - use SensorCore APIs

(Note: Places and Track features of SensorCore is Out of Scope)

Lumia SensorCore API

Partner Demo MS Health

Additional Tips to knowReadingTransform• Rotates sensor data for you to a display orientation• 1 line change to port apps from portrait-first devices to landscape-first devices (vice-

versa)

Data is streamed consistently across platforms• UWP apps on multiple platforms get the same behavior for ReportInterval

Using Simulator to build and Test your app• Download universal driver samples from Github to emulate new sensor types

Privacy• Users can control privacy via Privacy Settings

• Attend Quick Start Challenge to start building Sensor apps

• Universal driver samples • UWP SDK samples• API Reference

Related sessions:Build 2014 Sensors Session: Sensors Platform Enhancements in WindowsBuild 2015 Sessions:1. 3-610 Compiling Objective-C Using the VS 2015 C++ code generation (that also builds Windows, SQL, .Net

and Office2. 2-702 “PROJECT ASTORIA“: Build great Windows apps with your Android code

Get Started

© 2015 Microsoft Corporation. All rights reserved.