the physical world meets the web

Maximiliano Firtman @firt

the Physical World meets the Web

mobile+web developer & trainer

Slides, links and tests firt.mobi/talks/physical

questions

yes, please

QA at the end

Let’s Start!

Agenda

1- Mobile and IoT 2- The Physical Web 3- Progressive Web Apps 4- Connecting through Web APIs 5- VR|MR Worlds and the Web

1- Mobile and IoT

The web comes to save us

1- Mobile and IoT

- Every gadget, sensor, device will have mobile support - We don’t want to install apps - We don’t want to make everything public

2- The Physical Web

The Physical Web is an open approach to enable quick and seamless interactions with physical objects and locations.

1- Open Spec 2- Beacons BLE 3- Also mDNS over WiFi

Goal: broadcast URLs

image from physical web website

Eddystone protocol

1- Buy beacons 2- Install “Beacon Toy” for Android 3- Also mDNS over WiFi

Start broadcasting URLs!

Chrome on Android (49+) Chrome on iOS

Current support

Live demo

2- The Physical Web

- Broadcast URL in physical places - Open standard - Anything can broadcast a URL

3- Progressive Web Apps

Progressive Web Apps

1- Open Definition 2- Embraced by Google 3- Followed by Firefox, Opera, Samsung and Microsoft

A Progressive Web App is a model for creating app-like experiences using latest Web technologies progressively

Features

1- Instant Loading 2- Discoverable 3- Network independent 4- Responsive 5- Installable

6- Secure 7- Linkable 8- Re-engageable 9- Works everywhere 10- Fast

progressive enhancement

1- It’s a website! 2- Adds native installation 3- Adds Web Push Notifications 4- Adds hardware & platform access

FUTUREPARTIAL

But they work anyway, anywhere!

Live demo

1- HTML5 2- Service Workers 3- Web App Manifest 4- Web Push Notifications 5- (optionally) Hardware APIs

3- Progressive Web Apps

- Best of native and web worlds - We can serve app-like experiences - The user can use it from the browser, or install it - Push messages for re-engagement

4- Web APIs

Device Sensors Web Audio Web Bluetooth

web apis

DEVICE SENSORS

1- Mostly useful on mobile devices 2- Use device’s hardware & sensors 3- Different Levels

a- Mature APIs b- Experimental APIs c- Future APIs

a- Mature APIs

Geolocation (GPS & WiFi) Accelerometer and Magnetometer Compass Camera (still picture)

Know more about these APIs…

Thursday 14.30 Breaking the Limits of the Web

b- Experimental APIs

Get User Media Ambient Light Speech Recognition and Synthesis WebMIDI

Get User Media

Get the stream video from the camera Recognize objects, faces, QR codes Augmented reality

<code>

navigator.getUserMedia( {video:true,audio:true}, function(stream){ var videoSource = URL.createObjectURL(stream); }, function(){ alert("Error getting video feed"); } );

JAVASCRIPT

<code>

navigator.getUserMedia( {video:true,audio:true}, function(stream){ var source = URL.createObjectURL(stream); }, function(){ alert("Error getting video feed"); } );

JAVASCRIPT

Get User Media

Live demo

Web Midi

Connect to music devices: synthesizers, keyboards, guitars, drum machines, lightning systems

Send and receive MIDI messages

<code>

navigator.requestMIDIAccess({sysex: false}) .then(onMIDISuccess, onMIDIFailure);

function onMIDISuccess(midi) { var inputs = midi.inputs.values(); for (var input=…) { input.value.addEventListener(“midimessage”, function(event) { // event.data has the bytes }); } }

JAVASCRIPT

<code>

JAVASCRIPT

<code>

JAVASCRIPT

<code>

JAVASCRIPT

Web Midi

Ambient Light

Get info about current light conditions Measures in lux

<code>

window.addEventListener('devicelight', function(event) { var lux = event.value; });

JAVASCRIPT

<code>

window.addEventListener('devicelight', function(event) { var lux = event.value; });

JAVASCRIPT

Live demo

Ambient Light

FLAG FUTURE

Speech Synthesis and Recognition

Let us interact with the user with voice Synthesis: let the Web speak Recognition: let the Web listen to user

<code>var u = new SpeechSynthesisUtterance(); u.rate = 1; u.volume = 1; u.pitch = 1; u.text = “Hello World”; u.lang = “en-GB”;

speechSynthesis.speak(u);

JAVASCRIPT

<code>recognition = new SpeechRecognition(); recognition.continuous = true; recognition.interimResults = true; recognition.addEventListener(“result”, function(e) { if (e.type==“result") { for (…) { var result = e.results[i][0]; var confidence = result.confidence*100); var text = result.transcript; } } };

JAVASCRIPT

<code>recognition = new SpeechRecognition(); recognition.continuous = true; recognition.interimResults = true; recognition.addEventListener(“result”, function(e) { if (e.type==“result") { for (…) { var result = e.results[i][0]; var confidence = result.confidence*100); var text = result.transcript; } } };

JAVASCRIPT

<code>recognition = new SpeechRecognition() recognition.continuous = true; recognition.interimResults = true; recognition.addEventListener(“result”, function(e) { if (e.type==“result") { for (…) { var result = e.results[i][0]; var confidence = result.confidence*100); var text = result.transcript; } } };

JAVASCRIPT

Live demo

Speech Synthesis

FLAG FUTURE

Speech Recognition

c- Future APIs

Geofencing WebUSB WebNFC

Geofencing

Get a web or PWA activation when user enters or leaves a geo area

WebUSB

Let us connect with any USB hardware connected to the device

WebNFC

Near Field Communication Access to Tap to Pay, Tap to Share, etc.

WEB AUDIO

1- Generate dynamic audio 2- Listens to low level audio 3- Useful for ultrasound communication with devices!

<code>

// Sending ssocket = new SonicSocket({alphabet: '0123456789'}); ssocket.send(‘31415');

// Receiving sserver = new SonicServer({alphabet: '0123456789'}); sserver.on('message', function(message) { // Expect message to be '31415'. console.log(message); }); sserver.start()

JAVASCRIPT

Using Sonicnet.js library

<code>

JAVASCRIPT

<code>

JAVASCRIPT

<code>

JAVASCRIPT

<code>

JAVASCRIPT

Live demo

WEB AUDIO

WEB BLUETOOTH

1- Scan BLE devices 2- Scan services available 3- Connect to these services 4- Send and receive data

WEB BLUETOOTH

Huge access for IoT Mixed with the Physical Web Better with Bluetooth 5.0

<code>navigator.bluetooth.requestDevice({ filters: [‘services’: ‘heart_rate’] }).then(function (device) { return device.gatt.connect(); }).then(function (server) { return server.getPrimaryService(“heart_rate”); }).then(function (service) { return service.getCharacteristic( “heart_rate_control_measurement”); }).then(function (ch) { return ch.startNotifications().then(function() { ch.addEventListener('characteristicvaluechanged', function() { event.target.value }); }); });;

JAVASCRIPT

<code>navigator.bluetooth.requestDevice({ filters: [‘services’: ‘heart_rate’] }) .then(device => { device.gatt.connect() }) .then(server => server.getPrimaryService(“heart_rate”); }) .then(service => service.getCharacteristic( “heart_rate_control_measurement”);}) .then(ch => return ch.startNotifications().then(() => { ch.addEventListener('characteristicvaluechanged', function() { event.target.value }); }); });

JAVASCRIPT

Heart Rate Sensor Demo

BB8 Controller

Drone controller

Live demo

WEB BLUETOOTH

4- Web APIs

- Device Sensor APIs: mature, experimental, future - Web Audio - Web Bluetooth - TV Streaming

5- VR|MR World and the Web

Virtual Reality Mixed Reality

new worlds

VR and the Web

Samsung Internet Browser for Gear VR Oculus Web Browser Third party browsers

Live demo

Access to VR devices in your browser (Oculus, Cardboard, Vive)

Experimental support Available as a polypill

BUILD DEV FLAG

DEVPOLYFILL POLYFILL

MR and the Web

Mixed Reality with Windows Holographic Hololens

Live demo

MR and the WebWe are in very early stages We need content to get out of the flat hologram

- We are in early stages - We will need new APIs and concepts - VR and MR mixed with Physical Web and Web Bluetooth

5- VR|MR World and the Web

the physical world meets the web

1- Mobile and IoT 2- The Physical Web 3- Progressive Web Apps 4- Connecting through Web APIs 5- VR|MR World and the Web

Foto de freefoto.com

[email protected]@firt

firt.mobi/hpmw firt.mobi/mh5

the physical world meets the web

Technology