open standards for 3d and ar on the web - khronos group...javascript. javascript calls wasm modules...

© Copyright Khronos Group 2017 -

Open Standards for

3D and AR on the WebNeil Trevett | Khronos President

NVIDIA VP Developer Ecosystem

[email protected] | @neilt3d

mailto:[email protected]


Topics

1 2 3

Khronos and its

API ecosystem

Latest updates

relevant to 3D and

AR on the Web

Lessons from the

past and looking

to the future


Khronos Mission

Software

Silicon

Khronos is an International Industry Consortium of over 100 companies creating

royalty-free, open standard APIs to enable software to access hardware acceleration for

3D graphics, Virtual and Augmented Reality, Parallel Computing,

Neural Networks and Vision Processing


Khronos Standards Ecosystem

VR, Vision, Neural Networks- VR/AR system portability

- Tracking and odometry

- Scene analysis/understanding

- Neural Network inferencing

Real-time 2D/3D- Cross-platform gaming and UI

- VR and AR Displays

- CAD and Product Design

- Safety-critical displays

3D for the Web- Real-time apps and games in-browser

- Efficiently delivering runtime 3D assets

Parallel Computation- Machine Learning acceleration

- Embedded vision processing

- High Performance Computing (HPC)


Augmented Reality - Native Acceleration

Vision

sensor(s)

Tracking and

Positioning

Geometric scene

reconstruction

Semantic scene

understanding

(Neural Networks)

Generate Low Latency

3D Augmentations for

display by optical system

Interact with sensor, haptic

and display devices

Download 3D

augmentation object

and scene data

AR Application

How can we bring

ACCELERATED AR to

the Web?


Compact to Transmit

Fast to Load

Describes Full Scenes

Runtime Neutral

Open and Extensible


glTF Milestones

glTF 1.0 Spec

Ratified and

Released

Dec 2015

2012 thru 2014

Design Iteration and

Multiple

Implementations

Original motivation:

standardized way to

deliver 3D into

WebGL applications

Mar2017

glTF 2.0

Provisional spec for

industry freedback

Significant

Industry Adoption

June2017

We Are

Here!

Physically Based Rendering

for higher-quality materials

and rendering API

independence

glTF 2.0 Spec

Release

Microsoft joins

Specification Editors

Oct2016

Industry

Roadmap

Meeting

7 Months

glTF 2.0 launched today - here at Web3D!


What’s New in glTF 2.0• Physically Based Rendering (PBR) material definitions

- Material information stored in textures

• Graphics API neutral

- Proven by engine implementations using WebGL, Vulkan and Direct3D

- GLSL materials moved to extension for existing content and specialized use cases

• Morph Targets

- Enhanced animation system

• Improvements

- Binary glTF in core

- Dozens of refinements for enhanced performance

and a tighter, clearer specification

Sketchfab User: theblueturtlehttps://sketchfab.com/models/b81008d513954189a063ff901f7abfe4

https://sketchfab.com/models/b81008d513954189a063ff901f7abfe4


glTF 2.0 Scene Description Structure

.gltf (JSON)

Node hierarchy, PBR material textures, cameras

.binGeometry: vertices and indices

Animation: key-frames

Skins: inverse-bind matrices

.png

.jpg

...Textures

Texture based

PBR materials

Geometry


glTF 2.0 Scalable, Portable PBR• glTF 2.0 PBR Requirements

- Simple and inexpensive to implement - So can be everywhere – even mobile devices

- Scalable- Two combinable models – but materials continue to work even if just core supported

• Metallic-Roughness Material model

- baseColor — base color

- metallic — metalness

- roughness — roughness

- Simple to implement - mandated in core

• Specular-Glossiness Material model

- diffuse — reflected diffuse color

- specular — specular color

- glossiness — glossiness

- Slightly more resources - optional extension

Illustrations by Fraunhofer


Industry Transitioning to glTF 2.0• Breaking changes from 1.0 to 2.0 – but processing is streamlined and simplified

- NOT significant work and great benefits by upgrading to 2.0

• Industry moving quickly to glTF 2.0 — lots of early adopters

- BabylonJS, three.js, Cesium, xeogl, instant3Dhub

• gltf-pipeline integrating glTF 2.0 updates — has glTF 1.0 to glTF 2.0 translator

- Open source - use this to support both glTF 1.0 and 2.0 or move your users to 2.0

• Converters/Translators/Validators glTF 2.0 updates nearly ready

- COLLADA2GLTF and obj2gltf translators

- Khronos Validator and Gltf-test

• Samples and Tutorials

- glTF 2.0 sample models with PBR

- Extensive glTF tutorial series in draft

Move your pipeline to glTF 2.0

Its time!

https://github.com/AnalyticalGraphicsInc/gltf-pipeline/pull/191

https://github.com/KhronosGroup/COLLADA2GLTF/pull/20

https://github.com/AnalyticalGraphicsInc/obj2gltf

https://github.com/KhronosGroup/glTF-Validator

https://github.com/cx20/gltf-test

https://github.com/KhronosGroup/glTF-Sample-Models/tree/master/2.0#gltf-20-sample-models

https://github.com/KhronosGroup/glTF-Tutorials/


glTF Roadmap Discussions• Incrementally ship new functionality as extensions

- For testing out new features, or for long-term optional functionality

- glTF baseline needs to remain easy to process and deploy

• Mesh Compression

- Google Draco team

• Progressive Geometry Streaming

- Fraunhofer SRC

• Basis unified compressed texture format for transmission from Binomial

- Optimized transmission format with efficient local expansion to any GPU format

• Enhanced PBR

- E.g. NVIDIA MDL

• Point Clouds

- Generated by geometry capture

• Lighting Extension

- Enhanced lighting control

• Extensions for API and language specifics

- Optional hooks for enhanced perf/functionality

- Vulkan, DX12, Metal, GLSL, HLSL, SPIR-V, Metal C++

glTF 2.0 PBR Rendering - Image courtesy instant3Dhub / instantUV - Max Limper

Roadmap Meeting at SIGGRAPH

Let me know if you would like an invite


Khronos 3D APIs already

powering AR/VR Rendering

The standard for desktop VR today

On millions of mobile VR devices

Powering WebVR in browsers

The future of explicit, low-latency VR

Khronos APIs for AR/VR

But What About

INPUT Hardware for AR/VR..Device discovery

Multiple sensor tracking

Device Events

Haptics

Parameters for optics corrections

etc. etc…

OpenXR!Cross-Platform, portable AR/VR


OpenXR – Solving AR/VR Fragmentation

Proprietary

Engine

VR App

1

VR App

2

VRApp

4

VRApp

3

VR Device

1

VR Device

2

VR Device

3

VR Device

5

VR Device

4

After OpenXR Wide interoperabilityof

VR apps and devices

VR App

1

VR App

2

VRApp

4

VRApp

3

Proprietary

Engine

VR Device

1

VR Device

3

VR Device

5

VR Device

2

VR Device

4

Before OpenXRVR Market

Fragmentation

Device Layer

Application Interface


OpenXR Working Group Members

Design work started in December 2016

Typically 12-18 months to develop a V1.0 specification


OpenXR and VR Run-times – a Win-Win

VR Run-time

Application Interface

ProprietaryAPIs

Extensions

Device Layer

ProprietaryDriver

InterfacesExtensions

Access to any

OpenXR Application

Access to any

OpenXR Device

Any successful standard encourages and

enables healthy industry competition

OpenXR will not replace

VR run-times – or outlaw existing

interfaces

OpenXR will simply provide cross-vendor

APIs that can be exposed by a runtime to

access more apps and devices

OpenXR for portable

AR/VR apps and devices –

with initial focus on VR


API Adoption VectorsTo be relevant to the Web – a native API has to be available everywhere a Browser runs

i.e. key mobile and desktop OS

OpenGL and OpenGL ES

ships on every desktop and

mobile OS

OpenCL not adopted by

Android.

Apple switches to Metal

WebGL Next?

WebVX?

Apple not permitting Vulkan

to ship on macOS or iOS

OpenVX targeted at

embedded systems – not

desktop or mobile


Vulkan Explicit GPU Control

GPU

High-level Driver

AbstractionLayered GPU Control

Context managementMemory allocation

Full GLSL compiler

Error detection

ApplicationSingle thread per context

GPU

Thin DriverExplicit GPU Control

ApplicationMemory allocation

Thread managementSynchronization

Multi-threaded generation of command buffers

Multiple Front-end

CompilersGLSL, HLSL etc.

Loadable debug and

validation layers

Vulkan 1.0 provides access to

OpenGL ES 3.1 / OpenGL 4.X-class GPU functionality

but with increased performance and flexibility

SPIR-V pre-compiled shaders

Complex drivers

cause overhead

and inconsistent

behavior across

vendors

Always active

error handling

Full GLSL

preprocessor and

compiler in

driver

OpenGL vs.

OpenGL ES

Resource management

offloaded to app:

low-overhead, low-latency

driver

Consistent behavior:

no ‘fighting with driver

heuristics’

Validation and debug layers

loaded only when needed

SPIR-V intermediate

language: shading language

flexibility

Multi-threaded command

creation. Multiple graphics,

command and DMA queues

Unified API across all platforms with feature set

flexibility

Vulkan = high performance and

low latency 3D and GPU Compute

Ideal for VR/AR applications


Vulkan – Portable New Generation GPU API

Cross Platform

7

All Major GPU Companies shipping Vulkan Drivers – for Desktop and Mobile Platforms

Mobile, Embedded and Console Platforms Supporting Vulkan

Embedded LinuxAndroid TVAndroid 7.0 Nintendo Switch

Vulkan support

in Unity 5.6

Vulkan support in V5.4

Late May 2017

DOTA 2 on Vulkan

port of Source 2

DOOM shipping on

Vulkan port of id Tech 6

Initial Vulkan mobile

renderer in 4.12

Vulkan support

since V1.8

http://www.amd.com/


WebGL and WASM (Web Assembly)

C/C++ Code with

Native API calls

Emscripten

Clang/LLVM

JVM

WASM Modules

+ JS Glue Code

JavaScript

JavaScript

calls WASM

Modules

WASM modules

access Web APIs

(like WebGL)

through JS call-outs

For calling Web APIs in WASM Modules there needs to be: 1) a native API for WASM source code

2) a JS API for run-time invocation

3) a simple mapping between them to minimize JS Glue Code

JavaScript WebAPI

WebGL Next will need:1) a portable native GPU API that can run on any system

2) a JS binding of that API for use via JavaScript

3) a simple mapping between the two


Vulkan Portability – Call for Participation

‘WebGL Next’Lift ‘Vulkan Portability’ to

JavaScript and use in

WebAssembly native code

For Web nexgen graphics

and GPU compute

API

Overlap

Analysis

Portable ‘Vulkan

Subset’ API

Specification

A Portability Solution needs to address

APIs and shading languages

Open source compilers/translators for

shading and intermediate languages

MIR

DX IL

GLSL

MSL

HLSL

Vulkan Portability SolutionC/C++ Portability API Library

+ Shading Language tools

All open source

Use Feature

Sets to remove

non-portable

functionality

Use Extensions

to add

functionality

e.g. security and

robustness for

the Web

Vulkan is non-proprietary and is

already designed to be portable

No single API

on all systems

https://github.com/KhronosGroup/WebGLNext-Proposals/tree/master/Obsidian-Mozilla

Mozilla Obisidian ‘WebGL Next’ Proposal

https://github.com/KhronosGroup/WebGLNext-Proposals/tree/master/Obsidian-Mozilla


2007OpenGL ES 2.0

2012OpenGL ES 3.0

32-bit integers and floats

NPOT, 3D/depth textures

Texture arrays

Multiple Render Targets

Programmable

Vertex and

fragment shaders

2016

General Purpose Compute

Multi-threaded Rendering

Low-latency renderring

2011

WebGL 1.0

March 2017

WebGL 2.0Conformance Testing is vital for

Cross-Platform ReliabilityWebGL 2.0 conformance tests are very thorough

10x more tests than WebGL 1.0 tests

‘WebGL

Next’?Pervasive OpenGL ES 2.0OpenGL and OpenGL ES ship on

every desktop and mobile OS

WebGL Evolution

Vulkan

Portability

Apple

GPUWeb W3C

Community Group?

??

WebGL 2.0 now available

in Chrome/Firefox!

Edge/Safari: plan to ship

Already up to 50% of all

browsers

2014OpenGL ES 3.1Compute Shaders

Apple not shipping ES 3.1

Compute shader extension?


OpenVX - Portable, Efficient Vision Processing• OpenVX developers express a graph of vision and Neural Network operations (‘Nodes’)

- Using a C API

• Graphs isolate the programmer from hardware and enable significant optimizations

- Implementers can optimize – including node fusion and memory tiled caching

• Nodes can be executed on any hardware or processor coded in any language

- Including Vulkan for mobile and embedded OS

• BUT – native OpenVX is targeted at embedded – not desktop/mobile OS

- Perhaps a JavaScript Web graph API over Vulkan is still useful?

VisionNode

VisionNode

VisionNodeCNN Nodes

OpenVX graph mixing CNN nodes with traditional vision nodes

‘WebVX’??

Possible Native StackC API to Build Graph

Nodes accelerated by Vulkancompute shaders

Potential Web Stack??JS API to Build Graph

Nodes accelerated by Vulkancompute shaders


Why AR Needs Standard Acceleration APIs

Platform

Fragmentation

Silicon

Acceleration

Without API Standards With API Standards

Application

Portability

Everything

runs on CPU

Standard Acceleration APIs provide PERFORMANCE, POWER AND PORTABILITY


Consequences of Not Enough Acceleration


Layered API Ecosystem Examples

Application

Middleware

Browser

Native

WebVX?

Portability

3D Web Apps VR Web Apps AR Web Apps

High-level AR APIs similar to

proprietary platforms:

Android Tango, Apple ARKit?

Use

Case

Targ

ete

d

Div

ers

e O

pen S

ourc

e P

roje

cts

Sin

gle

im

ple

menta

tions

Genera

l Purp

ose

Defi

ned s

pecif

icati

on

Mult

iple

Im

ple

menta

tions

GPU Compute for Vision

and Machine Learning +

Rendering

General purpose vision

processing/Neural Net

for the Web – enables

Web innovation

In Discussion!!

= WebXR?


Augmented Reality - Web Acceleration

Vision

sensor(s)

Tracking and

Positioning

Geometric scene

reconstruction

Semantic scene

understanding

(Neural Networks)

Generate Low Latency

3D Augmentations for

display by optical system

Interact with sensor, haptic

and display devices

Download 3D

augmentation object

and scene data

AR Application

‘WebVX’

‘WebGL Next’


Please Get Involved!• Information on all these standards at Khronos

- www.khronos.org

• Khronos is driving to new levels of community engagement

- Specifications and resources increasingly open for industry input and feedback

•Any company or organization is welcome to join Khronos

- For a voice and a vote in any of these standards

• If joining is not possible – ask about a Khronos Advisory Panel

- Free of charge – enables design reviews, requirements and contributions

• Khronos is always open to liaison projects with Web3D

- The industry benefits from everyone working more closely

•Neil Trevett

- [email protected]

- @neilt3d

http://www.khronos.org/

open standards for 3d and ar on the web - khronos group...javascript. javascript calls wasm modules...

Documents