forge - devcon 2016: collaborative vr using google cardboard & the view & data api
TRANSCRIPT
![Page 1: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/1.jpg)
Kean Walmsley
Platform Architect & Evangelist, Autodesk Research
Collaborative VR using Google Cardboard and the Forge Viewer
![Page 2: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/2.jpg)
@AutodeskForge#ForgeDevCon
![Page 3: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/3.jpg)
About me
• 20 years at Autodesk in 4 countries
• 10 years blogging at http://autode.sk/ttif
• 9 years tweeting at @keanw
• Current role is in Autodesk Research• Focused on IoT and VR
![Page 4: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/4.jpg)
To participate in the experiment…
![Page 5: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/5.jpg)
Your mobile phone will need…
Or search for “QR Code Reader”
![Page 6: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/6.jpg)
VR comes of age
![Page 7: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/7.jpg)
![Page 8: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/8.jpg)
![Page 9: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/9.jpg)
© 2016 Autodesk
Gartner Hype Cycle 1995
![Page 10: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/10.jpg)
© 2016 Autodesk
Gartner Hype Cycle 2015
![Page 11: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/11.jpg)
© 2016 Autodesk
![Page 12: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/12.jpg)
© 2016 Autodesk
![Page 13: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/13.jpg)
Using the Autodesk Viewer to implement VR
![Page 14: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/14.jpg)
The original application
• Web-page containing two viewer instances
• Camera-changed event maintains stereo effect
• Memory inefficient: better to use two viewports on a single viewer/scene
• View adjusts with device tilting
• HTML5’s deviceorientation event
• Gives handy roll, pitch & yaw
• http://autode.sk/gcb
![Page 15: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/15.jpg)
Navigation choices
• “Look around” or walkthrough viewer
• Sub-optimal for systems without parallax
• Really requires head-tracking
• Cybersickness: “vergence-accommodation conflict”
• “Widget viewer” / turntable navigation
• Sensible option for smaller models
• Made sense for this prototype
![Page 16: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/16.jpg)
VR has serious challenges around user input
• Opportunities around gesture-based input
• Leap Motion has an Oculus mount & Android SDK
• Webcam-based hand tracking may also be good enough
• Voice is often greatly overlooked
• In many ways a natural way to interact within VR
• No need to worry about displaying hands virtually
![Page 17: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/17.jpg)
Adding speech recognition
• Google Chrome has a Speech Recognition API
• Requires an internet connection
• Handy JavaScript wrapper called Annyang
• Easy to register commands for common operations
• EXPLODE, COMBINE, zoom IN and OUT
• FRONT, BACK, LEFT, RIGHT, TOP, BOTTOM
• http://autode.sk/gcbv (https://youtu.be/0csk5bK8Iyk)
![Page 18: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/18.jpg)
Using mobile VR SDKs with our web-based viewer
![Page 19: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/19.jpg)
![Page 20: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/20.jpg)
Android SDK for Google Cardboard
• Provides additional capabilities for Android apps
• Magnet-based trigger, uses the phone’s magnetometer
• Disables the display’s auto-off feature
• Also does view management (which we don’t need)
• Simple to use with Android Studio
• NFC can also be used to launch a Cardboard app
![Page 21: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/21.jpg)
![Page 22: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/22.jpg)
![Page 23: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/23.jpg)
Oculus Mobile SDK for the Samsung Gear VR
• Great optics – wider field of view
• Low latency/more responsive due to additional IMU
• It doesn’t use the phone’s
• Adds more UI capabilities
• Touchpad on the side + Bluetooth gamepad
• No longer a walled garden
• Samsung has a WebVR-capable browser
• Works with Samsung Galaxy Note 5 or S6/S7
• A bargain at $99 (plus phone)
![Page 24: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/24.jpg)
Calling into our code
• Native apps embedding WebViews can call JavaScript
mWebView.loadUrl("javascript:openSelected()");
• Same approach works for Cardboard or Gear VR apps
• Allows us to wire-up “hardware” events to our code
• http://autode.sk/gvr
![Page 25: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/25.jpg)
Daydream
• Latest entrant into the mobile VR space
• Announced at Google I/O 2016
• Based on Android N, requires high-end phone hardware
• Nexus 6P is the closest currently available
• A new Wiimote-like controller is mandated
• Could be a major step in democratizing VR
![Page 26: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/26.jpg)
Implementing collaborative features in web-based VR
![Page 27: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/27.jpg)
User input in VR is tricky, remember?
• Another approach is to let someone else guide you…
• Your teacher, professor, architect, engineer or doctor
• What if they could lead you/a group through a virtual environment?
• VR becomes more social, more personal
• Building walkthroughs, site flythroughs
• Google has “Expeditions” for virtual fieldtrips
![Page 28: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/28.jpg)
The Internet has a key role to play
• The web enables collaboration and inter-device comms
• Existing infrastructure makes this easy
• Forge Viewer is a great place to implement this
• Web-based, 3D visualization technology
• Pre-integrated with Google Cardboard (as we’ve just seen)
• The basis for Autodesk’s entry in last year’s VR Hackathonin San Francisco
![Page 29: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/29.jpg)
![Page 30: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/30.jpg)
The “VR Party” team’s project
• Allow a master web-page to control a number of clients
• Master page embeds the full “GUI” viewer
• Clients have a stereo view on the same model
• The master propagates certain operations
• Isolate, Show, Hide, Explode, Section and Zoom
• Importantly not camera/target position changes
![Page 31: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/31.jpg)
How does it work?
• Unique session created when master page loads
• Custom QR code presented for clients to scan
• GUI viewer events capture interesting operations
• Transmitted to the back-end via Web Sockets
• Passed on to connected clients
• Applied in the client’s viewer(s)
• Page close/disconnect cleans up session state
• Informs clients of disconnection
![Page 32: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/32.jpg)
Technologies
• HTML5, JavaScript, CSS3, jQuery
• Forge Viewer (Three.js & WebGL)
• Node.js
• Socket.io
• GitHub for source control
• Heroku for hosting
• CloudFlare for SSL, DNS, caching
![Page 33: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/33.jpg)
Vrok-It!
“Best Web-based VR Project” at the
SF VR Hackathon in May 2015
![Page 34: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/34.jpg)
50% will fail
http://vrok.it
![Page 35: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/35.jpg)
If you’re interested in going further…
• Try uploading your own models• http://www.vrok.it
• http://www.vrok.it/v2 (WIP with single viewer)
• Clone the source from GitHub• https://github.com/KeanW/vr-party
• Send feedback• [email protected] or @keanw
![Page 36: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/36.jpg)
Merten Stroetzel, from our Office of
the CTO, has taken this approach
and integrated it with VRED. He’s
showing it here in the Exhibit Hall.
![Page 37: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/37.jpg)
Future directions
• Tune the viewer for VR• Complete the single viewer work
• Focus on supporting larger models
• Improve WebVR support• Now uses pose info, not deviceorientation
• Test support with tethered headsets• Oculus Rift CR, HTC Vive Pre
![Page 38: Forge - DevCon 2016: Collaborative VR using Google Cardboard & the View & Data API](https://reader036.vdocuments.us/reader036/viewer/2022062522/5879a94b1a28ab082c8b77bd/html5/thumbnails/38.jpg)