Interactive STEM Learning with zSpace

Will Holderness, Sarah Buchanan, Dr. Joseph J. LaViola Jr.

Overview• Motivation

• Related Work

• System Overview

• Challenges

Motivation• Incorporate handwriting input on

zSpace Display

• Educational application – Math and Physics

• zSpace Display incorporates 3D visualization & interaction

Related Work• Physics Book: a sketch-based

interface for animating physics diagrams by Salman Cheema– Allows users to sketch diagrams and

equations and then animate the results using a 2D physics engine

• Math Pad: another sketch based math tutor by Dr. Joseph J. LaViola Jr.

Math PadProjectile Motion Example

Physics Book Examples

3D Physics Book• Our goal is to make a 3D version of

Physics Book using the zSpace display

• The zSpace stylus will work as pen and gestural input

• Unity 3D to create the environment and leverage existing 3D physics engine

System Components• Handwriting Recognition– StarPad handwriting recognition

• Runs as server in separate process• Recognizes mathematical equations, returns

results• Same engine as Physics Book

• Stylus stroke collection– Stylus position is tracked using IR

cameras

– Challenge: No touch input on the screen• Need to detect touch input by stylus up &

down

System Components• Mouse Emulation by Stylus

– Implemented by Jerry Tu and his team at zSpace

– Track Stylus optically and with onboard inertial sensors

– Taps can be detected as rapid changes in acceleration

– Combination of position, orientation, and tap-detection allows us to determine when stylus starts writing on screen and where it goes.

– Inject these measurements into events on the Windows mouse event queue • mouse down, mouse up, double-click, and position

System Components• 3D Stereo & Head Tracking– zSpace SDK provides Unity Prefabs for

3D Stereo & Stylus setup.

– Objects in scene must use realistic sizes for 3D to work properly. • For the object to appear 4inches from the

screen, it must be 4 inches from the virtual camera in Unity.

Math and Physics Lessons• Projectile Motion

• Kinetic vs. Potential Energy

• Torque

• 3D Graphing

Projectile Motion

Kinetic Energy

Torque

Graphing

Challenges

1. Taps and lift are difficult to detect

2. Not all .NET libraries compatible with Unity

3. Changing object size and location affected the stereo & head tracking experience

4. The stylus position is relative to the main camera’s position

Challenge – .NET Libraries in Unity• StarPad is based on F# – Not compatible

• Windows built in Ink & Recognition libs– Not compatible

• Sockets that allow us to connect to external servers– Easy and Fast!

– Doesn’t cause lag in Unity

Challenge – Stereo & Head Tracking• Motion Parallax incorrect

• Can cause the stylus pointer not to follow the stylus exactly

• The zSpace Unity SDK includes a number of variables such as World Scale– Still difficult to find exactly right values

• zSpace Diagnostic Window has sliders for each variable

• Plan scenes in advance

Challenge – It’s All Relative.• We wanted to be able to “pick up”

the camera with the stylus and move the viewpoint around

• The stylus position is relative to the camera position– This prevented us from simply moving

the camera the same delta distance as the stylus

Summary• Adding the zSpace package to Unity

was trivial– zSpace elements to a project is drag and

drop

• Handwriting is still a challenge

• Graphing turned out to be the most interesting/engaging lesson – promoted most exploration from users

Questions?