oculus rift dk2 + leap motion tutorial
TRANSCRIPT
![Page 1: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/1.jpg)
Oculus Rift DK2 + Leap Motion Unity
Tutorial
Chris Zaharia @chrisjz
![Page 2: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/2.jpg)
Content• Leap Motion VR• Hand tracking using Leap
SDK V2 (Skeletal Tracking)• Object interaction• Movement using Rift DK2
positional tracking
![Page 3: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/3.jpg)
Leap Motion SDK V2
•Currently in beta•Introduces skeletal tracking
![Page 4: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/4.jpg)
Leap Motion VR
![Page 5: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/5.jpg)
Augmented Reality• Image API• Infrared & Night Vision• Enable in Leap Motion Control Panel > Settings > General > Allow
Images• Leap Oculus Passthrough Unity Package• Next Leap Motion may have colour camera
![Page 6: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/6.jpg)
Requirements• IDE• Unity Pro
• Leap• Leap Motion Driver• Leap Motion V2 Tracking Beta (v.2.1.1.21671+)• Leap Motion V2 Skeletal Assets (beta)
• Rift DK2• Unity 4 Pro Integration
![Page 7: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/7.jpg)
Scope•Map physical hand and finger movements to 3D hand
model and physics•Grab physics objects (rigidbodies) by pinching•Movement and jump using DK2’s positional tracker
![Page 8: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/8.jpg)
Leap Motion - Unity File Structure• LeapMotion
• Materials• Blinn2.mat• Gloves.mat• Hands_Alt_blinn1.mat
• Models• HandsRealistic.fbx
• Prefabs• Hand Graphics
• RigidLeftHand.prefab• RigidRightHand.prefab
• Hand Physics• ThickRigidHand.prefab
• HandController.prefab
![Page 9: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/9.jpg)
• LeapMotion• …• Scripts
• Hand• FingerModel.cs• HandModel.cs• RiggedFinger.cs• RiggedHand.cs• RigidFinder.cs• RigidHand.cs• SkeletalFinger.cs• SkeletalHand.cs
• Tools• ToolModel.cs
• Utils• LeapRecorder.cs• LeapUnityExtensions.cs• MagneticPinch.cs
• HandController.cs
![Page 10: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/10.jpg)
• Plugins• [Include all files]
![Page 11: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/11.jpg)
Attach hands to player• Assumed that you’ve already integrated Rift with player• Is compatible with Sixense’s Razer Hydra integration via script (see
LeapHandExtendController.cs in tutorial)
![Page 12: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/12.jpg)
1. Create empty GameObject called “LeapHandController” and place under OvrCameraController.
2. Either attach the HandController prefab to game object or attach HandController.cs script.
3. Fill out variables with the following:
![Page 13: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/13.jpg)
Setup RigidLeftHand + RigidRightHand prefabs• Check if both prefabs are set as follows or change
accordingly.
MagneticPinch.cs script allows user to grab rigidbodies by pinching fingers
![Page 14: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/14.jpg)
For right hand, choose the [..]Arm3 mesh
Each finger must have the following variables, based on type of finger (i.e. index, pinky) and if it’s left (starts with L) or right (starts with R):
![Page 15: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/15.jpg)
Setup ThickRigidHand prefab• Gives the hand collisions and
physics• Can therefore push objects
![Page 16: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/16.jpg)
Magnetic Pinch Script• Allows user to grab closest object which has a rigidbody• Force Spring Constant sets the elasticity of the grip to object• Magnetic Distance determines how close an object must be from a
hand for it to be grabbed
Warning: There is a conflict between Magnetic Pinch and enabling the Left/Right Physics Models. You’d need to fix this conflict by making the magnetic script ignore the hand’s physics
![Page 17: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/17.jpg)
Grab Hand and Grabbable scripts
• As an alternative to Magnetic Pinch script• Assign GrabHand.cs to hand graphic prefab• Assign Grabbable.cs to rigidbody object to be grabbed
![Page 18: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/18.jpg)
Hand Physics
• Interact with objects in a realistic way• Grab small objects with one hand• Grab larger objects with multiple hands• Push/Pull other objects• Other possibilities
• Grabbable objects must have colliders
Currently, grasping objects with hands is still quite jittery. Future SDK or Leap hardware updates should improve on this hopefully.
![Page 19: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/19.jpg)
![Page 20: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/20.jpg)
Handle conflicts with other hand trackers• This approach will hide/disable Sixense hands if leap motion detects
leap hands in scene• Start by extending Leap Motion’s HandController.cs
![Page 21: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/21.jpg)
LeapHandExtendController.cs (1)using UnityEngine;using System.Collections;using Leap;
public class LeapHandExtendController : HandController { protected Controller leap_controller_;
protected void Awake () { leap_controller_ = new Controller();}
![Page 22: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/22.jpg)
LeapHandExtendController.cs (2) protected void LateUpdate () { if (leap_controller_ == null) return;
CheckIfHandsEnabled ();}
![Page 23: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/23.jpg)
LeapHandExtendController.cs (3) protected void CheckIfHandsEnabled () { Frame frame = leap_controller_.Frame(); HandList hands = frame.Hands; int num_hands = hands.Count;
if (num_hands > 0) { TriggerSixenseHands (false); } else { TriggerSixenseHands (true); } }
![Page 24: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/24.jpg)
Movement - DK2 Positional Tracking• Move/Jump by using DK2’s positional tracking• Move forward or backward by moving head in those directions• Either rotate or strafe sideways by moving head left/right• Jump by popping head upwards• Could crouch too by popping head downwards
• User should normally be positioned directly in front of DK2’s tracker
![Page 25: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/25.jpg)
Logic - Positional Tracking Movement• Create/modify your player input controller script, attached to player• Head position is calculated by subtracting the initial local position of the main
camera, of when scene is loaded, from its current position• Create configurable Vector3 variables for:
• Sensitivity – strength multiplier of movement or jump actions• Minimum – The minimum position that the Rift must be away from the centre position, for
the action to actually be triggered• Movement will use the X (-left/+right) and Z axis (+forward/-backward) in the
direction of the camera as a variable sent to the Character Motor’s inputMoveDirection variable• Jump will use the Y axis and will be mapped to the Character Motor’s inputJump
variable
![Page 26: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/26.jpg)
![Page 27: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/27.jpg)
Code – Positional Track Movement[RequireComponent(typeof(CharacterMotor))]
public class FPSInputController : MonoBehaviour {
…
public bool ovrMovement = false; // Enable move player by moving head on X and Z axis
public bool ovrJump = false; // Enable player jumps by moving head on Y axis upwards
public Vector3 ovrControlSensitivity = new Vector3(1, 1, 1); // Multiplier of positiona tracking move/jump actions
public Vector3 ovrControlMinimum = new Vector3(0, 0, 0); // Min distance of head from centre to move/jump
public enum OvrXAxisAction { Strafe = 0, Rotate = 1 }
public OvrXAxisAction ovrXAxisAction = OvrXAxisAction.Rotate; // Whether x axis positional tracking performs strafing or rotation
private GameObject mainCamera; // Camera where movement orientation is done and audio listener enabled
private CharacterMotor motor;
// OVR positional tracking, currently works via tilting head
private Vector3 initPosTrackDir;
private Vector3 curPosTrackDir;
private Vector3 diffPosTrackDir;
![Page 28: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/28.jpg)
Start + Update void Start() { … initPosTrackDir = mainCamera.transform.localPosition; }
void Update() { // Get the input vector from OVR positional tracking if (ovrMovement || ovrJump) { curPosTrackDir = mainCamera.transform.localPosition; diffPosTrackDir = curPosTrackDir - initPosTrackDir; }
![Page 29: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/29.jpg)
Update (continued)… if (ovrMovement) { if (diffPosTrackDir.x <= -ovrControlMinimum.x || diffPosTrackDir.x >= ovrControlMinimum.x) { if (ovrXAxisAction == OvrXAxisAction.Strafe) { diffPosTrackDir.x *= ovrControlSensitivity.x; } else { transform.Rotate(0, diffPosTrackDir.x * ovrControlSensitivity.x, 0); diffPosTrackDir.x = 0; } } else { diffPosTrackDir.x = 0; }
![Page 30: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/30.jpg)
Update (continued)… if (diffPosTrackDir.z <= -ovrControlMinimum.z || diffPosTrackDir.z >= ovrControlMinimum.z) { diffPosTrackDir.z *= ovrControlSensitivity.z; } else { diffPosTrackDir.z = 0; } directionVector = new Vector3(diffPosTrackDir.x, 0, diffPosTrackDir.z); }
![Page 31: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/31.jpg)
Update (continued)if (ovrJump) {
if (diffPosTrackDir.y > ovrControlMinimum.y) {
motor.inputJump = true;
} else {
motor.inputJump = false;
}
} else {
motor.inputJump = Input.GetButton ("Jump");
}
…
motor.inputMoveDirection = mainCamera.transform.rotation * directionVector; // performs actual movement
![Page 32: Oculus Rift DK2 + Leap Motion Tutorial](https://reader035.vdocuments.us/reader035/viewer/2022081513/58f9aadb760da3da068b7d77/html5/thumbnails/32.jpg)
}