android and openni - nui application treasure hunter robot

©SIProp Project, 2006-2012 11

Android and OpenNI - NUI Application Treasure Hunter Robot

Hirotaka Niisato@hirotakaster


About Myself

SpecialServer/Network Technology

Community (OSS)SIProp.org

CompanyGMO Internet, Inc


NUI - Natural User Interface


NUI Device


Device Specifications

Kinect

(Windows)

Xtion Pro Live

RGB Camera ○ ○

Depth Sensor ○ ○

Audio ○ ○

Commercial Use ○ ○

Library Kinect for

Windows SDK

OpenNI

(GPL/LPGL)


Shop


Robotics


others


OpenNI

Open Source

Environment

● OS:Windows/Linux(+ARM)/MacOS/Android

● Java/C++/C#

Modules

● OpenNI

● Sensor

● Middleware(NITE)


OpenNI with Android Application Pattern.1

USB


OpenNI with Android Application Pattern.2

USB NW

Android : Toshiba T-01C(Regza Phone)


Treasure Hunting Robot


Summary

This is an "AR(augmented reality) Treasure Hunting Game“

You get virtual treasures by controlling real robot!


Manual

Look at radar window like dragon radar.Show some treasure on radar.

Center is a place in which a robot is present.

N(North) is a robot's front.

If you move the robot to a treasure place, you get some point.

Controlling the robot is by brain wave.


How to control

Keep 3 plots/5 plots in same area.

Upper(exciting) areaTurn left

Middle(active) areaGo toward

Lower(relax) areaTurn right


How to make


NUI Devices


Hardwares & Softwares

HardwaresBase computer

Pandaboard

Brain Wave SensorMindWave

NeuroSky

Depth SensorXtion pro live

ASUS

DisplayAiRscoter

Brother Industries

Walking RobotKHR-3WL

Kondo science

SoftwaresUbuntu & Android

Linaro11.11

Depth SensorOpenNI

Bone skeleton trackerNITE for ARM

Only binary…

UI FrameworkopenFrameworks

Android


Hardwares’ photo


Softwares’ photo


Calculate robot position

Recognizing a robot by bone skeleton.Using OpenNI

Calculate direction & position of robot.Calculate by moving robot.

Using depth camera.


Recognizing bone skeleton

NITE is NOT OSS!Only binary for ARM/Ubuntu.

Want to use Android&openFrameworks UI.

Made dual boot OS for Ubuntu&Android.

Linaro Kernel for Ubuntu&Android on Pandaboard

UserLand: Linaro/Ubuntu (OpenNI runs on this userland)

UserLand: Linaro/Android (Run on Ubuntu by chroot)


Recognizing bone skeleton

NITE can NOT recognize our robot…Head is too SMALL as human.

Legs & arms are too THIN as human.

Upright pose is not like human.


Calculate robot position

Calculate a current center position.Moving distance is calculated by using depth camera.

Calculate a robot's front.This robot has just 3 ways action.

Go toward, Turn left, Turn right

Direction of “Go toward” is the robot's front.


How to control

Keep 3 plots in same area.

Upper(exciting) areaTurn left

Middle(active) areaGo toward

Lower(relax) areaTurn right


Brain wave 1/2

What wave is What feeling?γ(gamma) wave is exciting.

β(beta) wave is active.

α(alpha) wave is relax.

θ(theta ) wave is sleep.

δ(delta) wave is deep sleep.

ExcitingRelax

β γαθδ


Brain wave 2/2

Upper area γ(gamma) wave is exciting.

Middle areaβ(beta) wave is active.

Lower areaα(alpha) wave is relax.

ExcitingRelax

β γαθδ


Download source code

We release all source code on our site.http://www.siprop.org/ja/2.0/index.php?product%2FTreasureHuntingRobot

　 Do It Yourself ⇒Do It With Others!

http://www.siprop.org/ja/2.0/index.php?product/TreasureHuntingRobot




Thank you!