idea lab presentation

Using wearable motion sensors to recognize human gestures

Peng Deng, Qifeng Mao{pdeng,qmao}@students.csse.unimelb.edu.au

CSSE University of MelbourneLabSUM∑

2

Agenda

• Introduction– Wireless sensor network– Normal applications– WSN in HCI

• Our HCI project– Idea– How it works using Sun SPOT– Challenges and solutions

• Demos

• Future apps and comments

3

Introduction: Sensor Networks1. Deploy2. Network setup3. Query and response

User

Event

CommunicationProcessing Element

Sensing Element

P S O U W PE PR L

Y

SENSORS

ADC MICRO

PROCESSOR

MEMORYRADIO

REAL TIME OS

ALGORITHMS

Limited Lifetime

Require Supervision Slow

Processing

Limited Memory

1 kbps – 1 Mbps, 3 – 100 m, Lossy

Transmission

4

Applications

• Environmental monitoring

• Security

• Defence

• Bioinformatics and health

• Transportation management

• Chemical detection and emergency response

Pictures from [3]

5

Accelerometer based HCI

6

Our idea

• “Off the desk” interaction

• Multiple sensor nodes on body– 1*hand held � 2*wrists+1*waist+2*feet

• Recognize gestures � recognize actions

• Use different sensors (not limited to accelerometer) to interact with environment near by

7

Sun SPOT

8

Sun SPOT

• 3-axis accelerometer• Temperature sensor• Light sensor• LEDs• Analog inputs• Switches• General purpose I/O

Embedded sensorsEmbedded sensors

2.4 GHz IEEE 802.15.4 radiowith integrated antenna

RadioRadio

512K RAM/4M FlashMemoryMemory

180 MHz 32 bit ARM920TCPUCPU

32 uADeep sleepDeep sleep

720 mAh lithium-ion batteryBattery capacityBattery capacity

Sun SPOTPlatformPlatform

NetBeans 5.0IDEIDE

JavaProgramming LanguageProgramming Language

Sun Java Squawk VMFrameworkFramework

[7] [8]

9

Sun SPOT Applications [7]

10

3-axis Accelerometer

General purpose. Depends on developers’ implementation

Carefully tuned to optimize performance in determining hand and arm motion

PurposePurpose

250~30050Acceleration Noise DensityAcceleration Noise Density

850 uA @ 3.3 VSleep mode support

300 uA @ ~3VPower consumptionPower consumption

+/- 2G (600 mv/g)+/- 6G (200 mV/g)

+/- 3 G (300 mv/g)Range & SensitivityRange & Sensitivity

USD $10.82USD $8.97PricePrice

ST Microsystems LIS3L02AQAnalog Devices ADXL330 ChipChip

Sun SPOT [18]Wii Remote Controller [17]

Hzg /µ Hzg /µ

11

How it works

Base

Station

CU-

HTKApps

12

How it works cont.

• CU-HTK: a speech recognition toolkit– Supervised machine learning (HMM)

• Training phase– 5 samples per gesture

• Recognition phase– Find most similar trained gesture

13

Challenges and solutions

• Human gestures are inexact and vary every time– HMM

• Real time data stream segmentation– Manually– Threshold level– Machine Learning (DTW, SVM, LDA …)

14

Demos

Pictures from [15] [16]

Q & A

16

Peng DengMEDC Student

SUM Research LabCSSE University of Melbourne

pdeng@students.csse.unimelb.edu.au

Thank you