it starts with igaze: visual attention driven networking ... · visual attention driven networking...
TRANSCRIPT
![Page 1: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/1.jpg)
It Starts with iGaze: Visual Attention Driven Networking
with Smart Glasses
Lan Zhang, Xiang-Yang Li, Wenchao Huang, Kebin Liu, Shuwei Zong, Xuesi Jian, Puchun Feng, Taeho Jung, Yunhao Liu
Tsinghua University, ChinaIllinois Institute of Technology, USA
University of Science and Technology of ChinaSuzhou Institute for Advanced Study, China
![Page 2: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/2.jpg)
You're beautiful, you're beautiful,
You're beautiful, it's true.
I saw your face in a crowded place,
And I don't know what to do,
'Cause I'll never be with you….
“You are beautiful”
![Page 3: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/3.jpg)
You're beautiful, you're beautiful,
You're beautiful, it's true.
I saw your face in a crowded place,
And I start an eye gaze with you,
I'll be a friend of you.
It Starts with iGaze
![Page 4: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/4.jpg)
Our iGaze Glasses
![Page 5: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/5.jpg)
Outline
•Background
•Challenges
•System Design
• Implementation
•Evaluation
•Future Work
![Page 6: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/6.jpg)
Next Biggest Thing: Wearable
2016, 170 millionwearable devices
2018:30 billion $ market
![Page 7: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/7.jpg)
Smart Glasses
• Cameras: world/eye camera• Processing and Storage Units• Sensors: GPS, gyroscope, accelerometer• Networking modules: Wi-Fi, Bluetooth
Smart Glasses
![Page 8: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/8.jpg)
Voice commands
Head gesture control
Hand control
• Photo and video capture• Social status updates• Navigation
• Augmented reality overlay
Existing Applications
User interactions Functionalities
![Page 9: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/9.jpg)
Voice commands
Head gesture control
Hand control
• Photo and video capture• Social status updates• Navigation• …
Existing Applications
User interactions Functionalities
Just head-mounted smart phone?
![Page 10: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/10.jpg)
Voice commands
Head gesture control
Hand control
• Photo and video capture• Social status updates• Navigation• …
Existing Applications
User interactions Functionalities
Just head-mounted smart phone?
Smart glasses are still looking for revolutionary applications!
![Page 11: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/11.jpg)
iGaze: New Networking Mode New Applications
Can understand the user's visual attention using an eye camera
Can automatically connect to the target of interest.
Can run on top of existing networking protocols, e.g., Wi-Fi.
![Page 12: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/12.jpg)
White shirt
Red tie...
Brown hair
Blue Tshirt...
Brown hair
Pink skirt...Orange sweater
Black hair...
Vision Plane
Device Plane
Vision Plane
Device Plane
e8:99:c4:9b:ab:56
e8:66:c3:7a:bc:32
e8:55:14:2c:36:ab
e8:29:44:3c:a6:18
e8:55:15:26:13:14
e8:99:c4:9b:ab:56
e8:37:21:a3:85:bc
“Store”
Red-orange Canopy
Brown brick wall
...
Bidirectional Unidirectional
Application Modes of iGaze
Both users wear smart glasses.
Social networking scenario.
One user wears smart glasses.
Object-oriented augment reality.
![Page 13: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/13.jpg)
Social
Po
ten
tia
l A
pp
lica
tio
ns
People can exchange information such as interests by simply looking at each other.
![Page 14: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/14.jpg)
Museum
Po
ten
tia
l A
pp
lica
tio
ns
Visitors can obtain the detailed descriptions of art works by glazing at it instead.
![Page 15: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/15.jpg)
Device Control
Po
ten
tia
l A
pp
lica
tio
ns
A house owner can turn on a smart appliance only by taking a glance at them.
![Page 16: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/16.jpg)
Advertising
Po
ten
tia
l A
pp
lica
tio
ns
When people look at a signboard or product, relevant promotion information can be sent to the potential customer immediately.
Targeted ads big impulse buying market
![Page 17: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/17.jpg)
Image recognition
QR Code Recognition
Scanning distance < 3mScanning angle <30 °
Extensive training
High computation cost
Vulnerable to environment noise
Existing Methods
![Page 18: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/18.jpg)
Image recognition Hand gesture based
Point&Connect Spartacus[MobiSys 09’] [MobiSys 13’]
Ranging based.Displacement > 20cm.
FFT based.Speed: 2~6 m/s.
Existing Methods
![Page 19: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/19.jpg)
Image recognition Hand gesture based
Point&Connect Spartacus
Ranging based.Displacement > 20cm.
FFT based.Speed: 2~6 m/s.
Existing Methods
Targeting handhold smart devices.Hard to achieve by head mounted devices.
![Page 20: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/20.jpg)
Outline
•Background
•Challenges
•System Design
• Implementation
•Evaluation
•Future Work
![Page 21: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/21.jpg)
White shirt
Red tie...
Brown hair
Blue Tshirt...
Brown hair
Pink skirt...Orange sweater
Black hair...
Vision Plane
Device Plane
Vision Plane
Device Plane
e8:99:c4:9b:ab:56
e8:66:c3:7a:bc:32
e8:55:14:2c:36:ab
e8:29:44:3c:a6:18
e8:55:15:26:13:14
e8:99:c4:9b:ab:56
e8:37:21:a3:85:bc
“Store”
Red-orange Canopy
Brown brick wall
...
Vision Plane
Device Plane
![Page 22: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/22.jpg)
White shirt
Red tie...
Brown hair
Blue Tshirt...
Brown hair
Pink skirt...Orange sweater
Black hair...
Vision Plane
Device Plane
Vision Plane
Device Plane
e8:99:c4:9b:ab:56
e8:66:c3:7a:bc:32
e8:55:14:2c:36:ab
e8:29:44:3c:a6:18
e8:55:15:26:13:14
e8:99:c4:9b:ab:56
e8:37:21:a3:85:bc
“Store”
Red-orange Canopy
Brown brick wall
...
Vision Plane
Device Plane
![Page 23: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/23.jpg)
White shirt
Red tie...
Brown hair
Blue Tshirt...
Brown hair
Pink skirt...Orange sweater
Black hair...
Vision Plane
Device Plane
Vision Plane
Device Plane
e8:99:c4:9b:ab:56
e8:66:c3:7a:bc:32
e8:55:14:2c:36:ab
e8:29:44:3c:a6:18
e8:55:15:26:13:14
e8:99:c4:9b:ab:56
e8:37:21:a3:85:bc
“Store”
Red-orange Canopy
Brown brick wall
...
Vision Plane
Device Plane
Gap
![Page 24: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/24.jpg)
White shirt
Red tie...
Brown hair
Blue Tshirt...
Brown hair
Pink skirt...Orange sweater
Black hair...
Vision Plane
Device Plane
Vision Plane
Device Plane
e8:99:c4:9b:ab:56
e8:66:c3:7a:bc:32
e8:55:14:2c:36:ab
e8:29:44:3c:a6:18
e8:55:15:26:13:14
e8:99:c4:9b:ab:56
e8:37:21:a3:85:bc
“Store”
Red-orange Canopy
Brown brick wall
...
Vision Plane
To accurately capture the vision plane attention of a user in real time
Goal
![Page 25: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/25.jpg)
White shirt
Red tie...
Brown hair
Blue Tshirt...
Brown hair
Pink skirt...Orange sweater
Black hair...
Vision Plane
Device Plane
Vision Plane
Device Plane
e8:99:c4:9b:ab:56
e8:66:c3:7a:bc:32
e8:55:14:2c:36:ab
e8:29:44:3c:a6:18
e8:55:15:26:13:14
e8:99:c4:9b:ab:56
e8:37:21:a3:85:bc
“Store”
Red-orange Canopy
Brown brick wall
...
Vision Plane
Challenge
• One low-resolution camera.• No training and manual calibration.• No restriction on users’ movement.• Low computing and energy cost.
![Page 26: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/26.jpg)
White shirt
Red tie...
Brown hair
Blue Tshirt...
Brown hair
Pink skirt...Orange sweater
Black hair...
Vision Plane
Device Plane
Vision Plane
Device Plane
e8:99:c4:9b:ab:56
e8:66:c3:7a:bc:32
e8:55:14:2c:36:ab
e8:29:44:3c:a6:18
e8:55:15:26:13:14
e8:99:c4:9b:ab:56
e8:37:21:a3:85:bc
“Store”
Red-orange Canopy
Brown brick wall
...
Device Plane
Find another identifier of devices good for matching
Candidate: device direction
Goal
![Page 27: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/27.jpg)
White shirt
Red tie...
Brown hair
Blue Tshirt...
Brown hair
Pink skirt...Orange sweater
Black hair...
Vision Plane
Device Plane
Vision Plane
Device Plane
e8:99:c4:9b:ab:56
e8:66:c3:7a:bc:32
e8:55:14:2c:36:ab
e8:29:44:3c:a6:18
e8:55:15:26:13:14
e8:99:c4:9b:ab:56
e8:37:21:a3:85:bc
“Store”
Red-orange Canopy
Brown brick wall
...
Device Plane
Convenient user controlAccuracy, Delay, Energy cost
Easy 1-1 mapping with users in vision plane!
Challenge
![Page 28: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/28.jpg)
Principle of Gaze-based Networking
Observation 1
Given an observer and his/her gaze vector, only the device vector to the correct visual target’s device is consistent with the gaze vector.
![Page 29: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/29.jpg)
Principle of Gaze-based Networking
Observation 2
Given a pair of users who are looking at each other, their gaze vectors have opposite directions.
![Page 30: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/30.jpg)
White shirt
Red tie...
Brown hair
Blue Tshirt...
Brown hair
Pink skirt...Orange sweater
Black hair...
Vision Plane
Device Plane
Vision Plane
Device Plane
e8:99:c4:9b:ab:56
e8:66:c3:7a:bc:32
e8:55:14:2c:36:ab
e8:29:44:3c:a6:18
e8:55:15:26:13:14
e8:99:c4:9b:ab:56
e8:37:21:a3:85:bc
“Store”
Red-orange Canopy
Brown brick wall
...
Solution
• Develop a low-cost glasses hardware• embedded with our attention capture software,• capture the gaze direction using the orientation
sensor and eye camera.
SolutionV
isio
n
Pla
ne
Dev
ice
Pla
ne • Capture the direction between glasses
• leveraging Doppler effect caused by arbitrary mild movement of head-mounted speakers.
![Page 31: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/31.jpg)
Outline
•Background
•Challenges
•System Design
• Implementation
•Evaluation
•Future Work
![Page 32: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/32.jpg)
Hardware & Software Architecture
Real-time Eye Tracking
Gaze Direction Determination
Play Stereo Sound
Phase TrackingDevice Direction
Determine
SoftwareHardware
Social Device Control Targeted Ads Museum Reality Game
Direction Matching
AttentionAcquisition
Eye CameraStereo
Speakers
Orientation Sensor
Microphone
![Page 33: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/33.jpg)
Hardware & Software Architecture
Real-time Eye Tracking
Gaze Direction Determination
Play Stereo Sound
Phase TrackingDevice Direction
Determine
SoftwareHardware
Social Device Control Targeted Ads Museum Reality Game
Direction Matching
AttentionAcquisition
Eye CameraStereo
Speakers
Orientation Sensor
Microphone
Gaze vector acquisition
![Page 34: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/34.jpg)
Hardware & Software Architecture
Real-time Eye Tracking
Gaze Direction Determination
Play Stereo Sound
Phase TrackingDevice Direction
Determine
SoftwareHardware
Social Device Control Targeted Ads Museum Reality Game
Direction Matching
AttentionAcquisition
Eye CameraStereo
Speakers
Orientation Sensor
Microphone
Device vector estimationDevice vector estimation
![Page 35: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/35.jpg)
Hardware & Software Architecture
Real-time Eye Tracking
Gaze Direction Determination
Play Stereo Sound
Phase TrackingDevice Direction
Determine
SoftwareHardware
Social Device Control Targeted Ads Museum Reality Game
Direction Matching
AttentionAcquisition
Eye CameraStereo
Speakers
Orientation Sensor
Microphone Matching and Networking
![Page 36: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/36.jpg)
Hardware & Software Architecture
Real-time Eye Tracking
Gaze Direction Determination
Play Stereo Sound
Phase TrackingDevice Direction
Determine
SoftwareHardware
Social Device Control Targeted Ads Museum Reality Game
Direction Matching
AttentionAcquisition
Eye CameraStereo
Speakers
Orientation Sensor
Microphone
Application Layer
![Page 37: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/37.jpg)
Visual Attention Networking (VAN) Protocol
![Page 38: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/38.jpg)
Networking Protocol
![Page 39: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/39.jpg)
Networking Protocol
![Page 40: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/40.jpg)
Networking Protocol
Time Time
Initiator Neighbors
Issue connection request
Emit acoustic signalwith head gesture
Match gaze direction Gaze match time window
Open microphone
Match device direction
Establish connection
![Page 41: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/41.jpg)
System Design
Real-time Eye Tracking
Gaze Direction Determination
Play Stereo Sound
Phase TrackingDevice Direction
Determine
SoftwareHardware
Social Device Control Targeted Ads Museum Reality Game
Direction Matching
AttentionAcquisition
Eye CameraStereo
Speakers
Orientation Sensor
MicrophoneGaze Vector Acquisition
Eye image
• Real-time eye tracking: captures the movement of a user’s eye.
• Attention acquisition: detect a visual attention when the gaze lasts for a reasonable time.
• Gaze vector determination: calculates the corresponding gaze vector to the visual target.
![Page 42: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/42.jpg)
Visual Attention Detection
• Two basic eye movement statuses: fixation and saccade.
• visual attention - eye fixation.
• We use • an eye-movement velocity
threshold to detect the fixation• a fixation duration window threshold
for visual attention
Real-time Eye Tracking
![Page 43: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/43.jpg)
Visual Attention Detection
• Continuous image capturing and analysis could be power and computation intensive.
• To further reduce the cost, we detect head fixation using gyro data before invoking the eye camera.
Gyro data of head movement.
On-demand Eye Tracking
![Page 44: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/44.jpg)
Determine Gaze Direction
rOe
Ze
Yc
Xc
φ a b
Zc
Oi Xe
Ye
Ix
Iy
Image Plane
Gaze
Direction
Camera Coordinate
Eye Coordinate
Iris Plane
Projection Model Basic Idea
• The iris contour is a circle, but its projection on the image plane is elliptical.
• When look straight ahead, the projection looks more like a circle;
• When look off to one side, it looks more close to an ellipse.
• We can estimate the pose of the iris circle by back-projecting the ellipse onto a circle in 3D space.
![Page 45: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/45.jpg)
Determine Gaze Direction
Projection Model Basic Idea
• The iris contour is a circle, but its projection on the image plane is elliptical.
• When look straight ahead, the projection looks more like a circle;
• When look off to one side, it looks more close to an ellipse.
• We can estimate the pose of the iris circle by back-projecting the ellipse onto a circle in 3D space.
The challenge is that with only a single elliptical image, there are many circles satisfying the projection cone.
Need to remove ambiguities caused by single camera!!
![Page 46: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/46.jpg)
Estimate Device Vector
Point&Connect Spartacus[MobiSys 09’] [MobiSys 13’]
Ranging based.Displacement > 20cm.
FFT based.Speed: 2~6 m/s.
The main challenge is how to determine accurate device direction with only mild head gesture?
![Page 47: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/47.jpg)
System Design
Real-time Eye Tracking
Gaze Direction Determination
Play Stereo Sound
Phase TrackingDevice Direction
Determine
SoftwareHardware
Social Device Control Targeted Ads Museum Reality Game
Direction Matching
AttentionAcquisition
Eye CameraStereo
Speakers
Orientation Sensor
MicrophoneDevice Vector Estimation
Stereo Acoustic Signal
Accel & Gyro
• Use two head-mounted speakers, based on the Phase Locked Loop (PLL) technique.
• The measuring accuracy of the relative displacement is less than 1mm.
![Page 48: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/48.jpg)
Makes a head gesture (e.g., head nod, shake).
Two speakers play two sine inaudible waves at different frequencies.
The receiver can track the precise phase of the received signal.
The phase shift is in proportion to relative displacement.
Obtain the relative direction of the receiver to the initiator via therelative displacement.
Make head gesture Get relative displacement Get relative direction
Device Direction Estimation
![Page 49: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/49.jpg)
Makes a head gesture (e.g., head node).
Two speakers play two sine inaudible waves at different frequencies.
The receiver tracks the precise phase of the received signal by PLL.
The phase shift is in proportion to relative displacement.
Obtain the relative direction of the receiver to the initiator via therelative displacement.
Make head gesture Get relative displacement Get relative direction
Device Direction Estimation
![Page 50: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/50.jpg)
Makes a head gesture (e.g., head node).
Two speakers play two sine inaudible waves at different frequencies.
The receiver tracks the precise phase of the received signal by PLL.
The phase shift is in proportion to relative displacement.
Obtain the relative device direction via the relative displacements of two speakers.
Make head gesture Get relative displacement Get relative direction
Device Direction Estimation
![Page 51: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/51.jpg)
Outline
•Background
•Challenges
•System Design
•Implementation
•Evaluation
•Future Work
![Page 52: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/52.jpg)
• We design and implement a low cost smart glasses.
• We are keeping improving and upgrading our hardware design (4 different versions now) Upgrade
Hardware Specifications
![Page 53: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/53.jpg)
Eye CameraStereo
Speakers
Orientation Sensor
Microphone
Hardware Specifications
Now supports smart-phone pairing
![Page 54: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/54.jpg)
Software Specifications
• We implement two versions of all software blocks of iGazefor both Linux and Android using C++ and Java.
• It supports both unidirectional and bidirectional application modes.• social application and smart device application
• Visual components are developed based on OpenCV library.
• Acoustic signal: 19kHz and 19.5kHz.
![Page 55: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/55.jpg)
Outline• Background
• Challenges
• System Design
• Implementation
• Evaluation
• Future Work
![Page 56: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/56.jpg)
• 47 inch screen.• Marks on the screen: 5° resolution.• 30 Volunteers: 12 f, 18 m• For each volunteer, about 100 gaze data are collected.
Gaze Experiment Setting
![Page 57: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/57.jpg)
Attention & Gaze Direction Accuracy
False rate vs. Time window• Time-Window
• Select 0.6s when false positive and negative equal
False rate vs. Users• Eye Moving Speed
• We set 20 pixel/s as the eye-moving speed threshold.
• Attention capture accuracy: ~95%
• Gaze direction accuracy: • <5° (91%)
• <10° (99%)
![Page 58: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/58.jpg)
Device Direction Accuracy
Head nod and shake
Accuracy vs. Amplitude
• High accuracy (<4°) can be achieved by mild head gesture (30 °).
Accuracy vs. Number
• Increasing head nod number does not improve accuracy.
• Direction estimation is robust against different head gestures and patterns.• Only one mild head gesture is sufficient for highly accurate estimation.
![Page 59: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/59.jpg)
Device Direction Accuracy
Accuracy vs. Relative angle•For the head shake gesture, the mean
error increases from 6° to 10°.•Head nodding suffers less from NLoS
effect, and its mean error remains <7°.
Accuracy vs. Distance•When the distance gets larger, the
accuracy decreases.
•Error is <5° within 4 meters & < 9 °error within 10 meters.
![Page 60: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/60.jpg)
Computation Cost &Energy Consumption
• 30 ms to process one eye image.• 200ms to process the one second
acoustic signal and gyro.• Other computations are negligible.
Computation Cost Energy Cost
![Page 61: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/61.jpg)
• Considering the human factors, in most cases a connection can be built within about 3 seconds from the initiator starts to pay attention.
• In two cases, with a proper threshold, false positive is less than 3% while the false negative is about 15%.
Case Study
![Page 62: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/62.jpg)
Outline• Background
• Challenges
• System Design
• Implementation
• Evaluation
• Future Work
![Page 63: It Starts with iGaze: Visual Attention Driven Networking ... · Visual Attention Driven Networking with Smart ... Suzhou Institute for Advanced Study, ... It Starts with iGaze: Visual](https://reader030.vdocuments.us/reader030/viewer/2022020303/5b8716547f8b9a28238c0697/html5/thumbnails/63.jpg)
• Intention Capture and Understanding.dynamics: multiple objects in the gaze direction, multiple users may be
interested in a same object, a user may be interested in a moving object…
• Privacy and security.How to avoid insecure connection.
• Performance improvements using a hardware-only instrumentation.
• Aesthetic design.
Fu
ture
Wo
rk