long range gaze estimation with multiple near-infrared

Long Range Gaze Estimation with

Multiple Near-infrared Emitters

International Automatic Control Conference CACS 2017

Zhi-Xiang Liu, Zheng-Hong Ma, Ming-Chih Ho,

Jia-Yush Yen, Yung-Yaw Chen

智慧型微創手術內視鏡機器人系統研發-子計畫二：微創手術內視鏡機器人智慧型導航系統設計計畫編號：MOST 104-2221-E-002-136-MY3

智慧型微創手術內視鏡機器人系統研發-子計畫二：微創手術內視鏡機器人智慧型導航系統設計計畫編號：MOST 104-2221-E-002-136-MY3

1

Outline

Introduction

Related works

– 3D eye gaze estimation

– Interpolation based eye gaze estimation

Methods

– Process of algorithm

– Multiple NIRs compensation method

Experiment and Results

Conclusions

References

智慧型微創手術擴增實境系統研發-總計畫兼子計畫二:智慧型微創手術焦點區域即時影像系統

計畫編號：MOST 104-2221-E-002-136-MY3

2

Introduction

Gaze trackers capture human’s visual attention.

Applications:

– Human-computer interaction, human behavior research, eye

disease diagnosis…etc.

Difficulties in gaze estimation:

– degenerate accuracy under head motion

– large distance applications (larger than 1.0 m)

Goal:

– develop a long range eye signal control system that allow free

head motion.


計畫編號：MOST 104-2221-E-002-136-MY3

3

Based on geometric relationship.

Identify line of eye sight.

Advantage: flexible, free head motion

Disadvantage: complicated calibration

Related work of 3D method

Fig.1 3D eye model 智慧型微創手術擴增實境系統研發-總計畫兼子計畫二:智慧型微創手術焦點區域即時影像系統

計畫編號：MOST 104-2221-E-002-136-MY3

4

Related work of interpolation method

Interpolation gaze estimation consists of two major parts:

– Feature extraction

Pupil / Iris center

Reference point: corneal reflection

– Determine gaze mapping functions

Obtain mapping relation from image coordinate to screen .


計畫編號：MOST 104-2221-E-002-136-MY3

5

Overview of Gaze Estimation Algorithm

Methods


計畫編號：MOST 104-2221-E-002-136-MY3

6

Process 1: Pupil Extraction

Fig.2 Bright pupil image Fig.3 Dark pupil image

Fig.4 pupil contour 智慧型微創手術擴增實境系統研發-總計畫兼子計畫二:智慧型微創手術焦點區域即時影像系統

計畫編號：MOST 104-2221-E-002-136-MY3

7

Process 2: Corneal Reflections extraction

Flow Chart:


計畫編號：MOST 104-2221-E-002-136-MY3

8

Process 4: Mapping Function

Construct a mapping from image coordinate to screen

coordinate via PCCR (pupil center – corneal reflection)

vector.

The different mapping functions were investigated.

Second order polynomial is used as mapping function.

Ramanauskas, N. (2015)

Calibration technique:

1. Linear calibration model using 5 calibration points

2. Second order polynomial using 9 calibration points

3. Second order polynomial using 25 calibration points

4. Linear interpolation using 25 calibration points

5. Model based mapping

Averaged Mapping Error

(Second order polynomial)


計畫編號：MOST 104-2221-E-002-136-MY3

9

Calibration Procedure

Calibration points on screen.

C1

C4

C7

C2

C5

C8

C3

C6

C9

Calibration Points

Coordinate

C1 (𝐶𝑥1, 𝐶𝑦1)

C2 (𝐶𝑥2, 𝐶𝑦2)

C3 (𝐶𝑥3, 𝐶𝑦3)

C4 (𝐶𝑥4, 𝐶𝑦4)

C5 (𝐶𝑥5, 𝐶𝑦5)

C6 (𝐶𝑥6, 𝐶𝑦6)

C7 (𝐶𝑥7, 𝐶𝑦7)

C8 (𝐶𝑥8, 𝐶𝑦8)

C9 (𝐶𝑥9, 𝐶𝑦9)


計畫編號：MOST 104-2221-E-002-136-MY3

10

Multiple NIRs compensation method

Motivation: Using multiple NIRs to effectively reduce

the error under head motion situation.

Construct a mathematical for simulation.


計畫編號：MOST 104-2221-E-002-136-MY3

11


Two factors related to gaze point estimation error under

head motion circumstances:

– 1. Eye position

– 2. Gaze position.

Dividing the gaze range into 3 areas.


計畫編號：MOST 104-2221-E-002-136-MY3

12


Begin with estimating rough gaze point by using (G, D)

mapping function.

Choose the NIR and mapping function nearest to the

rough estimated gaze point , then compute the gaze point

again.


計畫編號：MOST 104-2221-E-002-136-MY3

13

Experiment Setting

9 calibration points and 16 testing points


計畫編號：MOST 104-2221-E-002-136-MY3

14

Experiment 1

Multiple NIR

Single NIR

MeanError(px)

82.36 87.58

std(px) 46.45 60.35

MeanError(cm)

2.22 2.36

MeanError

0.84 0.90

std 0.47 0.62


計畫編號：MOST 104-2221-E-002-136-MY3

15

Experiment 2


計畫編號:NSC 101-2221-E-002-146-MY3

16

Results

X MOVE FB(1st) NFB(1st) FB(2nd) NFB(2nd) FB(avg.) NFB(avg.) improve percentage(%)

NO MOVE 0.72 0.72 0.81 0.79 0.765 0.755 -1.324503311

8cm 0.9 0.88 0.75 0.84 0.825 0.86 4.069767442

16cm 1.05 1.23 0.94 1.09 0.995 1.16 14.22413793

24cm 1.42 1.21 1.21 1.01 1.315 1.11 -18.46846847

32cm 1.02 1.64 1.14 1.79 1.08 1.715 37.02623907

40cm 1.11 1.63 1.07 1.64 1.09 1.635 33.33333333

48cm 1.41 1.66 1.48 1.6 1.445 1.63 11.34969325

56cm 1.71 1.82 1.48 1.59 1.595 1.705 6.451612903

64cm 1.58 1.79 1.43 1.47 1.505 1.63 7.668711656

72cm 1.27 1.58 1.12 1.01 1.195 1.295 7.722007722

Overall AVG. 1.219 1.416 1.143 1.283 1.181 1.3495 12.48610597


計畫編號：MOST 104-2221-E-002-136-MY3

17 智慧型微創手術擴增實境系統研發-總計畫兼子計畫二:智慧型微創手術焦點區域即時影像系統

計畫編號:NSC 101-2221-E-002-146-MY3

18

Experiment 3


計畫編號:NSC 101-2221-E-002-146-MY3

19

Result


計畫編號:NSC 101-2221-E-002-146-MY3

Z MOVE(cm) FB(1st) NFB(1st) FB(2nd) NFB(2nd) FB(avg.) NFB(avg.) improve percentage(%)

0 0.67 0.51 0.61 0.53 0.64 0.52 -23.07692308

8 0.67 0.8 0.58 1.01 0.625 0.905 30.93922652

16 1.28 2.44 1.51 2.79 1.395 2.615 46.65391969

24 1.87 3.42 1.9 3.37 1.885 3.395 44.47717231

32 2.5 3.85 2.97 4.18 2.735 4.015 31.88044832

Overall avg. 1.398 2.204 1.514 2.376 1.456 2.29 26.17476875

20 智慧型微創手術擴增實境系統研發-總計畫兼子計畫二:智慧型微創手術焦點區域即時影像系統

計畫編號:NSC 101-2221-E-002-146-MY3

21

Conclusions

Our system successfully reduces the head motion

error about twenty percent.

Our system maintain the error within one degree over

the range of 16cm head movement in both x and z

direction.

We propose a new system configuration that can be

used in long rang and free head motion applications.


計畫編號：MOST 104-2221-E-002-136-MY3

22

References [1] Z. X. Liu, "A Novel Configuration and Algorithm of Long Distance Gaze

Estimation System for Gaze Point Error Compensation.", Department of Electrical

Engineering, College of Electrical Engineering and Computer Science, National

Taiwan University, Master Thesis, July 2016.

[2] Suh, K. H., Kim, Y.-J., Kim, Y., Ko, D., & Lee, E. C. (2015). Monocular Eye

Tracking System Using Webcam and Zoom Lens Advanced Multimedia and

Ubiquitous Engineering (pp. 135-141): Springer.

[3] Ramanauskas, N. (2015). Calibration of video-oculographical eye-tracking

system. Elektronika ir Elektrotechnika, 72(8), 65-68.

[4] Jixu Chen,Qiang Ji “3D Gaze Estimation with a Single Camera without IR

Illumination” ,2008 IEEE

計畫編號：MOST 104-2221-E-002-136-MY3 智慧型微創手術內視鏡機器人系統研發-子計畫二：微創手術內視鏡機器人智慧型導航系統設計

23

Thank you for your attention

long range gaze estimation with multiple near-infrared

Documents