![Page 1: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/1.jpg)
Non-invasive Techniques for Human Fatigue
MonitoringQiang Ji
Dept. of Electrical, Computer, and Systems Engineering
Rensselaer Polytechnic [email protected]
http://www.ecse.rpi.edu/homepages/qji Funded by AFOSR and Honda
![Page 2: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/2.jpg)
![Page 3: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/3.jpg)
Visual Behaviors
Visual behaviors that typically reflect a
person's level of fatigue include– Eyelid movement – Head movement – Gaze – Facial expressions
![Page 4: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/4.jpg)
Eye Detection and Tracking
![Page 5: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/5.jpg)
Eye Detection
![Page 6: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/6.jpg)
Eye Tracking Develop an eye tracking technique
based on combining mean-shift and Kalman filtering tracking.
It can robustly track eyes under different face orientations, illuminations, and large head movements.
![Page 7: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/7.jpg)
![Page 8: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/8.jpg)
Eyelid Movements Characterization
Eyelid movement parameters
Percentage of Eye Closure (PERCLOS)
Average Eye Closure/Open Speed (AECS)
![Page 9: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/9.jpg)
![Page 10: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/10.jpg)
Gaze (Pupil Movements)
Real time gaze tracking Develop a real time gaze tracking
technqiue. No calibration is needed and
allows natural head movements !.
![Page 11: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/11.jpg)
Gaze Estimation
Gaze is determined by Pupil location (local gaze)
Local gaze is characterized by relative positions between glint and pupil.
Head orientation (global gaze) Head orientation is estimated by pupil
shape, pupil position, pupil orientation, and pupil size.
![Page 12: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/12.jpg)
![Page 13: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/13.jpg)
Gaze Parameters Gaze spatial distribution over time
PERSAC-percentage of saccade eye movement over time
![Page 14: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/14.jpg)
Gaze distribution over time while alert
![Page 15: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/15.jpg)
Gaze distribution over time while fatigue
![Page 16: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/16.jpg)
Gaze distribution over time for inattentive driving
![Page 17: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/17.jpg)
Plot of PERSAC parameter over 30 seconds.
![Page 18: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/18.jpg)
![Page 19: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/19.jpg)
Head Movement Real time head pose tracking
Perform 3D face pose estimation from a single uncalibrated camera.
Head movement parameters Head tilt frequency over time
(TiltFreq)
![Page 20: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/20.jpg)
The flowchart of face pose tracking
![Page 21: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/21.jpg)
Examples Face Model Acquisition
![Page 22: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/22.jpg)
![Page 23: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/23.jpg)
Head pitches (tilts) monitoring over time (seconds)
![Page 24: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/24.jpg)
![Page 25: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/25.jpg)
Facial Expressions Tracking facial features
Recognize certain facial expressions related to fatigue like yawning and compute its frequency (YawnFreq)
Building a database of fatigue expressions for training
![Page 26: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/26.jpg)
![Page 27: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/27.jpg)
The plot of the openness of the mouth over time
![Page 28: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/28.jpg)
Facial expression demo
![Page 29: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/29.jpg)
Fatigue Modeling
• Observations of fatigue is uncertain, incomplete, dynamic, and from different from perspectives
• Fatigue represents the affective state of an individual, is not observable, and can only be inferred.
![Page 30: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/30.jpg)
Overview of Our Approach
Propose a probabilistic framework based on the Dynamic Bayesian Networks (DBN) to
systematically represent and integrate various sources of information related to fatigue over time.
infer and predict fatigue from the available observations and the relevant contextual information.
![Page 31: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/31.jpg)
Bayesian Networks Construction
• A DBN model consists of target hypothesis variables (hidden nodes) and information variables (information nodes).
• Fatigue is the target hypothesis variable that we intend to infer.
• Other contextual factors and visual cues are the information nodes.
![Page 32: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/32.jpg)
Causes for Fatigue
Major factors to cause fatigue include: Sleep quality. Circadian rhythm (time of day). Physical conditions. Working environment.
![Page 33: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/33.jpg)
Bayesian Fatigue Model
![Page 34: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/34.jpg)
Dynamic Fatigue Modeling
![Page 35: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/35.jpg)
Bayesian Fatigue Model Demo
MSBNX (3).lnk
![Page 36: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/36.jpg)
Interface with Vision Interface with Vision ModuleModule
An interface has been developed to connect the output of the computer vision system with the information fusion engine.
The interface instantiates the evidences of the fatigue network, which then performs fatigue inference and displays the fatigue index in real time.
![Page 37: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/37.jpg)
![Page 38: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/38.jpg)
Conclusions
• Developed non-intrusive real-time computer vision techniques to extract multiple fatigue parameters related to eyelid movements, gaze, head movement, and facial expressions.
• Develop a probabilistic framework based on the Dynamic Bayesian networks to model and integrate contextual and visual cues information for fatigue detection over time.
![Page 39: Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu](https://reader036.vdocuments.us/reader036/viewer/2022062715/56649d7a5503460f94a5e60f/html5/thumbnails/39.jpg)
Effective Fatigue Monitoring The technology must be non-intrusive and
in real time. It should simultaneously extract multiple
parameters and systematically combine them over time in order to obtain a robust and consistent fatigue characterization.
A fatigue model is needed that can represent uncertain and dynamic knowledge associated with fatigue and integrate them over time to infer and predict human fatigue.