isr – institute of systems and robotics university of coimbra - portugal

38
University of Coimbra ISR – Institute of Systems and Robotics University of Coimbra - Portugal Institute of Systems and Robotics http://paloma.isr.uc.pt

Upload: kane

Post on 04-Feb-2016

15 views

Category:

Documents


0 download

DESCRIPTION

Institute of Systems and Robotics. http://paloma.isr.uc.pt. ISR – Institute of Systems and Robotics University of Coimbra - Portugal. Human-Robot Interaction. Determining face orientation for a robot able to interpret facial expressions Carlos Simplício, José Prado and Jorge Dias - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

ISR – Institute of Systems and RoboticsUniversity of Coimbra - Portugal

Institute of Systems and Robotics

http://paloma.isr.uc.pt

Page 2: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Determining face orientationfor a robot able to

interpret facial expressions

Carlos Simplício, José Prado and Jorge Dias

Presented by José Prado

2010

03 - 10

Human-Robot Interaction

Page 3: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Summary

Human-Robot Interaction

Introduction(Interactive Mobile Robots)

Autonomous Mobile Agent (AMA)Robotic System Controller (RSC)

Face Pose Identification System (FPIS)

Automatic Facial Expressions Recognition System (AFERS)

(Structure of a DBN classifying facial expressions)

Page 4: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Summary

Human-Robot Interaction

IIntroductionntroduction(Interactive Mobile Robots)(Interactive Mobile Robots)

Autonomous Mobile Agent (AMA)Robotic System Controller (RSC)

Face Pose Identification System (FPIS)

Automatic Facial Expressions Recognition System (AFERS)

(Structure of a DBN classifying facial expressions)

Page 5: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Introduction

We are developing a service/assistant robot, an Autonomous Mobile Agent (AMA).

This agent, will be used in the context of assisted ambiance.

The global project addresses the emergent tendencies to develop new devices for assistance and services.

Page 6: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Introduction

Human beings express their emotional states through:• facial expressions• gestures• voice• etc.

We propose:• a technique to determine face orientation based in human face symmetry;

• a DBN to classify human facial expressions.

Page 7: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Introduction

The AMA must observe and react according facial expressions of a person.

Facial expressions recognition becomes easier if done in frontal face images.

The robotic system will be used to follow the human being movements and keeps always a frontal face.

Page 8: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Summary

Introduction(Interactive Mobile Robots)

Autonomous Mobile Agent (AMA)Autonomous Mobile Agent (AMA)Robotic System Controller (RSC)

Face Pose Identification System (FPIS)

Automatic Facial Expressions Recognition System (AFERS)

(Structure of a DBN classifying facial expressions)

Page 9: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

AMA - architecture

Page 10: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Summary

Introduction(Interactive Mobile Robots)

Autonomous Mobile Agent (AMA)Robotic System Controller (RSC)Robotic System Controller (RSC)

Face Pose Identification System (FPIS)

Automatic Facial Expressions Recognition System (AFERS)

(Structure of a DBN classifying facial expressions)

Page 11: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Robotic System Controller - RSC

Robotic Platform movements:– Longitudinal translations;– Transversal translations;– Rotations.

Rotations correspond to an arc of circle centered in the human being.

Objective is to follow the rotation done by the human being, getting always an image of a frontal face.

Robotic Head can move in synchronization.

1

2

Page 12: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Summary

Introduction(Interactive Mobile Robots)

Autonomous Mobile Agent (AMA)Robotic System Controller (RSC)

Face Pose Identification System (FPIS)Face Pose Identification System (FPIS)

Automatic Facial Expressions Recognition System (AFERS)

(Structure of a DBN classifying facial expressions)

Page 13: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

In a perfect symmetric image,pixels positioned symmetricallyhave the same gray-level value: difference is zero.

We use this principle to verify if an image is symmetric: frontal face.

Example 1

Example 2

Page 14: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

In a perfect symmetric image,pixels positioned symmetricallyhave the same gray-level value: difference is zero.

Problems:• By nature, human faces are not perfectly symmetric;• There are shadows.

But it works!!!

Page 15: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

Define a vertical axis (always in the same position);

Calculate differences of gray-levels between symmetric (position) pixels. Build Normalized Gray-level Difference Histogram (NGDH).

In a frontal face, the vertical axis bisects the face and the information collected in the NGDH is strongly concentrated near the mean.

Else, the information is scattered along the NGDH.

NGDH withscattered information

NGDH withconcentrated information

Page 16: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

Algorithm:Find and extract face region in the image;

Define a vertical axis (dividing the region in two parts with equal number of pixels);

Synthesize face images - use vertical axis to perform a 3D transformation (rotation);Synthesized images are “hypotheses” to find the face out-of-plane rotation;

Built NGDH's;

Find the pseudomean – number of occurrences in a narrow region around the NGDH's mean;

Synthesized image with great pseudomean has the frontal face!!

Page 17: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

Page 18: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

Algorithm:Find and extract face region in the image;

Define a vertical axis (dividing the region in two parts with equal number of pixels);

Synthesize face images - use vertical axis to perform a 3D transformation (rotation);Synthesized images are “hypotheses” to find the face out-of-plane rotation;

Built NGDH's;

Find the pseudomean – number of occurrences in a narrow region around the NGDH's mean;

Synthesized image with great pseudomean has the frontal face!!

Page 19: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

Page 20: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

Algorithm:Find and extract face region in the image;

Define a vertical axis (dividing the region in two parts with equal number of pixels);

Synthesize face images - use vertical axis to perform a 3D transformation (rotation);Synthesized images are “hypotheses” to find the face out-of-plane rotation;

Built NGDH's;

Find the pseudomean – number of occurrences in a narrow region around the NGDH's mean;

Synthesized image with great pseudomean has the frontal face!!

Page 21: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

Page 22: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

Page 23: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

Algorithm:Find and extract face region in the image;

Define a vertical axis (dividing the region in two parts with equal number of pixels);

Synthesize face images - use vertical axis to perform a 3D transformation (rotation);Synthesized images are “hypotheses” to find the face out-of-plane rotation;

Built NGDH's;

Find the pseudomean – number of occurrences in a narrow region around the NGDH's mean;

Synthesized image with great pseudomean has the frontal face!!

Page 24: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

Page 25: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

OriginalAngle = 0º

Rotation

Rotation

+30º

Result -30º

Result 0º

Result +30º

Rotation

-30º

Page 26: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

OriginalAngle = -30º

Rotation

-30º

Rotation

Rotation

+30º

Result -60º

Result -30º

Result 0º

Page 27: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Face Pose Identification System - FPIS

OriginalAngle = +30º

Rotation

-30º

Rotation

Rotation

+30º

Result 0º

Result +30º

Result +60º

Page 28: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Summary

Introduction(Interactive Mobile Robots)

Autonomous Mobile Agent (AMA)Robotic System Controller (RSC)

Face Pose Identification System (FPIS)

Automatic Facial Expressions Recognition System (AFERS)Automatic Facial Expressions Recognition System (AFERS)

((Structure of a DBN classifying facial expressions)Structure of a DBN classifying facial expressions)

Page 29: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Facial Expressions

We only consider five emotional states.

Each emotional state has a characteristic facial expression.

A facial expression is a set of Action Units (AUs).

Paulo José Carlos Carlos Alex

anger fear happy neutral sad

AUs are “distortions” of facial features.

Ex: lips smile.

Page 30: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

DBN's Structure

Page 31: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

DBN's Structure

Level 1

Emotional States considered are: anger fear happy sad neutral other

Node (variable) that probabilistically reflect the existence of an Emotional State.

otherneutral,sad,happy,fear,anger,StateEmotional

Page 32: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

DBN's Structure

Level 2

Expressions considered are: anger fear happy sad neutral

Nodes (variables) that probabilistically reflect the existence of a facial expression.

yesno,AngerE

yesno,NeutralE

...

Page 33: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

DBN's Structure

Level 311 AUs are considered in each facial expression.

Page 34: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

DBN's Structure

Level 4

Nodes (variables) that probabilistically reflect the strength of the evidences (positive or negative).

maximumhigh,medium,low,minimum,AU1EH

maximumhigh,medium,low,minimum,AU25EH

...

Page 35: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

DBN's Structure

Level 5

Here, information is propagated between time slices.

These nodes (variables) combine / fuse probabilistically, through inertia, information coming from the low level in present time slice with that from the previous instant.

maximumhigh,medium,low,minimum,tAU1SensorEH

maximumhigh,medium,low,minimum,tAU1SensorEH 1

...

...

Page 36: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

DBN's Structure

Level 6

Nodes (variables) collecting the evidences provided by the sensors.

maximumhigh,medium,low,minimum,AU1SensorEH

maximumhigh,medium,low,minimum,AU25SensorEH

...

Page 37: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

Conclusions

It was developed:

An architecture for an Autonomous Mobile Agent;

A Face Orientation Identification Technique;

A structure for a DBN.

The Face Pose Identification Technique has a good performance and is very fast.

Classification of facial expressions using positive and negative evidences is very promising.

Page 38: ISR – Institute of Systems and Robotics University of Coimbra - Portugal

University of Coimbra

END

Thanks for your attention!!!

Questions ?