1. 8 Emotion Effects
2. CNN accuracyØ Around 60%, drop whenthere are large backgroundIntensity variation or object.Fed in cropped data to improve.
Future Work
Expression Recognition Shading/Restyling1. Dataset/PreparationØ Cohn-Kanade(CK+) datasetØ 5876 images, 8 Emotions
2. Convolutional Neural NetworkØ Modify and fine tune a Googlenet modelwith 8 expression categories. Trained usingCaffe on the farm, Matcaffe as local interface.
1. Depth Based Visual EffectMatthias Ziegler, Andreas Engelhardt, Stefan Mller, Joachim Keinert, Frederik Zilly, Siegfried Foessel. Multi-camera system for depth based visual effects and compositing. CVMP, 2015.
2. Facial RelightingYang Wang, Lei Zhang, Zicheng Liu et.al. Face Relighting from a Single Image under Arbitrary UnknownLighting Conditions. IEEE Transactions on Pattern Analysis and Machine Intelligence, Nov. 2015.
3. RGB-D RegistrationLembit Valgma. 3D reconstruction using Kinect v2 camera. Bachelor's thesis (12 ECTP). 2016
4. Expression RecognitionYu, Zhiding, and Cha Zhang. Image Based Static Facial Expression Recognition with Multiple Deep NetworkLearning - Microsoft Research. Microsoft Research. IEEE, Nov. 2015.
Kinect
• RGB&DepthCalibration• RGBDRegistration/Denoising• FaceRecognition/Tracking
Expression• CNNModelTraining• ExpressionRecognition
Restyling
• 3DFaceModelCalculation• FaceImageCropping/Masking• Shading/Relighting/Restyling
Related Work
Motivation System Architecture
Automated Restyling of Human Portrait Based onFacial Expression Recognition and 3D Reconstruction
Cheng-Han(Dennis) WuDepartment of Electrical Engineering, Stanford University
Original
0 neutral1 angry2 contempt3 disgust4 fear5 happy6 sad7 surprise
Front(+z) white light
Left(-x) blue light
Top(+y) red light
Depth
Normal Map(R:x G:y B:z)
Original Depth Face Mask
Surface NormalCalculation
Experimental Results
1. Lighting improvements with surface textile characteristiccalculation. 2. Improve expression recognition accuracy.
3. Improve bilateral filtering efficiency and system efficiency.
Kinect
Neutral Angry
Contempt Disgust Fear
Sad Happy Surprise
Experimented automatic restyling of an ordinary photo to a selfportrait with effects that correspond to his/her facial expression.This project wants to showcase and inspire audiences the capabilitiesof AI in visual effects automation in photography or film making.
1. Kinect for Windows V2 Interface:Ø Libfreenect2/OpenCV on MacOSX2. RGB/Depth Camera Calibration/RegistrationØ Undistort depth cam with camera matrix anddistortion coefficients and projectonto RGB
3. Depth DenoisingØ Bilateral filter depth map
4. Face Tracking/ Locating ROIØ OpenCV Haar-Cascade
This part collaborated with Hsin Chen([email protected]) 1. Cropping ROI and Image Masking
Ø Crop with OpenCV tracked face ROIØ Mask with dilated, hole-filled, and
thresholded depthmap.2. Shading/RelightingØ Calculate surface normal from depth.Ø Calculate directional lighting intensity
and color for the face.
2. RestylingØ Apply mood lighting and background,processed using above mask applied onboth images.
