structured face hallucination · 2013. 7. 5. · title: structured face hallucination author:...

Structured Face HallucinationChih-Yuan Yang, Sifei Liu, Ming-Hsuan Yang

Electrical Engineering and Computer Science, University of California, Mercedcyang35@ucmerced.edu, sliu32@ucmerced.edu, mhyang@ucmerced.edu

code available http://eng.ucmerced.edu/people/cyang35

Challenges

•How to handle various poses and expressions?•How to retrieve effective exemplars?•How to preserve the visual consistency in outputs?•How to reconstruct sharp contours?

Main Idea•Split a face image into three image structures: facialcomponents, contours, and smooth regions

•Label exemplar images in terms of pose and glasses•Exploit landmarks to handle component variety andtransfer high-frequency details of a whole exemplarcomponent

•Reconstruct contours with high-quality directionsand sharpness through novel priors

•Transfer details of smooth regions from exemplarpatches

Visual Consistency

•Gradients of a whole region → consistency of awhole region

•Direction-preserving upsampling → consistent edgedirection

•Sharpness priors → consistent and visually correctedge contrast

Algorithm

Integrate gradient maps U = wcUc + (1−wc)(weUe + (1−we)Ub

)Generate an output image Ih = argmin

I‖∇I −U‖2 s.t. (I ⊗G) ↓= Il

Facial components

Contours

Directional similarity fk(p) = exp(−‖P −Qk‖/σ), k = 1, . . . ,KDirection-preserving HR image Id = argmin

I

∑k ‖fk(I)−Tk‖

2 s.t. (I ⊗G) ↓= Il

Restore edge sharpness Ue(p) =m̄′pmp·Ud(p)

Smooth regions

Patch search accelerated by the PatchMatch method

Experimental Results

(a) Input (b) Irani91 (c) Yang10 (d) Ma10 (e) Liu07 (f) Proposed (g) Original

Conclusions•Contributions

I Split-and-merge approach to reconstruct details of a face imageI Exploit landmarks for effective exemplar searchI Novel statistical priors and upsampling method for simultaneously preserving edge

direction and sharpness•Advantages

I Effective and consistent high-frequency detailsI Robustness for various poses and expressions

Chih-Yuan Yang, Sifei Liu, Ming-Hsuan Yang (EECS, University of California, Merced) 26th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013)