semantic soft segmentation - korea...

Semantic Soft Segmentation

YAĞIZ AKSOY (MIT)et al.ACM Transactions on Graphics 2018

Copyright of figures and other materials in the paper belongs original authors.

Presented by Qi-Meng Zhang

2019. 10. 24

Computer Graphics @ Korea University

Qimeng Zhang| 2019. 10. 24 | # 2Computer Graphics @ Korea University

What is Semantic Soft Segmentation?

Semantic Segmentation

https://towardsdatascience.com/semantic-segmentation-the-easiest-possible-implementation-in-code-193bf27b86b8

Image from:

Input

Hard segmentation

+Soft transitions

Soft segmentation

SSS

https://towardsdatascience.com/semantic-segmentation-the-easiest-possible-implementation-in-code-193bf27b86b8


• Accurate representation of soft transitions between image regions is essential for high-quality image editing and compositing.

When fuzzy boundaries and transparency and involved

• Heavily on interaction

• Tedious task of object selection

Ex: Photoshop tool (magnetic lasso, magic wand)

1.Introduction

https://codebridgeplus.com/photoshop-magic-wand/

https://codebridgeplus.com/photoshop-magic-wand/


• In this work

Provide distinct segments of the image, while also representing the soft transitions between them accurately

Done fully automatically

1.Introduction

Contribution

Fig. 1.


• Based on SPECTRAL DECOMPOSITION

• Graph structure considering both

Low-level information

• Texture and color

High-level information

• Semantic cues

By deep learning

• Corresponding Laplacian Matrix

Reveals the soft transitions between semantic objects in eigenvectors

• Spatially varying model of layer sparsity

Generates high-quality layers from the eigenvectors

1.Introduction

How?


• Define the digital matting

1.Introduction

Matting

𝐼 = 𝛼 × 𝐹 + (1 − 𝛼) × 𝐵


1.Introduction

Matting Interface

𝛼 = 1𝛼 = 0 𝛼 ∈ [0,1]𝛼 = 1

𝛼 = 0


• Laplacian Matrix

1.Introduction

Graph Laplacian

𝐿 = 𝐷 −𝑊

Degree matrix Adjacency matrix

1 2

3

4

5

2 -1 -1 0 0

-1 2 -1 0 0

-1 -1 2 0 0

0 0 0 1 -1

0 0 0 -1 1

Laplacian Matrix

1. Symmetric2. Positive semi-definite3. Smallest eigenvalue of

L is 0.

*Also called affinity matrix


1 0

1 0

1 0

0 1

0 1

1.Introduction

Laplacian Matrix in Spectral Clustering

1 2

3 4

5

2 -1 -1 0 0

-1 2 -1 0 0

-1 -1 2 0 0

0 0 0 1 -1

0 0 0 -1 1

Laplacian Matrix

f1 f2

1 0

1 0

1 0

0 1

0 1

First K Eigenvectors

f1 f2


1.Introduction

Matrix with Image

Eigendecomposition

512×512

E.x: If only save the first(maximum) 50 eigenvalues, others set to 0

512×512


1.Introduction

Eigenvector with Segmentation

Smallest eigenvectors of the matting Laplacian

Spectral Matting[Anat Levin (MIT)et al./ CVPR 2007]


2.Related Work

Soft Segmentation(1/2)

Unmixing-Based Soft Color Segmentation for Image Manipulation[Yagiz Aksoy(Simon Fraser University.) et al./ TOG2017]

Color-based


2.Related Work

Soft segmentation(2/2)

Spectral Matting[Anat Levin (MIT)et al./ CVPR 2007]

A Closed-Form Solution to Natural Image Matting[Anat Levin (MIT)et al./ CVPR 2006]

Spatially connected soft segments

Matting Laplacian Matrix

Spectral Analysis


• Estimation of per-pixel opacities of a user-defined foreground region

Affinity-based method

2.Related Work

Natural Image Matting

KNN Matting [Chen, Li, and Tang (The Hong Kong University of Science and Technology). TPAMI, 2013.]

k nearest neighbors


2.Related Work

Targeted Edit Propagation

DeepProp: Extracting Deep Features from a Single Image for Edit Propagation[Yuki endo (University of Tsukuba) et al./ EUROGRAPHICS 2016]


2.Related Work

Semantic Segmentation

Pyramid Scene Parsing Network[Zhao(The Chinese University of Hong Kong)et al./ CVPR 2017]

Semantic Instance Segmentation via Deep Metric Learning[Alireza Fathi(Google) et al./ArXiv 2017]


• The problem description

Automatically generate a soft segmentation of the input image

• A decomposition into layers that represent the objects in the scene

• Including transparency and soft transitions

3.Method

𝛼: opacity value (𝛼 ∈ 0,1 )when 𝛼 = 0: fully transparent, 𝛼 = 1: fully opaque


3. Method

Overview

Relaxed sparsification


3.1 Background

Spectral Matting(Brief Summary)

Slide by Levon


• Recall the compositing equation

3.1 Background

The Matting Laplacian(1/3)

Slide by: CVFX @ NTHU


3.1 Background




3.1 Background



Rewrite as matrix

(𝑎𝑘𝐼1 + 𝑏𝑘 − 𝛼1)2

Least squares problem

……..

http://ocw.nthu.edu.tw/ocw/index.php?page=course&cid=125&More detail in:

http://ocw.nthu.edu.tw/ocw/index.php?page=course&cid=125&


3.1 Background

From Eigenvectors to Matting Components

𝛼𝑖𝑝:

𝐸: is a matrix containing the K eigenvectors of L with smallest eigenvalues

𝑦𝑖: is a the linear weights on the eigenvectors that define the soft segments

𝛾: the parameter controls the strength of sparsity prior

Build the corresponding normalized Laplacian matrix


• Defined a additional low-level affinity term

Represents color-based longer-range interactions

• Proposed a guided sampling based on an

over-segmentation of image

Generate 2500 superpixels

• Using SLIC(simple linear iterative clustering)

Estimate the affinity between each superpixel and all the superpixelswithin a radius that corresponds to 20% of the image size.

3.2 Nonlocal Color Affinity

SLIC Superpixels Compared to State-of-the-Art Superpixel Methods

[R.Achanta(EPFL) et al./ IEEE TRANS. PATTERN ANAL. MACH.INTELL 2012]


• The color affinity between the centroids of two superxiels s and t:

𝑐𝑠 , 𝑐𝑡: average colors of the superpixels of s and t

• Lies in [0,1]

erf: Gauss error function

𝑎𝑐,𝑏𝑐: controlling how quickly the affinity degrades and the threshold

where it becomes zero

• 𝑎𝑐 =50, 𝑏𝑐=0.05

3.2 Nonlocal color affinity

Equation


• This affinity essentially makes sure the regions with very similar colors stay connected in challenging scene structures

3.2 Nonlocal Color Affinity (cont’)

Fig. 3.


• A term that encourages the grouping of pixels that belong to the same scene object

The feature vectors are generated such that for two pixels 𝑝 and 𝑞 that belong to the same object 𝑓𝑝 and 𝑓𝑞 are similar.

𝑎𝑐,𝑏𝑐: controlling the steepness of the affinity function

෩𝑓𝑠 , ෩𝑓𝑡: average feature

3.3 High-level Semantic Affinity


•

3.3 High-level Semantic Affinity

Fig. 4.

Fig. 5.


• The Laplacian matrix L by adding the affinity matrices together

𝑊𝐿: is the matrix with the matting affinities

𝑊𝑐: is the matrix with the nonlocal color affinities

𝑊𝑆: is the matrix with the semantic affinities

𝜹𝑺, 𝜹𝑪 set to be 0.01

3.4 Creating the layers

Forming the Laplacian Matrix


• We extract the eigenvectors corresponding to the 100 smallest eigenvalues of L. 𝛾 = 0.8

K-means Algorithm on feature vectors

• 5 layers


Constrained Sparsification

Fig. 7.Before grouping After grouping


• We define an energy function that promotes matte sparsity on the pixel-level while respecting the initial soft segment estimates from the constrained sparsification and the image structure


Relaxed Sparsification (Energy1,2)

ො𝛼 is the layers created with the constrained sparsification

1

2

Relaxed Eqn (1)



Relaxed Sparsification(Energy3,4)

3

Energy defining the spatial propagation of information in Eqn(6)

4

𝛻𝑐𝑝 is the color gradient in the image at pixel p computed using the separable kernels

• Differentiation of discrete multidimensional signals.[H.Farid(NYU) and E.P.Simoncelli(CNS)/ Image Process 2004]


• Final energy

• Matrix form

Solve this equation using preconditioned conjugate gradient optimization


Relaxed Sparsification(Final Energy)

Fig. 6.



Matrix Form of the Energy Function(1,2)

𝑁𝑖: number of layer𝑁𝑖𝑝: pixel in layer i

𝐶: 𝑁𝑖 × 𝑁𝑖𝑝



Matrix Form of the Energy Function(3,4)

𝐷𝑢: diagonal matrix built with 𝑢𝑖𝑝

𝐷𝑣: diagonal matrix built with 𝑣𝑖𝑝


• Using DeepLab-ResNet-101 as feature extractor

3.5 Semantic Feature Vectors


• Train this network on the semantic segmentation task of the COCO-Stuff dataset

• Refine the feature map generated by this network to be well-aligned to image edges using the guided filter

• Use principal component analysis (PCA) to reduce the dimensionality to three

3.5 Semantic Feature Vectors

Fig. 8.


• Sparse eigendecomposition

MATLAB

640×480 image

• This step takes around 3 minutes

• Relaxed sparsification

Preconditioned conjugate gradient optimization(MATLAB)

50~80 iterations

This step takes around 30 seconds

• The run-time of our algorithm grows linearly with the number of pixels

3.6 Implementation Details


4.Experiment analysis

4.1 Spectral Matting & Semantic Segmentation

Fig. 9.



4.1 Spectral Matting & Semantic Segmentation

Fig. 10.



4.2 Natural Image Matting

PSPNet [Zhao et al. 2017] Mask R-CNN [He et al. 2017]

（f）

Fig. 11.



4.2 Natural Image Matting

Fig. 12.



4.3 Soft Color Segmentation

Fig. 13.



4.4 Using SSS for Image Editing

Fig. 14.


• Not optimized for speed

• One object may be divided into several layers

• Not provide instance-aware semantic information

• Fail at the initial constrained sparsification step when the object colors are very similar

• Grouping of soft segments may fail due to unreliable semantic feature vectors around large transition regions

5. Limitations and Future Work

Fig. 15.


• Proposed a method that generates soft segments that correspond to semantically meaningful regions in the image

Fusing the high-level information with low-level image features fully automatically

• Shown the soft segments with the semantic boundaries can be revealed by spectral analysis of the constructed Laplacian matrix.

• The proposed relaxed sparsification method for the soft segments can generate accurate soft transitions while also providing a sparse set of layers.

6. Conclusion

semantic soft segmentation - korea...

Documents