temporal adaptive alignment network for deep video ...temporal redundancy information can be...

Temporal Adaptive Alignment Network for Deep Video Inpainting

Ruixin Liu , Zhenyu Weng , Yuesheng Zhu∗ and Bairong LiCommunication and Information Security Lab, Shenzhen Graduate School, Peking University

{anne xin, wzytumbler, zhuys, lbairong}@pku.edu.cn

Abstract

Video inpainting aims to synthesize visually pleas-ant and temporally consistent content in missing re-gions of video. Due to a variety of motions acrossdifferent frames, it is highly challenging to utilizeeffective temporal information to recover videos.Existing deep learning based methods usually esti-mate optical flow to align frames and thereby ex-ploit useful information between frames. How-ever, these methods tend to generate artifacts oncethe estimated optical flow is inaccurate. To allevi-ate above problem, we propose a novel end-to-endTemporal Adaptive Alignment Network(TAAN)for video inpainting. The TAAN aligns referenceframes with target frame via implicit motion esti-mation at a feature level and then reconstruct tar-get frame by taking the aggregated aligned refer-ence frame features as input. In the proposed net-work, a Temporal Adaptive Alignment (TAA) mod-ule based on deformable convolutions is designedto perform temporal alignment in a local, denseand adaptive manner. Both quantitative and qual-itative evaluation results show that our method sig-nificantly outperforms existing deep learning basedmethods.

1 IntroductionVideo inpainting is a task of synthesizing visually realis-tic and semantically plausible contents in missing regionsof the given video sequence in temporal coherence. It canbe used in many applications such as unwanted object re-moval, damaged parts recovery, visual privacy filtering, etc.Significant progress has been made recently in single-imageinpainting[Pathak et al., 2016; Yu et al., 2018] thanks to thedeep generative networks. However, because of the additionaltime dimension, video inpainting method not only needs torepair the missing region for each frame but also has to en-sure the temporal consistency across frames. While tempo-ral relationship brings challenges for video inpainting, thetemporal redundancy information can be exploited by it to

∗Contact Author

(a)

Inp

ut

fra

mes

(b)

Ima

ge

inp

ain

tin

g(c

) O

ur

met

hod

Figure 1: (a) Input video with masks in red. (b) Video inpaintingresults from image inpainting method [Nazeri et al., 2019]. (c) Ourvideo inpainting method.

obtain better results at the same time. As illustrated in Fig-ure 1, compared with image inpainting based method, videoinpainting method performs better results and preserves thevideo coherence. However, it is difficult to directly utilizeinformation from reference frames due to the misalignmentbetween frames which is caused by complex motion of cam-era or objects. Therefore, how to efficiently utilized temporalinformation is the essential issue for video inpainting prob-lem.

Traditional patch-based methods[Newson et al., 2014;Huang et al., 2016] find the similar spatio-temporal patchesfrom the known regions of videos to fill the holes, which for-mulate the problem as a patch-based optimization task. Al-though some good results have been shown, they usually suf-fer from the high computational complexity, which results invery slow processing speed.

Motivated by the success of the deep neural networks insingle image inpainting task, several deep learning basedmethods for video inpainting have been proposed recentlyand achieve significant results in terms of quality and speed.[Lee et al., 2019] align frames firstly by performing globalaffine transformation, and copy valuable pixels from alignedframes to complete the missing regions. It shows that tem-

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)

927

poral alignment is important for exploiting information fromreference frames. However, such transformation could notwell align the frames with more complex motions, which arecommon in realistic scenes.

Optical flow contributes to obtaining temporal informationbetween frames in complex motion scenes. [Xu et al., 2019;Kim et al., 2019] utilize the optical flow to align frames, vis-ible information from reference frames is collected via flowwarping operation. These methods highly depend on the ac-curacy of the estimated optical flow. However, completingoptical flow between frames with missing region is quite chal-lenging and any errors in optical flow computation will influ-ence the final quality of results.

In this paper, a novel end-to-end Temporal Adaptive Align-ment Network(TAAN) without using optical flow is sug-gested by us to solve the video inpainting problem. Givena sequence of video frames with holes, the network processesvideo frame-by-frame. The TAAN aligns target frame withreference frames firstly at a feature level without explicit mo-tion estimation through a TAA module. Specifically, inspiredby the deformable convolution [Dai et al., 2017], the pro-posed TAA module utilizes the features from target frameand corresponding reference frame to predict the offsets ofsampling convolution kernels, and applies the kernels on thereference frame features to perform temporal adaptive align-ment. In this way, the final reconstructed target frame willhave less artifacts and the capability of handling various mo-tion conditions in temporal scenes will be improved. In ad-dition, benefited by deformable convolution, more informa-tion with the same structure as the sampled position will beexplored by TAA module, which strengthens the alignmentaccuracy between frames. Finally, a reconstruction networkwhich takes the aggregated aligned reference frame featuresand target features is developed to recover target frame. Theend-to-end network design helps our network pay more at-tention to the missing region and further improve the perfor-mance of the model.

We conduct extensive experiments on Youtube-VOS [Xuet al., 2018] and DAVIS [Perazzi et al., 2016] datasets. Theexperimental results show that our framework could achievevisually pleasing results at complex motion scenes. The ma-jor contributions of our paper are summarized as follows:

• We propose a novel end-to-end network for video in-painting, which aligns the frames at a feature level viaimplicit motion estimation and aggregates temporal fea-tures to synthesize missing content.

• We propose a temporal adaptive alignment modulebased on standard deformable convolution to adaptivelyalign frames which contain holes in a local and densemanner.

• We quantitatively and qualitatively evaluate our methodand show its efficacy.

2 Related Work2.1 Traditional MethodsTraditional patch-based approaches typically use the priorsuch as patch similarity to propagate information from the

known regions. [Patwardhan et al., 2005; Patwardhan etal., 2007] complete videos by sampling non-local spatio-temporal patch assuming static camera and constrained cam-era motion. They find the patch based on the greedy algo-rithm, which inevitably propagate the early errors and leadto globally inconsistent results. To enforce global spatio-temporal consistency, [Wexler et al., 2007] cast the prob-lem as global optimization task which constrain missing val-ues form coherent structures with respect to reference exam-ples. Further, to strengthen the temporal consistency, [New-son et al., 2014] use a robust affine estimation to compensatecamera motion and perform an extension of PatchMatch al-gorithm [Barnes et al., 2009] to accelerate the patch match-ing process. [Granados et al., 2012] compensate geomet-ric distortion by utilizing a set of homographies to performframe-to-frame alignments. However, they have difficultiesin handling general scenes which have more complex geo-metric variations. [Huang et al., 2016] combine the advan-tages of optical flow and color information, and formulate theproblem as a global optimization of color and flow. They per-form significantly results in complex motion scenes, achiev-ing the state-of-the-art results.

2.2 Learning-based MethodsA significant advantage of deep learning based methods istheir ability to learn semantics from large scale datasets. Thefirst deep learning based method for video inpainting is pro-posed by [Wang et al., 2019], they perform 3D convolution onlow resolution input to provide the temporal guidance and use2D convolution based on the low resolution result to recoverthe spatial coherent frame. [Chang et al., 2019] propose3D gated convolution network for video inpainting based onthe work of image inpainting, which tackle the uncertaintyof free-form masks problem. Further, they introduce a Tem-poral PatchGAN discriminator to enhance temporal consis-tency. However, 3D convolution is hard to train and theirtemporal receptive fields are too limited or directional. [Kimet al., 2019] collect information by flow warping from neigh-bor frames to the target frame to recover the target frame, andutilize a recurrent feedback and memory layer to stable tem-poral consistency. [Xu et al., 2019] address video inpaintingproblem as a pixel propagation problem by recovering accu-rate flow field from missing region and the synthesized flowfield is used to guide the pixel propagation to generate seman-tically plausible contents. Despite the fact that motion can bethe useful guidance for propagating information [Oh et al.,2019] aggregate temporal information through an asymmet-ric attention block to progressively fill the hole from the holeboundary. [Lee et al., 2019] utilize a self-supervised net-work to estimate affine matrices between frames for the align-ment and aggregate information from aligned frames througha copy-and-paste network to fill the missing regions.

3 Proposed Algorithm3.1 OverviewFigure 2 presents the workflow of the proposed video inpaint-ing network. Given a sequence of video frames X anno-tated with missing region M (1 indicates invalid pixels, 0 on


928

…

TAA module

Target frame

and mask

Reference

frames and

masks

…

……

Feature

Aggregation

Encoder

5*

5, s=

2

3*

3

3*

3

3*

3, s=

2

3*

3

3*

3

TAA module

TAA module

Down-sample & Duplicate

……

Reference

features

……

3*

3

3*

3,d

=2

3*

3

3*

3

3*

3

5*

5, s=

2

3*

3,d

=1

6

Reconstruction

Up

sam

ple

Up

sam

ple

num_blocks=4,

…

Input video

frames

Aggregation

features and

masks

Update the

output

Aligned reference

features and masks

Target

features

… …… …

Output

,

,

,

,

…

……

Figure 2: The schematic illustration of the our TAAN-based video inpainting model.

the contrary), the network processes video frame-by-frame inthe temporal order and generates the final outputs Y . Thewhole network consists of four modules: encoder module,TAA module, feature aggregation module and reconstructionmodule.

To complete a target frame, our network first takes the tar-get frame XT , reference frames XR and corresponding bi-nary masks into an encoder module to extract feature mapsF . All of them share the same encoder. [Lee et al., 2019]have demonstrated that taking account into the missing regionwhen aggregating information will help improve the effec-tiveness of inpainting results. So we down-sample the maskM and then extend it to the same number of channels as theextracted feature maps by duplication operation. Next, wesend the feature maps that extracted from the target frameXTt and the reference frame XR

i , corresponding duplicatedmasks MD into TAA module to get aligned results (FRi→tand MR

i→t, where i → t indicates reference features/masks iis aligned to target t). After alignment, all aligned referenceframe features and aligned masks are aggregated by featureaggregation module. Finally, we concatenate target featuresFTt , aggregated features FA and aggregated masks MA torecover the target frame.

To enforce the temporal consistency, we update the inputvideo sequence with completed frame over time and transferthe previous completed frame as one of the reference framesof the current target frame.

3.2 Network ArchitectureEncoderThis module concatenates the frame and corresponding bi-nary mask along the channel axis to form a 4-channel imageand take it as input to extract visual feature maps. The con-volutions with stride of 2 are utilized to decrease the reso-

lution twice and get the 1/4 scale of original size, which isimportant to maintain the high frequency details in the miss-ing region [Iizuka et al., 2017]. The extracted features will beemployed for feature-wise temporal alignment.

Temporal Adaptive Alignment(TAA) ModuleBy learning offsets of the sampling convolution kernels andapplying the learned kernels to the feature maps, deformableconvolution [Dai et al., 2017] could obtain information awayfrom its regular local neighborhood, improving the complexgeometric transformation capability. Motivated by the capa-bility of the deformable convolution, we introduce it into ourTAA module to perform temporal alignment. A detailed il-lustration of our TAA module is presented in Figure 3.

Given the visual features FTt and FRi from target frameand reference frame respectively, TAA module concate-nates these features and feeds them into an offset esti-mator network to predict offsets {∆pn | n = 1, · · · , |R|}of the convolution kernels. For example, R ={(−1,−1), (−1, 0), · · · , (0, 1), (1, 1)}, when a regular gridwith a 3×3 kernel, and a dilation factor of 1. The existenceof missing regions bring challenges for alignment. To solvethis problem, we adopt the U-Net architecture for the offsetestimator network which has been widely used in pixel-wiseestimation tasks. Further, a 3 × 3 convolution layer is uti-lized to predict the final offsets of the reference features andduplicated mask.

With the predicted offsets {∆pn} and reference frame fea-tures FRi , each position p0 in the aligned feature maps FRi→tare computed by the deformable convolution operation as fol-lows:

y(po) =∑pn∈R

w(pn) · x(p0 + pn + ∆pn). (1)

It is worth noting that the adaptively learned offsets will


929

Offset

Estimator

conv

conv

Deformable

conv

Deformable

conv

offsets

Reference

masks

Figure 3: The schematic illustration of how the TAA module alignsfeatures and masks.

implicitly capture motion information and contribute to align-ing frames in the complex motion scenes. In addition, as il-lustrated in Figure 3, for each sampled position p0, our TAAmodule could explore more features that may share the sameimage structure as p0 by deformable convolution operation,which helps to collect more information in reference framefeatures and further improve alignment capability.

The aligned masks are computed by the offsets and dupli-cated masks in a similar fashion. For better results, we use theDCNv2 [Zhu et al., 2019] in our implements, which has morestronger modeling power. We will analyze the effectivenessof TAA module in Sec.4.3.

Feature AggregationDifferent frames and locations are not equally beneficialto the reconstruction. Inspired by strong results presentedin [Lee et al., 2019], we pick up the most relevant informa-tion in the reference frames when aggregating the features asfollows:

We first measure the global similarities θi,t between thetarget frame feature and each aligned reference frame feature,while ignoring the invalid location in the aligned mask.

θi,t =1∑

(x,y) Vi,t(x, y)·∑(x,y)

Vi,t(x, y)·F Tt (x, y)·FR

i (x, y).

(2)Where Vi,t = (1−MDT

t )� (1−MRi→t) is the visibility

map.Then, we multiply the similarity with corresponding

aligned masks and use a softmax function across temporal di-mension to weigh the features in the aligned reference frames.The weight is computed as follows:

Wi(x, y) =exp(θi,t · (1−MR

i→t))∑i exp(θi,t · (1−MR

i→t)). (3)

The final fusing features are computed by summing the ref-erence frames features with the weight.

FA(x, y) =∑i

FRi→t ·Wi(x, y). (4)

The aggregation masks are computed as follows:

MA(x, y) = 1− (∑i

Wi(x, y)). (5)

ReconstructionThe reconstruction network is used to restore the target frameby taking the aggregated reference features, aggregated ref-erence masks and the target features as input. Four dilationblocks with dilation factor of 2n are designed to enlarge thereceptive field, which is beneficial to fill the region not exist-ing in the reference frames [Iizuka et al., 2017]. Finally, thenearest neighbor up-sampling is used to enlarge the featuremap to the target frame.

3.3 Loss FunctionsTo guarantee the completion quality of the results, we adoptreconstruction loss, perceptual loss, style loss and total varia-tion loss function to effectively train the proposed network.

Reconstruction loss is used to constrain pixel-level restora-tion. Where Igt represents the ground truth image, M is givenmask, Iout is the network prediction.

Lhole = ‖M · (Iout − Igt)‖1 . (6)

Lvalid = ‖(1−M) · (Iout − Igt)‖1 . (7)We also adopt perceptual loss Lprec and style loss Lstyle

to further improve the visual quality of the whole recoveredimages.

Lprec = E

[∑i

1

Ni‖φi(Igt)− φi(Iout)‖1

]. (8)

Lstyle = Ej[∥∥∥Gφj (Iout)−Gφj (Igt])

∥∥∥1

]. (9)

Where φi is the activation map in a pretrained VGG-19 onImageNet. In our paper, φi corresponds to activation mapsfrom layers relu1 1, relu2 1, relu3 1, relu4 1 and relu5 1. Theactivation maps also will be sent to calculate gram matrix instyle loss. G represents Gram Matrix for computing covari-ance.

Furthermore, we utilize the total variation loss to smooththe checkerboard effect. The total loss Ltotal is as follows:Ltotal = Lvalid+ 6Lhole+ 0.05Lprec+ 120Lstyle+ 0.1Ltv.

(10)For our experiments, the loss term weights are adopted

from [Liu et al., 2018].

4 Experiments4.1 Experimental SettingsDatasets. Keeping with the goal of synthesizing plausiblecontents of the missing region in videos, two datasets areemployed in this work to demonstrate the effectiveness ofthe proposed method. The first is YouTube-VOS [Xu et al.,2018] Dataset, which is a large-scale video object segmenta-tion dataset with a wide variety of scenes. It contains 4,453YouTube video clips and 94 object categories and is splitinto 5,471 for training, 474 for validation and 508 for testing.We train our video inpainting network on the YouTube-VOStraining set. The second dataset is DAVIS [Perazzi et al.,2016] Dataset. It includes 50 high quality video sequenceswith covering dynamic scenes, large occlusions, motion blur,complex camera movements, and each frames are annotatedwith the pixel-accurate foreground object masks.


930

(kite-walk)

Ou

r m

eth

od

CP

Net

Hu

an

gV

INet

Inp

ut

DF

G

(dog-agility)Figure 4: Qualitative comparison results of TAAN video inpainting approach for the scenes dog-agility(left) and kite-walk(right) from DAVISDataset.

Training Details. We select five reference frames(Xt−4,Xt−2, Xt−1, Xt+2, Xt+4) and resize them into 256×256 asinputs when training the network. To accelerate the trainingprocess while reducing over-fitting, we initialize parametersof our neural network by using the initialization method in[He et al., 2015]. Adam optimizer with the initial learningrate to 10−4 is utilized, we decayed the learning rate by 0.1every 1 million iterations.Baseline. We compare the proposed algorithm with thestate-of-the-art methods including a traditional patch-basedmethod:Huang [Huang et al., 2016] and three deep learningbased methods:DFG [Xu et al., 2019], VINet [Kim et al.,2019], CPNet [Lee et al., 2019]. For deep learning basedmethods, we directly conduct the experiment on the trainedmodel provided by authors. All experiments are done on the256×256 frames.

4.2 Experimental ResultsQualitative EvaluationsTo validate the generalization ability of our model, we com-pare the proposed method with other methods in real world

scenes. The results as shown in Figure 4.We can observe that the optical flow based methods [Xu

et al., 2019; Kim et al., 2019] always tend to produce ar-tifacts due to the wrong estimation of flow. CPNet adoptssimple global motion estimation to align frames, which lim-its performances for scenes with complex motion(e.g., scenedog-agility: the rods produce deformation over time).

Our model adaptively aligns frames in a local and densemanner at feature level, which can implicitly capture motioncues and aggregate more information to perform alignment,showing favorably inpainting results.

Quantitative EvaluationsSince there is no existing dataset to evaluate the video in-painting task. When testing on the Youtube-VOS dataset,we apply the video mask generation algorithm [Chang et al.,2019] to simulate masks of the holes and synthesize videosby applying the generated masks on the background imagesequences. Our method is supposed to recover the originalvideos. DAVIS dataset provides the dense object mask an-notations. To get closer to the reality, we directly shuffle the


931

Reference frames & Aligned reference features

Target frame

Target feature

Output

Aggregation

feature

Figure 5: The effectiveness of TAA Module. The gray hole in thefirst row indicates the missing area.

MethodDAVIS Youtube-VOS

PSNR SSIM PSNR SSIMHuang 30.607 0.927 29.015 0.885DFG 29.557 0.911 27.333 0.879VINet 29.765 0.901 27.542 0.874CPNet 29.121 0.886 28.879 0.865Ours 30.829 0.913 29.987 0.887

Table 1: Results of quantitative evaluation. The best result is labeledwith boldface.

pairs of videos and masks from DAVIS to test the modelswhen we conduct experiments on the DAVIS dataset.

The metrics results conducted on the Youtube-VOS andDAVIS datasets are summarized in Table 1. We can ob-serve that our method is superior to all deep learning basedmethods. Although Huang [Huang et al., 2016] achievescomparable results with our method, it formulates the videopainting problem as a patch-based optimization task and ismuch slower than deep learning based methods, which hasbeen indicated in our paper and other papers(e.g., [Xu et al.,2019], [Kim et al., 2019]).

4.3 Effectiveness of TAA ModuleTo validate the effectiveness of the proposed TAA modulefor alignment, the intermediate learned feature maps are vi-sualized in Figure 5. As for target frame XT

t , we selected 3representative reference frames:XR

t−12 is far from the targetframe, XR

t−1 and XRt+1 are close to it.

The missing region of XRt−1 and XR

t+1 overlap the targetframe, so they only capture part of the information of themissing region in XT

t . The green and red oval circles in tar-get feature FTt are invisible, while FR(t−1)→t and FR(t+1)→tare able to predict the details, and our module will not cap-ture corresponding information in the overlap region of tar-get frame and reference frames. This shows that our modulehas excellent performance on frames alignment at the featurelevel. The aligned feature map FR(t−12)→t is similar to theaggregated feature map FAt , both of them obtain the fully de-tails of the missing region, which indicates our TAA modulehas advantage on capture long temporal range of information

(a)Input frame sequence (b)Sample frame from the blue line

(c)Image inpainting based method (d)Our method

Figure 6: Temporal coherent completion.

and could effectively aggregate aligned feature map to helpour network to reconstruction the frame. As shown in the fig-ure, TAA module pays more attention to the missing regionand further improve the performance of the model.

4.4 Temporal ConsistencyTo enforce temporal consistency, we update the input videoframes with completed frame at each iteration, and proposeTAA module to propagate information between consecutiveframes via implicit alignment. We also process the framesone by one and run the network from the first frame to thelast frame, then reverse the order during the test stage.

To show the temporal consistency between frames, wesample the slices of video frames along the blue line, andstack them along the vertical-axis as [Huang et al., 2016].In Figure 6, our method performs more smoother result thanimage inpainting based method [Nazeri et al., 2019], whichindicates our method shows better temporal consistency.

5 ConclusionIn this paper, we present an end-to-end temporal alignmentnetwork for video inpainting. A TAA module is proposedbased on deformable convolution to perform temporal adap-tive alignment in a feature domain without explicit motionestimation (e.g., optical flow). Our network significantly uti-lizes the temporal information from reference frames basedon TAA module, which is important for video inpainting, andproduces visually pleasing and temporally coherent results.In the future research, we will extend our framework to solveother video restoration tasks in practical applications.

AcknowledgementsThis work was supported in part by NSFC-Shenzhen RobotJointed Founding under Grant U1613215, in part by the Shen-zhen Municipal Development and Reform Commission (Dis-ciplinary Development Program for Data Science and Intelli-gent Computing), and in part by the Key-Area Research andDevelopment Program of Guangdong Province under Grant2019B010137001.


932

References[Barnes et al., 2009] Connelly Barnes, Eli Shechtman,

Adam Finkelstein, and Dan B Goldman. Patchmatch:A randomized correspondence algorithm for structuralimage editing. In ACM Transactions on Graphics (ToG),volume 28, page 24, 2009.

[Chang et al., 2019] Ya-Liang Chang, Zhe Yu Liu, Kuan-Ying Lee, and Winston Hsu. Free-form video inpaintingwith 3d gated convolution and temporal patchgan. In Pro-ceedings of the IEEE International Conference on Com-puter Vision, pages 9066–9075, 2019.

[Dai et al., 2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong,Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. De-formable convolutional networks. In Proceedings of theIEEE International Conference on Computer Vision, pages764–773, 2017.

[Granados et al., 2012] Miguel Granados, Kwang In Kim,James Tompkin, Jan Kautz, and Christian Theobalt. Back-ground inpainting for videos with dynamic objects and afree-moving camera. In European Conference on Com-puter Vision, pages 682–695. Springer, 2012.

[He et al., 2015] Kaiming He, Xiangyu Zhang, ShaoqingRen, and Jian Sun. Delving deep into rectifiers: Surpass-ing human-level performance on imagenet classification.In Proceedings of the IEEE International Conference onComputer Vision, pages 1026–1034, 2015.

[Huang et al., 2016] Jia-Bin Huang, Sing Bing Kang, Naren-dra Ahuja, and Johannes Kopf. Temporally coherent com-pletion of dynamic video. ACM Transactions on Graphics(TOG), 35(6):196, 2016.

[Iizuka et al., 2017] Satoshi Iizuka, Edgar Simo-Serra, andHiroshi Ishikawa. Globally and locally consistent im-age completion. ACM Transactions on Graphics (ToG),36(4):107, 2017.

[Kim et al., 2019] Dahun Kim, Sanghyun Woo, Joon-YoungLee, and In So Kweon. Deep video inpainting. In Pro-ceedings of the IEEE Conference on Computer Vision andPattern Recognition, pages 5792–5801, 2019.

[Lee et al., 2019] Sungho Lee, Seoung Wug Oh, DaeYeunWon, and Seon Joo Kim. Copy-and-paste networks fordeep video inpainting. In Proceedings of the IEEE Interna-tional Conference on Computer Vision, pages 4413–4421,2019.

[Liu et al., 2018] Guilin Liu, Fitsum A Reda, Kevin J Shih,Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Im-age inpainting for irregular holes using partial convolu-tions. In Proceedings of the European Conference onComputer Vision, pages 85–100, 2018.

[Nazeri et al., 2019] Kamyar Nazeri, Eric Ng, Tony Joseph,Faisal Qureshi, and Mehran Ebrahimi. Edgeconnect:Structure guided image inpainting using edge prediction.In Proceedings of the IEEE International Conference onComputer Vision Workshops, pages 0–0, 2019.

[Newson et al., 2014] Alasdair Newson, Andres Almansa,Matthieu Fradet, Yann Gousseau, and Patrick Perez. Video

inpainting of complex scenes. SIAM Journal on ImagingSciences, 7(4):1993–2019, 2014.

[Oh et al., 2019] Seoung Wug Oh, Sungho Lee, Joon-YoungLee, and Seon Joo Kim. Onion-peel networks for deepvideo completion. In Proceedings of the IEEE Interna-tional Conference on Computer Vision, pages 4403–4412,2019.

[Pathak et al., 2016] Deepak Pathak, Philipp Krahenbuhl,Jeff Donahue, Trevor Darrell, and Alexei A Efros. Contextencoders: Feature learning by inpainting. In Proceedingsof the IEEE Conference on Computer Vision and PatternRecognition, pages 2536–2544, 2016.

[Patwardhan et al., 2005] Kedar A Patwardhan, GuillermoSapiro, and Marcelo Bertalmio. Video inpainting of oc-cluding and occluded objects. In IEEE International Con-ference on Image Processing, pages 69–72, 2005.

[Patwardhan et al., 2007] Kedar A Patwardhan, GuillermoSapiro, and Marcelo Bertalmıo. Video inpainting underconstrained camera motion. IEEE Transactions on ImageProcessing, 16(2):545–553, 2007.

[Perazzi et al., 2016] Federico Perazzi, Jordi Pont-Tuset,Brian McWilliams, Luc Van Gool, Markus Gross, andAlexander Sorkine-Hornung. A benchmark dataset andevaluation methodology for video object segmentation. InProceedings of the IEEE Conference on Computer Visionand Pattern Recognition, pages 724–732, 2016.

[Wang et al., 2019] Chuan Wang, Haibin Huang, XiaoguangHan, and Jue Wang. Video inpainting by jointly learn-ing temporal structure and spatial details. In Proceedingsof the AAAI Conference on Artificial Intelligence, pages5232–5239, 2019.

[Wexler et al., 2007] Yonatan Wexler, Eli Shechtman, andMichal Irani. Space-time completion of video. IEEETransactions on Pattern Analysis & Machine Intelligence,29(3):463–476, 2007.

[Xu et al., 2018] Ning Xu, Linjie Yang, Yuchen Fan,Dingcheng Yue, Yuchen Liang, Jianchao Yang, andThomas Huang. Youtube-vos: A large-scale videoobject segmentation benchmark. arXiv preprintarXiv:1809.03327, 2018.

[Xu et al., 2019] Rui Xu, Xiaoxiao Li, Bolei Zhou, andChen Change Loy. Deep flow-guided video inpainting.pages 3723–3732, 2019.

[Yu et al., 2018] Jiahui Yu, Zhe Lin, Jimei Yang, XiaohuiShen, Xin Lu, and Thomas S Huang. Generative imageinpainting with contextual attention. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recog-nition, pages 5505–5514, 2018.

[Zhu et al., 2019] Xizhou Zhu, Han Hu, Stephen Lin, andJifeng Dai. Deformable convnets v2: More deformable,better results. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition, pages 9308–9316, 2019.


933

temporal adaptive alignment network for deep video ...temporal redundancy information can be...

Documents