eulerian video magnification - iit kanpurmshishir/cs300a/... · this variation, while invisible to...
TRANSCRIPT
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Eulerian Video Magnification
Shishir Mathur
Authored byHao-Yu Wu Michael Rubinstein Eugene Shih John Guttag
Frdo Durand William T. FreemanPublished in SIGGRAPH 2012
MIT CSAIL
October 20, 2014
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Outline
1 Abstract
2 Eulerian Video Magnification for Revealing Subtle Changesin the World
IntroductionSpace-time video processing
3 Eulerian motion magnificationFirst-order motion
4 Final Results
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Contents
1 Abstract
2 Eulerian Video Magnification for Revealing Subtle Changesin the World
IntroductionSpace-time video processing
3 Eulerian motion magnificationFirst-order motion
4 Final Results
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Introduction
Figure : An example of using our Eulerian Video Magnificationframework for visualizing the human pulse. (a) Four frames from theoriginal video sequence. (b) The same four frames with the subject’spulse signal amplified. (c) A vertical scan line from the input (top)and output (bottom) videos plotted over time shows how our methodamplifies the periodic color variation. In the input sequence the signalis imperceptible, but in the magnified sequence the variation is clear.
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Abstract
The goal of the paper is to reveal temporal variations in videosthat are difficult or impossible to see with the naked eye anddisplay them in an indicative manner. The method, is calledEulerian Video Magnification, takes a standard video sequenceas input, and applies spatial decomposition, followed bytemporal filtering to the frames. The resulting signal is thenamplified to reveal hidden information. Using this method, weare able to visualize the flow of blood as it fills the face andalso to amplify and reveal small motions. The technique canrun in real time to show phenomena occurring at temporalfrequencies selected by the user.
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Contents
1 Abstract
2 Eulerian Video Magnification for Revealing Subtle Changesin the World
IntroductionSpace-time video processing
3 Eulerian motion magnificationFirst-order motion
4 Final Results
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Introduction
The human visual system has limited spatio-temporalsensitivity, but many signals that fall below this capacity can beinformative.For example, human skin color varies slightly withblood circulation. This variation, while invisible to the nakedeye, can be exploited to extract pulse rate In this paper, we seea combination of spatial and temporal processing of videos canamplify subtle variations that reveal important aspects of theworld around us.
Our basic approach is to consider the time series of color valuesat any spatial location (pixel) and amplify variation in a giventemporal frequency band of interest.The temporal filtering approach not only amplifies colorvariation,but can also reveal low-amplitude motion.
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Introduction
The human visual system has limited spatio-temporalsensitivity, but many signals that fall below this capacity can beinformative.For example, human skin color varies slightly withblood circulation. This variation, while invisible to the nakedeye, can be exploited to extract pulse rate In this paper, we seea combination of spatial and temporal processing of videos canamplify subtle variations that reveal important aspects of theworld around us.Our basic approach is to consider the time series of color valuesat any spatial location (pixel) and amplify variation in a giventemporal frequency band of interest.
The temporal filtering approach not only amplifies colorvariation,but can also reveal low-amplitude motion.
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Introduction
The human visual system has limited spatio-temporalsensitivity, but many signals that fall below this capacity can beinformative.For example, human skin color varies slightly withblood circulation. This variation, while invisible to the nakedeye, can be exploited to extract pulse rate In this paper, we seea combination of spatial and temporal processing of videos canamplify subtle variations that reveal important aspects of theworld around us.Our basic approach is to consider the time series of color valuesat any spatial location (pixel) and amplify variation in a giventemporal frequency band of interest.The temporal filtering approach not only amplifies colorvariation,but can also reveal low-amplitude motion.
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Space-time video processing
Figure : Overview of the Eulerian video magnification framework.The system first decomposes the input video sequence into differentspatial frequency bands, and applies the same temporal filter to allbands. The filtered spatial bands are then amplified by a given factor,added back to the original signal, and collapsed to generate theoutput video. The choice of temporal filter and amplification factorscan be tuned to support different applications.
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Space-time video processing
Our approach combines spatial and temporal processing toemphasize subtle temporal changes in a video. The process isillustrated in the Figure.We first decompose the video sequenceinto different spatial frequency bands. These bands might bemagnified differently because (a) they might exhibit differentsignal-to-noise ratios or (b) they might contain spatialfrequencies for which the linear approximation used in ourmotion magnification does not hold (Sect. 3). In the lattercase, we reduce the amplification for these bands to suppressartifacts. When the goal of spatial processing is simply toincrease temporal signal-to-noise ratio by pooling multiplepixels, we spatially low-pass filter the frames of the video anddownsample them for computational efficiency.
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Contents
1 Abstract
2 Eulerian Video Magnification for Revealing Subtle Changesin the World
IntroductionSpace-time video processing
3 Eulerian motion magnificationFirst-order motion
4 Final Results
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
First-order motion
To explain the relationship between temporal processing andmotion magnification, we consider the simple case of a 1Dsignal undergoing translational motion. This analysisgeneralizes directly to locally-translational motion in 2D.
Figure : Temporal filtering can approximate spatial translation. Thiseffect is demonstrated here on a 1D signal, but equally applies to 2D.The input signal is shown at two time instants
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
First-order motion
To explain the relationship between temporal processing andmotion magnification, we consider the simple case of a 1Dsignal undergoing translational motion. This analysisgeneralizes directly to locally-translational motion in 2D.
Figure : Temporal filtering can approximate spatial translation. Thiseffect is demonstrated here on a 1D signal, but equally applies to 2D.The input signal is shown at two time instants
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Contents
1 Abstract
2 Eulerian Video Magnification for Revealing Subtle Changesin the World
IntroductionSpace-time video processing
3 Eulerian motion magnificationFirst-order motion
4 Final Results
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Results
The results were generated using non-optimized MATLAB codeon a machine with a six-core processor and 32 GB RAM. Thecomputation time per video was on the order of a few minutes.We used a separable binomial filter of size five to construct thevideo pyramids. We also built a prototype application thatallows users to reveal subtle changes in real-time from livevideo feeds, essentially serving as a microscope for temporalvariations. It is implemented in C++, is entirely CPU-based,and processes 640X480 videos at 45 frames per second on astandard laptop
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
Figure : Eulerian video magnification used to amplify subtle motionsof blood vessels arising from blood flow. For this video, we tuned thetemporal filter to a frequency band that includes the heart rate0.88Hz (53 bpm)and set the amplification factor to alpha= 10. To reducemotion magnification of irrelevant objects, we applied a user-givenmask to amplify the area near the wrist only. Movement of the radialand ulnar arteries can barely be seen in the input video (a) taken witha standard point-and-shoot camera, but is significantly morenoticeable in the motion-magnified output (b).
Eulerian VideoMagnification
ShishirMathur
Abstract
Eulerian VideoMagnificationfor RevealingSubtleChanges inthe World
Introduction
Space-time videoprocessing
Eulerianmotionmagnification
First-ordermotion
Final Results
The end
Thank You