University of LjubljanaFaculty of Mathematics and Physics

Department of Physics

Seminar Ib - 1. year, 2nd grade

Eulerian Video Magnification

Author: Tilen Brecelj

Mentor: doc. dr. Daniel Svensek

Ljubljana, April 2013

Abstract

The seminar talks about Eulerian Video Magnification, which is a computationalmethod, that reveals subtle temporal motions and colour changes in videos, that are im-possible or very difficult to see with the naked eye. It presents, how this method analysesthe video sequence by performing spatial decomposition and temporal filtering to videoframes and amplifies the perceived motions or colour changes, that would otherwise re-main unseen.

Contents

1 Introduction 2

2 Human Eye and Cameras Spatial resolution 2

3 The basic idea 3

4 Mathematical basis 6

4.1 Signal amplification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

4.2 Amplification bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

5 Filter and parameters selection 10

6 Sensitivity to noise 12

7 Conclusion 13

References 14

1

1 Introduction

Eulerian Video Magnification [1] is a computational method, which reveals subtle tem-poral motions and colour changes in videos, that are impossible or very difficult to seewith the naked eye. This method analyses the video sequence by performing spatial de-composition and temporal filtering, which reveals changes in specific temporal intervals,to video frames. The perceived subtle motions or colour changes are then amplified anddisplayed in a way, to be easily seen with a naked eye. Subtle (almost) invisible motion cangive us a lot of important information. For example human skin slightly changes colourwith heartbeats and blood circulation, which cannot be seen with a naked eye, but canserve to extract pulse rate or diagnose an irregularity in blood flow. There are also a lotof interesting motions with very low spatial amplitudes and as such cannot be seen with anaked eye, but can reveal a lot about certain mechanisms or mechanical behaviours thatcan serve in different fields of research such as medicine, biology, engineering etc.

2 Human Eye and Cameras Spatial resolution

To get an idea of why something cannot be seen with a naked eye, but can be perceived bya camera, let us take a look at the capabilities of a human eye and cameras to distinguishsubtle motions. Let us compare their spatial resolutions, which describes the ability ofan image device or organ, to distinguish between two points, that are located at a smallangular distance - the smaller is the angular distance, the higher is the spatial resolutionand more details can be seen. For describing spatial resolution mathematically, we canuse the Rayleigh’s Criterion. When light enters a lens, a diffraction pattern is produced.The Rayleigh’s Criterion defines the minimal distance between diffraction patterns of twodistinguishable objects as the distance, when the diffraction minimum of one source pointcoincides with the diffraction maximum of the other source point. According to Rayleigh’sCriterion, we can calculate the minimal angular distance Θ, which must separate twodistinguishable points of an object as

Θ = 1.22λ

D, (1)

where λ is the wavelength of the light and D is the diameter of the light gatheringregion - eye pupils or camera aperture. In our case the wavelength will bi taken as λ = 550nm, because it is the approximate wavelength of the human eye’s greatest sensitivity.

The eye pupils’ diameter depends on the brightness of the environment and varies from3 to 9 mm, but for our calculations will be used the smallest diameter possible, when ourangular resolution is the smallest, so Dh = 3 mm. When inserting these two parameters inequation 1 and presuming the best observational conditions and a very good sight, we gethumans’ spatial resolution, Θh = 2.24×10−4 rad or 1.28×10−2 deg which is approximately46”. For a better illustration, this would be the angular distance between two points thatare 1 mm apart and cca 4.46 m away from our eye. The properties of human eye lightperceptive cells have no influence to the spatial resolution. The spatial resolution on thehuman light perceptive cells can be expressed as ∆l = Θf , where f is the focal length of

2

the eye and is approximately 22 mm. Inserting the angular resolution calculated aboveand the focal length, we get ∆l ≈ 5×10−6 m. In contrast, the size of a light perceptive cellis about 2× 10−6 m, so enough small to perceive the angular resolution as a consequenceof eye pupils’ diameter.

Let us now take a look at cameras spatial resolution, that can also depend on theirdetector properties. That is why we have to discuss spatial resolutions on cameras detec-tors, ∆l, as a consequence of cameras aperture size and camera sensor pixel sizes. Spatialresolution on cameras detectors is expressed as ∆l = Θf , where f is the focal length ofthe camera. Inserting the equation 1, we get

∆l = 1, 22fλ

D(2)

where f/D is a typical property of cameras called focal ratio (or relative aperture) andvaries for middle class cameras from 1.4 to 22. We will take the smallest ratio for ourcalculations, to get the minimal distance ∆l (and as such, the greatest spatial resolution).Inserting these numbers in equation 2, we get ∆l = 0.94 µm. This means that a typicaldetector with pixel size approximately 5 µm, would not be able to distinguish betweenthese two points and that middle class cameras’ resolutions are not determined by thediameter of their aperture, but by the pixel sizes of their detectors. Let us assume, thatthe distance between two photons on the detector, ∆ldet, to be perceived as photons fromdifferent sources, must be two pixel sizes, so ∆ldet ≈ 10 µm, for an average middle classcamera. This way, they can be detected by two pixels, that lay one pixel apart. Knowingthe value of a typical focal length, f , of a middle class camera, which is about 15 mm, wecan calculate the spatial resolution of an average camera, determined by the pixel size asΘ = ∆ldet/f . This estimate leads to the result, Θ ≈ 6.7×10−4 rad or 3.8×10−2 deg whichis approximately 2’17”. This means that the Eulerian video magnification algorithm doesnot base on cameras capabilities to reveal unseen motion, but on other principles.

As detection of motion, colour detection does not base on the capabilities of the cameraeither (the signal to be revealed has frequently lower changes of amplitudes than the noise),so the colour magnification algorithm also bases on other principles.

3 The basic idea

The basic approach of Eulerian Video Magnification for colour magnification is to amplifythe variation of colour values at any spatial location (pixel) in a temporal frequency band,that suits a certain phenomenon. For example, by amplifying the skin colour in a bandof temporal frequencies, that include plausible human heart rates, reveals the variation inredness as blood flows through the face - figure 1.

3

Figure 1: An example of how Eulerian Video Magnification amplifies subtle colour changesas a consequence of heart bits and blood flow. In (a) we can see four frames of the originalinput video sequence; in (b) we can see the same frames as in (a) with amplified coloursby a factor α = 100; in (c) we can see a vertical scan line from the original video (top) andfrom the amplified video (bottom), plotted over time - the colour variation in the bottomfigure can be very well seen. Source: [1]

As mentioned before, Eulerian Video Magnification can also reveal low-amplitude mo-tion. Figure 2 shows how subtle motion of a blood vessel can be magnified.

Figure 2: This two figures show how Eulerian Video Magnification can amplify subtle mo-tions of blood vessels arising from blood flow - to reduce motion magnification of irrelevantobjects, only the area around the wrist was amplified. Figure (a) shows how a slice ofthe wrist evolves in time; figure (b) shows the same slice, after motion was magnified by afactor α = 10, evolving in time. The temporal filter was tuned to a frequency band thatincludes the heart rate — 0.88 Hz (53 bpm). Source: [1]

The technique used to extract and reveal the wanted signal is localised spatial pool-

4

ing1 and bandpass filtering2. The whole process is schematically shown on figure 3. Toobtain a good quality of the output video in a reasonable time period, before analysing,the videos should be downsampled and filtered by a spatial lowpass filter (Laplacian pyra-mid3), to reduce the noise and to boost the subtle changes in the video. Than the videois decomposed into different bands of wavenumbers k (k = 2π

λ). After spatial processing,

temporal processing is performed on each band of wavenumbers. A bandpass temporalfilter is applied to each band of wavenumbers, to extract (pass) the motions, that suit thefrequency bands of the observed phenomenon. Motions of different passed wavenumbersare then magnified differently because of two reasons: firstly, they might have differentsignal-to-noise ratios and secondly, they might contain wavenumbers for which the linearapproximation used in motion magnification does not hold. In the second case, the mag-nification is reduced to suppress artifacts. After the magnification, magnified bandpassedsignals are added to the original signal.

Figure 3: Eulerian video magnification framework. The video is firstly decomposed intodifferent bands of wavenumbers. The same temporal filter is than applied to all bands ofwavenumbers, to reveal the time interval of the motion and the motion of each band ofwavenumbers. Than a passband filter is applied to pass the bands of wavenumbers thatsuit the time interval of the observed phenomenon. The filtered bands of wavenumbersare then amplified by a given factor α, added back to the original signal and collapsed togenerate the output video. Source: [1]

1Spatial pooling identifies frequently observed patterns and memorizes them as coincidences. Patternsthat are significantly similar to each other are treated as the same coincidence. [8]

2A band-pass filter is a filter, that passes frequencies within a certain range and rejects (attenuates)frequencies outside that range. [9]

3Laplacian pyramid is a set of band pass filters, used while pooling pixels into one. [10]

5

4 Mathematical basis

4.1 Signal amplification

Let us now describe mathematically how Eulerian motion magnification works for trans-lational motion in one dimension (this analysis can be than generalized directly to twodimensions). The image intensity function I(x, t) determines the image intensity at thelocation x at time t. After translational motion, lasting t, the initial image intensity func-tion I(x, 0) = f(x) evolves in I(x, t) = f(x+δ(t)), where δ(t) is the displacement function.The goal of the Eulerian video magnification is to get the magnified intensity function

I(x, t) = f(x+ (1 + α)δ(t)), (3)

with the magnification factor α. If the image intensity function can be approximatedby a first order Taylor series expansion about x, we can express it at time t as

I(x, t) ≈ f(x) + δ(t)∂f(x)

∂x. (4)

When we apply a broadband temporal bandpass filter to the image intensity functionI(x, t), over the whole area of interest, we get the intensity variation function B(x, t), whichexpresses the change of the image intensity over the whole area of interest, after time t. Inother words, by applying a broadband temporal bandpass filter, we pick form equation 4everything except f(x). If the displacement δ(t) is within the passband4 of the broadbandtemporal bandpass filter, we can express the intensity variation function B(x, t) as

B(x, t) = δ(t)∂f(x)

∂x(5)

To get the amplified intensity function I(x, t), all we have to do is sum the originalintensity function I(x, t) and the intensity variation function B(x, t), amplified by factorα, as

I(x, t) = I(x, t) + αB(x, t). (6)

After inserting equations 4, 5 and 6, we can express our amplified intensity function as

I(x, t) ≈ f(x) + (1 + α)δ(t)∂f(x)

∂x. (7)

Finally, if the first order Taylor expansion of the temporally bandpassed image intensityfunction I(x, t) is a good approximation of the magnified intensity function I(x, t), themagnified motion is

I(x, t) ≈ f(x+ (1 + α)δ(t)). (8)

This means that the spatial displacement δ(t) of the initial image, denoted by theintensity function f(x), after time t, has been amplified to the displacement (1 + α)δ(t).

4A passband is the range of frequencies that can pass through a filter without being attenuated [11]

6

On figure 4 we can see, how this method works in practice for a single sinusoid and howthe amplified signal looks like. We can notice, that the approximated signal with the firstorder Taylor series expansion at time t+ 1 matches very well with the input signal at thesame time.

Figure 4: The figure above shows, how can spatial translation be approximated and am-plified in one dimension (it can be equally applied to two dimensions). The input sig-nal is shown at times t as I(x, t) = f(x) and t + 1 as I(x, t + 1) = f(x + δ), the ap-proximated signal with the first order Taylor series expansion is shown at time t + 1 asI(x, t) ≈ f(x) + δ(t)∂f(x)

∂xand the magnified approximation of the signal at time t + 1 is

shown as I(x, t) ≈ f(x) + (1 +α)B(x, t), for α = 1. The temporal filter used to get B(x, t)is a finite difference filter, that subtracts the curves f(x) and f(x+ δ). Source: [1]

All of the above assumptions hold in practice for smooth images (i.e. images with slowchanges in contrast) and small motions. The inaccuracy of the magnification increaseswith the magnification factor α and the displacement function δ(t). This is illustrated infigure 5.

7

Figure 5: The figures above illustrate motion amplification on a one dimensional signal fordifferent wavenumbers and magnification factors α. For the images on the left side, λ = 2πand δ(1) = π

8, for the images on the right side λ = π and δ(1) = π

8. Figure (a) shows

the true displacement of I(x, 0) from equation 3 at time t = 1, for different amplificationfactors α, denoted with different colours. Figure (b) shows the amplified displacementproduced by the filter from equation 7. Different amplification factors α have the samecolours as in figure (a). Referencing equation 13, the red curves of each plot correspondto (1 + α)δ(t) = λ

4for the left plot and (1 + α)δ(t) = λ

2for the right plot, showing the

artifacts introduced in the motion magnification from exceeding the bound on (1 + α) byfactors 2 and 4, respectively. Source: [1]

4.2 Amplification bounds

Let us now derive the maximum value of α as a function of wavenumber k for a givenobserved motion δ(t), so that the gained amplification error would not be too large. Letus start with the approximate equalisation of the processed signal I(x, t) (equation 7) andthe true magnified motion I(x, t) (equation 3)

f(x) + (1 + α)δ(t)∂f(x)

∂x≈ f(x+ (1 + α)δ(t)). (9)

Let us take for example f(x) = cos(kx) and denote β = 1 + α. We get

cos(kx)− βkδ(t) sin(kx) = cos(kx) cos(βkδ(t))− sin(kx) sin(βkδ(t)). (10)

8

Hence, the following should hold

cos(βkδ(t)) ≈ 1 (11)

sin(βkδ(t)) ≈ βkδ(t) (12)

If we want for example our small angle approximation to hold within 10%, the followingvalues should be βkδ(t) ≤ π

4(as sin(π

4) = 0.9π

4). Denoted with the spatial wavelength of

the moving signal λ = 2πk

, this gives

(1 + α)δ(t) ≤ λ

8. (13)

This way we get the largest motion magnification factor α, that magnifies motionfor the given accuracy. On figure 6 we can see, how the error increases with increasingmagnification factor α and displacement function δ(t).

Figure 6: Motion magnification error as function of wavelength, computed as L1-norm(distance) between the true motion amplified signal (figure 5(a)) and the temporally filteredresult (figure 5(b)): (a) for different values of δ(t), where α = 1; (b) for different values ofα, where δ(t) = 2. The triangular marks on each curve represent the cutoff point derivedin equation 13. Source: [1]

In practice, the amplification factor α is fixed (at an optional value, as we define it)for spatial bands that are within the bounds of equation 13 and is attenuated linearly forhigher wavenumbers, as shown in figure 7. This way we get rid of the undesirable artifactsthat would arise for high wavenumbers, seen on figure 6(b).

9

Figure 7: Amplification factor α as function of spatial wavelength λ. The amplificationfactor is fixed for spatial bands that are within the derived bound in equation 13 and isattenuated linearly for higher wavenumbers. Source: [1]

5 Filter and parameters selection

To enable the amplification of motion from a video, the user has to select some of thebasic parameters of the Eulerian video magnification algorithm.

Firstly, a temporal bandpass filter that suits the observed phenomenon must be selected,to extract the desired motions or signals - the importance of the coincidence of the temporalfilter passband and the observed phenomenon temporal interval can be seen on figure 8.

Figure 8: (a) shows a frame from a video, on which blobs oscillate at different temporalfrequencies, as noted under each blob. Figure (b) shows the spatio-temporal slices fromthe magnified motion, after an ideal temporal bandpass filter of 1 − 3 Hz was applied, toamplify only the motions occurring within the specified passband. Source: [1]

There exist many different temporal filters that can be used, depending of the signalto be extracted. For broad but subtle motion magnification, temporal filters with a broad

10

passband are preferred. For colour amplification, for example of blood flow, narrow tempo-ral passband filters are preferred, to produce a more noise free result. Motions at specifictemporal frequencies (such as colour amplification) demands ideal temporal bandpass fil-ters with sharp cut off frequencies. There are also some other types of filters, for exampleIIR filters5, that are used for both, motion and colour magnification. On figure 9 we cansee some types of temporal filters.

Figure 9: Different types of temporal filters. (a) and (b) show ideal filters with sharp cut offfrequencies, (c) shows a so called Butterworth filter, that has as flat a frequency responseas possible in the passband [13] and (d) shows a so called second-order IIR filter with abroad frequency passband. Under each image there is a note in brackets that specifies onwhich video was each filter used. All of the videos with the same names as noted in thebrackets can be found on [3]. Some sequences from these videos can also be seen on figure1 - face and figure 2 - pulse detection. Source: [1]

Secondly, the user must select the amplification factor α and a wavenumber cut off(specified by spatial wavelength λc) beyond which an attenuated version of α is used - itcan be either forced to zero for all λ < λc (this was used for the human pulse amplification,shown in figure 1), or linearly scaled down to zero. Note that equation 13 is an examplederived for a sine wave for a small angle approximation that holds within 10% and is nota principle for how α and λc are connected and can only be used as a guide. Variouscombinations of α and λc can be used, also those that violate the bounds to exaggeratethe specific motions or colour changes at the cost of increasing noise or introducing moreartifacts.

5IIR filters - infinite impulse response filters are filters, with an infinite impulse response function, thatis non-zero over an infinite range of frequencies [12]

11

6 Sensitivity to noise

To extract as many information from a video as possible, this should be filmed in the bestconditions, i.e. in an enough bright environment and with a low ISO value, to produce asminimal noise as possible. But even so, Eulerian video magnification can amplify signals,that have smaller magnitude variations than the noise inherent in the video. Let us assumethat the noise is zero mean white and equally distributed all over the image. With suitablespatial low pass filters, the subtle signals can be enhanced over the area of filtering. Bycomputing the sum x of N pixels as x = Nx0, where x0 is the signal, and the sum of noisepower σ2 over N pixel values as σ2 = Nσ2

0, where σ20 is the average noise power of a pixel,

we can increase the signal to noise ratio as

x

σ=

Nx0√Nσ2

0

=√Nx0σ0. (14)

We can see, that the ratio increases with the square root of the number of pixels taken.As the number of pixels is proportional to the size of the area, N ∝ r, over which thefiltering is made, we can assume, that the bigger is the area of averaging, the greater isthe signal to noise ratio. Considering equation 14 and the relation between the number ofpixels N and the area size r, we can also estimate the area size that needs to be filtered,to reveal the signal at a certain noise power level. The importance of the correctly chosenfilter size is shown on figure 10.

Figure 10: This figure shows the importance of proper spatial pooling. (a) shows a framefrom a video on which colour magnification was performed with σ = 0.1 pixel noise added.(b) and (c) show intensity traces over time, for the pixel on (a) marked blue. (b) showsthe trace obtained when the (noisy) sequence is processed with the same spatial filter usedto process the original face sequence. In this trace the pulse signal is not visible. (c) showsthe trace when using a filter with the filter size estimated in equation 14. The pulse isvisible as periodic peaks about one second apart. Source: [1]

12

7 Conclusion

The seminar describes the Eulerian video magnification, a straightforward method thatamplifies subtle changes of signal (either colour changes or translational motions) by per-forming spatial and temporal processing, with no need of feature tracking and motionprediction, as other, computationally expensive methods with similar goals created so farused to do (for example Lagrangian method of processing). The seminar also compares thehuman eye spatial and cameras spatial resolution. Further, it describes the basic idea ofthe Eulerian video magnification algorithm, discusses its mathematical bases and derivesthe bounds of magnification, within which the magnification does not product to many ar-tifacts. In the end we become aware of some typically used filters and get an idea of how todeal with noise. With more improvements, Eulerian video magnification could be in futureused on different fields of science to reveal subtle, unseen motions or colour changes withthe naked eye, that could help us explore different subtle dynamical processes, understandcertain mechanism, diagnose illnesses etc.

13

References

[1] Hao-Yu Wu et al: Eulerian Video Magnification for Revealing Subtle Changes in theWorld (ACM Transactions on Graphics (TOG) - SIGGRAPH 2012 Conference Pro-ceedings, Volume 31 Issue 4, Article No. 65, July 2012)

[2] Hao-Yu Wu et al: Eulerian Video Processing and Medical Applications (Degree ofMaster of Engineering at Massachusetts Institute of Technology, June 2012)

[3] http://people.csail.mit.edu/mrub/vidmag/

[4] http://hyperphysics.phy-astr.gsu.edu/hbase/phyopt/raylei.html

[5] http://webphysics.davidson.edu/physlet_resources/bu_semester2/c27_

rayleigh.html

[6] http://www.wikilectures.eu/index.php/Resolution_of_human_eye

[7] http://en.wikipedia.org/wiki/Angular_resolution

[8] http://en.wikipedia.org/wiki/Hierarchical_temporal_memory

[9] http://en.wikipedia.org/wiki/Band-pass_filter

[10] http://www.cs.utah.edu/~arul/report/node12.html

[11] http://en.wikipedia.org/wiki/Passband

[12] http://en.wikipedia.org/wiki/Infinite_impulse_response

[13] http://en.wikipedia.org/wiki/Butterworth_filter

[14] http://www.cambridgeincolour.com/tutorials/cameras-vs-human-eye.htm

[15] http://en.wikipedia.org/wiki/Retina

Note: All the web pages were active in April, 2013.

14