A SMARTPHONE APPLICATION FOR REMOVING HANDSHAKE BLUR ANDCOMPENSATING ROLLING SHUTTER

Ondrej Sindelar?, Filip Sroubek?, Peyman Milanfar† ∗

?UTIA, Academy of Sciences of the Czech Republic, Prague, Czech Republic†University of California, Santa Cruz, U.S.A.

ABSTRACT

Smartphones are now widely used as photographic devices.Equipped with cheap cameras they are prone to many degra-dations, most notably handshake in combination with rollingshutter causes severe space-variant blur. Removing blur with-out any information about the camera motion is a computa-tionally demanding and unstable process. We use built-in gy-roscopes to record the motion trajectory of the camera duringexposure and then remove blur from the acquired photographbased on the reconstructed trajectory. The proposed deblur-ring application is implemented on Android smartphones withclose-to-real-time performance.

Index Terms— space-variant deconvolution; gyroscope;mobile phone; rolling shutter

1. INTRODUCTION

One of the most frequent problems in photography is blurinduced by camera motion under poor light conditions. Asthe exposure time increases, involuntary camera motion hasa growing effect on the acquired image. Image stabilization(IS) devices that help to reduce the motion blur by moving thecamera sensor in the opposite direction are becoming morecommon. However, such hardware remedy has its limita-tions as it can compensate only for motion of a very smallextent and speed. In addition, mobile phones are currently notequipped with IS. Deblurring the image offline using mathe-matical algorithms is usually the only choice we have in orderto obtain a sharp image. Arbitrary camera motion blur can bemodeled by space-variant (SV) convolution and the deblur-ring process is referred to as SV deconvolution [1]. This isa hard ill-posed problem if the blur shape is completely un-known.

Camera motion blur is SV for several reasons. First, itis caused by the camera projection itself. Phone cameras areusually equipped with wide-angle lenses (field of view around60◦), which distort objects close to image borders. The blurcaused by rotation around x and y axes is therefore different

∗This work was supported in part by the Academy of Sciences of theCzech Republic under project M100751201 and by the Grant Agency of theCzech Republic under project GA13-29225S.

in the image center and borders. The SV blur are particularlynoticeable when rotation around z axis is significant. Sec-ond, the camera-object distance influences the blur caused bycamera translation and the knowledge of depth map is thusnecessary. However, phone cameras have a focal length of afew millimeters and the scene projected into the camera im-age plane moves by less than a pixel if the objects are morethan 2m away, so the camera translation in such cases canbe neglected [2]. In our work we consider purely rotationalmotion of the camera, which has additional advantages. Un-like accelerometers, gyroscopes are sufficiently accurate forangular speed estimation but drift. We use gyroscope data toestimate rotation and compensate for the drift by calibratinga still camera.

Another reason for SV blur, unrelated to camera motionbut intrinsic to camera hardware design, is rolling shutter [3].In image sensors on mobile devices, contrary to systems withmechanical shutter, values of illuminated pixels are read suc-cessively line by line while the sensor is exposed to light. Thereadout from the CMOS sensor takes several tens of millisec-onds, which results in a picture not taken at a single moment,but with a slight time delay between the first and last row ofpixels. The rolling shutter effect is therefore another cause ofspace variance as the blur depends on the vertical position inthe image. An example in Fig. 1 illustrates the rolling shuttereffect. We took a snapshot of a LCD screen displaying a gridof white points on black background. Due to camera motion,the points appear as streaks on the captured image. To modelaccurately the blur at every position, it is necessary to shiftthe exposure-time window in which the gyroscope data arefetched according to the vertical position.

Our work demonstrates the use of built-in gyroscopes insmartphones for accurate blur estimation. The proposed solu-tion is simple and practical. It removes blur induced by cam-era rotation and simultaneously overcomes rolling-shutter ef-fect, which, to our knowledge, has not been considered in thedeconvolution problem before. As a testing platform we havechosen a Samsung Galaxy S II smartphone with Android op-erating system.

A similar system was proposed by Joshi et al. in [4] butthey have designed an expensive measuring apparatus con-sisting of a DSLR camera and an external inertial module,

Fig. 1. Rolling shutter effect: A snapshot (exposure time1/14s) of a point grid. The right column shows a series ofblur kernels rendered using data from the gyroscope sensorshifted in time. Blurs were created from sensor data starting0–60 ms after a synchronization timestamp.

and perform image deblurring offline on a computer. Con-trary to low-cost cameras, rolling shutter is not present inDSLR cameras. Sindelar et al. [2] tested simple deconvo-lution running on smartphones, but they have considered onlyspace-invariant blur, which limits applicability of their solu-tion. A different approach to minimize handshake in smart-phones was proposed in [5], where they take a burst of short-exposure noisy images, align them using gyroscope data andaverage them. Gyro-based deconvolution in [6] assumes mul-tiple blurred photos and the authors argue that deconvolutionin general outperforms the align-and-average approach.

2. SMARTPHONE APPLICATION

The tested device is equipped with all the apparatus neededfor our demo system, namely a relatively high-quality cam-era, inertial sensors, fast CPU (ARM Cortex-A9) and enoughRAM to perform computations. A block diagram of the de-blurring application is in Fig. 2.

We first perform offline calibration to obtain camera in-trinsic parameters, rolling shutter delay and gyroscope drift.

During the photo acquisition, samples of angular velocityare recorded using the embedded gyroscopes, which are af-terwards trimmed to match the exposure period. Integratingthe position track from the recorded gyroscope data allows usto render a correct blur at every pixel of the image. State-of-the-art non-blind deconvolution methods use sparse image

priors and the solution is usually found by some iterative min-imization algorithms, such as Alternating Direction Methodof Multipliers (ADMM). To perform full image deblurringwith SV blur would be computationally very expensive andnot feasible on a mobile device. Instead, we split the imageinto overlapping patches and generate one blur for each patch.We use a division to 6 × 8 squares with 25% overlap in ev-ery directions. Each patch is then reconstructed individuallyusing the Wiener filter for the corresponding blur:

U = GH∗

|H|2 + Φ, (1)

where Φ is the inverse signal to noise ratio, and G, H and Uare discrete Fourier transforms of the observed image patch,blur and the estimated image patch, respectively. To avoidringing artifacts around patch borders, edge tapering is ap-plied prior to filtering. Due to patch overlaps, we blend thereconstructed patches by weighting them with Hamming win-dows, which results in virtually seamless images.

The intensity values of the reconstructed image some-times lie outside the working bit-depth range (0-255), there-fore we added optional normalization with clipping of out-liers. The normalization is especially useful in the case oflarger blurs and scene with high luminance.

For the Fourier transformation, we use the FFTW libraryported to ARM CPUs, supporting two cores and a SIMD in-struction set (NEON). FFTW proved to be remarkably fast onthe tested smartphone.

The acquired images with native camera resolution of3264 × 2448 are by default scaled down to 2048 × 1536to take advantage of better performance of FFTW when theimage size is a factor of small primes.

The Wiener filtering consists of several FFTs: one for theblur and two (forward and backward for inverse) for eachcolor channel. That yields a total of 7 FFT operations foreach patch. The deconvolution of the image enlarged by theoverlaps takes about 7s; the whole process starting from thecamera shutter is done in a little over 10s. This includes imageresizing, blur estimation, compressing and saving the originaland deblurred image files.

We have identified several issues that hamper our solution.Correct synchronization of camera shutter with the gyroscopesamples is critical. Even a few millisecond error can produceannoying artifacts. We managed to find a good synchroniza-tion mechanism for our test device, which will be unfortu-nately hard to port to other models, because Android providesno general aid for precise camera handling. Gyroscope drift issubstantial and without any compensation results in a biasedblur estimator. Correct calibration is still an open question.Internal image post-processing done by the phone presentsanother serious problem for deconvolution. Since the origi-nal raw data from the image sensor are not available, we areforced to work with JPEG (compressed) images, which areprocessed by gamma correction and most likely also by un-

Fig. 2. The block diagram of the smartphone application:During camera exposure, the application records data fromthe built-in gyroscopes. The data are processed and blursare estimated. The captured photo is divided into overlap-ping patches, Wiener deconvolution is performed on everypatch and the reconstructed patches are blended to generatethe sharp photo. The whole process, entirely done on thesmartphone, takes around 10s.

Fig. 3. Examples of captured and reconstructed images us-ing our demo system. Best viewed on a computer screen andzoomed in.

documented image enhancement steps. We have employedthe inversion of gamma correction, which indeed improvesthe results to some degree.

Three examples of the application output are in Fig. 3.More examples and a demo video showing how the applica-tion runs are available on http://zoi.utia.cas.cz/mobile.

3. CONCLUSIONS

This work presents an image deblurring method that can ef-fectively remove blur caused by camera motion using infor-mation from gyroscopes. The deconvolution method incor-porates spatially varying blur, which allows us to handle bothcomplex camera motion and rolling shutter. The proposedmethod runs on a smartphone device.

There are several topics for future research. Implement-ing smarter deblurring algorithms that avoid ringing artifactsis viable. Gyroscope data are not precise and one could usethe calculated blurs as an initial estimate and apply modi-fied blind deconvolution methods to improve their accuracy.Camera-gyroscope synchronization errors could be solved byformulating a minimization problem over a single parameter– the synchronization shift.

4. REFERENCES

[1] O. Whyte, J. Sivic, A. Zisserman, and J. Ponce, “Non-uniform deblurring for shaken images,” InternationalJournal of Computer Vision, vol. 98, no. 2, pp. 168–186,2012.

[2] Ondrej Sindelar and Filip Sroubek, “Image deblurring insmartphone devices using built-in inertial measurementsensors,” Journal of Electronic Imaging, vol. 22, no. 1,pp. 011003–011003, 2013.

[3] P Forssen and Erik Ringaby, “Rectifying rolling shuttervideo from hand-held devices,” in Computer Vision andPattern Recognition (CVPR), 2010 IEEE Conference on.IEEE, 2010, pp. 507–514.

[4] Neel Joshi, Sing Bing Kang, C. Lawrence Zitnick, andRichard Szeliski, “Image deblurring using inertial mea-surement sensors,” ACM Trans. Graph., vol. 29, pp.30:1–30:9, July 2010.

[5] E. Ringaby and P.-E. Forssen, “A virtual tripod for hand-held video stacking on smartphones,” in ComputationalPhotography (ICCP), 2014 IEEE International Confer-ence on, 2014, pp. 1–9.

[6] S.H. Park and M. Levoy, “Gyro-based multi-image de-convolution for removing handshake blur,” in Proceed-ings of IEEE CVPR, 2014, pp. 3366–3373.