Computational Photography on the Cordova FrameworkCourse Project - CS 534, Fall 2015https://www.victorylap.co/imp-camera

Pat Murray & John Meurer

Abstract. The Cordova framework allows developers to create cross-platform applications using web technologies. Although this methods allows for easy creation of efficient and maintainable applications it has disadvantages that realized when the application accesses the hardware on the device, such as the camera. This article describes the process of utilizing the GPU of an iOS device in order to create an efficient camera to be used by the Cordova framework and the possible processes that can be run on these images via objective-c after capture.

1 Introduction & MotivationIonic is a SDK for building cross-platform mobile applications on top of an AngularJS backbone. Developers can program a mobile application using core web technologies (HTML, CSS, JS), which allows the app to run on both iOS and Android, within the device webviews. In order to run cross-platform applications, Ionic utilizes the Cordova framework. Essentially, Cordova wraps Ionic’s www/ code with the respective operating systems native code. Due to the limitations of web technologies, native operating system code is still needed in order to access a device's hardware, such as the camera. Cordova is able to wrap native code into plugins, used by Ionic developers in order to connect their mobile applications to device hardware, specifically the device camera.

There is a growing demand for developers to be able to utilize a device's camera while using frameworks such as Ionic. Ionic has a rapidly growing user base and with that there is a high demand amongst Ionic developers for a simple and efficient camera that operates via the Cordova framework. This growing demand has provided us with a clear motivation to supply not only fast operating camera to be easily used by Ionic developers but a camera that has post-processing capabilities that can easily be added and accessed from the plugin.

As far as which computational capabilities we strive to implement, we have chosen to focus on simulating double exposure to create a hybrid image and performing selective gaussian blurring. Double exposure is a technique in traditional film photography in which the film is exposed to light on two separate occasions. This creates a “ghosting” effect on the final image

in which both images are visible. We are motivated to implement this feature for both its artistic results and the fact that it can only truly be done with film cameras and not digital cameras. Focus blur is a concept that is based off of a mobile phones tap to focus feature. We think it would be interesting if everything but the subject being focused and its immediate surroundings were computationally blurred from the image. We are motivated to implement this feature because of the interaction that it creates between the user and the camera.

2 BackgroundDouble exposure was commonly used as a technique with disposable cameras. By hitting a disposable camera while taking a picture one is able to expose the same section of film multiple times. The first image captured acts as a primary image whereas the images taken afterwards appear to be overlaid with an alpha value. This is primarily called “ghosting”. Below is an example of a double exposure image captured with a disposable camera.

(2a) double exposure

Tapping to focus (2b) was a simple concept that has experienced a big rise given the popularity of mobile devices. The concept is to simply tap the touch screen in order to tell the camera what are of the frame it should auto focus on. Being that the user would typically want to focus on the primary object it is an interesting feature to apply a basic gaussian blur to the areas that the user is not interested in. This would create an image that emphasises the primary object even more than the original photograph.

(2b) tap to focus

3 Problem StatementThe problem is reflective of the motivation; Ionic developers need an camera that easy to utilize over the Cordova framework and is fast, simple, and provides a way to crop and perform computations on output images all within one application, that is, the application these developers are creating. The first objective is to create a camera that abides by the standards of Cordova plugins in order for other developers to seamlessly plug our camera into their application. The second objective is to use Objective-C in combination with the GPUImage library created by Brad Larson in order to create a camera that can quickly take captures and quickly render the output images ready for both cropping and post-processing tasks. Objective-C’s Core Image and GPUImage will allow for the creation of a camera with all contemporary smartphone camera tasks such as a front-facing camera toggle, touch to focus functionality, and pinch to zoom functionality. The third objective is to take the captured image crop off the top and bottom of the image in order to make a square (our application will force the device to stay in a portrait orientation). The fourth objective is to perform a single computation on the output image from a selection of possible computations.

The computations discussed in this article involve simulating double exposure and generating a blurred image based on the inverse selection of the user while utilizing tap to focus functionality. Objective-C is not optimized for bitmap computation in the same way as MATLAB

is and therefore will pose complications in performing complex bitmap operations. The double exposure image will be generated by computing a hybrid image out of two input images. The focus blur image will computed by performing a gaussian blur computation on all the pixels that are not selected. Also, because the plugin will be developed in Objective-C it will only be able to be used on iOS devices.

4 Related WorkThere is really only one alternative camera to be used by the Cordova framework, which is the Cordova Camera (4a) created by ngCordova. This camera poses many issues. One visibly notable issue is that cropping images relies on third party applications that can take a long time to open, render the image, and transition back to the original application. The Cordova Camera also does not utilize the GPU and is therefore a very slow performing camera. The average opinion of the Cordova Camera amongst the Ionic community is that it lacks the basic functionality and efficiency to be utilized by any popular mobile application (as seen on Ionic forums). Through all of these negative attitudes towards the camera it still remains the main camera for Ionic developers to use in their applications.

In addition to its limitations as a basic functioning camera plugin it also offers no post-processing abilities to any of the output images. This is a valuable capability for the camera to have as it will allow developers to perform computations simply by querying the plugin rather than designing their own implementation to handle the process.

(4a) cordova camera

5 TheoryHybrid images are typically created by first filtering one image with a low pass filter and then filtering another image with a high pass filter. The high pass image is then overlayed on top of the low pass image by adding each pixels frequency value together. Below is an example in which Marilyn Monroe is the low frequency image and Albert Einstein is the high frequency image overlaid on top.

(5a) hybrid image

It should be possible to create a double exposure images by taking two input images, respectively adjusting the alpha values of each pixel, and then creating a hybrid image similarly to the image above. The main difference is that we must retain both the low and high frequencies of both images.

After implementing a tap to focus functionality it is easy to reference to location of the screen that has been tapped. Using this location we will be able to set a radius of X pixels from this center point as an area not change the value of the pixel. A gaussian blur can then be applied to the image for every pixel except the ones in the circle. After applying a gaussian blur our resulting image will be left with an output similar to below.

(5b) focus blur concept

6 Method & ImplementationThe implementation of the camera's interface and basic functionality (including capturing an image) is all developed in Objective-C. In order to perform computations on the captured photos, we modified and implemented a library called GPUImage. GPUImage hides the complexity of OpenGL, a graphics library that utilizes the Graphics Processing Unit for Embedded Systems API, in a simplified Objective-C interface. This interface allows a developer to easily define an input image, perform bitmap modification, and the computations on the input image are incredibly fast because GPUImage utilizes the GPU rather than the CPU.

To begin our implementation of the camera plugin we first downloaded the GPUImage library from the Github page❑[4 ]. In order to utilize more of the code we modified the GPUImage repository directly by adding a new file rather than directly working with the API GPUImage provided. The formation of the static library necessary for Objective-C usage was actually incredibly difficult. It took some very specified terminal commands that were difficult to understand and even harder to find on our resources. From there we began implementing double exposure and focus blur.

Double exposure was implemented by first taking in two input images. The “ghosting” effect of the resulting image was then created by setting the alpha value of each pixel on the second image to be lower than the alpha of each pixel on the first image. We then proceeded to create a hybrid image by blending the alpha values and color values at each respective pixel❑[ 3]. The resulting image successfully simulated double exposure on a disposable camera.

Focus blur was implemented by referencing the center point (which was calculated earlier) and then performing a gaussian blur on every pixel that was not within the selection area. To create the blur effect we sampled color values of pixels within a ten pixel radius of the central pixel.

These pixels are then given weights as expressed by the gaussian distribution function (a) and creating for each resulting pixel to appear blurred❑[1]. Our resulting concept had no edge detection and therefore was an attempt to have the appearance of image.

(a) gaussian distribution

Including the front end styling and backend processing we developed around 2,000 lines of code which is all visible in the VLCamera.m file. In order to compute our resulting images we used many helper functions from the GPUImage library.

We then tested the camera by plugging it into an already existing application that was using the Cordova Camera. This was a simple process because all we had to do was replace the Cordova Camera files with our new files and keep the references the same. The process worked flawlessly and we were able to fine tune our application by adjusting parameters and experimenting with the results.

7 Experimental ResultsThe results for the Cordova/Ionic plugin are a success. From a user’s perspective the camera offers a toggle to switch between the devices available cameras (front facing and back facing), an action button for capturing the photograph, and a shaded header and footer in order to specify the square image that the user will be capturing. This camera is able to capture any image without postprocessing and be easily utilized in any Ionic application as a plugin. In order to activate any of the postprocessing (double exposure or focus blur) we have two toggles on either side of the the action button.

(7a) double exposure (7b) focus blur

In terms of the postprocessing features, they also produced images that were generally a success. We were not entirely sure what the double exposure was supposed to look like but we tried to simulate the result that double exposure creates with a disposable camera while using techniques to create hybrid images. Overall it creates a result where both the first and second image are visible even though they convolved together. Image (7a) above is the result for taking an image of a restaurant and then a selfie with two individuals. Looking at the image one can see that both individual photographs are still recognizable, making the resulting hybrid image a successful simulation of double exposure.

The implementation of focus blur outputted an image that utilized the gaussian blur in order to blur the inverse of an area the user selected. In image (7b) the user tapped to focus on the capital and then after capturing the image it outputted an image where every pixel outside of the selected area had been blurred using a gaussian blurring implementation. Looking closely at the image it is visible that the capital and the area around it are the only parts of the landscape that are not blurred.

8 ConclusionOverall, the resulting Ionic plugin was a success. The double exposure post-processing outputted a convincing hybrid image and the focus blur outputted a dynamically generated gaussian blur via user selected areas. Although the original goal of creating an efficient and easily usable camera to be made available to Ionic developers has been accomplished, there are slight improvements that can be made to the post-processing implementations. First off,

the original idea of focus blur was to blur everything around a primary object that the user focused on by tapping on the screen. The current implementation of focus blur is able to simply blur outside of a radius of the selected section of the screen. In order to accomplish blurring everything around the primary object it is necessary to perform an edge detection algorithm and then only apply the gaussian blur around these detected edges. Secondly, using Objective-C in terms of computational photography and bitmap processing is comparably more difficult than MATLAB. This has limited the scope of possible post-processing and forced the applications core features into directions that were more attainable rather than envisioned. Finally, it would be valuable to chain multiple computations on top of each other.

There is a good amount of improvement that can be made to the camera in the future but this implementation is a very good start and will hopefully satisfy Ionic developers looking for both a working camera alternative to Cordova camera and interesting post-processed image computation to be plugged into their application. The development of this application will continue to both improve upon and create new features in the realm of computational photography.

9 References[1] Larson, B. (2013). Optimizing Gaussian Blurs on a Mobile GPU. Retrieved 6 December, 2015, from

http://www.sunsetlakesoftware.com/2013/10/21/optimizing-gaussian-blurs-mobile-gpu

[2] Wu, J. (2014). Image Processing in iOS Part 1: Raw Bitmap Modification. Retrieved 8 December, 2015, from http://www.raywenderlich.com/69855/image-processing-in-ios-part-1-raw-bitmap-modification

[3] Wu, J. (2014). Image Processing in iOS Part 2: Core Graphics, Core Image, and GPUImage. Retrieved 8 December, 2015, from http://www.raywenderlich.com/71151/image-processing-ios-part-2-core-graphics-core-image-gpuimage

[4] https://github.com/BradLarson/GPUImage

Team Member ContributionsWe split the work relatively evenly.

We pulled the GPUImage framework from Brad Larson’s Github, and had to build it into a static binary library for use in an Objective-C project (this proved a ridiculously complicated task and we wouldn’t recommend attempting it).

Pat manipulated the main Cordova files, VLCamera.m, lines 120-212, for use with our modified View Controller. John wrote the core of the VLCameraViewController.m, around 450 lines, and Pat added gesture recognizers for touch motions and button presses, around 150 lines.