LYU0503

Document Image Reconstructionon Mobile

Using Onboard Camera

Supervisor: Professor Michael R.Lyu

Group Members: Leung Man Kin, Stephen Ng Ying Kit, Tommy

2

Outline

• Motivation• Objective• Symbian Operating System• Previous Work• Implementation• Experimental results• Difficulties• Conclusion

3

Motivation

• Most people posses mobile phones

• Quality of onboard camera improves

• Processing power increases

• Mobile phones have many functions– Take photos, Music Jukebox– How to add more values?

4

Objective

• Reconstruct documents on mobile phones• Input: Different parts of document

• Output: A high resolution document

5

Symbian Operation System

• A mobile OS from Symbian Ltd.• Design for 2G, 2.5G and 3G mobile phones• 32-bit preemptive multi-tasking OS• Features:

– Multi-tasking kernel– Mobile telephony support– Communication– Data management– Etc.

6

Symbian Operating System

• Symbian phone models– Nokia 6600– Nokia 6630– Nokia N90

• Symbian SDK provided to develop program• It enables programmers to gain more control on

phone compared with java environment

7

Symbian Operating System

• Some testing results using Nokia N90– It can allow use allocate at most 1M memory

manually (e.g. malloc in C).– It can allow us to open 8-10 2Mega pixel gray scale

image (using class CFbsBitmap).

• Why different?– The architecture of Symbian OS is client-server.– malloc in C is for client side.– The class is in the server side.

8

Previous Work

• Study basic idea of different image alignment and stitching techniques.

• Implemented & tested on PC– Use SIFT to find matching points– Stitch the image using the matching points

• Tested some related APIs on Symbian OS– Processing image– Fax API– Etc.

9

Implementation

• The implementation is divided into 5 parts– Take photos of the document– Lens correction of the photos– Mark matching points of the photos– Stitch the photos– Optimize the photo

10

Take photos of the document

1. Take the whole document– Align the distance between the camera and

document to take photo of the whole document.

11

Take photos of the document

2. Take the individual part of the document– The camera is taken closer to the document

to take the required image

1

3

2

4

12

Graphical User Interface

• To instruct the user to take photo of the individual part of the document– Overlaying is used.

13

Graphical User Interface

• Overlaying– Output.R = (Picture1.R + Picture2.R) / 2– Output.G = (Picture1.G + Picture2.G) / 2– Output.B = (Picture1.B + Picture2.B) / 2– R = Red– G = Green– B = Blue

• The process is simple and quick– Complexity is O(Width * Height)– Involved one addition and one shift for each color

component for each pixel.– Can be used in Symbian phone real-time

14

Take photos of the document

• After taking photo of the whole document, divide the whole document into several part like

• Use Overlay to display both the individual part and the current frame from the camera

1 2

3 4

15

Take photos of the document

• Before the overlaying, the size of both image should be the same– Cannot instruct the user to take photo correctly.– Resizing is used.– It would take time if interpolation is needed.

• Careful use of image size– The image resolution 1600 x 1200 (whole document)– The resolution from the camera 800 x 600

• If bigger (the same as the image), the camera would be lagging– The wanted resolution is about 400 x 300.– To reduce 1600 x 1200 to 400 x 300 and 800 x 600 to 400 x 300

• Just sub-sample the whole document.• No interpolation is needed

16

Take photos of the document

• User is instructed to take photos correctly.

17

Take photos of the document

• Pseudo code

Take phone of whole documentDivide the whole document into fourFor each sub document

While (there is next frame from camera)Sub-sampled sub-document Sub-sampled of the current frameOverlay the above two image and displayif (User take photo of sub-document)

Store the photobreak

18

Lens correction

• Photos suffer from lens distortion– Barrel type– Pincushion type

19

Lens correction

• Image transformation– Transform the images using the equations

20

Lens correction

21

Mark matching points manually

• In order to reconstruct the photos well– Need to find match points between photo– Transform the individual photos according to

the whole document

• The finding and matching feature points using SIFT need much time

22

Mark matching points manually

• On PC (3.0Ghz), it need about:– finding feature points:

• 4~5min each image– matching feature points:

• 4~5 min each pair of image– TOTAL = 45 min

• The processing power of Symbian phone is much less than the PC.– Nokia N90: RISC 220Mhz processer

• Mark the points manually would be faster and more reasonable.

23

Mark matching points manually

• Ask the user to mark the matching points

24

Mark matching points manually

• Like overlaying, resize is needed.– Only sub-sampling is needed.

• We use (x, y) coordinates to store for each points.

• In previous example, we would obtain 16 pairs of (x, y) coordinates.– Each individual image contributes 4 pairs– There are 4 individual images.

• The coordinates are stored and used to transform the individual photos in the later stage.

25

Mark matching points manually

• Pseudo code

For each sub-documentDisplay left half of the sub-document and

corresponding part of whole documentMarking two pairs of pointsDisplay right half of the sub-document and

corresponding part of whole documentMarking two pairs of pointsStore the coordinates into the file.

26

Mark matching points manually

MarkedMarker

27

Stitch the photos

1. Find transform

Parameter Set 1

2. Transform correspondingcoordinates using set 1

28

Stitch the photos

3. Find transform parameter set 2by sub-image2 and transformed coordinates

4. Using set 2 to transformwhole sub-image2

29

Stitch the photos

5. Similarly for all other sub-images

6. Stick them together

and high resolution image formed!

30

Image Blending

• Bilinear Interpolation– A weighted mean of nearby pixel values

31

Image Blending

• Averaging Mask– Mean of pixel values of 8 neighbors

32

Image Blending

• Threshold– Quantize the pixel values by a threshold value

33

Experimental Results

34

Experimental Results

35

Experimental Results

36

Experimental Results

37

Experimental Results

38

Difficulties

• Limitation of processing speed– Image processing involves much computation– Processing in phone would take much time than PC.

• Limitation of memory– PC have much more memory and is allowed to use

virtual memory.– Image processing, like image stitching, involves much

memory

39

Conclusion

• We have studied different stitching algorithms

• We have implemented the stitching algorithm on both PC & Symbian phone

• Defects are still exist in the resulting image

40

~ THE END ~