Mobile image processing – Part 2

Mobile application development on Android“Hello World” projectHello World project“Viewfinder EE368” projectImage processing libraryg g yMobile application examples

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 1

Google GogglesProduct recognition

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 2

http://www.google.com/mobile/goggles

How to download Google Goggles

Open the Android Market applicationSearch for “Google Goggles”Search for Google Goggles

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 3

Recognizing video at a glanceVisual bookmarks for movies and TV shows

(1) User snaps a photo of screen (3) User resumes video on the phone(1) User snaps a photo of screen. (3) User resumes video on the phone.

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 4

(2) Our system identifies video andframe within the video.

Recognizing video at a glanceVisual bookmarks for movies and TV shows

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 5

http://msw3.stanford.edu/~dchen/EE368/Video-Search-Android.wmv

Mobile devices versus personal computers

Feature Motorola DROID Typical PCScreen size 4.6 in x 2.4 in 25 in x 16 in

Processor speed 550 MHz 3.0 GHzRAM 256 MB 4 GBRAM 256 MB 4 GB

Disk space 133 MB internal8 GB external flash 500 GB disk

Internet access Typically 3GWiFi at hotspots WiFi or wired Ethernet

Telephony Core feature Supplementary featurep y pp y

Camera 5 MP camera embedded External webcam

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 6

“Viewfinder EE368” project

Goals of this projectLearn how to access frames from the viewfinderLearn how to modify camera parametersLearn how to augment the viewfinder frames

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 7

Beginning the project

Download the tutorial from class pagehttp://ee368 stanford edu/Androidhttp://ee368.stanford.edu/Android

Create a new Android projectRecommended settingsg

Project Name: ViewfinderEE368Android 2.0.1Application Name: Viewfinder EE368Application Name: Viewfinder EE368Package Name: com.example.viewfinderee368Create Activity: ViewfinderEE368Min SDK Version: 6

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 8

Android manifest file (AndroidManifest.xml)

Landscape mode

…

Landscape mode

Main Activityy…

Set the proper hardware permissions

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 9

Class hierarchy

Viewfinder Manage two views andViewfinder EE368

(Activity)

Manage two views and manage program open/close

Draw onPreview(View)

Draw on Top

(View)

Handle incoming viewfinder frames and adjust camera

Augment viewfinder frames with a new layer of information

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 10

jparameters

yon top

Viewfinder class: full screen mode

Icon and title bars visible Icon and title bars hidden

Viewfinder EE368

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 11

Class hierarchy

Viewfinder Manage two views andViewfinder EE368

(Activity)

Manage two views and manager program open/close

Draw onPreview(View)

Draw on Top

(View)

Handle incoming viewfinder frames and adjust camera

Augment viewfinder frames with a new layer of information

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 12

jparameters

yon top

Timeline of events on the mobile device

N f N fNew frame New frame

CapturePreview CaptureCapture

Augment

Preview

DrawOnTop

Capture

Augment

TimeFrame Interval

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 13

Preview class: frames go down two paths

Preview frames Display on phone screen

Preview(View)

…phone screen

Forward to callback functioncallback function

…Data in YCbCr 4:2:0 format

width w

heig

ht h

w/2 w/2h/

2

h/2

Cb Cr

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 14

h Y

Preview class: set camera parameters

Preview(View)(View)

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 15

Class hierarchy

Viewfinder Manage two views andViewfinder EE368

(Activity)

Manage two views and manage program open/close

Draw onPreview(View)

Draw on Top

(View)

Handle incoming viewfinder frames and adjust camera

Augment viewfinder frames with a new layer of information

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 16

jparameters

yon top

DrawOnTop class: create a paint brush

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 17

DrawOnTop class: YCbCr to RGB

width w

t h

w/2 w/2

2 2

w

heig

ht h/2

h/2

YCb Cr

h

RGB

921.22225658.408

25608.298

−⋅

+⋅

=CrYR

58.13525612.208

25629.100

25608.298

256256

+⋅

−⋅

−⋅

=CrCbYG

84.27625641.516

25608.298

−⋅

+⋅

=CbYB

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 18

DrawOnTop class: draw histogram bars

Bin 0 Bin 1 Bin 2 Bin 3 Bin 4

Rect rect = new Rect( left x, top y, right x, bottom y );Canvas.drawRect( rect );

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 19

Viewfinder EE368 demo

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 20

http://ee368.stanford.edu/Android/ViewfinderEE368

Real-time debugging using DDMS (1)

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 21

Real-time debugging using DDMS (2)

03EH368929E3K

DeviceFile system on device

Device

Messages on the devicethe device

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 22

Danger of unresponsiveness

Users notice delays longer than ~0.5 secAndroid only lets your program “hang” or “freeze” for ~5 secAndroid only lets your program hang or freeze for 5 sec before bringing up warning dialog

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 23

Activity life cycle

[ Professional Android Application Development ]

Bernd Girod, David Chen: EE368 Mobile Image Processing – Part 2 no. 24

[ Professional Android Application Development ]