android opencv first application

7/23/2019 Android OpenCV First Application

1/11

Android OpenCVInterface

Pi19404

March 30, 2013


2/11

Contents

Contents

A n d r o i d O p e n C V I n t e r f a c e 3

0.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

0.2 Packages used . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

0.3 Processing Raw Camera Data . . . . . . . . . . . . . . . . . . .3

0.4 Implementation Details . . . . . . . . . . . . . . . . . . . . . . .6

0.5 Compilation Details . . . . . . . . . . . . . . . . . . . . . . . . .7

0.5.1 Native Code . . . . . . . . . . . . . . . . . . . . . . . . . . 70.5.2 Java Code . . . . . . . . . . . . . . . . . . . . . . . . . . .

7

0.5.3 Project Setup . . . . . . . . . . . . . . . . . . . . . . . . .8

0.5.4 Compilation Setup . . . . . . . . . . . . . . . . . . . . . .9

0.5.5 Compilation . . . . . . . . . . . . . . . . . . . . . . . . . . .1 0

0.6 Launch Application . . . . . . . . . . . . . . . . . . . . . . . . . .1 1

0.7 Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 1

2 | 1 1


3/11

Android OpenCV Interface


0.1 IntroductionIn this article will look at simple application to demonstrate AndroidOpencv Interface on android platform using OpenCV and javaCVlibraries on Ubuntu Linux Platform.

0.2 Packages used1. OpenCV

2. JavaCV

3. Android

0.3 Processing Raw Camera Data

The android application provides the raw camera image in YUV formatas byte image in the registered preview Callback function.

The first step is to convert this to a format suitable for in-terface with Java or Open CV. Thus the aim is to obtain a bitmapimage of type ARGB_8888.

Different methods can be used to perform this operation.Onemethod is to use this data to create YuvImage object which can be

converted to bitmap object.The data represents the byte array which is input to the preview.

The functions framework() and getFrameHeight() return the widthand height of the image as set in camera parameters.

1

Y u v I m a g e y u v _ i m a g e = n e w Y u v I m a g e ( d a t a , I m a g e F o r m a t . N V 2 1 ,

g e t F r a m e W i d t h ( ) , g e t F r a m e H e i g h t ( ) , n u l l ) ;

2

R e c t r e c t = n e w R e c t ( 0 , 0 , g e t F r a m e W i d t h ( ) , g e t F r a m e H e i g h t ( ) ) ;

3 B y t e A r r a y O u t p u t S t r e a m o u t p u t _ s t r e a m = n e w B y t e A r r a y O u t p u t S t r e a m ( ) ;

4 y u v _ i m a g e . c o m p r e s s T o J p e g ( r e c t , 1 0 0 , o u t p u t _ s t r e a m ) ;

3 | 1 1


4/11


5 m B i t m a p = B i t m a p F a c t o r y . d e c o d e B y t e A r r a y ( o u t p u t _ s t r e a m . t o B y t e A r r a y ( ) ,

0 , o u t p u t _ s t r e a m . s i z e ( ) ) ;

The next method is to decode the YUV byte array and constructthe BitMap image. The function to decode YUV byte array is givenbelow.

1

p u b l i c v o i d d e c o d e Y U V 4 2 0 2 R G B A ( i n t w i d t h , i n t h e i g h t , b y t e [ ] d a t a ,

B i t m a p b i t m a p )

2 {

3 i n t [ ] r g b a = n e w i n t [ d a t a . l e n g t h ] ; ;

4 i n t f r a m e S i z e = w i d t h * h e i g h t ;

5

f o r ( i n t i = 0 ; i < h e i g h t ; i + + )

6 f o r ( i n t j = 0 ; j < w i d t h ; j + + ) {

7

i n t i n d e x = i * w i d t h + j ;

8 i n t s u p p l y _ i n d e x = f r a m e S i z e + ( i > > 1 ) * w i d t h + ( j & ~ 1 ) ;

9 i n t y = ( 0 x f f & ( ( i n t ) d a t a [ i n d e x ] ) ) ;

1 0

i n t u = ( 0 x f f & ( ( i n t ) d a t a [ s u p p l y _ i n d e x + 0 ] ) ) ;

1 1 i n t v = ( 0 x f f & ( ( i n t ) d a t a [ s u p p l y _ i n d e x + 1 ] ) ) ;

1 2

y = y < 1 6 ? 1 6 : y ;

1 3 f l o a t y _ c o n v = 1 . 1 6 4 f * ( y - 1 6 ) ;

1 4

i n t r = M a t h . r o u n d ( y _ c o n v + 1 . 5 9 6 f * ( v - 1 2 8 ) ) ;

1 5 i n t g = M a t h . r o u n d ( y _ c o n v - 0 . 8 1 3 f * ( v - 1 2 8 ) - 0 . 3 9 1 f * ( u -

1 2 8 ) ) ;

1 6

i n t b = M a t h . r o u n d ( y _ c o n v + 2 . 0 1 8 f * ( u - 1 2 8 ) ) ;

1 7 r = r < 0 ? 0 : ( r > 2 5 5 ? 2 5 5 : r ) ;

1 8

g = g < 0 ? 0 : ( g > 2 5 5 ? 2 5 5 : g ) ;

1 9 b = b < 0 ? 0 : ( b > 2 5 5 ? 2 5 5 : b ) ;

2 0

r g b a [ i * w i d t h + j ] = 0 x f f 0 0 0 0 0 0 + ( b < < 1 6 ) + ( g < < 8 ) + r ;

2 1 }

2 2 m B i t m a p . s e t P i x e l s ( r g b a , 0 , w i d t h , 0 , 0 , w i d t h , h e i g h t ) ;

2 3

r g b a = n u l l ;

2 4 }

The second method is slower compared to the first method . Thethird method is the fast method i have found for decoding thecamera data. is to use JNI interface.The decoding method is writ-

ten in C and the call is made from Java to C method using Javanative interface.

In the below code yuv is represents the camera input data which isa byte array and bgra is the output data which is integer array.

The bitmap can be created from the output data.The ARGB bitmapcan now be converted to IplImage object provided by Java which isequivalent to Open CV IplImage or Mat data structure.

4 | 1 1


5/11


This object can also be passed to the native opencv calls andtype-casted as Mat data type process_image functions performsthe desired computation and passes as output the result of the

computation which is returned to the android application.The _yuv variable represents the camera byte array. The _bgrarepresents the integer array after decoding and processing to bepassed back to the camera application.

Open CV cvtColor has bultin decoding routine which will be used todecode the byte array to a BGRA Mat image.

After BGRA to GRAY processing is performed on the image and

then results is sent back to the android application.1 c v t C o l o r ( m y u v , m b g r a , C V _ Y U V 4 2 0 s p 2 B G R , 4 ) ;

2 p r o c e s s _ i m a g e ( m b g r a ) ;

3

4 / / c r e a t i n g b i t m a p f r o m o u t p u t d a t a

5

f B i t m a p . s e t P i x e l s ( r g b a , 0 , w , 0 , 0 , w , h ) ;

6 / / d e c o d i n g t h e b i t m a p d a t a t o I p l I m a g e

7 I p l I m a g e o r g I m g = I p l I m a g e . c r e a t e ( c v S i z e ( f B i t m a p . g e t W i d t h ( ) , f B i t m a p

. g e t H e i g h t ( ) ) , 8 , 4 ) ;

8 f B i t m a p . c o p y P i x e l s T o B u f f e r ( o r g I m g . g e t B y t e B u f f e r ( ) ) ;

The task is to compile this native code into a shared library(so)that can be used by the android application.

The easiest method is to place all the code in a single headerfile.But is a bad designing practice Hence the file can be pre-pared manually from different C/C++ files as part of pre processingstep.The advantage is that there is not need to explicitly writethe native interface files as it is auto generated.

The header file used for the present application will be calledopencv_androidh

Now we have a interface between camera data and OpenCV datatype and further image processing task can be performed.

A simple color conversion example is used to demonstrate theapplication

The entire processing is performed in the native code and re-

5 | 1 1


6/11


sult is passed to the android application after the BGR2HSV colorconversion has been completed.

0.4 Implementation Details

The AndroidActivityMain is the main file responsible for launching theapplication. In the present android framework the camera frame canbe accessed as long as we display is allocated for displaying thecamera preview frame.

Hence the processed image needs to be written over the camerapreview frame. The Class DrawOnTop is responsible of generating aview which is drawn on top of the camera preview .

The Preview class is responsible for setting the camera param-eter and preview callback routines which provides access to thecamera data and opening and closing the camera etc. The Previewclass contains an instance of DrawOnTop class.

One the camera data is available the native routine is called to

process the camera frame and result is store in an array whichis used by DrawOnTop class to convert the integer array into abitmap and display on top of the camera preview.

6 | 1 1


7/11


0.5 Compilation Details

0.5.1 Native Code

The first step of compilation using the opencv,javacv method is towrite the header files.

1 n a m e s p a c e O p e n C V

2 {

3

c l a s s O p e n C V P r o c e s s {

4

5

O p e n C V P r o c e s s ( )

6 {

7

}

8

9 v o i d r u n ( i n t w i d t h , i n t h e i g h t , u n s i g n e d c h a r * _ y u v , i n t * _ r g b a )

1 0

{

1 1 / / a l l o c a t i n g t h e a r r a y u s e d b y o p e n c v

1 2

M a t m y u v ( h e i g h t + h e i g h t / 2 , w i d t h , C V _ 8 U C 1 , ( u n s i g n e d c h a r * )

_ y u v ) ;

1 3 M a t m b g r a ( h e i g h t , w i d t h , C V _ 8 U C 4 , ( u n s i g n e d c h a r * ) _ b g r a ) ;

1 4

1 5 / / d e c o d i n g t h e c a m e r a b y t e a r r a y

1 6

c v t C o l o r ( m y u v , m b g r a , C V _ Y U V 4 2 0 s p 2 B G R , 4 ) ;

1 7

1 8

/ / p e r f r o m i n g B G R t o G R A Y c o n v e r s i o n a n d w r i t i n g t h e r e s u l t b a c k

t o t h e

1 9 / / o r i g i n a l a r r a y

2 0 M a t i m a g e ;

2 1 c v t C o l o r ( m b g r a , i m a g e , C V _ B G R A 2 G R A Y ) ;

2 2

c v t C o l o r ( i m a g e , m b g r a , C V _ G R A Y 2 B G R A ) ;

2 3 }

2 4

2 5 } ;

2 6

2 7

}

0.5.2 Java CodeThe next step is to write the Java Files to native interface withthe header files

1 @ N a m e s p a c e ( " O p e n C V " )

2

p u b l i c c l a s s P r o c e s s I m a g e {

3 / / s p e c i f y i n g t h e p a t h w h e r e t h e h e a d e r a n d l i b r a r y i s t o b e

f o u n d

4 p u b l i c s t a t i c f i n a l S t r i n g a n d r o i d I n c l u d e p a t h = " . . / i n c l u d e

/ " ;

5

p u b l i c s t a t i c f i n a l S t r i n g a n d r o i d L i n k p a t h = " . . / l i b s /

a r m e a b i " ;

7 | 1 1


8/11


6 p u b l i c s t a t i c f i n a l S t r i n g g e n e r i c I n c l u d e p a t h = " / o p t / l o c a l

/ i n c l u d e / " ;

7 p u b l i c s t a t i c f i n a l S t r i n g g e n e r i c L i n k p a t h = " / u s r / l o c a l

/ l i b / "

8 p u b l i c s t a t i c f i n a l i n t

9 C V _ M A J O R _ V E R S I O N = 2 ,

1 0

C V _ M I N O R _ V E R S I O N = 4 ,

1 1 C V _ S U B M I N O R _ V E R S I O N = 2 ;

1 2

1 3 p u b l i c s t a t i c f i n a l S t r i n g C V _ V E R S I O N = C V _ M A J O R _ V E R S I O N + " . "

+ C V _ M I N O R _ V E R S I O N + " . " + C V _ S U B M I N O R _ V E R S I O N ;

1 4

1 5

1 6

/ / l o a d i n g t h e o p e n c v l i b r a r i e s a t i n s t a n t i a t i o n

1 7 s t a t i c {

1 8

i f ( l o a d ( ) ! = n u l l ) {

1 9 S t r i n g p l a t f o r m N a m e = g e t P l a t f o r m N a m e ( ) ;

2 0 }

2 1

}

2 2

2 3

/ / d e c l a r i n g t h e i n t e r f a c e o f n a t i v e O p e n C V c l a s s

2 4 p u b l i c s t a t i c c l a s s O p e n C V P r o c e s s e x t e n d s P o i n t e r {

2 5

s t a t i c { L o a d e r . l o a d ( ) ; }

2 6 p u b l i c O p e n C V P r o c e s s ( ) { a l l o c a t e ( ) ; }

2 7 p r i v a t e n a t i v e v o i d a l l o c a t e ( ) ;

2 8

p u b l i c n a t i v e v o i d r u n ( i n t w i d t h , i n t h e i g h t , b y t e y u v [ ] , i n t

[ ] r g b a ) ;

2 9

}

3 0

3 1

}

0.5.3 Project Setup

The next step is to copy the libraries and set up project depen-dencies.

The the javacv and javacpp files found in the javacv package to

the libs directory of the android project and also add the jar filesto the project.

Copy all the opencv libraries to libs/armeabi directory of the androidproject

The javacv package also provides javacv-android-arm.jar file.Unzipcontents of the files which are .so library files and copy them tothe libs/armeabi directory of the android project.

8 | 1 1


9/11


0.5.4 Compilation SetupThe next step is code compilation.First step is to compile the nativecode. Go to the libs/armeabi directory in the file system.Open the

javacpp.jar file and navigate to the path/ c o m / g o o g l e c o d e / j a v a c p p /

p r o p e r t i e s and open the file android-arm.properties.

make changes in the files as per the local android ndk pathand set the appropriate flags as per the arm architecture of thedesired target devise. below is the sample of the file that is used

in the present project1

p l a t f o r m . n a m e = a n d r o i d - a r m

2 p l a t f o r m . r o o t = / o p t / a n d r o i d - n d k - r 7 /

3

p a t h . s e p a r a t o r = :

4 c o m p i l e r . p a t h = t o o l c h a i n s / a r m - l i n u x - a n d r o i d e a b i - 4 . 4 . 3 / p r e b u i l t /

l i n u x - x 8 6 / b i n / a r m - l i n u x - a n d r o i d e a b i - g + +

5

c o m p i l e r . s y s r o o t . p r e f i x = - - s y s r o o t =

6 c o m p i l e r . s y s r o o t = p l a t f o r m s / a n d r o i d - 1 4 / a r c h - a r m /

7

c o m p i l e r . i n c l u d e p a t h . p r e f i x = - I

8 c o m p i l e r . i n c l u d e p a t h = s o u r c e s / c x x - s t l / g n u - l i b s t d c + + / i n c l u d e / :

s o u r c e s / c x x - s t l / g n u - l i b s t d c + + / l i b s / a r m e a b i - v 7 a / i n c l u d e / : : / u s r /

l o c a l / i n c l u d e : / u s r / l o c a l / i n c l u d e / o p e n c v : / u s r / l o c a l / i n c l u d e /

o p e n c v 2

9

c o m p i l e r . o p t i o n s = - m a r c h = a r m v 7 - a - m f l o a t - a b i = s o f t f p - m f p u = v f p -

f f a s t - m a t h - W l , - - f i x - c o r t e x - a 8

1 0

c o m p i l e r . o u t p u t . p r e f i x = - W l , - r p a t h , l i b / - D A N D R O I D - f f u n c t i o n -

s e c t i o n s - f u n w i n d - t a b l e s - f s t a c k - p r o t e c t o r - f u n s w i t c h - l o o p s -

f i n l i n e - l i m i t = 3 0 0 - W a l l - O 3 - n o d e f a u l t l i b s - f P I C - s h a r e d - W l , - -

n o - a l l o w - s h l i b - u n d e f i n e d - s - o \ u 0 0 2 0

1 1 c o m p i l e r . o p t i o n s . f a s t f p u = - m a r c h = a r m v 7 - a - m f l o a t - a b i = s o f t f p - m f p u =

v f p - f f a s t - m a t h - W l , - - f i x - c o r t e x - a 8

1 2 c o m p i l e r . o u t p u t . p r e f i x = - W l , - r p a t h , l i b / - D A N D R O I D - f f u n c t i o n -

s e c t i o n s - f u n w i n d - t a b l e s - f s t a c k - p r o t e c t o r - f u n s w i t c h - l o o p s -

f i n l i n e - l i m i t = 3 0 0 - W a l l - O 3 - n o d e f a u l t l i b s - f P I C - s h a r e d - W l , - -

n o - a l l o w - s h l i b - u n d e f i n e d - s - o \ u 0 0 2 0

1 3

c o m p i l e r . l i n k p a t h . p r e f i x = - L

1 4 c o m p i l e r . l i n k p a t h = s o u r c e s / c x x - s t l / g n u - l i b s t d c + + / l i b s / a r m e a b i - v 7 a / :

l i b s / a r m e a b i 2

1 5 c o m p i l e r . l i n k . p r e f i x = - l

1 6

c o m p i l e r . l i n k . s u f f i x =

1 7 c o m p i l e r . l i n k = l o g : g n u s t l _ s t a t i c : g c c : d l : z : m : c

1 8 c o m p i l e r . f r a m e w o r k . p r e f i x = - F

1 9

c o m p i l e r . f r a m e w o r k . s u f f i x =

2 0 c o m p i l e r . f r a m e w o r k =

2 1

s o u r c e . s u f f i x = . c p p

2 2 l i b r a r y . p r e f i x = l i b

9 | 1 1
http://localhost/var/www/apps/conversion/tmp/scratch_2/com/googlecode/javacpp/propertieshttp://localhost/var/www/apps/conversion/tmp/scratch_2/com/googlecode/javacpp/properties


10/11


2 3 l i b r a r y . s u f f i x = . s o

0.5.5 CompilationPlace the commands required for compilation in the script file andexecute from the workspace/main directory of the android project.

First step is to copy the native header file OpenCVProcessIm-age.hpp to the ndk package include directory.The base directoryis as per compiler.sysroot path configured in the above file(

/ o p t /

a n d r o i d - n d k - r 7 / p l a t f o r m s / a n d r o i d - 1 4 / a r c h - a r m / ).

Next set the java home and other variable required by Java.ThisJava SDK should be same as one used by the Java IDE used tocompilation of android application.

Next step is to compile the Java file which provides the native in-terface passing the javacv and Java jar files as class path arguments.

The final step is to run the Java command passing the jar filesargument as javacpp.jar ,properties file name as android-arm and

platform.root argument as the local android ndk path and compilednative Java class files as input and output is the shared library.

1

c p / h o m e / p i 1 9 4 0 4 / O p e n C V P r o c e s s I m a g e . h p p / o p t / a n d r o i d - n d k - r 7 /

p l a t f o r m s / a n d r o i d - 1 4 / a r c h - a r m / u s r / i n c l u d e / o p e n c v _ s u p p o r t _ c l a s s .

h

2 e x p o r t h o m e 1 = ` p w d `

3

c d $ h o m e 1

4 e x p o r t J A V A _ H O M E = / o p t / s o f t w a r e s - l i n u x / j d k 1 . 7 . 0 _ 0 4 /

5 J A R = $ h o m e 1 / l i b s / j a v a c p p . j a r

6

J A R 1 = $ h o m e 1 / l i b s / j a v a c v . j a r

7 S R C = $ h o m e 1 / s r c / c o m / a n d r o i d / o p e n c v / P r o c e s s I m a g e . j a v a

8

L I B S = $ h o m e 1 / l i b s / a r m e a b i

9 $ J A V A _ H O M E / b i n / j a v a c - c p $ J A R - c l a s s p a t h b u i l d / c l a s s e s : $ J A R 1 : $ J A R

$ S R C

1 0 $ J A V A _ H O M E / b i n / j a v a - D p l a t f o r m . r o o t = / o p t / a n d r o i d - n d k - r 7 - j a r $ J A R

- p r o p e r t i e s a n d r o i d - a r m - c l a s s p a t h $ h o m e 1 / s r c : $ J A R 1 : $ J A R c o m .

a n d r o i d . o p e n c v . P r o c e s s I m a g e - d l i b s / a r m e a b i

This will generate the libjniProcessImage.so file in the libs/armeabidirectory.

1 0 | 1 1
http://localhost/var/www/apps/conversion/tmp/scratch_2/opt/android-ndk-r7/platforms/android-14/arch-arm/http://localhost/var/www/apps/conversion/tmp/scratch_2/opt/android-ndk-r7/platforms/android-14/arch-arm/


11/11


Next compile the android java application .

0.6 Launch ApplicationTransfer the apk file generated to device and test the application.

(a) Screenshot ImageImage

0.7 CodeThe code can be found in code repository

h t t p s : / / g i t h u b . c o m /

p i 1 9 4 0 4 / m 1 9 4 0 4 / t r e e / m a s t e r / A n d r o i d / A n d r o i d O p e n C V 1 or

h t t p s : / / c o d e .

g o o g l e . c o m / p / m 1 9 4 0 4 / s o u r c e / b r o w s e / A n d r o i d / A n d r o i d O p e n C V 1 . The header

files is located in jni directory.The library files are not placed inthe repository download them from appropriate packages on send amail separately for download link.

1 1 | 1 1
https://code.google.com/p/m19404/source/browse/Android/AndroidOpenCV1https://code.google.com/p/m19404/source/browse/Android/AndroidOpenCV1https://github.com/pi19404/m19404/tree/master/Android/AndroidOpenCV1https://github.com/pi19404/m19404/tree/master/Android/AndroidOpenCV1

android opencv first application

Documents