91.204.201 computing iv

91.204.201 Computing IV

Chapter Two: Core Module. The Core Functionality

Xinwen Fu

CS@UML

References Application Development in Visual Studio Reading assignment: Chapter 2

An online OpenCV Quick Guide with nice examples

By Dr. Xinwen Fu 2

CS@UML

A few things Blackboard submission Report format Sreenshots

By Dr. Xinwen Fu 3

CS@UML

Outline 2.1 Mat - The Basic Image Container 2.2 How to scan images, lookup tables and time

measurement with OpenCV 2.3 Mask operations on matrices 2.4 Adding (blending) two images using OpenCV 2.5 Changing the contrast and brightness of an image 2.6 Basic drawing 2.7 Random generator and text with OpenCV 2.8 Discrete Fourier Transform 2.9 File input and output using XML and YAML 2.10 Interoperability with OpenCV 1

By Dr. Xinwen Fu 4

CS@UML

2.1 Mat - The Basic Image Container We have multiple ways to acquire digital images from the

real world: digital cameras, scanners, computed tomography or magnetic resonance imaging to just name a few. In every case what we (humans) see are images.

When transforming this to our digital devices what we record are numerical values for each of the points of the image.

By Dr. Xinwen Fu 5

CS@UML

Storing Images A black-white image is nothing more than a

matrix containing all the intensity values of the pixel points.

How we get and store the pixels values may vary according to what fits best our need In the end all images inside a computer world may be

reduced to numerical matrices and some other information describing the matric itself.

OpenCV is a computer vision library whose main focus is to process and manipulate these information to find out further ones. The first thing to learn and get accommodated with is

how OpenCV stores and handles images.By Dr. Xinwen Fu 6

CS@UML

Mat Basically a class with two data parts:

the matrix header (containing information such as the size of the matrix, the method used for storing, at which address is the matrix stored and so on)

a pointer to the matrix containing the pixel values (may take any dimensionality depending on the method chosen for storing)

OpenCV is an image processing library, doing image processing with its functions

By Dr. Xinwen Fu 7

CS@UML

Copy Mat OpenCV uses a reference counting system.

The idea is that each Mat object has its own header.

However the matrix may be shared between two instance of them by having their matrix pointer point to the same address.

Copy operators will only copy the headers, and as also copy the pointer to the large matrix too, however not the matrix itself.

By Dr. Xinwen Fu 8

CS@UML

1. Mat A, C; // creates just the header parts

2. // here we'll know the method used (allocate matrix)3. A=imread(file, CV_LOAD_IMAGE_COLOR);

4. Mat B(A); // use the copy constructor5. C=A; // assignment operator

All the above objects, in the end point to the same single data matrix.

Their headers are different, however making any modification using either one of them will affect all the other ones too.

In practice the different objects just provide different access method to the same underlying data. Nevertheless, their header parts are different.

By Dr. Xinwen Fu 9

CS@UML

Refer only to a subsection of the full data The real interesting part comes that you

can create headers that refer only to a subsection of the full data.

For example, to create a region of interest (ROI) in an image you just create a new header with the new boundaries:

By Dr. Xinwen Fu 10

Coordinates of the top-left cornerRectangle width and height

CS@UML

Cleaning Mat You may ask if the matrix itself may belong to

multiple Mat objects who will take responsibility for its cleaning when it’s no longer needed. The short answer is: the last object that used it.

For this a reference counting mechanism is used Whenever somebody copies a header of a Mat object a

counter is increased for the matrix. Whenever a header is cleaned this counter is decreased. When the counter reaches zero the matrix too is freed.

By Dr. Xinwen Fu 11

CS@UML

Copy the matrix itself Because, sometimes you will still want to

copy the matrix itself too, there exists the clone() or the copyTo() function.

Now modifying F or G will not affect the matrix pointed by the Mat header

By Dr. Xinwen Fu 12

CS@UML

Tips of using Mat Output image allocation for OpenCV functions

is automatic (unless specified otherwise).

No need to think about memory freeing with OpenCVs C++ interface.

The assignment operator and the copy constructor (ctor)copies only the header.

Use the clone() or the copyTo() function to copy the underlying matrix of an image.

By Dr. Xinwen Fu 13

CS@UML

Storing methods for pixel values. You can select color space and data type used

The color space refers to how we combine color components in order to code a given color.

The simplest one is the gray scale. For colorful ways we have a lot more of methods

to choose from. However, every one of them breaks it down to three or four basic components and the combination of this will give all others. The most popular one is RGB, mainly because this is also

how our eye builds up colors in our eyes. Its base colors are red, green and blue.

To code the transparency of a color sometimes a fourth element: alpha (A) is added.

By Dr. Xinwen Fu 14

CS@UML

Color Systems RGB is the most common as our eyes use

something similar, used by our display systems. The HSV and HLS decompose colors into their

hue, saturation and value/luminance components, which is a more natural way for us to describe colors. You may dismiss last component, making your algorithm

less sensible to light conditions of the input image. YCrCb is used by the popular JPEG image format. CIE L*a*b* is a perceptually uniform color space,

which comes handy if you need to measure the distance of a given color to another color.

By Dr. Xinwen Fu 15

CS@UML

Data Types for Color Each of building components has their own valid

domains. This leads to the data type used. The smallest data type possible is char, which

means one byte or 8 bits This may be unsigned (so can store values from 0 to

255) or signed (values from -127 to +127). In case of three components this gives 16 million

possible colors to represent (like in case of RGB) Even finer control by using float (4 byte = 32 bit) or

double (8 byte = 64 bit) data types for each component. However, increasing the size of a component also

increases the size of the whole picture in the memory.

By Dr. Xinwen Fu 16

CS@UML

Creating explicitly a Mat object Although Mat is a great class as image

container it is also a general matrix class. Therefore, it is possible to create and

manipulate multidimensional matrices.

You can create a Mat object in multiple ways

For two dimensional and multichannel images we first define their size: row and column count wise.

By Dr. Xinwen Fu 17

CS@UML

We need to specify the data type to use for storing the elements and the number of channels per matrix point. To do this we have multiple definitions made according to the following convention:

CV_[The number of bits per item][Signed or Unsigned][Type Prefix]C[The channel number]

CV_8UC3 means we use unsigned char types that are 8 bit long and each pixel has three items of this to form the three channels. This are predefined for up to four channel numbers.

The Scalar is four element short vector. Only 2 dimension matrix can use cout

By Dr. Xinwen Fu 18

CS@UMLBy Dr. Xinwen Fu 19

CS@UML

Data Type A primitive OpenCV data type is one of unsigned char, bool,

signed char, unsigned short, signed short, int, float, double, or a tuple of values of one of these types, where all the values in the tuple have the same type.

Any primitive type from the list can be defined by an identifier in the form CV_<bit-depth>{U|S|F}C(<number_of_channels>), for example:

uchar ~ CV_8UC1, 3-element floating-point tuple ~ CV_32FC3

A universal OpenCV structure that is able to store a single instance of such a primitive data type is Vec. Multiple instances of such a type can be stored in a std::vector, Mat, Mat_, SparseMat, SparseMat_, or any other container that is able to store Vec instances.

By Dr. Xinwen Fu 20

CS@UMLBy Dr. Xinwen Fu 21

CS@UML

Print for other common items OpenCV offers support for print of other

common OpenCV data structures too via the << operator like

2D Point

3D Point

std::vector via cv::Mat

By Dr. Xinwen Fu 22

CS@UML

Print (Cont’d) std::vector of points

By Dr. Xinwen Fu 23

CS@UML



By Dr. Xinwen Fu 24

CS@UML

Goal How to go through each and every pixel of

an image? How is OpenCV matrix values stored? How to measure the performance of our

algorithm? What are lookup tables and why use

them?

By Dr. Xinwen Fu 25

CS@UML

Color space reduction Divide the color space current value with a

new input value to end up with fewer colors

For instance every value between zero and nine takes the new value zero, every value between ten and nineteen the value ten and so on.

By Dr. Xinwen Fu 26

CS@UML

How the image matrix is stored in the memory? - gray scale image The size of the matrix depends of the color

system used. More accurately, it depends from the number of channels used.

By Dr. Xinwen Fu 27

CS@UML

How the image matrix is stored in the memory? - RGB color system For multichannel images the columns

contain as many sub columns as the number of channels

Note that the order of the channels is inverse: BGR instead of RGB

By Dr. Xinwen Fu 28

CS@UML

Color reduction formula When you divide an uchar (unsigned char - aka

values between zero and 255) value with an int value the result will be also char. These values may only be char values. Therefore, any fraction will be rounded down.

Taking advantage of this fact the upper operation in the uchar domain may be expressed as:

By Dr. Xinwen Fu 29

CS@UML

Measure time code runs Another issue is how do we measure time? OpenCV offers two simple functions to achieve this

getTickCount() and getTickFrequency(). The first returns the number of ticks of your systems CPU

from a certain event (like since you booted your system). The second returns how many times your CPU emits a tick

during a second. So to measure in seconds the number of time elapsed between two operations is easy as:

By Dr. Xinwen Fu 30

CS@UML

Lookup table for color reduction how_to_scan_images imageName.jpg

intValueToReduce [G] The final argument is optional. If given the image will be

loaded in gray scale format, otherwise the RGB color way is used. The first thing is to calculate the lookup table.

By Dr. Xinwen Fu 31

CS@UML

Th

e E

ffic

ien

t W

ay

By Dr. Xinwen Fu 32

CS@UML

Itera

tor

By Dr. Xinwen Fu 33

CS@UML

On-the-fly address calc

In case of color images we have three uchar items per column.

This may be considered a short vector of uchar items, that has been baptized in OpenCV with the Vec3b name.

By Dr. Xinwen Fu 34

CS@UML

On-the-fly address calc

By Dr. Xinwen Fu 35

CS@UML

The Core Function

By Dr. Xinwen Fu 36

CS@UML

Performance Difference For the best result compile the program and run it on your

own speed. For showing off better the differences I’ve used a quite large (2560 X 1600) image.

The performance presented here are for color images. For a more accurate value I’ve averaged the value I got from the call of the function for hundred times.

By Dr. Xinwen Fu 37

CS@UML

Mat_ If you need multiple lookups using this method for an image

it may be troublesome and time consuming to enter the type and the at keyword for each of the accesses. To solve this problem OpenCV has a Mat_ data type.

It’s the same as Mat with the extra need that at definition you need to specify the data type through what to look at the data matrix, however in return you can use the operator() for fast access of items.

To make things even better this is easily convertible from and to the usual Mat data type. A sample usage of this you can see in case of the color images of the upper function. Nevertheless, it’s important to note that the same operation (with the same runtime speed) could have been done with the at() function. It’s just a less to write for the lazy programmer trick.

By Dr. Xinwen Fu 38

CS@UML



By Dr. Xinwen Fu 39

CS@UML

Mask operation Recalculate each pixels value in an image according

to a mask matrix (also known as kernel). This mask holds values that will adjust how much influence

neighboring pixels (and the current pixel) have on the new pixel value.

From a mathematical point of view we make a weighted average, with our specified values.

By Dr. Xinwen Fu 40

CS@UML

Basic Method1. void Sharpen(const Mat& myImage,Mat& Result)2. {3. CV_Assert(myImage.depth() == CV_8U); // accept only uchar images

4. const int nChannels = myImage.channels();5. Result.create(myImage.size(),myImage.type());

6. for(int j = 1 ; j < myImage.rows-1; ++j)7. {8. const uchar* previous = myImage.ptr<uchar>(j - 1);9. const uchar* current = myImage.ptr<uchar>(j );10. const uchar* next = myImage.ptr<uchar>(j + 1);

11. uchar* output = Result.ptr<uchar>(j);

By Dr. Xinwen Fu 41

CS@UML

12. for(int i= nChannels;i < nChannels*(myImage.cols-1); ++i)

13. {

14. *output++ = saturate_cast<uchar>(5*current[i]

15. -current[i-nChannels] - current[i+nChannels] - previous[i] - next[i]);

16. }

17. }

18. Result.row(0).setTo(Scalar(0));

19. Result.row(Result.rows-1).setTo(Scalar(0));

20. Result.col(0).setTo(Scalar(0));

21. Result.col(Result.cols-1).setTo(Scalar(0));

22. }

By Dr. Xinwen Fu 42

CS@UML

filter2D function Applying such filters is so common in image

processing that in OpenCV there exist a function that will take care of applying the mask (also called a kernel in some places). 1. define a Mat object that holds the mask:

2. Then call the filter2D function specifying the input, the output image and the kernell to use:

By Dr. Xinwen Fu 43

CS@UML



By Dr. Xinwen Fu 44

CS@UML

Theory From our previous tutorial, we know

already a bit of Pixel operators. An interesting dyadic (two-input) operator

is the linear blend operator:

By varying from 0 to 1 this operator can be used to perform a temporal cross-dissolve between two images or videos, as seen in slide shows and film productions

By Dr. Xinwen Fu 45

CS@UML

Example1. #include <opencv/cv.h>2. #include <opencv/highgui.h>3. #include <iostream>4. using namespace cv;5. int main( int argc, char** argv )6. {7. double alpha = 0.5; double beta; double input;8. Mat src1, src2, dst;9. 10. /// Ask the user enter alpha11. std::cout<<" Simple Linear Blender "<<std::endl;12. std::cout<<"-----------------------"<<std::endl;13. std::cout<<"* Enter alpha [0-1]: ";14. std::cin>>input;

By Dr. Xinwen Fu 46

CS@UML

Example15. /// We use the alpha provided by the user iff it is between 0 and 116. if( alpha >= 0 && alpha <= 1 ) { alpha = input; }

17. /// Read image ( same size, same type )18. src1 = imread("../../images/LinuxLogo.jpg");19. src2 = imread("../../images/WindowsLogo.jpg");20. if( !src1.data ) { printf("Error loading src1 \n"); return -1; }21. if( !src2.data ) { printf("Error loading src2 \n"); return -1; }

22. /// Create Windows23. namedWindow("Linear Blend", 1);24. beta = ( 1.0 - alpha );25. addWeighted( src1, alpha, src2, beta, 0.0, dst);

26. imshow( "Linear Blend", dst );

27. waitKey(0);28. return 0;29. }

By Dr. Xinwen Fu 47

CS@UML



By Dr. Xinwen Fu 48

CS@UML

Image Processing A general image processing operator is a

function that takes one or more input images and produces an output image.

Image transforms can be seen as: Point operators (pixel transforms) Neighborhood (area-based) operators

By Dr. Xinwen Fu 49

CS@UML

Pixel Transforms In this kind of image processing transform,

each output pixel’s value depends on only the corresponding input pixel value (plus, potentially, some globally collected information or parameters).

Examples of such operators include brightness and contrast adjustments as well as color correction and transformations.

By Dr. Xinwen Fu 50

CS@UML

Brightness and contrast adjustments Two commonly used point processes are

multiplication and addition with a constant: g(x) = α f(x) + β

The parameters α > 0 and β are often called the gain and bias parameters Sometimes these parameters are said to control

contrast and brightness respectively. You can think of f(x) as the source image pixels

and g(x) as the output image pixels. Then, more conveniently we can write the expression as: g(i; j) = α f(i; j) + β

where i and j indicates that the pixel is located in the i-th row and j-th column.

By Dr. Xinwen Fu 51

CS@UML

Example1. #include <opencv/cv.h>2. #include <opencv/highgui.h>3. #include <iostream>

4. using namespace cv;

5. double alpha; /**< Simple contrast control */6. int beta; /**< Simple brightness control */

7. int main( int argc, char** argv )8. {9. /// Read image given by user10. Mat image = imread( argv[1] );11. Mat new_image = Mat::zeros( image.size(), image.type() );

12. /// Initialize values13. std::cout<<" Basic Linear Transforms "<<std::endl;14. std::cout<<"-------------------------"<<std::endl;15. std::cout<<"* Enter the alpha value [1.0-3.0]: ";std::cin>>alpha;16. std::cout<<"* Enter the beta value [0-100]: "; std::cin>>beta;By Dr. Xinwen Fu 52

CS@UML

17. /// Do the operation new_image(i,j) = alpha*image(i,j) + beta18. for( int y = 0; y < image.rows; y++ )19. { 20. for( int x = 0; x < image.cols; x++ )21. { 22. for( int c = 0; c < 3; c++ )23. {24. new_image.at<Vec3b>(y,x)[c] =25. saturate_cast<uchar>( alpha*( image.at<Vec3b>(y,x)[c] ) + beta );26. }27. }28. }

29. /// Create Windows30. namedWindow("Original Image", 1);31. namedWindow("New Image", 1);

32. /// Show stuff33. imshow("Original Image", image);34. imshow("New Image", new_image);

35. /// Wait until user press some key36. waitKey();37. return 0;38. }

By Dr. Xinwen Fu 53

CS@UML

Who is Lena? Lena

Söderberg, a Swedish model

cropped from the centerfold of November 1972 issue of Playboy magazine

By Dr. Xinwen Fu 54

CS@UML

Core function Instead of using the for loops to access each

pixel, we could have simply used this command:

image.convertTo(new_image, -1, alpha, beta);

where convertTo would effectively perform new_image = a*image + beta.

However, we wanted to show you how to access each pixel. In any case, both methods give the same result.

By Dr. Xinwen Fu 55

CS@UML



By Dr. Xinwen Fu 56

CS@UML

Example in Visual Studio 2010 Use Point to define 2D points in an image.

Use Scalar and why it is useful

Draw a line by using the OpenCV function line

Draw an ellipse by using the OpenCV function ellipse

Draw a rectangle by using the OpenCV function rectangle

Draw a circle by using the OpenCV function circle

Draw a filled polygon by using the OpenCV function fillPoly

By Dr. Xinwen Fu 57

CS@UML



By Dr. Xinwen Fu 58

CS@UML

Example in Visual Studio 2010 Use the Random Number generator class

(RNG) and how to get a random number from a uniform distribution. RNG rng( 0xFFFFFFFF ); rng.uniform(a,b); // This generates a

randomly uniformed distribution between the values a and b (inclusive in a, exclusive in b).

Display text on an OpenCV window by using the function putText

By Dr. Xinwen Fu 59

CS@UML



By Dr. Xinwen Fu 60

CS@UML

Skipped What is a Fourier transform and why use

it?

How to do it in OpenCV?

Usage of functions such as: copyMakeBorder(), merge(), dft(), getOptimalDFTSize(), log() and normalize() .

By Dr. Xinwen Fu 61

CS@UML



By Dr. Xinwen Fu 62

CS@UML

Skipped How to print and read text entries to a file

and OpenCV using YAML or XML files?

How to do the same for OpenCV data structures?

How to do this for your data structures?

Usage of OpenCV data structures such as FileStorage, FileNode or FileNodeIterator.

By Dr. Xinwen Fu 63

CS@UML



By Dr. Xinwen Fu 64

CS@UML

Skipped What changed with the version 2 of

OpenCV in the way you use the library compared to its first version

How to add some Gaussian noise to an image

What are lookup tables and why use them?

By Dr. Xinwen Fu 65

CS@UML

References OpenCV documentation

By Dr. Xinwen Fu 66

91.204.201 computing iv

Documents