Experimental Learning Experiences for Effective Computer Vision Instruction

Lynne Grewe

with support from the following Computer Imaging students: Paul Steffen, Zak Meekins, David Wilhelm, Eddie Padilla, Brian Tao, Candice Weiss,

Edwin Laranang, Walter Manalu, Robin Mack, James Green

Institute of Communication Science and TechnologyCalifornia State University Monterey Bay

Seaside, CA 93955 USAlynne_grewe@monterey.edu

Abstract

With the explosion of multimedia in the last two decades and the inclusion of video and image-based communications over the Internet, there is a demand for education in computer imaging and vision at the Undergraduate level. This paper discusses curriculum developed for such an audience that features experimental-based learning experiences as a vehicle for student education and exploration in the field of computer imaging/vision. As computer imaging/vision has many unsolved problems and challenges, it is crucial that we impart to students not only knowledge but, the investigative skills necessary for tackling these problems. This paper presents the results of student work on a number of such experiences involving both software development and vision hardware system configuration. The paper culminates with the results of a student-composed survey discussing their experience with one of these experimental learning experiences, which illustrates the usefulness of this form of learning.

1. Introduction: Curriculum Structure featuring Experimental Learning Experiences

The course discussed in this paper, titled Computer Imaging, covers topics in both Image Processing and Computer Vision as well as their applications in the fields of engineering, entertainment, manufacturing, multimedia, environmental sensing and exploration, robot vision and more. In this class, students gain the skills necessary to understand, manipulate and create images for a variety of purposes. The student learns how to useCCD cameras and digitizers, as well as use Software tools and C imaging program libraries. A major emphasis is placed on the participation and creativity in both individual, group and class projects.

A unique feature of these projects is that they are experimental in nature. They do not simply ask students to implement algorithms but to solve a problem with no presented solution. This involves experimentation to first define the problem and restrict the domain, research what has been done in the past, to break down the problem into manageable units, then to propose and investigate potential solutions, to implement, test and document the results. These projects offer experimental learning experiences that teach students critical research and problem solving skills as well as project organization and management useful for the working world today.    For a detailed outline the reader is referred to [1]. The course heavily uses web-based instruction and simulation material [1]. The main text [2] covers much of the material and is supplemented by the material in [3-7]. The Imaging Laboratory the students have access to includes a number of cameras, a PC with digitizer, lens, lights, a mobile robot with camera and other sensors. A C library is used from the Matrox Corporation named MIL, Matrox Imaging Library,[8] supports the hardware that implements various lower-level imaging routines.

A difficulty in teaching Computer Vision at the Undergraduate level is the limited mathematics background of the students. To accommodate this situation, the class concentrates on spatial domain processing and only discusses the concept of frequency analysis with regards to applications like compression. In addition, various math modules, such as probability, are introduced as needed.

In the following sections, three sample experimental learning experiences are described. Sample student work is given and discussed. Finally, reflection of the student experience is given via the discussion of a student developed and distributed survey and monitoring process on the last experiment.

2. Green Screen Experiment

This is the first experimental learning experience the students in the class are assigned. The objectives of this experiment are to:

Gain experience with real imaging hardware and creation of software to control it and process images.

Experiment with the setup of lighting and camera for a real-world system. To create a Green-Screen system that would filter out the green-screen backdrop and

insert any image/video in the background. This involves experimentation into what it means to be green.

Understand and explain to a user the constraints and operation of the produced Green-Screen system.

2.1 Results

The students worked collectively on this project and organized themselves into a managerial structure. The first stage of the experiment involved literature review. Besides traditional index services, the students relied heavily on information on the Internet. The following web-site documents one group's work http://classes.monterey.edu/graphics/world/labweb/Green/index.htmlThe students broke down the project into the following stages: Lab Configuration, Experimentation - what is green, Interface Design, Program Development, Documentation Development and Testing. Various approaches to measure "green" of the green screen have been taken including experimental measurement, statistical extrapolation of a small data set and hypothesis and test. Some groups have looked to controlling the environment through precise lighting and positioning whereas others have looked at solutions involving training the system to measure the current characteristics of the green screen. The various approaches are discussed and compared. This gives the students a good understanding of environmental considerations and their first experience controlling imaging hardware and implementing image processing routines. Below are some of the results of these projects.

Some groups look at improving robustness of their system by carefully positioning the sensor, lights, green screen and subject in the scene. Figure 1 shows some of the documentation describing the system configuration. Simple interfaces are developed. The system uses the Matrox Imaging Library (MIL) and Microsoft Visual C++.

Below is a timeline of the project and Figure 2 shows some snapshots of the students hard at work.

Green Screen Progress Timeline

9-22-99 Windows GUI Interface Completed 9-23-99 Green Screen Filming 10-1-99 Completed Green Screen Algorithm 10-11-99 First Html Code Completed 10-11-99 Green Screen Program completed the troubleshooting 10-18-99 Word Documentation and Html code completed 10-25-99 Program installed in the imaging lab. 11-06-99 Program presented.

Behind the Green ScreenLab Set-up, Step 2

Camera set-upThis documentation assumes the camera is already correctly connected to the computer. If this is not the case, ask for assistance in its setup. Take the connected camera and place it on the tripod, being certain that the lever just beneath the camera locks it into place. Turn the camera power adapter (on shelf above computer) to the on position.

This is what you will see when you look This is focused in a bit to cut out the wall through the camera. around the green screen

Figure 1: Step in Laboratory Configuration Documentation for Green Screen System.

Figure 2: Students developing system and testing, final results on right column.

3. Pseudo-Coloring Experiment

In this experiment the students were to work in small groups of two to three to experiment with developing a pseudo-coloring application. Specifically, there has been a recent pilot study out of Australia showing that breast-cancer can be detected in patients using an alternative to mammography that is much less invasive. The system uses a special X-ray scattering machine that takes a picture of a woman's hair. In this pilot study it was observed that certain concentric rings towards the outer-part of the hair indicated the presence of cancer to a high-degree. The application the students were to work on was to develop a system that would pseudo-color the input grayscale images from the X-ray scattering machine and highlight the photometrically continuous rings observed. Such a system could be used to aid the human observer of such images for the detection of cancer.

3.1 Results

Figure 3 shows the results produced by one of the student groups. A difficulty faced by the students was the extremely limited set of images (only two) that the students had to use as samples to base their pseudo-coloring algorithms on. Their goal was to emphasize the concentric rings.

(a) (b)

Figure 3: (a) Original Image, cross-section human hair. (b) Pseudo-colored image.

4. Dog-Tracker Experiment

The objective of this experiment is for the students to design a camera-based system that would capture video live or take it from a stored file, to process it and detect the motion in the scene resulting from a dog. Of the three experiments described here, this is the most complicated and open-ended. The students were told that they were to use the indoor environment of the Imaging Laboratory and its hardware and imaging software base. The only other direction the students were given was that the resulting system should operate in either of the following modes as selected by the user:

System Requirements:1. Use Camera system, machine and software tools in Imaging Lab2. User has option to run system in following modes:

Relative mode: involves comparison of subsequent frames. Background mode: involves comparing each new frame against a static

background image.

3. Organize class into company structure by electing minimally the following personnel: Manger, Lead Development Engineer, and Lead Documentation Engineer.

Manger = oversees entire project, reports to client (Professor), direct team members to self-select tasks or delegates them as necessary, makes weekly reports on status (using BlackBoard) of various deliverables/tasks.

Lead Development Engineer = oversee development of system (system design, code development, testing), reports to Manager status bi-weekly, setup file system.

Lead Document Engineer = oversee development of documentation in HTML and Word that documents the following:

1.How system works, algorithms used. Document accuracy and testing of system. 2.Document the process the class went through in this project. 3.Document the reflections of the class team - design and administer a survey asking what they learned, how this project/experience involved creative inquiry and development of unique ideas, etc.

4. Research ideas, brainstorm: 1.Should you first find the dog or first find the moving pixels and then determine which belongs to the dog?2.Should you have user assistance…let the user indicate a visual area where the dog is in the first frame….or show a picture of the dog…or train the system on the color range and possible size range of the dog (assumptions about how close or far he is)?3.Would a pixel-based approach or one that groups the moving pixels and then reasons about the groupings be better? 4.What are the assumptions/restrictions of the system? 5.How will you indicate where the dog is in each processed frame?

Calculate center of all the dog pixels and label with an X. Calculate center of the grouped dog pixels and label with an X. Color dog pixels or grouped dog pixels. What if the dog is stationary between two frames……what will this

mean? How will you measure the accuracy of the system? Will you capture a bunch of different image sequences of the dog

moving in different environments and test with this? Will you follow the dog at different ranges in the different image

sequences? Will you gather sequences where either at the beginning or middle or

ending frames he does not appear….or he disappears…..or he reappears?

In each processed frame will you have a user classify the dog moving pixels and then compare this with the output from the system? Or will you measure a user-specified center of movement compared to the system-determined center of movement?

3.Produce System in C++ using MIL as appropriate. 4.Test System on real video sequence.

5.1 Results

The students made a number of key decisions in creating the dog-tracker system. The first was that the system would not track just any dog but, that it would on some level be required to "learn" what dog it was being asked to find. This made the problem more solvable for the students. By exploring their options and choosing this requirement they have learned the important skill of defining your problem sufficiently enough to be able to frame a potential solution.

Another question they had to ask and answer was what was the most efficient way to detect a dog or parts of a dog. This was very challenging especially given their introductory skills, the timeline and their skills as programmers. They resolved to use some of their knowledge gained in the "Green Screen Experiment" by distinguishing a dog by its colors as they had distinguished the screen by its green value(s). They realized the limitation of this was that anything moving in the scene with similar color content as the learned/trained dog color(s) may be labeled as dog. To reduce this possibility, they looked into subsequent grouping of pixels. This translated into the understanding that sometimes higher-level features or semantic grouping of primitives are needed for some object detection tasks. Figure 4 shows some sample results of the system.

(a) (b)

(c ) (d) (e) (f)

Figure 4 (a,b) two subsequent images in sequence. (c,d) detected motion in the scene (e,f) dog detected motion.

5. Student FeedbackThis section presents the results of a survey the students of the Computer Imaging class created to reflect on their experience with the Dog-Tracker Experiment. Below are a series of questions followed by representative answers to these questions. The creation, distribution of the survey and assembly of the results presented below were part of the assigned learning experience. Except for small grammatical changes and some abbreviation for brevity sake, the results are unedited. You can see by the results, that the students have learned not only skills in programming but also important research skills and a real appreciation for the difficulty of challenging vision problems.

Research Outcome:1. What was the outcome of the research? Which ideas were incorporated into the actual program? We had many outcomes. We found that through image smoothing image subtraction and thresholding,

we could find and isolate movement in an image. We found that by smoothing an image before any further processing we could reduce the noise of a low resolution image.

To gain more experience with imaging libraries (MIL). To make architectural decisions: Detect only a known dog, or any dog? We used blob detection as well as motion detection. We decided not to worry about partial blocking

because the code was going to be based on motion-detection first, and then perform blob-analysis as opposed to blob analysis then motion-detection.

We learned the important steps involved: Gaussian blur, edge detection, outline objects, and spatial pattern matching.

Program Development and Testing:1. What was the most difficult aspect of developing the motion detection phase of the program? Converting the macrocode generated by inspector to actual C code. Using the computer in the imaging lab.

Identify one shape to target and outline. With motion detection it is kind of easy. Trying to figure out how to program in C. Knowing the necessary libraries and function calls. Determining if we wished to use color or greyscale analysis. Identifying an object by a single frame or by its motion. Debugging.

2. What was the most difficult aspect of developing the object recognition aspect of the program? Getting a program to recognize an object with any degree of integrity. The MIL libraries do have image recognition, but do it rather poorly. To train and add the shape as a specific item. Identify an object by subtracting two images or find it all based in one image. Identify an object by motion characteristics or temporal texture. Doing the actual coding.

3. What did you learn from this project? How to implement the MIL functionality into C programs. How image data is stored and also how to manipulate it. That pattern recognition isn’t easy. Getting more familiar with MIL calls. Reviewing the File I/O in C++ Need to better allocate time for a project this size. Being flexible with other people’s meeting schedules as part of the role of president. Asking people what they have done, and what they can do. How to take images and manipulate them.

4. What was the first idea you incorporated in developing this project? We used subtraction or absolute of difference. To identify motion by difference. Using subtraction in motion detection. Edge detection. The menu (user interface) as well as the file I/O (save and retrieve) algorithm 5. What was the next step you took after implementing the first idea? A way to reduce the noise in low resolution images. We used smoothing. Use the difference and where to output information. Change the implementation to absolute subtraction. Pattern recognition involving reducing noise in subtracted images, then blob, finally matching blob

characteristics to trained dog source images. Writing the code.

6. Did these ideas lead to further developments, and if so please list them. They led us to develop the software using high-resolution images rather than low resolution. Taught us that three displays helped. Yes. The pattern recognition function already in Inspector application satisfies our requirement for

detecting a dog along with our training procedure.

7. What are the problems encountered when testing the software? The whole concept of motion detection was new to all of the developers, so trial and error and time

were the major hold ups. Matrox errors. There are problems when thresholding color images and problems converting color to greyscale. Managing limited hardware buffer resources on video and memory. How to do pattern recognition.

Coding errors.

8. How were these problems resolved? We continued to use MIL libraries to reduce our problems and get a clearer picture of our goals each

time. Experimentation was the main way we solved our problems. Three displays were used. Using system RAM to dump video card/buffer/data into arrays. Used Matrox Inspector to find acceptable tolerance for pattern matching. Debugging.

9. Does the program require user assistance? i.e. let the user indicate a visual area where the dog is in the first frame… or show a picture of the dog… or train the system on the color range and possible size range of the dog. This program provides both these functions. Trains the program. Yes. The user has to show images of the dog. Trains the system on the color range.

10. Did you choose a pixel-based approach or one that groups the moving pixels and then reasons about the groupings? Why did you choose this method over the other one? Pixel based. Because it is what we have been doing all semester. Motion—used pixel based approach. Pattern-Detection—grouped pixels to establish a blob analysis. Pixel based. Haven’t thought about the grouping method. I think we chose pixel-based detecting motion in each pixel one by one, and then grouping as a mask.

This is better because it will also detect small movements.

11. If the dog is stationary between two frames what will this mean? There is no motion detected.9. Use background—Would show if the dog was originally in the picture. Relative—no motion detected. The dog will not be highlighted. It will not return any movement. Only the part of it moving will be detected.

12. How did you measure the accuracy of the system? We watched it work as we shot the film or visually compared the pictures. Used several pieces of film. For dog training we need three pictures of a dog. Front back and side.

User Interface Development:1. Please list relevant information about the development of the user interface. How it works and any problems that occurred during the development. Did not work on the interface. Simple text based questions that call Imaging routines. Was trying to figure out how to use a GUI, but haven’t come up with any yet. Right now it works as a

text-based console program. Lots of algorithms. It shows a menu for the user to choose from. Mode choice leads to submenus where the user is

prompted to retrieve a file or grab it.

6. Conclusion

The skills gained by the students through the series of experimental learning experiences assigned, like the ones discussed above, are invaluable. Students not only gain experience with real-world problems, they work with both hardware and software platforms and must solve many of the related systems problems that crop-up along the way. While this experience is very rewarding, it also makes the class challenging to run as an instructor. Because students come in with differing skill sets, it helps to run these learning experiences in small and large group settings. This allows students to share with each other and builds a stronger sense of community, which goes a long way in furthering the resulting learning experience. These students are not only prepared for research activities if they choose to continue into graduate school but, also have experience solving problems in a collaborative environment preparing them for the work environment.

7. References

[1] Lynne Grewe, Computer Imaging Homepage, CSUMB, http://classes.monterey.edu/CST/CST332-01/world/index.htm

[2] Randy Crane, A simplified approach to Image Processing, Prentice Hall, 1997,

[3] Rafael Gonzalez, and Richard Wood, Digital Image Processing, Addison-Wesley, 1993.

[4] V. Nalwa, A Guided Tour of Computer Vision, Addison-Wesley, 1993.

[5] C. Lindley, Practical Image Processing in C, John Wiley & Sons, 1991.

[6] R. Jain, R. Kasturi, et. al., MachineVision, MCGraw Hill, 1995.

[7] B. Horn, "Robot Vision", MIT Press, 1986

[8] MIL (Matrox Imaging Laboratory) documentation, Matrox Corporation, http://www.matrox.com.

[9] Veronica James, et. al. "Using hair to screen for breast cancer", Nature,VOL 398, March 4, 1999, pp. 33-34.