SmartMScope Final Report

September 2017

Project Title: SmartMScope: A Microscopy Image Analysis App for Mobile Plat-formsFunded by: Interdisciplinary Innovations Fund (IIF) of the MU Information Tech-nology CommitteeProject Report Period: January 2015 to September 2017Principal Investigators: Filiz Bunyak Ersoy, Kannappan Palaniappan, ComputerScience Department; Tommi White, Biochemistry Department; Michele Warmund,Division of Plant Sciences, University of Missouri; Mingzhai Sun, Ohio State Uni-versityProject Student Leaders: Ke Gao and Hang Yu Electrical & Computer Engi-neering Department, University of Missouri

1

Project Summary:

Primary goals and objectives listed in our original MU-IIF proposals were:

• Development of a mobile image-based plant phenotype and disease quantifica-tion tool that can help scientists, foresters, farmers, educators, and students.

• Building of a development platform to design and implement image analysissoftware on mobile platforms that can be used beyond the aforementioned spe-cific application.

• Promotion of collaboration between computer science and life sciences students.Help students learn to produce interdisciplinary innovative solutions throughcollaboration.

• Promotion of STEM disciplines in Missouri secondary education students, byenabling hands on activities with a mobile microscope and the proposed app.

In order to satisfy these goals and objectives, in collaboration with MU Divisionof Plant Sciences and MU Electron Microscopy Core, we have developed two mobileapps (SmartMScope-Leaf and SmartMScope-Seed), an online searchable databaseSmartMScope-DB, and a new web site to disseminate our apps and database. TheseSmartMScope image analysis apps provide automated and interactive methods fornon-destructive and non-invasive plant phenotype quantification. The SmartMScopeimage analysis apps run on mobile handheld devices and enable fast sample collectionand analysis in the field. Demos for our apps can be found on our project web page(www.smartmscope.com) and on YouTube. The three sub-projects developed forMU Interdisciplinary Innovations Fund (IIF) are further described in the followingsections.

The SmartMScope project started a new collaboration between MU ComputerScience department, MU Division of Plant Sciences and MU Electron MicroscopyCore (EM Core). The project introduced graduate and undergraduate students tointerdisciplinary research. The project lead to two apps, one online database, one website, one masters thesis, two student awards, and numerous posters and presentations(listed at the end of this report). The mobile application development experiencegained from the SmartMScope project also helped in other proposal applications (Dr.Bunyak has been awarded a seed grant by MU Coulter Translational PartnershipProgram for another mobile application).

During our extension period (May 2016-May 2017), we have (i) built a desktopversion of our seed image analysis software for batch processing in labs; (ii) developeda user interface for desktop version for sample selection; (iii) presented one posterMU Life Sciences week; (iv) published two papers at Microscopy and Microanalysis

2

Meeting and IEEE International Conference on Image Processing; and (v) startedthree new collaborations on seed image analysis.

SmartMScope-LEAF

Overview:

SmartMscope-LEAF is a mobile plant phenotype quantification app. It has beendeveloped for quantification of morphology of plant leaves, grape leaves in particularaccording to the needs of our collaborators. The morphological information of leavesplays an important role in scientific studies, for example leaf area is intricately linkedto crop productivity. Advancing the current understanding of leaf development willpotentially benefits genetics, physiology, plant breeding and developmental biology.This sub-project aims to develop a mobile, convenient, cheap tool for non-destructive,in the field quantification of plant leaf morphology. The SmartMscope-LEAF app hasthree main modules: (1) Android user interface; (2) image processing for leaf shapequantification; (3) image processing for leaf venation detection and quantification.

Figure 1: Leaf image quantification system flow.

Progress:

All modules, Android user interface, image processing for leaf shape quantification,and image processing for leaf venation detection and quantification are functional.The main functionalities of the developed app can be summarized as follows:

1. Capturing or importing a leaf image; segmentation of the leaf from its back-ground (leaf needs to be positioned on a white background);

2. Extraction of the leaf contours; fitting ellipse and convex-hull to the extractedleaf ;

3

3. Quantification of the leaf shape by computing measurements such as area,perimeter, compactness, eccentricity etc;

4. Analysis of the leaf contour, computation of its curvature, identification of theleaf sinuses and lobes, and computation of the associated measurements.

5. Detection, tracing, and quantification of leaf veins.

The SmartMscope-LEAF app is running on Android smart phones. It is writtenin Java with Android API with the support of openCV library. It has been testedon a test set of 105 grape leaf images, from 24 different cultivars. Interface of theapp and some of the intermediate results can be seen in Figures 2, 3,4.

Figure 2: Leaf image quantification user interface.

SmartMScope-SEED

Overview:

SmartMScope-SEED is a mobile app that has been developed for quantificationof seed/grain morphology. Seeds are deciding factor in germination, survival, anddevelopment of new generations of plants. Size and shape are among the most im-portant traits of seeds. The project aims to develop a mobile, convenient, cheaptool for quantification of seed/grain morphology. The developed app uses imageprocessing techniques to quantify size and shape features of individual seeds/grainsin a picture of group of seeds captured by a smartphone. Unlike some of the ex-isting tools, SmartMscope-SEED can handle touching seeds/grains and does notimpose strict restrictions on spatial layout for the seeds/grains getting imaged. The

4

Figure 3: Leaf shape quantification intermediate results.

Figure 4: Leaf veins.

SmartMscope-SEED app has four main modules: (1) Android user interface; (2)foreground/background segmentation (detection of seed groups); (3) cluster decom-

5

position (identification of individual seeds within a group); (4) quantification of in-dividual seed features.

Figure 5: Seed image quantification system flow.

Progress:

All modules (user inteface, seed detection, cluster decomposition, and seed quantifi-cation) are fully functional. The main functionalities of the developed app can besummarized as follows:

1. Foreground/Background Segmentation: Images are captured using the smart-phone camera with the seeds being placed onto a sheet of single color paper(homogeneous background is required for segmentation). Any color backgroundis supported as long as it produces high contrast against the imaged seeds. Toreduce the influence of shadows and illumination variations on segmentation,RGB color space is converted into RG-chromaticity space. K-means clusteringis used to segment the seeds from the background.

6

2. Cluster decomposition: A custom image analysis method reliably segments indi-vidual seeds from touching group of seeds. This module incorporates innovativemethods for boundary analysis and cluster decomposition.

3. Quantification: Finally, individual seeds are labeled and measurements suchas area, perimeter, major/minor axis are extracted. The measurement unit isconverted from pixel to centimeter based on the size of background sheet thatis used as a scale reference.

The SmartMscope-SEED app is running on Android smart phones. It is written inJava with Android API with the support of openCV library. It has been benchmarkedagainst a commercial plant phenotype quantification product (WinRHIZO) and anopen source tool (ImageJ). The results indicate that the SmartMScope-SEED appperforms better than both of them in terms of speed and accuracy. System flow,inermediate results, and user interface for SmartMScope-SEED can be seen in Figures5,6.

SmartMScope-DB: Web Application and Database

Overview:

We have created an online searchable database and a functional web application tointeractively display plant phenotype quantification results. The developed web ap-plication allows users to select options, pushes the associated requests to the server,which in turn queries the database, information is pulled off of the database andserved to the user. Application functionalities include image searches and dynami-cally produced and displayed charts. The web application is developed using PHP,JavaScript, and HTML, the database is constructed using MySQL. System flow anduser interface are shown in Figures 7,8.

Future Work:

1. Additional functionalities to the web application, such as a user favorites list.

2. App-database integration, secure communication.

3. Moving to cloud environment.

7

Figure 6: Seed image quantification user interface.

8

Figure 7: Web application and database.

9

Figure 8: Sample pages from the web application. The web application is still under-constructionand is not public.

10

Figure 9: Seed image quantification system desktop version.

Project Outcome & Impact

Software:

• SmartMScope-Seed: mobile application for seed morphometry analysis.

• SmartMScope-Leaf: mobile application for quantitative leaf image analysis.

• SmartMScope-DB: web application database for dissemination of results fromSmartMScope-Seed and SmartMScope-Leaf apps.

• www.SmartMScope.com: web site for dissemination of the SmartMScope apps.

• SmartMScope-Desktop: desktop version of SmartMScope-Seed app (Figure 9)and associated user interface for rapid batch processing of the seed imaged(developed during extension period May 2016-May 2017

Posters, Presentations, Demos:

• Plant Phenotype Analysis on Smart Phone, presented by Hang Yu and Ke Gao,31st Missouri Life Sciences Week, April 2015.

• Plant Phenotype Analysis on Smart Phones, presented at Missouri InformaticsSymposium (MIS 2015), April 2015.

• Plant Phenotype Analysis on Smart Phones, presented by Mitchell Battles,Undergraduate Forum, Spring 2015.

11

• Integrated Web and Mobile Application Database for Plant Image Analysis,presented by Mitchell Battles, Undergraduate Forum, Summer 2015.

• Plant Phenotype Analysis on Smart Phone, presented by Ke Gao, 32st MissouriLife Sciences Week, April 2016.

• Plant Phenotype Analysis on Smart Phones, presented by Mitchell Battles,Undergraduate Forum, Spring 2016.

• SmartMScope-Seed: Seed Size and Shape Analysis on Smart phones, presentedby Yuhang Ming, Undergraduate Forum, Spring 2016.

• SmartMScope apps and brochure showcased at MU EM Core exhibit at 32stMissouri Life Sciences Week, April 2016.

• SmartMScope apps and brochure showcased to Fulton High Schools AdvancedPlacement (AP) Biology Class during their visit to MU EM Core .

• SmartMScope apps presented to Monsanto team during their visit to MU Col-lege of Engineering, April 2016.

• Image Processing for High-Throughput Plant and Seed Phenotyping, presentedby Ke Gao, 33rd Missouri Life Science Week, April 2017. (extension period May2016-May 2017)

• Gao, Ke, Michele Warmund, Tommi White, Ruthie Angelovici, and Filiz Bun-yak. ”Mobile Image Analysis for Microscopic Images of Seeds.” Microscopy andMicroanalysis 23, no. S1 (2017): 1326-1327. Presented by Ke Gao, August2017. (extension period May 2016-May 2017)

• Ke Gao, Tommi White, Kannappan Palaniappan, Michele Warmund, Filiz Bun-yak. MUSEED: A Mobile Image Analysis Application for Plant Seed Morphom-etry. IEEE International Conference on Image Processing (ICIP) (2017), BeijingChina, September 2017. (extension period May 2016-May 2017)

12

Other

• Graduate student Ke Gao successfully defended his MSc Thesis entitled ”MobileImage Analysis for Seed Phenotyping” on December 2015. Thesis includedmethods and programs developed for the SmartMScope project.

• Graduate student Hang Yu has been accepted for an internship in NIH NationalLibrary of Medicine based on his accomplishments in SmartMScope project. Heis currently developing mobile medical image processing applications in NIH-NLM.

• Yuhang Ming received 2016 Mizzou Advantage Excellence in Undergraduate Re-search Award (Honorable Mention), for his work on SmartMScope (SmartMScope-Seed: Seed Size and Shape Analysis on Smart phones), May 2016.

• Mitchell Battles received 2016 Mizzou Advantage Excellence in UndergraduateResearch Award (Honorable Mention), for his work on SmartMScope (PlantPhenotype Analysis on Smart Phones), May 2016.We have started new collab-orations on seed image analysis with the following groups:

• Started collaboration with Dr. Edward Large and Dr. Andrew Scaboo, Divisionof Plant Sciences, on soybean seed analysis.

• Started collaboration with Dr. David G. Mendoza-Cozatl, Division of PlantSciences, on Arabidopsis seed analysis.

• Started collaboration with Dr. Ruthie Angelovici, Division of Biological Sci-ences, on Arabidopsis seed analysis.

Fund Expenditure:

We have spent $18,972 from MU-IIF funds. $183 has been used for computing ex-penses, $150 for poster printing,$956 has been used to support the undergraduatestudents Mitchell Battles and Yuhang Ming from Computer Science department.Both of the undergraduate students have been employed through College of Engi-neering Undergraduate Research Program. Part of their salaries to work on thisproject have been supplied by College of Engineering and Computer Science Depart-ment. The remaining funds were used to support graduate research assistants HangYu and Ke Gao (from Electrical & Computer Engineering Department), and (Table1).

13

Awarded Funds $18,972Graduate Research Assistants $17,682Undergraduate Research Assistants $956Equipment $183Poster printing $150Total Expenses $18,972Remaining Funds $0

Table 1: Fund Expenditure

14