A Cloud-Based Infrastructure for
Caloric Intake Estimation from
Pre-Meal Videos & Post-Meal Plate Waste Pictures
Vladimir Kulyukin Vikas Reddy Sudini
Department of Computer ScienceUtah State University
Heidi Wengreen Jennifer Day
Department of Nutrition, Dietetics, & Food SciencesUtah State University
Outline
● Background● Human-Centered Caloric Estimation
Infrastructure● Pilot Deployment● Discussion
Motivation● Accurate caloric intake estimation is an open
research problem in health informatics, dietetics, and nutrition management
● Obesity, diabetes, blockage of coronary arteries are related to mismanaged diets
● There is a growing need for robust methods for accurate and periodic caloric intake estimation
Vision-Based Approaches● Many current R&D efforts approach the problem of caloric
intake estimation through complete automation by means of computer vision [1, 2]
● Vision-based approaches estimate caloric intake from static pictures and, increasingly, videos
● Vision-based approaches exhibit high percentages on selected food items [3] but underperform due to low image quality, absence of reliable ground truth baselines, and highly variable food textures
Disengagement of the Nutritionist
● One reason, relatively unexplored in the literature, why completely automated solutions may underperform is that they do not integrate nutritionists into the process
● Nutritionists are typically asked for feedback post-factum when the ground truth is needed to estimate the performance of a given algorithm on a given set of images
● Some nutritionists disengage because they may feel that they have little or no stake in the system
PNUTS: Persuasive NUTtrition Mangement System
● To make nutritional data collection and estimation more manageable and enjoyable both to end users and nutritionists, we have been developing PNUTS
● PNUTS seeks to shift research, training, and clinical practices in nutrition management toward persuasion and community-oriented, context-sensitive nutrition decision support
● PNUTS is inspired by the Fogg Behavior Model [4]: motivation alone is insufficient; triggers and abilities are fundamental
Proposed Infrastructure● An objective of this paper is to propose a nutrition data
collection and analysis infrastructure ● The infrastructure enables clients to submit pre-meal videos
and post-meal plate waste pictures● The infrastructure recognizes the critical role of human
nutritionists and aims to keep them integrated into the process
● The infrastructure is designed to educate new generations of nutritionists
Three Main Systemic Roles
● The system has three main systemic roles:ClientNutritionistProgram Coordinator
● There is also the system administrator but this role is outside the scope of this paper
Client● The client is any user interested in accurate caloric intake
estimations of consumed foods and beverages● The client uses a web-enabled device to register with the
system● At breakfast, lunch, and dinner the client takes a short pre-
meal video and does an optional voice recording of the foods and beverages to be consumed
● At the end of the meal the client takes a still picture of plate waste
Client's Energy Intake Estimation Request (EIER)
● When the client submits a pre-meal video, post-meal static plate waste picture, these data, together with the client's ID and a time stamp, constitute an energy intake estimation request (EIER)
● Each EIER is saved in an SQL database and viewed subsequently by the nutritionist
Nutritionist
● The nutritionist also registers with the system to review and evaluate submitted EIERs
● The nutritionist watches the videos and looks at plate waste pictures
● Caloric intake estimates are based on USDA's National Nutrient Database (ndb.nal.usda.gov/ndb)
Five States of EIERs● An EIER is in one of the five states: unprocessed, pending,
processed, conflicting, and resolved
● Unprocessed – received by the system but not evaluated by any registered nutritionist
● Pending – evaluated by one nutritionist
● Processed – evaluated by two nutritionists without a conflict● Conflicting – evaluated by three nutritionists whose evaluations
disagree by more than 10%● Resolved – conflicting request evaluated by a program coordinator
Program Coordinator
● The Program Coordinator is also a registered nutritionist who has the authority to resolve EIER conflicts
● In principle, the system is designed to have multiple program coordinators, each supervising a number of nutritionists
● When a conflicting EIER is detected by the system, an appropriate program coordinator receives an email
● When the program coordinator logs in, the program coordinator sees a list of conflicting EIERs and resolves the conflict
Implementation Details
● The proposed infrastructure has been implemented in Java using Java Servlets, Java Server Pages (JSPs), and JBOSS
● The current cluster has two nodes: one master and one slave
● All databases are implemented with MySQL
Student Training
● The pilot version of the infrastructure was deployed and integrated into NDFS 4750: Transition to Professional Practice taught at Utah State University in the spring 2015 semester
● All students enrolled in NDFS 4750 received caloric estimation training prior to acting as clients and nutritionists
● Each student watched and evaluated 14 training pre-meal and post-meal video pairs and was required to complete by hand a caloric intake estimation form
● Some students had to undergo additional training
Data Collection
● 28 students, acting as clients, provided EIERs for digital data of 61 eating occasions
● Each client completed a paper-pencil three day food record of all foods eaten and submitted to the system
● 28 students, acting as nutritionists, provided energy intake estimates for these EIERs
● 407 pre-meal videos, 407 post-meal pictures were collected
Preliminary Analysis
● The mean estimate of the 28 nutritionists was 383 kcalories (range: 35 – 936)
● Agreement among the nutritionists was examined with intra-class correlation coefficients (ICCs): ICC = 0.89, p<0.001
● Agreement was lower for digital data from eating occasions that included a single food or five or more food items: ICC=0.78, 0.77, respectively; p-value=0.023 and p-value=0.002, respectively
● These values are at least as high as the ICCs reported with similar studies in cafeteria settings and free living adults [5, 6]
Observations
● The pilot version of the infrastructure functioned flawlessly during the entire semester
● There were two main complaints:
The clients complained that the system did not allow the to submit partial caloric estimations with an op-tion to complete them later
The nutritionists complained about the necessity to use the external database at ndb.nal.usda.gov and retype the information into a separate form
Observations● In an informal qualitative survey conducted with the students they
said that the system made them feel as part of a community of users
● Many students said that they would use the system as registered nutritionists after they graduate
● When asked about the data entry difficulties, the students said that these difficulties are compensated by not having to meet with the clients in person
● The system has turned out to be a valuable training tool for undergraduate nutritionist students
Acknowledgments
● We are grateful to Dr. Sheryl Aguilar for letting us pilot the system in NDFS 4750 she taught at USU in the spring 2015 semester
● We express our gratitude to all students enrolled in NDFS 4750 for testing the system as clients and nutritionists for their effort and feedback
Selected References
[1] Bosch M., Zhu F., Khanna N., Boushey C.J., and Delp E. “Combining global and local features for food identification in dietary assessment.” IEEE transactions on Image Processing . 2011:1789-1792. doi:10.1109/ICIP.2011.6115809.
[2] Martin, C.K., Han, H., Coulon, S.M., Allen, H.R., Champagne, C.M., and Anton, S.D. (2009). “A novel method to remotely measure food intake of free-living people in real-time: The Remote Food Photography Method (RFPM).” British Journal of Nutrition, 101, 446-456. PMCID: PMC2626133.
[3] Chen, M.Y., Yang,Y., Chia-Ju Ho, C., Wang, S., Liu, S., Chang, E., Yeh, C., & Ouhyoung. M. “Automatic Chinese food identification and quantity estimation.” In Proceedings of SIGGRAPH Asia Technical Briefs (SA '12). ACM, New York, NY, USA, , Article 29 , 4 pages, 2012. DOI=10.1145/2407746.2407775.
[4] Fogg, B.J. “A behavior model for persuasive design.” In Proceedings of the 4th International Conference on Persuasive Technology. Arctile 40. ACM, New York, USA, 2009.
[5] Wengreen, H.J., Madden G.J., Aguilar S.S., Smits R.R., Jones B.A. “Incentivizing children’s fruit and vegetable consumption : results of a United States pilot study of the Food Dudes program.” Journal of Nutrition Education Behavior. 2013;45(1):54-9.
[6] Martin C.K., Han H., Coulon S.M., Allen H.R., Champagne C.M., Anton S.D. “A novel method to remotely measure food intake of free-living individuals in real time: the remote food photography method.” British Journal of Nutrition. 2009;101:446-456.