Slide 1

Food Recognition Using Statistics of Pairwise Local Features Shulin (Lynn) Yang University of Washington Mei Chen Intel Labs Pittsburgh Dean Pomerleau Robotics Institute Rahul Sukthankar Carnegie Mellon 1 Slide 2 Abstract Food items are deformable objects that exhibit significant variations in appearance Food recognition is difficult the key to recognizing food is to exploit the spatial relationships between different ingredients (such as meat and bread in a sandwich). 2 Slide 3 Introduction The goals of such systems are to enable people to better understand the nutritional content of their dietary choices and to provide medical professionals with objective measures of their patients food intake. 3 Slide 4 Pairwise local feature distribution (PFD) 1. Soft labeling of pixels 2. Global Ingredient Representation (GIR) 3. Pairwise Features 4. Histogram representation for pairwise feature distribution 5. Histogram normalization 6. Classification with local feature distributions 4 Slide 5 Semantic Texton Forest (STF) 5 Slide 6 6 Slide 7 7 Slide 8 8 Slide 9 Pairwise local feature distribution (PFD) 1. Soft labeling of pixels 2. Global Ingredient Representation (GIR) 3. Pairwise Features 4. Histogram representation for pairwise feature distribution 5. Histogram normalization 6. Classification with local feature distributions 9 Slide 10 Global Ingredient Representation (GIR) 10 Slide 11 Pairwise local feature distribution (PFD) 1. Soft labeling of pixels 2. Global Ingredient Representation (GIR) 3. Pairwise Features 4. Histogram representation for pairwise feature distribution 5. Histogram normalization 6. Classification with local feature distributions 11 Slide 12 Pairwise Features 12 Slide 13 Between-pair category Between-pair category : B(P1,P2) The feature for each pixel pair has t discrete values, t being the number of pixels exist along the line between a pair of pixels. We use to represent the feature set for pixels P1 and P2. 13 Slide 14 14 Slide 15 15 Slide 16 Joint pairwise features 16 Slide 17 Pairwise local feature distribution (PFD) 1. Soft labeling of pixels 2. Global Ingredient Representation (GIR) 3. Pairwise Features 4. Histogram representation for pairwise feature distribution 5. Histogram normalization 6. Classification with local feature distributions 17 Slide 18 Histogram representation for pairwise feature distribution 18 Slide 19 19 Slide 20 Pairwise local feature distribution (PFD) 1. Soft labeling of pixels 2. Global Ingredient Representation (GIR) 3. Pairwise Features 4. Histogram representation for pairwise feature distribution 5. Histogram normalization 6. Classification with local feature distributions 20 Slide 21 Classification with local feature distributions 21 Slide 22 Experimental Methodology 1. Dataset 2. Baseline approaches 3. Preprocessing with STF 22 Slide 23 Pittsburgh Food Image Dataset(PFID) 23 Slide 24 Experimental Methodology 1. Dataset 2. Baseline approaches 3. Preprocessing with STF 24 Slide 25 Bag of SIFT features 25 Slide 26 SVM(Support Vector Machine) 26 Slide 27 SVM Hyper-plan H1 H2 Support Hyper-plans Classification Hyper-plan Support Hyper-plans Margin 27 Slide 28 Experimental Methodology 1. Dataset 2. Baseline approaches 3. Preprocessing with STF 28 Slide 29 Preprocessing with STF 29 Slide 30 Results 1. Classification accuracy on the 61 categories 30 Slide 31 Confusion matrix Rows: the 61 categories of food Columns: the ground truth categories 31 Slide 32 Such cases are challenging 32 Slide 33 Even for humans, to distinguish. So 61 PFID food categories 7 major groups 33 Slide 34 2. Classification accuracy into 7 major food types 1.sandwiches 2.salads/sides 3.chicken 4.breads/pastries 5.donuts 6.bagels 7.tacos 34 Slide 35 35 Slide 36 Confusion matrix Rows: the major 7 food categories Columns: the ground truth major categories 36 Slide 37 Result (OM) Orientation and midpoint is the higher-order feature that gives the best accuracy. This pair of features is able to leverage the vertically-layered structure of many fast foods. 37 Slide 38 In future work We plan to extend our method to: (1) Perform food detection and spatial localization in addition to whole-image recognition (2) Handle cluttered images containing several foods and non-food items (3) Develop practical food recognition applications (4) Explore how the proposed method generalizes to other recognition domains 38