goals& metrics datasets& - computer science · detector/descriptor& brief orb brisk...

1
Datasets Sample datasets from [4], [5], and our work. Descriptor Patterns BRIEF [1] ORB [2] Rotation Invariant BRISK [3] Rotation Invariant Scale Invariant Bibliography 1. Calonder, M., Lepe2t, V., Strecha, C., Fua, P., “BRIEF: Binary Robust Independent Elementary Features,” ECCV, 778792 (2010). 2. Rublee, E., Rabaud, V., Konolige, K., Bradski, G., “ORB: An Efficient Alterna2ve to SIFT or SURF,” ICCV, 25642571 (2011). 3. Leutenegger, S., Chli, M., Siegwart, R., “BRISK: Binary Robust Invariant Scalable Keypoints,” ICCV, 25482555 (2011). 4. Mikolajczyk, K., et al., “A Comparison of Affine Region Detectors,” IJCV 65(12), 4372 (2005). 5. Strecha, C., et al., “On Benchmarking Camera Calibra2on and Mul2View Stereo for High Resolu2on Imagery,” CVPR, 18 (2008). Compute Storage Real Value Parameterization PatchBased Binarized Binary SIFT SURF BRIEF ORB BRISK LDAHash PCADSC SSD NCC MI PCASIFT DAISY Taxonomy Descriptor is built from a set of pairwise intensity comparisons Each bit in the descriptor is the result of exactly one comparison The sampling pattern is fixed (except for possible scaling/rotation) Hamming distance can be used as a similarity measure Provide robustness as well as high computational efficiency Binary Descriptor Properties Results Detector/Descriptor BRIEF ORB BRISK SURF SIFT FAST Harris MSER Detector ms/image n/a 17 43 377 572 2.7 78 117 Descriptor μs/feature 4.4 4.8 12.9 143 314 n/a n/a n/a Storage bytes/feature 16,32,64 32 64 64(256) 128(512) n/a n/a n/a Goals Develop benchmark for binary features that is compatible with existing evaluations Perform quantitative analysis on detector/descriptor pairings Provide a comprehensive set of evaluation datasets Conclusions Avoid using descriptors that are invariant to a transform not present in your dataset Use a detector/descriptor pairing where both are invariant to the same set of transforms Orderofmagnitude speed gains are achievable by binary descriptors, resulting (at worse) in marginal matching performance penalties Metrics # Correct Matches # Putative Matches # Features # Correspondences = # Ground Truth Matches # Correct Matches Precision = / # Putative Matches # Putative Matches Putative Match Ratio = # Features / # Correct Matches Matching Score = # Features / # Correct Matches Recall = # Correspondences / Detector Entropy = Entropy of 2D histogram of feature loca2ons Detector FAST SIFT BRISK SURF ORB Harris MSER Avg # Features 8166 4788 7771 3766 13427 2543 693 Entropy 12.52 12.34 12.33 12.26 12.10 11.84 10.74 Blur Putative Match Ratio BF OB BK SF ST H 25 11 11 MR 19 10 9 FT 23 10 11 OB 13 10 8 BK 20 9 9 SF 31 15 12 15 ST 19 8 9 10 Precision BF OB BK SF ST H 67 78 84 MR 65 71 69 FT 68 72 85 OB 70 78 87 BK 67 77 73 SF 72 82 79 71 ST 63 76 76 73 Matching Score BF OB BK SF ST H 17 8 9 MR 12 7 6 FT 15 7 9 OB 9 8 7 BK 13 7 6 SF 22 12 9 11 ST 12 6 7 7 Recall BF OB BK SF ST H 44 22 24 MR 40 22 19 FT 39 10 24 OB 19 16 12 BK 31 9 14 SF 55 29 22 28 ST 38 18 18 24 Compression Putative Match Ratio BF OB BK SF ST H 68 54 51 MR 43 31 29 FT 50 33 35 OB 35 36 30 BK 50 35 33 SF 59 43 30 47 ST 35 23 28 29 Precision BF OB BK SF ST H 95 97 97 MR 83 88 86 FT 91 96 97 OB 95 97 98 BK 93 97 94 SF 92 96 94 90 ST 85 93 94 91 Matching Score BF OB BK SF ST H 64 53 49 MR 36 27 25 FT 46 32 34 OB 34 35 30 BK 46 34 32 SF 54 41 28 42 ST 30 22 27 26 Recall BF OB BK SF ST H 80 64 63 MR 53 40 33 FT 69 47 54 OB 47 49 35 BK 59 43 41 SF 74 56 39 61 ST 55 40 40 55 Exposure Putative Match Ratio BF OB BK SF ST H 39 28 26 MR 45 31 30 FT 53 38 39 OB 36 36 35 BK 44 32 28 SF 43 31 25 30 ST 34 25 29 34 Precision BF OB BK SF ST H 80 91 93 MR 76 76 72 FT 84 93 95 OB 89 93 96 BK 81 90 87 SF 80 88 82 78 ST 76 88 87 86 Matching Score BF OB BK SF ST H 31 26 24 MR 34 24 22 FT 44 36 37 OB 32 34 34 BK 36 29 24 SF 34 28 21 23 ST 26 22 25 29 Recall BF OB BK SF ST H 63 51 52 MR 63 44 39 FT 65 51 58 OB 51 53 45 BK 55 43 37 SF 73 58 44 53 ST 58 48 48 74 DayNight Illumination Putative Match Ratio BF OB BK SF ST H 13 8 7 MR 15 10 9 FT 11 7 7 OB 6 5 5 BK 9 6 6 SF 10 5 4 8 ST 7 5 6 10 Precision BF OB BK SF ST H 64 76 82 MR 52 41 36 FT 71 76 84 OB 78 80 89 BK 69 73 79 SF 57 60 57 37 ST 49 54 61 55 Matching Score BF OB BK SF ST H 8 6 5 MR 8 4 3 FT 7 5 6 OB 4 4 4 BK 6 4 4 SF 6 3 2 3 ST 4 3 3 6 Recall BF OB BK SF ST H 32 23 22 MR 47 24 18 FT 25 17 21 OB 13 13 11 BK 20 14 15 SF 35 21 15 19 ST 24 17 18 40 Scale Putative Match Ratio BF OB BK SF ST H 19 10 12 MR 10 8 9 FT 15 7 12 OB 5 21 5 BK 11 7 40 SF 15 8 8 56 ST 14 7 10 57 Precision BF OB BK SF ST H 80 80 87 MR 69 48 53 FT 77 73 89 OB 79 92 91 BK 74 73 78 SF 75 67 70 81 ST 76 72 82 96 Matching Score BF OB BK SF ST H 15 8 11 MR 7 4 5 FT 11 6 10 OB 4 19 5 BK 8 5 31 SF 11 5 6 45 ST 11 5 8 54 Recall BF OB BK SF ST H 24 13 17 MR 10 6 7 FT 19 9 17 OB 6 29 6 BK 11 7 42 SF 17 8 8 69 ST 16 7 10 80 Rotation Putative Match Ratio BF OB BK SF ST H 3 69 64 MR 5 44 42 FT 3 62 68 OB 1 62 58 BK 2 52 51 SF 4 44 38 43 ST 3 41 50 58 Precision BF OB BK SF ST H 14 93 88 MR 18 90 78 FT 15 91 93 OB 12 86 94 BK 12 89 86 SF 13 88 79 82 ST 13 93 84 95 Matching Score BF OB BK SF ST H 0 64 56 MR 1 40 33 FT 0 57 64 OB 0 53 55 BK 0 46 44 SF 0 38 30 35 ST 0 38 42 55 Recall BF OB BK SF ST H 0 68 62 MR 1 50 39 FT 0 63 74 OB 0 62 53 BK 0 52 48 SF 1 57 45 55 ST 1 50 48 79 Scale & Rotation Putative Match Ratio BF OB BK SF ST H 4 6 6 MR 6 6 5 FT 2 4 6 OB 1 8 4 BK 2 4 13 SF 3 4 4 14 ST 3 4 4 13 Precision BF OB BK SF ST H 34 65 81 MR 19 40 35 FT 23 64 83 OB 20 83 86 BK 27 68 84 SF 20 48 58 68 ST 18 55 72 84 Matching Score BF OB BK SF ST H 1 4 5 MR 1 2 2 FT 1 3 5 OB 0 7 3 BK 0 3 11 SF 1 2 2 9 ST 0 2 3 11 Recall BF OB BK SF ST H 2 6 9 MR 5 9 8 FT 1 4 7 OB 0 8 3 BK 1 3 14 SF 1 5 6 25 ST 1 5 7 30 Perspective Planar Scene Putative Match Ratio BF OB BK SF ST H 29 14 17 MR 15 9 9 FT 25 13 17 OB 12 15 13 BK 19 11 13 SF 18 10 10 15 ST 18 9 13 24 Precision BF OB BK SF ST H 88 87 90 MR 76 68 70 FT 84 85 89 OB 89 88 92 BK 86 83 87 SF 78 73 75 73 ST 83 80 81 86 Matching Score BF OB BK SF ST H 26 13 15 MR 11 6 6 FT 21 11 15 OB 11 13 12 BK 16 9 11 SF 14 7 7 11 ST 15 7 11 20 Recall BF OB BK SF ST H 36 18 22 MR 25 13 14 FT 28 8 21 OB 13 15 12 BK 22 7 15 SF 32 17 17 25 ST 28 14 17 42 Perspective NonPlanar Putative Match Ratio BF OB BK SF ST H 22 13 14 MR 17 11 11 FT 21 12 15 OB 9 13 9 BK 15 9 15 SF 16 9 9 18 ST 15 9 13 22 Precision BF OB BK SF ST H 60 66 73 MR 55 39 37 FT 73 73 81 OB 77 80 85 BK 63 62 72 SF 58 54 58 61 ST 69 68 75 84 Matching Score BF OB BK SF ST H 13 9 10 MR 9 4 4 FT 15 9 12 OB 7 10 8 BK 9 5 11 SF 10 5 5 11 ST 11 6 10 19 Recall BF OB BK SF ST H 30 19 23 MR 32 15 13 FT 30 17 24 OB 12 18 11 BK 19 10 20 SF 28 14 15 33 ST 24 13 18 44 Putative Match Ratio BF OB BK SF ST H 11 6 6 MR 11 7 7 FT 9 5 5 OB 4 6 4 BK 7 4 7 SF 10 6 5 11 ST 8 4 5 11 Precision BF OB BK SF ST H 81 78 84 MR 76 72 63 FT 84 77 86 OB 87 84 88 BK 83 77 86 SF 79 73 75 79 ST 76 71 75 83 Matching Score BF OB BK SF ST H 9 5 5 MR 8 5 4 FT 7 4 5 OB 4 5 3 BK 6 3 6 SF 8 4 4 8 ST 6 3 4 10 Recall BF OB BK SF ST H 8 5 5 MR 8 5 4 FT 7 4 5 OB 4 5 3 BK 6 3 5 SF 8 4 4 9 ST 6 3 3 10 Photo Collection H MR FT OB BK SF ST = = = = = = = Harris MSER FAST ORB BRISK SURF SIFT Detectors BF OB BK SF ST = = = = = BRIEF ORB BRISK SURF SIFT Descriptors Computation Requirements Feature Distribution Evaluation of Detector/Descriptor Pairings

Upload: others

Post on 26-Jul-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Goals& Metrics Datasets& - Computer Science · Detector/Descriptor& BRIEF ORB BRISK SURF’ SIFT FAST’ Harris’ MSER Detector&ms/image& n/a’ 17 43 377 572 2.7 78 117 Descriptor&μs/feature&

Datasets  

Sample  datasets  from  [4],  [5],  and  our  work.  

Descriptor  Patterns  

BRIEF  [1]   ORB  [2]  Rotation  Invariant  

BRISK  [3]  Rotation  Invariant  Scale  Invariant  

Bibliography  1.  Calonder,  M.,  Lepe2t,  V.,  Strecha,  C.,  Fua,  P.,  “BRIEF:  Binary  Robust  

Independent  Elementary  Features,”  ECCV,  778-­‐792  (2010).  2.  Rublee,  E.,  Rabaud,  V.,  Konolige,  K.,  Bradski,  G.,  “ORB:  An  Efficient  

Alterna2ve  to  SIFT  or  SURF,”  ICCV,  2564-­‐2571  (2011).  3.  Leutenegger,  S.,  Chli,  M.,  Siegwart,  R.,  “BRISK:  Binary  Robust  Invariant  

Scalable  Keypoints,”  ICCV,  2548-­‐2555  (2011).  4.  Mikolajczyk,  K.,  et  al.,  “A  Comparison  of  Affine  Region  Detectors,”  IJCV  

65(1-­‐2),  43-­‐72  (2005).  5.  Strecha,  C.,  et  al.,  “On  Benchmarking  Camera  Calibra2on  and  Mul2-­‐View  

Stereo  for  High  Resolu2on  Imagery,”  CVPR,  1-­‐8  (2008).  

Compute  

Storag

e  

Real  Value  Parameterization  Patch-­‐Based  

Binarized  Binary  

SIFT  

SURF  

BRIEF   ORB  

BRISK  

LDAHash  

PCA-­‐DSC  

SSD   NCC   MI  

PCA-­‐SIFT  

DAISY  

Taxonomy  

•  Descriptor  is  built  from  a  set  of  pairwise  intensity  comparisons  •  Each  bit  in  the  descriptor  is  the  result  of  exactly  one  comparison  •  The  sampling  pattern  is  fixed  (except  for  possible  scaling/rotation)  •  Hamming  distance  can  be  used  as  a  similarity  measure  •  Provide  robustness  as  well  as  high  computational  efficiency  

Binary  Descriptor  Properties  

Results  

Detector/Descriptor   BRIEF   ORB   BRISK   SURF   SIFT   FAST   Harris   MSER  Detector  ms/image   n/a   17   43   377   572   2.7   78   117  Descriptor  μs/feature   4.4   4.8   12.9   143   314   n/a   n/a   n/a  Storage  bytes/feature   16,32,64   32   64   64(256)   128(512)   n/a   n/a   n/a  

Goals  •  Develop  benchmark  for  binary  features  that  is  compatible  with  existing  evaluations  

•  Perform  quantitative  analysis  on  detector/descriptor  pairings  •  Provide  a  comprehensive  set  of  evaluation  datasets  

Conclusions  •  Avoid  using  descriptors  that  are  invariant  to  a  transform  not  present  in  your  dataset  

•  Use  a  detector/descriptor  pairing  where  both  are  invariant  to  the  same  set  of  transforms  

•  Order-­‐of-­‐magnitude  speed  gains  are  achievable  by  binary  descriptors,  resulting  (at  worse)  in  marginal  matching  performance  penalties    

Metrics  

#  Correct  Matches   #  Putative  Matches   #  Features  

#  Correspondences  =  #  Ground  Truth  Matches  

#  Correct  Matches  Precision  =   /  #  Putative  Matches  #  Putative  Matches  Putative  Match  Ratio  =   #  Features  /  #  Correct  Matches  Matching  Score  =   #  Features  /  #  Correct  Matches  Recall  =   #  Correspondences  /  

Detector  Entropy  =  Entropy  of  2D  histogram  of  feature  loca2ons  

Detector   FAST   SIFT   BRISK   SURF   ORB   Harris   MSER  Avg  #  Features   8166   4788   7771   3766   13427   2543   693  Entropy   12.52   12.34   12.33   12.26   12.10   11.84   10.74  

Blur  1 BlurPutative Match Ratio

BF OB BK SF STH 25 11 11MR 19 10 9FT 23 10 11OB 13 10 8BK 20 9 9SF 31 15 12 15ST 19 8 9 10

Precision

BF OB BK SF STH 67 78 84MR 65 71 69FT 68 72 85OB 70 78 87BK 67 77 73SF 72 82 79 71ST 63 76 76 73

Matching Score

BF OB BK SF STH 17 8 9MR 12 7 6FT 15 7 9OB 9 8 7BK 13 7 6SF 22 12 9 11ST 12 6 7 7

Recall

BF OB BK SF STH 44 22 24MR 40 22 19FT 39 10 24OB 19 16 12BK 31 9 14SF 55 29 22 28ST 38 18 18 24

Compression  2 Compression

Putative Match Ratio

BF OB BK SF STH 68 54 51MR 43 31 29FT 50 33 35OB 35 36 30BK 50 35 33SF 59 43 30 47ST 35 23 28 29

Precision

BF OB BK SF STH 95 97 97MR 83 88 86FT 91 96 97OB 95 97 98BK 93 97 94SF 92 96 94 90ST 85 93 94 91

Matching Score

BF OB BK SF STH 64 53 49MR 36 27 25FT 46 32 34OB 34 35 30BK 46 34 32SF 54 41 28 42ST 30 22 27 26

Recall

BF OB BK SF STH 80 64 63MR 53 40 33FT 69 47 54OB 47 49 35BK 59 43 41SF 74 56 39 61ST 55 40 40 55

Exposure  3 Exposure

Putative Match Ratio

BF OB BK SF STH 39 28 26MR 45 31 30FT 53 38 39OB 36 36 35BK 44 32 28SF 43 31 25 30ST 34 25 29 34

Precision

BF OB BK SF STH 80 91 93MR 76 76 72FT 84 93 95OB 89 93 96BK 81 90 87SF 80 88 82 78ST 76 88 87 86

Matching Score

BF OB BK SF STH 31 26 24MR 34 24 22FT 44 36 37OB 32 34 34BK 36 29 24SF 34 28 21 23ST 26 22 25 29

Recall

BF OB BK SF STH 63 51 52MR 63 44 39FT 65 51 58OB 51 53 45BK 55 43 37SF 73 58 44 53ST 58 48 48 74

Day-­‐Night  Illumination  4 Day-Night

Putative Match Ratio

BF OB BK SF STH 13 8 7MR 15 10 9FT 11 7 7OB 6 5 5BK 9 6 6SF 10 5 4 8ST 7 5 6 10

Precision

BF OB BK SF STH 64 76 82MR 52 41 36FT 71 76 84OB 78 80 89BK 69 73 79SF 57 60 57 37ST 49 54 61 55

Matching Score

BF OB BK SF STH 8 6 5MR 8 4 3FT 7 5 6OB 4 4 4BK 6 4 4SF 6 3 2 3ST 4 3 3 6

Recall

BF OB BK SF STH 32 23 22MR 47 24 18FT 25 17 21OB 13 13 11BK 20 14 15SF 35 21 15 19ST 24 17 18 40

Scale  6 ScalePutative Match Ratio

BF OB BK SF STH 19 10 12MR 10 8 9FT 15 7 12OB 5 21 5BK 11 7 40SF 15 8 8 56ST 14 7 10 57

Precision

BF OB BK SF STH 80 80 87MR 69 48 53FT 77 73 89OB 79 92 91BK 74 73 78SF 75 67 70 81ST 76 72 82 96

Matching Score

BF OB BK SF STH 15 8 11MR 7 4 5FT 11 6 10OB 4 19 5BK 8 5 31SF 11 5 6 45ST 11 5 8 54

Recall

BF OB BK SF STH 24 13 17MR 10 6 7FT 19 9 17OB 6 29 6BK 11 7 42SF 17 8 8 69ST 16 7 10 80

Rotation  5 RotationPutative Match Ratio

BF OB BK SF STH 3 69 64MR 5 44 42FT 3 62 68OB 1 62 58BK 2 52 51SF 4 44 38 43ST 3 41 50 58

Precision

BF OB BK SF STH 14 93 88MR 18 90 78FT 15 91 93OB 12 86 94BK 12 89 86SF 13 88 79 82ST 13 93 84 95

Matching Score

BF OB BK SF STH 0 64 56MR 1 40 33FT 0 57 64OB 0 53 55BK 0 46 44SF 0 38 30 35ST 0 38 42 55

Recall

BF OB BK SF STH 0 68 62MR 1 50 39FT 0 63 74OB 0 62 53BK 0 52 48SF 1 57 45 55ST 1 50 48 79

Scale  &  Rotation  7 Scale-Rotation

Putative Match Ratio

BF OB BK SF STH 4 6 6MR 6 6 5FT 2 4 6OB 1 8 4BK 2 4 13SF 3 4 4 14ST 3 4 4 13

Precision

BF OB BK SF STH 34 65 81MR 19 40 35FT 23 64 83OB 20 83 86BK 27 68 84SF 20 48 58 68ST 18 55 72 84

Matching Score

BF OB BK SF STH 1 4 5MR 1 2 2FT 1 3 5OB 0 7 3BK 0 3 11SF 1 2 2 9ST 0 2 3 11

Recall

BF OB BK SF STH 2 6 9MR 5 9 8FT 1 4 7OB 0 8 3BK 1 3 14SF 1 5 6 25ST 1 5 7 30

Perspective  Planar  Scene  8 Perspective-Planar

Putative Match Ratio

BF OB BK SF STH 29 14 17MR 15 9 9FT 25 13 17OB 12 15 13BK 19 11 13SF 18 10 10 15ST 18 9 13 24

Precision

BF OB BK SF STH 88 87 90MR 76 68 70FT 84 85 89OB 89 88 92BK 86 83 87SF 78 73 75 73ST 83 80 81 86

Matching Score

BF OB BK SF STH 26 13 15MR 11 6 6FT 21 11 15OB 11 13 12BK 16 9 11SF 14 7 7 11ST 15 7 11 20

Recall

BF OB BK SF STH 36 18 22MR 25 13 14FT 28 8 21OB 13 15 12BK 22 7 15SF 32 17 17 25ST 28 14 17 42

Perspective  Non-­‐Planar  9 Perspective-Nonplanar

Putative Match Ratio

BF OB BK SF STH 22 13 14MR 17 11 11FT 21 12 15OB 9 13 9BK 15 9 15SF 16 9 9 18ST 15 9 13 22

Precision

BF OB BK SF STH 60 66 73MR 55 39 37FT 73 73 81OB 77 80 85BK 63 62 72SF 58 54 58 61ST 69 68 75 84

Matching Score

BF OB BK SF STH 13 9 10MR 9 4 4FT 15 9 12OB 7 10 8BK 9 5 11SF 10 5 5 11ST 11 6 10 19

Recall

BF OB BK SF STH 30 19 23MR 32 15 13FT 30 17 24OB 12 18 11BK 19 10 20SF 28 14 15 33ST 24 13 18 44

10 Photo-CollectionPutative Match Ratio

BF OB BK SF STH 11 6 6MR 11 7 7FT 9 5 5OB 4 6 4BK 7 4 7SF 10 6 5 11ST 8 4 5 11

Precision

BF OB BK SF STH 81 78 84MR 76 72 63FT 84 77 86OB 87 84 88BK 83 77 86SF 79 73 75 79ST 76 71 75 83

Matching Score

BF OB BK SF STH 9 5 5MR 8 5 4FT 7 4 5OB 4 5 3BK 6 3 6SF 8 4 4 8ST 6 3 4 10

Recall

BF OB BK SF STH 8 5 5MR 8 5 4FT 7 4 5OB 4 5 3BK 6 3 5SF 8 4 4 9ST 6 3 3 10

Photo  Collection  

H  MR  FT  OB  BK  SF  ST  

=  =  =  =  =  =  =  

Harris  MSER  FAST  ORB  BRISK  SURF  SIFT  

Detectors  

BF  OB  BK  SF  ST  

=  =  =  =  =  

BRIEF  ORB  BRISK  SURF  SIFT  

Descriptors  

Computation  Requirements   Feature  Distribution  

Evaluation  of  Detector/Descriptor  Pairings