using relevance feedback in multimedia databases chotirat “ann” ratanamahatana eamonn keogh 7 th...
Post on 21-Dec-2015
218 views
TRANSCRIPT
![Page 1: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/1.jpg)
Using Relevance Feedback in Multimedia Databases
Chotirat “Ann” Ratanamahatana
Eamonn Keogh
7th International Conference on VISual Information Systemsat 10th International Conference on Distributed Multimedia Systems
September 9, 2004
![Page 2: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/2.jpg)
Roadmap
• Time series in multimedia databases and their similarity
measures
• Euclidean distance and its limitation
• Dynamic time warping (DTW)
• Global constraints and R-K Band
• Relevance Feedback and Query Refinement
• Experimental Evaluation
• Conclusions and future work
![Page 3: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/3.jpg)
What are Time Series• A collection of observations made sequentially
in time.• People measure things…
and things…change over time…
• Their blood pressure• George Bush's popularity rating• The annual rainfall in San Francisco• The value of their Google stock
• Their blood pressure• George Bush's popularity rating• The annual rainfall in San Francisco• The value of their Google stock
![Page 4: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/4.jpg)
Time Series in Multimedia Databases
Image data may best be thought of as time series…
![Page 5: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/5.jpg)
Image to Time Series
![Page 6: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/6.jpg)
Video to Time Series
0 50 100 150-1.5
-1
-0.5
0
0.5
1
1.5
Hand moving above holster
Hand moving downto grasp gun
Steady pointing
Hand moving toshoulder level
Hand at rest
![Page 7: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/7.jpg)
Time Series in Multimedia Databases
0 10 20 30 40 50 60 70 80 90
Hand at rest
Aiming gun
Hand moving toshoulder level
Hand moving downto grasp gun
Hand movingabove holster
0 10 20 30 40 50 60 70 80 90
Hand at rest
Aiming gun
Hand moving toshoulder level
Hand moving downto grasp gun
Hand movingabove holster
0 10 20 30 40 50 60 70 80 90
Hand at rest
Aiming gun
Hand moving toshoulder level
Hand moving downto grasp gun
Hand movingabove holster
0 10 20 30 40 50 60 70 80 90
Hand at rest
Aiming gun
Hand moving toshoulder level
Hand moving downto grasp gun
Hand movingabove holster
0 10 20 30 40 50 60 70 80 90
Hand at rest
Aiming gun
Hand moving toshoulder level
Hand moving downto grasp gun
Hand movingabove holster
0 10 20 30 40 50 60 70 80 90
Hand at rest
Aiming gun
Hand moving toshoulder level
Hand moving downto grasp gun
Hand movingabove holster
0 10 20 30 40 50 60 70 80 90
Hand at rest
Aiming gun
Hand moving toshoulder level
Hand moving downto grasp gun
Hand movingabove holster
Video
0 50 100 150 200 250 300 350 400 4500
0.5
1
0 50 100 150 200 250 300 350 400 4500
0.5
1
George Washington’sManuscript
![Page 8: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/8.jpg)
Classification in Time Series
Pattern Recognition is a type of supervised classification where an input pattern is classified into one of the classes based on its similarity to these predefined classes.
Class BClass BClass AClass A
Which class does
belong to?
![Page 9: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/9.jpg)
Euclidean Distance MetricGiven 2 time series
Q = q1, …, qn and
C = c1, …, cn
their Euclidean distance is
defined as
n
iii cqCQD
1
2)(),(
0 50 100 150-1.5
-1
-0.5
0
0.5
1
1.5
0 50 100 150-1.5
-1
-0.5
0
0.5
1
1.5
Q
C
![Page 10: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/10.jpg)
Limitations of Euclidean MetricVery sensitive to some distortion in the data
Training data consistsof 10 instances fromeach of the 3 classes
Training data consistsof 10 instances fromeach of the 3 classes
Perform a 1-nearest neighbor algorithm, with “leaving-one-out”
evaluation, averaged over 100 runs.
Euclidean distance Error rate:29.77%
DTW Error rate:3.33 %
![Page 11: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/11.jpg)
Dynamic Time Warping (DTW)
Euclidean DistanceOne-to-one alignments
Time Warping DistanceNon-linear alignments are allowed
![Page 12: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/12.jpg)
How Is DTW Calculated? (I)
QC
K
k kwCQDTW1
min),(
Warping path w
Q
C
Q
C
![Page 13: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/13.jpg)
How Is DTW Calculated? (II)Each warping path w can be found using dynamic programming to evaluatethe following recurrence:
)}1,(),,1(),1,1(min{),(),( jijijicqdji ji
where γ(i, j) is the cumulative distance of the distance d(i, j) and its minimumcumulative distance among the adjacent cells.
(i-1, j)
(i, j-1)
(i, j)
(i-1, j-1)
![Page 14: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/14.jpg)
Global Constraints (I)
C
Q
C
Q
C
Q
C
Q
Sakoe-Chiba Band Itakura Parallelogram
Prevent any unreasonable
warping
Prevent any unreasonable
warping
![Page 15: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/15.jpg)
Global Constraints (II)
Ri
Sakoe-Chiba Band Itakura Parallelogram
A Global Constraint for a sequence of size m is defined by R, whereRi = d 0 d m, 1 i m.
Ri defines a freedom of warping above and to the right of the diagonal at any given point i in the sequence.
![Page 16: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/16.jpg)
Ratanamahatana-Keogh Band (R-K Band)
Solution: we create an arbitrary shape and size of the band that is appropriate for the data we want to classify.
![Page 17: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/17.jpg)
How Do We Create an R-K Band?First Attempt: We could look at the data and manually create the shape of the bands.
(then we need to adjust the width of each band as well until we get a good result)
50 100 150 200 250
50
100
150
200
250
50 100 150 200 250
50
100
150
200
250
0 50 100 150 200 250-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
0 50 100 150 200 250-2.5
-2
-1.5
-1
-0.5
0
0.5
1
100 % Accuracy!
![Page 18: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/18.jpg)
Learning an R-K Band Automatically
50 100 150 200 250
50
100
150
200
250
0 50 100 150 200 250-2.5
-2
-1.5
-1
-0.5
0
0.5
1
0 50 100 150 200 250-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
50 100 150 200 250
50
100
150
200
250
Our heuristic search algorithm automatically learns the bands from the data.(sometimes, we can even get an unintuitive shape that give a good result.)
100 % Accuracy as well!
![Page 19: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/19.jpg)
Calculate h(1)
Calculate h(2)
h(2) > h(1) ? Yes No
Calculate h(1)
Calculate h(2)
h(2) > h(1) ? Yes No
R-K Band Learning With Heuristic Search
![Page 20: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/20.jpg)
R-K Band Learning in Action!
![Page 21: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/21.jpg)
Classification Examples with R-K Bands
Error rate
Euclidean 32.13%
DTW 10% 4.52%
R-K Bands 0.9%
![Page 22: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/22.jpg)
Face Classification
![Page 23: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/23.jpg)
Relevance Feedback
• A well-known and effective method in improving the query performance, especially in text-mining domains.– Refining the query based on user’s reaction
• Only relatively little research has been done on relevance feedback in images or multimedia data.
![Page 24: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/24.jpg)
Query Refinement
Averaging a collection of time series using DTW, according to their weights and warping (DTW) alignments.
Averaged SequenceAveraged Sequence
![Page 25: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/25.jpg)
1. Gun Problem
2. Leaf Dataset
3. Handwritten Word Spotting data
Experiment: Datasets
0 50 100 150-1
-0.5
0
0.5
1
1.5
2
2.5
0 50 100 150-1
-0.5
0
0.5
1
1.5
2
2.5
![Page 26: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/26.jpg)
Experimental Design
Given an initial query, we measure the precision and recall for each round of the relevance feedback retrieval.• Show the 10 best matches (k-nearest neighbors).• User ranks each result.• Accumulatively build the training set.• Learn an R-K band according to the current training data.• Generate a new query (query refinement), and repeat.
![Page 27: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/27.jpg)
![Page 28: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/28.jpg)
Results: Gun
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cisi
on
Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5
Gun
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cisi
on
Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5
Gun
![Page 29: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/29.jpg)
Results: Leaf
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cis
ion
Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5
Leaf
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cis
ion
Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5
Leaf
![Page 30: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/30.jpg)
Results: Wordspotting
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cisi
on
Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5
WordSpotting 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cisi
on
Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5
WordSpotting
![Page 31: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/31.jpg)
Conclusions
• Different shapes and widths of the band contributes to the classification accuracy / precision.
• We have shown that incorporating R-K Band into relevance feedback can reduce the error rate in classification, and improve the precision at all recall levels in video and image retrieval.
![Page 32: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/32.jpg)
Future Work
• Investigate other choices that may make envelope learning more accurate.– Heuristic functions– Search algorithm (refining the search)
• Is there a way to always guarantee an optimal solution?• Examine the best way to deal with multi-variate time
series for more complex data.• Explore other utilities of R-K Band and relevance
feedback, specifically on real-world problems: music, bioinformatics, biomedical data, etc.
![Page 33: Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7 th International Conference on VISual Information Systems](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d635503460f94a456ea/html5/thumbnails/33.jpg)
UCR Time Series Data Mining Archive: http://www.cs.ucr.edu/~eamonn/TSDMA
Contact: [email protected] [email protected]
Homepage: http://www.cs.ucr.edu/~ratana
All datasets are publicly available at: