automatic photo selection for media and entertainment applications
TRANSCRIPT
![Page 1: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/1.jpg)
Automatic Photo Selection for Media and Entertainment
ApplicationsEkaterina Potapova,
Marta Egorova, Ilia Safonov
National Nuclear Research University MEPhI Moscow, Russia
GraphiCon 20095-9 October
![Page 2: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/2.jpg)
Applications
Automatic Photo Selection for Media and Entertainment Applications
GraphiCon 2009 2
![Page 3: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/3.jpg)
Applications
Automatic Photo Selection for Media and Entertainment Applications
GraphiCon 2009 2
![Page 4: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/4.jpg)
Applications – photo book
Images are taken from printbook.ru, ehow.com, snapfish.com.au, smilebooks.co.ukGraphiCon 2009 3
Automatic Photo Selection for Media and Entertainment Applications
![Page 5: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/5.jpg)
Applications – slide show
Photos from ITaS’2008GraphiCon 2009 4
Automatic Photo Selection for Media and Entertainment Applications
![Page 6: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/6.jpg)
General workflow
GraphiCon 2009 5
Automatic Photo Selection for Media and Entertainment Applications
![Page 7: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/7.jpg)
Detection of low-quality photos
GraphiCon 2009 5
Automatic Photo Selection for Media and Entertainment Applications
General workflow
![Page 8: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/8.jpg)
General workflow
Detection of low-quality photos
Adaptive quantization on
time-camera plane
GraphiCon 2009 5
Automatic Photo Selection for Media and Entertainment Applications
![Page 9: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/9.jpg)
Selection of appealing photos
Detection of low-quality photos
Adaptive quantization on
time-camera plane
General workflow
GraphiCon 2009 5
Automatic Photo Selection for Media and Entertainment Applications
![Page 10: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/10.jpg)
Detection of low-quality photos
GraphiCon 2009 6
Automatic Photo Selection for Media and Entertainment Applications
![Page 11: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/11.jpg)
Estimation of JPEG qualityA.Foi et al.,2007
3
1
3
1,9
1
i jjiqK
Images are taken from en.wikipedia.org
Quantization Table
GraphiCon 2009 7
Automatic Photo Selection for Media and Entertainment Applications
![Page 12: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/12.jpg)
Detection of backlit, low-contrast & blurred photos
+
Good photo
Bad photo
True
False
…
…
11 TF
ii TF
NN TF
}{ iF
1w
iw
Nw
N
ii Tw
1
Two Ada Boost classifiers committee: -for detection of low-contrast and backlit photos-for detection of blurred photos
GraphiCon 2009 8
Automatic Photo Selection for Media and Entertainment Applications
![Page 13: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/13.jpg)
Detection of backlit and low-contrast photos - 1 S1/S2 - ratio of tones in shadows to midtones
)/()(]85,0[
1 NMiHS )/()(]170,85(
2 NMiHS
GraphiCon 2009 9
Automatic Photo Selection for Media and Entertainment Applications
![Page 14: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/14.jpg)
S11/S12 - ratio of tones in first to second part of shadows
)(/)(]42,0[
11 NMiHS )(/)(]85,42(
12 NMiHS
Detection of backlit and low-contrast photos - 1
GraphiCon 2009 9
Automatic Photo Selection for Media and Entertainment Applications
![Page 15: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/15.jpg)
M1/M2 - ratio of the histogram maximum in shadows to the maximum in midtones
]255,0[]85,0[1 ))(max(/))(max( iHiHM
]255,0[]170,85(2 ))(max(/))(max( iHiHM
Detection of backlit and low-contrast photos - 1
GraphiCon 2009 9
Automatic Photo Selection for Media and Entertainment Applications
![Page 16: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/16.jpg)
P1 - location of the histogram maximum in shadows
]85,0[1 ))(max()(| iHlHlP
]85,0[))(max( iH
P1
Detection of backlit and low-contrast photos - 1
GraphiCon 2009 9
Automatic Photo Selection for Media and Entertainment Applications
![Page 17: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/17.jpg)
C – global contrastlowhighC
})][|min{},][|min(min{0
00
i
k
CkHiHiHilow })][|max{},][|max(max{1
11
ik
RR CkHiHiHihigh
H0
C0
C1
H1
Detection of backlit and low-contrast photos - 1
GraphiCon 2009 9
Automatic Photo Selection for Media and Entertainment Applications
![Page 18: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/18.jpg)
Training set: 480 photos
Error rate on cross-validation test : ~0.055
Testing set: 1830 with 2% affected by backlit and low-contrast photos
The number of False Positives (FP) is 10 The number of False Negatives (FN) is 3
Low-contrast photoBacklit photo
Detection of backlit and low-contrast photos - 2
GraphiCon 2009 10
Automatic Photo Selection for Media and Entertainment Applications
![Page 19: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/19.jpg)
Image
Intensity image
Z1=[-1 1]Z2=[-1 0 1]
Z3=[-1 0 0 1]
Z10=[-1 0 0 0 0 0 0 0 0 0 1]
I.Safonov et al.,2008
…
Edge image
ii ZIE
3
bgrI
Histogram
iHe
Normalized entropy
k
kiHeiA )1)(log(
Entropy to [0, 1]
iAn
121 An An F
10
12
iiAnF
23 A F
?
?
?
?
An An
GraphiCon 2009 11
Detection of blurred photosAutomatic Photo Selection for Media and Entertainment Applications
![Page 20: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/20.jpg)
SDI
SVSDI F h )(
4
Crete et al., 2007
cr
crDISDI,
),(
crcrDBcrDIhh
hecrDBcrDISV
,)),(),((1001
1)),(),((
F.Crete et al.,2007
LPFIBh
HPFIDI
HPFBDB hh
231 An An F
11
22
iiAnF
23 A F
?
Image
Blurred image Edge image
Edge image
Comparison of the images
HPF=[1 -1]LPF=[1 1 1 1 1 1 1 1 1]/9
Detection of blurred photos
GraphiCon 2009 11
Automatic Photo Selection for Media and Entertainment Applications
![Page 21: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/21.jpg)
Training set: 416 photos
Error rate on cross-validation test : ~0.07
Testing set: 1830 with 171 blurred photos
The number of False Positives (FP) is 34
The number of False Negatives (FN) is 10SDI
SVSDI F h )(
4
231 An An F
11
22
iiAnF
23 A F
Detection of blurred photos
GraphiCon 2009 11
Automatic Photo Selection for Media and Entertainment Applications
![Page 22: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/22.jpg)
Time and camera-based quantization
Photo creation time
Photo source
1
243
L
H
evenisiiNpsH
oddisiiHYpi :)2/1(
:2/)1(
i is an index of source
L is time between the least and the most time for the largest source
Nps is a number of sources
H = L/M
M is count of images
Nregion < M
Calculation of bounding boxes
Partition into 2 app. equal subregions
Seeking for the biggest region
1200
3600
2400
72000 36000 T, s21600GraphiCon 2009 11
Automatic Photo Selection for Media and Entertainment Applications
Nregion < Ngroup
![Page 23: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/23.jpg)
GraphiCon 2009 12
Automatic Photo Selection for Media and Entertainment Applications
Salient Photo SelectionThe most appealing photo is the most salient photo
L.Itti, C.Koch et al.
Images are taken from the Internet
![Page 24: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/24.jpg)
Conspicuity maps
Gaussian pyramids
Image
Intensity image
r-channel g-channel b-channel
R-channel G-channel B-channel Y-channel
Orientation map
Intensity map
Color map
Saliency map
Feature maps
Gabor pyramids
GraphiCon 2009 13
Automatic Photo Selection for Media and Entertainment Applications
Salient Photo Selection
![Page 25: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/25.jpg)
original image
saliency map
intensity map
color map
orientation map
ROI
Automatic Photo Selection for Media and Entertainment Applications
Salient Photo Selection
GraphiCon 2009 14Image is taken from the Internet
![Page 26: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/26.jpg)
Automatic Photo Selection for Media and Entertainment Applications
Salient Photo Selection
GraphiCon 2009 15
Saliency Index
4),( maxS
yxS 124
88
11
100
81 9262
83105 70
![Page 27: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/27.jpg)
83
11124
Automatic Photo Selection for Media and Entertainment Applications
Salient Photo Selection
GraphiCon 2009 15
Saliency Index
4),( maxS
yxS
81
88 62 92
105 70
100
![Page 28: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/28.jpg)
Main Disadvantages:
average number of FP increases a lot with picture size
0
1
2
3
4
5
6
7
0 500 1000 1500 2000 2500
S, pixelav
gn
FP
0
5
10
15
20
25
30
0 500 1000 1500 2000 2500 3000
S, pixelt,
s.
0
1
2
3
4
5
6
7
0 500 1000 1500 2000 2500S, pixel
avg
nF
PBefore skin tone detection After skin tone detection
0
5
10
15
20
25
30
0 1000 2000 3000S, pixel
t, s.
After modification Before modification
We consider, that images of people attracts more attention
processing time also increases a lot with picture size
Six places were detected erroneously
Modifications: image down sampling is applied at preprocessing step
optimization of search using color information – skin tone detection
P.Viola, M.Jones, 2001
Automatic Photo Selection for Media and Entertainment Applications
Face Detection
GraphiCon 2009 16
Viola-Jones, Intel OpenCV
Before modifications After modifications
![Page 29: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/29.jpg)
Photos ranking
Heuristic formula, experiments have shown that value w=25 gives the best result
Automatic Photo Selection for Media and Entertainment Applications
GraphiCon 2009 17
124
88
11
116 92
118148 95
62
100
![Page 30: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/30.jpg)
118
62
Photos ranking
Heuristic formula, experiments have shown that value w=25 gives the best result
Automatic Photo Selection for Media and Entertainment Applications
GraphiCon 2009 17
124
88
11
100
116
92
148 95
![Page 31: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/31.jpg)
Automatic Photo Selection for Media and Entertainment Applications
Results and discussion
GraphiCon 2009 18
![Page 32: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/32.jpg)
Automatic Photo Selection for Media and Entertainment Applications
Results and discussion
GraphiCon 2009 18
Autocollage choiceOur choice
![Page 33: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/33.jpg)
Automatic Photo Selection for Media and Entertainment Applications
Results and discussion
GraphiCon 2009 18
![Page 34: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/34.jpg)
Automatic Photo Selection for Media and Entertainment Applications
Results and discussion
GraphiCon 2009 18
Autocollage choiceOur choice
![Page 35: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/35.jpg)
Automatic Photo Selection for Media and Entertainment Applications
Results and discussion
GraphiCon 2009 18
![Page 36: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/36.jpg)
Automatic Photo Selection for Media and Entertainment Applications
Results and discussion
GraphiCon 2009 18
Autocollage choiceOur choice
![Page 37: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/37.jpg)
Automatic Photo Selection for Media and Entertainment Applications
Results and discussion
GraphiCon 2009 19
Set 1Set 1 Set 2Set 2 Set 3Set 3 Set 4Set 4 Set 5Set 5 SumSum
Agree with experts 6 5 6 5 7 29
Acceptable 3 4 4 4 2 17
Unacceptable 1 1 0 1 1 4
Agree with experts 2 2 2 6 5 17
Acceptable 6 7 7 0 4 24
Unacceptable 2 1 1 4 1 9
Agree with experts 2 2 3 4 4 15
Acceptable 5 5 4 2 5 21
Unacceptable 3 3 3 4 1 14
Prop
osed
Auto
Colla
geRa
ndom
![Page 38: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/38.jpg)
?Automatic Photo Selection for Media and Entertainment Applications
Questions & Answers
GraphiCon 2009 8
![Page 39: Automatic Photo Selection For Media And Entertainment Applications](https://reader038.vdocuments.us/reader038/viewer/2022110118/554a6fb1b4c90573288b4d82/html5/thumbnails/39.jpg)
Automatic Photo Selection for Media and Entertainment Applications
GraphiCon 2009 9
Thank you for your attention
=)