ieee international conference on image processing...
TRANSCRIPT
![Page 1: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/1.jpg)
![Page 2: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/2.jpg)
IEEE International Conference on Image ProcessingICIP 2007, 16-19 September 2007
Session:MP-L6: Image Scanning, Display, Printing, C olor and Multispectral Processing ITopic:Image Scanning, Display and Printing: Image Q uality Assessment
Does where you gaze on an image affect you percepti on of quality? Applying visual attention to image quality metric
A. Ninassi ¹ O. Le Meur ¹, P. Le Callet ², D. Barba ² ,²׳
¹ THOMSON R&D, VCL, Cesson-Sevigné, France
² IRCCyN UMR 6597 CNRS, Nantes, France
![Page 3: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/3.jpg)
3
Introduction
“Most of the efficient quality metrics are based on the Human Visual System ”
An important component of the HVS=
Visual attention
Image and Video QualityEvaluation
Human Visual System(HVS)
![Page 4: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/4.jpg)
4
Outline
1. Image quality assessment and visual attention2. The ground truth3. Saliency-based quality metrics4. Results5. Conclusion
1.
2.
3.
4.
5.
![Page 5: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/5.jpg)
5
Quality assessment and visual attention 1/3
Visual attention ?
“ Visual attention allows us to select the relevant i nformation in our environment, in connection or not with a particular ta sk ”
Two mechanisms are involved in visual attention controlBottom-up
(involuntary attention)Salient elements of the
visual field catch the attention�
Mechanism based on signal
Top-down(voluntary attention)
Our attention is guided by the task to accomplish
�
Mechanism based on task
![Page 6: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/6.jpg)
6
Quality assessment and visual attention 2/3
Visual attention in image quality assessment ?
Basic idea :“A distortion that appears on region of interest is more annoying than a
distortion appearing on an inconspicuous area”
Simple application to image quality metric :Give more weight to the distortions appearing on
the saliency areas
Distortion weightDistortion weight
![Page 7: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/7.jpg)
7
Quality assessment and visual attention 3/3
Image distortion
evaluation
Pooling
function
Quality
score
Real
Saliency Map
Reference
image
Impairedimage
Full Reference quality metric
Distortionmap
Visual attention information
Hypothetical Saliency Map (model) [Osberger98][Barland06]
?
![Page 8: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/8.jpg)
8
Outline
1. Image quality assessment and visual attention2. The ground truth3. Saliency-based quality metrics4. Results5. Conclusion
1.
2.
3.
4.
5.
![Page 9: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/9.jpg)
9
The ground truth 1/4
Eye gaze tracking experimentsduring a quality assessment campaign
An eye-tracker is a device elaborated to track and to record the position where a human observer
is looking at
Collected data :– Mean observer score (MOS, from
observers’ scores)– Real saliency information (from eye gaze
position)
![Page 10: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/10.jpg)
10
The ground truth : Mean Observer Score 2/4
Picture quality assessment campaign :– 20 observers – 130 pictures (10 unimpaired references,120 impaired
versions)– Visualization distance 4H– DSIS protocol (Double Stimulus Impairment Scale)
Real Saliency Map
![Page 11: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/11.jpg)
11
The ground truth : Mean Observer Score 3/4
Degradation category rating (5 scores) :– Imperceptible– Not annoying– Slightly annoying– Annoying– Very annoying
How to rate ?– Scoring screen– selection and confirmation based on eye
gaze positionScore is Not annoying
![Page 12: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/12.jpg)
12
The ground truth : Saliency Information 4/4
FixationFixation
Sac
cade
Short
Fixation
Human saliencymap computation
Based on fixation duration (FD)
Based on fixation number (FN)
Raw eye tracking data
![Page 13: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/13.jpg)
13
Outline
1. Image quality assessment and visual attention2. The ground truth3. Saliency-based quality metrics4. Results5. Conclusion
1.
2.
3.
4.
5.
![Page 14: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/14.jpg)
14
� Absolute difference� SSIM [Wang04]
Saliency-based quality metrics 1/2
Image distortion
evaluation
Pooling
Function
Quality
score
Saliency map SM(x,y)
Reference
image
Impairedimage
Distortion Mapd(x,y)
![Page 15: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/15.jpg)
15
Saliency-based quality metrics : Pooling function 2/2
( ) ( )( )
( )( )
picturetheof
heighttheandwidththeareHandW
functionweighttheisyxiw
mapdistortiontheisyxd
scorequalitytheisQWhere
W
x
H
y i
W
x
H
y i
yxw
yxdyxwQ ,,
,,
,
1 1
1 1
,
,,
∑ ∑
∑ ∑
= =
= =⋅
=
( ) ( )( ) ( )( ) ( )( ) ( )
( )( ) mapsaliencynormalizedtheisyxnSMand
mapsaliencytheisyxSMWhere
yxSMyxw
yxSMyxw
yxSMyxw
yxSMyxw
n
n
,
,
,1,
,,
,1,
,,
4
3
2
1
+==
+==
Saliency-based spatial pooling :(according to previous literature works [Osberger98][Barland06])
4 simple weight functions :
![Page 16: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/16.jpg)
16
Outline
1. Image quality assessment and visual attention2. The ground truth3. Saliency-based quality metrics4. Results5. Conclusion
1.
2.
3.
4.
5.
![Page 17: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/17.jpg)
17
Results : Predicted MOS vs. MOS 1/5
Mean observer behavior Comparison between MOS and MOSp (predicted MOS) :
� Of simple quality metrics (with and without saliency weighting)– 2 distortion maps
� Absolute difference� Structural similarity (SSIM [Wang04] )
– 4 spatial pooling functions (wi)– 2 “mean observer” saliency maps
� 1 based on fixation number (FN)� 1 based on fixation duration (FD)
� On the whole database� In terms of :
– Correlation coefficient (CC)– Root mean square error (RMSE)
![Page 18: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/18.jpg)
18
Results : Predicted MOS vs. MOS 2/5
Correlation coefficient
00,10,20,30,40,50,60,70,80,9
w1 w2 w3 w4
FN saliency FD saliency
Correlation coefficient
00,10,20,30,4
0,50,60,70,80,9
w1 w2 w3 w4
FN saliency FD saliency
→ Some positive impact of the saliency information
→ The prediction improvement depends on :- weight function
- saliency map without normalization is better
- saliency based on Fixation Duration (FD) is better
RMSE
0
0,2
0,4
0,6
0,8
1
1,2
w1 w2 w3 w4
FN saliency FD saliency
RMSE
0
0,2
0,4
0,6
0,8
1
1,2
w1 w2 w3 w4
FN saliency FD saliency
WithoutSaliencyweighting
WithoutSaliencyweighting
Mean observer behaviorDistortion maps based on the absolute difference
![Page 19: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/19.jpg)
19
Results : Predicted MOS vs. MOS 3/5
Correlation coefficient
00,10,20,30,40,50,60,70,80,9
w1 w2 w3 w4
FN saliency FD saliency
Correlation coefficient
00,10,20,30,4
0,50,60,70,80,9
w1 w2 w3 w4
FN saliency FD saliency
RMSE
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
w1 w2 w3 w4
FN saliency FD saliency
RMSE
00,10,20,30,40,50,60,70,8
w1 w2 w3 w4
FN saliency FD saliency
No meaningful prediction improvement
→ No positive impact of the saliency information
WithoutSaliencyweighting
WithoutSaliencyweighting
Mean observer behaviorDistortion maps based on the SSIM
![Page 20: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/20.jpg)
20
Results : Predicted MOS vs. Observer Score 4/5
Particular observer behavior Comparison between observer score and MOSp (predict ed MOS) :
� Of simple quality metrics (with and without saliency weighting)– 2 distortion maps
� Absolute difference� Structural similarity (SSIM [Wang04] )
– 4 spatial pooling functions (wi)– 2 “particular observer” saliency maps
� 1 based on fixation number (FN)� 1 based on fixation duration (FD)
� On the whole database� In terms of :
– Correlation coefficient (CC)– Root mean square error (RMSE)
![Page 21: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/21.jpg)
21
Results : Predicted MOS vs. Observer Score 5/5
Particular observer behavior� Distortion maps based on the absolute differenceMean of the ∆CC are -0.08 and -0.07 for the FN and FD saliency maps respectivelyNo meaningful prediction improvement
� Distortion maps based on the SSIMMean of the ∆CC are -0.01 and -0.02 for the FN and FD saliency maps respectivelyNo meaningful prediction improvement
→ The general non-improvement of the prediction are not due to the average saliency map building
![Page 22: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/22.jpg)
22
Outline
1. Image quality assessment and visual attention2. The ground truth3. Saliency-based quality metrics4. Results5. Conclusion
1.
2.
3.
4.
5.
![Page 23: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/23.jpg)
23
Conclusion
→ Prediction improvement is not clearly established
Simple weight functions using the real saliency inf ormation, do not clearly improve the prediction capability of simple metrics
→ Prediction improvement on some particular cases is promising (but not acceptable)
→ The way to take into account the visual attention i n a quality metric cannot be limited to a simple spatial pooling
Future work :More studies are required to understand well the vi sual attention mechanisms in
an image quality assessment context,It seems that the saliency information and the dist ortion intensity have to be
jointly considered in the pooling function
![Page 24: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/24.jpg)
24
Thank you for your attention !
![Page 25: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/25.jpg)
25
Appendix
![Page 26: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/26.jpg)
26
References
[Wang04] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity”, IEEE Trans. on Image Processing, vol. 13, pp. 600–612, 2004.
[Osberger98] W. Osberger, N. Bergmann, and A. Maeder, “An automatic image quality assessment technique incorporating higherlevel perceptual factors”, IEEE ICIP, pp. 414–418, 1998.
[Barland06] R. Barland and A. Saadane, “Blind quality metric using a perceptual importance map for JPEG-2000 compressed images”, IEEE ICIP, pp. 2941–2944, 2006.
![Page 27: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/27.jpg)
27
Based on fixation duration (FD) : Based on fixation number (FN) :
Human saliency map computation
( ) ( ) ( )jj
N
jii
kFD yxdyyxxyxMS
FP
,,,1
)( ⋅−−∆=′ ∑=
( ) ( ) ( )yxgyxMSyxSM ,,, σ∗′=
( ) ( )∑=
′=′K
k
k yxMSK
yxMS1
)( ,1
,
( ) ( )∑=
−−∆=′FPN
jii
kFD yyxxyxMS
1
)( ,,
Saliency map (average) :
Density saliency map :
![Page 28: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/28.jpg)
28
� Examples
![Page 29: IEEE International Conference on Image Processing …people.irisa.fr/Olivier.Le_Meur/publi/Ninassi_ICIP2007... · IEEE International Conference on Image Processing ICIP 2007,](https://reader031.vdocuments.us/reader031/viewer/2022020100/5b85f9327f8b9a162d8bee52/html5/thumbnails/29.jpg)
29
Experiments : The ground truth