perceval - perceptual and cognitive quality evaluation techniques for audiovisual systems, ulrich...
DESCRIPTION
VERDIKT conference 2013TRANSCRIPT
![Page 1: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/1.jpg)
1
PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for
Audiovisual Systems
Ulrich [email protected]
![Page 2: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/2.jpg)
2
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
![Page 3: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/3.jpg)
3
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
![Page 4: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/4.jpg)
4
Multimedia systems rely on compression and encoding to speed up transmission
As a result, media quality goes down and the audio and video streams can become misaligned
Compression reduces the file size by removing redundant or unessential information
Ulrich Reiter – PERCEVAL
![Page 5: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/5.jpg)
5
Perceptual consequences of distortion:
- Face identification- Scene/object recognition- Speech comprehension- Audiovisual speech
integration- Temporal order of events
- Audiovisual temporal integration?
- Long duration quality perception?
Ulrich Reiter – PERCEVAL
![Page 6: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/6.jpg)
6
PERCEVAL project
Ulrich Reiter – PERCEVAL
![Page 7: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/7.jpg)
7
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
![Page 8: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/8.jpg)
8
Do quality expectations decrease over time and with increased involvement in the content?
How fast can users notice quality changes and at which quality level does this happen?
Is the quality level at which the change is noticed similar to the desired quality level?
Questions to be answered
Ulrich Reiter – PERCEVAL
![Page 9: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/9.jpg)
9
Quality assessment methods for long duration AV content
Ulrich Reiter – PERCEVAL
![Page 10: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/10.jpg)
10
SSCQE(Single Stimulus Continuous Quality Evaluation)
Ulrich Reiter – PERCEVAL
![Page 11: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/11.jpg)
11
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
![Page 12: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/12.jpg)
12
Method description
Quality adjustment
Ulrich Reiter – PERCEVAL
![Page 13: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/13.jpg)
13
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
![Page 14: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/14.jpg)
14
Experimental conditions
Ulrich Reiter – PERCEVAL
![Page 15: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/15.jpg)
15
Stimulus material
30min 55sec long
HD resolution (1080p)
H.264/AVC compression – Quantization parameter11 versions, JND
Conversion to 16 bit YUV 4:2:0
Original audio track
Ulrich Reiter – PERCEVAL
![Page 16: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/16.jpg)
16
Participants22 subjects (8 female, 14 male) in the age of 24-61 years
Test procedure and duration
Experimental procedure
Ulrich Reiter – PERCEVAL
Quality degradation initiated every 3 minsOne JND drop every 10 sec
![Page 17: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/17.jpg)
17
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
![Page 18: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/18.jpg)
18
Do the quality expectations decrease over time and with increased involvement in the content?
How fast can participants notice quality changes and at which quality level does this happen?
Is the quality level at which the change is noticed similar to the quality level set by the participant?
Questions to be answered
Ulrich Reiter – PERCEVAL
![Page 19: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/19.jpg)
19
Three subsets of data (20x10 each) created:
1) average quality level of the last minute of each 3min time slot
2) response time to the automatic quality degradationright after the start of each 3 min time slot
3) quality level at the time when a user reacted to quality change
Data preparation
Ulrich Reiter – PERCEVAL
![Page 20: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/20.jpg)
20
Results for subset data 1)
Main effects plot for quality levels averaged over last minute of each 3 min time section.
Ulrich Reiter – PERCEVAL
![Page 21: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/21.jpg)
21
Main effects plot for reaction time to quality changes.
Results for subset data 2)
Ulrich Reiter – PERCEVAL
![Page 22: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/22.jpg)
22
Main effects plot for quality levels at time when a userreacted to quality change
Results for subset data 3)
Ulrich Reiter – PERCEVAL
![Page 23: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/23.jpg)
23
Results cont.
Comparison of sensitivity to quality changes underdifferent conditions.
Ulrich Reiter – PERCEVAL
QLRT: Quality Level at Reaction Time
AQL: Absolute Quality Level
![Page 24: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/24.jpg)
24
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
![Page 25: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/25.jpg)
25
Conclusions
Time dimension does not influence participants’ expectations with respect to the perceived quality. The reaction time to quality changes is relatively constantacross all time intervals.Participants are less sensitive to quality changes when the process is controlled externally than when they are in charge of the quality adjustment.
Ulrich Reiter – PERCEVAL
![Page 26: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/26.jpg)
26
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
![Page 27: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/27.jpg)
27
Food for thought• Content dependency
– Ideal bandwidth trade-off for audio vs. video quality depends on content type
• Impact of emotional state, fatigue– On perceived quality, on quality requirements / expectations– Use of EEG and body sensors
• Data analysis of non-traditional data– E.g. quantitative and qualitative, combined– Multivariate statistical methods
Ulrich Reiter – PERCEVAL
![Page 29: PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for Audiovisual Systems, Ulrich Reiter, NTNU](https://reader035.vdocuments.us/reader035/viewer/2022081602/548e32f9b479599d2d8b46fa/html5/thumbnails/29.jpg)
29
References
[1] http://perceval.no[2] ITU-R Recommendation BT.500-7: Methodology for the Subjective Assessment of the Quality of Television Pictures. International Telecommunication Union, Geneva (1996)[3] N. Staelens, S. Moens, W. Van den Broeck, I. Marien, B. Vermuelen, P. Lambert, R. Van de Walle and P. Demeester, “Assessing Quality of Experience of IPTV and Video on Demand Services in Real-Life Environments”, IEEE Transactions on Broadcasting, Vol.56, No.4, Dec. 2010.[4] Borowiak, A., Reiter, U., Svensson, U.P.: Quality Evaluation of Long Duration Audiovisual Content. In: The 9th Annual IEEE Consumer Communications and Networking Conference –Special Session on Quality of Experience (QoE) for Multimedia Communications, pp. 353--357, Las Vegas (2012)[5] Yang, X., Tan, Y., Ling, N.: Rate control for H.264 with two-step quantization parameter determination but single-pass encoding. In: EURASIP Journal on Applied Signal Processing, pp. 1--13, (2006)[6] Wang, H., Qian, X., Liu, G.: Inter Mode Decision Based on Just Noticeable Difference Profile. In: Proceedings of 2010 IEEE 17th International Conference on Image Processing, Hong Kong (2010)Thanks to Adam Borowiak and Ragnhild Eg for providing figures and slides!
Ulrich Reiter – PERCEVAL