event report

ETSI STQ Workshop17-19 June 2008 – Prague, Czech RepublicWorkshop Report

ETSI STQ Workshop17-19 June 2008 – Prague, Czech Republic

Workshop Report

1


Overview........................................................................................................................3

WELCOME AND INTRODUCTION..........................................................................4

Session 1: KEYNOTE SPEAKERS..............................................................................6

Session 2: VIDEO QUALITY RELATED TO TRANSMISSION AND OTHER PARAMETERS.............................................................................................................7

Session 3: MULTIMEDIA / VIDEO QUALITY ASSESSMENT METHODS & METRICS (IPTV).........................................................................................................8

Session 4: KPIs............................................................................................................10

Session 5: SPEECH QUALITY..................................................................................11

Session 6: SPEECH QUALITY – continue...............................................................12

ROUND TABLE: QUALITY OF EXPERIENCE FOR E2E MONITORING........14

Session 7: MULTIMEDIA QUALITY - CONVERSATIONAL SERVICES AND IPTV.............................................................................................................................16

Session 8: NETWORK PERFORMANCE AND QoS FOR MULTIMEDIA SERVICES..................................................................................................................19

CONCLUSIONS AND RECOMMENDATIONS...................................................20

2


OverviewThe ETSI STQ Workshop, organised by ETSI and hosted by Mesaqin, took place on 17-19 June 2008 in Prague, Czech Republic, at the premises of the FEE CTU (Faculty of Electrical Engineering). It counted around 70 participants with a vested interest in Multimedia QoS.

The agenda included eight sessions and a round table, with presentations given by experts representing several organisations from around the world.

The workshop provided interesting information on all topics covered, an excellent opportunity for professional networking, co-operation opportunities, and directions, recommendations and potential inputs for future standardisation work.

3


WELCOME AND INTRODUCTIONModerator: Jean-Yves Monfort, ETSI STQ Chairman

Welcome - Prof. Skvor on behalf of Boris Simak, Dean of FEE CTU

On behalf of the Dean of FEE CTU Boris Simak, Prof. Skvor welcomed this ETSI Workshop at the University of Electrotechnical Engineering in Prague, thus opening the Workshop.

Objectives of the workshop - Jean-Yves Monfort, ETSI STQ Chairman

The Chairman of the ETSI Technical Committee (TC) STQ, Jean-Yves Monfort, outlined the objectives of the Workshop. This Workshop offers the opportunity to obtain guidance about future standardisation work in the area of Multimedia QoS. In more detail, the objectives are to provide latest results of QoS modelling experiments, analyse implications for the delivery of new services on NGNs, discuss the effect of transmission performance of access and core networks on the quality of multimedia services as perceived by the user, and to exchange information and foster professional networking for future standardisation work.

Practical arrangements - Jan Holub, Mesaqin

Jan Holub welcomed the participants to this workshop, in Prague, in the oldest technical university in Europe! He provided practical arrangements for the work during the workshop and about social events which will offer opportunity for professional networking.

ETSI introduction - Carmine Rizzo, ETSI STQ Technical Officer

Carmine Rizzo provided a brief overview with regards to what ETSI is. ETSI is a European Standards Organization setting globally-applicable standards for Telecommunications and other Electronic Communications networks and services. ETSI is an independent, not-for-profit, organisation created in 1988. Among various globally recognised achievements, ETSI created the GSM standard. ETSI is an ISO 9001:2000 certified organisation offering direct participation to the Members. At present ETSI counts over 700 Members in 62 countries across 5 continents.ETSI has produced more than 20 000 publications, all freely available!Carmine Rizzo briefly explained the three roles of ETSI, as a European Standard Organisation (ESO), Global Standards Producer (GSP) and Service Providing Organisation (SPO). He discussed in some detail the working process for standardisation tasks, from the initial adoption of new work, which needs to be supported by at least four ETSI Members, to the final publication of standards, for which Technical Committes normally take decisions by consensus.Carmine Rizzo briefly outlined the ETSI structure, the highest decisional authority being the General Assembly, formed by all the ETSI Members, which delegate powers to the ETSI Board, consisting of 24 ETSI Members.

4


He gave an overview of ETSI placed in the standardisation world, showing the high numbers of partnerships, co-operation agreements, partnership projects (3GPP, MESA), and the ETSI participation in the Global Standards Collaboration (GSC) with all other most important standards organisations from around the world.Carmine Rizzo discussed what it means to be an ETSI Member for an organisation, the benefits of participating directly in the ETSI standardisation processes, the resulting competitive advantage, the professional networking opportunities, the credibility on the market, and ultimately the possibility to participate directly in the production of standards for the benefit of the Members’ business. He showed the various classes of contribution, and explained that ETSI has made strong financial efforts in order to reduce, or keep as low as possible, the fees for SMEs, Micro-Enterprises and Universities. Various contacts for further information were provided.

STQ introduction - Jean-Yves Monfort, ETSI STQ Chairman

Jean-Yves Monfort provided a brief introduction about the ETSI TC STQ. He presented the structure and the officials, including Vice Chairs, Technical Officer, STQ support, rapporteurs responsible for the development of standards documents, the STQ Mobile Working Group and related rapporteurs.Jean-Yves Monfort highlighted the main ETSI TC STQ results obtained until now, related to VoIP (including wideband speech terminals), user related QoS parameter definitions and measurements, specification and measurement of speech transmission quality, and the main results of the STQ Mobile Working Group related to QoS aspects for popular services in GSM and 3G networks.Jean-Yves Monfort pointed out that the standards are regularly reviewed and updated, and new ones are created according to the directions of the ETSI Membership.Ongoing work covers three documents on QoS and network performance metrics and related measurement methods, and a forth one is planned on Multimedia for the near future. He also highlighted that ETSI provides service for interoperability tests (Plugtests). Finally he presented the work programme dedicated to NGN related QoS for speech and multimedia services. A roadmap of ongoing works is available in the presentation given by Trond Ulseth in session 7.

5


Session 1: KEYNOTE SPEAKERSModerator: Jean-Yves Monfort, ETSI STQ Chairman

Progress and future plans for VQEG - Arthur Webster, US Department of Commerce, NTIA, Institute for Telecommunication Sciences

Arthur Webster, VQEG co-Chair, provided historical background information about VQEG (Video Quality Expert’s Group), from the first meeting in Turin, Italy, in 1997 to the last one in Kyoto, Japan, in 2008.VQEG does not produce standards; it plans and performs tests, and provides results to standards organisations. VQEG is an unofficial, technical group, open to all, with links to several standards organisations (ITU-T, ATIS, IEC, ETSI STQ). Arthur Webster showed the VQEG structure and outlined the most relevant results, related to the phases of Full Reference TV (FRTV) lab tests and several other projects. In particular the first Recommendation for video Quality assessment is J.144 and two Recommendations planned for approval on video only quality :J.mmvofr (full reference), including four models and J.mmredref (reduced reference) including one modelFinally, he gave a summary of VQEG, which combines expertise of academia, industry, government labs, SDOs, and conducts tests and reports results to SDOs.Arthur Webster confirmed that at the beginning of VQEG work, there was mostly an interest in objective testing methods, and certainly more could have been done, and could be done in the future, on subjective testing methods. He stressed that work is made on a voluntary basis. Following other comments from participants, Arthur Webster provided his views about the current and possible future evolution of the work within VQEG related to subjective/objective testing.

Perceived QoS and Test suites - Kjell Brunnström, ACREO AB Sweden

Kjell Brunnström gave a presentation on perceived QoS and test suits. He discussed the EU project Multi Service Access Everywhere related to Broadband for all, a future low cost multi-service access network, and showed the test suite. He gave details of what and how to test for streaming video (audio/video quality, synchronisation, zapping quality) and for videoconference, the perceived QoS for VoIP (and also web browsing, interactive gaming), and conversational quality.Finally, Kjell Brunnström summarized his presentation by stating that a huge effort has been made to put together state-of-art testing methodology in one document, and that the MUSE test suite gives guidelines for what to test and how to test. He stressed that there is a lack of standardised test suite for video (they exist for audio) and the perceived QoS testing part have been trying to fill this gap.Following a discussion with a participant, he provided clarifications about the test methods, and informed that the documents are freely available on the internet.

6


Session 2: VIDEO QUALITY RELATED TO TRANSMISSION AND OTHER PARAMETERSModerator: Arthur Webster, US Department of Commerce, NTIA, Institute for Telecommunication Sciences

Looking at the relationship between video bitrates & end-user quality: subjective tests approach - Ricardo Pastrana-Vidal, France Telecom, Orange Labs

Ricardo Pastrana provided an explanation of the problem statement related to degradation of signal for his specific test approach, and how to measure the signal quality. He described the subjective tests principles and methods. He outlined the SAMVIQ (Subjective Assessment Methodology for Video Quality) method, a repeatable method standardised at EBU and ITU-R.Ricardo Pastrana showed the results of tests related to bit rate vs end-user quality assessment (multimedia, TV). The conclusions of the tests are that bit rate vs. perceived video quality are not directly related, whereas they are content dependent. Besides, he pointed out the limits or tradeoffs of bit rate vs. quality. Finally, he explained that the same results are obtained independently of codec solutions & image formats. Future work might be done to study the variations of effective bit rate.

QoS assessment method for video quality and pricing in IPTV Services - Takahori Hayashi & Akira Takahashi, NTT

Takahori Hayashi outlined the motivations and goals of the study whose results are shown in his presentation. As IPTV services are becoming some of the most promising applications over the Next-Generation Networks (NGNs), the main motivation is to provide users with sufficient quality, while IPTV service providers must properly evaluate, design, and manage users' quality of experience (QoE). This requires a methodology for assessing overall QoE, defined as “service value”, which is a function of the quality of the provided service and its cost. Establishing a service value evaluation method is the goal of this study. The conclusions are that a suitable assessment method for both video quality and pricing was determined, and it was found that the willingness-to-pay a certain price decreases exponentially with decreasing video quality under the condition of the same QoE. Besides, a service value evaluation model was established, as a linear function of video quality and pricing on a psychological interval scale, and the effects of other QoE factors on service value is currently evaluated for further study.

Conclusion for day 1

This presentation concluded the first day of Workshop. The Chairman Jean-Yves Monfort pointed out that ideas brought today, e.g. MUSE, will be considered for future work within the ETSI TC STQ. Finally, all participants briefly introduced themselves with their names and the organisations they represent.

7


Session 3: MULTIMEDIA / VIDEO QUALITY ASSESSMENT METHODS & METRICS (IPTV) Moderator: David Hands, ETSI STQ, BT Group Plc

An E2E quality measurement framework - David Hands ETSI STQ, BT Group CTO, Perceptual Engineering Research Group

David Hands outlined his presentation, which provided an overview of Objective Quality Measurement Metrics and related applications, an explanation about why a Quality Measurement Framework (QMF) is necessary, a description of QMF, and the application of metrics within QMF.He pointed out that this work is in progress and needs to be further developed. The conclusions reached so far are that a QMF has been introduced and the application of different measurement and monitoring tools has been defined. The following recommendations have been formulated: to use of perceptual models for edge measurements, while perceptual models are not appropriate for in-service network monitoring. Finally David Hands stressed the importance of measurement and monitoring tools, which must be accurate as this is essential irrespective of the measurement approach.

No reference video quality assessment metrics for multimedia : State of the Art of signal-based approaches - Nicolas Ramin, Ricardo Pastrana-Vidal, France Telecom, Orange Labs

Nicolas Ramin started his presentation by explaining the problem statement: currently multimedia services over packet networks do not guarantee the integrity of the transmitted signal, which can affect the end-user perceived quality. As a result, quality needs to be assessed in order to work towards an improvement of multimedia communication chain, for which purpose subjective assessment methods can be used. However, subjective test implementations are expensive and this issue can make them unfeasible. Objective measures can be an alternative solution to perform real-time quality assessments.Nicolas Ramin gave an overview of the state of the art NRQM (No Reference Quality Metric) signal-based approaches. These are global multimedia NRQM, temporal NRQM and spatial NRQM (artefact-based and blur-based). He explained characteristics and limitations of each approach.As a conclusion with regards to state of signal-based NRQM research, it is considered that temporal NRQM are at a state of maturity, while there are open issues related to spatial NRQM and global multimedia NRQM, which need further investigation.

Video quality measurement for TV, multimedia and beyond – Marcus Barkowsky, Opticom

Marcus Barkowsky started his presentation by showing typical measurement scenarios, including calibration setup for development and validation, and measurement setup. He outlined the basic framework for Video Quality Measurements (VQMs) and showed an example of temporal degradation. Then he

8


described the enhanced VQMs, which encompass additional requirements for multimedia scenarios, and showed related examples. Marcus Barkowsky provided an overview of spatiotemporal modeling, and the work done for simulating the participant in a subjective test.Marcus Barkowsky described the performance results, and explained the conclusions, with regards to degradations, that need to be handled by Video Quality Models for Multimedia Scenarios (spatial and temporal degradations and spatiotemporal effects). He explained that existing models (e.g. PEVQ) focus on spatial distortion and basic modeling of temporal degradations, while the proposed TetraVQM framework models focus on the spatiotemporal effects (masking by temporal changes) and improve the modeling of the viewer. Future work will include the enhancement of PEVQ by using the TetraVQM framework.Following a question about motion estimation, Marcus Barkowsky explained that they are using a fast search algorithm which only searches for a few positions.

9


Session 4: KPIsModerator: Klemens Adler, STQ Mobile, Vodafone D2 GmbH

Monitoring the End User quality of experience using the derivative Mean Opinion Score (MOS) key performance indicator - Filippos Kyriazidis, Manager of Network Performance Monitoring, Cosmote Mobile Communications

Filippos Kyriazidis outlined the objectives of his presentation: to indicate the importance of MOS to the end user perceived speech quality, to introduce the MOS derivative Key Performance Indicator (KPI) which comes from Drive Tests Data and Network Statistics, to implement a mathematical formalism in order to systemize the MOS calculation procedure, and to compare different speech codecs implementations in BSC areas using MOS distribution patterns.A method of evaluating speech codecs’ effectiveness is to calculate the percentage of samples in each MOS class, and to achieve this aim drive tests data should be post-processed differently with respect to the original data. Filippos Kyriazidis explained that the example shown is very specific, and should indeed be compared to other examples of measurements in the network.

KPIs for mobile services – Multimedia Telephony Services over IMS - Klemens Adler, STQ Mobile, Vodafone D2 GmbH

Klemens Adler, Chaiman of the ETSI STQ Mobile working group, gave an overview of the work done until now and ongoing efforts within this working group. Then he outlined the content of his presentation, which started with an explanation of the MTSI Interoperability concept (Multimedia Telephony). He described the Key Performance Areas of MTSI: MTSI Control Plane, MTSI User Plane (realtime RTP/UDP) and MTSI User Plane (non-realtime RTP/UDP).Klemens Adler listed the Control Plane KPIs and provided two examples (MTSI Registration Time and MTSI Session Add Time).This was followed by a list of Realtime User Plane KPIs, and again two examples were provided (MTSI Speech Quality and MTSI Real-Time Text Delivery Time).Finally the list of non-realtime User Plane KPIs was shown, and one related example (MTSI Messaging Delivery Time).Klemens Adler summarised the main points of his presentation: IMS-based services promise an enhanced end-user experience and allow easy combination of different media types within one session; achieving good end-user quality is vital for success; and it is important to use relevant KPIs for the MTSI services. He pointed out that ETSI STQ Mobile is well prepared with ongoing standardization work for MTSI services, and that ITU-T is also currently standardizing new algorithms for objective speech and video quality assessment, suitable for MTSI services.

10


Session 5: SPEECH QUALITYModerator: Joachim Pomy, TIA, ETSI STQ Vice Chairman

Speech quality in modern network-terminal configurations - Hans W. Gierlich, HEAD acoustics GmbH

Hans Gierlich started his presentation by providing an overview of modern network terminal configurations (the typical elements) and their impact on speech quality.More specifically, he provided a number of examples, and related explanations, of the impact of modern network terminal configurations on several speech quality parameters: delay, coding, noise and noise cancellation, insufficient echo loss / improper echo cancellation, cascading of signal processing, and missing signalling information between endpoints and network elements.Hans Gierlich summarised his presentation by pointing out that before launching new network infrastructures, it is indispensable to test in labs the impact of new network elements on the performance of the network from the speech quality perspective. He explained that currently no signalling is provided between terminals and networks for information exchange about the signal processing active in the different devices. Finally he outlined related work (G.mdcspne started in SG16 and the G.115.x series).To answer a question about the acceptance by the users of poor performance, and what the operators plan to do about this, Hans Gierlich said that the users normally accept more easily worse quality in mobile communications, however it depends heavily on the needs and expectations of the specific user.

Impact of multimedia services interaction on speech quality in VoIP & VoWLAN - Peter Počta, University of Zilina

Peter Počta outlined his presentation, which started with a description of the influences on speech quality in VoIP (delay, jitter, packet loss) and VoWLAN (delay, jitter, packet loss, signal strength and noisy environment). He showed a couple of experimental scenarios and provided a description of speech samples used.He listed the current multimedia services and described the related multimedia services modeling. The measurements results were shown.Peter Počta explained some possibilities for the purpose of more accurate and reliable objective speech quality assessment in terms of the reference signal which could be used, and he showed the results of Reference-Signal-Length experiments and Active-Speech-Ratio experiments.His conclusion were that the quality declines faster in the IEEE 802.11b scenario in comparison with the IEEE 802.3i scenario in the case of different service settings (higher bit rates), and that a more complex MAC (Medium Access Control) scheme for better supporting of Multimedia services should be implemented to Access points (e.g. HCCA). Future work will cover the impact of user’s mobility on speech quality in VoWLAN in the case of multimedia services interaction. Other conclusions related to improving the PESQ estimation were that longer durations of the reference signal imply lower standard deviations of dropped packets and of MOS-LQO’s scores, and an increase in amount of speech in the reference signal implies increase of the reference signal sensitivity to packet loss change. Further investigations will be made.

11


Session 6: SPEECH QUALITY – continueModerator: Hans W. Gierlich, ETSI STQ Vice Chairman, HEAD acoustics GmbH

Methods for multimedia service adaptation in Next Generation Networks - Blazej Lewcio, DT Labs

Blazej Lexcio explained that, in Next Generation Networks (NGNs), vertical handovers between different network technologies provide seamless roaming during active multimedia sessions. Both transmission quality and user experience quality depend on the network handover policies, link layer characteristics, and codecs used. He pointed out that a thorough analysis of user perception of this phenomenon has not been done yet. When this will be done, it would be possible to successfully design and further improve mobility management solutions for always-on multimedia services.Blazej Lexcio presented an overview of the current approaches for voice quality assessment, and then he described an NGN “mobilsense” testbed and a project methodology applied to map user experience to network conditions, focusing on phenomena caused by a user roaming across diverse wireless technologies. Results were shown of experiments carried out to study user perception of VoIP quality in NGNs, which highlighted that the speech quality is a function of audio bandwidth and network characteristics, as well as of the temporal handover position within a call.The tests results help specifying the situations in which a high-quality network handover should be scheduled, which is essential for the development of always-on multimedia services that provide optimum user satisfaction.Future work will include: investigate the correlation of network traces analysis and results from subjective tests, perform a thorough evaluation of the observed phenomena, prepare the testbed for online experiments, extend experiments to include conversational subjective tests, and extend the research to include mobile TV and video conferencing in NGNs.

Protecting voice calls from the background traffic in a mixed voice/data transmission over wireless networks - Miroslaw Narbutt, Communication Network Research Institute

Miroslaw Narbutt presented an experimental study about how real-time voice transmission can be supported by tuning MAC layer parameters in wireless networks. In particular it was investigated how the tuning of these parameters affects both the voice transmission quality and background data throughput.In his experiments, Miroslaw Narbutt used a prediction of end-to-end voice transmission quality from time varying transmission impairments using the latest Appendix to the ITU-T E-model. The experimental results show that the tuning of the AIFSN and CWmin parameters can be used to successfully prioritize voice transmission over data in real 802.11e wireless networks. The experimental results also demonstrate that the AIFSN parameter more effectively protects voice calls against background data traffic, compared to CWmin.Finally, he explained that it is possible to protect voice calls from heavy background in a wireless environment via voice prioritization at L2/MAC (IEEE802.11e standard), that proper tuning of L2/MAC parameters is essential, that the side effect of

12


voice prioritization at L2/MAC is a substantial reduction of the background traffic, and that the real-time voice transmission quality prediction is desired for proper tuning of L2/MAC parameters.

Speaker identification in low-rate coded speech - Andrew Catellier, Institute for Telecommunications Sciences, Boulder

Andrew Catellier started his presentation by explaining the motivations for performing studies about Speaker Identification (SID). Experiments were performed to help improve the platform. He gave an overview of the experiments design, the specifications the realization, and the related issues.Andrew Catellier said that 3 male and 3 female speakers, and 15 male and 10 female listeners, of various ages and professional occupations, were involved in the experiments of SID.The experimental results highlighted that male-female confusion is very low, that two of the three males were most often confused, and that two of the three females were easily recognized. With regards to per-length results, an interesting outcome was that no length is significantly easier than others. The results are consistent with prior research, however they are not intuitive. The results about SID vs. Intelligibility and Stress Detection showed that SID is not as robust as Dramatized Urgency (DU) detection, and that it is about 3 times more robust than intelligibility.Andrew Catellier concluded his presentation by listing future possible topics of investigation and by explaining open questions such as the intention to consult with experts in psychology and neurology to design lab tests that more closely model real world situations, and to attempt an experiment with better controlled recordings and familiar speakers.

13


ROUND TABLE: QUALITY OF EXPERIENCE FOR E2E MONITORING

The ETSI TC STQ Chairman Jean-Yves Monfort encouraged all participants to have an open discussion on the following items.

From Operators and FAI: What are the expected tools needed to monitor the E2E Quality, and for

Quality Assurance? What are the drawbacks of the existing methods? What are the services for which there is a need for new tools for Quality

monitoring? Can a FAI/Operator make money without quality monitoring?

What is the role of regulators to ensure that the minimum (normal) quality is provided?

From quality measurement tools manufacturers, what are the needed standards? Or the needed improvements?

Subjective methodology: What are the solutions to develop new methodologies? Could we define objective tools without comparing the results with user’s

subjective quality?

More generally, what are the expected standards related to E2E quality monitoring?

During the discussions several participants pointed that there was a need for exchanges on quality information among operators and standardization bodies.Trond Ulseth pointed out that, as network operator, it is not easy to guarantee how users perceive the quality; besides, it is not easy to know how many users use the bandwidth, which affects the quality; at present this is not an easy issue to solve.In particular the quality also depends on the type of content.Akira Takahashi also indicated there is a need of means to exchange between the different players (eg for IPTV).

Hans Gierlich pointed out that STQ has good tools for quality monitoring.David Hands pointed out that from the video perspective, an issue regarding to measuring E2E is need to work together proactively, e.g. to work towards a single measuring unit.Arther Webster said that VQEG is working on this, however there are organisations which do not want to work towards appropriate quality monitoring.

Hans Gierlich made the point that even with the best possible core and access network, if the end terminals do not comply with high quality standards, then the overall E2E quality is degraded.

14


Markus Vaalgamaa stressed that the open market encourages a wider range of offers, and a result the user perception range is also wider, which poses challenges both from the commercial point of view and from the standardisation perspective.It was also reminded that PC have never been designed as "terminals".The quality is sometimes very poor (SNR). User acceptation is strongly depending on the application. For business a strong need for high quality.

Jean-Yves Monfort pointed that some operators are involved in the definition of standards for high quality for wideband speechPrivate networks and terminal are outside the control of operators; so, operators may control the quality in the part on which they have control. Services are distributed over IP packets creating new challenges (it was different with the switched networks). Tools exist in packet networks.There is a need to standardize more in the "IP world".

It has also been noted that highly secured services & networks may create impairments. It is an area where there are very few ongoing activity.

The regulator has not the role to guarantee a minimum quality. What is needed from the regulator point of view is the consumer protection and addressing competition market (competitors, alternative operators), should provide services. It was finally pointed that it is difficult for a regulator to construct remedies while the network operators cannot provide reliable and standardised E2E quality measurements method. The regulators need the operators and standardisation bodies to work proactively together in order for the regulators to build an adequate regulatory framework.Arthur Webster indicated that US regulator tries to be outside the competition, but the transition concerning TV; there will be forced by the public to have accurate tests and results on the quality.

15


Session 7: MULTIMEDIA QUALITY - CONVERSATIONAL SERVICES AND IPTVModerator: Trond Ulseth, ETSI STQ, Telenor ASA

Audiovisual QoS for communication over IP Networks - Trond Ulseth, ETSI STQ, Telenor ASA

Trond Ulseth started his presentation by providing an overview of the path towards the Next Generation Network (NGN) and the related technological challenges. He described the work within ETSI on IP QoS: TC TISPAN is responsible for standardising network QoS mechanisms, while TC STQ is responsible for standardising end to end performance characteristics. He pointed out that the network mechanisms need to be aligned with the performance characteristics, and QoS mechanisms that meet the needs of multimedia communication might be a challenge.Trond Ulseth showed the roadmap of the upcoming ETSI TC STQ on IP performance. He explained the IP network characteristics determining perceived media quality, the related IP network performance and the access technologies. Then he gave a brief overview of the audiovisual media and applications. He discussed in more detail speech (telephony), audio and video aspects, and experimental results related to perceived audio quality and video quality. He pointed out that the results shown are partial as more information is needed.Trond Ulseth concluded his presentation by summarising that as the telecommunications industry is moving towards IP-based communication, the IP technology introduces new opportunities and new challenges to operators and application providers. At present, the effects of QoS mechanisms on network performance are not fully understood, and more knowledge about effects of IP network bandwidth and network degradations on perceived media quality is required. At the same time, the quality levels preferred by the user need to be identified for different media and usage scenarios. Finally, Trond Ulseth pointed out that it is hardly possible to develop a single set of recommendations covering all applications and presentation formats.Following a discussion with the participants, Trond Ulseth expressed his view that it is not possible to establish a clear relationship between packet loss and QoS.

Mutlimedia quality in a conversational video-conferencing environment - Quan Huynh-Thu, Psytechnics Ltd

Quan Huynh-Thu gave at first a brief introduction to video-conferencing and related technological challenges (delay, media synchronisation, coding artefacts, transmission errors and network congestion), and user expectations, which vary from the public to the enterprises.Quan Huynh-Thu described the main aspects of the future High Definition (HD) video-conferencing: real-time audio-video communication, enhanced features for tele-collaboration, and face-to-face feeling (tele-presence). He explained his HD video-conferencing subjective experiment (plan, facilities, setup). The trials were task-driven 2-min conversations: No. 1 task (named “shape matching”) required focus on visual terminal, and No.2 task (named “who’s in the bag?”) did not especially require

16


the use of visual information. Trials were carried out for various bandwidth/network conditions, and 20 subjects of age between 18 and 72 participated.Quan Huynh-Thu showed samples of various tests and summarised the results as follows: most participants unexpectedly provided high quality ratings and found the quality acceptable even if video was severely degraded (user expectation); the system in test produced similar video quality at both 3Mbps and 1.4Mbps (without packet loss); packet loss ratio was found to be the most important factor influencing multimedia quality amongst the 3 variables considered; statistical analysis showed an interaction effect between the visual impact of packet loss and task on multimedia quality.He finally pointed out that naïve participants (public) might have lower quality expectation than target users (experienced with video / business users). He also stressed that conversational tests represent heavy investment for relatively small amounts of data.

Applications of objectives quality assessment models in IPTV - Akira Takahashi, NTT Services Integration Labs

Akira Takahashi gave an introduction of the concept of Quality of Experience (QoE), defined as “the overall acceptability of an application or service, as perceived subjectively by the end user”. He provided an overview of the factors affecting QoE of IPTV services, and explained the relationship between QoS and QoE.Akira Takahashi stressed that there surely is a market need for QoE evaluation methods/tools, as recent surveys indicate. He showed scenarios of QoE evaluation (what to measure, when, where and how). Then he provided an overview of the classification and application of objective QoE models (media-layer, parametric packet-layer, bitstream, parametric planning and hybrid models).Akira Takahashi’s concluding remarks were the following. With regards to the market trend: QoE evaluation technologies are needed in IPTV market, and testing equipment market will shift from R&D usage to maintenance and monitoring applications. With regards to application scenarios: there are various applications of objective quality models in different quality design/management phases; standardization activities are quite active in ITU-T SG9, SG12, and ITU Video Quality Experts Group (VQEG).Future issues include real-time quality estimation for monitoring, and light-weight but accurate models for end user monitoring.

SUIT: Scalable Ultra-fast and interoperable interactive television - Yves Dhondt, IBBT, Ghent University, Multimedia Lab

Yves Dhondt started his presentation by explaining that SUIT is a IST European FP6 STREP project that ran from February 2006 till April 2008, with a consortium of 11 partners (4 industrial and 11 from academy/research labs). He outlined the SUIT objectives, including the introduction and promotion of scalable video for broadcasting services enabling robust communication for high speed mobile terminals (up to 150 km/h) and providing an end-to-end chain from provider to consumer as well as interactivity.Yves Dhondt described the SUIT architecture and profiles (building, portable, rural and vehicular profile), and he explained in some detail the work undertaken in terms

17


of SUIT and video quality. The conclusions of this work were summarised as follows: within the SUIT project, overall video quality is optimized in four different ways: multiple description coding, joint source channel coding, unequal power allocation, and smart rate control. He finally pointed out that the use of multiple description coding over different networks outperforms the other methods.

18


Session 8: NETWORK PERFORMANCE AND QoS FOR MULTIMEDIA SERVICESModerator: Elisa Boschi, ETSI STQ, Hitachi Europe / Fraunhofer FOKUS

Network performance metrics and measurement methods in IP networks - Salvatore D'Antonio, CINI

Salvatore D’Antonio outlined the motivations for the work he presented. He explained that network performance measurements base their interoperability on common metric definitions, and informed that two different standardisation bodies have addressed this issue (IETF IPPM working group and ITU-T Study Group 12).The work Salvatore D’Antonio presented had the following purposes: to discuss and compare definitions of metrics used to specify and assess layer-3 level performance in IP networks, to define/discuss measurement methods for selected performance metrics in IP networks, and to make a survey on the existing network performance-related IETF standards.Salvatore D’Antonio compared the work of the above mentioned groups in terms of emphasis and terminology, he provided several performance metrics definitions and related comparisons (one way delay vs, IP packet transfer delay, delay variation, one way packet loss vs. IP packet loss ratio), and he listed other metric.Salvatore D’Antonio described the measurement methods used (active vs/ passive), and suggested to refer to recommendations and best practices given by IETF / ITU-T, whose work is ongoing. He concluded by informing the participants that more information and details are available in DEG-STQ-000104-3.

Enabling carrier evolution for triple play services - Roman Osredkar, Iskratel

Roman Osredkar started with an overview of the telecommunication services in Slovenia, including number of operators, services offered and broadband penetration. He provided a brief summary of the evolution of QoS in access and core networks, and explained the QoS requirements for voice, video and data, and the quality of experience.Roman Osredkar explained the discovered triple play issues (related to delays, bandwidth for video, CPE/STB, and the new services) and related solutions. Finally, he provided the following conclusions: the implementation of triple play solutions starts with correct network design, the integration of equipment from chosen equipment providers is the next step to the working solution, and the triple play issues can be solved only with cooperation of equipment vendors and service providers.

19


CONCLUSIONS AND RECOMMENDATIONSSession moderators and Chair

The ETSI TC STQ Chairman Jean-Yves Monfort invited the moderators and all participants to have a final discussion on the topics of this Workshop, aiming at shaping the future standardisation activities within the ETSI TC STQ.The summaries of presentations and recommendations/conclusions are available in the slideware "STQ-workshop-conclusions".

World Class Standards

ETSI WORKSHOP

"EFFECTS OF TRANSMISSIONPERFORMANCE

ON MULTIMEDIA QoS"

Summary/Conclusions

Only some of the conclusions and recommendations are highlighted in this section, in complement to the summary of the round table :- It was noted that there is a need for more activity on subjective methodology and VQEG is the right platform. These new subjective methods are needed when developing objective model evaluation. So any laboratory is invited to collaborate on this topic.- It was stressed that there is a lack of standardised test suite for video (they exist for audio) and the perceived QoS testing part have been trying to fill this gap.- Subjective assessed quality is related to bit rate, but only within a particular scene. Different scenes have different quality scores for the same bitrate. In each case the relation was content dependent. Even when the source scenes receive the same subjective quality assessment, nevertheless the scenes have different scores for the same bitrate after compression is applied.- Based on the tests on video quality in IPTV services, a service value evaluation model is proposed offering a linear function of quality and pricing. Service value can be accurately evaluated from video coding rate and pricing by using the proposed model.- Tandeming of signal processing in network elements and terminals becomes an increasing problem: Information exchange logic & architecture concerning signal processing elements and their settings needed at (inter)-connection points- It was pointed out that private users accept lower quality than enterprises, and it is to be understood how this will impact standardisation efforts; packet loss is the most important factor influencing multimedia quality. Finally, more information is needed about the performance of IP fixed/wireless networks in order to obtain a better understanding of the evaluation of QoS parameters. He suggested that a database could be built with performance information.

20


Jean-Yves Monfort thanked the FEE CTU, the host Mesaqin and his representative Jan Holub, the ETSI STQ support staff (Nathalie Guinet, Jane Lloyd, Marlene Forina), the STQ Steering Committee, all moderators and speakers, and the ETSI Technical Officer Carmine Rizzo.

Finally, Jan Holub from Mesaqin thanked all participants in this ETSI STQ Workshop.

21

event report

Documents

etsi workshop

etsi members

etsi structure

roles of etsi

present etsi

etsi stq workshop17

etsi stq chairmanwelcome

workshop jean