characterisation of processing artefacts in high dynamic ... · pdf fileolie baumann , alex...

International Broadcasting Convention 2017 (IBC2017)

14-18 September 2017

Characterisation of processing artefacts inhigh dynamic range, wide colour gamut

ISSN 2515-236Xdoi: 10.1049/oap-ibc.2017.0316

www.ietdl.org

video

Olie Baumann ✉, Alex Okell, Jacob StrömEricsson, Southampton, United Kingdom✉ E-mail: [email protected]

Abstract: A new, more immersive, television experience is here. With higher resolution, wider colour gamut andextended dynamic range, the new ultra high definition (UHD) TV standards define a container which allowscontent creators to offer the consumer a much more immersive visual experience. Although very little contentexploiting the full range of the container is yet available, some artefacts associated with the compression of highdynamic range content have already been identified and reported in the literature. Specifically, the chromasubsampling process has been shown to cause disturbing artefacts for image regions of certain colour andluminance. This study quantifies the distortion and identifies regions of the extended colour volume whereartefacts associated with standard image processing techniques are more likely to occur. In doing so, it highlightsthat the problems will become greater as more content exploiting the full UHD container becomes available,requiring additional care and processing in content production and delivery. Finally, the study references ways ofovercoming these issues.

1 Introduction

The DVB UHD1 Phase 2 [1]) and ATSC 3.0 [2] standards arebecoming well established and allow significant benefits overhigh definition television standards beyond simply more pixels.The technological enhancements of high dynamic range (HDR),wide colour gamut (WCG) and high frame rate all contribute tomore life-like images and hence a more immersive viewingexperience.

ITU-R Recommendation BT.2100 (Rec. 2100) [3] definesparameters and formats for HDR television, using a minimumY ′CbCr component bit-depth of 10-bits, colour primaries fromITU-R Recommendation BT.2020 (Rec. 2020), and two alternativeelectro-optical transfer functions (EOTFs): perceptual quantiser(PQ, standardised as SMPTE ST 2084) and hybrid log-gamma(HLG, standardised as ARIB STD-B67).

This paper examines a system using the PQ transfer function,but many of the principles will also apply to HLG. The PQtransfer function is specifically designed to exploit the relativeinsensitivity of the human visual system to absolute differencesin luminance when the luminance is high. It achieves this byallocating more code words to lower values of the red, greenand blue component signals, than to higher values [4], and assuch is much more non-linear than traditional SDR transferfunctions.

A typical signal processing chain to convert linear light RGBcamera data to Rec. 2100 4:2:0 Y ′CbCr format using the PQtransfer function is shown in Fig. 1. Conversion from Y ′CbCr4:2:0 format to linear light RGB for display is the reverseprocessing chain, with chroma upsampling, and EOTFs applied toR′, G′ and B′.

The non-linearity associated with the PQ transfer function canresult in significant chroma leakage for some colours, meaningthat the luma component Y ′, does not completely represent thedesired luminance output from the display, but part of that signalis instead carried in the Cb and Cr components. This in turnmeans that errors introduced on the chroma components by, forexample, chroma subsampling, can result in visible distortion in

IBC, Open Access Proc. J., 2017, pp. 1–5This is an open access article published by the IET under the CreativeAttribution License (http://creativecommons.org/licenses/by/3.0/)

the output luminance [5]. It is the study of this distortion, where itoccurs, and how it can be avoided, which forms the basis of thispaper.

1.1 Note on colour spaces

The Rec. 2100 and ITU-R Recommendation BT.709 (Rec. 709)colour spaces are shown in the CIE 1931 xy coordinates alongwith the familiar horseshoe of the visible spectrum in Fig. 2. Itis worth noting that the chromaticity points, and therefore thecolour space of Rec. 2100 are identical to those of Rec. 2020[3]. Clearly, Rec. 2100 allows a far greater proportion of visiblecolour s to be represented than Rec. 709. In this paper, we useFig. 2 – plots of the extent of the Rec. 2100 and Rec. 709colour spaces in CIE xyY colour space for analysing theproperties of visible light. The x and y components define thehue and purity, and the y component the luminance incandelas-per-meter-squared (cd/m2) [6].

2 Chroma subsampling and luminance errors

It is well understood that the human visual system has a lower acuityto chrominance than to luminance and as such the Cb and Crcomponents of the Y ′CbCr container are often subsampled eithervertically as in 4:2:2, or both horizontally and vertically as in4:2:0 [6]. To avoid aliasing in the reconstructed chromacomponents, the full-resolution chroma components are low-passfiltered prior to decimation.

This filtering process is a weighted average of neighbouringsamples. Let us take an example, previously reported in [5], oftwo neighbouring pixels in linear light RGB. Both pixels are verysimilar colour and luminance

RGB1 = 1000, 0, 100( )

RGB2 = 1000, 4, 100( )

1Commons

Fig. 1 Typical processing chain to convert linear light RGB camera data to Rec. 2100 Y ′CbCr 4:2:0 format

Fig. 2 Plots of the extent of the Rec. 2100 and Rec. 709 colour spaces inCIE xy

Fig. 3 Squares of plain colour scaled from the RGB values of a workedchroma resampling example

The two colours are shown in the top two squares of Fig. 3. It shouldbe noted that all four squares in the figure have been identicallyscaled in order to see them in a non-HDR workflow.

2 This is an open

Converting these neighbouring pixels to Y ′CbCr and back to RGBas per the workflow presented in Fig. 1 results in the following RGBvalues

RGB1 = 484, 0.03, 45( )RGB2 = 2061, 2.2, 216( )

Clearly these are quite different, not only from one another but eachis also different from the original colour. This can be seen from thelower two squares of Fig. 3. We can look at the difference inluminance in the CIE colour space by converting to xyY

xyY1 = 0.6405183, 0.26326786, 129.9302( )xyY2 = 0.63216754, 0.26031566, 555.785( )

Whilst the hue and purity have remained close to one another, theluminance is significantly different. In real images, this effectmanifests itself as adding noise artefacts in chroma subsampledimages. It has been previously observed in MPEG by Lopez et al.[7]) and Francois [8]. Examples from the Market sequence areshown in Fig. 4. Since the original capture was in Rec. 709 colourspace, a Rec. 709 container has been used for this sequence inorder to simulate Rec. 2100 content that fills up the Rec. 2100colour space.

It has been reported that the distortion induced is greatest forcolours at the edges of the colour space [8]), i.e. for moresaturated colours, and [9] provided some examples, but to theauthors’ knowledge no systematic analysis of where the distortionoccurs has yet been undertaken.

3 Identifying where errors occur

Ideally, we seek to identify an error surface in the xy plane of CIE1931, showing how the distortion varies throughout the colourspace. This has two complicating factors:

1. The distortion is not simply a function of the colour of a singlepixel, but the combination of neighbouring pixels.2. The region of the colour space with greatest distortion maychange as a function of luminance.

An exhaustive search of every combination of neighbours wouldbe not only time consuming, but very difficult to visualise.

Instead, we create a synthetic image, or ‘swatch’, of 16 × 16 pixels,consisting of a single colour of known linear light RGB value (andtherefore known xyY value) and add zero-mean Gaussian noise.No attempt has been made to simulate the characteristics ofcamera or film noise. The resulting swatch is referred to as the

IBC, Open Access Proc. J., 2017, pp. 1–5access article published by the IET under the Creative CommonsAttribution License (http://creativecommons.org/licenses/by/3.0/)

Fig. 4 Left column: original 4:4:4. Right column: after conversion to 4:2:0and back. Image sequence courtesy of Technicolor and the NevEx project

reference swatch. We apply the Rec. 2100 processing chain to obtain4:2:0 Y ′CbCr data. Converting back to linear light RGB results inwhat we refer to as the reconstructed swatch. We are then able tocompare the reference and reconstructed swatches either byviewing or by measuring the error objectively. Most of theprocessing in this analysis is performed in Python using the opensource Colour Science library [10].

3.1 Objective analysis

We have analysed values of Y∈{20, 50, 100, 200, 500, 1000, 2500,5000, 7500, 9000}, and for each value, we define a linearly spacedgrid of 100 × 100 points in xy. For each point on this grid that resultsin a ‘legal’ linear light RGB value, we create a swatch and performthe processing described above. By ‘legal’ we mean that the xyYtriple corresponds to a Rec. 2100 normalised linear light RGBtriple with each component in the interval [0,1]. This allows us tovisualise how the error changes through an xy plane for several‘slices’ of Y.

For the objective analysis of the error, we convert both swatchesfrom linear light RGB to xyY, and calculate the sum of the squareddifference (SSD) between the luminance samples of thereconstructed swatch, Yj, and the luminance samples of thereference swatch, Yj

SSD =∑256i=1

Y j − Y j

( )2

This provides a single value of the expectation of the error for everycolour in the colour space which we refer to as the reconstructionerror. Fig. 5 shows xy planes for the set of luminance values, Y,described above. The reconstruction error for a swatch with eachcolour defined on the 100 × 100 grid is represented as a circularmarker. The colour of the marker is a representation of the colour(subject to the constraints already discussed) and the radius of themarker is linearly proportional to the magnitude of thereconstruction error.

We see that for all Y values, the errors are largest at the edges ofthe colour gamut. For example, for Y= 100, the largest errors occur atthe lower edge of the colour space in what might be referred to as thesaturated blues, purples and reds.


Furthermore, we see that for different values of Y, the area of thecolour space where larger reconstruction errors occur changes.Specifically, at a value of Y= 200, we start to see increasingreconstruction errors in the blue-green axis. At a value of Y= 1000we see that errors begin to increase in the yellows. It should benoted that the total addressable area of the xy plane decreases withincreasing Y since the range of ‘legal’ linear light RGB valuesdecreases.

3.2 Subjective assessment

The reconstruction error above represents one simple metric of theerror introduced by chroma subsampling in a Rec. 2100, PQcontainer. No account has been made for how the noise levelmight be assessed subjectively. For example, an absolute error inthe luminance will be easier to see when the luminance of theimage is lower than when it is very bright. With this in mind, weviewed swatches for the xy values which resulted in the maximumand minimum objective reconstruction error for several values ofluminance, Y, on the SIM2 HDR47 monitor.

We observed that for very low luminance values (e.g. Y= 10) thenoise on the 4:4:4 version was very significant, so large in fact that itmasked any errors introduced by the subsampling process in the4:2:0 version. For Y= 100 however, the swatch resulting in themaximum reconstruction error had noise that was barelyperceptible on the 4:4:4 version but very clear on the 4:2:0version. The swatch with a minimum error, occurring at xy= (0.33,0.34), showed barely perceptible noise on both the 4:4:4 and the4:2:0 versions.

For a luminance of Y= 2500, the swatch having the greatestreconstruction error can be described as a light cyan colour. Thenoise is not visible in 4:4:4 but can still be clearly seen in 4:2:0.Increasing the luminance further to Y= 7500, the colour of theswatch with largest error is yellow. The noise is now not visibleon either the 4:4:4 or the 4:2:0 versions, despite the reconstructionerror being 9.7 × 105 which is twice that of the error for Y= 100.

In conclusion, the subjective assessment of the swatches revealedthat for a fixed noise level, the perception of noise is higher when theswatch luminance is lower as one might expect. Furthermore, theperception of noise increases with increasing reconstruction errorfor a fixed luminance level. The precise relationship, however, hasnot been investigated.

4 Artefact avoidance

The artefacts analysed in the preceding can be avoided in two mainways. The first involves ensuring that the content does not have pixelvalues at the edge of the colour gamut. This will be the case where,for example, the camera used to capture the content cannot addressthe whole colour volume. This restricts the benefit of the increasedcolour volume and is therefore sub-optimal. The second involvesmodifying the process of converting RGB to Y ′CbCr. One coulduse a constant luminance workflow. This decouples the luminancefrom the chroma components and in so doing eradicates theartefact. It is accepted, however, that this would require a completeoverhaul of the broadcast chain.

Finally, we can modify the subsampling process. Two classes oftechnique to avoid luminance artefacts for non-constant luminanceY ′CbCr processing are presented in the following.

4.1 Luma adjustment

The first class of techniques is collectively called luma adjustment,and here the idea is to compensate for the error in the luminanceby changing the luma value Y ′ in each pixel. Since its introductionby Ström et al. [5], it has been the subject of severalimplementation optimisations (cf. Norkin [11], Ström et al. [12,13], Rosewarne and Kolesnikov [14]). It has also been included asthe enhanced processing mode for the technical report on

3Commons

Fig. 5 Plots of the reconstruction error planes for values of Y between 20 and 9000. The point of maximum reconstruction error is marked with a whitetriangular marker in each case

IBC, Open Access Proc. J., 2017, pp. 1–54 This is an open access article published by the IET under the Creative Commons

Attribution License (http://creativecommons.org/licenses/by/3.0/)

Fig. 6 Left column: after subsampling to 4:2:0 using traditionalprocessing. Right column: after subsampling to 4:2:0 using lumaadjustment. Image sequence courtesy of Technicolor and the NevEx project

conversion and coding practices for HDR/WCG [15]. An example ofartefact reduction using luma adjustment can be seen in Fig. 6.

4.2 Chroma adjustment

The second type of technique for artefact avoidance is known aschroma adjustment, and was introduced by Ström and Wennersten[16]. Instead of changing only the Y′ of the Y ′CbCr representation,the original colour of each pixel is changed slightly in all threecomponents so that it is more similar to the colour of itsneighbours. The change is small enough so that the differenceshould not be visible, yet it has the effect of almost completelyremoving the subsampling artefacts. After that, the luminance isreintroduced to avoid any luminance smoothing, and finally a stepof luma adjustment is used.

In performing chroma adjustment on the artificially generatedswatches as part of the subsampling process, it was observed thatthe reconstruction error is several orders of magnitude lower thanthat observed using standard subsampling techniques.

5 Conclusions

This paper has presented an analysis of the error induced by chromasubsampling in an HDR, Rec. 2100, non-constant luminanceworkflow. By creating synthetic images of plain colour


contaminated by noise we have been able to objectively assess themagnitude of these errors as a function of both colour andluminance. The artefacts have been demonstrated to be worse forcolours of high purity, i.e. occupying space at the edge of thecolour space as previously observed. Additionally, thiscontribution has demonstrated that the colours for which the erroris worse change as a function of the luminance.

Visible artefacts are unlikely to occur when the captured videodata does not fill the Rec. 2100 colour space. As more WCGcontent becomes available, however, the industry is likely to seemore artefacts associated with chroma subsampling and willtherefore need to take steps to avoid it. These fall in to two mainareas:

1) Prior to conversion to Y ′CbCr: e.g. avoiding regions of the colourspace known to cause the issue.2) Conversion to Y ′CbCr:o Using a constant luminance workflow – it is acknowledged by theauthors that this may not be practical where existing broadcastequipment is to be used.o Using chroma/luma adjustment algorithms in the subsamplingprocess.

The method used in this paper could be significantly enhanced byderiving a noise model more commonly observed in camera data,and using a modified error metric which reflects the subjectiveassessment of the noise artefacts.

6 References

1 DVB: ‘Specification for the use of video and audio coding in broadcastingapplications based on the MPEG-2 transport stream’, DVB Document A157, 2016

2 ATSC: ATSC proposed standard: video – HEVC (A/341), 20173 ITU-R: ‘Image parameter values for high dynamic range television for use in

production and international programme exchange’, recommendation ITU-RBT.2100-0, 2016

4 SMPTE: ‘High dynamic range electro-optical transfer function’, ST 2084, 20145 Ström, J., Samuelsson, J., Dovstam, K.: ‘Luma adjustment for high dynamic range

video’. Proc. of the IEEE Data Compression Conf. (DCC), Snowbird, 20166 Poynton, C.: ‘Digital video and HDTV algorithms and interfaces’ (Morgan

Kaufmann, 2003)7 Lopez, P., François, E., Yin, P., et al.: ‘Not public: generation of anchors for the

explorations for HDR/WCG content distribution and storage, m34077’. 109thMPEG Meeting, Sapporo, Japan, 2014

8 Francois, E.: ‘Not public: MPEG HDR AhG: about using a BT.2020 container forBT.709 content’. 110th MPEG Meeting, Strasbourg, France, 2014

9 Ström, J.: ‘Not public: investigation of HDR color subsampling, m35841’. 111thMPEG Meeting, Geneva, Switzerland, 2015

10 Mansencal, T., Mauderer, M., Parsons, M., et al.: ‘Colour 0.3.9’, Zenodo, 201711 Norkin, A.: ‘Fast algorithm for HDR video pre-processing’. Proc. of the IEEE

Picture Coding Symp. (PCS), Nuremberg, 201612 Ström, J.: ‘AhG-13 related: multi-LUT luma adjustment implementation’, JCT-

VC, JCTVC-Y0030, Chengdu, 201613 Ström, J., Andersson, K., Pettersson, M., et al.: ‘High quality HDR video

compression using HEVC main 10 profile’. Proc. of the IEEE Picture CodingSymp. (PCS), Nuremberg, 2016

14 Rosewarne, C., Kolesnikov, V.: ‘AHG13: further results for luma sampleadjustment’, JCT-VC, JCTVC-Y0034, Chengdu, 2016

15 ISO/IEC TR 23008-14, conversion and coding practices for HDR/WCG Y ′CbCr4:2:0 video with PQ transfer characteristics

16 Ström, J., Wennersten, P.: ‘Chroma adjustment for HDR video’. 2017 IEEE Int.Conf. Image Process, 2017

5Commons

characterisation of processing artefacts in high dynamic ... · pdf fileolie baumann , alex...

Documents