image deblocking using local segmentation lukasz kizewski supervisor:dr. peter tischer second...
TRANSCRIPT
Image Deblocking Using Local Segmentation
Lukasz Kizewski
Supervisor: Dr. Peter Tischer
Second Examiner: Dr. Andrew Paplinski
Presentation Outline
Introduction Lossy Image Compression - JPEG Discrete Cosine Transform (DCT) Subbands
Present Research Research gap
DCT coefficient study Deblocking Filter Conclusions Further Research
Lossy Image Compression
JPEG, MPEG is lossy Maintaining high image quality and high compression
ratios is a major, widespread issue Used everywhere!
Internet, digital photography, digital camcorders, DVDs,Digital TV, video telephony/conference, mobile phones, etc.
JPEG/MPEG uses ‘Transform Coding’ technique coupled with data quantization JPEG/MPEG uses Discrete Cosine Transform (DCT) DCT is energy-preserving and reversible Quantization step is the lossy part
Discrete Cosine Transform
Image divided into 8x8 pixel blocks DCT applied to each block independently Block decomposed into basis functions
First basis function (0,0) termed ‘DC coefficient’ Remaining 63 basis functions termed ‘AC coefficients’
[1]
DC coefficient- Average brightness
AC coefficients-White: Add to average-Black: Subtract from average
DCT - Subbands
For every 8x8 pixel block, output of DCT is64 DCT coefficients Each coefficient corresponds to one basis function “Image” of 1 DCT coefficient termed a “subband”
[1]
2x2 blockDCT example
Subbands
First 4 (out of 64) subbands of ‘Lenna’
[1]
After performing the DCT, resulting coefficients are quantized Divided by quantum value, rounded to integer
Quantum value dictated by ‘quality’ parameter and quantization table
High-order DCT coefficients more severely quantized (usually to zero)
During de-quantization, mid-point of quantization interval is chosen Usually incorrect
DCT Coefficient Quantization
Q=10
5 14
1
5 1410
JPEG – Decompression Quality
JPEG decoded image at different ‘quality’ parameter settings:
Q = 100 Q = 50 Q = 1
JPEG Image artifacts
Incorrectly reconstructing DCT coefficients results in unwanted image artifacts: Smooth regions are blocky, edges are jagged,
discontinuities appear near edges Aim of project – decrease severity of artifacts
Smooth region(shoulder)
Staircase effect(edge of hat)
Ringing effect(edge of mirror)
Present Research
Dozens of filters exist to increase the quality of highly-compressed imagesSome filter DCT coefficients (subbands)Some filter reconstructed pixel valuesOthers filter bothMost filters target only one type of image
artifact Some filters reduce one type of artifact whilst
making another more prominent
Research Gap
Natural/photographic images have high correlation between neighbouring pixels Neighbouring pixels are similar in brightness Property fails when edge in image is encountered
Lack of image segmentation in most filters Results in blurred/smoothed edges
Encountering an edge implies more than one segment Segments should be filtered independently
Filling the Research Gap
This project differs in that:Local segmentation is used
Pixels possibly split into 2 groups
Each segment filtered independently“Do No Harm” policy used – avoid further
image quality degradation If filtering model fails, it’s filtered value is
disregarded Worst-case scenario – image is not filtered at all
Filling the Research Gap (cont.)
Filter can be applied toDCT Coefficients (filtering subbands)Pixels (filtering an image)Pixels may be filtered after subbands are
reconstructed as accurately as possible Quality loss occurs at subband level
First filtering step should be reconstructing DCT coefficients better
DCT Coefficient Study
Determine each subband’s contribution to overall image quality
Selected subbands were not quantized Simulates subbands being reconstructed perfectly Subband’s contribution measured by increase in
Peak Signal-to-Noise Ratio (PSNR) PSNR: Logarithmically-scaled, mean-squared-error metric NOTE: PSNR value doesn’t always reflect viewer-subjective
image quality assessment
DCT Coefficient Study (cont.) Results of study:
Image Deblocking Filter
Main goal is to reconstruct DCT coefficients better to reduce severity of image artifacts
DC subband filtered DCT study shows DC subband is best
filter candidate A DCT coefficient is a subband “pixel”
Uses 3x3 weighted mask Mask center is pixel being filtered Mask scans entire image, filtering
each pixel
1 2 1
2 4 2
1 2 1
3x3 filtering mask
DC subband of ‘Lenna’
“Do No Harm”
Filter employs “Do No Harm” (DNH) policy If a filtered value is implausible, reject it and
leave value unfiltered Implausible pixel value is one that falls
outside the quantization interval Quantization interval: midpoint +/- ½ Quantum
Guarantees image quality cannot degrade further
Sequential Filter
Try 1-segment filtering Replace pixel with mask average If DNH not triggered, accept
Otherwise try 2-segment filtering Use average-value thresholding to
classify pixels in mask Replace pixel with average of
segment to which it belongs to If DNH not triggered, accept
Otherwise trigger DNH and leave unfiltered value
100 100 100
100 15 15
100 10 5
Weighted average= 50.94
100 100 100
100 15 15
100 10 5
Blue class average=12.78
Sequential Filter - Results
Blocky effect reduced to 8x8 pixels Contrast between blocks minimised Filtered image can be used as input to another filter
DC Subband Expansion
Resolution of DC subband is increased 8-fold To match the resolution of the original image Inverse-DCT modified to use expanded DC subband
Linear-interpolation used to fill the gaps between original DC subband values
Applies a “gradient” to DC subband
128x128 pixel image 128x128 expanded DC subband
DC subband
DCT DC expansion
DC Subband Expansion - Results
Smooth regions completely void of blocky artifacts Expansion applied to filtered DC subband
What about the edges?
‘Staircase’ and ‘ringing’ effects still visible in all results
Sequential deblocking filter inappropriate for AC subband filtering AC subbands have very low
inter-pixel correlation Filtering AC subbands with
this filter introduces ringing DNH isn’t triggered because
AC quantization intervals are very wide
What about the edges? (cont.)
AC subbands describe edges, with respect to DC value Modifying AC coefficients adds/removes edges/textures Filtered image (far right) shows a light edge added next
to the actual edge – a ringing artifact! AC filtering – topic for further research
Original Reconstructed Filtered-reconstructed
Unsuccessful Additions
Always use 2-segment filtering Contrast between blocks increased ‘Chessboard’ effect
Threshold segmenting If class representatives are close, treat mask as 1
segment Finding correct threshold value impossible
3-segment filtering If 2-segment filtering triggered DNH,
3-segment filtering was attempted Little-to-no improvement in image quality Concluded that 2-segment model is sufficient for
95% of cases
Unsuccessful Additions (cont.)
Overlapping mask filtering All pixels’ filtered values in a mask were recorded Filtered value equal to average of all possible
recorded reconstructions (most pixels had up to9 possible filtered values)
Little-to-no improvement in image quality Applying sequential filter to reconstructed pixel
values “Quantization interval” parameter not known Guessing interval resulted in:
Blurry images – blurred edges Overly-sharpened images – “cardboard cut-out” effect with
flat colors
Conclusions
Filtering DC subband has been successful in improving overall image quality As predicted by the DCT coefficient study Due to high inter-pixel correlation
AC subbands must be filtered in some other way This filter produces ringing artifacts
Due to averaging Due to little inter-pixel correlation Due to extremely wide quantization intervals – allowing large
change to an AC coefficient’s value Little or no information left in subband
Perhaps try median-filtering, instead of averaging May introduce blurring of edges
Conclusions (cont.)
Knowledge of a subband’s quantization interval aids in segmentation Maximum quantization error is known DNH “switches” between 1- or 2-segment filtering
Sequential (adaptive) filtering model proved successful DNH policy ensured no further image quality loss Sometimes too strict
Some blocks “don’t fit” in with surrounding pixels Conforms to JPEG specifications
Critical for video sequence coding (MPEG) – reconstruction errors propagate to subsequent frames
Future Research
AC subbands must be filtered in a different manner altogether Perhaps use DC subband segmentation to drive AC
subband filtering Resulting image from filtered,
non-expanded DC subband filter can be used as a starting point for a secondary filter This filter should target edge artifacts
Once edges are also filtered, DC expansion could be applied as a third step
Try median-filtering, instead of averaging
For More Information…
Consult Thesis See Website
http://www.csse.monash.edu.au/~kizewski/
E-mail me [email protected]
Read referenced material Read filter source code Ask a question
[1]: Rabbani, M. and Jones, P. W. (1991). Digital Image Compression Techniques, Vol. TT7 of SPIE Tutorial Texts.[2]: http://www.mat.univie.ac.at/~kriegl/Skripten/CG/node53.html
Thank-You
Questions?
…and now, for some eye-candy!