copyright © magnum semiconductor, unpublished introduction to deinterlacing by mark korhonen
TRANSCRIPT
Copyright © Magnum Semiconductor, UnpublishedCopyright © Magnum Semiconductor, Unpublished
Introduction to Deinterlacing
by Mark Korhonen
Copyright © Magnum Semiconductor, Unpublished
2
Overview:Introduction To Deinterlacing
Background• Progressive video• Interlaced video• Deinterlacing
Basic Deinterlacing Algorithms• Weave• Vertical Interpolation (aka Bob)
Advanced Deinterlacing Algorithms• Diagonal Interpolation• Cadence Detection• Motion Adaptive Deinterlacing (MADI)• Motion Compensated Deinterlacing (MCDI)
Deinterlacing Applications
Picture source: http://lurkertech.com/lg/fields/Copyright © Magnum Semiconductor, Unpublished
3
Background: Progressive Video
A complete video frame is displayed at regular intervals
View on computers and digital televisions
Example resolutions and frame rates:• SD: 720x480p30, 720x576p25• HD: 1280x720p60, 1920x1080p30
Sources: film (movies), animation, progressive cameras
Example: 30p = 30 frames/s
Picture source: http://lurkertech.com/lg/fields/Copyright © Magnum Semiconductor, Unpublished
4
Background: Interlaced Video
Frames of video are sampled at two time intervals• Even rows sampled at: 2t• Odd rows sampled at: 2t + 1
Terminology:• Top field: even rows in the frame• Bottom field: odd rows in the frame• Field polarity: indicates if a field is a top field or bot. field
Example: 60i = 60 fields/s
Copyright © Magnum Semiconductor, Unpublished
5
Background: Interlaced Video
Why interlaced video exists• Double frame rate w/ same bandwidth = smoother motion• Original television formats are all interlaced
– NTSC=720x480i60– PAL=720x576i50
• 1920x1080i60 exists because– Processing requirements for 1920x1080p60 are very high– Motion is smoother than 1920x1080p30– More detail than 1280x720p60
Copyright © Magnum Semiconductor, Unpublished
6
Background: Deinterlacing
Deinterlacing: convert interlaced video to progressive• Input: X field/s (e.g. i60)• Output: X frames/s (e.g p60)• This process generates X “missing” fields/s
Copyright © Magnum Semiconductor, Unpublished
7
Background: Deinterlacing
Fields typically used to generate the missing field• The current field is the opposite polarity of the field that
needs to be generated, but displayed at the same time– E.g. output = bottom field, curr = top field
• The previous and next fields are the same polarity as the field that needs to be generated
– E.g. output = bottom field, prev & next = bottom fields
Picture source: doom9.orgCopyright © Magnum Semiconductor, Unpublished
8
Basic Deinterlacing: Weave
Weave: use “prev” or “next” as the missing field• For still images and progressive content, no missing data• Very severe artifacts for moving interlaced content
Copyright © Magnum Semiconductor, Unpublished
9
Basic Deinterlacing: Weave
Extension• If stationary, average the previous and next fields
together as the missing field– Noise reduction (averaging two identical images reduces noise)– Handles fades better if luminance changing every field
Output[x,y] = m prev[x,y] + (1-m) next[x,y]• m is typically 0.0, 0.5 or 1.0
Copyright © Magnum Semiconductor, Unpublished
10
Basic Deinterlacing: Bob
Bob (aka vertical interpolation)• Generate the opposite polarity by interpolating along
vertical columns in the current field
Implementations• Line doubling – just use current field
– e.g. Output[x,y] = curr[x,y]
• 2-tap FIR filter – average the line above and below– e.g. Output[x,y] = 0.5 curr[x,y] + 0.5 curr[x,y+1]
• 8-tap FIR filter – better frequency response
Copyright © Magnum Semiconductor, Unpublished
11
Basic Deinterlacing: Bob
Bob Video Quality• Looks good for most video content
– Still content without a lot of detail– Moving content (hard to see detail on moving objects)
• Interpolation fails if a lot of detail – causes flickering
Picture source: Computer Desktop EncyclopediaCopyright © Magnum Semiconductor, Unpublished
12
Advanced Deinterlacing:Diagonal Interpolation
Problem: Bob isn’t ideal for diagonal edges
Solution: Apply FIR filter along a diagonal
Copyright © Magnum Semiconductor, Unpublished
13
Advanced Deinterlacing:Diagonal Interpolation
Diagonal Edge Detection Algorithm• Try a bunch of different angles and see which is best
– Pattern recognition problem: classify edges as a supported angle• Determine which angle is most correlated at each pel
• We can assume that the angle of the edge is wide
» Allows using neighbouring pels to improve accuracy
Picture source: http://www.infocellar.com/televisionCopyright © Magnum Semiconductor, Unpublished
14
Advanced Deinterlacing:Cadence Detection
Observation: If interlaced video was generated from a progressive source, we can safely weave it with no artifacts
Sample cadences
Copyright © Magnum Semiconductor, Unpublished
15
Advanced Deinterlacing:Cadence Detection
Detection Area• Size of cadence area can be variable
– e.g. entire field, 16x16 block, every pel
• Detecting entire fields sufficient for ~99% of video• Smaller areas only needed if video is a mixed source
– moving interlaced text over progressive video• e.g. weather warnings on a TV movie
– different frame rates of progressive video edited together• e.g. source of border was 30p, but source of contents were 24p
Picture source: http://www.infocellar.com/televisionCopyright © Magnum Semiconductor, Unpublished
16
Advanced Deinterlacing:Cadence Detection
Cadence Algorithm 1: detect regular repeated fields• For 3:2, every 5th field is a repeat
– SAD will be very low every 5th field called Inverse Telecine
• Very robust for a lot of content• Doesn't work for 2:2 (no repeats), or changing cadences
(e.g. slow-motion replays, edited video)
Copyright © Magnum Semiconductor, Unpublished
17
Advanced Deinterlacing:Cadence Detection
Cadence Algorithm 2: detect weaving artifacts• 1) Weave curr with prev, 2) Weave curr with next• Determine if weaving artifacts are less in a field
– Pattern recognition problem: classify into three states• Progressive – weave with previous
• Progressive – weave with next
• Interlaced – perform other deinterlacing
Copyright © Magnum Semiconductor, Unpublished
18
Advanced Deinterlacing:Cadence Detection
Cadence Algorithm 2: detect weaving artifacts• Tricky for stationary content and at scene changes
– Fairly easy to compensate
• Tricky for vertical motion of detailed video– e.g. pan of venetian blinds
• Can be tricky for video that looks like it already contains weaving artifacts
– Artifacts introduced by video compression can look like weaving artifacts
– Certain textures
Copyright © Magnum Semiconductor, Unpublished
19
Advanced Deinterlacing:Motion Adaptive Deinterlacing (MADI)
Idea: weave stationary areas, interpolate moving areas
Stationary detection = pattern recognition problem• Size of stationary area can vary (e.g. entire field, 16x16
block, every pel)– Smaller stationary areas = less flickering but more computation
• Possible implementations:– SAD of prev and next
• Watch out for periodic motion
– min SAD of last X fields• Watch out for how long it takes stationary regions to be detected
Copyright © Magnum Semiconductor, Unpublished
20
Advanced Deinterlacing:Motion Adaptive Deinterlacing (MADI)
Basic algorithm:• Inputs:
– Weave – generated from previous and/or next– Inter – generated from current (diagonal interpolation)
• Output[x,y] = (K) Weave[x,y] + (1-K) Inter[x,y]– K ~= 1 for stationary/progressive areas (weave)– K ~= 0 for moving areas (interpolation)
NOTES:• Fades may need special handling
– e.g. bias more towards interpolation
• x and y advance through regions in the video– Could be every pel, 16x16 blocks, or the entire field
Copyright © Magnum Semiconductor, Unpublished
21
Advanced Deinterlacing:Motion Compensated Deinterlacing (MCDI)
Idea:• If we can’t weave a particular region, could we weave a
motion compensated version of this region from curr, prev or next?
Comments:• Potential for preserving all of the detail• Very difficult to do correctly
– Especially rotations, zooms, morphing, lighting changes, etc.– Objects can get covered/uncovered– Even small errors can look very bad
• weaving artifacts, ghosting around edges, weird motion, etc.
• Motion compensation should be done for every pel– Very computationally demanding
Copyright © Magnum Semiconductor, Unpublished
22
Advanced Deinterlacing:Motion Compensated Deinterlacing (MCDI)
Basic Algorithm:• Input fields
– MC_curr[x,y] = curr[x + dx_c, y + dy_c]– MC_prev[x,y] = prev[x + dx_p, y + dy_p]– MC_next[x,y] = next[x + dx_n, y + dy_n]– MADI[x,y] = pel generated using motion adaptive deinterlacing
• Output:
• p + q + r ~= 1 if high confidence in motion compensation
• p + q + r ~= 0 if low confidence in motion compensation
– Confidence in the motion compensation can be based on• The smoothness of the motion vector field
• Estimated SAD (true SAD unknown because the field is missing)
• Consistency of motion from field to field
yxMADIrqpyxnextMCryxcurrMCqyxprevMCpyxMCDI ,)1(,_,_,_,
Copyright © Magnum Semiconductor, Unpublished
23
Deinterlacing Applications
Frame rate conversion from interlaced to interlaced• No deinterlacing = motion judder
– Need to drop a pair of fields – increases motion judder
• Deinterlacing = smoother motion– Requires excellent quality deinterlacing
• Deinterlaced fields will be deinterlaced twice
Copyright © Magnum Semiconductor, Unpublished
24
Deinterlacing Applications
Upscaling interlaced video requires deinterlacing• Perserves more detail• Minimizes aliasing
Copyright © Magnum Semiconductor, Unpublished
25
Deinterlacing Applications
If the output frame rate is low, flicker can be severe
• If input frame rate is high enough– In the output, pick a polarity that only use fields from the input– The other polarity will always be generated (e.g. via interpolation)
• This works because flicker is caused by alternating rows between a) original and b) interpolated – we have removed the alternating
Copyright © Magnum Semiconductor, Unpublished
27
Video Resolutions
Common video resolutions• SD: 720x480 (NTSC), 720x576 (PAL)• HD: 1280x720, 1920x1080• Sub SD: VGA, QVGA, CIF, QCIF, etc.
Definitions• pels = 1x1 area of video data• pixel = 1x1 area on video display• E.g. 720x480 video is 720 pels wide, but may be
displayed on a television with 1920 pixels