image processing on gpu
TRANSCRIPT
Image Image ProcessingProcessing
on GPUon GPU
Image ProcessingImage Processing• Define new images from existing image
for different purposes• Used for texture preprocessing for 3D
graphics and visualization• Simple processing:
– Transform each pixel independently• Pixel to pixel operations• Example: RGB image to grey scale image
– Move pixels inside without modifying intensities
• Complex processing– Many advanced technologies for different
applications
Noise ReductionNoise Reduction• An important application for image
processing• Important in many areas• Sources:
–Digital camera devices–Scanned documents–Satellite images–Medical imaging–…
Example Noise TypesExample Noise Types• Salt and pepper:
random white and black pixels
• Impulse: random white pixels
• Gaussian noise: variation in intensity from Gaussian distribution
• …• We need to “SMOOTH”
the image to reduce noises
Linear FiltersLinear Filters• Construct a new but same-size image
from original one• Fill new pixel with a weighted sum of
old pixels -- filtering• Using the same weights for each pixel• Linear filtering
–Output for the sum of tow images is the same as the sum of the outputs obtained for the images separately
ConvolutionConvolution• Filter kernel: the pattern of weights• Convolution: the process of applying
the filter on an image• Mathematically:
SmoothingSmoothing• Reduce noise• Replace each pixel with a weighted
average of its neighbors• Local blur effects to remove noise• Low frequency filter to remove high
frequency noise
Averaging SmoothAveraging Smooth• (2k+1)*(2k+1) region
• Poor blurring• Too simple
∑ ∑+
−=
+
−=+=
ki
kiu
kj
kjvuvij F
kR 2)12(
1
Gaussian SmoothingGaussian Smoothing• Symmetric Gaussian kernel
– Weights distributed with Gaussian function– Sigma: standard deviation
– In applications, discrete kernel used:
)2
)(exp(2
1),( 2
22
2 σπσσyxyxG +
−=
)2
))1()1((exp(2
12
22
2 σπσ−−+−−
−=kjkiHij
Gaussian Lookup TableGaussian Lookup Table–A lookup table can be pre-generated to
accelerate the computation, such as
Gaussian KernelGaussian Kernel• If standard deviation is very small
(smaller than one pixel), the smoothing have very little effect
• For a larger one, the average will be strongly biased toward a consensus of the neighbors. Noise will largely disappear, at the cost of some blurring
• Finally, if it is very large, much of detail disappear along with noise
Why use GaussianWhy use Gaussian• Not the only smoothing filter• Convenient to use:
– Efficiency– Separable
– Performed separately for x, y, z dimensions
22 2121 σσσσ +=∗ GGG
Boundary EffectsBoundary Effects• In practical system, image has
boundaries• When computing convolution, need
to handle boundaries where the neighbors do not exist– Ignore these locations-image may
shrink–Pad the image with constant values–Pad with other methods: periodical …
ExampleExample
• Robert J. Woodham, UBC
Edge DetectionEdge Detection• Edges are important information on
images–Object detection …–Large gradient– Intensity derivatives
• Edge detection: estimate derivative and an image represented by a discrete set of pixels
What is an edgeWhat is an edge
• Easy for human• Difficult for computer
Estimate DerivativeEstimate Derivative• Finite difference
• The similar convolution kernel is
• Gradient vector
jiji hhxh
,1,1 −+ −≈∂∂
⎪⎭
⎪⎬
⎫
⎪⎩
⎪⎨
⎧−=000101
000G
⎟⎟⎠
⎞⎜⎜⎝
⎛∂∂
∂∂
=∇yf
xfyxf ,),(
SobelSobel KernelKernel• A popular gradient magnitude
computation
Second DerivativesSecond Derivatives• Sobel operator can produce thick edges• Ideally, we want thin boundaries• Alternative approach:
– Looking for local extrema in first derivative– Where the change in gradient is highest
2D 2D LaplacianLaplacian• An equivalent measure of second
derivative
Edge detection cg codeEdge detection cg code• Laplacian edge detection
static const char *edgeFragSource = {"uniform sampler2D texUnit;"
"void main(void)""{"" const float offset = 1.0 / 512.0;"" vec2 texCoord = gl_TexCoord[0].xy;"" vec4 c = texture2D(texUnit, texCoord);"" vec4 bl = texture2D(texUnit, texCoord + vec2(-offset, -offset));"" vec4 l = texture2D(texUnit, texCoord + vec2(-offset, 0.0));"" vec4 tl = texture2D(texUnit, texCoord + vec2(-offset, offset));"" vec4 t = texture2D(texUnit, texCoord + vec2( 0.0, offset));"" vec4 ur = texture2D(texUnit, texCoord + vec2( offset, offset));"" vec4 r = texture2D(texUnit, texCoord + vec2( offset, 0.0));"" vec4 br = texture2D(texUnit, texCoord + vec2( offset, offset));"" vec4 b = texture2D(texUnit, texCoord + vec2( 0.0, -offset));"" gl_FragColor = 8.0 * (c + -0.125 * (bl + l + tl + t + ur + r + br + b));""}"
};
Results not satisfiedResults not satisfied• Noise are emphasized
How to improve?How to improve?• Add smoothing filter• Smoothing a differentiated image
tends to get good result we want• Smooth the edge detected due to
noises
Improved ResultsImproved Results
Derivative of Gaussian FilterDerivative of Gaussian Filter• The improvement is implemented as:
Gaussian smoothing first and then use differential filter
• This can be replaced by convolution with the derivative of Gaussian
xIG
∂∗∂ )( σ
Ix
Gx
IG∗
∂∂
=∂∗∂ σσ )(
Steps in edge detectionSteps in edge detection• Typical steps:
–Filtering: cut down on noise–Enhancement: amplify the difference
between edges and non-edges–Detection: use threshold–More: estimate geometry and other
operations
Image Processing on GPUImage Processing on GPU• Processing an image is fairly simple and
straightforward. • The filter renders a screen-aligned quad
into an invisible pixel buffer.– The screen-aligned quad has the input image
bound as a texture. – For each rendered pixel, a Cg fragment
program is executed, which does the actual image processing in a local neighborhood.
– Because the quad is rendered using the input image as a texture, the Cg program can access the pixels in the input image though simple texture reads.
Convolution on GPUConvolution on GPU
Typical GPU FrameworkTypical GPU Framework• Load Sources• Apply filters as cg
programs• Render image on
screen• Must also provide
operators for saving images
An Example: An Example: ScotopicScotopic FilterFilter• Vision under reduced illumination,
– lose most of our ability to see color– Scenes appear blurred and shaded in blue
• Blur use Gaussian filter• Blue shaded use night-vision filter• Then edge detection• This procedure mimic the human night
vision system
• GPU implementation enable immediate processing, useful for many real applications, such as military
ExampleExample
More Techniques and ExamplesMore Techniques and Examples
Image GlowImage Glow
Add Glow on GPUAdd Glow on GPU• (a) The scene is rendered normally.• (b) A rendering of glow sources is blurred to
create • (c) a glow texture, which is added to the ordinary
scene to produce • (d) the final glow effect.
Glow on GPUGlow on GPU• The alpha channel of an object's
texture can be used to specify glow sources and the glow source brightness.
Color ControlColor Control• Color correction is part of almost all print
and film imaging applications. – move color images from one color space to
another (say, from Adobe RGB to sRGB); – to stylize images (creating illusions such as
faded film, cross-processing, or other stylistic variations);
– to combine art elements from different sources (such as matching color palettes between different video sources, or models made by different artists);
– to give broad coherence and mood to entire parts of a game or scene without changing the underlying models and textures.
General IdeaGeneral Idea• Change color of individual pixels
– Per channel: alter red, green, and blue components individually
– color-mixing: each output channel may be an operation based on the red, green, and blue components simultaneously
• The mathematics of color corrections can be compactly and easily described in a GPU shader.
• Just as important, they can be controlled effectively with common tools used widely by computer artists and programmers. – E.g.Relying on Adobe Photoshop to let artists
create control resources, which can then be applied in real time via pixel shaders.
Gamma AdjustmentGamma Adjustment• Adjusting Image Gamma and Overall Dynamic
Range in photoshop• Cg program implement: • outPixel = (pow(((inPixel * 255.0) - inBlack) / (inWhite - inBlack), inGamma) *
(outWhite - outBlack) + outBlack) / 255.0;
Curved Level AdjustmentCurved Level Adjustment• Curves tool in Photoshop provides arbitrary remapping of
the color channels
Curved Level AdjustmentCurved Level Adjustment• A 1x256 texture map should be defined,
although smaller maps can often be used effectively
• Before creating the map, the artist must first define the color transformation, typically by applying adjustment layers to existing still images using the Curves tool in Photoshop.
• Create a ramp texture from an original white-black ramp according to the color transformation
Curved Level AdjustmentCurved Level Adjustment• Cg code• float3 InColor = tex2D(inSampler,
IN.UV).xyz; • float3 OutColor; • OutColor.r = tex1D(ColorCorrMap,
InColor.r).r; • OutColor.g = tex1D(ColorCorrMap,
InColor.g).g; • OutColor.b = tex1D(ColorCorrMap,
InColor.b).b;
• Channel-by-Channel Results of the Red, Green, and Blue Remappings
PDE and Image ProcessingPDE and Image Processing• We introduce simple local pixel
filtering for image processing• Some more complex image
processing techniques involves PDEs• Thus, GPU-based PDE solver is very
important– Image diffusion– Image editing–…
Repeated Smoothing: DiffusionRepeated Smoothing: Diffusion• Usually we need to repeat smoothing
operation• Such operation actually is a solution to
the diffusion equation
Φ∇=∂Φ∂
+∂Φ∂
=∂Φ∂ 2
2
2
2
2
yxt
Anisotropic DiffusionAnisotropic Diffusion• Edge preserving smoothing• Use a function g to control diffusion• Smoothing noise while conserve edges
)),,(( Φ∇⋅∇=∂Φ∂ σyxct
Diffusion FunctionDiffusion Function• c() is the diffusion function • A monotonically decreasing function
of the image gradient magnitude • Generally g() can be implemented
Gaussian Diffusion FunctionGaussian Diffusion Function• g() can be implemented by the
function of derivative of Gaussian distribution
• Smoothing noise while preserving edges
DemosDemos• Copyright by Joachim Weickert
How to solve the equation?How to solve the equation?• Typical Partial Differential equation• Basic finite difference operations to
discretize the equation• Thus, a linear system is achieved
–Show how?
• How to use GPU solving linear system?–Linear algebra on GPU