image sensor pipeline

PowerPoint Purpose
The objective of this presentation is to describe the image processing algorithms of Digital Camera. Most of the algorithms along the data path will be described.
The intended audiences include algorithm designers, software programmers, IC engineers, application engineers, and project managers as well.
websitehttp// www.smit.com.cn
YCbCr422
AUTO
Black level with G1/G2 filter
This function allows user to adjust the black level so that blacks appear as true blacks.
Formula:
B’ = B + BL_B_OFFSET
DPD/DPC and Noise Reduction
The image from sensor may have noise and dead pixels. This module can do noise cancellation and DPC at the same time.
diff40 = abs(p4-p0); diff41= abs(p4-p1);
diff42 = abs(p4-p2); diff43 = abs(p4-p3);
(diff44>dpc_th) && (diff45>dpc_th) &&
(diff46>dpc_th) && (diff47>dpc_th))
pixel cancellation (DPC)
if((DV<=DH)&&(DV<=DDL)&&(DV<=DDR))
new = (pix1+pix7+1)/2;
do noise cleaning
if ((DV2<=DH2)&&(DV2<=DDL2)&&(DV2<=DDR2))
avg = (p1+p4+p7) / 3;
var = abs(p1-avg) + abs(p4-avg) + abs(p7-avg);
else output = p4 ;
Digital Gain
There are two types of digital gain: global digital gain and R/G/B digital gain. The digital gain control is implemented on the input RGB Bayer pattern stream.
All the pixel data (R, G and B) are multiplied by a digital global gain. The higher the global gain is, the brighter the image is. The global gain is used to adjust image brightness in Auto Exposure (AE) function.
Different from digital global gain, digital RGB gains affect RGB pixel separately. Different color channel uses different gain value, so there are three gain values for R, G and B pixels, respectively. Digital RGB gains are used in Auto White Balance (AWB) function.
Implementation:
LensShadingCorrection(Vignetting)
This module is used to compensate for the shading effect, which causes the image intensity to get darker on the outer range of the image. We will apply a different gain value for each different (x, y) position in the image. The algorithm works on Bayer image, and works separately on R/G/B channel.
0
0
1
0
0
0
0
0
0
0
1
0
1
0
1
XOR
CFA Interpolation
In Bayer format, each pixel has only one of R/G/B values. By CFA interpolation, each pixel will have complete R/G/B values.
Interpolation of G at R/B location
if (DH < DV)
else
Interpolation of G at B location can be done on the corresponding formula of B values.
n-2
n-1
n
Interpolation of G at R/B location
Interpolation of G at B location can be done on the corresponding formula of B values.
n-2
n-1
n
Interpolation of R at B location
Interpolation of B at R location can be done on the corresponding formula of B values.
n-2
n-1
n
Color Correction
Color correction is done using a color matrix and it compensates color deviation due to color filtering and sensing circuits.
RGB Gamma Correction
The transfer function of most CRT displays produces an intensity that is proportional to some power (referred to as gamma) of the signal amplitude. As a result, high-intensity ranges are expanded and low-intensity ranges are compressed.
ITU-R BT.709
SMPTE 240M
Formula 3:RGB range: 0-255; YCbCr range: 0-255
Floating-point:
Fixed-point:
Edge Extraction and Sharpness
Edge extraction is done on Y channel, and will be used for image sharpening. To calculate one pixel, a 3x5 filter is used.
n-2
n-1
n
The edge map (EM) is further modified through a lookup table (EMLUT). EMLUT is constructed as following:
The modified edge map is then added back to the Y channel. That is, Y’ = Y + EMLUT(x).
x1
x2
x3
-x1
-x2
-x3
When –x1 < x ≤ x1, we think x is the noise, not the edge. There are two methods to calculate the Y value in this range: one is Y = Y – m2x, the other is Y = Avg.
DDH = abs(2*Ym,n – Ym,n-1 – Ym, n+1) ;
DDV = abs(2*Ym,n – Ym-1,n – Ym+1,n) ;
DDL = abs(2*Ym,n – Ym-1,n-1 – Ym+1,n+1) ;
DDR = abs(2*Ym,n – Ym-1,n+1 – Ym+1,n-1) ;
Find the minimum of
To avoid over-shoot on edges, The Y’ is further modified as the following:
Find the minimum of the 15 Y’s, denoted by Ymin, and the maximum of the 15 Y’s, denoted by Ymax.
if ( Y’ >= Ymax )
else if ( Y’ < Ymin )
Geometrical image distortions
False Color Suppression
False color suppression is done on U, V channel. Y is not changed, U, V may be changed by false color suppression.
Implementation:
3. edge = abs(edge);
4. get temp by clipping the edge into [edge_min, edge_max];
Where edge_min and edge_max are user-programmable registers.

calculate u_gain and v_gain:
u_gain = K_edge*(edge_max - temp);
v_gain = K_edge*(edge_max - temp);
where K_edge=65536/(edge_max - edge_min) ;
6. do false color suppression on U, V channel when edge is larger than edge_min:
temp_u=(u_gain*(temp_u-128))/65536+128;
temp_v=(v_gain*(temp_v-128))/65536+128;
if edge is less than or equal to edge_min, just output u, v directly.
7. clip u, v into [0, 255] and output.
0
1
edge
edge_max
edge_min
u_gain
v_gain
1. Detection
The shape of histogram is very special in backlight condition. Two peaks lie on the two sides of the histogram: one on the low light side, and the other on the high light side. According to this fact, we may detect the backlight condition.
To detect Backlighting, we need histogram information , the mean brightness of the whole image (maybe weighted), iWinYmean and the mean brightness of 16 windows WinYmean[0] ~ WinYmean[15];
When the following adaptive condition is satisfied, it is regarded as in backlight condition:
if(int_Sum_White_Rate*(iThresHold+int_Sum_Black_Rate)*20>=(BackLight_White_Thres*iThresHold*iThresHold)&&
m_iBkLNeedAddET = FALSE;
When the backlight is detected, and the following adaptive condition is satisfied, it is regarded as needn’t add ET time or sensor gain or digital gain or all:
Brightness
Input:
Y, U, V in the range of [0, 255], as source image
BRIGHTNESS, in the range of [-128, 127], as brightness adjustment
Output:
Implementation:
Y’ = CLIP255(Y + BRIGHTNESS);
Contrast
Input:
Y, U, V in the range of [0, 255], as source image
Output:
Parameters:
Implementation:
Hue
Input:
YUV422 data, Y, U and V are in the range of [0, 255].
Register Hue_sin and Hue_cos are in the range of [-256, 256]
Hue angle, within the range of [-45, 45] for case 2.
Output:
Implementation:
If U and V fall out of [0, 255], clipping operation is applied as following:
Clipping(x) = (x<0) ? 0 : ( (x>255)?255:x )
Saturation
Input:
Output:
Parameter:
Formula:
If U and V falls out of [0, 255], clipping operation is applied as following:
Clipping(x) = (x<0) ? 0 : ( (x>255)?255:x )
-
-
-
-
128
Cr’
Cb’
Y
dy
dy
Y’
Saturation Value [0,1.992x] in 8-bit
Note: Y in [0,255], CbCr in [0,255]. Default value for dy is 128.
Auto Expose
The luminance of the image changes when the environment varies. In order to keep the luminance in the given range, we must adjust exposure time (ET) of the sensor.
If the ET is small, changing ET by one unit (one flickering period) can make the brightness of image change much. In order to make the brightness of image change smoothly, digital global gain is combined with ET adjustment
An image is divided into 16 windows with different weights.
Ymean of the frame and Ymean of the 16 windows are calculated as described below.:
Win1
Win13
Win16
Win15
Win14
Win4
Win3
Win2
Win5
Win8
Win7
Win6
Win9
Win12
Win11
Win10
Ha
Hb
Hc
Hd
He
Va
Vb
Vc
Vd
Ve
Integration time must be integer of bank 1/120 ,2/120….. N/120 ,
Example: if exposure line =1.5 bank=1.5/120 sec
, banding
Auto White Balance
When the white paper moves from one light source to another light source, an image sensor sees different colors under different conditions. Similarly, when a digital camera is moved from outdoors (sunlight) to indoor fluorescent (or incandescent) light conditions, the color in the image shifts.
To correct for light source color-temperature changes, the balance among red, green and blue has to be shifted. This “white balancing” is performed by the algorithm below.
Auto White Balance
The number (Counter) of the pixels which satisfy the following conditions and their sums (Rsum, Gsum and Bsum)) of red, green and blue are calculated along with image data stream.
YbotReg<Y<YtopReg
Abs(I)<ItopReg
Abs(Q)<QtopReg
To deal with the issue of big region of uniform color, we will add the following judgment
if (Counter>=TH_GRAY_COUNT) //there is enough valid gray pixels
To update RGain, BGain to adjust AWB
else //there is not enough gray pixels
Keep the Rgain, BGain of the last frame.
Auto White Balance
The average value (AwbRmean, AwbGmean and AwbBmean) of the tricolor is calculated by the following formulas:
AwbRmean=Rsum/Counter
AwbGmean=Gsum/Counter
AwbBmean=Bsum/Counter
RDiff = abs( AwbRmean-AwbGmean )
BDiff = abs( AwbBmean-AwbGmean )
Based on these differences, Rgain and Bgain will be updated. AWB range control register (AWB threshold1, AWB threshold2, AWB gain step) will control this operation.
Auto White Balance
Rstep = 0 Rstep = 1 Rstep = AwbStep
RgainReg – Rstep ( AwbRmean>AwbGmean)
ISO 12231: 1997
ISO 12233 :
ISO/DIS 12234-1 : Part1 : removable memory reference model
ISO/DIS 12234-2 : Part2 :– TIFF/EP
ISO/CD 15739 : Noise Measurements
ISO/WD 17321 :
ISO 15740 :Picture Transfer Protocol(PTP)

Default value for dy is 128.
Win1Win4Win3Win2
Win5Win8Win7Win6
Win9Win12Win11Win10
Win13Win16Win15Win14
HaHbHcHdHe
Va
Vb
Vc
Vd
Ve

image sensor pipeline

Documents