website : http : // www.smit.com.cn
Image Sensor Pipeline
CCD & CMOS Module
website : http : // www.smit.com.cn
CCD ISP
CMOS ISP
VD
CCD module Video Decoder
CMOS SOC
website : http : // www.smit.com.cn
CMOSCCD
Color SpaceColor transformResize色彩调整测试设备与人员国家图像质量标准新技术点
CCDInterlaceDeinterlaceDenosie色彩增强
CMOSLens(very important,but no mention)Bad pixelInterpolationGammaBacklightingColor Correction3AvignettingEdge Extraction and SharpnessDetection and CompensationAnti FlickerTemporal Noise ReductiondistortionsDeblur(no mention)
website : http : // www.smit.com.cn
Outline• Sensor Performance • Color Space • Color Transform• Interlace & deinterlace• Denoise&Deblur• Scale & Crop• New scheme
website : http : // www.smit.com.cn
典型的安防市场用 CCD 模组
典型的安防市场用 CMOS 模组
website : http : // www.smit.com.cn
Sensor Performance Summary• Sensor Pixel Size• Read Noise• AD Converter• Signal-to-Noise Ratio• Full Well Capacity• Dynamic Range• Unity Gain• Low Light Sensitivity Factor• Noise Sources• Pixel Density
website : http : // www.smit.com.cn
Sensor Size
Typical sensor size of 3, 4, and 5 megapixel digital compact cameras
Typical sensor size of 6 megapixel digital SLRs
背景是 35mm 标准胶卷
website : http : // www.smit.com.cn
Sensor Pixel Size
Full Well Capacity
website : http : // www.smit.com.cn
Color Array
The New Foveon Sensors
CFA Bayer
website : http : // www.smit.com.cn
Super CMOS SensorNormally sensor pixels are square. Fujifilm's Super CCD sensors have octagonal pixels, as shown in this diagram. Therefore, the distance "d2" between the centers of two octagonal pixels is smaller than the distance "d1" between twoconventional square pixels, resulting in larger (better) pixels.
website : http : // www.smit.com.cn
However, the information has to be converted to a digital image with square pixels. From the diagram you can see that, for a 4 x 4 area of 16 square pixels, only 8 octagonal pixel measurements were used: 2 red pixels, 2 blue pixels, and 4 greenpixels (1 full, 4 half, and 4 quarter green pixels). In other words, 6 megapixel Super CCD images are based on the measurement by only 3 million effective pixels,similar to the above interpolated example, but with the advantage of larger pixels. In practice the resulting image quality is equivalent to about 4 megapixel. The drawback is that you have to deal with double the file size (leading to more storage and slower processing), while enjoying a quality improvement equivalent to only 33% more pixels.
简单的说,就是同样感光面积的的 Sensor ,能提供约多 44% 的有效像素,但是它需要多处理 1 倍大小的图像。得不偿失?????????
website : http : // www.smit.com.cn
Sensitivity (ISO)
ISO 100 - Red Channel ISO 800ISO 100 ISO 800 - Red Channel
Increase sensitivity-amplified -Noise increase
website : http : // www.smit.com.cn
Sensor Linearity
Sensors respond in a linear way (*) to light while human vision responds in a non-linear way which is often approximated by a power curve of about 0.45. So what the sensor measures as "127" is perceived by the human vision as about "186".
website : http : // www.smit.com.cn
Sensor Linearity
这就是需要做 gamma 校正的原因,当然实际的校正曲线往往是 S 型的曲线这是因为为显示设备压缩动态范围的需要。
website : http : // www.smit.com.cn
AD Converter
Sensor Dynamic Range ADC Type Image Tonal Range
Low (e.g. around 256:1)8 bit 8 bit
10 or 12 bit 8+ bit (*)
High (e.g. around 4,000:1)
8 bit 8 bit
10 or 12 bit 10 or 12 bit if RAW
website : http : // www.smit.com.cn
Dynamic RangeWell Capacity (electrons) / Read Noise (electrons),
理想 Dynamic Range = Full Sensor ,在完全黑暗中,读出应该为 0.根据 sensor 的噪声模型,最小信号为 1 时,噪声为 1.假设动态范围为 1000 ,那么需要用 2^12bits 的 AD 表示这颗 sensor 。另外: sensor 的放大器,和 AD 会带来噪声,所以在不同的 ISO 值会有不同的噪声叠加值。所以,不同的 ISO 情况下,有不同的动态范围。
所以 read noise, full well capacity, and dynamic range. 三者关系复杂
Photons
Noise
Signal-to-noiseRatio
9 3 3
100 10 10
900 30 30
10000 100 100
40000 200 200
90000 300 300
Table 1a
Poisson Statistics
website : http : // www.smit.com.cn
High Dynamic Range影响动态范围的参数:1.噪声大小(越小越好)2.数据位宽(越宽越好)3.Pixel 物理面积大小(越大越好)
说明: Pixel 物理尺寸越大, HDR 性能越好AD 位宽越高,性能越好
说明:在高 ISO 时, HDR 性能急速下降,因为 ISO 增加, AD 增益加大,噪声增加。
website : http : // www.smit.com.cn
The More Pixels, the Worse the Image!
"As an independent test lab for digital cameras, we have observed a decrease in the image quality of digital cameras during the last 3 years,“ saysImage Engineering.www.image-engineering.de
Newer cameras have an increasingly worse image quality and the reason for that is obvious:"The more pixels a camera has, the better it is” was true in the beginning of digital photography when compact cameras had a VGA resolution (640 x 480)
When the pixel count exceeded 6 Megapixels in 2004, this was no longer the case but consumers still buy cameras with the highest pixel count.
website : http : // www.smit.com.cn
Color Space
•YUV
•YIQ
•YCbCr
•HSV
•HSB
•RGB
•Lab
•CMYK
luma/chroma(TV) Hue/Saturation Mixtrue
website : http : // www.smit.com.cn
BT656,PAL/SECAMATV
TV Color Space
A B
To Be Created
DBT601SDTV,BT1120
BT709HDTVDTV
C
YUVYIQ
YCbCr
NTSCATV
website : http : // www.smit.com.cn
YUV
Y' value of 0
Y' value of 0.5
Y' value of 1
Note :From http://en.wikipedia.org/wiki/YCbCr#ITU-R_BT.601_conversion
YUV YIQ
website : http : // www.smit.com.cn
YCbCr
Y' value of 0
Y' value of 0.5
Y' value of 1
Note :From http://en.wikipedia.org/wiki/YCbCr#ITU-R_BT.601_conversion
website : http : // www.smit.com.cn
RGB/CMY(K)RGB, 表示范围大,适用主动发光物体色彩空间,如显示器。
CMKY ,表示范围小,使用与被动显示物体,如打印相片
website : http : // www.smit.com.cn
RGB/HSB(HSV)RGB, 非常适合产生 R,G,B 分量色彩。
RGB, 非常适合调整饱和度,亮度,色度 。
website : http : // www.smit.com.cn
Color TransformColor Transform :基本上任何一款视频处理芯片都需要。YUV-RGB ,注意处理精度问题HSB-RGB ,算法复杂度,色彩损失
website : http : // www.smit.com.cn
YUV Transform
YUV 的色彩排列很重要,每颗芯片可能有自己定义的数据输入格式,这些格式在FOURCC 中定义。 YUV 转换涉及到 DownSampling 和Upsampling 技术
YUYV
YVYU
YV12Y655
Y41P
GREY
http://www.codecs.onerivermedia.com
website : http : // www.smit.com.cn
RGB Transform
RGB 的数据格式定义:主要是某些数码相机中,输出的RGB 格式排列有不同。
RGBA
RAW
BI_RGBRGB
BI_RLE8
RLE
http://www.codecs.onerivermedia.com
website : http : // www.smit.com.cn
Q&A about Color Space &
Transform
website : http : // www.smit.com.cn
Interlace & Deinterlace
• What is interlace?Is interlace a bug? Why interlace ?• How to Deinterlace?• Telecine
website : http : // www.smit.com.cn
AD. :Interlace! Since 1930s!
website : http : // www.smit.com.cn
From : http://en.wikipedia.org/wiki/Deinterlacing
Resize
Deinterlace
This process of dividing frames into half-resolution fields at double the frame rate is known as interlacing.
website : http : // www.smit.com.cn
( 1 ) Clever processing for TV system Save half bandwidth.(Data compress)
( 2 ) Most modern displays, such as LCD, DLP and plasma displays, are not able to work in interlaced mode, because they are fixed-resoution displays and only support progressive scanning. (Bad luck to ATV)
( 3 ) In order to display interlaced signal on such displays, the two interlaced fields must be converted to one progressive frame with a process known as de-interlacing.
Why interlace
website : http : // www.smit.com.cn
How to deinterlace
Deinterlace
3D deinterlace• Memory cousume
复杂,效果最好复杂,效果最好
Blend(to blur)• Area Base
• To blur the mice teeth• Discard
Motion CompensationMotion detection
Weave(do nothing)Field combination
deinterlacing
BOBDisplay every fields by
50FPSField extension deinterlacing
• Lose nothing
Note:None of them is perfect!!!!!!!!!!!(The main argument is that no matter how complex the deinterlacing algorithm may be, the artifacts in the interlaced signal cannot be completely eliminated because some information is lost between frames.)
website : http : // www.smit.com.cn
When is deinterlace bad?
( 2 ) Film to TV(just de telecine)
( 1 ) when video source is not interlaced
website : http : // www.smit.com.cn
telecine
FilmTV (telecine)
Interlaced video can carry progressive scan signal, and deinterlacing process should consider this as well. (More unfortunately)
24Fps25Fps on PAL TV system 24Fps59.94Fps on NTSC SECAM
system
website : http : // www.smit.com.cn
2:2 2:3
website : http : // www.smit.com.cn
Does deinterlacing work?
• Interlace will be disappeared in future!
• The European Broadcasting Union has argued against the use of interlaced video in production and broadcasting
website : http : // www.smit.com.cn
Q&A about Deinterlace
website : http : // www.smit.com.cn
Scale & Crop
How to Scale?,( 等比,非等比 )How to Crop?Rotation? (no need)
website : http : // www.smit.com.cn
ResizeResizeScale
Crop
website : http : // www.smit.com.cn
原图
近邻取样缩放到 0.6 倍 近邻取样缩放到 1.6 倍
二次线性插值缩放到 0.6 倍 二次线性插值缩放到 1.6 倍
三次卷积插值缩放到 0.6 倍 三次卷积插值缩放到 1.6 倍
Scale
website : http : // www.smit.com.cn
原图
近邻取样缩放到 8 倍 二次线性插值缩放到 8 倍
三次卷积插值缩放到 8 倍 二次线性插值 ( 近似公式 )
Scale
website : http : // www.smit.com.cn
Crop
website : http : // www.smit.com.cn
Denoise & Deblur
website : http : // www.smit.com.cn
• 背景噪声,白噪声• TBD
website : http : // www.smit.com.cn
Video Enhancement
website : http : // www.smit.com.cn
• Luminance peaking ,补偿信道过程中高频损失,主要在 YUV 域中处理 Y 分量
• 数字亮度瞬态增强技术 (DLTl ) (sharpness)• 黑电平扩展,只处理 Y• 白电平扩展,只处理 Y• 对比度,亮度,处理 Y• 色度,饱和度调整 , 之处理 UV• 肤色校正
website : http : // www.smit.com.cn
参考 PW328
website : http : // www.smit.com.cn
website : http : // www.smit.com.cn
肤色校正原理,肤色在 YUV 空间中分布范围:
正负 20 度之间
调整方法:
website : http : // www.smit.com.cn
YCbcr444
Luminance Peaking
YCbcr444 YCbcr444
DLT1
BLE/WLE
YCbcr444
BLE/WLE
BLUE leGamma
YCbcr444->420/422
website : http : // www.smit.com.cn
CMOS Sensor Image Pre_processing
ISP
website : http : // www.smit.com.cn
New Scheme
website : http : // www.smit.com.cn
• ROI• SVC/MVC/3D• Panorama Image(1.3M=4D1)• HDR
website : http : // www.smit.com.cn
谢 谢!