elaborazioni di immagini per dispositivi mobilebattiato/ei_mobile0708/sensori2(a... · 2020. 3....
TRANSCRIPT
1
STMicroelectronics
Advanced System Technology13 March 2008
Elaborazioni di Immagini per Dispositivi Mobile
Ing. Alessandro Capra
ADVANCED SYSTEM TECHNOLOGY13 March 2008 2
Agenda� Introduction
� Mobile camera Devices
� Pre-processing: Auto Focus, Auto Exposure
� Color interpolation� Color interpolation� Noise management� Noise estimation� Croma management� Codecs still/video� Codecs still/video� Applications� Red eye
2
ADVANCED SYSTEM TECHNOLOGY13 March 2008 3
Miniature camera: inside view
Sensor CMOS
IR Filter
Lens
Lens Holder
Substratus
ADVANCED SYSTEM TECHNOLOGY13 March 2008 4
System: The Yellow Duck
3
ADVANCED SYSTEM TECHNOLOGY13 March 2008 5
EXPOSURE CORRECTION
ADVANCED SYSTEM TECHNOLOGY13 March 2008 6
Exposure
One of main problems affecting image quality comes from improper exposure adjustment.
Most existing techniques to improve tonal quality (i.e. histogram equalization, gray level slicing…) are blind tovisual content of image.
There’s no way to define exactly a correct exposure but ideally most relevant features of the scene should occupymid tone gray levels.
4
ADVANCED SYSTEM TECHNOLOGY13 March 2008 7
CMOS Sensor
Exposure
CFAB
GR
Integrationtime
Analoggain
Digitalgain
EC
WB
Statistic
ADVANCED SYSTEM TECHNOLOGY13 March 2008 8
Exposure
Anti Flickering Filter: reduce the light flickering during the videoacquisition
Leaky Integrator
FinFout
1
)1
1()1(1
)()(
≥
−⋅−+⋅=
KK
tFK
tFtF outinout
5
ADVANCED SYSTEM TECHNOLOGY13 March 2008 9
Visibility Image 1/3
Since extraction of features can be difficult in highly underexposed or overexposed images, the mean gray level of the Y channel is forced to be ≈128.
ORIGINAL CORRECTED
ADVANCED SYSTEM TECHNOLOGY13 March 2008 10
Visibility Image 2/3
Classical EC can be considered as a “roughly” correction. Correction based on relevant zones in most cases produces better results.
FLAT CORRECTION RELEVANT ZONES CORRECTION
6
ADVANCED SYSTEM TECHNOLOGY13 March 2008 11
Computation Of Measures
N
igradtrhF
N
i
)|)((|1∑
==
• N number of pixels in the block • grad(i) output of Laplacian (or Sobel)
3××××3 filter for pixel i• Trh operator discards low values
(noise)
∑
∑
=
=⋅−
= 255
0
255
0
)(
)(
k
k
kH
kHMkC
• M mean gray level of block• H(k) histogram bin for gray value k
Contrast
Focus
-1
4
-1
0
-1
0
0
-1
0
Laplacian filter
ADVANCED SYSTEM TECHNOLOGY13 March 2008 12
Relevant blocks identification
Y channel
FC FC FC FC FC FC FC
FC FC FC FC FC FC FC
FC FC FC FC FC FC FC
FC FC FC FC FC FC FC
FC FC FC FC FC FC FC
FC FC FC FC FC FC FC
FC FC FC FC FC FC FC
FC FC FC FC FC FC FC
Measures on non-overlapping blocks
FC FC FC FC FC FC FC
FC FC FC FC FC FC FC
FC FC FC FC FC FC FC
FC FC FC FC FC FC FC
FC FC FC FC FC FC FC
FC FC FC FC FC FC FC
FC FC FC FC FC FC FC
FC FC FC FC FC FC FC
Relevant blocks identification:VS=a*F+b*C>T (a+b=1)
All measures are normalized in the range [0:1].
7
ADVANCED SYSTEM TECHNOLOGY13 March 2008 13
Relevant regions example
ADVANCED SYSTEM TECHNOLOGY13 March 2008 14
Examples
input output
8
ADVANCED SYSTEM TECHNOLOGY13 March 2008 15
Examples
input output
ADVANCED SYSTEM TECHNOLOGY13 March 2008 16
Skin detection 1/2
• Probabilistic model of distribution of Cr, Cb channels used to assign reliability of skin pixels classification
• Pixels with probability of being skin>T are used by exposure correction
INPUT SCALED SKIN PROBABILITIES
9
ADVANCED SYSTEM TECHNOLOGY13 March 2008 17
Skin detection 2/2
( ) [ ]
−=
−
2
)x(dexp
2
1s|xp
22
1 rr Σ
π
[ ] )x()'x()x(d1
2 µµ Σ rrrrr−−=
−
Skin PDF over Cr,Cb plane can be modeled as a 2D gaussian function. Skin probabilities are well identified on chromatic plane.
Skin probabilities
Possible solutions to reduce complexity:1. Store portions of skin probabilities on 2D LUTS (indexed on Cr, Cb);2. Store bounds (on CrCb plane) of skin region.
vectorechrominanc matrix, covariance xr
∑
rmean vecto metric, or,color vect usMahalanobidxrr
ADVANCED SYSTEM TECHNOLOGY13 March 2008 18
Example
input output
10
ADVANCED SYSTEM TECHNOLOGY13 March 2008 19
Possible speed improvement 1/3
Input imageFeature extraction
Correctioncurve
Output image
Scaled image(i.e. preview,
RGB from BPmacro pixels)
ADVANCED SYSTEM TECHNOLOGY13 March 2008 20
Possible Speed improvement 2/3
INPUT
FULL RESOLUTION
1/4 RESOLUTION
11
ADVANCED SYSTEM TECHNOLOGY13 March 2008 21
Possible Speed improvement 3/3
FULL RESOLUTION
1/4 RESOLUTION
INPUT
ADVANCED SYSTEM TECHNOLOGY13 March 2008 22
Local exposure correctionExposure correction often works poorly especially in presence ofareas of very different illumination.
A better strategy would be applying different corrections in different areas.
INPUT FAILED OUTPUT
12
ADVANCED SYSTEM TECHNOLOGY13 March 2008 23
Local Correction 1/4
INPUT OUTPUT
ADVANCED SYSTEM TECHNOLOGY13 March 2008 24
Local Correction 2/4
input output
13
ADVANCED SYSTEM TECHNOLOGY13 March 2008 25
Local Correction 3/4
input output
ADVANCED SYSTEM TECHNOLOGY13 March 2008 26
Local Correction 4/4
input output
14
ADVANCED SYSTEM TECHNOLOGY13 March 2008 27
Comparison 1
OLD NEW
ADVANCED SYSTEM TECHNOLOGY13 March 2008 28
Comparison 2
OLD NEW
15
ADVANCED SYSTEM TECHNOLOGY13 March 2008 29
Comparison 3
OLD NEW
ADVANCED SYSTEM TECHNOLOGY13 March 2008 30
AUTOFOCUS
16
ADVANCED SYSTEM TECHNOLOGY13 March 2008 31
Optical principles
f1
D1
B1 =+
ADVANCED SYSTEM TECHNOLOGY13 March 2008 32
Optical principles
e= focus error
c= diameter of circle of confusion
g= N·max(c)
17
ADVANCED SYSTEM TECHNOLOGY13 March 2008 33
Optical principles
e= focus error
c= diameter of circle of confusion
g= N·max(c)
ADVANCED SYSTEM TECHNOLOGY13 March 2008 34
Lens: the focus system
Most single-lens reflex cameras use an autofocus method called the "phase detection system." Using a separator lens in autofocus module, this system produces two images from the image information of the subject captured through the lens. It then measures the distance between those two images using a line sensor, and detects defocus amount.
18
ADVANCED SYSTEM TECHNOLOGY13 March 2008 35
Lens: the focus system
Autofocus of compact cameras uses a mechanism called the "contrast detection system." Based on the principle that "in focus = highest contrast," this system analyzes the image information of the subject obtained by an image sensor. Then, by moving the lens, this system seeks the lens position where the image contrast is highest.
ADVANCED SYSTEM TECHNOLOGY13 March 2008 36
AutoFocus: Development� What is the end-users desire?
� To not know that an Auto-Focus algorithm is operating!
� Perfect pictures every time at a push of a button� The same intelligence as you and me!!!
� Why?
� They don’t care
� Often hi-res pictures are viewed later, they do not know the picture is in/out of focus until it is too late
� Latency to snap is the number 1 reason for bad pictures (particularly of children and animals)
� They think it is easy (just like their eyes) …. So do you?
19
ADVANCED SYSTEM TECHNOLOGY13 March 2008 37
AutoFocus
RequirementsRequirements� In-focus positioning related to frequency information
� Stable video focusing
� Object tracking
� Multi-objects multi-focus management
� Fast execution and low complexity
Disturbance ElementsDisturbance Elements� Light condition
� Number of edges
� Noise
� Homogeneous regions
� Motion Blur
ADVANCED SYSTEM TECHNOLOGY13 March 2008 38
Statistics Examples
∑∑
∑∑
=
=
n
x
m
y
n
x
m
y
yxureaplaceMeas
yxureaplaceMeas
),(LAbsoluteL
),(LSquareL 2
0-10
-14-1
0-10
Classical Laplacian:
-10-1
040
-10-1
Diagonal Laplacian
∑∑
∑∑
=
=
n
x
m
y
n
x
m
y
yxureaplaceMeas
yxureaplaceMeas
),(LagonalLAbsoluteDi
),(LonalLSquareDiag 2
20
ADVANCED SYSTEM TECHNOLOGY13 March 2008 39
Focus Measure
Motor Steps
Measure window
140 160 180 200 220 240 2602
2.2
2.4
2.6
2.8
3
3.2
3.4
3.6x 10
7
Step motor
Foc
us M
easu
re Square Laplace Measure
140 160 180 200 220 240 2605
5.1
5.2
5.3
5.4
5.5
5.6
5.7x 10
5
Step motor
Foc
us M
easu
re
Absolute Laplace Measure
Step motor
Foc
us M
easu
re Square Diagonal Laplace Measure
140 160 180 200 220 240 2601.3
1.4
1.5
1.6
1.7
1.8
1.9
2
2.1
2.2
2.3x 10
7
Absolute Diagonal Laplace Measure
Step motorF
ocus
Mea
sure
140 160 180 200 220 240 2604.5
4.55
4.6
4.65
4.7
4.75
4.8
4.85
4.9
4.95
5x 10
5
ADVANCED SYSTEM TECHNOLOGY13 March 2008 40
Advanced AF
BB
BB
FF
FF
FF
BB
BB
�Multi-zone analysis�Tracking�Object detection & Motion Analysis
21
ADVANCED SYSTEM TECHNOLOGY13 March 2008 41
AF Functional View
Actuator
DefCor GStatistics
AE CtrlLoop
AF CtrlLoop
Param
s
ADVANCED SYSTEM TECHNOLOGY13 March 2008 42
Statistics
GZoning HP Filters Accum Weighting Fm
22
ADVANCED SYSTEM TECHNOLOGY13 March 2008 43
AF Control loop
Fm
Do_Nothing
Start End
LoopSupervisor
PeakSearch
LeakyIntegrat.
TrackingDecision
Position Internal Representation
Mapping to Actuator command
Reset
Params
Do_Nothing
Coarse
Fine
Tracking
ADVANCED SYSTEM TECHNOLOGY13 March 2008 44
Actuator Review
1. Electromagnetic Actuators1. Stepper Motors2. Simple Solenoids3. Voice Coil Solenoids
2. Piezoelectric Actuators1. Stacked Piezo Devices2. Bimorphs3. Disk Translators4. Moonie Motors5. Helimorphs6. Oscillating Bimorphs7. Inch Worms8. Ultrasonic disk motors
3. Electrostatic Actuators4. Electrostrictive Actuators5. Magnetostrictive Actuators6. Shape memory alloys7. MEMS Actuators
Available Technologies For electro-mechanical Micro Actuation:
23
ADVANCED SYSTEM TECHNOLOGY13 March 2008 45
Stepper Motor: MaxEmil, ALPS
Largely used in DSC and SRL where actuator size is not relevant. Quick and precise actuator
ADVANCED SYSTEM TECHNOLOGY13 March 2008 46
Piezo motion solution: Miniswys
24
ADVANCED SYSTEM TECHNOLOGY13 March 2008 47
Established technologies: voice coil solution
Type of motor used for most Speakers for Hi-Fi
�Used in Sony-Ericsson K750i camera phone�Proposed by SEMCO (Samsung Electro-Mechanics)
Magnet
Winding / sheath
ADVANCED SYSTEM TECHNOLOGY13 March 2008 48
What is MSM?
� MSM is a voicecoil driven actuator with a twist – it has a static holding force
� Technology developed by Philips Applied Technologies
� Available to Philips High Technology plastics for use in camera modules
� A very clever and simple trick is used…
25
ADVANCED SYSTEM TECHNOLOGY13 March 2008 49
MSM concept
Lens carriage
Module housing
Guidance pole (ferrous)
Coil
Magnet
ADVANCED SYSTEM TECHNOLOGY13 March 2008 50
MSM concept – no current in coil
Magnet attracted to poles – holding force generated by pole/carriage friction
26
ADVANCED SYSTEM TECHNOLOGY13 March 2008 51
MSM concept – pulsed current in coil
ADVANCED SYSTEM TECHNOLOGY13 March 2008 52
1Limited actuator� Actuator is based on piezo in 2 or more layers
� By putting a voltage on the layers the different expansion of the layers can cause a bend
� 1Limited use a patented shape, the helimorph, to magnify the deflection to make it larger than a simple bender
27
ADVANCED SYSTEM TECHNOLOGY13 March 2008 53
1Limited piezo design
ADVANCED SYSTEM TECHNOLOGY13 March 2008 54
Electro-wetting: Varioptic
Varioptic lens technology
Innovative solution. Low cost and high miniaturisation. Still under development to improve speed, HQ optical response, mechanical resilience.
28
ADVANCED SYSTEM TECHNOLOGY13 March 2008 55
DGITAL AUTOFOCUS
ADVANCED SYSTEM TECHNOLOGY13 March 2008 56
CDM
29
ADVANCED SYSTEM TECHNOLOGY13 March 2008 57
ADVANCED SYSTEM TECHNOLOGY13 March 2008 58
� Conventional concept
� Eyesquad
Object Best lens ImageSensors Processing
ObjectDistorted lens
ImageSensorsEyesquad’s
Processing
Both parts are inter-related
30
ADVANCED SYSTEM TECHNOLOGY13 March 2008 59
ADVANCED SYSTEM TECHNOLOGY13 March 2008 60
Q&A
31
ADVANCED SYSTEM TECHNOLOGY13 March 2008 61
Bibliography
� http://www.dpreview.com� http://www.nikon.co.jp/main/eng/portfolio/about/technology/index.htm� http://www.canon.com/technology/index.html� http://www.kodak.com� http://www.learningcenter.sony.us� http://www.st.com/stonline/products/families/imaging/imaging.htm� http://www.samsung.com� http://www.philips.com
� http://www.trenholm.org/hmmerk/index.html
� Image Sensor Architectures for Digital Cinematography, DALSA Digital Cinema� Active Pixel Sensor (Aps) Design - From Pixels To Systems, Orly Yadid-Pecht� CCD vs CMOS, Nicolas Blanc� An advanced video camera system with robust AF, AE, and AWB control, IEEE� Selecting the Optimal Focus Measure for Autofocusing and Depth-From-Focus,
Murali Subbarao and Jenn-Kwei Tyan