framework for digital camera in linux
DESCRIPTION
Framework for digital camera in Linux. Dongsoo Kim ( [email protected] ) Heungjun Kim( [email protected] ) Samsung Electronics. Contents. Introducing mobile camera devices SoC camera subsystem Mobile camera module devices Framework for digital camera - PowerPoint PPT PresentationTRANSCRIPT
![Page 2: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/2.jpg)
2
Introducing mobile camera devices- SoC camera subsystem
- Mobile camera module devices
Framework for digital camera- Video4Linux2 and camera device
- New APIs for high-end mobile camera devices
- Dual camera framework for 3G handset
Porting issues in SoC camera subsystem
![Page 3: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/3.jpg)
3
AP : Application Processor
Camera interface : interface device which is syncing image data from external camera device and gives some commands to them
Camera module : external camera device packaged with lens, sensor and ISP on it
ISP : Image Signal Processor. Also supports lots of additional functions
V4L2 : Video for Linux 2
![Page 4: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/4.jpg)
4
This document contains some animations.
Slideshow mode is recommended
![Page 5: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/5.jpg)
5
Exposure control- Aperture (Iris)
- Shutter
- ISO Lens control- Focus
- Zoom
Strobe support- Various program
mode : slow sync, red-eye reduction and on….
![Page 6: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/6.jpg)
6
Introducing mobile camera devices
![Page 7: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/7.jpg)
7
History of camera phone
First Camera Phone in 2000 with 3.5MP
1.3MP in 20032MP in 20043MP in20045MP in20047MP in 200510MP in 200612MP in 2009
![Page 8: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/8.jpg)
8
Old and low-end mobile camera modules
Feature Low end camera
Exposure
Aperture Not supported
Shutter Just change frame rate
ISO Low sensitivity & High noise level
Lens
Focus Pan focus
Zoom Digital zoom
Strobe Just LED light most of time
![Page 9: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/9.jpg)
9
Brand new and high-end mobile camera modulesFeature High end camera
Exposure
Aperture Supported with lens packaging
Shutter Supported with lens packaging
ISO High sensitivity & Low noise level
Lens
Focus Movable focus point (Auto/manual)
Zoom Optical zoom & Digital zoom
Strobe XENON flash is also supported
![Page 10: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/10.jpg)
10
What is SoC camera subsystem?
ARM SoC solution
ARMCore
Peripheral1(Display)
Peripheral2 (MMC)
Peripheral3 (USB)Camera interface
Peripherals
External
Camera module
External
Camera module
![Page 11: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/11.jpg)
11
Camera interface
Buffer handlingBuffer
handling
Resolution
handling
Resolution
handling
Data format handling
(YUV/RAW/JPEG…)
Data format handling
(YUV/RAW/JPEG…)
Camera module
I2CI2C
Exposure
control
Exposure
control
Lenscontr
ol
Lenscontr
ol
Strobe
control
Strobe
control
White balanc
e control
White balanc
e control
Effect contr
ol
Effect contr
ol
Facedetec
t
Facedetec
t
Zoomcontr
ol
Zoomcontr
ol
.
.
.
.
.
.
CommandinterfaceCommandinterface
RAW DATA
Image data in expected formatRequest for data
![Page 12: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/12.jpg)
12
Various ways of preview (Live view) in SoC camera subsystems
CameraModule
Memory
CameraInterfa
ce
ARMcore
Frame buffer
OKCanc
el
User Applicati
onCOPY!
DATA
DA
TA
![Page 13: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/13.jpg)
13
Various ways of preview (Live view) in SoC camera subsystems
CameraModule
Memory
CameraInterfa
ce
ARMcore
Frame buffer
OKCanc
el
User Applicati
on
COPY!
DMA Through
DMA
DATA
DA
TA
Saveas file
![Page 14: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/14.jpg)
14
How to capture still shots in SoC camera subsystem- Your camera module has a dedicated JPEG encoder
on it
Memory
CameraInterfa
ce
ARMcore
User Applicati
onCOPY!
DATAMEMORY
Camera module
RAWDATA
PROCESS
JPEGPROCES
S
DATA
DA
TA
Save as JPEG file
![Page 15: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/15.jpg)
15
How to capture still shots in SoC camera subsystem- No JPEG encoder in your camera module
CameraModule
Memory
CameraInterfa
ce
ARMcore
User Applicati
onCOPY!
DATA
DA
TA
RAWDATA
Codec S/W
Save as JPEG file
![Page 16: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/16.jpg)
16
How to take motion pictures in SoC camera subsystem
CameraModule
Memory
CameraInterfa
ce
ARMcore
Frame buffer
OKCanc
el
User Applicati
on
COPY!
DATA
DA
TA
COPY!
Codec S/W
Save as…
![Page 17: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/17.jpg)
17
In Kernel
V4L2 (AKA Video For Linux 2)- The second version of the Video For Linux API
- Kernel interface for analog radio and video capture and output drivers
In user space
Multimedia middleware
Gstreamer- Pipeline based multimedia framework written in the C programming
language with the type system based on GObject
OpenMAX- Cross-platform set of C-language programming interfaces that provides
abstractions for routines especially useful for audio, video, and still images
![Page 18: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/18.jpg)
18
Is V4L2 enough for mobile
digital camera modules?
No way
![Page 19: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/19.jpg)
19
Framework for digital camera
![Page 20: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/20.jpg)
20
Camera subsystem in Linux kernel aspect
Camera interfaceCamera interface
Buffer handlingBuffer
handlingResolution handling
Resolution handling
Data format handling
(YUV/RAW/JPEG…)
Data format handling
(YUV/RAW/JPEG…)
CommandinterfaceCommandinterface
V4L2 driverV4L2 driver
Camera module
Exposure
control
Exposure
control
Lenscontro
l
Lenscontro
l
Strobecontro
l
Strobecontro
l
White balance control
White balance control
Effect contro
l
Effect contro
l
FacedetectFace
detect
Zoomcontro
l
Zoomcontro
l
.
.
.
.
.
.
V4L2 driverV4L2 driver
videobufvideobuf-core
videobuf-sg-dma
videodevV4l2-dev
V4l2-ioctl
V4l2-subdev(new)
V4l2-device
V4l2-int-
device(old)
V4L2
![Page 21: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/21.jpg)
21
How does it work with V4L2?
Open deviceOpen
device
VIDIOC_QUERYCAP
VIDIOC_QUERYCAP
Application or Middleware
VIDIOC_S_INPUT
VIDIOC_S_INPUT
VIDIOC_REQBUF
VIDIOC_REQBUF
VIDIOC_QBUF
VIDIOC_QBUF
VIDIOC_STREAMON
VIDIOC_STREAMON
VIDIOC_QBUFVIDIOC_QBUF VIDIOC_DQBUF
VIDIOC_DQBUF
VIDIOC_QUERYBUF
VIDIOC_QUERYBUF
Copy memory to userspace
![Page 22: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/22.jpg)
22
New V4L2 APIs for digital camera
Exposure control
![Page 23: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/23.jpg)
23
Need for detailed control in exposure- We’ve got camera devices with mechanical shutter
- Iris could be handled also
- ISO controlled output is somehow useful
Why not using enhanced exposure control like a regular digital camera?
![Page 24: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/24.jpg)
24
What we’ve got in mainline V4L2 right now?
#define V4L2_CID_EXPOSURE_AUTO (V4L2_CID_CAMERA_CLASS_BASE+1)enum v4l2_exposure_auto_type { V4L2_EXPOSURE_AUTO = 0, V4L2_EXPOSURE_MANUAL = 1, V4L2_EXPOSURE_SHUTTER_PRIORITY = 2, V4L2_EXPOSURE_APERTURE_PRIORITY = 3};#define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2)#define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3)
No way to control iris
No way to control exposure metering method
![Page 25: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/25.jpg)
25
Additional exposure control
enum v4l2_exposure_auto_type {V4L2_EXPOSURE_AUTO = 0,V4L2_EXPOSURE_MANUAL = 1,V4L2_EXPOSURE_SHUTTER_PRIORITY = 2,
V4L2_EXPOSURE_APERTURE_PRIORITY = 3,+ /* Additional features for digital camera */+ V4L2_EXPOSURE_ISO_PRIORITY =4,};#define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2)#define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3)
. . .
+#define V4L2_CID_EXPOSURE_LOCK (V4L2_CID_CAMERA_CLASS_BASE+18)
![Page 26: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/26.jpg)
26
Supported aperture stages, shutter speed, and ISO speed could be different between every different camera module products
+#define V4L2_CID_CAM_APERTURE(V4L2_CID_CAMERA_CLASS_BASE+19)+#define V4L2_CID_CAM_SHUTTER (V4L2_CID_CAMERA_CLASS_BASE+20)+#define V4L2_CID_CAM_ISO (V4L2_CID_CAMERA_CLASS_BASE+21)
Supported values for aperture, shutter and ISO should be made in V4L2_CTRL_TYPE_MENU
{ .id = V4L2_CID_CAM_APERTURE, .type = V4L2_CTRL_TYPE_MENU, .name = "Aperture", .minimum = 0, .maximum = 4, .step = 1, .default_value = 0,},
static const char *camera_iris_stages[] = {/* This module supports 5 Iris stages * on it's own but F number depends * how you package Lens module */
"2.8“, "4“, "5.6“, "8“, "11“, NULL};
![Page 27: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/27.jpg)
27
New exposure metering control
+/* Exposure Methods */+#define V4L2_CID_PHOTOMETRY (V4L2_CID_CAMERA_CLASS_BASE+17)+enum v4l2_photometry_mode {+ V4L2_PHOTOMETRY_MULTISEG = 0, + V4L2_PHOTOMETRY_CWA = 1,+ V4L2_PHOTOMETRY_SPOT = 2,+ V4L2_PHOTOMETRY_AFSPOT = 3, +};
![Page 28: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/28.jpg)
28
New V4L2 APIs for digital camera
Lens control
![Page 29: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/29.jpg)
29
Need for enhanced Focus mode API
- Current V4L2 framework just slightly covers MANUAL & AUTO FOCUS mode for usb webcam
- High-end camera modules support for various focus mode presets like
•AF Macro
•AF-S / AF-C
For compatibilityFocus control API should be defined
![Page 30: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/30.jpg)
30
Focus Mode & Enumeration values
#define V4L2_CID_FOCUS_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+10)#define V4L2_CID_FOCUS_RELATIVE (V4L2_CID_CAMERA_CLASS_BASE+11)+#define V4L2_CID_FOCUS_MODE (V4L2_CID_CAMERA_CLASS_BASE+13)
+/* Focus Methods */+enum v4l2_focus_mode {+ V4L2_FOCUS_AUTO = 0,+ V4L2_FOCUS_MANUAL = 1,+ V4L2_FOCUS_MACRO = 2,+ V4L2_FOCUS_CONTINUOUS = 3,+};
![Page 31: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/31.jpg)
31
New V4L2 APIs for digital camera
Object recognition
![Page 32: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/32.jpg)
32
Object detection
Gee!It recognized my
face!Hello everyone
![Page 33: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/33.jpg)
33
What do we set?- Detect mode
- Type of object
- How many objects to detect
- Triggered action
What do we get?- How many objects are detected
- Detected object’s type
- Detected object’s coordinates
![Page 34: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/34.jpg)
34
New API for object recognition- New capability
#define V4L2_CAP_ASYNCIO 0x02000000 /* async I/O */#define V4L2_CAP_STREAMING 0x04000000 /* streaming I/O ioctls */
+#define V4L2_CAP_OBJ_RECOGNITION 0x10000000
+#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect)+#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)
- New IOCTL
![Page 35: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/35.jpg)
35
+/* Object detection and triggered actions */+enum v4l2_recog_mode {+ V4L2_RECOGNITION_MODE_OFF = 0,+ V4L2_RECOGNITION_MODE_ON = 1,+ V4L2_RECOGNITION_MODE_LOCK = 2,+};++enum v4l2_recog_action {+ V4L2_RECOG_ACTION_NONE = 0,/* only detection */+ V4L2_RECOG_ACTION_BLINK = 1,/* Capture on blinking */+ V4L2_RECOG_ACTION_SMILE = 2,/* Capture on smiling */+};++enum v4l2_recog_pattern {+ V4L2_RECOG_PATTERN_FACE = 0; /* Face */+ V4L2_RECOG_PATTERN_HUMAN = 1; /* Human */+ V4L2_RECOG_PATTERN_CHAR = 2; /* Character */+}
New API for object detection
![Page 36: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/36.jpg)
36
+struct v4l2_recog_rect {+ enum v4l2_recog_pattern p; /* What pattern detected */+ struct v4l2_rect o; /* detected area */+ __u32 reserved[4];+}++struct v4l2_recognition_data {+ __u32 detect_cnt; /* detected object counter */+ struct v4l2_recog_rect obj; /* detected area */+ __u32 reserved[4];+};++struct v4l2_recognition {+ enum v4l2_recog_mode mode;+ enum v4l2_recog_pattern pattern; /* What pattern to detect */+ __u32 obj_num; /* How many object to detect */+ __u32 detect_idx; /* select detected object */+ struct v4l2_recog_data data; /* read only */+ enum v4l2_recognition_action action;+ __u32 reserved[4];+};
New API for object detection
![Page 37: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/37.jpg)
37
What do we set?- Detect mode
- How many objects to detect
- Triggered action
What do we get?- How many objects are detected
- Detected object’s coordinates
![Page 38: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/38.jpg)
38
New API for object recognition- New capability
#define V4L2_CAP_ASYNCIO 0x02000000 /* async I/O */#define V4L2_CAP_STREAMING 0x04000000 /* streaming I/O ioctls */
+#define V4L2_CAP_OBJ_RECOGNITION 0x10000000
+#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect)+#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)
- New IOCTL
![Page 39: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/39.jpg)
39
+/* Object detection and triggered actions */+enum v4l2_recognition_mode {+ V4L2_RECOGNITION_MODE_OFF = 0,+ V4L2_RECOGNITION_MODE_ON = 1,+ V4L2_RECOGNITION_MODE_LOCK = 2,+};++enum v4l2_recognition_action {+ V4L2_RECOG_ACTION_NONE = 0,/* only detection */+ V4L2_RECOG_ACTION_BLINK = 1,/* Capture on blinking */+ V4L2_RECOG_ACTION_SMILE = 2,/* Capture on smiling */+};++struct v4l2_recognition_data {+ __u8 detect_cnt; /* detected object counter */+ struct v4l2_rect o; /* detected area */+};++struct v4l2_recognition {+ enum v4l2_recognition_mode mode;+ __u8 obj_num; /* How many object to detect */+ __u8 detect_idx; /* select detected object */+ struct v4l2_recognition_data data; /* read only */+ enum v4l2_recognition_action action;+};
New API for object detection
![Page 40: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/40.jpg)
40
New V4L2 APIs for digital camera
Dual camera
![Page 41: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/41.jpg)
41
What is dual camera?
Dual camera applications- Video telephony
- Camera applications• Taking self portrait shots
Flip side camera: for still shots
Front side camera: for special purpose
![Page 42: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/42.jpg)
42
H/W restrictions in Application Processor’s camera peripheral interface
Megacamera
VGAcamera
Datapath
![Page 43: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/43.jpg)
43
H/W restrictions in Application Processor’s camera peripheral interface
“SINGLE” camera interfaceon Application Processor !“SINGLE” camera interfaceon Application Processor !
Only one camera could be using data pins at a time
Only one camera could be using data pins at a time
Each camera is different inEach camera is different in
Working resolution
(SYNC)
Working resolution
(SYNC)Working MCLKWorking MCLK Working PCLKWorking PCLK
Only one camera could be handled at a timeOnly one camera could be handled at a time
![Page 44: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/44.jpg)
44
We already have API for dual camera- Enumerate video inputs
- Query or select the current video input
VIDIOC_ENUMINPUTVIDIOC_ENUMINPUT
VIDIOC_G_INPUTVIDIOC_G_INPUT
VIDIOC_S_INPUTVIDIOC_S_INPUT
![Page 45: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/45.jpg)
45
What do we need to consider?
•Context of each camera module•Capability : Supported color spaces, resolutions, additional functions…•Latest working status : resolution, color space, effect and so on
•Context of each camera module•Capability : Supported color spaces, resolutions, additional functions…•Latest working status : resolution, color space, effect and so on
•Context of camera interface•Working information of the camera module getting switched : Clock information and so on
•Context of camera interface•Working information of the camera module getting switched : Clock information and so on
•Let user space application get noticed about camera input devices
•Don’t forget VIDIOC_ENUMINPUT•Check out supported pixel format using VIDIOC_ENUM_FMT and VIDIOC_ENUM_FRAMESIZES
•Let user space application get noticed about camera input devices
•Don’t forget VIDIOC_ENUMINPUT•Check out supported pixel format using VIDIOC_ENUM_FMT and VIDIOC_ENUM_FRAMESIZES
![Page 46: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/46.jpg)
46
How to switch active camera
Condition 2Shared things
Camera power enable pinCamera data pinsMCLK
Condition 3Dedicated things
Camera enable pinsCamera reset pins
Condition 1How to enable camera module:
Give power sourceEnable expecting cameraGive MCLK CAM
POWERMCLK DATA PINS
MegaEnable
MegaReset
VGAEnable
VGAReset
Mega camera VGA camera
CPU
![Page 47: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/47.jpg)
47
STREAMOFF
Camera (A)
initialize
Camera (A)
initialize
Start preview
Start preview
Switch to camera (B)
Camera (B)
initialize
Camera (B)
initialize
Camera interface reset
& setting
Start preview
Start preview
Reset camera interface’s
current setting
Make a proper MCLK for camera
(B)
Setup resolution,
color-space, DMA
Stop previewStop DMA
Cam power off
Camera (B) Enable pin on
Camera (A) enable pin off
Campower on
Stop previe
w
Stop previe
w
STREAMON
G_INPUT/
S_INPUT
G_INPUT/
S_INPUT
![Page 48: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/48.jpg)
48
Porting issues in SoC camera subsystem
![Page 49: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/49.jpg)
49
I2C protocol has start and stop condition
Some camera modules generate unexpected noise until their power up sequence- Makes I2C bus in bus busy state
![Page 50: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/50.jpg)
50
How to prevent this malfunctioning issue
Use a “noise free on power up” camera module
Use a level shifter to block and ignore noise from camera module
Change functionality of I2C pins while turning on camera module
OR
OR
![Page 51: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/51.jpg)
51
All about between camera interface and camera module- Expected resolution from camera interface
- Served resolution from camera module
- Symptoms depend on camera interface
Expecting resolution is bigger than served resolution- A “select time out” error in user space application or
- Preview image is floating in some direction
Serving resolution is bigger than expected resolution- A “select time out” error in user space application or
- Preview angle looks like to be zoomed in
![Page 52: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/52.jpg)
52
Color order mismatched between camera interface and camera module
![Page 53: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/53.jpg)
53
Preview freezing while checking lens response- Using while() loop while it checks response, preview will
freeze until focusing job gets finished
- If the ISP driver use the kernel thread when it check response from current lens status could make smooth preview on the LCD while lens is on focusing.
![Page 54: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/54.jpg)
54
Solving preview freezing while checking lens statusstruct platform_isp {
const struct platform_data *pdata;struct v4l2_int_device *v4l2_int_device;struct i2c_client *i2c_client;int reschk;
. . . . . . . . .}
static int ioctl_s_ctrl(struct v4l2_int_device *s,struct v4l2_control *vc)
{ struct platform_isp *isp = s->priv; struct i2c_client *client = isp->i2c_client; switch (vc->id) { case V4L2_CID_FOCUS_AUTO: isp->vfm = vc->value; if (vc->value == V4L2_CID_FOCUS_MACRO) { err = i2c_command_for_isp( client, focus_macro ); isp->reschk = 1; } break; }}
If isp->reschk is Set, The kernel thread is active.
![Page 55: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/55.jpg)
55
Solving preview freezing while checking lens statusint pseudo_reschk_thread (void *data){ int err = 0;……… do { switch (isp->reschk) { case 1: err = i2c_command_for_isp(client, focus_check); if (err < 0) { isp->reschk = 1; /* i2c command function failed. */ } else { isp->reschk = 0; /* if succeed, reschk ended. */ } break;…… } } while(1); return 0;}
This function return when the ISP ismatched the focus(+ return) or timeout(- return).
This is able to re-check to the next check time.
Platform Data Vairable for isp, or static variable.
![Page 56: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/56.jpg)
56
JPEG has no synchronization protocol when it gets transferred
All about camera interface and camera module’s data synchronization
- When camera interface supports JPEG sync• Follow user manual and make camera module to follow that
protocol
- When camera interface does not support JPEG sync• Make synchronization protocol: make JPEG data to be
synchronized with VSYNC
![Page 57: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/57.jpg)
57
Q&A
![Page 58: Framework for digital camera in Linux](https://reader033.vdocuments.us/reader033/viewer/2022051214/56812cf0550346895d91bcb1/html5/thumbnails/58.jpg)
58
Thank you