cameras in embedded systems: device tree and acpi view · cameras in embedded systems: device tree...
TRANSCRIPT
![Page 1: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/1.jpg)
Cameras in embedded systems: Device tree and ACPI view
Sakari Ailus - Intel
![Page 2: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/2.jpg)
A typical embedded system with a camera
SoC
cameramodule
lens voice coil
sensor
lens voice coil
lens
ISPCSI-2
I²CI²C
GPIO
regulator
reset
vana
vdig
clock
● Image Signal Processor● Raw camera sensor● Lens voice coil
![Page 3: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/3.jpg)
Raw sensors
● Raw sensors have little processing logic in the sensor itself– Analogue and digital gain but not much more
This is how white looks like! -->
![Page 4: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/4.jpg)
Image signal processors
● Process the imagefor viewing
After ISP processing whitelooks like this --->
![Page 5: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/5.jpg)
Video4Linux and Media controller
● Video4Linux (V4L2) is the Linux API for capturing images– Video capture cards
– USB webcams
– Cameras in embedded devices
● Media controller is a control interface for complex media devices– Image pipeline discovery and configuration– Device discovery
![Page 6: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/6.jpg)
Example of amedia graph
ISP
sens
or
![Page 7: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/7.jpg)
Probing
● Each driver isprobed separately
● How to tell drivers they all are part of the same media device?
PCI, AMBA etc.
ISPI2C
controller
sensorlens
voice coil
![Page 8: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/8.jpg)
Media device setup
1.media_device_init()
2.v4l2_device_register()
3.video_register_device()
5.v4l2_device_register_subdev(isp)
6.v4l2_register_subdev_nodes()
7.media_device_register()
4.v4l2_device_register_subdev(sensor)
Media device driver sensor driver
![Page 9: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/9.jpg)
V4L2 async
![Page 10: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/10.jpg)
V4L2 async
● The V4L2 async framework facilitates sub-device registration
● V4L2 sub-device device node creation and media device registration postponed after probe
● To do its job, the V4L2 async framework makes use of firmware provided information
![Page 11: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/11.jpg)
V4L2 async example (ISP)
device base ISP driver ISP DT node sensor DT node V4L2 async async sub-device list async notifier list
probe(dev)
parse local endpoints
of_graph_get_remote_port_parent(endpoint)
v4l2_async_register_notifier(notifier)
Look for sub-devices matching the notifier list
No match found
Add ISP notifier
![Page 12: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/12.jpg)
V4L2 async (sensor)
device base ISP driver sensor driver sensor DT node V4L2 async async notifier list
probe(dev)
v4l2_of_parse_endpoint()
endpoint properties
v4l2_async_register_subdev(sd)
Look for a notifier matchin the sub-device
Found it!
notifier->bound()
notifier->complete()
![Page 13: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/13.jpg)
Device tree
![Page 14: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/14.jpg)
Device tree
● System hardware description in a human readable format● Originates from Sparc / Open Firmware● Primarily used on embedded systems
– ARM
– Also PowerPC, Sparc and x86
● Tree structure– Nodes
– Properties
● Source code compiled into binary before use
![Page 15: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/15.jpg)
Device Tree standard and bindings
● Device Tree specification maintained by devicetree.org– Syntax and some semantics
● Bindings define the interface between the firmware and the software– Bindings are Operating System specific
– Linux Device tree binding documentation part of the Linux kernel source
– FreeBSD developers appear to be converging towards using Linux DT bindings
![Page 16: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/16.jpg)
Device tree graphs
● phandle properties can be used to refer to other nodes in the tree
● Port is an interface in a device (as in an IP block)
● Endpoint describes one end of a connection to a port [7]
![Page 17: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/17.jpg)
Sensor node
&i2c2 {smia_1: camera@10 {
compatible = "nokia,smia";reg = <0x10>;/* No reset gpio */vana-supply = <&vaux3>;clocks = <&isp 0>;clock-frequency = <9600000>;nokia,nvm-size = <(16 * 64)>;port {
smia_1_1: endpoint {link-frequencies = /bits/ 64 <199200000 210000000 499200000>;clock-lanes = <0>;data-lanes = <1 2>;remote-endpoint = <&csi2a_ep>;
};};
};};
source: arch/arm/boot/dts/omap3-n9.dts
![Page 18: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/18.jpg)
ISP node board specific part
&isp {vdd-csiphy1-supply = <&vaux2>;vdd-csiphy2-supply = <&vaux2>;ports {
port@2 {reg = <2>;csi2a_ep: endpoint {
remote-endpoint = <&smia_1_1>;clock-lanes = <2>;data-lanes = <1 3>;crc = <1>;lane-polarities = <1 1 1>;
};};
};};
source: arch/arm/boot/dts/omap3-n9.dts
![Page 19: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/19.jpg)
OF graph API
● Parse port and endpoint nodes under device nodes
● Enumerate over endpoints● Obtain remote endpoint
– Based on the phandle value
![Page 20: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/20.jpg)
ACPI
![Page 21: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/21.jpg)
ACPI
● Advanced Configuration and Power Interface● Operating system independent● Origins in x86 and PC
– Increasingly used in embedded systems
● Device discovery and enumeration● Power management● ACPI methods
– Runnable code
– ACPI virtual machine
![Page 22: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/22.jpg)
ACPI
● ACPI specifications developed by UEFI Forum– Roughly one specification per year
● What do you do if you need to add a new kind of a device?– A new ACPI specification?
![Page 23: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/23.jpg)
ACPI Device Specific Data
● _DSD object type part of ACPI 5.1 and later– Key-value pairs (property extension) and
– Tree structures (hierarchical data extension)
● Together property and data extension could be used to implement very similar functionality to Devicetree
● _DSD property registry [6]– Light-weight approach for registering _DSD properties
![Page 24: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/24.jpg)
fwnode property API
● Access properties independently of underlying firmware implementation– Device Tree
– ACPI
● Makes use of ACPI _DSD property extension [2]
![Page 25: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/25.jpg)
Future work
![Page 26: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/26.jpg)
Fwnode graph API
● Functionally the same as the OF graph API– But is firmware independent
● Device tree implementation is used on Device tree
● Makes use of the _DSD hierarchical data extension [3] on ACPI
● Implementation at RFC level [4]
![Page 27: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/27.jpg)
V4L2 fwnode API
● ”V4L2 ACPI support”– Embedded systems with I²C components
– Requires both fwnode graph API and V4L2 fwnode API
● Same functionality as V4L2 OF API● V4L2 fwnode and V4L2 OF fully interoperable
– Sub-device driver using V4L2 fwnode works with a media device driver using V4L2 OF
– and vice versa!
● RFC implementation available [5]
![Page 28: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/28.jpg)
Flash
● LED flash devices supported● But the kernel has no knowedge which sensor
they're related to– This is rather important if there are multiple
cameras in the system, such as most mobile phones nowadays
● Standardise phandle property for this?
![Page 29: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/29.jpg)
Camera module
● Currently there's no ”camera module” concept in the kernel (nor DT or ACPI)
● Camera module construction is important for the user space– Which sensor and lens are related?
– What kind of lens is there?
– What's the voice coil spring constant?
– Is there an infra red filter? What kind of filter is it?– What's the aperture size?
![Page 30: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/30.jpg)
Camera module power on and power off sequences
● Regulators, clocks and / or GPIOs may be shared between module components
● Power on and power off sequences device component specific– Which order and when each resource may be enabled?
– E.g. regulator and clock are enabled, then after 10 ms the reset GPIO can be lifted and the device is ready for use
● Requirements of bothlens and sensor must beconsidered for modulepower-up sequence
cameramodule
lens voice coil
sensor
lens voice coil
lens
CSI-2
I²C
reset
vana
vdig
clock
![Page 31: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/31.jpg)
Questions?
![Page 32: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/32.jpg)
References
[1] http://www.uefi.org/acpi
[2] http://www.uefi.org/sites/default/files/resources/_DSD-device-properties-UUID.pdf
[3] http://www.uefi.org/sites/default/files/resources/_DSD-hierarchical-data-extension-UUID-v1.pdf
[4] http://www.spinics.net/lists/linux-acpi/msg69547.html
[5] http://www.spinics.net/lists/linux-media/msg106160.html
[6] https://github.com/ahs3/dsd
[7] Documentation/devicetree/bindings/graph.txt
![Page 33: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/33.jpg)
Firmware logistics
Device tree ACPI
BIOSvendor
systemvendor
BIOS in flashmemory
SoCvendor
OSbinaries
runtimeACPI tables
mother-
board
supportw
ebsite
sele
ctive ta
ble
rep
lace
me
nt
(initrd
)SoC
vendor
OSbinaries
systemvendor
flashmemory
Linuxkernel
dts
Device tree
appended to
kernel image
![Page 34: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/34.jpg)
ACPI camera example
![Page 35: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/35.jpg)
ACPI camera example
Scope (\_SB.PCI0.I2C2) {
Device (CAM0){ Name (_DSD, Package () {
/* device specific data */Package () { Package () { "compatible", Package () { "nokia,smia" } }, Package () { "lanes", 4 }, Package () { "clock-frequency", 24000000 },},/* data extension */Package () { Package () { "ports", "PRTS" },}
})
![Page 36: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/36.jpg)
ACPI camera example
Name (PRTS, Package() {/* data extension */Package () { Package () { "port@0", "PRT0" },}
}) Name (PRT0, Package() {
/* device specific data */Package () { Package () { "port", 0 },},/* data extension */Package () { Package () { "endpoint@0", "EP0" },}
})
![Page 37: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/37.jpg)
ACPI camera example Name (EP0, Package() {
/* device specific data */Package () { Package () { "endpoint", 0 }, Package () { "clock-lanes", 0 }, Package () { "data-lanes", Package () { 1, 2, 3, 4 } }, Package () { "link-frequencies",
Package () { 209600000, 342000000, 451200000 } }, Package () { "remote-endpoint", Package() { \_SB.PCI0.ISP, 0, 0, 0 } },},
})}
}
![Page 38: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/38.jpg)
ACPI ISP example Scope (\_SB.PCI0) {
Device (ISP){ Name (_DSD, Package () {
/* data extension */Package () { Package () { "ports", "PRTS" },}
}) Name (PRTS, Package() {
/* data extension */Package () { Package () { "port@4", "PRT4" },}
})
![Page 39: Cameras in embedded systems: Device tree and ACPI view · Cameras in embedded systems: Device tree and ACPI view ... device base ISP driver ISP DT node sensor DT node V4L2 async async](https://reader030.vdocuments.us/reader030/viewer/2022021801/5b238aba7f8b9aff718b474a/html5/thumbnails/39.jpg)
ACPI ISP example
Name (PRT4, Package() {/* device specific data */Package () { Package () { "port", 4 }, /* CSI-2 port number */},/* data extension */Package () { Package () { "endpoint@0", "EP0" },}
}) Name (EP0, Package() {
/* device specific data */Package () { Package () { "endpoint", 0 }, Package () { "clock-lanes", 0 }, Package () { "data-lanes", Package () { 1, 2, 3, 4 } }, Package () { "remote-endpoint", Package () { \_SB.PCI0.I2C2.CAM0, 0, 0, 0 } },},
})}
}