android's new stream-based camera architecture
TRANSCRIPT
![Page 1: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/1.jpg)
Android 5.x’s
Stream-Based Camera Architecture
Balwinder Kaur Senior Member, Technical Staff
ON Semiconductor
Android Builder’s Summit | San Jose | 3.23.15
![Page 2: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/2.jpg)
2
Tools of the Trade
![Page 3: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/3.jpg)
3
Agenda
• The Basic Idea • Enabling New Experiences • Camera2 Architecture • Camera2 APIs
– Ready. Set. Go! • The Other Use Cases • The Devil is in the Details • Play Store Filters • Demo • Summary
![Page 4: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/4.jpg)
The Basic Idea Limitations of original Camera Architecture
![Page 5: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/5.jpg)
5
Limitations of android.hardware.Camera
http://source.android.com/devices/camera/camera.html
3 Primary Opera,ng Modes • Preview, Capture & Video
Recording
New Features – hard to implement • Burst mode photography • Zero Shutter Lag, Multi-Shot
HDR, Panoramic Stitch
No Per-Frame Control • Not Possible
No Support for RAW • Most products returned null
for the raw callback
Minimal Metadata • Support for face detection
data
Primi,ve Custom Se@ngs • Through
Camera.Parameters class but no guarantees when the settings would go into effect
![Page 6: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/6.jpg)
Enabling New Experiences
![Page 7: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/7.jpg)
7
Camera2 Architecture Enables -
Point and Click Camera
Professional Camera
Professional Camera + Post Processing on-‐device
Innova,ve Mobile Camera ++
![Page 8: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/8.jpg)
8
Features and Requirements
Point and Click • Preview • S,ll • Video Recording
Professional Camera • Fine Grained Control of Lens, Sensor, Flash • RAW Sensor Output with capture metadata
Computa4onal Photography • Individual Frame Control • Algorithms & Compute Power • HDR, Focus Stacking, Exposure Bracke,ng
Innova4ve Mobile Cameras • Sensors • Loca,on • Connec,vity/Cloud Power • Compute Power
![Page 9: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/9.jpg)
9
android.hardware.camera2
• Enables Professional Quality Photography – Think DSLR instead of Point-and-Shoot
• Enables Access to RAW Images • Enables On-device processing of Camera Data
– High Dynamic Range, Focus Stacking • Enables New Use Cases combining Imaging with
– Rich Sensor input • Inertial sensors, altitude et. al.
– Compute Power • Multi-core CPU & GPU
– Connectivity • Power of the Cloud & Access to Proximity (BLE, NFC)
– Context • Location & User History
Major Limita,on-‐ Op,cs
DSLR++ ?
Computa,onal Photography
![Page 10: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/10.jpg)
10
Professional Quality Photography
• Fine grained control of the – Sensor – Flash – Lens – Image Signal Processing Pipeline
• Control on a per-frame basis, and deterministic behavior • Processing Images at full resolution and full frame rate(30 fps) • Meta data
– Every frame is returned with the actual settings that it was taken with, and requested for.
Professional Camera
![Page 11: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/11.jpg)
11
Computational Photography
Definition
• Computational photography refers to computational image capture,
processing, and manipulation techniques that enhance or extend the
capabilities of digital photography.
Source: Wikipedia
Professional Camera + Post Processing on-‐device
![Page 12: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/12.jpg)
12
Multi Shot HDR
• HDR = High Dynamic Range • Multiple shots at different exposures and then blended together
Computa,onal Photography
![Page 13: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/13.jpg)
13
Panoramic Stitching
Source: commons.wikimedia.org
Computa,onal Photography
Multiple Images with orientation information and fixed Exposure
![Page 14: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/14.jpg)
14
Flash No-Flash Photography
Source: MicrosoW Research
Computa,onal Photography
![Page 15: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/15.jpg)
15
Flash No-Flash Photography
Source: Microsoft Research
Computa,onal Photography
• Take 2 shots of the scene– one with flash on, the other with flash off
• Computationally blend the 2 frames together
![Page 16: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/16.jpg)
16
Focus Stacking
Source: Wikipedia
Focus Stacking for macro photography. Also known as – all-in-focus photography
Computa,onal Photography
![Page 17: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/17.jpg)
17
View Finder
• Today’s smartphones enable features like – Touch to Expose & Touch to
Focus • Computational
Photography can take it to the next level – Edit the frame in the view
finder by selecting areas to sharpen, blur, brighten and maintain these during capture mode Source: SIGGRAPH 2012
Computa,onal Photography
![Page 18: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/18.jpg)
18
Mobile Embedded Vision Applications
• Gesture Recognition • Face Recognition, Object Tracking • Visual Search • 3D Mapping of the environment • Augmented Reality
Natural User Interface Companies are using the Front Facing Camera to provide gesture recogni,on solu,ons.
![Page 19: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/19.jpg)
19
Innovative Mobile Camera ++
You Fine Grained Control
Lens, Sensor, Flash
Cloud Compu,ng
On-‐Device Compute Power
Sensors
Loca,on
User History User Preferences
![Page 20: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/20.jpg)
Camera2 Architecture
![Page 21: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/21.jpg)
21
Limitations of android.hardware.Camera
http://source.android.com/devices/camera/camera.html
3 Primary Opera,ng Modes
New Features – hard to implement
No Per-Frame Control
No Support for RAW
Minimal Metadata
Custom Se@ngs
![Page 22: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/22.jpg)
22
Camera2 Core Operation Model
http://source.android.com/devices/camera/camera.html
• Per Frame Settings • The Settings travel with individual
requests and are no longer globally applied
• Image buffer delivered to all configured and requested Surfaces
• Metadata returned back to the application as requested separately
• Multiple requests and results in queue simultaneously
![Page 23: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/23.jpg)
23
Camera3 HAL Model
http://source.android.com/devices/camera/camera.html
![Page 24: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/24.jpg)
The Camera APIs
![Page 25: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/25.jpg)
25
Android Lollipop
Camera related classes and interfaces from android.hardware deprecated 2 new packages
android.hardware.camera2 android.hardware.camera2.params 25+ classes, 1 Exception
![Page 26: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/26.jpg)
26
Camera Model
From the Javadoc “This package models a camera device as a pipeline, which takes in input requests for capturing a single frame, captures the single image per the request, and then outputs one capture result metadata packet, plus a set of output image buffers for the request. The requests are processed in-order, and multiple requests can be in flight at once. Since the camera device is a pipeline with multiple stages, having multiple requests in flight is required to maintain full frame-rate on most Android devices.”
![Page 27: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/27.jpg)
27
Architecture walk through for primary use cases
Live Preview
S,ll Capture (JPG)
S,ll Capture (RAW/DNG)
S,ll Capture (Burst)
Video Recording
![Page 28: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/28.jpg)
Ready. Set. Go! Figure out what’s available
![Page 29: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/29.jpg)
29
Concepts and Terms from Camera 2
• Camera Device • Streams • Capture Session • Target Surfaces
It’s a paradigm s h i f t !
• Capture Request • Capture Result
– Image Data – Metadata – Total Capture Result
![Page 30: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/30.jpg)
30
1. How many Cameras do I have?
Query the System for capabilities CameraManager provides information on the number of available cameras, or CameraDevices.
CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);
String[] camids = manager.getCameraIdList();
![Page 31: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/31.jpg)
31
2. What kind of a CameraDevice is it?
CameraCharacteristics provide static metadata for a given CameraDevice. This information is immutable for a given camera.
CameraCharacteristics characteristics = manager.getCameraCharacteristics(camid);
CameraCharacteris,cs Keys INFO_SUPPORTED_HARDWARE_LEVEL 3 Classes of Cameras are supported
LEGACY LIMITED FULL
Additionally, RAW may be supported ① FULL + RAW ② LIMITED + RAW
![Page 32: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/32.jpg)
32
3. How many maximum streams does the system support?
PROCESSED & NON_STALLING (like YUV)
PROCESSED & STALLING (like JPEG)
RAW
FULL >=3 >=1 >=0
LIMITED >=2 >=1 >=0
LEGACY >=2 >=1 0
3 Classes of Streams ① Processed & Non-Stalling ② Processed & Stalling ③ RAW (Bayer Domain)
The Maximum Number of Streams = REQUEST_MAX_NUM_OUTPUT_PROC1
+ REQUEST_MAX_NUM_OUTPUT_PROC_STALLING1 + REQUEST_MAX_NUM_OUTPUT_RAW1
1 CameraCharacteris,cs.Key
![Page 33: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/33.jpg)
33
Guaranteed Stream Configurations - LEGACY
Target 1 Target 2 Target 3
Type Max Size Type Max Size Type Max Size
PRIV1 Maximum2
JPEG Maximum
YUV Maximum
PRIV Preview JPEG Maximum
YUV Preview JPEG Maximum
PRIV Preview PRIV Preview
PRIV Preview YUV Preview
PRIV Preview YUV Preview JPEG Maximum 1 Device Opaque Format, No Visibility into Format for Applica,on 2 Device Maximum Sensor Output
S,ll capture plus in-‐app processing.
![Page 34: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/34.jpg)
34
Guaranteed Stream Configurations - LIMITED
Target 1 Target 2 Target 3
Type Max Size Type Max Size Type Max Size
PRIV Preview PRIV Record 1
PRIV Preview YUV Record
YUV Preview YUV Record
PRIV Preview PRIV Record JPEG Record
PRIV Preview YUV Record JPEG Record
YUV Preview YUV Preview JPEG Maximum
1 Maximum resolu,on supported by the device for media recording
![Page 35: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/35.jpg)
35
Guaranteed Stream Configurations - FULL
Target 1 Target 2 Target 3
Type Max Size Type Max Size Type Max Size
PRIV Preview PRIV Maximum
PRIV Preview YUV Maximum
YUV Preview YUV Maximum
PRIV Preview PRIV Preview JPEG Maximum
YUV 640x480 PRIV Preview YUV Maximum
YUV 640x480 YUV Preview YUV Maximum
![Page 36: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/36.jpg)
36
Guaranteed Stream Configurations - RAW
Target 1 Target 2 Target 3
Type Max Size Type Max Size Type Max Size
RAW Maximum
PRIV Preview RAW Maximum
YUV Preview RAW Maximum
PRIV Preview PRIV Preview RAW Maximum
PRIV Preview YUV Preview RAW Maximum
YUV Preview YUV Preview RAW Maximum
PRIV Preview JPEG Maximum RAW Maximum
YUV Preview JPEG Maximum RAW Maximum
In-‐app processing with simultaneous JPEG and DNG.
![Page 37: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/37.jpg)
37
4. How can these streams be configured?
CameraCharacteristics characteristics = cameraManager.getCameraCharacteristics(cameraId); StreamConfigurationMap configs = characteristics.get(
CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);
StreamConfigurationMap provides information on • Supported Stalling and Non-Stalling Image Formats • Supported Image Sizes for Supported Image Formats • Supported Video Sizes and fps(frames per Second) Speeds • Minimum Frame Duration and Stall Duration for supported formats and sizes • Determine whether a given Surface is supported by this CameraDevice or not
![Page 38: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/38.jpg)
38
FULL Camera Devices
30 FPS at sensor max resolu,on more than 20fps is required
Per Frame Control android.sync.maxLatency PER_FRAME_CONTROL
Manual Sensor Control Manual Post Processing
Control android.request.availableCapabili,es
Arbitrary Cropping Region
android.scaler.croppingType == FREEFORM
Minimum 3 Processed Non-‐Stalling Output
Streams (think YUV)
Stream Configura,ons Available
as a Configura,on Map
![Page 39: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/39.jpg)
39
LIMITED and LEGACY Camera Device
LEGACY • NO Per-‐ Frame Control • NO Manual Sensor Control • NO Manual Post Processing • NO Arbitrary Cropping Regions • No Stringent Performance Constraints
• Not a candidate for a Superior Camera Experience
LIMITED
• Query for individual capabili,es!
LEGACY LIMITED FULL
That’s a lot of checks!
① FULL + RAW ② LIMITED + RAW
![Page 40: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/40.jpg)
Ready. Set. Go! Open the Camera
![Page 41: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/41.jpg)
41
1. Open a Full Camera Device
① Use the instance of the CameraManager to open a Camera passing in its camera id.
② Get a handle to a CameraDevice via the CameraDevice.StateCallback method when it is successfully opened.
CameraDevice.StateCallback mStateCallback = new CameraDevice.StateCallback () {
…};
manager.openCamera(camid, mStateCallback, ..);
![Page 42: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/42.jpg)
42
2. Create a Capture Session
• Expensive Operation – Involves allocation of buffers beneath the HAL
– May take up to several hundred milliseconds • All modes cannot be supported concurrently
– so Sessions may have to be torn down and setup again.
CameraCaptureSession.StateCallback mCaptureCallback= new CameraCaptureSession.StateCallback () {...};
List<Surface> outputs = new ArrayList<Surface>(...);
createCaptureSession (outputs, mCaptureCallback, ...);
// a handle to a capture session in mCaptureCallback method
Minimize calls to createCaptureSession
Failure to create a capture session may result in excep,ons being thrown
![Page 43: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/43.jpg)
43
Different Target Output Streams
![Page 44: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/44.jpg)
44
3. Create a Capture Request
① Pick the Capture Request Template o TEMPLATE_PREVIEW o TEMPLATE_STILL_CAPTURE o TEMPLATE_VIDEO_SNAPSHOT o TEMPLATE_RECORD o TEMPLATE_MANUAL
CaptureRequest.Builder mPreviewBuilder = mCameraDevice .createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
![Page 45: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/45.jpg)
45
3. Create a Capture Request – contd.
mPreviewBuilder.addTarget(mPreviewSurface); //where mPreviewSurface is a Surface initialized to the width and height of the Preview Desired
② Pick an appropriate target output o e.g. TextureView for the correct width/height for Preview
③ Pick the Frequency of the Capture Request o capture o captureBurst o setRepeatingRequest o setRepeatingBurst o stopRepeating o abortCaptures()
![Page 46: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/46.jpg)
Ready. Set. Go! Get your pictures
![Page 47: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/47.jpg)
47
Receiving Image and Metadata
• The Image data is typically received in a listener associated with the Output Surface.
• The Metadata is received in the onCaptureCompleted callback method of CameraCaptureSession.CaptureCallback via the TotalCaptureResult object.
How does the metadata ,e to the correct image
data frame?
Through the Time Stamp! CaptureResult.SENSOR_TIMESTAMP
& image.getTimestamp()
mSession.setRepeatingRequest(mPreviewBuilder.build(), listener,backgroundHandler);
![Page 48: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/48.jpg)
The Other Use Cases
![Page 49: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/49.jpg)
49
Still Capture - JPEG
① Use TEMPLATE_STILL_CAPTURE for the CaptureRequest ② Create an ImageReader object with format ==
ImageFormat.JPEG ③ Pick CameraCaptureSession.capture for the frequency of
capture ④ Image Data is received in the
ImageReader.onImageAvailableListener ⑤ Metadata is received in the
CameraCaptureSession.CaptureCallback. onCaptureCompleted callback
![Page 50: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/50.jpg)
50
Still Capture - RAW
① Use TEMPLATE_STILL_CAPTURE for the CaptureRequest ② Create an ImageReader object with format == ImageFormat.
ImageFormat.RAW_SENSOR ③ Pick CameraCaptureSession.capture for the frequency of capture ④ Image Data is received in the ImageReader.onImageAvailableListener ⑤ Metadata is received in the
CameraCaptureSession.CaptureCallback.onCaptureCompleted callback ⑥ Use an instance of DngCreator to convert RAW to DNG (Digital Negative)
final File dngFile = new File(path, filename); outs = new FileOutputStream(dngFile); if (mRawTotalResult != null) { DngCreator d = new DngCreator(characteristics, mRawTotalResult); d.writeImage(outs, image); }
![Page 51: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/51.jpg)
51
Still Capture - BURST
① Use TEMPLATE_STILL_CAPTURE for the CaptureRequest ② Create an ImageReader object with format == ImageFormat.
ImageFormat.JPEG ③ Pick CameraCaptureSession.captureBurst for the frequency of capture ④ Image Data is received in the ImageReader.onImageAvailableListener ⑤ Metadata is received in the CameraCaptureSession.CaptureCallback.
onCaptureCompleted callback
![Page 52: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/52.jpg)
52
Video Recording
① Use TEMPLATE_RECORD for the CaptureRequest ② Create a MediaRecorder Object and use its surface
as the output stream ③ Pick CameraCaptureSession.setRepeatingRequest
for the frequency of capture ④ Image Data is recorded in the Image file set via
MediaRecorder.setOutputFile call. ⑤ Metadata is received in the
CameraCaptureSession.CaptureCallback. onCaptureCompleted callback
![Page 53: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/53.jpg)
53
In-App Processing of Camera Data
① Use an appropriate Template for the CaptureRequest; including TEMPLATE_MANUAL if the use case so demands
② Create an object like Allocation Renderscript, ImageReader (with Format YUV_420_888), SurfaceTexture, or GLSurfaceView that provides a Surface
③ Pick an appropriate frequency for capture ④ Image Data can be found in an appropriate callback for the surface
chosen ⑤ Metadata is received in the CameraCaptureSession.CaptureCallback.
onCaptureCompleted callback
![Page 54: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/54.jpg)
54
Some use cases for In-App Processing
• Use Cases – Filters – Computational Photography Use cases – Custom Post Processing of the Imaging Pipeline – Computer Vision Applications
• Uses – Compute Framework
• Renderscript, OpenCL – Low level Graphics APIs
• OpenGL ES
![Page 55: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/55.jpg)
55
And then there was Metadata…
CameraMetadata
Controls
CaptureRequest
Information
Static Metadata Fixed for a given CameraDevice CameraCharacteristics
Dynamic Metadata Settings associated with each
frame CaptureResult
Key Value Pairs for Hardware Control (Lens, Sensor, Flash), Image Processing Pipeline, Control Algorithms and Output Buffers
![Page 56: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/56.jpg)
Fine Grained Control a.k.a “The Devil is in the Details”
![Page 57: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/57.jpg)
57
Manual Sensor Control
Sensor
Sensi,vity (Gain/ISO)
Sensor Exposure Time
Sensor Frame Dura,on
Flash (if present)
Flash Mode
Lens (if adjustable)
Lens Aperture
Lens Filter Density
Lens Focal Length
Lens Focus Distance
Black Level Lock
Black Level Lock On/Off
![Page 58: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/58.jpg)
58
Manual 3A Control
• A Camera Request can override the 3A Algorithm Controls – Auto Exposure (AE) – Auto Focus(AF) – Auto White Balance (AWB)
• There is an overall control for all Auto Algorithms android.control.mode
– When set to OFF, application has full control of the pipeline – When set to AUTO, the individual controls are in effect
![Page 59: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/59.jpg)
59
Application Controlled (Auto) Algorithms
Auto Focus
• Applica,on uses android.lens.focusDistance to control the Lens Auto Exposure
• Applica,on uses android.sensor.exposureTime, android.sensor.sensi,vity and android.sensor.frameDura,on + android.flash.* to control exposure
Auto White Balance • Applica,on uses android.colorCorrec,on.transform & android.colorCorrec,on.gains to control the white balance
android.control.mode= AUTO android.control.a*mode = OFF
![Page 60: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/60.jpg)
60
Manual Post Processing
http://source.android.com/devices/camera/camera.html
![Page 61: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/61.jpg)
61
Hot Pixel Mode
Black Level Correc,on Demosaic
Noise Reduc,on Mode
Shading Mode (lens shading)
Color Correc,on Mode
Tone Curve Adjustment
Edge Enhancement
Image Processing Pipeline
![Page 62: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/62.jpg)
62
Manual Post Processing
Manual Tonemap control
• android.tonemap.*
Manual White Balance control
• android.colorCorrec,on.*
Manual Lens Shading map control
• android.shading.*
Manual aberra,on correc,on control (if supported)
• android.colorCorrec,on.aberra,on.*
![Page 63: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/63.jpg)
63
The other camera2 package android.hardware.camera2.params
• BlackLevelPattern • ColorSpaceTransform • Face • LensShadingMap • MeteringRectangle • RggbChannelVector • StreamConfigurationMap • TonemapCurve
![Page 64: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/64.jpg)
Play Store Feature Filters
Android Manifest <uses-feature>
![Page 65: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/65.jpg)
65
Play Store Feature Filters
• android.hardware.camera.hardware_level.full • android.hardware.camera.capability.raw • android.hardware.camera.capability.manual_sensor • android.hardware.camera.capability.manual_post_processing
See Also: Versioning
![Page 66: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/66.jpg)
Demo Time
![Page 67: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/67.jpg)
Summary
![Page 68: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/68.jpg)
68
Its all about you and your power tools!
You Fine Grained Control
Lens, Sensor, Flash
Cloud Compu,ng
On-‐Device Compute Power
Sensors
Loca,on
User History User Preferences
![Page 69: Android's New Stream-Based Camera Architecture](https://reader034.vdocuments.us/reader034/viewer/2022051521/585407901a28abfa398fbf30/html5/thumbnails/69.jpg)
69
Q & A and THANK YOU for your time.
Balwinder Kaur [email protected] [email protected]
http://www.slideshare.net/lbk003 https://github.com/lbk003