capturing from the camera using av foundation on...
TRANSCRIPT
![Page 1: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/1.jpg)
These are confidential sessions—please refrain from streaming, blogging, or taking pictures
API enhancements and performance improvements
Session 419
Capturing From the CameraUsing AV Foundation on iOS 5
Brad FordiOS Engineering
1
![Page 2: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/2.jpg)
What You Will Learn
• Which iOS 5 captures APIs to use in your app• The AV Foundation capture programming model• iOS 5 Performance improvements in AV Foundation• iOS 5 AV Foundation API enhancements
2
![Page 3: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/3.jpg)
Sample Code for This Session
• ChromaKey• RosyWriter• ‘StacheCam• StopNGo (iOS Edition)
Materials available at:https://developer.apple.com/wwdc/schedule/details.php?id=419
3
![Page 4: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/4.jpg)
Technology Framework
CoreAnimationCoreAudio CoreMedia
AVFoundation
UIKit
UIImagePickerController
Camera app
4
![Page 5: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/5.jpg)
00:00:15
Using the Camera in iOS 5Simple programmatic access
Hideable camera controls UI
API for high, medium, or low quality recording
-(void)takePicture-(BOOL)startVideoCapture
User touch to focus, like Camera appUser touch and hold to lock AE and AF
-(void)setCameraDevice:-(void)setCameraFlashMode:
5
![Page 6: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/6.jpg)
Why Use AV Foundation Capture?
• Independent focus, exposure, and white balance controls■ Independent locking■ Independent points of interest for focus and exposure
• Access to video frame data■ Accurate timestamps■ Per-frame metadata■ Configurable output format (e.g. 420v, BGRA)■ Configurable frame rate■ Configurable resolution
Full access to the camera
6
![Page 7: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/7.jpg)
Why Use AV Foundation Capture?Flexible output
• Still Image Capture■ YUV and RGB output■ Exif metadata insertion
• QuickTime Movie Recording■ Movie metadata insertion■ Orientation lock
• Video Preview in a CALayer■ Aspect fit, aspect fill, and stretch
• Audio level metering• Access to audio sample data
7
![Page 8: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/8.jpg)
Programming modelAV Foundation Capture
8
![Page 9: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/9.jpg)
Capture Basics—Inputs and Outputs
Video
Outputs
InputsAudio
9
![Page 10: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/10.jpg)
Video
Outputs
InputsAudio
Capture Basics—Inputs and Outputs
AVCaptureDeviceInputAVCaptureDeviceInput
AVCaptureSession
AVCaptureMovieFileOutput
AVCaptureAudioDataOutput
AVCaptureVideoDataOutputAVCaptureStillImageOutput
AVCaptureVideoPreviewLayer
10
![Page 11: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/11.jpg)
Capture Use Cases We Will Cover
• Process video frames from the camera and render with OpenGL• Process live video and audio, and write them to a QuickTime movie• Scan video frames for patterns using the flash and VideoDataOutput• Process captured still images with CoreImage
11
![Page 12: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/12.jpg)
Capture Use Cases We Will Cover
• Process video frames from the camera and render with OpenGL• Process live video and audio, and write them to a QuickTime movie• Scan video frames for patterns using the flash and VideoDataOutput• Process captured still images with CoreImage
12
![Page 13: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/13.jpg)
ChromaKeyDemo
Sylvain NuzCore Media Engineering
13
![Page 14: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/14.jpg)
Outputs
Inputs
Process and render with OpenGL
ChromaKey
Video
14
![Page 15: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/15.jpg)
Outputs
Inputs
Process and render with OpenGL
Video
AVCaptureSession
AVCaptureDeviceInput
AVCaptureDevice
CMSampleBuffer
AVCaptureVideoDataOutput
ChromaKey
15
![Page 16: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/16.jpg)
AVCaptureDevice *device = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
ChromaKey
• Create an AVCaptureSession
• Find a suitable AVCaptureDevice
• Create and add an AVCaptureDeviceInput
AVCaptureSession *session = [[AVCaptureSession alloc] init];session.sessionPreset = AVCaptureSessionPreset1280x720;
AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:device error:&error];[session addInput:input];
16
![Page 17: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/17.jpg)
ChromaKey
• Create and add an AVCaptureVideoDataOutput
• Implement your delegate callback
AVCaptureVideoDataOutput *output = [[AVCaptureVideoDataOutput alloc] init];[session addOutput:output];
• Configure your output, and start the session dispatch_queue_t serial_queue = dispatch_queue_create(“myQueue”, NULL);[output setSampleBufferDelegate:self queue:serial_queue];[session startRunning];
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection{ // CFShow(sampleBuffer);}
17
![Page 18: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/18.jpg)
ChromaKey
• Lock autoexposure, focus, and white balanceAVCaptureDevice *device = self.device;[device lockForConfiguration:&error];
[device setFocusMode:AVCaptureFocusModeLocked];[device setExposureMode:AVCaptureExposureModeLocked];[device setWhiteBalanceMode:AVCaptureWhiteBalanceModeLocked];
[device unlockForConfiguration];
18
![Page 19: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/19.jpg)
Bridging CoreVideo and OpenGLESChromaKey
Brandon CoreyiOS Engineering
19
![Page 20: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/20.jpg)
ChromaKey
• Bridges CoreVideo PixelBuffers to OpenGLES textures■ Avoids copies to and from the GPU■ Allows recycling of textures
• Supported in OpenGL ES 2.0 and later• Defined in <CoreVideo/CVOpenGLESTextureCache.h>
CVOpenGLESTextureCache
20
![Page 21: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/21.jpg)
ChromaKey
• Efficiently maps a BGRA buffer as a source texture (OpenGL ES 2.0)CVOpenGLESTextureCacheCreateTextureFromImage( kCFAllocatorDefault, openGLESTextureCacheRef, pixelBuffer, // a CVPixelBuffer received from the delegate callback NULL, // optional texture attributes GL_TEXTURE_2D, // the target GL_RGBA, // internal format width, height, GL_BGRA, // pixel format GL_UNSIGNED_BYTE, // data type of the pixels 0, // plane index to map (BGRA has only one plane) &outTexture );
21
![Page 22: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/22.jpg)
Standard Texture Binding
• Uploading data to a Texture ImageGLuint texture;
glGenTexture(1, &texture);
void *pixelData = CVPixelBufferGetBaseAddress(pixelBuffer);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, 1280, 720,
0, GL_BGRA, GL_UNSIGNED_BYTE, pixelData);
Texture Object
Texture ImageCVPixelBuffermemcpy()
22
![Page 23: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/23.jpg)
CVOpenGLES Texture Binding
• Binding data to a Texture ObjectGLuint texture;
CVPixelBufferRef pixelBuffer;
CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
textureCacheRef, pixelBuffer, NULL,
GL_RGBA, 1280, 720, GL_BGRA, GL_UNSIGNED_BYTE, 0, &texture);
Texture Object
Texture ImageIOSurface backed CVPixelBuffer memcpy()
23
![Page 24: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/24.jpg)
Getting Data Back Out
• Have OpenGL ES render to a Framebuffer Object (FBO)• Done one of two ways
■ Create and attach a Renderbuffer to an FBO■ Create and attach a Texture to an FBO
• glReadPixels or glTexImage2D to move data into a buffer
Framebuffer Object (FBO)
GL_COLOR_ATTACHMENT0
GL_COLOR_ATTACHMENT1
GL_COLOR_ATTACHMENTn
GL_DEPTH_ATTACHMENT
GL_STENCIL_ATTACHMENT
...Texture Object
Renderbuffer Object
Texture Image
Renderbuffer Image
24
![Page 25: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/25.jpg)
CVOpenGLES Framebuffer Binding
• Binding to a Texture and attaching to a Framebuffer Object GLuint texture, fbo;glGenFrameBuffer(1, &fbo);CVOpenGLESTextureCacheCreateTextureFromImage(... &texture);glBindFrameBuffer(GL_FRAMEBUFFER, fbo);glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, texture, 0);
Texture or Renderbuffer Object
Texture or Renderbuffer Image
OpenGL ES Renderer
Texture or Renderbuffer Object
IOSurface backed CVPixelBuffer
glReadPixels()CVPixelBuffer
25
![Page 26: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/26.jpg)
CVOpenGLESTextureCacheUsage notes
• Buffers from AVCapture and AVAssetReader are created with the appropriate formatting
• If you create your own CVPixelBuffers, you must pass kCVPixelBufferIOSurfacePropertiesKey as a pixelBufferAttribute to CVPixelBufferCreate() or CVPixelBufferPoolCreate()
• If you use AVAssetWriter, be sure to use the pixel buffer pool it provides
26
![Page 27: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/27.jpg)
CVOpenGLESTextureCacheUsage notes (continued)
• BGRA, 420v, and 420f are supported• OpenGLES now supports GL_RED and GL_RG single channel render targets (iPad 2 only)
27
![Page 28: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/28.jpg)
AVCaptureVideoDataOutputPerformance Enhancements in iOS 5
Brad FordiOS Engineering
28
![Page 29: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/29.jpg)
AVCaptureVideoDataOutputUsage
• setSampleBufferDelegate:queue: requires a serial dispatch queue to ensure properly ordered buffer callbacks
• Note: Do not pass dispatch_get_current_queue()• By default, buffers are emitted in the camera’s most efficient format• Set the videoSettings property to specify a custom output format, such as ‘BGRA’
• Hint: Both CoreGraphics and OpenGL work well with ‘BGRA’
29
![Page 30: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/30.jpg)
AVCaptureVideoDataOutputPerformance tips
• Set the minFrameDuration property to cap the max frame rate• Configure the session to output the lowest practical resolution• Set the alwaysDiscardsLateVideoFrames property to YES (the default) for early, efficient dropping of late video frames
• Your sample buffer delegate’s callback must be FAST!
30
![Page 31: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/31.jpg)
AVCaptureVideoDataOutputSupported pixel formats
• ‘420v’■ Planar yuv 4:2:0 video format (luma and chroma are in separate planes)■ Chroma is subsampled in horizontal and vertical direction■ “Video Range” samples (16-235)■ This is the default format on all iOS 5 supported cameras
• ‘420f’■ Full color range planar yuv (0-255)■ This is the default format when using AVCaptureSessionPresetPhoto
• ‘BGRA’■ Blue / Green / Red / Alpha, 8 bits per pixel (more than 2x bandwidth)
31
![Page 32: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/32.jpg)
AVCaptureVideoDataOutputEnhancements in iOS 5
• Support for rotation of CVPixelBuffers■ Rotation is hardware accelerated■ Use AVCaptureConnection’s setVideoOrientation: property■ All 4 AVCaptureVideoOrientations are supported■ Default is non-rotated buffers
■ Front camera = AVCaptureVideoOrientationLandscapeLeft■ Back camera = AVCaptureVideoOrientationLandscapeRight
32
![Page 33: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/33.jpg)
AVCaptureVideoDataOutputEnhancements in iOS 5
• Support for pinning of minimum frame rate■ Enables fixed frame rate captures■ Use AVCaptureConnection’s -setMaxFrameDuration: property■ WARNING: Fixed frame rate captures may result in reduced image quality in low light
33
![Page 34: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/34.jpg)
AVCaptureSession + AVCaptureVideoDataOutputAVCaptureSession’s setSessionPreset: affects video data output resolution
High
Medium
Low
640x480
1280x720
Photo
352x288
iFrame 960x540
iFrame 1280x720
Highest recording quality
Suitable for WiFi sharing
Suitable for 3G sharing
VGA
720p HD
Full photo resolution
CIF for streaming apps
540p HD for editing
720p HD for editing
34
![Page 35: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/35.jpg)
AVCaptureVideoDataOutputSupported resolutions
Preset
Back CameraBack CameraBack Camera Front Camera
iPhone 3GS iPhone 4 iPad 2 / iPod Touch All models
High
Medium
Low
Photo
640x480 1280x720 1280x720 640x480
480x360480x360480x360480x360
192x144192x144192x144192x144
512x384 852x640 960x720 640x480
35
![Page 36: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/36.jpg)
AVCaptureVideoDataOutputSupported resolutions (continued)
• Special consideration for Photo mode as of iOS 4.3• AVCaptureSessionPresetPhoto delivers
■ Full resolution buffers to AVCaptureStillImageOutput■ Preview sized buffers to AVCaptureVideoDataOutput
• Aspect ratio is unchanged
36
![Page 37: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/37.jpg)
AVCaptureVideoDataOutputSupported resolutions (continued)
Preset
Back CameraBack CameraBack Camera Front Camera
iPhone 3GS iPhone 4 iPad 2 / iPod touch All models
352x288
640x480
1280x720
iFrame960x540
iFrame1280x720
37
![Page 38: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/38.jpg)
What is iFrame?
38
![Page 39: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/39.jpg)
What Are I Frames?
• I—Intra
I I I I I I I I I I
I P P P P I P P P P
I B B P B B I B B P
• P—Predictive
• B—Bidirectionally predictive
39
![Page 40: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/40.jpg)
What Is iFrame?
• Apple “ecosystem friendly” video• Supported by 30+ camera/camcorder models
■ H.264 I frame only video + AAC or PCM audio■ Constant frame rate (25 or 29.97 fps)■ ~30 megabits/s @ 960x540 or ~40 megabits/s @ 1280x720■ MOV or MP4 container
• Great for editing• Produces larger files
Supported on all iOS devices with HD cameras
40
![Page 41: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/41.jpg)
Capture Use Cases We Will Cover
• Process video frames from the camera and render with OpenGL• Process live video and audio, and write them to a QuickTime movie• Scan video frames for patterns using the flash and VideoDataOutput• Process captured still images with CoreImage
41
![Page 42: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/42.jpg)
RosyWriterDemo
Matthew CalhouniOS Engineering
42
![Page 43: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/43.jpg)
RosyWriter
Outputs
Inputs VideoAudio
43
![Page 44: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/44.jpg)
Outputs
Inputs VideoAudio
AVCaptureSession
AVCaptureDeviceInput
AVCaptureDevice
AVCaptureDeviceInput
AVCaptureDevice
AVCaptureVideoDataOutput AVCaptureAudioDataOutput
AVAssetWriterInput (video) AVAssetWriterInput (audio)
AVAssetWriter
RosyWriter
44
![Page 45: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/45.jpg)
What Is an AVAsset?
• Defined in <AVFoundation/AVAsset.h>• Abstraction for a media asset
■ Can be URL-based■ Can be stream-based■ Can be inspected for properties■ Can be played with an AVPlayer■ Can be read with an AVAssetReader■ Can be exported with an AVAssetExportSession
Can be written with an AVAssetWriter
45
![Page 46: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/46.jpg)
RosyWriter
• Create an AVAssetWriter
• Append samples in AVCaptureDataOutputs’ delegate callbacks
AVAssetWriter *writer = [[AVAssetWriter alloc] initWithURL:url fileType:AVFileTypeQuickTimeMovie error:&error];
• Create and add AVAssetWriterInputsAVAssetWriterInput *videoInput = [[AVAssetWriterInput alloc] initWithMediaType:AVMediaTypeVideo outputSettings:outputSettings];[videoInput setExpectsMediaDataInRealTime:YES];[writer addInput:videoInput]; dispatch_queue_t serial_queue = dispatch_queue_create(“myQueue”, NULL);[output setSampleBufferDelegate:self queue:serial_queue];[session startRunning];
[writer startWriting];if ( [videoInput isReadyForMoreMediaData] ) [videoInput appendSampleBuffer:sampleBuffer];
46
![Page 47: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/47.jpg)
AVAssetWriter or AVCaptureMovieFileOutputWhich one to use?
• AVCaptureMovieFileOutput■ Requires no set up (automatically uses AVCaptureSessionPreset)■ Can record multiple movie files■ Supports max file size, duration, and minimum free disk space limits■ Does not allow client access to video/audio buffers before writing
• AVAssetWriter■ Requires set up of output settings■ Is a one-shot writer■ Allows client access to video/audio buffers before writing(via AVCaptureVideoDataOutput / AVCaptureAudioDataOutput)
■ Incurs more overhead than AVCaptureMovieFileOutput
47
![Page 48: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/48.jpg)
AVAssetWriterSample video settings
• Settings are defined in <AVFoundation/AVVideoSettings.h> NSDictionary *outputSettings = [NSDictionary dictionaryWithObjectsAndKeys:! ! ! ! AVVideoCodecH264, AVVideoCodecKey,! ! ! ! [NSNumber numberWithInt:1280], AVVideoWidthKey,! ! ! ! [NSNumber numberWithInt:720], AVVideoHeightKey,! ! ! ! [NSDictionary dictionaryWithObjectsAndKeys:! ! ! ! [NSNumber numberWithInt:10500000], AVVideoAverageBitRateKey, ! ! ! ! [NSNumber numberWithInt:1], AVVideoMaxKeyFrameIntervalKey,! ! ! ! AVVideoProfileLevelH264Main31, AVVideoProfileLevelKey, ! ! ! ! nil], AVVideoCompressionPropertiesKey,! ! ! ! nil];
48
![Page 49: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/49.jpg)
AVAssetWriterSample audio settings
• Settings are defined in <AVFoundation/AVAudioSettings.h>
AudioChannelLayout acl = {0};acl.mChannelLayoutTag = kAudioChannelLayoutTagMono;
NSDictionary *outputSettings = [NSDictionary dictionaryWithObjectsAndKeys:![NSNumber numberWithInt:'aac '], AVFormatIDKey,![NSNumber numberWithInt:64000], AVEncoderBitRatePerChannelKey,![NSNumber numberWithInt:1], AVNumberOfChannelsKey,![NSNumber numberWithDouble:44100.], AVSampleRateKey,![NSData dataWithBytes:&layout length:sizeof(layout)], AVChannelLayoutKey,!nil];
49
![Page 50: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/50.jpg)
AVAssetWriterDo’s and don’ts
• DO:■ Set -expectsMediaDataInRealTime to YES for each AVAssetWriterInput■ Set AVAssetWriter’s movieFragmentInterval to preserve recordings in the event of an interruption
■ Set AVCaptureVideoDataOutput’s -alwaysDiscardsLateVideoFrames to NO
• DON’T:■ Hold on to sample buffers outside the data output callbacks■ Take a long time to process buffers in the data output callbacks
50
![Page 51: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/51.jpg)
What Is a Movie Fragment?
• Fast start QuickTime movie
• Non fast start (captured) QuickTime movie
• QuickTime movie with movie fragments
Movie Header Movie Data
Movie Data Movie Header
Header Data F Data F Data F Data F Data F Data F Data F Data F Data F Data F Data
51
![Page 52: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/52.jpg)
Capture Use Cases We Will Cover
• Process video frames from the camera and render with OpenGL• Process live video and audio, and write them to a QuickTime movie• Scan video frames for patterns using the flash and VideoDataOutput• Process captured still images with CoreImage
52
![Page 53: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/53.jpg)
MorseMeDemo
Valentin BonnetiOS Intern
53
![Page 54: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/54.jpg)
LED Torch
MorseMe (Sender)
54
![Page 55: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/55.jpg)
LED Torch
AVCaptureDevice
MorseMe (Sender)
55
![Page 56: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/56.jpg)
MorseMe
Outputs
Inputs Video
AVCaptureDeviceInput
AVCaptureDevice
AVCaptureVideoPreviewLayer
AVCaptureVideoDataOutput
AVCaptureSession
56
![Page 57: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/57.jpg)
MorseMeTorch support
• Torch modes
• Use -hasTorch to determine if the AVCaptureDevice has one• Call -lockForConfiguration: before attempting to set the torch mode
if ( YES == [device lockForConfiguration:&error] ) { [device setTorchMode:AVCaptureTorchModeOn]; [device unlockForConfiguration];}
Flashlight apps no longer need to run an AVCaptureSession to use the torch
AVCaptureTorchModeOff
AVCaptureTorchModeOn
AVCaptureTorchModeAuto
57
![Page 58: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/58.jpg)
MorseMeTorch API enhancements in iOS 5
• Torch availability accessors■ Torch may become unavailable as phone gets too hot■ Key-value observe isTorchAvailable to know when the unit gets too hot, and when it cools down enough
■ Key-value observe the torchLevel property to know when the torch illumination level is decreasing due to thermal duress
58
![Page 59: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/59.jpg)
Capture Use Cases We Will Cover
• Process video frames from the camera and render with OpenGL• Process live video and audio, and write them to a QuickTime movie• Scan video frames for patterns using the flash and VideoDataOutput• Process captured still images with CoreImage
59
![Page 60: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/60.jpg)
‘StacheCamDemo
Using AVCaptureStillImageOutput with CoreImage in iOS 5
60
![Page 61: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/61.jpg)
Outputs
Inputs
Process with CIFaceDetector
‘StacheCam
Video
61
![Page 62: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/62.jpg)
Outputs
Inputs
Process with CIFaceDetector
Video
AVCaptureDeviceInput
AVCaptureSession
AVCaptureStillImageOutput
AVCaptureVideoPreviewLayer
‘StacheCam
62
![Page 63: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/63.jpg)
‘StacheCamiOS 5 enhancements for still images
• AVCaptureStillImageOutput new property isCapturingStillImage■ Key-value observe to know exactly when a still image is being taken■ Handy for driving a shutter animation
• AVCaptureDevice new property subjectAreaChangeMonitoringEnabled■ Allows you to lock focus or exposure, but still receive a notification when the scene has changed significantly, so you can re-focus / expose
63
![Page 64: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/64.jpg)
‘StacheCamiOS 5 enhancements for still images
• CoreImage CIFilters make their iOS debut■ Red-eye reduction■ “Auto-enhance”
• CIDectector finds faces and features• CIImage interfaces with CVPixelBuffers• Specify ‘BGRA’ output for CIImage compatibility
64
![Page 65: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/65.jpg)
‘StacheCamiOS 5 enhancements for still images
CVPixelBufferRef pb = CMSampleBufferGetImageBuffer(imageDataSampleBuffer);CIImage *ciImage = [[CIImage alloc] pb options:nil];
NSDictionary *detectorOptions = [NSDictionary dictionaryWithObject: CIDetectorAccuracyLow forKey:CIDetectorAccuracy];CIDetector *faceDetector = [CIDetector detectorOfType:CIDetectorTypeFace context:nil options:detectorOptions];
NSArray *features = [faceDetector featuresInImage:ciImage options:imageOptions];for (CIFaceFeature *face in features) { // iterate through the faces, find bounds, // left eye, right eye, mouth position}
65
![Page 66: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/66.jpg)
Using Core Image on iOS & Mac OS X MissionThursday 2:00PM
For More on CoreImage in iOS 5
66
![Page 67: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/67.jpg)
SummaryiOS 5 AV foundation capture
• Gives you more CPU cycles• Bridges CoreVideo and OpenGL• Exposes more resolutions• Adds more flexibility to still image capture• Enables you to deliver even cooler apps
67
![Page 68: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/68.jpg)
Eryk VershenMedia Technologies [email protected]
DocumentationAV Foundation Programming Guidehttp://developer.apple.com/library/ios/#documentation/AudioVideo/Conceptual/AVFoundationPG/
Apple Developer Forumshttp://devforums.apple.com
More Information
68
![Page 69: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/69.jpg)
AirPlay and External Displays in iOS apps PresidioTuesday 3:15PM
HTTP Live Streaming Update Nob HillTuesday 4:30PM
Working with Media in AV Foundation Pacific HeightsWednesday 2:00PM
Introducing AV Foundation Capture For Lion Pacific Heights Wednesday 3:15PM
Related Sessions
Exploring AV Foundation PresidioTuesday 2:00PM
69
![Page 70: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/70.jpg)
AV Foundation Lab Graphics, Media & Games Lab CWednesday 9:00AM-1:30PM
QT Kit Lab Graphics, Media & Games Lab AWednesday 9:00AM-1:30PM
AV Foundation Lab Graphics, Media & Games Lab BThursday 9:00AM-1:30PM
HTTP Live Streaming Lab Graphics, Media & Games Lab DWednesday 9:00AM-1:30PM
Labs
QuickTime Lab Graphics, Media & Games Lab DThursday 9:00AM-1:30PM
AirPlay Lab Graphics, Media & Games Lab BWednesday 9:00AM-1:30PM
DAL Lab Graphics, Media & Games Lab CThursday 9:00AM-1:30PM
70
![Page 71: Capturing From the Camera Using AV Foundation on …docs.huihoo.com/apple/wwdc/2011/session_419__capture_from_camera...Blue / Green / Red / Alpha, 8 bits per pixel ... 720p HD Full](https://reader031.vdocuments.us/reader031/viewer/2022020122/5aa201917f8b9a1f6d8ca4cd/html5/thumbnails/71.jpg)
71