photo-realistic 3d model extraction from camera array capture
Post on 14-Apr-2015
Embed Size (px)
DESCRIPTIONThis is a paper that I wrote and co-presented with Callum Reid, CEO of Digicave. We presented it at that year's Hollywood Post Alliance tech retreat, and SMPTE's 2nd international conference on stereo and 3D. This is a really cool topic! Check out digicave's website for example content.
Author(s)First Name John Callum Middle Name Robert Rex Surname Naylor Reid Role Author Author Email firstname.lastname@example.org email@example.com SMPTE Member? Yes No
AffiliationOrganization Timeslice USA Digicave Ltd Pub ID2011 Stereo 3D Conference
Address 415 N. State St, Ste 190, Lake Oswego, OR, 97034 3 Orange Row Brighton, East Sussex, BN1 1UQ Pub Date June 20, 2011
Country USA UK
SMPTE Meeting Presentation
Photo-realistic 3D Model Extraction from Camera Array CaptureJohn R. Naylor B.Sc. (Hons.), M.B.A., C.Eng., M.I.E.TTimeslice USA, Lake Oswego, Oregon, USA, firstname.lastname@example.org
Callum Rex Reid B.A. (Hons.)Digicave Ltd, Brighton, East Sussex, UK, email@example.comThe authors are solely responsible for the content of this technical presentation. The technical presentation does not necessarily reflect the official position of the Society of Motion Picture and Television Engineers (SMPTE), and its printing and distribution does not constitute an endorsement of views which may be expressed. This technical presentation is subject to a formal peer-review process by the SMPTE Board of Editors, upon completion of the conference. Citation of this work should state that it is a SMPTE meeting paper. EXAMPLE: Author's Last Name, Initials. 2010. Title of Presentation, Meeting name and location: SMPTE. For information about securing permission to reprint or reproduce a technical presentation, please contact SMPTE at firstname.lastname@example.org or 914-761-1100 (3 Barker Ave., White Plains, NY 10601).
Written for presentation at theInternational Conference on Stereoscopic 3D for Media and Entertainment Abstract. In which the authors describe a process of capturing photo-realistic 3D computer models using purely passive methods based on subject capture with arrays of cameras, and image analysis to permit the instant capture of both form and texture of the subject or talent. The key performance characteristics of the array are discussed, particularly the challenges of triggering, and the limits imposed on the content created by the native resolution of the cameras used in the array. Details of the rig design and camera layout and configuration for efficient and effective subject capture are presented. The process by which multiple still photographs are processed to produce a point cloud which in turn becomes the model mesh are presented, together with examples of the current state of the art of this approach. Tradeoffs such as the decision to eschew the use of active techniques such as laser scanning, structured light projection, or time-of-flight techniques are discussed, together with their benefits. Keywords. 3D scanning; Full Body; Camera array; Photometrics; Passive scanning; Sculptural Photography, Free Viewpoint Media.
The authors are solely responsible for the content of this technical presentation. The technical presentation does not necessarily reflect the official position of the Society of Motion Picture and Television Engineers (SMPTE), and its printing and distribution does not constitute an endorsement of views which may be expressed. This technical presentation is subject to a formal peer-review process by the SMPTE Board of Editors, upon completion of the conference. Citation of this work should state that it is a SMPTE meeting paper. EXAMPLE: Author's Last Name, Initials. 2010. Title of Presentation, Meeting name and location.: SMPTE. For information about securing permission to reprint or reproduce a technical presentation, please contact SMPTE at email@example.com or 914-761-1100 (3 Barker Ave., White Plains, NY 10601).
IntroductionThe system described in this paper specifically looks at achieving sculptural photography within standard creative industry working practices within realistic budget constraints to be used in online interactive content as Free Viewpoint Media (FVM). Lets start by defining sculptural photography, as an expression of 3D scanning technology.
3D Scanning as Sculptural PhotographyCurrently 3D scanning is mainly used for applications such as approximating someones dress size or creating 3D references for special effects. But we think that it is important to look at what 3D scanning can offer as a way of creating content in its own right; as a tool to empower traditional photographic techniques that can give artists the ability to capture form as well as image. We term this simultaneous capture of form and image Sculptural Photography. With this in mind, sculptural photography systems have to perform under a wide variety of lighting conditions to which some 3D scanning techniques are better suited than others. They also need to be affordable, and the choice of a 3D scanning system is the single biggest cost driver in current solutions. The scanning cost is dictated by simple economics that cause special purpose solutions for niche applications to be orders of magnitude more costly than those that repurpose off-the-shelf components. Sculptural photographic content can be experienced in current interactive platforms as a form of free viewpoint media whereby the end user can explore a real moment from virtual camera perspectives. Technological considerations about the final delivery platform can often influence capture procedures and should be taken into account throughout the process; with the choice between passive and active 3D scanning methods having most influence. Passive Scanning versus Active Scanning 3D Scanning falls into two main categories, passive scanning and active scanning. The active scanning category covers all the techniques that require the projection of light onto the subject to estimate depth. From structured light and infra red (IR) to time-of-flight scanners, these invariably require the form to be captured separately from the natural colours of the subject because the subject is briefly exposed to a patterned light, or they are known to fail on a wide variety of surfaces if they use invisible wavelengths of light due to the different reflection behavior of light at these wavelengths. In contrast, passive scanning uses only the ambient light reflected by the subject, as captured by passive sensors, and sophisticated algorithms to infer the geometry of the form. As an emerging science, a rich variety of techniques have been explored for this purpose; techniques that include stereo photo-metrics, artificial intelligence, Bayesian algebra, and other image analysis techniques. Reliance on tightly calibrated multi-camera systems is a common aspect of most of these, because the capability of passive capture systems depends on the calibration, resolution and image quality of their input. Table 1 introduces some of the key characteristics of 3D scanning, and compares active with passive methods.
Characteristic Typical Capture Components
Active Scanning. Lasers, IR lamps, structured filters, special purpose sensors and processors (e.g. timeof-flight) Normally constrained to be flat Usually progressive
Passive Scanning. Digital stills cameras, ranging from specialist instrumentation units, to off-the-shelf commercial products Can be what the Director of Photography wants Whole frame
Economics Main capability drivers
Niche Market Supply Resolution of custom sensors, some of which are only 200x200. Degree to which wavelengths used behave the same as visible light.
Mass Market Supply Camera resolution, dynamic range, triggering, and sophistication of analysis algorithms
Rolling shutter effects of progressive scan restrict talents ability to move Mass market demand potential
Table 1 Comparison of Passive and Active 3D Scanning Methods From this analysis Digicave and Timeslice chose to develop a sculptural photography system based on passive scanning to better satisfy the technical, artistic, and economic criteria that will enable sculptural photography as a mass market phenomenon. The rest of this paper describes the capture system, and image analysis pipeline that has been developed for sculptural photography; and some of its constraints, and trade-offs. It concludes by discussing delivery mechanisms for sculptural photographs, and steps towards the capture of photographic sculptures in motion.
The Capture SystemCapture systems for sculptural photography are derived from more normal camera array systems and share all of a camera arrays requirements for high quality results. These have been described previously Macmillan (2010), so here we will concentrate on the requirements that are altered or special to sculptural photography: Portability and Calibration Configuration On-set Preview
Portability and CalibrationIt may seem strange to mix the two requirements of portability and calibration under the same heading, until one considers that a system that takes more than an hour to calibrate isnt really portable. Which is why we have developed a calibration method that is quick to execute, yet3
robust, at the cost of a 10% loss in maximum resolution provided by the cameras in use. The process relies on targeting each camera at a common fixation point at which we have placed a tracking target; and centering, leveling, focusing, and framing each camera in the array by eye. The array is then triggered and off-the-shelf motion tracking software used to stabilise the tracking target in the array sequence. This process produces stable-looking sequences, at the cost of lost pixels around the border of each individual frame. The entire initial calibration can be accomplished by an experienced crew in less than an hour. The level of calibration achieved by this process is good enough to produce stabilized sequences for on-set preview, but the stabilized sequence is not used for the extraction of 3D models. This 2nd level of calibration is covered later in this paper. Stayi