drone cinematographyicarus.csd.auth.gr/.../08/1-drone-cinematography.pdf · • novel aerial shot...

This project has received funding from the European Union’s Horizon 2020 research

and innovation programme under grant agreement No 731667 (MULTIDRONE)

Contributors: N. Nikolaidis, I. Mademlis, I. Pitas

Aristotle University of Thessaloniki, Greece

Presenter: Prof. Ioannis Pitas

Aristotle University of Thessaloniki [email protected] www.multidrone.eu

Presentation version 1.2

Drone Cinematography

http://www.multidrone.eu/




• Unmanned Aerial Vehicles (UAVs, or “drones”) have made their way

in the media and entertainment industry. • Movies

• TV: sports, newsgathering, documentaries

• Advertising.

• Drones allow filmmakers to capture shots that were impossible,

extremely difficult or costly to capture so far: • Wide shots of remote or hard-to-reach landscapes

• Follow-along close up views of sports or other action

• Impressive fly-throughs, fly-bys and fly-overs.

• Essentially drones provide a level of camera motion freedom that

was so far available only in animation.



Skynamic GmbH Demo Reel, www.skynamic.tv




• Value of current business services and labor that will

be replaced in the near future by drone-based

solutions, in the media & entertainment sector:

$8.8bn (Source: PricewaterhouseCoopers).




• Example: rooftop motorbike chase scene from the J. Bond film 007:

Skyfall (MGM, Sony Pictures, Columbia Pictures, 2012).

Drone Cinematography in

Movies



• Example: party scene from the film The Wolf of Wall Street

(Paramount Pictures, 2013).


Movies



• Numerous other films, TV series and documentaries

include scenes shot using drones:

• 007: Spectre (MGM, Columbia Pictures, 2015)

• Jurassic World (Universal Pictures, 2015)

• Captain America: Civil War (Walt Disney Studios, 2016)

• Game of Thrones (HBO, 2011-)

• Narcos (Netflix, 2015-)

• Planet Earth II (BBC, 2016).


Movies and TV series




Movies and TV series

GOT (HBO) Planet Earth II (BBC)



• Major channels including Fox Sports, BBC, and NBC are using

drones to cover outdoor sports events (live or post-produced):

• Golf, auto-racing, skiing, rowing (2016 Olympics).

Dakar Rally 2016. (source Time.com, Franck Fife—

AFP/Getty Images)

Golf Channel


Sports Coverage



• TV channels are using drones to cover large-scale events

(e.g., demonstrations), or capture videos from war zones,

natural disasters etc.

War-torn Aleppo, Syria (AMC)


News

Moscow protest



• Drones can replace spidercams, camera dollies, cranes

and helicopters.




• UAVs advantages over dollies, cranes and helicopters:

• Reduced cost:

• Cost of renting a helicopter for a few hours equals the cost of buying a well-

equipped drone.

• Fast and adaptive shot setup.

• Access to narrow spaces, versatility, flights close to humans and

buildings.




• Fewer legal and safety restrictions compared to helicopters.

• Novel aerial shot types.

• Continuous shootings with seamless transitions between very

different viewpoints/environments:

• E.g.: the camera moves from ground level to several meters above ground

and then indoors, in a single shot.




• The media and entertainment industry has already praised the

contribution of UAVs in modern cinematography:

• A number of drone companies (DJI, PictorVision, Aerial MOB, Flying Cam, etc)

won a 2017 Emmy Award for Technology and Engineering for their Low

Latency Remote Controlled Airborne Video Platforms for Television.

• G. Hooper and P. George of HoverCam were awarded a 2013 Academy of

Motion Picture Arts and Sciences (AMPAS) Technical Achievement Award

for the Helicam miniature helicopter camera system.

• E. Prévinaire, J. Sperling, E. Brandt and T. Postiau were awarded a 2013

AMPAS Scientific and Engineering Award for the Flying-Cam SARAH 3.0

system.




• UAV cinematography is mainly derived from traditional

ground and aerial cinematography, but must also take

into account UAV-specific limitations, capabilities,

properties.

• A visual vocabulary of UAV cinematography can be

defined, consisting in:

• Shot types

• Combinations of framing shot types + UAV / camera

motion types.

© 2009, Jim Zuckerman

UAV Cinematography



• Composition principles:

• Central Composition.

• Rule of Thirds.

• Lighting rules.

• Depth-of-Field / Focus settings.

• Need to define a standardized UAV shot type taxonomy.

UAV Cinematography



• Actual drone footage and related articles / guidelines were used to this

end:

• A total of 8 framing (static) shot types and 26 UAV / camera motion types

suitable for UAV media production have been identified.

• Camera motion types were clustered into groups according to their

characteristics.

• Visually pleasing combinations of framing shot types and camera motion types

were identified.

UAV Cinematography



• Framing Shot Types are more or less those of traditional

cinematography.

• Most are defined based on the percentage of the video frame width /

height covered by the single target / subject.

FRAMING SHOT TYPE Percentage of frame width/height

covered by target

Extreme Long Shot (ELS) <5%

Very Long Shot (VLS) 5-20%

Long Shot (LS) 20-40%

Medium Shot (MS) 40-60%

Medium Close Up (MCU) 60-75%

Close Up (CU) >75%

Framing Shot Types



• A couple of Framing Shot Types deal with two or more

subjects / targets:

• 2 Shot / 3 Shot: 2/3 subjects appear on frame, equally visible

(typically LS or MS).

• Over the Shoulder (OTS): Adapted from traditional

cinematography OTS. Main target fully visible, secondary target

visible at the video frame edge.

Framing Shot Types



• Example UAV shot types when shooting boat targets from the side.

Extreme Long Shot Long Shot

Medium Close Up Two Shot

Framing Shot Types



• UAV / camera motion types can be considered as either

“scene-oriented” or “target-oriented”.

• Four groups of UAV / camera motion types were defined.

• Static shots (6). No UAV motion, target may or may not be

present:

• Static Shot (SS)

• Static Shot of Still Target (SSST)

• Static Shot of Moving Target (SSMT)

• Static Aerial Pan (SAP)

• Static Aerial Tilt (SAT).

UAV / Camera Motion Types



• Dynamic shots (6). Moving UAV, no target:

• Moving Aerial Pan or Tilt (MAP, MAT)

• Pedestal / Elevator Shot (PS)

• Bird’s Eye Shot (BIRD)

• Moving Bird’s Eye Shot (MOVBIRD)

• Survey Shot (SURVEY)

• Fly-Through (FLYTHROUGH).




• Target Tracking shots (11). UAV motion depends on target

motion:

• Moving Aerial Pan with Moving Target (MAPMT)

• Moving Aerial Tilt with Moving Target (MATMT)

• Lateral Tracking Shot (LTS)

• Vertical Tracking Shot (VTS)

• Orbit (ORBIT)

• Fly-Over (FLYOVER)

• Fly-By (FLYBY).




• Target Tracking shots (continued)

• Chase/Follow Shot (CHASE)

• Descent (DESCENT)

• Descent-Over (DESCENTOVER)

• Ascent (ASCENT)




Source: Youtube: "5 Drone Moves Every Flier

Should Know",

https://www.youtube.com/watch?v=1hz-

lkx4o6c

Lateral Tracking Shot (LTS)

• Camera stays focused on the

moving target.

• UAV flies sideways / in parallel to

the target, matching its speed.


https://www.youtube.com/watch?v=1hz-lkx4o6c





Source: Youtube: "Drone footage of Cycle-

Racing",

https://www.youtube.com/watch?v=vQ94R

4LC9ig

Chase/Follow (CHASE)

• Camera stays focused on the

moving target.

• UAV follows / leads the target

from behind / from the front,

matching its speed.


https://www.youtube.com/watch?v=vQ94R4LC9ig

https://www.youtube.com/watch?v=vQ94R4LC9ig



Source: Youtube: "Drone Chasing

Horses",

https://www.youtube.com/watch?v=O0

uw8py9qmw

Orbit (ORBIT)

• Camera gimbal is slowly rotating,

so as to keep the still or moving

target properly framed.

• UAV circles around the target while

following its trajectory (if any).


https://www.youtube.com/watch?v=O0uw8py9qmw

https://www.youtube.com/watch?v=O0uw8py9qmw




Static Aerial Tilt (SAT)

• UAV hovers.

• Camera gimbal rotates slowly

around the pitch axis in order to

capture the scene context.

Source: Youtube: "5 Drone Moves

Every Flier Should Know",

https://www.youtube.com/watch?v

=1hz-lkx4o6c








Source: Youtube: "How to

film amazing aerials with

your drone |

DroneFilmSchool",

https://www.youtube.com/wat

ch?v=zmQcSdj7vbE

Moving Bird’s Eye Shot

(MOVBIRD)

• Camera remains stable facing

vertically down.

• UAV is slowly flying parallel to

the terrain with constant velocity.

https://www.youtube.com/watch?v=zmQcSdj7vbE

https://www.youtube.com/watch?v=zmQcSdj7vbE




Source: Youtube: "Manor

House Stables - Drone Film -

Horses",

https://www.youtube.com/wat

ch?v=56qOnxc28dw

copyright owner: M7 Aerial

Survey Shot

• Camera remains stable facing

ahead or backwards.

• UAV is slowly flying parallel to

the terrain with constant velocity.

https://www.youtube.com/watch?v=56qOnxc28dw

https://www.youtube.com/watch?v=56qOnxc28dw



• Dynamic Target shots (3): a target exists, but UAV trajectory also depends on

other factors

• Constrained Lateral Tracking Shot (CONLTS)

• Pedestal/Elevator with Target (PST)

• Reveal Shot (REVEAL) Constrained Lateral Tracking Shot

(CONLTS)

• Camera remains stable, focused on the

moving target.

• UAV follows the target but it is

constrained to move onto a pre-defined

“flight plane” vertical to the ground plane.

• Useful in sports, e.g. football.




• Motion types involving 2 or more drones in orchestrated

motion can be also considered.

Multiple UAV Motion Types



• The drone trajectories and the camera gimbal motions of the

identified UAV motion types can be modeled mathematically, using

two coordinate systems:

• A global, fixed, World Coordinate System (WCS).

• A Target Coordinate System (TCS), attached and moving along with the target.

• Such a modeling can prove useful for several tasks.

• Example: analytically determine maximum allowable camera focal length fmax

when the UAV orbits a moving target, so that 2D visual tracking does not fail.

• fmax is a function of the current UAV position relatively to the target and the

current deviation of the target velocity from the expected one.

UAV Motion Type Modeling



• Example: ORBIT. • The camera always looks at the target while the drone moves on a circle in the

TCS

• 𝐏𝑡 = 𝑝𝑡1, 𝑝𝑡2, 𝑝𝑡3𝑇 : Target position in WCS

• 𝐗𝑡 = 𝑥𝑡1, 𝑥𝑡2, 𝑥𝑡3𝑇 : Drone position in TCS

• 𝐋𝑡: Camera Look-At Point in WCS




• The following expression is derived for maximum focal

length, so that 2D visual tracking does not fail:




• A UAV cinematographer has the following means for

achieving a certain framing shot type or UAV/camera

motion type or for transitioning between types:

• UAV location and orientation.

• Camera orientation through

gimbal control (usually 3 degrees

of freedom: roll, pitch, yaw).

• Camera focal length.

Tools of the Trade



• Certain shot types can be obtained by more than one camera / UAV

configurations:

• The simplest, less energy consuming shall be chosen.

• Obtaining a certain framing shot type can simply be achieved by

setting the focal length f (zoom level).

• Zoom in to obtain a Medium Close Up.

• The same shot type can be achieved by moving the drone with

respect to the subject :

• More complex and consumes battery power.

Tools of the Trade



• Certain UAV / camera motion types can also be

approximately materialized by means of changing only the

camera / gimbal configuration.

• These approximations can / shall be preferred, due to a

number of reasons:

• Inability to fly along the trajectory specified by the motion type

(e.g., due to proximity to humans).

• Inability to match the target speed.

• Power consumption considerations.

Tools of the Trade



• Example: CHASE (follow) from behind a target moving uphill.

• Due to energy consumption considerations, the same shot can be

approximately materialized by:

• Having the drone stop following the target and start hovering.

• Continuously changing the camera zoom and camera gimbal orientation so as

to always have the target in the frame center and with the same size.

Tools of the Trade



• Current state of the art in UAV cinematography: single

drone, manually operated. • Two persons are needed for professional productions: drone

operator, cameraman.

• Single-UAV shooting limitations:

• A single target/subject can be captured at each time instance.

• The target may only be captured from a specific viewpoint and

with specific shot type (e.g., Close-Up) at a given time instance.

• When UAV travels from one location to another to, e.g., aim at a

different target, the corresponding video might not be useful for

broadcasting.

Current State of the Art, Limitations & Challenges



• Multiple UAVs (“swarm”) can alleviate these limitations:

• Can cover multiple targets simultaneously or the same target from different

viewpoints, allowing the director to choose the preferred video feed.

• Can cover wide-area events.

• Can be “relayed” in order to capture the moving target for a longer period/distance.

• Example: a UAV following a target is soon expected to enter a low-battery mode and land.

• Another drone can take off and continue the task.




• Multiple UAV shooting

challenges: • Drone to drone collision

avoidance.

• Avoid having one drone entering

the field of view of another

drone.

• Coordination.

• Crew size may increase.




• Manual control limitations:

• Difficult to achieve complex UAV motions especially when

combined with coordinated camera gimbal movement.

• Orbit around a static target, camera always

pointing to the target.

• Additional difficulties when the target is

moving :

• Orbit around a moving target.

• Follow a fast/unpredictably moving target

physically or with the camera.




• Autonomous / intelligent drone / camera operation can solve those issues or

greatly facilitate drone / camera control.

• Many technologies, sensors and research areas are involved:

• Cameras (standard, stereo, depth, infrared), LIDARS, ultrasonic sensors, GPS on the

target.

• Vision and machine learning on the generated data, path planning, control, etc.

• Moving target detection and 2D / 3D tracking

• Detection of events (e.g., fall of a cyclist) and points of interest

• Automatic planning and execution of shots

• Automatic framing / camera control:

• Rule of 3rds or central composition

• avoid having a drone in the field of view of the other

• 180 degrees rule.

• Selection of the most informative or best framed video stream.




• The technology is already here!?

• Example. DJI Phantom 4 Pro can:

• Physically follow a moving target from behind or from the side.

• Orbit around a moving target by just using the roll stick.

• Follow (lock) a moving target with the camera while the drone moves under the

operator commands.

• Recognize various targets such as humans, bikes or motorcycles.

• Obvious questions that need to be answered: accuracy, failure rates

and robustness in:

• Cluttered environments.

• Dynamic scenes with harsh / changing lighting conditions.

• Fast and unpredictably moving targets.




• Each UAV / camera motion type requires a certain

technology level to be performed autonomously.

• Three technology levels can be identified:

• “Minimum”: No visual data analysis or 3D target localization

information is required for shooting such motion types.

• Example: scene-oriented shots with simplistic UAV trajectories

• “Vision”: Visual data analysis is required.

• “Vision+3D”: Both visual data analysis and 3D target (and drone)

localization are needed.




• Automated / intelligent UAV cinematography has

recently arise as an interesting R & D area.

• Algorithms and software products have been developed for

automating aspects of drone flight and shooting, while

obeying cinematographic restrictions and director

guidelines.

• The field lies at the intersection of: • Aerial cinematography.

• Aerial robotics.

• Computer vision/machine learning.

Research and Products



• Joubert et al (2015) present a tool to support drone video capture of

static scenes.

• The tool enables users to: • Specify shots visually, using keyframes for the drone camera location and the

camera look-at point. • The system then generates drone trajectories that

satisfy the equations of motion.

• Precisely control the timing of shots using easing

curves. • When the user-defined trajectory exceeds any

physical limits the system warns the user to modify it.

N. Joubert, M. Roberts, A. Truong, F. Berthouzoz, and P. Hanrahan,

“An interactive tool for designing quadrotor camera shots,”

ACM Transactions on Graphics (TOG), vol. 34, no. 6, pp. 238, 2015 (SIGGGRAPH Asia 2015).




• Preview the resulting

shots in a 3D

environment.

• Capture the resulting

shots in the real world

using a UAV.




• Joubert et al (2016) present a system to capture drone video footage

of human subjects performing rather limited movements:

• Subjects are tracked using RTK GPS and IMU sensors.

• The system automatically captures static (framing) shots that respect visual

composition principles (rule of 3rds).

• It also automatically calculates transitions (drone trajectories and camera

parameter configurations) between these shots.

• Evaluated transitions are feasible, safe (e.g., not too close to humans), and

visually pleasing.

N. Joubert, D. B. Goldman, F. Berthouzoz,

M. Roberts, J. A. Landay, and P. Hanrahan,

“Towards a drone cinematographer: Guiding

quadrotor cameras using visual composition

principles”, arXiv:1610.01691, 2016.




• Nageli et al (2017) propose a method for aerial videography planning in cluttered

and dynamic environments.

• The method takes as input user specified, high-level plans (paths) and framing

objectives (position and size of filmed person in the frame).

• The algorithm adapts the high-level plans in real-time to produce feasible drones

trajectories, while also taking the motion of the subjects into account, e.g., to avoid

collisions.

[Nageli2017b]: T. Nageli, L. Meier, A. Domahidi, J. Alonso-Mora, O. Hilliges, “Real-time Planning for Automated Multi-view Drone Cinematography”,

ACM Transactions on Graphics, vol. 36, no. 4, pp. 132:1-132:10, SIGGRAPH 2017.

[Nageli2017a]: T. Nageli, J. Alonso-Mora, A. Domahidi, D. Rus, O. Hilliges, “Real-Time Motion Planning for Aerial Videography With Dynamic

Obstacle Avoidance and Viewpoint Optimization”, IEEE Robotics and Automation Letters, vol. 2, no. 3, pp. 1696-1703, 2017.




• The algorithm can also handle multiple drones:

• Drone to drone collision avoidance.

• Drone path modification to prevent a drone from entering the field of view of

other drones.

[Nageli2017b]: T. Nageli, L. Meier, A. Domahidi, J. Alonso-Mora, O. Hilliges, “Real-time Planning for Automated Multi-view Drone Cinematography”,

ACM Transactions on Graphics, vol. 36, no. 4, pp. 132:1-132:10, SIGGRAPH 2017.

[Nageli2017a]: T. Nageli, J. Alonso-Mora, A. Domahidi, D. Rus, O. Hilliges, “Real-Time Motion Planning for Aerial Videography With Dynamic

Obstacle Avoidance and Viewpoint Optimization”, IEEE Robotics and Automation Letters, vol. 2, no. 3, pp. 1696-1703, 2017.




• Skywand: • A VR system allowing the user to explore a 3D model of the scene and place desired,

example key-frames in the environment.

• The system computes the UAV trajectory and the sequence of camera motions, so as

to capture smooth footage containing the key-frames.

• Previsualization of the trajectory and the video is available.

• Freeskies CoPilot : • Similar functionality with a 3D map instead of VR.




Open Issues

• What traditional cinematography rules persist in UAV cinematography?

• What traditional rules can or should be broken to create engaging /

interesting videos?

• What opportunities are there for new rules of cinematography, new

shot types, camera movements and visual effects?

• Preserve / enhance narrative and engagement, create a wow factor, while

preserving viewing quality and comfort.

• Establish a grammar & taxonomy of drone cinematography

• How these new visual paradigms can influence visual narrative?



Open Issues

• What are the operating envelopes for the involved parameters that

lead to acceptable / engaging visual results?

• Drone movement and speed relative to the target.

• Camera zoom factor & orientation and their rate of change.

• Target dynamics.

• What is the effect of errors (target tracking errors, drone position and

orientation errors) in the visual results?

• Subjective and, if possible, objective tests and metrics are needed.

• Media professionals shall be consulted.



Open Issues

• What are the appropriate types of transitions between

different shots?

• How one can use the new capture and display formats (360

video, VR, plenoptic) within the scope of UAV

cinematography?



Tools and Research Needed

• R & D in computer vision and

machine learning.

• Intuitive and informative graphical

user interfaces for planning the

drone and camera / gimbal

movements.

• Tools (simulators) for the pre-

visualization of the drones

trajectories and the videos

acquired by the cameras.



Q & A

Thank you very much for your attention!

Contact: Prof. I. Pitas

[email protected]

www.multidrone.eu

drone cinematographyicarus.csd.auth.gr/.../08/1-drone-cinematography.pdf · • novel aerial shot...

Documents