virtual cinematography theory and practice for automatic real- time camera control and directing...
TRANSCRIPT
Virtual CinematographyVirtual CinematographyTheory and Practice for Automatic Real-Theory and Practice for Automatic Real-
Time Camera Control and DirectingTime Camera Control and Directing
Liwei He Liwei He Microsoft ResearchMicrosoft Research
http://research.microsoft.com/users/lhehttp://research.microsoft.com/users/lhe
MotivationMotivation
There are 3 elements in computer graphicsThere are 3 elements in computer graphics– lights, scene objects, andlights, scene objects, and
– the camera (focus of this talk)the camera (focus of this talk) Camera control is hard (7 DOFs)Camera control is hard (7 DOFs)
– position (3), direction (3), field of view (1)position (3), direction (3), field of view (1) We may learn from cinematographyWe may learn from cinematography
Roles in cinematographyRoles in cinematography
Low-level responsibilitiesLow-level responsibilities– CameramanCameraman
• Position and move the cameraPosition and move the camera
– Film editorFilm editor• Keep the film to proper lengthKeep the film to proper length• Decide shot transitionsDecide shot transitions
High-level responsibilitiesHigh-level responsibilities– Script writer and directorScript writer and director
• Story-tellingStory-telling
Camera control level 1Camera control level 1
Input: camera position and directionInput: camera position and direction Output: camera transformationOutput: camera transformation Application: low-level graphics library Application: low-level graphics library
routines (Direct3D)routines (Direct3D)
Camera control level 2Camera control level 2
Input: show both A and B, follow C, etc.Input: show both A and B, follow C, etc. Output: camera position and directionOutput: camera position and direction Application: 3D game, computer Application: 3D game, computer
animation packageanimation package
Camera control level 3Camera control level 3
Input: high-level user directionsInput: high-level user directions– show a conversation show a conversation
– show a car chase sceneshow a car chase scene Output: a sequence of level 2 camera specOutput: a sequence of level 2 camera spec Applications: graphical chat, tele-Applications: graphical chat, tele-
conferencing, Virtual Reality gamesconferencing, Virtual Reality games
Camera control level 3 (cont.)Camera control level 3 (cont.)
Encodes cinematographic expertiseEncodes cinematographic expertise– an interesting problem in itselfan interesting problem in itself
Provides an interface that isProvides an interface that is– real-timereal-time
– visually entertainingvisually entertaining
– more informativemore informative
Eyepeep BARG07011 Feldegast
Microsoft V-Chat Lunar Island
Eyepeep says "really"Feldegast says "yup"BARG07011 has joined the conversationEyepeep says "im still older"Feldegast says "lol"DGeste309 does something silly.
BARG07011DGerste309EyepeepFeldegastGrenDayGod
Principles of CinematographyPrinciples of Cinematography
Film structureFilm structure
FilmFilm
......
......
ScenesScenes
ShotsShots
Camera distanceCamera distance
Close upClose up Close shotClose shot Medium shotMedium shot
Full shotFull shot Long shotLong shot
The line of interestThe line of interest
A B
External cameraExternal camera
A B
External cameraExternal camera
A B
Internal cameraInternal camera
A B
Apex cameraApex camera
A B
Moving camerasMoving cameras
A B
pan(A) track(B)
Some rules in film editingSome rules in film editing
Don't cross the line of interestDon't cross the line of interest Avoid jump cuts Let the actor lead Break movement
Don’t cross the line of interestDon’t cross the line of interest
1
2
Some rules in film editingSome rules in film editing
Don't cross the line of interest Avoid jump cutsAvoid jump cuts Let the actor lead Break movement
Avoid jump cutAvoid jump cut
Some rules in film editingSome rules in film editing
Don't cross the line of interest Avoid jump cutsAvoid jump cuts Let the actor lead Break movement
Two-person conversation Two-person conversation
A B
11 22
33 44
2
3
1
A BA B
A
4 B
Three-person conversationThree-person conversation
22
44
33
11
B
A
C
43
2
A B
C
A B
1
A B
C
Bad
Good
x
y
z
U n iv e rse
Three spacesThree spaces
The universe space
Three spacesThree spaces
The eye space (z-axis is the look at direction)
x
y
z
U n iv e rse
zE y e
x y
Three spacesThree spaces
The screen space (screen is ctan(fov/2) away from eye point)
x
y
z
E y eS c re e n
-1 + 1
+ 1
-1
Standard look-at transformationStandard look-at transformation
Given eye position Pfrom, a look at direction
Want a rotation matrix R and a translation to transform a vector from universe space to eye space
x y
TT
x
y
z
U n iv e rse
P f ro m
E y e
x
y
z
U n iv e rse
P f ro m
E y e
Standard look-at transformationStandard look-at transformation
But this is not uniqueT
xy
x
y
z
U n iv e rse
P f ro m
E y e
Standard look-at transformationStandard look-at transformation
But this is not uniqueT
x y
x
y
z
U n iv e rse
P f ro m
E y e
Standard look-at transformationStandard look-at transformation
But this is not uniqueT
yx
x
y
z
U n iv e rse
P f ro m
E y e
Standard look-at transformationStandard look-at transformation
We will need to specify an up vector , usually [0,1,0]U
U
T
x
y
z
U n iv e rse
P f ro m
E y e
Standard look-at transformationStandard look-at transformation
In eye space
Z-axis is
X-axis is
Y-axis is
Rotation matrix
Translation is -Pfrom
T
ZUV ˆˆˆ
VTU ˆˆˆ
TUVR ˆˆˆ33
UU V
T
Internal close-up of actor AInternal close-up of actor A
x
y
z
U n iv e rse
P f ro m
E y eP a t
Look at [xLook at [xatat,y,yatat] transformation] transformation
U V
TGiven eye position Pfrom, look at point Pat, and Pat in screen space [xat, yat]
Want
In universe space
T
A
)(ˆfromat PPunitA
Look at [xLook at [xatat,y,yatat] transformation] transformation
In screen space:
H = [xat, yat, ctan(fov/2)]
Solve from the following equations: x
y
z
E y eS c re e n
-1 + 1
+ 1
-1
P f ro m
P a t
H
1ˆˆ
ˆˆ]0,1,0[ˆ
ˆˆ]1,0,0[ˆ
UT
UAH
TAH
T
External of actor A and actor BExternal of actor A and actor B
x
y
z
U n iv e rse
P f ro m
E y eP B
P A
Look at [xLook at [xAA,y,yAA] and [x] and [xBB,y,yBB]]
U V
T
Given PA,PB in universe space and eye space, and distance of eye position Pfrom and Pa.
Want Pfrom and T
A
Use numerical method:
1. Set Pfrom to 0
2. Solve R using the Look at [xat,yat] method
3. Transform E by inverse of R, getting a new Pfrom approximation
4. Goto 2
Look at [xLook at [xAA,y,yAA] and [x] and [xBB,y,yBB]]
x
y
z
E y eS c re e n
-1 + 1
+ 1
-1
P f ro m
P B
H
P A
Medium panning shot of actor AMedium panning shot of actor A
ReferencesReferences
Jim Blinn’s CG&A ’88 articleJim Blinn’s CG&A ’88 article– Where am I? What am I looking at?Where am I? What am I looking at?
Steve Drucker’s Interactive 3D ’92, ’94, Steve Drucker’s Interactive 3D ’92, ’94, ‘95 papers‘95 papers– Cast camera positioning as an Cast camera positioning as an
optimization problemoptimization problem
System DesignSystem Design
Overall system diagramOverall system diagram
Real-timeApplication
Virtual Cinematographer
Renderer
At each time stepAt each time step
1. Application generates events to VC1. Application generates events to VC– ( subject, verb, object )( subject, verb, object )
2. VC determines– camera specifications
– acting hints
3. Renderer outputs the image
Real-timeApplication
Virtual Cinematographer
Renderer
At each time stepAt each time step
1. Application generates events to VC– ( subject, verb, object )
2. VC determines2. VC determines– camera specificationscamera specifications
– acting hintsacting hints
3. Renderer outputs the image
Real-timeApplication
Virtual Cinematographer
Renderer
At each time stepAt each time step
1. Application generates events to VC– ( subject, verb, object )
2. VC determines– camera specifications
– acting hints
3. Renderer outputs the image3. Renderer outputs the image
Real-timeApplication
Virtual Cinematographer
Renderer
A networked virtual party gameA networked virtual party game
Actors are simulatedActors are simulated
Actor’s mind
LonelinessLoneliness
ThirstThirst
BoredomBoredom
Walk, converse, look around, Walk, converse, look around, drink, etc.drink, etc.
Can be controlled by the usersCan be controlled by the users
Networked virtual party gameNetworked virtual party game
Server
Client
Client
Client
User actionsUser actions
EventsEvents
Inside the Virtual Inside the Virtual CinematographerCinematographer
VC architectureVC architecture
Idioms
Camera modules
VC architectureVC architecture
Idioms
Camera modules
Camera modulesCamera modules
Geometric placement of specific Geometric placement of specific cameras for each shotcameras for each shot
Choose the side of the line of interest Influence acting
Camera module: Camera module: ext(B,A)ext(B,A)
A B
Static camera modulesStatic camera modules
A B
ext(B,A) ext(A,B)
apex(A,B)
int(B)
Moving camera modulesMoving camera modules
A B
pan(A) track(B)
Camera modulesCamera modules
Geometric placement of specific cameras for each shot
Choose the side of the line of interestChoose the side of the line of interest Influence acting
Don’t cross the line of interestDon’t cross the line of interest
1
2
Camera modulesCamera modules
Geometric placement of specific cameras for each shot
Choose the side of the line of interest Influence actingInfluence acting
Acting hintsActing hints
A BA’A’ B’B’
VC architectureVC architecture
Idioms
Camera modules
Previous worksPrevious works
Dave Christianson’s AAAI ’96 paperDave Christianson’s AAAI ’96 paper– Declarative camera control for automatic Declarative camera control for automatic
cinematographycinematography
– An off-line algorithmAn off-line algorithm
– Need to know all events aheadNeed to know all events ahead
– Use plan algorithms in AIUse plan algorithms in AI
Film idiomsFilm idioms
Capture a particular type of sceneCapture a particular type of scene Register relevant eventsRegister relevant events Select shot typesSelect shot types Determine the transitions between shotsDetermine the transitions between shots
Hierarchical idiom structureHierarchical idiom structure
LookaroundLookaround
MasterMaster
ConverseConverse DrinkingDrinking MovingMoving
3Talk3Talk
2Talk2Talk
Hierarchical idiom structureHierarchical idiom structure
Avoids exponential growth of statesAvoids exponential growth of states Allows expertise to be reusedAllows expertise to be reused Provides robustnessProvides robustness
MovingMoving idiom idiom
track(C)pan(A)
ext(A,B) ext(B,A)
apex(A,B) apex(A,B)
MovingMoving idiom idiom
track(C)pan(A)
ext(A,B) ext(B,A)
apex(A,B) apex(A,B)
T > 8
MovingMoving idiom idiom
track(C)pan(A)
ext(A,B) ext(B,A)
apex(A,B) apex(A,B)
T > 8
MovingMoving idiom idiom
track(C)pan(A)
ext(A,B) ext(B,A)
apex(A,B) apex(A,B)D < 10
MovingMoving idiom idiom
track(C)pan(A)
ext(A,B) ext(B,A)
apex(A,B) apex(A,B)
D < 5
MovingMoving idiom idiom
apex(A,B)
D < 5
apex(A,B)
pan(A)
ext(B,A)
T > 8
ext(A,B)
D < 10
T > 8
3Talk3Talk idiom idiom
int(C)
int(C)
ext1to2
2Talk(A,B)
Invoking a sub-idiomInvoking a sub-idiom
int(C)
int(C)
ext1to2
2Talk(A,B)
ext(B,A)ext(B,A)
ext(A,B)
ext(A,B) 2Talk
ContributionsContributions
Real-time camera controlReal-time camera control– lightweightlightweight
– automatic camera placementautomatic camera placement
– automatic shot transitionsautomatic shot transitions Do a reasonable jobDo a reasonable job
– not to replace human in computer animationnot to replace human in computer animation
Work in ProgressWork in Progress
Camera control for chatCamera control for chat
Camera control for live meetingCamera control for live meeting
Image after warpingImage after warping
Camera control for classroomCamera control for classroom
Multiple cameras per classroomMultiple cameras per classroom– Lecturer camerasLecturer cameras
– Audience cameraAudience camera Use audio and vision techniques toUse audio and vision techniques to
– Track the lecturerTrack the lecturer
– Activate the audience cameraActivate the audience camera
Dramatic lightingDramatic lighting
Virtual CinematographyVirtual CinematographyTheory and Practice for Automatic Real-Theory and Practice for Automatic Real-
Time Camera Control and DirectingTime Camera Control and Directing
Liwei He Liwei He Microsoft ResearchMicrosoft Research
http://research.microsoft.com/users/lhehttp://research.microsoft.com/users/lhe