1intelligent tools for media workshop in media authoring automatic generation of animation and video...
TRANSCRIPT
11Intelligent Tools for MediaIntelligent Tools for Media
Workshop in Media AuthoringAutomatic Generation of Animation and Video
ContactDoron Friedman
[email protected]: 054-461807
22Intelligent Tools for MediaIntelligent Tools for Media
AgendaAgendaIntroductionIntroduction Motivation and BackgroundMotivation and Background Course MaterialCourse Material
Logistics and AdministrationLogistics and Administration
Description of ProjectsDescription of Projects
33Intelligent Tools for MediaIntelligent Tools for Media
Motivation – Media Authoring Motivation – Media Authoring
3D:3D:Professional animators – several years of studyProfessional animators – several years of study
Professional softwareProfessional software
Low leve of abstaction Low leve of abstaction
Very little reuseVery little reuse VideoVideo
Professional editors Professional editors
Professional softwareProfessional software
Pixel based (rather than object based)Pixel based (rather than object based)
44Intelligent Tools for MediaIntelligent Tools for Media
Smart GraphicsSmart GraphicsWhat this is NOT aboutWhat this is NOT about How to achieve efficient 3D renderingHow to achieve efficient 3D rendering
What this IS aboutWhat this IS about How to quickly generate animation from high level How to quickly generate animation from high level
inputinput
55Intelligent Tools for MediaIntelligent Tools for Media
Making movies (Simplified)Making movies (Simplified)
Story/idea Script Shooting Script Animation File
•Story- understanding•Storytelling•Directing
•NLU •Reusable animation •Animation composition•Text-to-speech
•Theory of space and time •Camera-control principles•Model of actions
66Intelligent Tools for MediaIntelligent Tools for Media
Script -> AnimationScript -> Animation
Formal Script Shooting Script Animation File
•Theory of space and time•Camera-control principles•Model of actions
•Reusable animation •Animation composition•Text-to-speech
77Intelligent Tools for MediaIntelligent Tools for Media
Interactive Virtual EnvironmentsInteractive Virtual Environments
Linear AnimationLinear Animation AuthoringAuthoring
Interactive Virtual EnvironmentsInteractive Virtual Environments AuthoringAuthoring AgentAgent Real-time DirectorReal-time Director
88Intelligent Tools for MediaIntelligent Tools for Media
ApplicationsApplicationsProduction IndustryProduction Industry
Rapid prototypingRapid prototyping StoryboardingStoryboarding
Gaming IndustryGaming Industry Engine for interactive Engine for interactive
gamesgames A movie-maker gameA movie-maker game Toys (Lego/AIBO)Toys (Lego/AIBO)
Education Education
TrainingTraining
SimulationsSimulations IndustrialIndustrial MilitaryMilitary Traffic Traffic ScientificScientific Medical Medical
99Intelligent Tools for MediaIntelligent Tools for Media
State of the trade todayState of the trade today Leading tools (Maya, RenderMan, ..)Leading tools (Maya, RenderMan, ..) Leading standards/trends (VRML, Java3D, …)Leading standards/trends (VRML, Java3D, …) How authors work todayHow authors work today
1010Intelligent Tools for MediaIntelligent Tools for Media
SMIL-1SMIL-1- - TimeTime
ParPar SeqSeq
beginbegin
•Layout–Relative/absolute positioning–Z-index–Fit (meet, slice, scroll)
1111Intelligent Tools for MediaIntelligent Tools for Media
SMIL-2SMIL-2
45 Modules / Language Profiles45 Modules / Language ProfilesIntegration with other languagesIntegration with other languagesVector graphicsVector graphicsExcl (exclusive par)Excl (exclusive par)Relative synchronizationRelative synchronizationTime manipulation (speed, reverse)Time manipulation (speed, reverse)Animation Animation AudioLayoutAudioLayoutTransitionsTransitions
1212Intelligent Tools for MediaIntelligent Tools for Media
MPEG-4 as a Rich Media MPEG-4 as a Rich Media Representation SchemeRepresentation Scheme
Several goalsSeveral goals
Profile ApproachProfile Approach
1313Intelligent Tools for MediaIntelligent Tools for Media
Mpeg-4 Relevant FeaturesMpeg-4 Relevant Features
•TermCap node:
–Frame rate
–Memory/CPU load
•BIFS switch node
1414Intelligent Tools for MediaIntelligent Tools for Media
MPEG-7 – the Bits about the MPEG-7 – the Bits about the BitsBits
D – Descriptors – representation of features (syntax + D – Descriptors – representation of features (syntax + semantics)semantics)
DS – Description Schemes – structure and semantics of DS – Description Schemes – structure and semantics of relationships between componentsrelationships between components
DDL – Description Defintion Language – to create new DDL – Description Defintion Language – to create new DS, and perhaps new DDS, and perhaps new D
1515Intelligent Tools for MediaIntelligent Tools for Media
Example – Multiple Description Example – Multiple Description TypesTypesStructural – right, left, moving towardsStructural – right, left, moving towards
Physical – color, shape, motionPhysical – color, shape, motionConceptual – goalkeeper, player, ballConceptual – goalkeeper, player, ball
1616Intelligent Tools for MediaIntelligent Tools for Media
MPEG-21MPEG-21
Define a multimedia framework to enable Define a multimedia framework to enable transparent and augmented use of transparent and augmented use of multimedia resources across a wide range multimedia resources across a wide range of networks and devices of networks and devices 7 Items7 Items
Digital Item DeclarationDigital Item Declaration Digital Item Identification and DescriptionDigital Item Identification and Description Content Handling and UsageContent Handling and Usage Intellectual Property Management and ProtectionIntellectual Property Management and Protection Terminals and NetworksTerminals and Networks Content RepresentationContent Representation Event ReportingEvent Reporting
1717Intelligent Tools for MediaIntelligent Tools for Media
Course MaterialCourse MaterialBackground: OO Design and Programming (C++/Java, Background: OO Design and Programming (C++/Java, tbd)tbd)
3D technologies: 3D technologies: VRMLVRML HANIMHANIM Java3D ?Java3D ? MPEG-4/BIFS/XMTMPEG-4/BIFS/XMT
Video/Mixed TechnologiesVideo/Mixed Technologies MPEG-4MPEG-4 JMFJMF SMIL-1/SMIL-2SMIL-1/SMIL-2
1818Intelligent Tools for MediaIntelligent Tools for Media
Course LogisticsCourse LogisticsCommunication channelsCommunication channels
GroupsGroups
Development methodologyDevelopment methodology
DatesDates
1919Intelligent Tools for MediaIntelligent Tools for Media
Communication ChannelsCommunication ChannelsMailing list: tbaMailing list: tba
Web-site: Web-site: http://www.math.tau.ac.il/~doronf/workshop_200http://www.math.tau.ac.il/~doronf/workshop_2002.html2.html
Includes links to resourcesIncludes links to resources
My e-mail: My e-mail: [email protected]@post.tau.ac.il
2020Intelligent Tools for MediaIntelligent Tools for Media
GroupsGroups1-4 people in group (recommended: 2)1-4 people in group (recommended: 2)
Group gets one gradeGroup gets one grade
Inter-group integration – big bonus!Inter-group integration – big bonus!
2121Intelligent Tools for MediaIntelligent Tools for Media
Development MethodologyDevelopment MethodologySome thoughts on Software Engineering:Some thoughts on Software Engineering: Writing programs is easy, writing software is Writing programs is easy, writing software is
impossibleimpossible % of code used% of code used Legacy softwareLegacy software
2222Intelligent Tools for MediaIntelligent Tools for Media
Development Process (waterfall)Development Process (waterfall)
MRD (Marketing Requirements Document)MRD (Marketing Requirements Document)
HLD (High Level Design)HLD (High Level Design)
FRS (Functional Requirement Specification)FRS (Functional Requirement Specification)
ATP (Acceptance Testing Procedures)ATP (Acceptance Testing Procedures)
CodeCode
DebugDebug
Testing / Q.A.Testing / Q.A.
2323Intelligent Tools for MediaIntelligent Tools for Media
ProjectsProjects1.1. Animation LibraryAnimation Library
2.2. Automatic SummariesAutomatic Summaries
3.3. Video Template AuthoringVideo Template Authoring
4.4. Object-based VideoObject-based Video
5.5. Film & TV School projectsFilm & TV School projects
2424Intelligent Tools for MediaIntelligent Tools for Media
Animation LibraryAnimation LibraryObjects with animationObjects with animation HumanoidHumanoid CarCar DoorDoor 4-legged animal4-legged animal ElevatorElevator Bottle, glass and drinkBottle, glass and drink Cigarette and smokeCigarette and smoke WeaponsWeapons ……
2525Intelligent Tools for MediaIntelligent Tools for Media
Animation Library - HumanoidAnimation Library - HumanoidStandStand
Walk (from, to), (path)Walk (from, to), (path)
Run Run
JumpJump
Sit (on)Sit (on)
take (object)take (object)
Put (object, location)Put (object, location)
Touch (object/organ)Touch (object/organ)
WaveWave
TalkTalk
Eat/drinkEat/drink
Climb (wall, stairs,..)Climb (wall, stairs,..)
2626Intelligent Tools for MediaIntelligent Tools for Media
Animation Library - CarAnimation Library - CarDrive (from, to), (path)Drive (from, to), (path)
Open (door)Open (door)
Close (door)Close (door)
Traffic simulationTraffic simulation
2727Intelligent Tools for MediaIntelligent Tools for Media
Animation Library - SpeechAnimation Library - SpeechText-to-speech Text-to-speech
Lip animation (sync?)Lip animation (sync?)
2828Intelligent Tools for MediaIntelligent Tools for Media
Animation Library – Facial ExpressionsAnimation Library – Facial Expressions
Basic emotions (smile, angry, afraid) Low level animation
Eyes (look-at, close, open, blink) Eye-brows Mouth
Face features (hair color, hair style, eye color, glasses)
2929Intelligent Tools for MediaIntelligent Tools for Media
Animation Library - OtherAnimation Library - OtherDoor (open, close)Door (open, close)
4-legged (share animation with humanoid)4-legged (share animation with humanoid)
Elevator (open, close, go-to-floor)Elevator (open, close, go-to-floor)
Bottle, glass, drink (pour, person drinking, spill)Bottle, glass, drink (pour, person drinking, spill)
Gun (shoot, bullet/missile)Gun (shoot, bullet/missile)
……
3030Intelligent Tools for MediaIntelligent Tools for Media
Animation CompositionAnimation CompositionSequence (states)Sequence (states)
ParallelParallel (walk, run, Jump) x (hold, wave)(walk, run, Jump) x (hold, wave)
Compound Animations (hierarchy)Compound Animations (hierarchy) E.g: eat = take (food), touch(mouth)E.g: eat = take (food), touch(mouth)
Inverse KinematicsInverse Kinematics
3131Intelligent Tools for MediaIntelligent Tools for Media
Smart ObjectsSmart ObjectsDifferent Complexity Levels of BehaviorDifferent Complexity Levels of Behavior
Default behaviorsDefault behaviors Rules and constraints (trigger -> action)Rules and constraints (trigger -> action) Behavior model (perception, motivation, planning)Behavior model (perception, motivation, planning)
High level interface to expose capabilitiesHigh level interface to expose capabilities
Naïve physicsNaïve physics
ExamplesExamples AvatarAvatar Tamaguchi/petTamaguchi/pet CarCar ElevatorElevator
3232Intelligent Tools for MediaIntelligent Tools for Media
Problem specification -exampleProblem specification -example
Smart lift in crowd simulation Smart lift in crowd simulation
3333Intelligent Tools for MediaIntelligent Tools for Media
Defining object behaviors - Defining object behaviors - exampleexample
smart tablesmart table in in party simulationparty simulation
graspable fruitsgraspable fruits get_closest_ get_closest_
fruit fruit (and eat it)(and eat it)
replace_fruitsreplace_fruits
3434Intelligent Tools for MediaIntelligent Tools for Media
Script ParsingScript ParsingDefine syntaxDefine syntax
Define internal representationDefine internal representation
Parse input, create representation, and write to Parse input, create representation, and write to filefile
ChallengesChallenges ActionAction Space (near, behind, path planning, obstacles)Space (near, behind, path planning, obstacles) Time (before, after, concurrent, fast)Time (before, after, concurrent, fast)
3535Intelligent Tools for MediaIntelligent Tools for Media
Along Example
3636Intelligent Tools for MediaIntelligent Tools for Media
Animation GenerationAnimation GenerationDefine internal representationDefine internal representation
Read representation from fileRead representation from file
Generate VRMLGenerate VRML
ChallengesChallenges Re-use VRML codeRe-use VRML code Performance and optimizationPerformance and optimization scalabilityscalability
3737Intelligent Tools for MediaIntelligent Tools for Media
Automatic EditingAutomatic EditingFormalize cinematic principlesFormalize cinematic principles
Analyze screenplayAnalyze screenplay
Hints (Annotations) in screenplay Hints (Annotations) in screenplay
EditingEditing Separate into shotsSeparate into shots Decide shot typeDecide shot type Camera motionCamera motion LightLight SoundSound DissolvesDissolves EffectsEffects
3838Intelligent Tools for MediaIntelligent Tools for Media
Example - shotsExample - shotsCamera placementCamera placement
The Line
External ExternalInternal
Apex
3939Intelligent Tools for MediaIntelligent Tools for Media
ExamplesExampleshttp://www.math.tau.ac.il/~doronf/se-anim_files/http://www.math.tau.ac.il/~doronf/se-anim_files/frame.htmframe.htm
4040Intelligent Tools for MediaIntelligent Tools for Media
Template-based Video EditingTemplate-based Video Editing
4141Intelligent Tools for MediaIntelligent Tools for Media
Automated template-based generationAutomated template-based generationWizard approach: user fills in the slotsWizard approach: user fills in the slots
Performance: static elements of presentation are pre-cached Performance: static elements of presentation are pre-cached (unless very large)(unless very large)
Allows for dynamic, on-the-fly generation of presentationsAllows for dynamic, on-the-fly generation of presentations
DOM
MPEG-4
Abstract Template
XSL
material 1….
XML
XSL XSL
SMIL-1
caching
caching
material n
DB
4242Intelligent Tools for MediaIntelligent Tools for Media
Rich Media TranscodingRich Media Transcoding