music-driven motion editing marc cardle rainbow group computer laboratory university of cambridge...
TRANSCRIPT
Music-Driven Motion EditingMusic-Driven Motion EditingMusic-Driven Motion EditingMusic-Driven Motion Editing
Marc CardleMarc Cardle
Rainbow GroupRainbow Group
Computer LaboratoryComputer Laboratory
University of CambridgeUniversity of Cambridge
[email protected]@cl.cam.ac.uk
Marc CardleMarc Cardle
Rainbow GroupRainbow Group
Computer LaboratoryComputer Laboratory
University of CambridgeUniversity of Cambridge
[email protected]@cl.cam.ac.uk
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Overview of TalkOverview of Talk
• Background: Example AnimationBackground: Example Animation
• Research goals and framework overviewResearch goals and framework overview
• Musical FeaturesMusical Features
• Motion EditingMotion Editing
• ExamplesExamples
• Background: Example AnimationBackground: Example Animation
• Research goals and framework overviewResearch goals and framework overview
• Musical FeaturesMusical Features
• Motion EditingMotion Editing
• ExamplesExamples
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
What is driving what?What is driving what?
Film SoundtracksFilm Soundtracks
Music VideosMusic Videos
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Animusic ApproachAnimusic Approach
(Lytle ’94)
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
• A framework for motion editing using A framework for motion editing using music analysismusic analysis
• Simplify and automate the generation of Simplify and automate the generation of synchronised musical animationssynchronised musical animations
• A framework for motion editing using A framework for motion editing using music analysismusic analysis
• Simplify and automate the generation of Simplify and automate the generation of synchronised musical animationssynchronised musical animations
Research GoalResearch GoalResearch GoalResearch Goal
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
MIDIMIDIMIDIMIDI
• Parametric representation Parametric representation
• Pitch, velocityPitch, velocity
• Start-time and End-timeStart-time and End-time
• Parametric representation Parametric representation
• Pitch, velocityPitch, velocity
• Start-time and End-timeStart-time and End-time
SynthesizerSampler
Audio Signal
(Lytle ’94)
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Audio and MIDIAudio and MIDI
• Analysis is based on both MIDI and AudioAnalysis is based on both MIDI and Audio
• Complementary perspectives on the Complementary perspectives on the analysed musicanalysed music
• Analysis is based on both MIDI and AudioAnalysis is based on both MIDI and Audio
• Complementary perspectives on the Complementary perspectives on the analysed musicanalysed music
++AudioMIDI
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
System OverviewSystem Overview
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Musical FeaturesMusical Features
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
MIDI-Based AnalysisMIDI-Based Analysis
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
MIDI-Based AnalysisMIDI-Based Analysis
• Low-level MIDI note information Low-level MIDI note information
• MIDI Channel and Instrument informationMIDI Channel and Instrument information
• Chord RecognizerChord Recognizer
• Virtual PitchVirtual Pitch
• Beat TrackingBeat Tracking
• MeterMeter
• Low-level MIDI note information Low-level MIDI note information
• MIDI Channel and Instrument informationMIDI Channel and Instrument information
• Chord RecognizerChord Recognizer
• Virtual PitchVirtual Pitch
• Beat TrackingBeat Tracking
• MeterMeter
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
MIDI-Based AnalysisMIDI-Based Analysis
• SegmentationSegmentation
• Pattern MatchingPattern Matching
• SegmentationSegmentation
• Pattern MatchingPattern Matching
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
MIDI-Based AnalysisMIDI-Based Analysis
• RegisterRegister
• LoudnessLoudness
• DensityDensity
• DurationDuration
• Attack SpeedAttack Speed
• Focus and DelayFocus and Delay
• RegisterRegister
• LoudnessLoudness
• DensityDensity
• DurationDuration
• Attack SpeedAttack Speed
• Focus and DelayFocus and Delay Time
Max Pitch
Min Pitch
Pit
ch
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Audio-Based AnalysisAudio-Based Analysis
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Audio-Based AnalysisAudio-Based Analysis
• Zero-Crossing RateZero-Crossing Rate
• Spectrum Amplitude VariationsSpectrum Amplitude Variations
• Spectral CentroidSpectral Centroid
• Zero-Crossing RateZero-Crossing Rate
• Spectrum Amplitude VariationsSpectrum Amplitude Variations
• Spectral CentroidSpectral Centroid
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Features AnimationFeatures Animation
MIDISoundtrack
Register
Density - spread
Density - notes
Novelty
Frequency
Duration
Dynamics
Attack Speed
AudioSoundtrack
Mean Roll-off
Mean Zero-Cr.
Std Centroid
Std Flux
Std RMS
Mean Centroid
Mean Flux
Mean RMS
Std Roll-off
Std Zero-Cr.
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Motion EditingMotion Editing
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Motion EditingMotion Editing
• Cyclification Cyclification (Silva ‘99)
• Time warping Time warping (Bruderlin ‘95)
• Various motion linear and non-linear Various motion linear and non-linear filtering techniquesfiltering techniques
• Cyclification Cyclification (Silva ‘99)
• Time warping Time warping (Bruderlin ‘95)
• Various motion linear and non-linear Various motion linear and non-linear filtering techniquesfiltering techniques
(Gleicher ’00)
(Litwinowicz ‘91) (Bruderlin ‘95)
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Motion EditingMotion Editing
• Frequency spectrum re-scaling Frequency spectrum re-scaling (Bruderlin ‘95)
• Standard and multi-target blendingStandard and multi-target blending
• Frequency spectrum re-scaling Frequency spectrum re-scaling (Bruderlin ‘95)
• Standard and multi-target blendingStandard and multi-target blending
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Motion EditingMotion Editing
• Emotional Transforms Emotional Transforms (Polich. ‘01)
• Noise addition Noise addition (Perlin ‘02)
• Motion warping Motion warping (Witkin ’95)
• Emotional Transforms Emotional Transforms (Polich. ‘01)
• Noise addition Noise addition (Perlin ‘02)
• Motion warping Motion warping (Witkin ’95)
(Gleicher ’00)
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Example - 1Example - 1
Original Keyframe Motion
Boring…Boring…
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Example - 1Example - 1
Modified Keyframe Motion
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Example - 2Example - 2
Vel
oci
ty
End-effectors Time
Minima
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Example - 2Example - 2
Time
Mo
tio
n A
mp
litu
de
Density Event
New Motion
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Animation - 2Animation - 2
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Example - 3Example - 3
Zero-Crossing Rate
RMSVolume
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Animation - 3Animation - 3
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Future WorkFuture Work
• Add more music analysis and motion Add more music analysis and motion editing methodsediting methods
• Maya Plugin with high-level descriptionsMaya Plugin with high-level descriptions
• User testing (interested ?)User testing (interested ?)
• Add more music analysis and motion Add more music analysis and motion editing methodsediting methods
• Maya Plugin with high-level descriptionsMaya Plugin with high-level descriptions
• User testing (interested ?)User testing (interested ?)
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
Thank youThank you
Any Questions?Any Questions?
Marc CardleMarc CardleRainbow GroupRainbow Group
Computer LaboratoryComputer LaboratoryUniversity of CambridgeUniversity of Cambridge
[email protected]@cl.cam.ac.ukhttp://www.cl.cam.ac.uk/users/mpc33/http://www.cl.cam.ac.uk/users/mpc33/
Thank youThank you
Any Questions?Any Questions?
Marc CardleMarc CardleRainbow GroupRainbow Group
Computer LaboratoryComputer LaboratoryUniversity of CambridgeUniversity of Cambridge
[email protected]@cl.cam.ac.ukhttp://www.cl.cam.ac.uk/users/mpc33/http://www.cl.cam.ac.uk/users/mpc33/
Music-Driven Motion Editing – Marc CardleMusic-Driven Motion Editing – Marc Cardle
S2002S2002 Bookman Old Bookman Old Style, Bold, 37 pointsStyle, Bold, 37 pointsS2002S2002 Bookman Old Bookman Old Style, Bold, 37 pointsStyle, Bold, 37 points
This subtitle is 31 points This subtitle is 31 points • Bullets are orange; text is 26 pointsBullets are orange; text is 26 points
• They have 110% line spacing, 6 points They have 110% line spacing, 6 points before/afterbefore/after
• Longer bullets in the form of a paragraph Longer bullets in the form of a paragraph are harder to read if there is insufficient line are harder to read if there is insufficient line spacing. This is the maximum recommended spacing. This is the maximum recommended number of lines per slide (seven).number of lines per slide (seven).• Sub-bullets look like this.
This subtitle is 31 points This subtitle is 31 points • Bullets are orange; text is 26 pointsBullets are orange; text is 26 points
• They have 110% line spacing, 6 points They have 110% line spacing, 6 points before/afterbefore/after
• Longer bullets in the form of a paragraph Longer bullets in the form of a paragraph are harder to read if there is insufficient line are harder to read if there is insufficient line spacing. This is the maximum recommended spacing. This is the maximum recommended number of lines per slide (seven).number of lines per slide (seven).• Sub-bullets look like this.