Advanced real-time sound and data
processing with FTM&Co STEIM Workshop 10–13.5.2010
presented by Diemo Schwarz
IMTR Real-Time Music Interaction TeamIrcam–Centre Pompidou
http://ftm.ircam.frFTM 2.5.0 beta release (BETA 14) for
Max 5 on Mac OS X
Wifi: Studio 3PW: AACCBBDDAA
1
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
PlanFTM
Introduction
Basics
Examples
Persistency: SDIF, SQLite
GaborPrinciple
Examples
MnMGesture Follower
CataRT
2
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
What is FTM&Co?a real time object system for Max/MSPrepresentation of musical structuresrepresentation of sound data and structuresabstraction from the real-time system (Max, PureData, ...)Timeline:
Reference article:Norbert Schnell, R. Borghesi, D. Schwarz, F. Bevilacqua, R. Müller « FTM — Complex data structures for Max », International Computer Music Conference (ICMC), Barcelona, 2005
— Max/ISPW — Max/FTS — jMax — FTM — t
3
Introduction
courtesy of Mikhail Malt
4
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
FTM MotivationsAudio analysis/synthesis (Gabor)
representation of grains, waves/periods and framesrepresentation and organisation of sound descriptors
Gesture analysis, classification and recognition (MnM)(linear) algebra and matrix processingrepresentation and (statistical) modelling of gestures gestalts
Score representation and following (FTM, suivi)representation of a score or time series of datarepresentation of objects in a score
➡ modularity & flexible structuring of memory
5
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
FTM ComponentsFTMlib shared library for Max
message system extensionintroducing complex data types
+Max externals
+Editors and graphical externals
6
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
FTMlib FeaturesObjects of predefined classes
mat, dict, track, fmat, expr, bpf, ...
Expression evaluationFile import/export (soundfiles, MIDI, SDIF)Object serialization and persistence
7
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
TerminologyReal World Object-Oriented
Programming FTM Max
a thing with a name, state, and properties (an alert black-and-
white dog named Otto)
object (data) object (some specific Max objects = externals)
an action and reaction(“sit!”)
method and return value
message andreturn value
message(and output)
a process (the vet) (function, process) module
(external)Max object (external)
a class of similar things (dogs)
class class –
a more general or more specific class (pug dogs, animals)
superclass, derived class
– (types, related classes) –
8
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
FTM Conceptsseparation of data and operators
➡ data objects of optimised classes with methodsfile import & export (text, audio files, MIDI files, SDIF)
persistence (save within patcher and in separate files)
conversion to Max values and listssent within Max values and lists (as references)
simple garbage collector (by reference count)
➡ operators
9
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
The FTM Package Familiy
Gesture Follower CataRT
Gaboratomic sound
MnMmaths+stats
Suiviscore
following
FTM externals
FTMdata classes, visualisation, editing
SDIF, SQLite
application packages
basic library
http://ftm.ircam.frhttp://imtr.ircam.fr
applications
10
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
DistributionFTM 2.5 current stabilizing versions for Max5
FTM 2.3 and 2.4 for Max 4.6
FTM is free (libre) open source software (LGPL)Gabor&MnM are free (as in free beer)
both are distributed together
Web site http://ftm.ircam.frUser mailing list [email protected] on http://lists.ircam.frDeveloper list [email protected]
11
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
Credits and ReferencesFTM&Co is developed in the Real-Time Musical Interactions team at IRCAM by Norbert Schnell and Riccardo Borghesi, Diemo Schwarz, Frederic Bevilacqua, Remy Müller, Jean-Philippe Lambertwith contributions by Julien Bloit, Baptiste Caramiaux, Arshia Cont, Pierre Duquesne, Sebastien Gulluni, Baptiste de la Gorce, Bruno Zamborlin
Reference article:Norbert Schnell, R. Borghesi, D. Schwarz, F. Bevilacqua, R. Müller « FTM — Complex data structures for Max », International Computer Music Conference (ICMC), Barcelona, 2005
12
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
G a b o r :Dennis Gabor, Acoustical Quanta and the Theory of Hearing, Nature, 1947
Multi-Representation grains, waveforms, spectral frames, partial sets
Real-Time extension modules for Max/MSP, PD
Analysis–Synthesis...and transformation in a unified framework
Reference article:Norbert Schnell et al. « Gabor, Multi-Representation Real-Time Analysis/Synthesis », COST-G6 Conference on Digital Audio Effects (DAFx), Madrid, 2005
13
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
Two Fundamental Ideas...1. Atomic sound particles in various descriptions
time-domain: grains, wave periodsspectral domain: FFT frames, phase vocoder frames, sinusoidal partialsmetadata: cepstrum spectral envelopes, filter coefficients, sound descriptors
2. Arbitrary rate processingpicking grainspitch synchronousmultiple frame rates
14
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
Overview
15
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
...their Realisation...1. Unified framework
sound particles represented as generic data objectsmatrix of floating point numbers
streams as time-sequence of objects
(others: hash-table or matrix of objects)
sent in between specific analysis or synthesis modules
2. Sound processing via message passing (control)64 bit time tags for sub-sample precise reconstitution of the signal input/output buffers with interpolation
16
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
...and their Advantages1. Unified framework allows generic
manipulation, storage, and display of atomshigher modularitycan use mathno longer needing a monolithic analysis+synthesis+manipulation module
2. Arbitrary control rate processingis adaptable to frequency/texture/rhythm of sound or analysis algorithmnot limited to fixed-rate audio DSP chain
17
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
From Signal to AtomWays to cut the input buffer into vectors:
arbitrary grain extraction: gbr.grain~
pitch synchronous elementary waveforms: gbr.psy~based on yin pitch estimation algorithm
(de Cheveigné and Kawahara, JASA 2002)
constant rate frames: gbr.slice~FFT
phase vocoder analysis
additive analysis
18
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
From Atom to SignalWindowingInverse FFTFFT-1 (additive synthesis)Overlap add: gbr.ola~
shortcut for paste and drain (output delay line buffer)
19
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
...and in between...Manipulation
of the dataof the timing
DisplayStorage and retrieval
files (audio, SDIF)databases
Future: network or inter-application send/receive via OSC
20
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
Gabor Summary
Ideas Realisation
atomic sound particles in multiple representations
unified framework of specific analysis–
synthesis, generic data
arbitrary rate processingevent processing with time-tags + delay lines
21
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
MnM package
Classification and recognitionmotion capture data, sound and music
Basic matrix calculationsLinear algebraPCA, GMM, HMM, NMD, EM, etc.
Reference article:Frederic Bevilacqua, R. Muller, N. Schnell « MnM: a Max/MSP mapping toolbox », New Interfaces for Musical Expression, Vancouver, 2005
“Mapping is not Music”
22
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
MubuPackaging of frequently used signal analysis, representation and synthesis algorithms with perfect synchronization as MSP externalsMailing list [email protected] article:Norbert Schnell, et al. R. Borghesi, D. Schwarz, F. Bevilacqua, R. Müller « Mubu & Friends - Assembling Tools for Content Based Real-Time Interactive Audio Processing in Max/MSP », International Computer Music Conference (ICMC), Montreal, 2009
23
Real-Time
Corpus-Based
Concatenative Synthesis:
CataRT
24
Diemo Schwarz, Etienne Brunet– Synthèse sonore concaténative par corpus en temps-réel
IntroductionReal-time Synthesis interactive + explorative (navigation)concatenative grains + overlap + chainingcorpus-basedsounds + segmentation + descriptors (timbre, metadata, classes)
25
Diemo Schwarz – Real-Time Data-Driven Sound Synthesis
Principle of CBCSLarge sound database
Segmented into units
si la do
si do dore
la do domi sol
si la do
■ synthesis target
■ unit selection from the sound database to best match the target
■ Synthesis by concatenation of the units
26
Motivation and Approachwork with all the nuances of real sound
large sound databases exist, ready to use
new method → new sound creation
data-driven vs. rule-based approach
general superiority of data-driven approaches in many other domains
27
Influences and LinksInspired by concatenative speech synthesis
CBCS in real-time (as implemented in CataRT) can be seen as a content-aware extension to granular synthesis:
providing direct access to specific sound characteristic
not restricted to selection by position in a sound
28
Diemo Schwarz – Real-Time Data-Driven Sound Synthesis
Applications of CBCS
Interactive exploration of sound databases (browsing)
Resynthesis of an audio target (mosaicing)
Texture synthesis (ambiences, soundtracks)
High-level instrument synthesis (example: Synful)
Expressive speech synthesis
29
A Short History of Data-Driven SynthesisI.Prehistory (1948)
Concrete and early electronic music
II.The Classics — revisited (1980, 1990)Sampling
Granular Synthesis
John Oswald
III.The Modern Times (2000)many research projects, often in mutual ignorance
30
Tape splicing
soft attack or decay
combined attack and decay of two sounds
medium attack or decay
hard attack or abrupt finish
softer and less abrupt than D
[figures: Mikhail Malt]
Karlheinz Stockhausen: Étude des1000 collants (1952)
31
John Cage: Williams Mix (1953)
[http://www.medienkunstnetz.de/works/williams-mix]
32
Level of “Data-Drivenness”
1948
19801990
1993
20002001
2001
20012002
2003
2003
2003
2003
2003
20042004
2004
20042005
2005
2005
Selection
Let them sing it for you
Granular
Samplers
Synful
Musescape
CataRT
MoSievius
stochastic SoundscapesSoundclustering
La legende des sieclesN.A.G.
MATConcat
targeted Directed Soundtracks
Soundmosaicing
Input driven resynthesisMPEG!7 Audio Mosaics
Soundspotter
automatic
Ringomatic
Musical Mosaicing
Caterpillar
Audio Analogies
manual segmental high!level
Musique Concretemanual
Plunderphonics
similarity descriptorssimilarityframe spectrum
Analysis
fixed
2005
2003
[Schwarz, JNMR 2006]
33
Diemo Schwarz – Real-Time Data-Driven Sound Synthesis
CataRT Introductionflexible testbed to experiment continuation and interaction: CataRT
realisation of CBCS in real-timepatch for Max/MSP with FTM, Gabor, MnM by Norbert Schnell et al. from http://ftm.ircam.fr
released as Free Software under GPL at http://imtr.ircam.fr
modular MVC architecture:building blocks for you to use
34
CataRT Overview
CorpusAnalysis
DB
Synthesis
Audio Input
35
Analysis
Corpus
Soundfile+markers
+descriptors
Persistent Sound and Unit
Database
descriptor analysis
Soundfile+grain sizeor markers
RT audio input
Audio Input
onset detection
Soundfile
36
Diemo Schwarz, Etienne Brunet– Synthèse sonore concaténative par corpus en temps-réel
Synthesis
Corpusnearest
neighbour selection
Target descriptor values Control
transformation + concatenation
descriptor analysis
RT audio input
onset detection
controller
37
Diemo Schwarz – CataRT workshop
Architectureone catart.data per corpus
one catart.import for segmentation and analysis
any number of catart.data.proxy to access data
any number of catart.lcd for display and control
any number of catart.selection per corpus
any number of catart.synthesis per selection or per corpus
38
Composers and Projects using CataRT
Sound design and compositionLuis Naõn, Hans Tutschku, Matthew Burtner, Sebastien Roux, Hector Parra, Roque Rivas, Bruno Ruviaro
InstallationPlumage (project Enigmes about the navigable score, Roland Cahen), Pierre Jodlowski, Cécile Babiole
Live electro-acoustic music Dai Fujikura, Stefano Gervasoni, Sam Britton, Aaron Einbond, Sebastien Gaxie, Miguel Angel Ortiz-Pérez, theconcatenator
Examples: myspace.com/catartsoftware
!"
!"
18
""
""
q = 60
q = 60
!"
!"
q = 102
q = 102
!#
!#
""
""
!"
!"
A. Fl.(smallgroup)
B. Cl.(smallgroup)
Tpt.(smallgroup)
1 Vln.(smallgroup)
1 Vc.(smallgroup)
electronics
Fl.
Ob. I
Ob. II
Cl. I
Cl. II
Bsn. I
Bsn. II
Hn. I
Hn. II
Hn. III
Hn. IV
Tpt.
Tbn. I
Tbn. II
Timp.
B. D.
Vln. I
Vln. II
Vla.
Vc.
Db.
!
"
!
"
!
"
!
"
!
"
!
"
!"
$SMALL GROUP
%&'%( % % %) % # % #
p
* + + ,ca. 20 - 21 sec.
, ,- , , , ,
$ % %( %&'(% % % #
p
%) * , , , ,- , , , ,
$ % % #
p
% #) . + , , , ,- , , , ,
$ %&'( #% % #
% % %p
+ / , , ,- , , , ,
0 $% %1 % % %p
% #) . , , , ,- , , , ,
$
section 3
*
4
%%%%%%%%%(
211( %%%%%%%
((
more and more dense and violent(grain size gets smaller and smaller)Harmony 3Harmony 2
gliss.
gliss.
+3 semitones,
-4 semitones,
%%%%%%%%%1
2222 %%%%%%%22(1 ,
CUT OFF
5
34 5 . * + /
$ , , , , ,- +marcato
*ff
$%'6 $%
'6$%
'6%7( %7($%
'6
%7 . . %7 . $%'
( 6$%
'6%7( %7 .$%
'6 %78 . * $%'6%78 . * .
$%'
6$%
'( 6
%7 .%7
*sffffz
$%'6%7'( % * * espress.
fff
. $%'6 %8 %( %
$%'6
%% #
$ , , , , ,- +marcato
*ff
$%'( 6 $%'
( 6%7 %71 * $%
'6 %78 . + $%'( 6 $%'6
%7 .%7(
.$%
'6$%'6%7 %7 * %
%')$%
'( 6
$%'6
%7 %7
. %7( *sffffz
%7'(8 . *
espress.
*fff
%) $%'6% %( # % %
$ , , , , ,- + marcato*ff
. %%'()
$%'6
%74 $%'6%7(
. . %7 .
$%'
( 6 %78 *$%
'6 %7(8 . * * $%'6
$%'6
%7 %7 . $%
'( 6 %7(8 * .
$%'6
sffffz
%78fff
%7'(. . % %8 . * + *
espress.
%6
$ , , , , ,- +marcato
*ff
%%') %74 .
%%
'
2 )%%')
$%'6
%7%7 .
%7
(.
%%')
%%
'
)%%
'
)%7( %7 %7* $%'( 6 $%
'6
%%
'
)
%7 %7(%%')
%%
'
)%7. %7 %7
. %%')
%%
'
)$%
'6
%7
( .%7( %7 .
sffffz fff
$%
'
( 6
%7'
(
%(. + *
espress.
%%')
%%(
%%'
()%
%
$ORCHESTRA
, , , , ,- +marcato
* .ff
%%')
$%'( 6
%74%%
'
2 )%7 .
%7. .
%%
'
) %%')%7 %7 . %
%')%%
'
) %%')
$%'
( 6
%7 %72. %7
$%'
( 6
%%')
%7( . . %71%%
'
)%72. %7
.%%
'
)%%')
sffffz
%7 %7fff
espress.
%%'
()$
%
'
(6
%7'% %( %
% #%
% #%
%%
'
)%
% %
9 , , , , ,- +marcato
*ff
$%'2 6%78 .$%
'6
$%'6%7 . %7 .
$%'
6$%
'6$%'2 6 $%
'6%7 %72 . %7 $%'6 $%'6
%%')
%7 %7 . %7%%')
$%'6
%7 . . %7$%
'6%7 .%7
.$%
'6 %78 .
sffffz fff
$%'
( 6 espress.
%7'
2%
%( . $%'
( 6%( %( % %) *
$%'6 % # %
9 , , , , ,- +marcato
*ff
%%') %74 . .
$%'6
$%'6%7
%7 . %
%'2) %%') $%
'6%7 %7 . %7 * %
%') %%')
$%'6
%7 %7$%
'6%7 . %7 . . %%') %74 .
$%'
6 %78 *sffffz
espress.
fff
%7' % %( % %
) * $%'
( 6 %( # % %) *
$ , , , , ,- + + marcato
ff
. $%'6
%74 * 9 . $%'6
$%'
2 6%7. %7 . $%
'6 $%'6%7 . %7 * . $%'6%78 * $%
'6 %78 . *sffffz
ff
$%'
( 6%7' * %7( * $%'6 %7(8 . + .$%'6 %78 *
$ , , , , ,- + 9+ marcato*ff
$%'2 6 %78 . .$%
'6 %728 *$%
'6%78 . *$%
'6%78 . * $%'6 %78 . * $%
'6 %728 .sffffz ff
$%'6%7'
. . %( . $%'( 6 %78 * $%'6
$%'
( 6%7 . . %7 + $
$ , , , , ,- + marcato9 *ff
$%'6%78 . .
$%'6%78 * * . $%
'6$%
'6%728 $%'6%7 . . %7 + * $%'
( 6 %78 . * $sffffz
%7'8 . *ff
*$%
'6 %7( 4 . * *
$%'
6%74 . * 9
$ , , , , ,- + 9*ff
marcato$%'2 6%78 . * $%'6%78 . . $%
'6%74 * * . %
%') %74 * . $%'6
$%'6
%74 %728 . * .$%
'6
sffffz
%78ff
%%')%7'( %7( * * $%
'( 6 %78 . * $ . $%'6 %7( 4 *
%%'
() %7( 4 .
$ , , , , ,- + + marcato
ff
$%'( 6%78 . * *$%
'6 %78 . .
$%
'6 %%
')%72
. %7. $%'( 6%78 * . $%
'6 %7(8 * *sffffz ff
$%'6%7' %7 * * . $%'6 %78 + .$%'6 %78 *
9 , , , , ,- , , ,(senza sord.)
sffffz
%7'2 4 . * + /
9 , , , , ,- , , ,straight mute
sffffz
%7'(8 . *super short staccato
+p
%7'( 7' 7' 7' 7'
5
gliss.
9 , , , , ,- , , ,p
soft sticks
%'( ' ' ' ' ' ' ' ' ' ' '3 3 3 3
gliss.
: , , , , ,- , , ,sffffz
%7'4 . * + /
$ , , , , ,- marcato+ *ff
$%'6 %7(8 .
$%'6
$%'
6%7( . . %7 * $%
'6 %7(8 .$%
'6 $%'6%7(
. .%7
* $%'6
%74 . . $%'
( 6 %7(8 *$%
'6 %74 .
sffffz
unis.
fff
$%'6 espress.%7'( %
%($%'6
% %% %( % %4 . *
$ , , , , ,-div.
marcato+ * .ff
$%'6 %7(8 * $%
'6 %78 . . $%'( 6 %7(8 * *
$%'6 %78 .
$%'6 $%
'( 6%7 . . %71 . $%
'( 6 $%
'6%%')%7(
.%7
. %7
$ , , , , ,-
div.
marcato
+ *ff
$%'( 6%78 .$%
'6$%
'6%7(. .
%7(*
$%
'6 %74 . %
%'()%%')
%7
( . . %7* . %
%') %78 .$%
'( 6
$%'
6sffffz
unis.%7* .
%7
2fff
$%'6 espress.
$%'
6%7'( %7 . %(
%) * /
$ , , , , ,-marcato
+ff
* .$%
'( 6
%718 * . $%'6 %7(8 .
$%'
( 6 %78 *
$%'6
$%'6%7( . %7 .
$%'6
$%'
6%7
. %7 .
$%'
6 %%')%72
. . %7
( *
0 , , , , ,-div.
marcato+ *ff
$%6%74 . $%
'2 6 %
%')%7 . . %7 * $%
'6 %78 . . $%'6 %72 4 *
$%'
( 6$%
'6%7 . %7 . %
%') %78 . $%'
2 6%74 .
$%'
6 %74 .
sffffz
unis.
fff
espress.%7' %( %$%
'6%
%( % %% #
% % *
0 , , , , ,- marcato+ff
* .$%( 6
%78 * $%'
2 6%74 . . $%'6 %72 4 *
$%'
( 6 %78 . *$%
'6$%
'6%7 . %7 . . $%
'6%78 * . %%') %72 4
9 , , , , ,- +marcato
*ff
$%'6
%74 . .$%
'6 %78 * $%'2 6$%
'6%7 . . %7 * $%
'6%74 . $%'6
%%')%7 . . %7 * .
$%'
6 %78 *
sffffz
unis.
%%'()
espress.
%7'%( .
fff
. %%')
% %($%
'( 6
% % % +
9 , , , , ,-div.
+marcato
*ff
.$%
'6%78 * $%
'6%74 . . $%
'6 %78 *$%
'6
$%'6%7
. %7 . $%'2 6
%%')%7 . . %7 +
$%'
6 %78 .
9 , , , , ,- +marcato
*ff
$%'6
%78 . $%'6$%
'6%7. %7 . .
$%'6
%78 * .$%
'2 6 %78 *
$%'6
%78 . * .$%
'6%78 +
sffffz
%7'%7( * *
fff
$%'6 %(
) + * %)
2
39
Installation GrainStick“Rainstick” gestural control metaphor
Demo of SAME (Sound And Music for Everyone Everyday Everywhere Every way) EU project
Composition of sounds and ambiences by Pierre Jodlowski
WFS spatialisation and Motion Tracking by Grace Leslie and the Room Acoustics team
Gesture analysis by Bruno Zamborlin and Diemo Schwarz
40
Installation Xe-RocksTurning photocopiers into musical instruments, by video and sound artist Cécile Babiole
On show at the ECM Gantner, Bourogne
41
Current research and applications at the Real-Time Music Interaction Team (IMTR) — http://imtr.ircam.fr — Diemo Schwarz
Installation PlumageCollaboration with national design school ENSCI led by Roland Cahen, Yoan Ollivier, Benjamin Wulf (ENSCI), Christian Jacquemin, Rami Ajaj (LIMSI), Diemo Schwarz (Ircam)
within the project ENIGMES about the navigable score3D interface using Virtual Choreographer http://virchor.sourceforge.net
42
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
Plumage
43
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
Plumage
44
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
Plumage
45
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
Musical Applications (1)Corpus-based improvisation with live sound input
Diemo Schwarz with Etienne Brunet
Live Algorithms for Music (LAM) concerts:Improvisation of George Lewis, Evan Parker, Sam Britton, Diemo Schwarz and othersRien du tout by Sam Britton, Diemo Schwarz
Rearranging, navigation, live-recording, re-orchestration
46
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
47
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
Musical Applications (3)Whisper Not by Stefano Gervasoni for viola and electronics, computer music realization by Thomas Goepfer
“response” of CataRT to the instrumentmorphing between corpora of pizzicato viola and water-drops
Interaction with pre-recorded sound,
interpolation, rearranging, navigation
48
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
Musical Applications (4)swarming essence by Dai Fujikura for orchestra and electronics, computer music realization by Manuel Poletti
harmonic selection of specially composed small instrument group and extended playing stylecorpus-based orchestration, separate treatment
Re-orchestration, rearranging, navigation
49
Current research and applications at the Real-Time Music Interaction Team (IMTR) — http://imtr.ircam.fr — Diemo Schwarz
Musical Applications (5)Beside Oneself, Aaron Einbond, 2008 for viola and electronics:
real time control of CataRT from audio analysis of the viola
What the blind see 2009 for small ensemble and electronics
real time control of CataRT from audio analysis of the viola
corpus-based “orchestration” or transcription
texture example:1. Rain on leaves
2. CataRT resynthesis with extended technique instrument sounds
50
Aaron Einbond – What the blind see
B
&
&
ã
ã
t
&
ã
Alto
Cl.B.en Sib
Hp.
Pno.
Vib.
F
!
" " "
C10
C10
C10
a tempo
a tempo
a tempo
1936
T T .T#
Œ Ù .T
$T
%6
& . ym
bruit des clefs : doigtés ad lib.
"f "yn y
7
TmflM
ßym y ym ym y Ù TM
flßy ym y y yb
7
® x xx x xm
œSF neige to concatenation
Œ
7
‰ ® jœb œbp
œœ œ œm œ œ
7
‰ œ œ> œ œ .œ
#&
œétouffé avec papier sempre
p
œm
Ó
Ó ?
Œ ‰ jœb˝
‰ &6
˝superball
Vib.
p
3‰6˝ ‰
° sempre
ongles
%
%
6
® Tn -
pT JT Tb T
6
ym"f "
ym ym yy T- JT
%
Jœ
&
6 6 6 6 6>paume
ß
6
œ
pœ œ œm œb
œ œ
6æ j6æ
jyO>߉ 5& .
ym>
pplectre
y
˝
> . .y
yO
>
3
‰ œn ˝
® .
.œb˝
6&̋
6
œn ˝
& 6˝
.6˝
7
7
7‰ ® yæ gliss.(main)
arco
#IVyæ 3
yb y. y. y. y.%
6
Tym
ym y y y
6
yn ym ym yn ymT- JT Tb T .TbflM
ßTb Mflß
%
6
® œm œ
.œœ œb œ œ
œ .œ>7
œ .œ œ œ
7
‰ ® œ œb7
%
jyO
7
.yO
yb
#̋y
6
‰ ®6
6
6˝
œb ˝
6. ‰6
®œ˝
6>ß
28
1. snow melting on a metal roof 2. CataRT resynthesis with instrument sounds3. manually edited transcription 4. live reading by ensemble
51
New Concepts for Musical ApplicationsRe-arranging units by other rules than temporal order
Composition = Navigation through the sound space of heterogeneous sound databases
exploit richness of detail of recorded sound
efficient control by perceptually and musically meaningful descriptors
Interaction with self-recorded sound: sound of a musician is available for interaction beyond simple repetition
Cross-selection and interpolation: apply sound characteristics from one corpus to another, morphing between corpora
Orchestration and re-orchestration: matching a mass of sounds to a harmonic or sonic target while retaining precise control over the result
52
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
Exercise SuggestionsAdapting CataRT
Integration in your concert patchAdapting to your musical needs/ideas
Improving CataRTCorpus Data
soundfile editor (list/delete sounds in corpus)
soundset editor (define sound sets, add/remove units)
descriptor range viewer/editor (e.g.: exclude units below threshold)
Analysissegmentation editor
Selectionconstraints: don’t repeat unit
Synthesisforced pitch/loudness
multi-channel output
Controlby MIDI notes
by envelopes, sequencer
by live sound analysis
53
FTM & Co — IMTR Team — IRCAM - Centre Pompidou
AcknowledgementsCataRT is based on FTM&Co by Norbert Schnell et al.Thanks go to:
Alexis Baskind, Julien Bloit, Greg Beller, Miguel Angel Ortiz PérezRoland Cahen, Yoan Olliver, Benjamin Wulf (ENSCI),Christian Jacquemin, Rami Ajaj (LIMSI)
Norbert Schnell, Riccardo Borghesi, Fréderic Bevilacqua, Rémy Muller, Jean-Philippe Lambert (IMTR Team)the French National Agency of Research ANR and the RIAM project Sample Orchestrator
Links:CataRT wiki on imtr.ircam.fr,
mailinglist [email protected] on http://lists.ircam.frand [email protected]
54