music and shape: exploring cross-modal representation examples, a model and analyses professor adam...
TRANSCRIPT
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Music and shape:Exploring cross-modal
representation
Examples, a model and analyses
Professor Adam OckelfordUniversity of Roehampton
11th March 2010
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Music and shape
How can the two relate?
One set of dynamic relationships can be created through movement:
• more or less freely associated with sound
(eg, dance, gestures in performance)
• creating, causing or controlling sound(eg, playing instruments, conducting)
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Music and shape
Another set of relationships can arise through the visual representation of sound – the focus of this presentation
• Seven examples• Attempt to construct common theoretical
framework• Indicative analyses• Discussion
Children’s ‘picture scores’
7 year old’s representation of a rhythm(after Bamberger, 1982)
Children’s ‘picture scores’
7 year old’s representation of a rhythm(after Bamberger, 1982)
Blind children’s drawings on ‘German Film’
After Welch (1981)
Blind children’s drawings on ‘German Film’
Conventional Westernmusic notation
Any features in common with children’s informal notations?
Braille music
Key differences with print?
Guitar chord tablature
What processes are at work here?
Similarities and differences with previous examples?
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Graphic notation
Excerpt from Stockhausen, Electronic Study No. 2 (1954)
Jamie Roberts’s score of Jean Michel Jarre’s Oxygene (Track 4)
Score produced through synaesthetic response to
music
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
How does it all work?
A common underlying principle or principles?
Or fundamentally different processes at work?
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Perceptual domains
• ‘Domains’ = way of modelling the fact that one sensation can have different ‘states’
• Touch: eg, temperature and pressure.• Vision: eg, hue, saturation and
brightness.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Perceptual domains
In music:
- Pitch: (chroma, octave), harmony, tonality
- Perceived time: onset, duration
- Timbre
- Loudness
- Perceived location of sound source
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Different categories of domain
Can be conceived as being
single dimensional, bidimensional or multidimensional
and as being
linear or cyclical or both, or complex
Examples …
PitchQuickTime™ and a
TIFF (Uncompressed) decompressorare needed to see this picture.
Duration
‘Values’• Each state in which a domain is perceived to exist can be
conceived of as a ‘value’.• May be simple or complex (like the domains to which they
pertain)• Example:
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Relationships within domains
• Musical structure is not to do with individual values, but the relationships between them within each domain - particularly pitch and perceived time
• We can assume that the relationships between values normally pass listeners by as series of qualitative experiences
• However, they may be conceptualised (culture specifically?) as differences or ratios, or they may reflect the complexity of the values to which they pertain
Examples of intra-domain relationships
‘Primary’ relationships
Examples of intra-domain relationships
‘Secondary’ relationships
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Relationships between domains
• Two main types
– ‘Regular’ (isomorphic in the domain of ‘shape’)
– ‘Irregular’ (through association)
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Regular inter-domain relationships
Guiding principle:to map values systematically
between domains,a relationship must involve
a function that isnot specific
to the perceptual domains concerned.
Regular inter-domain relationshipsFor example, differences are domain-specific, so
single values of difference cannot be mapped systematically between domains.
Regular inter-domain relationshipsWhereas, ratios are abstract, are not bound by the
context in which they occur. Hence they permit regular mapping between domains.
Regular inter-domain relationshipsRatios may occur between differences (as secondary
relationships). Hence the regular inter-domain relationships would be tertiary.
‘Irregular’ inter-domain relationships
Guiding principle:formed through association …
either through repetitionor indirect connection
Irregular inter-domain relationships
Different types:
(a) arbitrary (repetition)
(b) indirect (indirect connection)
(c) synaesthetic (indirect connection)
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
Arbitrary inter-domain relationships
> •
Any relationship between a sound and its visual representation (shape) is possible.
For example:
accent staccato fermata
Example (‘staccato’) – relationship created through repetition (or verbal explanation = proxy repetition):
Arbitrary inter-domain relationships
Example (‘note cluster’) – relationship created through action / object as common visual and auditory source
Indirect inter-domain relationships
Can be reinforced through repetition
Example – green corresponds to Eb major (and vice versa)
Synaesthetic inter-domain relationships
Again, can be
reinforced
(for others)
through repetition
From theory to analysis
Children’s ‘picture score’
Simplest interpretation: arbitrary relationships – given shape (circle) maps onto note
Children’s ‘picture score’
More advanced interpretation: regular mapping of distance and duration; sequence and temporal order
Score on German Film of pitch glide
Coordinated regular mapping of vertical and horizontal distances with pitch and perceived time (implied tertiary relationships)
Print score
Arbitrary and semi-regular mappings combine(the latter reinforcing the former)
Braille score
Only arbitrary mappings survive transcription into Braille(which makes using this medium more challenging)
Guitar tablature
Combines both indirect and regular representations of pitch
Stockhausen score:
comprises regular
representations of pitch,
perceived time and loudness
Jamie’s score: comprises
regular and indirect
representations of pitch,
perceived time and timbre
Conclusions
• Evidence from children’s untutored representations of music suggests that sophisticated inter-domain mapping between sound and shape occurs early and intuitively
• This view is reinforced by blind children’s representations of changing pitch – which are also sophisticated, and exist in the absence of any visual model to guide them
Conclusions
• The precise nature of inter-domain mapping may vary from individual to individual, and according to cultural convention.
• However, it appears to be relatively easy to grasp regular mappings (consider, for example, widespread use of horizontal dimension for time and vertical dimension for pitch) or arbitrary symbols (as in standard notations)
Conclusions
• Some inter-domain mappings may be hard-wired through synaesthesia.
• These seem to vary from individual to individual.
• However, such representations can be learnt and appreciated by others
Conclusions
• What next?• More exhaustive search of different forms of
symbolic visual representation of sound to check broad applicability of the model
• Use the model to generate new forms of sound / shape installations
• More research into how mappings aries and are learnt, and into sonic/visual synaesthesia
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Adam Ockelford
07818-456 472