large-scale capture of producer-defined musical semantics - ryan stables (semantic media @ the...

Large-Scale Capture of Producer-Defined MusicalSemantics

Ryan StablesSchool of Digital Media Technology

Birmingham City University

Problem...

Problem Definition

Producer:

I Audio effects parametersusually refer to low-levelattributes.

I Professionally produced audiooften requires extensivetraining.

Researcher:

I Lack of semantically annotatedmusic production datasets.

I How can we map low-leveldescriptors to perceivedmuscial timbre?

Problem Definition

I Descriptors need to representthe views of music producers.

I These may change with genre,musical instruments, etc...

I Various terms may be used todefine similar things (colour,texture etc...)

Project Aims

1. Gather large amounts of semantics data during the musiccreation/production process.

I Develop a series of DAW plug-ins.I Extract information and anonymously upload it to a server.

2. Identify correlation and patterns in the semantics data.

3. Use the data to improve/aid music production tasks.

Model...

Project overview

Server

Descriptor name...

Save...Load...

Save...

Semantic Descriptor

Parameter Space

Feature Set

Pre/Post Gain

Analysis...

Natural Language

Processing

Dimensionality

Reduction

Etc...

(1)

(2)(3)

Figure : Schematic Overview of the Semantic Audio Feature Extraction Project.

(1) Plug-in interface

I Parameters can be setexperimentally.

I Semantic descriptors to bestored in text field.

I Descriptors can be loadedthrough same interface.

I Parameters are stored and/orset.

Figure : Semantic Audio plug-in: Multi-band distortion

(2) Feature Extraction

I Features are extracted from theselected region.

I The parameter space is stored.

I Semantic descriptors are sentas targets.

I Additional metadata is sent, ifavailable.

Server

Descriptor name...

Save...Load...

Save...

Semantic Descriptor

Parameter Space

Feature Set Pre/Post Gain

Analysis...

Natural Language Processing

Dimensionality Reduction

Etc...

Figure : Stored attributes.

(3) Mapping

I NLP Algorithms to identifysemantic correlation.

I Dimensionality reduction tofind correlation infeatures/parameters.

I Additional data partitionsbased on metadata (Genre,instrument, etc...)

I Results sent back to userplug-in.

Server

Descriptor name...

Save...Load...

Save...

Semantic Descriptor

Parameter Space

Feature Set Pre/Post Gain

Analysis...

Natural Language Processing

Dimensionality Reduction

Etc...

Figure : Results processing

Design Constraints...

Architecture

I Requirements:I Maximisation of user-base.I Transparency: Access to the processing chain.

I Design decisions:I Stand-alone plug-ins.I MultiFX.I Plug-ins within a plug-in.I Analysis-only.

I Other:I Free field vs. fixed word.I Before and after.I Metadata pane.

Analysis framework

I LibXtract.

I Hard-coded, C library.I Around 400 combined

audio features*.I [Bullock, 2007]

I Vamp.

I Plug-in within a plug-in.I Hosts LibXtract features,

amongst others.I [Cannam et al., 2006]

Mini-Project...

Mini-Project: Aims

I Analyse the production requirements of musicians.I Birmingham ConservatoireI The Music Producers GuildI The Birmingham Music Network

I Build a series of prototype systems for the collection ofmusical semantics data.

I Use these systems to collect data from a small group ofmusicians during the production process.

I Evaluate the results in order to identify a suitable system forfuture research.

I Demonstrate the feasibility of a wider research project in thisarea.

Mini Project: Schematic

Plug-in

development

Interface design

Algorithm

Development

Server, network,

data distribution

User Testing Data

Aquisition

Results

Analysis

Figure : Schematic Overview of the Mini-Project.

Positions and Timescale

I 2 x PhD Students: 1 x C4DM (QMUL) & 1 x DMT (BCU).

I 3 x Advisory roles.

I Timescale: 6-months from September 2013.

I Future: collaborative grant application.

Thanks!ryan.stables@bcu.ac.uk

References

Bullock, J. (2007).Libxtract: A lightweight library for audio feature extraction.In Proceedings of the International Computer MusicConference, volume 43.

Cannam, C., Landone, C., Sandler, M. B., and Bello, J. P.(2006).The sonic visualiser: A visualisation platform for semanticdescriptors from musical signals.In ISMIR, pages 324–327.