Toward Certain Sonic Properties of an Audio Feedback System

by Evolutionary Control of Second-Order Structures

EvoMUSART 2015

2015/04/08 Seunghun Kim1, Juhan Nam1, and Graham Wakefield2

1Graduate School of Culture Technology, KAIST

2Digital Media, Visual Art & Art History, York University

• An audio feedback system adapting toward certain sonic properties

• Real-time feature objectives

• Second-order signal-processing structure

• 2 microphones and 4 loudspeakers are connected through a network of various DSP components

• Evolutionary process to develop the internal signal processing algorithms

• Sequential operation of developed algorithms -> analysis based on target conditions -> generation of the following algorithms

Abstract

Audio Feedback

• Audio Feedback

• Sound is generated through the connection of a microphone, signal processing components and a loudspeaker

• Circular structure using the output signals as re-input signal, beyond the one-way relationship

Feedback-based Music Systems

• LIES

• Feedback delay network(FDN): delay lines connected by a feedback matrix

• Some loops apply signal processing components: formation of a complex network

• Audible Eco-Systemic Interface (AESI)

• Compositional work interacting with its acoustic environment through sonic feedback, depending on ambient noise as energy source

• Features extracted from the received sound are compared with the original signal, and the difference is used to control parameters for sound synthesis, and thereby the system adapts toward the room resonances

Lack of Intentional Control in Previous Feedback Systems

• Implication from the emergence feature (overall sound quality is by-product of the system interactions)

• Specific performances cannot be accurately predicted in advance

• Attention to the technical conditions/sonic interactions

• Critique: loss of control over the overall sonic shape

• However, every adaptive system preserves balance between bottom-up and top-down processes

• Regulative processes in audio feedback systems: 1) placement of microphone/speaker, 2) resonant modes of the chamber and 3) dependencies of the relationships in the system

• In Ephemeron, a performer controls the overall sonic result by modulating system parameters, resulting in preventing the system’s tendency toward a stable state

Intentional Control of an Audio Feedback Toward Certain Sonic Properties

• Limitation: direction of the tendency is unknown because of nonlinear dynamics

• Our motivation: deep exploration of intentional control toward certain sonic properties without sacrificing nonlinearities of the audio feedback itself

• Intentional control: desired tendencies in the feedback sound

• The intentional control would support idiosyncratic interactive applications that combine context-specificity, nonlinearity and interactivity

System Design

• Purpose: A feedback system adapting toward certain sonic properties

• Three main ideas

• 1. Goal-directedness: sonic features as target condition, which can be controlled by users in real-time

• 2. Second-order feedback structure: self-organization and replacement of internal signal processing algorithms

• 3. Evolutionary process for design of the second-order feedback structure

System Design

System Design 1: Goal-directedness toward a specific sonic condition

• With suggestion of sonic properties as the target condition, a feedback system could adapt toward the conditions -> implication of the intentional control

• Feature extraction from a feedback sound is then necessary to measure a present state and its deviation from a target state

• Since feedback sound is not note-based music, candidates for target sonic must be broadened to encompass timbral characteristics

System Design 1: Goal-directedness toward a specific sonic condition

• Selected sonic properties of feedback sound for target conditions

• Average pitch: Average fundamental frequency (YIN algorithm)

• Vibrato (Pitch fluctuation): Standard deviation of the fundamental frequency curve

• Tremolo (Volume fluctuation): Standard deviation of the amplitude curve

• Tonality (distinction between tone-like or noise-like signal): Spectral flatness

• Brightness: Spectral centroid

• User selects one or a combination of the properties and designates a value for each properties (Example: 500Hz average pitch and 0.9 tonality)

• Higher-level criteria (continuity, contrast, surprise, tranquility, tension, …) will be considered (future works)

mic 1speaker 1

speaker 2

speaker 3

speaker 4

mic 2

Analyzer & Controller Each line consists of 8 DSP components

-> organized by the controller

System Design 2: Second-order Feedback Structure

System Design 2: Second-order Feedback Structure

• No single algorithm satisfying feature objectives within unknown environmental conditions: modular approach as an effective alternative

• Replacement of live algorithm by another algorithm

• Facilitation of creativity by providing the basis for combinatoric search through a space of possibilities

• Second-order feedback structure based on the modularity

• The system switches internal structures and types/parameters of signal processing components for greater dynamics/variety in shaping the feedback

System Design 2: Second-order Feedback Structure

Types/Ranges of the DSP components used in second-order structure

• Lowpass Filter: 600~1200Hz (cutoff frequency)

• Bandstop Filter: 600~900Hz (first cutoff frequency) and 3600~5400Hz (second cutoff frequency)

• Bandpass Filter: 30~200 (first cutoff frequency) and 430~600Hz (second cutoff frequency)

• Amplifier: 3~100 (amplification degree)

• Frequency Shifter: -300~300Hz

• Delay Line: 3~1000 samples

• Sinewave Generator: 100~600Hz

• Feedback: 0.5~3 (amplification degree)

• Feedforward: 0.5~3 (amplification degree)

• Bypass

System Design 3: Evolutionary Process For Second-order Structure Control

• Evolutionary process: a well-established method to explore huge parameter spaces, including tasks in music composition/sound synthesis + musically interesting results in which target behaviors and other behaviors not specified by the goals

• In our work,

• Genotype: pairs of genes to specify the types/parameters of DSP components

• Phenotype: signal-processing algorithm

System Design 3: Evolutionary Process For Second-order Structure Control

1 - The system randomly creates 8 genotypes, which each consists of 64 pairs of genes corresponding to types/parameters of DSP components

2 - Feedback sounds are generated by sequentially developed DSP algorithms from the genotypes

3 - The analyzer evaluates fitness of each genotype by analyzing the received signals based on the target conditions, which could be changed in real-time by a user

4 - Genotypes in the following generation are selected randomly by proportion to the fitness, with mutation (replacement of types/parameters of the DSP components)

5 - Repeat from step 2

System Design 3: Evolutionary Process For Second-order Structure Control

• Reproduction uses mutations in the parameters or types of a component

• Default probabilities are 0.08 for parameter change and 0.05 for type change in each gene (controllable by a user)

• Rather frequent reproductions because of “fitness bottleneck”: the evaluation of each individual must take place in real-space over a reasonable duration

• Adaptive control depending on current fitness will be used (future work)

Implementation: Controller + Analyzer (OpenFrameworks)

Feature measurement for fitness evaluation

UI for selecting target conditions

Received signal & analysis

Genotype currently being developed

Genotype values

Implementation: Controller + Analyzer (OpenFrameworks)

Feature measurement for fitness evaluation

UI for selecting target conditions

Received signal & analysis

Genotype currently being developed

Genotype values

Implementation: Second-order DSP Structure (Max/MSP)

Input signals

Output signals before mixing

Subpatchers for DSP algorithms

Sending singnals to analyzer

Installation

mic1

speaker 1

mic2speaker 2

speaker 3

speaker 4

Demo

Evaluation

• Evaluation to show how sonic behaviors of the system would change according to the target conditions

• Average (solid lines) and optimal values (dash lines) of individuals in each generation

• Red and blue lines when the target value is maximum (red) and minimum (blue)

• Green dash horizontal lines represent the target values

• Red vertical line represents the change of the target state

Evaluation

• Demo sounds

• High average pitch

• Low average pitch

• High vibrato

• Low vibrato

• High tremolo

• Low tremolo

• High spectral flatness (noise signal)

• Low spectral flatness (harmonic tone)

• High spectral centroid (bright)

• Low spectral centroid (dark)

Conclusion

• High-level intentional control of audio feedback

• Evolutionary process to control a second-order structure + User control by selecting and changing target conditions

• Expectations: Enhancement of interactivity in the overall sonic behaviors of feedback systems and sound/music applications featuring balance between nonlinear emergence and regulation

• Future works 1: High-level control

• Continuity, contrast, surprise, tranquility, tension, …

• Based on timbre studies which relates perceptual and acoustic properties

• Future works 2: Improvement in the evolutionary process

• Adaptive control of mutation probabilities

• Improvement of fitness criteria

Thank you