hard real-time constraints in the digital music...

Hard real-time constraints in the Digital MusicIndustry

Leonardo Gabrielli

Universita Politecnica delle Marche, Ancona, Italy

June, 26th, 2014 - RTS-LIKE - Pisa

June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 1

3MediaLabs - what

3MediaLabs: Multimedia Information Processing

Made of two research sub-groups: A3LAB and SEMEDIA.LEADER: Prof. Francesco Piazza

A3LAB: DSP Algorithms and Adaptivesystems for Audio applications

Data Processing Approach

People: 2 Assistant Professors, 5PostDocs, 4 Phd Students

SEMEDIA: Semantic Web and Multime-dia

MetaData Processing Approach

People: 3 PostDocs


3MediaLabs - A3LAB

Research Topics

Computational Intelligence: Neural Networks, Extreme LearningMachine, Blind Signal Processing.

Speech and Music Processing : Speech Enhancement, Speech/SpeakerRecognition, Acoustic Echo and Feedback Cancellation,Speech-interfaced Systems for HMI, Music Signal Analysis-Synthesis.

Immersive Audio Rendering : Digital Equalization, Wave-field Synthesisand Analysis, Loudspeakers array, Digital effects, 3D Audio.

In-car Audio: Advanced Car Information System, Professional Audio.

Projects

Privately and publicly (EU, National, Regional) funded

More than 1.5 Million Euro in the last 5 years


Collaborations

Companies: Texas Instruments, Faital Spa, Leaff Engineering,VideoWorks, CRN Yacht, FBT, Radvision, and so on so forth.

Universities: Technical University of Munich, Aalto University, Universityof Illinois at Chicago, South China University of Technology, ImperialCollege (UK), Chinese Academy of Sciences and others.

Member of the Texas Instrument European University Program

Related Companies

Leaff Engineering : former Spin-off operating in the Digital Audio field.

Sensible Logic: former Spin-off operating in the Semantic Web andDigital Libraries field.

DowSee: brand-new Spin-off operating in the Energy Management andSmart Grid fields.


The New Lab

Big acoustic semi-anechoic chamber (11m x 9m x 6m, ISO3744, ISO3745, Ft100Hz)

Media-lab, Texas Instrument DSP lab, and more


The New Lab


Speaker’s Bio

LEONARDO GABRIELLI

PhD candidate at UNIVPM:Physics modeling of ElectroacousticMusical InstrumentsMusic Technology: wirelessconnection and embedded platforms

Co-founder and CTO of DowSee srl:

Spinoff company of UNIVPMWireless Sensor Networks for theWater GridComputational Intelligence forDomestic Gas, Water andElectricity


Physically Informed Sound Synthesis

Analysis and Synthesis of Stringed Instruments

Virtual Acoustic Feedback: digital emulation of feedbacks betweensound sources and capture devices

e.g. guitar - loudspeaker feedbackcan be used to augment any kind of pitched instrument


WeMUST

A tool for live music at home / studioslow-latency, uncompressed audiocan be integrated into existingtechnologies (e.g. PCs, tablets)built on existing technologieseasy configuration and connectionaudio + MIDI + OSC...

Emphasis is put on embedded devices

Applications

Live: rehearsal studios

Laptop orchestras

Art installations and exhibitions

Currently we are improving reliability and studying techniques foroutdoor/stage use


Some history


40 years of Digital Music Industry

Prophet 5 (1978)

Arch: MCU controlling a full Analog signal path

5-voices polyphony

MCU: (Zilog Z80)

VCO, VCF: SSM (Rev.1,2), Curtis CEM (Rev.3)

6000 units sold



Fairlight CMI (1979)

Sampling system

dual Motorola 6800

Memory: 16 kB per voice, System: 64 kB, Video: 16 kB (512x256pixels)

street price: 28,000£!!

Compare: KORG M-1 (1988): cost 1500£, 250,000 units sold



MIDI (Musical Instrument Digital Interface) (1981)

Current-loop UART at 31250 baud

3-byte control messages



Yamaha DX7 (1983)

Digital FM synthesis, 6 operators

Patches, Storage disk

early MIDI adopter

200,000 units sold



Computer Music SW

Max Mathews’ MUSIC (1957), computer language

Miller Puckette’s Max (mid-1980s): visual programming language

...



Virtual Analog

DSP to emulate analog waveshapes: Virtual Analog

Motorola 56002 DSP (80MHz, 40MIPS, 24bit fixed point)

Motorola 68331 MCU


RECAP: HW

Transition era: late 70s to the 80s:MCUs control a full Analog/Discrete Digital signal path

1990s: DSP era (TI, AD, Motorola)Sample-wise, Assembly, Fixed-pointEarly integration with MCU cores (e.g. TMS320C5x + ARM7)

2000s: PCs get to grips with real musicIntel SSE (1999) enhances FP calculus in x86 CPUsMany contemporary music software houses take offNote: Atari/Apple music SW since the late 80s

2000s: Peak point in discrete DSPs usage:Floating+Fixed point, 100MHz-1GHz, C coding, frame-wiseApplication processors: DSP + MCU

Note: ARM attempts successfully at Sig.Proc in 2005 with NEON


Current HW /2

2008: begin of the smartphone eraARM cores everywhere!

Consequences:ARM dominating in handheldsx86 dominating in PCsLast years: end of the all-SW era: musicians want tangible interfaceshybridization between interface (control) and engine (synthesis):



Tux is playin’ (and payin’)

Viscount UNICO Organs (Italy, 2004)

KORG Oasys (2005): Intel Pentium 4 and a ”customized Linux”

KORG Triton (2011): Intel Atom and a ”custom OS over Linux”

Viscount Physis piano (Italy, 2011)



Physis Piano (2011)

6 TI OMAP L137 (C67x + ARM9) + ARM9 master MCU

Master core runs Debian

Kernel drivers contributions

And latest news: Japanese manufacturer to release a low-end workstationbased on all-ARM processing?


all-ARM musical instruments

converging to standardized HW platforms

no SW convergence

side note: Android audio API are lame!


Constraints


Technical Constraints and Issues

Response below perceptual latency (ON-LINE vs REAL-TIME!)

No dropouts

FLOPS!

Robust against EM noise and surge

Gig-proof Build quality

User Requirements

Aesthetics

Sound Quality

Flexibility

User-friendliness


Typical Team

2 HW Eng

N SW Eng (2-3x Algo, 1x Control, 1x DSP-Sys Integration, 1x Linuxspecialist)

1-3 ”Musicians”

1 Tech Officer

Manufacturing staff


Tech stuff


ARM-based systems and GNU/Linux

ARM cores are not designed for DSP, but:leveraging SDKs reduces time to marketSW libraries include optimized DSP routinesReference designs facilitate HW designEVM boards allow parallel HW/SW developmentDSP programming headaches avoided (parallelism, mem sharing, ...)

Viable solutions for ARM development:

eCos

freeRTOS

QNX

Windows CE

Linux


Current SW stack for audio processing


Audio servers

JACK is *the* solution

Pulseaudio resamples everything (no synch, adds overhead)

ALSA can only give exclusive control


Latency: HW and SW

With JACK, latency is HW-dependent only

synchronous processing

Time unit = 1 period p of 2i samples

Total input-output latency:

Lio = (Ni +No)× pFs

+AL

with Ni, No number of buffers (min:2, ping-pong)

AL is the algorithmic latency (0 if no components with memory)


Scheduling Jack

Jack currently runs as a SCHED FIFO thread plus some SCHED OTHER

threads.

Jack clients run as a DSP SCHED FIFO thread plus SCHED OTHER threads

every Jack client gets a RT priority lower than that of Jack

so Jack can preempt clients breaking the deadline

when a client does not finish within the next IRQ: XRUN!


Scheduling Jack


Scheduling Jack

SCHED DEADLINE is aware of periodicity and CPU bandwidth

Can SCHED DEADLINE yield better performance than SCHED FIFO?


Previous Works

Cucinotta et al. ”Low-latency Audio on Linux by Means of Real-TimeScheduling”, LAC2011


Scheduling ALSA

Remove the need for:

synchrony

interconnection

extra features of JACK

Take exclusive control of ALSA and have a single audio process running on thesystem.Periodicity can be gained with SCHED DEADLINE


References

Cucinotta, Faggioli, and Bagnoli. ”Low-latency audio on linux by meansof real-time scheduling.” In Proceedings of the Linux Audio Conference(LAC 2011)

Gabrielli, Leonardo, Stefano Squartini, Emanuele Principi, and FrancescoPiazza. ”Networked beagleboards for wireless music applications.” InEducation and Research Conference (EDERC), 2012 5th European DSP,pp. 291-295. IEEE, 2012.

Gabrielli, Leonardo, Stefano Squartini, and Francesco Piazza.”Advancements and Performance analysis on the Wireless Music Studio(WeMUST) framework.” In Audio Engineering Society Convention 134.Audio Engineering Society, 2013.


hard real-time constraints in the digital music...

Documents