a different flavor of audio programming

A different flavor of audio programming

We won’t talk about

● Real time programming● Signal processing● C++

A different flavor of audio programming

We won’t talk about

● Real time programming● Signal processing● C++

We will talk about

● Digital Audio Workstation (DAW) internals○ File formats○ Representation of time○ Automation points○ Audio rendering details

● Python

About me (@offlinemark)

● Previously security engineer○ Reversed binaries, file formats,

network protocols○ Lead developer for Manticore project:

analysis tool for reversing x86/ARM● Music producer & DJ

○ 10 years using DAWs○ REAPER, Garageband, Ableton Live, FL Studio

Disclaimer

Digital Audio Workstations (DAWs)

DAW Core

DAW Core

Reverb Distortion EQ Limiter

Plugins

DAW Core

Reverb Distortion EQ Limiter

Plugins

Audio Subsystem

Needs of DAW users

Audio

AudioPlugins

Needs of DAW users

Audio

“Workflow”

Plugins

???

Needs of DAW users

Workflow Problem Example:

Time Marker Export

DAW Core

Time Marker Export

DAW Core

Time Marker Export

Audio Subsystem

Time Markers

DAW Core

Time Marker Export

Audio Subsystem

Time Markers

DAW Core

.als, .flp

Project File

Time Markers

SaveProject

.als, .flp

DAW Core

Project File

Time Markers

Time Markers

.als, .flp

DAW Core

Project File

Time Markers

DAW CoreTime

Markers

LoadProject

.als, .flp

DAW Core

Third PartyTool

Project File

Time Markers

LoadProject

Time Markers

.als, .flp

DAW Core

Third PartyTool

Project File

Time Markers

Export

Time Markers

00:00 - A05:22 - B10:59 - C

markers.txt

DAW Core

???????????????????????????????

Third PartyTool

Project File

Time Markers

LoadProject

Time Markers

???

ReverseEngineering

Taking things apart to see how they work

● Fix● Learn● Extend

???????????????????????????????

��

???????????????????????????????

DAW Core

Third PartyTool

Project File

Time Markers

LoadProject???

DAW Core

Third PartyTool

Project File

Time Markers

LoadProject???

DAW Core

Third PartyTool

Project File

Time Markers

LoadProject???

DAW Core

Third PartyTool

Project File

Time Markers

LoadProject

Time Markers

Prior Art: Solving workflow problems

Live Enhancement Suite● By @DylanTallchief and @InvertedSilence

● Extends Live via desktop automation (AutoHotkey, Hammerspoon) ● New hotkeys, menus, features● http://enhancementsuite.me/

@bcrypt’s rekordbox scripts

● ableton-to-cues.py - convert between Ableton Live Warp Markers and Rekordbox cues

● https://github.com/diracdeltas/rekordbox-scripts

Time Marker Extraction

Our goal:● Third party tool● Analyze DAW project file● Extract time markers● Output as plain text

○ Marker text○ Marker time (mm:ss format)

The map:

1. Easy Mode2. Hard Mode3. Unreal Mode

Easy Mode

No tempo automation(Single static tempo)

Tempo automation: Dynamic BPM changes

Tempo automation: Dynamic BPM changes

Hard Mode

With tempo automation (naive impl, inaccurate)

(Dynamic tempo)

With tempo automation (fancy impl, accurate)

(DAW rendering engine details)

Unreal Mode

The map:

1. Easy Mode2. Hard Mode3. Unreal Mode4. Super Unreal Mode 😱

1. Easy Mode2. Hard Mode3. Unreal Mode4. Super Unreal Mode 😱

The map:

With nonlinear tempo automation

1. Easy Mode2. Hard Mode3. Unreal Mode4. Super Unreal Mode 😱

The map:

With nonlinear tempo automation

1. Easy Mode2. Hard Mode3. Unreal Mode4. Super Unreal Mode 😱

The map:

We will only coverlinear tempo automation today

Easy mode: No tempo automation

My first guess at the Marker structure:

struct Marker {int time; // millisecondsstring text;

};

Wrong! Times are stored in beat time

struct Marker {int beat_time; // beat timestring text;

};

DAWS store object times in beat time

Beat time is stable

Beat time is convertible to real timeusing the BPM (Beats per Minute)

Pseudocode

// 1. Parse markers and BPM// 2. Print outfor marker in markers { sec_time = marker.beat_time * (60/BPM) print_out(sec_time, marker.text)}

Pseudocode

// 1. Parse markers and BPM// 2. Print outfor marker in markers { sec_time = marker.beat_time * (60/BPM) print_out(sec_time, marker.text)}

Pseudocode

// 1. Parse markers and BPM// 2. Print outfor marker in markers { sec_time = marker.beat_time * (60/BPM) print_out(sec_time, marker.text)}

Easy ModeWhat we need:

● Markers (beat time, text)● Tempo (BPM)

Ableton Live Format (.als)

$ file demo.alsdemo.als: gzip compressed data...

Ableton Live Format (.als)

● Compressed XML document● Declarative object hierarchy● Human readable

(but undocumented)

Ableton Live Format (.als): Locators

Ableton Live Format (.als): Tempo (BPM)

FL Studio Format (.flp)

$ file demo.flpdemo.flp: data

FL Studio Format (.flp)

● Raw binary format● Not human readable,

“undocumented”

FLP Format Resources: Notes from devs (1999)

FLP Format Resources

● github.com/LMMS/lmms/ (10 years old)● github.com/andrewrk/PyDaw (10 years old)● github.com/monadgroup/FLParser

Helpful, but missing parts of the format.

FL Studio Format (.flp)

Header ● Header section: file signature, header len, ...● Data section: array of Event structures● Iterate and interpret events to construct state● Each event carries 1 piece of data (int/string) about

project*

*variable length data can contain multiple pieces of data via an array of structs

Events

FL Studio Format (.flp)

Header ● Event Structure○ ID field: 1 byte ID (0-255)○ Data field: Depends on ID

■ 0-63: int8/uint_8■ 64-127: int16/uint_16■ 128-191: int32/uint_32■ 192-255: variable length (additional size field)

Event 1

Event 2

Event 3

Event 4

...*variable length data can contain multiple pieces of data via an array of structs

FL Studio Format (.flp)

Header ● Up to 256 events● Relevant events:

○ MARKER_TIME (0x94) [uint32]: marker beat*○ MARKER_TEXT (0xcd) [UTF-16]: marker text○ TEMPO (0x9c) [uint32]: millibeat per minute

Event 1

Event 2

Event 3

Event 4

...*technically is a “pulse”, which is a beat subdivision

FL Studio Format (.flp)

Header ● Up to 256 events● Relevant events:

○ MARKER_TIME (0x94) [uint32]: marker beat*○ MARKER_TEXT (0xcd) [UTF-16]: marker text○ TEMPO (0x9c) [uint32]: millibeat per minute

Event 1

Event 2

Event 3

Event 4

...*technically is a “pulse”, which is a beat subdivision

Easy ModeWhat we need:

● ✅ Markers (beat time, text)● ✅ Tempo (BPM)

Pseudocode

// parse markers (beat, text) and BPMfor marker in markers { sec_time = marker.beat_time * (60/BPM) print_out(sec_time, marker.text)}

Hard mode: Tempo automation (naive)

Tempo automation

Handling tempo automation

Abstract Algorithm● Start at beginning of timeline● For each automation segment

○ Compute time elapsed during segment● Continue until we reach marker● Sum all elapsed time segments

How are automation points stored?

struct Point {int beat_time;int BPM_value;

};

Hard mode pseudocode

// parse markers (beat, text)// and tempo automation pointsfor marker in markers { sec_time = compute_sec_time(marker.beat_time, points) print_out(sec_time, marker.text)}

Hard mode pseudocode

// parse markers (beat, text)// and tempo automation pointsfor marker in markers { sec_time = compute_sec_time(marker.beat_time, points) print_out(sec_time, marker.text)}

Hard mode pseudocode

// parse markers (beat, text)// and tempo automation pointsfor marker in markers { sec_time = compute_sec_time(marker.beat_time, points) print_out(sec_time, marker.text)}

int compute_sec_time(marker_beat_time, points) { // Accumulation algorithm here}

Hard ModeWhat we need to do

● ✅ Markers (beat time, text)● 🤔 Automation points● 🤔 How to compute

elapsed time?

Computing time elapsed between points

Computing time elapsed = Physics I

🚗💨

Computing time elapsed = Physics I

Integrate to calculatedistance elapsed

Computing time elapsed = Physics I

✅

Can’t integrate the BPM curve directly

Can’t integrate the BPM curve directly

Can’t integrate the BPM curve directly

Must invert BPM to Minutes Per Beat (MPB)

✅

..or even Seconds Per Beat (SPB)

✅

Computing Time Elapsed

for a BPM automation

1. Construct BPM function2. Convert to SPB form3. Integrate SPB function

1. Construct BPM Function

2. Convert to SPB form

3. Integrate SPB function

✅

3. Integrate SPB function

3. Integrate SPB function

✅Time Elapsed: 2.77 seconds

Hard ModeWhat we need:

● ✅ Markers (beat time, text)● ✅ How to compute

elapsed time?● 🤔 Automation points

Ableton Live Format (.als): Tempo Automation Points

Ableton Live: Tempo Automation Points

Ableton Live: Tempo Automation Points

Ableton Live: Tempo Automation Points

Ableton Live: Tempo Automation Points

Ableton Live: Tempo Automation Points

Fl Studio: Tempo Automation Points

Fl Studio: Tempo Automation Points

Fl Studio: Tempo Automation Points

Channels

Playlist Items

Structures

struct Channel { int id; int dest_id; int param_id; Point[] points; // ...};

struct PlaylistItem { int channel_id; int start; int length; // ...};

Structures

struct Channel { int id; int dest_id; int param_id; Point[] points; // ...};

struct PlaylistItem { int channel_id; int start_beat; int length; // beat time // ...};

Approach● Parse all channels and playlist items

○ Events: CHANNEL_NEW (0x40), AUTOMATION_CHANNEL (0xe3), AUTOMATION_DATA (0xea), PLAYLIST_ITEMS (0xe9)

Approach● Filter channels for tempo automation channels

○ Channel dest_id = 0x4000 (Master Track), param_id = 0x5 (Tempo Parameter)

Approach● Filter playlist items for tempo automation playlist items

○ Using channel IDs of tempo automation channels

Approach● Render each playlist item’s points

○ Resolve global timeline positions of points

Approach● Render each playlist item’s points

○ Resolve global timeline positions of points

Channel 1:Point 1: Start + 2Point 2: Start + 10Point 3: Start + 14

Approach● Render each playlist item’s points

○ Resolve global timeline positions of points

Channel 1:Point 1: Start + 2Point 2: Start + 10Point 3: Start + 14

Item 1: Start: 10 Item 2: Start: 60

Approach● Render each playlist item’s points

○ Resolve global timeline positions of points

Channel 1:Point 1: Start + 2Point 2: Start + 10Point 3: Start + 14

Item 1: Start: 10Point 1: 12Point 2: 20Point 3: 24

Item 2: Start: 60Point 4: 62Point 5: 80Point 6: 84

Approach● Render each playlist item

○ Resolve global timeline positions of playlist item’s points

Approach● Render each playlist item

○ Resolve global timeline positions of playlist item’s points

Approach● Render each playlist item

○ Resolve global timeline positions of playlist item’s points

Approach● Merge playlist items

Hard ModeWhat we need:

● ✅ Markers (beat time, text)● ✅ Automation points● ✅ How to compute

elapsed time?

One little problem..

It should take 2.77 seconds

✅Time Elapsed: 2.77 seconds

What does the DAW say?

The DAW doesn’t match the theoretical result!

Lesson 1: Accuracy is relevant!

The quality of implementation will affect our accuracy.

Lesson 2: Accuracy defined by the DAW

At the end of the day, we must match the DAW.

Unreal mode: Tempo automation (fancy)

Enough physics, let’s talk audio rendering

DAWs can’t truly render audio at changing tempo

Approximate by sampling automation curve

Approximate by sampling automation curve

Requires new approach for calculating time elapsed!

1. Find BPMs at step points

2. Convert BPMs to SPB

3. Construct SPB stepwise function

4. Compute area under SPB stepwise function

Time Elapsed

One Key Parameter: Tempo Quantization Value

??

One Key Parameter: Curve Quantization

➔ DAW implementation detail➔ Not documented➔ Not stored in project file

Unreal ModeWhat we need:

● ✅ Markers (beat time, text)● ✅ Automation points● ✅ How to compute

elapsed time?● 🤔 Quantization value?

How to find tempoquantization?

???

How to find tempoquantization?

Give up and ask for help :(

How to find tempoquantization?

● ✅ Give up● Brute force

Brute forcing the quantization value

➔ Quantization is happening.➔ Will be a power of 2. (16th, 32rd, 64th note, ...)

➔ Limited possibilities in practice. (214th notes are impractical)

What we know:

1. Pick a known automation curve (60-120, 4 beats)

1. Pick a known automation curve (e.g. 60-120, 4 beats)

And remember what the DAW reported

2. Create experimental test harness

Standalone time calculation, parameterized on quantization

2. Create experimental test harness

Standalone time calculation, parameterized on quantization

Input: Start BPM, End BPM, # Beats, Quantization Value

2. Create experimental test harness

Input: Start BPM, End BPM, # Beats, Quantization Value

Output: Time Elapsed

2. Create experimental test harness

Output: Time Elapsed

Input: Start BPM, End BPM, # Beats, Quantization Value

2. Create experimental test harness

3. Try a bunch of values! See which matches the DAW.

3. Try a bunch of values! See which matches the DAW.

Ableton Live’s quantization value

FL Studio’s quantization value?

FL Studio’s quantization value?

How to find tempoquantization?

● ✅ Give up● ✅ Brute force● 👂Observe using

DAW

Hearing the quantization

Unreal ModeWhat we need:

● ✅ Markers (beat time, text)● ✅ Automation points● ✅ How to compute

elapsed time?● ✅ Quantization value

.als

Ableton Live Project Export

Time Markers

00:00 - A05:22 - B10:59 - C

markers.txt

.flp

FL Studio Project

Automation Points

Third Party Tool

⚙

.als

Ableton Live Project Export

Time Markers

00:00 - A05:22 - B10:59 - C

markers.txt

.flp

FL Studio Project

Automation Points

Third Party Tool

✅ Third party tool✅ Extracts time markers✅ Robust & high accuracy

⚙

● ✅ Third party tool● ✅ Analyze DAW project file● ✅ Extract time markers● ✅ Output as plain text

○ Marker text○ Marker time (mm:ss format)

Our goal:

Implementation & Evaluation

dawtool 🛠

● Python 3● Parsers for DAW formats

○ Ableton Live v8-10○ FL Studio v11-12, 20

● Time marker extraction● Theoretical & DAW modes● No dependencies*

Evaluation: Speed

Evaluation: Speed

Evaluation: Accuracy (Stress Tests)

DAW Max Observed Err.

Ableton Live

DAW Max Observed Err.

Ableton Live .001 s

Evaluation: Accuracy (Stress Tests)

1 ms error over 16 hour Live set 🤷‍♂

DAW Max Observed Error % Error

Live .001 s .000002

FL Studio

Evaluation: Accuracy (Stress Tests)

DAW Max Observed Error % Error

Live .001 s .000002

FL Studio .56 s

Evaluation: Accuracy (Stress Tests)

DAW Max Observed Error % Error

Live .001 s 0.000002

FL Studio .56 s 0.0029

Evaluation: Accuracy (Stress Tests)

Demo

Conclusion

Audio

“Workflow”

Plugins

Needs of DAW users

AudioPlugins

Project file manipulation

Needs of DAW users

AudioPlugins

Project file manipulation

Desktop Automation

Needs of DAW users

AudioPlugins

Project file manipulation

Time Marker Exporter

Needs of DAW users

AudioPlugins

Project file manipulation

What else?? 🤔

Needs of DAW users

Timestamps (https://timestamps.me)

People actually use it!

You are more powerful than you think.

Acknowledgements

Luke Van Seters

Ableton & Image Line

All Timestamps users

ADC ‘20 Organizers

Thank you.Mark Mossberg @offlinemarkgithub.com/offlinemark/dawtool

Timestampstimestamps.me