bpm js

jsbpm

onset detection in the BrowserJanessa Det - WEB eng @TWITTERNYC

hello.bpmjs

janessa det (@jandet)

twitter nycSoftware engineerWeb Team

columbia universityM.S. Computer scienceGraphics & Vision

duke universityB.S.E. biomedical engineering

IRON MAN

onset detectionbpm. what?

bpmjsjanessa det (@jandet)

onset detectionfundamental to:sonar, speech, EKG's,movement detection

thisthing

IRON MAN

AUTOMATICTRANSCRIPTIONOF MUSIC

onset detectionbpm. what?


AUTOMATICTRANSCRIPTIONOF MUSIC

thisthing

1. instrument identification

2. instrument isolation

3. pitch detection

4. dynamics

accessibleapplicable

sharablecross-platform

fast

digital signals processing

in the browser. why?bpmjs


accessibleapplicable

sharablecross-platform

fast

digital signals processing

javascript

in the browser. why?bpmjs


because

javascript

and it would be cool

is fun

to play with

diverse

diversity is powerful.bpmjs


the browser is

diversethe community is



yeah,that’s me

reality

the dream

"finding ways to breedengineering research, music, and artwith the trusty steed of javascriptto make the magical unicorns of my imagination a reality"



yeah,that’s me

reality

the dream

"finding ways to breedengineering research, music, and artwith the trusty steed of javascriptto make the magical unicorns of my imagination a reality"

ok, let’sbreed some

unicorns

onset detection?how do we doonset detection?

GENRES

tempos

beats

onsets

what is bpm?bpmjs


1. pitch

2. notes

3. percussive beats

DRUM AND BASS 160-180 BPM

House 118-135 BPM

hip-hop 115 BPM

Concert marches 120 BPM

Screamers 130-150 BPM

beatsGENRES

what is a beat?

1. pitched non-percussive

2. pitched percussive

3. non-pitched percussive

4. complex mixed

onsetstempos

Largo 40-60 BPM

Larghetto 60-66 BPM

Adagio 66-76 BPM

Andante 76-108 BPM

Moderato 108-120 BPM

Allegro 120-168 BPM

Presto 168-200 BPM

Prestissimo 200+ BPM

bpmjsjanessa det (@jandet)detection method.

DETECTIONFUNCTIONstft

DETECTIONFUNCTIONSTFT PEAK PICKINGPEAK PICKING BPMBPM

Prep the signalfor Detection

Functions

Selectively maximize for onset properties

Pick out peak times from Detection Function

CalculateBPM

ಠ_ಠ

http://disapprovalface.com/

http://disapprovalface.com/

bpmjsjanessa det (@jandet)detection method.

DETECTIONFUNCTIONstft

DETECTIONFUNCTIONSTFT PEAK PICKINGPEAK PICKING BPMBPM

Prep the signalfor Detection

Functions

Selectively maximize for onset properties

Pick out peak times from Detection Function

CalculateBPM

stft. short-time fourier transform.


4. zero paddingZero pad to nearest power of 2

2. hamming windowApply windowing function -- Hamming

3. normalizationNormalize magnitude

5. fast fourier transformConvert to frequency domain

1. window slicingSlice signal into overlapping windows

detection functions.bpmjs


frequency based methods

3. phase deviation

1. High frequency content

2. spectral difference

4. euclidian distance

peak pickingbuild threshold

peak picking.bpmjs


build threshold1. median filterSmooth out the signal for thresholding

2. adaptive thresholdingThreshold by median-filtered signal

3. initial peak findingIdentify local maxima

4. thresholding peaksApply adaptive threshold to peak findings

5. refractory periodApply minimum time between peaks as threshold

peak picking

120 BPM

bpm calculation.bpmjs


0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0peak times (in seconds)

0.5

1 beat / 0.5 sec * 60 sec / 1 min

= 120 BPM

in javascript?how do we do itin javascript?

color profile of

music characterization.bpmjs


color profile ofso insane by discovery

average ~ 115 bpm

0:30 1:000:15 0:45 1:15 1:45 2:151:30 2:000:00 2:30 3:002:45 3:15

dsp.jsweb audio

complex arithmetic

a lot of loops & mathbrowserify

tools used.bpmjs


dsp.jsweb audio

complex arithmetic

a lot of loops & mathbrowserify

i

modules.bpmjs


peak picking

detection

bpmweb audioAudioBuffer node

hfc sd pd e. dist.

dsp.js

fft windowing

complexstft

how to not break chrome

js challenges.bpmjs


float32array


compatibility

complex arithmeticmatrices


js challenges.bpmjs


float32array


compatibility

complex arithmeticmatrices

NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN... NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN NaNNaNNaNNaNNaNNaNNaNNaN

0. signal winndow

stft. short-time fourier transform.


3. ZERO PADDING

1. HAMMING WINDOW

2. NORMALIZATION

4. FFT -> Complex objects


detection functions.hfc

sd

pd

e. dist.

not bad...


peak picking.detectionfunction (ed)

median filtered

thresholdedetc

THE DREAM:

matlab!!benchmarks

optimizations& performance


1. web workers for stft2. SLICE WELL-SELECTED WINDOWS

benchmarksvs. matlab!!

THE DREAM: “REAL-TIME”

3. asm.js, enscripten, LLjs

weaknesses.

future work.

weaknesses.bpmjs


1. sensitive tuning parameters

2. a priori knowledge helpful to select detection functions

3. median filtering or aggressive thresholding can cripple

the detection function

future work.1. only scratching the surface of research

2. many detection functions

3. onset detection for different applications

is the unicorn beautiful?ok so how did i do?is the unicorn beautiful?


ok, it feels alittle forced

but we’re getting there!!

the web is changing.bpmjs


web audio api

dsp.jsGod Bless npm

rethinkingweb engineering.


what does it mean to be aweb engineer?how about more engineering?

this is not the final frontier


embrace diversityopen community of learning

knowledge builds upon itselfcombining fields is very powerful

think outside the boxbreed those unicorns

rethinkingweb engineering.


thanks.

go forth and breed unicorns

main referencesBeat Detection for Automated Music Transcription:An exploration of Onset Detection Algorithms.http://bingweb.binghamton.edu/~ahess2/Onset_Detection_Nov302011.pdf

The Scientist and Engineer's Guide toDigital Signal ProcessingBy Steven W. Smith, Ph.D.http://www.dspguide.com/pdfbook.htm

additional resources:Gamedev: beat detection algorithmshttp://archive.gamedev.net/archive/reference/programming/features/beatdetection/index.html

BPM Databasehttp://www.bpmdatabase.com/

http://bingweb.binghamton.edu/~ahess2/Onset_Detection_Nov302011.pdf












http://archive.gamedev.net/archive/reference/programming/features/beatdetection/index.html




http://www.bpmdatabase.com/

http://www.bpmdatabase.com/

bpm js

Technology