Epic NHL goal celebration hackwith a hue light show andreal time machine learning

François Maillet, Datacratic

PyCon Canada - Toronto November 7th, 2015

@mailletf blog.francoismaillet.com

YouTube demoyoutube.com/watch?v=tmh_eAq9yp4

About me

• Head of Machine Learning at Datacratic in Montréal

• Working on Machine Learning Database (MLDB.ai)

• Studied computer science and ML at Université de Montréal

• Research on music related problems: playlist generation, auto tagging, recommendation

blog.francoismaillet.com/epic-celebrationFor code snippets

and all technical details

How it all started…

Can we take it further?

Detect goals automatically?

Flash Philips Hues to the music?

Image source: http://www.chinadaily.com.cn/life/images/attachement/jpg/site1/20120807/001ec9790963118abddc19.jpg

The intuition

High level view of components

Light show API

Machine learning

Real-time detection

The machine learning problem

• Binary classification task • Classes

• positive: the Canadiens scored a goal • negative: the Canadiens did not score a goal

• P(Habs goal | some audio)

Creating the dataset

• Habs website has 4 minutes highlights of previous games

• Extracted audio to wav files of 10 highlights

• Created CSV file containing time at which goals occurred

Audio feature extraction

• Library: librosa(github.com/bmcfee/librosa)

• Used mel power spectrogram

MPS of Habs Goal

MPS of goal against Habs

Feature vector

• Vectorize MPS

• 2 second clips, frequencies up to 8KHz with 128 Mel bands, at a sampling rate of 22.5KHz

• MPS have a shape of 128×87 -> feature vector of 11,136 elements when vectorized

Selecting training examples

• Positive examples: Canadians goals • Negative examples:

• Goals against • 50 random segments per clip

Training a classifier

• Library: scikit-learn

• Models: logistic regression and SVM

• Performance: 0.97AUC, 0.63 F1-Score

Picking a decision heuristic

• Prediction != Decision

• Can use many predictions to make a decision

Decisioning with a rolling window

Evaluation metricsWHAT HAPPENED MODEL SAID RESULT

Habs scored! GOAL! True positive :D

Bad guys scored GOAL! REALLY BAD false positive

No goal GOAL! False positive

Habs scored! No goal False negative

What we used

• SVM: more true positives, but did worst job on false positives

• Used logistic regression

• To trigger a goal: 5 positives votes out of the last 20 and votes cast. Votes YES if the P(goal) > 90%.

• Cross-validation results: 23 Habs goals detected, 11 not detected, 2 opposing team goals falsely detected and no other false positives.

Looking at Habs’ 2014-15 season stats

• Scored an average of 2.61 goals per game; got scored 2.24 times

• Can loosely expect: • not detect 0.84 Habs goal per game

• go off for a goal by the opposing team once every 4 games.

Living room light show

Two parts to the light show

• Play Le goal song, by L’Oreille Cassée

• Flash lights with phue library

• github.com/studioimaginaire/phue

• Package it behind rest API using bottle

Hooking up the live audio stream

• Soundflower (rogueamoeba.com/freebies/soundflower) • virtual audio device • allows audio to be passed between apps

• pyaudio (people.csail.mit.edu/hubert/pyaudio) • play and record audio in Python

Real-time setupAudio from

browser

Soundflower

HDMI audio output

pyaudio

/goal endpoint

last 2 seconds of

audio

ML model P(goal | audio) 10 times / sec

last 20 predictions

Decision heuristic

Still using emergency

button

Putting together the Youtube video

• Library: matplotlib

• Generated each frame and saved to PNGs

• Stitched with ImageMagick

• Converted to video with FFmpeg

Future work

• Which team’s goalie is on screen?

• P(Habs Goal on screen | image)

• Generalize to the other Francophone station in Montréal

Thank you!

@mailletf François Maillet, Datacratic

PyCon Canada - Toronto

November 7th, 2015

blog.francoismaillet.com/epic-celebration