MUSIC TO COLOR BARCODEZOË JAMES, MOLLY ROBINS, ELIZABETH SHAHER

UNIVERSITY OF ROCHESTER ELECTRICAL & COMPUTER ENGINEERING DEPARTMENT

INTRODUCTIONWhen consuming music we rely solely on ourears to help us understand different characte-ristics of a piece. These characteristics, suchas loudness, pitch, and timbre, are importantdistinguishing features that can define the mu-sic we listen to. We believe that we could useother senses to understand these characteris-tics and how they relate to a piece as a whole.

We came across a TED talk by Neil Harbis-son who is color blind and sees the world throughfrequencies. He created a piece of hardware thatwould see colors and translate them to differentfrequencies. We sought out to do the opposite,by creating a program that could take in musicand sounds and output color. We decided thatprinting a color for every frame over time as a sin-gle static barcode would best represent our data.

BACKGROUND & RESEARCH

Relationships between Sound and ColorFrom our research we found that there is no directrelationship between color and sound. Theyare both frequency based, and color is createdthrough additive wave synthesis, but there isno corresponding color that can be associatedwith a specific sound. With no direct mathe-matical, or psychological relationship betweencolor and sound we had to design our own.

Key Component AnalysisThe components that were chosen for musicalanalysis were inspired by components used forgenre classification. Genre classification usestimbre, rhythmic content, and pitch content tocharacterize a piece and its content. We wereable to decide on three components that could beused to describe a piece, varied noticeably overtime, and were relevant to calculate. We choseloudness, to mirror how energy was includedin genre classification; the spectral centroid, torepresent timbre and brightness; and the loudestfrequency as this would be largely affected byany prominent instruments present in the piece.

METHODSThis project was written in MATLAB, as theprogram provides many tools for audio analysisand visualization. The code is comprised of asingle main script that calls two functions: onethat calculates the key components, and anotherthat translates those key components into an RGBcolor code. Each of these functions operate on asingle frame. The main script itself is where thesound file is input, and the barcode is generated.

Before analysis, the input is buffered into frameswith a frame length of 512 samples, using a rec-tangular window and a hop size of zero samples.After the buffering process, the input is processedframe-by-frame through the spectral analysisfunction followed by the sound-to-color conver-ting function. The output of the sound-to-colorconversion is then plotted as a stem plot witha wide line thickness, without the stem heads.

To create the RGB color code, the red va-lue was associated with loudness, the greenvalue was associated with brightness (spec-tral centroid), and the blue value was as-sociated with pitch (loudest frequency).

RESULTS & DISCUSSION

The red color was associated with Loudness. Thevalue was calculated using the formula below.

Our reason for choosing to multiply our decibellevel by 3 is that the most quiet sections of musicare about -80dB. When there is no scaling on thedecibel value the barcodes become overwhelmin-gly red. Since green and blue are both based onfrequency, they can be calculated the same way.To associate a value between 0 and 255 with a fre-quency that can range anywhere from 0 to 20,000we took the common frequency bands used formixing into account. We multiplied the bandnumber of the input frequency by 20 to accountfor the specific band. We then modulated thefrequency over 135 and added it to the multipleof 20. This allowed us to account for differen-ces between frequencies within the same band.

One of the best applications of our visualiza-tion software is the ability to use it as a wayto represent and recall data. Since it is notpossible to perfectly recreate the song, or anypiece of sound from the color barcode, wecan use them as representations of the audio.

To show how the barcodes are individual to thesong, we created barcodes for two different ver-sions of the same song, “Bohemian Rhapsody”.The first barcode in the figure below was madeusing original song, and the second barcode in thefigure was created using a cover of the song thatemulated the style, but was performed by a dif-ferent artist and produced by a different team.

The barcodes for these two versions of the samesong are visually different, and can be properlymatched within our database. However, it is alsoclear that these songs are very similar. The fre-quencies present and the volume of each sectionof the song are relatively similar. They look likethe same song, but can be differentiated, the sameway that they sound like the same song but caneasily be differentiated.

COLORBAR DATABASEWe created a small database of songs withtheir associated barcode. Our code canread and accurately match the barcode tothe correct song, including matching spe-cific versions of the same piece of music.

Our database currently works by matchingan input matrix with one of the matrices storedfrom previous musical outputs, meaning a newsong can not be identified, the barcode must existin the database beforehand. When a song doesexist in the database, the identification programwill output the title and artist of the song, thenplay a 30 second clip of the piece of music.

REFERENCES1. Libretexts, “Electromagnetic Radiation,” Chemistry LibreTexts, 23-Apr-2019. [Online]. [Accessed:

30-Apr-2019].2. Musical genre classification of audio signals - IEEE Journals Magazine. [Online].[Accessed: 30-

Apr-2019].3. N. Harbisson, “I listen to color,” TED. [Online]. [Accessed: 01-May-2019].4. “Understanding Hexadecimal Colors is Simple,” Pluralsight, 19-Jan-2015. [Online]. [Accessed:

30-Apr-2019].