By: Michael Pouris, BSc, Deborah I. Fels, PhD, P.EngIMDC, Ryerson University, Toronto, Canada

1. Background information2. Research and Model

Purpose of the visualization

Past Industry attempts

The priorities

Psychological model

3. The implementation

Translating to a visual medium

4. Study results and discussion

Music is a major art form Present in all cultures worldwide Transcends cultural and language boundaries

Portal to a cultural shared experience (McDermott, 2004)

Hard of hearing and deaf have limited access to a shared hearing experience

Serve as a tool that depicts music visually for hard-of-hearing or deaf audiences

Sensory substitution is possible (Nanayakkara, Taylor, Wyse and Ong, 2009)

Research questions:

Can sensory substitution models be used to provide access to music using the visual channel?

What are user reactions to MusicViz?

Industry attempts have failed

Does not use justified psychological model which describes auditory mappings to visual system

Three primary goals are to develop a visualization that is:

1. Aesthetically pleasing

2. Emotionally moving

3. Adaptable

Interpretation is to explain the meaning of something with subjective bias

Interpreting feelings of the music is not feasible

There is not one “feeling” for a musical piece

Everyone interprets music differently

Goal: Visualization is open to each individuals interpretation

Translation is to change or convert to another form

Translate auditory cues to visual medium

Give enough information

Goal: Individual can interpret information that is translated

Auditory System Visual System

Pitch height and Pitch differences • Higher pitches = smaller objects• Lower pitches = larger objects• Pitches = altitude

Volume changes • Bigger objects = more prominent

Tempo and Beats • Indicate BPM and rhythm (repeated pattern)

Auditory Visual

Based on Ilie & Thompson (2006)

Issue between pitch and volume in visual system:

Higher pitch smaller object

Higher amplitude (volume) larger object

Problem: amplitude and frequency are integrated within visual system

Solution:

Auditory System Visual System

Pitch height and Pitch differences • Associate with altitude

Volume changes • Associate with size

Implemented using JOGL 2.0 (Java OpenGL)

Is a wrapper class for C OpenGL calls

Uses GLUEGEN to bind Java to C

X-Axis: Instrument arrangementY-Axis: Pitch scaleZ-Axis: Time progression

Pitch: Pitch height determines height of the pipe

Timbre: Different “sound families” are defined through color

Volume: Depicted through using thickness of pipes

Tempo: Clusters of activity

Rhythm: Visual depiction of beats and interactivity

12 participants 3 hearing, 3 hard of hearing and 6 deaf ASL users

Procedure: 6 one minute-songs, each from different genres in

random order

Genres: classical, country, jazz, pop, rap/hip-hop/R&B (RHRB), rock▪ Commonly used groupings in industry

Pre-study, post-song, post-study questionnaires Eye tracking data recorded with FaceLab 5 Data analysed with repeated measures ANOVA

Feedback on enjoyment and emotional experience Enjoyment:

5 point Likert scale (1-not enjoyable at all to 5-enjoyable)

Emotions:

1. Valence and arousal model (Russell, 1972), overbearingness added

2. Discrete model (happy, sad, anger, fear) (Ekman, 1972)

Level of focus

No statistical difference between genres Descriptive analysis:

All genres are rated between 4-somewhat enjoyable and 5-enjoyable

Rock is most enjoyable (M=4.5, SD=0.85)

RHRB (M=4.0, SD=1.054) and classical (M=4.0, 1.247) are least enjoyable

▪ Initially hypothesized to be the most enjoyable due to bass

Overbearingness: 1-subtle to 9-overbearing:

Between pop (M=6.42, SD=1.084) and RHRB (M=4.92, SD=1.564)

Between pop and country (M=4.25, SD=1.603)

Trend: an overbearing song can still be happy.

Pop is most overbearing (M=6.42, SD=1.084), highest arousal (M=6.08, SD=2.021) and happiest (M=6.75, SD=2.179)

Valence from 1-unhappy to 9-happy: not significant

Arousal: 1-calm to 9-excited: not significant

Four 7-point scales (1-weak to 7-strong) for:

No significant difference between genres

A song may have multiple emotions and people were not able to choose one with limited set of words

1. Happiness2. Sadness

3. Anger4. Fear

5-point likert scale: “1-my mind wondered a lot” to “5-I was always focused on the visualization”

No significant difference between genres Trends:

Paid most attention to rock (M=4.91, SD=0.302), country (M=4.73, SD=1.206) and jazz (M=4.64, SD=0.505)

Paid least attention to pop (M=4.09, SD=1.378), RHRB (M=4.09, SD=1.578)

Major limitation is the lack of participants Additional participants are needed to determine

whether trends observed are actual statistical differences

Additional participants are needed to explore the differences between the hearing statuses

Participants showed preference for rock, pop and classical genres

MusicViz provided enjoyment and information

Funding provided by GRAND, CFI, NSERC Co-workers at the IMDC Study participants

Questions?

Michael Pouris, Deborah FelsRyerson University, Toronto, Ontario, Canada

mpouris@ryerson.ca, dfels@ryerson.ca