Sonification of Complex Data Sets: An Example from Basketball

Frances L. Van Scoy

West Virginia UniversityMorgantown, West Virginia 26506-6845 USA

fvanscoy@wvu.edu

Abstract

We are seeking ways to display via music relationships in complex data sets. As initial data sets we study some simple mathematical functions and scores from basketball games.

GoalsNew multimedia virtual systems use a combination of visualization and sonification to convey information to users. The term visualization has been in widespread usage; the term sonification, meaning "the transformation of numerical data into sound for purposes of observing that data" (NCSA ADG, 1999), or, more simply, "data representation through audio" (ICAD, 1999), is somewhat less widely used.

In describing goals in visualization, Edward R. Tufte (1983) wrote:

Excellence in statistical graphics consists of complex ideas communicated with clarity, precision, and efficiency. Graphical displays should

• show the data

• induce the viewer to think about the substance rather than about methodology, graphic design, the technology of graphic production, or something else

• avoid distorting what the data have to say

• present many numbers in a small space

• make large data sets coherent

• encourage the eye to compare different pieces of data

• reveal the data at several levels of detail, from a broad overview to the fine structure

• serve a reasonably clear purpose: description, exploration, tabulation or decoration

• be closely integrated with the statistical and verbal descriptions of a data set.

We ask how well these requirements can be fulfilled by a good sonification application. Our initial approach investigates approaches to mathematical representations of mathematical functions and data from basketball games.

Previous Work by Others

Others have addressed this issue in various ways.

Crumbs

The NCSA Biological Imaging Group (Brady, 1996) observed that many early attempts at sonification were difficult to listen to for long periods of time, because the tones generated were repetitious and the chords typically dissonant. Their goal was to build a system that would generate sound which would be both melodic and unobtrusive. The Crumbs software they developed rotates through several different melodies, all based on the dorian scale. As the user begins searching with a probe for fibrous tissue in the data set (generated from an MRI scan of a horse's leg), the sound of a guitar playing a melody is generated. As the user gets closer to the goal, other instruments are added in order, a piano in single register, a piano in multiple registers, a voice-like synthesized instrument, and a flute. All five instruments are heard when the user reaches the most dense region of fiber.

The vOICe

Peter Meijer has developed The vOICe, an experimental system for auditory image representation (Meijer). His system coverts a picture into a 64 x 64 pixel digital image in gray scale. The picture is then scanned from left to right, one column at a time. Each row of the picture is interpreted as representing a particular sound frequency. While a particular column is being scanned, the 64 sound frequencies are generated with the volume of each frequency based on the brightness (within 16 levels) of the corresponding pixel. The result is a changing sound pattern with duration of about 1.5 minutes. The goal of his work is to assist blind people in understanding the environment around them (for example, recognizing the position of doors in a large room).

Adam Burford's Masters Project

The work presented in this paper has an intellectual origin in the masters project of Adam Burford (1989). Combining his undergraduate major in music and his graduate work in computer science, Adam generated music from changes in cellular automata over time. In particular, he generated one measure of music from each configuration in an instance of the Game of Life. Figure 1 shows an example of an oscillating sequence of configurations, and Music 1 presents the music generated from that sequence.

Figure 1 Oscillating Sequence of Game of Life Configurations

Music 1 Music Generated from Oscillating Sequence of Game of Life Configurations

Representation of Simple Functions

We initially began looking at ways to represent minute-by-minute scores from basketball games. We mapped each nonnegative integer between 0 and 108 to a note in the chromatic scale. That is, we had two functions f and g each with domain the integers between 0 and 40 and range the integers between 0 and 108. The value of f(t) corresponded to the score of one team when 40-t minutes of playing time in the game remained and g(t) to the score of the other team at that time. For example, Music 2 shows the music generated this way from the first 10 minutes of the basketball game described later in this paper.

Music 2 Music from Basketball Game Scores

This approach has several disadvantages: nine octaves are required (beyond the range of a piano keyboard or other instrument), the two notes representing the current score of each team often are dissonant, and unless a different instrument is chosen for each team it is difficult to determine which team is ahead.

We then began looking at simpler functions.

Chromatic Scale for Simple Functions

Simple functions of a single variable can be represented by selected notes of the chromatic scale. For example the function f(t) = t can be represented by the chromatic scale.

Figure 2 Graph of f(t) = t

Music 3 Musical representation of f(t) = t

A semicircle eliminates some notes of the chromatic scale and repeats others. We show this in the following table in which we compute the value of sqrt(144-t2) for 0 <= t <=12 as we determine how to represent a quarter circle.

x 0 1 2 3 4 5 6 7 8 9 10 11 12

x2 0 1 4 9 16 25 36 49 64 81 100 121 144

144-x2 144 143 140 135 128 119 108 95 80 63 44 23 0

sqrt( 144 - x2) 12.00 11.96 11.83 11.62 11.31 10.91 10.39 9.75 8.94 7.94 6.63 4.80 0.00

rounded 12 12 12 12 11 11 11 10 9 8 7 5 0

note C C C C B B B Bb A Ab G F C

A semicircle, then can be represented by the notes C F G G# A A# B B B C C C C C C C B B B Bb A Ab G F C.

Figure 3 Graph of f(t) = sqrt(k2 -t2)

Music 4 Musical representation of f(t) = sqrt(k2 -t2)

Functions over the Same Domain

Next, let us draw in music a full circle. That is, we will present the two functions f(t) = sqrt(k 2 - t2) and g(t) = - sqrt(k2 - t2).

Figure 4 Graph of f(t) = sqrt(k2 -t2) and g(t) = sqrt(k2 -t2)

We now play two notes simultaneously, one representing the value of f(t) and one representing the value of g(t).

Music 5 Musical representation of graph of f(t) = sqrt(k2 -t2) and g(t) = sqrt(k2 -t2)

This works, to a certain extent, but has the disadvantage of producing sounds which are disharmonic.

Consider the pair of functions f(t) = t and g(t) = -t.

Figure 6 Graph of either f(t) = t and g(t) = -t or f(t) = abs(t) and g(t) = - abs(t)

These two functions can be represented by two chromatic scales, one ascending and one descending. The resulting music has the same problem of that generated by the circle above of being unpleasant to the Western ear.

Music 6 Musical representation of graph of f(t) = t and g(t) = -t

It has another problem, however, in that the music is indistinguishable from that generated by the pair of functions

f(t) = abs(t) and g(t) = - abs(t).

Two possible solutions are choosing easily distinguished instruments for the two functions, or using pitch for one function and duration for the other. For example, we might represent a relatively small value by a whole note and a relatively large value by eight eighth notes. Simplifying the number of pitches for this example, we have two distinguishable melodies for f(t) = t, g(t) = -t and for f(t) = abs(t), g(t) = - abs(t).

Music 7 Musical representation using pitch and duration of f(t)=t and g(t)=-t

Music 8 Musical Representation using Pitch and Duration of f(t) = abs(t )and g(t) = - abs(t)

Permutations as Proposed Solution

Our interest in sonifying basketball scores minute-by-minute was more than showing which team is ahead by how much. We were looking for a way to show which combinations of five players are more effective (in terms of increasing their team's score relative to the other team's).

We tried to represent each of the twelve players by a different musical instrument. We also tried to represent each player by a different note in a chord, and each minute by a measure, in which the number of notes played indicated roughly the difference in score between the two teams. Neither of these generated music which was pleasant to listen to or easy to understand.

To avoid producing dissonant chords we decided to represent the players on the court in a given minute not by a chord but by a permutation of the notes (from a major scale) representing those players. In this way we do not have to be concerned with generating chords or sequences of chords which are pleasing to the ear but rather with generating short melodies which change over time as function values change.

The permutation varies according to any of several patterns. For example, if we present the permutations in lexicographic order (defined by pitch) we would play the series of notes (representing three players):

C D E, C E D, D C E, D E C, E C D, E D C

Any sequence of permutations of these notes can indicate that the team of interest is ahead in points. To indicate that this team is tied or behind, we sharpen or flatten one note to make the sequence of notes be from a minor scale.

This work is inspired by work in comma-free codes (Ridley, 1993; Buchanan, 1992). In constructing a comma-free code, an n element alphabet and a set of code words each of length k over that alphabet are chosen in such a way that if any sufficiently long fragment of a series of code words is selected the code words can be recognized uniquely. For example when n=4 and k=3 there are potentially 43 = 64 possible code words but it can be shown that the largest possible set of code words contains only 20 in order to maintain this "comma-free" property.

However, in our work we are not looking for comma-free sequences which can be recognized logically but rather for melodies which are recognized more subtly and intuitively by a listener.

There are two main kinds of permutations we are currently studying: permutations in lexicographic order and in bell change ringing order. We are also beginning to look at a subset of permutations, those forming a comma-free code.

Permutations in Lexicographic Order

We generate permutations of a set of notes not in strict alphabetical order but in lexicographical order based on pitch.

For example, with base set {C, D, E }, we generate the string of notes CDE CED DCE DEC ECD ECD (with spaces added here for legibility).

Music 9 Musical Representation of the Permutation in Lexicographic Order of C D E

Note that this same perumutation on a different set of notes, such as {C, E, G} or {C, D#, E} sounds somewhat similar but different.

Music 10 Musical Representation of the Permutation in Lexicographic Order of C E G

Music11 Musical Representation of the Permutation in Lexicographic Order of C D# E

We represent a function by beginning with the melody of the permutations of a set of notes in lexicographic order and then as the value of the function changes we change, generally one at a time, the notes in the base set.

Change Ringing

We have also begun to study those permutations that correspond to the notes produced during change ringing of bells. Over simplifying the definition, we begin with a downward scale and then for each successive permutation exchange some pairs of adjacent notes. The result is a fairly complex melody with a recognizable structure to it. We perturb this structure by gradually sharpening or flattening certain notes in the base set. Appendix B shows an example of change ringing.

Permutations Based on Comma-Free Codes

One student who joined the project in summer 1999 has a degree in psychology. We hope to use him to design and perform experiments to determine whether computer users can distinguish among melodies generated by different comma-free codes over the same alphabet.

This will be a change from our current work of conveying information by choice of notes. In theory, we could use comma-free codes to represent 20 different values using only 4 notes.

A Specific Example of Representation of a Basketball Game

We are searching for a scheme for music generation which is robust enough to take data collected at basketball games which show which five players of one team are on the playing court at a given time and how the relative score changes with each combination of players. This problem is interesting because it is simple enough to understand (help the coaching staff recognize which combinations of players are most effective) but because the data is relatively complex (there are 12 players on the team, only 5 play at a given time, and player performance is of course not consistent).

As a case study we initially hand coded simplified data from the first half (20 minutes) of the West Virginia Universitgy versus University of Georgia basketball game of December 2, 1998.

Table 1 shows which five players (by player number) were playing at each minute of the first half of the game as well as the score of each team at the end of that minute.

minute Center Guard Guard Forward Forward WVU Georgia

1 35 25 33 0 32 3 3

2 35 25 33 0 32 3 5

3 35 25 33 0 32 8 7

4 35 25 33 0 32 8 10

5 35 25 33 0 32 10 10

6 35 25 33 0 32 12 10

7 35 34 33 0 32 15 11

8 35 25 33 0 32 17 11

9 35 25 33 0 32 19 17

10 35 25 33 0 32 24 22

11 31 25 13 0 30 26 22

12 31 34 33 0 32 26 23

13 31 25 33 0 32 26 25

14 31 25 33 0 32 26 27

15 31 25 33 0 32 28 27

16 31 25 33 0 32 28 34

17 31 25 33 0 32 30 34

18 31 25 33 0 32 34 36

19 31 25 33 0 32 36 36

20 31 25 33 0 32 39 38

Table 1 First Half Data for WVU versus Georgia Basketball Game of December 2, 1999

We somewhat arbitrarily assigned notes to each of the players as shown in Table 2.

Player 35 0 32 31 25 34 13 33 30

Note middle C G E high C A D B F D above high C

Table 2 Assignment of Notes to Players

The melody generated from this first assignment of notes to players is given in Music 13. In this example the number of notes per measure indicates the relative amount by which our team is ahead.

We chose a permutation of notes based on change ringing:

1 2 3 4 5; 2 1 4 3 5; 2 4 1 5 3; 4 2 5 1 3; 4 5 2 3 1; 5 4 3 2 1; 5 3 4 1 2; 3 5 1 4 2; 3 1 5 2 4; 1 3 2 5 4

We chose this particular permutation in large part because it repeats every ten steps, or four times in a 40 minute basketball game.

In the first measure we play the notes for the starting players in order of center, two guards, and two forwards. In the second measure we permute the position of the notes for the center and the first guard and the notes for the second

guard and first forward. As one player is substituted for another we change the pitch associated with that playing role but continue with the permutation. For example, in minute 7, player 34 replaces player 25. By our permutation sequence, measure 7 of the music should have contained the notes in order E F G C A but instead uses D for A, giving E F G C D. Appendix C gives the music generated by this scheme.

We used Finale Allegro to transpose measures in which our team was behind into a minor key, trying several keys until we found one that for this combination of players/notes sounded "sad." The resulting music does indicate even to first time listeners that something is different in that minute/measure, but by choosing the minor key arbitrarily we lost the consistent mapping of a player to a note.

In another attempt, rather than transposing losing measures to a minor key we lowered the pitch by an octave. The result is given in Appendix D.

We have generated various other melodies based on this same basketball game. Other variations include changing how notes are associated with players, using one instrument when our team is ahead and a different instrument when our team is behind, and using percussion instrument accompaniment to indicate relative score.

While we would prefer to have losing measures in a minor key, we have not yet developed an algorithm which regardless of the combination of players always produces a measure in a minor key while retaining the identification of players and notes.

Status and Goal

We have now written a C++ application which takes as input our rules for generating music and the data for a particular basketball game and generates a MIDI file for the resulting music which is then played on a Macintosh G3 computer using the Finale Allegro 98 package. We are now constructing by hand data files from hand written notes taken at WVU basketball games during the 1997-8 and 1998-9 seasons. We anticipate that the music generated from these two seasons will be noticably different. During 1997-8 the team had five strong seniors who played almost the entire game, with rest breaks provided by two other seniors, and the team was quite successful, going to the Sweet Sixteen. In 1998-9 the team had few experienced players, and a core of five to seven strong players never emerged. This team lost more games than it won. We anticipate that more frequent substitutions and playing time by more different players during the 1998-9 season will lead to obviously different sounding music than that generated from games of the 1997-8 season. (To test this, we will have to disable indicating relative score and which team is leading, or compare only games in which WVU won.)

So far our evaluation of different schemes for generating music has been quite informal, testing melodies on each other, on others who work in our lab, and on visitors to the lab. One team member has an undergraduate degree in psychology, and during summer 1999 he will develop and perform more formal experiments with naive users.

More importantly than improving our method of representing basketball games, we intend to apply the lessons learned from working with basketball to analysis of three-dimensional data sets. In particular we want to provide a viewer of a 2-d projection of 3-d data with clues about how to change viewing perspective to see interesting properties of the data. Figure 7 shows a simple example in which we project into the x-y plane two "clouds" of points, each cloud parallel to the x-y plane.

Figure 7 Projection of 3-Data into the x-y Plane

This projection could represent data arranged in many ways in Cartesian space, including those shown in Figure 8, in which the two data clouds are either disjoint or intersecting.

Figure 8 3-D Views of Data, Showing Disjoint and Intersecting Red anGreen Clouds

We plan to generate music that will give auditory clues about how far apart these clouds are along the z axis by generating music in which the relative distance of a point in a particular cloud from the x-y plane is mapped to pitch. This will require a more sophisticated scheme for generating music than is currently being used with the basketball data

As in Meijer's The vOICe, we will scan the 2-d projection from left to right, generating music (as distinguished from sound) for each column. We will provide a bar which shows which column in the image is producing the current music. The user may click on the mouse button to mark that particular column and may later zoom in to the data using two of these bars to bracket the region of interest. The user may also choose to scan the projection from top to bottom. We hope to generate music that will guide the user in choosing viewing perspectives of the data.

Bibliography

Brady, Rachel, Robin Bargar, Insook Choi, and Joseph Reitzer, "Auditory Bread Crumbs for Navigating Volumetric Data," presented at IEEE Visualization '96, San Francisco, October 1996, available on-line at

http://mayflower.ncsa.uiuc.edu/sonification.html

Buchanan, Hollie L. and Michael E. Mays, "Retrograde codes and bounded synchronization delay," Information and Computation 96 (1992) 55-64

Burford, Adam, Cellular Automaton Systems/Self-Reproducing Turing Machine, masters problem report, MS CS, West Virginia University, Department of Statistics and Computer Science,1989.

ICAD (International Community for Auditory Display), "Joint ICAD/ASA/EAA Workshop on Auditory Display," http://www.santafe.edu/~icad/websiteV2.0/Conferences/Berlin99/cfp.html

Meijer, Peter, "The vOICE - Seeing with Sound," http://ourworld.compuserve.com/homepages/Peter_Meijer/voice.htm

NCSA Audio Development Group, "Sonification," http://cage.ncsa.uiuc.edu/adg/Sonification/

Ridley, J. M. and Michael E. Mays, "Generalized Witt formulae and comma-free codes," Journal of Pure and Applied Algebra 88 (1993)177-185

Tufte, Edward R., The Visual Display of Quantitative Information, Cheshire, Connecticut: Graphics Press, 1983.

Acknowledgements

This work is supported by the EPSCoR programs of the National Science Foundation and the State of West Virginia and by the Department of Computer Science and Electrical Engineering of West Virginia University.

Undergraduate students contributing to this project are Eric Martino, Radhika Mukamala, and Young Oh of West Virginia University and James Keefover of Bluefield State College.

Macintosh and G3 are registered trademarks of Apple Computer, Inc.

Finale Allegro 98 is a registered trademark of Coda Music Technology.

Appendix A Basketball

Some relevant facts of basketball for this paper:

(1)there are two teams

(2) each team typically has twelve members, but only five from each team are on the court at a given time

(3) at the university level, a game consists of two periods of twenty minutes each

At the end of the season the National Collegiate Athletic Association holds a single elimination tournament in which the ostensibly best 64 college teams in the country play. Those sixteen teams remaining in the tournament after two rounds of play are informally called the "Sweet Sixteen."

Appendix B Musical Representation of a Permutation of the C Major Scale based on Change Ringing

Appendix C Music Based on Basketball Game, Losing Minutes in Arbitrary Minor Key

Appendix D Music Based on Basketball Game, Losing Minutes Lower Octave