discrimination of musical instrument sounds resynthesized with simplified spectrotemporal parameters

Redistr

Discrimination of musical instrument sounds resynthesizedwith simplified spectrotemporal parametersa)

Stephen McAdamsb)

Laboratoire de Psychologie Expe´rimentale (CNRS), Universite´ ReneDescartes, EPHE, 28 rue Serpente,F-75006 Paris, France and Institut de Recherche et de Coordination Acoustique/Musique(IRCAM/CNRS), I place Igor-Stravinsky, F-75004 Paris, France

James W. Beauchampb)

School of Music and Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, 2136 Music Building, 1114 West Nevada Street, Urbana, Illinois 61801

Suzanna MeneguzziLaboratoire de Psychologie Expe´rimentale (CNRS), Universite´ ReneDescartes, EPHE, 28 rue Serpente,F-75006 Paris, France and IRCAM, I place Igor-Stravinsky, F-75004 Paris, France

~Received 17 November 1997; revised 21 September 1998; accepted 23 September 1998!

The perceptual salience of several outstanding features of quasiharmonic, time-variant spectra wasinvestigated in musical instrument sounds. Spectral analyses of sounds from seven musicalinstruments~clarinet, flute, oboe, trumpet, violin, harpsichord, and marimba! produced time-varyingharmonic amplitude and frequency data. Six basic data simplifications and five combinations ofthem were applied to the reference tones: amplitude-variation smoothing, coherent variation ofamplitudes over time, spectral-envelope smoothing, forced harmonic-frequency variation,frequency-variation smoothing, and harmonic-frequency flattening. Listeners were asked todiscriminate sounds resynthesized with simplified data from reference sounds resynthesized with thefull data. Averaged over the seven instruments, the discrimination was very good for spectralenvelope smoothing and amplitude envelope coherence, but was moderate to poor in decreasingorder for forced harmonic frequency variation, frequency variation smoothing, frequency flattening,and amplitude variation smoothing. Discrimination of combinations of simplifications wasequivalent to that of the most potent constituent simplification. Objective measurements were madeon the spectral data for harmonic amplitude, harmonic frequency, and spectral centroid changesresulting from simplifications. These measures were found to correlate well with discriminationresults, indicating that listeners have access to a relatively fine-grained sensory representation ofmusical instrument sounds. ©1999 Acoustical Society of America.@S0001-4966~99!00202-7#

PACS numbers: 43.66.Jh, 43.75.Wx@WJS#

minfrhanterel0;-grriath

cealso

eral,

ta-

n topa-le,r-al-eten-

co

INTRODUCTION

It has been traditional to view musical sounds in terof a spectral model that describes them as a series of ssoidal components, each having an amplitude and aquency. Often, as is the case in this article, these soundsfrequencies which are harmonically related to a fundamefrequency, or at least approximately so. While many expments on timbre have used fixed frequencies and fixed rtive amplitudes~Miller and Carterette, 1975; Plomp, 197Preis, 1984; von Bismarck, 1974!, analyses of musical instrument sounds reveal that these parameters have adeal of variation, leading to the conjecture that these vations are responsible, in large part, for the uniqueness ofindividual sounds.

For example, we can think of the amplitudes~A! andfrequencies~f ! varying over time~t! and having two parts, a

a!Portions of these results were presented at the 133rd meeting of the Atical Society of America~Beauchampet al., 1997!.

b!Address correspondence to either S. McAdams at IRCAM~Electronicmail: [email protected]! or to J. Beauchamp at UIUC~Electronic mail:[email protected]!.

882 J. Acoust. Soc. Am. 105 (2), Pt. 1, February 1999 0001-4966/99

ibution subject to ASA license or copyright; see http://acousticalsociety.org/

su-e-ve

ali-a-

eat-e

smoothly or slowly moving part~1! and a more rapidlychanging microvariation part~2!:

Ak~ t !5A1k~ t !1A2k~ t !, ~1!

f k~ t !5 f 1k~ t !1 f 2k~ t !, ~2!

wherek refers to the harmonic number. Alternatively, sinwe consider only quasiharmonic sounds here, we canbreak the frequency into two other parts:

f k~ t !5kf 0~ t !1D f i k~ t !, ~3!

wheref 0 is the fundamental frequency averaged over sevharmonics andD f i k is an inharmonic frequency deviationboth varying over time.

Figure 1 gives a block diagram of a spectral represention model using the parameters of Eqs.~1! and~2!, which isalso an additive, sine-wave-synthesis model. The questiobe explored in this article is: to what degree can theserameters be simplified, without making them discriminabwith respect to sounds containing the full amount of infomation? A given sound can be reconstituted with high quity from the full representation using time-varying additivsynthesis. However, such a representation is quite data in

us-

882/105(2)/882/16/$15.00 © 1999 Acoustical Society of America

content/terms. Download to IP: 128.123.35.41 On: Mon, 08 Sep 2014 21:37:54

teeist

ewo

vitnnaui

nd

leio-eo

asus-li-,ox

c

ternt ier

es

li-ngheheof

overandnot

nds.

ces

ns

ge

ri-thatalogkedthe

n-ng

y

re,dim-ve-lopepeendce

av-tal,nice

toate

-, asct

otiondatarlyralpri-ner-orsy to

e

Redistr

sive. Any possibility of reducing the data would alleviastorage problems and accelerate the process of synthwhich is particularly important for real-time sound synthesAlso, one might hope that such simplifications would leadthe possibility of streamlined synthesis control using a fwell-chosen, perceptually relevant parameters. Most imptant for us, however, is that knowledge about the sensitiof human listeners to certain kinds of sound simplificatiomay provide clues for understanding the sensory represetion of musical sounds. Specifically, this study is aimeddetermining the relative perceptual importance of variospectrotemporal features which we have suspected areportant for making timbral distinctions and for judging souquality.

A few researchers have already addressed the probof perceptually relevant data reduction using discriminatparadigms. Grey and Moorer~1977! used a rectangularwindow, heterodyne-filter analysis algorithm and timvarying additive synthesis to prepare their stimuli based16 sounds from various bowed-string, woodwind, and brinstruments of duration 0.28 to 0.40 s. They asked their sjects~musical listeners! to discriminate between five versionof the sounds:~1! the digitized original analog tape recording, ~2! a complete synthesis using all time-varying amptude and frequency data resulting from the analysis stage~3!a synthesis using a small number of line-segment apprmations to the amplitude and frequency envelopes,~4! thesame modification as version~3! with removal of low-amplitude initial portions of attack transients, and~5! thesame modification as~3! with frequencies fixed in harmonirelation to the fundamental frequency~frequency-envelopeflattening!. Listeners heard four tones in two pairs and haddetermine which pair contained a different tone. They wallowed to respond ‘‘no difference heard.’’ Discriminatioscores were computed as the probability that the correcterval was chosen plus half the probability that a no diffence response was given~ostensibly to simulate randomguessing on those trials!.

An important result was the low discrimination scorfor comparisons of versions~2! and ~3!, which ranged from0.48 to 0.80~depending on the instrument!, with an average

FIG. 1. Spectral-representation model using smooth and microvariationvelopes for amplitude and frequency. Each harmonick is summed with theothers to form the total output by additive synthesis.

883 J. Acoust. Soc. Am., Vol. 105, No. 2, Pt. 1, February 1999 Mc


sis,.o

r-ysta-tsm-

mn

-ns

b-

i-

oe

n--

of only 0.63. This indicated that microvariations in amptude and frequency are usually of little importance, implyithe possibility for significant data reduction. However, tauthors gave no algorithm for fitting the straight lines to tdata or criteria for error, but stated only that the numbersegments varied between four and eight per parametereach tone’s duration. Also, since the tones were shortsome segments were needed to fit attack transients, it isclear how these results can be extrapolated for longer souDiscrimination rates between versions~3! and ~4! and be-tween~3! and~5! were similarly low, averaging 0.65~range:0.55 to 0.74! and 0.68~range: 0.56 to 0.92!, respectively.The results indicated that there were significant differenamong the 16 instruments.

In general, discrimination rates for single simplificatiowere low, and relatively high rates~above 0.85! only oc-curred for multiple simplifications. For example, the averadiscrimination rate between versions~1! and~5!, where threesimplifications were combined, was 0.86. From our expeence, these figures seem low. We can only conjecturethis was due to the short tones used, to noise on the antape used for stimulus presentation which may have massome parameter variation details, and perhaps even toexperimental instructions which specifically oriented listeers toward differences in quality of articulation and playistyle rather than toward any audible difference.

Charbonneau~1981! extended Grey and Moorer’s stud@based on their version~3! representation# by constructinginstrumental sounds that maintained their global structuwhile simplifying the microstructure of the amplitude anfrequency envelopes of each harmonic partial. The first splification was applied to the components’ amplitude enlopes, each component having the same amplitude enve~calculated as the average harmonic-amplitude envelo!scaled to preserve its original peak value and start- andtimes. ~This is similar to our amplitude-envelope coherensimplification; see Sec. I below.! The second simplificationwas similarly applied to the frequency envelopes, each hing the same relative frequency variation as the fundamenmeaning that the sound remained perfectly harmothroughout its duration~similar to our frequency-envelopcoherence simplification; see Sec. I below!. The third simpli-fication resulted from fitting the start- and end-time datafourth-order polynomials. Listeners were asked to evaluthe timbral differences between original@version ~3!# andsimplified sounds on a scale from 0~no difference! to 5~large difference!. Results indicated that the amplitudeenvelope simplification had the greatest effect. Howeverfor the Grey and Moorer study, the strength of the effedepended on the instrument.

Sandell and Martens~1995! used a different approach tdata reduction. The harmonic time-frequency representaderived from a phase-vocoder analysis was treated as amatrix that could be decomposed into a number of linearecombinable principal components from either a tempoor a spectral perspective. The recombination of the approately weighted principal components can be used to regeate the signal of a given instrument sound. These authestimated the number of principal components necessar

n-

883Adams et al.: Discrimination of spectrotemporal simplifications


i

ulitheteOor

cn-eran

sfroa

ere

ondethnxisri

rantere

of

ofe

t,m

a-, on

inane

dat

sodte-e-th

fre-the

tely

thatns.ach

fied

-so-

was

rd,r toin-nic.

ty

u-igh

had

d one-e

s of

Redistr

achieve a simplified sound that was not reliably discrimnated from a sound reconstructed from the full~thoughdown-sampled! analysis data. From these results, they cothen compute the proportion of data reduction possible wout compromising perceived sound quality. They achievconsiderable data reduction for the three instruments tesbut the amount varied a great deal across instruments.interpretation problem that often plagues perceptuallyented principal components analyses on acoustic data~seealso, Repp, 1987! is that the perceptual nature and relevanof the individual components is most often difficult to coceive. For example, it is not clear that they could represperceptual dimensions with clearly defined acoustic chateristics along which stimuli could be varied intuitively isound synthesis.

This reservation notwithstanding, the results of thethree studies demonstrate that timbre changes resultsimplification of the signal representation. In fact, it is clefrom the two earlier studies that the simplifications pformed on temporal parameters, and specifically on timvarying functions of amplitude and frequency, influence tgreater or lesser degree the discrimination of musical sou

In the present study, we sought to determine precisthe extent to which simplified spectral parameters affectperception of synthesized instrumental sounds, using toof 2-s duration and without the use of straight-line appromations. We measured the discrimination of several kindsimplifications for sounds produced by instruments of vaous families of resonators~air column, string, bar! and typesof excitation~bowed, blown, struck!. Two of the simplifica-tions we chose~amplitude-envelope coherence and spectenvelope smoothness! were derived from previous studies otimbre perception and corresponded to acoustic paramethat are highly correlated with perceptual dimensionsvealed by multidimensional scaling techniques~Grey andGordon, 1978; Iverson and Krumhansl, 1993; Krimphet al., 1994; McAdamset al., 1995!. The other four simpli-fications related to the perceptibility of microvariationsamplitude and frequency over time, with much having bewritten about the latter~Brown, 1996; Dubnov and Rode1997; McAdams, 1984; Sandell and Martens, 1995; Schucher, 1992!.

In addition, various combinations of these simplifictions were applied to the sounds in groups of two, threefour. We hypothesized that accumulated simplification aloseveral perceptual dimensions would increase discrimtion. Below, we present the technique used for analyzingsynthesizing the stimuli, followed by a description of thdiscrimination experiment. The results are then discusseterms of musical synthesis and the perceptual representof musical signals.

I. ANALYSIS/SYNTHESIS METHOD

Seven prototype musical instrument sounds werelected for analysis using a computer-based phase-vocmethod~Beauchamp, 1993!. This phase vocoder is differenthan most in that it allows tuning of the fixed analysis frquency (f a) to coincide with the estimated fundamental frquency of the input signal. The analysis method yields



-

d-dd,nei-

e

ntc-

emr--

as.

lyees-of-

l-

rs-

f

n

a-

rga-d

inion

e-er

e

frequency deviations between harmonics of this analysisquency and the corresponding frequency components ofinput signal, which are assumed to be at least approximaharmonic relative to its fundamental.

A. Signal representation

For each sound, an analysis frequency was chosenminimized the average of the harmonic frequency deviatioThus, a time-varying representation was achieved for esound according to the formula

s~ t !5 (k51

K

Ak~ t !cosS 2pE0

t

~k fa1D f k~ t !dt!1uk~0! D ,

~4!

where

s~ t !5sound signal,t5time in s,k5harmonic number,K5number of harmonics,Ak(t) is the amplitude of thekth harmonic at timet,f a5analysis frequency,D f k(t) is the kth harmonic’s frequency deviation,such thatk fa1D f k(t) is the exact frequency of thekthharmonic, anduk(0) is the initial phase of thekth harmonic.

The parameters used for synthesis that were simpliin this study areAk(t) andD f k(t). No attempt was made tosimplify uk(0). AlthoughAk(t) andD f k(t) were only storedas samples occurring every 1/(2f a) s, the signal was approximated with reasonable accuracy at a much higher relution ~sample frequency 22 050 or 44 100 Hz! by using lin-ear interpolation between these values. Synthesisaccomplished by additive~or Fourier! synthesis of the har-monic sine waves.

B. Prototype sounds

Sounds of the instruments clarinet, flute, harpsichomarimba, oboe, trumpet, and violin were selected in ordehave one representative from each of several families ofstruments whose tones are at least approximately harmoFive of the sounds were taken from the McGill UniversiMaster Samples recordings, one from Prosonus~oboe!, andone~trumpet! had been recorded at the UIUC School of Msic. An attempt was made to select sounds that were of hquality, that represented the instruments well, and thatfundamental frequencies close to 311.1 Hz~E-flat 4!, a notewithin the normal playing range of these instruments.1 Sincesynthesis was accomplished by an additive method baseEq. ~1!, it was easy to alter the stimuli’s fundamental frquencies (f a) to be exactly 311.1 Hz. Table I gives sombasic characteristics of the prototype sound signals.

C. Analysis method

The phase vocoder method used for analysis consistthe following steps:



mbersAttack

lsivelyon, was

885 J. Acoust. S

Redistribution subject to ASA

TABLE I. Data for the seven instrument sounds used in the study. For McGill source recordings, the nuindicate volume:track-index. For Prosonus recordings, they indicate woodwinds volume:band-index.(t1) is the time in the original sound at which the attack was estimated to end. Decay (t2) is the time in theoriginal sound at which the decay was estimated to begin. The marimba and harpsichord, being impuexcited instruments, have no sustain portions. The marimba, being shorter than the target 2-s duratistretched rather than shortened, and so the attack and decay values were not used.

Source oforiginal

recording

Originalfundamentalfrequency

~Hz!

Originalduration

ofsound,tL ~s!

Number ofharmonics

used inanalysis,K

Attack,t1 ~s!

Decay,t2 ~s!

Clarinet ~Cl! McGill ~2:10-14! 311.4 3.81 70 0.05 3.50Flute ~Fl! McGill ~9:86-04! 311.0 2.31 70 0.25 2.10Harpsichord~Hc! McGill ~11:95-06! 311.1 2.97 70 0.04 2.97Marimba ~Mb! McGill ~3:04-23! 312.2 1.83 70 ¯ ¯

Oboe~Ob! Prosonus~W1:04-04! 311.8 2.98 30 0.15 2.20Trumpet~Tp! UIUC 350.0 2.43 31 0.32 1.30Violin ~Vn! McGill ~9:63-03! 311.1 4.94 66 0.22 4.10

ceiom

eiod

-ds

s

ge

gia

arlyac

aisis

a

ig

enaaivth

coththsu

sys-re-

sisitiver inura-lingerveor--

the

ing

ure,in-

s,

de

~1! Band-limited interpolation of the input signal to produa power-of-two number of samples per analysis per(1/f a), which is the lowest possible to exceed the nuber of original samples in this time interval.

~2! Segmentation of the input signal into contiguous framwhose lengths are equal to twice the analysis per(2/f a) and which overlap by half an analysis perio( f a/2).

~3! Multiplication of each signal frame by a Hamming window function whose length is two analysis perio(2/f a).

~4! Fast Fourier transform~FFT! of the resulting product toproduce real and imaginary components at frequencief a/2, f a , 3f a/2,..., f s/22 f a , where f s is the samplingfrequency. Components which are not positive intemultiples of f a are discarded.

~5! Right-triangle solution of each retained real and imanary part to give the amplitude and phase of each hmonic.

~6! Computation of the frequency deviation for each hmonic by a trigonometric identity which essentialgives the difference in phase between frames for eharmonic.

~7! Storage of the harmonic-and frequency-deviation datan ‘‘analysis file.’’ The number of harmonics storedless thanf s /(2 f a). The analysis file for each soundthe basis for further sound processing.

Further details of this procedure are discussed by Bechamp~1993!.

The analysis system may be viewed as a set of contous bandpass filters which have identical bandwidths (f a)and are centered at the harmonics of the analysis frequ( f a). The basic assumption is that the signal consists of hmonic sine waves which line up with the filters such theach filter outputs one of the sine waves. The analysis gthe amplitude and frequency of each sine wave. Whensine waves are summed, the signal is almost perfectly restructed. In fact, the sine-wave sum can be viewed ascreated by processing the input signal by the sum ofbandpass-filter characteristics. It can be shown that this

oc. Am., Vol. 105, No. 2, Pt. 1, February 1999 Mc

license or copyright; see http://acousticalsociety.org/

d-

sd

0,

r

-r-

-

h

in

u-

u-

cyr-tesen-atem

is flat within 61 dB over the range@ f a/2,f s/2#. Figure 2shows a block diagram of the basic analysis/ synthesistem and Fig. 3 shows a typical set of amplitude and fquency data.

D. Types of simplification

Spectral simplifications were performed on the analydata, after which the sounds were synthesized by the addmethod. In order that sound duration would not be a factothe study, most of the sounds were shortened to a 2-s dtion by resampling the analysis data. Rather than resampat a uniform rate, the sounds were resampled to prestheir attack and decay portions and shorten their interior ptions while retaining their microstructural variations in amplitude and frequency. This was done by first observingsound’s rms amplitude given by

Arms~ t !5A(k51

K

Ak2~ t !, ~5!

and then identifying by eye the time intervals correspondto the attack and decay as (0,t1) and (t2 ,tL) ~see Table I forchosen values oft1 and t2!, wheretL is the original soundduration. The marimba was an exception to this procedsince its original duration was 1.83 s. The data for thisstrument were simply stretched to obtain a duration of 2

FIG. 2. Method for time-varying spectral analysis that yields the amplituand frequency deviations for each harmonick. The exact frequency forharmonick is given by f k5k fa1D f k(t), where f a is the analysis funda-mental frequency.



es;

Redistr

FIG. 3. Example spectral-analysis data for original violin tone~left column: first harmonic; right column: fourth harmonic; upper row: amplitude enveloplower row: frequency envelopes!. Note the difficulty in reliably estimating the frequency of harmonic 4 when its amplitude approaches zero. Attack (t1) anddecay (t2) boundaries are indicated.

th

d

on

e

alew

to

on4

tio

ndneat

esb

thd

sth

ren

for

a-asious

g aof

li-or-

deuldi-

ef-acktherre-ckera-

ve-e-d to

and no notable degradation of the musical quality oforiginal was noted by the authors.

Second, for each harmonic amplitude and frequencyviation, the time intervals (t1 ,t112) and (t222,t2) werecross-faded using a cubic function to minimize any disctinuities. Thus, between the timest1 and t2 , the sound wastransformed from what it sounded like in the region of timt1 to what it sounded like in the region of timet2 over aperiod of 22(t11tL2t2) s. This gave each sound a totduration of 2 s. In order for this method to work properly, wassumed that each sound had a microstructure whichstatistically uniform over the interval (t1 ,t2). Since thesounds selected had no vibrato, this assumption seemedvalid, and the resulting synthesized sounds were judgedthe authors to be free of artifacts. Details on the duratishortening algorithm are given in Appendix A. Figureshows a set of data corresponding to Fig. 3 after applicaof the duration-shortening algorithm. Note thatt1 and t2 areindicated in Fig. 3.

Finally, the seven duration-equalized prototype souwere compared, and amplitude multipliers were determisuch that the sounds were judged by the authors to hequal loudness. When the sounds were synthesized withshortened duration, the amplitude multipliers, and a synthfundamental frequency of 311.1 Hz, they were judged toequal in loudness, pitch, and duration.~It should be men-tioned, however, that this equalization was not central forpresent study, since each discrimination pair was alwaysrived from a single prototype sound.! The equalized soundthen served as the reference sounds for this study, andcorresponding data sets are henceforth referred to asanalysis data.

Six primary simplifications of the analysis data weperformed prior to synthesis. Each of these simplificatio



e

e-

-

as

beby-

n

sdveheise

ee-

eirthe

s

constitutes a major reduction in the amount of data usedsynthesis.

1. Amplitude-envelope smoothness (AS)

The objective of this operation was to remove microvriations or noise in the harmonic amplitude over time,these had been shown to be perceptually salient in prevwork by Charbonneau~1981!. These envelopesAk(t) wereprocessed by a second-order recursive digital filter havinButterworth response and a smoothing cutoff frequency10 Hz. This essentially removed all microdetail in the amptude envelopes. However, we did not smooth the attack ptions of the envelopes (0<t<t1) since we only wished todetermine the importance of microdetail in the amplituenvelopes thereafter. Smoothing the attack portions wohave slowed the attacks, unintentionally affecting discrimnation of the simplified sounds from their corresponding rerence sounds. In order to avoid discontinuity, the attportion of each amplitude envelope was cross-faded intosubsequent smoothed portion over a few frame points cosponding to the delay of the filter. In this way, the attaportions were essentially unaltered by the smoothing option ~see Table I fort1 values!.

2. Amplitude-envelope coherence (AC) (spectralenvelope fixing)

The objective was to test the effect of eliminatingspec-tral flux ~defined as the change in shape of a spectral enlope over time! without changing the rms amplitude envlope or the average spectrum. Spectral flux has been founbe an important perceptual dimension of timbre~Grey, 1977;Krumhansl, 1989; Krimphoffet al., 1994!. To eliminatespectral flux, the amplitude envelopeAk(t) for each har-



:

Redistr

FIG. 4. Example spectral-analysis data for violin tone with duration reduced to 2 s~left column: first harmonic; right column: fourth harmonic; upper rowamplitude envelopes; lower row: frequency envelopes!.

ahas

erTh

aap

nell

aln

hism

l’ser,

-

p-rd-one

ns-

umingher-havence

cyi-

-hingat-re-pe

ng

o-urs

monic k was replaced by a function which was proportionto the rms envelope and the average amplitude of themonic. Thus, the harmonic-amplitude ratio@A2(t)/A1(t), etc.# were fixed during the course of thsound. In addition, the amplitudes were scaled in ordepreserve the rms envelope under this transformation.formula for this transformation is:

Ak~ t !←AkArms~ t !

A(k51K Ak

2, ~6!

whereAk signifies the time average of thekth harmonic am-plitude over the sound’s duration and← signifies the re-placement operation. Note that with this transformation,amplitude envelopes of all harmonics have the same shalbeit with different scale factors.

3. Spectral envelope smoothness (SS)

The question to be answered here is whether jaggedor irregularity in the shape of a spectrum is perceptuaimportant. For example, the clarinet has a characteristic‘‘jagged’’ up-and-down spectral envelope due to weak eergy in the low-order, even harmonics. A smoothing of tspectral envelope would give it more of a low-pass forSpectral-envelope smoothness was found by Krimphoffet al.~1994! to correspond to the third dimension of Krumhans~1989! 3D space. To test this, the time-varying spectra wsmoothed with respect to frequency. To accomplish thiseach time frame each harmonic amplitude was replacedthe average of itself and its two neighbors~except for end-point harmonics number 1 andK, where averages of themselves and their neighbors were used!

A1~ t !←A1~ t !1A2~ t !

2, ~7a!



lr-

toe

lle,

ssyly-

.

eatby

Ak~ t !←Ak21~ t !1Ak~ t !1Ak11~ t !

3, k¹$1,K%, ~7b!

AK~ t !←AK21~ t !1AK~ t !

2. ~7c!

This smoothing algorithm is not unique and may not be otimal, but it is perhaps the simplest one can imagine. Accoing to this algorithm, the smoothest possible spectrum isthat follows a straight-line curve~i.e., Ak5a1b•k!, sincesuch a spectral envelope would not be altered by this traformation.

Figure 5 compares the time-varying amplitude spectrof a reference sound with those obtained after increasamplitude-envelope smoothness, amplitude-envelope coence, and spectral-envelope smoothness algorithmsbeen applied. The effect of these operations on the referetime-varying spectrum is readily apparent.

4. Frequency envelope smoothness (FS)

We wished to test the auditory importance of frequenmicrovariations in a parallel fashion to that of amplitude mcrovariations. Therefore, the envelopesD f k(t) were pro-cessed similarly to theAk(t) envelopes in amplitudeenvelope smoothing described above, except that smootwas done over the entire sound’s duration, including thetack phase. This operation did not grossly affect the fquency variation during the attack, as amplitude-envelosmoothing would have affected amplitude variation durithat period had it included the attack.

5. Frequency envelope coherence (FC) (harmonicfrequency tracking)

Here, we wanted to test the discriminability of inharmnicity among a sound’s partials, even if it sometimes occ



Redistr

FIG. 5. Simplifications of amplitude envelopes for harmonics 1 to 8:~a! full violin-tone analysis data~reference sound! ~b! after amplitude-envelopesmoothing,~c! after rms envelope substitution~amplitude-envelope coherence!, ~d! after spectral-envelope smoothing.

peti

aglopniei

theurtebesipe

i-inarf

q

nnd

ncyfiedco-

untin

rted

ls,n anedo 3lay-hery-

mn

only momentarily. Analogously to the amplitude-envelocoherence case, all frequency envelopes over time aretogether in a perfect harmonic relation. First, an avertemporal-frequency contour was computed on the envefor the first five harmonics, and then the individual harmocontours were set equal to this contour multiplied by threspective harmonic numbers.

f K~ t !←kf 0~ t !, ~8!

where f 0(t) is defined by

f 0~ t !5(k51

5 Ak~ t ! ~1/k! f k~ t !

(k515 Ak~ t !

. ~9!

With this method, the strongest harmonics amongfirst five receive the ‘‘highest votes’’ for determining thaverage fundamental frequency of the sound. The measfrequency of the first harmonic could have been used insof f 0. However, it is possible that the first harmonic mayweak in amplitude, which with phase-vocoder analywould result in a poorly defined frequency envelo~Moorer, 1978!. This method obviates that problem.

6. Frequency envelope flatness (FF)

This simplification tested listeners’ abilities to discrimnate the combination of no frequency variations and noharmonicity, as after this operation is performed, neitherpresent in the synthesized sounds. Indeed, there is noquency envelope, as each harmonic’s frequency is set eto the product of its harmonic number~k! and the fixed



edee

cr

e

edad

s

-ere-ual

analysis frequency (f a). This operation had previously beefound to have an effect on discrimination by Grey aMoorer ~1977! and Charbonneau~1981!.

Figure 6 shows a reference set of harmonic-frequeenvelopes in comparison to those which have been simpliby frequency-envelope smoothing, frequency-envelopeherence, and frequency-envelope flattening.

Each simplification is accompanied by a certain amoof data reduction. Formulas for data reduction are givenAppendix B.

II. EXPERIMENTAL METHOD

A. Subjects

The 20 subjects were aged 19 to 35 years and repono hearing problems. They included ten musicians~sixmales, four females! and ten nonmusicians~four males, sixfemales!. Musicians were defined as being professionasemiprofessionals, or having at least 6 years of practice oinstrument and playing it daily. Nonmusicians were definas having practiced an instrument for not more than 2 tyears in their childhood or adolescence, and no longer ping. The subjects were paid for their participation with texception of three who were members of the auditoperception team atIRCAM.

B. Stimuli

The seven instruments chosen belong to the air colu~air reed, single reed, lip reed, double reed!, string ~bowed,



Redistr

FIG. 6. Simplifications of frequency envelopes for harmonics 1 to 4:~a! full violin-tone analysis data~reference sound!, ~b! after frequency-envelopesmoothing,~c! after average frequency-envelope substitution~frequency-envelope coherence!, ~d! after replacement by fixed harmonics~frequency-envelopeflattening!.

-ndadreanon

e

eitelee

thes

nnte

ron

e ofr-ed.onpair

erednhehe

ndsing

ted

rfor-alscor-

ofcts,on

reewastru-

-nd-terXTrd00

plucked!, and bar~struck! families: clarinet, flute, harpsichord, marimba, oboe, trumpet, and violin. Each was alyzed and synthesized with the reference sound-analysis~before modification!. In no case could the original recordesound be discriminated from the full synthesis when psented in an AA–AB discrimination paradigm at better th64%.2 The sounds were stored in 16-bit integer formathard disk. All ‘‘reference’’ sounds~full synthesis! wereequalized for fundamental frequency~311.1 Hz or E-flat 4!and for duration~2 s! ~see Sec. I D for a description of thtechnique for equalizing duration in synthesis!. They werealso equalized for loudness in an adjustment procedurethe authors. The different kinds of simplifications and thcombinations that were applied to the stimuli are illustragraphically in Fig. 7. Six simplifications concerned a singparameter, three concerned two parameters, and oneconcerned three and four parameters.3 The 11 simplifiedsounds for each instrument were synthesized withmethod described above on a NeXT computer. They wequalized for loudness within each instrument in an adjument procedure by the authors.

C. Procedure

A two-alternative forced-choice~2AFC! discriminationparadigm was used. The listener heard two pairs of sou~AA–AB ! and had to decide if the first or second pair cotained two different sounds. The dependent variable wasd8 measure of sensitivity to the physical difference derivfrom signal-detection theory using a 2AFC model~Greenand Swets, 1974; Macmillan and Creelman, 1991!. The trialstructure could be one of AA–AB, AB–AA, BB–BA, oBA–BB, where A represents the reference sound and B



a-ta

-

byrd

ach

eret-

ds-hed

e

of the 11 simplifications. This paradigm has the advantagpresenting to the listener both a ‘‘same’’ pair and a ‘‘diffeent’’ pair between which the different one must be detectAll four combinations were presented for each simplificatiand for each instrument. The two 2-s sounds of eachwere separated by a 500-ms silence, and the two pairs wseparated by a 1-s silence. On each trial, a button labele~inFrench! ‘‘The first pair was different: key 1’’ appeared othe left of the computer screen and a button labeled ‘‘Tsecond pair was different: key 2’’ appeared on the right. Tcomputer would not accept a response until all four souin a trial had been played. This was indicated by a dimmof the labels on the buttons during sound presentation.

For each instrument, a block of 44 trials was presento the subjects~four trial structures311 simplifications!.Each block was presented twice in succession, and pemance for each simplification was computed on eight trifor each subject. Seven pairs of blocks were presentedresponding to the seven instruments. The total durationthe experiment was about two h and 20 min. For 13 subjethe experiment was divided into two sessions performeddifferent days, with four instruments on one day and thinstruments on the other. For seven other subjects, itperformed in one day with several pauses between insments.

The experiment was controlled by thePSIEXPinteractiveprogram~Smith, 1995! running on a NeXT computer. Subjects were seated in a Soluna S1 double-walled souisolation booth facing a window through which the compuscreen was visible. Sounds were converted through Nedigital-to-analog converters, amplified through a Canfopower amplifier, and then presented through AKG K10



Pte

inpeeon

errath

atrao

d-

ss

alc

e

tilyisrysts

rid

es

elly

Ds-t austlta-

sle-

ofith

ce

ralal

on

llyichlywe

-

im-

ns

-

tch-

6

7

9

Redistr

open-air headphones at a level of approximately 70 dB Sas measured with a Bruel & Kajer 2209 sound-level mefitted with a flat-plate coupler.

At the beginning of the experiment, the subject readstructions and asked any necessary questions of the exmenter. Five or six practice trials~chosen at random from thinstrument being tested! were presented in the presencethe experimenter before the first block for each instrumeThen the two experimental blocks for that instrument wpresented. The order of presentation of the 44 trials wasdom within each block, and the order of presentation ofinstruments was randomized for each subject.

III. RESULTS

Discrimination rates were computed for each simplifiction of each instrument’s reference sound across the fourstructures and two repetitions for each subject. The meacross both groups of subjects for the 11 simplificationsseven instrument sounds are given in Table II and plotteFig. 8. Accumulated simplifications involving amplitudeenvelope coherence~AC!, amplitude-envelope smoothne~AS!, and spectral-envelope smoothness~SS! are joined bylines to visualize the effect of accumulation. In generspectral-envelope smoothness and amplitude-envelopeherence simplifications were the most easily discriminatfollowed by coherence~FC! and flatness~FF! of frequencyenvelopes, and finally amplitude-~AS! and frequency-~FS!envelope smoothness. With one exception, the accumulaof simplifications improved discrimination, attaining nearperfect discrimination for all instruments. The pattern of dcrimination differences across simplification types is vedifferent for each instrument, suggesting that the acoustructure of each sound is affected differentially by thesimplifications.

To evaluate the different factors included in this expement, several statistical analyses were performed. Thependent variable in these analyses was thed8 index of sen-sitivity ~derived from proportion-correct discrimination ratin Table A.5.2 from Macmillan and Creelman, 1991!. A glo-bal mixed analysis of variance~ANOVA ! was performed onbetween-subjects factor musical training~2! and within-subjects factors instrument~7! and simplification ~11!.Mixed ANOVAs on musical training and simplification weralso performed for the data of each instrument individua

FIG. 7. Schema illustrating the accumulation of stimulus simplificatioFor key, see Table II.



Lr

-ri-

ft.en-e

-ialnsnin

,o-

d,

on

-

ice

-e-

.

For the data within each instrument, Tukey–Kramer HS~‘‘honestly significant differences’’! were computed to determine the critical difference between condition means asignificance level of 0.05. This technique allows a robcomparison among all means of a data set by the simuneous construction of confidence intervals for all pairs~Ott,1993!. Finally, in order to determine which simplificationwere reliably different from chance performance, singsamplet-tests were performed against a hypothetical mean0.50 with probabilities being corrected for multiple tests wthe Bonferroni adjustment.

A. Effects of musical training

Musicians discriminated simplifications from referensounds slightly better overall than nonmusicians~86.8% vs82.2%! by 3.0% to 7.1% across instruments@F(1,18)58.05, p,0.05#.4 There was no interaction of this factowith other factors in the global analysis. In the individuANOVAs, there were significant main effects of musictraining for four of the seven instruments@flute: F(1,18)55.01, p,0.05; marimba:F(1,18)59.76, p,0.01; oboe:F(1,18)56.99, p,0.05; violin: F(1,18)55.70, p,0.05#,and there were significant musical training by simplificatiinteractions for two instruments@clarinet:F(10,180)52.93,p,0.05; violin: F(10,180)52.55, p,0.05#. So overall,there was a small effect of musical training that was globareliable and present in the majority of instruments but whvaried differently across simplification conditions in ontwo of the instruments. Given the small size of the effect,will not consider it any further.

B. Effects of instrument

In the global ANOVA, there were highly significant effects of instrument@F(6,108)528.80,p,0.0001#, simplifi-cation @F(10,180)5237.97, p,0.0001#, and their interac-tion @F(60,1080)59.87,p,0.0001#. This strong interactionrevealed very large differences in the effects of a given s

.

TABLE II. Results of discriminating six basic simplifications and five combinations of simplifications compared to the reference sounds~completeresynthesis of the originals after frequency, duration, and loudness maing!. Key: AC5amplitude-envelope coherence, AS5amplitude-envelopesmoothness, SS5spectral-envelope smoothness, FC5frequency-envelopecoherence, FS5frequency-envelope smoothness, FF5frequency-envelopeflatness, Cl5clarinet, Fl5flute, Hc5harpsichord, Mb5marimba, Ob5oboe,Tp5trumpet, Vn5violin.

Simplification

Instrument

Cl Fl Hc Mb Ob Tp Vn Mean

AC 0.81 0.96 0.97 0.97 0.75 0.98 0.95 0.91AS 0.56 0.80 0.79 0.59 0.54 0.73 0.59 0.66SS 0.98 0.97 0.96 0.99 0.99 0.82 0.99 0.9FC 0.69 0.72 0.93 0.50 0.53 0.77 0.70 0.69FS 0.56 0.59 0.84 0.67 0.72 0.81 0.69 0.70FF 0.70 0.72 0.91 0.62 0.48 0.82 0.73 0.71AC/FF 0.86 0.98 0.97 0.96 0.94 0.99 0.99 0.95AS/FF 0.69 0.94 0.92 0.65 0.81 0.86 0.71 0.80SS/FF 1.00 0.99 0.97 0.98 0.98 0.89 0.98 0.9AC/AS/FF 0.87 0.98 0.98 0.97 0.86 1.00 0.98 0.95AC/AS/SS/FF 0.99 0.98 0.98 0.98 0.99 0.97 1.00 0.9



na

t

f

42

.te,e,

isee-omr tolifi-eanriti-n ablee

i-is-is-

-ea-

na-0%FFlene.-mu-

anterrenlyin as-hatced/orsultsdi-ra-

imi-for

tlyal-ede

no

Tnc

Redistr

plification across instruments. We will therefore only cosider differences among simplifications within the individuANOVAs for each instrument.

C. Effects of the simplifications and theiraccumulation

The main effect of simplification was highly significan(p,0.0001) for all seven instruments@clarinet: F(10,180)540.14; flute:F(10,180)541.14; harpsichord:F(10,180)511.54; marimba: F(10,180)571.82; oboe: F(10,180)543.40; trumpet: F(10,180)522.05; violin: F(10,180)581.65#, indicating a large variation in discriminability othe different types of simplification. Single-samplet-tests ad-justed for multiple tests indicated that only nine of the

FIG. 8. Discrimination rates as a function of the number of simplificatioperformed on sounds from seven instruments. The letter codes refer tsimplification types~see Table II caption!. Simplifications involving AS,AC, and SS are connected to visualize the effect of their accumulation.vertical bars represent61 standard error of the mean. Chance performais at 0.5. Some points have been displaced laterally for visibility.



-l

single simplifications werenot discriminated above chanceThese include AS and FS for the clarinet, FS for the fluAS and AC for the marimba, AS, FC, and FF for the oboand AS for the violin. Note that no single simplification‘‘successful’’ ~i.e., indistinguishable from the referencsound! for all seven instruments. However, amplitudenvelope smoothness was only reliably discriminated frthe reference in flute, harpsichord, and trumpet. In ordeevaluate the significance of the differences among simpcations, a clustering organization is projected onto the mdata in Fig. 9 in which means that are smaller than the ccal Tukey–Kramer HSD for that instrument are enclosed ibounded region. The critical differences are listed in TaIII. In general, simplifications involving amplitude-envelopcoherence~AC! and spectral-envelope smoothness~SS! arefound in the highest cluster, showing near-perfect discrimnation for most instruments, although AC is less well dcriminated in the clarinet and oboe, and SS is less well dcriminated in the trumpet.

As a general rule, the discrimination of a multiple simplification was roughly equal to the discrimination of thconstituent simplification which had the highest discrimintion rate. For example, take the clarinet sound. Discrimition was near chance for AS, around 70% for FF, about 8for AC, and nearly perfect for SS. Accumulating AS and Fgave a rate no different from that for FF. Similarly, AC/Fand AC/AS/FF had rates no different from that of AC, whiSS/FF and AC/AS/SS/FF were not different from SS aloThis rule held for 32 of the 35 multiple-simplification conditions. Thus, there were only three cases where the acculation of two simplifications was better discriminated theither of the constituent simplifications: AS/FF was betthan AS and FF for the flute, and AC/FF and AS/FF webetter than their constituents for the oboe. There was oone case where an accumulated simplification resulteddecreasein discrimination performance: AC/AS/FF was dicriminated worse than AC/FF for the oboe, suggesting tthe addition of the amplitude-envelope smoothness reduthe effect of amplitude-envelope coherence andfrequency-envelope flatness. Taken together, these resuggest that it is generally sufficient to examine the invidual effects of a single, ‘‘most-potent’’ simplification foeach instrument to explain the behavior of their combintions. In order to compare across instruments, the discrnation rates for the six single simplifications are showneach instrument in Fig. 10.

IV. MEASUREMENTS OF SPECTRAL DIFFERENCESBETWEEN REFERENCE AND SIMPLIFIEDSOUNDS

A. Amplitude and frequency errors

The effect of the simplifications on sounds was direcmeasured from the analysis file data by computing normized rms differences between reference and simplifisounds. Accordingly, for the amplitude simplifications, wmeasured the relative difference between reference~Ar! and

sthe

hee



qu

n.58.ede

h-al-udeoverrgueesateany

s,

m-fied

er-er-eiress

ageed

sixin

or

er-resses,n

orx-Cusir

-ting

%oe

eae. 8ticoet ism

a-

Redistr

simplified ~As! time-varying amplitude spectra~which areassumed to represent sounds having the same mean frecies and same duration! using

ERRamp51

I (i 51

I S (k51K ~Ask~ i !2Ark~ i !!2

(k51K Ask~ i !•Ark~ i ! D 1/2

, ~10!

FIG. 9. Schematic representation of significant differences between mas revealed by Tukey–Kramer HSD tests. Discrimination performancorganized along the vertical dimension within each panel, as in FigSimplifications with means whose differences are not bigger than the cridifference~see Table III! are enclosed within a bounded region. In the obdata, for example, FF is not significantly different from AS and FC budifferent from FS. However, AS and FC are not significantly different froFS.

TABLE III. Critical Tukey–Kramer differences for the mean discrimintions of simplifications computed across both groups of subjects.

Instrument Critical difference

Clarinet 0.217Flute 0.196Harpsichord 0.226Marimba 0.187Oboe 0.207Trumpet 0.215Violin 0.170



en-

wherei is the number of the analysis time frame andI is thetotal number of frames. ERRampcan vary between 0 and 1. Iour set of sounds, it varied between about 0.01 and 0With this formula, the error at any instant relative to thamplitude at that instant is computed. Due to the amplituproduct in the denominator, Eq.~10! accentuates low-amplitude portions, giving them the same weight as higamplitude portions. It is assumed here that proportionamplitude errors are more relevant than absolute-ampliterrors. The normalized squared errors are accumulatedharmonics and are then averaged over time. One could athat this might be improved by first accumulating amplitudby critical bands before averaging, but this would complicthe calculation considerably and would not guaranteeimproved result.

In a similar manner, for the frequency simplificationwe measured the difference between reference~fr! and sim-plified ~fs! series of time-varying frequency data using

ERRfreq51

I f a(i 51

I S (k51

K S ~Ark~ i !„f sk~ i !2 f r k~ i !…

k D 2

(k51K Ark

2~ i !D 1/2

.

~11!

Frequency differences are divided by the harmonic nuberk, because we assume that they are intrinsically amplilinearly withk. The frequency difference for each harmonickis weighted by its amplitude, giving greater votes to highamplitude harmonics. This is beneficial because lowamplitude harmonics tend to have more oscillation in thfrequency data, which is an artifact of the analysis procand not representative of the sound itself~Moorer, 1978!.Besides averaging over time, we normalize by the averfundamental frequency (f a), so that the results are presentas a proportion of the fundamental. The values of ERRfreq inour set of sounds were very low~between 0.0009 and0.0134!.

The amplitude and frequency-error results for thebasic simplifications for the seven instruments are shownTables IV and V, respectively. The meand8 scores are plot-ted in Fig. 11 as a function of the logarithm of the errvalues for the amplitude~a! and frequency~b! simplifica-tions. Although there is some dispersion in the plot, the ovall relations between listener-obtained discrimination scoand the objective measurements are clear. For most calarger errors predict higher sensitivity. If discriminatioscores are expressed in terms ofd8, log(ERRamp) explains77% of the variance in discrimination performance fsingle-amplitude simplifications. The amount of variance eplained increases to 88% if the outlying point due to the Acondition for the marimba is removed. Note that the varioamplitude simplifications are quite different overall in thediscriminability ~AS,AC,SS!.

The picture is quite different for the frequency simplifications. First, the data are much more scattered, indicathat ERRfreq explains much less variance than did ERRampforcorresponding conditions; explained variance ind8 bylog(ERRfreq) is only 34% but increases dramatically to 57when the outlying point due to the FF condition for the ob

nsis.

al



efene

onra-

deoiA

coa

ic

b

id

one-

f the

nd

estond.forAS

am-e

ca-

onbleses

aysheingex-

nd

fiedluessbekey,

0510

11

fiedRis

334

Redistr

is removed. Second, there is a much greater overlap betwthe conditions indicating that there is a less systematic efof the simplification condition and that each simplificatiotype affects the various instruments to very different degre

B. Effect of spectral-amplitude changes on centroid

Since the centroid of the spectrum has been shown tstrongly correlated with one of the most prominent dimesions of multidimensional-scaling representations of timbdifferences~Grey and Gordon, 1978; Iverson and Krumhansl, 1993; Kendall and Carterette, 1996; Krimphoffet al.,1994; Krumhansl, 1989; Wessel, 1979!, one might conjec-ture that a listener’s ability to detect spectral-amplitumodifications is due to detection of attendant centrchanges rather than to the modifications themselves.though in synthesized tones spectral centroid can betrolled independently of other spectral-amplitude modifictions, they are not necessarily separable in musinstrument tones. Nonetheless, we have found them tostatistically independent to a substantial degree in a numof our stimuli.

We define time-varying normalized spectral centro~SC! to be

FIG. 10. Discrimination rates for the seven different instrument souhaving been simplified in six ways~see the text for a complete description!.For instrument key, see Table II.



enct

s.

be-l

dl-n--albeer

SC~ i !5(k51

K kAk~ i !

(k51K Ak~ i !

. ~12!

To test the degree to which an amplitude simplificatiaffects the centroid, we calculate the rms-amplitudweighted mean centroid change based on the centroids osimplified ~SCs! and reference~SCr! spectra:

DSC5

(i 51

I USCs~ i !

SCr ~ i !21U•Arms~ i !

( i 51I Arms~ i !

. ~13!

This quantity is zero if there is no difference in centroid ait is unbounded, although for our simplificationsDSC at-tained a maximum value of 0.3.

Of course, the amplitude-envelope coherence~AC! sim-plification may result in a large centroid change for tonwith a great deal of spectral flux, since it was designedeliminate any centroid change during the course of a souHowever, centroid effects, some quite sizable, also occurAS and SS operations, although the changes induced byare generally much less than those due to the other twoplitude simplifications. Table VI gives a list of the averagrelative centroid changes for the three amplitude simplifitions. Mean discrimination data (d8) are plotted as a functionof DSC in Fig. 11~c!. Note that these averages are basedmagnitudes of the SC changes. Further inspection of TaVI reveals that for the instruments tested, centroid increain stimuli with spectral-envelope smoothness are alwpositive, whereas for the other two simplifications, tchange in centroid can go in either direction—even durthe sounds. The logarithm of the mean centroid change

s

TABLE IV. Relative spectral differences between reference and simplispectra for basic and accumulated amplitude simplifications. The varepresent ERRamp @Eq. ~10!#. Note that the values for basic simplificationand those simplifications accumulated with the FF simplification wouldidentical, since the FF operation has no effect on the amplitudes. Forsee Table II caption.

Simplification

Instrument

Cl Fl Hc Mb Ob Tp Vn

AC 0.100 0.164 0.204 0.033 0.122 0.280 0.35AS 0.017 0.024 0.035 0.016 0.020 0.024 0.01SS 0.565 0.324 0.258 0.505 0.377 0.143 0.40AC/AS/FF 0.101 0.165 0.207 0.035 0.124 0.280 0.35AC/AS/SS/FF 0.578 0.342 0.282 0.508 0.418 0.299 0.5

TABLE V. Relative spectral differences between reference and simplispectra for basic frequency simplifications. The values represent ERfreq

@Eq. ~11!#. Note the values for FF would override all accumulations of thoperation with other simplifications. For key, see Table II caption.

Instrument

Simplication Cl Fl Hc Mb Ob Tp Vn

FC 0.001 0.002 0.010 0.001 0.003 0.003 0.00FS 0.001 0.004 0.013 0.003 0.008 0.004 0.00FF 0.002 0.005 0.013 0.003 0.010 0.007 0.00



oS

s-de-und

icalim-ontheig.andble

essthe

rumnt,im-t inetheo as

onr per-cient

onessrp-m-the

thethe

turede-t al-e-pe

s

acoi

ins

Redistr

FIG. 11. Discrimination scores (d8) plotted as functions of the logarithmof three objective measures@ERRamp ~a!, ERRfreq ~b!, andDSC ~c!# for thesingle amplitude and frequency-envelope simplifications indicated for epanel. The linear regression lines were computed without one outlying pindicated in each panel: Mb/AC in~a!, Ob/FF in ~b!, and Mb/AS in ~c!,since the removal of these single points resulted in dramatic increasescorrelation coefficients in each case~see the text for complete descriptionof the objective measures!.



plains 54% of the variance ind8, but this value increases t65% when the outlying point due to the marimba in the Acondition is removed.

V. DISCUSSION

The discrimination data show that the sensitivity of liteners to simplifications of musical instrument spectrapends on the parameter modified and the instrument soprocessed, while being relatively unaffected by the mustraining of the listeners. The interaction between type of splification and instrument is most likely due to a combinatiof the perceptual salience of the parameter simplified andstrength of that parameter in the particular sound. From F10 it is quite obvious that spectral-envelope smoothnessamplitude-envelope coherence are the most discriminasimplifications. However, spectral-envelope smoothncauses a smaller perceptual change for the trumpet thanother instruments, which is not unexpected since its spectis quite smooth to begin with. For this latter instrumeamplitude-envelope coherence is the most discriminable splification, due to the strong degree of spectral flux presenbrass tones~Grey, 1977!. Further, the amplitude-envelopcoherence simplification is much less discriminable forclarinet and oboe because their spectra do not undergmuch spectral flux as most other instruments~Grey, 1977!.

The other simplifications result in lesser discriminatiscores, either because these involve parameters of lesseceptual salience or because the parameters have insuffistrength to result in higher scores. Again, scores dependthe instrument tested. Amplitude-envelope smoothnseems to be most important for the flute, trumpet, and hasichord, the former two because of their relatively large teporal variations and the latter because of its effect ondecay curves.

Objective measures of average spectral (DSC) and spec-trotemporal change (ERRamp,ERRfreq) were developed in anattempt to quantify the acoustic cues that give rise todiscrimination performance. Figure 11 clearly shows thatthree amplitude simplifications~AC, AS, and SS! have dif-ferent effects on changes in amplitude-envelope strucand consequently that they are discriminated to differinggrees as well. Spectral-envelope smoothness is almosways the most easily detected, followed by amplitudenvelope coherence, and finally by amplitude-envelo

hnt

the

ciablyificantsound

TABLE VI. Average relative-magnitude change of centroid@DSC, Eq.~13!# for the three basic amplitudesimplifications and two accumulations of those simplifications. Note that centroids would not be appreaffected by the FF simplification since the frequency variations were less than 1% during the signamplitude portions of the sounds. A minus sign indicates that, on average, the centroid for the simplifieddecreased compared to the reference sound. For key, see Table II caption.

Simplification

Instrument

Cl Fl Hc Mb Ob Tp Vn

AC 0.031 0.045 0.279~2! 0.118~2! 0.038 0.166 0.155AS 0.005~2! 0.023 0.015~2! 0.054~2! 0.002~2! 0.005 0.012~2!SS 0.039 0.041 0.047 0.299 0.023 0.012 0.042AC/AS/FF 0.031 0.046 0.282~2! 0.117~2! 0.038 0.166 0.156AC/AS/SS/FF 0.058 0.061 0.284~2! 0.359 0.038 0.168 0.156



nc

ofanpen-peean

th

-

dau

edngmopswffetrunceroob

inc

ioeeutthntegr

dere

erng

i

twc

ifi-blto

to,d,fre-re-rderby

ng,urehe

95;esantrta-

ds isn ofs,

ndileri-in

hisove,er inta,

tionm-o-ent

areerstheors.

u-

in

jec-the

ated

theen-ian

Redistr

smoothness, which gives performance not far from chafor four of the seven instrument sounds.

Similarly, note that for most sounds the effectamplitude-envelope smoothness on spectral centroid ch~mean50.016! is much less than that of amplitude-envelocoherence~0.119!, with the flute being the only exceptio~0.023 vs 0.045, respectively!. Indeed, the flute has a comparatively high discrimination score for amplitude-envelosmoothness~0.80!, whereas the marimba with amplitudsmoothness has a moderately high spectral-centroid ch~0.054! and a relatively low discrimination rate~0.59!. Also,surprisingly, the flute has a high discrimination score foramplitude-coherence simplification~0.96!, even though itschange of centroid~0.041! is slightly less than the amplitudebased simplifications~overall mean for the three, 0.045!. Onthe other hand, we see that in comparison to the amplituenvelope coherence, spectral-envelope smoothness cmoderate to high centroid changes~0.02–0.30!, with thetrumpet ~0.01! being the obvious exception as mentionabove. The marimba exhibits a large relative centroid cha~0.30!, but this is true because the marimba’s sound is donated by its fundamental. In this case, spectral-envelsmoothness makes a profound change by introducing aond harmonic which was originally nonexistent. Note, hoever, that spectral-envelope smoothness will have an eon any jagged spectrum, regardless of whether the specis changing or not, whereas amplitude-envelope cohereonly affects sounds with time-varying spectra. Sinspectral-envelope smoothness inherently affects the centwe cannot tell whether discrimination is due to this effectdirectly to the change of spectral-envelope fine structure,it is probably due to a combination of these effects.

All of the amplitude simplifications produce changesboth ERRamp and DSC measures. Further, these two objetive measures partially explain the variance in discriminatperformance and yet are not strongly correlated betwthem (r 50.61). This suggests that they may both contribto the discrimination of amplitude-related changes inspectrotemporal morphology of the simplified instrumesounds. To test this idea, the logarithms of both paramewere selected as independent variables in a stepwise resion across single-amplitude simplifications withd8 as thedependent variable. This technique tests the indepencontribution of each parameter, which only enters the regsion if its contribution is statistically significant~F-to-enter54, in our case!. Both parameters successfully entinto the regression. The final result is given by the followilinear regression equation, by which 83% of the variancethe data is explained:

d854.3411.35• log~ERRamp!10.64• log~DSC!. ~14!

It becomes clear from this analysis that there are at leastperceptual cues contributing to discrimination performanin these sound simplifications.

The striking thing about the frequency-related simplcations is their relative weakness in creating discriminadifferences in the stimuli. This result may be due primarily



e

ge

ge

e

e-ses

ei-e

ec--ctm

ce

id,rut

-nn

eetrses-

nts-

n

oe

e

the fact that, in normal instrument sounds without vibrathe amount of frequency variation is relatively small. Indeeas can be gleaned from Table V, the largest change inquency variation created by flattening or smoothing the fquency envelopes is for the harpsichord and is on the oof 1.3%; the next largest is on the order of 1% producedfrequency flattening of the oboe. It is perhaps surprisitherefore, that so much has been written in the literatabout the importance of frequency microvariations in tcreation of naturalness in synthetic sounds~Dubnov and Ro-det, 1997; McAdams, 1984; Sandell and Martens, 19Schumacher, 1992!. Nonetheless, there are certainly classof musical sounds where pitch contour plays an importrole in musical expressiveness, such as vibrato and pomento, particularly in vocal and bowed-string sounds.

The effect of combining amplitude-related anfrequency-related cues for the accumulated simplificationless clear in the data, however. In a stepwise regressiothe entire set ofd8 scores on all three objective measureonly ERRamp entered significantly into the regression athen explained only 63% of the total variance. So, whindividual cues seem to explain a large portion of the vaance for the basic simplifications, their combined usejudging accumulated simplifications remains uncertain. Tmay be due in part to the judgment strategy discussed abnamely that listeners respond to the most salient parametan accumulation of parameters. In the discrimination dathere are only four out of 35 cases where an accumulagives higher discrimination scores than the best of its coponent simplifications: AC/FF is better than either compnent for the oboe, and AS/FF is better than either componfor clarinet, flute, and oboe. If our objective measurestruly indicative of the perceptual cues being used by listento perform the task, they should have a similar pattern todiscrimination data with accumulations having the sameslightly higher values than their constituent simplificationGlobally this is the case~see Tables IV, V, and VI!: AC/AS/FF is approximately equal to AC for all seven instrments in terms of both ERRamp andDSC, and AC/AS/SS/FFis approximately equal to or slightly higher than SS or ACall seven instruments in terms of ERRamp, although it is quitea bit higher for clarinet, flute, and marimba in terms ofDSC.The combination of the psychophysical data and the obtive measurements would thus seem to globally supportmost-potent cue judgment strategy.

VI. CONCLUSIONS

The results of this study point very strongly to~1!spectral-envelope shape~jagged vs smooth! and ~2! spectralflux ~time variation of the normalized spectrum! as being themost salient physical parameters that we have studied relto timbre discrimination, followed in order by~3! the pres-ence of frequency variation,~4! frequency incoherence~in-harmonicity!, ~5! frequency microvariation, and~6! ampli-tude microvariation. Simplifications ~reductions oreliminations! of these parameters give rise to changes inspectrotemporal morphology of an instrument sound’s ssory representation, to which both musician and nonmusic



tlyis

llyee

pae.

rea

pi

nctththwca

ttta

tioaeica

wferu

b-nu

d

-

ic-

fre-

or

h-

dlytheatee

nd

dataof2,

en-ata

re

70,

ero.s

annd

-flattonetly

re-withthe

net,.64,50 inove

Redistr

listeners are very sensitive. This sensitivity is only slighgreater in musicians than in nonmusicians. The level of dcrimination resulting from the modifications was globagreater for the amplitude simplifications than for the frquency simplifications, with the exception of amplitudsmoothing. Thus, musical-sound synthesis should payticular attention to spectral-envelope fine-structure and sptral flux if a high degree of audio quality is to be ensured

Objective measures were defined that predict a gdeal of the discrimination performance. These measuresrelated to changes in the amplitude envelopes and the stral centroid for amplitude simplifications, and to changesthe frequency envelopes for frequency simplifications. Sidiscrimination can be predicted by physical measuremendifferences in the time-varying spectra, it appears thatimportance of these parameters is in direct proportion toextent to which they actually vary in musical sounds, ashave shown with the strong interaction between simplifition type and musical instrument. Further work is neededexamine the relative perceptual sensitivity of listenersthese different physical factors. We have also shown thaseveral parameters are varied simultaneously, listenerspear to use the most salient one, and their discriminaperformance can, for the most part, be predicted on the bof it alone. While it is likely that this acute sensitivity to thfine-grained spectral and temporal structure of the mussounds exists across the entire range of pitch, dynamics,articulation possible on each instrument, further researchbe needed to determine the relative importance of the difent objective parameters in different regions of an instment’s musical ‘‘space.’’

ACKNOWLEDGMENTS

We would like to thank Sophie Savel for running sujects in the control experiment, Bennett Smith for the woders ofPSIEXP, as well as John Hajda and two anonymoreviewers for helpful comments.

APPENDIX A

We can write the reduced-duration harmonic amplituenvelope as

Ak~ t !←5Ak~ t !, 0<t,t1 ,

~12a~x!!•Ak~ t !1a~x!•Ak~ t1t22t3!,

t1<t,t3 ,

Ak~ t1tL22!, t3<t<2,~A1!

where

t3522~ tL2t2!, a~x!53x222x3, x5~ t2t1!

~ t32t1!,

and tL is the duration of the original sound.Note thata(x) is a cubic spline with the following prop

erties:



-

-

r-c-

atre

ec-neofeee-ooifp-n

sis

alndillr--

-s

e

~1! The derivative ofa(x) with respect tox is zero atx50 andx51,

~2! a~0!50,~3! a~1!51.

The same method obviously applies to the harmonfrequency envelopes.

APPENDIX B

Since the data rate for each harmonic amplitude orquency envelope is originally 2f a , the overall data rate forKharmonic amplitude and frequency envelopes is 4K• f a .

Amplitude-envelope smoothing~AS! and frequency-envelope smoothing~FS! essentially reduce the data rate feach harmonic envelope from 2f a to 2f c , where f c is thefilter cutoff frequency. If only amplitude-envelope smooting were applied, the data rate forK harmonics would bereduced to 2K• f c12K• f a52K•( f c1 f a). In our case, sincef a5311 Hz andf c510 Hz, the data reduction factor woulbe 4.311/@2•~101311!#51.94. The same result would appif only frequency-envelope smoothing were applied. Onother hand, if both were applied the new total data rwould be 4K• f c , and the data reduction factor would bf a / f c . In our case, this is 311/10531.1, i.e., there is only asubstantial overall data reduction if both amplitude- afrequency-envelope smoothing are applied.

Spectral-envelope smoothing does not reduce therate very much, at least not with the current definitionsmoothing. Since the order of the smoothing function isthe reduction is approximately a factor of 2.

Amplitude- ~AC! and frequency coherence~FC! simpli-fications essentially replace multiple envelopes by singlevelopes. If one of these simplifications were applied, the drate falls from 4K• f a to 2K• f a12 f a52(K11)• f a . So, thedata reduction factor would be approximately 2. If both weapplied, the data reduction factor would be exactlyK, thenumber of harmonics. In our case, this varies from 30 todepending on the instrument.

The data rate for flattened frequency envelopes is zSo, if frequency flattening~FF! is applied, the data rate goefrom 4K• f a to 2K• f a , a factor of 2 reduction.

Data rates after combinations of data simplifications cbe calculated from the individuals. For example, if AC aFF are combined, the data rate becomes 2f a . For AS and FF,it would be 2K• f c . For SS and FF, it isK• f a . For AC, AS,and FF, it is 2f c . For AC, AS, SS, and FF, it is justf c . Thecorresponding data-reduction factors are for AC/FF, 2K; forAS/FF, 2f a / f c ; for SS/FF, 4; for AC/AS/FF, 2K• f a / f c ; forAC/AS/SS/FF; 4K• f a / f c .

1We were unable to find a trumpet tone of suitable quality recorded at E4, so we used a tone recorded by author J.B. which was within a wholeof that pitch, F4. When resynthesized at E-flat 4, it sounded perfecnatural to all of the authors.

2A control experiment designed to test discrimination of the digitizedcordings and the fully analyzed–resynthesized sounds was conductedsix listeners. Each subject performed 40 trials for each instrument usingparadigm described in Sec. II C. The discrimination rates for oboe, clariflute, harpsichord, marimba, trumpet, and violin were 0.62, 0.54, 0.59, 00.53, 0.57, and 0.53, respectively. Chance performance would be at 0.this two-alternative forced-choice paradigm. Discrimination rates ab



lsse

64ate

posysurthentatos,

lifits:

hror

-ad

evonim

pefr

s a

ho-

-

en

al

e

-cen-

.

-

-

l

-

-

psy-

’’

ionity,

ff, J.di-

l

li-

is

x

he

y

ng.

i-

dy

-

Redistr

0.55 are significantly different from chance (p,0.05) by an exact binomiatest for 240 trials~6 listeners340 trials!. Therefore, the full reconstructionwere discriminated from the original recordings in four of the seven ca~oboe, harpsichord, marimba, and trumpet!, although their discriminationrates are still quite low. That the discrimination scores are as high asis actually quite surprising, given that the authors could not discriminthem informally. The subjects must have been operating from extremsubtle cues which only became obvious upon repeated listening. Twosible cues are phase differences and low-frequency noise differencetween the original and synthetic cases. Improvements in the analsynthesis algorithms should be able to close this gap in the futHowever, for this study we are confident that the vast majority ofspectrotemporal features survived the analysis/synthesis process iMoreover, the duration-shortened resynthesized sounds were createdthe reference sounds, so our conclusions, which relate to these soundnot affected by this finding.

3Audible artifacts were sometimes noticeable with the AC and SS simpcations. Amplitude-envelope coherence created two types of artifac‘‘shh’’ noise at the end of the sound~clarinet! or a kind of general mutingof the sound~harpsichord, marimba, and trumpet!. In fact, in rendering theamplitude envelopes coherent, one increases the amplitude of weakmonics that start later and end earlier in parts of the signal that are neanoise floor. The frequency estimation is not very precise for these tempregions of such harmonics~Moorer, 1978!, and by increasing their amplitude, the existing imprecise fluctuations in frequency become clearlydible. The spectral-envelope smoothing creates a kind of gargling sounthe flute, marimba, and trumpet. This simplification also increases the lof weak harmonics whose representations have been corrupted by strneighbors due to channel leakage, thus amplifying fluctuations due toprecise estimation of their frequency. These two kinds of artifacts disapor are notably reduced when combined with frequency coherence orquency flatness, since the frequency fluctuations producing the artifactthen reduced or eliminated.

4Probabilities are corrected where necessary by the Geisser–Greenepsilon ~Geisser and Greenhouse, 1958!, which is a conservative adjustment to account for inherent correlations among repeated measures.

Beauchamp, J. W.~1993!. ‘‘Unix workstation software for analysis, graphics, modifications, and synthesis of musical sounds,’’94th Convention ofthe Audio Engineering Society, Berlin ~Audio Engineering Society, NewYork!, Preprint 3479~L-I-7!.

Beauchamp, J. W., McAdams, S., and Meneguzzi, S.~1997!. ‘‘Perceptualeffects of simplifying musical instrument sound time-frequency represtations,’’ J. Acoust. Soc. Am.101, 3167.

Brown, J. C.~1996!. ‘‘Frequency ratios of spectral components of musicsounds,’’ J. Acoust. Soc. Am.99, 1210–1218.

Charbonneau, G. R.~1981!. ‘‘Timbre and the perceptual effects of thretypes of data reduction,’’ Comput. Music J.5„2…, 10–19.

Dubnov, S., and Rodet, X.~1997!. ‘‘Statistical modeling of sound aperiodicities,’’ in Proceedings of the International Computer Music Conferen,1997, Thessaloniki~International Computer Music Association, San Fracisco!, pp. 43–50.

Geisser, S., and Greenhouse, S. W.~1958!. ‘‘An extension of Box’s resultson the use of theF distribution in multivariate analysis,’’ Ann. Math. Stat29, 885–891.



s

%elys-

be-is/e.

ct.beare

-a

ar-theal

u-inelger-

are-re

use

-

Green, D. M., and Swets, J. A.~1974!. Signal Detection Theory and Psychophysics~Krieger, Huntington, NY!.

Grey, J. M.~1977!. ‘‘Multidimensional perceptual scaling of musical timbres,’’ J. Acoust. Soc. Am.61, 1270–1277.

Grey, J. M., and Gordon, J. W.~1978!. ‘‘Perceptual effects of spectramodifications on musical timbres,’’ J. Acoust. Soc. Am.63, 1493–1500.

Grey, J. M., and Moorer, J. A.~1977!. ‘‘Perceptual evaluations of synthesized musical instrument tones,’’ J. Acoust. Soc. Am.62, 454–462.

Iverson, P., and Krumhansl, C. L.~1993!. ‘‘Isolating the dynamic attributesof musical timbre,’’ J. Acoust. Soc. Am.94, 2595–2603.

Kendall, R. A., and Carterette, E. C.~1996!. ‘‘Difference thresholds fortimbre related to spectral centroid,’’ inProceedings of the 4th International Conference on Music Perception and Cognition, Montreal, pp.91–95~Faculty of Music, McGill University!.

Krimphoff, J., McAdams, S., and Winsberg, S.~1994!. ‘‘Caracterisation dutimbre des sons complexes. II. Analyses acoustiques et quantificationchophysique,’’ Journal de Physique4~C5!, 625–628.

Krumhansl, C. L.~1989!. ‘‘Why is musical timbre so hard to understand?,in Structure and Perception of Electroacoustic Sound and Music, editedby S. Nielzen and O. Olsson~Excerpta Medica, Amsterdam!, pp. 43–53.

Macmillan, N. A., and Creelman, C. D.~1991!. Detection Theory: A User’sGuide ~Cambridge U.P., Cambridge!.

McAdams, S.~1984!. ‘‘Spectral Fusion, Spectral Parsing, and the Formatof Auditory Images,’’ unpublished Ph.D. dissertation, Stanford UniversStanford, CA, App. B.

McAdams, S., Winsberg, S., Donnadieu, S., De Soete, G., and Krimpho~1995!. ‘‘Perceptual scaling of synthesized musical timbres: Commonmensions, specificities, and latent subject classes,’’ Psychol. Res.58, 177–192.

Miller, J. R., and Carterette, E. C.~1975!. ‘‘Perceptual space for musicastructures,’’ J. Acoust. Soc. Am.58, 711–720.

Moorer, J. A.~1978!. ‘‘The use of phase vocoder in computer music appcations,’’ J. Audio Eng. Soc.24, 717–727.

Ott, R. L. ~1993!. An Introduction to Statistical Methods and Data Analys~Duxbury, Belmont, CA!.

Plomp, R. ~1970!. ‘‘Timbre as a multidimensional attribute of completones,’’ in Frequency Analysis and Periodicity Detection in Hearing, ed-ited by R. Plomp and G. F. Smoorenburg~Sijthoff, Leiden!, pp. 397–414.

Preis, A. ~1984!. ‘‘An attempt to describe the parameter determining ttimbre of steady-state harmonic complex tones,’’ Acustica55, 1–13.

Repp, B. H.~1987!. ‘‘The sound of two hands clapping: An exploratorstudy,’’ J. Acoust. Soc. Am.81, 1100–1109.

Sandell, G. J., and Martens, W. L.~1995!. ‘‘Perceptual evaluation ofprincipal-component-based synthesis of musical timbres,’’ J. Audio ESoc.43, 1013–1028.

Schumacher, R. T.~1992!. ‘‘Analysis of aperiodicities in nearly periodicwaveforms,’’ J. Acoust. Soc. Am.91, 438–451.

Smith, B. K. ~1995!. ‘‘ PSIEXP. An environment for psychoacoustic expermentation using theIRCAM Musical Workstation’’, in Program of theSMPC95: Society for Music Perception and Cognition, Berkeley, CA, ed-ited by D. L. Wessel~University of California, Berkeley!.

von Bismarck, G.~1974!. ‘‘Sharpness as an attribute of the timbre of steasounds,’’ Acustica30, 159–172.

Wessel, D. L.~1979!. ‘‘Timbre space as a musical control structure,’’ Comput. Music J.3„2…, 45–52.



discrimination of musical instrument sounds resynthesized with simplified spectrotemporal parameters

Documents