score-informed networks for music performance …...score-informed models as compared to...

SCORE-INFORMED NETWORKS FOR MUSIC PERFORMANCEASSESSMENT

Jiawen Huang ∗ Yun-Ning Hung ∗ Ashis PatiSiddharth Gururani Alexander Lerch

Center for Music Technology, Georgia Institute of Technology, USA{jhuang448,amyhung,ashis.pati,siddgururani,alexander.lerch}@gatech.edu

ABSTRACT

The assessment of music performances in most cases takesinto account the underlying musical score being performed.While there have been several automatic approaches forobjective music performance assessment (MPA) based onextracted features from both the performance audio and thescore, deep neural network-based methods incorporatingscore information into MPA models have not yet been inves-tigated. In this paper, we introduce three different modelscapable of score-informed performance assessment. Theseare (i) a convolutional neural network that utilizes a simpletime-series input comprising of aligned pitch contours andscore, (ii) a joint embedding model which learns a joint la-tent space for pitch contours and scores, and (iii) a distancematrix-based convolutional neural network which utilizespatterns in the distance matrix between pitch contours andmusical score to predict assessment ratings. Our results pro-vide insights into the suitability of different architecturesand input representations and demonstrate the benefits ofscore-informed models as compared to score-independentmodels.

1. INTRODUCTION

A performance is a sonic rendition of a written musicalscore (in the case of Western classical music). The charac-teristics of a music performance play a major role in howlisteners perceive music, even if performances are basedon the same underlying score [14]. To perform a musicalpiece, the performer must first parse the score, interpret ormodify the musical information, and utilize complex motorskills to render the piece on their instrument [21].

From the perspective of the performer, mastery over theart of music performance is often a journey spanning severalyears of instruction and practice. A major factor in learningand improving one’s skill as a performer is to analyze and

∗ These authors contributed equally to this work.

c© Jiawen Huang, Yun-Ning Hung, Ashis Pati, SiddharthGururani, Alexander Lerch. Licensed under a Creative Commons At-tribution 4.0 International License (CC BY 4.0). Attribution: JiawenHuang, Yun-Ning Hung, Ashis Pati, Siddharth Gururani, Alexander Lerch.“Score-informed Networks for Music Performance Assessment”, 21st In-ternational Society for Music Information Retrieval Conference, Montréal,Canada, 2020.

obtain feedback regarding the performance. Due to the com-plex nature of music performance, students require regularfeedback from trained professionals. Teachers are expectedto grade or rate students based on various performance cri-teria such as note accuracy or musicality. These criteriaare often ill-defined and subject to interpretation, thus mak-ing objective and consistent music performance assessment(MPA) rather difficult [26, 29]. Regardless, this subjec-tive manner of MPA is still used, e.g., in school systemswhere ensemble members are selected based on instructors’assessments of student auditions.

Wu et al. discussed the notion of objective descrip-tors (features) which are potentially useful for automaticMPA [30]. Such features are computed by applying signalprocessing methods to recorded performances and are usedto model teachers’ assessments of the performances usingmachine learning. With the rise of deep learning, neuralnetworks were found to outperform the classical pipelineof feature extraction followed by regression [22]. However,one issue with these approaches is that they ignore the scorethat the students are meant to play. We will refer to suchapproaches as score-independent. The idea of incorporat-ing score-based features utilizing audio to score alignmentwas explored, e.g., by Vidwans et al. [27]. Further analy-sis of hand-crafted features for MPA showed the relativeimportance of score-based features over score-independentones [8]. Therefore, the design of deep architectures thatincorporate score information is an obvious and overdueextension of previous approaches.

The goal of this paper is to explore different methods toincorporate this score information. Our hypothesis is thatincluding score information will lead to improved perfor-mance of deep networks in the objective MPA task. To thisend, we present three architectures which combine scoreand audio features to make a score-informed assessment ofa music performance. First, we concatenate aligned pitchcontours and scores into a 2-dimensional time-series featurerepresentation that is fed to a convolutional neural network(CNN). Second, we propose a joint embedding model foraligned score and pitch contours. The assessment ratingsare predicted using the cosine similarity between the scoreand performance embeddings. Third, we utilize the distancematrix, a mid-level representation combining both the scoreand pitch contour, as the input to a deep CNN trained topredict the teachers’ assessments. Finally, using a fairlylarge scale dataset of middle school and high school student

auditions, we perform an in-depth evaluation comparingthese proposed architectures against each other and with ascore-independent baseline approach for MPA .

2. RELATED WORK

MPA deals with the task of assessing music performancesbased on audio recordings. Progress in MPA is roughlycategorized into feature design-based approaches [8, 13, 20,24, 30] and feature learning-based approaches [9, 22, 31].Feature design-based methods rely on signal processingtechniques to either extract standard spectral and temporalfeatures [13], or use expert knowledge to extract perceptu-ally motivated features capable of characterizing music per-formances [20, 24]. The extracted features are then fed intosimple machine learning classifiers to train models whichpredict different performance assessment ratings. Featurelearning-based approaches, on the other hand, stem from theargument that important features for modeling performanceassessments are not trivial and cannot be easily described.Hence, they rely on using mid-level representations (suchas pitch contours or mel-spectrograms) as input to sophisti-cated machine learning models such as sparse coding [9,31]and neural networks [22].

Most performances of Western music, however, arebased on written musical scores. Hence, performancesare also assessed based on their perceived deviations fromthe underlying score. There has been some prior researchon incorporating the score information into the assessmentmodeling process. Most of the these approaches rely oncomputing descriptive features using some notion of dis-tance between the score representation and the performancerepresentation [3, 6, 11, 17, 19]. The most common ap-proach has been to first use an alignment algorithm, e.g.,Dynamic Time Warping (DTW) [25], to temporally alignthe performance recording with the score and then computedescriptive features which characterize the deviations of theperformance from the score [1, 27]. However, to the bestof our knowledge, incorporating score information directlyinto neural network-based models for MPA has not beeninvestigated before.

Score-informed approaches have helped improve resultsfor both related performance analysis tasks and other musicinformation retrieval tasks. Most of these methods havealso relied on an alignment between the audio recordingand the score as the primary tool for incorporating scoreinformation. Aligning audio recordings with scores hasbeen useful for several tasks such as detecting expressivefeatures in music performances [15], identifying missingnotes and errors in piano performances [5], and segmentingsyllables in vocal performances [23]. Scores have also beenused to generate soft labels and/or artificial training data fortasks such as source separation [4, 18].

3. METHODS

We propose and compare three different approaches to incor-porate the score information with audio features for MPA. 1

1 The code is available at: https://github.com/biboamy/FBA-Fall19

Scor

e

Pitc

h C

onto

ur

Assessment

Linear

1D Convolutional Layer, k-kernal size, s-stride size, c-number of features

1D Batch Normalization Layer ReLU (Rectifier Linear Unit)

c-4 c-8 c-16

k-3,

k-7

Con

v

Con

v

Con

v

c-16

Leak

y R

eLU

Figure 1. Schematic for the SIConvNet. The aligned scoreand pitch contour are stacked together and fed into a 4-layerCNN to directly predict the assessment ratings.

The score information is represented as the MIDI pitch se-quence (in ticks) obtained from the sheet music of the scoreto be performed. Henceforth, the MIDI pitch sequence willbe referred to as the score. The student’s performance isrepresented by the pitch contour of the audio. We use pitchcontour since it captures both pitch and rhythmic informa-tion. Musical dynamics and timbre are ignored in this study;while dynamics are an important expressive tool for the per-former [14], the score usually lacks specificity in dynamicsinstructions and cannot serve as the same absolute referenceas for pitch and rhythm.

3.1 Score-Informed Network (SIConvNet)

The first approach that we use is probably the most straight-forward way of incorporating score information into theassessment model. A simple CNN is used that relies onboth the score and performance as the input and directlypredicts the assessment ratings.

3.1.1 Input Representation

The input representation for this approach is constructed bysimply stacking an aligned pitch contour and score pair tocreate a N × 2 matrix, where N is the sequence length ofthe pitch contour. The two channels correspond to the pitchcontour and score, respectively.

In order to obtain this representation, we first consider apitch contour snippet of length N (sequence of logarithmicfrequencies). Then, we find the corresponding part of thescore using a DTW-based alignment process. The obtainedscore snippet (sequence of MIDI note numbers) is thenresampled to have the same length N as the pitch contour.

3.1.2 Model Architecture

A schematic of the model architecture is shown in Figure 1.We use a simple 4-layer CNN based on the architectureproposed by Pati et al. [22] and append a single linearlayer which predicts the assessment. Each convolutionalstack consists of a 1-D convolution followed by a 1-D batchnormalization layer [12] and ReLU non-linearity. The linearlayer at the end comprises of a dense layer followed byLeaky ReLU non-linearity.

Score

Pitch Contour

Embe

ddin

gEm

bedd

ing

Cosine Similarity(Assessment)

k-3,

k-7

c-4

Con

v

Con

v

Con

v

c-8 c-16 c-16

k-3,

k-7

Con

v

Con

v

Con

v

Figure 2. Schematic of the JointEmbedNet architecture.

3.2 Joint Embedding Network (JointEmbedNet)

The second approach is based on the assumption that perfor-mances are rated based on some sort of perceived distancebetween the performance and the underlying score beingperformed. Consequently, we use two separate encodernetworks to project the score and the pitch contour to ajoint latent space and then use the similarity between theembeddings to predict the assessment ratings.


This approach uses the same input representation as SICon-vNet (see Section 3.1.1). However, instead of stacking thealigned pitch contour and the score, the individual N × 1sequences are fed separately to the two encoders.


This network (see Figure 2) uses two 1-D convolutionalencoders having the same architecture as SIConvNet. Eachencoder has 4 convolutional blocks to extract high levelfeature embeddings. The performance encoder is expectedto extract relevant features pertaining to the performancefrom the pitch contour. On the other hand, the score en-coder is expected to extract the important features from thescore. Assuming that the assessment rating for the perfor-mance is high if these two embeddings are similar, we usethe cosine similarity cos(Escore, Eperformance) between thetwo embeddings to obtain the predicted assessment rating.Escore and Eperformance are the embeddings obtained fromthe score and performance encoders, respectively. If thetwo embeddings are similar, the cosine similarity is closeto one, and the model will predict a higher rating.

3.3 Distance Matrix Network (DistMatNet)

The final approach uses a distance matrix between the pitchcontour and the score as the input to the network. Giventhe information from both the pitch contour and the score,the task of performance assessment might be interpreted asfinding a performance distance between them. Thus, thechoice of the distance matrix as the input representationallows the model to learn from the pitch differences. AResidual CNN [10] architecture is chosen for the network.


The distance matrix elements are the pair-wise wrappeddistances between the pitch contour and the MIDI pitchsequence. The octave-independent wrapped-distance is

Score

Pitc

h C

onto

ur

Assessment

2D Convolutional Layer

2D Batch Normalization ReLU

Residual Block Dropout

MaxPool2D

Linear Layer

Figure 3. Schematic of the DistMatNet architecture.

used to compensate for possible octave errors made by thepitch tracker. To ensure a uniform input size to the network,the matrix is resampled to a square shape of a fixed size.Thus, a performance with constant tempo would result in analigned path located on the diagonal. Unlike the previoustwo methods where the input pitch contour and the scoreare aligned using DTW, the distance matrix input avoidsany error propagation caused by alignment errors. Thechoice of this input representation stems from the successof distance matrices (or self-similarity matrices) in otherareas of MIR such as structural segmentation [2, 7] andmusic generation [28].


The model architecture is shown in Figure 3. It is composedof 3 residual blocks. Each residual block has 2 convolu-tional layers. Dropout and max-pooling are added betweeneach residual block. A classifier with two linear layers (128features) with one ReLU and dropout layer in between isused after the residual network to perform regression pre-diction. We use (3,3) kernal size and 4 feature maps for allconvolutional layers, 0.2 dropout rate, and a (3,3) kernalsize for all max-pooling layers.

4. EXPERIMENTS

4.1 Dataset

The dataset we use to evaluate our methods is a subsetof a large dataset of middle school and high school stu-dent performances. These are recorded for the FloridaAll State Auditions, which are separated into three bands:(i) middle school band, (ii) concert band, and (iii) sym-phonic band. The recordings contain auditions spanning 6years (from 2013 to 2018), and feature several monophonicpitched and percussion instruments. Each student performsrehearsed scores, scales and a sight reading exercise. Forthe purpose of this study we limit our experiments to thetechnical etude for middle school and symphonic band au-ditions. We choose Alto Saxophone, Bb Clarinet and Fluteperformances due to these being the most popular acrossall pitched instruments. Table 1 shows the distribution ofdata across different instruments. The average duration ofeach performance is 30 s for middle school and 50 s forsymphonic band students. The dataset also includes themusical scores that the students are supposed to perform foreach exercise. The average length (in notes) of the musical

Middle School Symphonic BandAlto Saxophone 696 641Bb Clarinet 925 1156Flute 989 1196

Table 1. Number of performances for the different instru-ments per band.

scores are 136 for middle school and 292 for symphonicband. Note that these scores are the same across all studentsperforming the same instrument in the same year but varyacross years and instruments.

The dataset also contains expert assessments for eachexercise of a student performance. Each performance israted by one expert along 4 criteria defined by the FloridaBandmasters’ Association (FBA): (i) musicality, (ii) noteaccuracy, (iii) rhythmic accuracy, and (iv) tone quality. Allratings are on a point-based scale and are normalized torange between 0 to 1 by dividing by the maximum. Sincewe focus on pitch contours as the primary audio feature,tone quality is excluded from this study.

4.1.1 Data pre-processing

The pitch contours are extracted using the pYIN algo-rithm [16] with a block size and hop size of 1024 and 256samples, respectively. The audio sampling rate is 44100 Hz.The extracted frequencies are converted from Hz to MIDIpitch (unlike the MIDI pitches from the musical score, thesecan be floating point numbers). Both the resulting pitchcontour and musical score are normalized by dividing by127. Finally, for the purpose of model training and evalu-ation, we divide our dataset into three randomly sampledsubsets: training, validation, and testing. We use a ratio of8: 1 : 1 for splitting the dataset.

We use random-chunking as a data augmentation toolwhen training SIConvNet and JointEmbedNet since it hasshown to be useful in improving model performance [22].First, the pitch contour is chunked into snippets of length Nby randomly selecting the starting position. The correspond-ing aligned and length-adjusted score snippet is obtainedusing the method described in Section 3.1.1. We assumethe chunked segment has the same assessment score as thewhole recording. We do not perform chunking on our dis-tance matrix since the matrix has already been resampledinto a smaller resolution. Instead, we discuss how varyingthe resampling size could effect the performance in one ofthe experiments.

4.2 Experimental Setup

We present three experiments to evaluate our proposed meth-ods. First, we compare the overall performance of the pro-posed architectures against a score-independent baselinesystem PCConvNet [22] which uses only the randomly-chunked pitch contour as input. This experiment also givesus an indication of the effectiveness of each of the proposedmethods. Second, we look at the sensitivity of the SICon-vNet and JointEmbedNet to the chunk size N . Finally, weinvestigate the effect of varying the resolution of the input

SIConvNet JointEmbedNet DistMatNet3,089 6,144 63,417

Table 2. Number of parameters for each model.

distance matrix for the DistMatNet model. The latter twoexperiments were aimed at understanding the effects of thedifferent hyper-parameters used while constructing the in-put data for each model. These helped us arrive at the bestparameters for each approach.

The number of trainable parameters for each methodis shown in Table 2. DistMatNet has a higher number ofparameters because it uses a higher-dimensional input witha deeper architecture to capture high level information [10].

For each method, we trained separate models to predicteach assessment criterion. Moreover, to measure the varia-tion of each model, we trained each model on 10 differentrandom seeds. We used Mi to represent the model trainingon different random seed where i = 0 . . . 9. A boxplot withmedian and variation of each Mi is shown to demonstratethe results. All the models are trained based on the meansquared error between estimated and ground truth ratings.All the models are trained with a stochastic gradient descentoptimizer with a 0.05 learning rate. We apply early stoppingif the validation loss does not improve for 100 epochs. Theperformance of all models is measured using the coefficientof determination (R2 score):

R2 = 1 −∑

i (yi − ŷi)2∑i (yi − ȳi)2

, (1)

where yi is the ground truth rating, ŷi is the estimated rating,and ȳi is the average of the ground truth rating. R2 is acommon metric to evaluate the fit of a regression predictionto the ground truth value.

5. RESULTS & DISCUSSION

5.1 Overall Performance

Figure 4 shows the comparative performance for all modelsfor middle school and symphonic band. We can makethe following observations (with independent t-test resultsreported):

(i) We compare the performance of various modelstrained on different band performances. All systemsperform better (higher R2 value) on the middle schoolrecordings than on the symphonic band recordings(p < 0.01 except JointEmbedNet for musicality). Onepossible explanation for this is that symphonic bandscores are usually more complicated and longer. Forexample, symphonic band scores tend to be performedat high tempo with high note density. The chunkinginto smaller lengths (and the downsampling of thedistance matrix) compared to the score length mightlead to a less accurate mapping to the assessmentrating. An additional factor is that most performersin the symphonic band auditions exhibit greater skilllevel than middle school performers thus making it

Musicality Note Accuracy Rhythmic Accuracy0.30

0.35

0.40

0.45

0.50

0.55

0.60

R-S

quar

ed

PCConvNet (Baseline)SIConvNetDistMatNetJointEmbedNet

(a) Middle School


0.35

0.40

0.45

0.50

0.55

0.60

R-S

quar

ed

(b) Symphonic Band

Figure 4. Box plots showing comparative performance (higher is better) across different models and assessment criteria.

potentially more difficult to model the differences inproficiency levels.

(ii) All score-informed models generally outperform thebaseline, implying that score information is indeedhelpful for MPA (p < 0.01 except SIConvNet for mu-sicality, DistMatNet for musicality and note accuracyon middle school). We notice, however, that the dif-ference between the score-independent baseline andthe score-informed models is smaller for the middleschool than for symphonic band. Given the signifi-cant improvement over the baseline for symphonicband performances (which have complicated scores),we speculate that the score-informed models bene-fit more from access to score information. In otherwords, a score reference becomes more impactful withincreasing proficiency level while the pitch contouralone contains most relevant information for mediumproficiency levels.

(iii) While the two models SIConvNet and JointEmbed-Net both use the same input features, JointEmbedNeteither outperforms or matches SIConvNet in all ex-periments. The main difference between these twoarchitectures is that SIConvNet simply performs aregression to estimate the assessments while JointEm-bedNet learns a similarity in the embedding space tomodel the assessments. Therefore, we can assumethat JointEmbedNet is able to explicitly model thedifferences between the input pitch contour and scoreespecially in the case of symphonic band where thescores are more complicated.

(iv) We observe that while DistMatNet and JointEmbed-Net both utilize the similarity between the score andpitch contour, albeit at different stages of the network,JointEmbedNet typically performs better across cate-gories and bands, and the gap is larger for musicalitythan for the other two categories. It is possible that theabsolute pitch at the input may be important for thefinal assessment (octave jumps, for example, wouldnot be properly modeled in the distance matrix). Morelikely, however, is that the significantly larger input

dimensionality of the matrix (compared to the alignedsequences for JointEmbedNet) negatively impacts per-formance. Most of the relevant information for MPAcenters around the diagonal of the distance matrixwith relatively small deviations depending on the stu-dents’ tempo variation. Most of the distance matrixelements far from the diagonal contain redundant orirrelevant information, thus complicating the task. An-other advantage that JointEmbedNet might have overDistMatNet in terms of overall assessment is that thedistance is computed on the whole performance whileDistMatNet computes a frame-level pitch distance,potentially complicating the task for overall qualitymeasures like musicality.

5.2 Chunk Size

In this experiment, we look at the impact of two differentchunk sizes for the first two methods. Figure 5 shows theresults on middle school (a) and symphonic band (b). Forboth SIConvNet and JointEmbedNet, a chunk size of 10 soutperforms that of 5 s across all the bands.

Chunking with random sampling is a form of data aug-mentation. By using the ground truth rating of the wholeperformance, the chunks are assumed to reflect the qualityof the whole performance. The results show that 5 s chunksmight be too short to evaluate the whole performance while10 s chunks are much better suited regardless of categoryand score complexity. Chunk lengths greater than 10 s werenot tested because we restricted ourselves to the length ofthe shortest performance in the dataset. Consequently, weused a 10 s chunk size for the experiment in Figure 4.

5.3 Distance Matrix Resolution

In this experiment, we study the impact of the different inputmatrix resolutions 400 × 400, 600 × 600, and 900 × 900,for the DistMatNet model. The results for both middleschool and symphonic band are shown in Figure 6. First,the performance of rhythmic accuracy criterion tends to de-crease with increasing distance matrix resolution. It might


0.35

0.40

0.45

0.50

0.55

0.60R

-Squ

ared

SIConvNet-chunk size: 5 secsSIConvNet-chuck size: 10 secsJointEmbedNet-chunk size: 5 secsJointEmbedNet-chuck size: 10 secs

(a) Middle School


0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

R-S

quar

ed

(b) Symphonic Band

Figure 5. Box plots showing comparative performance(higher is better) across different chunk sizes for SIConvNetand JointEmbedNet.

be more difficult for the same model structure to capturethe complexity inside a larger matrix. This can also explainthe result for middle school: although increasing the inputresolution from 400 × 400 to 600 × 600 will capture moredetails, the performance decreases when the matrix resolu-tion is further increased. Second, an input matrix size of600 × 600 leads to a slightly higher average score (0.46)on both symphonic and middle school than the other tworesolutions (0.45 for 400×400 and 0.43 for 900×900). Weended up using the 600× 600 resolution for the experimentin Figure 4.

6. CONCLUSION

This paper presents three novel neural network-based meth-ods that combine score information with a pitch representa-tion of an audio recording to assess a music performance.The methods include: (i) a CNN with aligned pitch contourand score as the input, (ii) a joint embedding model thatlearns the assessment as the cosine similarity of the embed-dings of both the aligned pitch contour and the score, and(iii) a distance-matrix based CNN, using a differential repre-


0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

R-S

quar

ed

matrix size: 400matrix size: 600matrix size: 900

(a) Middle School


0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

R-S

quar

ed

(b) Symphonic Band

Figure 6. Box plots showing comparative performance(higher is better) across different matrix sizes for the dis-tance matrix network (DistMatNet).

sentation of pitch contour and score at the input. The resultsshow that all the methods outperform the score-independentbaseline model. The joint embedding model achieves thehighest average performance.

Beyond the obvious applications in software-based mu-sic tutoring systems, score-informed performance assess-ment models (and objective MPA in general) can benefitthe broader area of music performance analysis. Models ca-pable of rating performances along different criteria couldserve as useful tools for objective evaluation of generativesystems of music performance. In addition, such mod-els could also be explored for objective analysis of inter-annotator differences in rating music performances.

In the future, we plan to incorporate timbre and dynamicsinformation into the models as it has been shown to improveaccuracy [22]. This will also enable the model to assessperformances in terms of tone quality, the criterion ignoredin this study. We also plan to investigate other instrumentsand to examine cross-instrument relationships by traininginstrument-specific models. Furthermore, the musical scorereference could be replaced with other representations suchas the pitch contour of a highly-rated performance.

7. ACKNOWLEDGMENT

We would like to thank the Florida Bandmasters Associa-tion for providing the dataset used in this study. We alsogratefully acknowledge Microsoft Azure who supportedthis research by providing computing resources via the Mi-crosoft Azure Sponsorship.

8. REFERENCES

[1] B. Bozkurt, O. Baysal, and D. Yuret. A dataset and base-line system for singing voice assessment. In Proc. ofInternational Symposium on Computer Music Multidis-ciplinary Research (CMMR), pages 430–438, Matosin-hos, Porto, 2017.

[2] Alice Cohen-Hadria and Geoffroy Peeters. Music struc-ture boundaries estimation using multiple self-similaritymatrices as input depth of convolutional neural net-works. In Proc. of Audio Engineeing Society (AES) In-ternational Conference on Semantic Audio, Erlangen,Germany, 2017.

[3] Johanna Devaney, Michael I Mandel, and Ichiro Fu-jinaga. A Study of Intonation in Three-Part Singingusing the Automatic Music Performance Analysis andComparison Toolkit (AMPACT). In Proc. of 13th Inter-national Society of Music Information Retrieval Confer-ence (ISMIR), Porto, Portugal, 2012.

[4] Sebastian Ewert and Mark B Sandler. Structureddropout for weak label and multi-instance learning andits application to score-informed source separation. InProc. of IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP), pages 2277–2281, New Orleans, USA, 2017.

[5] Sebastian Ewert, Siying Wang, Meinard Müller, andMark Sandler. Score-informed identification of missingand extra notes in piano recordings. In Proc. of 17thInternational Society for Music Information RetrievalConference (ISMIR), New York City, USA, 2016.

[6] Felipe Falcao, Baris Bozkurt, Xavier Serra, NazarenoAndrade, and Ozan Baysal. A Dataset of Rhythmic Pat-tern Reproductions and Baseline Automatic AssessmentSystem. In Proc. of 20th International Society for MusicInformation Retrieval Conference (ISMIR), Delft, TheNetherlands, 2019.

[7] Thomas Grill and Jan Schluter. Music boundary detec-tion using neural networks on spectrograms and self-similarity lag matrices. In Proc. of 23rd European Sig-nal Processing Conference (EUSIPCO), pages 1296–1300, Nice, France, 2015.

[8] Siddharth Gururani, Ashis Pati, and Alexander Lerch.Analysis of objective descriptors for music performanceassessment. In Proc. of International Conference onMusic Perception and Cognition (ICMPC), Montréal,Canada, 2018.

[9] Yoonchang Han and Kyogu Lee. Hierarchical approachto detect common mistakes of beginner flute players. InProc. of 15th International Society of Music Informa-tion Retrieval Conference (ISMIR), pages 77–82, Taipei,Taiwan, 2014.

[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and JianSun. Deep residual learning for image recognition. InProc. of IEEE Conference on Computer Vision and Pat-tern Recognition (CVPR), pages 770–778, Las Vegas,USA, 2016.

[11] Jiawen Huang and Alexander Lerch. Automatic assess-ment of sight-reading exercises. In Proc. of 20th Inter-national Society for Music Information Retrieval Con-ference (ISMIR), Delft, The Netherlands, 2019.

[12] Sergey Ioffe and Christian Szegedy. Batch normaliza-tion: Accelerating deep network training by reducinginternal covariate shift. In Proc. of 32nd InternationalConference on Machine pyin (ICML), pages 448–456,Lille, France, 2015.

[13] Trevor Knight, Finn Upham, and Ichiro Fujinaga. Thepotential for automatic assessment of trumpet tone qual-ity. In Proc. of 12th International Society of Music Infor-mation Retrieval Conference (ISMIR), pages 573–578,Miami, USA, 2011.

[14] Alexander Lerch, Claire Arthur, Ashis Pati, and Sid-dharth Gururani. Music performance analysis: A sur-vey. In Proc. of 20th International Society for MusicInformation Retrieval Conference (ISMIR), Delft, TheNetherlands, 2019.

[15] Pei-Ching Li, Li Su, Yi-Hsuan Yang, Alvin WY Su,et al. Analysis of expressive musical terms in violin us-ing score-informed and expression-based audio features.In Proc. of 16th International Society of Music Infor-mation Retrieval Conference (ISMIR), pages 809–815,Málaga, Spain, 2015.

[16] Matthias Mauch and Simon Dixon. pYIN: A fundamen-tal frequency estimator using probabilistic threshold dis-tributions. In Proc. of IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP),pages 659–663, Florence, Italy, 2014.

[17] Oscar Mayor, Jordi Bonada, and Alex Loscos. Perfor-mance analysis and scoring of the singing voice. In Proc.of Audio Engineering Society (AES) Convention, pages1–7, London, UK, 2009.

[18] Marius Miron, Jordi Janer Mestres, and EmiliaGómez Gutiérrez. Monaural score-informed source sep-aration for classical music using convolutional neuralnetworks. In Proc. of 18th International Society for Mu-sic Information Retrieval Conference (ISMIR), Suzhou,China, 2017.

[19] Emilio Molina, Isabel Barbancho, Emilia Gómez,Ana Maria Barbancho, and Lorenzo J Tardón. Funda-mental frequency alignment vs. note-based melodic sim-ilarity for singing voice assessment. In Proc. of IEEEInternational Conference on Acoustics, Speech and Sig-nal Processing (ICASSP), pages 744–748, Vancouver,Canada, 2013.

[20] Tomoyasu Nakano, Masataka Goto, and Yuzuru Hi-raga. An automatic singing skill evaluation method forunknown melodies using pitch interval accuracy and vi-brato features. In Proc. of International Conference onSpoken Language Processing (INTERSPEECH), pages1706–1709, Pittsburg, USA, 2006.

[21] Caroline Palmer. Music performance. Annual review ofpsychology, 48(1):115–138, 1997.

[22] Ashis Pati, Siddharth Gururani, and Alexander Lerch.Assessment of student music performances using deepneural networks. Applied Sciences, 8(4):507, 2018.

[23] Jordi Pons Puig, Rong Gong, and Xavier Serra. Score-informed syllable segmentation for a cappella singingvoice with convolutional neural networks. In Proc. of18th International Society for Music Information Re-trieval Conference (ISMIR), Suzhou, China, 2017.

[24] Oriol Romani Picas, Hector Parra Rodriguez, DaraDabiri, Hiroshi Tokuda, Wataru Hariya, Koji Oishi, andXavier Serra. A real-time system for measuring soundgoodness in instrumental sounds. In Proc. of AudioEngineering Society Convention, Philadelphia, USA,2015.

[25] Hiroaki Sakoe and Seibi Chiba. Dynamic ProgrammingAlgorithm Optimization for Spoken Word Recognition.Transactions on Acoustics, Speech, and Signal Process-ing, 26(1):43–49, 1978.

[26] Sam Thompson and Aaron Williamon. Evaluating eval-uation: Musical performance assessment as a researchtool. Music Perception: An Interdisciplinary Journal,21(1):21–41, 2003.

[27] Amruta Vidwans, Siddharth Gururani, Chih-Wei Wu,Vinod Subramanian, Rupak Vignesh Swaminathan, andAlexander Lerch. Objective descriptors for the assess-ment of student music performances. In Proc. of AudioEngineeing Society (AES) International Conference onSemantic Audio, Erlangen, Germany, 2017.

[28] I-Chieh Wei, Chih-Wei Wu, and Li Su. Generating str-cutured drum pattern using variational autoencoder andself-similarity matrix. In Proc. of 20th InternationalSociety for Music Information Retrieval Conference (IS-MIR), Delft, The Netherlands, 2019.

[29] Brian C Wesolowski, Stefanie A Wind, and GeorgeEngelhard. Examining rater precision in music perfor-mance assessment: An analysis of rating scale structureusing the multifaceted rasch partial credit model. Music

Perception: An Interdisciplinary Journal, 33(5):662–678, 2016.

[30] Chih-Wei Wu, Siddharth Gururani, Christopher Laguna,Ashis Pati, Amruta Vidwans, and Alexander Lerch. To-wards the objective assessment of music performances.In Proc. of International Conference on Music Percep-tion and Cognition (ICMPC), pages 99–103, San Fran-cisco, USA, 2016.

[31] Chih-Wei Wu and Alexander Lerch. Learned featuresfor the assessment of percussive music performances.In Proc. of International Conference on Semantic Com-puting (ICSC), Laguna Hills, California, USA, 2018.

score-informed networks for music performance …...score-informed models as compared to...

Documents