· web view[48] "itu-r bs.1387-1: method for objective measurements of perceived audio...

D7.1 Initial User Requirements & Evaluation Procedurespublic

1

D7.1Initial User Requirements &

Evaluation Procedures

Abstract

This document describes the baseline initial user requirements and evaluation procedures for the EASAIER project. It describes essential information for Work Package 7: Evaluation and Benchmarking. It is based upon previous user requirement studies and recently published work from the Arts and Humanities research sector. It also takes into account the results of interviews and visits to sound archives from across Europe. The user requirements form the basis of the function specification for the EASAIER system, and in turn determine criteria and standards for evaluation. Testing and evaluation methods are reviewed and mapped against EASIAER outcomes, with an indicative schedule. The Evaluation Procedures will guarantee quality assurance in development of the EASAIER system.

Version 2.6Date: 01/02/2007Editor: Josh ReissContributors: Celia Duffy, Joseph Harrop


Table of ContentsTable of Contents................................................................................................................................................................2

List of Tables.......................................................................................................................................................................3

List of Figures.....................................................................................................................................................................3

1. Executive summary of document............................................................................................................................4

2. The current context: exemplar digital sound collections......................................................................................5

2.1. The current context............................................................................................................................................5

2.2. Exemplars..........................................................................................................................................................5

3. User requirements for digital sound archives........................................................................................................7

3.1. Previous user requirements studies...................................................................................................................8

3.2. User needs and EASAIER deliverables.............................................................................................................9

4. EASAIER testing and evaluation..........................................................................................................................13

4.1. Listening Tests: Guidelines for subjective evaluation of audio processing components................................13

4.2. The testing and evaluation cycle.....................................................................................................................14

4.3. EASAIER testing and evaluation: objectives and roles..................................................................................154.3.1 Defining Objectives....................................................................................................................................154.3.2 Defining Roles............................................................................................................................................15

4.4. User groups.....................................................................................................................................................16

4.5. EASAIER partner in-house testing scenarios..................................................................................................174.5.1 Component testing at DIT..........................................................................................................................174.5.2 Listening test procedures at QMUL...........................................................................................................19

4.6. Expert User Advisory Board (EUAB).............................................................................................................21

4.7. Criteria for testing and evaluation..................................................................................................................23

5. Implementation schedule and reporting mechanisms.........................................................................................25

5.1. Schedule of testing and evaluation, with dependencies..................................................................................26

5.2. Sample testing and evaluation template for an EASAIER component............................................................27

5.3. Reporting testing and evaluation activity........................................................................................................27

6. Conclusion...............................................................................................................................................................28

7. References................................................................................................................................................................29

Appendices........................................................................................................................................................................32

Appendix 1: Samples of initial HOTBED User Needs Analysis Instruments (2001)...................................................32

Appendix 2: British Library ASR Project Usability Survey, 2006..............................................................................36

2


List of Tables

Table 1. Harmonica user needs, exemplars fulfilling those needs and EASAIER functionalities.......8Table 2. EASAIER roles and testing/evaluation activities.................................................................16Table 3. EASAIER end-user and sub end-user groups......................................................................16Table 4. DIT listening test..................................................................................................................18Table 5. EUAB member outline.........................................................................................................21Table 6. Testing and evaluation methods...........................................................................................23Table 7. Areas of investigation and specific questions for testing and evaluation............................24Table 8. EASAIER products/activities, question areas and testing/evaluation methods...................25Table 9. Indicative schedule of testing/evaluation activity................................................................26Table 10. Populated testing/evaluation template for D5.2, Time Scale Modification Tool...............27

List of Figures

Figure 1. EASAIER testing and evaluation cycle..............................................................................15Figure 2. Screenshot of the MUSHRAM interface............................................................................20Figure 3. Subjective ratings of the anchors. Bars indicate 95% confidence intervals........................20Figure 4. Subjective ratings of FDICA, DUET and ASB on speech mixtures with different

reverberation times and signal-to-noise ratios. Bars indicate 95% confidence intervals...........21

3


1. Executive summary of documentThere are two main objectives of this report:

to place EASAIER in its wider context and relate the results of previous user requirements studies to EASAIER products; and

to discuss testing and evaluation methods and procedures that will be employed by WP7: Evaluation and Benchmarking.

Section 2 is concerned with the wider context. After discussion of the current stage of development of digital sound archives, this section presents seven exemplar collections which have carried out user evaluation studies. These collections represent the present state of digital sound archives and their access systems. The manner in which their functionalities relate to the needs of their respective target audiences provides a background for the EASAIER user-to-functionality relationship, which is discussed in Section 3.

In Section 3 we briefly discuss previous user requirement studies, noting any significant outcomes, and map these outcomes to the exemplars noted in Section 2 and EASAIER functionalities. This allows for an appreciation of how the user requirements that inform EASAIER functionalities relate to user requirements of other projects and commercial applications. It also further clarifies the user contexts in which EASAIER functionalities will be employed.

Section 4 considers testing and evaluation objectives, outlining the iterative nature of testing and evaluation cycles and the roles of EASAIER partners in this process. It maps various methods to EASAIER products, with accompanying rationale. Classifications of the EASAIER user groups are given here, with special consideration of the role of the Expert User Advisory Board (EUAB). Case scenarios relating to EASAIER partner in-house testing of functionalities will also be presented here.

Section 5 discusses the complex partner-dependent scheduling issues. Having several partners working on EASAIER functionalities exclusively as well as in various modes of collaboration, alongside maintaining our association with the EUAB and target user groups, makes for a intricate corporate working relationship with regard to levels of formative testing and evaluation. Here, we propose a draft implementation schedule for formative and summative testing activity, and describe our reporting mechanisms with EASAIER partners and the EUAB. The testing and evaluation template for a specific EASAIER component is also given here.

There are several appendices which provide example questionnaires or surveys from HOTBED and the ASR project. They will be the basis for similar surveys in later stages of the EASAIER project.

4


2. The current context: exemplar digital sound collectionsIn this section we review the current context for digitised sound collections and present seven exemplar collections which have carried out user evaluation studies.

2.1. The current context Digitised sound collections have reached an interesting moment in their development. It is only in recent years that it has become technically feasible to hold large amounts of digital sound data and to make it accessible to a wide audience. Up until now sound archive research has been a fairly niche activity, limited to those enthusiasts who could navigate specialised, non-standard cataloguing systems and then negotiate the practicalities of access to the sound content via a wide variety of carrier formats, some very delicate.

This state of affairs has changed – and is still changing – rapidly. There is a growing number of important sound collections being digitised, several of which will be discussed in this report. There is also a growing number of networked access services, which have moved successfully beyond the initial project stage. UK examples include the JISC’s Film and Sound Online Culverhouse Collection[1] available to UK Higher and Further Education institutions, as well as commercial subscription services such as the Naxos Music Library[2] or the Classical Music Library[3].

This change has seen the role of the sound archive and sound archivist evolve dramatically due to the advances made in technology and to demands on behalf of the sound archive user. For example, the Radio Television Eire archive (RTÉ) maintains that it now performs a dual role within the company, on the one hand as curator of collections to the public and on the other with the responsibility of maintaining the high standards of librarianship of a national broadcast archive. With access to digital material so easy via the internet, issues of copyright and licensing are matters of heated debate amongst sound archive managers.

Personal digital audio collections are now commonplace. People carry their digital repositories of sound and music in their back pockets, downloading and uploading content with ease. However, there seems to be just as much evidence that, at least for the Higher Education learning and teaching community, audio collections are still relatively underused [4, 5]. In the research community, there is a growing awareness of the potential of digital resources. However, a recent study found that current technical facilities at the majority of universities and colleges are significantly less advanced than those of a few leading institutions, thereby inhibiting greater uptake of these resources[6].

The music community which uses audio as primary source material is used to accessing content in any manner possible, for as long as possible. But even the growing musicological interest in historical recordings – for example, the AHRC Research Centre for the History and Analysis of Recorded Music (CHARM)[7] – is compromised by vagaries of the marketplace. For example, recordings do not necessarily remain available forever from record company catalogues. The provision of stable, available, sustainable, quality-assured digital audio collections is a tempting prospect. Examples of such collections are discussed below, and as will be seen, these exemplars cater to various user communities.

2.2. Exemplars

There have been several user evaluation studies with regard to digital collections. However, the number of evaluation studies specific to sound archive use and usability is relatively small. These exemplars have been chosen because:

They are functioning digital sound archives; They have carried out and published user evaluation studies during their development.

They are presented here, regionally arranged, to establish a context for this report and provide practical background information concerning the state of digital sound archives today. First, there are two significant ‘early adopter’ examples from the US. Four very different UK examples are then presented. Finally, there are French and Finnish broadcast archives, and IRCAM. Features of the exemplars will be compared to similar ones in the EASAIER project, for example, enriched access tools and user evaluation methods. A number of these exemplar projects and services are members of EASAIER’s Expert User Advisory Board (see Section 4.6 below).

5


American Memory (2000), US Library of Congress (USA)

The American Memory was one of the first major audio collections to be made accessible online, and continues to lead the way in this area. With the aim of documenting the ‘American experience’, the project provides free and open access via the web to sound recordings, still and moving images and printed material including maps and sheet music. These materials are taken from the collections of the Library of Congress and other institutions, and serve as a public resource for education and lifelong learning.More than 9 million items documenting U.S. history and culture are organised into more than 100 thematic collections based on their original format, their subject matter, or who first created, assembled, or donated them to the Library. For the extensive audio collections, playback of sound recordings is available for users, at three levels of sound quality. Downloading of sound recordings is not available.

Variations2 (2000) and Variations3 (2005), Indiana University Digital Music Library (USA)

Variations has led the field in digital music library developments for a number of years, providing online access to selected recordings and scores from the Indiana University Cook Music Library. As the first major digital recorded music library project, it has inspired several similar projects, such as University of Surrey’s PATRON project. Variations2 features several useful tools that allow for searching, annotation and marking of a recording, analysis of musical form using the ‘timeliner’ tool, and the creation of custom playlists. A ‘score with audio’ function enables concurrent listening and reading of records online (including automatic page turning). The Variations3 project builds on the work of its predecessor. In September 2005, the Indiana University Digital Library Program was awarded a grant to extend its online learning and research tools to be deployed at a wide range of US college and university libraries. At the completion of the three-year project, American institutions will be able to introduce, expand or upgrade their current online music archives to include those features of Variations2 alongside any developments of the new project.

The HOTBED Project (2001–4), Royal Scottish Academy of Music and Drama (UK)

HOTBED (Handing on Tradition By Electronic Dissemination) was a JISC funded project headed by staff at Glasgow's Royal Scottish Academy of Music and Drama (RSAMD). HOTBED staff developed a system allowing staff and students in the RSAMD's BA in Scottish Music course and staff and students at Edinburgh University's School of Scottish Studies quick and useful access to digital sound and video resources online. These resources were pooled from both tutors' performances and collections and existing archive materials and made available through HOTBED for learning and teaching purposes. The HOTBED system currently holds over 43 hours of audio-visual material, sourced from reel-to-reel tapes, 78 rpm records, cassettes, CDs, and DV videos. Supporting materials, such as song transcriptions and hyperlinks are included with some items. In addition to providing a searchable, web based database for the materials, HOTBED features bespoke manipulation tools for musical use. Students and staff can customise any resource they find by 'marking up' the item to allow quick access to points of interest. They can then choose to loop any section within the item for practice or analysis.Organisational tools are also provided, allowing each registered user to gather items together in groups. An intra-system email system is included that allows users to communicate easily and share harvested resources with other users on the system, thereby providing a framework for collaboration. Furthermore, staff can share collected resources with all students on the system by means of ‘public lists’ that they can post and dynamically update as required.

Spoken Word Services (from 2003), Glasgow Caledonian University (UK)

Spoken Word Services is based in the Saltire Centre at Glasgow Caledonian University. Its core aim is to enhance and transform educational experience through the integration of digitised spoken word audio and video into learning and teaching. With project partners the BBC, and Michigan State and Northwestern Universities, Spoken Word Services are attempting to make a substantial portion of the BBC’s radio archive accessible online for educational purposes only.The Spoken Word project, funded as part of the JISC/NSF Digital Libraries in the Classroom Programme, features web-based audio collection and annotation tools. Audio annotation, amongst other online tools, helps students analyse and organise notes about specific materials and allows students and scholars to share their observations.

Culverhouse Collection (from 2006), Film and Sound Online, Joint Information Systems Committee (JISC), (UK)

JISC is the UK academic organisation that supports education and research by promoting innovation in new technologies and by the central support of ICT services. JISC manages research and development programmes in the

6

http://www.nsf.gov/

http://www.jisc.ac.uk/

http://www.gcal.ac.uk/

http://www.gcal.ac.uk/thesaltirecentre/

http://www.rsamd.ac.uk/

http://www.rsamd.ac.uk/

http://www.jisc.ac.uk/


use of ICT in teaching, learning and research and develops services, infrastructure and colllections. JISC also supports user requirements projects within the ICT sector, such as the User Requirement Study for a Moving Pictures and Sound Portal in 2003. The Culverhouse Classical Music Collection is a new collection of music and scores to support media studies, media production, music, the performing arts and creative industries, presented as part of the JISC’s multimedia Film and Sound Online collections. JISC has worked closely with Brian Culverhouse Productions to make this collection of music and supporting materials available online. The Culverhouse Collection is exceptional in that it delivers a large volume of completely copyright-cleared classical music (over fifty hours) from the seventeenth to the twentieth centuries for use by subscribed UK Further and Higher Education institutions. This audio collection allows download in four formats – MP3, .WAV, Windows Media Player and Quicktime. While the collection offers no functions for the manipulation or editing of audio files, the user is free to make changes on the material once downloaded.

Archive Sound Recordings Service (from 2006), The British Library (UK)

The Archival Sound Recordings (ASR) service is the result of a two-year development project to increase access to the Sound Archive's extensive collections. Funded by JISC, ASR gives free access (under license) to over 12,000 recordings to UK Higher and Further Education staff and students. These recordings presently consist of eleven collections of material ranging from Soundscapes from Great Britain and Canada to interviews of important figures in British and European art and design to 750 recordings of Beethoven string quartets from the last 100 years. The recordings’ metadata has been derived from the British Library Sound Archive Catalogue.Academics, teachers, learners and researchers working in Higher and Further Education can play and download the audio files via the Athens access management system. The site was launched in September. Users outside UK Further and Higher Education can browse the site but, due to copyright restrictions, are denied any listening or downloading capabilities.

The Institut National de l'Audiovisuel (INA), (France)

The Institut National de l'Audiovisuel is a public organisation with an industrial and commercial role. Funded by the French state, the INA engages with those responsibilities of a state broadcaster, including the preservation of the national audiovisual heritage, collecting audiovisual programmes, preserving and restoring its archive collections. It aims to improve the already impressive access to its images and sounds within the web environment, using digital technology to the full and making material more readily available. It encourages those in research and experimental activities to work together on heritage projects, fostering the innovative side to production in the creative and research fields.

The IRCAM Digital Sound Archive, (France)

IRCAM has been in the forefront of digital developments and digital thinking and philosophy in music and sound for decades. It comments that digitisation has “brought about a major change in the fundamentals of the whole process, and therefore constitutes a vast opportunity for change, in particular with respect to music, and even more so learned music” and has “led us to redefine at IRCAM the policy of production, use and broadcast of digitised documents, and in particular of sound documents. It doubtlessly affects the process of music creation; the issue of digitisation is therefore not just a question of patrimonial archival and collection production or of access modes, but a reinvention of the whole field of lutherie (or instrument-making).IRCAM is thus in transition from the age of computer music to that of music creation in the era of generalised digitization …” [8].

The YLE Digital Sound Archive (Finland)

Yleisradio (YLE) is Finland's national public service broadcasting company. It is among the first broadcasters to face the challenge of creating a digital archive from radio broadcast material. YLE has adopted a full digital radio archive solution where the digitised material can be located, listened to and transferred to the user by browser-based technology. The archive has found solutions to problems of deteriorating (analogue) and unstable (digital) carriers and has developed a user-friendly system for both radio and music holdings.

3. User requirements for digital sound archivesIn this section we note outcomes from previous user requirement studies, and map them to the exemplars noted in Section 2 and EASAIER functionalities.

7


3.1. Previous user requirements studiesRecent work from Downie, Cunningham and others has provided a clear picture of what end users want from music retrieval systems and the types of queries that they make[9-11]. User needs studies and extensive research by the JISC have identified a number of key features that are necessary to enrich sound archives[5, 12, 13] These features include web-based access, integration of other media, time-stretching functionality, and alignment of scores with audio, among many others.

Studies from within the EASAIER consortium, such as the HOTBED project, on audio archive evaluation and enrichment[14], have also contributed significant findings (e.g. the need for real-time, user-defined mark-up and looping functions). They confirm and build on previous work on user needs for digitized audio collections of speech and music such as the ground-breaking exploratory studies carried out in 1993 by the Library of Congress[15], or in the mid 1990s by the pioneering European-funded Jukebox[16] and Harmonica[17] projects. For example, a user needs study carried out in 1997 for the Harmonica project presents a wish-list of functionalities for a music library, as suggested by MA in Musicology students at the Sorbonne, Paris:

Internet access Access to cd-rom on jukebox Simultaneous listening and score display Computer access to texts Computer access to scores Personal tags on documents Computer access to recordings Recording comparison Excerpts saving Instrument track decomposition Automatic sound to score transcription Graphical sound display

The extent to which digital accessibility has come of age is illustrated below (see Table 1), where the user needs listed on the Sorbonne students’ wish-list are aligned with functionalities found in the exemplars above, alongside those functionalities in the EASIAER project that address these same user needs.

Table 1. Harmonica user needs, exemplars fulfilling those needs and EASAIER functionalities

Sorbonne student user need Exemplar fulfilling this need EASAIER functionalityInternet access All exemplars EASAIER will enable sound archives to be

accessed online.Access to cd-rom on jukebox All exemplars EASAIER D6.1, user interface will play

material.Simultaneous listening and score display

Variations 2&3, IRCAM EASAIER D6.3, cross & multimedia interface will display material in a variety of media.

Computer access to texts American Memory EASAIER D6.3, cross & multimedia interface will display text metadata to audio files, including transcriptions.

Computer access to scores Variations 2&3 EASAIER D5.3, multimedia alignment and enrichment will display scores as metadata if available.

Personal tags on documents Spoken Word Project, HOTBED, Variations2&3, IRCAM, YLE

EASAIER D6.2, cognitive interface will allow users to save and share tags.

Computer access to recordings All exemplars EASAIER D3.1 & 3.2, Music & Speech Retrieval will enable content-based retrieval on speech and music audio material.

Recording comparison Variations2&3 EASAIER D3.1 & 3.2, Music & Speech Retrieval uses content-based searching to produce ranked lists of musically similar material

Excerpts saving Variations2&3, YLE EASAIER D6.2, cognitive interface will enable personal saving and sharing of user interaction with material.

8


Sorbonne student user need Exemplar fulfilling this need EASAIER functionalityInstrument track decomposition IRCAM EASAIER D4.2, sound source separation

singles out instruments and voices from an audio file.

Automatic sound to score transcription

None of the exemplars EASAIER D4.4, transcription of speech and music.

Graphical sound display Variations 2&3, IRCAM EASAIER D5.3, multimedia alignment and enrichment will display audio files in several formats

The Harmonica user report noted that the wishes of the librarians and of the students were different. The librarians were all interested in unified access to materials within the collections, but the students were interested in new ways of engaging with those materials. The EASAIER project, in this case, addresses both the wishes of librarian and student, developing powerful retrieval software alongside enriched access tools for innovative interaction with audio and audio-visual material.

The exemplars discussed above demonstrate that digital libraries have been developing systems for their users to access and interact with digitally stored materials in their collections. While the tools created for this purpose have evolved as technology has advanced, the attitude and approach of their respective user groups has not evolved at a similar rate. The recent UK Arts and Humanities Research Council (AHRC) ICT Strategy Project report into ICT tools for searching, annotation and analysis of audiovisual media, presents a ‘state of the nation’ paper with regard to the usefulness and usability of audiovisual media and ICT tools in arts and humanities research[18]. It makes observations with regard to the audiovisual materials available and the methods by which these materials are accessed. The report also discusses the engagement on behalf of researchers with these methods and materials. It states that researchers could work more effectively with better knowledge of ICT, remarking that ‘a common failing is not so much ignorance of how to use particular tools as a misunderstanding of the process the computer carries out and the validity of its results.’ It continues, ‘the development of strategies to aid scholars in the use or reuse of existing tools is considered by some to be more important than creation of wholly new tools that are specifically designed for the humanities’ [19].

In 2006, the ARHC Methods Network Workgroup convened a group of senior academics, all experienced in the development, support and implementation of projects to develop digital tools in the arts and humanities. Alongside a digital tools ‘wish-list’, the meeting report outlines strategic issues, for example, the importance of focusing on end user needs to ensure that the tools developed are useful to the user community[20]. While this presents a clearer picture of user needs in areas of research, there is a deficit of publications specific to user needs amongst the general public or in other educational fields. Commercial companies carry out extensive user needs studies on their respective target audiences, prior to and during product development. However, as these studies typically hold valuable information concerning potential customers, they are rarely published or available in the public domain.

Within the EASAIER project, knowledge of potential user communities within the general public is small compared to what we know about the research and Higher and Further Education communities. By way of testing, evaluation and information gathering, we hope to gain a clearer appreciation of those users not addressed in previous studies. For example, a member of the general public is likely to interact with a sound archive using the EASAIER software in a different manner than that of a researcher, music student or sound archive enthusiast. These latter groups have already been the focus of a number of user studies while the former, perhaps more unpredictable, group has not.

The easy accessibility of digital sound archives has resulted in new types of user behaviour. The sound archive, in digital format, is no longer a source for users of a small number of specialist items. Rather, users will now employ online access as a means of ‘browsing’ potential areas and objects of interest. This is evident in commercial websites where customers are presented with a list of products in addition to their initial purchase, based on similar product features - the “Other people who bought this product, also bought … ” marketing ploy, but which may also have an effect on researcher users’ behaviour.

3.2. User needs and EASAIER deliverables

This section discusses each EASAIER deliverable (D3.1, D3.2 etc.) that has specific impact on the user experience, and outlines the user need or demand that prompted that feature’s development. It also notes commercial software products with similar features.

9


Music retrieval (D3.1): According to the AHRC ICT Strategy Project report, content-based searching is ‘still beyond the horizon for music’ and cites (as regards query by humming interfaces) ‘very significant technical issues to be addressed before this can be achieved and questions about the degree to which it would ever be a simple-to-use and effective tool’[18]. This assertion is contradicted by recently built content-based searching systems using musical features that approach commercial quality (see for example [21]). Overall, the field has also moved beyond the niche area of query-by-humming. This demonstrates a mismatch in understanding of the state of the art between the arts and humanities research community and the audio engineering development communities.

The EASAIER music retrieval function allows users to input an excerpt of an audio file into the search engine, and have a list of tracks that are the same or similar (i.e. displaying similar characteristics) presented to them. This function works on searching and organising audio files according to their relevance to music-related queries, using various features to produce a ranked list of audio files related to an audio query through various metadata and automatically extracted features.

This functionality caters to the ‘I know what I like when I hear it’ user approach. Presenting the user with a list of tracks, ranked according to the users’ own parameters, encourages further engagement and interaction with the available materials. An example is the commercial mobile phone and internet-based service, Shazam – a real-time song identification service[22]. When users hear a song they like, they dial a four-digit code from their mobile phone, whereupon Shazam records a short audio file via the phone connection. Users then receive an SMS text message from Shazam with the name of the track and the artist. Following identification of the desired track, the tune is ‘tagged’ and users can then select from an array of options to interact with the song, including access to a personalised, stored ‘tag list’ on the Web. Users have the ability to share an excerpt of the song, and/or purchase the single or album directly from their phone or online. Shazam works only with popular music.

An important distinction between Shazam and EASAIER us that Shazam has an exact match content-based search to label the song and a user tagging system for recommendation, whereas EASAIER uses content-based searching to produce a ranked list of musically similar material.

The music retrieval function will also search on manually entered and automatically generated metadata.

Speech retrieval (D3.2): This function alleviates the dependence on transcription of spoken word material for retrieval purposes and eliminates the need to listen to the entirety of an audio track, or set of audio tracks, to source desired material. Content-based searching in speech is also addressed in the AHRC ICT Strategy Project report [18]. This is an example of what researchers in the humanities view as the use of technology to automate what users are already doing, but doing it better, faster and more cheaply.

The speech retrieval function searches for the occurrence of a word or phrase within an audio file, including those audio files not fully transcribed. This function will enable users to quickly search and find material relevant to their requests within sources that previously required a great deal of time to investigate thoroughly.

Cross media information retrieval (D3.3): The JISC User Requirements Study for a Moving Pictures and Sound Portal identified the need to have ‘cross-searching between still and time-based collections’[5]. The HOTBED user study identified the use of video as having a strong impact in aural learning. The AHRC ICT Methods Network expressed the wish for a tool that links visual and graphic media to text resources[20].

The EASIAER cross media information retrieval functionality allows the user to enter a piece of media as a search query and the system will retrieve results relating to the search, but including types of media different to that entered as the query.

Vocal query interface (D3.4): The obvious uses for accessibility and typing-free word processing are reflected in the demand for this function as seen in products like ‘Dragon NaturallySpeaking’ – a leading brand speech recognition software. By ruling out spelling mistakes, this EASAIER function can also increase search reliability and accuracy. However, inaccurate speech recognition could negate any advantage with regard to improving spelling.

With this functionality, users can search for words by pronouncing them into a microphone attached to the computer. This function uses a phoneme level speech recognition system. The user can use this function as an alternative to using the keyboard to enter text.

Audio Stream Segmentation (D4.1): Audio and audio-visual material will often have a mixture of music and

speech. Ethnomusicologists have pointed out that field work often includes making recordings that feature both

10


speech and music. The wish for quick access to particular segments of either speech or music in a mixed audio or audio-visual file has been noted within this specialist group[23].

Classifications of speaker (in the case of speech), or segmentation according to structure in a musical audio track will have immediate applications in music learning environments. The time spent scrutinising an audio file will be reduced by employing these automatically generated segments.

This functionality segments audio into its macroscopic elements such as silence, music, or speech, and further into classifications, for example identifying each of the speakers in a mixed speech audio, or defining verses and chorus within a popular song.

Sound Source Separation (D4.2): The AHRC ICT strategy project report noted that audio files with mixed speech, music and noise are difficult to search[18]. The source separation function, in conjunction with audio stream segmentation (D4.1) and sound object identification (D4.3), will help to remedy this situation.

Sound source separation refers to the task of taking a signal containing a mixture of many different sources and extracting each of these individual sources from the mixture. Alongside those wishes of the students in the Harmonica project user study for ‘instrument track decomposition’, the popularity of commercial applications like the ‘Music Minus One’ recordings, and the more recent ‘play-along’ tracks such as those from Smartmusic [24], is evidence of demand for this function. This technology also has applications within the study of ensemble performance practice, where one performer can be extracted from the group for close listening.

Sound Object Identification (D4.3): Recognising individual speaker’s voices will have a large and immediate impact on those spoken word archives featuring a high number of identifiable speakers. This tool has obvious use in the classroom, where school pupils unfamiliar with instruments’ respective timbres will be able to learn in an interactive manner.

This functionality informs users of where in an audio stream a certain person speaks, in addition to this person being identified. This function will automatically identify the types of sources being used within an audio stream. That is, speakers and musical instruments will be automatically recognised and labelled.

Transcription of Speech and Music (D4.4): The AHRC ICT Strategy Project states ‘many researchers make some kind of transcription of audio-visual material, and would value tools to automate this process.’ It also states that ‘full music transcription tools do not exist’ and ‘for speech, such tools exist with important limits to their accuracy’. User demand for automatic speech transcription is high, and can be seen in the development of the HTK Hidden Markov Model Toolkit[25]. Almost 20 years after its initial development, there is continued user demand for this tool, and its subsequent versions. The popularity of commercial applications such as ‘Dragon NaturallySpeaking’ (see above) is further evidence of user demand for speech transcription tools.

Transcription of polyphonic music remains problematic[26, 27]. EASAIER software draws on advances in chord recognition, onset detection, key recognition and extraction of harmonic and melodic contours to develop a more accurate transcription function.

Looping and Marking (D5.1): In the user needs analysis sessions carried out for the HOTBED project, this function, ‘to repeat phrase-by-phrase and to loop phrases’, was specifically identified as a desirable feature[28].

This function automatically marks an audio or audio-video file. Any section so marked may be looped or repeated seamlessly.

Time scale/Pitch scale modification (D5.2): In the HOTBED project user needs analysis sessions, the facility to slow down playback without altering pitch was specified as a desirable feature to aid musicians in their study and analysis[28]. A study for the RSAMD by the ethnomusicologist Peter Cooke observed that ‘often performers can only really satisfy themselves that they have accurately “heard” a performance by checking at slower playback speeds. For example, fiddlers need slow playback in order to be able to note bow changes (especially when visual evidence is not available)’[23].

Cooke observes that users of video material can benefit from time scale modification: ‘slow playback also has real value when dancers are attempting to study and learn steps and gestures and stylistic features associated with a particular dance’[23]]. Time scale modification of audio allows the playback rate of the content to be slowed or speeded to any desired speed without affecting the local pitch content. This is of great importance where the intelligibility of the audio is less than adequate.

11


Pitch scale modification will allow the user to change the pitch of a piece of music at access time. The usefulness of this function is clear with regard to users playing or singing along with audio files.

Multimedia alignment and enrichment (D5.3): The AHRC ICT strategy project report stated that users ‘can

benefit from other sorts of visualisations’[18]. Cooke also observes that a variety of visualization, presentation and file formats can enrich user interaction, for example, displaying the spectral content of notes and phrases, displaying specific pitches in hertz and conversion to MIDI files[23]. The EASAIER project aims to use audio-oriented formats with event-based annotation and extraction of score-related features to provide a comprehensive approach to musical audio representation for users.

Audio archives often contain several different representations of musical content. This content can be placed into three basic categories: ‘score-orientated’, ‘event-orientated’ and ‘audio-orientated’. Dealing with score-oriented content is outside the scope of the EASAIER project, and may be considered a user need that is not addressed. Popular commercial editing environments such as Finale and Sibelius exist which use the open standardized format MusicXML for representing and converting musical scores.

MIDI encoding is typically used for ‘event-orientated’ content. The mapping between the score timescale and event time scale captures an important role of the interpretative musician. Neither XML nor MIDI accurately represents those instruments on which continuous control is exerted, for example, brass and woodwind instruments. Continuously controlled instruments have important variations in loudness, pitch and timbre within a note that cannot be easily captured by a score- or event-based format. The EASAIER system will resolve this issue by treating the events as annotation and mark-up of the audio, such as the storing of markers at note onsets. Thus, the flexibility of audio representation is maintained while the content may still be handled in an ‘event-oriented’ approach.

Sound Enhancement (D5.4): In an interview with staff at the Irish Traditional Music Archives, the option for noise reduction at access time was seen as a desirable feature[29]. Giving the user choice to enhance the audio material is in keeping with current practice among ethnographic disciplines, where ‘untouched’ audio is viewed to have inherent qualities that are lost in the process of noise reduction. For example, ITMA keep two kinds of digitised audio files from its collections of 78’s and wax cylinders, one with noise reduction and one without.

EASAIER proposes a multipurpose noise reduction and sound enhancement system allowing the user to improve the playback quality at access time.

Operational Interface Design (D6.1): The EASAIER interface will offer a large number of functions in an intuitive manner. User evaluation will play an influential role in the development of this function.

As the report into the Alexandria Digital Library Project at the University of California puts it, ‘we are not building something as simple, nor as functionally static, as a door handle. For a digital library – specifically, the interface to a digital library – it is an iterative process of discovering the conceptual models …’[30].

Cognitive Interface (D6.2): The AHRC ICT Strategy Project report views ‘the main problem for researchers’ being ‘how to locate the material of interest in the vast quantity available, and how to organise material once collected’[18]. This function is a step towards resolving this problem. EASAIER will develop audio-visual signal representation, manipulation and management tools allowing users to create, exchange and consume multimedia data easily. Using a cognitive approach, EASAIER will offer transparent access to music resources to users with differing levels of musical knowledge. An appropriate interface will be proposed depending on the user’s profile, allowing a rapid access to the items of interest, audio features and descriptors.

Cross and Multi-Media interface (D6.3): A report from a recent meeting of the AHRC ICT Methods Network Workgroup stated that ‘an integrated interface can link resources in multiple media types to reveal creative process, especially in relation to “live” or ephemeral forms such as dance and body-based performance practices’ [20].

Integration of retrieval systems will allow users to retrieve multimedia objects through various means from the same interface. As noted above, audio resources often need to be linked to associated materials such as video, images, text, transcripts or scores. This function will offer interactive and intuitive representation and tools for users to easily access and manipulate audio and linked material through the same interface.

Several of the functions above are available in the commercial and academic fields. It could be suggested that there is a large degree of overlap between these and the EASAIER deliverables discussed above. For example, EASAIER’s Time

12


Scale Modification tool is similar to the commercial application ‘The Amazing Slow Downer’ from Roni Music software[31]. Another commercial software package with similar tools as EASAIER is NCH Swift Sound’s ‘WavePad’ [32], which allows for editing audio, adding audio effects and noise reduction. WavePad is a wav and MP3 editor, and supports a number of other file formats. As discussed below in the in-house testing scenario, the EASAIER functions are performance-tested against these commercial and academic ‘competitors’. However, rather than presenting the user with just one or a small number of features, EASAIER brings together a large number of functionalities (some already available, some specifically developed) to build a useful, user-friendly sound archive software package, thereby adding value to most available commercial or academic software packages.

4. EASAIER testing and evaluationIn this section we define testing and evaluation objectives and roles with the EASAIER project. We describe the characteristics of user groups and map testing and evaluation methods to EASAIER functionalities. Criteria for different testing and evaluation methods are discussed.

4.1. Listening Tests: Guidelines for subjective evaluationof audio processing components

Several studies have pointed out that standardized test procedures are crucial to guarantee the accuracy and the reproducibility of the results. Ad-hoc procedures suffer some drawbacks in this respect and better procedures could be obtained by adapting existing standards to this context. This paper intends to provide a tutorial review of the issues regarding this adaptation and some preliminary guidelines for the evaluation of basic audio quality.

Listening tests can answer a wide range of questions, such as finding verbal attributes to describe a set of signals, rating their quality and intelligibility and ordering them by preference. Several organizations, including the International Telecommunication Union (ITU), the European Broadcasting Union (EBU), the International Electrotechnical Commission (IEC), the International Organization for Standardization (ISO) and the Audio Engineering Society (AES), have published standardized procedures for various types of listening tests in other areas of audio signal processing. This section reviews the issues regarding this adaptation and provides some preliminary guidelines for the evaluation of basic audio quality.

4.1.1. Basic audio quality rating

The simplest family of tests aims to quantify the basic audio quality of each signal, that is to provide a single rating embracing all the attributes of perceived quality. Such ratings suffice to find the best algorithm overall among several tested algorithms. Various tests are used depending on the quality level and on the availability of a reference. The standard ITU-R BS.1284-1 [33] describes three rating scales measuring respectively the absolute quality of one signal, the comparative quality of two signals and the quality degradation relative to a reference. A test procedure for signals of intermediate quality is defined in the standard ITU-R BS.1534 [34] called MUSHRA (Multiple Stimulus with Hidden Reference and Anchors). The test consists in rating the quality degradation of some test signals, displayed in a random sequence, relative to a known reference. The rating scale spans numbers between 0 and 100 and contains five intervals labelled “excellent”, “good”, “fair”, “poor” and “bad”. All the signals are presented simultaneously and can be played as often as needed. The test signals include the signals under evaluation, plus the reference and some standardized anchors. Correct identification of the reference (corresponding to a rating of 100) is used to check that the subjects perceive the distortions. Anchors are used to measure absolute degradation. Well-defined standardized anchors are crucial for the accuracy and the reproducibility of the test, since they allow the comparison of ratings obtained in different listening conditions or with different signals. An anchor for coding is defined in MUSHRA as the reference signal low-pass filtered at 3.5 kHz. Anchors for other types of distortion can be defined using basic signal processing tools. The test is repeated for each sequence of test signals, and results are displayed in terms of mean ratings and confidence intervals. In practice, due to the subjective nature of the test, differences between subjects often appear. Some subjects tend to be less critical than others and use only part of the rating scale. To avoid this as much as possible, subjects undergo a prior training consisting in listening to all the signals to learn their whole quality range. Ten to twenty trained subjects are typically sufficient to obtain statistically significant results. Significance can be further improved by using high-quality sound reproduction material, selecting critical test signals and removing outlier subjects (post-screening). Recommendations about these issues are given in [34]. Subjects generally listen to each test signal once or twice only, thus the overall test time can be small.

13


Signals of near-transparent quality are better evaluated using the standard ITU-R BS.1116-1 [35], which aims to determine whether a test signal contains an audible distortion with respect to a reference, without quantifying the amount of distortion. Specific standards exist for narrowband speech signals, such as ITU-T P.800 [36] and its variants for echo cancellation and noise suppression.

4.1.2. Individual attribute rating

During the development of an algorithm, it is often beneficial to get several quality ratings corresponding to multiple perceptual attributes instead of a single rating. Specific parameters can then be tweaked to improve the quality regarding the most critical attributes. Attributes preselected by experts may cover only part of the perceptual structure of the signals and be ambiguous for non-experts. Thus they are often completed or replaced by elicited attributes chosen by a panel of subjects. These attributes can be defined explicitly by words or drawings, or implicitly by their numerical value for each signal. A review of standard elicitation methods is given in [37]. Popular methods include Descriptive Analysis[38], which aims to obtain a common set of verbal attributes through consensus, and Multidimensional Scaling[39], which maps each signal into a point in a low-dimensional space whose axes represent orthogonal attributes based on perceptual similarity ratings. These methods have proved successful for the description of spatial and timbre attributes. In practice, the signals need to be well chosen to avoid the nondetection of weak attributes in the presence of more dominant ones. Once verbal attributes have been fixed, further listening tests can be conducted to rate the quality regarding each attribute. Post-screening is crucial to ensure that all subjects have the same internal definition of the attributes. Advanced statistical post-screening methods used for various applications are reviewed in [40].

4.1.3 Speech intelligibility rating

In the case of speech data, quality is not always proportional to intelligibility, that is the ability to understand what is being said. Some distortions, such as low-pass filtering, can degrade quality but not intelligibility. A review of intelligibility tests for telecommunications and speech synthesis is provided in [41]. Segment-level tests, including the Diagnostic Rhyme Test[42] and its variants, ask subjects to point the word spoken among several rhyming words. Sentence-level tests, including the Harvard Psychoacoustic Sentences[43], require subjects to transcribe full sentences. Note that, in order to avoid biases, reference signals or transcriptions are not provided to the subjects. Similar tests could be devised for music data, based on a score transcription task. Although their results are easily interpretable, intelligibility tests are generally more difficult to implement than quality tests. Also, the preference expressed by the subjects for some signals does not depend on intelligibility only [41].

4.2. The testing and evaluation cycleThe main stages of testing and evaluative activity that will be adopted for EASAIER are represented diagrammatically in Figure 1. This cycle of activity forms the basis of the activity template that will be adopted for all testing and evaluation and is a simplified version of standard usability measures [44]. A sample, populated template for EASAIER Deliverable 5.2 (Time/Pitch Scale Modification) is given in Section 5.2.

Enhancing the user experience and meeting identified user needs is at the heart of the EASAIER project. One of the major project objectives is to study the effectiveness and impact of the system through close attention to evaluation. This is discussed in section 4.3.

The Expert User Advisory Board (EUAB) has a key role as the main source of feedback to EASAIER. EUAB members will participate in formal testing and evaluative activities, in addition to providing advice and guidance throughout the project. Details of EUAB membership and each member’s association with the EASAIER project are outlined in Table5. The RSAMD has experience of conducting through-going user testing and evaluation via the HOTBED project and its recent involvement in the British Library National Sound Archives Archival Sounds Recordings (ASR) project. Instruments from HOTBED and the ASR project are included for information in Appendices 1 and 2.

14


Figure 1. EASAIER testing and evaluation cycle.

4.3. EASAIER testing and evaluation: objectives and roles

When the cook tastes the soup, it’s formative evaluation; when the dinner guest tastes the soup, it’s summative evaluation.

In a formative evaluation, results can be fed back to the designers/developers to move the object under evaluation forward in a different way. In a summative evaluation, the results are intended to give an overall picture at the end of a stage as a fixed point of reference. In the EASAIER context, the summative evaluation at the end of the project (Deliverable 7.2) will measure its success at that fixed point; and formative evaluation activities during the course of the project will be useful to shape and point EASAIER products towards their final form. An indicative schedule is given in Table 8. A mapping of products and activities to questions and methods is given in Table 7.

4.3.1 Defining Objectives

Usefulness and usability is at the heart of the EASAIER project, so the overall objective is to ensure EASAIER products are:

useful and fit for purpose: this will encompass answering needs identified from previous users needs studies in this area, and soliciting additional users needs to fill in gaps and in response to ideas generated from working with users. This will result in more qualitative data.

and secondly,

useable, both by content manager specialists and by non-expert end users: this involves observation and measuring of user interactions with EASAIER tools. This will result in more quantitative data.

This dual qualitative/ quantitative perspective relates to another duality, that of evaluation vs testing. In the EASAIER context, evaluation sets out to answer the more exploratory types of question, e.g. “What sort of users tools do you want?” and is more likely overall to use open, qualitative methods such as interviews, observations and ethnographic methods. Testing, on the other hand, sets out to answer the more controlled types of question, e.g. “Does this work properly?” and is more likely to use closed, quantitative methods such as checklists and fixed activities. This dual approach is further described below.

4.3.2 Defining Roles

The interactions between EASAIER partners are complex, with more than one partner contributing to workpackages. As a normal and routine part of the design and development cycles, partners will be carrying out their own testing and

15


(to a lesser extent) evaluation activities (see Section 4.5 for in-house testing procedures from DIT and QMUL). WP7 expects to contribute to these in a formative sense during the development of modules, prototypes and demonstrators, feeding back results in an iterative process. WP7 will also carry out the major summative evaluation at the end of the project when the complete EASAIER system is in place.

Table 2 describes the roles and different sorts of interactions between WP7 and developer partners for both testing and evaluation activities.

Table 2. EASAIER roles and testing/evaluation activities.

Roles Quantitative/Testing activities Qualitative/Evaluationactivities

Initial phase EASAIER developer partners

Initial functionality tests of tools against identified internal specifications (“Does it work?”) See Section 4.5 for initial tests concerning the time-scale modification and source separation tools.

Design tools according to previously identified needs and through two pre-project meetings with content managers in 2005 (“This is what we think you want”)

Initial usability tests (e.g. interface, users help and instructions). See Section 4.5 as an example for the Time Scale modification tool.Request/suggest testing criteria and objectives for WP7

RSAMD WP7 Defines testing criteria and objectives. See Section 5.2 example for the testing and evaluation template.

Assesses utility of tools according to previously identified needs (in existing literature)

Tests usability with different user groups (“Does it work for these groups?”)

Gathers data on additional user needs and feeds back to developers

Gives feedback to developers on usability (iteratively where possible/appropriate). See Section 5.3.

Evaluates tools with different user groups (“Is this what you want? What else would be useful?”) See Section 4.4.Gives feedback to developers on usefulness

Subsequent phase(s)Developer groups

Amend according to feedback Re-specify/re-design/augment features according to feedback.

RSAMD Re-tests with different user groups Re-evaluates with user groupsFinal phaseRSAMD Final testing Summative evaluation, concentrating on the

EUAB.

4.4. User groups

The user community for sound archives using EASAIER is potentially large and wide ranging. Due to advances in technology, online and open access systems, and public familiarity with digital resources, what was once a specialist area of investigation is now a general area of investigation for anyone at anytime. For testing and evaluation purposes, it is useful to classify the wider user community. The broadest division is that of content provider, represented by the Expert User Advisory Board (EUAB) and end-user (i.e. those who access a sound archive material using an EASAIER system).

Content providers – sound archives – manage a large volume of material from a wide range of subject areas. However, they form only one small community of EASAIER’s users when compared to the number of envisaged end user groups.

Charles Inskip, in his dissertation on the Vaughan Williams Memorial Library, divides end users into four groups that relate to their involvement in folk music[45]. He attempts to make these as mutually exclusive as possible. These four end-user groups, and their respective sub-user groups, cover the potential target audience for EASAIER content providers. As such, they will be adopted in this report and our testing and evaluation methodology (see ).

16


Table 3. EASAIER end-user and sub end-user groups.

Performer Musician, singer, dancer, actor, student, composer, author

Academic Teacher, lecturer, student, school

Professional Journalist, media researcher, record company, historian, concert promoter

Enthusiast Recreational user, shopper, music consumer At the RSAMD, we have in-house access to, and institutional contact with, representatives from all the above user sub-groups. We have direct access to musicians (both classical and traditional), actors, lecturers, academics, concert promoters and composers. Moreover our close association with the Spoken Word project gives us access to users, external to RSAMD, of a speech-based sound archive. While the EUAB is EASAIER’s primary test user group, we will also address the end user sub-groups as much as is practical. We believe this is important, as the American Memory user evaluation report observes:

… the sound of the recordings awakens an interest in the topic represented; it is these motion and auditory collections that seem most interactive. Thus, the ‘format’ of the collection is not viewed in a library sense, but as the thing that gives added value to the content. Multimedia's capacity to draw users into the content of the collections is particularly valued in school and public library environments[15].

Inskip’s groups and sub-groups do not address the different subject areas end users might be interested in. This range of potential subjects is very wide indeed and impossible to sample comprehensively. We will take advice from our Expert User Advisory Board in this respect and also consider inviting general web-based calls for feedback.

4.5. EASAIER partner in-house testing scenariosThe first stage of testing of any EASAIER system component is done internally by the main partner involved in development. This is proof-of-concept and feasibility testing. For most components, it relies on comparison of automatically extracted features or retrieved content against a detailed annotation of content in an established testbed. For this reason, ALL maintain a testbed of transcribed English language broadcast recordings, supplied by the Spoken Word project, and NICE maintain their own internal transcribed collection of Call Centre recordings. A wide variety of music collections are also used by DIT and QMUL for testing the extraction of musical features and the functionality of the music retrieval system.

However, for the evaluation of audio processing methods which modify the content or result in imprecise or highly subjective labels, a combination of subjective and objective evaluation procedures are often used. These methods include the enriched access tools described in Work Package 5, as well as some Sound Object Identification methods, such as separation of speakers or musical instruments. In this section, we describe two procedures that have been employed by EASAIER partners when rigorous in-house listening tests are required.

4.5.1 Component testing at DIT

This testing scenario is an example of in-house, developer testing protocols. The EASAIER team at Dublin Institute of Technology, who lead WP5, Enriched Access Tools, supplied this test scenario.

These tests assess the technical functionality of the tool, the interface controls, and compare the tool to similar tools from other developers using blind tests. This forms the first phase of testing and after this in house testing phase, RSAMD takes the tool to ‘real’ users and follows up on both technical aspects with expert, sharp-eared musicians (e.g. for detection of artifacts) and presentational aspects. RSAMD also gathers qualitative data on the usefulness of this tool in different user scenarios, as described in Section 5.2.

EASAIER team at DIT, Ireland:

We usually do two levels of testing, the first level of testing being a combination of subjective and analytical. We use standard metrics to ensure that our algorithm is behaving technically as we expect but our ears will sometimes catch an error we couldn’t have found easily using analytical methods. As we build and design the algorithms we constantly check the results by running test signals through them and audition the results. We use a synthetic test signal with which we can predict the output, for example, white noise, impulses and sinusoids. We then compare our perditions with the actual algorithm outputs.

17


Sometimes our processes inherently introduce artifacts, those undesirable sounds that are a direct result of our processing. We need to ensure that the artifacts do not negate the benefits of the using the algorithm in the first place, for example, we can build a noise reduction algorithm that completely removes all the hiss but introduces a single high pitch whistle which is even more annoying than the original hiss. We try to avoid these sorts of artifacts where possible, but sometimes it’s not possible, so we look to the RSAMD to inform us when the ‘disease’ is better than the ‘cure’.

To focus on a particular function, namely the Time Scale Modification, we ask ourselves these questions:

Does it successfully slow down or speed up the audio?

Is the quality of the slowed audio satisfactory?

Is there any perceived change in the tonal quality or timbre of the audio? If yes, this change is an artifact.

Can we improve upon this artifact?

Is there any amplitude modulation (volume pulsing up and down)?

Is there any frequency modulation (tonal content distortion)?

Are the transients smeared? This is a problem in time scaling. When employing large slow down factors, sharp, percussive sounds – for example, drums – are changed from snappy and punchy to floppy and mushy. This is called transient smearing.

Is the quality of the time scaling the same regardless of the time scaling factor?

A major question is whether anything other than the perceived tempo has been altered. If yes, then it is an artifact. We can then try to get rid of this artifact.

Are the interface controls responsive? Does using them feel like real-time interaction?

Is this level of control sufficient or insufficient for real users?

Would visual feedback – for example, graphs – help?

Are there any other ways in which we can make it better?

Once we’ve answered all these questions we then set up a series of single blind experiments. We take approximately 10 representative examples of various types of signals, including ones we know will be problematic for the algorithms: male and female speech, and music of all sorts (monophonic, polyphonic, percussive and tonal, etc.). We then run all these text signals through our algorithm and also the best of the rest in academia and the commercial world. The parameters for each algorithm are the same.

We construct a listening test (see Table 4) for each example, allowing the user to hear the original signal and two differently scaled versions of the same signal – one of them our version. These versions are labelled either 1 or 2. The user is allowed to listen to the original and either result as often as they like. We then ask users to judge which they prefer between our algorithm and another, the user not knowing which is which. The users most often pick one as being preferable after listening to each result a few times. We also give the users the option of specifying that the algorithms are indistinguishable.

A typical table of test results is depicted in Table 4.

Table 4. DIT listening test.

1 > > 2 1 > 2 1=2 2 > 1 2 >>1

Example 1 tick

Example 2 Tick

Example 3 tick

Example 4 tick

Example 5 tick

To explain the table, the user has 5 options (from left to right):

18


1 algorithm 1 is much better than algorithm 2

2 algorithm 1 is slightly better than algorithm 2

3 algorithm 1 equal to algorithm 2

4 algorithm 2 is better than algorithm 1

5 algorithm 2 is much better than algorithm 1

In each example our algorithm might be labelled 1 or 2 and the user never knows. We’ve found this method works well.

4.5.2 Listening test procedures at QMUL

This testing scenario outlines the formal listening test procedure used by QMUL for benchmarking and assessment of their source separation tool, to be deployed in WPs 4 and 5. Audio source separation is the problem of recovering source signals from a mixture where several audio sources are active [46]. Evaluating audio source separation algorithms means rating the quality or intelligibility of separated source signals. While objective criteria fail to account for all auditory phenomena so far, precise subjective ratings can be obtained by means of listening tests. In practice, the accuracy and the reproducibility of these tests depend on several design issues. In this paper, we discuss some of these issues based on ongoing research in other areas of audio signal processing. We propose preliminary guidelines to evaluate the basic audio quality of separated sources and provide an example of their application using a free Matlab graphical interface. Evaluating the performance of a separation algorithm means quantifying the perceived quality or intelligibility of the separated sources using one or several ratings corresponding to different distortions. Objective criteria adapted to this problem have been proposed in the literature, including the Signal-to-Distortion Ratio (SDR), the Signal-to-Interference Ratio (SIR) and the Signal-to-Artifacts Ratio (SAR)[47]. Although they are related to the perceived audio quality in many cases[47], they do not model auditory phenomena of loudness weighting and spectral masking. Objective criteria designed for audio coding [48] or denoising [49] better account for these phenomena, but they provide a single rating which becomes invalid when different types of distortion are present[49]. A more principled way of obtaining perceptually relevant ratings is to perform listening tests involving human subjects. This has rarely been done in the context of source separation [50-53], maybe because of the misconception that all listening tests are time-consuming and do not provide as precise ratings as objective criteria. In practice, simple listening tests often provide statistically significant results with less than ten non-expert subjects. Intelligibility tests do not rely on particular distortions and can be directly used in a source separation context. However standard quality tests are designed for coding and de-noising, and must be adapted to this different context. The key points allowing reproducibility are the use of standardized rating scales and anchors, and prior training and post-screening of the subjects. Ad-hoc procedures used so far [50, 52, 53] are insufficient in this respect. In the following, we propose some preliminary guidelines to evaluate the basic audio quality of separated sources. These guidelines should not be considered as final recommendations but as a possible starting point towards a collaborative standard definition.

Adaptation of MUSHRA

We adapted the MUSHRA standard as follows. In order to obtain reference signals, mixtures are generated by recording physical sources successively as proposed in [54] or by convolving single-channel source signals with synthetic filters. The spatial image[46] of each source on all the mixture channels is used as a reference. The test is repeated for each reference by grouping the corresponding estimated sources (possibly extracted from different mixtures). Three anchors were used to provide absolute quality ratings of interference, noise and artifacts. Interference anchors were obtained by adding a scaled version of the sum of the other source images to the target source image, and noise anchors by adding scaled white noise to the target source image. The scaling factors were defined so that the ratio between the loudness of the distortion signal alone and the loudness of the anchor equals 0.5. Loudness is evaluated using a Matlab routine 1

based on the standard ISO 532B[55]. Artifacts anchors were computed by random cancelling of half of the time-frequency points in the short-term Fourier transform of the target source image using half-overlapping sine windows of length 1024. Other features of MUSHRA, including the rating scale, were kept unchanged. In order to facilitate running MUSHRA tests, we built a Matlab interface called MUSHRAM distributed under GPL2. A screenshot of the interface is shown in Figure 2.

1 http://www.auditory.org/mhonarc/2000/zip00001.zip2 www.elec.qmul.ac.uk/digitalmusic/downloads/#mushram

19


Figure 2. Screenshot of the MUSHRAM interface.

Application We used the above guidelines to evaluate Frequency-Domain Independent Component Analysis (FDICA) [56], the Degenerate Unmixing Estimation Technique (DUET)[52] and the Adaptive Stereo Basis (ASB) method[57] on synthetic mixtures of two male speech signals with various Reverberation Times (RT) and Signal-to-Noise Ratios (SNR). More details about the experiment were given in [57]. Subjective ratings obtained from eight subjects are shown in Figure 3 and Figure 4. The results showed that ASB performed best in clean conditions (SNR=40 dB, RT=20 ms) but worst in noisy reverberant conditions (SNR=20 dB, RT=320 ms). Rating differences are less significant in other conditions. Interestingly, anchors were given better ratings than actual estimated sources. This is partly due to the difficulty of the considered mixtures. Also, signals containing several types of distortion at the same time, including heavily filtered sources and noise, could be more annoying than signals containing a single type of natural distortion with the same loudness.

Figure 3. Subjective ratings of the anchors. Bars indicate 95% confidence intervals.

20


We showed that listening tests are a practical way to compare audio source separation algorithms and used an adapted MUSHRA procedure to evaluate basic audio quality, along with a free Matlab interface.

Figure 4. Subjective ratings of FDICA, DUET and ASB on speech mixtures with different reverberation times and signal-to-noise ratios. Bars indicate 95% confidence intervals.

4.6. Expert User Advisory Board (EUAB)

The Expert User Advisory Board (EUAB) consists of content managers external to the EASAIER consortium. They are potential EASAIER users, with a strong interest in the outcomes of this project. This board will serve as expert testers and evaluators. Members of the EUAB provided the EASAIER project with initial user needs specifications.

We aim to solicit the most useful kinds of feedback from the EUAB while placing the least demands on its members. For example, due to their subject area, individual members may be more suitable and willing to provide feedback on specific elements of the project. Some members, such as Spoken Word Services, are prepared to provide testbeds for the trial of EASAIER prototypes. It is envisaged that the EUAB will also provide informed opinion regarding industry issues – as opposed to user needs – pertaining to the EASAIER project, for example, search tools, players and interfaces, copyright and licensing. EUAB members specific roles and requirements within EASAIER is outlined in Table 5. A model for the EASAIER project’s summative evaluation will become more obvious as the project develops and this will be a discussion point for the Board’s first meeting. Crispin Jewitt (Director, The British Library Sound Archive) has suggested that the EUAB meet in person twice during the project period for the purposes of discussion and presentation of EASAIER prototypes. Tentative dates for these meetings have been set for project months 12 and 24, with locations to be confirmed. We have met with EUAB members several times, before and since the start of the project, discussing user needs issues.

Table 5. EUAB member outline.

Member The British Library Sound ArchiveArchives One of the largest sound archives in the world, the British Library’s Archival Sound

Recordings Project gives UK Higher and Further Education staff and students free access to over 12,000 recordings in a variety of fields and genres.

EUAB Role The British Library Sound Archive brings its experience as a major national sound archive and leader in digitisation of and networked access to holdings.

Contributions to EASAIER Advice and guidance on continuing user requirements from Harmonica, Jukebox and the ASR project as well as wealth of contacts and networksIndustry issuesFormative testing of toolsTechnical adviceTestbed material (rights dependent)Implementation of final EASAIER toolkit and summative testing and evaluation

Member The National Library of ScotlandArchives The National Library of Scotland is Scotland's largest library. The Library acquires

music through its legal deposit privilege, as well as through bequests, donations and purchases. The music collections now include extensive British holdings, especially Scottish music. There is a wide selection of foreign music editions, special collections of early editions of Handel, Berlioz and Verdi, special collections of Scottish music collectors, and music sound recordings.

EUAB Role A national library with growing digitised music collections, the National Library of Scotland was an early supporter of EASAIER and helped shape the initial

21


specifications of the project.Contributions to EASAIER Advice and guidance on user requirements from current digital collections

Formative testing of tools Industry issues Testbed material (rights dependent)Implementation of final EASAIER toolkit and summative testing and evaluation

Member The Irish Traditional Music ArchiveArchives The Irish Traditional Music Archive is a reference archive and resource centre for

the traditional song, instrumental music and dance of Ireland. Established in 1987, the Archive has a comprehensive collection of materials – sound recordings, books, photographs, videos, etc. – for the appreciation and study of Irish traditional music. It now holds the largest such collection in existence, with coverage of the island of Ireland, of areas of Irish settlement abroad, and of non-Irish performers of Irish traditional music.

EUAB Role Perspective as a small specialist traditional music archive, which is in the very early stages of digitisation of holdings. As such a particular role for ITMA will be to evaluate the effectiveness of EASAIER tools for smaller archives.

Contributions to EASAIER Advice and guidance on user requirements for traditional music collectionsFormative testing of tools (in conjunction with DIT)Implementation of final EASAIER toolkit and summative testing and evaluation

Member The Institut National de l’Audiovisuel (INA)Archives Funded by the French state, the INA has responsibilities of a state broadcaster,

including the preservation of the national audiovisual heritage, collecting audiovisual programmes, preserving and restoring its archive collections. It aims to improve the already impressive access to its images and sounds within the web environment, using digital technology to the full and making material more readily available.

EUAB Role Perspective as major national broadcast archive and a leader in digital archiving (INA has the first audiovisual archiving centre in the world) in addition to its research arm.

Contributions to EASAIER (To be confirmed)Advice and guidance on user requirements Formative testing of toolsIndustry issues Technical adviceImplementation of final EASAIER toolkit and summative testing and evaluation

Member Spoken Word Services Archives Spoken Word Services is based at Glasgow Caledonian University. Its core aim is to

enhance and transform educational experience through the integration of digitised spoken word audio and video into learning and teaching. Spoek Word Services are attempting to make a substantial portion of the BBC’s radio archive accessible online for educational use.

EUAB Role Spoken Word services holds a rich collection of digitized audio resources, associated texts and images, and a set of integrated online annotation tools. Its purpose is to promote the usability and integration of digital spoken word repositories to improve undergraduate teaching and as such has a great deal to offer EASAIER in both user requirements and technical guidance.

Contributions to EASAIER Advice and guidance on speech-based user requirements from its Spoken Word project Formative testing of tools, as an archive that already employs similar toolsTechnical adviceContacts with developers at Spoken Word partner organisationsTestbed material (rights dependent)Implementation of final EASAIER toolkit and summative testing and evaluation

Member Borthwick Institute for ArchivesArchives The Borthwick Institute for Archives is one of the biggest archive repositories

outside London and holds a number of different formats.EUAB Role The University of York recorded music collections (part of the Borthwick

collections) give the perspective of a specialist music archive in the very early stages of digitisation of holdings. As such a particular role for ITMA will be to

22


evaluate the effectiveness of EASAIER tools for smaller archives. Contribution to EASAIER (To be confirmed)

Advice and guidance on user requirements for specialist music collections (e.g. jazz and contemporary classical music)

Member Swedish National Archive of Recorded Sound and Moving ImagesArchives The Swedish National Archive of Recorded Sound and Moving Images (SLBA)

collects the Swedish output of audiovisual media: broadcast radio and TV, film, video, records, CDs, multimedia. The archive also collects older items and the ultimate aim is to cover the entire national production of recorded sound and moving pictures from the late 1800s onwards.

EUAB Role Perspective as major national broadcast archive which is expanding its digital collections.

EASAIER requests (To be confirmed)Advice and guidance on user requirements particularly in popular music area. Industry issues Implementation of final EASAIER toolkit and summative testing and evaluation

Member Discoteca di Stato, RomeArchives The Italian Discoteca di Stato (Italian National Sound Archive) holds over 230,000

sound and video recordings from the 1920s to the present. Maintained by the Ministry for Cultural Heritage and Activities, the Archive comprises material including historical speeches, theatrical works and documentaries. It also consists of a large catalogue of music (some unpublished) and songs in dialect and Italian.

EUAB Role Perspective as major national broadcast archive with digital collections, particularly in the regional and traditional music areas.

EASAIER requests (To be confirmed)Advice and guidance on user requirements particularly in regional and traditional music area. Industry issues Implementation of final EASAIER toolkit and summative testing and evaluation

4.7. Criteria for testing and evaluationTesting and evaluation methods to be employed are described in Table 6 with a summary of the advantages of each and the sorts of data they yield, and the user groups that they will be implemented with. Reporting methods are discussed in Section 5.2.

Our choice of methods is based on the following parameters:

Type of information (user need/wish-list/feedback) Types of user participating in testing and evaluation Number of test users recruited Time, cost and logistical limitations.

We are seeking more qualitative than quantitative feedback, leaning towards evaluation rather than formal testing. We will ask, “Is this useful?” and, “How could it be more useful?” Questions such as, “Does it work?” will be addressed during EASAIER partner in-house testing (see Section 4.5).

Our main test users are the EUAB members. These are small in number, but rich as potential sources of qualitative data. Numbers in other target user groups will be small, precluding statistically significant results, but enabling and facilitating qualitative feedback.

We will seek to use methods that complement one another and that, in combination, provide the richest yield of feedback. Logistical issues may demand some flexibility in the methods adopted.

Table 6. Testing and evaluation methods.

Quantitative testing methods Type of data yieldedChecklists against set task scenarios or different design options

Quick, easy feedback (e.g. Yes/No answers) yielding comparable, measurable data on controlled, closed-format task scenarios.

Questionnaires (against set task scenarios)

Forced choice, Likert-type[58] questions will be used, yielding comparable, measurable data on controlled question areas. Used in conjunction with set

23


Quantitative testing methods Type of data yieldedtask scenarios increases comparability. Set, closed-format task scenarios facilitate measurable and comparable results. Can also gather focused data on user characteristics/interests/expertise level. See Appendix 2 for an example of this approach.

Part quantitative, part-qualitative methodDocumented walk through/talk through protocols against set task scenarios

Documented either by third person evaluators or users themselves, gives rich feedback data on user experience. Set, closed-format task scenarios to facilitate measurable and comparable results.

Documented walk through/talk through protocols, free choice

Documented either by third person evaluators or users themselves, gives rich feedback data on user experience and how users interact with tools or the system.

Supplemental observation/video recording (watching how users interact with tools)

As above, and can be useful to provide material for follow-up interview/focus group schedules.

Walk through/talk through protocols, with interventions

Reactive interventions from tester/evaluator (“Why did you do that? Was that what you were expecting? How could that work better for you?”) enable drilling down to detailed level.

Qualitative testing methodsSemi-structured interviews Previously defined question areas, but with relatively open-ended questions,

will provide data from various sources and groups of users that is comparable, but broad-brush, e.g. on user expectations, and requirements, as well as being able to include unexpected topic areas and ideas that users contribute.

Focus groups As above, with advantage of group discussions sparking ideas and unexpected areas for discussion.

Questionnaires As above, can also gather focused data on user characteristics/interests/expertise level. See the Appendices for examples of this approach.

Testing and evaluation activities will vary according to:

the nature of the tool/module/prototype under testing/evaluation whether formative or summative feedback is required (for example, the stage of the tool/module being

tested/evaluated) advice and requests from other partners (for example, tools developers may wish WP7 to test specific aspects

with different user groups) and will specify requirements in the testing/evaluation template (see Section 5.2) collaborative working to be agreed with interface developers on WP6

These areas and the questions specific to them are displayed in Table 7. It is envisaged that more than one testing/evaluation method will be employed to ensure a thorough investigation of these areas and questions (see Table 8 for mapping of question area to method). Where appropriate, standard criteria such as Cleverdon’s for evaluating retrieval systems: coverage, time, effort, presentation, recall and precision [59] will be used.

Table 7. Areas of investigation and specific questions for testing and evaluation.

User characteristics:Level of experience and engagement with sound archives, digital archives and digital toolsLevel of user expertise Broad/specific areas of interest

Presentation, interface and usability:Look and feelUsabilityHelp/instructionsGeneral/specific difficulties Any unexpected/innovative usability

Functionality:

24


Tools function according to developers’ expectationsTools function according to users’ expectationsSearches and retrieval function according to standard criteria[59, 60]

User needs and requirements To what extent functionalities, demonstrators and prototypes meets user needsMost important functionalities, with reasons givenFunctionalities to be added/abandonedPost-usage requirements (e.g. profiling, storage of annotations)Value to sound archive users

Implementation (for content managers only)Ease of implementation within existing system and archive contentUsability for managers and end usersValue for sound archive content managers

Table 8 is an indicative map of how question areas and methods will relate to different features, modules, and prototypes to be tested and evaluated. Note that there is no intended direct representational correlation here between question areas and methods (e.g. on the first line, functionality will not be necessarily assessed via checklist, or user needs by walk/talk throughs).

Table 8. EASAIER products/activities, question areas and testing/evaluation methods.

Module, feature, prototype Question areas Methods

Looping and marking module FunctionalityUser needsPresentation

ChecklistWalk/talk throughFocus group

Ongoing liaison with EUAB User characteristics InterviewsFocus groupQuestionnaire

Recruitment of other user groups User characteristics Focus groupQuestionnaire

Prototype segmentation, separation and speaker/instrument identification system

FunctionalityUser needsPresentation


Prototype speech and music retrieval systems with vocal query interface


Supplemental observationWalk/talk throughFocus group

Prototype cross media retrieval system FunctionalityUser needsPresentation

Supplemental observation ChecklistWalk/talk throughFocus group

Prototype transcription system FunctionalityUser needsPresentation


Time stretching modules with synchronized multimedia prototype



Demonstrator of deployed access tools in sound archives (subsuming tools built in previous deliverables)

ImplementationFunctionalityUser needsPresentation

InterviewQuestionnaire

5. Implementation schedule and reporting mechanismsIn this section we propose an outline schedule of testing and evaluation activity and briefly consider reporting mechanisms.

25


5.1. Schedule of testing and evaluation, with dependenciesWe will agree indicative schedules with relevant partners building modules, prototypes and demonstrators along the following lines:

For WP7 schedules for access to interim modules etc. for formative evaluation before due completion dates; Deadlines for WP7 to feed back to partners on results.

There are several layers of dependency in this schedule (e.g. feed back of formative testing and evaluation to partners is dependent on timely delivery of interim modules to WP7) and concomitant risks of slippage particularly as the building of most modules is dependent on collaboration between two or more partners.

Therefore, the following schedule (see Table 9), particularly between months 10 and 20, should be seen as indicative only at this stage. It will be used as a planning tool and is likely to be subject to frequent adjustment. This schedule notes only those modules, prototypes and demonstrators we expect to subject to specific user testing and evaluation.

Following the current Deliverable (7.1), the next formal Deliverable report is 7.2 Report on formal evaluations and deployment, due in month 30. This final report will be in effect the summative evaluation report. In the intervening months, WP7 will be working on a schedule of formative testing and evaluation of modules, prototypes and parts of these (e.g. specific functions within modules) as they are produced by the EASAIER partners throughout the life of the project. Inevitably, many of the modules, prototypes and demonstrators are scheduled to be completed at the latter end of the project (between months 20 and 27).

In order to avoid squeezing the testing and evaluation activities into too short a timeframe, we expect to liaise closely with partners and engage with them at earlier stages in development of the tools in order to provide some less formal formative testing and feedback before they are completed. These formative evaluations will result in reports back to partners and these interim reports will be fed into the final summative report, 7.2. In addition, we will be running less formal information-gathering protocols and instruments as part of our constant liaison with the Expert User Advisory Group over the whole course of the project.

Table 9. Indicative schedule of testing/evaluation activity

Project Month WP leader/Deliverable number Testing/evaluation activity by WP710–11 WP7 Agree indicative schedule with all partners10 onwards WP7 Further recruitment and ongoing liaison with Expert

User Advisory group10 WP7 Recruitment and specification of other user groups10–12 WP5: Looping and marking

module, 5.1Formative testing and feedback

12 WP7 Expert User Advisory Board meeting12–14 WP4: Prototype segmentation,

separation and speaker/instrument identification system, 4.1

Formative testing/evaluation and feedback

12–30 WP6 Liaison and discussion over interface issues14 4.1 delivered Feedback on whether D4.1 addresses user needs and

requirements for Sound Object Representation15–18 WP3: Prototype on speech and

music retrieval systems with vocal query interface, 3.2


20 3.2 delivered Feedback on whether D3.2 addresses user needs and requirements for the Retrieval System

18–24 WP3: Prototype cross media retrieval system, 3.3


18–24 WP4: Prototype transcription system, 4.2

Formative testing/evaluation and feedback, including comparisons of automatic transcriptions with known transcriptions available in archives or the public domain, identification of common transcription errors

18–24 WP5: Time stretching modules with synchronized multimedia prototype, 5.2


25–26 WP7 Preparation for summative evaluation and 7.2 report with user groups

26


24 WP7 Expert User Advisory Board meeting, focused on deployment of the EASAIER system in EUAB sound archives

26–27 3.3, 4.2, 5.2 delivered27–30 WP8: Demonstrator of deployed

access tools in sound archives, 8.2 (subsuming tools built in previous deliverables)

Summative evaluation, leading to 7.2 report

5.2. Sample testing and evaluation template for an EASAIER component

Testing of the various EASAIER components will require significant collaboration between developers and the evaluation team. This necessitates the establishment of agreed pbjectives and testing procedures which guarantee that the evaluation expectations of both developers and test organizers A template like that displayed in Table 10 will be used by DIT as the developer and RSAMD as the testing/evaluation partner to set the parameters for the testing and evaluation activities. Similar templates will be used for all EASAIER tools during the evaluation stage.

Table 10. Populated testing/evaluation template for D5.2, Time Scale Modification Tool.

Challenge Agreed criteria and mechanismsObjectives Assess if the time scale modification tool presentation and HCI is effective and

workable for expert musicians, but non-expert users and identify improvements. Identify tolerance to different artifacts with real users.Evaluate utility of tool in two musical genres Gain insight into use in other applications.

User groups and target audiences Classical and Scottish music students at RSAMD.Testing and Evaluation methods to be used

Checklist (tests functionality, fitness for purpose, detection of artifacts using non-technical language).Walk/talk through (more focus on presentation and HCI aspects).Focus group (usefulness, uses for other groups).

Reporting/discussion mechanisms

Graphical presentation of checklist dataSummary account of Focus GroupSummary account of walk-through/talk throughs with any problem areas specified in detailDecision on any desirable adjustments

Schedule Month 18DIT and RSAMD agree objectivesDIT deliver tool RSAMD recruits users and runs first phase tests and evaluationsMonth 20RSAMD feeds back via reports to DIT and initiates dialogue and discussionMonth 22DIT deliver revised tool if necessaryRSAMD runs second phase tests and evaluationsMonth 23RSAMD feeds back via reports to DIT Month 24DIT makes any adjustments

5.3. Reporting testing and evaluation activityThe Expert Advisory Board, as our main user group, will be kept in close and frequent contact with EASAIER developments and test or evaluation results, both though regular formal reporting and informal channels, such as mailing lists, teleconferences and visits.

Testing and evaluation findings will be communicated by formal and less formal means, according to the audience [61]. The main deliverable, as noted above, is the summative evaluation report (D7.2) at the end of the project. The format and communication of interim, formative findings to other partners will take the form of summary reports for

27


qualitative tests using graphical representations of measurable or comparable data and summary reports of qualitative measures synthesizing main results. These will be supplemented by discussion by email, teleconference and visits.

6. Conclusion

Attention to user needs and requirements is at the heart of the EASAIER project. This early deliverable demonstrates explicit links between identified user needs and requirements in the existing literature and proposed EASAIER outputs. It also indicates how emerging user needs and requirements will be monitored throughout the project by close collaboration with varied user groups, but chiefly with the Expert User Advisory Board, and fed back into EASAIER development. This iterative, collaborative approach is mirrored in the interactions between EASAIER partners in their formative testing and evaluation activities. It also demonstrates the multidisciplinary nature of the project in which technical development partners work alongside those with content and evaluation expertise, thus ensuing the best possible fit to ‘real life’ needs. We have described how testing and evaluation methods will be selected, evaluation objectives identified, how activities will be set up and reported and how results will be actioned. The timeframe for this complex and inter-dependent set of activities is limited and scheduling of formative activities remains a significant challenge. Nevertheless, these formative stages are vital to ensuring that the final summative evaluation of EASAIER at the end of the project is best prepared and that the chances of successful outcome for all partners and all users, expert and non-expert are maximised.

28


7. References

29


Appendices

Appendix 1: Samples of initial HOTBED User Needs Analysis Instruments (2001)

These questionnaires used task scenarios on different sites to elicit data from RSAMD students on a number of criteria including:

usefulness to their studies performance and functionality, e.g. of searching mechanisms user characteristics.

The questionnaire instruments were specifically targeted at students from the BA Scottish Music Course at the Royal Scottish Academy of Music and Drama. This is different from the wider-ranging audience for EASAIER and the orientation in EASAIER towards tools rather than content itself. That is to say, the EASAIER testing and evaluation work package would not be measuring usefulness of content to informants’ studies, but rather usefulness of an approach to searching or of specific tools. However, the samples below give comparable illustrative examples of how task scenarios and user characteristics can yield useful data.

------------------------------------------------------------------------------------------------------------------------

QUESTIONNAIRE USED BY YEAR 1 STUDENTS

HOTBED USER NEEDS ANALYSIS Year 1 BA Scottish Music

GAELIC STUDIES FORTNIGHT 13/6/01

STUDENT NAME: ________________________________________________

Task 1: Imagine that you have been asked to write an essay on the following four songs collected on Tiree and to be found on the Tiree Project website www.tiriodh.ed.ac.uk

Fodar dha na gamhna beaga Mrs Neil LamontMoch an diugh a rinn mi èirigh Mrs Neil LamontOran an t-saighdear Margaret MacArthurÓ Hó Ró ‘llle Dhuinn Margaret MacArthur

Listen to them.

Look at the information given about each track.

1. How useful do you think this information would be if you were to do an essay on these songs and/or the two singers? Please give reasons.

1.1. Is there any other information you would like with regard to the songs, the singers or the actual collection of the material? Please list.

1.2. Using the search facility what related material can you find which might be useful in the essay. Please list.(For example you might wish to discover if these songs were also collected from any other singers, or what other material the two singers may have recorded.)

1.3. Are there any limitations to, or problems with the search facility that you have found? Please list.

1.4. How easy did you find switching between tracks in order to compare them for the essay?

1.5. If you were doing this essay for real how many times do you estimate you would visit the site?

1.6. What would you estimate the duration of each visit to be?

30

http://www.tiriodh.ed.ac.uk/


1.7. Would you also look elsewhere for supporting information for your essay? If so, where would you look?

1.8. Please indicate your level of fluency in Gaelic: FLUENT COMPETENT INTERMEDIATE BEGINNER

1.9. If you are fluent, did you use the Gaelic or the English version of the site? Please state why.

1.10. If you are not fluent, do you think this hindered you in your search for material or was the English version sufficient for your needs? Please note any language problems that you encountered.

1.11. Any other comments you have about this site or more generally about the idea of doing your research in such a manner would be welcome, whether positive or negative! Please list your thoughts below, and continue on the next sheet.

1.12. Have you ever used a music site similar to this before? Please give details and state whether you found it better or less useful than the Tiree site.

Task 2: Learn the following song, which can be found on the PEARL site www.pearl.arts.ed.ac.uk under TOCHER:

Fhir a’chinn duibh Alasdair Boyd

2.1. How many times did you need to replay the song before you felt you knew it?

2.2. Did the supporting information answer any questions you might have had about either the song or the singer? If not, please list your queries or thoughts.

2.3. Did you find the transcription helpful when learning the song? Or would you rather not have seen it until you felt that you knew the piece? Would you like the option of only viewing the transcription if you request it? Please give your reasons.

2.4. How do you feel about learning a song in this way? Do you think that it would aid your course work if you had access to more such material? Please write down any thoughts or comments you have about this.

Thank you for your co-operation in this user needs analysis. Your answers are of tremendous importance in helping us to plan and design the best way of working with digitized archive recordings.

31

http://www.pearl.arts.ed.ac.uk/


QUESTIONNAIRE 2 USED BY 2ND & 3RD YEAR STUDENTSAND STAFF

HOTBED USER NEEDS ANALYSIS Years ll & lll

STUDENT NAME: Year:

FIDDLE TUNES OF THE OLD FRONTIER

Go to www.loc.gov (the Library of Congress site) then click through the following stages:American MemoryCollection FinderPerforming ArtsFiddle Tunes of the Old Frontier.

Take some time and browse the site thoroughly, listening to some of the sound recordings, looking at transcriptions, fieldnotes, background information and glossary.

Please note your thoughts and comments, whether positive or negative, about the site.

What are the most useful features?

Is it worthwhile being able to see the original transcriptions, and notebooks?

Would you want to see the transcriptions as the music is played or would you rather have to ask to see them?

SEARCHING On this site you can browse for tunes by title, by genre or by the session in which they were recorded. On a scale of 1-10, with 10 being the highest, how useful do you find each of these?

By title

By genre

By recording session

Would one of these facilities be sufficient or do you need all three?

Try out several searches for tunes that might be there e.g. Old Joe Clark, Arkansas Traveler, Fisher’s Hornpipe etc.Did this work? Were there any problems? How do you find using the search facility?

Try finding and then listening to some tunes that you don’t think you know.Would you be able to learn them from this source? (Assuming you had a computer in the practice or teaching room) How would you feel about this style of learning? How many times would you need to replay the tune?

If you had to do an essay or project on Henry Reed or on fiddle tunes from this area how useful would you find this site? Please give reasons and comments. Is there anything not included which you would like to see or know more about?

If you were doing this essay how many times do you think you’d visit the site?

How long would you spend each time?

Any other thoughts about this site?

32

http://www.loc.gov/


SITE TWO : THE TIREE PROJECT

Go to www.tiriodh.ed.ac.uk

Using some or all of the following material, please give us your comments.

Fodhar dha na gamhna beaga Mrs Neil LamontMoch an diugh a rinn mi èirigh Mrs Neil LamontOran an t-saighdear Margaret MacArthurÓ Hó Ró ‘Ille Dhuinn Margaret MacArthur

How easily or otherwise did you locate this material?

Is the information given about each track sufficient, or is there anything else you would need to know if you were to use these recordings in an essay or project?

Try to search for other material that you think might be there. What do you think of the search facilities?

How would you compare this site to the Fiddle Tunes of the Old Frontier site, which you visited first?

Any other comments about the Tiree site?

SITE THREE : THE TOCHER MATERIAL FROM THE SCHOOL OF SCOTTISH STUDIES.

Go to www.pearl.arts.ed.ac.uk

This site houses some of the material from the journal Tocher. Browse through it then note your thoughts below. Bear in mind that with stories and Scots songs as well as Gaelic material this could be a site you might want to use in folklore work.

Find and listen to Fhir a’chinn duibh sung by Alasdair Boyd.How many times would you need to listen to the song to learn it?

Did the supporting information answer any questions you might have had about either the song or the singer?

How does it compare to the supporting information available on the Tiree site?

Was it helpful to be able to see the transcription?

Any other thoughts?

ABOUT YOUR USE OF COMPUTERS AND THE INTERNET

Have you used music sites like these before?

If so are there any you’ve found particularly helpful, or that we should have a look at?

On a scale of 1 to 10 with 10 being the highest how comfortable are you with using the internet to research either course work or personal repertoire?

Any comments or thoughts about using computers?

Thank you for your co-operation in this user needs analysis. Your answers are of tremendous importance in helping us to plan and design the best way of working with digitized archive recordings.

33

http://www.pearl.arts.ed.ac.uk/

http://www.tiriodh.ed.ac.uk/


Appendix 2: British Library ASR Project Usability Survey, 2006This instrument sought to gather wide-ranging data on:

user characteristics initial impressions finding and listening to sound recordings (using task scenarios) using the help overall assessment.

Although some aspects are comparable with metrics planned for EASAIER (e.g. the task-orientated approach, look and feel, presentation, searching and user characteristics) there are important differences. Firstly, we will not phrase questions with Likert-type responses all in a positive mode (“Do you agree …?”, “How easy did you find this?”), but, following standard practice, will mix positive and negative modes. Although used as a checklist, this type of instrument will yield a certain amount of quantitative data, we will also seek to maximize its added value by using it as a basis for walk/talk through protocols that give user and evaluator the chance to elaborate on their feedback, rather than as a standalone metric. Finally, this instrument takes a very broad-brush approach; many of the EASAIER interventions will be more task- or approach-specific.------------------------------------------------------------------------------------------------------------------------

Thank you for agreeing to help the British Library to develop the Archival Sound Recordings website. The Library is creating the site in collaboration with the JISC to bring 12,000 recordings from ten diverse and interesting collections to every student, teacher, and researcher in UK further and higher education institutions.

This demonstrator version of the site features sample content and the basic functionality that will help you find recordings and listen to them. We need your feedback now to ensure that the final version of the site will meet the requirements of the people who will use it.

Please note the following: The demonstrator site contains recordings from just three of the ten collections to be included on the final site. The three collections are the African Writers’ Club broadcasts, the Rycroft South Africa recordings, and the Visual Arts Interviews. Metadata for the 400 Popular Music Tracks, Beethoven String Quartets, and Sony Radio Awards is also included. You can search and browse for this content, but full item details are unavailable at present, and you cannot play the recordings. Also unavailable are the ‘Advanced search’, ‘Playlist’, and ‘My project’ features.

Familiarising yourself with the site and completing this online questionnaire will take no longer than half an hour. Please follow the instructions provided, click the boxes or buttons which best reflect your views, and add extra comments where appropriate.

If you supply your email address we will enter you in a prize draw for a video iPod.

Question Options Supplementary questionA. About you1. What is your current occupation? Undergraduate student

(university/higher education)Postgraduate student (university/higher education)Teaching staff (university/ higher education)Student (further education college)Teaching staff (further education college)Library and information staffAudio-Visual staffOther

[Radio buttons permitting just one option to be ticked].

If ‘Other’, please specify.

[Box for inserting free text.]

34


Question Options Supplementary question2. How often do you use the Internet? Several times a day

About once a daySeveral times a weekAbout once a weekLess than once a week


3. Where do you use the Internet most frequently?

College/universityWorkHomeLibraryOther


If ‘Other’, please specify.


4. Had you heard of the British Library before encountering this website?

YesNo


If ‘yes’ please describe your impressions of the Library before encountering this website.[Box for inserting free text.]

5. Have you used the following websites to find and listen to sound recordings?

The British LibraryNapsteriTunesSound ArchiveBBCiOther

[Yes/No tick boxes for each site].6. Do you use any accessibility tools to help you with using a computer and the Internet?

YesNo

[Tick boxes].

If ‘yes’ please specify.


B. Initial impressions7. Do you agree that the home page explains what the website is about?

1 (Strongly disagree)2 (Disagree)3 (Neither agree nor disagree)4 (Agree)5 (Strongly agree)


8. Do you agree that the sound recording collections available on the site are immediately identifiable from the home page?



9. Do you agree that the visual appearance of the site is aesthetically pleasing?



10. Do you agree that the layout of text, navigation links, and other elements on

1 (Strongly disagree)2 (Disagree)

35


Question Options Supplementary questionthe home page is clear? 3 (Neither agree nor disagree)

4 (Agree)5 (Strongly agree)


11. Do you agree that the text on the home page is easy to read?



C. Finding sound recordings12. Use the search box on the home page to look for recordings featuring Princess Magogo. How easy did you find this?

1 Very easy2 Easy3 Neither easy nor hard4 Hard5 Very hard/Did not finish task


Please explain any problems you experienced.

[Box for inserting free text].

13. Now return to the home page and use the search box find recordings of the flute. How easy did you find this?





14. On the results page for your search for flute recordings, use the search box to search for violin recordings in the South Africa collection. How easy did you find this?





15. Do you agree that the search facility returns relevant results?



16. Do you agree that the layout of search results is easy to understand?



17. Please return to the home page. Now use the browse facility to find the two recordings of Beethoven’s 16th String Quartet by the Amadeus ensemble. How

1 Very easy2 Easy3 Neither easy nor hard4 Hard



36


Question Options Supplementary questioneasy did you find this? 5 Very hard/Did not finish task


18. Now use the browse facility to find recordings featuring Peter Kennard talking about photography and the Guildford Four. How easy did you find this?





19. Do you agree that the browse option returns relevant results?





20. Do you agree that the layout of browse results is easy to understand?



D. Listening to sound recordings21. Find Princess Magogo’s Zulu language recording of the folk tale song ‘Uhamba namanzini’. Click on the entry in the search results or browse display to reach the ‘item page’ for the recording. Use the British Library audio player to listen to the recording. How easy did you find this?





22. Use the audio player to make a clip of the first ten seconds of ‘Uhamba namanzini’, then play the clip back as a loop. How easy did you find this?





23. Do you agree that the language used to describe the recordings on the ‘item page’ is clear and understandable?



24. Do you agree that it is easy to learn how use the Archival Sound Recordings website and audio player for listening?


[Radio buttons permitting just one

37


Question Options Supplementary questionoption to be ticked].

E. Help features25. While using the website, did you use the help features?

YesNo

[Tick boxes].26. If you did use the help features, did you find an answer for the problem you had encountered?

YesNo

[Tick boxes].

If ‘No’, how can the help features be improved?

[Box for inserting free text].F. Overall assessment27. How do you rate the content of this website? [Please make this assessment on the basis of the full range of collections to be available when the site is complete].

1 (lowest score)2345 (highest score)


Please explain


28. How do you rate the design of this website?

1 (lowest score)2345 (highest score)


29. Once the website is complete, how likely are you to visit it again?

1 (least likely)2345 (most likely)


How frequently would you expect to visit?

DailyWeeklyMonthlyAd hoc


30. How likely are you to recommend the site to a friend?

1 (least likely)2345 (most likely)


31. Please use this box for any further comments you may have about the Archival Sound Recordings website or this questionnaire.

[Large box for inserting free text.]

G. Contact details32. Please provide your email address. [We need this to enter you in the prize draw and in case we need to clarify any of your answers].


33. Please name the college, university, or other institution you are affiliated with.


Thank you for your help – and don’t forget to include your email address!

38

· web view[48] "itu-r bs.1387-1: method for objective measurements of perceived audio...

Documents