Designing the HCI Living Curriculum

by

Andrea Maria Marta Jovanovic

A thesis submitted in conformity with the requirements for the degree of Master of Applied Science

Department of Mechanical and Industrial Engineering University of Toronto

© Copyright by Andrea Jovanovic 2018

ii

Designing the HCI Living Curriculum

Andrea Jovanovic

Master of Applied Science

Department of Mechanical and Industrial Engineering University of Toronto

2018

Abstract

The Association for Computing Machinery (ACM) Special Interest Group on Computer-Human

Interaction (SIGCHI) has been supporting research into Human-Computer Interaction (HCI)

education for many years, most actively in the last six years. This thesis proposes the preliminary

framework for a Community of Practice (CoP) of HCI scholars and educators, sharing and

collaborating to develop teaching materials. Two studies were carried out. The first involved a

set of 12 interviews with HCI educators and practitioners that examined requirements for the

HCI Living Curriculum. The second study was a questionnaire with 28 respondents which

carried out further requirements analysis. Based on the findings of the two studies, use cases and

design requirements for the HCI Living Curriculum are presented. Guided by these use cases and

design requirements, a prototype HCI Living Curriculum was then constructed. The thesis

concludes with a discussion of lessons learned and prospects for future development of the

Living Curriculum.

iii

Table of Contents Table of Contents ......................................................................................................................... iii

List of Tables ................................................................................................................................. v

List of Figures ............................................................................................................................... vi

List of Appendices ....................................................................................................................... vii

Chapter 1 Introduction ................................................................................................................ 1

1.1 Problem Formulation .................................................................................................................1 1.2 Research Strategy .......................................................................................................................2 1.3 Thesis Roadmap .........................................................................................................................3 1.4 Thesis Notes .................................................................................................................................4

Chapter 2 Literature Review ....................................................................................................... 5

2.1 Communities of Practice ............................................................................................................5 2.2 Open Educational Resources .....................................................................................................7 2.3 The HCI Living Curriculum .....................................................................................................8

Chapter 3 Study 1: Interviews ................................................................................................... 10

3.1 Method .......................................................................................................................................10 3.2 Procedure ..................................................................................................................................10 3.3 Data Collection and Analysis ...................................................................................................11 3.4 Results ........................................................................................................................................15 3.5 Design Requirements ...............................................................................................................15

Chapter 4 Study 2: Questionnaires ........................................................................................... 18

4.1 Method .......................................................................................................................................18 4.2 Procedure ..................................................................................................................................18 4.3 Data Collection and Analysis ...................................................................................................18 4.4 Results ........................................................................................................................................19

4.4.1 Description of the data ...........................................................................................................19 4.4.2 Stem-and-leaf plots ................................................................................................................20 4.4.3 Cluster analysis ......................................................................................................................22

4.5 Discussion ..................................................................................................................................25

Chapter 5 Design: Use Cases and Prototype ............................................................................ 26

iv

5.1 Design ........................................................................................................................................26 5.2 Use Cases ...................................................................................................................................26 5.3 Prototype ...................................................................................................................................27

5.3.1 Adding content .......................................................................................................................27 5.3.2 Searching and viewing content ..............................................................................................29 5.3.3 Connecting with others ..........................................................................................................35

5.4 Conclusion .................................................................................................................................36

Chapter 6 Contributions, Limitations, and Future Work ...................................................... 37

6.1 Introduction .................................................................................. Error! Bookmark not defined. 6.2 Contributions ............................................................................................................................37 6.3 Limitations ................................................................................................................................37 6.4 Future work ..............................................................................................................................38 6.5 Final words ................................................................................................................................38

References .................................................................................................................................... 39

Appendices ................................................................................................................................... 43

v

List of Tables Table 1 - Affinity Diagram ........................................................................................................... 12

vi

List of Figures Figure 1 - Normed Incentive Value .............................................................................................. 23

Figure 2 - Normed Evaluation Intensity ....................................................................................... 23

Figure 3 - Add course wireframe .................................................................................................. 28

Figure 4 - Add material wireframe ............................................................................................... 29

Figure 5 - Search and browse wireframe ...................................................................................... 30

Figure 6 - View search results wireframe ..................................................................................... 32

Figure 7 - View material wireframe .............................................................................................. 33

Figure 8 - View course wireframe ................................................................................................ 34

Figure 9 - View profile wireframe ................................................................................................ 36

vii

List of Appendices Appendix A: Questionnaire ……………………………………………………………………..43

Appendix B: Additional Data from Study 2 …………………………………………………….47

Appendix C: Use Cases………………………………………………………………………….50

Appendix D: Ethics Protocol …………………………………………………………………....53

1

Chapter 1 Introduction

asdf

1.1 Problem Formulation Human Computer interaction (HCI) is a diverse and constantly evolving discipline. Many

educators around the world have to solve similar problems in communicating the essence of HCI

to a diverse student population in a variety of contexts. These educators often have to re-invent

the wheel, solving the same problems over and over again in constructing use cases, projects, and

examples that will aid in the understanding of HCI and the development of associated skills. In

1992 a HCI curriculum was developed by Hewett and other leading HCI educators to serve the

needs of the HCI community (Hewett et al., 1992). This HCI curriculum contained an

introduction to HCI, examples of courses in HCI, examples of HCI curriculum designs, a

collection of resources, and case studies. While a major contribution in its time, the early HCI

curriculum was somewhat limited in scope.

The early HCI curriculum focused on topics that were central to HCI at the time (Hewett et al.,

1992). The use and context of computers were largely limited to workplace settings. Human

characteristics were predominantly those of skilled and trained personnel. Interfaces and systems

architecture were constrained to desktop computer input and output methods at the time. The

scope of HCI has grown and changed dramatically since 1992. Today there are new methods of

interaction, such as mobile and sensor-based interaction, new design approaches such as

experience design and participatory design, and new paradigms, such as pervasive and

ubiquitous computing. Sensitivity to user diversity has also become more critical (Churchill,

Bowser & Preece, 2013; Churchill, Bowser & Preece, 2016).

It is clear that a new HCI curriculum is needed, and it should be a “living curriculum” (Churchill,

Bowser & Preece, 2016) so that it can grow and change with the discipline. However, the

development of the HCI Living Curriculum is a massive undertaking and no one person has the

detailed knowledge of all the various areas of the discipline that are necessary for this

development project. Thus the problem that this thesis will address is not the development of the

curriculum itself, but rather the development of a collaborative tool that can be used by HCI

2

educators in jointly building the HCI Living Curriculum. Although not constructed, for ease of

reference this envisioned tool will be referred to throughout this thesis as “TEACH-HCI” (Tools

for Educators and Academics to Collaborate in Helping HCI).

1.2 Research Strategy The first steps in developing any software tool involve requirements analysis and typically this is

facilitated through use cases, compelling and relevant examples of how the intended software

should work. Design also works best when target participants are closely involved with the

development process so that their needs and expectations are intimately associated with the

resulting tool. One of the challenges in the research reported below is that HCI educators are

busy people, and are geographically scattered around the world. Thus, in carrying out this

research I used the following research strategies.

1. I developed a set of use cases that could be used in envisioning how a collaborative tool

for developing the HCI Living Curriculum should work. In developing the use cases I

focused on broad scenarios that would have wide applicability.

2. I chose to collect data at the world’s premiere conference on HCI (Association for

Computing Machinery (ACM) Special Interest Group on Computer-Human Interaction

(SIGCHI) Conference, referred to as CHI) with the data being collected in Denver

Colorado at the CHI 2017 Conference in May 2017. This allowed me to carry out

interviews and administer the questionnaire efficiently to some of the world’s top

educators and practitioners.

3. I used semi-structured interviews to guide interviewees towards the topics of most

relevance to this thesis, while also providing interviewees with the ability to extend the

scope of the requirements analysis and introduce new topics. Six of the interviews were

done prior to the conference and the results of these interviews were used to revise and

inform the final version of the questionnaire, with a further six interviews being carried

out at the conference.

4. The questionnaire was administered to 28 CHI 2017 conference attendees. Based on the

wording in session titles, I went to the conference sessions that were most closely related

3

to the topic of HCI education and made announcements where I invited people to fill in

the questionnaire, and then handed the questionnaire out to those who expressed an

interest in participating.

5. An Affinity diagram was created to capture the most important themes that were

expressed in the interviews. The affinity diagram then guided the development of the

TEACH-HCI prototype for the collaborative tool.

6. Analysis of the questionnaire explored the extent to which the respondents diverged in

their assessments of requirements for TEACH-HCI and also characterized different

groupings of respondents as a step towards future development of personas that could

guide further development of the TEACH-HCI tool.

1.3 Thesis Roadmap This chapter has presented the motivation for the thesis research and the strategies that were used

in conducting the thesis research. The work then carried out in this thesis is reported in the

following chapters.

Chapter 2 is a review of relevant literature. My strategy in selecting relevant literature was to

focus on communities of practice and on collaboration in online communities. Since the purpose

of the TEACH-HCI tool is not to be prescriptive, but rather to serve as a container for Living

Curriculum content, I focused on the kind of collaboration required, rather than the specifics of

HCI curriculum content. I also reviewed HCI use cases that have been reported in the literature.

The chapter concludes with a brief review of research relating to some of the research

methodologies used in this thesis.

Chapter 3 reports on study 1, where the 12 interviews were carried out. The focus of the analysis

reported is on the development of an affinity diagram, and the use of that diagram to develop a

set of design requirements.

Chapter 4 reports on study 2, where the TEACH-HCI questionnaire was administered to 28

participants at the CHI 2017 conference. The pattern of responses across the various question

types is reported, features are identified that link questions with similar patterns of responding,

and different types of participant are identified through cluster analysis.

4

The TEACH-HCI prototype that was developed, based on the results of studies 1 and 2, and the

design requirements derived from study 1, is then presented in chapter 5. This prototype is

presented as a set of wireframes relating to use cases, with descriptions concerning flow through

the wireframes. A set of use cases is also presented in this chapter. The labeling for these use

cases was derived from the interview responses, while the content of these use cases was derived

from the literature.

The body of the thesis concludes with discussion of the contributions made, limitations, and

prospects for future research and development of TEACH-HCI. A set of Appendices is also

provided where the details of the questionnaire from study 2, additional data analysis from study

2, content of the use cases, and research ethics protocol are presented.

1.4 Thesis Notes The content in 2.1, 2.3, and chapter 3 has already been published in Jovanovic, St-Cyr and

Chignell (2017).

5

Chapter 2 Literature Review

asdf

2.1 Communities of Practice A Community of Practice (CoP) is a group “of people who share a concern, a set of problems, or

a passion about a topic, and who deepen their knowledge and expertise in this area by interacting

on an ongoing basis” (Wenger, 2002). CoPs are viewed as an inventive way to combine working,

learning, and innovating (Brown & Duguid, 2001). Learning and knowledge creation occur

through socialization, and members of the community often develop a set of shared resources to

engage more effectively (Bolsani & Scarso, 2014).

As Web 2.0 technologies have increased in popularity, virtual CoPs have incorporated social

networking functionalities to increase social transparency and ameliorate interaction (Marlow &

Dabbish, 2014; Stuart et al., 2012). Some of these functionalities include the means to upload

content, communicate with other members, and view member profiles (Boyd & Ellison, 2007;

Joinson, 2008).

The Wikipedia is perhaps the largest community authored repository on the internet, and has

been considered by some to be a CoP although it is a very large and diffuse community that

contributes to Wikipedia. As noted by O’Sullivan (2009), earlier and more focused CoPs include

the British Royal Society and authors and editors of the Oxford English Dictionary. A more

focused community within the Wikipedia consists of people who contribute to health-related

pages. Farič and Potts (2014) surveyed 32 people who contribute to Wikipedia health pages and

interviewed 17 of those people. They noted five factors that motivated people in their sample to

contribute to Wikipedia health pages: “education (learning about subjects by editing articles),

help (wanting to improve and maintain Wikipedia), responsibility (often a professional

responsibility, to provide good quality health information to readers), fulfillment (editing

Wikipedia as a fun, relaxing, engaging, and rewarding activity), and positive attitude to

Wikipedia (belief in the value of Wikipedia).” It seems likely that similar motivations might also

be applicable to a Human-Computer Interaction (HCI) Living Curriculum CoP. However,

members of a CoP may vary in terms of their commitment to, and productivity within the

6

community. Farič and Potts (2014) noted that “the community who most actively monitor and

edit health-related articles is very small” with some community members focusing on

“improving spelling and grammar, organization, and handling vandalism”.

While much of the motivation for contribution to CoPs seems to be altruistic and intrinsic,

feedback mechanisms have been proposed or developed for providing additional external

motivation. For instance, Rotman et al. (2012) cited the Encyclopedia of Life as using featured

contributor pages to recognize important contributors. Impact factors of various kinds can also be

used. For instance, Research Gate’s RG score is a composite metric taking into account social

interactions and reputation alongside traditional publication-based metrics (Jordan, 2015).

Various forms of badges, awards, and recognitions (McInnis et al., 2016) may also be used to

reward and recognize contributions within a CoP.

One concern with any CoP is sustainability. Community members must be able to see clear

benefits from their participation that outweigh any associated costs. Perceived benefits that

should enhance sustainability include extrinsic returns such as status, reciprocity, and reputation

(Wasko & Faraj, 2005), and intrinsic returns such as enjoyment and social interaction with other

community members (Cho, Chen & Chung, 2010; Hew & Hara, 2007).

Another concern for CoPs is the usability and effectiveness of collaborative writing applications.

Archambault et al. (2013) surveyed the literature on the use of collaborative writing applications

in healthcare. They concluded that “More primary research is needed to find ways to address the

different barriers to their use and to make these applications more useful for different

stakeholders.”

CoPs typically publish documents to the entire community, not to the public at large. They

generally use an open publishing model where there is no strong organizational ownership of

material (e.g., a fairly liberal creative commons type of license is provided, where reuse is

permitted as long as there is attribution). Bates et al. (2007) examine attitudes towards the rights

of authors in a teaching and learning community. In this case, many authors were willing to share

their resources with others because of the value that they received from feedback provided and

the assistance they got from others in improving the quality of materials.

7

One of the strongest arguments for participation in a CoP is skill development. While authorship

is not explicitly noted on Wikipedia pages, professional CoPs often have strong role models that

may assist people in developing their professional identity (cf. Andrew et al., 2009). In a

community like the HCI Living Curriculum, participants can expect to view and discuss best

practices, thereby honing their skills as HCI educators. Marlow and Dabbish (2014) discussed

this type of professional development in a graphic design community. Activities that they noted

as supporting skills development included exposure to others’ work and interaction around

created artifacts and the observation of role models and imitation of their practices. In addition to

role models, more active mentoring may occur in a CoP, either through design, or through

serendipitous relationships that occur. Given that many members of the HCI Living Community

will be academics it seems likely that the skills and practices that they have developed in

mentoring graduate students may also transfer to the community, leading to beneficial mentoring

relations that should strengthen the community. Informal mentoring in the form of advisory

feedback below the threshold of a personal relationship may also occur, as was demonstrated in

the case of distributed mentoring within an online fan community (Campbell et al., 2016).

A professional CoP such as the HCI Living Curriculum needs to ensure that quality is

maintained. This can be done by using both technological and social approaches. Social tools

include user generated collaborative quality methods such as rating (Nesbit, Belfer & Vargo,

2002), and technological tools that automatically check for indicators of quality reflected in the

interface (Clements, Pawlowski & Manouselis, 2015).

One final key consideration for a CoP is how it should be governed and administered (e.g.,

Barnes et al., 2008). For instance, although the Wikipedia does rely on a distributed collection of

editors and contributors there is in fact a central governance structure that sets policy and

arbitrates disputes where necessary. The hierarchical governance of Wikipedia and some

challenges with that structure are discussed by Jemielniak (2016).

2.2 Open Educational Resources Open Educational Resources (OERs) support collaborative teaching and learning, and a growing

number are being shared, reused, and republished (Dimitriadis et al., 2009). OERs are usually

available through database portals, known as Learning Object Repositories (LORs) (Clemens,

Pawlowski & Manouselis, 2015). The fundamental functionalities of LORs have been identified

8

as searching/browsing OERs, viewing OERs, downloading OERs, storing OERs, and

downloading OERs metadata (McGreal, 2008). Thus, while LORs support the storage and

retrieval of OERs, the functionalities for collaboration between teachers or learners usually are

not supported. Monge, Ovelar and Azpeitia (2008) propose the implementation of social

software tools to encourage participation and collaboration within repository communities. They

explain that “extending the role of learning object repositories from a storage system to an open

platform where users can participate and contribute would benefit innovation in teaching and

learning” (Monge, Ovelar & Azpeitia, 2008).

2.3 The HCI Living Curriculum From 2011 to 2014, the Association for Computing Machinery (ACM) Special Interest Group on

Computer-Human Interaction (SIGCHI) Executive Committee sponsored a project to investigate

the present and future of HCI education (Churchill, Bowser & Preece, 2013). Data collected in

the research included 52 interviews conducted with SIGCHI community members, as well as 616

surveys completed in English, 156 in Brazilian Portuguese, 52 in Mandarin Chinese, and 48 in

Chilean Spanish. Questions focused on what educators, practitioners, and students considered to

be top priorities for the field of HCI. Additionally, educational resources were compiled and

discussions were hosted at the annual CHI conferences, including discussion lunches, HCI

education workshops, and SIGCHI Town Hall meeting discussions (Churchill, Bowser & Preece,

2013).

A recurring theme that emerged throughout the project was participants’ desire for a collection of

online resources shared among HCI educators. The goal formulated is to create a CoP of HCI

scholars and educators, sharing and collaborating to develop course outlines, curricula, and

teaching material, known as a Living Curriculum (Churchill, Bowser & Preece, 2016). In order

to be effective for the fast moving HCI field, these resources would need to be frequently

updated and maintained. Functionality for discussion and commenting would also be essential in

supporting the objective of content co- development.

The current infrastructure of the SIGCHI HCI Education Community (SIGCHI Education

Community) does not support the envisioned communication and sharing of resources. Before

the Living Curriculum can become a reality, there is a need to investigate outside community

platforms or methods of extending the current infrastructure.

9

A workshop on developing the HCI Living Curriculum was held at the CHI2014 conference

(Churchill, Preece & Bowser, 2014). Several visions of what this new curriculum could be were

presented. However, the proposed HCI Living Curriculum has yet to be designed or

implemented, and the research carried out in this thesis sought to take next steps towards closing

this gap.

10

Chapter 3 Study 1: Interviews

asdf

3.1 Method This research aims to investigate the requirements and use cases for the design of the Human-

Computer Interaction (HCI) Living Curriculum. In particular, I was interested in answering the

following research questions:

1. How do stakeholders envision a HCI Living Curriculum? �

2. What are the requirements for a HCI Living Curriculum? �

3. What are the barriers to a HCI Living Curriculum? �

To answer these questions, I conducted a qualitative study consisting of a series of individual

semi-structured interviews. The goal of the individual interviews was to collect data from HCI

practitioners and educators who will be potential end users of the HCI Living Curriculum

Community of Practice (CoP). In total, I conducted interviews with 12 participants. Participants

were contacted via an announcement in the HCI Education Facebook Group. They were selected

based on their willingness to participate and were not compensated for their participation. All

participants were HCI educators or had been involved in HCI education projects in their

professional careers. They consisted of a mix of academics (nine participants) and industry

professionals (three participants) from the SIGCHI community representing several regions such

as: North America, Africa, Europe, and the Middle East. Interviews took place online using

Skype, on the phone, and in person at the CHI 2017 Conference in Denver, Colorado, May 6-11,

2017.

3.2 Procedure Each interview lasted 15 minutes to 45 minutes and was conducted using the same procedure.

First, the researcher introduced herself to the participant and provided a brief introduction to the

study background and procedures. The participant was asked to read a consent form and provide

verbal consent. Second, the researcher began the semi-structured interview following a preset

11

guide. A background description of the motivation for a Living Curriculum in HCI was provided

to the participant. The interviewee was then asked a series of questions regarding their opinions

about a Living Curriculum in HCI. The researcher also explored unanticipated themes as they

were raised during the session. Below are the initial questions that were used for each interview:

1. How do you envision a HCI Living Curriculum?

2. What do you think are the requirements for a HCI Living Curriculum?

a. What would participation look like? �

b. How should a Living Curriculum behave? �

3. Do you see any barriers to a HCI Living Curriculum?

a. Do you have any concerns? �

b. Are there obstacles that would discourage �you from contributing? �

At the conclusion of the interview, the participant was debriefed and thanked for their

participation.

3.3 Data Collection and Analysis Interviews were audio recorded and content was transcribed into a computerized database and

analyzed to extract common themes and requirements through an affinity diagram (see Table 1).

12

Table 1 - Affinity Diagram

Motivation Quality Contribution Copyright Content Membership Other Barrier to entry could make participation more attractive P2

More concerned about enactment than content quality P4

Co-creation of material like slides in a Wikipedia style interaction P1, P2, P6, P10

Copyright is a reason for not having open membership P1

Framework for what is important in the field P2, P8, P9, P11, P12

Institution email required P1

Must be very simple to participate P1, P7

Seeing number of downloads and where resources are used is motivating P1, P2, P3, P5, P6, P7, P10, P11, P12

Review committee P2, P3, P5, P11, P12

Remixing material P1, P3, P5, P6, P8, P9, P10

Concern about not attributing properly P1, P7, P11, P12

Information about how to teach material P2, P8, P9, P10, P11, P12

Institutional email not required P2

Very quickly changing field of HCI; broadness of HCI P1, P3, P4, P5

Concern about criticism P1, P2, P3, P5, P7

Designated person to vet material, organize, and catalogue P1, P2

Metadata entry should be simple and optional P1, P7

Open sharing for educational purposes; Creative Commons licensing P1, P5, P6, P10

Importance of pedagogical practices P4, P7, P8, P9, P10, P11, P12

Personal profiles for members P3, P5

Importance of involving industry practitioners P3, P4, P5

13

Gamification and leaderboards P5, P6

Dislike ratings P1, P2, P5

Some may not want to share their materials P2, P3, P7

Suggest topics that don’t have (as much) material P1

Ability to create smaller communities (i.e. around countries, topics) P5, P8, P9, P10, P11, P12

Disconnect between industry and academic P1, P4, P5

Getting credit for contributing can offset feeling proprietary P3

Don’t trust ratings from other people P2, P5

Social space and commenting platform where content is dominant P3, P5, P10

Global content and local content; different contexts P2, P8, P9, P11, P12

Mentorship opportunities P5, P11, P12

Different needs in different parts of the world P2, P3, P5, P8, P9, P11, P12

Likes/thumbs up can be okay P3, P5, P8, P9, P10

General discussion area (not based on a material) P5, P8, P9

Commentary given different weight depending on experience in HCI P6

Consider how to handle multiple languages P3

Likes can be a popularity contest P3, P7

May need to create material from scratch to avoid attribution and copyright issues P1, P7

Participation should not be anonymous P10

Student engagement considerations P5, P11, P12

Ratings may act as a disincentive P1, P5

Concern about unequal contributions and commitment P6

Some educators are not very knowledgeable and are only steps ahead of the students P2

14

Institutional buy-in or accrediting body as incentive P4, P11, P12

Reviews including how resources were used P6, P8, P9, P11, P12

Different methods of contribution so that junior members can also contribute P6

Extra effort such as following formatting guides would be discouraging P8, P9

Ratings on different dimensions (such as HCI content and pedagogy) P6, P8, P9

Ability to annotate resources and see others’ annotations P6, P10

Tiers of reviews of materials that are indicated as being reviewed P8, P9

A place to share and ask for advice P6

Once something is rated it is more likely to be viewed again P7

Open contribution from members P1, P6, P7, P8, P9, P10

Diverse or regional review committee to incorporate range of perspectives P11, P12

Ability to upload drafts and look for feedback P7, P10, P11, P12

15

3.4 Results

The transcribed interviews were reviewed and comments made by participants were summarized.

When a comment was restated by another participant, the new participant ID was added to a

previous card. Comments were then organized into groups to create an affinity diagram. The

groupings that emerged out of our affinity diagram were: motivation, quality, contribution,

copyright, content, and membership. I also included a grouping for “other”. Some comments

overlapped groupings and are indicated by merged cells. Both common (expressed by multiple

participants) and uncommon (expressed by only one participant) comments are included in the

affinity diagram. Comments summarized in the affinity diagram range from participants’ feelings

about contributing to an online community to their beliefs about how features in such a

community should be organized and implemented.

The overarching theme of the affinity diagram is collaboration. Every theme relates to

collaboration in some way. For example, the motivation theme encompasses participants’

feelings towards and reasons for participating in or abstaining from the community. The

contribution theme comprises comments about working together and communicating with others.

A number of the issues represented in the affinity diagram also reaffirm what has been seen in

the literature, presented in chapter 2. For example, participants brought up views regarding

ownership of material (see Bates et al., 2007). Additionally, participants emphasized the

importance of maintaining quality of the resources (Nesbit, Belfer & Vargo, 2002; Clements,

Pawlowski & Manouselis, 2015), and shared several ideas of how this could be done.

The affinity diagram informed the design requirements outlined below.

3.5 Design Requirements

In order to increase motivation to participate in the CoP, the design should display data about use

of resources such as number of downloads and who downloaded what. Moreover, several

participants who were interviewed suggested incorporating some aspects of gamification and

leaderboards when uploading resources, contributing to discussions, and reviewing content as

well as “likes” or “thumbs up” feature for resources. It was also proposed that authors should be

able to add resources in their current forms without strict formatting constraints.

16

Quality of the content available in the CoP was also a concern raised by many participants. Some

suggested that a review committee be created to oversee resources and to ensure a high level of

quality. Members of this review committee should be diverse or possibly regional to incorporate

a range of perspectives and acknowledge that HCI can be implemented differently in different

global contexts. Moreover, designated administrators should help with organizing and

cataloguing material in an appropriate manner. Finally, once resources are available in the CoP,

members should be able to leave reviews of the materials by describing how they used the

materials and by rating resources on various dimensions (e.g. rating on HCI content quality,

rating on pedagogy, etc.).

Several participants provided suggestions regarding the contribution process to the CoP. There

should be different ways of contributing to the community and different ways of participating

(e.g. a way to encourage junior members to contribute, even if they are not confident in

contributing resources). Some also mentioned that metadata entry should be simple and quick to

perform, with the option to bypass it. Thus, the contribution process should be as simple as

possible. In addition, comments and discussions should be organized around the resources to

increase participation and to encourage users to provide feedback related to resources. Finally,

contributions could also be made in the form of general discussions. Thus, the CoP should

include a general discussion area which is not based on particular resources, but where general

topics can be discussed (e.g., a discussion forum to share and ask for advice).

Copyright was a major concern raised by participants in our interviews. To address this issue, all

materials and resources should be shared openly for educational purposes, such as using Creative

Commons licensing.

According to the interviewees, content throughout the community should be based on a

framework that is relevant for HCI educators and practitioners. Previous research by Churchill,

Bowser, and Preece (2016) can be used to inform content organization. These authors have

already proposed a way to organize HCI education content based on their research with

educators, students, and practitioners about which topics in HCI are important for the future.

Content should not be limited to teaching materials, but should also include information on how

to teach the materials and other teaching practices/methods in HCI. Finally, the CoP should

17

support content that is relevant globally as well as content that is relevant only locally in specific

contexts.

When it comes to membership, participants in our interviews suggested that members of our CoP

should have personal profiles that are tied to their interaction in the community, such as

resources that are uploaded and downloaded, comments that are posted, reviews that are left, etc.

They also commented about the ability to incorporate sub-groups or smaller communities

(around countries, around subtopics, etc.) within the larger community to provide spaces to

discuss specialized and specific needs/topics.

18

Chapter 4 Study 2: Questionnaires

asdf

4.1 Method

A study was conducted consisting of individual printed questionnaires. The goal of administering

questionnaires was to collect data from Human-Computer Interaction (HCI) practitioners and

educators who will be potential end users of the HCI Living Curriculum Community of Practice

(CoP). The questionnaire questions were determined based on themes that emerged from the first

half of the study 1 interviews that were conducted prior to the CHI 2017 conference. The

questions were concerning versatility of the CoP, the necessity of incentives, methods of

evaluating quality, and feelings towards membership. In total, 28 participants were surveyed.

Participants were contacted via an announcement in person before sessions at the CHI2017

Conference in Denver, Colorado, May 6-11, 2017. Sessions were chosen based on their

relevance to HCI education.

4.2 Procedure

Questionnaires were completed in under 15 minutes and were conducted using the same

procedure. First, the researcher introduced herself to the participant and provided a brief

introduction to the study background. The participant was asked to read a consent form and

provide written consent. A background description of the motivation for a Living Curriculum in

HCI was provided to the participant. Then, the participant completed the questionnaire and

returned it to the researcher. The participant was then thanked for her participation. A copy of the

questionnaire used in study 2 is provided in Appendix A.

4.3 Data Collection and Analysis

Questionnaire responses were transcribed into a spreadsheet. Quantitative responses were

imported into SPSS and analyzed in terms of descriptive statistics and measures of association

between the measures collected. Participants were clustered into types using k-means cluster

analysis. To facilitate interpretability, questions were given short names. Question one is named

Versatility, question two is Incentive Value, question three is Evaluation Intensity, and question

19

four is Membership. Qualitative textual responses were analyzed to extract common themes to

describe the clusters.

4.4 Results

4.4.1 Description of the data

4.4.1.1 Correlations

The following Pearson Correlation tables were performed on data split by participant responses

to question four. Participants were categorized into those who did not agree that membership in

the community was required for access to resources (N=24), and those who did agree that

membership in the community was required for access to resources (N=4). It can be seen that the

correlations are fairly consistent for the entire sample of 28 vs. for the 24 people who did not

think that membership was required. There was a significant correlation between Evaluation

Intensity and Incentive Value, a modest (non-significant correlation) between Evaluation

Intensity and Versatility, and a close to zero correlation between Versatility and Incentive Value.

The results of additional data analysis of study 2 can be found in Appendix B.

Pearson Correlation Membership = 0 (24 participants who did not select that membership was required) Normed Evaluation

Intensity Normed Versatility Normed Incentive

Value Normed Evaluation Intensity

1 .285 .534**

Normed Versatility 1 -0.14 Normed Incentive Value

1

** Correlation is significant at the 0.01 level (2-tailed) Pearson Correlation for all 28 participants Normed Evaluation

Intensity Normed Versatility Normed Incentive

Value Normed Evaluation Intensity

1 .308 .542**

Normed Versatility 1 0.24 Normed Incentive Value

1

** Correlation is significant at the 0.01 level (2-tailed)

20

4.4.2 Stem-and-leaf plots

I used stem-and-leaf plots to visualize how measures differed between those who thought that

membership was needed and those who did not. The stem-and-leaf plots for Normed Evaluation

Intensity, Normed Versatility and Normed Incentive Value are described below. Since there were

only four people who thought that membership should be required, any differences observed

between the plots can only be suggestive.

4.4.2.1 Normed Evaluation Intensity

For Normed Evaluation Intensity, over half of the people in the no membership required group

had values between 0.25 and 0.5, where three of the four people in the membership required

group had a value of 0.5. The no membership required group ranges from 0 to 1, but the

membership required group only has values of 0.25 and 0.5.

Normed Evaluation Intensity Stem-and-Leaf Plot Membership = 0 (24 participants who did not select membership was required)

Frequency Stem & Leaf 3 0.000 8 0.255555555 10 0.50000000000 1 0.75 2 Extremes (>=1.00)

Normed Evaluation Intensity Stem-and-Leaf Plot Membership = 1 (4 participants who did select membership was required)

Frequency Stem & Leaf 1 0.25 3 0.5000

21

4.4.2.2 Normed Versatility

Taking into account the difference in frequency, the stem plots for the two groups were fairly

similar with about half of each group having a normed value of 0.66 with the remainder split

between values of 0.33 and 1.

Normed Versatility Stem-and-Leaf Plot Membership = 0 (24 participants who did not select membership was required)

Frequency Stem & Leaf 5 0.33333 11 0.66666666666 8 1.00000000

Normed Versatility Stem-and-Leaf Plot Membership = 1 (4 participants who did select membership was required)

Frequency Stem & Leaf 1 0.3 2 0.66 1 1.0

4.4.2.3 Normed Incentive Value

For Normed Incentive Value, over half of the people in the no membership required group had

values between 0.25 and 0.5, where 3 of the 4 people in the membership required group had a

value of 0.5. The no membership required group ranged from -0.25 to 0.75, but the membership

required group only has values of 0.25 and 0.5.

Normed Incentive Value Stem-and-Leaf Plot Membership = 0 (24 participants who did not select membership was required)

Frequency Stem & Leaf 2 Extremes (=<-0.25) 12 0.2555555555555 6 0.5000000 4 0.75555

Normed Incentive Value Stem-and-Leaf Plot Membership = 1 (4 participants who did select membership was required)

Frequency Stem & Leaf 1 0.25 3 0.5000

22

4.4.3 Cluster analysis

K-means analysis is a form of partitioning where the data analyst chooses the number of

partitions for each run. For the two-cluster solution that I selected, the procedure starts by

randomly selecting two cluster centroids. In the first iteration, all the participants are assigned to

the cluster with the closest centroid to them. The cluster centroids are then re-calculated based on

mean values of each clustering variable across the people who have been assigned to each

cluster. Participants are then reassigned to the other cluster if they are now closer to the

reassigned centroid for the other cluster than they are to the reassigned centroid that they were

previously closest to. Cluster centroids are again recalculated based on the new assignments. The

process continues until a new iteration does not produce a reassignment of any participants (i.e.,

the partitioning process has converged). Typically, convergence happens in relatively few

iterations in k-means cluster analysis. In this thesis, when the two-partition clustering converged

there were 16 participants in one cluster and 12 in the other cluster. Clustering variables were

normalized to be on the same scale for the analysis, and distances to centroids were calculated as

Euclidean distances.

Analysis of variance was used to identify which variables were distinguishing between the

clusters. Evaluation Intensity (F[1,26]=15.86, p<.001) and Incentive Value (F[1,26]=26.30,

p<.001) differed significantly between the clusters whereas Versatility did not (F=1.03).

4.4.3.1 Histograms

The following histograms illustrate the differences in Evaluation Intensity and Incentive Value

between Cluster 1 and Cluster 2. Cluster 1 is lower (than Cluster 2) in Normed Incentive Value

(see Figure 1). This suggests that people in Cluster 1 require fewer incentives to participate in the

HCI Living Curriculum, compared to Cluster 2. Cluster 1 is lower (than Cluster 2) in Normed

Evaluation Intensity (see Figure 2). This suggests that people in Cluster 2 desire more rigor

before material is accepted into the HCI Living Curriculum, as compared to those in Cluster 1.

23

Figure 1 - Normed Incentive Value

Figure 2 - Normed Evaluation Intensity

4.4.3.2 Qualitative analysis of clusters

Qualitative analysis of the clusters was performed based on the textual responses to question

five, regarding goals of the Living Curriculum, and question six, regarding obstacles preventing

participation in the Living Curriculum.

Overall, more responses to question five (goals of the Living Curriculum) related to community

from Cluster 1 (seven responses, 43.75% of respondents) than Cluster 2 (four responses, 33.33%

24

of respondents). These responses were classified by use of words such as “community,” “share,”

“sharing,” and “relations.” Additionally, more responses to question five related to education

from Cluster 1 (four responses, 25% of respondents) than Cluster 2 (two responses, 16.67% of

respondents). These responses were classified by use of words such as “education,” “educate,”

“curricula,” and “teaching.”

Overall, more responses to question six (obstacles to participation in the Living Curriculum)

related to concern about sharing personal materials from Cluster 2 (three responses, 25% of

respondents) than Cluster 1 (one response, 6.25% or respondents). Examples of responses

include “Desire to keep to oneself the stuff I have developed that sets me apart,” “It should be

clear who is using my material,” “Concerns about uncredited re-use of my materials,” and “Some

of my work is not publicly shareable.”

4.4.3.3 Summary of clusters

Cluster 1 is lower in Normed Incentive Value, suggesting a lower need for incentives to

participate in the Living Curriculum, and lower in Normed Evaluation Intensity, suggesting a

desired lower barrier for sharing content in the Living Curriculum. Additionally, Cluster 1 had

more responses relating to community and education when asked about goals of the Living

Curriculum, and fewer responses relating to concern about sharing personal materials when

asked about obstacles to participation in the Living Curriculum. This suggests that respondents in

Cluster 1 are more open and willing to participate in communities of practice such as the HCI

Living Curriculum, with little need for incentives, and a desire for more community members to

participate through sharing content.

Cluster 2 is higher in Normed Incentive Value, suggesting a higher need for incentives to

participate in the Living Curriculum, and higher in Normed Evaluation Intensity, suggesting a

desired higher barrier for sharing content in the Living Curriculum. Additionally, Cluster 2 had

fewer responses relating to community and education when asked about goals of the Living

Curriculum, and more responses relating to concern about sharing personal materials when asked

about obstacles to participation in the Living Curriculum. This suggests that respondents in

Cluster 2 are less open and willing to participate in communities of practice such as the HCI

Living Curriculum, with increased need for incentives, and a desire for increased rigor/scrutiny

before content can be shared.

25

4.5 Discussion

When I asked people explicitly about whether membership in the CoP should be needed to get

access to the CoP generated materials, only 4 of the 28 participants agreed that membership was

needed. However, cluster analysis based on Evaluation Intensity, Versatility, and Incentive

Value as clustering variables, produced a fairly even split of the participants into two groups

(with 16 and 12 participants, respectively). The 16 people in group 1 reported a lower need for

incentives to participate in the Living Curriculum, and lower Evaluation Intensity, suggesting a

desired lower barrier for sharing content in the Living Curriculum. These results suggest that it

may be difficult to assume that a large majority of potential users of TEACH-HCI would be

willing to share all their materials with the general public. While 24 out of 28 people in my

sample stated that membership was not needed, a significant fraction of those people appears to

hold attitudes that might in fact favour the membership approach. Further research, with a more

detailed questionnaire, is needed to determine the proportion of people who have attitudes that

are conducive to a membership required solution even if they don’t explicitly state that

membership is required (possibly due to social desirability factors).

26

Chapter 5 Design: Use Cases and Prototype

asdf

5.1 Design

This chapter presents the Human-Computer Interaction (HCI) Living Curriculum use cases and

prototype that were developed based on the results of studies 1 and 2, and the design

requirements derived from study 1. The use cases represent the possible user interactions with

the system. The prototype is presented as a set of wireframes relating to the use cases, with

descriptions concerning flow through the wireframes.

5.2 Use Cases

I developed a set of preliminary use cases based on the interview data collected and the design

requirements that resulted, presented in chapter 3. For example, participants described methods

of finding resources and types of resources they would expect to be available. Additionally, the

review of Open Educational Resources (OERs) in chapter 2 outlined the fundamental

functionalities expected in a Learning Object Repository (LOR). The use cases delineate the

various means in which users will use the system to accomplish various goals, including the

specific actions the user will take and those the system will take. Full use cases can be found in

Appendix C. The use case titles are as follows:

• Add course

• Add material

• View course

• Browse

• Search

• Advanced search

• View results

27

5.3 Prototype

I created low fidelity prototype using the software Axure. While the overall purpose of each

wireframe was from the use cases, many of the details of the interactions were from the affinity

diagram, such as the desire to have multiple measure of quality, peer and user reviews, and

discussions centered around materials. The incorporation of these details are described in the

sections below.

5.3.1 Adding content

An important aspect of these wireframes is how contributors will upload resources to the

Community of Practice (CoP), considered a critical feature by many participants. Given that

most resources will be tied to academic courses, a course-based framework has been adopted to

organize the uploaded resources. This will minimize the amount of metadata that needs to be

entered when multiple resources from the same course are uploaded. This section describes the

wireframes used for adding a course and adding material.

5.3.1.1 Add course

Users may add a course to TEACH-HCI (see Figure 3). Courses have associated attributes to

provide valuable information to the viewer, and to make them easier to find through searching

and browsing. Users fill out these attributes when adding a course. These attributes include

mandatory course details such as the creator, course name, description, level (undergraduate or

graduate), and associated keywords. Additionally, there are optional course details such as

course code, year, semester, and website. Institution details may be pre-populated from the user’s

profile, and include information about the institution, faculty, department, and program of the

course. When material is added, associated metadata is populated such as the date added.

28

Figure 3 - Add course wireframe

5.3.1.2 Add material

Users may add material to TEACH-HCI (see Figure 4). Users may choose to add material to a

course, in which case they are catalogued together. Associating material to a course is optional.

Materials have associated attributes to provide valuable information to the viewer, and to make

them easier to find through searching and browsing. Users fill out these attributes when adding

material. These attributes include mandatory material details such as material name, description,

type, associated keywords, and files. There are also optional material details such as associated

course. Additional details may be pre-populated from its course (if included), and include the

creator and level. When material is added, associated metadata is populated such as the date

added.

29

Figure 4 - Add material wireframe

5.3.2 Searching and viewing content

Searching will be one of the main ways that users will retrieve resources on TEACH-HCI. This

section describes the wireframes for searching, viewing search results, viewing material, and

viewing courses.

5.3.2.1 Search and browse

The Search page (see Figure 5)will often be the access point for users who are looking for

resources. This page contains a search field where users may enter search terms. Search terms

may include keywords, course or material titles, creator names, or other terms. The Advanced

Search button also gives users access to additional input areas where they may specify which

fields they would like to search and enter details. When a user enters a query in the search field

and selects the magnifying glass, they are taken to the search results page, described in the next

section.

The Popular Keywords section of the Search page depicts a word cloud with the associated

keywords from courses and materials that have been input most frequently. This will be the most

30

common method of browsing material in TEACH-HCI. The size of the text of the term

represents its popularity. When a user selects a term from the Popular Keywords section, they are

taken to the Search Results page, described in the next section.

The Featured Material section of the Search page showcases material that has received high

ratings and significant attention from the community. High ratings are measured by peer review

ratings as well as user ratings. Attention from the community is measured by the number of user

ratings, number of discussion comments, and number of views. The formatting and contents of

the Featured Material section resemble entries in the Search Results page, described in the next

section.

Figure 5 - Search and browse wireframe

5.3.2.2 View search results

When a user is searching or browsing TEACH-HCI, they will encounter the Search Results page

(see Figure 6). In the search field, this page shows the term that was searched or the keyword that

was selected from the Search page. In this example, the keyword prototype was selected from the

Search page. If the user wants to enter a new search into the search field, they have the option of

31

keeping search filters or starting a new search. The Advanced Search button may be accessed at

any time and provides detailed searching information as described in the previous section.

Below the search field, the Search Results page shows the number of resources returned from the

search, and the numbers of the entries that are currently being displayed on the page. In this

example, seven items were returned and items one through seven are shown on the page. On the

left side of the page, users may select options from the Refine your Search panel. Refinements

may be made based on material type, course level, keywords, or author. In this example, the

search has been restricted to in-class delivery material. On the right side of the page, users may

select a method to sort their search. Search results may be sorted by overall rating, date (newest

first), date (oldest first), material type, title, or author. In this example, the search has been sorted

by overall rating.

The majority of the page is used to display the actual results that met the search criteria. In this

example, two of the results are visible before the user must scroll down. The results show a

number of details that were entered at the time the material was created, including name of the

material, description, creator, material type, course level, and keywords. The right side of each

result panel contains a Save to Collection button. Selecting this button saves the material to the

user’s collections, and facilitating quick access at a later time. The right side of each result panel

also displays the material’s community information. This includes its average peer review rating,

its average user rating, and the number of comments made in the discussion section for that

material. Ratings and discussions are discussed in the next section.

32

Figure 6 - View search results wireframe

5.3.2.3 View material

When a user chooses to view a material, they are taken to a page dedicated to that material (see

Figure 7). This page contains the same resource details and community details described in the

Search Results page in the previous section. Additionally, the right side of the page contains two

methods of interacting with the material. Users may select the Download Material button to

access the resource. Users may select a number of stars to rate the material. This rating

represents a user rating. Any submitted values will be added to the overall calculation of the

average user rating for that material.

Below these sections are the Peer Review panel, User Rating panel, and Users who viewed this

material also viewed… panel. The Peer Review and User Rating panels show the average rating,

the number of total ratings, bars depicting a breakdown of the ratings, and a link to all submitted

reviews for peer reviews and user ratings, respectively. In this example, the material has a peer

review rating of five stars, has received three peer reviews, and all three peer reviewers rated the

material with five stars. The Users who viewed this material also viewed… panel displays the

titles, peer ratings, and user ratings or materials that are recommended for the user by the system.

The system indexes the other materials that users who have accessed the selected material have

33

accessed. The materials displayed here constitute the other materials that previous users have

accessed most frequently. Three materials are displayed on the panel at one time. The left and

right arrow buttons may be selected to change the panel to display three additional materials.

The Discussion section is below these panels. This section displays comments that have been

made related to the selected material. Additionally, this section also includes details related to the

discussion, such as the comment author’s name and display picture, and the time the comment

was made. The comment author’s name and display picture are linked to the profile page of that

user, described below. There are also buttons for the actions that can be taken related to the

discussion, such as leaving a new comment, replying to an existing comment, or quoting an

existing comment. In this example, the most recent comment is visible before the user must

scroll down. This comment was made two weeks ago by Jane Doe who is the author of the

material. The right side of the discussion section shows the display picture of the participants

who have contributed to the discussion. The display pictures are linked to the profile page of

each user. In this example, there have been seven participants.

Figure 7 - View material wireframe

34

5.3.2.4 View course

When a user chooses to view a course, they are taken to a page dedicated to that course (see

Figure 8). This page contains the same resource details and community details described in the

Search Results page above. Additionally, the right side of the page contains two methods of

interacting with the course. Users may select the Download Material button to access all

resources associated with the course. This downloads a .zip file with all the course materials.

Users may select a number of stars to rate the course. This rating represents a user rating. Any

submitted values will be added to the overall calculation of the average user rating for that

course.

The Course Material tab is displayed below this section. This tab resembles content on the

Search Results page described above, and functions in the same way. In this example, the course

contains 41 materials. One course material is displayed before the user must scroll down. The

materials are sorted by material type. The Course Comments tab resembles the discussion section

of the material page described in the previous section, and functions in the same way.

Figure 8 - View course wireframe

35

5.3.3 Connecting with others

It is important to allow users to connect with others for TEACH-HCI to be a successful CoP.

Users’ Profile pages are one way for community members to connect and learn about each other.

5.3.3.1 View profile

A user’s Profile page (see Figure 9) contains information about the user, information about the

user’s participation in the community, and methods of interacting with the user. The first section

of the page displays the user’s name, display picture, description, position, institution, faculty,

department, and keywords describing their expertise. The right side of the page contains a

Follow User button and a Contact User button. Selecting the Follow User button will create a

subscription to updates about that user’s actions in the community, such as added material,

content, or comments. Selecting the Contact User button allows for communication with that

user through a messaging system.

The User’s Material tab, User’s Comments tab, and User’s Community tab. The User’s Material

tab resembles the Search Results page described above, and functions in the same way. The

resources included in this tab consist of materials and courses that that user has posted. In this

example, one course and one material are shown before the user must scroll down. The User’s

Comments tab is similar to the discussion section of the material page described above, and

functions in the same way. The User’s Community tab displays the names and display pictures of

other users that that user has subscribed to.

36

Figure 9 - View profile wireframe

5.4 Conclusion

The use cases and wireframes represent a collection of preliminary envisioned interactions with

the HCI Living Curriculum. These range from uploading materials to the system, to searching for

resources, and interacting with others in the community. The evaluation of the use cases and

wireframes was not in the scope of this thesis, but should be pursued in future work.

37

Chapter 6 Contributions, Limitations, and Future Work

asdf

This chapter lists the contributions that were made in this thesis and also discusses the limitation

of this research and prospects for future research. The chapter concludes with some final words.

6.1 Contributions

The following research contributions were made in this thesis. All of these contributions are

relevant to the design and evaluation of the Human-Computer Interaction (HCI) Living

Curriculum.

1. Identification of stakeholder requirements for how a HCI Living Curriculum should

operate and work

2. Identification of barriers to the use of a HCI Living Curriculum

3. Identification of different types of potential user for a HCI Living Curriculum

4. Initial design of community of practice for the HCI Living Curriculum

5. Identification of features that can be used to evaluate a HCI Living Curriculum

6.2 Limitations

There are a few limitations to this thesis. First, the prototype was not evaluated with potential

end users, as this was not within the scope of the thesis. Second, only 28 people were surveyed.

HCI educators are busy people, and it was difficult to get access to them. Third, the coding of

interview data was only done by one person, because the affinity diagram was used as an

ideation technique. Reliability metrics are typically not calculated for affinity diagrams. Finally,

the questionnaire used was relatively short. Based on the results obtained, there seemed to be a

mismatch between the attitudes that some people reported, and their opinion of whether

membership was required or not. A longer questionnaire might help to more precisely delineate

the issues concerning attitudes towards incentives and sharing of content, and perhaps provide

insight into whether the clusters differ on additional values like demographics and trust.

38

However, a longer questionnaire might also lower the response rate that could be achieved with a

reasonable amount of effort.

6.3 Future work

Further development of the HCI Living Curriculum should proceed beyond the prototyping

stage. First, the prototype should be evaluated by conducting a cognitive walkthrough with

representative end users. There should also be investigation into the issues of who or what will

host the HCI Living Curriculum, and how the required infrastructure will be funded. In addition,

the questionnaire should be redesigned so as to develop a more sensitive measure of the

membership required construct, and one that may be less susceptible to social desirability bias.

6.4 Final words

HCI is a fast-moving field and the HCI Living Curriculum is sorely needed. Ideally, educators

would share materials with each other liberally, and best practices would emerge that could be

promoted by the community at large.

My research findings suggest that attitudes towards the need for membership, and towards

evaluation intensity and incentives need to be better understood before implementing TEACH-

HCI.

In this thesis, I collected a novel set of data concerning some attitudes that people have that are

relevant to the development of the HCI Living Curriculum. While the research has raised some

issues that need to be addressed, I believe that the HCI Living Curriculum should be developed

as soon as is practicable. However, more research needs to be carried out on some of the relevant

attitudes that people have, prior to the development and implementation of the HCI Living

Curriculum.

39

References

Andrew, N., Ferguson, D., Wilkie, G., Concoran, T., & Simpson, L. Developing professional

identity in nursing academics: the role of communities of practice. Nurse Education

Today, 29(6), 607-611. https://www.ncbi.nlm.nih.gov/pubmed/19250718

Archambault, P. M., van de Belt, T. H., Grajales III, F. J., Faber, M. J., Kuziemsky, C. E.,

Gognan, S. … Légaré, F. (2013). Wikis and collaborative writing applications in health

care: a scoping review. Journal of medical Internet research, 15(10), e210.

https://www.ncbi.nlm.nih.gov/pubmed/24103318

Barnes, S., Li, F., Polyakov, S., Xu, H. & Moen, W. (2008). A repository for learning objects:

supporting the reuse and repurposing of redsigned courses and their content. American

Society for Information Science and Technology, 45(1).

http://dx.doi.org/10.1002/meet.2008.1450450282

Bates, M., Loddington, S., Manuel, S., & Oppenheim, C. (2007). Attitudes to the rights and

rewards for author contributions to repositories for teaching and learning. Research in

Learning Technology, 15(1), 67-82. http://dx.doi.org/10.1080/09687760600837066

Bolisani, E., & Scarso, E. (2014). The place of communities of practice in knowledge

management studies: A critical review. Journal of Knowledge Management, 18(2), 366-

381. http://dx.doi.org/10.1108/JKM-07-2013-0277

Boyd, D. M. & Ellison, N. B. (2007). Social network sites: definition, history, and scholarship.

Journal of Computer-Mediated Communication, 13(1), 210-230.

http://onlinelibrary.wiley.com/doi/10.1111/j.1083-6101.2007.00393.x/full

Brown, J. S., & Duguid, P. (2001). Structure and spontaneity: knowledge and organization. In I.

Nonaka & D. J. Teece (Eds.), Managing Industrial Knowledge: Creation, Transfer and

Utilization (44-67). London: Sage.

Campbell, J., Aragon, C., Davis, K., Evans, S., Evans, A., Randall, D. Thousands of positive

reviews: distributed mentoring in online fan communities. Proceedings CSCW ’16: The

40

19th ACM Conference on Computer-Supported Cooperative Work & Social Computing.

San Francisco, CA: ACM.

Cho, H., Chen, M. & Chung, S. (2010). Testing an integrative theoretical model of knowledge-

sharing behavior in the context of Wikipedia. Journal of the American Society for

Information Science and Technology, 61(6), 1198-1212.

http://onlinelibrary.wiley.com/doi/10.1002/asi.21316/full

Churchill, E. F., Bowser, A., & Preece, J. (2013). Teaching and learning human-computer

interaction: past, resent, and future. interactions, 20(2), 44-53.

http://dx.doi.org/10.1145/2427076.2427086

Churchill, E. F., Bowser, A., & Preece, J. (2016). The future of HCI education: a flexible, global,

living curriculum. interactions, 23(2), 70-73. http://dx.doi.org/10.1145/2888574

Churchill, E. F., Preece, J., & Bowser, A. (2014). Developing a living HCI curriculum to support

a global community. Proceedings from CHI EA ’14: CHI ’14 Extended Abstracts on

Human Factors in Computing Systems. Toronto, ON: ACM.

Clements, K., Pawlowski, J., & Manouselis, N. (2015). Open educational resources repositories

literature review: towards a comprehensive quality approaches framework. Computers in

Human Behavior, 51(B), 1098-1106. https://doi.org/10.1016/j.chb.2015.03.026

Dimitriadis, Y., McAndrew, P., Conole, G., & Makriyannis, E. New design approaches to

repurposing open educational resources for collaborative learning using mediating

artefacts. Proceedings from ascilite 2009: Same places, different spaces. Auckland, NZ.

Farič, N. & Potts, H. W. (2014). Motivations for contributing to health-related articles on

Wikipedia: an interview study. Journal of medical Internet research, 16(1), e260.

http://doi.org/10.2196/jmir.3569

Hew, K. F., & Hara, N. (2007). Empirical study of motivators and barriers of teacher online

knowledge sharing. Educational Technology Research and Development, 55(6), 573-595.

https://doi.org/10.1007/s11423-007-9049-2

41

Hewett, T. T., Baecker, R., Card, S., Carey, T., Gasen, J., Mantei, M., Perlman, G., Strong, G., &

Verplank, W. (1992). ACM SIGCHI: Curricula for Human-Computer Interaction. New

York: ACM.

Jemielniak, D. (2016). Wikimedia movement governance: the limits of a hierarchical

organization. Journal of Organizational Change Management, 29(3), 361-378.

https://doi.org/10.1108/JOCM-07-2013-0138

Joinson, A. N. (2008). Looking at, looking up or keeping up with people?: motives and use of

facebook. Proceedings from CHI ’08: CHI08. New York, NY: ACM.

Jordan, K. (2015). Exploring the ResearchGate score as an academic metric: reflections and

implications for practice. Proceedings from ASCW ’15: Quantifying and Analysing

Scholarly Communication on the Web. Oxford, UK.

Jovanovic, A., St-Cyr, O., & Chignell, M. (2017). Designing the HCI living curriculum.

Proceedings from CEEA ’17: Canadian Engineering Education Association Conference.

Toronto, ON: Canada.

Marlow, J. & Dabbish, L. (2014). From rookie to all-star: professional development in a graphic

design community of practice. Proceedings from CSCW ’14: CSCW14. Baltimore, MD:

ACM.

McGreal, R. (2008). A typology of learning object repositories. In Handbook on information

technologies for education and training (5-28). Berlin Heidelberg: Springer.

McInnis, B. J., Murnane, E. L., Epstein, D., Cosley, D., & Leshed, G. (2016). One and done:

factors affecting one-time contributors to ad-hoc online communities. Proceedings from

CSCW ’16, 19th ACM conference on Computer Supported cooperative work & social

computing. San Francisco, CA: ACM.

Monge, S., Ovelar, R., & Azpeitia, I. (2008). Repository 2.0: Social dynamics to support

community building in learning object repositories. Interdisciplinary Journal of E-

Learning and Learning Objects, 4(1), 191–204.

42

Nesbit, J., Belfer, K. & Vargo, J. (2002). A convergent participation model for evaluation of

learning objects. Canadian Journal of Learning and Technology/La revue canadienne de

l’apprentissage et de la technologie, 28(3).

https://www.cjlt.ca/index.php/cjlt/article/view/26569/19751

O’Sullivan, D. (2009). Wikipedia: A new community of practice?. Burlington, VT: Ashgate

Publishing.

Rotman, D., Procita, K., Hansen, D., Parr, C. S., Preece, J. (2012). Supporting content curation

communities: the case of the encyclopedia of life. Journal of the American Society for

Information Science and Technology, 63(6), 1092-1107.

http://onlinelibrary.wiley.com/doi/10.1002/asi.22633/full

SIGCHI Education Community. http://www.sigchi.org/communities/hci-ed/

Stuart, C., Dabbish, L., Kiesler, S., Kinnaird, P. & Kang, R. (2012). Social transparency in

networked information exchange: a theoretical framework. Proceedings from CSCW ’12:

CSCW. Seatlle, WA: ACM.

Wasko, M. M. & Faraj, S. (2005). Why should I share? Examining social capital and knowledge

contribution in electronic networks of practice. MIS Quarterly, 29(1), 35-37.

https://www.jstor.org/stable/25148667

Wenger, E., McDermott, R. A., & Snyder, W. (2002). Cultivating communities of practice: A

guide to managing knowledge. Boston, Mass: Harvard Business School Press.

43

Appendices

Appendix A: Questionnaire

Client Information Sheet and Informed Consent Form for the Study: Study Name: Developing a living curriculum for HCI education Researchers: Dr. Mark Chignell, Andrea Jovanovic, Dr. Olivier St-Cyr Research Rationale: Contribute to the advancement of knowledge concerning the requirements and implementation of a living curriculum. Research Procedure You are invited to voluntarily take part in this study if you are 18 years of age or older, fluent in English, and a current practitioner or educator in the field of HCI. You are free to refuse to participate or withdraw from the study at any time without penalty. However, data cannot be withdrawn after the data have been aggregated and de-identified. You will be asked to answer questions and give your opinions concerning the requirements and implementation of a living curriculum. No personal or identifying information will be included in written reports or presentations and all data will be kept in a locked office and on a password encrypted computer, accessible by only the investigators. All data will be securely stored until December 2022, after which all data will be destroyed. Interested participants may submit their email address to receive a summary of the results of the study. This summary will be emailed within three months of completion of the study. Contact Information If you have any questions about this study, please email andrea.jovanovic@mail.utoronto.ca. If you have any questions about your rights as a research participant, please contact the Office of Research Ethics (ethics.review@utoronto.ca, 416-946-3273). The research study you are participating in may be reviewed for quality assurance to make sure that the required laws and guidelines are followed. If chosen, (a) representative(s) of the Human Research Ethics Program (HREP) may access study-related data and/or consent materials as part of the review. All information accessed by the HREP will be upheld to the same level of confidentiality that has been stated by the research team. Please read the following and choose the appropriate box. I ___________________________, have read this consent form and understand its contents. I understand that my responses will be kept confidential, except that the resulting information will be summarized and may be presented in publications of the results. I understand that I am free to withdraw before or anytime during the study without penalty. � I voluntarily agree to participate in this study � I DO NOT agree to participate in this study

44

HCI Education Questionnaire Communities of Practice are groups “of people who share a concern, a set of problems, or a passion about a topic, and who deepen their knowledge and expertise in this area by interacting on an ongoing basis” (Wenger, McDermott, & Snyder, 2002, p.4). We envision a virtual Community of Practice for educators and practitioners based on HCI education, and are looking for your feedback about how this online community should be implemented. This research is inspired by the work done on the HCI Living Curriculum (Churchill, Bowser, & Preece, 2016). Elizabeth F. Churchill, Anne Bowser, and Jennifer Preece. (2016). The future of HCI education:

a flexible, global, living curriculum. interactions 23, 2 (February 2016), 70-73. DOI=http://dx.doi.org/10.1145/2888574

Wenger, E., McDermott, R., & Snyder, W. M. (2002). Cultivating communities of practice.

Boston: Harvard Business School Press. 1. How do you envision a virtual Community of Practice based on HCI education? Select all that apply. � A social network for discussion about HCI education-related topics � A collection of HCI teaching resources such as course syllabi, slides, assessments, etc. � A collection of research-based best practices for teaching and learning HCI � Other(s): ________________________________________________________________

________________________________________________________________ ________________________________________________________________

Comments: ________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ 2. Do you think incentives would be required to encourage participation in this community? � I do not think incentives would be required � Recognition by an accredited body or institution � Information about the use of my uploaded resources, such as number of downloads � Knowing that my contribution may help others in the community � Other(s): ________________________________________________________________

________________________________________________________________ ________________________________________________________________

45

Comments: ________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ 3. What method(s) of quality control do you think would be effective to differentiate between the quality of different resources? Select all that apply. � “Likes” or “Thumbs up” determined by the community � Ratings (e.g. out of 5 stars) determined by the community � Ratings (e.g. out of 5 stars) determined by a review committee of experts � Review committee of experts vets all resources and only selects the highest quality � Other(s): ________________________________________________________________

________________________________________________________________ ________________________________________________________________

Comments: ________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ 4. What are your thoughts about the use and copyright of resources? Select all that apply. � All resources should be made available under Creative Commons licenses (or other licenses that allow others to use, alter, and distribute works at least non-commercially) � Access to resources should require membership in the community � Other(s): ________________________________________________________________

________________________________________________________________ ________________________________________________________________

Comments: ________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________

46

5. What do you think are the goals of a Community of Practice based on HCI education? __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ 6. Do you see any obstacles that would prevent you from contributing to a Community of Practice based on HCI education? __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ 7. Do you have any additional comments? __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ Thank you for completing this questionnaire. We would like to discuss more with our participants by conducting one-on-one interviews. If you are able to participate in a short interview about your visions for a Community of Practice in HCI education, please provide your name and email.

Name: _______________________________________

Email: _______________________________________

47

Appendix B: Additional Data from Study 2

Box plots

48

Cluster 1 and 2 box plots

49

Crosstabulations

NormedIncentiveValue * NormedVersality Crosstabulationa Count

NormedVersality

Total .33 .67 1.00

NormedIncentiveValue -.25 1 0 1 2

.25 2 5 5 12

.50 1 5 0 6

.75 1 1 2 4

Total 5 11 8 24

a. 4b Membership = 0

NormedIncentiveValue * NormedVersality Crosstabulationa Count

NormedVersality

Total .33 .67 1.00

NormedIncentiveValue .25 1 0 0 1

.50 0 1 1 2

.75 0 1 0 1

Total 1 2 1 4

a. 4b Membership = 1

50

Appendix C: Use Cases

Add course

1. User selects to add course �

2. System prompts for course attributes �

3. User enters course attributes �

4. User selects to save course �

5. System adds course �

6. System displays course page �

Add material

1. User selects to add material �

2. System prompts for material �

3. System prompts for material attributes �

4. User selects to save material �

5. System adds material

6. System displays material page

51

View course

1. User selects to view course �

2. System displays course attributes �

3. System displays course material �

4. System displays subscription options �

5. System displays ratings and discussion information �

6. System displays history �

Browse

1. System displays topics �

2. User selects topic �

3. System displays courses and course items tagged for the selected topic �

4. Go to “View results” use case �

Search

1. User selects search bar �

2. User enters search criteria �

3. User selects to search �

4. System display search results �

5. Go to “View results” use case �

52

Advanced search

1. User selects advanced search �

2. User enters search criteria �

3. User selects any of the advanced search filtering options �

4. Go to “View results” use case �

View results

Pre-condition: User has selected a topic to browse, performed a search, or performed an

advanced search

1. System displays results grouped by type and sorted by rating �

2. User selects a sorting option �

3. System displays results sorted by selected option �

4. User selects a filtering option �

5. System displays results filtered by selected option �

53

Appendix D: Ethics Protocol

Office of the Vice-President, Research and Innovation Human Research Ethics Program

UT HREP – Application Form for Supervised/Sponsored Research 1 of 25

12 Queen’s Park Crescent West – McMurrich Building, 2nd

floor Version Date: May/16

Office Use Only Application Number:

ETHICS REVIEW APPLICATION FORM FOR

SUPERVISED AND SPONSORED RESEARCHERS (For use by graduate students, post-docs, residents, external investigators, and visiting

professors/researchers)

SECTION A – GENERAL INFORMATION

1. TITLE OF RESEARCH PROJECT

Developing a living curriculum for HCI education 2. INVESTIGATOR INFORMATION Investigator:

Title (e.g., Dr., Ms., etc.): Ms.

Name: Andrea Jovanovic

Department (or organization if not affiliated with U of T): Mechanical and Industrial Engineering Mailing address: Attn: Interactive Media Lab, 5 King’s College Road, Toronto, ON, M5S 3G8 Phone: 647-637-0499 Institutional e-mail: andrea.jovanovic@mail.utoronto.ca

Level of Project:

Student Research: Doctoral ☐ Masters ☒ Post-Doctoral Research ☐ Visiting professor/External researcher ☐ Course Based ☐ CBR/CBPR ☐ Other ☐ (specify: )

Supervisor/Sponsor (must be a UofT faculty member with research privileges):

Title: Dr. Name: Mark Chignell Department: Mechanical and Industrial Engineering Mailing address: Attn: Interactive Media Lab, 5 King’s College Road, Toronto, ON, M5S 3G8 Phone: 416-978-8951 Institutional e-mail: chignell@mie.utoronto.ca

Co-Investigators: Are co-investigators involved? Yes ☒ No ☐

Before you start, familiarize yourself with:

TCPS2 Application instructions

Office FAQs

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 2 of 25

Title: Dr. Name: Olivier St-Cyr Department (or organization if not affiliated with U of T): Faculty of Information Mailing address: 140 St. George Street, Bissell Building, Room 614, Toronto, Ontario, M5S 3G6 Phone: 416-978-8876 Institutional e-mail: olivier.st.cyr@utoronto.ca

Title: Name: Department (or organization if not affiliated with U of T): Mailing address: Phone: Institutional e-mail:

Please append additional pages with co-investigators’ names if necessary. 3. UNIVERSITY OF TORONTO RESEARCH ETHICS BOARD:

Social Sciences, Humanities and Education Health Sciences

HIV/AIDS

To determine which Research Ethics Board (REB) your application should be submitted, please consult: http://www.research.utoronto.ca/about/boards-and-committees/research-ethics-boards-reb/

4. LOCATION(S) WHERE THE RESEARCH WILL BE CONDUCTED:

(a) If the research is to be conducted at a site requiring administrative approval/consent (e.g., in a school), please include all administrative consent letters. It is the responsibility of the researcher to determine what other means of approval are required, and to obtain approval prior to starting the project.

University of Toronto Hospital specify site(s) School board or community agency specify site(s) Community within the GTA specify site(s) International specify site(s) CHI 2017 Conference in Denver, Colorado Other specify site(s) online using Skype, on the phone

(b) For all off-campus research, whether in the local community or internationally, the researcher should consult with the Framework on Off-Campus Safety, Guidelines on Off-Campus Safety, and Guidelines on Safety in Field for institutional requirements. (c) The University of Toronto has an agreement with the Toronto Academic Health Sciences Network (TAHSN) hospitals regarding ethics review of hospital-based research where the University plays a peripheral role. Based on this agreement, certain hospital-based research may not require ethics review at the University of Toronto. If your research is based at a TAHSN hospital, please consult the following document to determine whether or not your research requires review at the University of Toronto. http://www.research.utoronto.ca/faculty-and-staff/research-ethics-and-protections/humans-in-research/ - “Administrative review” heading toward the bottom of the page. 5. OTHER RESEARCH ETHICS BOARD APPROVAL(S) (a) Does the research involve another institution or site? Yes No (b) Has any other REB approved this project? Yes No

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 3 of 25

If Yes, please provide a copy of the approval letter upon submission of this application. If No, will any other REB be asked for approval?

Yes (please specify which REB) No 6. FUNDING OF THIS PROJECT

(a)

Funding Status Source and Type Details

Funded Agency: Fund #: 4 (6 digits)

Agency: Fund # :4 (6 digits)

Applied for funding

Agency: Submission date:

Agency: Submission date:

Unfunded If unfunded, please explain why no funding is needed: This research will be undertaken as part of an MASc thesis. Participants will not be compensated financially for their participation.

7. CONTRACTS AND AGREEMENTS (a) Is this research to be carried out as a contract or under a research agreement? Yes No If yes, is there a University of Toronto funding or non-funded agreement associated with the research? Yes No

If Yes, please append a copy of the agreement with of this application.

Is there any aspect of the contract that could put any member of the research team in a potential conflict of interest? Yes No If yes, please elaborate under #10. (b) Is this a Division 5, Health Canada regulated clinical trial that involves drugs, devices or natural health products? Yes No (if so, the application must be reviewed by the full board) 8. PROJECT START AND END DATES Estimated start date for the component of this project that involves human participants or data: April 2017 Estimated completion date of involvement of human participants or data for this project: December 2017

9. SCHOLARLY REVIEW:

(a) Please check one:

I. The research has undergone scholarly review by thesis committee, departmental review committee, peer review committee or some other equivalent (Specify review type – e.g., departmental research committee, supervisor, CIHR, SSHRC, OHTN, etc.):

II. The research will undergo scholarly review prior to funding (Specify review committee – e.g., departmental research committee, SSHRC, CIHR peer-review committee, etc.):

III. The research will not undergo scholarly review (Please note that all research greater than minimal risk requires scholarly review)

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 4 of 25

(b) If box I or II above was checked, please specify if:

The review was/will be specific to this application

The review was/will be part of a larger grant

10. CONFLICTS OF INTEREST (a) Will the researcher(s), members of the research team, and/or their partners or immediate family members: (i) Receive any personal benefits (e.g., financial benefit such as remuneration, intellectual property rights, rights of employment, consultancies, board membership, share ownership, stock options, etc.) as a result of or in connection with this study? Yes No (ii) If Yes, please provide further details and discuss how any real, potential or perceived conflicts of interest will be managed during the project. (Do not include conference and travel expense coverage, or other benefits which are considered standard for the conduct of research.)

Not applicable.

(b) Describe any restrictions regarding access to or disclosure of information (during or at the end of the study) that have been placed on the investigator(s). These restrictions include controls placed by the sponsor, funding body, advisory or steering committee.

Not applicable.

(c) Where relevant, please explain any pre-existing relationship between the researcher(s) and the researched (e.g., instructor-student; manager-employee; clinician-patient; minister-congregant). Please pay special attention to relationships in which there may be a power differential – actual or perceived. Not applicable.

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 5 of 25

SECTION B – SUMMARY OF THE PROPOSED RESEARCH 11. RATIONALE Describe the purpose and scholarly rationale for the proposed project. State the hypotheses/research questions to be examined. The rationale for doing the study must be clear. Please include references in this section.

Purpose and Scholarly Rationale for the Proposed Project From 2011-2014, the ACM SIGCHI (Association for Computing Machinery, Special Interest group on Computer and Human Interaction) Executive Committee sponsored a project to investigate the present and future of Human-Computer Interaction (HCI) education [1]. Research consisted of 52 interviews conducted with SIGCHI community members, as well as 616 surveys completed in English, 156 in Brazilian Portuguese, 52 in Mandarin Chinese, and 48 in Chilean Spanish. Questions focused on what educators, practitioners, and students considered to be top priorities for the field of HCI. Additionally, educational resources were compiled and discussions were hosted at the annual CHI conferences, including discussion lunches, HCI education workshops, and SIGCHI Town Hall meeting discussions [1]. A recurring theme that emerged throughout the project was participants’ desire for a collection of online resources shared among HCI educators. The goal would be to create a connected social network of HCI scholars and educators, sharing and collaborating to develop course outlines, curricula, and teaching material, known as a living curriculum [2]. In order to preserve effectiveness, these resources would need to be frequently updated and maintained. Functionality for discussion and commenting would be essential in supporting the objective of content co-development. A workshop on developing the HCI living curriculum was held at the SIGCHI 2014 conference [3]. Several visions of what this new curriculum could be were presented. However, to this day, the proposed HCI living curriculum has not been designed nor built. Rationale for Conducting This Study The current infrastructure of the SIGCHI HCI Education Community does not support the envisioned communication and sharing of resources. Before the living curriculum can become a reality, there is a need to investigate outside community platforms or methods of extending the current infrastructure, and visions for the living curriculum. This research aims to investigate the use cases and requirements for the design of the HCI living curriculum. Research Questions to be Examined 1. How do stakeholders envision a HCI living curriculum? 2. What are the requirements for a HCI living curriculum? 3. What are the barriers to a HCI living curriculum? References 1. Elizabeth F. Churchill, Anne Bowser, and Jennifer Preece. 2013. Teaching and learning human-computer interaction: past, present, and future. interactions 20, 2 (March 2013), 44- 53. DOI=http://dx.doi.org/10.1145/2427076.2427086 2. Elizabeth F. Churchill, Anne Bowser, and Jennifer Preece. 2016. The future of HCI education: a flexible, global, living curriculum. interactions 23, 2 (February 2016), 70-73. DOI=http://dx.doi.org/10.1145/2888574 3. Elizabeth F. Churchill, Jennifer Preece, and Anne Bowser. 2014. Developing a living HCI curriculum to support a global community. In CHI '14 Extended Abstracts on Human Factors in Computing Systems (CHI

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 6 of 25

EA '14). ACM, New York, NY, USA, 135-138. DOI=http://dx.doi.org/10.1145/2559206.2559236

12. METHODS (a) Please describe all formal and informal procedures to be used. Describe the data to be collected, where and how they will be obtained and how they will be analyzed.

The proposed study is a qualitative study that will consist of a series of individual semi-structured interviews and in-person paper-based questionnaires (20-40 participants total). Interviews will take place online using Skype or Google chat, on the phone, in person at the University of Toronto St. George Campus, and in person at the SIGCHI 2017 Conference in Denver, Colorado, May 6-11, 2017. Questionnaires will be paper-based and will take place in-person. One researcher will conduct each individual interview and administer the in-person paper-based questionnaires. The goal of the individual interviews and questionnaires is to investigate the use cases and requirements of the living curriculum with HCI practitioners and educators who are potential end users of the system. Procedure Each interview and questionnaire is expected to last 15 minutes to one hour. The procedure for each interview session and questionnaire is as follows: 1. The researcher will introduce him or herself to the participant and provide a brief introduction to the study background and procedures. The participant will be asked to read the consent form (Appendix A) and provide verbal consent. 2. The questions will follow a preset guide (Appendix B). 3. The study will conclude. The participant will be debriefed and thanked for their participation. Data Collection and Analysis Interviews will be audio recorded so that content can be transcribed into a computerized database and analyzed using qualitative data analysis software. Questionnaires will be transcribed into a computerized database and analyzed using qualitative data analysis software. (b) Attach a copy of all questionnaires, interview guides and/or any other instruments. (c) Include a list of appendices here for all additional materials submitted (e.g., Appendix A – Informed Consent; Appendix B – Interview Guide, etc.):

Appendix A –Consent Forms Appendix B – Question Guide Appendix C – Recruitment Notice

13. PARTICIPANTS, DATA AND/OR BIOLOGICAL MATERIALS (a) Describe the participants to be recruited list the eligibility criteria, and indicate the estimated sample size (i.e. min-max # of participants). Where applicable, please also provide a rationale for your choice in sample size and/or sample size calculation.

Participants to be Recruited 20-40 participants will be recruited from the HCI professional and academic community. Inclusion Criteria The inclusion criteria for participants are as follows: 1. 18 years of age or older

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 7 of 25

2. Fluent in English 3. Current practitioner or educator in the field of HCI Exclusion Criteria The exclusion criteria for participants are as follows: 1. Under 18 years of age 2. Not fluent in English 3. Not a current practitioner or educator in the field of HCI Personally Identifiable Information No personally identifiable data will be collected from the subjects. Sample Size This is a qualitative and exploratory study; thus, no formal sample size calculations will be performed. The numbers of participants should be sufficient to get the findings we need to set the direction for subsequent research.

(b) Where the research involves extraction or collection of personally identifiable information, please describe the purpose, from whom the information will be obtained, what it will include, and how permission to access the data is being sought. (Strategies for recruitment are to be described in section #15.)

Not applicable.

(c) Is there any group or individual-level vulnerability related to the research that needs to be mitigated (for example, difficulties understanding informed consent, history of exploitation by researchers, power differential between the researcher and the potential participant)? If so, please provide further details below.

Not applicable.

(d) If your research involves the collection and/or use of biological materials (e.g. blood, saliva, urine, teeth, etc.), please provide details below. Be sure to indicate how the samples will be collected and by whom. Not applicable.

14. EXPERIENCE OF INVESTIGATORS WITH THIS TYPE OF RESEARCH (a) Please provide a brief description of previous experience by (i) the principal investigator/supervisor or sponsor, (ii) the research team and (iii) the people who will have direct contact with the participants. If there has not been previous experience with this type of research, please describe how the principal investigator/research team will be prepared.

Mark Chignell is a Professor of Mechanical and Industrial Engineering (since 1990), and a Professor in the Faculty of Information (since 2013) at the University of Toronto. He was an Assistant Professor in Industrial and Systems Engineering at the University of Southern California from 1984 to 1990. He has a Ph.D in Psychology (University of Canterbury, New Zealand, 1981), and an M.S. in Industrial and Systems Engineering (Ohio State, 1984). Mark is Director of the Knowledge Media Design Institute at the University of Toronto, BUL Chair in Human Computer Interaction, Director of the Interactive Media Lab within the Department of Mechanical and Industrial Engineering, and a visiting scientist at the IBM Centre for Advanced Studies. He is a recognized expert in Human Computer Interaction and User Interface Design and he has extensive experience in experimental design, statistical analysis, and data mining.

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 8 of 25

Andrea Jovanovic is a MASc student in the Department of Mechanical and Industrial Engineering. She has a BASc in Industrial Engineering from the University of Toronto. She has worked as a Human Factors Analyst at Healthcare Human Factors where she facilitated interviews and conducted usability evaluations with medical professionals and patients. She has been involved in several research studies in the Interactive Media Lab that involved facilitating experiment sessions. Olivier St-Cyr is an Assistant Professor, Teaching Stream in the Faculty of Information – iSchool at the University of Toronto. Prior to joining the iSchool, he was a full-time Professor in the School of Information & Communications Technology at Seneca College (Seneca@York) and a Status-Only Professor in the Department of Mechanical and Industrial Engineering at the University of Toronto. He also spent eight years teaching several courses in Computer Programming, User Interface Design, and HCI at Seneca College, the University of Toronto, York University, and the Ontario College of Art and Design (OCAD) University. He received a Honours BA in Computer Science and Psychology from York University (2000), a MASc in Systems Design Engineering from the University of Waterloo (2002), and a PhD in Industrial Engineering from the University of Toronto (2006). He is a Limited Engineering Licensee (LEL) of Professional Engineers Ontario (PEO).

15. RECRUITMENT OF PARTICIPANTS Where there is recruitment, please describe how, by whom, and from where the participants will be recruited. Where participant observation is to be used, please explain the form of insertion of the researcher into the research setting (e.g., living in a community, visiting on a bi-weekly basis, attending organized functions). If relevant, describe any translation of recruitment materials, how this will occur and whether or not those people responsible for recruitment will speak the language of the participants.

Participants will be recruited by posting in the HCI Education Facebook Group and Twitter pages (Appendix C).

Attach a copy of all posters, advertisements, flyers, letters, e-mail text, or telephone scripts to be used for recruitment as appendices.

16. COMPENSATION Please see U of T’s Compensation and Reimbursement Guidelines. (a) Will participants receive compensation for participation? Financial Yes No In-kind Yes No Other Yes No (b) If Yes, please provide details and justification for the amount or the value of the compensation offered.

Not applicable. (c) If No, please explain why compensation is not possible or appropriate.

Participants are HCI educators and practitioners who have an interest in the HCI living curriculum and who would be interested in its implementation.

(d) Where there is a withdrawal clause in the research procedure, if participants choose to withdraw, how will compensation be affected?

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 9 of 25

Participants may withdraw during the study at any time. However, participants cannot withdraw their data after the data have been aggregated and de-identified. At any point in time, any individual who consents to using their name may choose request that their name no longer be used. Interested participants may still submit their email address to receive a summary of the results of the study. This summary will be emailed within three months of completion of the study.

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 10 of 25

SECTION C –DESCRIPTION OF THE RISKS AND BENEFITS OF THE PROPOSED RESEARCH 17. POSSIBLE RISKS (a) Please indicate all potential risks to participants as individuals or as members of a community that may arise from this research: (i) Physical risks (e.g., any bodily contact or administration of any substance): Yes No (ii) Psychological/emotional risks (e.g., feeling uncomfortable, embarrassed, or upset): Yes No (iii) Social risks (e.g., loss of status, privacy and/or reputation): Yes No (iv) Legal risks (e.g., apprehension or arrest, subpoena): Yes No (b) Please briefly describe each of the risks noted above and outline the steps that will be taken to manage and/or minimize them.

Not applicable.

18. POSSIBLE BENEFITS ● Describe any potential direct benefits to participants from their involvement in the project ● Describe any potential direct benefits to the community (e.g., capacity building) ● Comment on the potential benefits to the scientific/scholarly community or society that would justify

involvement of participants in this study

There are no direct benefits to participants from their involvement in the study. Interested participants may submit their email address to receive a summary of the results of the study. This summary will be emailed within three months of completion of the study.

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 11 of 25

SECTION D – INFORMED CONSENT 19. CONSENT PROCESS (a) Describe the process that will be used to obtain informed consent and explain how it will be recorded. Please note that it is the quality of the consent, not the form that is important. The goal is to ensure that potential participants understand to what they are consenting. (b) If the research involves extraction or collection of personally identifiable information from or about a research participant, please describe how consent from the individuals or authorization from the data custodian (e.g., medical records department, district school board) will be obtained.

The researchers will contact potential participants to answer questions about the study background and procedure. Throughout the process, all questions, comments, and concerns voiced to the research team will be met with due diligence, to the researchers’ best abilities. If a potential participant agrees to participate, a session time will be scheduled. Participants will be given a consent form (Appendix A) prior to the interview or questionnaire. Participants will be given time to review the form and ask questions. If consent is provided, the session will proceed.

20. CONSENT DOCUMENTS (a) Attach an Information Letter/Consent Form For details about the required elements in the information letter and consent form, please refer to our informed consent guide (http://www.research.utoronto.ca/wp-content/uploads/documents/2014/10/GUIDE-FOR-INFORMED-CONSENT-V-Oct-2014.pdf) Additional documentation regarding consent should be provided such as:

- screening materials introductory letters, letters of administrative consent or authorization

(b) If any of the information collected in the screening process - prior to full informed consent to participate in the study - is to be retained from those who are later excluded or refuse to participate in the study, please state how potential participants will be informed of this course of action and whether they will have the right to refuse to allow this information to be kept.

Not applicable.

21. COMMUNITY AND/OR ORGANIZATIONAL CONSENT, OR CONSENT BY AN AUTHORIZED PARTY (a) If the research is taking place within a community or an organization which requires that formal consent be sought prior to the involvement of individual participants, describe how consent will be obtained and attach any relevant documentation. If consent will not be sought, please provide a justification and describe any alternative forms of consultation that may take place. Not applicable.

(b) If any or all of the participants are children and/or individuals that may lack the capacity to consent , describe the process by which capacity/competency will be assessed and/or, the proposed alternate source of consent.

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 12 of 25

Not applicable.

(c) If an authorized third party will be used to obtain consent:

i) Submit a copy of the permission/information letter to be provided to the person(s) providing the alternative consent

ii) Describe the assent process for participants and attach the assent letter.

Not applicable.

22. DEBRIEFING and DISSEMINATION (a) If deception or intentional non-disclosure will be used in the study, provide justification. Please consult the Guidelines for the Use of Deception and Debriefing in Research

Not applicable.

(b) Please provide a copy of the written debriefing form, if applicable. (c) If participants and/or communities will be given the option of withdrawing their data following the debriefing, please describe this process.

Not applicable.

(d) Please describe what information/feedback will be provided to participants and/or communities after their participation in the project is complete (e.g., report, poster presentation, pamphlet, etc.) and note how participants will be able to access this information.

Interested participants may submit their email address to receive a summary of the results of the study. This summary will be emailed within three months of completion of the study.

23. PARTICIPANT WITHDRAWAL (a) Where applicable, please describe how participants will be informed of their right to withdraw from the project and outline the procedures that will be followed to allow them to exercise this right.

Before the study begins, participants will be informed that they may withdraw from the study at any time and for any reason, without the need to provide an explanation. However, participants cannot withdraw their data after the data have been aggregated and de-identified. At any point in time, any individual who consents to using their name may choose request that their name no longer be used.

(b) Indicate what will be done with the participant’s data and any consequences which withdrawal may have on the participant.

If a participant withdraws from the study, his or her data will be destroyed and will not be used in any analyses. However, participants cannot withdraw their data after the data have been aggregated and de-identified.

(c) If participants will not have the right to withdraw from the project at all, or beyond a certain point, please

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 13 of 25

explain. Ensure this information is included in the consent process and consent form.

Not applicable.

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 14 of 25

SECTION E – CONFIDENTIALITY AND PRIVACY 24. CONFIDENTIALITY Data security measures must be consistent with UT's Data Security Standards for Personally Identifiable and Other Confidential Data in Research. All identifiable electronic data that is being kept outside of a secure server environment must be encrypted.

(a) Will the data be treated as confidential? Yes No (b) Describe the procedures to be used to protect the confidentiality of participants or informants, where applicable

Participants may choose to participate anonymously. In this case, participant names will be replaced with a unique identification number to maintain data integrity while protecting confidentiality. It will not be possible to re-identify participants from the identification numbers assigned to them. Participants may withdraw during the study at any time. However, participants cannot withdraw their data after the data have been aggregated and de-identified.

(c) Describe any limitations to protecting the confidentiality of participants whether due to the law, the methods used, or other reasons (e.g., a duty to report)

Participants who wish to be acknowledged and possibly quoted in publications related to this research may choose to do so by consenting to have their name mentioned. In this case, participant names will be associated with their data. At any point in time, any individual who consents to using their name may choose request that their name no longer be used.

25. DATA SECURITY, RETENTION AND ACCESS (a) Describe how data (including written records, video/audio recordings, artifacts and questionnaires) will be protected during the conduct of the research and dissemination of results.

Electronic files will be protected by passwords. Hard copies of collected data will be kept in a locked cabinet in an office accessible only by the researchers.

(b) Explain how long data or samples will be retained. (If applicable, referring to the standard data retention practice for your discipline) Provide details of their final disposal or storage. Provide a justification if you intend to store your data for an indefinite length of time. If the data may have archival value, discuss how participants will be informed of this possibility during the consent process.

Data will be stored for a maximum of five years, after which all hard and soft copies will be destroyed. Hard copies will be shredded and electronic copies will be deleted from hard drives.

(c) If participant anonymity or confidentiality is not appropriate to this research project, please explain.

Participants who wish to be acknowledged and possibly quoted in publications related to this research may choose to do so by consenting to have their name mentioned. At any point in time, any individual who consents to using their name may choose request that their name no

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 15 of 25

longer be used.

(d) If data will be shared with other researchers or users, please describe how and where the data will be stored and any restrictions that will be made regarding access.

Not applicable.

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 16 of 25

SECTION F – LEVEL OF RISK AND REVIEW TYPE See the Instructions for Ethics Review Submission Form for detailed information about the Risk Matrix. 26. RISK MATRIX: REVIEW TYPE BY GROUP VULNERABILITY and RESEARCH RISK (a) Indicate the Risk Level for this project by checking the intersecting box

______________________Research Risk____________________________

Group Vulnerability Low Medium High Low 1 1 2 Medium 1 2 3 High 2 3 3

(b) Explain/justify the level of research risk and group vulnerability reported above:

Only non-vulnerable subjects will be recruited. The research risk is low. This study involves no experimental treatment and is considered a minimal risk study.

(Please note that the final determination of Review Type and level of monitoring will be made by the reviewing University of Toronto REB) Based on the level of risk, these are the types of ethics review that an application may receive: Risk level = 1: Delegated Review; Risk level = 2 or 3: Full Board Review For both delegated and full reviews (SSH&E, HS, or HIV), please submit one electronic copy of your application and all appendices (e.g., recruitment, information/consent and debriefing materials, and study instruments) as a single Word document or a pdf. Do not submit your entire research proposal. Please ensure that the electronic signatures are in place and e-mail to new.ethics.protocols@utoronto.ca The deadline for delegated review (SSH&E or HS) is EVERY Monday, or first business day of the week, by 4 pm. Information about full REB meeting and submission due dates are posted on our website (SSH&E, HS or HIV). HIV REB reviews all applications at full board level but applies proportionate review based on the level of risk. All other submissions (e.g., amendments, adverse events, and continuing review submissions) should be sent to ethics.review@utoronto.ca

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 17 of 25

SECTION G – SIGNATURES 27. PRIVACY REGULATIONS My signature as Investigator, in Section G of this application form, confirms that I am aware of, understand, and will comply with all relevant laws governing the collection and use of personally identifiable information in research. I understand that for research involving extraction or collection of personally identifiable information, provincial, national and/or international laws may apply and that any apparent mishandling of personally identifiable information must be reported to the Office of Research Ethics. For U of T student researchers, my signature confirms that I am a registered student in good standing with the University of Toronto. My project has been reviewed and approved by my advisory committee or equivalent (where applicable). If my status as a student changes, I will inform the Office of Research Ethics.

Signature of Investigator: _______________________________ Date:

***For Graduate Students, the signature of the Faculty Supervisor is required. For Post-Doctoral Fellows and Visiting Professors or Researchers, the signature of the Faculty Sponsor is required. In addition to the supervisor/sponsor, the chair or the dean of the UoT sponsor’s/supervisor’s department is required to approve

and sign the form*** As the UofT Faculty Supervisor of this project, my signature confirms that I have reviewed and approve the scientific merit of the research project and this ethics application submission. I will provide the necessary supervision to the student researcher throughout the project, to ensure that all procedures performed under the research project will be conducted in accordance with relevant University, provincial, national or international policies and regulations that govern research involving human subjects. This includes ensuring that the level of risk inherent to the project is managed by the level of research experience that the student has, combined with the extent of oversight that will be provided by the Faculty Supervisor and/or On-site Supervisor. As the UofT Faculty Sponsor for this project, my signature confirms that I have reviewed and approve of the research project and will assume responsibility, as the University representative, for this research project. I will ensure that all procedures performed under the project will be conducted in accordance with all relevant University, provincial, national or international policies and regulations that govern research involving human participants.

Signature of Faculty Supervisor/Sponsor: ______________________ Date:

As the Departmental Chair/Dean, my signature confirms that I am aware of the requirements for scholarly review and that the ethics application for this research has received appropriate review prior to submission. In addition, my administrative unit will follow guidelines and procedures to ensure compliance with all relevant University, provincial, national or international policies and regulations that govern research involving human participants. My signature also reflects the willingness of the department, faculty or division to administer the research funds, if there are any, in accordance with University, regulatory agency and sponsor agency policies.

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 18 of 25

Print Name of Departmental Chair/Dean (or designate) : Signature of Departmental Chair/Dean: ___________________________ Date: (or authorized designate)

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 19 of 25

APPENDIX A – CONSENT FORMS Client Information Sheet and Informed Consent Form for the Interview Study: Study Name: Developing a living curriculum for HCI education Researchers: Dr. Mark Chignell, Andrea Jovanovic, Dr. Olivier St-Cyr Research Rationale: Contribute to the advancement of knowledge concerning the requirements and implementation of a living curriculum. Research Procedure You are invited to voluntarily take part in this study if you are 18 years of age or older, fluent in English, and a current practitioner or educator in the field of HCI. You are free to refuse to participate or withdraw from the study at any time without penalty. This interview will take between 15 minutes and one hour, and there is no foreseeable risk or harm to you. You will be asked to participate in an interview where you will answer questions and give your opinions concerning the requirements and implementation of a living curriculum. You may choose to participate by name or anonymously. If you choose to participate by name, you will receive acknowledgement for participation in publications related to this research and may possibly be quoted. If you choose to participate anonymously, all data collected will be coded to protect identity and kept in a secure office. No personal or identifying information will be included in written reports or presentations and all data will be kept in a locked office and on a password encrypted computer, accessible by only the investigators. All data will be securely stored until December 2022, after which all data will be destroyed. Interested participants may submit their email address to receive a summary of the results of the study. This summary will be emailed within three months of completion of the study. Contact Information If you have any questions about this study, please email andrea.jovanovic@mail.utoronto.ca. If you have any questions about your rights as a research participant, please contact the Office of Research Ethics (ethics.review@utoronto.ca, 416-946-3273). Please read the following and choose the appropriate box. I ___________________________, have read this consent form and understand its contents. I understand that what I say will be audio-recorded but kept confidential, except that the resulting information will be summarized and may be presented in publications of the results. I understand that I am free to withdraw before or anytime during the study without penalty.

I voluntarily agree to participate in this study by name I voluntarily agree to participate in this study anonymously I DO NOT agree to participate in this study

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 20 of 25

Client Information Sheet and Informed Consent Form for the Questionnaire Study: Study Name: Developing a living curriculum for HCI education Researchers: Dr. Mark Chignell, Andrea Jovanovic, Dr. Olivier St-Cyr Research Rationale: Contribute to the advancement of knowledge concerning the requirements and implementation of a living curriculum. Research Procedure You are invited to voluntarily take part in this study if you are 18 years of age or older, fluent in English, and a current practitioner or educator in the field of HCI. You are free to refuse to participate or withdraw from the study at any time without penalty. However, data cannot be withdrawn after the data have been aggregated and de-identified. You will be asked to answer questions and give your opinions concerning the requirements and implementation of a living curriculum. No personal or identifying information will be included in written reports or presentations and all data will be kept in a locked office and on a password encrypted computer, accessible by only the investigators. All data will be securely stored until December 2022, after which all data will be destroyed. Interested participants may submit their email address to receive a summary of the results of the study. This summary will be emailed within three months of completion of the study. Contact Information If you have any questions about this study, please email andrea.jovanovic@mail.utoronto.ca. If you have any questions about your rights as a research participant, please contact the Office of Research Ethics (ethics.review@utoronto.ca, 416-946-3273). The research study you are participating in may be reviewed for quality assurance to make sure that the required laws and guidelines are followed. If chosen, (a) representative(s) of the Human Research Ethics Program (HREP) may access study-related data and/or consent materials as part of the review. All information accessed by the HREP will be upheld to the same level of confidentiality that has been stated by the research team. Please read the following and choose the appropriate box. I ___________________________, have read this consent form and understand its contents. I understand that my responses will be kept confidential, except that the resulting information will be summarized and may be presented in publications of the results. I understand that I am free to withdraw before or anytime during the study without penalty.

I voluntarily agree to participate in this study

I DO NOT agree to participate in this study

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 22 of 25

Sample Questionnaire HCI Education Questionnaire Communities of Practice are groups “of people who share a concern, a set of problems, or a passion about a topic, and who deepen their knowledge and expertise in this area by interacting on an ongoing basis” (Wenger, McDermott, & Snyder, 2002, p.4). We envision a virtual Community of Practice for educators and practitioners based on HCI education, and are looking for your feedback about how this online community should be implemented. This research is inspired by the work done on the HCI Living Curriculum (Churchill, Bowser, & Preece, 2016). Elizabeth F. Churchill, Anne Bowser, and Jennifer Preece. (2016). The future of HCI education:

a flexible, global, living curriculum. interactions 23, 2 (February 2016), 70-73. DOI=http://dx.doi.org/10.1145/2888574

Wenger, E., McDermott, R., & Snyder, W. M. (2002). Cultivating communities of practice.

Boston: Harvard Business School Press. 1. How do you envision a virtual Community of Practice based on HCI education? Select all that apply.

A social network for discussion about HCI education-related topics

A collection of HCI teaching resources such as course syllabi, slides, assessments, etc.

A collection of research-based best practices for teaching and learning HCI

Other(s): ________________________________________________________________

________________________________________________________________ ________________________________________________________________

Comments: ________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ 2. Do you think incentives would be required to encourage participation in this community?

I do not think incentives would be required

Recognition by an accredited body or institution

Information about the use of my uploaded resources, such as number of downloads

Knowing that my contribution may help others in the community

Other(s): ________________________________________________________________

________________________________________________________________ ________________________________________________________________

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 23 of 25

Comments: __________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ 3. What method(s) of quality control do you think would be effective to differentiate between the quality of different resources? Select all that apply.

“Likes” or “Thumbs up” determined by the community

Ratings (e.g. out of 5 stars) determined by the community

Ratings (e.g. out of 5 stars) determined by a review committee of experts

Review committee of experts vets all resources and only selects the highest quality

Other(s): ________________________________________________________________

________________________________________________________________ ________________________________________________________________

Comments: ________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ 4. What are your thoughts about the use and copyright of resources? Select all that apply.

All resources should be made available under Creative Commons licenses (or other

licenses that allow others to use, alter, and distribute works at least non-commercially)

Access to resources should require membership in the community

Other(s): ________________________________________________________________

________________________________________________________________ ________________________________________________________________

Comments: ________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 24 of 25

5. What do you think are the goals of a Community of Practice based on HCI education? __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ 6. Do you see any obstacles that would prevent you from contributing to a Community of Practice based on HCI education? __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ 7. Do you have any additional comments? __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ Thank you for completing this questionnaire.

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 21 of 25

APPENDIX B – QUESTION GUIDE

Interview Question Guide After informed consent, the facilitator will provide a background description of the motivation for a living curriculum in HCI. Participants will then be asked a series of questions regarding their opinions about a living curriculum in HCI. Props such as use cases and prototypes may be used to prompt further discussion. Audio recordings and field notes will be transcribed and analysed. Initial Questions

1. How do you envision a HCI living curriculum? 2. What do you think are the requirements for a HCI living curriculum?

a. What would participation look like? b. How should a living curriculum behave?

3. Do you see any barriers to a HCI living curriculum? a. Do you have any concerns? b. Are there obstacles that would discourage you from contributing?

Additional Probing Questions

1. Do you think incentives would be required to encourage participation? a. What kinds of incentives do you think would be effective in this community?

2. Do you have any concerns about the quality of the resources in the curriculum? a. What methods of quality control do you think would be appropriate? b. What do you think about social quality measures such as likes and ratings? c. What do you think about a review committee to ensure high quality?

3. Do you have any concerns about intellectual property or copyright in resources?

UT - ROCO-HREP – Application Form for Supervised/Sponsored Research 25 of 25

APPENDIX C – RECRUITMENT NOTICE

This notice will be posted on the HCI Education Facebook Group and Twitter pages. We are looking for educators and practitioners in HCI who are willing to participate in an interview about possible use cases and requirements of a living curriculum in HCI. If you are interested in participating or would like more information, please contact Andrea Jovanovic at andrea.jovanovic@mail.utoronto.ca

Office of the Vice-President, Research and Innovation Human Research Ethics Program

UT Office of Research Ethics – Amendment Form Version Date: Oct. 2015 12 Queen’s Park Crescent West – McMurrich Building, 2nd floor, Toronto, M5S 1S8 Page 1 of 3

Office Use Only Protocol Number:

AMENDMENT REQUEST FORM

This form is to be submitted when changes to the approved U of T ethics application form are required. Substantive changes to the ethics protocol should not be implemented until ethics approval has been received, unless the changes are required to eliminate immediate risk(s) to study participants. A copy of the complete U of T ethics application form with the proposed changes indicated in bold text or tracked changes should be submitted with this form.

1. TITLE OF RESEARCH PROJECT

Developing a living curriculum for HCI education U of T Protocol reference number: 34384 2. INVESTIGATOR INFORMATION Investigator: Title (e.g., Dr., Ms., etc.): Ms.

Name: Andrea Jovanovic

Department (or organization if not affiliated with U of T): Mechanical and Industrial Engineering Mailing address: Attn: Interactive Media Lab, 5 King’s College Road, Toronto, ON, M5S 3G8 Phone: 647-637-0499 Institutional E-mail: andrea.jovanovic@mail.utoronto.ca Level of Project Faculty Research CBR/CBPR Research Post-Doctoral Research Student Research: Doctoral Masters Course-based Faculty Supervisor/Sponsor (If applicable): Title: Dr. Name: Mark Chignell Department (or organization if not affiliated with U of T): Mechanical and Industrial Engineering Mailing address: Attn: Interactive Media Lab, 5 King’s College Road, Toronto, ON, M5S 3G8 Phone: 416-978-8951 Institutional E-mail: Institutional e-mail: chignell@mie.utoronto.ca Primary Alternate Contact (e.g., research manager/coordinator): Title: Dr. Name: Olivier St-Cyr Phone: 416-978-8876 Institutional E-mail: olivier.st.cyr@utoronto.ca

UT Office of Research Ethics – Amendment Form Version Date: Oct-15 12 Queen’s Park Crescent West – McMurrich Building, 2nd floor, Toronto, M5S 1S8 Page 2 of 3

3. PROPOSED CHANGES

a) Please describe the proposed amendment or modification in the space provided below. List the sections of the ethics protocol that have been revised and submit a copy of the complete revised protocol with the changes indicated in bold text.

We would like to take the questions from the interviews and get people to answer them in the form of an in-person paper-based questionnaire. This will allow us to collect data more efficiently and get more responses. The rest of the protocol including the sample population, and the questions being asked will not change. b) Will the proposed amendment change the overall purpose or objective of the study?

Yes No If Yes, a new protocol may be requested by the REB.

c) Will the proposed amendment affect the vulnerability of the participant group or the research risk?

Yes No

If Yes, please indicate the new overall risk level on the Risk Matrix below d) What follow-up action do you recommend for study participants who are already enrolled in the study?

Inform study participants Revise consent/assent forms (please attach a copy with the changes) Other (please describe) No action required

4. RISK MATRIX: REVIEW TYPE BY GROUP VULNERABILITY AND RESEARCH RISK Please check one: (Check the box that best reflects the overall risk of your revised protocol in light of the changes you have made) Research Risk Group Vulnerability Low Medium High Low 1 1 2 Medium 1 2 3 High 2 3 3 Risk level = 1: Delegated Review Risk level = 2 or 3: Full Review Note: Final determination of review type will be made by the University of Toronto REB 5. SIGNATURES

UT Office of Research Ethics – Amendment Form Version Date: Oct-15 12 Queen’s Park Crescent West – McMurrich Building, 2nd floor, Toronto, M5S 1S8 Page 3 of 3

My signature certifies that the above information is correct and that no unapproved procedures will be used on this study.

Ø U of T Human Research Ethics Program accepts e-mailed or scanned submissions as long as it is sent from a faculty researcher's/supervisor's institutional e-mail account. Please send the completed documents via e-mail to ethics.review@utoronto.ca.

Signature of Investigator: Date: July 19, 2017 AND (if applicable) Signature of Faculty Supervisor/Sponsor: Date: (for student or sponsored research only) NOTE: We cannot process the form until all the signatures are in place. Please submit your protocol and supporting documents as a single attachment (if possible). Signatures should be included as an inserted image into the document, or the hard copy can be signed, then scanned and e-mailed to the office. If neither method is possible, the HREP will accept confirmation of Investigator or Supervisor representation, provided that an institutional email is used.