!!!

! !Evaluating!VirBELA!!

Studying!usability!and!effectiveness!of!VirBELA!for!distance!learning!!

!!!!!!

Authors:!!Sanna!Jamila!Ali!Jamie!Kumar!Brown!Clara!Caldeira!Andrea!Renika!D’Souza!Han!Ke!!Sanket!Subhash!Khanwalkar!Shibani!Konchady!Rohit!Malhotra!Anjanakshi!Prakash!Noopur!Raval!David!Wu!Dinghang!Yu!

INF!231:!Group!2!Professor!Alfred!Kobsa!

Evaluating VirBELA Studying the effectiveness of VirBELA for distance learning

Abstract Virtual reality applications have been used traditionally to overcome the barriers of distance and time in order to facilitate computer supported collaborative work (CSCW). Some important considerations while designing a virtual learning environment (VLE) include understanding instructor and student needs, building features that can assist in the learning process, ensuring smooth navigation as well as encouraging participants to learn actively. This study focuses on VirBELA, a virtual reality VLE that was originally developed at UC San Diego and is now being used in a handful of the classes offered by the Distance Learning Center at UC Irvine. Based on existing scientific literature and on our own ethnographic, participant research as well as UX research based on standard guidelines, this study seeks to determine the effectiveness of VirBELA as a software product as well as a distance learning tool. Based on the findings and insights we have received, this study also proposes design recommendations to enable a better learning experience within VirBELA.

Keywords: VLE, virtual reality, VirBELA, distance learning

Acknowledgements We would like to thank Mr. Mathew Williams, our client from the Distance Learning Center at UC Irvine and Dr. Alfred Kobsa, our professor at the Donald Bren School of ICS. We would also like to thank Dr. Sarah Eichhorn, Alex Howland, and Dr. Kara Pham for their time and their helpfulness in conducting this research. Finally we would like to thank all of our participants who made this research possible.

Table of Contents

Introduction 1

Methods

Participant Observation 1

Ethnographic Study of a VirBELA session 2

Literature review of Virtual Learning Environments 4

Focus Group 5

Interview with Course Designer and VirBELA developer 7

Heuristic Evaluation 9

Interviews with Students 10

Usability Testing 12

Survey Research 20

Recommendations for Software Design Changes 21

Concluding Remarks for UCI Distance Learning Center’s Implementation 21

References 23

Appendix

Usability Testing Screenshots 24

Student Interview Questions 26

Interview with Course Designer 28

Interview Questions with Developer 33

Survey Results 33

Heuristic Evaluation Table 37

1

1. Introduction

Distance learning is becoming an increasingly popular option for students as the price of higher education continues to climb and technology allows for greater and easier access to knowledge. One distance learning method currently being investigated by the University of California, Irvine Center for Distance Learning and Education is VirBELA, a virtual learning environment (VLE) designed at the University of California, San Diego. VirBELA allows students create and control avatars they use to interact in real-time with their fellow students and professors. In addition to VLEs, distance learning students at UC Irvine use other tools such as message boards, videotaped lectures and a “virtual learning system” called ALEKS.

While pilot testing has been performed with the VirBELA environment, this is the first time it is being used alongside other technologies for actual classes. We evaluated the VirBELA virtual learning environment from a usability perspective in terms of its design as well as its effectiveness in distance learning.

1.1 Related Work

While distance learning seems to have popped up in the last few years with the rise of Massive Open Online Courses (MOOCs), it is a concept that has been around for years, though it’s exact definition is not quite agreed upon. The basic concept involves students and instructors that are interacting at different places and/or times (Moore, Dickinson-Deane, Galyen, 2010). Virtual learning environments (VLEs) are a relatively new addition to the field of distance learning. Some VLEs are constructed as virtual worlds, often sporting high-fidelity graphics and set within a 3D environment, whereas others like Coursera utilize recorded lectures and assignment based peer evaluations. Like virtual worlds, users navigate the environment through the use of avatars, 3D representations of the user. VLEs have been shown to enable social interaction in real time, creating a sense of place, engaging learners in role-playing, teaching collaboration (DeFreitas & Neumann 2009, Johnson & Levine 2008) and how to co-construct shared knowledge (Vygotsky, 1978).

2. Research Methods and Results

2.1 Participant Observation of Classroom

VirBELA is currently used to offer a collaboration facilitating virtual environment for online instruction in multiple subjects. In order to better judge the suitability of VirBELA for various kinds of pedagogy, we sought to understand what it is like to study a course that is being offered inside VirBELA, in a real classroom instead.

We chose to partake in two classes of Math 2A, a subject that is also offered to UCI distance learners, currently with the use of VirBELA as a learning tool. We perceived this subject as an interesting case to study because some of its characteristics make it challenging to translate it into an online collaborative environment, such as a focus on individual problem-solving and the presence of formulas that are easier to write by hand.

2

Following were our observations:

Class #1 Participant Attributes Instructor Male, Late 20s Students Undergraduates, 18-20 yr olds, Mostly Asians Class #2 Participant Attributes Instructor Female, Late 30s Students Undergraduates, 18-20 yr olds, Mostly Asians General Observations Aspect Observation Classroom Lecture Hall, > 100 capacity Pedagogy The pedagogy was very conventional, no use of

educational technology and minimal student-teacher interaction.

Collaboration No collaboration, students solved problems in isolation

Conclusions

The pedagogical methods adopted by math instructors were found to be very traditional and students were only passively engaged in the classroom. This could be because of many reasons:

● Instructor predisposition towards conventional pedagogy ● Austere nature of the subject (Math) ● Lack of a framework to facilitate collaboration

As a collaborative virtual learning environment, VirBELA is designed to facilitate teamwork between a small group of people. Replicating a big class in a subject that by its very nature requires one to be proficient at individualistic problem solving goes against the philosophy behind its design. To leverage VirBELA best:

● Pedagogy must encourage group work ● Subject must lend itself well to collaborative learning

2.2 Ethnographic study of a VirBELA session Emerson, Fretz and Shaw explain that ethnographic field research involves the study of groups and people as they go about their everyday lives. It allows us first hand participation in an initially unfamiliar social world through passive and active observation. Ethnography as a reliable method is often employed in usability testing to embody the original experience of a software as felt by its primary users.

The team undertook an ethnographic study of an ongoing VirBELA session to familiarize itself with how students at UCI are actually using the software. Four researchers from the group

3

logged into an ongoing assignment solving session for class 1B of mathematics, facing multiple technical challenges while doing so. Two researchers initially faced difficulty in starting the software and reaching the classroom within the environment where students had convened. For one, the software kept crashing at the start. The other encountered a blank screen after reaching the classroom. Both the researchers tried looking for troubleshooting and online software support but could not find relevant information. As for the other two who could access and attend the class, they observed that a few students had logged in, of whom two were actually writing/drawing the equations on Twiddla (an integrated whiteboard for doodling) while the others observed. They observed that a common chat window popped up whenever someone asked a question or wanted to discuss. This could potentially annoy others who did not want to chat or see the chat window appear time and again.

Fig.1. Students from class Math 1B in an actual

class in progress

Fig 2. HCI experts trying to write a math equation in

VirBELA

4

2.3. Literature review of Virtual Learning Environments

Research questions:

Some research questions that we seek to answer through the survey of existing literature are as following:

● What impact do Virtual Learning Environments have on social interaction among students?

● How can they be used best as stand- alone methods or assistive tools to traditional classroom teaching?

In order to contextualize our study and gain a better understanding of previous research around virtual reality and education, we studied existing literature and studies conducted around virtual reality software, learning environments, 3D gaming and their implementation for learning different topics over time and distance. These covered varied disciplines such as medical science, geography, mathematics, information science and art education to find the variations and behaviours associated with learning a specific topic. From a study conducted by DeFreitas et al, we learned that VLEs (Virtual Learning Environments) enable social interaction in real time, creating a sense of place, engaging learners in role-playing and teaching collaboration (DeFreitas & Neumann 2009, Johnson & Levine 2008). A study conducted by Vygotsky in 1978 demonstrated that participants learned to co-construct knowledge through virtual participants (Vygotsky, 1978). Case studies by Michele Dickey on a 3D virtual world called Active Worlds lends support to the constructivist idea, showing how the unique environment and interaction afforded by the virtual world helped students synthesize knowledge (Dickey, 2005).

Some popular virtual training applications are used to train pilots, army personnel, truck drivers, disabled persons and teaching medical science, astronomy and architecture - basically by simulating high fidelity environments without incurring actual risks and damage to the physical and mental health of participants. These applications also helped mitigate time, distance and cost barriers for organizations.

Research on students’ participation and attitudes towards virtual learning states that due to the novelty of the technology involved, the study materials appear more exciting. It also revealed that having to perform tasks within VLEs compelled students to learn, since it requires them to participate as much as or more than the instructor themselves.

Case Studies on Virtual Learning Environments:

One study involving 218 findings on distance education and VR concluded that VLEs are less effective in independent and asynchronous learning but support collaborative discussion and synchronous/live learning among students better.

A study on fourth graders that visited the Great Barrier Reef virtually stated that while the ‘real’ experience was superior to the virtual one, a VLE let them explore the space in different angles individually. Similarly, another study on Art Cafe, a 3D virtual learning environment for art education in real time, explained that users enjoyed more autonomy in the setup because they could create their own identities as avatars according to name, gender and appearance. However,

5

using a singular chatbox for group discussions made their conversations chaotic. A paper by Minocha and Roberts highlights the ability of 3D virtual worlds to enhance collaboration and interaction through avatars and proposes a new knowledge framework for merging the best of 2D VLEs and 3D worlds (Minocha & Roberts, 2008).

In a research study devised to test the effectiveness of web-based VR environments for education, it was noted that they help with student retention. In this case effectiveness was measured in terms of inadequate use by teachers and pupils and faulty design “Technologies that promote participant communication are best suited for subject matter or course designs that emphasize discussion, brainstorming, problem-solving, collaboration, and reflection"

Alternately, when students were administered geography Decision-Making Exercises (DME) in a VLE, it was found that while synchronous activities were tedious to arrange, students engaged more with their peers than in physical classrooms. They also felt less awkward since face-to-face interaction was reduced. And, some students emerged as leaders in the discussions.

Conclusions:

● VLEs seem to promote collaboration and social interaction. 3D virtual worlds enhance this collaboration with avatars and a greater sense of place. Research into how students feel about collaboration and interaction in VirBELA as well as other VLEs like ALEKS is needed to compare and contrast.

● VLEs seem to support the constructivist paradigm of students synthesizing knowledge. Research into the types of tasks students do in VirBELA and whether they are geared towards constructivist principles is needed.

2.4. Focus Group Since focus groups are excellent for the generation of ideas, and to quickly gauge user impressions about a topic or concept, we chose to use this form of user studies in our initial stages of the project to better understand user views about learning environments, virtual reality and viability of virtual reality in learning environments. Hence, the goal we set was as follows:

Goal:

Discover perceptions of people regarding Virtual Reality and expectations about it as a learning environment.

Steps taken:

● Brainstorming session for the questions/ topics to be discussed in the Focus Group

Participants: Shibani, Anjanakshi, Jamie, Andrea

We aimed to formulate questions that, while not leading the participants towards a particular point of view, would help us to obtain different perspectives and learn about experiences people may have had in classrooms (physical and online) as well as virtual reality and its use as a learning environment.

6

Sample Questions/ Topics from the Brainstorming session Describe your activities in a typical classroom

Group Partner Assignment for Projects

Specific things you like/ dislike about classes

Experiences working in groups

Study Preferences Perceptions about ‘Virtual Environments’

Experience in a Math Class Possible Uses of Virtual Environments What would you do if there was a class that you wanted to attend /but can’t because of time/ location constraints

Virtual Environment as a Classroom

● Identifying user profiles for focus group and recruitment

Since VirBela is an application intended for students to use for their online classes, we chose to recruit students, more specifically graduate students from various disciplines as they would be convenient to contact and recruit. We did not enquire about prior experience with virtual environments and online classes since we wanted to know their opinions without giving them a chance to think about it too much.

We invited 10 graduate students who live around the same area as most of the facilitators as it would be easier for participants to reach the location for the focus group, and there would also be a buffer in case a few participants were not available on the day.

● Reminders for participants a day before the focus group for confirmation The recruited users were sent messages and also reminded in person about the time, date and location of the focus group to be conducted the next day. One person informed us that she was out of town and could not be present for the session.

● Focus Group

○ Demographics of Participants On the day of the session, seven participants arrived at the given location and time for the focus group. So we were still within the safe range of about 6-10 in general for a focus group.

Demographics of participants

Gender Female- 2 , Male- 5

Age 22-24 Disciplines Biotechnology- 1

Chemical- 1 Computer Engineering- 2 Computer Science- 1 Networked Systems- 1 Software Engineering- 1

7

○ Facilitators

Moderator: Jamie

Note- Taker: Shibani

Videographer: Andrea

○ Session The focus group took around one and a half hours to complete. We gained insight about how different people like to learn and their preferences regarding instructor behaviour, peer interactions and collaborations in a group. We also learnt what they thought about virtual environments, if they knew what it was. For those who didn’t, responses from other participants made them understand the concept and in some cases, even made them realise that they had used it before but didn’t realize it. They talked about what they do when they miss classes and also considered the possibility of using virtual environments for learning while discussing the pros and cons of doing so.

○ Incentives The participants were given chocolates and a home- cooked meal as agreed upon.

○ Inferences from the discussion and the views presented in the focus group

■ Classrooms should relate the application of concepts to the real world. ■ There should be more interaction between the instructor and students to

ensure more participation and to keep them engaged. ■ Majority thought that smaller classroom sizes may also improve

engagement levels as it would be easier to have everyone participate but one said that the class size should not make a difference if you are really interested in participating in class.

■ It would be easier to study if the course material for the class was available online.

■ Discussion and collaboration in groups is carried out in order to understand difficult concepts more easily or to work on team projects. In other cases, individual study is preferred.

■ Virtual Environments are perceived as something that involves long sessions on a computer, used for games and simulations

■ As a classroom, a Virtual Environment is not considered as effective with lethargic involvement and potential for distraction in the physical absence of authority figures and peers.

■ Virtual Environments are a good option to work on things at your own pace or in situations where you are unable to attend a class.

2.5. Interview with Course Designer and VirBELA developer

An important part of developing an understanding of an artifact is to comprehend the guiding principles behind its creation, the vision with which it was created and used. In order to better understand the purpose behind the use of VirBELA at UC Irvine, we interviewed Dr. Sarah

8

Eichhorn, Associate Dean at the UCI Distance Learning Center and a tenured lecturer in the Department of Mathematics who taught one online course that used VirBELA this quarter (Fall of 2014). We also wanted to learn about how and to what extent this tool was being used and to get an instructor’s perspective on its benefits and disadvantages.

The steps we followed are -

1. Objective Setting - We brainstormed the objectives behind choosing to interview these people. We came up with the following list of reasons -

a. To understand the vision behind VirBELA’s creation b. To understand the guiding principles behind course design for VirBELA c. To understand how these principles are reflected in VirBELA’s design d. To know how VirBELA is being used currently e. To understand the reasons behind VirBELA’s current technical limitations/flaws f. To get to know the future vision for VirBELA

2. Deciding Interview Structure - Given our initial lack of understanding of VirBELA we

decided to conduct a semi structured interview. We prepared a set of questions for both the interviews but were open to improvisation based on interviewee’s response.

3. Creating Questions - We came up with a list of questions for each interview keeping in mind the guidelines for creating a questionnaire. Some of the questions we created are -

Course Designer Interview

a. What are the purposes for which VirBELA is being used currently? b. What subjects are being taught within VirBELA? c. What is the pedagogical philosophy behind course creation in VirBELA? d. Have you tested your courses on a wider audience? (ethnographic, age

considerations) e. What kind of feedback have you received so far from students/instructors? f. How do you see VirBELA being used in the future? g. Anything you would like to improve in VirBELA?

Developer Interview

a. What was the goal behind creating VirBELA? b. Did you have any pedagogical predispositions while creating VirBELA? c. What is the design process like for VirBELA? d. How did you choose the collaboration tools (that are there in VirBELA)? e. Why did you choose the paradigm of virtual reality for VirBELA? f. What are the technical limitations in VirBELA? g. What are your future plans for VirBELA?

4. Recruiting Interviewees - With the help of our client we were able to recruit one course designer and one developer for the interview, Dr. Sarah Eichorn, Associate Dean of the UCI distance learning center and Alex Howland respectively.

9

5. Deciding the Venue/Mode - We interviewed the course designer in her office at UCI distance learning center. The developer was in San Diego so we interviewed him over video chat.

6. Deciding Roles - Interviewers: Rohit Malhotra, Sanket Khanwalkar and Jamie Brown Note takers: Clara Caldeira

Conclusion Based on the interviews with both the designer and developer we have come to understand that VirBELA’s motto is to be a collaborative learning software that leverages virtual reality to achieve immersion. The collaboration tools provided within VirBELA only reflect this philosophy. From a technical standpoint, the current release has issues with web synchronization, VoIP, managing concurrency etc. However, the VirBELA team is continuously fixing existing issues and researching upcoming web and VR technologies to improve their product successfully.

2.6. Heuristic Evaluation

Our entire team participated in conducting a heuristic evaluation through exploration of the virtual environment and by attempting to complete tasks as part of a mock class. We collectively recorded over 30 bugs and issues encountered during these sessions, ranging in severity from cosmetic to serious. These bugs and issues were then mapped to violations of Nielsen’s heuristics as well as Sutcliff and Gault’s heuristics for Virtual Environments (VE’s). Here, we will discuss more broadly the themes of heuristics we encountered in the software in relation to specific issues. To learn in more detail about each bug, heuristics violated, and change proposed to VirBELA developers, readers should look to our Appendix for the full heuristic evaluation table.

1) Many of the more severe issues encountered were software errors. While “error prevention” is an important heuristic in this context, almost as important was the heuristic “Help users recognize, diagnose, and recover from errors.” When shared screens were black or blank or refreshing in infinite loops, there was no help available to the user, through a forum or user support service, to rectify the situation. When error messages appeared, they were often vague or unhelpful (“You don’t have permission” or “The game crashed”). Finally, in attempting to resolve an issue without help, it would be beneficial to give users the option to “refresh” or close and reopen individual modules of the software or to refresh the session as a whole without having to log out or reinstall the software.

2) The user experience as a whole, and especially the onboarding experience, would be

improved by taking advantage of heuristics such as “Recognition rather than recall” and “Consistency and standards” as well as the VE heuristic “Support for learning.” Some of the suggestions we make are to follow VE conventions for the interface (such as placement of the chatbox and user list, adding functionality like jumping and improved control over the map, and using conventions such as ellipses and thought bubbles rather than lightbulbs to connote typing) and to include tutorials or in-product guidance to help users new to VE’s understand commands and functions. Without this guidance and

10

adherence to conventions, it is difficult for both users new to VE and users familiar with VE to operate the software using recognition rather than recall.

3) Finally, for a virtual environment, the Nielson heuristic “Match between system and real

world” is especially important, and it can be related to the VE heuristics “Natural engagement,” “Natural expression of action,” and “Sense of presence.” For example, in a traditional classroom, there are desks and chairs. Upon entering a virtual classroom, there is some discomfort and uncertainty about where to stand without blocking other students’ view or just generally feeling awkward. There is also incongruity in using the microphone as compared with physical-world conversation, due to garbled audio when multiple students speak at once as well as distracting echoes.

Our findings inform the user testing we did in our next phase of research. Based on difficulties of HCI experts in completing various tasks in VirBELA, we would like to better understand whether and how new and average users encounter these issues. We will also use these takeaways to frame our survey research and interview questions in order to evaluate the severity of the concerns that were highlighted from an expert perspective. 2.7. Interviews with Students

Since Math 1B Pre- calculus class was already using VirBELA, we wanted to conduct interviews to gauge the effectiveness and usability of Virbela and its features, and also get to know more about the issues faced by the students while carrying out their tasks in VirBELA. The interviews gave us an opportunity to understand and explore the way VirBELA is currently being used by distance learning students to learn pre- calculus directly from the source.

Goal:

To obtain rich, detailed data and get an holistic view of student’s reactions to using VirBELA for learning pre-calculus and learn more about their experiences, likes, dislikes etc.

Before the interviews:

We began the process by identifying the objectives for the interviews, which were as follows:

● To obtain feedback from the students on their expectations when they first learnt they were going to be using VirBELA vs their actual experience once they started using it.

● To learn about their overall impressions of the software. ● To understand the major problems they faced. ● To identify what feature(s) in VirBELA they found most useful. ● To list out the pros and cons of using the software. ● To point out applications of Virbela in subjects other than pre calculus.

We decided on the interview being semi-structured with a set of both open and closed ended questions and obtain both quantitative and qualitative data from the participants by giving them an opportunity to explain their choice while analysing their comments.

We formulated questions so that they reflect and cover all aspects of the above mentioned objectives. After coming up with an initial draft of questions, we conducted pilot test with other

11

team members to test for clarity and validity and check for bias. We checked if the all the questions were clear and if we were able to complete the interview within the allotted time frame of half an hour. At the end of this test, we reformulated our questions, making sure that they were clear, understandable and impartial so as to not mislead the participants. We also tested the devices to be used for recording the actual interview sessions to ensure that everything was in place.

We sent out emails to all the distance learning students inviting them to volunteer as participants for our interview, with the promise of pizza at the end of it as incentive. After hearing back from those who volunteered, we set a time and place for conducting the one-on-one interviews. After a turnout of only two participants for the in-person interviews, we contacted students again and conducted two more interviews via phone. We attribute the low turnout to the fact that since they were distance learning students, it could be difficult for most of them to come to campus for an in-person interview.

Activity Materials:

Our activity materials included the list of questions and demographics form to be filled out by the participants and a comfortable location for participant and interviewer.

Interviewers:

● Andrea D’Souza ● Anjanakshi Prakash ● Han Ke

During the Interviews:

Once the students arrived, we initially asked them to fill out the demographics form in which they would provide information pertaining to their personal details such as name, gender, major, and for how long they had been using VirBELA. We explained to them the purpose of the interview and obtained their verbal consent to audio record the session so that it would be available for analysis later. We encouraged them to answer honestly and to feel free to say when they cannot answer a question. We thanked them for their participation at the end of the session.

Results and Inferences:

What they use it for: The students we interviewed use VirBELA to take quizzes and solve group assignments. They meet once a week in world and collaborate. They are sometimes joined by the instructor who clears doubts etc. All of them use ALEKS to learn various topics and keep track of their progress.

Expectations: When asked what they thought it was going to be like when they heard they were going to use VirBELA, One student said S/he expected it to be like a chatroom for discussions. Two students expected that it would help them communicate better with instructors and their classmates.

Actual Experience: All of them said VirBELA met their expectations although one of the students thought there would be more students in world. One of the student had knowledge of

12

other virtual reality environments and “was very happy to find it was like a complete virtual reality with really good graphics”.

Most useful feature: One student said they found the Mic feature to be most useful and said “it is easier to talk rather than type or chat” while another said they liked the chat option the most instead. One student mentioned how useful it was to have everything was separated into different windows. Almost all the participants found the Whiteboard tool most useful to collaborate on assignments.

Major Problems faced: Two of the students pointed out that it was hard initially to figure out how to use the software, move around, find classrooms etc. one of them said “I felt like they had been thrown into the world and had to learn the way around on my own”. The same student also talked about a system crash after the teacher had put up questions which didn’t show up for some people because of which they couldn’t complete the task on time. One student said that their avatar sometimes doesn’t move where s/he points and suggested that VirBela needed better mouse navigation. All of them wished for more help options to initially explain what each tool does and make their function more obvious with relevant icons.

Pros of using the software: One student, whom we found to be a shy person, said that “the Virtual classroom makes students less afraid to raise questions”. Another student mentioned that having a web browser allowed them to look up information on the Internet quickly, making their tasks quicker. All of them agreed that VirBELA made their class more hands-on through the simultaneous use of the whiteboard and mic features.

Cons of using the software: The students could not think of any particular cons of using VirBELA.

Conclusions: All of the students said they would return to taking a class in VirBELA in the future and would also recommend it to their friends. One of them said that “no requirement of physical presence and the ability to communicate freely” is what would entice them to return. Online classes generally make it hard to talk to other students and the ability to communicate on VirBELA makes it enticing to continue using it. When asked what other subjects they think could be taught on VirBELA, one of them felt an English class would be suitable; they felt “the teacher could put up pieces of literature on the shared resources, which the students could then read and discuss”.

2.8. Usability Testing

Usability is not a single, one-dimensional property of a user interface. Usability has multiple components and is traditionally associated with these five usability attributes:

● Learnability: The system should be easy to learn so that the user can rapidly start getting some work done with the system.

● Efficiency: The system should be efficient to use, so that once the user has learned the system, a high level of productivity is possible.

13

● Memorability: The system should be easy to remember, so that the casual user is able to return to the system after some period of not having used it, without having to learn everything all over again.

● Few and Non-catastrophic Errors: The system should have a low error rate, so that users make few errors during the use of the system, and so that if they do make errors they can easily recover from them. Further, some errors, which are more catastrophic in nature, would should not occur. Those errors would destroy the user’s work.

● Satisfaction: The system should be pleasant to use, so that users are subjectively satisfied when using it. So they would like to use it.

Therefore, our usability test and analysis was conducted based on these attributes.

2.8.1. Usability Testing

One of the methods we used to evaluate the general usability of Virbela was usability testing. The purpose of these tests was to determine how effectively new users could use the system and how intuitive or easily understandable the user interface is. Our tests involved four participants, two male and two female, from varying academic backgrounds who had never used VirBELA before. We selected these participants because we wanted to determine if novice/new users would be able to make effective use of Virbela’s core features.

The user tests were structured around a set of user tasks that tested the fundamental functions and features of the distance learning software. The testing session was moderated by one team member who took notes and was accompanied by a second note taker and a technical aide.

2.8.2. Tasks

The user tests were structured around a set of user tasks that tested the fundamental functions and features of the distance learning software. Tasks were selected that address the key functions of VirBELA, functions that are commonly used and are integral to the purpose and vision of the software. These tasks include:

● Change environment settings

○ Change avatar; ○ Turn on sound;

● Ask students to navigate to different classrooms

○ Go to lecture hall; ○ From lecture hall go to room D1; ○ Try to run; ○ Find and meet another user in an unknown room;

● Use screens and communicate with other students

● Screen use

○ Try to use google spreadsheet

14

■ Make a table; ■ Enter some data (letters, words);

○ Try to use the google doc screen

■ Try to share it outside of VirBELA;

○ Screens in general

■ Try to change some of the screens; ■ Go to Youtube and watch a video; (The purpose of this is to see how

apparent the functions of the Screens are, whether they will figure out the functions inherent to each window)

○ For specific screens, ask them to access certain websites, use gmail, etc

■ check email:

● Account: virbela.uci@gmail.com ● Password: testvirbela

■ check news on CNN;

● Utilize methods of communication (keyboard and speech)

○ Find difference in two articles with another user; ○ Solve a math problem together in three ways; ○ Share text, picture, video with other player;

In addition to the tasks that users were asked to complete, a set of pre-test and follow-up questions were drafted in order to obtain more information about the users and their impressions of VirBELA. The pre-test questions were used to establish user demographics such as gender, year, experience with 3d virtual environments, etc. The follow-up questions were used to gauge the overall impressions each user had after using the system. They also explored memorable moments (good or bad) and their opinions on the effectiveness of the software.

The pre-test questions are as follows:

● What is your name? ● What is your year? ● What is your major? ● What is your cultural background?

The follow-up questions are as follows:

● What are your impressions from this experience? ● What stood out most for you during this simulation? Why? ● Would you recommend this to a friend? Why or why not? (shows their general opinion

about it and doesn’t place a tremendous amount of pressure on them to make a decision)

15

2.8.3. Usability Testing Overview

During the usability test, the participant would be instructed by the moderator to attempt to complete the specified tasks. There was a think-aloud protocol adopted during these tests which involved users vocalizing their thoughts while attempting the assigned tasks. We felt that this approach gave us a deeper insight into the thoughts and impressions users had about the system.

While the participant was working, the two note takers (moderator and dedicated note taker) would record their observations about both the participant’s behaviors and his or her proficiency with the assigned tasks. Recorded observations consisted of salient changes in mood or noticeable difficulties in completing the assigned tasks. Failures to complete the assigned task were also noted. These observations were obtained through close examination of the user’s facial expressions and body language. For example, when a user would furrow his/her brow and pause for a moment, this was taken as an indication of frustration/hinderance and was recorded as such. The think-aloud protocol helped in marking noteworthy observations since it allowed users to vocalize their frustrations. In addition to the physical and aural observations made during the tests, we also utilized screen capture software to record the mouse movements of the user for later analysis. Voice capture software was also used, though some users did not wish to be recorded. For these users, only screen capture software was used.

Other matters need attention:

1) if we are allowed to record the screen or both the screen and voice during the test, those materials would be used for the later analysis.

2) Since many functions involve interaction among users and maybe require more than one user at a time, an experimenter would act as another user to assist participant.

2.8.4. Steps taken for Usability Testing

1. Define user groups a. Based on gender primarily, determined academic year unimportant due to

similarity of activities in VirBELA (similar functions used by all participants)

2. Recruited participants: 2 males, 2 females

3. Prepared usability tasks

a. Determined these tasks by taking a look at the fundamental features required to effectively learn (or do anything) in VirBELA as well as general usability functions

i. Navigation/movement ii. Chat

iii. Settings

b. Tested the main features of system

i. Various screens and the possible actions one could take with them

16

4. Conducted tests

a. Used quiet room b. Welcomed user and briefed him/her while walking to test location c. Gathered background information (year, major, experience with virtual learning

tools/3d worlds) d. Began test

i. Used a think aloud protocol to determine what users truly thought about the experience

ii. Had one moderator who took notes, one dedicated note taker, one tech support

5. Concluded test, thanked participant and walked them out

6. Compiled notes on electronic document for each participant

7. Analyzed notes from all participants and created document with major problems

a. Problems defined as issues that 2 or more people experienced or issues that caused a critical failure (complete inability to use function)

b. Used resulting document to generate re-design recommendations (more on this in “Recommendations” section)

2.8.5. Findings in usability testing

A. Common Problems/Observations

1. Navigation is sub par

○ Shift to run is not apparent to everyone, only people who have experience with virtual environments (CS: GO, other fps) intuitively know to press shift (3). This is related with the usability attributes: Learnability and Efficiency.

○ Fast Travel function is not apparent, takes some a long time to locate (3). This is related with the usability attributes: Learnability and Efficiency.

○ Camera movements when using mouse to move are disorienting/annoying (3). This is related with the usability attribute: Satisfaction.

○ Lack of camera control was frustrating, couldn’t use mouse to direct camera(2). This is related with the usability attribute: Satisfaction.

(see youtube for illustration: https://www.youtube.com/watch?v=pyY8b_lqVgE)

○ Disappointed in lack of jump function (2). This is related with the usability attribute: Satisfaction.

○ Run animation looks strange (1). This is related with the usability attribute: Satisfaction.

17

○ Disappointed that user couldn’t see avatar’s face(1). This is related with the usability attribute: Satisfaction.

○ Most preferred keyboard movements. This is related with the usability attributes: Efficiency and Satisfaction.

2. Map is not very useful

○ First instinct is to click on the map to make avatar move to location (2). This is related with the usability attributes: Learnability, Efficiency, as well as Few Errors.

○ Not very useful outside of displaying current location (GoTo function makes map mostly obsolete) (1). (see Appendix 1.1). This is related with the usability attributes: Efficiency and Satisfaction.

3. Characters could use some more options

○ Not enough clothing options and colors (skin colors as well) (3). This is related with the usability attribute: Satisfaction.

○ Casual options would be nice: This is related with the usability attribute: Satisfaction.

4. Chat is confusing and difficult to use

○ No way to clear chat history (3). This is related with the usability attributes: Few Errors and Satisfaction.

○ Chat box does not look like a chat box (the little bar thingy is confusing) (2). This is related with the usability attributes: Learnability, Few Errors, as well as Satisfaction.

○ No way to manually minimize chat (3). This is related with the usability attribute: Satisfaction.

○ Click to chat is annoying (2). This is related with the usability attribute: Satisfaction.

○ microphone options could become very disruptive if multiple people talk at the same time (3). This is related with the usability attribute: Satisfaction.

5. Classroom screens are unintuitive and hard to use

○ Twiddla screen not helpful, mouse/trackpad is too imprecise (4). This is related with the usability attributes: Few errors and Satisfaction.

○ Trying to change screens in the lecture hall requires permission which confused users (3). This is related with the usability attributes: Few errors and Satisfaction.

18

○ Screen labels (instructor screen, shared screen, etc.) are not noticed which leads to confusion during use (3). This is related with the usability attributes: Learnability, Few errors, as well as Satisfaction.

○ Zooming into screens is difficult and unintuitive (3) (see Appendix 1.2). This is related with the usability attributes: Leanability, Few errors, as well as Satisfaction.

○ Sharing documents was difficult since share button did not work (3). This is related with the usability attribute: Few errors.

○ Unable to load videos due to lack of flash player (3) (see Appendix 1.3). This is related with the usability attributes: Few errors, Satisfaction.

○ Url bar is placed in an unintuitive location, user’s first instinct is to manipulate bar in main screen (3). This is related with the usability attributes: Learnability, Few errors, as well as Satisfaction.

○ Pointing dot of other users is annoying (2). This is related with the usability attribute: Satisfaction.

○ Plug in tools are out of date and error messages appear a lot (2) (see Appendix 1.4). This is related with the usability attributes: Few errors and Satisfaction.

6. General observations

○ Outside environment looks nice but disappointing how little exploration there is. This is related with the usability attribute: Satisfaction.

○ Novel idea but interest wanes due to inefficient features and clunky interactions. This is related with the usability attribute: Learnability and Satisfaction.

○ Most users would rather use other tools due to imprecise nature of the 3d world and its avatar interactions. This is related with the usability attribute: Satisfaction.

○ Tutorial would be helpful in acclimating users to the 3d world (which panels to use, what hotkeys are available). This is related with the usability attribute: Satisfaction.

○ Lack of social features (friend lists, clicking on other avatars to interact with them) was disappointing. This is related with the usability attribute: Satisfaction.

Main takeaways

1. Many things are unintuitive and hidden (number of hotkeys programmed into system but never revealed to users).

2. Clunky controls make the experience frustrating.

3. Using features in the system is more difficult than using them outside the system and some don’t work at all (video, google doc sharing).

19

B. Suggestions based on usability testing:

1. Some screens students can interact with and some not. These two kinds of screens should be separated physically so students can perceive them as different. And the label of “instructor screen” should be more obvious.

2. The pointing dot is useful when students discuss on a same screen. But many users report it is annoying sometimes. We think in the environment setting, there should be a switch to turn on or turn off the pointing dot function.

3. The map is only useful for new users, and it can be replaced by a fast travel function. We suggest combining these two functions, by making the map into a button. Upon clicking on the button, the map appears, and users can click on map for fast travel.

2.8.6. Usability Testing Results

This round of usability testing highlighted a number of usability issues that stemmed from unintuitive functions and clunky controls. One of the biggest sources of frustration during these tests stemmed from the use of the screens in the smaller classrooms. These screens are very similar with a small, difficult to notice label being the only differentiating feature. In all our tests, none of the users noticed this and attempted to perform their assigned task on the wrong screen. This example highlights just one of the functionality issues within Virbela.

Asides from issues with key features of Virbela, there are a number of smaller issues that detract from the overall experience. Awkward navigation due to a fixed camera was a very common complaint, as was the obtrusive chat box. Though these smaller issues may not cripple the system, they worked against it to build an increasingly negative impression in the minds of the new users that participated in the tests.

2.9 Competitive Analysis

After the user testing was completed, we conducted a competitive analysis in which we compared the relevant features of 5 other products similar to Virbela. These products were other successful commercial 3d environments that revolved around the premise of controlling a virtual avatar from a third person perspective. The products examined were World of Warcraft, Guild Wars 2, Second Life, Star Wars the Old Republic and Age of Conan: Hyborian Adventures.

Across all the user interfaces of these products, there were a number of frequently appearing interface elements: map screen, chatbox, character icon, social icon, settings icon, and a help icon. Though some of these elements are present in the current version of Virbela, others aren’t. Additionally, the elements present in Virbela are much more simplified than those in the other products which could be the cause of some of the usability frustrations experienced by users during testing.

Based off of this competitive analysis, we recommend that the default user interface in Virbela include a social, character, and help icon in order to increase the visibility of currently hidden functions as well as to add additional ones that are an industry standard.

20

2.10 Survey Research

2.10.1 Survey Design

In order to quantitatively analyze the user experience of students using VirBELA this quarter and to generalize insights from observations of individual users or experts, we distributed a cross-sectional survey via Google Form to the 200 pre-calculus students who used VirBELA for collaborative assignments.

The survey was mostly closed-ended with rating questions (unipolar constructs with a midpoint), multiple-choice, and one ranking question as well as 3 completely open-ended questions. The content of the survey was related to ease of use, enjoyment, comfort, learning, comparison to other tools, rating individual features and activities, general likes and dislikes, and technical issues. The survey in its entirety as well as aggregated responses are included in the appendix.

2.10.2 Survey Results

We received 25 responses, giving us a response rate of 12.5%. 72% of respondents had never taken an online course before, and 16% had taken just one before (see Appendix 4.1). 92% of respondents were lower division undergraduate students (see Appendix 4.2), which is consistent with our expectations for a pre-calculus course. We are concerned about a response bias, as the survey is likely to attract students who had the best and worst experiences with the software.

The software itself and most of its functionality (with the exception of the Twiddla) were commonly rated as easy to use. On the other hand, activities such as collaboration, discussion, and problem solving were still rated positively though not as strongly (see Appendix 4.7). Students were generally neutral to positive on the software as a whole, but overall, they did not feel that using VirBELA helped them learn the material better or that they got to know their classmates better (see Appendix 4.3). 84% of respondents said they would not like to use VirBELA for a different purpose (such as a lecture) or even at all, and they did not generally see value in using it for a different subject (see Appendix 4.5, Appendix 4.3). 40% of students experienced technical difficulties, and consistent with the results of our heuristic evaluation, when they did experience technical difficulties, they found it difficult to understand the issue, resolve it on their own, or find help (see Appendix 4.6)

Some comments confirmed our observations from our ethnographic study of the VirBELA session: participants complained that often times, one or two students would do all the work while the other students in the classroom just “slack off.” Also, 68% of respondents mostly used the chatbox and possibly one other feature, whereas 24% said they used most of the features fairly equally (see Appendix 4.8). As noted in our ethnographic study, if the chatbox is the most utilized feature, we have concerns about the added value of the virtual environment to online collaboration.

As such, we asked respondents to rank their desire to use VirBELA as compared with an online discussion forum, video chat with screensharing (such as Skype or Google Hangout), and a live chatroom with screensharing. Video chat with screen sharing was the most highly ranked option, followed by the live chatroom, and then VirBELA. Nevertheless, several commenters did mention that using VirBELA was “interesting,” “different,” and “fun,” and students liked the idea and the concept of the virtual environment.

21

3. Recommendations for Software Design Changes

Throughout various stages of our usability study of VirBELA, we encountered many issues with the software. With the heuristic evaluation and usability testing in particular, we were able to recommend specific changes to improve the user interface and experience. The primary issues dealt with clumsy implementation of the core features of VirBELA and technical failures that were poorly explained, if at all. Features like the fast travel system and screen privileges were hard to find/understand, which caused users to become frustrated and confused. Adding to this were the technical failures like screen malfunctions, where student screens appeared blank or unresponsive and were accompanied by jargon ladened error messages. With 40% of users experiencing technical difficulties with no recourse, the lack of error prevention and help was a serious barrier to successful use of the software.

Based on these issues, we recommend that the software undergo changes in feature implementation and technical programming.

Regarding the features, some of the most common errors would be prevented by making information about the virtual world more visible. Student and instructor screens should be made more distinct and should use something more salient than a small label to differentiate the two. The fast travel function should be moved to the map screen so that related tasks are grouped together. In addition to the restructuring of interface elements, some kind of help function or tutorial should be introduced. This would ideally improve error prevention and provide support, two factors that could have reduced the number of usability issues encountered.

With regards to the technical changes, we suggest that the primary features of VirBELA (such as screens and microphones) be optimized. A number of complaints were raised against the slow loading times of the student screens and their inability to play videos. Users also expressed concern when using the microphone, stating that multiple individuals speaking through this feature at the same time would produce too much noise. We had also discovered other issues through our heuristic evaluations, such as the software’s excessive power consumption and texture rendering bugs. Addressing these problems would help improve the overall experience by providing a seamless environment unhindered by technical malfunctions, in which students could focus on their given tasks without distraction.

4. Concluding Remarks for UCI Distance Learning Center’s Implementation

Having made design recommendations to the VirBELA software itself, we would also like to tie together our research insights to evaluate the overall value of VirBELA and its current implementation to UCI’s Distance Learning Center.

Keeping in mind our takeaways from the literature review and interview with VirBELA developer Alex Howland, we understand the primary goal of this virtual learning environment to be facilitation of collaborative learning amongst a small group of people. As such, use of VirBELA would couple best with pedagogy that encourages group work, low student teacher ratio and/or requires some form of immersion to teach concepts that would otherwise be lost on students (for example, simulating a courtroom scenario for lawyers). VirBELA is like an oddly shaped knife that can only cut through certain parts of the whole 'online learning' problem. We

22

would therefore not expect this precalculus course to be an especially fitting course for this particular software.

As far as actual student use, the idea that VirBELA was not well suited for this particular learning scenario was reinforced. Based on our survey research and our ethnographic study of a VirBELA session, it seems common that only one or two students would actually work out the problem and collaborate in the environment. Meanwhile, other students in the classroom would merely observe, claim participation in the assignment, and record the answers. This situation may be common in a physical-world scenario as well, but regardless, it does not seem to accomplish the goal of the software: truly enabling collaboration in learning. Furthermore, for the activities assigned, a majority of the students used only the chatbox and possibly one other feature (such as the shared resource screen). If the chatbox is the most utilized feature of the software, it is difficult to justify the “added value” of the virtual environment for online collaboration. We also note from surveys that students would prefer to use video chat (or text chat) with screen sharing over VirBELA.

It is difficult to make claims, at this time, about the use of VirBELA in other learning scenarios based on our usability study. In surveys and interviews, users were mostly unable to imagine an effective repurposing of VirBELA for a subject besides math or for an activity besides group problem-solving. For future studies, we would recommend examining what tasks can be performed in VirBELA that are geared towards constructivist paradigms of students synthesizing knowledge (as the literature states VLE’s seem to support these better) rather than individualistic problem solving common in precalculus courses. We would like to note that that current courses seem to introduce interactivity by replicating real world activities virtually; it may be worthwhile to instead imagine new ways of interacting in a learning environment that fit the strengths of the virtual world.

23

5. References

Barreau,Deborah. Distance Learning: Beyond Content. Journal of Education for Library and Information Science, Vol. 41, No. 2 (Spring, 2000), pp. 79-93

Bray, Nathaniel J., Michael S. Harris, and Claire Major. "New Verse or the Same Old Chorus?: Looking Holistically at Distance Education Research." Research in Higher Education 48.7 (2007): 889-908. Print.

Courage, Catherine, and Kathy Baxter. Understanding Your Users: A Practical Guide to User Requirements Methods, Tools, and Techniques. San Francisco, CA: Morgan Kaufmann, 2005. Print.

De Freitas, S. & Neumann, T., (2009). The use of ‘exploratory learning’ for supporting immersive learning in virtual environments, Computers & Education, 52:343-352

Dickey, Michele D. Three-dimensional virtual worlds and distance learning: two case studies of Active Worlds as a medium for distance education. British Journal of Educational Technology. Vol. 36, No. 3, 2005, pp 439-451.

Farkas, D.K. Guidelines for Designing Web Navigation. Technical Communication 4(2000), 341-358.; Farkas, J.B (2000)

Harrington, Maria C.R. An Ethnographic Comparison of Real and Virtual Reality Field Trips to Trillium Trail: The Salamander Find as a Salient Event.. Children, Youth and Environments. Vol. 19, No. 1, 2009, pp. 74-101

Li-Fen, Lilly Lu. Art Café: A 3D Virtual Learning Environment for Art Education. National Art Education Association, Vol 61, No. 7, Nov., 2008, pp. 48-53

Lou, Yiping, Robert M. Bernard, and Philip C. Abrami. "Media and Pedagogy in Undergraduate Distance Education: A Theory-Based Meta-Analysis of Empirical Literature." Educational Technology Research and Development 54.2 (2006): 141-76. Print.

Minocha, Shailey, and Dave Roberts. "Laying the Groundwork for Socialisation and Knowledge Construction within 3D Virtual Worlds." Alt-J 16.3 (2008): 181-96. Print.

Moore, Joi L., Camille Dickson-Deane, and Krista Galyen. "E-Learning, Online Learning, and Distance Learning Environments: Are They the Same?" The Internet and Higher Education 14.2 (2011): 129-35. Print.

Nielsen, J. "What is Usability. Usability Engineering", Morgan Kaufmann, 1993

Rubin, Jeffrey. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. New York: Wiley, 1994. Print.

Stott, D. "Learning the Second Way." Bmj 335.7630 (2007): 1122-123. Print.

Summers, Carolyn. Immersive Interactive Learning Labs for STEM Education., Schloss Annette, Kerry Handron, JeffreyJacobson. 2012

24

6. Appendix

1) Screen Shots

Fig 1.1 Useless Map Function

Fig 1.2. Zoom in Button and URL input are unintuitive

25

Fig 1.3. Flash player is out of date

Fig 1.4. Error messages appear frequently

26

Fig 1.5. The application tends to crash at times. Needs to be reinstalled to be used again.

2) Questions for Student Interviews

INITIAL THOUGHTS:

● what were you expecting when you found out you were going to be using a (virtual environment(VE)) for distance learning? Initial thoughts/expectations?

● How different was the actual experience from your expectations? ● what are your overall impressions of virBela? ease of use/ease in learning/ease to recall remember

what you’ve learnt.

PREVIOUS EXPERIENCE:

● is this the first distance learning class you've taken? ● if yes, what are your thoughts so far? ● If no, how different was the previous one?

○ which software did you use in your previous class? ○ which method do you prefer and why?

● Do you use Aleks along with VirBela? ● Can you explain what you use Aleks and VirBela for? //common utilities ● Mention some advantages of Aleks over VirBela and vice versa.

27

IN WORLD USAGE

● what can you achieve through virBela that you think is difficult in an actual classroom? ● what aspects of an actual classroom is missing in virBela? ● List some pros and cons of VirBela. ● talk about major problems you faced with VirBela that were a hindrance to your activity? ● did you manage to resolve it? how?

○ if yes, did you like using the software after it was resolved? ● what feature do you like best and why? ● what features would you like to remove/add/modify and why? ● where are you usually physically placed when you're within the world?

VIRBELA TOOLS

● How do you move around in the world? ● How was your experience using the interface? Likes/ Dislikes ● mention any problems you have with the interface - creating avatar, moving around, finding

places ● what kind of tasks do you do for your class? ● how do you go about doing that in the world? ● How was your experience performing the activities? Likes/Dislikes ● what difficulties do you face with the activities (using the inworld tools like twiddla, shared

resources etc.)? ● how do you think they can be resolved? ● what is the most useful in world tool and why? ● what is the most difficult in world tool and why?

COMMUNICATION

● how many students are there in world during a typical class? ● how do you communicate with classmates ● how do you communicate with instructor? ● rate the quality of communication on a scale of 1-10, 1 being the worst and 10 being the best and

why? ● mention any communication problems you have faced

CONCLUSIONS

● If you could make one significant change to this software, what would it be? ● Would you return to taking a class on virBela in the future? Why/why not? ● What aspect of this experience would make you to return? ● Would you recommend your friends to consider it? why/why not? ● Any other thoughts/ suggestions/ opinions?

28

3) Interview Transcript with Dr. Sarah Eichhorn R: What is the process you follow for designing the courses for VirBELA? S: I don’t think we have a good process in place because it’s very new and we kind of piloted it. For my class, I already knew what I wanted to use the tool for. I was building it into an existing class, replacing another technology with VirBELA within that course. We had instructors who were describing what they wanted to do in other classes class so we’d recommend VirBELA as a tool for them. And then they would start to think about how they might use that in their class. One instructor did the opposite, which was start with the tool and then find the usage. R: So you design the classes for both meetings in classrooms and design classes that happen totally virtually. S: Yes. R: In what capacity did you include the users - the instructors, students, etc - in the course design process? S: It’s different depending on the different models. When I originally designed this math class that I teach, we thought about what the instructors would do, what the students would do, what would the learning be like for the students. And when we incorporated VirBELA, instead of using the instructor doing a lot of the talking, the students could work on a group first and then come back and interact with the instructor, so they would have tried the problem first themselves. As opposed to just listening to the instructor describing it. R: What kind of interaction do you emphasize during the course designing process? Do you design courses keeping in mind the instruction or keeping in mind collaboration between students? S: We use VirBELA both ways. One instructor predominantly uses VirBELA for office hours in the large lecture hall. It’s sort of interactive because the students can ask questions, but it’s mostly like a discussion section in a typical face to face class. Whereas other instructors are using just the breakout rooms. The students go there, they interact with their group, they do their project, they submit their answers and they leave. Sk: In this case, are the learning objectives different from the traditional style? S: The learning objectives for the courses may be the same but the activity that they are doing - one is just a Q&A, it’s more like office hours. Whereas the other one is more replacing homework. Instead of working alone, they’re working in groups together on an assignment. The role of the instructor is very different, in one the instructor is providing information, on the other the students are processing the information. So that’s a big difference in learning. R: Are your courses designed keeping in mind a fixed number of students? S: Each class will have a particular class size. Because VirBELA is kept at a hundred, when we design activities in there it’s always for less than 100 students. In the breakout rooms we usually put them in groups of 3 or 4 because we want them to be able to talk and interact. The nature of the activity dictates the group size. R: What courses does VirBELA cover right now, apart from math? S: There’s a math class, an economics class, a human resources class and a chemistry course using it. R: Does VirBELA affect the way you design your courses? Are there certain features of VirBELA that help you design your courses or are there certain challenges in VirBELA that impede your process of designing courses?

29

S: We haven’t designed a lot of new courses using VirBELA, but going forward one of the things we’re hoping to be able to do is design classes where you can have a lot of small group interactions and then debrief as a class. That’s one thing that you can do in VirBELA that we don’t really have any other substitute for. R: The students can break themselves into groups and then do their projects in their own classrooms with minimum interference from other groups. S: Right. The problem with most web conferencing systems is that it’s usually one line for everybody. In a web conference you can’t break people into groups so that they can only hear each other. VirBELA does this well and lets us design classes with that principle. That will fundamentally change how we develop classes. Sk: How do you decide on the scalability of a certain task or assignment or a project? How big the project should be or how small the task should be so that the success or the overall return on the invested time is maximum? S: Good question. I think it varies a lot for a subject and it’s no different from a face to face class than it is for an online class. Depends on how much overhead there is as far as giving instructions to students. VirBELA is relatively low overhead because if an instructor just wants to use it for one project once in a class, you don’t want a huge learning curve. VirBELA is relatively easy to get in, I timed myself and it took me 6 minutes to download and install, make an account and start running around. The biggest barrier is just that learning curve, and then it could be used multiple times throughout a class. Somehow no assignment is too small if it’s being utilized frequently and if it’s being utilized for a reason. R: Is VirBELA used more for large classes, for projects or for doing homework and office hours? S: It’s relatively new so i don’t think we have a good weighting on it yet. My vision for it is that it’d me mostly be used for projects and group work. I think that’s the real value of it, you don’t really need the innovative tool to just lecture at the class. It’s also the one thing that is missing in online classes, a lot of online classes feel non-human because you’re spending so much time sitting at a computer watching lecture videos. In VirBELA you feel like there’s other students there, you can feel like you get to know somebody and you can recognize them from session to session. Whereas in most webinars there’s just a list of people’s names who are there. R: Have the students reported that they’re finding a hard time learning or there’s some problem with the course design or VirBELA itself? S: VirBELA occasionally freezes up a little bit. The google doc integration hasn’t been perfect. For example, if the students copy and paste something on the google doc, it’ll freeze up. That was our number one complaint, that the google doc was a little temperamental. R: Have they reported any issues with the learning aspect of VirBELA, are they able to learn or collaborate properly? S: We haven’t had any issues. We only ran 3 classes over the summer that finished and now we have 4 in progress. We haven’t gotten much feedback from what’s in progress. We did a survey of the ones from the summer and they seemed pretty happy with it. Sk: Is the success of the course decided based on the survey that you are conducting with the students? S: The summer was just a pilot, so we did the survey. In the future we’ll be looking at more indicators. With the math class we’re doing this kind of comparative study. Some students are using it, some students are not and we are checking their relative performances to see if having that extra tool helps the performance at all.

30

R: Will you also keep in mind that a certain percentage of the students belong to this age group, they come from this ethnic group, and so on? S: Purposefully on the pilot we used classes that were very homogeneous. Because we were a little worried about the technology, it does require higher end graphics capabilities and you have to have a pretty good connection. One of the classes we piloted with was a MBA class. The MBA program here gives all their students a laptop when they get here, so all of the students had the exact same computer. And we knew they were all on campus so they’d have stable internet connections. That class was not very diverse, they were all incoming MBA students who are very similar in background. What we’re doing now is try it with classes that are a little bit more diverse so we can get a sense of the technology limitations. People with different backgrounds, if they had different access to technology, that affects their ability to use VirBELA. And also just user experience, are younger people more apt to like the avatar, or are older people going to be put off by this kind of virtual world navigation. Are people who play a lot of video-games, are males more likely to like it than females. Although I know that that statistic is not actually true. Everybody assumes men play more than women but that’s actually not quite right. We’re definitely interested in that and the math course that’s running is a remedial math class, which tends to be more diverse than most courses on campus. Because unfortunately the underrepresented minority students tend to be overly represented in those remedial courses. R: And most of the students in the classes are undergrads or are there grads as well? S: They are all undergrads in that class. R: And in general, in VirBELA? S: The econ classes are graduate students. Those are pre-MBA classes. R: Are there any special considerations that you pay when you design classes for grads versus undergrad students? S: The graduate class we did was an incoming graduate students and we felt like they weren’t that different from undergraduates because they’re only a couple of months removed from being undergraduates themselves. We are looking to use VirBELA for our continuing education classes and extension. And those are adult learners, people who were out of school and are going back to earn their degree. We’re very interested to see if there is a big difference in how they use VirBELA. These are people who are going to school for very serious reasons, they want to get a certificate which will allow them to get ahead in their job. We’re wondering if that focus makes the fun aspects of VirBELA less enticing, if they’ll feel like this isn’t serious because it looks like a video game. And also if the navigation will be confusing to the older populations. Although I will say, we did a session where we showed VirBELA to the staff, and we have staff in a wide range of ages, we have a couple who are seventy. And they loved VirBELA, they were running around instantly, they were jumping, they were doing the little dancing… it was funny because a couple of them were skeptics at first, and then they they got in there and they were picking out their outfits and running around… Sk: How are the instructors taking VirBELA? Are they enjoying the process of redefining the way they’re conducting the classes, designing the courses? S: We’ve had a physics instructor use it this summer and he basically did a technology replacement. He was using an interactive whiteboard called scriblar, he replaced it with VirBELA. He said just how it was way better in VirBELA because he felt like he got to know the students, he said it was a little more natural to talk to them. It was easier to talk back and forth in VirBELA. Scriblar was much more set up to be the one way push of the audio. R: Do the instructors report any difficulty in using VirBELA in the courses?

31

S: We had one instructor who had never used google docs before, so that was a learning curve for them. The others, it was pretty quick on the uptake. Sk: So there is no separate need for tutorials for the instructors? S: We have a pdf document that we give to the students, it’s 2 pages, and we have a 6 page that we give to the instructors. It’s extra things that they need to do, like setting up the rooms. We had one instructor who forgot to use a headset, so he was getting reverb like crazy. J: Could you tell us a little bit more about the chemistry class and what are the specific features that made it a great fit for VirBELA? S: We didn’t really design it, that’s a face to face class. And she is having extra office hours in there, before tests she’s using it to do a quick review. She teaches a large number of students but she hasn’t had a lot of response yet, she’s got a handful of students which was a little disappointing. The reason it was a great fit for chemistry was the whiteboard. It’s hard to find tools that do that whiteboard drawing. And she wanted the students to be able to draw chemical diagrams together in groups. R: Is there any set of established rules that you adhere to when you create the courses? S: I’d say, using the technology for what it’s good for, making the instructions as streamlined as possible for the students so that there’s clarity on where they’re supposed to go, when they’re supposed to be there.To get started quickly and not spend a lot of time on the technology instead of spending the time on the learning. R: Does the virtual reality system change the way you design courses? S: We’re not really using the VR features, we’re really using the interactive tools and audio. We haven’t done much with the fact that there are physical avatars. I know the VirBELA team has ways in which you can really exploit the virtual reality. For example, they have a classroom that looks like a courtroom, for legal students. So they can practice what the courtroom would look like. In that sense, they are really taking advantage of the virtual space, making the space look right. We just have this generic classroom set up in ours. The VirBELA team has the ability to make a custom space for whatever we want but we haven’t really had a need for that. Sk: For example, if an instructor wants to instruct a course in biology in VirBELA, how do go you about it? Do you go through a lot of iterations designing the course, testing it on a sample space and then doing it again and eventually making it live? Or do you just have a set of patterns, if it adheres to those patterns you’re through? S: Pretty much the latter. We’d figure out what is it he wants to use it for, we’d have a talk about that and then we’d set it up and they’d pretty much just go do it. A lot of the times we’ll have instructor make sure they can do all the tasks they need to do, so they’re not trying to do it for the first time while the students are there. One thing is thinking about the logistics of how you’re going to communicate to the students what room to go to. If they’re breaking out, how are you going to call them back. When the students log in, they need some sort of information, you can’t just tell them to log in to VirBELA. So you want to have a communication system outside of it. That’s probably the biggest prep work that you kind of plan for. R: Say an instructor is holding a class within VirBELA, and the class is only being offered inside VirBELA. How does the instructor go about giving assignments and taking back the assignments that have been done? S: We have a learning management system, it’s called Moodle. It’s a lot like EEE. So we use Moodle to do the turning things in and stuff like that. It depends on the nature of the assignment, so for example, students could go in, do their assignment on the board in VirBELA and the instructor could grade it and

32

enter their grades into the Moodle system. He could also have them copy the thing that’s in the google doc that they worked on and upload it into Moodle. Sk: Are there any special advantages to the instructors on doing something on VirBELA? Like being invisible while something is going on, for example? S: One big advantage is that the instructor could be in their pajamas, at home, they could be on vacation, anywhere they wanna be in the world. One big advantage is that you can use a tool to record your screen and make it available to students. If you’re teaching a face to face class it’s a little bit harder to record. Students, if they’re sick, they could still log in. Just the breakout rooms, having the students being able to work on a project together and have all those tools at their disposal is nice. Sk: Is it also possible that the students might feel like the professor is invisibly out here, so that may alter their discipline or their productivity? S: Group work is a big challenge in classes, even face to face classes, telling your students to work in a group outside of class is challenging. But if you provide them an environment and schedule and time where they can get together and you know they’re gonna have all the tools they need for everything… that I think is an advantage. Being able to see what the students have done and kind of spy on them is also nice, I think. I sat in some of the math classes and I learned a lot about how they’re talking and what they’re thinking about and what they’re doing. Sk: Do you have access to what these guys are messaging amongst each other? S: There’s an invisible mode, you can listen to them and see what they’re doing. They can use that private chat and the instructor can’t see that. I think the students mostly haven’t figured that out or use it that much. Sk: Is there any special emphasis that you guys are giving to a synchronized tasks against asynchronized tasks? S: If you’re really doing it just to lecture at students, that’s probably not a good use of synchronous time. When we’re using time synchronously in VirBELA we’re trying to make it as interactive as possible. I think scheduling convenience is one thing that is kind of nice with VirBELA, students don’t have to drive into campus and be there at a particular time. Sk: I’m sure you’re analysing the statistics in terms of when a student has entered, how much time has he spent, how much time did it take him from the point that he entered the system and then finished the assignment. S: We actually don’t have any of that data because of the way the system is. We don’t have any tracking on the students because it’s open access. I say it when I teach instructors how to use it, “think of it like a classroom”. There’s no operating mechanism to track who’s in there with it so we haven’t been looking at that. R: Is there any way that you think the collaboration feature can be improved in VirBELA? S: We have little things we want to do, for example, we want a second whiteboard in there, and that’s been a programming challenge. Because of the way twiddler is set up, we can’t put a second whiteboard in the room. They’re working in fixing that. As far as them talking to each other, it seems pretty good. I don’t have any particular feature that needs to be added there. I like the toggle with the mic, that I think is a nice feature. It does lead to a little bit stilted conversations but it solves that problem of the open mics with multiple users. Even if they don’t have headsets it really cuts down the reverb a lot, which is nice. It’s really hard to speak when you’re hearing yourself in echo. R: I think you’re done with the questions, Thank you for your time. S: No problem, happy to help!

33

4) Interview Questions with Alex Howland

1. What were the goals behind creating Virbela? 2. Any specific way in mind you had for teaching in Virbela? For ex, system for creating

collaborative learning? 3. Regarding tech research, any influences for features that VB going to have thats going to

make it stand out? 4. Has there been research on what tools you've incorporated will be the most appropriate? 5. Is there a specific reason to not allow mobile? 6. Regarding UI design - what was the process used to design and how did you test? 7. Questions on scalability, server downtime, synchronization etc. 8. What were tech limitations while building VB? 9. Are there "best ways" to achieve tasks in VirBELA 10. Why virtual reality instead of something like Coursera? 11. Any features you guys are looking forward to incorporating into VB? 12. What other types of things are you incorporating into VB? 13. How do you address hardware dependent problems like the difficulty writing something

on Twiddler?

5) Survey Results Table 4.1 How many online courses have you taken before this one?

0 courses 72%

1 course 16%

2 courses 4%

3 courses 4%

4 courses 0%

5 or more 4%

Table 4.2 What is your academic standing?

Undergraduate, lower division 92%

Undergraduate, upper division 4%

High school student or lower 4%

34

Table 4.3 Please read the following statements and indicate how much you agree or disagree with them. “Strongly disagree” was given the value -2, “Disagree” -1, “Neither agree nor disagree” 0, “Agree” 1, and “Strongly agree” 2. Mean Mode

VirBELA made the course more fun. -0.32 0

VirBELA was easy to use. 0.64 1

I felt more engaged by participating in a VirBELA session than I would have in a "regular" live online session that did not make use of virtual reality.

-0.04 2

I felt comfortable interacting with my classmates in VirBELA.

0.4 2

I felt comfortable interacting with my instructor in VirBELA.

0.2 0

Using VirBELA helped me understand the material I was learning better.

-0.4 -2

When I first started using VirBELA, it was easy for me to learn to use.

0.56 1

I would prefer to use VirBELA for a different subject than math.

-0.28 0

I got to know one or more of my classmates well using VirBELA.

-0.52 0

I would recommend VirBELA to a friend. -0.36 1

Table 4.4 Please rank the tools you would like to use to interact with students and instructors in an online course in order of preference. Mean Mode

VirBELA 2.64 2

Live Chatroom with Screensharing 2.48 2

Videochat with Screensharing (e.g., Skype, Google Hangout)

2.12 1

Online Discussion Board/Forum 2.76 4

35

Table 4.5

Would you prefer to use VirBELA for a different purpose in the future (for example for lectures instead of discussions)?

84.0% No

Table 4.6 40% of respondents said that they experienced technical difficulties while using VirBELA and were asked to respond to more questions about that experience. Please read the following statements and indicate how much you agree or disagree with them. “Strongly disagree” was given the value -2, “Disagree” -1, “Neither agree nor disagree” 0, “Agree” 1, and “Strongly agree” 2. Mean Mode

When I encountered a technical issue, I understood what it was.

-0.7 -2

When I encountered a technical issue, it was easy for me to resolve it on my own.

-0.6 0

When I encountered a technical issue, it was easy for me to find help.

-0.9 -1

Table 4.7 Please rate the following activities in terms of how easy it was to do in VirBELA. “Very difficult” was given the value -2, “Somewhat difficult” -1, “Neutral” 0, “Fairly easy” 1, and “Very easy” 2. Mean Mode

Download and install the software 1.16 1

Create your avatar 1.4 2

Move through the world 1.2 2

Find your classroom 1.4 2

Communicate with students 1 2

Use the Twiddla (drawing/sketching app) 0.3 0

Use the shared resource document 0.57 1

Do collaborative problem solving and teamwork 0.21 0

Have a discussion with classmates 0.52 1

Brainstorm with classmates 0.48 2

36

Table 4.8 Was there a particular feature you used most in VirBELA?

Chatbox 68%

Used all fairly equally 24%

Shared resource document 20%

37

3. Heuristic Evaluation

Problem Nielson Heuristics Violated

VE Heuristics Violated

Severity Rating Proposed Recommendation

Students talking at the same time ===> Instructor receives everyone’s responses at once and it’s jumbled. He can’t even call on someone because he can’t look at the white board and look at who’s speaking at the same time.

Match between system and the real world

Natural engagement, natural expression of action, clear turn-taking 3

Give instructor ability to view white board or document while viewing class.

The student resource board went into an infinite loop (gets stuck for everyone) when someone doesn’t have flash and tries to download it.

Error prevention, Help users recognize, diagnose, and recover from errors 4

Give users some ability to refresh a particular module (i.e. if the student resource board doesn't work, close and reopen it).

For the student resource board, one user received the error message “You don’t have permission to change this link.”

Help users recognize, diagnose, and recover from errors 3

Give users better description of this error and how to fix it.

One user received the error message "The game crashed." and had to reinstall the application in order to use it.

Help users recognize, diagnose, and recover from errors 3

Give users better description of this error and how to fix it.

If something isn't working, users have to quit the application and log in again

Help users recognize, diagnose, and recover from errors 3

Give users ability to refresh session rather than close and log in again, if something isn't working.

Twiddla: very difficult to do freehand doodling with trackpad.

Flexibility and efficiency of use

Compatibility with user's task and domain, sense of presence 3

Compatibility/support for tablets (at least for instructors)

Shared screens show up blank or black for half the students Error prevention 4

Give users option to refresh screen until it appears correctly.

Graphics incompatibility

Error prevention; Help users recognize, diagnose, and recover from errors 3

38

A lot of echo when people use the microphone

Match between system and the real world, Consistency and standards

Natural engagement, natural expression of action, sense of presence 3

Not apparent how to open chat box

Flexibility and efficiency of use, recognition rather than recall

Support for learning 2

Can’t close chat box unless you wait and don’t move anything

User control and freedom, aesthetic and minimalist design 2 Give users option to minimize chat box

If someone types and enters something into chat, window pops up again

User control and freedom, aesthetic and minimalist design 2

If chat window is minimized, highlight it without re-opening if a new message appears.

There is an error message displayed at the top of the chat box which says "Compiled userconfig guilayer URL is different than user defined serverconfig". The chat function still works but it is very confusing for the user to see such messages and the inherent meaning is not clear. There is no way to close or hide this message.

Help users recognize, diagnose, and recover from errors 1

Chat: lightbulb is used for typing (that’s not common practice and the implied message is incongruous with the action) Consistency and standards 2

If someone is typing, there should be a speech bubble with the ellipsis rather than a lightbulb

No jump option (which is a staple in most 3D 3rd person simulated environments) Consistency and standards

natural expression of action 1 Provide jump option

Avatar and avatar names block screens

User control and freedom, aesthetic and minimalist design 2

An option to become transparent or disappear when focused on screens would address this to some degree, currently, people can walk into your field of vision and obstruct the screen you’re looking at.

General commands: no tutorials of any kind, certain functions left unexplained (running, special select, etc.)

Recognition rather than recall

Support for learning 3

Tutorials, support for learning during onboarding

39

Student list takes up too many spaces once the class becomes populated, takes up screen space and clutters UI (no option to minimize?)

User control and freedom, aesthetic and minimalist design 1 Inlcude option to minimize student list

Student list is also in a strange location, typically those lists are in the bottom right hand corner (top left is for your avatar and his/her related actions/stats) Consistency and standards 1 Move student list to bottom right the “Go To” function is hidden under your name for whatever reason (conflicting ideas; clicking on your name would typically lead to functions related to your avatar status like emotes and stats, GoTo is a form of fast travel more semantically related to maps/functions) Consistency and standards

Navigation and orientation support 1 Move GoTo to Maps/Functions menu

Map not adjustable, must actually move in the virtual world in order to move map (very inefficient way to plot path, when you’ve found a new route you’ve already changed position. Or you might be going in the wrong direction while looking for a building and get even more lost)

Flexibility and efficiency of use

Navigation and orientation support 2 Allow user to adjust/move map

Can't enlarge map (limits your scope and loses the opportunity to add low level details to increase map specificity)

Flexibility and efficiency of use

Navigation and orientation support 2 Allow user to enlarge map

screens in each classroom tend to blend in with background: borders on the screens are white and the wall is a very light off-white

Aesthetic and minimalist design 1

Darken everything outside the screen (like what Hulu and Youtube do with their “theatre view” function)

application takes too much battery power, not conducive to learning from distances unless you’re near an outlet. But then you become stationary to a degree and the intent of the application is constrained.

Flexibility and efficiency of use 3

Names of classrooms not very unique - confusion with B and D when using the microphone to tell people where to go

Navigation and orientation support 1

Give classrooms unique names (possibly relate back to UCI for "campus feel")

No chairs or desks in classroom: people don't know where to stand and can block each other's view

Match between system and real world

Natural engagement 1

Create seating to avoid awkwardness, people blocking each other's view

40

Not enough context/information upon login (user’s class schedule, subject, instructor name and other details) Visibility of system status 1

There should be a welcome screen modelled as a notice board that displays the user’s class schedule, subject, instructor name and other details when one logs in.

Also, currently there is no ambient sound or auditory cues to guide users.

Visibility of system status, match between system and real world 1

If a class is happening, the user should be able to faintly hear some sounds to guide them to the class. Similarly, other sounds like bells for breaks should be integrated as auditory cues.

There is no forum or service that provides help information for people to troubleshoot their issues within the VE

Help and Documentation, help users recognize, diagnose and recover from errors 2

Create VirBELA Help Center offering best practices to deal with common issues, or potentially a means of contacting someone to troubleshoot their problem. In-product support even better.