Tsiatsos, T., Andreas, K., & Pomportsis, A. (2010). Evaluation Framework for Collaborative Educational Virtual Environments. Educational Technology & Society, 13 (2), 65–77.

65 ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.

Evaluation Framework for Collaborative Educational Virtual Environments

Thrasyvoulos Tsiatsos, Konstantinidis Andreas and Andreas Pomportsis Department of Informatics, Aristotle University of Thessaloniki, GR-54124, Thessaloniki, Greece //

tsiatsos@csd.auth.gr // akons@csd.auth.gr // apombo@csd.auth.gr ABSTRACT

In this paper we will focus on a specific category of Collaborative Virtual Environments that aims to support Collaborative Learning. We call these environments Collaborative Educational Virtual Environments. Our aim is to analyze the evaluation process through the study of relevant bibliography and by doing so reveal the existing research gap which fails to evaluate the twin nature of CEVEs. Our proposed framework aims to evaluate both the pedagogical and educational nature of a CEVE by incorporating two consecutive and recurrent cycles, each consisting of distinct evaluation phases. Furthermore, each phase is designed to address specific evaluation goals. Finally, the paper presents two case studies, applying the proposed theoretical methodology in a pragmatic 3D educational environment.

Keywords

Collaborative virtual environments, Evaluation framework, Educational virtual environments, Usability evaluation Introduction In the past few years and due to the evolving growth in networking and telecommunication technologies a number of interactive Virtual Environments (VEs) have been developed. To discern VEs from typical software applications, they can be defined as interactive, multisensory, three-dimensional, computer synthesized environments (Barfield et al., 1995). Three-dimensional (3D) VEs come with varying features; however, typically most provide three main components: the illusion of 3D space, avatars that serve as the visual representation of users, and an interactive chat environment for users to communicate with one another (Dickey, 2005). Specific types of VEs can be distinguished based on their use or purpose. A Collaborative Virtual Environment (CVE) is a computer-based, distributed, VE or set of places. In such places, people can meet and interact with others, with agents, or with virtual objects (Churchill et al., 2001). CVEs are powerful and engaging collaborative environments for e-learning, because they are capable of supporting several important learning objectives (Chee & Hooi, 2002). These objectives include experiential learning, simulation-based learning, inquiry-based learning, guided exploratory learning, community-based learning and Collaborative Learning (CL). It is probable that CVEs will play an important role in future education due to continuous enhancements in computer technology and the current widespread computer literacy among the public. To keep up with such expectations, e-learning systems have gone through a radical change from the initial text-based environments to more stimulating multimedia systems.

Figure 1. Clarifying the definition of CEVEs

66

This paper focuses on a specific category of CVEs that aims to support CL. We call these environments Collaborative Educational Virtual Environments (CEVEs). Figure 1 clarifies the position of CEVEs in relation to CVEs and VEs in general. In CEVEs the users can undertake different roles and rights; the learners should not be passive, but should be able to interact; they should support various e-learning scenarios and they should have common features with a physical space. Furthermore, the educational interactions in the CEVE should change the simple virtual space into a communication space. This means that users should be provided with multiple communication channels, which enable them to interact with each other, inside the virtual space. CL is group-based learning, regardless of whether this takes place face-to-face, via computer networks, or a through a mixture of both modalities. According to Petraglia (1997), a technologically sophisticated CL environment, designed following cognitive principles, could provide advanced support for a distributed process of inquiry, and facilitate advancement of a learning community’s knowledge as well as the transformation of the participants’ epistemic states through a socially distributed process of inquiry. There are many important benefits to using Computer Supported Collaborative Learning (CSCL) in education (Bruckman & Bandlow, 2002). Therefore, the combination of collaborative e-learning and CVEs (i.e., CEVEs) seems to be an effective solution for supporting CSCL processes. However, there is no concrete and focused evaluation framework which can assess this combination. The evaluation phase in a software application’s development cycle is an important and difficult task of paramount importance (Turner & Turner, 2002). Through expedient evaluation, software houses can cut down on development cost and time. Evaluation can also aid in the demonstration of a software application’s benefits to funding sources and can improve its overall effectiveness and appeal. Better evaluations can lead to better designs, based on users’ and experts’ recommendations and suggestions. Moreover, according to the above definition of CEVEs it is obvious that the evaluation of a CEVE needs a different approach than that used for a typical VE. Pedagogical issues, like conceptual learning, collaboration, constructivism and adaptability, should be taken under serious consideration. Without a descriptive model of the ideal CL process, we cannot easily find indicators that will tell us whether effective learning is taking place (McGowan, 2002). This paper focuses on the presentation of a proposed evaluation framework for 3D CEVEs. This proposal is justified by the research gap revealed in the following section, which briefly presents major frameworks used by researchers in the studied bibliography for the evaluation of CEVEs. Examples are formative evaluation, summative evaluation, predictive evaluation, iterative evaluation, interpretive evaluation and conversational framework. Related work As in any Human Computer Interaction (HCI) research, educational technology designers use formative evaluation to informally understand what needs improvement in their learning environment, and guide the process of iterative design. Formative evaluation must pay attention first to usability, and second to learning outcomes (Bruckman & Bandlow, 2002). It is a qualitative approach, centred mostly on the operational requirements or the user requirements (learner-centred approach). Through formative evaluation, there is an assessment of the concepts and contents of the cognitive objects taught as well as of the supportive material. According to Panagiotakopoulos et al. (2003) the types of formative evaluation are: requirements estimation, structure definition, development process and development fidelity. Furthermore, predictive evaluation can be part of the formative evaluation. It refers to the assessment of the quality and capabilities of the software, before its use by the target group. It is realized through means such as reports, reviews, and check-lists. Expert heuristic and formative evaluations are applied in alternating cycles in the early design state of the CEVE. Based on the expert's knowledge, problems concerning usability can be solved following the expert's recommendations. After these recommendations are considered in a new and better design of the CEVE, the summative evaluation is applied. The objective of the summative evaluation method is to compare different CEVEs designed with the information obtained from the User+Need Space (Goebbels et al, 2003) and to determine the effectiveness of the software compared to predefined goals.

67

One major problem of the current design process is that only after all the procedures have been completed, can we know whether a program currently being developed is efficient or not. A solution is to provide an iterative evaluation during the specific design phases (Song et al., 2007). This idea suggests an evaluation approach which provides all kinds of data that can be used as resources for course designers and innovators in networked learning (Asensio et al., 2006). Another form of evaluation is interpretive evaluation. The main idea of this method is observation of the use of the educational software by the target group. It can be software-oriented, development-oriented or concerning the evolving learning process through the use of the software. Gay & Airasian (2000) suggest a framework constructed out of two different but complementary theoretical models: the conversational framework and the Viable System Model (VSM). The conversational framework is a well-known model of effective teaching practice for academic learning. The VSM is a model for the design and diagnosis of effective organisational structures drawn from management cybernetics. Prasolova-Førland (2008) presented a case study where place metaphors in a number of 3D educational CEVEs are analyzed. The methodology followed was based on an exploratory case study and concludes that a characterization framework of 3D CEVEs could be based on the terms of learner, place and artefact. Even though this is not an evaluation framework, we believe that it could provide guidelines toward the determination of specific metrics in an evaluation framework for CEVEs. Another significant work concerning the way that virtual reality aids complex conceptual learning has been presented by Salzman et al. (1999). Although this work is focused on immersive VEs it is possible to adopt the model’s main ideas for scientific investigations concerning also other types of virtual environments such as desktop-based VEs (Koubek & Müller, 2002). According to this model the first step is to specify the learning concepts and which of the VEs characteristics are able to facilitate the knowledge acquisition. The next step is to gather background information on the learners. Finally, Lee & Wong (2008) have also observed that there is a need for a detailed theoretical framework for VR-based learning environments that could guide future development efforts. They point out that a critical step toward achieving an informed design of a VR-based learning environment is the investigation of the relationship between the relevant constructs or participant factors, the learning process and the learning outcomes. According to the above we can conclude that there is no focused and concrete evaluation framework for evaluating CEVEs. One the one hand, there are many techniques and evaluation frameworks for evaluating either the pedagogical or the technical nature of CSCL systems but none of them is focused on the CEVE nature of the e-learning system. One the other hand, there are several evaluation approaches for CEVE systems. However, none of them is focused both on the pedagogical and educational nature of the e-learning system. Therefore, after identifying the absence of an evaluation framework for CEVEs evaluation we propose a new hybrid model as described in the rest of this paper. Toward an evaluation framework for CEVEs It is important when planning an evaluation to determine which items are assessable. This is often the most complex part. This collection of items is necessary to formulate specific questionnaires and hence to find and eliminate disturbance factors from the implementation of a CEVE (Goebbels et al., 2003). The evaluation goals and the evaluation criteria are usually concurrently defined since they affect each other mutually. In our research we discovered that criteria varied considerably between evaluations. The common criteria used among the majority of researchers include: effectiveness, transparency, confidence, usability, interaction, application, collaborative work, system related criteria and the sense of co-presence. Considering all these evaluation items in one session is almost impossible, since the items mentioned above evaluate too many different aspects of HCI. Also, there is no mutually accepted array of criteria or evaluation framework which is compatible with all educational software. The broadness of CEVE evaluation criteria constitutes an obstruction in the evaluation process. This has been circumvented by many researchers who propose specific criteria

68

partitioning. These partitions differ greatly in number, purpose and scope. Some partitions simply help categorize the criteria while others constitute distinct phases or sessions of the evaluation process. Regardless of the type or model of methodology followed by the evaluator, software evaluation is comprised of certain successive phases (Panagiotakopoulos et al., 2003). The adoption and utilization of CEVEs for supporting the pedagogical process requires a new type of evaluation methodology or framework. In other words, evaluation should not be limited to system and usability assessment, but should also focus on the pedagogical added value of the medium, as well as the new affordances made available through its application. Our rationale concerning the evaluation framework for CEVEs is to organize the evaluation process based on the idea of iterative and incremental development process of an educational VE. An iterative and incremental development process specifies continuous iterations of design solutions together with users. According to Goransson (2001) iteration includes: (a) a proper analysis of the user requirements and the context of use; (b) a prototype design and development phase; and (c) a documented evaluation of the prototype. Therefore, concerning the evaluation process, that is the main interest in this paper, we need to conduct various evaluation cycles in order to evaluate each prototype of the system. The evaluation of each prototype system will result in suggestions for modifications in the following version of the prototype system design. In the case of a CEVE we propose to conduct three phases in each evaluation cycle (Figure 2), namely: Pre-analysis phase Usability phase Learning phase

Figure 2. Overview of an evaluation cycle for CEVEs

A brief and general description regarding the necessity of the above phases as well as useful techniques for their successful conduct is presented in the following paragraphs. Pre-analysis Phase Formative evaluation must pay attention first to usability, and second to learning outcomes. Enhancing this idea, as shown in Figure 3, evaluation goals can be defined in the form of questions. Two types of questions can be formed (Bruckman & Bandlow, 2002): (a) the evaluation methodology questions and (b) evaluation questions. The evaluation methodology (meta-evaluation) questions concern questions regarding the general process of evaluation. Usually, the selected evaluation methodology or framework contains specific inherent goals. For example, the experimental model checks and evaluates the technical characteristics of the software and its purpose is to reveal design issues concerning functional requirements (Lawton, 1978). On the other hand, the teacher-as-a-researcher model uses the teacher in order to assess the effect of the software in-class and can help reveal prejudice problems, role conflicts and student overworking (Panagiotakopoulos, 2003).

69

Figure 3. Organizing the evaluation goals

In any case, before selecting a specific evaluation methodology, the evaluator’s desired outcomes should be in consonance with the possible results of a framework. Asensio et al. (2006) presented a list of questions that can aid evaluators in detecting the appropriate metrics and selecting a fitting methodology. The evaluation questions concern questions regarding usability and learning outcomes. Usability refers to the ability of the system to support the learning process. This covers the effectiveness, efficiency and satisfaction with which specified users achieve specified goals in particular environments (ISO 9241-12, 1998). Furthermore, in the usability questions, the characteristics of the system (e.g., avatars, non-verbal signals, and communication tools) are evaluated. The learning outcome questions concern pedagogical research. Learning outcomes are derived from educational objectives since teaching, like research, is a purposeful activity. Learning outcomes represent the translation of aims into specific, tangible, attainable terms. For the definition of learning outcomes and objectives, Bloom’s taxonomy (Bloom, 1956) can be used. In order to help teachers understand and implement standard-based curriculum Bloom’s taxonomy has been revised (Anderson & Krathwohl, 2001). Classifying instructional objectives using the revised taxonomy helps to determine the levels of learning included in an instructional unit or lesson. Furthermore, in this phase, the evaluator can gather information regarding the individual learning styles. This information can then be used in conjunction with questionnaire or interview results to ascertain the validity and weight of a learner’s opinion recorded in the other phases. For example, during the Learning Phase, the tutor presents learning objects by utilizing a specific pedagogic approach. If the selected approach is not in accordance with a learner’s learning style, then his opinion will probably bear a fair amount of subjective negativity and must therefore be judged in a lighter objective fashion. According to (Graf & Kinshuk, 2006), the Felder-Silverman learning style model (FSLSM) seems to be the most appropriate for use in computer-based educational systems as it describes the learning style of learners in more detail than other models. For identifying the learning styles according to FSLSM, the Index of Learning Styles (Felder & Soloman, 1997), a 44-item questionnaire, can be used. Some example types of learners according to the FSLSM are: Active learners, which learn by trying things out and working with others. Reflective learners, which learn by thinking things through and working alone. Visual learners, which remember best what they have seen. Verbal learners, which get more out of words, regardless whether they are spoken or written. Usability Phase Usability inspections of the initial applications are necessary so as to uncover the main design flaws and allow a clean up of the design, meanwhile adapting the method to 3D collaborative aspects. Usability and interaction are very much interrelated. Concerning interaction, social-arbitrary knowledge (i.e., language, values, rules, morality, and symbol systems) can only be learned in interactions with others. Several HCI rules for display design must be taken into account when implementing any e-learning system. These include consistency of data display, efficient information assimilation by the user, use of metaphors, minimal memory load on user, compatibility of data display with data entry, flexibility for user control of data display,

70

presentation of information graphically where appropriate, standardized abbreviations, and presentation of digital values only where knowledge of numerical value is necessary and useful. Learning Phase The main goal of the learning evaluation phase is to conduct pedagogical research. As proposed by Lee & Wong (2008) the model presented by Salzman et al. (1999) can be a useful guide in designing, developing and evaluating a VR learning environment. Although this model is focused on immersive VEs it could also be used in desktop CEVEs in order to address which CEVE features are appropriate, what kind of students might gain benefits through learning in CEVEs, and how CEVEs enhance learning by looking into the interaction and learning experience. Therefore, our proposed evaluation framework is based on the rationale of this model in order to test scientific hypotheses concerning learning and communicating in VEs through experimental studies. Overview of the proposed framework Our proposed hybrid evaluation framework is structured by two evaluation cycles comprised of phases as presented in Table 1. However, as mentioned before, this set of cycles can be repeated as many times as necessary in order to achieve the system development and evaluation goals desired. In other words, for each additional version of the system an additional cycle can be added.

Table 1: Overview of the proposed framework Cycle 1

Phase Step Goals Pre-analysis Pre-test Set the evaluation goals and separate the target group (e.g., learners,

tutors, researchers, designers) into advanced and novice CEVE users and determine learners’ learning styles.

Usability Familiarization session (usability session 1a)

Uncover usability problems of the most important parts of the user interface concerning the basic functionalities of the first prototype.

Co-presence session (usability session 1b)

Uncover usability problems of the communication and collaborative functionalities of the first prototype.

Expert evaluation Similar to the “Familiarization session” and “Co-presence session”. It also integrates the know how of usability and pedagogical experts in the field of CVEs.

Learning Learning scenario-based session

(learning session 1)

Collect further requirements and additional functionality. Discover the pros and cons of the VE. Determine the appropriateness of different kinds of learning scenarios

Cycle 2 Phase Step Goals

Pre-analysis Meta-evaluation Insure that the evaluations of Cycle 1 are valid, accurate and unbiased.

Usability Usability session 2 Uncover usability problems of advanced features of the second prototype, including new concepts of the 3D-metaphor that were developed according to the results of the first evaluation cycle.

Learning Course scenario-based session

(learning session 2)

Collect information about the possible changing experiences and attitudes of the users toward the VE, discover appropriateness of different kinds of learning courses and collect data about absent possibilities and functions.

Each cycle is made up of the previously described phases conducted in individual steps. A diagram describing the proposed framework can be seen in Figure 4 at the concluding section of this paper.

71

The structure of the development and evaluation process presented here follows the idea of iterative design: a process of testing a system prototype, redesigning based on the results of evaluation and testing it again. This set of steps will constitute the framework for this paper, each step being presented in detail in the following sections. After each step there is a data collecting step which consists of several different types of means which are applied in order to mine information, opinions and thoughts from the evaluation participants. These means can be separated into two categories according to the type of collected data. Next, the data analysis step utilizes the results collected previously through the implementation of specific statistical models and statistical hypothesis tests, in order to convert the raw data into constructive information. The final step of each cycle includes the interpretation of the collected data and the production of information. This interpretation is made in correlation with the evaluation questions and the desired outcome set in the initial phases. In other words, as mentioned in the related work section, summative evaluation takes place in order to determine the effectiveness of the software compared to predefined goals. If the results provide answers to the evaluation questions and are in consonance with the desired outcome, conclusions can be made regarding the efficiency of the evaluation. After this process, evaluators can proceed with the augmentation of the prototype and if it is necessary and possible repeat the evaluation cycles. The following sections present in detail the two evaluation cycles of the proposed framework.

Evaluation Cycle 1 The main goal of evaluation cycle 1 is to conduct formative evaluation in the first version of the system in order to help the development process of the system. Another aim of this cycle is to determine the appropriateness of different kinds of learning scenarios. The following paragraphs present each step included in this cycle concerning pre-analysis, usability and learning phases.

Pre-analysis phase: Pre-test The choice of evaluators depends greatly on the intended target group of the CEVE and its ability to cooperate with the CEVE designers. According to our suggested framework (Table 1), before evaluating the CEVE, the evaluation candidates should participate in a two part pre-test questionnaire. The first part of the pre-test will inquire about facts such as previous experience with 3D VEs in general and CEVEs specifically. Also, information such as age, profession and education should be taken into consideration. Based on the results of the first part of the pre-test, the candidates can then be organized into groups of novices and advanced users if necessary. In this way, during the evaluation process the participants will be able to relate to each other more easily and thus facilitate collaboration. Also, being of the same level, users will be able to proceed in a more uniform manner. As mentioned in a previous section the second part of the pre-test can be used to identify learning styles. To this end, the Index of Learning Styles (Felder & Soloman, 1997), a 44-item questionnaire, can be used.

Usability phase The usability phase can consist of three sessions: familiarization, usability and expert evaluation, presented in the following paragraphs. Familiarization Session In this session, users familiarize themselves with the CEVE’s navigation control scheme and overall user interface. Through completion of this session they will be capable of answering a questionnaire regarding issues such as usability, user friendliness, appearance, aesthetics, interactivity and adaptability. Example evaluation subjects

72

include: avatar navigation, virtual object manipulation, virtual metaphors and use of symbols. The evaluation takes place in a laboratory test setting using data collecting tools which include: think aloud protocol, questionnaires with open/closed questions, interviews and audiovisual recordings. In other words, interpretive evaluation is used to observe the use of the educational software by the target group. After this session the users will be more capable of participating in the next session since the familiarity with the user interface will enable them to concentrate on evaluating the tools and services offered by the CEVE. Co-presence Session In this session, users evaluate the communication and collaboration capabilities of the CEVE through cooperation with others. In similar fashion to the previous session, after completing the session users answer a questionnaire regarding issues such as usability, user friendliness, appearance, aesthetics, interactivity and adaptability. Example evaluation subjects include: control of avatar gestures and facial expressions, communication and collaboration tools, group coordination tools, data sharing capabilities and security/privacy issues. As before, the evaluation takes place in a laboratory test setting using data collecting tools which include: think aloud protocol, questionnaires with open/closed questions, interviews and audiovisual recordings. Expert Evaluation The goals of expert evaluation are similar to those of the previous evaluation sessions. In addition, researchers can opt to integrate the know-how and opinions of external usability and pedagogical experts. Data collection for this session is based on: heuristic evaluation, consistency inspection and usability walkthroughs. Several guidelines and heuristics are available to check the usability of software interfaces. For web applications, specific style-guides are relevant. Additionally, general criteria like the ISO 9241-10 can be used. On the other hand, usability walkthroughs simulate typical sequences of actions of the user. Common and important tasks and the sequence of necessary subtasks are identified and performed by the evaluator. For each step, the cognitive processes of the user are considered, whether the system provides sufficient feedback so that the success or failure of an action with respect to the intended goal can be recognized by the user. Therefore, usability walkthroughs mostly focus on the dynamic and interactive features of the system.

Learning phase: Learning scenario-based session Tutors and learners comprise the evaluation group of this session. Relevant criteria examined here are: How users progress without strong guidance How users take advantage of CEVE functionality in order to carry out the learning process. Tools of great

interest here are: avatars, avatar gestures and facial expressions, communication tools (e.g., audio chat) Users’ rating of the system in terms of group awareness, social presence and immersion concerning learning in

VEs User’s acceptance of various CL scenarios in terms of the time devoted, the extent of the collaboration (e.g.,

number of messages exchanged) and percentage of learning task completion Data collection in this session is based on laboratory tests applying various learning scenarios such as tutor-centered or collaborative. Data collection tools could include the think aloud protocol, questionnaires with open/closed questions, interviews and audiovisual recordings.

Evaluation Cycle 2 The main goals of evaluation cycle 2 are (a) to validate the evaluation conducted in the first evaluation cycle; (b) to uncover usability problems of advanced features of the second prototype; (c) to collect information about the possible changing learning experiences and attitudes of the users toward VEs.

73

The following paragraphs present each step included in this cycle concerning pre-analysis, usability and learning phases.

Pre-analysis phase: Meta – evaluation As presented in Figure 3 goals can be defined with regard to the evaluation itself, the learning outcomes and the CEVE’s features. Meta-evaluation is the evaluation of an evaluation, evaluation system or evaluation device. In more detail, meta-evaluation is also defined as the process of delineating, obtaining, and applying descriptive and judgmental information about the utility, feasibility, propriety, and accuracy of an evaluation and its systematic nature, competent conduct, integrity/honesty, respectfulness, and social responsibility to guide the evaluation and/or report its strengths and weaknesses (Stufflebeam, 2001). Meta evaluation is important in order to insure that evaluations are valid, accurate and unbiased. The two primary instruments for judging program evaluations are The Program Evaluations Standards (Sanders & JCSEE, 1994), which present detailed criteria for determining an evaluation's utility, feasibility, propriety, and accuracy, and The AEA Guiding Principles (Bamberger et al, 2006) that were developed for evaluators to guide the evaluation of a broader range of evaluations.

Usability phase: Usability Session 2 After augmenting the system based on the results and suggestions derived from Cycle 1 of the evaluation, the research team can proceed with a second evaluation of the system’s usability. More specifically, this session can be used to uncover usability problems of advanced features of the second prototype including new concepts of the 3D-metaphor that were developed according to the results of the first evaluation cycle. The criteria evaluated and the data collection methods used are similar to those utilized in the “Co-presence Session”.

Learning phase: Course scenario-based session According to the description of Table 1 tutors and learners comprise the evaluation group of this session. Relevant criteria examined here are: How much the users use the CEVE functionality in orders to carry out the learning process. Tools of great

interest here are: avatars, avatar gestures and facial expression, communication tools (e.g., audio chat). For example, group average with regard to number of text statements, number of avatar gestures, etc.

Users’ assessment of the system in terms of group awareness, social presence and immersion concerning learning in VEs.

The subjective satisfaction regarding the learning outcomes. Data collection in this session is based on pre and post-session questionnaires to assess learners’ characteristics and their knowledge concerning the topics of the course. Also, interviews and audiovisual recordings could be used.

Case Studies In this section we present the results of a small scale study carried out in a tertiary education department, to assess two custom designed CEVEs exploiting the proposed evaluation framework. One CEVE was designed on top of the open source platform Croquet, to utilize the Jigsaw CL technique, while the other was implemented through the Second Life platform and examined the application of a combination of CL techniques (i.e., Jigsaw and Fishbowl).

The case of Croquet This section presents the first case study, which involved three components: the Croquet platform, the Jigsaw CL technique and the participation of undergraduate students. Although there are several successful CL techniques,

74

applied in traditional educational settings, we chose to utilize Jigsaw because it is a CL method with a three-decade track record of increasing positive educational outcomes (Aronson & Bridgeman, 1979). In Table 2 below, we describe the individual phases of the cycles based on their goals, and with regard to our evaluation study present proposed evaluation methods and indicative examples of expected outcomes.

Table 2: Application of the framework in the Croquet case study Evaluation Cycle 1 Evaluation Method Outcomes

Pre-test Questionnaires regarding previous experience.

The majority of participants had used distance collaboration software in the past.

Familiarization session

Users were allowed to explore the 3D environment and experiment with the user interface and the available tools and functionality.

Positive reaction. Users had no difficulty in learning how to operate the user interface or managing basic functionalities.

Co-presence session

Users were organized into groups of 2 or 3 and were allowed to communicate using the available functionality. As a collaboration activity users had to co-browse the Internet in search of specific information topics.

Disappointment in the networking performance of the platform. Dissatisfaction regarding system stability and system response time.

Expert evaluation Usability walkthroughs. Many problems concerning user navigation and networking performance.

Learning scenario-based session

Utilization of jigsaw CL technique. Here, group members must work together as a team to accomplish a common goal; each student depends on everyone else.

The majority said that a number of technical difficulties hindered the scenario process. Despite this, they still speak in favor of the platform albeit with a few suggested improvements.

Evaluation Cycle 2 Evaluation Method Outcomes

Meta-Evaluation

The two primary instruments for judging program evaluations are The Program Evaluations Standards and The AEA Guiding Principles.

Results based on comparison with the defined standards of feasibility, accuracy, propriety etc.

Usability Evaluation

Users were organized into groups of 2 or 3 and were allowed to communicate using the available functionality. As a collaboration activity, users had to co-author a document.

The major advantages of the platform are: 3D graphics, support for collaboration communication tools.

Course scenario-based session

Pre and post session questionnaires. Scenario based session with role play and simulation elements.

Uncertainty regarding the pedagogical value of the Croquet platform.

Every step of the framework has specific goals, achieved through the selection and application of established evaluation methods. Table 2 demonstrates how the evaluation model assists in the comprehensive assessment of the CEVE by collecting data and comments on both usability and educational issues. The case of Second Life This section presents the second case study, which involved four components: the Second Life platform, the Jigsaw CL technique, the Fishbowl CL technique and the participation of postgraduate students. From the results of the Croquet case study we concluded that although, both platforms support basic features that are crucial for a CSCL scenario (e.g., user representation, user-actions representation, etc.), it seems that SL is more technically stable and that it offers a more intuitive user interface than Croquet. Furthermore, based on user comments and on the analysis of the completed questionnaires for the Croquet case study, we proceeded to augment SL platform’s capacity to support the Jigsaw CL technique. The online

75

transferability of Jigsaw was enhanced by the development of specific virtual tools, metaphors and functionality such as distinct avatar clothes for each team, helpful avatar gestures, and query/idea visualization tools. Although the enhancements described above aided the overall execution of Jigsaw, the results of the first evaluation cycle reveal further limitations regarding its online transferability which were imposed by the inherent non-collaborative nature of SL. More specifically, the lack of a collaborative word processor restricted students in their work and necessitated face to face meetings or the use of other collaboration software. For this reason, in the second evaluation cycle, the scenario was altered by incorporating a new CL technique (i.e., Fishbowl) and adding new functionality (e.g., voice chat management, Jigsaw armchairs). As before, in Table 3 below, we describe the individual phases of the cycles based on their goals, and with regard to our evaluation study present proposed evaluation methods and indicative examples of expected outcomes.

Table 3: Application of the framework in the SL case study Evaluation Cycle 1 Evaluation Method Outcomes

Pre-test Questionnaires regarding previous experience.

The majority had used distance collaboration software in the past.

Familiarization session

Users were allowed to explore the 3D environment and experiment with the user interface and the available tools and functionality.

Over 70% of the students used the platform beyond the limits of the case study after familiarizing themselves with it.

Co-presence session

Users were organized into groups of 3 or 4 and were allowed to communicate using the available functionality. As a collaboration activity users had to co-browse the Internet in search of specific information topics.

Students are divided regarding the capabilities of the platform to support CL techniques, as opposed to the traditional face to face method.

Learning scenario-based session

Utilization of jigsaw CL technique. Here, group members must work together as a team to accomplish a common goal; each student depends on everyone else.

50% of the students believe that the effectiveness of collaboration was reduced by using the SL platform.

Evaluation Cycle 2 Evaluation Method Outcomes

Meta-Evaluation

The two primary instruments for judging program evaluations are The Program Evaluations Standards and The AEA Guiding Principles.

Results based on comparison with the defined standards of feasibility, accuracy, propriety etc.

Usability Evaluation

Users were organized into groups of 3 or 4 and were allowed to communicate using the available functionality.

More than half of the students said that they will continue to use the SL platform even after the collaborative activity was over

Course scenario-based session

Utilization of the fishbowl CL technique. Here, students form concentric circles with the smaller, inside group of students discussing and the larger, outside group listening and observing.

Most of the students (a) would like to see more CL techniques being implemented, and (b) would be interested in participating in online lectures through the SL platform.

Conclusion and Future work In this paper, based on the examination of specific evaluation frameworks as suggested by researchers, we pointed out the research gap which exists regarding CEVE evaluation In other words, most evaluation methodologies focus on either the technical or pedagogical side of CEVEs. Motivated by these findings we formed the base for our hybrid evaluation framework (Figure 4) by combining the work of individual researchers with our own innovative approach.

76

Figure 4. Diagram of the proposed framework

As revealed by the two case studies presented in a previous section, the education/pedagogical value of our approach can be concisely described as such: It completes the research gap, which stems from ignoring the twin nature of CEVEs It is an iterative approach to the design and implementation of pedagogical scenarios It provides a methodology for the examination of the online transferability of traditional CL techniques It facilitates the identification of missing functionality which restrict the effective online transferability of

traditional CL techniques Our framework consists of two consecutive cycles. This set of cycles can be repeated as many times as necessary in order to achieve evaluation goals. Each cycle is made up of individual phases consisting of three steps: main phase step, data acquisition step and data analysis step. There are three types of phases: pre-analysis, usability and learning. Future research work includes the application of our suggested framework on more CEVE platforms and the continuing augmentation of our framework based on evaluation results and relevant bibliography. In addition, we will examine several more CL techniques which can be utilized in the Learning Phase of our proposed framework in order to assess their applicability and effectiveness when applied through a 3D CEVE. More specifically, we can examine and increase the potential of CL techniques which have been tagged as having low or moderate online transferability by the research community. By implementing specific functionalities and realizing particular didactic approaches, we can effectively transfer these techniques to the 3D online environment. References Anderson, L.W. & Krathwohl, D.R. (Eds.) (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. NY: Addison Wesley Longman.

Aronson, E. & Bridgeman, D. (1979). Jigsaw groups and the desegregated classroom: In pursuit of common goals. Personality and Social Psychology Bulletin, 5, pp. 438-446.

Asensio, M., Hodgson, V. & Saunders, M. (2006). Developing an inclusive approach to the evaluation of networked learning: the ELAC experience, Proceedings of the Fifth International Conference on Networked Learning 2006, 10-12 April 2006, Lancaster University.

Bamberger, M.J., Rugh, J. & Mabry, L. (2006). RealWorld Evaluation: Working under Budget, Time, Data, and Political Constraints, Sage Publications, Inc; February 13, ISBN-13: 978-1412909464.

Barfield, W., Zeltzer, D., Sheridan, T. & Slater, M. (1995). Presence and Performance within Virtual Environments. In W. Barfield & T. Furness (Ed.). Virtual Environments and Advanced Interface Design, 473-513, Oxford University Press.

77

Bloom, B.S. (1956). Taxonomy of Educational Objectives: The Classification of Educational Goals, Handbook 1: Cognitive Domain. 1956, NY: McKay, ISBN: 978-0582280106.

Bruckman, A. & Bandlow, A. (2002). HCI for Kids, Published in: The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications, J. Jacko & A. Sears (Eds). Lawrence Erlbaum and Associates.

Chee, Y. & Hooi, C. (2002). C-VISions: Socialized learning through collaborative, virtual, interactive simulation, in Proceedings. of CSCL 2002: Conference on Computer Support for Collaborative Learning, Boulder, CO, USA, pp. 687-696. Hillsdale, NJ: Lawrence Erlbaum.

Churchill, E., Snowdon, D. & Munro, A. (2001). Collaborative Virtual Environments: Digital Places and Spaces for Interaction, Springer-Verlag, London Limited, Great Britain. ISBN:1852332441.

Dickey, M. (2005). Brave new (interactive) worlds: A review of the design affordances and constraints of two 3D virtual worlds as interactive learning environments, Interactive Learning Environments, 13(1), 121 -137.

Felder, R.M. & Soloman, B.A. (1997). Index of Learning Styles Questionnaire. Retrieved July 20, 2009 from http://www.engr.ncsu.edu/learningstyles/ilsweb.html

Gay, L. R., Mills, G. & Airasian, P. (2000). Education Research: Competencies for Analysis and Application. (6 ed.). Upper Saddle River, NJ: Merrill. ISBN: 978-0131185340.

Goebbels, G., Lalioti, V. & Göbel, M. (2003). Design and Evaluation of Team Work in Distributed Collaborative Virtual Environments, Virtual Reality Software and Technology Proceedings of the ACM symposium on Virtual reality software and technology, Osaka, Japan, 231-238, ACM Press, NY, USA.

Goransson, B. (2001). Usability Design: A Framework for Designing Usable Interactive Systems in Practice. IT Licentiate theses 2001-06, ISSN 1404-3203, Department of Human Computer Interaction, Information Technology, Uppsala University, Uppsala, Sweden.

Graf, S. & Kinshuk, K. (2006). Considering Learning Styles in Learning Management Systems: Investigating the Behaviour of Students in an Online Course, SMAP '06, First International Workshop on Semantic Media Adaptation and Personalization, Athens, Greece, ISBN: 0-7695-2692-6.

Graf, S. & List, B. (2005). An Evaluation of Open Source E-Learning Platforms Stressing Adaptation Issues, IEEE International Conference on Advanced Learning Technologies (ICALT 2005), July 5-8, Kaohsiung, Taiwan.

ISO (1998). ISO 9241-12: Ergonomic requirements for office work with visual display terminals (VDTs) Part 11: Guidance on usability. Geneva, Switzerland: International Organization for Standardization.

Koubek, A. & Muller, K. (2002). Collaborative and Virtual Environments for Learning, ACM SIG Proceedings, New Orleans, Louisiana, USA, November 16-20.

Lawton, D. (1978). Curriculum evaluation: new approaches. In Lawton, D. (ed) Theory and Practice of Curriculum Studies, Routledge, ISBN 0710000286, 9780710000286, 176-185.

Lee, E. A-L. & Wong, K. W. (2008). A Review of Using Virtual Reality for Learning, Lecture Notes in Computer Science, Z. Pan et al. (Ed.): Transactions on Edutainment I, LNCS 5080, 231–241, Springer, doi: 10.1007/978-3-540-69744-2.

McGowan, J. (2002). Learning Styles Modality Preference Inventory. Retrieved July 20, 2009 from http://homepages.wmich.edu/~jmcgowan/CTE344/session3/Modalityinventory.pdf

Panagiotakopoulos, H., Pierrakeas, H. & Pintelas, P. (2003). Educational software and its evaluation, in Greek. Metaihmio Publications, ISBN: 960-375-579-6.

Petraglia, J. (1997). The rhetoric and technology of authenticity in education. Mahwah, NJ: Lawrence Erlbaum. ISBN: 978-0805820423.

Prasolova-Førland, E. (2008). Analyzing place metaphors in 3D educational collaborative virtual environments, Computers in Human Behaviour, 24(2), 185-204, March.

Salzman, M., Dede, C., Loftin, R. & Chen, J. (1999). A model for understanding how virtual reality aids complex conceptual learning. Presence, 8 (3), 293-316.

Sanders, J.R. & JCSEE (The Joint Committee on Standards for Educational Evaluation) (1994). The Program Evaluation Standards: 2nd Edition How to Assess Evaluations of Educational Programs, Sage Publications, Inc; April 6, ISBN-13: 978-0803957329.

Song, S., Lee, J. & Hwang, I. (2007). A New Framework of Usability Evaluation for Massively Multi-player Online Games: Case Study of “World of Warcraft” Game, Human-Computer Interaction, Part IV, HCII 2007, LNCS 4553, 341–350, Springer-Verlag Berlin Heidelberg.

Stufflebeam, D. L. (2001). Evaluation checklists: Practical tools for guiding and judging evaluations. American Journal of Evaluation, 22(1), 71-79.

Turner, P. & Turner, S. (2002). An affordance-based framework for CVE evaluation. People and Computers XVI: Memorable Yet Invisible. Faulkner, X., Finlay, J. & Detienne, F. (Ed.), Springer, 1st Edition, October 3.