Evaluation Plan for “Teaching and Learning

with VMware Workstation Software”

TLT476: Technology Program Evaluation

Tonya B. Amankwatia, Ph.D.

Prepared by: Lifang Chang

Lehigh University, Spring 2008

1

Introduction and Background

Introduction

This evaluation plan describes the procedures that will be used to evaluate

“VMware Workstation (VMW)” implemented by the department of Networking, Security

and System Administration (NSSA) at Rochester Institute of Technology. VMware

Workstation is a virtual machine software suite for x86 or x86-64 computers and it allows

one physical machine to run multiple operating systems simultaneously. Currently, the

department of Networking, Security and Systems Administration (NSSA) at RIT uses

VMware Workstation to provide its computer science students remote access to multiple

virtual operating environments for distance learning. The document begins with

background information on the VMware workstation, followed by the purposes of the

evaluation and the audiences with whom the evaluation results will be shared. Next, the

intended decisions based on the evaluation results and questions to be addressed during

the evaluation process are described. In addition, the proposed methods and

instrumentation, the selected sample and inherent limitations, logistics and timeline are

explained including who, where, when, and how will be doing what. The final section in

this document proposes a budget for the evaluation.

This evaluation plan has been prepared by Lifang Chang, a graduate student in the

program of Teaching, learning and Technology at Lehigh University. The plan has been

informed by discussions with the client, Dr. Jorge Diaz-Herrera, Dean of the College of

Computing and Information sciences, the supervising instructor, Dr. Tonya Bevedare

Amankwatia of the program of Teaching, learning and Technology at Lehigh University.

Background

2

New York’s Rochester Institute of Technology (RIT), with an enrollment of more

than 15,000 students, is known for its professional, science and technology programs.

The Department of Networking, Security and Systems Administration (NSSA) is one of

six departments in the Golisano College of Computing and Information Sciences at RIT.

The department of NSSA has five computing labs that serve nearly 400 students, many of

whom are working and can’t be on campus every day. The IT (Information Technology)

administrators and managers chose VMware virtualization software to provide a way for

students to remotely access the multiple operating environments they normally work with

in the lab.

VMware Workstation (VMW) provides couple of virtual systems (such as

Windows and Linux) and configurations to multiple students with a single server. Using

personal computers, students can remotely log in and instantly be connected to an

available virtual machine rather than having to be physically present in the lab to log on

to individual systems.

As shown in the following screen capture of the system interface, VMM can run

Windows under Linux (P1),

3

(P1)

run Ubuntu (one type of Linux) under Windows (P2),

(P2)

run Windows Vista under Mac OS/X (P3),

4

(P3)

and also can support a virtual machine running Windows XP on a Windows Vista host

(P4).

(P4)

The VMW software suite is distributed by electric download from the VMware

website with discount cost to academic communities.

Purposes

5

The purpose of this evaluation is to collect data which will provide decision

makers at the College of Computing and Information Science in RIT with the timely,

accurate information required to support decisions regarding the extension, management,

and marketing of the VMware Workstation software.

The specific sub-purposes:

1. To collect information regarding the acceptability, applicability, and

effectiveness from the target audience of students using the virtualization software.

2. To collect information regarding the implementation of the various components

of VMware Workstation initiative into the overall teaching and learning environment,

with a goal of providing information that can be used to improve the future

implementation in the entire college.

Audiences

The primary audiences for the evaluation will be the dean council including the

college dean, Dr. Jorge Diaz-Herrera and other 16 council members. Important secondary

audiences include supporting faculties and staff of the VMware Workstation operating

environment in the department of NSSA. Potential audiences include the Information

Technology managers and administrators from other departments and organizations at

Rochester Institute of Technology.

Decisions and Questions

The evaluation will be used to inform two major decisions to be made by the dean

council and supporting faculties and staff. Also, most of these decisions will be made

regardless of the quantity and quality of information available to the decision makers.

The following decisions are assumed:

1. Should the department continue the use of VMware virtualization software?

6

2. What techniques and strategies should be used for effectively integrating

VMware virtualization system across the curriculum?

3. How should VMware workstation software be used for enhancing students’

real-world problem-solving skills?

4. How should VMware workstation software be used for improving server

utilization?

5. How should VMware workstation software be used for enhancing extending

access to and use of information resources?

6. What techniques and strategies should be used for integrating the VMware

Workstation virtualization solution in the wider Computing and Information Science

College?

7. How should people and other resources be deployed to support an everywhere

computing environment based upon the VMware workstation solution?

In order to have the information needed to guide the types of decisions noted

above, many questions must be addressed:

A. What are student reactions to using VMware Workstation system?

• Does the student enjoy using the module?

• Does the student think that the general skills learned with the VMware

software were relevant to their current research environment or future

career plans?

• What is their level of satisfaction with the learning experience in the

VMware virtualization environment?

• What do they learn from the VMware software?

7

B. What are reactions of the faulty and computing labs’ staff to the VMware

Workstation system?

• What is their expert opinion of the cost-effective of the VMware software?

• What is their expert opinion of the usability of the VMware software?

• What is their expert opinion of flexibility of the VMware software?

C. How can the department of NSSA best share its experience with VMware

virtualization software with other departments looking to implement similar programs?

D. What would the NSSA department like to work with the other VMware

virtualization technologies?

Methods

Couple of descriptive and interpretive methodologies will be used to assist the evaluators

in gathering data to answer the questions listed above as well as other questions that will

surely revealed during the evaluation time. The data collection methods include:

1. Interviews with faculties who have integrated VMware Workstation with their

curriculums.

2. Interviews with computing lab personnel

3. Observations in the computing lab

4. Questionnaires to be completed by students who have utilized LTTS modules

5. Expert review of the VMware Workstation software from three types of experts

(content, usability, and flexibility)

The primary source of data will be collected through interviews with faculties and

technology staff in the department. The evaluator will go to the computing lab making

the observations. At least three interviewees in the department of NSSA will be

interviewed based on their permission. Interview protocols include questions about

8

virtualization deployment environment, networking security, students’ academic

performance and open-ended questions (see Appendix A). Information will also be

gathered via a questionnaire what will be given to students (see Appendix B). The

questionnaire will be distributed on the web site with SurveyMonkey.com.

One final method of gathering information will be expert review. This evaluation

requires three different types of experts in content, usability, and distance learning area.

Ideally, an expert who can cover three different areas at the same time is preferred.

The relationships among these five methods and the evaluation questions

identified in the previous section are described in the evaluation matrix that appears

below.

Method What are student reactions to using VMM?

What are reactions of the faulty and computing labs’ staff to the VMW?

How can the department best share its experience with other departments?

What would the NSSA department like to work with the other VMware technologies?

Faculty Interview X X Staff Interview X X X Lab Observation X X Student Questionnaire X X Expert Review X X

Sample

Data will be gathered from several sources during the evaluation. The participants

for this evaluation will be:

• 25 – 30 students who has used VMware software from the NSSA department.

The participants in the evaluation include both undergraduate and graduate students, and

9

also represent a variety of age groups. The students are randomly selected with female

and male.

• 3 expert reviewers from other departments such as Computer Science,

Information Technology and Distance learning. Accordingly, they will focus on the

reviews on content, usability and distance learning area.

• At least 3 faculties who has used VMware software with their curriculums.

• 2 representatives from the department’s computing labs

Instrumentation

The instruments used to gather data for the evaluation include:

• A sample User Questionnaire that the students participating in this evaluation

will complete after they have used the VMware virtualization software. The questionnaire

will be administered during the last ten minutes of one class period. Moreover, the user

questionnaire will be reviewed by the supervising instructor, Dr. Tonya Bevedare

Amankwatia to help assure wording is appropriate.

• A protocol for the User Interviews that will be conducted with selected faculties

who have used the VMware Workstation software in their class.

• A protocol for the User Interviews that will be conducted with selected

computing lab staff who have supported the VMware Workstation operating environment.

• A description of the in-lab observation that will record student and instructor

interactions with the VMware operating environment.

• A expert review checklist that will be conducted with selected experts in

different area such as content, usability and distance learning.

10

• Brief resumes of six experts from whom a panel of three will be selected to

review the VMware software.

All instruments are subject to major revisions, and should only be considered as

rough drafts of the actual instruments to be developed and refined during the real

evaluation.

Limitations

Several constraints on this evaluation must be clarified because they constitute

limitations on the interpretation and utility of this evaluation. First, many of the

participants in this evaluation study are students with a wide range of technology skills.

Different levels of technology skills bring up different sets of needs and expectations

with respect to the technology itself and the context of the operating systems. Therefore,

the results from this evaluation will be affected by the background of the participating

students. On the other hand, a relatively small sample of students is also a limitation.

Students may have varying levels of interest in the topic of networking security,

operating system principle, and software programming depending on their personal

interests and future job plans. It can be anticipated that randomly selected students will

not represent the whole department student populations.

Second, only one virtualization solution will be evaluated. The results cannot be

generalized to all VMware virtualization software. Currently, the Department of NSSA

installs “Remote Laboratory Emulation Systems” on the computing lab. The evaluation

results may not be appropriate for the future ones that will be developed.

Third, the data for this evaluation will be collected over a brief period of time

during the Summer Semester of 2008. Given the brevity of this evaluation, it will be

especially difficult to find out how effective the software are in influencing transfer to

11

teaching and technology practices in the colleges. Consequently, longer term evaluations

will be needed in the future.

Fourth, the evaluator, Lifang Chang, is a novice undertaking this evaluation for

the first time. Although she will be supervised by an experienced evaluator- Dr. Tonya

Bevedare Amankwatia, it is still a student project and the results should be viewed from

that perspective.

Logistics and Time Line

Couple of people will participate in this evaluation, including faculty and graduate

students from Lehigh University. Their roles and responsibilities are displayed in the

following table. Roles and responsibilities for the Rochester Institute of Technology

participants will be clarified after consultation with the personnel there.

Personnel Role Responsibilities Lifang Chang Principle Investigator Direct the overall project effort;

Prepare initial evaluation plan; Direct and participate in data

collection and analysis; Prepare evaluation reports; Deliver evaluation updates info

to various audiences. Tonya Bevedare Amankwatia Co-Investigator Provide expert evaluation

consultation; Assist with project reports; Deliver evaluation updates info

to various audiences. To be determined IT Graduate Assistant Assist with the preparation of

instrumentation; Assist with data collection and

analysis; Assist in preparing evaluation

reports.

The majority of the evaluation work will take place at Rochester Institute of

Technology. The College of computing and Information Science has agreed to sponsor

12

the evaluation project at Lehigh University. Space and equipment for the IT graduate

assistant will be located in the RIT campus.

This evaluation plan will be revised based on an earlier draft prepared in April

2008. The final evaluation plan will be reviewed, revised, and approved by May 2008.

The initial data collection phase of the evaluation itself will commence June, 2008. The

first interim report will be delivered by July 1, 2008. The final report of this evaluation

will be submitted by July 10, 2008.

Budget

Some parts of the budget are hypothetical since resources are not available to pay

for this evaluation. The detailed proposed budget is showed in the following table.

Item 4/15/08~7/10/08 Personnel Proposed Budget Actual Expenditure Rate Hours Amount

Planning 20h $800 Conducting 4h $160 Analysis 10h $400

Evaluator-Chang

$40/hour

Reporting 20h $800 $2160

Planning 2h $400 Analysis 1h $200

Consultant- Dr. Amankwatia

$200/hour

Reporting 2h $400 $1000

Conducting 3h $195 Analysis 1h $65

IT Graduate Assistant

$65/hour

Reporting 0.5h $32.5 $292.5 Total Personnel $3452.5 Duplication

Clerical costs $150

13

Total Duplication $150 Misc. Administration

Participation Incentives $400

Evaluator Travel $200 Total Misc. Administration $600

Total Evaluation Budget $4202.5

14

Appendix A

Preliminary Interview Questions for faculties Note: This protocol is represented in this proposal as an example of the type of tool we might use in the VMware Workstation evaluation project. 1. Which courses are you teaching in RIT? a. For which courses do you use the VMware system? b. For which courses do you use VMware to assign homework? c. For which courses do you use VMware for in-class activities? d. Is it necessary to use VMware for these classes or is the use of VMware optional? e. Are there any courses that you never use VMware? Please list some? 2. What types of assignments and lab activities do you use VMware for (e.g., creating virtual local networks, programming exercises, etc.): a. on campus? b. off campus? c. Is it necessary to use VMware system for these activities or is the use of virtual system optional? 3. Which operating system do you use in your courses? 4. How could you provide help when students work on VMware system?

a. on campus? b. off campus? … … …

[Continued…]

15

Appendix B

Student Experience Survey Note: This survey is represented in this proposal as an example of the type of tool we might use in the VMware Workstation evaluation project. Circle all choices that apply to you.

1. Participate description: a. Undergraduate student b. MS student c. PhD. student d. Female e. Male

2. VMware system experience: a. I do not know how to use b. Less than a year’s experience c. 1-3 years d. Over 3 years … …. 3. Student use of “Remote Laboratory Emulation System”

a. on campus for in-class activities b. on campus for advanced exploring c. on campus for final assignments d. off campus for final assignments e. off campus for personal interests. f. …

[Continued….]