Evaluate if virtualization and simulated labs in Computer Hardware Technology course meet their stated objectives

A Report Submitted to American National University

By

Shobhana Ganapathi

7/30/2013

2

Table of Contents

Executive Summary ………………………………………………………………….. 3

Description of the course Computer Hardware Technology with simulated labs ……. 4

Evaluation Method……………………………………………………………………. .5

Evaluation Methods & Tools…………………………………………………….....5, 6

Results……………………………………………………………………………. 7, 8, 9

Assessment Performance…………………………………………………………… .7

Learner Completion Rate and Average Final Performance Scores……………….... .8

Learner perceptions and Learner satisfaction of Computer Hardware Technology

Course with Labsims………………………………………………………………… 9

SME Feedback……………………………………………………………………… .9

Discussion……………………………………………………………………….... .10, 11

Project Cost………………………………………………………………………….. .12

APPENDIX

Appendix A: EDP……………………………………………………………….13 to 15

Appendix B: Learner Perception and Satisfaction Questionnaire……………… 16 to 19

Appendix C: SME Questionnaire………………………………………………… .20, 21

Appendix D: Student interview & Evaluator Observations Report………………… 22

3

Executive Summary

This report details the results of evaluating ‘if virtualization and simulated labs meet their

stated objectives in the Computer Hardware Technology Course’. Data was collected and

analyzed to find the effectiveness and efficiency of the Computer Hardware Technology course

with the simulated labs and it was compared to the performance in face-to-face labs. The main

objective of the evaluation was to determine if replacing f-t-f labs with simulations

compromised student learning. The evaluation results are being provided to the program

directors of American National University (ANU) to enable them to make a decision to migrate

the course from a face-to-face setting to a fully online platform.

The individual components that were evaluated were the content and the instructional

support materials in the simulated labs. The technology components of graphics, video and

audio and all the exercises, discussion activities and assessments were examined in both

versions of the course. Quantitative data was collected from both versions of course offerings

for performance in tests, projects and assessments and qualitative data was collected from

questionnaires and interviews for learner perceptions, satisfaction and ease of use of technology

components.

An analysis of the results indicate that there is no significant difference in performance

scores in the Computer Hardware Technology course with simulated labs vs. the version with

face-to-face labs. Student perceptions and attitudes indicate they prefer to work with online

simulations as it gives them the flexibility of working from anywhere, anytime convenient and

more importantly the ability to repeat the labs any number of times was cited as a plus point in

its favor. (Appendix D for excerpt of student interview)

Computer Hardware Technology course with simulated labs meets all of its stated

objectives and is recommended as a viable option to the course with face-to-face labs. The

stakeholders can rest assured that student learning is not compromised and they can move

forward in migrating the course online.

4

Description of the course Computer Hardware Technology with

simulated labs

CHT 102- Computer Hardware technology is an Associates level computer science

course that is required of all Information Systems Engineering students at ANU. Enrolments are

typically between 10 to 15 students. The students who are enrolling for CHT should have

successfully completed the pre-requisite of ISE 100- Introduction to Computers course. The

course is broken into 5 modules. Each module has two distinct parts theory and practicals done

in labsims. Module 1 to Module 4 is of 11 hours duration and module 5 is of 6 hours. The total

duration of the course including the introduction and conclusion is about 60 hours.

Each module comprises of some instructional support materials and simulated labs for

practice. The student work in each module involves completing exercises, discussion activities

and assessments. The students navigate through graphics, video and audio clips to complete the

activities in each module.

The main objectives of this evaluation were: (Appendix A for details)

To check if Goal statement is clear

To check Goal & objective alignment

Is the content in the virtual lab presented in a clear and concise manner?

Is the content appropriate for Computer Hardware Technology?

Is the content up to date covering current hardware technologies?

Do the technology applications like embedded videos, graphics and lab simulations function

properly?

Is organization and structure of the message logical and coherent?

5

Evaluation Method

The evaluation was done from two orientations, participant orientation and objective

orientation. It was deemed important to evaluate the course with integrated labsims from a dual-

perspective as both the participants and the objectives were critical to the success of this course.

The participants were students enrolled in the Computer Hardware Tehnology course

who have successfully completed the pre-requisite of ISE100- introduction to computers. Other

participants were the SME (HOD computer science department), the instructor and the

evaluator.

The decision making methodology was used for this evaluation as the evaluation results

were going to directly impact the decision being made by program directors to migrate the face-

to-face course to online instruction.

The results of the evaluation are being presented in two parts. The reports detail both

short term and long term outcomes. The current report focuses on short term outcomes like

student learning, knowledge, attitude, performance, skill and understanding.

Long term outcomes will be examined to see if the online course produces a ROI by

increasing enrollments. Since it is beyond the scope of the current evaluation report this will be

presented in another report after collecting and analyzing data from four course offerings of labs

with simulations.

Evaluation Methods & Tools

Both qualitative and quantitative methods were used to collect data

• Qualitative: Valuable information was collected from interviews, observations,

surveys and responses provided via e-mail, messages and forum posts to explore the

context of participants' experiences of the course using simulated labs. (Appendix B)

6

• Quantitative: Data was collected through user survey's and questionnaires and

assessment data was collected from performance in lab sims, tests, quizzes and

project weight age mentioned in project rubrics. (Appendix B & C)

Specifically, the average performance scores from four course offerings were obtained, two from

the course with face-to-face labs and two from the course with simulated labs. This was

compared to find the difference in completion rates and average performances. Student interview

(Appendix D), SME feedback via questionnaire (Appendix C) & email and student questionnaire

(Appendix B) results were used to report learner satisfaction.

7

Results

Assessment Performance

All the scores from past four course offerings of tests quizzes and project work

based on assessment rubrics were obtained and tabulated to arrive at the average scores

and median value. It was determined that average performance scores were about the

same, in the 80% to 85% range in both versions of the course offerings as shown in the

table and graphs below.

It was determined that student completion rate was 100% in Term1 (Fall 2012

with face-t-face labs) and 80% in Term2 (Winter 2012 with face-t-face labs) and 84% in

Term3 (Spring 2013 with simulated labs) and 91.6% in Term 4 (Summer 2013 with

simulated labs). The face–to-face course offering had an average completion rating of

90% and the version with simulated labs had an average completion rate of 89.55%

Learner Completion Rate and Average Final Performance Scores

Course Offering

Number of Learners

Number who completed the course

Completion Rate

Average Final Scores

Lowest Score (including Fail)

Highest Score

Median Value of scores

Term 1 ( f-t-f labs)

8 8 100% 85.6% 61 98 91.5

Term 2 ( f-t-f labs)

10 8 80% 79.12% 74 100 74.5

Term 3 (Simulated labs)

8 7 87.5% 84% 57 97 93

Term 4 (Simulated labs)

12 11 91.6% 82.54% 47 98 85

8

8 100%

10

80% 8 87.5%

12 91.6%

0

2

4

6

8

10

12

14

No. of Students No. Who Completed the course

Learner completion Rate

Term 1 (f-t-f labs) Term 2 (f-t-f labs) Term 3 (simulated labs) Term 4 (Simulted labs)

85.60%

79.12% 84.00% 82.54%

50.00%

55.00%

60.00%

65.00%

70.00%

75.00%

80.00%

85.00%

90.00%

95.00%

100.00%

Average scores

Final Performance Scores

Term 1 (f-t-f labs) Term 2 (f-t-f labs) Term 3 (simulated labs) Term 4 (Simulted labs)

9

Learner Perceptions and Learner Satisfaction of Computer Hardware

Technology Course with Labsims

The survey was taken by 9 out of the 12 students enrolled for the summer course. 6 participants out of 9 agreed they liked working with simulated labs 2 were neutral and only 1 disagreed. 6 participants were happy with the clarity of the content, 2 were neutral and 1 disagreed.7 of nine participants agreed the content covered latest technologies and embedded graphics and simulations worked properly and 2 were neutral in their views. There was no disagreement in this regard.

QUESTIONNAIRE ITEM

STRONGLY AGREE N%

AGREE N%

NEUTRAL N%

DISAGREE N%

STRONGLY DISAGREE N%

I liked working with simulated labs in the course

1 (11.11%) 5 (55.56%) 2 (22.22%) 1 (11.11%) 0 (0%)

The content in the virtual lab was presented in a clear and concise manner

2 (22.22%) 4 (44.44%) 2 (22.22%) 0 (0%) 1 (11.11%)

The embedded videos graphics and simulations functioned properly

3 (33.33%) 4 (44.44%) 0 (0%) 2 (22.22%) 0 (0%)

The content was up to date and covered latest technologies

4 (44.44%) 3 (33.33%) 2 (22.22%) 0 (0%) 0 (0%)

N= 9 learners

SME Feedback

A questionnaire was given to the SME to give his opinion and feedback about the CHT course. The questions covered the goal, content, technology and message design to evaluate efficiency, effectiveness and appeal of the content and simulated labs in the course. (Appendix C)

The SME feedback via questionnaire and email indicated that the content was appropriate and aligned to the objectives of the Computer Hardware technology Course.

10

Discussion

Overall scores, averages and median values were used to investigate how (and if) scores

improved after using the simulated labs. The demographics of students enrolled in both

versions of the course were the same. They were students who had passed the pre-

requisite of introduction to computers course. The only variable was that students

enrolled in Term 1 and Term 2 did the labs in a face-to-face setting and the students

enrolled in Term 3 and Term 4 did the labs in an online simulated setting.

Learner completion rate and average performance scores in the course offering with face-

to-face labs were 90% and 82% respectively.

Learner completion rate and average performance scores in the course offering with

simulated labs were 85.5% and 83% respectively.

Since the scores are about the same it is determined that student learning is not

compromised while using simulated labs.

The analysis of Learner satisfaction data compiled from the questionnaire ( Appendix B)

shown in the table under ‘results’ as frequency and percentages indicate 66% of the

students indicated they liked working with the simulations and were happy with the

clarity of the content in the virtual labs. 22% were neutral in their views and only 11%

disagreed with this. 78% of the students were happy with the quality of the simulations

and alignment of content to latest technologies.

Since the survey was taken only by a small number of students student perceptions were

also gathered by interviewing two students who had worked with simulations in a

previous course offering. The SME feedback was collected via questionnaire (Appendix

C) and feedback via email.

11

Students had mixed perceptions about watching the video and learning before working

with simulated labs. See excerpts of interview below. (Details of interview available in

Appendix D)

Student 1: “I found the material more challenging to comprehend when I tried to read the textbook. But when I watched the video and practiced the lab I understood it better and did not have to read the chapter a second time.” Student 2: “Sometimes watching the video becomes tedious especially with these boring guys in the video”

But they seemed to be unanimous in their preference for working online with virtual labs. An excerpt from their conversation is given below.

Student 1: “I would prefer to do it online because I can work from home when I get time.” Student 2: “I prefer online labs because I can repeat it many times till I feel I am ready. This will help me prepare for my A+ certification”

After the analysis of survey results, student interview, SME questionnaire and feedback

via email it is determined that a majority of students were satisfied with using the

simulated labs (78%) and would prefer this mode of instruction (100%).

12

Project Cost

The following is the cost of the evaluation including travel, materials, and SME and PA fees.

Professional Salaries

Main Evaluator (PM) 15days @$500/day $7500

SME 5 modules @ $300/module $1500

Personal Assistant (PA)

10 days @$120/day $1200

$10,200

Travel to ANU

Estimated mileage 500 miles @ $0.30/mile $150

5 site visits @ estimated average of $150 travel & per diem per visit

$750

Supplies, Materials, Communication $1200 $2100

Total

$12,300

13

APPENDIX

Appendix A: EDP

Evaluation Questions

Program Objectives

Project Activities to Observe

Data Source Population/Sample

Data Collection Design

Data Analysis

Whose Responsibility

Goals: Is the

purpose stated

concisely?

To check if Goal statement is clear

Request SME to read and compare with checklists of F-T-F course materials

Reference checklists of F-T-F course materials

HOD Computer science Department

Prepared checklists of Objectives

Compare the two checklists

Evaluator

Do goal and lesson objectives align with each other?

To check Goal & objective alignment

Students read the goal and all the objectives

End-user survey

All students enrolled in Computer Hardware Technology

Administer end-user survey through webmonkey

Plot graphs

evaluator

Content: Is the content in the virtual lab presented in a clear and concise manner?

Subject Matter Expert (HOD Computer Science) browses the virtual lab with checklists of F-T-F course

Feedback via email & messages

SME (HOD) Prepare Checklist of course materials

Compare the two checklists

evaluator

14

materials ,

Is the content appropriate for Computer Hardware Technology?

HOD & Instructor browse the virtual labs and complete simulated lab activity

Interview report from SME (HOD Computer science) & instructor. Assessment data will be collected through lab sim and online quiz activity in the portal and discrepancies in check list of project rubrics.

SME & Instructor Report Look for negative remarks

evaluator

Is the content up to date covering current hardware technologies?

HOD & Instructor browse the virtual labs and complete simulated lab activity

Interview report from SME (HOD Computer science) & instructor

SME & Instructor Report Look for negative remarks

evaluator

Technology

Do the technology applications like embedded videos, graphics and lab simulations

Expert & end-user browse and review the online course. Data is

Expert review report & End-user survey

Expert & All students enrolled in Computer Hardware Technology

Administer end-user survey through webmonkey

Plot graphs

evaluator

15

function properly?

collected from portal to find time spent on each task.

Message Design

Is organization and structure of the message logical and coherent?

Expert & end-user browse and review the online course

Expert review report & End-user survey

Expert & All students enrolled in Computer Hardware Technology

Administer end-user survey through webmonkey

Plot graphs

evaluator

16

Appendix B: Learner Perception and Satisfaction Questionnaire

Student Opinions about Computer Course with Simulated Labs:

Your response to this questionnaire will be helpful and is greatly appreciated. Your input will

help us modify the course as necessary. Thank you.

1. Please fill out the following demographic data: Gender

Please fill out the following demographic data: Gender Male

Female

2. What is your age?

What is your age? 18 to 24

25 to 34

35 to 44

45 to 54

55 to 64

65 to 74

75 or older

3. Please select the option that best matches your response: I have a great deal of computer experience.

Please select the option that best matches your response: I have a great deal of computer experience.

Strongly Agree

17

Agree

Neutral

Disagree

Strongly Disagree

4. I liked working with simulated labs in this course.

I liked working with simulated labs in this course. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

5. The content in the virtual lab was presented in a clear and concise manner.

The content in the virtual lab was presented in a clear and concise manner. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

6. The content was up to date and covered latest technologies

The content was up to date and covered latest technologies Strongly Agree

18

Agree

Neutral

Disagree

Strongly Disagree

7. The embedded videos, graphics and simulations functioned properly.

The embedded videos, graphics and simulations functioned properly. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

8. The organization and structure of the message was logical and coherent.

The organization and structure of the message was logical and coherent. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

9. I think I learned as much from the simulations as I would have done in the physical labs.

I think I learned as much from the simulations as I would have done in the physical labs. Strongly Agree

19

Agree

Neutral

Disagree

Strongly Disagree

10. It prepared me well to take the A+ Certification

It prepared me well to take the A+ Certification Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

Next

Powered by SurveyMonkey

20

Appendix C: SME Questionnaire

Evaluation Criteria

Specific Evaluation Questions to Evaluate Computer Hardware Technology (CHT) with simulated labs

Effectiveness

Goals Is the information accurate?

Are the goals and objectives clear?

Are the goals and objectives achievable?

Are the goals and objectives appropriate for classroom instruction?

Content Does the information cover the content properly?

Does the content match the objectives?

Are activities and final project congruent?

Do activities promote learning?

Technology Are the videos and simulations working properly?

Are the simulations appropriate to learn computer hardware concepts?

Do they sufficiently meet learning objectives?

Message Design Are messages an integrated whole?

Do videos and simulations provide sufficient instruction leading to successful lab completion?

Do learners understand the instruction?

Are there design features which are distracting?

Are directions clear?

Is time allotted for the course appropriate?

Does the text stand alone if the graphics are unavailable?

21

Efficiency Is the purpose stated concisely?

Goals Do goal and lesson objectives align with each other?

Is the content presented in a clear and concise manner?

Content Is the content appropriate for Computer Hardware Technology?

Is the content up to date covering current hardware technologies and collaboration efforts?

Do the technology applications like embedded videos and links work properly?

Technology Does the firewall prevent students from accessing links in their search results?

Is organization and structure of the message logical and coherent?

Message Design Are titles and subtitles used while organizing the content?

Appeal

Goals Are the goals relevant to learners?

Content Is the content interesting?

Is the content in the virtual labs presented in a way that can be understood by students who have just completed only ISE 100?

Is the content challenging at the same time being enjoyable?

Technology Are instructions logical and easy?

Is it easy for students to access materials?

Are the videos presented well?

How about the audio?

Is the language level and tone suitable for adult learners?

Message Design Do graphics and video enhance the instruction?

General Comments

Based on content and quality do they qualify to replace face to face labs? Please provide your comments on this

22

Appendix D: Student interview & Evaluator Observations Report

Excerpts of Interview Conducted by Evaluator Comments Evaluator: Did you like watching the videos before working with the simulations?

Students have mixed perceptions about watching the video and learning

Student 1: I found the material more challenging to comprehend when I tried to read the textbook. But when I watched the video and practiced the lab I understood it better and did not have to read the chapter a second time. Student 2: Sometimes watching the video becomes tedious especially with these boring guys in the video Evaluator: Do you like to take tests and exams in the virtual labs? How does it compare to tests in face-to-face setting?

Students generally seem to be inclined towards taking tests which demonstrates their practical ability to do something rather than paper and pencil multiple choice tests. Student 1: After our virtual lab tests we were

able to view our scores immediately. This proved to us that we knew something. After our multiple choice test we were just glad that it was over! Student 2: I like the virtual labs because the exam tests us on what we can do and not what we can memorize. Evaluator: If you were given the choice to come to school to do your labs, would you prefer that to doing labs online?

Students seem to be unanimous in their preference for working online with virtual labs

Student 1: I would prefer to do it online because I can work from home when I get time. Student 2: I prefer online labs because I can repeat it many times till I feel I am ready. This will help me prepare for my A+ certification