project [email protected] course management system assessment methodology at wesleyan, brandeis, williams...

Project Project [email protected]@WBW.EDU

Course Management System Course Management System Assessment MethodologyAssessment Methodology

atat

Wesleyan, Brandeis, WilliamsWesleyan, Brandeis, Williams

NERCOMP 2003NERCOMP 2003

NERCOMP 2003NERCOMP 2003 22

Copyright InformationCopyright Information Copyright Wesleyan University, Brandeis Copyright Wesleyan University, Brandeis

University, and Williams College, 2003. This University, and Williams College, 2003. This work is the intellectual property of the authors. work is the intellectual property of the authors. Permission is granted for this material to be Permission is granted for this material to be shared for non-commercial, educational shared for non-commercial, educational purposes, provided that this copyright statementpurposes, provided that this copyright statementappears on the reproduced materials and notice appears on the reproduced materials and notice is given that the copying is by permission of the is given that the copying is by permission of the authors. To disseminate otherwise or to authors. To disseminate otherwise or to republish requires written permission from the republish requires written permission from the authors.authors.


Presenters for the TeamPresenters for the Team

Gayle Barton, Williams CollegeGayle Barton, Williams College [email protected]@williams.edu

Perry Hanson, Brandeis UniversityPerry Hanson, Brandeis University [email protected]@brandeis.edu

Mike Roy, Wesleyan UniversityMike Roy, Wesleyan University [email protected]@wesleyan.edu


CreditsCredits

Thanks to:Thanks to: The Team Members for their Hard WorkThe Team Members for their Hard Work The Eduworks Principals:The Eduworks Principals:

Geoff CollierGeoff Collier Robby RobsonRobby Robson

AT&T for providing us a Learning, AT&T for providing us a Learning, Network, Teaching and Technology grantNetwork, Teaching and Technology grant


WesleyanWesleyanJohn MeertsJohn Meerts

Emmanuel Paris-BouvretEmmanuel Paris-Bouvret

Mike RoyMike Roy

Dan SchnaidtDan Schnaidt

Raja SwamyRaja Swamy

Jolee WestJolee West

WilliamsWilliamsGayle BartonGayle Barton

Mika HiraiMika Hirai

Sharron MacklinSharron Macklin

Christine MChristine Méénardnard

Dinny TaylorDinny Taylor

Adam WangAdam Wang

BrandeisBrandeisPerry HansonPerry Hanson

Janet HillJanet Hill

Izabela KaminskaIzabela Kaminska

Tracey Leger-HornbyTracey Leger-Hornby

Scott RegensteinScott Regenstein

Debbie VerhoffDebbie Verhoff

Dave WedamanDave Wedaman

Ryan WilliamsRyan Williams

EduworksEduworksGeoff CollierGeoff Collier

Robby RobsonRobby Robson

The Project TeamThe Project Team


Project [email protected] [email protected] The ProblemThe Problem

How to evaluate CMSs?How to evaluate CMSs? What’s their purpose?What’s their purpose?

LiteratureLiterature Evaluation methodologyEvaluation methodology


WBW Context and MotivationWBW Context and Motivation Wesleyan Wesleyan

99% Residential99% Residential 2700 Undergraduate Students2700 Undergraduate Students 311 Full Time Faculty311 Full Time Faculty

BrandeisBrandeis 93% Residential93% Residential 3051 Undergraduate Students3051 Undergraduate Students 326 Full Time Faculty326 Full Time Faculty

WilliamsWilliams 99% Residential99% Residential 1936 Undergraduate Students1936 Undergraduate Students 263 Full Time Faculty263 Full Time Faculty


Research ApproachResearch Approach Utilized a broad institutional perspectiveUtilized a broad institutional perspective Gathered data from faculty, students, and staffGathered data from faculty, students, and staff

Nov ‘02 Dec ‘02 Jan ‘03 Feb ‘03

IT and Library staff interviews

Database audit of actual CMS usage data

Online survey (prep and execute)

Faculty and Student focus groups (prep and execute)

Project Timeline


Project DesignProject Design

Online surveyOnline survey

Focus groupsFocus groups

Usage data auditUsage data audit

e.g., what statement best characterizes your opinion about the benefits of using Web-based learning technology to improve the learning outcomes for your classes? Questionnaire – Faculty Questionnaire – Student

e.g., what do you see as existing or potential benefits from using the Web in classes? WBW FOCUS GROUPWBW FOCUS GROUP

e.g., how many classes utilized each major CMS feature?


Faculty ParticipationFaculty Participation

InstitutionInstitution Total Total Faculty Faculty FTEsFTEs

Online Online Survey Survey ParticipantsParticipants

Focus Focus Group Group ParticipantsParticipants

WesleyanWesleyan 311311 87 (28%)87 (28%) 1010

BrandeisBrandeis 326326 99 (30%)99 (30%) 2323

WilliamsWilliams 263263 155 (59%)155 (59%) 1515


Student ParticipationStudent ParticipationInstitutionInstitution Total Total

StudentsStudentsStudents Students Polled in Polled in Online Online SurveySurvey

ResponsesResponses Focus Group Focus Group ParticipantsParticipants

WesleyanWesleyan 27002700 00 N/AN/A 1111

BrandeisBrandeis 30513051 30513051 771771 66

WilliamsWilliams 19361936 500500 233233 77


Preliminary FindingsPreliminary Findings

CMS UsageCMS Usage Perceived benefits Perceived benefits UsabilityUsability Favorite FeaturesFavorite Features Participant opinionsParticipant opinions


CMS UsageCMS Usage (1 of 2)(1 of 2)

FACULTY QUESTION: Do you have a web site for any of the classes that you teach? Questionnaire - FacultyQuestionnaire - Faculty


CMS UsageCMS UsageSTUDENT QUESTION: Approximately how many of your classes over the last two years have made use of the following web learning technologies? Questionnaire - FacultyQuestionnaire - Faculty

(2 of 2)(2 of 2)

Blackboard (Williams) WebCT (Brandeis)

>53 - 51 - 20

Pe

rce

nt

80

60

40

20

0>53 - 51 - 20

Pe

rce

nt

80

60

40

20

0


Perceived BenefitsPerceived BenefitsFACULTY QUESTION: What statement best characterizes your opinion about the benefits of using Web-based learning technology to improve the learning outcomes for your classes?

(1 of 3)(1 of 3)

60

50

40

30

20

10

0

Integral to teaching, directly improves

outcomes

Students have better access; this enhances learning

I save time: more time left for students

No improve-

ment

No opinion


Perceived BenefitsPerceived BenefitsSTUDENT QUESTION: Do online course materials improve your learning?

(2 of 3)(2 of 3)

100

80

60

40

20

0

Helps me learn: access to info I

wouldn’t see

No effect Negative effect

Difficult to use


Perceived BenefitsPerceived BenefitsSTUDENT QUESTION: Which of the following statements most closely approximates your experience with the convenience and efficiency of accessing online course materials?

(3 of 3)(3 of 3)

100

80

60

40

20

0

Saved time: faster and easier

No effect Negative effect

Difficult to use


UsabilityUsability

According to Students:According to Students:

According to Faculty:According to Faculty:Depending on the CMS tool used, 30% to 75% of faculty Depending on the CMS tool used, 30% to 75% of faculty described CMS tools as very easy or usable with some described CMS tools as very easy or usable with some helphelp

More than 95% of students described their CMS tools as More than 95% of students described their CMS tools as very easy or usable with some helpvery easy or usable with some help


Favorite Features - BrandeisFavorite Features - Brandeis

Used more:Used more:Syllabus (74%)Syllabus (74%)Announcements(62%)Announcements(62%)On-line readings and text On-line readings and text materials (62%)materials (62%)

Used less:Used less:On-line exams (3%)On-line exams (3%)Online whiteboard (5%)Online whiteboard (5%)Real-time chat (12%)Real-time chat (12%)

Notable:Notable:Drop box: Only used by 18%, but a favorite for those usersDrop box: Only used by 18%, but a favorite for those usersOnline Face BookOnline Face Book


Favorite Features - WesleyanFavorite Features - Wesleyan

Used more:Used more:Syllabus (62%)Syllabus (62%)Announcements(59%)Announcements(59%)On-line readings and text On-line readings and text materials (51%)materials (51%)

Used less:Used less:Online whiteboard (0%)Online whiteboard (0%)Real-time chat (0%)Real-time chat (0%)Drop-box (12%)Drop-box (12%)

Notable:Notable:Calculated questions (on line quizzing)Calculated questions (on line quizzing)Discussion boardDiscussion board


Favorite Features - WilliamsFavorite Features - Williams

Used more:Used more:Announcements (81%)Announcements (81%)Course syllabus (77%)Course syllabus (77%)On-line readings and text On-line readings and text materials (63%)materials (63%)

Used less:Used less:Online whiteboard (0%)Online whiteboard (0%)Real time chat (2%)Real time chat (2%)Tracking student grades on Tracking student grades on assignments and tests (8%)assignments and tests (8%)

Notable:Notable:Online Face Book (51%)Online Face Book (51%)Sign-up sheets (12%)Sign-up sheets (12%)Wimba (<1%)Wimba (<1%)


Participant OpinionsParticipant Opinions

I tasked my class with evaluating a Web site on I tasked my class with evaluating a Web site on whales in Lake Michigan. A large percentage whales in Lake Michigan. A large percentage commented on the structure, graphical layout, commented on the structure, graphical layout,

navigational scheme, and overall design, but only a navigational scheme, and overall design, but only a small percentage recognized the site as bogus.small percentage recognized the site as bogus.

Faculty member

It’s “insulting” that faculty don’t think students can It’s “insulting” that faculty don’t think students can screen accurate information from fiction. screen accurate information from fiction.

Student

(1 of 3)(1 of 3)



I wrote a whole research paper without having to go I wrote a whole research paper without having to go to the library. This is a problem.to the library. This is a problem.

Student

I think it’s great that I can write research papers I think it’s great that I can write research papers without having to go to the library.without having to go to the library.

Student

(2 of 3)(2 of 3)



“…“…one of the top three uses of the web; it is an incredible one of the top three uses of the web; it is an incredible bonanza for this field . . . seeing stuff that I could never have bonanza for this field . . . seeing stuff that I could never have

seen before . . . provide links to seed them, start them off, and seen before . . . provide links to seed them, start them off, and the student then searches far beyond the initial seed…”the student then searches far beyond the initial seed…”

Faculty member

(3 of 3)(3 of 3)

Using a Wimba plug-in, I can post audio questions for my Using a Wimba plug-in, I can post audio questions for my Chinese class. The students post their individual responses, Chinese class. The students post their individual responses,

which lets me hear their individual pronunciation problems which lets me hear their individual pronunciation problems before class.before class.

Faculty member


ConclusionConclusion

Next stepsNext steps DiscussionDiscussion


The End The End


Presenters:Presenters:

Gayle Barton, Williams CollegeGayle Barton, Williams College [email protected] [email protected]

Perry Hanson, Brandeis UniversityPerry Hanson, Brandeis University [email protected] [email protected]

Mike Roy, Wesleyan UniversityMike Roy, Wesleyan University [email protected]@wesleyan.edu

For copies of this presentation, please go to:For copies of this presentation, please go to:

https://webserver.brandeis.edu/pages/view/ITS/CMSatWBWhttps://webserver.brandeis.edu/pages/view/ITS/CMSatWBW


LinksLinks

Questionnaire - FacultyQuestionnaire - Faculty Questionnaire – StudentQuestionnaire – Student WBW FOCUS GROUPWBW FOCUS GROUP

project [email protected] course management system assessment methodology at wesleyan, brandeis, williams...

Documents