Volume 2002, Issue 8

April 16, 2002

Selecting Course Management Software to

Meet the Requirements of Faculty and Students

Margaret E. Halloran, U.S. Air Force Academy*

4772 Walnut Street, Suite 206 ◆ Boulder, Colorado 80301-2538 ◆ www.educause.edu/ecar/

Research Bulletin

Center for Applied Research EDUCAUSE

Overview Colleges and universities have accepted the Web and its technologies as valuable tools, both to supplement traditional classroom instruction (delivering readings and course content, expanding the curriculum outside the classroom walls, engaging students in interactive learning environments) and to provide distance learning. Web-enabling courses, however, may require faculty and students to acquire new technical skills, such as the ability to write HTML. And converting a course to a Web-enabled platform can be very labor intensive.

One way to circumvent these drawbacks is to use a course management system (CMS), which uses templates specifically designed for Web-enabled instruction. Course management systems incorporate many different development tools into a seamless interface for Web-based course content. By uploading course handouts, images, and files created using other software, such as word processors or spreadsheets, a faculty member can develop a Web presence without knowledge of HTML or other programming languages.

Course management software enables faculty members to learn one software package for Web-based curriculum, assessment, synchronous and asynchronous discussions, collaborative work, multimedia, and interactive resource development. Tracking features enable instructors to examine the use of these online resources, which can lead to inferences about the interest and difficulty of the content or about an individual student’s level of effort. Deploying a CMS at the institutional level also offers students a consistent, virtual, interactive environment, which promotes learning and decreases the number of systems a student has to master to accomplish routine tasks, such as finding homework assignments or an instructor’s office hours.

While it is possible to author a custom CMS solution, the development costs of an enterprise-level database-driven solution are many times more than what commercial, off-the-shelf products typically charge for license agreements. In addition, many textbook publishing companies are now forming alliances with companies that produce CMS software, making these supplemental resources available to faculty and students who buy their textbooks.

Once an institution decides to use a CMS, how does it determine which one? There are as many as 109 different course management software packages on the market. Several studies that have evaluated and compared various aspects of these tools focused mainly on checklists of what these products can do, not what these products need to do, or can do well.1–4 For example, packages have been evaluated based on whether they have an asynchronous discussion feature, not whether the discussion group feature is easy to use or useful to faculty. Even studies that evaluated ease of use of the packages only addressed the development of resources, not the use of the resulting resource.

When focusing on the requirements of users, there are two main groups to consider: students and faculty. Their perspectives may be very different. Faculty need to be able

2

to develop resources quickly and efficiently using this software, whereas students need to be able to find and access the course materials their instructors have assigned. Although this system must be an easy developmental tool for the faculty, it also must provide a pedagogically sound and intuitive navigational structure for students. Therefore, these feature/function analyses may be of limited predictive power in determining the applicability of different course management systems for deployment by educational institutions.

Institutions can use other types of criteria in selecting a CMS. For example, institutions should be interested in a system that allows for flexibility in the type of content that can be uploaded by faculty. The selection process may involve evaluation of the vendor’s long-term viability and of the costs to the institution. Scalability of the solution also can be an important issue. A product that works well for 50 students may not be sufficient for enrollments of 5,000. And, to ensure long-term usefulness and interaction, more and more institutions are interested in products that adhere to standards, such as the Instructional Management System (IMS) standard, or that allow for reusable learning objects.

The objective of this Research Bulletin is to outline a methodology that institutions can use to evaluate the usability and usefulness of course management software from both the faculty and student perspectives. Although this study examines CMS products used in traditional classroom education, this process could be tailored for software used to support distance learning programs. The process involves faculty rating different software packages, not only on how easy they are to use but on how useful the features are to classroom instruction. Students evaluate the resulting Web-enabled courses for ease of navigation and for finding resources, as well as the usefulness of different product features.

Since there is a familiarity component to the users’ satisfaction with a product, these results may vary depending on the level of user experience. Therefore, feedback of both faculty and students who have used the software for an entire semester may provide a more complete analysis than data collected only from novice users.

Highlights of the Selection Process This methodology was developed based on a series of experiments conducted at the U.S. Air Force Academy in Colorado Springs, Colorado. The experiments, involving faculty and students, measured a number of different attributes of and reactions to CMS software used in resident courses. Through this process we were able to discern how to complete a thorough evaluation of CMS products based on the reactions of faculty and students, as well as novice and experienced users. Institutions selecting CMS vendors are making long-term and costly investments. A thorough methodology will enable institutions to make better decisions.

Although the sequence may vary somewhat, the following steps are important:

1. List the features and functions of potential packages.

3

2. Engage both faculty and students in the review.

3. Prioritize the importance of features and functions, and evaluate ease-of-use.

4. Pilot the product(s) with experienced users (faculty and students), not just novices.

5. Consider other criteria that may be important, such as vendor viability or the ability to integrate with legacy or existing software systems.

Step 1. List Features and Functions Determining the list of features and functions associated with various CMS packages can be based on previous studies, product reviews, or vendor literature. An initial step is to generate a checklist of what products can and cannot do. Table 1 represents a subset of such a list compiled for six CMS products. Note that a checklist does not provide any indication of whether or not the features are well constructed, easy to use, or valuable to users.

Table 1: Examples of Features Included in Six Commercially Available CMS Software Products

Product A B C D E F

General Features

Table of contents X X X X X

Customized “look and feel” (colors, icons, logos) X X X X

Automated indexing tool X X X

Integrated calendar tool X X X

Multi-language support X X X X X

Conformity with IMS specifications X X X X

Authoring Features

Does not require knowledge of HTML X X X X X

Allows Java applets X X

Has drag-and-drop authoring features X

4

Product A B C D E F

Course Management Features

Class list can be uploaded as a file X X X X X X

Includes an online student manual X X X X X X

Course can be downloaded for safekeeping X

Offers student tracking features X X X X

Communication Features

One-to-one course e-mail X X X X X X

One-to-many course e-mail X X X X X

Shared whiteboard X X X

Grading Features

Student access to progress data X X X X X X

Ability to add offline grades X X X X X

Grade statistics and/or histograms X X

Quizzing Features

Quizzes automatically graded and entered in grade book X X X X * Questions can have multiple correct answers X X X X

Can use a mix of question types in one exam X X X X

Can be Used with These Operating Systems

Unix X X X X X

Windows NT X X X X X X

Macintosh X X

* offered by a third party vendor

Step 2. Engage Both Faculty and Students in the Review There are two major groups of users of CMS products: faculty and students. Although the first impression may be that faculty or system administrators are more knowledgeable about the teaching and learning process or software systems and should therefore guide the selection of CMS tools, this assumption may prove false. Our

5

investigation showed that faculty and students did not always find the same features important. Because student achievement and satisfaction may be intertwined with the CMS tool chosen, we believe institutions should involve both faculty and students in any evaluation of CMS products.

Many people assume that students are more facile with technology than their instructors. In our study at the U.S. Air Force Academy, we found that faculty members had significantly more experience using and programming computers than did students. Students, however, spend significantly more time using the Web each week than do faculty.

Faculty and students had different requirements of the CMS product for developing and accessing resources. When faculty and students were asked to rank the features used to evaluate the course management software packages, they disagreed on which features were more and less important (see Table 2). For example, features used primarily by students, including finding grades or the ability to navigate to a certain location within the site, were not rated by faculty as more important than average, whereas students rated these features as some of the most important. Similarly, student ratings of features designed for faculty, such as tracking tools to determine if students had visited the site, were among the lowest.

These differing perceptions of importance have implications for how a CMS is selected. If only the faculty’s requirements are considered, or if the product is not evaluated by all of the user groups, the probability increases of purchasing a suboptimal product for a majority of the users.

Table 2: Faculty and Student Evaluations of CMS Features

Faculty Students

Does not require knowledge of HTML Ability to add offline grades

Classes can be uploaded as a file True/false, multiple choice, and matching quiz question types

Course can be downloaded for safekeeping Table of contents, searching features

Quizzes automatically graded Online quizzing features

High Importance

Can use mix of question types in one exam Other grading features

Can support foreign languages Student tracking features

Asynchronous and synchronous discussion Collaborative work areas

Shared whiteboards Asynchronous and synchronous discussion

Virtual field trips Virtual field trips

Low Importance

Operating system and industry standards Operating system and industry standards

6

Step 3. Consider the Importance of Features as Well as Ease of Use

Although the feature lists for many CMS products are extensive (discussion groups, student Web pages, collaborative work areas), long lists do not guarantee that those features will be widely used or deemed important by faculty or students. A CMS with an easy-to-use interface that contains a grade book, automated quizzes, and a place to put announcements and course documents will likely be preferred to one that contains many collaboration features. As faculty become more comfortable and familiar with software and pedagogy, they may start to use collaborative features more frequently, resulting in increased student use of those features. However, institutions should expect new users to rate features such as uploading lessons and grade books as the most important.

Time is one variable often cited by faculty as a reason for not adopting technology tools. As a result, evaluating the needed time-on-task may be an important determinant of which CMS package is right for your institution. In our study, for example, it took faculty an average of 1 hour and 31 minutes to create a Web-enabled test course using one of the products being reviewed and an average of 2 hours and 22 minutes to create a course identical in content using two alternative products. Measures of students’ time to complete tasks with different CMS products also vary. We found that students took significantly less time to complete quizzes and to find the syllabus and supporting documents for the course built with one of the CMS products under review compared to the others.

Ease of use of a CMS is another important variable. Side-by-side comparisons of products can assess their relative ease of use. One product was rated more useful than the others for 17 of 18 tasks, even though the tasks were identical. These tasks included creating announcements, composing quizzes, adding assignments, using collaboration features, and performing administrative tasks. The faculty scores seemed to depend on whether the tasks were easy to accomplish, giving the advantage to the product with the most intuitive interface. We believe the faculty perceptions of usefulness of particular CMS features are tied to ease-of-use. Interestingly, faculty did not use the online manual or ask for help when using any of the three products we evaluated in depth, even when they found the task difficult to accomplish.

Students had difficulty finding resources on Web pages created by the CMS software because course documents often were not located where students expected them to be. Students sometimes had to search under different headings to find all the handouts and assignments for a single day. Like faculty, students typically did not ask for help or use the online manuals. Unlike faculty, however, students did not tie the usefulness of the features to their ease of use. One explanation for this is that many Web sites have counterintuitive navigational layouts, and “hunt and click” navigation is used by those who browse the Web frequently.5 Because students are frequent Web browsers, they may be habituated to this type of navigation.

CMS packages can appear very similar based on a review of a checklist of features and functions. However, an evaluation of users’ overall experience using products can show

7

significant differences. In our study faculty rated one product significantly higher than the others for 10 statements (see Table 3). The product was perceived to have a significantly better navigational layout, be more intuitive in nature, and require little training to use, as well as having pedagogically useful features. Although the students also preferred the same CMS package overall, their decision was based more on overall look and feel rather than product features (see Table 4).

Table 3: Comparison of Mean Overall Assessment Scores by Faculty (n = 12) for the Preferred (Package A) and Other CMS Products*

Statement Pkg A Other p

Took a reasonable amount of time 76.4 40.0 0.034

Had a good navigational layout 80.0 24.1 0.001

Had a good pedagogical layout 73.5 24.0 0.001

Had aesthetically pleasing screen displays 75.0 48.6 0.015

Was easy to use 83.6 9.6 0.001

Had an easy-to-use manual 66.0 18.5 0.001

Had instructive directions on the screen 67.7 23.2 0.001

Had pedagogically useful features 80.0 40.0 0.002

Was intuitive in nature 83.6 12.7 0.001

Needs little training to use 79.6 9.1 0.001

* Anchors for this scale are 0 = decidedly disagree, 25 = moderately disagree, 50 = neutral, 75 = moderately agree, 100 = decidedly agree. A p-value less than 0.05 indicates a significant difference between the packages.

8

Table 4: Comparison of Mean Overall Assessment Scores by Students (n = 29) for the Preferred (Package A) and Other (Packages B and C) CMS Products*

Statement Pkg A Pkg B p Pkg A Pkg C p

Took a reasonable amount of time 98.1 71.5 0.009 84.0 72.5 0.288

Had a good navigational layout 90.6 45.6 0.001 90.0 73.0 0.144

Had a logical layout 92.5 55.4 0.002 87.0 77.0 0.358

Had aesthetically pleasing screen displays 87.5 72.5 0.088 88.0 61.0 0.048

Was easy to use 89.9 55.9 0.007 86.0 69.0 0.136

Had an easy-to-use manual 73.8 48.8 0.155 91.0 62.0 0.118

Had instructive directions on the screen 70.0 39.3 0.058 60.0 43.1 0.105

Had useful features 95.6 78.1 0.068 89.8 74.0 0.151

Was intuitive in nature 93.8 68.5 0.009 82.5 71.5 0.385

Needs little training to use 83.8 62.5 0.141 81.0 68.0 0.311

* Anchors for this scale are 0 = decidedly disagree, 25 = moderately disagree, 50 = neutral, 75 = moderately agree, 100 = decidedly agree. A p-value less than 0.05 indicates a significant difference between the packages.

When evaluating the usefulness of products, remember to include assessments based on things such as multi-section courses. For example, features such as the ability to divide grade books, announcements, or documents into multiple sections of the same course without duplicating the entire course are important for multi-section courses but not included in many CMS products. It may not be possible to share calendars, quiz questions, handouts, or content among instructors teaching the same course. Therefore, CMS software may be best suited for a faculty member teaching one section of a single course who does not share large amounts of content with instructors teaching other sections.

Step 4. Involve Experienced as Well as Novice Users Conducting a pilot of the “short list” of CMS products prior to making a final decision is an important part of selection. Novice users may change their priorities and preferences based on experience. Our research showed that requirements gathered from faculty and students who had been using a CMS for an entire semester were more similar than those gathered from similar groups of naïve users.

When novice participants were asked to determine what features were important and should be included in software to facilitate the development and use of Web-enabled courses, both faculty and students focused on their own use of the software and did not appear to consider the requirements of other user groups. Similarly, when participants

9

who had been using the preferred CMS product for one semester were asked to determine which features were important and should be included in software, both faculty and students still tended to focus on their own requirements. Students focused on the types of quiz questions that would appear on exams and the ability to check their scores online. Faculty continued to focus on administrative procedures and quiz construction.

However, there was more agreement between the two experienced user groups than between the naïve user groups. Both groups felt that student access to progress data, automatically graded quizzes, different question types on a single quiz, tables of contents, and the ability to create pages without knowledge of HTML were more important than other features required of a CMS product.

Step 5. Other Selection Criteria Implementing a CMS may cost a campus anywhere from a few thousand to millions of dollars, depending on the size of the institution, software chosen, and associated services. Not only can initial costs be significant, but costs for changing CMS platforms may be even higher. As a result, institutions must consider short- and long-term implications:

Ability of the CMS to integrate with existing applications and future systems

Skill base of existing staff, and availability and amount of training resulting from faculty and staff turnover and new software versions

Cost of maintenance, including recurring license fees and manpower costs

Adherence to standards that may allow for reusability of content

Scalability of the solution

Hardware and supporting software packages needed to run the system

What It Means to Higher Education Many campuses are currently facing a CMS decision or will be soon. Although there has been significant attention paid to the emerging CMS leaders, many institutions either use a homegrown package or allow faculty to select their own. A 2001 Eduventures survey found that 38 percent of campuses use multiple CMS products.6 However, more and more institutions are feeling pressure to standardize on a single CMS, in large part due to support issues and the need to create a transparent environment for students.

How institutions select a CMS can be very important. Eduventures found that the factors having the most significant influence on selecting a CMS were ease of use (64 percent), flexibility (47 percent), and price (46 percent).7 However, making selections on the basis of these three factors may ignore other important considerations.

Many CMS decisions are based on functional specifications. We found that of the six packages evaluated, two were almost identical with respect to the number and types of

10

features they offered. When the product evaluation focused on qualities such as ease of use or usefulness, a clear distinction appeared. In addition, many of the features, such as discussion groups, student Web pages, and collaborative work areas, were not widely used or deemed important by either faculty or students. Although some faculty may eventually use these features as they become more familiar with the software and pedagogy, a CMS with an easy-to-use interface that contains a grade book, automated quizzes, and a place to put announcements and course documents provides the greatest utility to the largest number of users. Thus, basing a decision on features may not suffice.

A CMS product will be used by both faculty and students. As a result, it is important to understand the requirements of both audiences. Our research verified that faculty and students have different requirements. If only the faculty requirements are considered, the probability of purchasing a suboptimal product for the majority of users increases. The requirements of faculty and students also vary based on whether the individual is a novice or has experience with the product. Therefore, it is useful to consult published accounts of experienced users and/or collaborate with similar institutions already using a CMS.

Allowing faculty and students to use the proposed product(s) is another essential step. Ease of use is an important consideration, particularly for institutions interested in expanding CMS use beyond the early adopters to mainstream faculty. (At many institutions where a CMS has been deployed, fewer than 50 faculty use the system,8 indicating a relatively modest adoption rate.)

Selection of a CMS is only the first step. Deployment can be much more challenging. While much of the process of product selection is based on features and user needs, it is also important to consider specific characteristics associated with the vendor. There has been significant flux in the online learning market in the past 18 months, with continuing predictions of a “shake out.” As a result, institutions should evaluate the long-term viability of a particular vendor, as well as that vendor’s reliability and the quality of the service and products provided. The successful deployment of a CMS will also depend on factors such as how well it fits with existing software and other institutional requirements.

The use of CMS software can be a significant investment that may have a large impact on the institution. If the implementation of a CMS is successful, instructors will become dependent on it and may tailor the development of the curriculum to take advantage of its features. Therefore, the decision to implement a CMS and the selection of a particular product should be considered carefully, based on the following questions:

Have we identified criteria upon which to base our decision?

Have we taken into account the needs of students and faculty?

Have we considered the opinions of experienced as well as novice users?

Do our selection criteria go beyond technical merits and include considerations such as vendor reliability and stability?

11

Do we have a plan in place that will allow us to transition if the company goes out of business or changes the pricing structure?

Where to Learn More To access the EDUCAUSE archive on course management systems, see

<http://www.educause.edu/asp/doclib/subject_docs.asp?Term_ID=493>.

Course Management Systems: Today and Tomorrow. CREN audio archive. <http://www.cren.net/know/techtalk/events/cms.html>.

To access the Institute for Information Technology Applications technical report on usability of course management software, see <http://www.usafa.af.mil/iita/Publications.htm>.

Endnotes 1. S. Gray, “Web-based Instructional Tools,” Syllabus Magazine, 12 (2) (1998).

2. S. I. Hazari, “Evaluation and Selection of Web Course Management Tools” (1998), retrieved January 2001 from the World Wide Web <http://sunil.umd.edu/webct/>.

3. InfoWorld, “The Virtual Classroom: Web-based Training Solutions” (November 1998).

4. B. Landon, “Online Educational Delivery Applications: A Web Tool for Comparative Analysis,” retrieved April 14, 2002 from the World Wide Web <http://www.c2t2.ca/landonline/>.

5. J. Nielsen, “Testing Whether Web Page Templates are Helpful,” The Alertbox (May 1998), <http://www.useit.com/alertbox/980517.html>.

6. P. Stokes, “CMS Users Still Waiting for the Killer App,” Eduventures (February 2001).

7. Ibid.

8. Ibid.

About the Author Dr. Margaret (Peg) E. Halloran (Peg.Halloran@usafa.af.mil) is the Director of Educational Technology for the Air Force Institute of Information Technology Applications, located at the U.S. Air Force Academy in Colorado Springs, Colorado.

12

* The views expressed in this paper are those of the author and do not necessarily reflect the official policy or position of the Institute for Information Technology Applications, the Department of the Air Force, the Department of Defense, or the U.S. Government.

Copyright 2002 EDUCAUSE and Margaret Halloran. All rights reserved. This ECAR Research Bulletin is

proprietary and intended for use only by subscribers. Reproduction, or distribution of ECAR Research Bulletins

to those not formally affiliated with the subscribing organization, is strictly prohibited unless prior permission is

granted by EDUCAUSE and the author.

13