1

Addressing the Questionof Quality in E-LearningEnvironments

Gonzalo Bacigalupe, EdD, MPHAssociate Professor & ChairDepartment of Counseling & School PsychologyUniversity of Massachusetts Bostongonzalo.bacigalupe@umb.edu

Distance Education Definitions

•Instructor and students need not be in the samephysical place. (NCES, U.S. DOE)

•A formal educational process … the major portionof the instruction occurs when the learner and theinstructor are not in the same place at the sametime through virtually any media. A DistanceLearning Program … over half of the requiredcourses in the program occur when the learner andthe instructor are not in the same place at thesame time. (NCATE GUIDELINES, 2005)

2

ODE Features

• Accelerated expansion

• Vast majority of new distance programs areoffered online with a cohort-based model (asopposed to self-paced) (DOE-OPE, 2006)

• Access (flexibility, working students, travel,technical application) is core in mission ofinstitutions offering ODE

ODE in the U.S.

(Sloan Consortium, 2005)

3

Why worryaboutquality?

•The days of astonishinggrowth in ODE may beover

•Brands based on studentoutcomes may provemore successful

•Faculty, curriculum, andinstitutional reputationare attributes onlinelearners care about.(Stokes, 2006)

Faculty have been proceeding foryears teaching courses, hopingthat they are effectivelypresenting what they themselveshave learned, and that studentsthen are demonstrating that theyhave learned what the facultythink they should be learning.

The advent of online learning hasbrought to the fore pressure onus in higher education todemonstrate outcomes in a waythat we have never felt before…(Hamlin, et.al., 2004, p.3)

Note for those not teaching online:This is not just in the case of e-learning

4

ODE Defenders Responses

• ODE program and courses are of the samecaliber as the one on campus

• ODE increases access to a more diverseconstituency and enhances inclusiveness

• ODE offer educational opportunities forall students that may not be possible incampus

• … ?

No significant difference hypothesis

•Instructors adopting learning management systems must considerhow students' evaluations of learning reflect the overall quality ofinstruction. Traditional course evaluations are used for facultyretention, tenure and promotion decisions, but also provide instructorswith valuable information on the quality of their instruction.

•Topper (2007) studied the response rates and compared instructionalquality, using student course evaluations along with additional datafrom online and face-to-face graduate education courses, to evaluatethe effectiveness of instruction. The statistical analysis of courseevaluations showed no significant difference in instructional qualitybased on the format used.

•Together with comparisons of student work, these results provideadditional evidence in support of the finding of no significant differencebetween formats in the area of instructional quality.

•Topper, (2007). Are they the same? Comparing the instructional quality of online and face-to-face graduate education courses. . Assessment & Evaluationin Higher Education 32 (6), 681-691.

5

Caveat

Quality is subject to interpretation

6

Measuring

• Everyone involved in online learning would agree that quality isparamount; however, objectively measuring quality is difficult andinfrequently undertaken.

• Quality is an unstated expectation, yet we rarely use a formalprocess for assessing quality of training products.

• Poor design, project under funding, overly optimistic schedules, andtechnical barriers are all enemies of quality.

• A methodical approach to evaluation can help remove subjectivebiases and achieve a more authoritative analysis.

• (WBTIC, 1994-2007)

Learning styles

“Educatorsmust do morethan provideaccess toinformation”(Campbell,1998, p.vii).

7

Sloan (Moore, 2005)

Sloan

8

Scorecard

Questions by Accreditation Body:Example

Unit Accreditation Board = UAB

9

Red Flags

ODE Curriculum approval differs from F2F

Large numbers of students in DE courses

Target population needs are not addressed

Single method of assessment

Students’ dissatisfaction

Red Flags

Unclear course objectives

Cookie cutter approach to course development

Very little board/threaded/lively discussion

Lack of substance and reflexivity of students’postings

Rapid turnover in DOE adjunct faculty teaching

No regular update of courses

10

Best Practices Principles

• Communicate clearly your expectations

• Timely submissions

• Prompt feedback

• brief assignments, and/or

• accumulative scaffolding

• High expectations for students

• Encourage faculty-student contact

Best Practices Principles

• Stimulate student collaboration and cooperation

• Uncover students’ knowledge and skills

• Different learning styles

• Resources for teaching and peer tutoring

• Make the course your “brand”

• Share the outcome (OCW) be accountable

11

NATIONAL EDUCATION ASSOCIATION (NEA)Benchmarks for Success in Internet-Based Distance Education

• Institutional Support Benchmarks• Technology plan to ensure information integrity

• Reliable technology system and centralized support system

• Course Development Benchmarks• Guidelines for minimum standards are used for course

development, design, and delivery, while learning outcomesdetermine the technology being used to deliver course content.

• Instructional materials reviewed periodically to meet programstandards

• Courses design requires students to engage themselves inanalysis, synthesis, and evaluation as part of their course andprogram requirements

NEA Benchmarks for Success in Online Distance Education

• Technology/Learning• Student-faculty and student interaction is essential

• Feedback is timely and constructive

• Students learn about proper methods of effective research

• Course Structure

• Students are advised before program start to determine self-motivationand commitment to distance learning and to assess technology access

• Course information for students is exhaustive (i.e., syllabus)

• Students have access to sufficient library resources

• Clear expectations about assignment completion

12

NEA Benchmarks for Success in Online Distance Education

•Student Support

•Students receive information about requirements, tuition and fees, student support, etc. Hands ontraining is available to students to secure digital information

•Faculty Support

•Course development technical assistance

•Support in transition from classroom to online

•Permanent training and peer mentoring

•Evaluation and Assessment

•Multi-method of assessment and specific standards to assess effectiveness and teaching/learningprocess

•Data on enrollment, costs, and successful/innovative use of technology are used to evaluateprogram effectiveness

•Intended learning outcomes are reviewed regularly

Some best practices at UMB

• Instructor and designer have been the same

• Curriculum is approved by faculty governance

• Multiple technologies are employed in delivery ofonline and hybrid courses

• Some programs have acquired a rich experiencein the use of synchronous modalities

• Wide variety of disciplines and programs havedeveloped ODE offerings

13

What needs improvement?• System wide decision-making may have

hindered the ability of campus to innovateand adopt appropriate resources

• Technology resources/support for hybridand on campus courses are not comparablewith ODL -> Pace of technology adoption mayslow down and hurt ability to mainstreaminguse across classrooms

• Resources for assessment, evaluation, andresearch have not accompanied allocationof resources for course development

ODE Quality: Emerging Questions

• Do synchronous versus asynchronous venuesimpact access?

• What criteria should we employ to measureadequate instructor’s availability?

• How can faculty and students successfullymanage the potential of social networking tools?

• Does ODE increases/enhances access ofmarginalized groups or foster the status quo?

• Is ODE generative of knowledge creation, qualityscholarship, collaboration, community service,etc.)?

14

Useful Resources• The Sloan Consortium Effective Practices Quality Framework www.sloan-c.org/effective/framework.asp

• ADEC Guiding Principles for Distance Teaching and Learning www.adec.edu/admin/papers/distance-teaching_principles.html

• EFA Global Monitoring Report portal.unesco.org/education/en/ev.php-URL_ID=35939&URL_DO=DO_TOPIC&URL_SECTION=201.html

• How effective is distance education? An AskEric Response www.eduref.org/cgi-bin/printresponses.cgi/Virtual/Qa/archives/Educational_Technology/Distance_Education/disted.html

• UMASS Information Technology Council www.massachusetts.edu/itc/mission.html

• Quality Principles University of Wisconsin-Madison www.wisc.edu/depd/html/quality3.htm

• ESAC Faculty Development, Quality Matters: Inter-Institutional Quality Assurance in Online Learningwww.esac.org/fdi/rubric/finalsurvey/demorubric.asp

• National Standards of Quality for Online Courses www.nacol.org/nationalstandards/

• What Constitutes Quality in Web-Based Training? www.webbasedtraining.com/primer_quality.aspx

• My e-learning links at delicious http://del.icio.us/bacigalupe/e-learning

• REFERENCES: In development, pleaseemail me for a copy in a couple of weeks.

• gonzalo.bacigalupe@umb.edu