A Holistic ApproachA Holistic Approach toto

Assessing eLearningAssessing eLearning

Dr. Tom Wilkinson, DirectorThe Institute for

Distance and Distributed LearningVirginia Tech

Appalachian College AssociationSummit 2004

Abingdon, Virginia

eLearning @ Virginia Tech…. excellence and access

• Nearly 60,000 credit and non-credit eLearning enrollments since Su98.

• 1,672 eLearning course offerings since Su98.

• 568 different eLearning courses offered in past 5 years

• 21 master degrees/certificate/ re-certification/licensure programs

• 85% of academic departments participating

• 89% of online learners were more confident they could reach their academic goals

• 88% rated the quality of instruction good or excellent

0

100

200

300

400

500

600

93/ 94 95/ 96 97/ 98 99/ 00 01/ 02 03/ 04

Growth in eLearning course offerings

Created in 1999, the Institute for Distance and Distributed Learning provides leadership, coordination, management and support to the distance and distributed learning (eLearning) activities of Virginia Tech. The Institute takes a holistic approach to distance learning where the students total educational experience is considered.

Electronically extend the campus

Provide an Open Campus

Research& Development

International Education

Specialized& Contractual

Virginia Tech’s ... Institute for Distance & Distributed Learning

AcademicAdministrative Support

Student & Faculty

Academic

A Holistic Approach to eLearning… the whole of the experience is greater than the sum of its parts

Information Systems

University Engagement … of distance and distributed learning at Virginia Tech

Bursar

Faculty

Library

Alumni

Enrollment Services

CEUT/CIL

Research

University Outreach

Budget &Finance

AdministrativeServices

GraduateSchool

IDDL

Colleges & Departments

Academic

StakeholdersFaculty

Cost

Access

Learning

Systems

Student

Stakeholders

A Holistic Model of eLearning … the whole of the experience is greater than the sum of its parts

Information Systems

Colleges & Departments

University Engagement … of distance and distributed learning at Virginia Tech

Bursar

Faculty

Library

Alumni

Enrollment Services

CEUT/CIL

Research

University Outreach

Budget &Finance

AdministrativeServices

GraduateSchool

Access

CostSystems

Faculty

Student Learning

Stakeholders

• 89% increase in eLearning enrollments• 37% increase in academic departments involvement• 20% of eLearning faculty are honored faculty• 151% increase in eLearning courses • 120% increase in degree & certificate programs • 100% increase in high enrollment online courses• 34% increase in centers & sites served

• 95% of barriers eliminated or significantly reduced • Nearly $2 million in IDDL Enterprise Fund eligible revenue • Award winning programs & the Chronicle article

eLearning at Virginia Tech…1998-2004

Assessment of Online Learners

• 91% were satisfied or very satisfied

• 96% rated services about the same, better or much better than with traditional courses

• 89% were more confident they could reach their academic goals

• 89% said they could not have taken this course in a traditional format

• 92% completion rate of “virtual” students

…over 1,500 respondents

SO WHAT…does that tell us about the quality of our efforts?

Why Assess eLearning?

Who Assesses eLearning?

What about eLearning do we assess?

When do we assess eLearning?

What is Quality eLearning?

“Quality is hard to define,

but I know it when we see it.”

Holistic Approach to eLearning Quality The whole of the experience is greater than the sum of its parts

Academic

Stakeholders

Institute for Distance and Distributed Learning

Faculty Satisfaction

Cost Effectiveness

Access

Learning Effectiveness

System Effectiveness

Student Satisfaction

Stakeholders

What are the Key Indicators of Quality?

• Accrediting agencies (SACS)

• State/Federal Agencies (SCHEV, DoE)

• Best Practices (WCET, ACHE)

• Sloan-C “Quality Pillars”

• Institutional Requirements

• Operational Requirements

eLearning Quality Matrix

Key Quality Indicator Documentation– KQI Source– Quality Metric– Process/Practice– Documentation– Assessment Schedule– Score Card

The KQIs….Key Quality Indicators

Distance and Distributed Learning at Virginia TechKey Quality Indicators

Key Quality Indicators

Criteria Process DocumentationAssessment

Schedule Metrics

Score

1. Institutional Indicators

1a. The program is consistent with the institution’s role and mission

- SACS criteria- SACS Best Practices- ACHE Guidelines- WCET Principles

Review for consistency at yearly IDDL strategic planning retreats

- University Strategic Plan- Academic Affairs Strategic Plan- DDL Strategic Plan

May each year

IDDL Mission and Strategic Plan is consistent with University Mission and Strategic Plan

3

1b. Compliance with SACS notification accrediting requirements

- SACS Criteria- SACS Best Practices- WCET Principles

Review of program offerings with SACS notification file

- VT SACS notification process.- Grad. School program list.- SACS notification file.

Juneeach year

SACS notified of all appropriate distance learning program changes.

3

1c. University’s budgets and policy statements reflect its commitment to Learners

- SACS Accreditation- SACS Best Practices- WCET Principles- ACHE Guidelines

Review university budget for DDL inclusion and IDDL budget for adequacy; Review policy statements for DDL

- University Strategic Plan- DDL strategic Plan- University Budget- IDDL budget allocation- IDDL Enterprise Fund report

July each year

IDDL budget on university base budget;

2

1d. Adequacy of technical & physical plant facilities including appropriate staffing and technical assistance

- SACS Accreditation- SACS Best Practices- WCET Principles- ACHE Guidelines

Conduct student & faculty course and support service evaluations;Review technology plan and budget

- student evaluations- faculty evaluations- support staff evaluations

End of each term

At least 85% of all respondents satisfied with technology and technical assistance;Budget plan provides for technology updating; Staffing structure is appropriate

3

Example 1

Key Quality Indicators Criteria Source (s) Process

1. Institution/Organization Indicators1a. The program is consistent with the institution’s role and mission.

• SACS criteria

• SACS Best Practices

• ACHE Guidelines

• WCET Principles

Review for consistency at yearly IDDL strategic planning retreats

Example 1 (continued)

DocumentationAssessment

Schedule Metric (s) Score

• University Strategic Plan

• Academic Affairs Strategic Plan

• DDL Strategic Plan

• IDDL Strategic Plan

May / June

each year

Program is consistent with University Mission and Strategic Plan

3

Score: 3 = meets or exceeds Key Quality Indicator 2 = meets most of KQI 1 = meets some of KQI 0 = does not meet KQI

Example 2

Key Quality Indicators Criteria Source (s) Process

4. Student Indicators4a. Appropriate services are available for students of electronically offered programs.

• SACS criteria

• SACS QEP

• SACS Best Practices

• ACHE Guidelines

• WCET Principles

Identify appropriate online student services through:

• consultation with university student affairs division

• student surveys

• utilization reports

• evaluation of services

Example 2 (continued)

DocumentationAssessment

Schedule Metric (s) Score

• Student Services for Distance Learners thesis results

• Online Wellness Resource Center dissertation results

• Hierarchy of Student Services needs for distance learners (independent study)

• Course evaluations

• OWRC utilization results

• Yearly identification of service needs

(May/June)

• Student satisfaction & service utilization each semester

•Online student services provided if 10% or more of students indicate the importance of service.

•90% or more of distance learners rate services good or excellent.

3

Contextually Designed Models… addressing changes within the system

Pedagogical Considerations- course management- opportunity for interaction and engagement- extending research- real-world application- assessment and evaluation- content life-cycle

Economic Considerations- business plan- costing break points- program maintenance including course re-design/development- faculty and department incentives- revenue distribution- return on investment

Administrative Considerations- course logistics: application, registration,

section assignment- infrastructure requirements- roles and responsibilities - faculty availability: qualifications, recruitment, hiring, orientation, training- policy adaptation/creation

Interactivity … in eLearning

Number of Students

Fully Interactive

Self-paced

Amount of interactivity

with faculty & other students

High

Low High

Program Development Process… key system elements

Analysis and Planning- need identification and assessment - congruency with mission/strategic plan- market analysis - SWOT analysis- business plan - approval process

Development and Delivery - faculty development - program and instructional design- prototype development- online course development/ production - scalable T & L technology deployment- deployment of support services - program delivery- ongoing assessment and continuous improvement

Evaluation and Assessment - learning outcomes- 5 key holistic factors

-education, access, communication, technology and satisfaction

- quality of the online experience- program effectiveness and impact- program return on investment (ROI)

Identifying a Niche … finding your spot in the eLearning marketplace

Important Factors for Individuals Interest in Taking Online Courses

• 98% - transferability

• 97% - online access to library

• 96% - interaction with instructor

• 94% - reputation of the institution

• 94% - technical support offered

• 93% - cost

• 93% - allowed to work at own pace

Market Analysis • Adults aged 18 – 50• Possess some college• 4 states in eastern U.S.A.

Course Development Model … a scalable system

PHASE 1Planning &Development

PHASE 2Component &Module Production

PHASE 3Component &Module Evaluation

Material Development Content Formatting Course Management Tool Streaming Production ADA Compliance Audio Graphics

Learning Outcomes Design and Flow Instructional Delivery Production Values

LEARNER

Course Feedback

Course Feedback

DELIVERY

PRINT

CD-ROM

INTERNET

SERVER

Distance Learning Instructional Design Online Education Course Transformation

Considerations Identify Delivery Methods Course Re-design Plan Component Design Plan Production Plan

DESIGN, DEVELOPMENTMaster Faculty and IDDL Staff

LEARNER

LEARNER

LEARNER

INSTRUCTOR

Online Course Delivery Model … a scalable system

MASTER FACULTY• Plan Course • Design Content• Maintain Content• Establish Assessment Metrics• Instructor of Record

IDDL• Programmatic Support• Course Transformation• Develop Course Media• Coordinate Delivery

DLINSTRUCTOR

DLINSTRUCTOR

DLINSTRUCTOR

DELIVERY

N

N

N

DESIGN, DEVELOPMENT, DELIVERY MANAGEMENT

Course Feedback

Course Feedback

LEARNERS

INSTRUCTOR• Monitor Student Progress • Manage Logistical Tasks • Guide, Mentor, Coach• Lecture & Provide Feedback• Help Design Learning

Scalability … Cost Model Comparison (Technology Costing Methodology – WCET)

Number of Students

$

Course-Related Costs

Classroom

Mediated

High

Low High

Student-Related costs

For more information please contact:

Dr. Tom Wilkinson – Director,

Institute for Distance and Distributed Learning

Virginia Tech

tom.wilkinson@vt.edu

www.iddl.vt.edu

www.vto.vt.edu