ESTABLISHING QUALITY STANDARDS FOR FACULTY DEVELOPMENT IN TEACHING ONLINE COURSES: LESSONS FROM THE TRENCHES

Gail HodgeAssociate Dean for Academic Affairs

Director of On-line Programming

Jonathan Helmke

Faculty Librarian

WHO WE ARE

University of DubuqueSmall, private, Christian college

1400 undergraduate 123 graduates (MBA and MAC) 136 Seminary

Mission drivenBegan offering undergraduate online

courses in Fall 2007

GOALS OF ONLINE PROGRAM

Accessibility: Provide Students with alternative ways to earn course credits (avoid institutional time conflicts)

Vocation Preparation: Build upon career skills for working in online environments

Revenue Generating: Provide course opportunities to non-traditional students and non-degree seeking students who are new to the University

Presented to Board of Trustees

The goal for online education is

to provide mission-driven,

competitively marketed,

income generating, and clearly

branded undergraduate and

graduate courses, as well as

selected graduate degree

programs, to enhance and

expand educational

opportunities.

Written by Online Advisory Committee

STANDARDS NEED TO REFLECT OUR MISSION: BRANDING What are we best known for? Incorporate these same attributes into the online product

Personal attention Two voice contacts

Learner-centered pedagogy Assignments that help students explore and discover

various concepts/theories (engage in active learning) Students interact with peers, instructors, and others

Serve first-generation students Diverse learning styles Teach to different senses: Text, graphics, audio,

interactive discussions Academically under-prepared when entering college

FRAMEWORK

FRAMEWORK FOR TODAY‘S PRESENTATION

Sloan-C Pillars

Lessons Learned integrated w/ Sloan-C Pillars Gap analysis 360 Assessment Recommendations for next steps

Learning Effectiveness

Cost Effectiveness

Access Faculty Satisfaction

Student Satisfaction

1: IDENTIFY CHAMPIONS – EVEN IF IT MEANS BUYING THEIR AFFECTION

Recruitment of online Faculty Offer course developmental dollars on top of

regular teaching compensation to encourage faculty to come out of the woodwork – our pioneers.

Open Learning Management System (LMS) to face-to-face (F2F) classes Builds faculty skills in authoring content and

knowledge of using LMS Exposes students to LMS

Sloan-C Pillar: Faculty Satisfaction and Cost Effectiveness

2: THE GREAT DIVIDE: DIGITAL IMMIGRANTS VERSUS DIGITAL NATIVES

Faculty nervous about students being ahead of them technologically Unsure if they can field/trouble-shoot technical

questions Digital natives catch-on to the LMS fairly quickly

No surprise as Digital Natives are the same audience who use social networking sites, like FacebookTM

Digital Natives are also the drivers, pushing faculty into adopting the LMS

Techno-phobia reinforces need for strong training program for faculty Beginning, middle, and end of academic year LMS Support Group meetings throughout year (MUG)

Sloan-C Pillars: Cost Effectiveness, Faculty Satisfaction and Student Satisfaction

AFTER YEAR 1

We began developing standards for Faculty Development Established set training requirements Instituted Peer and Self Review for assessment Incorporated course and student support

standards/requirements Common LMS interface

“Wall” consistency First block of content above the fold line

Welcome video to introduce students to course (makes instructor a “real” person)

Two voice contacts with each student All content needed to be developed prior to

start of class

FACULTY TRAINING ON LMS

Online Faculty: Mandatory 9 hours of training 1 hour – Quality standards overview/update 6 hours of LMS training, from intro to advanced

applications 2 hours of resource training (e.g. MediaSite, Skype)

LMS User Group Meets twice a semester Members discuss how-to’s/issues they have with

LMS F2F Faculty: Optional 2 – 6 hours of training

August and May (before and after academic year) 2 hours – introductory 2 -4 hours – advanced applications

TR

AIN

ING

SC

HED

ULE

FO

R O

NLIN

E

INS

TR

UC

TO

RS

Offered Between SemestersWeek 1

1 hour of QA tips, morning (repeated in Spring for past instructors)2 hours of Introduction to LMS, afternoon

Week 3, morning2 hours of Intermediate LMS

Week 3, afternoon2 hours of Advanced LMS

Week 5, morning1 hours of training on instructional design tools (MediaSite and Skype)

Week 5, afternoon1 hours of training on library resources (eReserve)

3: PROOF IS IN THE PUDDING: INSIST ON ADHERING TO MILESTONES IN THE COURSE DEVELOPMENT SCHEDULE!

Courses needs to be developed and QA’d (reviewed) before they go public Adherence to standards

Seems like a no-brainer; but was difficult to enforce Some faculty not confident in building whole

course without testing initial modules Procrastination

Some faculty approached course development like their f2f class -- prepped day before, or they think they can wing it

SUGGESTION: Consider developing the online course for delivery in a F2F class before it is offered online

Sloan-C Pillar: Learning Effectiveness; Access

4: BE PREPARED TO THROW-OUT THOSE OLD NOTIONS ABOUT STUDENT LEARNING

Behaviorist vs Constructivist Move from “Sage on the Stage” to “Guide on

the Side” Student-centered learning is putting students in

charge of their learning Role of instructor is to provide resources

Develop content that engages studentsProvide strong resources (e.g. library)

Take advantage of Web 2.0 and interactive media Measure/assess student engagement

Sloan-C Pillar: Learning Effectiveness

ONLINE TEACHING

What direction is communication taking place in your online course?

Instructor StudentStudent InstructorStudent Student

Learning Effectiveness

Cost Effectiveness

Access Faculty Satisfaction

Student Satisfaction

#3 Proof is in the Pudding: Insist on Adhering to Milestones in the Course Development Schedule!

#4 Throwing out old notion about student learning

#2 Training: Digital Immigrants versus Digital Natives

#1 Identify Champions – Even if It Means Buying Their Affection

#2 Training: Digital Immigrants versus Digital Natives

#3 Adhering to Milestones

#1 Identify Champions – Even if It Means Buying Their Affection

#2 Training: Digital Immigrants versus Digital Natives

#2 Training: Digital Immigrants versus Digital Natives

#4 Throwing out old notion about student learning

Applicable Sloan C- Pillars

GAP ANALYSIS

THE FIRST GAP

Faculty training has been primarily focused on learning the LMS “tool”

Shift to assisting faculty on how to develop sound instructional design activities that promote student learning in the online environment that is motivating, engaging, and meaningful.

EXAMPLE – FORUM DISCUSSION Reading: pp. 451-459. Sometimes

the most difficult part of an essay, when writing about an art work or an artist, is to describe it without using the word "masterpiece" or the idea of "the insane".

So, assuming that Starry Night on p. 457 is NOT a masterpiece and is NOT by an insane artist, what is going on in the picture? The internet and resources available in our Library might help you with some alternative ideas, though the picture is discussed in your book.

About 400 words, please.

Starry Night by Van Gogh

Challenge faculty to think of ways to improve upon

this activity

What kind of feedback might the student expect

after answering this question?

ASSESSMENT SHOULD CONNECT TO GOALS

Reminder of our Goals: Accessibility: Provide Students with Alternative

Ways to Enroll in classes (in order to avoid institutional conflicts with other activities/courses)

Vocation Preparation: Building upon career skills for working in online environments

Revenue: Increase new enrollment by attracting new markets

WHAT’S MISSING …

Sloan-C Pillars: Faculty and Student Satisfaction

THE SECOND GAP

What’s missing from our goals are the things we really want to know:

Have student learning outcomes (SLOs) been met, as much if not better, in the online environment as F2F classes? How can we get at direct assessment?

Are students more or less satisfied with taking online courses than taking the F2F courses? How do students measure course satisfaction? Are we asking the right questions of students? Are students answering the questions we are asking

or are they answering the questions we aren’t asking?

Sloan-C Pillars: Learning Effectiveness and Student Satisfaction

ASSESSMENT360 Peer, Self, and Student

360 ASSESSMENT

Peer review (COIL) Boiled down to 10 questions from 50

Self review Used same 10 questions from Peer Review

Student evaluations Separate instrument

Open-ended question on satisfactionQuantitative items on accessibility

Open forum discussion of all Online Faculty Provide lunch Discuss what went well; what didn’t work;

recommended changes for next time

COIL (PEER-SELF QUESTIONS)

1. Presents course content in a manner that hierarchically structures the sequence of information

2. Provides activities that actively engage students in interactions with web-based course content.

3. Provides activities that can engage students in understanding and mastering different learning strategies.

4. Provides clear and adequate guidance on how to navigate the course, engage in course work, and turn in assignments.

5. Clearly delineates institutional policy on cheating and plagiarism at the start of course.

6. Provides rules/expectations for active participation in all online activities.

7. Personalizes communication by/with student-student and student-instructor (e.g., feedback on assignments, replies to discussion forum postings)

8. Provides students with timely, continuous, frequent support, and feedback. (e.g., feedback on assignments came within 1 week after deadline. Feedback was instructional/remedial)

9. Provide opportunities for student to question instructor to insure accuracy of understanding. (e.g., a chat room or discussion forum that encourages students to post questions. Syllabus and site provide instructor contact information and times of availability)

10. Conducts a teleconference (or other “voice” contact) at least twice with each student during the course.

RESULTS OF 360 ASSESSMENT

Hypothesis 1: Do instructors rate themselves similarly to their peers?

T-test run Findings: Instructors tended to rate

themselves lower than their peers 14 cases 12 were lower than peer reviews 2 were higher then peer reviews

1 was within 1 STD below mean1 was outside 2 STD below mean

Second Peer Review reinforced first Peer Reviewers observation

T-TEST

Difference

Mean 0.08Standard Error 0.035019618Median 0.08Mode 0.2Standard Deviation 0.131031411Sample Variance 0.017169231Kurtosis -0.001776348Skewness -0.788888516Range 0.42Minimum -0.2Maximum 0.22Sum 1.12Count 14Largest(1) 0.22Smallest(1) -0.2Confidence Level(95.0%) 0.075655284

1 instructor fell below 2 STD of mean *

*

STUDENT EVALUATION OF ONLINE EXPERIENCE

Open-ended Qualitative Questions Was this class intellectually stimulating? Yes/No, Did this class stretch your thinking? Yes/No

1. Please write a few statements supporting your answer as to why or why not you felt the class was intellectually stimulating, and or stretched your thinking.

2. What aspects of this class contributed most to your learning?

3. What aspects of this class detracted from your learning?

4. What suggestions do you have for improving the class?

STUDENT EVALUATIONS … Quantitative Questions

Scale of 1 – 5 (strongly disagree – strongly agree)1. The online course was more challenging than a similar

face-to-face course.2. I think I learned more by taking this course online than

by taking it face-to-face.3. I felt “more connected” to my online instructors than to

my face-to-face instructor(s).4. Based on my experience with this online course, I

would take another online course.5. Knowing what I know now, I would NOT take this course

ONLINE.6. I interacted more with the other students in this online

course then I did in my face-to-face classes.7. I felt isolated in this online course.

EXAMINING THE OUTLIER TO STUDENT FEEDBACK

No real indication of problems How much weight do we assign student

evaluations? What aspects of a course are most important

to undergraduate students? Time demands Subject Matter Interest Rigor of assignments Developing a connection to the professor

Build in next student evaluation instrument question items that get to the heart of these attributes

WHERE DO WE GO FROM HERE?

WHAT DID WE LEARN?

Peer evaluation tends to be higher than instructor-self evaluation

Student evaluations had no correlation to peer or instructor evaluation

Student evaluations clustered around the mean

We need to improve our student course evaluation instrument to get feedback on what it is that students are really evaluating.

We need to implement direct assessment of student learning outcomes

NEXT STEPS

Re-write goals to include quality assurance of student learning outcomes, as well as faculty and student satisfaction

Adopt standards for faculty development Adopt assessment strategies for faculty

development. Are we asking the right questions that will help

faculty in their course development and delivery?

Are we providing adequate support for faculty development?

RECOMMENDED STANDARDS FOR FACULTY DEVELOPMENT

Require training on functions of LMS Require participation in faculty development

workshops for developing strong pedagogically student-centered activities that match-up with student learning outcomes

Provide professional development funds so faculty can attend online education conferences/workshops, as well as use as incentives for developing online courses

Include online teaching as a hallmark in faculty promotion and tenure guidelines

Standardize on an assessment tool that provides meaningful feedback to faculty and students that will help shape and improve online course delivery