March/April 2013

The Assessment Update is published by the Assessment Committee to share assessment taking place at jefferson college and to facilitate conversation about assessment best practices.

In response to a request by the Board Administration Team (BAT) and Faculty Association Team (FAT) to explore methods for evaluation of online courses, the Assessment Committee is researching Quality Matters, a faculty-driven process designed to help institutions improve their online courses through peer review and professional development. This program, based on best practices, has quickly gained national recognition and is currently being used by several peer colleges in the state to ensure the quality of their online and blended courses.

A Collaborative, Collegial ProcessIn January, the Assessment Committee hosted a teleconference with Dr. Leo Hirner, Director of Distance Learning at Metropolitan Community College and co-author of “Quality Indicators of Online Programs” (Community College Journal of Research and Practice, Volume 36, Issue 2, 2012). As a member of KC REACHE (Kansas City Regional Access Consortium for Higher Education), which includes several Missouri public and private universities and community colleges, MCC has successfully used Quality Matters to improve online instruction and to earn Quality Matters certification for several online courses.

Dr. Hirner noted that the process is faculty driven and diagnostic rather than evaluative, emphasizing that the goal is not to evaluate faculty or delivery of instruction but to improve and certify the quality of the instructional design of online courses. He pointed out that faculty must have training in applying the Quality Matters rubric as well as the opportunity to update courses before a review. The reviews are led by trained, certified faculty from participating institutions, and all faculty are invited to participate in training for reviewer certification. Dr. Hirner added that the collegial nature of the process has helped participants from different institutions develop positive relationships that have proven to be valuable in other collaborative efforts as well.

“Quality Matters” in Online Courses: Evaluation of Online Course Design

Continued...• 1 •Our Mission is Quality Education / Student-Centered • Comprehensive • Accessible

UNDERLYING PRINCIPLES OF QUALITY MATTERS

The Quality Matters Rubric and Processes Are...

CONTINUOUS•Theprocessisdesignedtoensurethatallreviewed

courseswilleventuallymeetexpectations.•Theprocessisintegraltoacontinuousquality

improvementprocess.

CENTERED•Onresearch:thedevelopmentoftherubricisbased

onnationalstandardsofbestpractice,theresearchliterature,andinstructionaldesignprinciples.

•Onstudentlearning:therubricandprocessaredesignedtopromotestudentlearning.

•Onquality:thereviewsetsaqualitygoalatthe85%levelorbetter(coursesdonothavetobeperfectbutbetterthangoodenough).

COLLEGIAL•AQualityMattersreviewispartofafaculty-driven,

peerreviewprocess.•Thereviewprocessisintendedtobediagnosticand

collegial,notevaluativeandjudgmental.

COLLABORATIVE•Thereviewisbasedoncollaborativelyidentified

evidencefoundinthecourseratherthanthepersonalpreferenceofanindividualreviewer.

•Thereviewisflexibleandnotprescriptive(manywaystomeeteachstandard).

•Thereviewteamconsistsofthreeexperiencedonlineinstructorsasreviewersalongwiththecoursefacultydeveloper.

http://www.qmprogram.org/research-grants/fipse/principles

Focus on AlignmentAccording to information on the Quality Matters web site, a unique attribute of the program is its specific standards for alignment of expected learning outcomes and “critical course components,” including assessment and measurement, instructional materials, learner interaction and engagement, and course technology. Jefferson College philosophy instructor Teresa Fiala, who has participated in Quality Matters training, spoke to the Assessment Committee about her experience with the program, affirming that the training deepened her awareness of the importance of designing all course components to directly align with specific learning outcomes. Dr. Fiala further confirmed that she found the training beneficial overall.

Supported by Research and ResultsThe Quality Matters Rubric, according to “QM Research,” was developed “based on a review of the literature, the expertise of experienced practitioners, and existing standard sets.” Updates and new editions have also been based on current literature reviews. Citations for these reviews are available on

the website.

Quality Matters has also conducted studies to assess the impact of its program. In “What We’re Learning from Quality Matters-Focused Research: Research, Practice, Continuous Improvement,” Kay Shattuck, Director of Research, notes the emergence of research on Quality Matters and provides an overview of findings, concluding that the program “can be an important component in an institution’s total quality improvement and assurance efforts” and that “QM-focused research can continuously contribute to the improvement of online education.”

Other ConsiderationsAlong with benefits are costs for subscriptions, reviews, and reviewer training, though Jefferson College would be able to join the KC REACHE consortium to receive a discounted subscription rate (see the 2012-2013 Fee Schedule).

Alternatively, Jefferson College could continue to develop its own program for evaluation of online courses. Some institutions have developed their own systems after participating in Quality Matters training, and some have made their own rubrics available to others online (e.g., California State University–Chico’s Rubric for Online Instruction and the Illinois Online Network’s Quality Online Course Initiative Rubric). Some researchers believe that evaluation of online courses must go beyond course design to include formal evaluation of faculty performance, but many also recognize that Quality Matters certification is considered a nationally recognized benchmark in the evaluation of online courses.

Online and blended course offerings at Jefferson College will almost certainly continue to expand, which calls for a tested method of assessing and certifying the design of these courses. The Assessment Committee is seeking additional feedback about the possibility of adopting Quality Matters at Jefferson College. Faculty and staff are encouraged to visit the Quality Matters website and to contact Allan Wamsley, Director of Online Learning and Instructional Technology, or other members of the Assessment Committee to share thoughts and ask questions.

• 2 •Our Mission is Quality Education / Student-Centered • Comprehensive • Accessible

GENERAL STANDARDSTheQualityMattersRubricincludesasetof

eightgeneralstandardsand41specificstandards.Theeightgeneralstandardsareasfollows:

Course Overview and Introduction

Learning Objectives

Assessment and Measurement

Resources and Materials

Learner Engagement

Course Technology

Learner Support

Accessibility

http://www.qmprogram.org/rubric

How Will My Institution Benefit From Implementing

QUALITY MATTERS?

Quality Matters processes benefit both individual faculty and their institutions in the following ways:

1. Improvedstudentlearningoutcomesandretention

2. Adoptionofasystematicandcomprehensivecontinuousqualityassuranceprocessthatincludesfacultytraining,coursedevelopment,andcourserevisionsthatarealignedwithaccreditationstandards

3. Incorporationofnewtechnologiesandresearchfindings

4. Opportunitytoengageinbenchmarkingactivitieswithpeerinstitutions

5. Ongoingfacultyprofessionaldevelopment

6. Increasedflexibility,creativity,anddivergentthinking

7. Increasedefficiencyinusinginstitutionalresources

http://www.qmprogram.org/faq

• 3 •Our Mission is Quality Education / Student-Centered • Comprehensive • Accessible

HELP WANTED!It is time to update the Jefferson College Assessment Guidebook, and the Assessment Committee is seeking volunteers to serve on an Assessment Guidebook subcommittee.

Opportunity for Service to the InstitutionWorking on this subcommittee is a great opportunity to participate in service to the institution. Volunteers do not have to be experts in Assessment. Anyone with ideas, suggestions, or willingness to help with revision of the guidebook is encouraged to serve on this subcommittee.

Multi-Section and Program-Level Assessment The current version of the Assessment Guidebook is a good source for faculty evaluating learning outcomes and designing classroom assessment techniques for course-level assessment. It includes general guidelines, samples, rubrics, scoring guides, etc. Goals for revision include expanding the information on multi-section, program-level, and institutional assessment. Anyone with experience or interest in any of these areas is especially encouraged to provide input.

ContactAnyone who is interested in serving on the subcommittee should contact Chris Otto, Assistant Professor of English and Chair of the Assessment Committee at cotto1@jeffco.edu.

Chris Otto, ChairArts & Science Faculty

Dedric LeeArts & Science Faculty

Sheba NitschCareer & Technical Education Faculty

Cindy RossiCareer & Technical Education Faculty

Holly Lincoln, Student Services

Jason Gardner, Student Services

Connie NashStudent Learning & Support Liason

Joan Warren, Institutional Research

Lisa Pritchard, Library

Mary Beth OttingerCareer & Technical Education Division Chair

and Curriculum Liason (ex officio)

TBA, Student

Shirley DavenportDean of Arts & Science Education (ex officio)

Dena McCaffrey Interim Dean of Career & Technical Education

(ex officio)

Julie FraserDean of Student Services (ex officio)

Sandra FreyArts & Science Division Chair (ex officio)

Linda AbernathyArts & Science Division Chair (ex officio)

Michael Booker, Division Coordinator of Communication and Fine Arts

Allan Wamsley, Director of Online Learning and Instructional Technology

It is the policy of Jefferson College that no person shall, on the basis of age, ancestry, color, creed, disability, gender, national origin, race, or religion, be subject to discrimination in employment or in admission to any educational program or activity of the college. If accommodations for a disability are needed, please contact Christine Platter at (636)481-3169 / 797-3000, ext. 3169; TDD (636)789-5772.

1000VikingDriveHillsboro,MO63050(636)481-3000/797-3000www.jeffco.edu

2012-2013 ASSESSMENTCOMMITTEE

UPCOMING IN THE NEXT ISSUELook for these stories in the August-September

issue of Assessment Update:

•Multi-SectionAssessmentinSociology

•AssessmentSurveyofFaculty