Transformative Assessment: A Progress Report from NLII

Vicki Suter, EDUCAUSE

Stephen C. Ehrmann, TLT GroupGary Brown, WSU

Joan Lippincott, CNI

Copyright Statement

• Copyright Gary Brown, Stephen Charles Ehrmann, and Joan K. Lippincott , 2002. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Outline

I. Introduction and some guidelines for assessment– Steve Ehrmann, The TLT Group’s Flashlight Program

II. Results of NLII Online workshop and a rubric for assessing transformative assessment – Gary Brown, Washington State University

III. Making changes – institutional and team issues – Joan Lippincott, Coalition for Networked Information

Steps to Date

• Focus Workshop in Denver, March, 2002• 26 institutions; 11 teams and 15 ‘singles’

• Online workshop in May-May, 2002• 11 institutions started; about 5 finished

• Largest teams from U Washington, U Maryland, Colorado system

• Gary and I will be talking about what we’ve learned about evaluation strategies

• Joan will discuss the team and institutional strategies and concerns

Seven Dangerous Questions

Using IT and Assessment to Transform Academic Programs

Stephen C. Ehrmann, Ph.D

Thanks!• 330+ institutional subscribers to TLT

Group services

• EDUCAUSE/NLII

OutlineI. “Transformation?”

II. Seven Dangerous Assessment Questions

III. “Transformative Assessment?”

IV. How can assessment promote improvement?

V. The READY web site

The Vision (1960 – present)• Technology, properly used, can improve education

and its outcomes• New participants (learners, experts)• New materials and resources• More options, responsibilities for learners, instructors, staff• New ways of organizing & supporting learning

• Therefore with every passing decade we ought to be able to make further improvements in education and its outcomes.

• Can we see them? Can we even make improvements if we can’t see them?

Def’n of Transformation?

• Important, qualitative change in a “triad”:

I.T.Goal - Outcome

Activity – structure in use

{

Not Enough

Outcome/Goal Activity-Structure

Technology(s) and other needed elements

Whatever Infrastructure used a lot, with great satisfaction

Infrastructure

Tech support

Not Enough, Either

• Reference, “What Outcomes Assessment Misses”

Outcome/Goal Activity-Structure

Technology(s) and other needed elements

Improve learning, retention, etc.

?Infrastructure

Tech support

Example of a TransformationOutcome/Goal Activity-

StructureTechnology(s) and other needed elements

Better learning outcomes (e.g., higher ‘scores’, better retention)

7 Principles on the upswing across the curriculum;

Wider variety of learners, teachers, other resources

Communications; Productivity tools

Skilled faculty, spread of low threshold activities (e.g.,, managing collaboration)

Another TransformationOutcome/Goal Activity-

StructureTechnology(s) and other needed elements

Community, during & after college

Plentiful interaction, formal and informal, including outsiders

Virtual spaces; communications; directories, etc.

Staff who are good at collaboration; outside partnerships

And Another

Outcome/Goal Activity-Structure

Technology(s) and other needed elements

Speed to degree; access

Hybrid courses;

Altered recruiting and student info

CMS, e-mail, Web, etc.

Help in redesigning courses so they take more advantage of online; less need for f2f

Assessment isn’t for Wimps

• “Dangerous questions” are:• Risky to ask!• Even riskier not to ask! (especially if other

institutions or your constituents are beginning to ask) [hypothesis]

Seven Dangerous Questions1. WHICH educational outcomes are we trying

improve (baseline)?2. We can’t change outcomes without changing what

we do. So far, is there enough change in how people teach and learn?

3. Where there isn’t, what’s standing in our way?4. Is IT useful in changing those T&L activities?5. Where IT hasn’t been used that way, why not?6. What are the costs and stresses of key activities?

Can they be reduced?7. Now that years have passed, what have we

achieved (compare with #1)

TA & AlignmentProgrammatic Outcome (who learns, what they learned, etc.) & Strategy

Assessment – Use of Data

Mission StatementBudget ProcessProgram DevStaff Dev AdmissionsLegislative/Alumni Capital plans(etc.)

Assessment’s Powers1. The discussion of how to collect data can help to

make discussion of goals and strategies less abstract, agreements more meaningful. Medium for people to talk with one another.

2. Discussion of alignment can help in those ways, too.

3. Periodic findings can refocus attention4. Spotlight problems as well as achievements in

order to attract resources5. Steering, e.g., grab hidden opportunities, reduce

or remove barriers

Getting READY

• Check out the READY website at EDUCAUSE, section on transformative assessment• Self-assessment checklist of elements

needed if your institution is to use data to help accomplish any transformative change

Summary: What Does TA/TLT Look Like?

Growing institution sense of direction(s) of progress for using IT (Triads)

Good infrastructure to support assessment at many levels

Already using data to help move in that direction (LOTS of relevant studies feeding on one another)

Alignment (two-way) of assessment with other functions (budgeting, staff development, fund-raising, recruitment, etc.)