authentic institutional assessment

36
Complex Questions & ‘Kitchen Stories’: Pursuing Authentic Institutional Assessment in an Age of Accountability William S. Moore, Ph.D. Coordinator, Assessment, Teaching & Learning WA State Board for Community & Technical Colleges [email protected] 360- 704-4346 National Student Affairs Assessment & Retention Conference June 2005 Atlanta, GA

Upload: wsm8851

Post on 15-Dec-2014

123 views

Category:

Documents


1 download

DESCRIPTION

arguing for the need to be true to the notion of "authentic" assessment in pursuing institutional assessment efforts in higher education, despite (or perhaps because of) the strong and growing pressure around a narrow vision of accountability for higher education

TRANSCRIPT

Page 1: Authentic Institutional Assessment

Complex Questions & ‘Kitchen Stories’:Pursuing Authentic Institutional

Assessment in an Age of Accountability

William S. Moore, Ph.D.Coordinator, Assessment, Teaching & Learning

WA State Board for Community & Technical Colleges [email protected] 360-704-4346

National Student Affairs Assessment & Retention Conference

June 2005Atlanta, GA

Page 2: Authentic Institutional Assessment

As you can clearly see

on slide 397…

Adapted from “Dilbert” & Scott Adams

Oh, no—not

another case of

PowerPoint

poisoning!!!

Page 3: Authentic Institutional Assessment

Key Messages• We’re in the Age of Accountability, not

Aquarius

• Assessment needs to be about learning, not accountability (or retention per se)

• Re-framing our core assumptions about assessment leads us to more complex, and more useful, questions about what matters

• Authentic approaches to institutional assessment will help us address these complex questions more effectively

Page 4: Authentic Institutional Assessment

Age of Accountability

Page 5: Authentic Institutional Assessment

Forces Contributing to an Age of Accountability View of postsecondary education

as strategic investment and societal benefit

Limited resources and competing governmental demands

Increasing public intrusion regarding “proof” of quality “performance” in higher education

Growing cynicism toward all major institutions (issues of trust)

Page 6: Authentic Institutional Assessment

Accounting or Accountability?

Assessment as learning can be reconciled with assessment for accountability only if we fundamentally rethink accountability…[as] a deeply human enterprise that depends on open sharing of information and continuing conversations…

Lorna Earl & Paul LeMahieu, 1997

Page 7: Authentic Institutional Assessment

• Need rigorous, useful peer review process that links or balances accountability & assessment perspectives

• Challenge is involving thoughtful peers, producing work meaningful for campus AND translatable to external audiences

Striking a Balance

Page 8: Authentic Institutional Assessment

Role of Underlying Beliefs in Accountability

The question of how we exhibit educational accountability hinges on beliefs about what constitutes knowledge and how it can be demonstrated.

Paul Theobald & Ed Mills, 1995

Page 9: Authentic Institutional Assessment

Assessment grows out of public discussions of the dimensions of excellence, exchanging informed clinical judgments and uncovering overlaps of or tensions between opinions. Its purpose is not accountability, but learning.

Dennie Palmer Wolf & Nancy Pistone, 1991

Page 10: Authentic Institutional Assessment

It’s Learning That Matters

Page 11: Authentic Institutional Assessment

Basically, you learn two kinds of things in college:

•Things you will need to know in later life (2 hours)…

•Things you will NOT need to know in later life (1198 hours). These are the things you learn in classes whose names end in ‘-ology’, ‘-osophy’, ‘-istry’, ‘-ics’, and so on. The idea is, you memorize these things, then write them down in little exam books, then forget them. If you fail to forget them you become a professor and have to stay in college the rest of your life.

College Learning???

Dave Barry, 1981

Page 12: Authentic Institutional Assessment

“The major aim of the freshman year

should be to win the student to the

intellectual enterprise.”

Nevitt Sanford, The Modern American College, 1962

Page 13: Authentic Institutional Assessment

…Being able to repeat facts and plug numbers into formulae to get the right answers is handy, even essential. But it is not what education is fundamentally about…

Learning should be about changing the ways in which learners understand, or experience, or conceptualize the world around them…

Learning as Transforming Understanding

Paul Ramsden

Page 14: Authentic Institutional Assessment

Hope & Loss: Real Learning Takes Courage

…It may be a great joy to discover a new and more complex way of thinking and seeing, but what do we do about the old simple world? What do we do about the hopes that we had invested and experienced in those simpler terms? When we leave those terms behind, are we to leave hope, too?

Bill Perry, 1978“Sharing in the cost of growth”

Page 15: Authentic Institutional Assessment

Exploring Assumptions and Complex Questions

Page 16: Authentic Institutional Assessment

“Assessment is…”

Page 17: Authentic Institutional Assessment

Re-Visioning Assessment from a Developmental Perspective

• Assessment as Learning

• Assessment as Meaning-making

• Assessment as Dialogue

Page 18: Authentic Institutional Assessment

Assessment as Learning

Observing & judging

performance based on explicit criteria

Providing feedback based

on those judgments

Reflecting regularly on the strengths & weaknesses of the

institution

Asking fundamental questions about learning and the

conditions for learning

Page 19: Authentic Institutional Assessment

Assessment as Meaning-Making

Ultimately, however, the real value of science may lie in uncertainty…Scientific knowledge is a body of statements of varying degrees of certainty…We scientists take it for granted that it is perfectly consistent to be unsure, that it is possible to live and not to know. But I don’t know whether everyone realizes that this is true…

Richard Feynman, 1970 from “The Value of Science”

Page 20: Authentic Institutional Assessment

What can we truly know about our students—and about each other—without some dialogue and human connection?

Assessment as Dialogue

Page 21: Authentic Institutional Assessment

Core Principles of Assessment• Assess the things that really

matter, not just the things easily assessed

• Emphasize the quality and quantity of conversations about assessment evidence

• Use a variety of approaches and multiple indicators

Page 22: Authentic Institutional Assessment

A Sampling of Assessment Approaches

• Research papers, essay tests

• Self-evaluations• Interviews• Performance tasks

(e.g., cases, problems, etc.)

• Multiple-choice tests• End of seminar surveys• Course evaluations• Student tracking

studies

• Projects, field work• Standardized tests,

surveys• Peer evaluations• Portfolio collections of

work• Video analyses• Focus groups (& SGIDs)• Student characteristic

profiles

Page 23: Authentic Institutional Assessment

Scary Issues Around Assessment

• Assuming “accountability” leads to real improvement

• Getting lost in bureaucratic minutiae

• Worshipping numbers• Obsessing over

“objectivity” & “being scientific”

Page 24: Authentic Institutional Assessment

What are our students really learning? What do they understand deeply?What kinds of human beings are they becoming—intellectually, morally, in terms of civic responsibility?How does our teaching affect that learning, and how might it do so more effectively?

Pat Hutchings & Lee Shulman,“The scholarship of teaching,” Change, Sept./Oct., 1999

…Imagine a kind of institutional research that asks:

Page 25: Authentic Institutional Assessment

Every complex question has a simple answer……and it’s wrong.

H.L. Mencken

Page 26: Authentic Institutional Assessment

Authentic Institutional Assessment

Page 27: Authentic Institutional Assessment

Authentic Institutional Assessment Emphasizes:

•Mission, not just methods

•Relationships, not just reports

•Long-term, ongoing process of self-reflection, not just a quick fix

•Faith, not just proof

Page 28: Authentic Institutional Assessment

Key Features of Authentic Institutional Assessment

Clarifying collective sense of purpose

Page 29: Authentic Institutional Assessment

Collective Sense of Purpose

Assessment...requires us to work together, and to do unfamiliar things like setting common goals and standards, devising methods of assessment, interpreting the results, and using them to improve and coordinate our teaching. [It thus] possesses all the appeal and efficiency of committee work, in particular the kind visited upon us by administrators.

Robert Holyer, Change, Sept./Oct., 1998

Page 30: Authentic Institutional Assessment

Key Features of Authentic Institutional Assessment

Clarifying collective sense of purpose

Developing shared notions of quality

Page 31: Authentic Institutional Assessment

Shared Notions of Quality

…It is through developing a shared language about student work and shared set of values and criteria that assessment can affect the larger educational system with the power reformers desire and in the direction they advocate...

Karen Sheingold & John Frederiksen, “Linking Assessment with Reform,” ETS, 1994

Page 32: Authentic Institutional Assessment

Key Features of Authentic Institutional Assessment

Clarifying collective sense of purpose

Developing shared notions of quality

Building trust & capacity

Page 33: Authentic Institutional Assessment

Public Trust

Much psychometric evaluation is ostensibly in the service of public trust, but there are more effective ways of establishing public trust than through the use of psychometrics.

Peter Johnston, 1989

Page 34: Authentic Institutional Assessment

StudentLearning

•Content knowledgein meaningful context

•Complex general education abilities

•Student attitudes/perspectives on

learning

An Overview of AuthenticInstitutional Assessment

InstitutionalEffectiveness

•Assessing mission achievement

•Understanding organizationalclimate/culture

•Analyzing support systems & processes

Builds communitythrough collaborativejudgment processes

Substantive & focused on the scholarship of

teaching and learning

ProfessionalDevelopment

Page 35: Authentic Institutional Assessment

In Conclusion…

I know I have not succeeded in answering all of your questions. Indeed, it feels as though I have not completely answered ANY of your questions. The answers I have given often serve simply to raise a new set of questions. In some ways it seems clear that you are as confused as ever. However, you’ll be happy to know that I believe firmly that you are now confused on a much higher level and about much more important things than before!

Adapted from John McMahon, Seattle Central Community College

Page 36: Authentic Institutional Assessment

Key Issues/Lingering Questions?