october 31, 2006sonoma state university, rohnert park, ca 1 (un)intended consequences: wasc...
Post on 21-Dec-2015
215 views
TRANSCRIPT
October 31, 2006 Sonoma State University, Rohnert Park, CA
1
(Un)Intended Consequences: WASC Accreditation and Implications of the Report of the Secretary of Education’s Commission on the Future of Higher Education
Barbara D. Wright,Associate DirectorWASC
October 31, 2006 Sonoma State University, Rohnert Park, CA
2
An overview …
Where assessment is today Where WASC is today Report of the Spellings Commission Implications, possibilities Discussion
October 31, 2006 Sonoma State University, Rohnert Park, CA
3
The pendulum has swung . . .
1983-1990: heavy focus on assessment for accountability, ascertaining level of achievement
1990-2003: more focus on assessment for quality improvement, less on quality assurance
2003 – present: renewed focus on quality assurance, standard-setting
October 31, 2006 Sonoma State University, Rohnert Park, CA
4
Shifts in our understanding of assessment
Isolated facts, skills
Memorization, reproduction
Comparing performance against other students
A full range of knowledge, skills, dispositions
Problem solving, investigating, reasoning, applying, communicating
Comparing performance to established criteria
from . . . to
October 31, 2006 Sonoma State University, Rohnert Park, CA
5
Shifts in assessment, cont. Scoring right,
wrong answers
a single way to demonstrate knowledge, e.g. m/c or short-answer test
Simplified evidence
Looking at the whole reasoning process
Multiple methods & opportunities, e.g., open-ended tasks, projects, observations
Complex evidence
October 31, 2006 Sonoma State University, Rohnert Park, CA
6
Shifts in assessment, cont. A secret,
exclusive & fixed process
Reporting only group means, normed scores
Scientific A filter An add-on
open, public & participatory
Disaggregation, analysis, feedback
Educative A pump Embedded
October 31, 2006 Sonoma State University, Rohnert Park, CA
7
Shifts in assessment, cont. “teacher-proof”
assessment
Students as objects of measurement
episodic, conclusive
Reliability
Respect, support for faculty & their judgments
Students as participants, beneficiaries of feedback
continual, integrative, developmental
Validity
October 31, 2006 Sonoma State University, Rohnert Park, CA
8
My pet assessment peeve . . .
Lots of focus on how, on process Some examples of use Nothing on the actual FINDINGS
Why not?
October 31, 2006 Sonoma State University, Rohnert Park, CA
9
The 2001 WASC Standards:
I: Defining Institutional Purposes and Ensuring Educational Objectives
II: Achieving Educational Objectives Through Core Functions
III: Developing and Applying Resources and Organizational Structures to Ensure Sustainability
IV: Creating An Organization Committed to Learning and Improvement
October 31, 2006 Sonoma State University, Rohnert Park, CA
10
What does WASC mean by EE?
Educational infrastructure (leadership, expertise, processes, resources)
Educational outcomes at all levels (student, program, institution, organization)
A culture of inquiry and evidence
A system of quality assurance (intentional, holistic, aligned) for student and organizational learning that demonstrates . . .
October 31, 2006 Sonoma State University, Rohnert Park, CA
11
An evolution in WASC thinking:
FROM Standards I & III as capacity issues, II & IV as educational effectiveness . . .
TO seeing ALL four Standards as having both a capacity and an educational effectiveness dimension (see “Two Lenses on Two Reviews”)
October 31, 2006 Sonoma State University, Rohnert Park, CA
12
An evolution . . .
FROM requesting a lot of different kinds of data about EE . . .
TO privileging direct evidence (i.e., student work and performances), although descriptive data and indirect evidence are still important
October 31, 2006 Sonoma State University, Rohnert Park, CA
13
An evolution . . .
FROM an expectation of program review . . .
TO insistence on robust program review, including an assessment plan and a focus on student learning as a central component of a broader review
October 31, 2006 Sonoma State University, Rohnert Park, CA
14
An evolution . . .
FROM expecting programs to define standards for student learning . . .
TO asking for evidence that students achieve those standards
October 31, 2006 Sonoma State University, Rohnert Park, CA
15
And finally. . .
Growing awareness of the need for member institutions – and WASC – to communicate more fully with the broad public About student learning About the results of accreditation
reviews
October 31, 2006 Sonoma State University, Rohnert Park, CA
16
The Spellings Commission Report --
A Test of Leadership. Charting the Future of U.S. Higher Education
Fall 2006
October 31, 2006 Sonoma State University, Rohnert Park, CA
17
The context:
Boomers – “the most highly educated generation in US history” – are about to retire
Replacement workers will need higher educational levels for a knowledge-based global economy
College costs are rising, results declining The “education gap” and “ambition gap” (Th.
Friedman) put US at competitive disadvantage Prosperity, national security are at risk
October 31, 2006 Sonoma State University, Rohnert Park, CA
18
The Commission on the Future of Higher Education – Issue Paper #2
National Assessment of Adult Literacy: less than 1/3 of college graduates could read complex texts, make inferences
National Survey of America’s College Students: significant numbers of college grads (20-30%) have only basic quantitative skills
Employers: college grads lack skills for the workplace
“Accountability/Assessment” by Charles Miller & Geri Malandra
October 31, 2006 Sonoma State University, Rohnert Park, CA
19
“Accountability/Assessment” by Charles Miller & Geri Malandra
Colleges’ standards are “fuzzy” HE is “focused inward,” doesn’t
communicate goals, outcomes to the public “There are no commonly used tests or
other assessments” “College courses are not designed to
foster critical thinking and problem solving, which are hard to address with lecture formats and multiple-choice tests.”
October 31, 2006 Sonoma State University, Rohnert Park, CA
20
“Accountability/Assessment” by Charles Miller & Geri Malandra
AAC&U’s approach (Our Students’ Best Work, 2004; and Liberal Education Outcomes, 2005)
Accreditation’s focus on student learning SHEEO’s endorsement of a focus on student
learning Business Higher Education Forum’s endorsement
of multiple methods and linking with K-12 education
New instruments (NSSE, CLA, ETS’ MAPP)
“Promising efforts” include:
October 31, 2006 Sonoma State University, Rohnert Park, CA
21
Spellings Commission concerns . . . HE grads too often deficient in
Reading and writing Math Critical thinking Problem solving Ability to be life-long learners
Other countries overtaking the US in HS graduation rates College-going rates Baccalaureate graduation rates
October 31, 2006 Sonoma State University, Rohnert Park, CA
22
Concerns, cont. . . .
Lack of alignment between HS and HE expectations for learning
Lack of information available to students, parents about costs, comparative quality of colleges
Achievement gaps (racial, ethnic, SES) HE insufficiently innovative, adaptable,
efficient, focused on outcomes Accreditation part of the problem, not a
solution
October 31, 2006 Sonoma State University, Rohnert Park, CA
23
On quality, transparency, and accountability:
“Despite increased attention to student learning results by colleges and universities and accreditation agencies, parents and students have no solid evidence, comparable across institutions, of how much students learn in colleges or whether they learn more at one college than another… Accreditation reviews are typically kept private, and those that are made public still focus on process reviews more than bottom-line results for learning or costs” (p.13-14).
October 31, 2006 Sonoma State University, Rohnert Park, CA
24
Commission’s Recommendations …
Seamless alignment of high school and college curricula
Performance outcomes as “the core” of accreditation, over inputs or processes
Technology and new pedagogies used to get more learning at lower cost
A “culture of accountability and transparency” with “consumer–friendly” information
October 31, 2006 Sonoma State University, Rohnert Park, CA
25
Recommendations, cont. “Institutions should measure and report
meaningful student learning outcomes” (CLA, NSSE, MAPP noted)
Availability of test scores, certification and licensure rates, time to degree, other data should be “a condition of accreditation”
User-friendly data should allow inter-institutional and interstate comparisons
October 31, 2006 Sonoma State University, Rohnert Park, CA
26
The challenge . . .
To find ways to assess student learning – and report the findings – in ways that support education and do no harm.
Standardized tests are a legitimate part of the picture, but they can’t be the whole picture.
October 31, 2006 Sonoma State University, Rohnert Park, CA
27
Choice of assessment method matters.
Students value and learn what we teach and test.
How we teach and test matters as much as what
What and how we assess also matters.
We get more of what we test or assess, less of what we don’t.
October 31, 2006 Sonoma State University, Rohnert Park, CA
28
Higher-order thinking …( adapted from L. Resnick, 1987)
It’s nonalgorithmic, i.e., the path of action is not fully specified in advance.
It’s complex, i.e., the total path is not “visible” from any single vantage point.
It often yields multiple solutions, each with costs and benefits.
It requires nuanced judgment and interpretation
It involves application of multiple criteria, which may conflict with one another
October 31, 2006 Sonoma State University, Rohnert Park, CA
29
Higher order thinking, cont …
It often involves uncertainty; not everything about the task is known or can be.
It requires self-regulation; someone else is not giving directions.
It involves making meaning, discerning patterns in apparent disorder.
It is effortful: the elaborations and judgments required entail considerable mental work and are likely to take time.
October 31, 2006 Sonoma State University, Rohnert Park, CA
30
Other ways to think about higher-level learning . . .
Bloom’s taxonomy (knowledge, comprehension, application, analysis, synthesis, evaluation)
Perry Scheme of Intellectual Development (dualism, multiplicity, relativism, commitment)
Biggs’, Entwistle’s “surface” vs. “deep” learning
Constructivist approaches ???
October 31, 2006 Sonoma State University, Rohnert Park, CA
31
Methods for complex outcomes …
are direct are open-ended focus on essentials, principles pose authentic, engaging tasks require meaning-making, judgment require active expression are scored for understanding, not just
regurgitation
October 31, 2006 Sonoma State University, Rohnert Park, CA
32
Methods for complex outcomes include …
Portfolios Capstones Performances Common assignments Secondary readings Course management programs Local tests, comps in the major The CLA
October 31, 2006 Sonoma State University, Rohnert Park, CA
33
Four dimensions of learning --
What students learn (cognitive as well as affective, social, civic, professional, spiritual and other dimensions)
How well (thoroughness, complexity, subtlety, agility, transferability)
What happens over time (cumulative, developmental effects)
Is this good enough?
October 31, 2006 Sonoma State University, Rohnert Park, CA
34
Thinking About Standards . . . Absolute standards: the knowledge/skill level
of champions, award winners, top experts Contextual standards: appropriate
expectations for, e.g., a 10-year old, a college student, an experienced professional
Developmental standards: amount of growth, progress over time, e.g., 2 years of college, 5 years
(Institutional, regional, national standards?)
We know “what,” but do we know “how well,” or what is “good enough”? How might we know?
October 31, 2006 Sonoma State University, Rohnert Park, CA
35
Once assessment is in place. . .
We need to not only use our findings to make improvements;
we need to reveal our findings and use them to set standards for
college-level learning
This is the next frontier in assessing and improving students’ learning and development. If we don’t do it, others will.
October 31, 2006 Sonoma State University, Rohnert Park, CA
36
We’re not starting from zero – assessment is a foundation to build on:
Outcomes – pervasive Gathering student work – in process Analysis and use – coming along Public sharing of findings – ? Developed institutional standards – a
handful? Cross-institutional standards, rubrics
and examples of student work – not yet, but why not?
October 31, 2006 Sonoma State University, Rohnert Park, CA
37
We all have implicit standards, e.g., for
Remedial writing, math Freshman writing, math Course-level judgments about grades,
especially Ds, Fs Competency-based curricula
As we did with outcomes, now we need to make the implicit explicit and collectively owned.
October 31, 2006 Sonoma State University, Rohnert Park, CA
38
What do students know, and how do we – all – know they know it? Together we …
What do students know? Is it good enough? A course of action might be to…
Analyze student work for quality, proficiency Determine the effective ceiling & floor of our
students’ learning at different points Compare it to work at other institutions Begin to reach a collective sense of “college-
level learning” in a few key areas And communicate that level of learning to
students, parents high school faculty and students policy makers the general public
October 31, 2006 Sonoma State University, Rohnert Park, CA
39
It won’t be easy. Plenty of factors can work against the use of standards.
Traditional proxies: requirements, credits, seat time
Grade inflation The unspoken faculty/student truce (I won’t
bother you if you don’t bother me.) Insufficiently rigorous AP and dual
enrollment courses Lack of shared culture, trust The complexity of the whole issue: what
standards, for whom, applied how, with what consequences?
October 31, 2006 Sonoma State University, Rohnert Park, CA
40
Are parents & students really interested in learning?
“Research suggests … otherwise… College students tend to look upon knowledge and ideas less as ends in themselves and more as a means toward accomplishing other goals, such as … achieving success in their careers … Rather than measures of student learning, indicators such as graduation rates, employment rates, and cost of attendance are some of the “consumer information” colleges and universities should make readily available” (NASULGC, 2006).
October 31, 2006 Sonoma State University, Rohnert Park, CA
41
Having those standards means We can have substantive
conversations with students, parents, employers, policy makers, and the public.
We have not just numbers (data) but specific, substantive, outcomes-based information to inform conversations across the educational spectrum, K-16 and beyond.
October 31, 2006 Sonoma State University, Rohnert Park, CA
42
Parade Magazine? Are you kidding?
respond to public need, interest inform the public, build intellectual
capital in the wider society – and among policy makers
allow parents, high school, others to internalize and reinforce our message
No. Communication is essential. It will
October 31, 2006 Sonoma State University, Rohnert Park, CA
43
Communication, cont. be educative for students, show them
what they have mastered or still need to do
shift some responsibility for education to these others
free us in HE from taking ALL the responsibility for a complex process that can never succeed if it is ours alone.
October 31, 2006 Sonoma State University, Rohnert Park, CA
44
Questions for discussion …
How could your institution use assessment to set standards? What efforts could you build on?
How might commercial and local approaches be combined?
What are the benefits/drawbacks of Institutional standards? Regional standards? Consortial standards? State or national Standards?
Should we be doing this at all?
October 31, 2006 Sonoma State University, Rohnert Park, CA
45
What about WASC?
A user group for campuses using NSSE A user group for campuses using the CLA A group to explore cross-institutional
rubrics for Capstones Portfolios
A group to inform WASC about institutional experience with standardized tests
WASC plans several projects in the coming year: