Assessing for Transformation
Gary BrownThe Center for Teaching, Learning & Technology
We should not expect the guidance for change of this magnitude—in institutional culture and values—to come from the faculty ranks. After all, faculty are deeply rooted in the traditional values of higher education. Fundamentally, this is a leadership issue. --Carol Barone, Vice President
Educause
Agenda
• Assumptions—the need for transformation• To arms, to arms…• About those jobs• What Transformation is NOT• What Transformative Assessment is NOT
• Stories• Goals, Activities, & Processes (GAPs)• Inquisitions and Assessments• Critical Thinking—the rubric paradigm• The Sleeping Bear & Principles of Transformative
Assessment• TAPS
To Arms, To Arms…
• PHOENIX • Phoenix Online is growing faster than
any of the university's 40 physical campuses
• 1,700 online support specialists• 7,000 mostly part-time faculty • 49,400 students
• 70% increase• $64.3 million in net income
Phoenix —from the Chronicle
• “Some critics note that Phoenix strips faculty members of their central role in higher education.” “One of the greatest risks we face as a nation in the growth of new educational providers is the unbundling of teaching, research, and service functions”—Art Levine
• However, counters Phoenix, “Professors at traditional universities who attempt online education are learning as they go, and often give students a bad experience as a result.”
Cost
• The USA currently has 3,500 colleges and universities with an enrollment of 14 million students.
• We spend $175 billion on education.• $12,500 per student • But now, eleven International Mega-
universities serve 3 million students.
Corporate Tax Support
0
20
40
60
80
100
1957 1990 2022
Corporate Share of Taxes
Job Growth: 1992-2005*• Predicted Percentage Increase…
020406080
100120140
Retail Cashiers Clerks Truck
Drivers
Nurses Aids Waiters Home Health Computers
*US Department of Labor Statistics (1993)
Job Growth: 1992-2005*
• Or in Total Numbers
0
200,000
400,000
600,000
800,000
Retail Cashiers Truck Drivers Home Health Technology
*US Department of Labor Statistics (1993)
Job Market Projections*
020406080
100
ProductionService
People Service Symbolic Analysts
*US Department of Labor Statistics (1993)
The Real StoryWe need:
• 9 cashiers for every technical worker.• 1.5 janitors for all the lawyers, accountants
investment bankers, stock brokers, & computer programmers combined.
• “The projected shift … in educational requirements… can be accomplished if those entering the labor force have…one fourth grade level more education than those retiring from the labor force.”
Economic Policy Institute
College Graduates @ Work
0
50,000
100,000
150,000
200,000
Retail Housekeeping Truck Drivers
*US Department of Labor Statistics (1993)
Already, In Fact…
% College Graduates working in Blue Collar Jobs
0
20
40
1968 1990*US Department of Labor Statistics (1993)
What Transformation Is NOT
Attrition Rates in WSU Distance Courses
0
10
20
30
40
fall1999
spring2000
fall2000
spring2001
fall2001
spring2002
Online Video
%
Student Enrollments in WSU Online Learning Spaces
0
4000
8000
12000
16000
20000
1998 1999 2000 2001 2002
Transformation?
What Transformative Assessment is NOT
Evaluations as Barriers to Improvement
• Many evaluation instruments are subtly biased in favor of traditional instruction.• They often penalize innovators,
measuring what faculty aren’t doing but failing to measure what they are doing.
Purpose of Evaluation is Not Clear
• Most evaluation focuses on personnel decisions.
• Inconsistent interpretation and use of evaluation results.
• Little focus on areas where that might best benefit from feedback
• Evaluation usually occurs at the very end of a semester
—Too late for change
Students Are in the Dark, Too
• The traditional evaluation process is not designed to help students become more involved with what will really help them learn.
• Students don’t take evaluations seriously.
Teaching-Learning Practices are NOT Linked to Learning
• Course evaluations rarely focus on practices that have been shown to produce better learning outcomes.
• Assessment rarely improves learning.
No Link Between
Evaluation & Faculty Development
• Inadequate support for addressing results. Literature from past 10-15 years emphasizes staff development as key to effective evaluation practices that promote teacher growth and improvement” (1997, Annunziata in Stronge, pp 289)
• Administrators frequently calculate an average score for all evaluation items and then rank faculty.
• The method does not foster dialogue among faculty about good practices.
The Goals, Activities, and Practices
• Invite faculty to join in the process of formulating the assessment.
• A series of three short, online surveys designed to provide formative assessment: one instructor survey and two student surveys.
The 3 Survey Goals1. To help faculty gather useful data about their
teaching goals, values and strategies…
2. To help faculty learn how those strategies address and influence their students’ goals, values, and learning behaviors…
3. And to examine the interaction between goals and practices that faculty might share with each other
—and so establish a culture of evidence.
GAPs: Encouraging Participation• Brown Bags - to initiate the conversation on assessing
teaching and learning.• We emailed each faculty approximately 10 times • Student surveys were linked to course sites of
instructors who answered the instructor survey.• We analyzed and returned results promptly...• Regular updates were sent to instructors with their class
results and overall results.• We offered and have co-authored papers with faculty...• We have talked with chairs, deans, and the assessment
coordinator’s to include GAPS in teaching portfolios.• We wore out our shoes…
Sample Findings• Significant mismatch between faculty and student goals• Online courses are significantly more effective than video based courses• Designed courses are significantly more likely than courses not formally designed to evidence principles
of good practice• Faculty motivation predicts perceptions of efficacy
• Faculty who want to keep abreast of the scholarship of T & L report that online learning is positive• Faculty who teach online primarily for the money …
• Student age and gender predict perceptions of testing efficacy
Time & CostEstimated Hours
01020
304050
607080
90
Design Development Delivery Assessment
Actual Design Target
The Assessment Gold Standard
Participants Who Used GAPs Data to Transform….
563
10655 16 2
0
300
600
Invited faculty Faculty who tookinstructor survey
Faculty w/ studentresponses
Faculty w/ >50%student response
rate
Faculty who usedthe data
However….• One of the two programs endured a budget related,
faculty initiated inquisition…..• Demonstrated, by virtue of GAPS, systematic, formative
data gathering and responsiveness to that data…• Demonstrated evidence of good practice.• Demonstrated the “creative use of technology” and gains in
critical information literacy• Demonstrated the program was cost effective
• The program improved freshman retention• Demonstrated the program improved student grades,
including “special admits” expected to struggle.
The Learning Context: Student Performance @ WSUCum GPA by Admissions Quartiles
1st Quartile2nd Quartile3rd Quartile 4th Quartile
English
GenEd
Math
FS
0
1
2
3
4English
GenEd
Math
FS
Which has resulted in….• The faculty committee that initiated the inquiry was commended by
administration for its attentiveness to issues of quality….• They have subsequently been invited to examine other programs… • Which has transformed the CTLT• We are seeing the inklings of a culture of evidence sprouting in a rough,
research dominated terrain…
small steps….
• Guide faculty grading
• Guide student learning
• Provide measures of growth
The Rubric Paradigm
Dimensions of Critical Thinking1. Identifies and summarizes the problem/question at issue (and/or the
source's position).
2. Identifies and presents the STUDENT’S OWN perspective and position as it is important to the analysis of the issue.
3. Identifies and considers OTHER salient perspectives and positions that are important to the analysis of the issue.
4. Identifies and assesses the key assumptions.
5. Identifies and assesses the quality of supporting data/evidence and provides additional data/evidence related to the issue.
6. Identifies and considers the influence of the context * on the issue.
7. Identifies and assesses conclusions, implications and consequences.
Critical Thinking and Measures of Growth
Emerging____________________________ Mastering
Identifies and summarizes the problem/question at issue (and/or the source's position).
Does not identify and summarize the problem, is confused or identifies a different and inappropriate problem.
Does not identify or is confused by the issue, or represents the issue inaccurately.
Identifies the main problem and subsidiary, embedded, or implicit aspects of the problem, and identifies them clearly, addressing their relationships to each other.
Identifies not only the basics of the issue, but recognizes
nuances of the issue.
Critical Thinking:8 Courses: 4 with CT, 4 w/o CT
0
1
2
3
4
5
6
Problem StudentView
Other Views
ContextEvidence
AssumptionsImplications
Average
Upper Division CT Lower Division CT No CT
Critical Thinking:8 Courses: 4 with CT, 4 w/o CT
2
2.5
3
3.5
4
Problem StudentView
Other Views
ContextEvidence
AssumptionsImplications
Average
Upper Division CT Lower Division CT No CT
Critical Thinking:One course—two semesters
0
1
2
3
4
5
6
Problem StudentView
Other Views
ContextEvidence
AssumptionsImplications
Average
Entomology--spring 99 Entomology--spring 00
Faculty Development
• The critical thinking rubric was valuable for helping faculty articulate their goals and communicate expectations to students.
• Faculty who used the rubric were enthusiastic and expressed plans to integrate the rubric more intensively in future courses.
Critical Thinking Study—Results
• Gains in courses when rubric is used—when the faculty in this project integrated the WSU Critical Thinking Rubric into their instruction and assessment, evidence of student gains in critical thinking increased dramatically.
• Gains from freshmen to junior years—Critical thinking was significantly higher among juniors than among freshmen.
• But even the writing of juniors had only a mean of 3.1 on a 6 point scale.
Additional Findings & Implications
• The dimension of least gain was in students’ abilities to articulate their own viewpoints.
• The greatest gains by juniors reflect improved abilities to analyze issues from multiple perspectives.
• Comparisons to WSU’s writing assessment— As critical thinking scores rise, writing placement
scores and portfolio exam scores sink...
• The faculty questionnaire revealed a focus on grading over fostering critical thinking for broader life-long learning.
Transformation?
The Sleeping Bear
• Those most oblivious of the teeth are often the first to fodder…
• The growling bear will elicit benchmarks & comparisons
• The bear will feast on standardization
• But bears are omnivorous…
Emerging Principles of Transformative Assessment
• Institutional leadership is imperative• Assessment focuses on institutional efforts to
provide students with rich learning experiences• Assessment includes student reports of their own
increasingly unique experiences• Qualitative measures are valued and may be
supported by quantitative measures• Emphasizes student learning defined by
development and change over time• Dissemination engages the responsibility for
shaping as well as reflecting society needs• “High Standards” is not the same as Standardization
Dimensions of Transformation
1. Purpose2. Data Acquisition3. Application4. Dissemination
Criteria for Rating Transformation
Rater: Project:
* Faculty * Designer
* Student * Assessment Specialist
* Community Colleague * Administrator
* Other _________
Rating
Assessment Purpose
Data & Data Acquisition
Application
Dissemination
Bonus—Faculty Rewarded for Assessment
Total Score
Comments:
The Scoring Form
U#1 U#2 U#3 U#4 U#5 DimensionAverage
Purpose 3 3.1 3.6 4 3 3.34Data 2.4 3.5 3.8 3.5 3 3.24Application 2.5 3.4 3.7 3.5 4.5 3.52Dissemination 3.3 3 3.5 2 4.5 3.26University Average
2.8 3.3 3.7 3.3 3.8 3.343.38
Pilot Findings
Pilot Findings
0
1
2
3
4
5
6
PurposeData
ApplicationDissemination
Average
U # 1
U # 2
U # 3
U # 4
U # 5
Key CommentsU#1
• Doesn’t stretch student learning or acknowledge what students may bring.
• Focus satisfies industry’s demand for skills.
• While faculty are included in this process, audience is clearly the upper echelon of the institution and future employers.
Key CommentsU#2
• Not many rewards for assessment.
• No mention of disseminating results beyond unit; no feedback to faculty and students.
• Limited to classes.
Key Comments U#3
• Assesses key concern: learning to learn.
• No plan to apply or fund results of findings. Faculty support neglected.
• Feedback to institution and external agencies is minimal—how are constituents engaged in process?
Key Comments U#4
• Design for infrastructure for hybrid classes—assumptions that tech alone equates with transformation
Key Comments U#5
• Good start but…