integrating performance measures into university endeavor

Integrating Performance Measures into University Endeavor

Victor M. H. Borden, Ph.D.Associate Vice President

University Planning, Institutional Research, and Accountability (IU)Associate Professor of Psychology (IUPUI)

http://www.ohio.edu/


Becoming an Evidence-Driven Learning Organization



Or



How I Learned to Stop Worrying and Love Performance Measures



Or




Victor M. H. Borden – February 9, 2006 – Ohio University

If this were a simple matter, you If this were a simple matter, you would have figured it out long ago andwould have figured it out long ago andI wouldn’t be here. Do not expect myI wouldn’t be here. Do not expect myexplanations to be simple nor myexplanations to be simple nor myadvice to be straightforward.advice to be straightforward.This will be more like a graduate-levelThis will be more like a graduate-levelseminar than an introductory courseseminar than an introductory course





The Institutional Research Credo

I realize that I will not succeed in answering all of your questions. Indeed, I will not answer any of them completely. The answers I provide will only serve to raise a whole new set of questions that lead to more problems, some of which you weren’t aware of in the first place. When my work is complete, you will be as confused as ever, but hopefully, you will be confused on a higher level and about more important things





Why Not “Data-Driven?”Data, per se, are not what we need





NMPSID Tracked DBIRTH Age GENDER RACE MARSTAT SATV SATM ACTC HSCTYP HSGPAAS0022 1 05/24/1962 43.3 1 1 2 -9 -9 -9 0.00 -9AS0023 0 09/09/1954 51.0 1 3 1 -9 -9 -9 -9.00 -9AS0024 1 07/02/1982 23.2 0 6 1 530 530 20 1.00 -9AS0201 1 8/9/76 26 1 6 -9 -9 -9 -9 -9.00 -9AS0202 1 8/17/83 19 0 6 1 -9 -9 -9 1.00 -9AS0203 1 11/9/70 32 1 6 -9 -9 -9 -9 1.00 -9AS0204 1 8/26/74 28 1 1 1 -9 -9 20 1.00 -9AS0205 1 3/26/68 34 0 6 -9 520 460 -9 1.00 -9AS0206 1 8/31/83 19 1 6 1 -9 -9 -9 1.00 -9AS0207 1 9/8/76 26 1 6 -9 -9 -9 -9 -9.00 -9AS0208 1 11/6/55 47 1 6 -9 -9 -9 -9 0 -9AS0209 0 1/7/1985 17.6 0 3 -9 -9 -9 -9 -9.00 -9AS0210 0 3/19/1977 25.5 1 6 1 -9 -9 20 1 -9AS0211 1 11/15/1971 30.8 1 3 -9 -9 -9 -9 -9 -9AS0212 0 6/29/1971 31.2 1 1 -9 -9 -9 -9 0 -9AS0213 0 -9 -9 7 -9 -9 -9 -9 -9 -9AS0214 0 7/5/1956 46.2 1 6 -9 -9 -9 -9 -9.00 -9AS0215 1 10/21/1960 41.9 1 1 4 -9 -9 -9 -9.00 -9AS0216 0 -9 -9 7 -9 -9 -9 -9 -9.00 -9AS0217 0 7/23/1975 27.1 1 3 1 -9 -9 -9 -9.00 -9AS0218 0 7/6/1972 30.2 1 6 1 -9 -9 -9 -9 -9AS0219 1 2/23/1977 25.5 1 4 1 -9 -9 -9 1 -9AS0220 0 2/5/1983 19.6 1 6 -9 -9 -9 -9 1 -9AS0221 1 8/15/1967 35 1 2 -9 -9 -9 -9 -9 -9

AS031282 0 11/21/1961 41.8 1 -9 -9 -9 -9 -9 -9 -9AS031284 0 7/21/1962 41.1 1 -9 -9 -9 -9 -9 -9 -9AS031288 0 3/17/1982 21.5 0 -9 -9 -9 -9 -9 -9 -9AS031310 0 9/2/1974 29 1 3 -9 -9 -9 -9 -9 -9AS031327 0 5/30/1975 28.3 1 3 -9 -9 -9 -9 -9 -9AS031328 0 7/30/1968 35.1 1 3 -9 -9 -9 -9 1 2.47





If Not Data-Driven, Then What?Evidence-based practice to decide…

What to do How best to do it If it is working as desired

So that we can learn from what we do and improve

We want to be part of a Learning Organization





Learning Organizations…organizations where people continually

expand their capacity to create the results they truly desire, where new and expansive patterns of thinking are nurtured, where collective aspiration is set free, and where people are continually learning to see the whole together. (Senge, 1990)





Learning Organizations…are characterized by total employee

involvement in a process of collaboratively conducted, collectively accountable change directed towards shared values or principles. (Watkins and Marsick 1992)





OverviewLessons I’ve learned (the hard way) about

developing university performance measures

Performance measures as the “tip of the evidence-based iceberg” Going below the surface

Applying an organizational learning lensSome implications and related thoughts





Lessons LearnedEarly lessons on measurement theory1994 NDIR VolumeMeasuring Institutional Performance

Outcomes (APQC-MIPO)Developing campus PIs to link planning,

budgeting, evaluation and improvementTaking it to the next level





Measurement Theory

Inductive – Deductive Cycle





Measurement Theory Validity

Warranted assertion (Dewey) Degree to which the measure accurately represents the concept

(what you are attempting to measure)• Size of a person (weight, height, circumference, body mass)• Quality of instruction (course ratings, peer review, student learning)?

Reliability Degree to which measure faithfully represents the concept

• Course ratings taken mid-term/end-term Unless very careful attention is paid to one’s theoretical

assumptions and conceptual apparatus, no array of statistical techniques will suffice – Blalock, 1982





1994 NDIR VolumeUsing Performance Indicators to Guide

Strategic Decision Making (Borden and Banta, Eds.)





Lessons Borden and Bottrill: Where you stand on PIs

depends on where you sit Ewell and Jones: Think before you count Joengblood and Westerheijden (Europe): PIs out,

Quality Assurance in Dorris and Teeter (TQM): PIs are fine, if P stands

for Process Dolence and Norris: KPIs are the fuel of a

strategic decision engine DeHayes and Lovrinic (ABC): Show me the

money…and what you use it for doing.





Lessons (continued)Banta and Borden Criteria for Effective PIs

Start with purpose Align throughout organization Align across input, process, output Coordinate a variety of methods Use in decision making





Measuring Institutional Performance Outcomes

An American Productivity and Quality Center (APQC) benchmarking study





APQC MIPO Findings The best institutional performance measures

communicate the institution’s core values Good institutional performance measures are carefully

chosen, reviewed frequently, and point to action to be taken on results

External requirements and pressures can be extremely useful as starting points for developing institutional performance measurement systems

Performance measures are best used as “problem detectors” to identify areas for management attention and further exploration

Clear linkages between performance measures and resource allocation are critical, but the best linkages are indirect (and non-punitive)





MIPO Cont. Performance measures must be publicly available,

visible, and consistent across the organization Performance measures are best considered in the

context of a wider transformation of organizational culture

Organizational cultures supportive of performance measures take time to develop, require considerable “socialization” of the organization’s members, and are enhanced by stable leadership

Performance measures change the role of managers and the ways in which they manage





MIPO – Boiling it DownYou cannot ‘lead’ with performance

measuresPerformance measures emerge from a

broader culture of evidence, that is, they are part of something bigger





E.G.: PIs@IUPUI

www.iport.iupui.edu



Taking it to the Next Level:Accountability at Indiana

University

Articulating and Attaining Strategic Goals and Objectives





AudiencesBoard of Trustees

Most comprehensive, University-wide viewCampus accreditors and (prospective)

partners Campus-specific objectives and indicators

Targeted packaging for… Media; legislators; alumni; current and

prospective students and their parents; research agencies and collaborators





Purposes Position IU strategically Improve the effectiveness and quality of

programs and services Provide a common framework to align efforts

across campuses Communicate a clear and consistent message

about IU’s broad goals Enhance IU’s image

Define and document IU’s contributions to the state, students, and communities

Demonstrate integrity in accounting for the use of public and private resources





PrinciplesMission-centeredResearch-drivenTransparency Inclusive dimensions of excellence and

qualityEmpowerment and responsibility Influenced by “best practices”

National Commission on Accountability in Higher Education





FrameworkUniversity-wide strategic goals and core

performance indicatorsCampus performance objectives and

indicators derived from mission, aligned to university goals and core indicators

Explicit link to administrative area goals and objectives

Annual performance reports and reviews University and campuses





Advance University Distinction and Distinctiveness

Rankings and recognitions

Focused areas of distinction

Centers of Excellence

Overall campus quality

Enhance Academic Program Quality

Quality of faculty

Program accreditation and review

Teaching and learning development

Information/technology resources

Physical resources

Program demand and delivery

Improve Student Achievement and Success

Preparation and support

Access and affordability

Student engagement

Progress

Outcomes

Expand the Scope and Impact of Research and Creative Activities

Funding

Research collaborations

Faculty participation/productivity

Space and equipment

Academic Impact

Practical Impact

Advancing Indiana Economic development and impact

Cultural development and impact

Educational development

Indiana professional practice: Preparation and service

Civic engagement

Increase Operational Efficiency and Effectiveness

Finances and budgeting

Enrollment

Leadership development

Administrative overhead

Quality of administrative services to Faculty/staff/student





Limitations of Measures/Metrics Inherently imperfectOverly simplisticNot everything that counts can be

counted, and not everything that can be counted counts – Albert Einstein





Accommodating the LimitationsAn imprecise answer to the right question

is much better than a precise answer to the wrong question (paraphrasing John Tukey)

Triangulation Using multiple, convergent measures to better

reflect the underlying Performance measures as the tip of the

evidence-based iceberg





Performance Measures as the Tip of the Evidence-Based Practice Iceberg

PlanImplement

AssessImproveVe

rtic

al (h

iera

rchi

cal)

alig

nmen

t Performance measures

Horizontal (cross-unit) alignment

Evidence Based Practice





Evidence-Based PracticeCommonly used in clinical domain

Validity derived from rigorous research conducted by others and believed to generalize to other settings

For university endeavor there are limits to generalizability across settings

Focus shifts to more continuous use of process-generated data using less rigorous methods to monitor, reflect, and adjust





Methods of Evidence-Based PracticeThe many faces of evidence-based

practice Student learning outcomes assessment Program evaluation Program review Quality improvement Balanced score card Benchmarking

The role of collaborative inquiry




Victor M. H. Borden – February 9, 2006 – Ohio UniversityAdapted from Norman Jackson

The Evaluation Cycle2. ENGAGE WITH THE

PROBLEM

5. EVALUATE IMPACT* did it work as I

intended?* how did people

respond?* what were the results?

6. PLAN TO IMPROVE

1. THINK ABOUT ISSUES

3. DEVELOP RESOURCES/ STRATEGIES TO IMPROVE

4. IMPLEMENT INTERVENTIONS

* experiment

Bac

k to

the

draw

ing

boar

d

On

to s

omet

hing

els

e





The Assessment Matrix





The Support Unit Matrix





Quality Improvement ModelsAdvantages

Focus on process provides best chances for identifying points of improvement

Collaborative teams empower staff and help improve communication across units

Formulaic method and external staff support help guide and keep on track

Sample methods Penn State’s Fast Track U of Wisconsin Accelerated Improvement





PSU Fast Track





UWisc Accelerated Improvement

Define Goals and measures of successDocument processUnderstand customer needsCheck/refine goals

Design Develop potential solutionsAnalyze solutions/optionsFinalize solution develop implementation plan

Implement Inform affected peopleConduct training, if neededExecute action plans w/timeline

Follow-up Collect data to track improvementReview and refine process changesIssue final report with results

http://www.wisc.edu/improve/improvement/accel.html





Program ReviewProgram self-study, site visit by “peers”Common method for academic programs

Increasing use for administrative programsFits well with accreditation frameworkGuidelines shape tone and tenor

Content standards Review team composition

Flexibility accommodates range of inquiry orientations





Limits of Program ReviewExpensive and time-consumingCan be done with little participation

Or with a lotResults not always directly useful for

change Memorandum of understanding helpful

Episodic nature not responsive to changing environment





Balanced Score Card (BSC)Kaplan & Norton propose business model

Financial performance Customer service and satisfaction Process effectiveness and efficiency Organizational learning





BSC in Higher Education Ruben (1999)

Teaching/Learning• Programs/Courses, Student Outcomes

Service/Outreach• University, profession, alumns, state, prospective students,

families employers Scholarship/Research

• Productivity/Impact Workplace satisfaction

• Faculty/staff Financial

• Revenues/expenditures





Benchmarking Best practices in organizations sharing similar

internal work procedures HE focus often on peer or aspirational

institutions NACUBO study searched for measures APQC introduces qualitative benchmarking to

higher education Measuring institutional performance outcomes Electronically supported student services





More Complex ModelsThe Evaluation Center

Stufflebeam, Eastern Michigan Universityhttp://www.wmich.edu/evalctr/checklists/ CIPP Model Constructivist Evaluation Deliberative Democratic Evaluation Key Evaluation Checklist Qualitative Evaluation Utilization-Focused Evaluation

http://www.wmich.edu/evalctr/checklists/





Limits of Complex ModelsToo complex and expensive to be practicalThey require an…

“evaluation unit as a staff operation at a high level of the organization in order to help insulate the unit from inappropriate internal influences and enhance its influence on decision making .”

Daniel J. Stufflebeam

http://www.wmich.edu/evalctr/checklists/institutionalizingeval.htm

http://www.wmich.edu/evalctr/checklists/institutionalizingeval.htm





Collaborative Action Inquiry Continuous cycle of data collection data

analysis data feedback action plans data collection

Stakeholder empowerment through active and on-going participation

Data feedback meetings promote collaboration, dialogue, and collective analysis

Active learning and discovery fostered by critical reflection process

Data-driven action plans developed = research linked to action





Linking Research and ActionWho does what?

Decides what actions are taken? Is responsible for effective implementation? Can devise appropriate evaluation protocols? Has access to or can collect appropriate

evidence? Reviews the results and decides what to do?

What can be done to get these people to work together and in concert?





A Learning ParadigmTypical data-driven focus supposes

rational worldLearning incorporates uncertainty,

ambiguity, and multiple styles Individual learning and organizational

learning are compatible conceptsEvidence-based practice is compatible

with learning approach





Single- and Double-Loop Learning Argyris and Schön Learning is the detection and correction of error

(unintended consequences) “Governing Variables” are those things what we

feel are important to keep within limits “Action Strategy” is what we do or plan to do to

keep the governing variables within limits “Consequences” are the intended and

unintended outputs and outcomes Intended: confirm our theory in use Unintended: suggests error in our theory in use





Single-Loop LearningGoverning variables not called into

questionAdjustments made to action strategies at

bestDefense mechanisms can readily arise to

maintain single-loop learningGoverningVariables

ActionStrategies

Conse-quences





Double-Loop Learning Questioning the role of the framing and learning

systems which underlie actual goals and strategies

Reflection is fundamental Basic assumptions are confronted Hypotheses publicly tested Falsification is sought Ego is laid aside

GoverningVariables

ActionStrategies

Conse-quences





Model I and II Org LearningSingle- and double-loop learning at the

organizational levelModel I: Organizational members

prescribe to a common theory in use Organizational policies and practices inhibit

changeModel II: Governing values, policies, and

practices promote double-loop learning



John Seely Brown & Paul DuguidThe Social Life of Information(2000) Harvard Business School Press

Organizational Learning and Communities-of-Practice:Toward a Unified View of Working, Learning, and Innovation. (1991) Organization Science, 2(1), 40-57.





Learning To Be / Know HowBased on collaborative practice

Communities of practiceKnowledge as inseparable from the

knowerEvidence from a variety of sources,

including practitioner experienceSharing interpretations as processCommon priorities and strategies as

output





Learning is GoodWe promote (lifelong) learning for studentsWe seek to contribute to the creation of

knowledge within our disciplines and professions

What about in our practice as... Classroom teachers Conferrers of degree credentials Managers and administrators Support staff





The Learning/Performance Measure Conundrum

If our general objective is to collectively learn how to do our work better, then we must accept that our current thinking, practices, structures, etc., need to change

Our current best thinking about what measures reflect progress toward desired changes may change through the learning process

We should not be rigid about our performance measures but rather allow our evidence-based collaborative learning efforts to guide their evolution





Implications for Faculty/Staff/Org Development

There are many viable ways to integrate inquiry into organizational practices

Administrative support focus may need to shift from information provision and toward collaborative inquiry

Someone needs to focus on how this all fits together The institutional portfolio provides one such

mechanism





Implications for Information UseData sourcesTypes of needsTypes of usersSources of informationTools for user needs





Data Sources Sources of evidence

Documented Provider/practitioner experience User/client experience Contextual

Derived from Institution’s operational information systems Student, Human Resource, Finances Space, program inventory, courseware

Surveys Students, faculty, staff, prospects, community

External data sources Federal and State (K-16) education data, national efforts (CDS,

rewards and recognitions, media) Census, labor, workforce development, licensing boards





Types of Information Needs Operational

Directly support the ongoing operation of a system Formatted presentations of transactional Often use data from a single operational domain

Tactical Monitor and respond quickly to a variety of short-term situations Typically more aggregate (less granular) than operational reports Includes both recurrent and ad hoc information needs Often requires merging data from multiple operational domains

as well as data from non-operational sources Strategic

Focuses on higher level policy and practice issues, often with longer timeframes

Often requires more significant analysis of institutional, survey, and external data sources





User Roles Casual

occasional use that demands relatively little technical expertise

Recurrent more frequent use but modest technical expertise OR

insufficient time to employ technical skills Power

modest to frequent use with capacity for using more complex technical systems

Individuals may occupy different roles at different times





Information Needs and UsersType of User

Casual Recurrent Power

Operational

Tactical

Strategic

Type

of U

se

Pre-packaged Operational

Reports

Report ModulesWith Parameter

Choices

ODBC Access to Data Warehouse

Tables

Research BriefsAnd Analyses

Web-basedReport

Generators

OLAPTools





Implications for IT Analytic data warehouse is essential, but Think more broadly about data sources

Not just enterprise system as we now know it Data from courseware platform Mechanisms for collecting “droppings” from other

important activities• Faculty vitae and annual reports• Portfolios of faculty and student work• Civic engagement inventory

Access/reporting technology should focus on enabling “value-added resellers” to deliver to broad range of users





Responsibility-Centered Budgeting

Similar to Churchill’s opinion of democracy It has been said that democracy is the worst

form of government except all the others that have been tried

Concerns about changing to RCB It changes everything and yet nothing really

changes [I] have known a great many troubles, but

most of them never happened ~Mark Twain





Parting Thought It is good to have an end to journey

towards; but it is the journey that matters in the end - Ursula Le Guin



integrating performance measures into university endeavor

Documents

university endeavorvictor

ohio universityif

ohio universitywhy

new set of questions

simple matter

evidencebased practice

hard way

total employee involvement