Institutional Researcher’s Credo

I realize that I will not succeed in answering all of your questions. Indeed, I will not answer any of them completely. The answers I provide will only serve to raise a whole new set of questions that lead to more problems, some of which you weren’t aware of in the first place. When my work is complete, you will be as confused as ever, but hopefully, you will be confused on a higher level and about more important things.

Developing & Sustaining a Culture of Institutional Research

Victor M. H. Borden, Ph.D.Professor of Educational Leadership and Policy Studies

Indiana University Bloomington

What’s Your Problem?

Select one current institutional issue that requires attention• What do you need to know to be better informed

to pursue that issue?

• What do you need to know to decide what course of action to take?

• How will you be able to tell if that action is working?

What is IR?

…research conducted within an institution of higher education to provide information which supports institutional planning, policy formation and decision making.• Saupe, 1990 – The functions of IR

What Does an IRer Do?

The institutional researcher serves higher education and, in turn, his institution through critical appraisal and careful investigation of its processes and programs.• Suslow, 1972– A Declaration on Institutional

Research

Who Does IR?

IR Staff

Anyone who gathers, evaluates, and interprets ‘data’ for use in decision making• Senior administrators and their senior staff

• Academic department chairs and program directors

• Administrative directors and managers

• Faculty working on committees and in administrative support roles

• Administrative assistants

Variations on U.S. IR Practice

From 1 to 25 person offices

Accountability

Enrollment management

Data administration and warehousing

Learning Assessment support

Survey research

Marketing and development

Planning and policy analysis

Other duties as assigned

An IR Office Case Study

IUPUI’s Office of Information Management and Institutional Research (IMIR)

IMIR mission

The mission of the Office of Information Management and Institutional Research (IMIR) is to provide and coordinate information support for planning, administering, and evaluating academic and administrative programs in ways that will continuously improve IUPUI.

IMIR Services and Functions

Management Reports and Analyses• developing for academic deans and other campus

administrators a series of management reports and analyses that integrate information from a variety of institutional and external data sources;

Ad Hoc Requests• providing academic and administrative managers with

information needed to address ad hoc problems and issues;

Information Management• creating organized, documented, and accessible data

resources based on institutional, survey, and external databases;

IMIR Services and Functions

Survey Research• conducting survey research to assess the expectations,

satisfaction, and outcomes of students, alumni, faculty, staff, employers, and other stakeholders;

Planning and Program Evaluation Support• providing direct support to specific campus, school, and

program evaluation and planning activities;

Information Technology Development• developing computer network-based systems for collecting,

accessing, and analyzing information in a more timely and cost-effective manner; and

Consulting• helping staff from other academic and administrative units to

conduct institutional research, reporting, and analysis

Principles of Effective IR (old school)

Rational, data-driven decision making processes yield more informed and successful decisions

Improved access to well-managed data sources provides decision makers with the tools they need to inform decision processes

To be useful, data must be turned into information, and information into knowledge

Data, information, and knowledge can all be managed The primary function of research and information

support is to provide information needed by decision-makers and decision-making groups

BUT

Heresy

Being “data-driven” is not, in and of itself, a good thing– e.g., “selective use” of evidence to support a

foregone conclusion Torture numbers long enough and they’ll confess to

anything

Effective use of data requires sharing diverse and often divergent perspectives– It’s not what the data say, it’s what you say about

the data– Some disagreement and dissent is important to

learning and innovation

Further Heresy

Building effective programs requires some level of irrationality and disorder• To learn from what we do requires that we

unlearn some things that we often don’t want to unlearn

As if doing this by ourselves were not difficult enough, we must do this together• Organizational learning requires communities

of practice

Beyond Data-Driven

Data-driven implies…– Rational, systematic testing of ideas through

inspection of facts– sequential, often individual decision-making

processLearning-driven implies…

– Going beyond what we already know and can do to gain new competencies

– Deconstruction and reconstruction of ideas and beliefs

– Becoming irrational to become re-rational

Beyond Data-Driven

We are engaged in the learning process as interdependent members of an organization (i.e., system)

Through collaborative, action-oriented, organizational learning, we learn (together) how to achieve organizational goals

Learning Organizations

…are organizations where people continually expand their capacity to create the results they truly desire, where new and expansive patterns of thinking are nurtured, where collective aspiration is set free, and where people are continually learning to see the whole together. (Senge, 1990)

Example: Learning Communities

Problem: Low retention rates at urban, commuter university associated with under-prepared new students, low grades, and non-existent campus life

Summer Retreat Involving Administrators, Faculty, Advisors, IR Staff– How do we increase engagement through the

classroom?– Constraint: can’t “block-schedule” working commuters

Common first-year seminar for all beginners– 100+ sections of 25 students, each with faculty member,

advisor, student mentor and librarian/technologist– Common outcomes template

E.G., LCs (2)

Bottom line evaluation shows modest impact• Improved retention and grades among “treated students”• Overall retention rate goes south

Further (qualitative) analyses reveal problems with “one-size-fits-all” approach• Focus groups; interviews; faculty fellows

Adjustments• Agreement with local CC to defer some students• School-based models to appeal to different interests• Companion supports: summer bridge, critical inquiry, structured

learning assistance, math assistance center• Block and thematic learning communities

E.G., LCs (3)

Improved Results• 10 percentage point increase in retention

over four years!• No improvement for two years!!

New problems—New approaches• Existing bridge programs adjusted to weed

out students who aren’t likely to succeedFurther increases in retention!!

The Collaborative Action-Inquiry Approach

An organizational learning framework for evidence-based practice

Action research and beyondFurther examples

Herbert Simon’s Challenge

What information consumes is rather obvious: it consumes the attention of its recipients. Hence, a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it. ~(Simon, 1971)

Kahneman & Tversky’s Challenge

Cognitive biases• We seek information based on what we know

and want to know• Biases and defense mechanisms prevent us

from processing information objectivelySelf-serving biasPrimacy and Recency

Challenges (cont.)

Decisions made within a social-organizational-political context• Rules and roles define/constrain process• Social and organizational relationships must

be accommodated

A Learning Paradigm

Planning, evaluation, and improvement all fit within a rational model

Learning incorporates uncertainty, ambiguity, and multiple styles

Individual learning and organizational learning are compatible concepts

Learning is mission critical for all educational institutions

Single- and Double-Loop Learning

Learning is the detection and correction of error (unintended consequences)

“Governing Variables” are those things what we feel are important to keep within acceptable limits

“Action Strategy” is what we do or plan to do to keep the governing variables within limits

“Consequences” are the intended and unintended outputs and outcomes• Intended: confirm our theory in use• Unintended: suggests error in our theory in use

Single-Loop Learning

Governing variables not called into question

Adjustments made to action strategies at best

Defense mechanisms can readily arise to maintain single-loop learning

GoverningVariables

ActionStrategies

Conse-quences

Double-Loop Learning

Questioning the role of the framing and learning systems which underlie actual goals and strategies

Reflection is fundamental• Basic assumptions are confronted• Hypotheses publicly tested• Falsification is sought• Ego is laid aside

GoverningVariables

ActionStrategies

Conse-quences

Model I and II Org Learning

Single- and double-loop learning at the organizational level

Model I: Organizational members prescribe to a common theory in use• Organizational policies and practices inhibit

changeModel II: Governing values, policies, and

practices promote double-loop learning

A Model I Learning Organization Governing Variables

– Tow the line– Win at all costs– Suppress negative feelings– Emphasize rationality

Action Strategies– Control environment and task unilaterally– Protect self and others unilaterally– Discourage inquiry

Consequences– Defensive relationships– Low freedom of choice– Reduced production of valid information– Little public testing of ideas

A Model II Learning Organization

Governing Variables– Valid information is most important– Free and informed choice– Shared internal commitment

Action Strategies– Shared control– Participation in design and implementation of action

Consequences– Minimally defensive relationships– High freedom of choice– Public testing of ideas

Jerome Bruner / Gilbert Ryle

• Learning about

vs.

• Learning to be

• Knowing that

vs.

• Knowing how

http://www.ukzn.ac.za/Homepage.aspx

Learning To Be / Know How

Based on collaborative practice• Communities of practice

Knowledge as inseparable from the knowerEvidence as inputSharing interpretations as processCommon priorities and strategies as output

What’s important is not what the data say,

It’s what we say and do about the data

Action Research

Continuous cycle of data collection data analysis data feedback action plans data collection

Stakeholder empowerment through active and on-going participation

Data feedback meetings promote collaboration, dialogue, and collective analysis

Active learning and discovery fostered by critical reflection process

Data-driven action plans developed = research linked to action

Action Research

1. Research Question/Evaluation Focus

2. Data Collection

3. Analysis and Interpretation

4. Presentation and Reflection

5. Action Plan

6. Evaluation of impact (Go to 2)

7. Focus on new issue (Go to 1)

Linking Research and Action

Who does what?• Decides what actions are taken?• Is responsible for effective implementation?• Can devise appropriate evaluation protocols?• Has access to or can collect appropriate

evidence?• Reviews the results and decides what to do?

What can be done to get these people to work together and in concert?

Determining the Research Question

Traditional IR• Given to researcher

– Top-down directive

– Bottom-up request• Clarification of

request– Discussion of

context and use

Action Research• Developed together

– Requester or researcher

• Specific questions often deferred until vested parties brought together

Data Collection

Traditional IR• Researcher finds

and collects data• Researcher

accountable for integrity of information

Action Research• Stakeholders have

role– Collecting data– Learning about

nuances• Shared responsibility

for integrity

Data Analysis and Interpretation

Traditional IR• Researchers

responsible through dissemination

• May consult with stakeholders to gain insight into the results

Action Research• Stakeholders involved

in stages of data analysis

• Preliminary results presented and discussed– Further analyses

shaped by those discussions

Report Presentation and Dissemination

Traditional IR• Researcher

prepares and often presents results to stakeholders

Action Research• Presentation and

report writing responsibilities shared

• Presentations involve– active discussion – facilitation of action

plan development

Follow-up

Traditional IR• Some additional

analyses may be requested or perhaps some clarification

• Often the end of the process

Action Research• Stakeholders design

action plan based on results

• Data collection included in follow-up plan

• Further lines of inquiry established for next cycle of research

Example: Evaluation of New Student Orientation

Research Question and Evaluation Focus – reassessment of goals; incoming students’

needs; impacts on knowledge, attitudes, and behaviors

Data Collection– focus groups and questionnaires, sought

perspectives of all major stakeholdersData Reporting and Feedback

– meetings with orientation leaders and faculty stakeholders

Example: Evaluation of New Student Orientation

Development of Action Plans• facilitation of dialogue and data-driven

proposalsAction

• implementation of proposed changesAssessment – on-going formative evaluation;

re-administration of process and outcome instruments

Example: Indiana Project on Academic Success (IPAS)

Research-based inquiry for enhancing academic success

Four-stage method• Assessment• Organizing• Action Inquiry• Evaluation

Supported by use of state and institutional student tracking records

Stage 1: Assessment

Compare campus assessment information to statewide assessment results; identify possible challenges

Collect additional information from campus sources, such as prior reports and studies and focus group interviews

Organize teams of administrators, faculty, professional staff, and students to identify critical challenges on the campus

Prioritize the challenges, identifying two or three that merit special attention at a campus level

Stage 2: Organizing

Coordinate the assessment and inquiry process with campus-level planning and budgeting; integrate the challenges with strategic plans; coordinate budgeting to provide necessary support.

Appoint workgroups to address critical, campus-wide challenges; consider providing release time to team leaders to work on tasks for the campus.

Coordinate the inquiry process (activities of the workgroups) with campus planning and budgeting.

Stage 3: Action Inquiry

Build an Understanding of the Challenge– What solutions have been tried in the past, and how well

did they work? What aspects of the challenge have not been adequately addressed? What aspects require more study? Develop hypotheses about the causes for the challenges using data to test the hypotheses. Do the explanations hold up to the evidence?

Look Internally and Externally for Solutions– Talk with people on campus about how they have

addressed related challenges. Consider best practices for retention and how they might be adapted to meet local needs. Visit other campuses that have tried out different approaches to the problem. How well would these alternatives address the challenge at your campus?

Stage 3: Action Inquiry

Assess Possible Solutions• Consider alternatives in relation to the

understanding of the problem developed in Stage 3, step 1. Will the solutions address the challenge at your campus? How can the solution be pilot tested? If you tried out the solution, how would you know if it worked? What information would you need to know how well it worked?

Stage 3: Action Inquiry (cont.)

Develop Action Plans– Action plans should address the implementation of

solutions that should be pilot tested. Consider solutions that can be implemented by current staff. If there are additional costs, develop budgets for consideration internally and externally. (Remember, seeking additional funds can slow down the change process.) Develop action plans with time frames for implementation and evaluation

Implement Pilot Test and Evaluate– Provide feedback to workgroups and campus coordinating

team. Use evaluation results to refine the solution. Also, evaluation can be used as a basis for seeking additional funding from internal and external sources, if needed

Institutional Support for Organizational Learning

How do we organize/staff to support the “culture of institutional research?”

There are many kinds of supports possible• Institutional research and analysis• Measurement and instrumentation• Program review and evaluation• Quality enhancement• Performance management

Planning for Improvement

Planning & Budgeting 1. Mission, Vision, Goals developed 2. Unit goals aligned 3. Annual reports on web 4. Programs based on assessable goals,

with performance indicators 5. Biennial planning/budgeting hearings

conducted

Implementation

(Everyone on campus implements goals)

Evaluation

1. Academic and administrative program reviews

2. Evaluation of process effectiveness 3. Assessment of learning outcomes

in major in general education

4. Course evaluations 5. Student assessment 6. Constituent surveys 7. Management information and analysis 8. Program cost analysis 9. Web-based evaluation tools 10. Annual campus performance report 11. NCA accreditation

Assessable

Outcomes

Inst

rum

enta

tion

Data Collection

Analysis

App

licat

ion

of F

indi

ngs Culture

OfEvidence

Improvement 1. Reporting to internal constituents

2. Demonstrating accountability to external stakeholders

3. Applying findings in campus improvement initiatives

4. Proposing improvement initiatives

5. Improving assessment methods · Web-based

data

· Electronic portfolios

Planning for Improvement

Planning & Budgeting 1. Mission, Vision, Goals developed 2. Unit goals aligned 3. Annual reports on web 4. Programs based on assessable goals,

with performance indicators 5. Biennial planning/budgeting hearings

conducted

Planning for Improvement

Evaluation1. Academic and administrative program reviews2. Evaluation of process effectiveness3. Assessment of learning outcomes

• In major• In general education

4. Course evaluations5. Student assessment6. Constituent surveys7. Management information and analysis8. Program cost analysis9. Web-based evaluation tools10.Annual campus performance report11.General and specialized accreditation

Planning for Improvement

Improvement1. Reporting to internal constituents2. Demonstrating accountability to

external stakeholders3. Applying findings in campus

improvement initiatives4. Proposing improvement initiatives5. Improving assessment methods

• Web-based data collection/reporting• Electronic portfolios

How Might You Do it?