architecting evaluation and assessment using a design ... · architecting evaluation and...

Architecting Evaluation and Assessment—Using a Design-based

Approach to Develop a Complete and Systematic Approach

Presenters:

Dr. Douglas Gilbert – University of the Rockies

Dr. Peter Pintus – University of the Rockies

Dr. Olin Oedekoven – Peregrine Academic Services

@ACBSPAccredited #ACBSP2015

Agenda

The Challenge of Assessment

ACBSP Assessment Requirements

Using a Design Approach to Architect a Model

Multi-Tiered Assessment Instrumentation

Examples

Conclusions

2 @ACBSPAccredited #ACBSP2015

The Challenge of Assessment

ACBSP Assessment Requirements

Using a Design Approach to Architect a Model

Multi-Tiered Assessment Instrumentation

Examples

Conclusions

3 @ACBSPAccredited #ACBSP2015

The Assessment Challenge: Four Key Questions

1. Do we know what graduates need to know and be able to do?

2. Does the curriculum provide what graduates need to know and be able to do?

3. Are students on a learning path for relevant knowledge, skills, and abilities?

4. Do graduates have the competencies demanded by employers?

4 @ACBSPAccredited #ACBSP2015

The Challenge of Assessment

ACBSP Assessment Requirements

Using a Design Approach to Architect a Model

Multi-Tiered Assessment Instrumentation

Examples

Conclusions

5 @ACBSPAccredited #ACBSP2015

Key ACBSP Assessment Requirements

• Criterion 4.1 – Must have a “learning outcomes assessment program”

a. Program learning objectives (PLOs) b. An assessment process c. Internal assessment approach d. External assessment approach e. Formative and summative assessment approach

• Criterion 4.2 – Trends

• Criterion 4.3 – Time series benchmarked comparisons

and also…

6 @ACBSPAccredited #ACBSP2015

Setting strategic direction

•Standard 2 – “Process … to better

address key student and program

performance requirements”

7 @ACBSPAccredited #ACBSP2015

The Challenge of Assessment

ACBSP Assessment Requirements

Using a Design Approach to Architect a Model

Multi-Tiered Assessment Instrumentation

Examples

Conclusions

8 @ACBSPAccredited #ACBSP2015

What is a “complete and systematic approach”?

• Complete — “[H]aving all necessary parts : not lacking anything”

(Merriam-Webster.com. 2015. http://www.merriam-webster.com)

• Systematic — “The term "systematic" refers to approaches that are repeatable

and use data and information so learning is possible.” (ACBSP Bac/Grad Standards and Criteria—Rev J., December 2014, p. 71).

— “Well-ordered, repeatable, and exhibiting the use of data and information so that learning is possible.” (Baldrige Excellence Framework 2015-16, p. 53).

• Approach — “[T]he methods used by an organization to address the criteria

requirements. Approach includes the appropriateness of the methods to the requirements and the effectiveness of their use.” (ACBSP Bac/Grad Standards and Criteria—Rev J., December 2014, p. 62).

9 @ACBSPAccredited #ACBSP2015

Is this your assessment design?

10 @ACBSPAccredited #ACBSP2015

A Problematic Assessment System

11

Data: Student surveys and

classroom work, accumulated by

faculty in classroom

Uses: Individual Faculty

Efforts

Data: Departmental studies, e.g.

accounting, OB, etc.

Uses: Departmental Improvement

Data: Institutional res--

• Surveys (NSSE, EOCS, etc.)

• Std-ized tests • IPEDS • Alumni

Uses: Reporting

Uses: Program reviews

Data: Programmatic

assessments, e.g. portfolios.

Uses: Program reviews

Data: Academic support

and co-curricular, e.g. library, IT, student

affairs, etc.

Uses: Individual dept. improvement

Uses: Program reviews

Faculty Governance

Curriculum approval

Provost / VPAA

Assessment Advisory Committee

Dean / Dept. Chair

Program Chairs

Strategic Planning

VPs

Institutional Research

Adapted from Wolvoord (2010) @ACBSPAccredited #ACBSP2015

Common “Disconnects”

• Structure — Academic data islands of classroom, departments, and programs

— Institutional and instructional data not integrated

• Process — Blurred or non-existent assessment roles and responsibilities

— Non-systematic or isolated cycles of improvement

• Technology — “One-off” faculty classroom technology approaches

— Wide range of surveys and assessment automation

• People — Separated, disjointed assessment responsibilities

— Holistic knowledge management competencies lacking

— Little cross-functional / inter-departmental perspective

12 @ACBSPAccredited #ACBSP2015

A Design-Based Assessment Architecture

13

Business Unit Strategy

Mission Vision Values

Structure

Data Definitions

KPIs / SLOs

Control Limits

Technology People

Architecture Enablers

Policies Procedures Instruments

Process

Performance

Measurement

Performance

Analysis / Review

Performance

Improvement

Technology

Standards

Organizational

Learning Focus

Analysis

Technology

Knowledge Mgmt

Competencies

Collection

Technology

Process-based

Organization

@ACBSPAccredited #ACBSP2015

The Common Singular Focus

14

Business Unit Strategy

Mission Vision Values

Structure

Data Definitions

KPIs / SLOs

Control Limits

Technology People

Architecture Enablers

Policies Procedures Instruments

Process

Performance

Measurement

Performance

Analysis / Review

Performance

Improvement

Technology

Standards

Organizational

Learning Focus

Analysis

Technology

Knowledge Mgmt

Competencies

Collection

Technology

Process-based

Organization

@ACBSPAccredited #ACBSP2015

The Challenge of Assessment

ACBSP Assessment Requirements

Using a Design Approach to Architect a Model

Multi-Tiered Assessment Instrumentation

Examples

Conclusions

15 @ACBSPAccredited #ACBSP2015

• Exams / Standardized tests

• Case study analyses

• Rubric scores for assignments, reports, and research

projects

• Rubric scores for writing, oral presentations, and

performances

• Observations of field work, internship performance,

service learning, consulting projects

• Quizzes

• Class discussion participation

Course evaluations

Grades not based on explicit criteria related to clear

learning goals

Test blueprints (outlines of the concepts and skills

covered on tests)

Percent of class time spent in active learning

Number of student hours spent on service learning

Number of student hours spent on homework

Number of student hours spent at intellectual or cultural

activities related to the course

Cours

e

Academ

ic

Pro

gra

m

Busi

ness

Unit

F

orm

ativ

e

Su

mm

ativ

e

Fo

rma

tive

S

um

ma

tive

Perfo

rmance

Managem

ent

Asse

ssment

Financial Focus

Student

Focus

Customer

Focus

Workforce

Focus

Leadership and Governance Focus

Yesterday

Today

Tomorrow

Operational Environment

Direct Indirect

Capstone projects, senior theses, exhibits, or

performances (internal to school only)

Student publications or conference presentations

Employer and internship supervisor ratings of students'

performance

Externally normed assessments, e.g. Peregrine, ETS,

etc.

Pass rates or scores on licensure, certification, or

subject area tests

Job Placement

Department or program review data

Employer or alumni surveys

Focus group interviews with students, faculty members,

or employees

Student perception surveys

Internal External

Sta

ndard

4.1

a,b

Sta

ndard

2

or

Outcome / Instrumentation Overview

16

Sta

ndard

4.1

c,d

,e

Course Level Elements

17

Program Level Elements

18

Business Unit Elements

19

The Challenge of Assessment

ACBSP Assessment Requirements

Using a Design Approach to Architect a Model

Multi-Tiered Assessment Instrumentation

Examples

Conclusions

20 @ACBSPAccredited #ACBSP2015

Overview of Examples

1. College-level KPI Reports (Standard 2)

2. Signature assignments using standard rubric (Standard 4.1c)

3. NPS – External Measure (Standard 4.1d)

4. Trending (longitudinal) dashboard (Standard 4.2)

5. External test benchmarking approach (Standard 4.3)

21 @ACBSPAccredited #ACBSP2015

College Dashboard - Layer 1 (National Baldrige Recipient Example)

22 Source: Dallas Community College District

ACBSP Standard 2

@ACBSPAccredited #ACBSP2015

College Dashboard - Layer 2 (National Baldrige Recipient Example)

23 Source: Dallas Community College District @ACBSPAccredited #ACBSP2015

College Dashboard - Layer 3

(National Baldrige Recipient Example)

24 Source: Dallas Community College District @ACBSPAccredited #ACBSP2015

Signature Assignments Using a Standardized Rubric (Standard 4.1c)

25

Standardized Rubric

@ACBSPAccredited #ACBSP2015

NPS as an External Measure (Standard 4.1d)

26

Source: Bain & Co.

http://www.netpromotersystem.com/about/measuring-your-net-promoter-score.aspx

@ACBSPAccredited #ACBSP2015

Trending (longitudinal) dashboard (Standard 4.2)

27 @ACBSPAccredited #ACBSP2015

External test benchmarking approach (Standard 4.3)

28 @ACBSPAccredited #ACBSP2015

The Challenge of Assessment

ACBSP Assessment Requirements

Using a Design Approach to Architect a Model

Multi-Tiered Assessment Instrumentation

Examples

Conclusions

29 @ACBSPAccredited #ACBSP2015

Putting it all together Signature Assignment Example

30

Signature assignment aligned to Mission, Vision, & Values

Structure

• Rubric structure

• Relevant student

learning

outcomes

• Targets

Technology

• LMS

• Rubric software

• Analysis

software

People

• Knowledge mgt

competencies

• CQI approach

• SIPOC for fit of

process

Architecture Enablers

Assessment Policies Overall assessment

cycle Rubric types

Process

• Signature

assignment

selection

• Scoring / rubric

process

• Cycles of review /

improvement

@ACBSPAccredited #ACBSP2015

Additional Resources

•Presentation available at

http://bitly.com/ACBSP2015

•White paper to be available at same

location

31 @ACBSPAccredited #ACBSP2015