from inputs to process to outcomes: the quality management program at the american university in...

From Inputs to Process to Outcomes: the Quality Management

Program at the American University in Bulgaria

Steven F. Sullivan

Dean of Faculty

How to assess quality?

Assess Inputs

Does the meat have the correct weight, fat content?

Is there the correct amount of cheese?

Are the other ingredients according to the recipe?

Input-based assessment

Advantage: If the inputs necessary

for a quality product are not there, the quality cannot be there.

Disadvantage: Having the inputs

doesn’t guarantee that the output is high-quality.

Standards may lag behind changes in technology

Process Evaluation

Was the meat cooked for the right amount of time at the right temperature?

Does the restaurant observe the limits on holding time?

Was the sandwich packaged correctly?

Does process evaluation guarantee high quality?

Following the correct processes ensures that the actions taken are those that were intended, but doesn’t guarantee that the desired outcome was achieved.

Outcome assessment

Were the organization’s goals achieved?

Are society’s goals achieved?

AUBG’s assessments of quality

External Constituencies– New England Association of Schools and

Colleges (US accreditation)– National Evaluation and Assessment Agency

(Bulgarian accreditation)– Employers– Donors

Internal Constituencies

Board of Trustees Students Parents

NEASC (US Accreditation)

The institution’s academic programs are consistent with and serve to fulfill its mission and purposes. 

NEASC (continued)

The institution works systematically and effectively to plan, provide, oversee, evaluate, improve, and assure the academic quality and integrity of its academic programs and the credits and degrees awarded. 

The institution develops the systematic means to understand how and what students are learning and to use the evidence obtained to improve the academic program.

Outcome assessment

Advantage: you set out to measure the very objective your organization is striving to achieve.

Disadvantage: It’s hard!

Do we engage in the processes that reflect our goal to accomplish the institutional mission?

Do we monitor our success in graduating students? Do we evaluate our students’ career success after

graduation? Do we take steps to ensure that the quality of our

academic programs are upheld? Do we monitor student, faculty, and institutional

performance and take appropriate steps to improve performance?

NEAA Criteria (Bulgarian accreditation)

1.2. HEI performs procedures for developing, approving, observing and renewing the educational documentation (qualification descriptions, curricula, syllabi)

3.2. HEI ensures professional level, qualification and development of academic staff

3.2.2. Ensures the normative minimum of habilitated teachers appointed

Examples of outcomes to track…

Student Graduation and Attrition Rates

84.5%

64.3%63.5%

52.1%55.3%

50.0%52.4%

60.8%58.8%

66.8%62.1%62.8%

59.2%

68.1%

72.7%

53.4%

81.2%

73.7%

80.7%77.5%

75.2%77.9%77.9%74.5%

67.1%

78.0%80.4%

75.5%75.4%

80.2%83.3%84.1%

78.4%78.4%81.6%80.7%

77.7%

69.1%

78.5%81.3%

77.9%79.9%

15.4%15.2%

20.6%21.6%18.4%17.9%

22.3%

28.9%

19.4%18.2%21.1% 19.0%

17.2%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Four year graduation rate Five year graduation rate Six year graduation rate Attrition Rate

Example: allocation of resources to fulfill academic mission

2008 - 2009 Number Of Sections Taught and Average Section Size by Course Abbreviation

0

5

10

15

20

25

30

35

40

ANT BUS COS ECO ENG EUR FAR HTY JMC MAT MIS MLL PHI POS RLG SCI SES SOC STA

0

10

20

30

40

50

60

70

80

90

Average Section Size - Left Scale Number Of Sections - Right Scale

Academic Planning

Faculty member’s observation: “I feel like my obligation to try to reach out to the bottom 15% of the class drags down my whole class. Why are these students even taking my course?”

What does the data say?

Student 1st 4 GPA CGPA BUS GPA

11 1.35 2.65 2.43

4 1.67 2.3 2.20

18 1.68 2.64 2.59

2 1.83 2.59 2.17

56 1.85 2.97 3.05

9 1.90 2.99 2.40

5 1.93 2.5 2.22

44 1.93 2.85 2.97

28 2.00 3.04 2.72

40 2.08 3.11 2.92

7 2.08 2.87 2.31

12 2.08 2.46 2.44

22 2.08 3.05 2.66

47 2.08 3.01 2.99

51 2.15 3.02 3.01

17 2.18 2.37 2.59

Overall conclusion

Summary Data -- 2009 Business Majors

1st 4 GPA # of students overall BUS GPA (average)

below 2.0 8 2.5

from 2.0 to 2.49 23 2.68

from 2.5 to 2.99 43 3.03

from 3.0 to 3.49 27 3.29

from 3.5 to 4.0 50 3.69

Response

2010-11 Academic Catalog: Upper-level required courses in Business Administration (those

numbered 3XX and 4XX) are available only to declared Business Administration majors. To be qualified for declaring the Business Administration major, a student must earn a 2.5 GPA in the first four required Business Administration courses

– BUS 203, BUS 208, BUS 209, and BUS 210.

Example: fulfillment of mission

3. To what degree has the overall AUBG experience refined your personal values and integrity?

Marking of multiple answers allowed

A great deal, expectations were clearly modeled by faculty and administration 53 28.0%

Moderately, there has been only moderate discussion of matters of integrity 90 47.6%

Very little, integrity is not consistently valued at AUBG 26 13.8%

Not at all, AUBG has less impact than the overall environment in this regard 10 5.3%

Not applicable, integrity should not be a University issue but only a personal issue 10 5.3%

Example: survey of graduating students:

3. How important was AUBG for developing your leadership skills?

extremely important 59 32.4%

relatively important 78 42.9%

somewhat important 38 20.9%

not important at all 4 2.2%

does not apply 3 1.6%

Evaluation of instruction

Average Satisfaction with Professor

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

1 1.5 2 2.5 3 3.5 4

Average grade assigned to class

Avera

ge "

pro

fesso

r sati

sfa

cti

on

" sco

re

Average GPA vs Course Satisfaction

2.00

2.50

3.00

3.50

4.00

4.50

5.00

1.50 2.00 2.50 3.00 3.50 4.00

Average GPA of Class

Aver

age

Satis

fact

ion

with

Cou

rse

Series1

Distribution of coursework throughout the semester

Fall 2009 Academic Coursework Distribution

0

0.05

0.1

0.15

0.2

0.25

0.3

Per

cen

t o

f se

mes

ter

gra

de

Series1

Consistency of academic standards

Business Department 2008-09

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

A A- B+ B B- C+ C C- D+ D D- F P U W

Grades

percen

tag

e

Prof. A 126 students 3.7 GPA

Prof. B 180 students 3.6 GPA

All Other BUS 1948 students 2.9 GPA

Conclusions

Assessment occurs throughout the institution Types of assessment include inputs,

processes, and outcomes Facts and data can (and should) be collected

to support the decision-making process A commitment to quality requires that

decision-making be driven by relevant data.