from inputs to process to outcomes: the quality management program at the american university in...
TRANSCRIPT
From Inputs to Process to Outcomes: the Quality Management
Program at the American University in Bulgaria
Steven F. Sullivan
Dean of Faculty
How to assess quality?
Assess Inputs
Does the meat have the correct weight, fat content?
Is there the correct amount of cheese?
Are the other ingredients according to the recipe?
Input-based assessment
Advantage: If the inputs necessary
for a quality product are not there, the quality cannot be there.
Disadvantage: Having the inputs
doesn’t guarantee that the output is high-quality.
Standards may lag behind changes in technology
Process Evaluation
Was the meat cooked for the right amount of time at the right temperature?
Does the restaurant observe the limits on holding time?
Was the sandwich packaged correctly?
Does process evaluation guarantee high quality?
Following the correct processes ensures that the actions taken are those that were intended, but doesn’t guarantee that the desired outcome was achieved.
Outcome assessment
Were the organization’s goals achieved?
Are society’s goals achieved?
AUBG’s assessments of quality
External Constituencies– New England Association of Schools and
Colleges (US accreditation)– National Evaluation and Assessment Agency
(Bulgarian accreditation)– Employers– Donors
Internal Constituencies
Board of Trustees Students Parents
NEASC (US Accreditation)
The institution’s academic programs are consistent with and serve to fulfill its mission and purposes.
NEASC (continued)
The institution works systematically and effectively to plan, provide, oversee, evaluate, improve, and assure the academic quality and integrity of its academic programs and the credits and degrees awarded.
The institution develops the systematic means to understand how and what students are learning and to use the evidence obtained to improve the academic program.
Outcome assessment
Advantage: you set out to measure the very objective your organization is striving to achieve.
Disadvantage: It’s hard!
Do we engage in the processes that reflect our goal to accomplish the institutional mission?
Do we monitor our success in graduating students? Do we evaluate our students’ career success after
graduation? Do we take steps to ensure that the quality of our
academic programs are upheld? Do we monitor student, faculty, and institutional
performance and take appropriate steps to improve performance?
NEAA Criteria (Bulgarian accreditation)
1.2. HEI performs procedures for developing, approving, observing and renewing the educational documentation (qualification descriptions, curricula, syllabi)
3.2. HEI ensures professional level, qualification and development of academic staff
3.2.2. Ensures the normative minimum of habilitated teachers appointed
Examples of outcomes to track…
Student Graduation and Attrition Rates
84.5%
64.3%63.5%
52.1%55.3%
50.0%52.4%
60.8%58.8%
66.8%62.1%62.8%
59.2%
68.1%
72.7%
53.4%
81.2%
73.7%
80.7%77.5%
75.2%77.9%77.9%74.5%
67.1%
78.0%80.4%
75.5%75.4%
80.2%83.3%84.1%
78.4%78.4%81.6%80.7%
77.7%
69.1%
78.5%81.3%
77.9%79.9%
15.4%15.2%
20.6%21.6%18.4%17.9%
22.3%
28.9%
19.4%18.2%21.1% 19.0%
17.2%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Four year graduation rate Five year graduation rate Six year graduation rate Attrition Rate
Example: allocation of resources to fulfill academic mission
2008 - 2009 Number Of Sections Taught and Average Section Size by Course Abbreviation
0
5
10
15
20
25
30
35
40
ANT BUS COS ECO ENG EUR FAR HTY JMC MAT MIS MLL PHI POS RLG SCI SES SOC STA
0
10
20
30
40
50
60
70
80
90
Average Section Size - Left Scale Number Of Sections - Right Scale
Academic Planning
Faculty member’s observation: “I feel like my obligation to try to reach out to the bottom 15% of the class drags down my whole class. Why are these students even taking my course?”
What does the data say?
Student 1st 4 GPA CGPA BUS GPA
11 1.35 2.65 2.43
4 1.67 2.3 2.20
18 1.68 2.64 2.59
2 1.83 2.59 2.17
56 1.85 2.97 3.05
9 1.90 2.99 2.40
5 1.93 2.5 2.22
44 1.93 2.85 2.97
28 2.00 3.04 2.72
40 2.08 3.11 2.92
7 2.08 2.87 2.31
12 2.08 2.46 2.44
22 2.08 3.05 2.66
47 2.08 3.01 2.99
51 2.15 3.02 3.01
17 2.18 2.37 2.59
Overall conclusion
Summary Data -- 2009 Business Majors
1st 4 GPA # of students overall BUS GPA (average)
below 2.0 8 2.5
from 2.0 to 2.49 23 2.68
from 2.5 to 2.99 43 3.03
from 3.0 to 3.49 27 3.29
from 3.5 to 4.0 50 3.69
Response
2010-11 Academic Catalog: Upper-level required courses in Business Administration (those
numbered 3XX and 4XX) are available only to declared Business Administration majors. To be qualified for declaring the Business Administration major, a student must earn a 2.5 GPA in the first four required Business Administration courses
– BUS 203, BUS 208, BUS 209, and BUS 210.
Example: fulfillment of mission
3. To what degree has the overall AUBG experience refined your personal values and integrity?
Marking of multiple answers allowed
A great deal, expectations were clearly modeled by faculty and administration 53 28.0%
Moderately, there has been only moderate discussion of matters of integrity 90 47.6%
Very little, integrity is not consistently valued at AUBG 26 13.8%
Not at all, AUBG has less impact than the overall environment in this regard 10 5.3%
Not applicable, integrity should not be a University issue but only a personal issue 10 5.3%
Example: survey of graduating students:
3. How important was AUBG for developing your leadership skills?
extremely important 59 32.4%
relatively important 78 42.9%
somewhat important 38 20.9%
not important at all 4 2.2%
does not apply 3 1.6%
Evaluation of instruction
Average Satisfaction with Professor
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
1 1.5 2 2.5 3 3.5 4
Average grade assigned to class
Avera
ge "
pro
fesso
r sati
sfa
cti
on
" sco
re
Average GPA vs Course Satisfaction
2.00
2.50
3.00
3.50
4.00
4.50
5.00
1.50 2.00 2.50 3.00 3.50 4.00
Average GPA of Class
Aver
age
Satis
fact
ion
with
Cou
rse
Series1
Distribution of coursework throughout the semester
Fall 2009 Academic Coursework Distribution
0
0.05
0.1
0.15
0.2
0.25
0.3
Per
cen
t o
f se
mes
ter
gra
de
Series1
Consistency of academic standards
Business Department 2008-09
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
A A- B+ B B- C+ C C- D+ D D- F P U W
Grades
percen
tag
e
Prof. A 126 students 3.7 GPA
Prof. B 180 students 3.6 GPA
All Other BUS 1948 students 2.9 GPA
Conclusions
Assessment occurs throughout the institution Types of assessment include inputs,
processes, and outcomes Facts and data can (and should) be collected
to support the decision-making process A commitment to quality requires that
decision-making be driven by relevant data.