Outcomes Assessment Outcomes Assessment and Assurance of Learningand Assurance of Learning
Outcomes Assessment Outcomes Assessment and Assurance of Learningand Assurance of Learning
James M. KraushaarJames M. Kraushaar, Ph.D., Associate Professor , Ph.D., Associate Professor of MISof MIS
Nicole ChittendenNicole Chittenden, IT Manager, IT ManagerRobert RohrRobert Rohr, IT Professional Sr., IT Professional Sr.
The School of Business AdministrationThe School of Business Administrationat the University of Vermontat the University of Vermont
Track 4: Technology for Teaching and LearningTrack 4: Technology for Teaching and Learning
© © James Kraushaar et al., 2005.James Kraushaar et al., 2005. This work is the intellectual property of the authors. Permission This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from permission of the authors. To disseminate otherwise or to republish requires written permission from the authors.the authors.
Presentation for NERCOMP 2006, Mar. 20-22, Presentation for NERCOMP 2006, Mar. 20-22, Worcester, MAWorcester, MA
© 2005 The School of Business Administration at the University of Vermont
AgendaAgenda ToolsTools ModelModel Data CollectionData Collection Sample - Data AnalysisSample - Data Analysis ConclusionConclusion
© 2005 The School of Business Administration at the University of Vermont
School of Business Assessment School of Business Assessment SystemsSystems
Satisfaction:Satisfaction: Educational Educational
Benchmarking (EBI)Benchmarking (EBI) On-line course On-line course
evaluationsevaluations Advisor surveyAdvisor survey AACSB accreditationAACSB accreditation UVM Office of UVM Office of
Institutional Studies Institutional Studies (OIC) year out (OIC) year out surveysurvey
Competency:Competency: Educational Testing Educational Testing
Services (ETS) Services (ETS) Major Field Test in Major Field Test in BusinessBusiness
GradesGrades Faculty Activity Faculty Activity
ReportsReports Curriculum:Curriculum:
Faculty Curriculum Faculty Curriculum Coverage InventoryCoverage Inventory
© 2005 The School of Business Administration at the University of Vermont
Curriculum: Product deliveryCurriculum: Product delivery
MissionMission
CurriculumCurriculum
SatisfactionSatisfaction CompetencyCompetency
© 2005 The School of Business Administration at the University of Vermont
Are we delivering the academic program Are we delivering the academic program we think we are?we think we are?
Curricular ProgramsCurricular ProgramsCourse Content/DeliveryCourse Content/Delivery
Extracurricular ProgramsExtracurricular ProgramsAcademic AdvisingAcademic Advising
Career & Professional Dev.Career & Professional Dev.Leadership OpportunitiesLeadership Opportunities
OUTCOMES ASSESSMENTOUTCOMES ASSESSMENT
COMPETENCYCOMPETENCYGradesGrades
ETS Major Field TestETS Major Field Test
SATISFACTIONSATISFACTIONCourse EvaluationsCourse Evaluations
EBI Exit SurveyEBI Exit SurveyAdvising SurveyAdvising Survey
GOAL:GOAL:The BestThe Best
Small School ofSmall School ofBusinessBusiness
© 2005 The School of Business Administration at the University of Vermont
The Predictable QuestionsThe Predictable Questions
Are our students satisfied with Are our students satisfied with education?education? Course evaluationsCourse evaluations Exit interviewsExit interviews BenchmarkingBenchmarking
Are our students prepared?Are our students prepared? GradesGrades Standardized testsStandardized tests Employer surveysEmployer surveys
MissionMission
CurriculumCurriculumSatisfactionSatisfaction CompetencyCompetency
MissionMission
CurriculumCurriculumSatisfactionSatisfaction CompetencyCompetency
© 2005 The School of Business Administration at the University of Vermont
Less Obvious Less Obvious AnswersAnswers
Curricular choices drive results:Curricular choices drive results: Where should emphasis be placed to Where should emphasis be placed to
address competency shortfalls?address competency shortfalls? Which choices are working? Which choices are working? What changes will have greatest effect?What changes will have greatest effect? Are key concepts reinforced?Are key concepts reinforced? Can curricular focus on mission critical Can curricular focus on mission critical
aims be identified?aims be identified?
© 2005 The School of Business Administration at the University of Vermont
The plan… gather The plan… gather Curricular Curricular Coverage Inventory Coverage Inventory
Evaluate aggregate curricular focusEvaluate aggregate curricular focus Quantify the total experience of Quantify the total experience of
programprogram Target specific areas for review Target specific areas for review
and/or adjustmentand/or adjustment MissionMission
CurriculumCurriculumSatisfactionSatisfaction CompetencyCompetency
© 2005 The School of Business Administration at the University of Vermont
Design for alignmentDesign for alignment
Align faculty survey of curricular Align faculty survey of curricular coveragecoverage with mission statementwith mission statement with student competencywith student competency
Can now compare/contrast subjective Can now compare/contrast subjective and objective quality measuresand objective quality measures Student satisfaction vs. competencyStudent satisfaction vs. competency
© 2005 The School of Business Administration at the University of Vermont
Curricular Coverage InventoryCurricular Coverage Inventory
Collected from facultyCollected from faculty Info across every courseInfo across every course Three areas:Three areas:
Mission drivenMission driven Competencies drivenCompetencies driven Learning Assessment methodsLearning Assessment methods
Demo Demo
© 2005 The School of Business Administration at the University of Vermont
Sample Analysis: Topics bigger Sample Analysis: Topics bigger than any one coursethan any one course
Mission Statement Excerpt:Mission Statement Excerpt:
… … prepares students for … dynamic prepares students for … dynamic globalglobal economy … economy … ethical, social, ethical, social, and environmentaland environmental responsibility… responsibility…
© 2005 The School of Business Administration at the University of Vermont
20 27 27
Social Responsibility 2.39 9 8 14 27 25
Environmental/Sustainability Concerns 2.19 5 4
21 28 12
Legal Environment 2.77 16 6 23 19 19
Ethical Environment 2.78 12 10
28 18 13
Global Awareness 2.88 6 23 20 23 11
Use of Information Technology 2.92 13 11
13 5 3
Acquire, Analyze and Interpret Data 4.02 36 27 10 6 4
Critical Thinking 4.16 45 17
9 2 3
Solving Qualitative Problems 4.10 38 28 10 1 6
Defining Qualitative Problems 4.24 42 27
12 21 10
Solving Quantitative Problems 3.41 37 4 9 22 11
Defining Quantitative Problems 3.37 32 8
11 15 27
Interpersonal Skills 2.40 10 7 13 29 24
Ability to work in Teams 2.66 11 19
19 17 8
Leadership Skills 2.10 3 8 13 29 30
Writing Skills 3.31 20 19
19 23 14
Presentation Skills 2.57 7 18 19 10 29
Sense of History 2.81 8 19
Section I - Knowledge & Skill InventoryMission Elements Avg 5 4 3 2 1
Results: Curricular focus on Results: Curricular focus on MissionMission
GlobalGlobal EthicalEthical EnvironEnviron
--mentalmental
SocialSocial
© 2005 The School of Business Administration at the University of Vermont
Assessment Tools UsedWritten Examination(s)Written Quiz(zes)Oral Examination(s)Oral Quiz(zes)Individual Presentation(s)Inividual Paper(s)/Project(s)Group Presentation(s)Group Paper(s)/Project(s)Case(s)Problem/Homework Set(s) 4.24
2.770.750.541.220.43
0.0 91.60.06 0.0 1.2 0.0 0.0 94.00.01 0.0 1.2 0.0
24.9 13.43.81 19.4 13.9 12.7 2.4 48.12.20 20.7 13.4 26.4
Section III - Assessment Tools UsedFreq T/F ShA Qnt Ess Oth
Sample Report: How are Sample Report: How are students assessed in an students assessed in an
average courseaverage course
© 2005 The School of Business Administration at the University of Vermont
RecapRecap
Talked about curriculum coverage Talked about curriculum coverage data gathering systemdata gathering system
Nicole will discuss external Nicole will discuss external satisfaction and competency satisfaction and competency measurement systemsmeasurement systems
© 2005 The School of Business Administration at the University of Vermont
Incorporating Selected Outside Incorporating Selected Outside ToolsTools
Student SatisfactionStudent Satisfaction EBIEBI – – ((http://http://www.webebi.com/wesswww.webebi.com/wess))
Offers “Complete Assessment Solutions”Offers “Complete Assessment Solutions” Focus on annual comparison with Carnegie Focus on annual comparison with Carnegie
class institutionsclass institutions Select 6 - comparisons have limited value Select 6 - comparisons have limited value
given inconsistent participation by peersgiven inconsistent participation by peers
Student CompetencyStudent Competency ETSETS – – ((http://http://www.ets.orgwww.ets.org))
Major field testMajor field test Measures learning outcomes and Measures learning outcomes and
benchmarks against other institutionsbenchmarks against other institutions Required for completion of capstone classRequired for completion of capstone class
© 2005 The School of Business Administration at the University of Vermont
EBI – Measure of Student EBI – Measure of Student SatisfactionSatisfaction
Incorporates measures of school Incorporates measures of school initiativesinitiatives
Advising, Instruction in Ethics, Career & Advising, Instruction in Ethics, Career & Professional DevelopmentProfessional Development
Summary format for sharing results Summary format for sharing results with faculty, staff, Board of Advisors, with faculty, staff, Board of Advisors, ProvostProvost
Priority Matrix – identifies items that are highly Priority Matrix – identifies items that are highly correlated with student satisfaction correlated with student satisfaction
Home grown reportsHome grown reports
© 2005 The School of Business Administration at the University of Vermont
EBI – Priority Matrix 2005EBI – Priority Matrix 2005
© 2005 The School of Business Administration at the University of Vermont
Example: Analyzing Example: Analyzing SatisfactionSatisfaction
0
5
10
15
20
25
30
35
40
# of questions
2001 2002 2003 2004 2005
Year
Per question, how did our students perform vs. All Schools
top 20%
top 50%
bottom 20%
0
5
10
15
20
25
30
35
40
45
# of questions
2004 2005
Year
Trends by Category vs. Carnegie Class
Good
Bad
Ugly
© 2005 The School of Business Administration at the University of Vermont
ETS - Student competencyETS - Student competencyWhat has it told us?What has it told us?
% Mean by Indicator
FA04-UVM mean % Correct
FA04-UVM % at or below
SP05-UVM mean % Correct
SP05-UVM % at or below
All B-Schools Mean % Correct
All B-Schools % at or below
1-Accounting 54.0 90% 51.6 90% 44.6 50%2-Economics 52.1 90% 45.8 70% 42.8 50%3-Management 59.7 65% 56.3 40% 57.1 45%4-Quant Bus Analy/MIS 67.8 95% 66.2 90% 56.6 50%5-Finance 43.6 85% 42.8 85% 36.4 50%6-Marketing 55.0 90% 51.1 75% 46.8 50%7-Legal & Social Envir 50.4 50% 48.2 35% 49.8 50%8-International Issues 47.0 65% 42.3 35% 44.3 50%
n= 118 65 80,044Mean 159.5 156.2 152
σ= 13.6 13.5 13.7
Significant Fall/Spring variation. Why?Significant Fall/Spring variation. Why?
© 2005 The School of Business Administration at the University of Vermont
Student competency Student competency differences across semesters?differences across semesters?
0
20
40
60
80
100
120
# of students
Fall 2004 Spring 2005
Transfer vs. Non-transfer
Non-transfer
Transfer
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
percent students
0.0-1.0 1.0-2.0 2.0-2.5 2.5-3.0 3.0-3.5 3.5-4.0
GPA Range
GPA distribution
Fall 2004
Spring 2005
© 2005 The School of Business Administration at the University of Vermont
Multiple measuresMultiple measures
Moved from individual measurements Moved from individual measurements to composite viewto composite view Neither satisfaction nor competency Neither satisfaction nor competency
tells the whole storytells the whole story Combine them for more complete Combine them for more complete
picturepicture
© 2005 The School of Business Administration at the University of Vermont
Curriculum Change: Student Curriculum Change: Student Satisfaction (EBI) vs. Satisfaction (EBI) vs.
Student Competency (ETS)Student Competency (ETS)
Human Human Resource Resource
ManagementManagementLowLow LowLow
AccountingAccounting LowLow HighHigh
Legal and Legal and Social Social
EnvironmentEnvironmentHighHigh LowLow
SatisfactionSatisfaction CompetencyCompetency
© 2005 The School of Business Administration at the University of Vermont
Case StudyCase Study
Legal environmentLegal environment Courts & legal systemsCourts & legal systems Constitution and businessConstitution and business Administrative lawAdministrative law
Regulatory EnvironmentRegulatory Environment Employment & labor law, Employment & labor law,
Antitrust law, Antitrust law, Consumer protectionConsumer protection Tort lawTort law CrimesCrimes Environmental and Environmental and
international lawinternational law
Business relationshipsBusiness relationships Contract & commercial Contract & commercial
law (including UCC)law (including UCC) Business organizationsBusiness organizations Law of agencyLaw of agency Intellectual propertyIntellectual property
Ethics & Social Ethics & Social responsibilityresponsibility
EthicsEthics Social responsibilitySocial responsibility
10% of ETS questions center around Legal & Social Environment 10% of ETS questions center around Legal & Social Environment Legal & Social Environment Content Includes: Legal & Social Environment Content Includes:
© 2005 The School of Business Administration at the University of Vermont
Curricular Coverage Inventory = Curricular Coverage Inventory = Limited Review of Business Limited Review of Business
RelationshipsRelationships
8 8 5Social responsibility 3.14 9 5
Ethics & social responsibility
Ethics 3.11 10 7 3 7 8
5 3 27
Intellectual property 1.54 1 3 1 4 26
Law of agency 1.37 0 0
1 3 31
Business organizations 1.06 0 0 0 2 33
Contracts & commercial law (incl. UCC) 1.14 0 0
3 1 26
Business relationships
Environmental & international law 1.71 3 2
0 2 32
Crimes 1.20 0 1 1 2 31
Tort law 1.14 0 1
2 1 27
Consumer protection 1.46 2 1 2 1 29
Antitrust law 1.66 3 2
Regulatory environment
Employment & labor law 2.29 5 7 1 2 20
2 0 28
Administrative law 1.71 0 5 2 6 22
Constitution & business 1.63 3 2
Legal and Social Environment
Legal environment
Courts & legal systems 2.23 3 8 1 5 185 4 3 2 1Avg
© 2005 The School of Business Administration at the University of Vermont
Curricular Coverage Inventory:Curricular Coverage Inventory:Faculty focus within courseFaculty focus within course
Section II: Skills/Topics Mentioned in Syllabus
132 - Legal and Political Environment of Business
Question Value
Legal and Social Environment
Legal environment
Courts & legal systems 3.67
Averyt, William 5.00
Meier, Frederic 2.00
Monte, Peter 4.00
Constitution & business 4.33
Averyt, William 5.00
Meier, Frederic 4.00
Monte, Peter 4.00
Administrative law 2.67
Averyt, William 3.00
Meier, Frederic 2.00
Monte, Peter 3.00
Business relationships
Contracts & commercial law (incl. UCC) 1.33
Averyt, William 1.00
Meier, Frederic 1.00
Monte, Peter 2.00
Business organizations 1.33
Averyt, William 1.00
Meier, Frederic 1.00
Monte, Peter 2.00
Law of agency 1.33
Averyt, William 1.00
Meier, Frederic 1.00
Monte, Peter 2.00
Intellectual property 2.33
Averyt, William 4.00
Meier, Frederic 1.00
Monte, Peter 2.00
Ethics & social responsibility
Ethics 3.33
Averyt, William 2.00
Meier, Frederic 4.00
Monte, Peter 4.00
Social responsibility 3.33
Averyt, William 2.00
Meier, Frederic 3.00
Monte, Peter 5.00(continued, next column…)(continued, next column…)
© 2005 The School of Business Administration at the University of Vermont
ConclusionConclusion Based on data (NOT just intuition)Based on data (NOT just intuition) Both satisfaction and competencyBoth satisfaction and competency Better information about delivery of Better information about delivery of
topics across entire programtopics across entire program Better target curricular changesBetter target curricular changes
MissionMission
CurriculumCurriculum
SatisfactionSatisfaction CompetencyCompetency
© 2005 The School of Business Administration at the University of Vermont
AcknowledgementsAcknowledgements
AACSB Accreditation and Training AACSB Accreditation and Training ProcessProcess
University of Alaska, FairbanksUniversity of Alaska, Fairbanks Western Ontario – Outcomes Western Ontario – Outcomes
Assessment ConferenceAssessment Conference Our staff and facultyOur staff and faculty
© 2005 The School of Business Administration at the University of Vermont
QuestionsQuestions
© 2005 The School of Business Administration at the University of Vermont
Contact informationContact information web: web: http://http://www.bsad.uvm.eduwww.bsad.uvm.edu/assess/assess James M. Kraushaar, Ph.D.,James M. Kraushaar, Ph.D.,
Associate Professor of MISAssociate Professor of MIS [email protected]@bsad.uvm.edu
Nicole Chittenden, IT ManagerNicole Chittenden, IT Manager [email protected]@uvm.edu
Robert Rohr, IT Professional Sr.Robert Rohr, IT Professional Sr. [email protected]@uvm.edu