©k. martell southern business administration association annual meeting 2004 assurance of...
TRANSCRIPT
©K. Martell
Southern Business Administration Association
Annual Meeting 2004
Assurance of Learning—Are we making the grade?
Kathryn MartellAssociate Dean, Montclair State
©K. Martell
Today’s Talk• AACSB Expectations• Where we are compared to those
expectations• Advice and other musings
Based onsurvey
Survey sample…
66% Public Institutions69% AACSB accredited
82% Offer Masters 17% Offer Doctorate
©K. Martell
Number of Students
0% 10% 20% 30% 40% 50%
More than 3000
1500-3000
500-1500
Less than 500
©K. Martell
Business Schools annual budget
0% 10% 20% 30% 40%
Over $10million
$5-$10million
$2-$5million
Under $2million
©K. Martell
(review) Program Assessment
• Expert Learning is a curriculum function – not a course function.
• Program assessment evaluates student learning across the curriculum.
• Program assessment measures student learning, NOT faculty teaching.
©K. Martell
Assessing Institutional Effectiveness
©K. Martell
A quick review:AACSB Standard 16
“(Driven by mission)…the school specifies learninggoals and demonstrates achievement of learning goals for key general, management-specific,and/or appropriate discipline specific knowledge and skills that its students achieve in each undergraduate degree program.”
Old Days:
We taught it!
Now:
They learned it!
©K. Martell
What it does say• Program-level learning goals• Mission driven• Direct measures of assessment• Knowledge & skills• Use data for program
improvement
©K. Martell
What it doesn’t say…(i.e., it could be a lot
worse!)• Use a specific method• Assess at the concentration/major level• Benchmark w/ other schools• Assess every topic/skill in Standard 15• Value-added assessment required• That every student must be assessed• What percentage of students must meet
standard• Graduation must be tied to meeting
standards• That the complete assessment process must
be implemented all at once, and immediately• Refer to FAQ in ARC
©K. Martell
Review (continued)The assessment process:1. Define goals and objectives. (include both general and
management-specific goals).
2. Align curriculum with goals.3. Identify instruments and methods for assessing
student achievement for each goal.4. Implement. (Collect data)5. Disseminate data.
6. Feedback Loop! Evaluate Assessment data, make changes.
©K. Martell
AACSB Timetable: We have time
(but the sand is moving…)• Year 1 (2003): Set
Learning Goals• Year 2 (2004):
Curriculum Alignment, Measures
• Year 3-4 (2005&6): Collect data, analyze, curriculum changes
• By 2007, the full assessment cycle (including curriculum change) should be in place.
©K. Martell
Step one: Learning goals
• 4-10 per degree program• Faculty driven, mission driven• DUE: Last year!
©K. Martell
Does your school have…
0%
10%
20%
30%
40%
50%
60%
70%
writtenassessment
standards objectives formallearning
goals
Yes No
©K. Martell
The MOST IMPORTANT/RELEVANT learning goals for your School of Business’ undergraduate degree
programs.
0% 10% 20% 30% 40% 50% 60% 70%
Life long learners
Professional
Strategic thinkers
Entrepreneurial
Technically Proficient
Leaders
Analytical
Socially responsible
Team Players
Competent in their discipline
Global perspective
Knowledgable of all business disciplines
Able to integrate across disciplines
Problem Solvers
Eff ective decision makers
Critical Thinkers
Ethical
Eff ective Communicators
Top rated: ethics, communication, critical thinkers, decision makers, problem solvers, ability to integrate, knowledge, global perspective, disciplinary competence, team players.
©K. Martell
Kathryn’s advice on learning goals:
1. Look at your mission2. Top down, not bottom up.3. Don’t want to know, don’t
ask.4. Keep the one thing(s) , the
one thing(s).
©K. Martell
Step 1b: Translate those goals into something you can observe (objectives)
0%
10%
20%
30%
40%
50%
60%
70%
writtenassessment
standards objectives formallearning
goals
Yes No
©K. Martell
Goals vs. objectives• Goals express what we want our students to be.• Objectives describe what we want our students
to do.
Objectives are indicators of goals.
In order to be assessable, objectives must be written so they specify behaviors or products that we can observe.
©K. Martell
ExamplesGoal: Interdisciplinary thinking
Objective:When confronted with a business case
that doesn’t prompt them to do so, students will develop solutions or recommendations that include mutually supportive activities in the value chain.
When confronted with a case from one discipline, students will appropriately apply methods from another discipline.
©K. Martell
ExampleGoal: Good Team Member
Objectives: Students will demonstrate the
ability to work effectively in a team environment to solve business related scenarios.
©K. Martell
(Ambitious) ExampleGoal: Global perspectiveObjective:Students will conduct a business conversation
in a foreign language.Students will host a social function consistent
with the norms of the invited students’ culture
Students will analyze foreign financial statements using international accounting standards
Students will design a HRM program consistent with the norms of another culture
Students will complete the US citizenship exam
Students will complete the equivalent of the US citizenship exam for another country
©K. Martell
Next step: Choose Methods
• Tests, Exams• Essays• Embedded questions• Journals• Case Studies• Simulations• Internships• Assessment Centers• Written Assignments
• Portfolios• Presentations• Capstone or other
projects• Videotaping*******• Surveys (Student,
Alumni, Employer)• Interviews• Focus Groups
©K. Martell
Note:
• Course grades are not appropriate for program assessment.
©K. Martell
Why grades aren’t program
assessment….Exams 75
Presentation 85
Paper 90
Homework 80
Total 81
Exams 90Presentation 84Paper 60Homework 82Total 81
COURSE
Exams50%
Oral Presentation 15%
Paper25%
Homework10%
©K. Martell
0% 20% 40% 60% 80%
Syllabi Review
Pass rates on professional exams
exam: Analytical skills
exam: writing
Capstone or Senior Project
Student GPAs
Assessment Center
Comprehensive Exam (written)
Student Interviews
Employer Survey
Psychometric measures
ETS Field Test
Assessment methods:
Top Direct: ETS, Course embedded, capstone, student portfolio, pass rates on prof. exams
©K. Martell
Examples of Course-Embedded Measurement:
• Written assignments including cases, research reports, memos.
• Oral presentations, debates.• Computer projects• Simulations• Classroom exercises• Business plans and/or consulting projects.• Exams or embedded questions on exams.
©K. Martell
Difference between assessing student work for a course, and
assessing student work – in a class – for program assessment • For class: every student assessed,
professor determines criteria and standards (which may or may not be explicit), professor does the evaluation, student gets the result
• For program assessment: sampling acceptable, faculty determine criteria and standards (which are explicit), professor OR outsider does the evaluation, feedback goes to the program faculty
©K. Martell
Oral Communication
Objectives Low Performance Average Exemplary Performance EarnedPoints
Organization 0-1 pointNo opening statement, or irrelevant statement. Loses
focus more than twice. Does not manage time effectively.
2-3 points Has opening statement relevant to topic, and gives outline of speech. Mostly organized, loses focus only
once or twice. Some
transitions. Allows enough time to deliver speech, although it could have been better edited.
4-5 points Has a clear opening statement that catches audience’s interest and gives overview. Stays focused throughout.Has transition statements,
ties sections together. Very effective time management.
Content 0-1 point Does not meet assignment requirements, is not current, nor objective.Does not use appropriate
analysis.
2-3points Fulfills assignment. Current. Uses appropriate sources
and is objective. Reasonable analysis.
4-5 points Is more thorough than "average." Research goes beyond minimum
requirements. Strong analysis.
Quality of slides 0-1 pointSloppy and/or unprofessional. May be
difficult to read. Many slides are superfluous.
2-3 points Readable, professional,
appropriate number.
4-5 points Readable, professional, imaginative and/or of high quality (without being a
distraction).
©K. Martell
Oral Communication Evaluation: Checklist Traits: (3) (2) (1)1. Opening Statement with purpose ___ X__ ___2. Organization ___ _X_ ___
3. Content: currency & relevance _X_ ___ ___4. Quality of slides _x__ ___ ___5. Quality of conclusion ___ _x__ ___*** 6. Voice quality, pace ___ _X_ ___7. Mannerisms ___ ___ _x_8. Professionalism ___ _x_ ___9. Use of media _x_ ___ ___10. Ability to answer questions _x_ ___ ___
Scoring:
10-19 Does not meet expectations 2320-26 Meets expectations27-30 Exceeds expectations
©K. Martell
Turning Grading Into Program Assessment
Oral Communication Evaluation: Checklist (n=30) Traits: (3) (2) (1)1. Opening Statement with purpose _2__ _25_ _3__2. Organization _8__ _16_ _6__3. Content: currency & relevance _7__ _18_ _5__4. Quality of slides _20_ _5__ _5__5. Quality of conclusion _2__ _12_ _16_*** 6. Voice quality, pace _14_ _12_ _4__7. Mannerisms _9__ _16_ _5__8. Professionalism _3__ _10_ _17_9. Use of media _6__ _14_ _10_10. Ability to answer questions _10_ _14_ _ 6__
Scoring:Does not meet expectations: 7Meets expectations: 20Exceeds expectations 3
©K. Martell
Kathryn’s advice on choosing methods…
1. Choose a method that will generate data you can use.
2. Build on something you’re already doing…
3….but modify to be a program assessment measure
4. Whenever possible, kill 2 birds with 1 stone.
©K. Martell
2 birds for 1 stone!• Students will write a well-organized,
concise five page paper, identifying and comparing motivation theories, using appropriate spelling, punctuation, citations, grammar, and vocabulary.
• Business plan or case: analytical skills, business knowledge, integration, writing skills
Next step: share Next step: share (and store) and use (and store) and use
the datathe data
Next step: share Next step: share (and store) and use (and store) and use
the datathe data
©K. Martell
How often Business Schools share student learning assessment data with most of its
faculty
0 20 40 60
Most common venues:
Faculty meetings (52%),
Written reports (41%), dept. (36%) & committee meetings (37%).
©K. Martell
How schools are currently storing assessment data
0 50 100 150
Other
commercially developed assessmentsoftw are/database
internally developed data base
internally generated summary reports
computer spreadsheets (Excel, Access,etc.)
paper f iles
Number of schools
©K. Martell
Has ASSESSMENT of student learning LED to any significant curriculum changes in the past 3 years in your undergraduate
programs?
0 20 40 60 80 100
Don't Know
No
Yes
Make sure to document the link between direct measurement of student learning and curriculum change…
©K. Martell
Sample Annual Assessment Report
Learning Objectives
What did weassess?
Sample
Who did we assess? When? Where?
Methods
How did we assess?(attach rubric)
Findings/Evidence
What did we find out?
Interpretation/Action Items
What are we going to do about it?
Implementing your Implementing your assesment planassesment plan
Implementing your Implementing your assesment planassesment plan
Observations & adviceObservations & advice
©K. Martell
First…a little question of
money…
©K. Martell
65% of Business Schools devoted resources (financial or release time) to learning
assessment last year.
Yes
No
©K. Martell
The approximate FINANCIAL resources devoted to these
Business School assessments last year?
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
none under$1000
$1000-$5000
$5000-$10,000
over$10,000
Don'tknow
©K. Martell
Release time allocated to assessment activities within the
Business School
None
1 course per year
2 Courses per year
More than 3 courses peryear
©K. Martell
Money spent on assessment in the Business School—
where from?67% ad hoc spendingfrom Business Schoolbudget
13% from theUniversity budget
10% in the BusinessSchool budget as aline item
10% Don't know
©K. Martell
92 out of 154 respondents said their Universities offer resources to
support assessment within their academic units.
Faculty training
Books, Materials
$ for seminars
$ for assessmentinstruments
Questionnairedevelopment
©K. Martell
Kathryn’s observation: Assessment Programs can
not run on budget dust• Strategic Item (should be linked to
financial resources)• Trade-off: Faculty time vs. financial
resources• Make or Buy. Ways to “buy” time:
choice of methods, outside graders, centralize responsibilities/ release time, factor into job responsibilities (and take away something else!)
©K. Martell
Organizing for assessment: Primary responsibility for administering the
student learning assessment process.
0
20
40
60
80
100
54% Admin 12% facultysteward
22%Committees
3% StaffPerson
10%Combination
Deans Department heads Existing New All
©K. Martell
Bringing SAC & AACSB together
(Kathryn’s advice)• AACSB: program level• SAC: department level• One system for both:1. Include disciplinary competence as one of
your AACSB learning goals2. When gathering performance data from
students, keep track of student majors3. Disaggregate and analyze data by major/
department for SAC; report in aggregate for AACSB
©K. Martell
0% 10% 20% 30% 40% 50% 60% 70% 80%
AACSB’s new Assurance standards
Translate general student learning goals
a level of acceptable student competency
assessment data to promote curriculum change
Articulate key student learning goals
assessment plan on your programs’ learning goals
Keep records on assessment process for AACSB
Analyze assessment data, and disseminate it to faculty?
How confident are you?Pretty darn confident!
©K. Martell
What makes you nervous…
0% 10% 20% 30% 40% 50% 60%
AACSB expectations will change again
Time required
Reviewers’ interpretation of standards
Faculty knowledge about assessment
Financial resources
% of surveyed
©K. Martell
And what doesn’t…
0% 10% 20% 30% 40% 50% 60% 70%
What we’ll find out
Conflict w / School goals (e.g., research)
How to draw conclusions from data
Getting started
Storing the data
Lack of university support
Student resistance
“Closing the loop” – using data to change curriculum
% of respondents
©K. Martell
How satisfied are you with your Business School’s current process of
assessment student learning?
0% 10% 20% 30% 40% 50%
Not At All Satisfied
Moderately Satisfied
Very Satisfied
©K. Martell
Kathryn’s Advice on developing your
assessment program:Keep It Simple (….)
It doesn’t have to be that It doesn’t have to be that hard.hard.
Difference between scholarly Difference between scholarly and academic rigor.and academic rigor.
The enemy of the good is the The enemy of the good is the perfect.perfect.
©K. Martell
Anticipate Faculty Resistance
1. “I can’t”2. “I won’t”3. “I shouldn’t have to”4. “I’ve already done that”*• And Their Support
* source: C. Haessig & A. LaPotin, (1999), Assessment Update.
©K. Martell
Are we making the grade?
• Well…sorta• The good news:
anxiety is down, activity is up, sharing examples, goals established
• Bad news: Objectives & standards should be established by now and assessment of one goal should be started (or close to starting).
0%
10%
20%
30%
40%
50%
60%
70%
writtenassessment
standards objectives formallearning
goals
Yes No
©K. Martell
©K. Martell
AACSB/AIR Assessment in Business Schools Book outlineContext/Environment – Reasons for Assessment1. Assessment in Business Schools: where we were, where we
are, where we need to go
2. What the regional accreditors require3. School goals: branding4. School goals: student development5. School/university cultureInnovative Practice – each step of the wayChapters 6-14Assessing the (seemingly) “Unassessable” 15. Leadership16. Ethics 17. Innovation/entrepreneurship/creativity18. Global/cultural sensitivity19. Critical Thinking 20. Other topics 21. Assessment Practices in Accounting22. Assessing Disciplinary Competence23. Organizing for Assessment24. Overcoming Obstacles to assessment
©K. Martell
Kathryn’s STRONGEST Advice:
• Get started. Now.
2nd strongest:• Don’t let the curmudgeons get
you down…everyone doesn’t have to be on board.