institutional effectiveness at the university of north alabama
DESCRIPTION
IE Presentation, November 20, 2008. Institutional Effectiveness at the University of North Alabama. Dr. Andrew L. Luna Institutional Research, Planning, and Assessment. Connections to IE?. Telephones and IE?. Walter Shewhart. Edwards Deming. Joseph Juran. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/1.jpg)
Institutional Effectiveness at the Institutional Effectiveness at the University of North AlabamaUniversity of North Alabama
Dr. Andrew L. Luna
Institutional Research, Planning, and Assessment
IE Presentation, November 20, 2008
![Page 2: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/2.jpg)
Connections to IE?Connections to IE?
![Page 3: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/3.jpg)
Hawthorne Works, Bell Laboratories
Telephones and IE?Telephones and IE?
Walter Shewhart
Edwards Deming
Joseph Juran
![Page 4: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/4.jpg)
Shewhart CycleShewhart Cycle
FACT...
The Shewhart Cycle is the foundation for all quality and
Continuous improvement processes that we use today
Plan
Do
Check
ActContinuous
Improvement
![Page 5: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/5.jpg)
Points of DiscussionPoints of Discussion
Similarities between the Shewhart Cycle and Institutional Effectiveness
Overview of Institutional Effectiveness at UNA
Review of Outcomes and Improvement Processes
Review of AssessmentQuestions
![Page 6: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/6.jpg)
More on the Shewhart CycleMore on the Shewhart Cycle
Plan – Create a strategy as to what you want to do and how you will measure success
Do – Follow the plan and do what you say you will do
Check – Assess the effectiveness of the current plan by looking at the success outcomes measures
Act – Make changes to the strategies to improve the measured outcomes
Repeat the Cycle!
![Page 7: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/7.jpg)
Why is the Shewhart Cycle Important?Why is the Shewhart Cycle Important?
If you can’t measure something, you can’t understand it…
If you can’t understand it, you can’t control it… If you can’t control it, you can’t improve it… If you can’t improve it…then why the heck are
you doing it?
![Page 8: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/8.jpg)
So, What is Institutional Effectiveness?So, What is Institutional Effectiveness?
A sharpened statement of institutional mission and objectives
Identification of intended departmental/programmatic outcomes or results (Plan)
Establishment of effective means of assessing the accomplishments outcomes and results (Do, Act, Check)
FACT...
Institutional Effectiveness is primarily undertaken to improve what we do…not just to pass accreditation.
![Page 9: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/9.jpg)
Shewhart Cycle and SACSShewhart Cycle and SACSMacro IEMacro IE
Core Requirement 2.5:
“The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that (1) incorporate a systematic review of institutional mission, goals, and outcomes; (2) result in continuing improvement in institutional quality; (3) demonstrate the institution is effectively accomplishing its mission.”
PlanCheck and
Act
Do
![Page 10: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/10.jpg)
Key Points to Core Requirement 2.5Key Points to Core Requirement 2.5
Emphasizes an expectation that the institution is the primary focal point for compliance
Sets expectations for the description of planning and evaluation processes that are active and continuous rather than static or single occurrences.
Points to a clear and strong expectation for documentation of the systematic review of institutional mission, goals and accomplishments consistent with its mission
Sets expectations for the documented use of results of institutional planning and evaluation to achieve institutional improvements
![Page 11: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/11.jpg)
Shewhart and SACS, Cont.Shewhart and SACS, Cont.Micro IEMicro IE
Comprehensive Standard 3.3.1:
“The institution identifies expected outcomes for its education programs … and its administrative and educational support services; assesses whether it achieves those outcomes; and provides evidence of improvement based on analysis of those results.”
Plan
CheckDo and Act
![Page 12: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/12.jpg)
Key Points to Comprehensive Standard Key Points to Comprehensive Standard 3.3.13.3.1 Emphasizes the unit level of individual educational
programs and support services The expected achievements of educational programs and
support services should be articulated, and evidence presented concerning accomplishments
Distinguishes between program outcomes and learning outcomes
Sets expectations that improvement is guided by the establishment and evaluation of program and learning outcomes
![Page 13: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/13.jpg)
Shewhart and SACS, Cont.Shewhart and SACS, Cont.General Education and IEGeneral Education and IE
Comprehensive Standards 3.5.1
“The institution identifies college-level competencies within the general education core and provides evidence that graduates have attained those competencies.”
Plan
Do, Check, Act
![Page 14: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/14.jpg)
Key Points to Comprehensive Standard Key Points to Comprehensive Standard 3.5.13.5.1 General Education should be part of the institutional
mission The expected achievements of the General Education
program should be articulated, and evidence presented concerning accomplishments
Improvement should be guided by the establishment and evaluation of learning outcomes
![Page 15: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/15.jpg)
Overview of Institutional EffectivenessOverview of Institutional Effectiveness
Focus on Assessment
Comprehensive Dept./Program Review
Program Outcomes•Quality Indicators•Productivity Indicators•Viability Indicators
Continuous Improvement of Programs and Departments
Institutional Effectiveness
MissionStrategic Goals
Evaluation of LearningLearning Outcomes
•What graduates know•What graduates can do•What attitudes/values graduates possess
Continuous Improvement of Student Learning
![Page 16: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/16.jpg)
Institutional Effectiveness System at Institutional Effectiveness System at UNAUNA
Annual Report - Annual Action Plan and Assessment Report
Comprehensive Program and Department review – Five-year Review
Review of General Education – Five-year cycle of General Education assessment
![Page 17: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/17.jpg)
Schematic of Institutional Effectiveness Schematic of Institutional Effectiveness ProcessProcess
Year 1Annual Reports
Year 2Annual Reports
Year 3Annual Reports
Year 4Annual Reports
Year 5Annual Reports
Review of Strategic Goals
Five-Year Review for Selected Depts.
Five-Year Review for Selected Depts.
Five-Year Review for Selected Depts.
Five-Year Review for Selected Depts.
Five-Year Review for Selected Depts.
Area IFive-Year Assessment
Area IIFive-Year Assessment
Area IIIFive-Year Assessment
Area IVFive-Year Assessment
OverallGen. Ed. Assessment
•OIRPA•IE Committee•Gen. Ed. Committee
5-Year Cycle?
No
Yes
![Page 18: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/18.jpg)
Five-Year Program/Department Review Five-Year Program/Department Review Timeline Timeline (pending IE Committee Approval)(pending IE Committee Approval)
September October November December January February March April May June July August September
Last year’s Depts. that underwent 5-year review submits outcomes of review as AAPAR priority initiatives
OIRPA submits Five-Year Enrollment report to academic departments
OIRPA conducts assessment workshop for UNA campus
OIRPA meets with Deans/VP for overview
Deans/VPs meet with departments to discuss review
OIRPA meets with departments up for review
OIRPA initiates individual departments meetings
Five-Year Reviews completed and sent to Dean/VP
OIRPA submits overview of Five-Year process to IE Committee
![Page 19: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/19.jpg)
Annual Action Plan and Assessment Annual Action Plan and Assessment Report Timeline Report Timeline (pending IE Committee Approval)(pending IE Committee Approval)
September October November December January February March April May June July August September
President, VP, and Dean Initiatives Due
1st part of AAPAR due for current fiscal year w/ one Priority Initiative for next FY
Next FY Priority Initiatives by Deans
Next FY Priority Initiatives by VPs
SPBS reviews Next FY Priority Initiatives
2nd Part of AAPAR completed by depts.
OIRPA submits AAPAR overview to IE Committee
Budget initiatives based on Priority Initiatives are established
![Page 20: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/20.jpg)
OutcomesOutcomes
Operational Outcomes - measures of how well the institution/division/department is meeting/exceeding requirements
Learning Outcomes - statements of the knowledge, skills, and abilities the individual student possesses and can demonstrate upon completion of a learning experience or sequence of learning experiences (e.g., course, program, degree).
![Page 21: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/21.jpg)
Problems with OutcomesProblems with Outcomes
Outcomes are too broad Outcomes do not address core
requirements/competencies or mission Outcomes are not measurable
![Page 22: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/22.jpg)
Types of MeasurementTypes of Measurement
Discrete or Attributes data Binary data with only two
values
Continuous or Variable data
Information that can be measured on a continuum or scale
Yes/No Good Bad On/Off Male/Female Pass/Fail Height/Weight Temperature Test Scores Time Distance
![Page 23: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/23.jpg)
Bloom’s Taxonomy of Learning Bloom’s Taxonomy of Learning OutcomesOutcomes
Category Definition Related Behaviors
Knowledge
recalling or remembering something without necessarily understanding, using, or changing it
define, describe, identify, label, list, match, memorize, point to, recall, select, state
Comprehension
understanding something that has been communicated without necessarily relating it to anything else
alter, account for, annotate, calculate, change, convert, group, explain, generalize, give examples, infer, interpret, paraphrase, predict, review, summarize, translate
Application
using a general concept to solve problems in a particular situation; using learned material in new and concrete situations
apply, adopt, collect, construct, demonstrate, discover, illustrate, interview, make use of, manipulate, relate, show, solve, use
![Page 24: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/24.jpg)
Bloom’s Taxonomy, Cont.Bloom’s Taxonomy, Cont.Category Definition Related Behaviors
Analysis
breaking something down into its parts; may focus on identification of parts or analysis of relationships between parts, or recognition of organizational principles
analyze, compare, contrast, diagram, differentiate, dissect, distinguish, identify, illustrate, infer, outline, point out, select, separate, sort, subdivide
Synthesis
relating something new by putting parts of different ideas together to make a whole
blend, build, change, combine, compile, compose, conceive, create, design, formulate, generate, hypothesize, plan, predict, produce, reorder, revise, tell, write
Evaluation
judging the value of material or methods as they might be applied in a particular situation; judging with the use of definite criteria
accept, appraise, assess, arbitrate, award, choose, conclude, criticize, grade, judge, prioritize, recommend, referee, select, support
![Page 25: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/25.jpg)
Forms of MeasurementForms of Measurement
Longitudinal data is gathered over an extended period
Semester1 , ,Semester2 …Semester3 Semestert
![Page 26: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/26.jpg)
Forms of Measurement, Cont.Forms of Measurement, Cont.
Cross-sectional data represent a snapshot of one point in time
![Page 27: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/27.jpg)
What is Improvement?What is Improvement?
Measurable actions that increase learning, efficiency, effectiveness, and/or the bottom line
Decrease the BadIncrease the GoodDecrease Variability
![Page 28: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/28.jpg)
Decrease Variability?Decrease Variability? What the heck is that? What the heck is that?
Class A
100, 100 99, 98 88, 77 72, 68 67, 52 43, 42
Mean = 75.5
Class B
91, 85 81, 79 78, 77 73, 75 72, 70 65, 60
Mean = 75.5STD = 21.93 STD = 8.42
Mean
![Page 29: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/29.jpg)
Inputs, Processes, and OutcomesInputs, Processes, and Outcomes
OutcomesInput
Measurement Materials Methods
Environment People Machines
X’s
X’s
X’s
X’s X’s
X’sX’s Y’s
![Page 30: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/30.jpg)
Assessment StepsAssessment Steps
Develop learning/operational objectives Check for alignment between the
curriculum/business process and the objectives Develop an assessment plan Collect assessment data Use results to improve programs/department Routinely examine the assessment process and
correct, as needed
![Page 31: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/31.jpg)
Types of Assessment – DirectTypes of Assessment – DirectAcademicAcademic
Published Tests Locally Developed Tests Embedded Assignments and Course Activities Competence Interviews Portfolios
![Page 32: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/32.jpg)
Types of Assessment – DirectTypes of Assessment – DirectEducational Support/AdministrativeEducational Support/Administrative
People enrolled/participating/served Work accomplished Revenue generated Turnaround time Decrease in nonconformities
![Page 33: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/33.jpg)
Types of Assessment - IndirectTypes of Assessment - Indirect
Surveys Interviews Focus Groups Reflective Essays
![Page 34: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/34.jpg)
How Can OIRPA Assist?How Can OIRPA Assist?
Create university wide reports – Five-year departmental reports
Analyze university-wide assessment data – NSSE, CAAP
Hold workshops on assessment and IE Work with individual departments on annual reports,
program review, and outcomes assessment Provide ad hoc data reports for departments Work with committees to develop assessment plans –
IE Committee, Gen. Ed. Committee
![Page 35: Institutional Effectiveness at the University of North Alabama](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813866550346895da0164f/html5/thumbnails/35.jpg)
Questions or Comments?Questions or Comments?