the missing link: development of programmatic outcomes christopher meseke,ph.d. park university
TRANSCRIPT
What is Assessment? “As a whole, assessment is a framework for
focusing faculty attention on student learning and for provoking meaningful discussions of program objectives, curricular organization, pedagogy, and student development” (Allen, 2004).
Assessment is the quality control of the educational process
Common Reactions to Assessment Initiatives
• Ignoring it• Bribing someone else to do it• Complaining about it • Losing sleep over it• Sitting down and writing it
Levels of assessment: quality control and assurance Institutional/Programmatic National/State Licensure Exams Certain academic programs (Nursing,
Engineering, Physical Therapy, Social Work, etc) Accreditation Council for Graduate Medical
Education (ACGME) Higher Learning Commission of the North
Central Association of Colleges and Schools
Assessment is important to the accreditation process The Higher Learning Commission of the
North Central Association of Colleges and Schools (HLC-NCA) requires it.
Assessment of student learning provides evidence at multiple levels: course, program, and institutional.
Assessment of student learning includes multiple direct and indirect measures of student learning.
Assessment results inform improvements in curriculum, pedagogy, instructional resources, and student services.
Interpretation of the HLC-NCA Statement on Assessment on Student Learning, 2003. It is the faculty members who must
take ownership of the assessment process. Their buy-in, ownership, and implementation is directly related to both the mission of the institution and attitudes, knowledge, and skills required for the student to successfully complete the program requirements.
Goals of Program Assessment Planning
Measure student learning, not teaching, curricular content, processes or resources
Measure things that are important to us Involve faculty in development of the
assessment program, processes and instruments
Use multiple measures to produce valid and reliable data (triangulation)
Make it manageable and cost affordable
Goals of Program Assessment Planning
Map assessment data back to the curriculum for improvement
Produce annual reports of assessment outcomes that show the data, interpretation of data, and improvement plans where indicated
Same assessment procedures for all campuses and sister programs
Modified Hatfield Assessment Model
Meta-Competencies/Institutional Goals
objective
objective
objective
Competency
Learning Outcome
objective
CompetencyCompetency
indicators
Degree to which Competency is achieved
Learning events
Programmatic Competencies A measurable, complex behavioral
statement may be written for each Key Component (meta-competency) and competency.
Statements reflect what we think we can actually measure in an educational environment considering time and resources.
Program Assessment Learning Competencies Develop a set of Learning Goals (Meta-
competencies) which represent the attributes of the graduate
General Education (Competencies) Discipline (Competencies)
Course (Outcomes)
KNOWLEDGECOMPREHENSION
APPLICATIONANALYSISSYNTHESISEVALUATION
CiteCountDefineDraw
IdentifyList
NamePointQuoteRead
ReciteRecordRepeatSelectState
TabulateTell
TraceUnderline
AssociateClassifyCompareComputeContrast
DifferentiateDiscuss
DistinguishEstimateExplainExpress
ExtrapolateInterpolate
LocatePredictReportRestateReview
TellTranslate
ApplyCalculateClassify
DemonstrateDetermineDramatize
EmployExamineIllustrateInterpretLocate
OperateOrder
PracticeReport
RestructureScheduleSketchSolve
TranslateUse
Write
AnalyzeAppraiseCalculate
CategorizeClassifyCompareDebate
DiagramDifferentiateDistinguish
ExamineExperiment
IdentifyInspect
InventoryQuestionSeparate
SummarizeTest
ArrangeAssemble
CollectComposeConstruct
CreateDesign
FormulateIntegrateManageOrganize
PlanPrepare
PrescribeProduceProposeSpecify
SynthesizeWrite
AppraiseAssessChoose
CompareCriticize
DetermineEstimateEvaluate
GradeJudge
MeasureRankRate
RecommendReviseScoreSelect
StandardizeTest
Validate
Lower Order Bloom’s verbs
KNOWLEDGECOMPREHENSION
APPLICATIONANALYSISSYNTHESISEVALUATION
CiteCountDefineDraw
IdentifyList
NamePointQuoteRead
ReciteRecordRepeatSelectState
TabulateTell
TraceUnderline
AssociateClassifyCompareComputeContrast
DifferentiateDiscuss
DistinguishEstimateExplainExpress
ExtrapolateInterpolate
LocatePredictReportRestateReview
TellTranslate
ApplyCalculateClassify
DemonstrateDetermineDramatize
EmployExamineIllustrateInterpretLocate
OperateOrder
PracticeReport
RestructureScheduleSketchSolve
TranslateUse
Write
AnalyzeAppraiseCalculate
CategorizeClassifyCompareDebate
DiagramDifferentiateDistinguish
ExamineExperiment
IdentifyInspect
InventoryQuestionSeparate
SummarizeTest
ArrangeAssemble
CollectComposeConstruct
CreateDesign
FormulateIntegrateManageOrganize
PlanPrepare
PrescribeProduceProposeSpecify
SynthesizeWrite
AppraiseAssessChoose
CompareCriticize
DetermineEstimateEvaluate
GradeJudge
MeasureRankRate
RecommendReviseScoreSelect
StandardizeTest
Validate
Higher Order Bloom’s verbs/Upper division
Course / Program outcomes
An Example of Departmental/Programmatic Competencies Students will be able to:
Demonstrate biological knowledge appropriate for the course level.
Demonstrate a working knowledge of the scientific method.
Demonstrate the ability to communicate scientific concepts and findings, both in oral and written format.
Apply interdisciplinary knowledge to the biological sciences.
Demonstrate an awareness of ethical issues in the life sciences.
Program Competencies
Course1
x
x
x
Course2
x
x
x
x
Course3
x
x
x
Course4
x
x
x
x
x
x
x
x
Course5
Mapping Outcomes to Courses
Program Competencies
Course1
x
x
x
Course2
x
x
x
x
Course3
Course4
x
x
x
x
x
x
x
x
Mapping Outcomes to CoursesCourse
5
Triangulation of Assessment Strategies: Direct Measures (Pick 2)
Licensure Scores GRE/GMAT/MCAT/other Departmental Exit Exams Portfolio Core Assessment Capstone Experience Others
Triangulation of Assessment Strategies: Indirect Measures (Pick 1)
Advisory Boards Senior Survey Employer/Professional
School Survey Focus Groups/Interviews Other
Three-Year Assessment Plan All departments/programs should be
assessed on three-year cycles Not all department/program competencies
should/can be assessed Focus on 2-3 competencies at most
Establish meaningful criteria At end of three-year cycle examine data for
meaningful trends Only after the three-year cycle are changes
made, only if needed
An example of a three-year assessment cycle
Three-Year Schedule of Program Assessment Plans Submissions to the Assessment Committee 2013-2014 2014-2015 2015-2016
Program G/UG Accounting UG X Adult Education G X Athletic Training UG XBiology UG X Business Administration G XBusiness Core (CPC) UG XChemistry UG X Communication Arts UG X Communication and Leadership G X Computer Information Systems UG X Criminal Justice UG X Economics UG X
Early Childhood Education (Educational Studies) G/UG X Educational Leadership G X Elementary Education UG XEngineering UG X English UG X Finance UG X Fine Arts UG X Geography UG X Graphic Design UG X Healthcare Leadership G X Healthcare Management UG X History UG X Human Resource Management UG X Information and Computer Science UG X Interior Design UG X International Business UG X International Center for Music G/UG X Language and Literacy G X Legal Studies UG X Liberal Education UG X Liberal Studies UG X Logistics UG X Management UG X Management Information Systems UG XMarketing UG X Mathematics UG X Middle and Secondary Education UG X Modern Languages UG X Nursing UG X Political Science UG X Psychology UG X Public Administration (BPA) UG X Public Affairs (MPA) G X Social Psychology UG X Social Work UG X Sociology UG X Teacher Leadership G X
Summary Choose an Assessment Model Identify (develop) programmatic competencies Write program level competencies based upon
the Institutional meta-competencies. Establish key performance indications
(KPOs)based upon the competencies Identify 3 measures for each competency (2
direct, 1 indirect) and set success criteria (may be different across various university units)
Develop data gathering and reporting mechanisms and templates
Develop close-the-loop mechanisms
Big Mistakes in Assessment• Assuming that it will go away
• Trying to do too much, too soon
• Expecting to get it right the first time
• Not considering implementation issues when creating plans
• Borrowing plans and methods without acculturation
Big Mistakes in Assessment
• Demanding statistical research standards
• Doing it for accreditation instead of improvement
• Confusing assessment with student learning
• Making assessment the responsibility of one individual
• Assuming collecting data is doing assessment