1

CS-90Strategies for Aligning Assessment for

Program Improvement with Accountability for Individual Student

Learning

Thomas P. Pusateri, Ph.D.Florida Atlantic University

pusateri@fau.edu

Concurrent session110th Annual Meeting of the Commission on

CollegesSouthern Association of Colleges and Schools

Atlanta Hyatt Regency HotelDecember 5, 2005, 3:45-4:45 PM

2

http://iea.fau.edu

3

4

5

Background Information• Increased emphasis on identifying and assessing core

student learning outcomes in higher education (e.g., Measuring Up 2004).

• The Florida Board of Governors included student achievement as 1 of 7 of its accountability measures.

• A Board of Governors’ resolution (April 22, 2004) directed university personnel to develop Academic Learning Compacts (ALCs) and related processes.

6

What are ALCs? “University personnel will provide students and prospective

students with Academic Learning Compacts, which include concise statements of what active and successful participants in the joint teaching-learning process will know and be able to do, expressed in terms of the core student learning outcomes embodied in the requirements for each baccalaureate degree. The Compacts also will list the types of assessments used in the program. Compacts provided to students must be written in a user-friendly, jargon-free format.”

Policy Guideline 04.08.26State University System (of Florida)

Division of Colleges and Universities

Florida Atlantic University’s is realigning its assessment system to accommodate Academic Learning Compacts.

7

University personnel will develop Academic Learning Compacts that:

I.a. Identify, at a minimum, the expected core student learning outcomes for program graduates in the areas of

i. content/discipline knowledge and skills;

ii. communication skills*;

iii.critical thinking skills*; * these skills may be defined at either the university level or the program level

FAU’s programs identify one or more of each of the following outcomes:

• Content knowledge– Vocabulary, Theories– Research skills– Technical skills

• Communication skills*

Written–Oral–Other–Team

• Critical Thinking skills*

Analytical – Creative –Practical

* FAU’s Writing Across the Curriculum requirement

BoG Policy Guideline FAU’s Realignment

8

I.b. Identify corresponding assessments used to determine how well student learning matches those articulated expectations.

Where/When are students assessed?

How are students held to performance standards?

Departments assess all students through one or more of the following:

• Core required courses

• Capstone experience– Senior thesis– Senior

project/portfolio– Field placement

• Standardized test

BoG Policy Guideline FAU’s Realignment

9

Departments establish standards through one or more approaches:

• Standardizing courses– Common syllabi– Common

expectations– Common assignments

• Grading rubrics– Faculty define levels

of student performance

– Periodic calibration• Setting cut-off scores on

required exams

I.b. Identify corresponding assessments used to determine how well student learning matches those articulated expectations.

Where/When are students assessed?

How are students held to performance standards?

BoG Policy Guideline FAU’s Realignment

10

2. University personnel will develop the evaluation systems (including external validations) necessary to corroborate that the assessments referenced above measure student achievement against the expected core learning outcomes and results will be used to improve student achievement and program effectiveness.

FAU’s departments rely on one or more systems:

• Multiple raters– Juried performances– 2+ faculty members– Non-faculty raters

• SamplingCollection and periodic review of student work

• Comparisons to norms on standardized tests

BoG Policy Guideline FAU’s Realignment

11

[U]niversity personnel shall develop an action plan that will be approved by their university’s board of trustees and submitted to the Division of Colleges and Universities. The plan shall include a. a proposed timeline for

developing policies and implementing procedures to capture each element outlined above, as well as in making Academic Learning Compacts readily available to students.

FAU’s action plan• Briefings for Board of

Trustees and University Undergraduate Programs Committee (Fall 04)

• Consultations with academic departments (04-05)

• Progress report to Faculty Senate Chair (Mar 05)

• Board of Trustees - Workshop (Apr 05)- Approves timeline (May 05)

• Faculty Senate approves ALC Guidelines (Oct 05)

• Board of Trustees approves ALC Policies (Nov 05)

• ALCs will be made available to students (Dec 05)

BoG Policy Guideline FAU’s Realignment

12

13

14

REVIEW OF ALCsUniversity personnel are expected to demonstrate how information from the periodic review of student learning outcomes, as well as from the evaluation of corresponding assessment mechanisms, has been used to improve student achievement and program effectiveness.

Departments will report their evaluation

(assessment) strategies by

February 1, 2006.

BoG Policy Guideline FAU’s Realignment

15

Reporting samples from core required courses or capstones on the distribution of scores obtained on embedded questions, key assignments, rubrics, etc.

16

Assessing multiple raters

17

Assessing multiple raters

18

FAU’S DEFINITIONS OF CONTENT KNOWLEDGE IN THE DISCIPLINE

Declarative knowledge: Students will demonstrate knowledge of the vocabulary, history, theories or concepts specific to the discipline. Declarative knowledge is usually assessed via in-class or standardized tests, typically in an objective (multiple-choice; short answer) format.

Procedural knowledge: Research skills: Students will demonstrate knowledge of the procedures involved in discipline-specific research (e.g., idea generation, literature review, data collection, reporting). These skills are usually measured via the evaluation of a research project, thesis, or dissertation.

Procedural knowledge: Technical skills: Student will demonstrate technical skills related to the discipline (e.g., developing a business report or lesson plan, mastering a musical instrument, learning how to use statistical software). This type of procedural knowledge may be assessed via an in-class project (e.g., case study, paper), observation (e.g., juried performance), or portfolio.

19

FAU’S DEFINITIONS OF COMMUNICATION SKILLS

Written communication: Students will produce writing that is grammatically correct, well-organized, and properly formatted for the purpose of the assignment and the discipline. This is usually assessed via individual paper assignments where instructors assess the quality of written expression.

Oral communication: Students will prepare and deliver informative and/or persuasive oral presentations that attend to the audience and are well-organized. This is usually assessed via individual oral presentations in a classroom setting, although sometimes instructors and/or supervisors may assess oral communication across time (e.g., during classroom discussions or at an internship site).

Other forms of communication: Students will prepare and present information and/or persuasive material using media that may or may not include written or oral communication (e.g., a musical or theatrical performance; a visual, audiovisual, or graphical product). This is usually assessed via classroom projects, juried performances, and/or portfolios.

Team/Collaborative communication: Students will demonstrate team-oriented, collaborative skills in which they contribute to group products. This is usually assessed via group papers or presentations and may involve peer-assessment (e.g., group members assess each other) and/or assessment by audience members (e.g., an instructor, supervisor, or peer).

20

b. A description of how university personnel will certify that each baccalaureate graduate has completed a program with clearly articulated core student learning expectations… and that corresponding robust and effective assessment mechanisms have been used to ensure that graduates have met the criteria of the Compacts.

FAU will certify students through the awarding of a baccalaureate degree from programs that have articulated ALCs and that require students to achieve acceptable performance standards prior to graduation via completion of one or more of the following: (a) senior capstone course, (b) exit examination(c) a series of core required courses.

BoG Policy Guideline FAU’s Realignment

21

FAU’S DEFINITIONS OF CRITICAL THINKING SKILLS*

Analytical skills: Students will analyze, evaluate, compare/contrast or judge discipline-specific theories, issues, events, or other content. This is usually assessed through examining the quality of argument in a student essay, oral presentation, or formal report.

Creative skills: Students will create a product by synthesizing knowledge from a discipline (e.g., create a new piece of art, interpret a piece of music in a personal and appropriate way, develop a new theory or research proposal). This is usually assessed by an instructor or supervisor examining a student product for the quality of its synthesis of current knowledge into a new product.

Practical skills: Students will put into practice their knowledge and skills within a discipline (e.g., developing a business proposal or lesson plan, writing a grant). This is assessed ideally by examining an authentic product the student employs in an appropriate setting (e.g., a student teacher delivering a lesson, an intern proposing a plan to a supervisor); it may also be assessed via in-class projects in which students propose practical solutions based on their understanding of the discipline (e.g., case study analysis).

*Based on Robert Sternberg’s Theory of Successful Intelligence

22

EXAMPLES OF ALCs CONTENT KNOWLEDGE

PSYCHOLOGY (Terms, theories): Graduates in Psychology will understand basic concepts, theories, and experimental findings in four core areas of psychology (cognitive, developmental, social, and psychobiology).

HISTORY (Research & Technical skills): Students will demonstrate the ability to conduct archival research utilizing both primary and secondary sources. They will develop procedural and technical skills involved in historical research such as library research, oral history, book review writing, and proper citations.

FINANCE (Technical skills): Students will develop proficiency in the use of Microsoft Excel to solve financial problems. 

23

EXAMPLES OF ALCs COMMUNICATION SKILLS

GEOGRAPHY: Graduates in Geography will be able to produce writing that is grammatically correct and well-organized and to deliver clear and well-organized oral presentations. Both written and oral communications should illustrate clear communication skills utilizing graphic tools in geographic information systems, remote sensing and other techniques.

REAL ESTATE: Students will demonstrate their abilities to report the results from their feasibility analyses both orally and in written form in a clear and grammatically sound manner. Students will also demonstrate skills in discussing the analyses presented by their peers.

TEACHER EDUCATION: Students will demonstrate the ability to communicate effectively verbally and in writing (EAP 2, Indicator 2.2) by successfully developing and implementing a lesson plan.

24

EXAMPLES OF ALCs CRITICAL THINKING SKILLS

ENGLISH: Students will analyze/interpret a complex literary work, demonstrating the capacity to formulate and develop in a formal essay a significant topic in depth and in an independent and even original way. Essays are expected to exhibit mature creative and analytical thinking, and an ability to create a mature critical argument quickly with no external assistance.

COMPUTER ANIMATION IN ARTS: Students will demonstrate proficiency in understanding and apply the stages of problem-solving skills (e.g. research, thumbnail, roughs, comprehensive and mechanical preparation) as applied to 3D computer arts.

ELECTRICAL ENGINEERING: Design and conduct scientific and engineering experiments including analysis and interpretation of data. Deliver engineering results that meet performance standards for cost, safety, and quality. Make and defend ethical judgments in keeping with professional standards.

25

CORE REQUIRED COURSES

PSYCHOLOGY: For each of the courses listed below, faculty teaching that course will develop a master syllabus which will include student learning outcomes to be addressed in all sections of the course. Faculty will share strategies for developing course assignments (e.g., exams, exercises, term papers) to assess student attainment of the learning outcomes. The courses are:EXP 3505: Cognition,DEP 3054: Psychology of Human Development,SOP 3004: Social Psychology, and PSB 3002: Biological Bases of Behavior.

MUSIC: All students must complete four specific courses in the history and literature of music: MUH 3013 (Introduction to Music History), MUH 4211 (Music of Western Civilization I), MUH 4212 (Music of Western Civilization II), and MUH 4371 (20th Century Music History).

26

CAPSTONE EXPERIENCE

SOCIAL WORK: All BSW students are required to take SOW 4510 (Field Education in Social Work), which includes a classroom field seminar plus 400 agency-based hours supervised by a professional social worker. This experience integrates classroom knowledge from the Human Behavior in the Social Environment (HBSE) sequence, practice sequence, research sequence, and policy sequence; students use this knowledge base to develop skills to intervene with various clients.

HONORS COLLEGE: During the senior year, students will enroll in thesis research and thesis writing and will produce a draft honors thesis to be evaluated. To prepare for writing a senior thesis, students will complete (1) courses in their concentration aimed at providing the content knowledge needed to write a thesis, (2) writing courses aimed at providing competence in effective written communication, and (3) courses in the core and concentration designed to build critical thinking skills.

27

STANDARDIZED TEST

NURSING: Approximately three months prior to graduation, all seniors will take the Health Education Systems, Inc. (HESI) examination. This test is a valid predictor of performance on the National Licensing Examination (NCLEX). Students must achieve a score of 850 or better on the HESI to graduate. Students who do not receive a passing score are provided with advising and a remediation plan and the opportunity to re-take the exam; they will not graduate until their score is 850 or better.

OCEAN ENGINEERING: One of the requirements for graduation is that a student must take the nationally-administered Fundamentals of Engineering (FE) examination. The examination tests one's knowledge of basic and fundamental engineering subjects: statics, dynamics, thermodynamics, fluid mechanics, computer programming, engineering economics, electrical networks and strength of materials.

28

STANDARDIZING COURSES

PSYCHOLOGY: Faculty teaching core courses will develop a master syllabus which will include student learning outcomes to be addressed in all sections of the course. Faculty will share strategies for developing course assignments (e.g., exams, exercises, term papers) to assess student attainment of the learning outcomes.

ACCOUNTING: Faculty members teaching the undergraduate accounting core courses: ACG 3131 (Financial Reporting), ACG 3341 (Managerial Decision Making and Accounting)., and TAX 4001 (Federal Taxation I) are required to teach a set number of chapters from a common text selected by departmental decision. Faculty members teaching a particular course will develop and periodically review a common list of educational objectives and the topics and critical thinking skills appropriate to these objectives to teach in each course.

29

GRADING RUBRICS

ENGLISH: Essays in all 3000- and 4000-level English courses will be assessed holistically according to a rubric developed and shared by faculty and communicated to students.

SUPERIOR Addresses the question or prompt fully

Demonstrates substantial comprehension of relevant material Shows substantial depth, complexity, and creativity of thought

Demonstrates clear and coherent organization Develops arguments fully with ample supporting detail

Demonstrates superior control of diction, syntactic variety, and transitions

STRONGAddresses the question or prompt substantively, yet not fully

Demonstrates comprehension of relevant material Shows depth, complexity, and creativity of thought

Demonstrates clear organization Develops arguments with supporting detail

Demonstrates control of diction, syntactic variety, and transition

30

ENGLISH RUBRIC (continued)

COMPETENT Adequately addresses the question or prompt

Demonstrates adequate understanding of relevant material Shows clarity of thought but may treat the topic simplistically or

repetitively Demonstrates adequate organization

Develops arguments adequately, with some detail Demonstrates adequate facility with syntax, mechanics, and usage

but contains some errors

INADEQUATE May distort or neglect parts of the question or prompt, and/or

Fails to comprehend relevant material Lacks clarity of thought; demonstrates confused/simplistic thinking,

and/orLacks adequate organization, and/or

Fails to provide adequate or appropriate details to support generalizations, or may provide details without generalization, and/or

Demonstrates significant errors in language, syntax, or mechanics

31

HONORS COLLEGE RUBRIC:

Honors with distinction: The thesis demonstrates a profound knowledge of the fundamental concepts in the subject area. The thesis shows mastery of grammar, syntax, structure and style: it is lucid, well-organized, and stylistically elegant. The thesis engages in sophisticated critical thinking and rigorously considers alternate resolutions of a problem in an exemplary fashion or otherwise shows sophisticated critical thinking in an exemplary manner appropriate to the area of study.

Honors: The thesis shows a thorough knowledge of the fundamental concepts in the subject area. The thesis shows effective command of grammar, syntax, structure and style: it is clearly written and sensibly organized. The thesis engages in sophisticated critical thinking and explores alternate approaches or resolutions of a problem effectively or otherwise shows sophisticated critical thinking in a manner appropriate to the area of study.

Unacceptable: The thesis does not show competent knowledge of the fundamental concepts in the subject area. The thesis does not show mastery of grammar, syntax, structure, or style. The thesis does not competently engage in critical thinking or explore alternate approaches or resolutions of a problem.

32

CUT-OFF SCORE ON STANDARDIZED EXAM

NURSING: Approximately three months prior to graduation, all seniors will take the Health Education Systems, Inc. (HESI) examination. This test is a valid predictor of performance on the National Licensing Examination (NCLEX). Students must achieve a score of 850 or better on the HESI to graduate. Students who do not receive a passing score are provided with advising and a remediation plan and the opportunity to re-take the exam; they will not graduate until their score is 850 or better.

COMPARISON TO NORMS ON STANDARDIZED TESTS

33

MULTIPLE RATERS

MUSICCONTENT KNOWLEDGE (Technical skills) andCOMMUNICATION andCRITICAL THINKING (Creative skills): Graduates will have ability to perform a cross section of the music from the complete repertoire of the performance medium and exhibit the appropriate technical skills for artistic self-expression.

Students in the Bachelors Music program will be evaluated by the appropriate faculty at jury performances each semester. They are also required to successfully complete a senior recital, which must include representative examples from the appropriate repertoire for the performance medium. A pre-hearing evaluated by three faculty members must be approved before the recital is scheduled.

34

SAMPLING

LANGUAGES & LINGUISTICS: Will randomly sample the performance of students writing term papers for program-specific 3000- and 4000-level courses to compare demonstrated research skills for these with the research-skill performance of students taking and passing FOL 3880.

Will randomly sample the accuracy of linguistic structures and attention to linguistic nuance in the essays of students enrolled in selected 3000- and 4000-level courses to compare rankings of student performance in linguistics-specific courses with rankings of students’ linguistic awareness as evidenced in other courses.

JEWISH STUDIES: Periodically selected exams and papers will be read by members of the Jewish Studies Executive Committee to ensure that the criteria of the Academic Learning Compacts are being met by individual students.

35

National Center for Public Policy and Higher Education “Measuring Up” 2004 report

INCOMPLETE

PLUS

36

Florida Board of Governors Accountability Measures

1. Number/Percent of students from underserved populations enrolled/graduating

2. 4- and 6-year graduation rates

3. Access to and production of degrees

4. Meeting statewide workforce needs

5. Research expenditures from external sources

6. World-class academic research programs

7. Student learning: Academic Learning Compacts

37

38