creating a highly effective assessment system - abhe.org · students should be able to demonstrate,...

Creating a Highly Effective Assessment System Dr. Jennifer B. King ABHE | February 21, 2018

OUTCOMES FRAME THE ASSESSMENT SYSTEM

SESSION 1:

ENGAGE • EDUCATE • EQUIP • EMPOWER

(8:30AM – 9:30AM)

In this session…

© 2018 Dr. Jennifer B. King | Jen B. King's Consulting LLC 3

Participants will… �  Define assessment. �  Discuss assessing vs. grading. �  Compare grading rubrics and outcomes-only rubrics. �  Discover the six components of an assessment system. �  Define component #1 (Outcomes). �  Determine the three qualities of well-written outcomes. �  Use a checklist to evaluate outcomes.

What is Assessment?


Assessment is the systemic, methodical collection, review, and use of information about educational programs undertaken for the purpose of improving student learning (Palomba & Banta, 1999).

Assessment is for the benefit of the student!


… is not grading


grading


grading Skills are assessed. To evaluate skills, the assessor compares student work to the work of an expert.

Tasks are graded. The assessor knows what is typical at a given level and judges how close the student work is to that level. Student work is compared to other student work.


Advanced Proficient Basic In-Progress

CRITERIA 5 4 3 2 1 0

How well directions are followed

Directions are followed to the fullest extent.

All directions are followed.

Many directions were followed.

Some directions were followed.

Few directions were followed.

No directions were followed.

Neatness Completed artwork is exceptionally neat.

Completed artwork is neat.

Completed artwork is neat, with some aspects not neat.

Completed artwork is more messy than neat.

Artwork is incomplete and extremely messy.

Artwork is incomplete.

Work habits Stayed on task during the allotted time with superior focus.

Stayed focused during the allotted time without reminders to stay on task.

Worked for most of the allotted time, but needed a few reminders to stay on task.

Worked some of the allotted time and needed several reminders to stay on task.

Worked very little of the allotted time and did not heed the reminders to stay on task.

Did not work during the allotted time.

Expectations Exceeded expectations for his/her ability.

Fully met expectations for his/her ability level.

Met most expectations for his/her ability level.

Met some expectations for his/her ability level.

Met a few expectations for his/her ability level.

Has not met the expectations for his/her ability level.

Rubric for Grading


Novice/1 Emerging/2 Competent/3 Target/4 Exceptional/5

Imaginative Expression

Work reflects an indiscriminate understanding of an idea.

Work shows a basic understanding of conventional concepts, imagery, and forms.

Composition demonstrates a personal interpretation of an idea characterized by conventional visual representations using traditional forms and design.

Visual expression exhibits an individual approach in the development of concept and context, imagery is mostly original in source and form.

Visual expression embodies a strong and uniquely individual conceptual and contextual development of visual imagery presented in original forms and design.

Purposeful Exploration

Work portrays a minimal use of formal and personal knowledge in expressing an idea.

Work shows basic awareness of how context, personal knowledge and experience, and available resources inform the creative process.

Composition demonstrates a relevance of context, personal knowledge and available resources in forming and communicating ideas.

Visual expression exhibits a conscious and thoughtful reflection on cultural contexts and an intentional use of personal knowledge and available resources to inform the creative process.

Visual expression represents thoroughly examined cultural contexts, personal knowledge and available resources to inform creative intent, the creative process and individual choices.

Formal Qualities

Work reflects a basic knowledge of design qualities.

Work shows conservative application of design elements and principles.

Composition demonstrates understanding and competency in use of design systems, principles and elements in support of a personal perspective.

Visual expression exhibits a purposeful and thoughtful application of design structures and the elements and principles of design in support of creative intent.

Visual expression represents an excellent command of appropriate design systems, elements, and principles which strengthens creative intent.

Technical and Media Skills

Work reflects a limited proficiency of media techniques and tools.

Work shows sufficient control of media, techniques, and tools.

Composition demonstrates a personal preference and competent use of media, techniques and tools in support of a personal perspective.

Visual expression exhibits a purposeful and thoughtful application of media, techniques and tools in support of creative intent.

Visual expression represents adventurous choices and excellent command of appropriate media, techniques, and tools which strengthens creative intent.

Rubric for Assessing Outcomes


assessment is a system… We start with a seed, which grows into a plant. How do we measure plant growth over time? What is our measure of success? How will we know that we have succeeded?

What is an Assessment System?


Six Components 1.  Outcomes (Institutional, Program, Course) 2.  Programs/Courses 3.  Key Assessments 4.  Student Work/Exemplars 5.  Scoring Criteria/Rubrics 6.  Data/Decision-Making

#1 Component Outcomes – The Essential Component


Outcomes are statements describing the knowledge, skills and dispositions students should be able to demonstrate at the end

of their academic program.

Student Learning Outcomes Student learning outcomes focus on the more particular skills, knowledge, and attitudes that students learn in courses and programs.

Outcomes are not…


!! Vision Statements !! Mission Statements !! Values Statements !! Accreditation Benchmarks


!! STANDARD 1: Mission, Goals, and Objectives

!! STANDARD 2: Student Learning, Institutional Effectiveness, and Planning !! 2a Assessment of Student Learning and Planning

!! STANDARD 3: Institutional Integrity

!! STANDARD 4: Authority and Governance

!! STANDARD 5: Administration !! STANDARD 6: Institutional Resources

!! STANDARD 7: Enrollment Management

!! STANDARD 8: Student Services

!! STANDARD 9: Faculty

!! STANDARD 10: Library and Other Learning Resources !! STANDARD 11: Academic Programs

© 2018 Dr. Jennifer B. King | Jen B. King's Consulting LLC

Institutional Learning Outcomes Broad holistic statements that apply to all students regardless of field of study. ILOs guide individual departments and disciplines in the development of student learning outcomes for programs and courses and services

Program Outcomes Program holistic statements that apply to specific student groups within a given program of study.

Course Outcomes Explicit statements about knowledge, skills and behaviors to be demonstrated by the conclusion of the course.

16

Types of Outcomes

Communications SLOs Example

Institutional Level: Graduates will be able to communicate ideas, perspectives, and values clearly and persuasively, while listening openly

to others.

Program Level: Teacher candidates will be able to

communicate effectively by listening, speaking, writing, reading, and visual representations.

Course Level: “In this course, the student will use a variety of effective communication techniques to

foster active inquiry in the classroom.


Quantitative Reasoning SLOs Example

Institutional Level: Graduates from our campus can apply quantitative reasoning to real-world problems.

Program Level: Students who complete the Psychology program can use statistical tools to analyze and interpret data from psychological studies.

Course Level: Students who complete PSY303: Clinical Statistics can calculate and interpret a variety of descriptive and inferential statistics.


Three Qualities of (Really) Great Outcomes


1.! Realistic 2.! Measurable 3.! Observable

Realistic


Make sure that outcomes are attainable. Outcomes need to be reviewed in light of student ability, developmental levels, their initial skill sets, and the time available to attain these skills sets (i.e., four years).

SLO 1ST YR Novice

1

2ND YR Emerging

2

3RD YR Competent

3

4TH YR Advanced

4

Exceeds Expectations

5

Students will be able to produce written work that is substantive, organized, and grammatically accurate.

Realistic Make sure that outcomes are to be reviewed in light of student ability, developmental levels, their initial skill sets, and the time available to attain

Measurable


Explain in clear and concise terms the specific skills students should be able to demonstrate, produce, and know as a result of the program’s curriculum. “Students completing the BS in Chemistry should be well-practiced in the relevant skills of the field” is too vague. We do not know what the “relevant skills” of the field of chemistry include. This will create problems in measuring student behavior and drawing valid conclusions about the program’s success.

Measurable Explain in clear and concise terms the students should be able to demonstrate, produce, and know as a result of the program’s curriculum.

“Students completing the BS in Chemistry should be well-practiced in the relevant skills of the field” what the “relevant skills” of the field of chemistry include.

This will create problems in measuring student behavior

Observable


“Students will develop an appreciation of, and will be exposed to…” are latent terms that will be difficult to quantify. When composing learning outcomes, it is important to rely on concrete action verbs that specify a terminal, observable, and successful performance as opposed to passive verbs that are not observable.

Observable “Students will develop an appreciation of, and will be exposed to…”

When composing learning outcomes, it is important


Blo

om’s

Taxo

nom

y (1

956)

Low- to Higher-Order Skills

When in doubt, say “demonstrate.”

Learning Outcomes Checklist


! Observations about Outcomes Can be directly measured and observed

Maps directly to curriculum

Focuses on student learning outcomes and not teaching activities

Relies on action verbs in future tense

Is useful to identify areas to improve

Describes what students are intended to do, know and produce Adapted from Maki (2004) Assessing for Learning: Building a Sustainable Commitment Across the Institution

Visual Arts Example


Institutional Level: Graduates from our campus can apply artistic and reasoning skills to create new or improved products to impact our world.

Program Level: Students who complete the Art program can combine and apply artistic skills to imagine, create, realize and refine

artworks in conventional and innovative ways.

Course Level: Students in ART102: Visual/Media Arts will use common, every day materials to develop skills for creating visual artwork.

will use common, every day materials to develop skills for creating


Program: Students who complete the Art program can combine and apply artistic skills to imagine, create, realize and refine artworks in conventional and innovative ways. Visual Arts Student Learning Outcomes (SLOs)

Describes w

hat students are intended to do, know

, produce

Is useful to identify areas to im

prove

Relies on action verbs in

future tense

Focuses on student learning outcom

es and not teaching activities

Maps directly to

curriculum

Can be directly

measured and observed

Find curiosity, interest and enjoyment in their own creativity and that of others.

Have knowledge of the skills needed for the creation of visual and performing art work.

Understand the language, concepts and principles of visual and performing arts.

Make and appreciate visual and performing art work.

Share their thoughts and ideas through visual and performing arts.

Be open to visual and performing art forms across time, place and cultures, and perceive the significance of these art forms as an integral part of life.

Activity: Evaluate Outcomes (Consider the Three Qualities of (Really) Great Outcomes)

Adapted from Maki (2004) Assessing for Learning: Building a Sustainable Commitment Across the Institution


Q & A

OUTCOMES DETERMINE KEY ASSESSMENTS

SESSION 2:


(10:30AM – 11:30AM)

In this session…


Participants will… �  Write student learning outcomes. �  Assign outcomes to component #2 (Programs/Courses). �  Define component #3 (Key Assessments/Critical Tasks). �  Discover types of student work, component #4. �  Create an exemplar.

Component #2 Programs/Courses


Academic programs comprise the core, required, and elective courses that lead to a degree or certificate.

Visual Arts Example


Institutional Level: Graduates from our campus can apply artistic and reasoning skills to create new or improved products to impact our world.

Program Level: Students who complete the Art program can combine and apply artistic skills to imagine, create, realize and refine

artworks in conventional and innovative ways.

Course Level: Students in ART102: Visual/Media Arts will use common, every day materials to develop skills for creating visual artwork.

will use common, every day materials to develop skills for creating

32 © 2018 Dr. Jennifer B. King | Jen B. King's Consulting LLC

Activity: Write/Revise student learning outcomes

Visual Arts Student Learning Outcomes (SLOs)

Art Courses ART101 ART102 ART103

Experience and develop curiosity, interest and enjoyment in their own creativity and that of others.

Acquire and develop skills needed for the creation of visual and performing art work.

Use the language, concepts and principles of visual and performing arts.

Create visual and performing art work.

Communicate their thoughts and ideas through visual and performing arts.

Develop receptiveness to visual and performing art forms across time, place and cultures, and perceive the significance of these art forms as an integral part of life.


Activity: Write/Revise student learning outcomes

ABHE Feb. 21 Visual Arts Student Learning Outcomes (SLOs)


Express personal creativity through visual art.

Demonstrate skills by the creation of visual artwork.

Use the language, concepts and principles of visual arts to critique visual arts projects of various genres.

Create visual artwork that reflects mastery of the skills in visual arts.

Suggestions: Writing (or Rewriting) Outcomes


•  Make it realistic, attainable – Can an undergraduate student demonstrate the outcome? Is the outcome to simple for a graduate student?

•  Make it measurable – Are the specific skills to be demonstrated clear to instructors and students?

•  Make it observable – Use a strong, action verb to underscore what will be observed when the outcome is demonstrated.


Activity: Provide a key word(s) to simplify each SLO


Art Courses

Key Word(s) ART101 ART102 ART103

Experience and develop curiosity, interest and enjoyment in their own creativity and that of others.

Experience curiosity

Acquire and develop skills needed for the creation of visual and performing art work.

Acquire and develop skills

Use the language, concepts and principles of visual and performing arts.

Language, concepts, principles of the arts

Create visual and performing art work. Create artwork

Communicate their thoughts and ideas through visual and performing arts.

Communicate artwork

Develop receptiveness to visual and performing art forms across time, place and cultures, and perceive the significance of these art forms as an integral part of life.

Receptiveness of artwork


Activity: Provide a key word(s) to simplify each SLO


Art Courses

Key Word(s) ART101 ART102 ART103

Express personal creativity through visual art. Express creativity

Demonstrate the skills necessary to produce a variety of visual artwork.

Demonstrate skills


Use language, concepts and principles


Create visual artwork

Suggestions: Providing key words or phrases


•  Simplify the outcome in a few words or a simple phrase – How would the outcome be “boiled down” in the most simplest of terms?

•  It’s a label, really – The key word or phrase will serve as the label for future scoring criteria.

Activity: Assign Outcomes to Courses (Map your SLOs to each course)




Experience and develop curiosity, interest and enjoyment in their own creativity and that of others. x Acquire and develop skills needed for the creation of visual and performing art work. x

Use the language, concepts and principles of visual and performing arts. x Create visual and performing art work. x Communicate their thoughts and ideas through visual and performing arts. x x Develop receptiveness to visual and performing art forms across time, place and cultures, and perceive the significance of these art forms as an integral part of life.

x

Suggestions: Assigning Outcomes to Courses


•  It’s a buffet – Outcomes are spread across courses within a program of study

•  Don’t be ambitious – Every course will not need to address every outcome.

•  Look for coverage – Make sure every outcome is assigned to a course within a program of study

Component #3 Key Assessments


An assessment or critical task used during the course of instruction to provide feedback to the teacher and learner about the learner's demonstration of and progress toward desired educational outcomes.

Activity: Define Key Assessments (How will students demonstrate the SLOs?)




Experience and develop curiosity, interest and enjoyment in their own creativity and that of others. x

Acquire and develop skills needed for the creation of visual and performing art work. x x x

Use the language, concepts and principles of visual and performing arts. x

Create visual and performing art work. x

Communicate their thoughts and ideas through visual and performing arts. x

Develop receptiveness to visual and performing art forms across time, place and cultures, and perceive the significance of these art forms as an integral part of life. x

Key Assessments Research

Paper Interpretive

Artwork Art

Reflection

Suggestions: Defining Key Assessments


•  Package course outcomes to define key assessments – Consider all outcomes assigned to a course in order to define a key assessment

•  Key Assessments are not “Pet Projects” – Outcomes will not always align to existing assignments

•  Emphasize quality over quantity – Only one key quality assessment is needed per course, only a dozen or fewer key assessments are needed per program

Component #4 Student Work/Exemplars


Student work serving as a typical example or an excellent model.

Student Work (papers, projects, video, etc.)


Student work is the collection of experiences and activities that students engage in to demonstrate developing skills. Types of student work can include audio files, papers, videos, teamwork, clinicals, performances, and presentations.

Categories of Student Work

Direct Measures Indirect Measures


�  Exams and tests

�  Demonstrations �  Portfolios Papers

�  Projects

�  Presentations

�  Performances

�  Student satisfaction surveys

�  Alumni satisfaction surveys �  Employer satisfaction surveys

�  Grades

�  Retention rates

�  Graduation rates/surveys

�  Placement rates (employment or transfer institutions)

�  Focus groups/interviews

�  Advisory committee recommendations

�  Reflective essays


Activity: Create an Exemplar Key Assessment: “Aluminum Foil Interpretive Artwork Project”

Student Learning Outcome: Students in ART102: Visual/Media Arts will use common, every day materials to develop skills for creating visual artwork. Assignment: Create interpretive artwork using aluminum foil.


Q & A

OUTCOMES DEFINE SCORING CRITERIA

SESSION 3:


(1:00PM – 2:00PM)

In this session…


Participants will… �  Define component #5 (Scoring Criteria/Rubrics). �  Examine a rubric for a specific outcome. �  Examine a rubric for a specific key assessment. �  Define scoring criteria. �  Identify performance levels. �  Describe levels of performance.

Component #5 Scoring Criteria/Rubrics


Scoring Criteria are the components of rubrics used to evaluate skill development, specifically the instruments that collect data about the skills described in the outcome.

Rubrics are structures that group scoring criteria into a framework used to assess student work i.e. presentation, product, performance, paper, etc.

Visual Arts Outcome Rubric CRITERIA Novice/1 Developing/2 Competent/3 Exceptional/4

A Exceeds Expectations

B Exceeds Expectations

C Exceeds Expectations

D Exceeds Expectations


Rubric for each outcome (A rubric for each outcome; all student artwork is assessed using this rubric)

GOOD

Aluminum Foil Interpretive Artwork Rubric CRITERIA Novice/1 Developing/2 Competent/3 Exceptional/4

A Exceeds Expectations

B Exceeds Expectations

C Exceeds Expectations

D Exceeds Expectations

Rubric for each Key Assessment (An individual rubric for each key assessment)


BEST

Define Scoring Criteria


Outcome stated as a key word(s)

Criteria 1 ! ! ! !

Criteria 2 ! ! ! !

Criteria 3 ! ! ! !

Scoring Criteria

Scoring criteria are the components of rubrics used to evaluate skill development, specifically the instruments that collect data about the skills described in the outcome. Scoring criteria are refined as key words or phrases.

VISUAL ARTS STUDENT LEARNING

OUTCOMES KEY

WORD(S)

ART COURSES SCORING CRITERIA

101 102 103 1 2 3 Express personal creativity through visual art.

Express creativity x Develop a

philosophy of creativity

Analyze various creative processes of VA

Express personal creativity through visual art.


Demonstrate skills x x Explain skills

needed for VA Examine visual artwork to determine skills used




x x Acquire vocabulary expressing language, concepts and principles of VA

Examine VA projects using appropriate language, concepts and principles



Create visual artwork x Identify visual arts

skills Examine visual arts skills


KEY ASSESSMENTS Research

Paper Foil Art Art

Reflection

© 2018 Jen B King's Consulting LLC | [email protected] | (813) 545-0560 | ENGAGE EDUCATE EQUIP EMPOWER

Activity: Use outcome language to develop scoring criteria from ABHE Feb. 21 (Scoring criteria should indicate growth over time)

Activity: State criteria and write descriptive language for each level


Novice/1 Emerging/2 Competent/3 Target/4 Exceptional/5

Imaginative Expression

Language represents what a first-year student can do

Language represents what a second-year student can do

Language represents what a third-year student can do

Language represents what a fourth-year student can do

Language represents the rare student who goes above/beyond the Target

what a first-year student can do

1st Year

what a second-year student can do

2nd Year

what a third-year student can do

3rd Year

what a fourth-year student can do

4th Year

the rare student who goes above/beyond the Target

Above & Beyond

Expression

Key Word(s)

CRITERIA

Aluminum Foil Interpretive Art RUBRIC NOVICE

1 EMERGING

2 COMPETENT

3 EXCEPTIONAL

4

Demonstrate skills Explain skills needed for VA Examine visual artwork to determine skills used




Acquire vocabulary expressing language, concepts and principles of VA




Add Grading Criteria Here


Activity: State the Key Word(s) in the Rubric from ABHE Feb. 21 (Transfer the key word(s) to the column labeled “Criteria.”)


You can added grading criteria to an outcomes-only rubric, though it’s not recommended.

Identify Performance Levels


Criteria" Novice 1.0

" Emerging

2.0 "

" Competent

3.0 "

Exceptional

4.0 "

Criteria 1 ! ! ! !

Criteria 2 ! ! ! !

Criteria 3 ! ! ! !

Novice Emerging Competent Exceptional

} Performance Levels

Defines the extent to which one’s performance against a criteria is met. These levels should indicate growth of over time, not finite growth such as Poor, Good, Very Good and Excellent.

CRITERIA

RUBRIC NOVICE

1 EMERGING

2 COMPETENT

3 EXCEPTIONAL

4







Activity: Identify Performance Levels in the Rubric (Discuss the terms used below and make suggestions for other terms.)


Define Performance Levels


Criteria Novice 1.0

" Emerging

2.0"

" Competent

3.0"

Exemplary

4.0" Criteria 1 ! ! ! !

Criteria 2 ! ! ! !

Criteria 3 ! ! ! !

Indicators Indicators

Descriptive language, referred to as indicators, because it indicates the performance of the individual demonstrating the criteria. Like the levels themselves, the language should describe growth over time.

CRITERIA

RUBRIC NOVICE

1 EMERGING

2 COMPETENT

(Target) 3 EXCEPTIONAL

4

Demonstrate skills Explain skills needed for VA Examine visual artwork to determine skills used




Acquire vocabulary expressing language, concepts and principles of VA






Activity: Describe Levels of Performance from ABHE Feb. 21 (Transfer the scoring criteria to the rubric. Use only the criteria for assigned course outcomes.)


You can added grading criteria to an outcomes-only rubric, though it’s not recommended.


Q & A

OUTCOMES DELIVER DATA

SESSION 4:


(3:00PM – 4:00PM)

In this session…


Participants will… �  Grade an exemplar. (Go to slide #7)

�  Assess an exemplar. (Use workshop created rubric)

�  Use data to make decisions.

Component #6 Data/Decision-Making


Data driven decision-making refers to an educator's ongoing process of collecting and analyzing different types of data to guide decisions towards improvement of educational programs and processes.

Activity: Grade and Assess the Exemplar (Use the new rubric to grade and assess the exemplar.)


Refer to the rubrics on slides #8 and #9.


Grading Assessing

Unit of Measure Individual student Program

Purpose Measure student performance

Measure program performance

Evaluator Individual instructor Collective faculty

Data Single Aggregate

Uses Emphasis on judgment Emphasis on diagnostic

Activity: Discuss what was learned when grading versus assessing (What was the purpose of grading the exemplar versus assessing the exemplar?)


GRADING RUBRIC

CRITERIA POOR/1 GOOD/2 GREAT/3 EXCELLENT/4

1 "

2 "

3 "

4 "

Desmond

16pts/Grade = A 16pts/Grade = A


ASSESSMENT RUBRIC

CRITERIA NOVICE/1 EMERGING/2 COMPETENT/3 EXCEPTIONAL/4

1 " Exceeds Expectations




Desmond is

a 1st year =

Novice

Suggestions: Using Data to Make Decisions


The data will reveal: 1.  The extent to which students demonstrated the outcome 2.  Clarity or confusion in the outcome 3.  Clarity or confusion in the program/course 4.  Clarity or confusion in the key assessment 5.  Clarity or confusion in the rubric


Q & A


Dr. Jennifer King [email protected]

(813) 545-0560

creating a highly effective assessment system - abhe.org · students should be able to demonstrate,...

Documents