1 anne marie karlberg [email protected] january 30 th, 2009 creating effective assessment tools

59
1 Anne Marie Karlberg [email protected] January 30 th , 2009 Creating Effective Assessment Tools

Upload: hector-underwood

Post on 15-Jan-2016

213 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

1

Anne Marie Karlberg

[email protected]

January 30th, 2009

Creating Effective Assessment Tools

Page 2: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

2

Overview

1. What has been done at NWIC in terms of course outcomes?

2. Why use an assessment tool?

3. 2 Basic Assessment tools

(i) Checklists

(ii) Rubrics: a. The analytic rubric (scoring guide) b. The holistic rubric

4. Rubric websites

5. Break into groups: Work on creating rubrics for course outcomes

Page 3: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

3

Overview

1. What has been done at NWIC in terms of course outcomes?

2. Why use an assessment tool?

3. 2 Basic Assessment tools

(i) Checklists

(ii) Rubrics: a. The analytic rubric (scoring guide) b. The holistic rubric

4. Rubric websites

5. Break into groups: Work on creating rubrics for course outcomes

Page 4: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

4

1.Direct indicators (outcomes): e.g., essays, capstone projects, demonstrations, presentations

Assessment Data

Page 5: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

5

1.Direct indicators (outcomes): e.g., essays, capstone projects, demonstrations, presentations

2.Indirect indicators (perceptions): e.g., surveys, focus groups, interviews

Assessment Data

Page 6: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

6

1.Direct indicators (outcomes): e.g., essays, capstone projects, demonstrations, presentations

2.Indirect indicators (perceptions): e.g., surveys, focus groups, interviews

3.Institutional data: e.g., retention, graduation, enrollment, transfer trends

Assessment Data

Page 7: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

7

A. Course level

B. Program level

C. College level

Assessment Levels

Page 8: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

8

Examples of Activities at Each Level of NWIC’s Assessment Program

Type of assessment information College Program Course

Direct indicators of student learning (outcomes assessment)

Develop college outcomes

Implement college outcomes

Develop program outcomes

Implement program outcomes

Develop course outcomes

Implement course outcomes

Indirect indicators of student learning (surveys, interviews)

Graduate surveyFaculty peer review Student opinion survey

Alumni survey

Graduate survey

Student exit survey

Course evaluations

Institutional data (rates and numbers)

Track rates and numbers… Student enrollmentGraduation Retention TransferPerformance after transfer

Track rates & numbers…Student enrollmentRetention

Graduation

Track rates & numbers…Student enrollment

Course completion

Page 9: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

9

Examples of Activities at Each Level of a Tribal College Assessment Program

Type of assessment information College Program Course

Direct indicators of student learning (outcomes assessment)

Develop college outcomes

Implement college outcomes

Develop program outcomes

Implement program outcomes

Develop course outcomes

Implement course outcomes

Indirect indicators of student learning (surveys, interviews)

Graduate surveyFaculty peer review Student opinion survey

Alumni survey

Graduate survey

Student exit survey

Course evaluations

Institutional data (rates and numbers)

Track rates and numbers… Student enrollmentGraduation Retention TransferPerformance after transfer

Track rates & numbers…Student enrollmentRetention

Graduation

Track rates & numbers…Student enrollment

Course completion

Page 10: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

10

Beginning in 2003, faculty started consulting with one another and developed consistent outcomes for many courses – regardless of where or how the courses were being taught (i.e., course outcomes forms).

History of course outcomes at NWIC

Page 11: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

11

Each course outcomes form and syllabus identifies two types of learning outcomes that will be assessed during the course…

1. NWIC outcomes

2. Course specific outcomes (unique set of outcomes)

History of course outcomes at NWIC

Page 12: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

12

The outcomes are assessed in two ways:1. Directly: instructors assess

students’ work

2. Indirectly: students identify how well they think they have mastered each of the outcomes on course evaluation forms at the end of the quarter

History of course outcomes at NWIC

Page 13: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

13

The outcomes are assessed in two ways:1. Directly: instructors assess

students’ work

2. Indirectly: students identify how well they think they have mastered each of the outcomes on course evaluation forms at the end of the quarter

History of course outcomes at NWIC

Page 14: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

14

Characteristics of effective course outcomes

1. Out there Does each statement describe what we intend students will be able to do “out there” in life roles as opposed to an activity “in here”?

2. Active Does the statement describe what they can DO with what they know or understand?

3. Distinguishing

Do the statements as a whole clearly distinguish the levels of intended performance between courses?

4. Robust Are the statements “robust” (complex) enough to drive content and assessment tasks?

Page 15: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

15

Goal (% of college-level courses)

Baseline % of 198 courses (Dec. 2007)

2007 -

2008

2008 -

2009

2009 -

2010

2010 -

2011

2011 -

2012

The following are steps in developing the course outcomes process… % % % % % %

a. articulate course outcomes 41 60 75 100 100 100

b. include course outcomes on syllabi 41 60 75 100 100 100

c. develop course level assessment tools (e.g., rubrics) 0 0 15 30 45 60

d. determine or create the instructional activities, experiences, projects, or assignments that will be used to teach the outcomes

0 0 15 30 45 60

e. determine or create the activities, experiences, projects, essays, or assignments that will be used to assess the outcomes

0 0 15 30 45 60

f. attach anchor papers for each level of the rubric scale 0 0 15 30 45 60

g. assess students at the beginning and end of the course for outcomes

0 0 15 30 45 60

h. analyze the assessment data 0 0 15 30 45 60

i. use the data to improve and revise curriculum and to improve learning

0 15 30 45

j. share the results with faculty (occasionally) 0 15 30 45

k. compare the outcomes of different students in the same course over time

0 15 30 45

l. document the process 0 15 30 45

m. educate students about role of assessment in their education 0

Course Outcomes Plan

Page 16: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

16

Overview

1. What has been done at NWIC in terms of course outcomes?

2. Why use an assessment tool?

3. 2 Basic Assessment tools

(i) Checklists

(ii) Rubrics: a. The analytic rubric (scoring guide) b. The holistic rubric

4. Rubric websites

5. Break into groups: Work on creating rubrics for course outcomes

Page 17: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

17

2. Why use an assessment tool?

Makes the criteria for assessment explicit.

Guides the evaluation of students’ work and makes it less subjective.

Improves the assessment of learning.

Enhances learning.

Assists in targeting important content and in sustaining consistent attention to evaluative criteria.

Can be used for self and peer evaluation.

Increases consistency among instructors.

Page 18: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

18

Overview

1. What has been done at NWIC in terms of course outcomes?

2. Why use an assessment tool?

3. 2 Basic Assessment tools

(i) Checklists

(ii) Rubrics: a. The analytic rubric (scoring guide) b. The holistic rubric

4. Rubric websites

5. Break into groups: Work on creating rubrics for course outcomes

Page 19: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

19

The checklist

Isn’t about quality—it’s just about the parts that are necessary.

Is a good indicator of “can do–can't do” and “done–not done”.

Defines the parts that make up a complete task (either product or process).

Is useful for formative evaluation of a product or process.

Page 20: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

20

Checklists: Use checklists to…

Record observed performance.

For example,

Activities Checklist

Yes No Can name five sports.

Yes No Can name five activities other than sports.

Yes NoCan create simple sentences about likes & dislikes regarding activities & sports.

Yes No Can ask a question about what others do in their free time.

Page 21: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

21

Use checklists to…

Keep track of progress over time (e.g., as an inventory of skills at the beginning and end of a course). For example,Receptive oral skills

Understands simple directions.

Understands simple sentences.

Understands simple yes/no questions.

Understands vocabulary appropriate to age.

Understands meaning of different intonation patterns.

Understands more complex directions.

Understands rapid speech.

Understands language in classroom situation.

Understands language of peers.Adapted from Genesee, F. & Upshur, J.A. (1996). Classroom-based evaluation in second-language education . Cambridge: Cambridge University Press, p. 88.

Page 22: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

22

Use checklists to…

Indicate the minimum requirements of a product or performance, which are not included in your qualitative criteria (i.e., rubrics).

For example,Essay - Non-negotiables

Yes No My paper is typed, double-spaced.

Yes No I wrote at least 500 words.

Yes No My paper has an introduction, body, and conclusion.

Yes No I included examples from the story.

Yes No I proofread my paper.

Page 23: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

23

Use checklists to…

Help students fulfill task requirements.

For example,

Essay - Proofreading check - Draft 2

Length requirement

Spell-checked

Subject-verb agreement

Noun-adjective agreement

Verb conjugations

Page 24: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

24

An example of a “process” checklist

For working in a team, giving a presentation, working to solve a problem, etc.

Page 25: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

25

For example, papers, portfolios, projects, solutions, etc.

An example of a “product” checklist

Page 26: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

26

Advantages of checklists

• Easy to construct and use.

• Align closely with tasks.

• Effective for self and peer assessment.

• Make learners aware of task requirements, allowing them to self-monitor progress.

• Useful for sharing information with parents and others.

• Provides basic formative feedback.

Page 27: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

27

Disadvantages of checklists

• Provide limited information about how to improve performance or product.

• Do not indicate relative quality of performance or product.

• Do not provide information for assessing grades.

Page 28: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

28

Overview

1. What has been done at NWIC in terms of course outcomes?

2. Why use an assessment tool?

3. 2 Basic Assessment tools

(i) Checklists

(ii) Rubrics: a. The analytic rubric (scoring guide) b. The holistic rubric

4. Rubric websites

5. Break into groups: Work on creating rubrics for course outcomes

Page 29: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

29

Rubrics

• A scoring scale used to assess student performance along a task-specific set of criteria

• Comprised of two components: criteria and levels of performance

• Some rubrics include descriptors: describe more precisely what performance looks like at each level

Page 30: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

30

2. Written Communication Skills: Outcome a. The student writes standard English

Level of Performance

Criteria Exemplary (4)Accomplished

(3)Developing

(2) Beginning (1)1.Idea and

Content

2. Organization / Structure

3.Voice

Level of performance

Criteria

Page 31: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

31

2. Written Communication Skills: Outcome a. The student writes standard English

Level of Performance

Criteria Exemplary (4)Accomplished

(3)Developing

(2) Beginning (1)1.Idea and

ContentWrites clearly and with focus; relevant details support the central theme.

Maintains clear focus throughout the paper with sufficient appropriate details indicating awareness, knowledge, and insight.

Partially focuses on topic with minimal or no support of position. Writing is basic, too general for the reader to develop a clear understanding.

Writes with unclear purpose or central theme. Does not clearly define or support position on topic. Uses limited or disconnected details that disrupt the unity of the paper.

2. Organization / Structure

Provides clear introduction and reinforcing conclusion. Orders writing logically with effective transitions, providing sufficient information in the appropriate places.

Supports thesis and purpose through organization and paragraphing; most transitions are appropriate, but sequence of ideas may need improvement. Reiterates introductory elements in conclusion.

Writes with some signs of logical organization; may include abrupt or illogical shifts and ineffective flow of ideas. Makes few transitions between ideas.

Writes with organization that is unclear or inappropriate to the thesis; lacks transitions between ideas.

3.Voice Writes expressing own personality, with confidence and feeling. Individual, powerful commitment to the topic is obvious, as are strong connections to the audience and to the purpose; evokes strong emotion in the reader.

Writes so that own personality pokes through; confidence and feeling fade in and out. Commitment to the topic is apparent, and connection to the audience and to the topic are appropriate. The writing evokes some emotion in the reader.

Writes without revealing own personality; writing is cautious. Commitment to topic, and connection to the audience and to the purpose are limited. Writing evokes limited emotion in reader.

Writes without personality. Shows lack of commitment to topic, connection to the audience and to the purpose. Evokes no emotion in reader.

Level of performance

Criteria

Descriptors

Page 32: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

32

Possible terms for level of performance

4 3 2 1

Exemplary  Accomplished 

Developing Beginning

Professional Experienced Developing Novice

Frequently Sometimes Rarely Never

Distinguished

Proficient Apprentice Novice

Exemplary Excellent Acceptable Unacceptable

Exceeds expectations

Meets expectations

Progressing Beginning

Superior Good Fair Needs work

Page 33: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

33

Adjectives and Adverbs for Descriptors

Exemplary Proficient Progressing

Beginning / Not Meeting Expectations

All More than ___ (or most) Fewer than ___ (or some) None (or very few)

Always Often, usually Seldom, rarely, sometimes Never, not at all

Complete Somewhat complete Less than complete Incomplete

Superior Adequate Less than adequate Inadequate

Maximum Satisfactory Minimal Unsatisfactory

Articulate Understandable Vague Unclear

Clear, accurate Often clear, often accurate Sometimes unclear/inaccurate Rarely clear

…to the highest level …to an acceptable level …to a minimal level …to an unacceptable level

Includes all elements of… Includes most elements of… Includes few elements of… Includes no elements of…

Clear Somewhat proper Sometimes improper Improper

Proper Some degree of clarity Somewhat unclear Unclear

Appropriate Somewhat appropriate Limited Inappropriate

All Necessary… Adequate number of… Minimal amount of… Lacks enough of…

Significant Important Somewhat relevant Inconsequential, Unimportant

Critical, crucial Essential Somewhat useful Unnecessary

Logical, rational Reasonable Somewhat reasonable Illogical

Intuitive Somewhat intuitive Somewhat instinctive Random

Page 34: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

34

Level of Proficiency/Performance

CriteriaExemplary (4)

Accomplished (3)

Developing (2)

Beginning (1) Score

1.Idea and Content

Writes clearly and with focus; relevant details support the central theme.

Maintains clear focus throughout the paper with sufficient appropriate details indicating awareness, knowledge, and insight.

Partially focuses on topic with minimal or no support of position. Writing is basic, too general for the reader to develop a clear understanding.

Writes with unclear purpose or central theme. Does not clearly define or support position on topic. Uses limited or disconnected details that disrupt the unity of the paper. 3

2. Organization / Structure

Provides clear introduction and reinforcing conclusion. Orders writing logically with effective transitions, providing sufficient information in the appropriate places.

Supports thesis and purpose through organization and paragraphing; most transitions are appropriate, but sequence of ideas may need improvement. Reiterates introductory elements in conclusion.

Writes with some signs of logical organization; may include abrupt or illogical shifts and ineffective flow of ideas. Makes few transitions between ideas.

Writes with organization that is unclear or inappropriate to the thesis; lacks transitions between ideas.

2

3.Voice Writes expressing own personality, with confidence and feeling. Individual, powerful commitment to the topic is obvious, as are strong connections to the audience and to the purpose; evokes strong emotion in the reader.

Writes so that own personality pokes through; confidence and feeling fade in and out. Commitment to the topic is apparent, and connection to the audience and to the topic are appropriate. The writing evokes some emotion in the reader.

Writes without revealing own personality; writing is cautious. Commitment to topic, and connection to the audience and to the purpose are limited. Writing evokes limited emotion in reader.

Writes without personality. Shows lack of commitment to topic, connection to the audience and to the purpose. Evokes no emotion in reader.

4

Comments: Total 9 / 12

2. Written Communication Skills: Outcome a. The student writes standard English

Page 35: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

35

Level of Proficiency/Performance

Criteria Exemplary (4)Accomplished (3)

Developing (2)

Beginning (1)

Score

Weight

Subtotal (Score X Weight)

1.Idea and Content

Writes clearly and with focus; relevant details support the central theme.

Maintains clear focus throughout the paper with sufficient appropriate details indicating awareness, knowledge, and insight.

Partially focuses on topic with minimal or no support of position. Writing is basic, too general for the reader to develop a clear understanding.

Writes with unclear purpose or central theme. Does not clearly define or support position on topic. Uses limited or disconnected details that disrupt the unity of the paper. 3 X 2 6

2. Organization / Structure

Provides clear introduction and reinforcing conclusion. Orders writing logically with effective transitions, providing sufficient information in the appropriate places.

Supports thesis and purpose through organization and paragraphing; most transitions are appropriate, but sequence of ideas may need improvement. Reiterates introductory elements in conclusion.

Writes with some signs of logical organization; may include abrupt or illogical shifts and ineffective flow of ideas. Makes few transitions between ideas.

Writes with organization that is unclear or inappropriate to the thesis; lacks transitions between ideas.

2 X 3 63.Voice Writes expressing own

personality, with confidence and feeling. Individual, powerful commitment to the topic is obvious, as are strong connections to the audience and to the purpose; evokes strong emotion in the reader.

Writes so that own personality pokes through; confidence and feeling fade in and out. Commitment to the topic is apparent, and connection to the audience and to the topic are appropriate. The writing evokes some emotion in the reader.

Writes without revealing own personality; writing is cautious. Commitment to topic, and connection to the audience and to the purpose are limited. Writing evokes limited emotion in reader.

Writes without personality. Shows lack of commitment to topic, connection to the audience and to the purpose. Evokes no emotion in reader.

4 X 1 4

Comments: Total 16 / 24

2. Written Communication Skills: Outcome a. The student writes standard English

Page 36: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

36

Chocolate chip cookie rubric

What cookie elements could we judge?• Number of chocolate chips

• Texture

• Color

• Taste

• Richness (flavor)

Page 37: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

37

Chocolate chip cookie rubric

Level of performance?

  Criteri

a

Number of Chips

Texture

Color

Taste

Richness

Page 39: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

39

Chocolate chip cookie rubricLevel of performance

  Criteri

aDeliciou

s Tasty Edible Inedible

Number of Chips

Chocolate chip in every bite

Texture

Chewy

Color Golden brown

Taste Home-baked taste

Richness

Rich, creamy, high-fat flavor

Page 40: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

40

Chocolate chip cookie rubricLevel of performance

  Criteria

Delicious Tasty Edible Inedible

Number of Chips

Chocolate chip in every bite

Too few or too many chips

Texture

Chewy Texture resembles a dog biscuit

Color Golden brown

Burned

Taste Home-baked taste

Store-bought flavor; stale, hard, chalky;preservative after taste

Richness

Rich, creamy, high-fat flavor

Nonfat contents

Page 41: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

41

Chocolate chip cookie rubricLevel of performance

  Criteri

aDeliciou

s Tasty Edible Inedible

Number of Chips

Chocolate chip in every bite

Chips in about 75% of bites

Chocolate in 50% of bites

Too few or too many chips

Texture

Chewy Chewy in middle, crisp on edges

Texture either crispy/crunchy or 50% uncooked

Texture resembles a dog biscuit

Color Golden brown

Either light from overcooking or light from being 25% raw

Either dark brown from overcooking or light from undercooking

Burned

Taste Home-baked taste

Quality store-bought taste

Tasteless Store-bought flavor; stale, hard, chalky;preservative after taste

Richness

Rich, creamy, high-fat flavor

Medium fat contents

Low-fat contents Nonfat contents

Page 42: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

42

Rubrics

Advantages: provides

• Clearer expectations

• More consistent and objective assessment

• Better feedback

Page 43: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

43

Overview

1. What has been done at NWIC in terms of course outcomes?

2. Why use an assessment tool?

3. 2 Basic Assessment tools

(i) Checklists

(ii) Rubrics: a. The analytic rubric (scoring guide) b. The holistic rubric

4. Rubric websites

5. Break into groups: Work on creating rubrics for course outcomes

Page 44: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

44

a. Analytic rubrics (scoring guide)

• describes specific details we are looking for in the work (provides a lot of information)

• performance is judged separately for each criterion.

• useful for formative evaluation of a product or process

• many formats

Page 45: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

45

See hand-out

Student name: ____________________________________ Project title: _____________________________________

Course name and number: ________________________________ Date: _____________________________________ Learning outcome: Work collaboratively, in a team environment, to complete a project, to standards. Directions: Circle the number that best summarizes your assessment of a team member’s performance. Please write suggestions for improvement on the back of this page for any of the indicators you rated “beginning” or “developing”.

Beginning Developing Proficient Exemplary

1. Interpersonal communication, information exchange Withheld judgment while listening to others ......................1 2 3 4 NA Encouraged and valued the contribution of all members ....2 4 6 8 NA Enhanced the flow of information to all members ..............1 2 3 4 NA Used technology to stay connected .....................................1 2 3 4 NA

2. Work contribution Engaged willingly in the work ............................................3 6 9 12 NA Provided useful information and experience.......................1 2 3 4 NA Willingly gave and received constructive feedback............1 2 3 4 NA

3. Task coordination Helped clarify team’s task(s) ..............................................1 2 3 4 NA Helped organized and chart work flow ...............................1 2 3 4 NA Contributed to efficient use of team time............................1 2 3 4 NA

4. Individual responsibility Asked for verification of individual responsibilities ...........1 2 3 4 NA Assumed responsibilities.....................................................3 6 9 12 NA Stayed engaged until the team’s task(s) was finished .........3 6 9 12 NA NA Followed team’s timelines and deadlines............................1 2 3 4 NA

5. Adaptability Showed respect for differences in ability and work style....1 2 3 4 NA Adapted to change...............................................................1 2 3 4 NA Reflected on team’s process and interaction .......................2 4 6 8 NA

6. Interdependence Helped build internal/external networks to get work done .1 2 3 4 NA Helped synthesize input ......................................................1 2 3 4 NA Assisted other team members..............................................3 6 9 12 NA Trusted other team members ...............................................1 2 3 4 NA

Level of Achievement Grade Scale

A+ 90% - 100% A 85%-89%

Exemplary

A- 80%-84% B+ 76%-79% B 72%-75%

Proficient

B- 68%-71% C+ 64%-67% C 60%-63%

Developing

C- 56%-59% D 50%-55% Beginning F 0%-49%

Subtotal

Scoring Guide for Assessing Team Member Performance

Comments:

Subtotal

Subtotal

Subtotal

Subtotal

Subtotal

Total score (add subtotals from above)

____ / 124 x 100 = ____ %

Check one: Self assessment Peer assessment by ______________________ Instructor assessment by _________________

Page 46: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

46

Overview

1. What has been done at NWIC in terms of course outcomes?

2. Why use an assessment tool?

3. 2 Basic Assessment tools

(i) Checklists

(ii) Rubrics: a. The analytic rubric (scoring guide) b. The holistic rubric

4. Rubric websites

5. Break into groups: Work on creating rubrics for course outcomes

Page 47: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

47

b. Holistic rubrics

• assigns a level of performance by assessing performance across multiple criteria as a whole

• describes criteria more holistically (provides fewer details, does not list separate levels of performance for each criterion)

• is time-saving for the instructor

• is useful for summative evaluation of a product or process (tool to advance)

Page 48: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

48

See hand-out Student name: Joseph Salmon Project title: Nooksack River Stream Management

Course name and number: ENVS 263 Date: January 24th, 2009 Learning outcome: Work collaboratively, in a team environment, to complete a project, to standards. Directions: Circle the number that best summarizes your assessment of a team member’s performance. Please write suggestions for improvement on the back of this page for any of the indicators you rated “beginning” or “developing”. Level of Performance

Criteria Exemplary (4) Accomplished (3) Developing (2) Beginning (1) Score Weight

Subtotal (Score X Weight)

1. Interpersonal communication

Showed superb skill in listening to others, withholding judgment of others, advocating for own ideas and keeping information flowing in/out and through the team.

Participated fully to further the teamwork; had some difficulty listening to or valuing others and keeping the information flowing.

Actively communicated with others on the team but needed to either listen, contribute, advocate or make a genuine contribution to the discussion.

Communicated minimally with others on the team; did not listen to or express ideas.

4 X 2 8

2. Work contribution

Became one of the primary contributors to the work.

Willingly engaged in the work all the way through.

Made an adequate contribution.

Made a minimal contribution.

3 X 2 6

3. Task coordination

Exhibited a high level of organization and coordination skill.

Took responsibility for coordinating parts of the work.

Assisted in coordinating the work when directly asked by other members.

Made no effort to help organize the work.

2 X 1 2

4. Individual responsibility

Demonstrated a sense of responsibility for the quality and timely completion of the whole team task.

Took full personal responsibility for their portion of the task.

Assumed responsibility now and then during the work.

Showed no sense of real personal responsibility for the work.

3 X 3 9

5. Adaptability Used member differences and unpredictable happenings to realign and complete the work tasks to quality standards.

Adjusted well to individual differences among team members.

Tried hard to adjust to member differences and other unpredictable changes; still had difficulty.

Was unable to adjust to member differences and/or needs for change

2 X 2 4

6. Interdependence Networked with all team members, established trust and synthesized ideas to move the work forward.

Willingly sought and assisted other team members; strengthened the network.

Sought out team members and looked for ways to integrate their work; had difficulty.

Seemed to “stand alone”.

1 X 2 2

Total Score 31 / 48

Level of Achievement Grade Scale

A+ 90 - 100% A 85 - 89%

Exemplary

A- 80 - 84% B+ 76 - 79% B 72 - 75%

Proficient

B- 68 - 71% C+ 64 - 67% C 60 - 63%

Developing

C- 56 - 59% D 50 - 55% Beginning

F 0 - 49%

Holistic Rubric for Assessing Team Member Performance

Comments: Total score (add subtotals from above)

31 / 48 x 100 = 65 %

Check one: Self assessment Peer assessment by ______________________ Instructor assessment by _________________

Page 49: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

49

Compare analytic and holistic rubrics

Holistic AnalyticDescription provides a single score

based on an overall impression of a student’s performance on a task

provides feedback along several dimensions; descriptors may or may not be included

Advantages quick scoring provides an overview of student achievement

more detailed feedback; scoring more consistent across students and graders; effective tool for self, peer, and instructor evaluation

Disadvantages

does not provide detailed information, may be difficult to provide one overall score

time consuming to score

Use when you want

a quick snapshot of achievement and a single dimension is adequate to define quality

to see relative strengths and weaknesses or detailed feedback; to assess complicated skills or performance; to self- or peer assess performance

Page 50: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

50

Examples of rubrics

Page 51: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

51

Provides an insightful comparison of two or more organizations or initiatives. 2 6 8 10

Focuses on mission/goals/purpose/funding and approaches/strategies. 1 3 4 5

Presents in a coherent, articulate and systematic manner. 1 3 4 5

Provides necessary context and background for audience to understand presentation. 1 3 4 5

Uses visual aids (e.g., Internet or Powerpoint projections, other) in presentations. 1 3 4 5

Presents oral information in an engaging and rehearsed manner. 1 3 4 5

Stays within 10-15 minute presentation time-frame. 1 3 4 5

Is overall a creative and original presentation. 1 3 4 5

Pays attention overall to detail. 1 3 4 5

Total:                  / 50

Student: __________________________Date: ____________________________

 outstanding = 5

very good = 4 satisfactory = 3

unsatisfactory / not possible to evaluate = 1  

Media Literacy Reporting

Comments:

Outcome: To engage students in the process of educating themselves, and one another, about current developments in these areas of the media literacy movement, using the Internet as a source.  

Page 52: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

52

Student name: Adib Jamshedi Project title: Chocolate chip cookie bake-off

Course name and number: Culinary Arts 101 Date: January 30th, 2009 Learning outcome: Students will be able to bake a delicious chocolate chip cookie. Directions: Please write suggestions for improvement in the “Comments” section at the bottom of this page for any of the criteria you rated “inedible”. The cookies must…

measure between 1 and 2 inches in diameter have been baked within 2 hours of the competition

Level of Performance

Criteria Delicious (4) Tasty (3) Edible (2) Inedible (1) Score Weight

Subtotal (Score x Weight)

Number of Chips

Chocolate chip in every bite

Chips in about 75% of bites

Chocolate in 50% of bites

Too few or too many chips

3 X 4 12

Texture Chewy Chewy in middle, crisp on edges

Texture either crispy/crunchy or 50% uncooked

Texture resembles a dog biscuit

3 X 2 6

Color Golden brown Either light from overcooking or light from being 25% raw

Either dark brown from overcooking or light from undercooking

Burned 1 X 1 1

Taste Home-baked taste

Quality store-bought taste

Tasteless Store-bought flavor; stale, hard, chalky; preservative after taste

2 X 2 4

Richness Rich, creamy, high-fat flavor

Medium fat contents

Low-fat contents

Nonfat contents

3 X 1 3

Total Score 26

Level of Achievement Grade Scale

A+ 90 - 100% A 85 - 89%

Exemplary

A- 80 - 84% B+ 76 - 79% B 72 - 75%

Proficient

B- 68 - 71% C+ 64 - 67% C 60 - 63%

Developing

C- 56 - 59% D 50 - 55% Beginning

F 0 - 49%

Rubric for Assessing Chocolate Chip Cookies

Comments: The yellow color of the cookies was unappealing.

Total score (add subtotals from above)

26 /40 x 100 = 65 %

Check one: Self assessment Peer assessment by ______________________ Instructor assessment by AMK

Page 53: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

53

Other examples of rubrics?

Page 54: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

54

See hand-out:Instructions

for creating a rubric

Page 55: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

55

Overview

1. What has been done at NWIC in terms of course outcomes?

2. Why use an assessment tool?

3. 2 Basic Assessment tools

(i) Checklists

(ii) Rubrics: a. The analytic rubric (scoring guide) b. The holistic rubric

4. Rubric websites

5. Break into groups: Work on creating rubrics for course outcomes

Page 56: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

56

Rubric building websitesRubric Builders and Generators

The Canadian Teacher: www.thecanadianteacher.com/tools/assessment Tools to create a variety of assessments for student work

MyTL4:http://myt4l.com/index.php?v=pl&page_ac=view&type=tools&tool=rubricmaker

Make customized assessments for student work

NCRtec: http://goal.learningpt.org/spsg/GetProd.asp Create customized rubrics for student work

PBL Checklists: http://pblchecklist.4teachers.org/checklist.shtml Create checklists for writing, science, oral presentations and multimedia

RubiStar: http://rubistar.4teachers.org/index.php An online tool to develop scoring guides 

Rubric Builder: http://landmark-project.com/rubric_builder/index.php Create scoring guides online

Rubric Generator: http://www.teach-nology.com/web_tools/rubrics/ Personalize ready made scoring guides or create your own

Rubric Studio: http://www.rcampus.com/indexrubric.cfm Create a rubric or select from already created rubrics

Subject Specific and General Rubrics 

Rubrics 4 Teachers: http://www.rubrics4teachers.com/ Educational rubrics created and ready to use

Teacher Helpers Assessment and Rubric Information: http://school.discoveryeducation.com/schrockguide/assess.html

A variety of subject specific assessment rubrics

Rubric Construction Kit 

Assessment Rubrics: http://edtech.kennesaw.edu/intech/rubrics.htm Criteria for creating rubrics

Atomic Learning: http://movies.atomiclearning.com/k12/la_rubrics_wd03 Directions for creating a rubric in MS Word

2Learn: http://www.2learn.ca/projects/together/START/rubricc.html Planning worksheet and generator

Rubric Checklist:http://www.ncsu.edu/midlink/rubrics/Rubric.Checklist.ELS.11.00.doc

Helpful guide for constructing an effective rubric

Rubric Template: http://its.leesummit.k12.mo.us/Rubric%20Template%207.00.xls An Excel template for creating your own rubrics

Rubric Vocabulary: http://www.ncsu.edu/midlink/rubrics/Rubric%20Words.doc Aids for defining rubric dimensions

Page 57: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

57

Overview

1. What has been done at NWIC in terms of course outcomes?

2. Why use an assessment tool?

3. 2 Basic Assessment tools

(i) Checklists

(ii) Rubrics: a. The analytic rubric (scoring guide) b. The holistic rubric

4. Rubric websites

5. Break into groups: Work on creating rubrics for course outcomes

Page 58: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

58

Create rubrics for course outcomes

• Break into groups or work individually

• Hand-outs:

• How to create a rubric

• Blank templates

• Remember the key to successful rubrics:

→Steal good ideas!

Page 59: 1 Anne Marie Karlberg amkarlberg@gmail.com January 30 th, 2009 Creating Effective Assessment Tools

59

Thank you!