education 3504 week 3 reliability & validity observation techniques checklists and rubrics

Education 3504 Week 3

reliability & validityobservation techniqueschecklists and rubrics

Reliability and Validity

• validity – extent to which you are measuring

what you think you are measuring

• reliability – extent to which information you are

gathering is consistent and unambiguous

Reliability

• chance factors– characteristics of student– characteristics of test / evaluation– conditions of administration– variations in scoring

Validity

• danger of thinking you are measuring one thing, actually measuring another– trivial knowledge– lowest-common denominator– regional knowledge/one

culture/gender

Validity

• face validity– appears important, relevant, interesting

• construct validity– appears to capture intended quality or

behaviour - e.g. what does “good citizenship” consist of?

validity and reliability are related

• but can vary independently

Checklists and Rubrics

checklist construction

• Checklist must be based on objectives of the course, unit, or lesson

• each item should be clear and precise

• each item should avoid generalities and focus on specifics

checklist construction

• checklist should be designed for feedback to students and parents – so need to be understandable – might want to distribute to students

before hand • must do so if intended for grading

checklist construction

• if students can use them, can be used for peer and self-evaluation

• focus on a few specifics, so keep checklist short

– if list needs to be longer, focus on only small section at a time

• anecdotal summary comments can be tied to checklists

rating scales

5 -Extremely active

4 -somewhat active

3 -average active

2 -somewhat active

1 -extremely active

rating scales

Extremely Active (5 points)

• always out of seat

• rarely sits without squirming

• makes noise, yells a good deal of the time

Rubricsscoring schemes

clear expectationsGeneric Essay

Subjectcontent

awareness of issues

critical thinking

Takes a position

EXCELLENT

A (85 - 100)

A Markedly Exceptional Performance

a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed

an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant

an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information

a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion

SUPERIOR B (70 - 84)

Clearly Above Average Performance

a thorough grasp of the subject matter is demonstrated

an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant

the paper goes beyond description to interpretation, analysis, synthesis and evaluation

a position is adopted and logically argued; appropriate supporting

detail is supplied

SATISFACTORY

C (55 - 69)

A Fully Competent Paper

a basic grasp of the subject matter is

demonstrated

asserts viewpoint without acknowledging alternative viewpoints

accurate information incorporating relevant sources and references is conveyed

a position is adopted

and logically argued

…but flexibleGeneric Essay

Subjectcontent

awareness of issues

critical thinking

Takes a position

EXCELLENT

A (85 - 100)

A Markedly Exceptional Performance

a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed

an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant

an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information

a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion

SUPERIOR B (70 - 84)

Clearly Above Average Performance

a thorough grasp of the subject matter is demonstrated

an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant

the paper goes beyond description to interpretation, analysis, synthesis and evaluation

a position is adopted and logically argued; appropriate supporting

detail is supplied

SATISFACTORY

C (55 - 69)

A Fully Competent Paper

a basic grasp of the subject matter is

demonstrated

asserts viewpoint without acknowledging alternative viewpoints

accurate information incorporating relevant sources and references is conveyed

a position is adopted

and logically argued

levels of masteryGeneric Essay

Subjectcontent

awareness of issues

critical thinking

Takes a position

EXCELLENT

A (85 - 100)

A Markedly Exceptional Performance

a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed

an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant

an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information

a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion

SUPERIOR B (70 - 84)

Clearly Above Average Performance

a thorough grasp of the subject matter is demonstrated

an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant

the paper goes beyond description to interpretation, analysis, synthesis and evaluation

a position is adopted and logically argued; appropriate supporting

detail is supplied

SATISFACTORY

C (55 - 69)

A Fully Competent Paper

a basic grasp of the subject matter is

demonstrated

asserts viewpoint without acknowledging alternative viewpoints

accurate information incorporating relevant sources and references is conveyed

a position is adopted

and logically argued

multi-dimensionalGeneric Essay

Subjectcontent

awareness of issues

critical thinking

Takes a position

EXCELLENT

A (85 - 100)

A Markedly Exceptional Performance

a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed

an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant

an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information

a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion

SUPERIOR B (70 - 84)

Clearly Above Average Performance

a thorough grasp of the subject matter is demonstrated

an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant

the paper goes beyond description to interpretation, analysis, synthesis and evaluation

a position is adopted and logically argued; appropriate supporting

detail is supplied

SATISFACTORY

C (55 - 69)

A Fully Competent Paper

a basic grasp of the subject matter is

demonstrated

asserts viewpoint without acknowledging alternative viewpoints

accurate information incorporating relevant sources and references is conveyed

a position is adopted

and logically argued

parallelismGeneric Essay

Subjectcontent

awareness of issues

critical thinking

Takes a position

EXCELLENT

A (85 - 100)

A Markedly Exceptional Performance

a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed

an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant

an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information

a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion

SUPERIOR B (70 - 84)

Clearly Above Average Performance

a thorough grasp of the subject matter is demonstrated

an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant

the paper goes beyond description to interpretation, analysis, synthesis and evaluation

a position is adopted and logically argued; appropriate supporting

detail is supplied

SATISFACTORY

C (55 - 69)

A Fully Competent Paper

a basic grasp of the subject matter is

demonstrated

asserts viewpoint without acknowledging alternative viewpoints

accurate information incorporating relevant sources and references is conveyed

a position is adopted

and logically argued

high but attainable standards: cumulative criteria

Generic Essay

Originality Subjectcontent

awareness of issues

critical thinking

EXCELLENT

A (85 - 100)

A Markedly Exceptional Performance

originality, insight, and creativity are demonstrated; the paper goes beyond repeating what others have said and contributes something new to our understanding of the topic

a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed

an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant

an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information

SUPERIOR B (70 - 84)

Clearly Above Average Performance

a thorough grasp of the subject matter is demonstrated

an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant

the paper goes beyond description to interpretation, analysis, synthesis and evaluation

SATISFACTORY

C (55 - 69)

A Fully Competent Paper

a basic grasp of the subject matter is

demonstrated

asserts viewpoint without acknowledging alternative viewpoints

accurate information incorporating relevant sources and references is conveyed

allowance for part marksGeneric Essay

Subjectcontent

awareness of issues

critical thinking

Takes a position

EXCELLENT

A (85 - 100)

A Markedly Exceptional Performance

a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed

an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant

an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information

a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion

SUPERIOR B (70 - 84)

Clearly Above Average Performance

a thorough grasp of the subject matter is demonstrated

an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant

the paper goes beyond description to interpretation, analysis, synthesis and evaluation

a position is adopted and logically argued; appropriate supporting

detail is supplied

SATISFACTORY

C (55 - 69)

A Fully Competent Paper

a basic grasp of the subject matter is

demonstrated

asserts viewpoint without acknowledging alternative viewpoints

accurate information incorporating relevant sources and references is conveyed

a position is adopted

and logically argued

allowance for part marks

A - EXCELLENT (85 - 100) - A Markedly Exceptional Performance

√ originality, insight, and creativity are demonstrated; the paper goes beyond repeatingwhat others have said and contributes something new to our understanding of thetopic

a comprehensive grasp of the subject matter is demonstrated, including an in-depthunderstanding of the relevant concepts, theories, and issues related to the topicaddressed

an awareness of differing view-points is demonstrated and a rigorous assessment ofthese undertaken where relevant

an ability to think critically is demonstrated in the analysis, synthesis and evaluationof relevant information

a thoughtful statement of position is presented and defended through logicalarguments and carefully selected supportive detail; the arguments presented build to aconsistent conclusion

√ a clear, fluent, and concise style highlights a well-written, tightly argued, andlogically structured essay

√ a virtually flawless mastery of all aspects of grammar, structure, and style isdemonstrated

B - SUPERIOR (70 - 84) - Clearly Above Average Performance

a thorough grasp of the subject matter is demonstrated√ an awareness of differing view-points is demonstrated and an assessment of these

attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied a clear style which communicates well (but may contain occasional or minor flaws in

the mechanics of spelling, grammar, sentence structure, etc.) is evident in the logicalpresentation of a reasonable argument

C - SATISFACTORY (55 - 69) - A Fully Competent Paper

√ a basic grasp of the subject matter is demonstrated accurate information incorporating relevant sources and references is conveyed√ a position is adopted and logically argued an adequate attempt at analysis, synthesis, interpretation or evaluation is evident an acceptable style demonstrates an awareness of, and attention to, the principles of

paragraph development, sentence structure, grammar and spelling, etc.

correspondence to real worldGeneric Essay

Subjectcontent

awareness of issues

critical thinking

Takes a position

EXCELLENT

A (85 - 100)

A Markedly Exceptional Performance

a comprehensive grasp of the subject matter is demonstrated, including an in-depth understanding of the relevant concepts, theories, and issues related to the topic addressed

an awareness of differing viewpoints is demonstrated and a rigorous assessment of these undertaken where relevant

an ability to think critically is demonstrated in the analysis, synthesis and evaluation of relevant information

a thoughtful statement of position is presented and defended through logical arguments and carefully selected supportive detail; the arguments presented build to a consistent conclusion

SUPERIOR B (70 - 84)

Clearly Above Average Performance

a thorough grasp of the subject matter is demonstrated

an awareness of differing viewpoints is demonstrated and an assessment of these attempted where relevant

the paper goes beyond description to interpretation, analysis, synthesis and evaluation

a position is adopted and logically argued; appropriate supporting

detail is supplied

SATISFACTORY

C (55 - 69)

A Fully Competent Paper

a basic grasp of the subject matter is

demonstrated

asserts viewpoint without acknowledging alternative viewpoints

accurate information incorporating relevant sources and references is conveyed

a position is adopted

and logically argued

evolving

• changes if you discover papers that do not fit criteria – add descriptors that accurately describe

what you are seeing

evolving

• changes as your teaching changes

• changes as students change

• changes as environment changes

ease of useA - EXCELLENT (85 - 100) - A Markedly Exceptional Performance

originality, insight, and creativity are demonstrated; the paper goes beyond repeatingwhat others have said and contributes something new to our understanding of thetopic

a comprehensive grasp of the subject matter is demonstrated, including an in-depthunderstanding of the relevant concepts, theories, and issues related to the topicaddressed

an awareness of differing view-points is demonstrated and a rigorous assessment ofthese undertaken where relevant

an ability to think critically is demonstrated in the analysis, synthesis and evaluationof relevant information

a thoughtful statement of position is presented and defended through logicalarguments and carefully selected supportive detail; the arguments presented build to aconsistent conclusion

a clear, fluent, and concise style highlights a well-written, tightly argued, andlogically structured essay

a virtually flawless mastery of all aspects of grammar, structure, and style isdemonstrated

B - SUPERIOR (70 - 84) - Clearly Above Average Performance

a thorough grasp of the subject matter is demonstrated an awareness of differing view-points is demonstrated and an assessment of these

attempted where relevant the paper goes beyond description to interpretation, analysis, synthesis and evaluation a position is adopted and logically argued; appropriate supporting detail is supplied a clear style which communicates well (but may contain occasional or minor flaws in

the mechanics of spelling, grammar, sentence structure, etc.) is evident in the logicalpresentation of a reasonable argument

ease of useChecklist - Space Community Unit

1 = weak 2 = fair 3 = good 4 = very good 5 = superior

Name:

Date:

Grade:

The students demonstrate the ability to:

A Social 1 2 3 4 5 Comments

1. respond to stimuli

2. respect the space of others

3. discipline self

4. develop appreciation of thework of self and others

B Physical

1. become aware of body andvoice as instruments ofexpression

C Intellectual/Emotional

1. communicate through space

2. develop concentration

3. recognize form of drama(beginning, middle, end)

4. develop and exerciseimagination

5. explore emotion (role play)

KEY BEHAVIORS

A #1 Respond toStimuli

#2 Respect thespace of others

#3 Discipline Self #4 Developappreciation

B. #1 Becomeaware of Body andVoice

Superior

5

Acts on experienceand information in astrikingly co herentway

Respec ts the spaceof others every time

Readily takes ac tionto discipline self

Indica tes whole-hearted enthusiasm

Begins to use bodyand voice in a richvariety that reflec tspersonalknowledge, feelingsand values

Very Good

4

Act on experienceand information in aconsistentlycoherent way

Respec ts the spaceof othersconsistently

Takes action todiscipline self

Indica tesenthusiasm

Begins to use bodyand voice in avariety of ways thatreflects personalknowledge, feelingsand values

Good

3

Acts on experienceand information in agenerally coherentway

Respec ts space ofothers most of thetime

Sometimes takesaction to disciplineself

Indica tes interest Begins to use bodyand voice somedifferent ways thatreflect personalknowledge, feelingsand values

Fair

2

Acts on experienceand information in asomewhathaphazard way

Respec ts the spaceof others some ofthe time

With help takesaction to disciplineself

Indica tesacceptance

Begins to use bodyand voice withencouragementthat reflectspersonalknowledge, feelingsand values

Weak

1

Acts on experienceand informationrandomly;sometimes rambles

Respec ts space ofothers hardly ever

Even with help doesnot always disciplineself

Indica tesimpatience

Begins to use bodyand voice withdifficulty even whenencouraged toexpress personalknowledge, feelings

rubrics workshop

• Objective:Produce the best cookie possible

rubrics workshop

• Objective:Produce the best cookie possible

• Was it difficult to come to a consensus on criteria for “best”?

why we evaluate

• motivation • administration• accountability• certification

motivation

• problems: – test anxiety

Norm referenced

• describes student performance relative to other students– e.g.: ranks 11th in group of 30

Criterion referenced

• describes student performance compared to a clearly defined criteria or task– e.g.: “adds single-digit whole numbers”

motivation• problems:

– overemphasis distorts learning process

• emphasis on external motivation

motivation

• problems: – overemphasis distorts learning process

• not everything we teach can be assessed (e.g., non-cognitive skills, attitudes, etc.)

administration

• general trend towards greater emphasis on assessment

• general shift from selection to accountability

accountability

• government to taxpayers– 2003, K-12 = 3.8 billion dollars…

accountability

• teachers– accountable for mandated curriculum– accountable for student learning– to oneself over time

accountability

• students to their parents

certification

• right of passage• selection and

allocation (gatekeeper)