Student Learning Objectives Pilot Test
SLO Learning Goals and Quality Assessment
Aurora Public SchoolsFall 2013
Introductions Center for Transforming Learning and Teaching
Catalyzing and co-creating the transformation of learning environments through the use of assessment so that all are engaged in learning and empowered to positively contribute in a global society.
www.ctlt.org
Facilitator/Trainer:Julie Oxenford O’Brian
Coach/Trainer:Mary Beth Romke
Check-In Check-in with your table group:
How was your experience identifying an SLO Learning Goal?
Were you able to specify associated standards, determine the cognitive complexity, write your goal as an objective statement, and determine if the goal represents the learning needs of your students?
Did you try-out engaging your students with success criteria?
Capture remaining questions about identifying SLO Learning Goal(s) on a sticky note
Purpose of Session Two
Finalize SLO Learning Goals.
Introduce the roles of Assessment in the SLO Process and the key characteristics of Quality Assessment.
SLO ComponentsLearning Goal
Learning Goal
Standards Reference
Rationale
Success Criteria
Measures
Evidence Sources
Alignment of Evidence
Collection and Scoring
Performance Targets
Baseline Data
Performance Groups
Performance Targets
Rationale for TargetsProgress Monitoring
Check Points
Progress Monitoring Evidence Sources
Instructional Strategies
SLO Results Student Performance Results
Targets Met
Teacher Performance
Day OneDay Two
Day Three and Four
Day Five
Day Three
Day Five
Materials
Learning Outcomes: Session Two
Engage in learning activity during this session.
Complete follow-up readings and tasks.
Finalize an SLO Learning Goal including writing a rationale for the goal.
Understand the role of assessment in SLOs.
Define assessment and the key components of assessment.
Identify a variety of methods (informal and formal) for collecting data about student learning.
Describe the relationship between the method of assessment used and the information gained.
Identify baseline data sources.
Activity: Monitoring your learning Turn to Progress Monitoring (Note catcher, p. 2-3).
Re-write today’s learning outcomes in language that has meaning for you.
Create a bar graph which describes where you currently believe you are in relationship to each learning target.
Leave the “reflections” column blank for now.
Learning Target
I don’t know what this Is
I need more
practice
I’ve got It
I could teach some-
one about it
Reflections
Identify a variety of methods (informal and formal) for collecting data about student learning.
In my words:
I can describe how to collect learning data and list several different options.
Day Two Agenda
Quality Assessment
Practice
Using Assessment for
SLOs
Data Collection Methods
Goal Method Match
Finalize SLO Learning
Goals
Baseline Data
Sources
SLO Learning Goal Process1. Identify the “big ideas” for the grade level and content area.
2. Identify learning goals associated with at least one “big idea” that would be achieved across several units, and/or which have related goals in prior or subsequent grade levels. These become candidates to be the SLO Learning Goal.
3. Determine which standards are associated with each candidate SLO Learning Goal.
4. Prioritize possible Learning Goals based on the learning needs of the student population (identifying two or three top priorities).
5. Determine the cognitive complexity (depth of knowledge) of the priority SLO Learning Goals. Eliminate candidate SLO learning goals with a depth of knowledge less than 3 for secondary and less than 2 for elementary.
6. Select the SLO Learning Goal.
7. Describe the rationale for your selection.
Day One
SLO Rubric A tool for evaluating the quality of SLOs.
Used by:Teachers in the development of each SLO
component.
Supervisors as they vet SLOs with teachers.
District leaders to investigate the quality of SLOs being developed.
SLO Pilot will try-out this “draft” tool.
SLO Rubric Consider your SLO Learning Goal.
Is your SLO Learning Goal consistent with the Learning Goal component definition on the Rubric?
Does it meet the criteria for “acceptable quality”?
Note: you should not have completed a rationale yet.
Take a few minutes to make any needed revisions to meet the “acceptable quality” criteria.
Effective Feedback is
Clear, descriptive, criterion-based, and indicates:√ how the learning goal differed from that
reflected in quality criteria, and
√ how the receiver of the feedback can move forward (what they might do next to improve).
Provide feedback about SLO Learning Goals Choose a partner (different grade level and/or content
area).
Exchange your SLO Learning Goals with your partner.
Consider: To what degree does her/his Learning Goal meet the acceptable
quality criteria?
Is it clear how the Learning Goal relates to the identified standards?
Is the Learning Goal at an appropriate level of cognitive rigor (DOK level)?
How could the components of the Learning Goal be improved?
Share your feedback with your partner.
SLO Learning Goal Rationale Acceptable Quality Criteria for Rationale:
Clearly explains why the learning goal is an appropriate focus or need for students to learn.
Clearly explains how the learning goal addresses high expectations (DOK no less than 3 for secondary and no less than 2 for elementary).
Learning Goal Rationale Outline
Justify that your SLO learning goal is at the right level (it is an educational objective).
State how cognitively complex (as measured by Depth of Knowledge) and that it is a DOK>=3 for secondary and DOK>=2 for elementary.
Describe the data that justifies the learning goal is a need for the identified student population.
Practice: Your Rationale Write a rationale for your SLO Learning
goal.
Capture your rationale:On the SLO form or
In the note catcher for today (p. 3).
Share your Rationale
Stand up and find someone you haven’t spoken with today.
Share your SLO Learning Goal statement and your rationale.
Provide just-in-time feedback to your partner about his/her rationale.
Make any needed revisions to your rationale.
Day Two Agenda
Quality Assessment
Practice
Using Assessment for
SLOs
Data Collection Methods
Goal Method Match
Finalize SLO Learning
Goals
Baseline Data
Sources
Defining Educational Assessment What is assessment?
Write your working definition of assessment in your note catcher (p. 4).
Activating peers as resources:Find a partner
Share your definition
Update your definition (if appropriate)
Defining Educational Assessment Terms used synonymously in education:
assessment, educational measurement, and testing.
Educational Assessment is. . . A process by which educators use students’
responses to specially created or naturally occurring stimuli to draw inferences about the students’ knowledge and skills.
A process of reasoning from evidence.
Pellegrino, J., Chudowsky, N., and Glaser, R. Eds. (2001). Knowing what students know: The science and design of educational assessment. Washington DC: National Academy Press.
Assessment Components
1. The aspect(s) of student learning that are to be assessed (cognition).
2. The tasks used to collect evidence about students’ achievement (observation).
3. The approach used to analyze and interpret the evidence resulting from the tasks (interpretation).
Pellegrino, J., Chudowsky, N., and Glaser, R. Eds. (2001). Knowing what students know: The science and design of educational assessment. Washington DC: National Academy Press.
Assessment Triangle
Cognition
Observation Interpretation
(Tools, p. 1)
Pellegrino, J., Chudowsky, N., and Glaser, R. Eds. (2001). Knowing what students know: The science and design of educational assessment. Washington DC: National Academy Press.
Assessment Quality
Work with your table group to list three considerations for assessment quality.
Capture in your note catcher (p. 6).
Prepare to share your list. . .
Characteristics of Quality Assessment Accuracy
The assessment instrument measures what it is supposed to measure.
Consistency
Multiple data sources result in the same inferences.
Fairness (bias)
All students can access the materials in the assessment instrument and have the chance to show what they know.
Motivation
Students want to show what they know..
Instructional importance and utility
The use(s) for the results justify the investment of time and effort involved.
(Tools, p. 6)
Testing Axioms Turn to Testing Axioms (Tools, p. 5).
Talk with a partner about the following: Do you agree/disagree with each axiom?
What are the implications for using externally developed tests for SLOs?
What are the implications for other classroom uses of test results? Grading?
These axioms guide most large-scale assessment development (e.g. TCAP, Interim assessments).
Assessment Results Learning
Assessment results measure learning, but are not direct observations of learning.
All assessment instruments measure only a sample of the learning we care about.
All assessment results include “error” in their measurement of students’ learning.
Increasing assessment quality = reducing the error in our measurement of students’ learning.
Quality Assessment Criteria Select a partner and turn to the Quality Assessment
Criteria.
Individually and silently read the first row of the quality criteria.
Turn to your partner and “say something” about the criteria: A summary of what you have read.
A connection to something else.
An elaboration or explanation of what you have read.
Silently read the next row of quality criteria. Continue until you have read and “said something” about each of the quality criteria.
Day Two Agenda
Quality Assessment
Practice
Using Assessment for
SLOs
Data Collection Methods
Goal Method Match
Finalize SLO Learning
Goals
Baseline Data
Sources
Assessment in Student Learning Objectives
As part of the SLO Process, we use multiple evidence sources (data collected from a variety of assessment instruments) to “reason from evidence” about:
Student learning in relationship to our Learning Goal at the beginning of the instructional interval (baseline data).
Student progress towards the Learning Goal during the instructional interval (progress monitoring/formative assessment).
Student learning in relationship to our Learning Goal at the end of the instructional interval (summative assessment).
Teacher contribution to Student Learning Growth (aggregation of results across students in the class/course).
Levels of ObjectivesLevel of Objective
Global Educational Instructional
Scope Broad Moderate Narrow
Time needed to learn
Two or more years (often
many)
Weeks, months, or academic
yearHours or days
Purpose or function Provide vision Design
curriculum Prepare lesson
plans
Example of use Plan a multi-year curriculum (e.g.
elementary reading)
Plan units of instruction
Plan daily lessons, activities,
experiences and exercises
A Taxonomy for Learning, Teaching, and Assessing: A revision of Bloom’s taxonomy of educational objectives, 2001
SLO
Learn
ing
Goals
Less
on
Object
i
ves
or
Targ
ets
Tools, p. 7
“An assessment activity can help learning if it provides information to be used as feedback by teachers, and by their students in assessing themselves and each other, to modify the teaching and learning activities.”
Black, Harrison, Lee, Marshall & Wiliam, 2003
“Formative assessment is a planned process in which assessment-elicited evidence of students’ status is used by teachers to adjust their ongoing instructional procedures or by students to adjust their current learning tactics.”
Popham, 2008
Definitions of Formative Assessment
Tools, p. 9
Formative Assessment Episode
1. Determine the learning goal/target.
2. Gather/collect information about learning (in relationship to the target(s)).
3. Analyze and interpret the gathered information about learning.
4. Use the learning information to improve teaching and/or learning.
Formative Assessment Episodes
Learning Goal/Target
Collecting Learning
Information
Analyzing Learning
Information
Interpreting Learning
Information
Using Learning
Information
Oxenford-O’Brian, 2013Tools, p. 11
Summative vs. Formative Assessment
Summative Formative
Ranking/Sorting
CertifyingCompetence
Grading
Accountability
Collecting data about
learning
Analyzing &Interpretingdata about
learning
Questioning
Clarifying Targets w/Learners
Providing Useful
Feedback
Self- & Peer-Assessment
Setting Goals &
Monitoring Progress
Planning &EvaluatingInstruction
Adjusting Learning ActivityDefining the
Learning Target (s)
Tools, p. 15
Assessment in the SLO Form Take out the SLO Form and SLO Component
Descriptions.
Where in the form will you capture information about assessment occurring as part of the SLO Process? Measuring and Scoring - how you will observe and interpret
student learning at the end of the instructional interval.
Performance Targets/Baseline Data – how you will observe and interpret student learning at the beginning of the instructional interval.
Progress Monitoring – how you will observe and interpret student learning during the instructional interval (progress towards the Learning Goal(s)).
Results – how student learning results are aggregated into a teacher performance rating.
Day Two Agenda
Quality Assessment
Practice
Using Assessment for
SLOs
Data Collection Methods
Goal Method Match
Finalize SLO Learning
Goals
Baseline Data
Sources
Data Collection Methods
1. How we “collect data” determines our assessment method.
2. Use sticky notes to write down all of the strategies you currently use to collect data about student learning.
3. List as many as you can, capturing one per sticky note.
Jigsaw Reading: Collecting Data Select a partner and assign readings (one per person):
Evidence of Learning (Davies, 2000) – Tools, p. 21.
Assessment, Testing, Measurement and Evaluation (Russell & Airasian, 2012) – Tools, p. 27.
As you read, highlight: Assessment Methods: descriptions of different categories of data
collection techniques or sources of evidence.
Examples of strategies for each data collection method.
Share the descriptions and examples with your partner.
Data Collection Strategies Work as a table group.
Group your data collection strategies (sticky notes) into the following categories of data collection methods: Observation
Questioning
Student Products
Put similar strategies together.
Student Products – additional categories
Not all student products yield the same type of data about learning.
Additional “assessment methods” that can be part of student products include: Selected Response
Short Constructed Response
Extended Constructed Response
Performance/Demonstration
Portfolio
Informal vs. Formal MethodsInformal Assessment Methods: Collected in the moment
Take less time
May or may not be planned ahead of time
Individual, small group, or full class
May or may not result in documented evidence
Observation and Questioning
Formal Assessment Methods: Structured
Take more time
Planned in advance
Usually full class
Result in documented evidence of student learning
Student Products
Assessment Methods Continuum
Observation
Questioning (individual, group, full class)
SelectedResponse
Short ConstructedResponse
Demonstration or Performance
Portfolio
Time
Complexity of information
Informal Formal
ExtendedConstructedResponse
Student Products
Tools, p. 33
Organizing based on Continuum Sort your “student product” strategy examples into the
additional categories of the continuum. You may need to clarify
some of your examples.
If a strategy doesn’t fit into one of the categories, put it in an
“other” category.
Turn to the Assessment Methods Continuum (Note Catcher, p. 9-
10).
Make notes about assessment methods: Clarifications about the category
Example strategies
Day Two Agenda
Quality Assessment
Practice
Using Assessment for
SLOs
Data Collection Methods
Goal Method Match
Finalize SLO Learning
Goals
Baseline Data
Sources
Accuracy Alignment
Are the data we collect providing information about the learning goals we care about?
This is often referred to as “alignment”.
Alignment includes:
To what degree do the assessment tasks/items include the type of thinking/skills included in the learning goal?
To what degree do the assessment tasks include the knowledge/concepts included in the learning goal?
Are the assessment tasks as cognitively complex (DOK) as the learning goal?
Accuracy Starts with Learning Goal
Accurate assessment depends on knowing the kind of thinking and the complexity of the thinking that is being asked of students by the learning goal or target.
Clarifying the type of thinking and cognitive complexity of learning goals/targets helps us to better select a method of assessment that measures what we’re looking for.
This means. . . Deconstruct the learning goals/targets (identifying
the skills/type of thinking and the content/knowledge).
Categorize the type of thinking required by the learning goal/target (using Revised Bloom’s Taxonomy).
Establish the cognitive complexity of the learning goal/target (using the Depth of Knowledge Framework).
Remember we already did this!
Appropriate Assessment Method(s)
Once we are clear on the thinking and depth of knowledge required by a learning goal/target (deconstructing).
We can better determine what assessment methods to use to collect data about student learning in relationship to the goal/target.
Every assessment method is not equally accurate for assessing every type of goal/target.
Aligning Learning Goals and Assessment Methods
1. Use the “Learning Goal to Method Match” blank table (Note Catcher, p. 11-12).
2. Fill in why you think each cell represents a “match” or not.
3. Compare your completed table to a table of a partner.
4. Prepare to share out questions/conflicts.
Cognitive Processes vs. Assessment MethodsType of Learning Target
Assessment Method
Observation Questioning Selected Response
Short Constructed Response
Extended Constructed Response
Demonstration or Performance
Remember Only if student talk is factual
Yes if questions are factual
Good for assessment remembering facts
Good for assessing remembering facts
Good for assessing conceptual knowledge
Too time consuming, hard to distinguish specific gaps
Understand YesYes if questions are about understanding
Only for factual knowledge Possibly Yes
May be difficult to distinguish specific gaps
Apply YesDifficult to use for this type of thinking
Difficult to use for this type of thinking
Possibly, may be difficult Yes Yes
Analyze YesYes if questions are about analysis
Possibly Possibly Yes Yes
Evaluate YesYes if questions are about evaluation
Possibly Possibly Yes Yes
Create Yes No
No (only to assess pre-requisite knowledge)
No (only to assess pre-requisite knowledge)
Only if what is being created is a written product
Best Method (in general)
Tools, p. 35
Depth of Knowledge vs. Assessment MethodsType of Learning Target
Assessment MethodObservation Questioning Selected
ResponseShort Constructed Response
Extended Constructed Response
Demonstration or Performance
DOK 1: Recall and Reproduce
Only if student talk is factual
Yes if questions are factual
Good for assessing remembering facts
Good for assessing remembering facts
Yes, for reproducing procedures.
Too time consuming, hard to distinguish specific gaps
DOK 2: Skills and
ConceptsYes
Yes, depending on the questions
Possibly Possibly YesGood for assessing some skills
DOK 3: Strategic Thinking/ Reasoning
YesYes, depending on the question
Difficult to use for this complexity of thinking
Difficult to use for this complexity of thinking
Yes Yes
DOK 4: Extended Thinking
YesYes, depending on the question
Difficult to use for this complexity of content
Difficulty to use for this complexity of thinking
Yes Yes
Tools, p. 36
Activity: Practice Matching Learning Goals to Assessment Methods
1. Work with your content/grade level group.
2. Take out your SLO Learning Goal(s).
3. Use the “Learning Goal and Assessment Methods Match” table (Note Catcher, p. 11).
4. Identify the assessment method(s) you will use for your SLO Learning Goal(s) and explain why.
Day Two Agenda
Quality Assessment
Practice
Using Assessment for
SLOs
Data Collection Methods
Goal Method Match
Finalize SLO Learning
Goals
Baseline Data
Sources
What is baseline data? Student learning data collected before or at the
beginning of the instructional period.
Measures of student learning that relate to your SLO learning goal.
Could include: TCAP results (by student) from last year for current students. District interim/benchmark assessment results from beginning of
the year. Results from other district-or school-wide assessments. Results from classroom assessments.
Why analyze baseline data? Evaluate how much initial student performance
varied at the beginning of the instructional period.
Determine if students can/should be put into more than one group based on their initial performance.
Establish a “baseline” from which student learning growth can be measured for different performance groups.
Not to establish an initial score for every student.
What evidence do you have?Talk with a partner. . .
What sources of evidence (assessment results) are available about student learning in relationship to your SLO Learning Goal from the beginning of the instructional interval? How closely does each evidence source align with the SLO
learning goal?
How formal was the data collection method?
When was data collected?
How was it scored?
How many evidence sources do you have?
Triangulation
How much baseline data? Consider all of your evidence sources about
student learning in relationship to your SLO Learning Goal collected before or near the beginning of the instructional period.
Prioritize them.
List your top three evidence sources on the “Baseline Data” chart (Note Catcher, pg. 14).
Describe your level of confidence in your top three evidence sources.
Analyzing baseline dataFor each evidence source (Baseline data handout):
1. Determine what scores or metrics are provided by the evidence source?
2. Describe the performance of the student population, or the class. (e.g. 80 % of the students were proficient; 15% were partially proficient; and 5% were unsatisfactory)
3. Consider the range of student performance (low to high). Is the variability in student performance enough to form more than one group of students based on their performance?
4. If yes, describe the performance of the groups of students (2-4).
Combining evidence sourcesIn your Note Catcher (pg. 14):
Identify the number of performance groups you will have.
Identify a label for each (e.g. low, medium, high).
Describe student performance for each evidence source by performance group.
Create a combined description of student performance for each performance group.
Assign Students to Performance Groups
The simplest case: one performance group: Student performance does not vary
Baseline student performance can be characterized for the student population as a whole.
More than one performance group: Assign students (by name) to each performance group.
How will you assign students form whom performance was inconsistent across evidence sources?
How will you assign students for whom all baseline data is not available?
Use Performance Group Descriptions chart (Note Catcher, p. 16).
Before we see you again. . . Identify appropriate assessment methods for your SLO
Learning Goal example.
Bring at least one example instrument (if available) for the content area for your example learning goal(s).
Reflect and Consider your Learning
Return to your Progress Monitoring (Note Catcher).
Did you move to the right in your self-assessment? Add to your graph.
Make any notes about your own learning in the “reflections” column.
Give us Feedback!! Oral: Share one ah ha!
Written: Use sticky notes
+ the aspects of this session that you liked or worked for you.
The things you will change in your practice or that you would change about this session.
? Question that you still have or things we didn’t get to today
Ideas, ah-has, innovations
Leave your written feedback on the parking lot.