identification and articulaton of enduring student learning outcomes

Identification and Articulaton of Enduring Student Learning Outcomes

Karl A. SmithSTEM Education Center / Technological Leadership Institute /

Civil Engineering – University of Minnesota &Engineering Education – Purdue University

[email protected] - http://www.ce.umn.edu/~smith

King Fahd University of Petroleum and Minerals

Design and Implementation of Cooperative Learning

August 19-21, 2013

2

Session 2 Layout• Reflection on Session 1• Enduring student learning outcomes (BIG ideas)• Taxonomies of learning outcomes• Connecting outcomes and assessment strategies

Five Minute University

• http://www.youtube.com/watch?v=kO8x8eoU3L4

Streveler and Smith 3

http://www.youtube.com/watch?v=kO8x8eoU3L4

http://www.youtube.com/watch?v=kO8x8eoU3L4

Back

war

d D

esig

n

Context

Content

Assessment

Pedagogy

C & A & PAlignment?

End

Start

Yes

No

Understanding by Design (Wiggins & McTighe, 2005)

Content-Assessment-Pedagogy (CAP) Design Process Flowchart

4

Streveler, Smith & Pilotte (2012)

3 Stages of Backward Design

Are the desired results, assessments, and learning activities ALIGNED?

Identify the Desired Results

Determine Acceptable Evidence

Plan Learning Experiences

What should learners know, understand, and be able to do?







How will we know if the learners have achieved the desired results? What will be accepted as evidence of learners’ understanding and proficiency?







What activities will equip learners with the needed knowledge and skills? What materials and resources will be useful?


Elements of Content

• Knowledge-centered aspects of Content– Focuses on the information, topics, “stuff” of the

learning unit• Curricular priorities

• Learning-centered aspects of Content – Focuses on how the learner interacts with the

content

8Streveler and Smith

Designing Learning Environments Based on HPL (How People Learn)


KNOWLEDGE-CENTERED What do you want learners to learn?


To understand a topic or subject is to use knowledge in sophisticated, flexible ways. Knowledge and skill are necessary elements of understanding, but they are not synonymous with understanding. Matters of understanding require more: [Learners] need to make conscious sense and apt use of the knowledge they are learning and the principles underlying it.

-Understanding by Design

Wiggins and McTighe (1998)

Understanding Big Ideas


Wiggins & McTighe Curricular Priorities

Good to be familiar with

Important to know

Enduring outcomes

12

Click icon to add picture

Streveler and Smith

14

Understanding Misunderstanding

A Private Universe – 21 minute video available from www.learner.org

Also see Minds of our own (Annenberg/CPB Math and Science Collection – www.learner.org)

1. Can we believe our eyes? 2. Lessons from thin air3. Under construction

Exercise

Determine for your curriculum (re)design

Enduring outcomes

Important to know or understand

Good to be familiar with


LEARNER-CENTEREDHow do people learn?


Information Processing Model of Learning

• Input via attention – to short term memory – to long term memory – retention and retrieval


Information Processing Model of Learning

• Key areas for instruction– Attention and processing power = cognitive load (bandwidth)

• LIMITED – need to be careful how one uses the learner’s bandwidth– Connected nature of memory

• Link to what is already known– PRIOR KNOWLEDGE

• Need for organization– Structural knowledge – ADVANCED ORGANIZERS

– Multiple inputs (both in timing and in modes of input) = number and strength of connections = richer learner that will be retained longer

• PRACTICE• MULTIPLE MODES OF INPUT (visual, audio, explanation to yourself and

others)


EXERCISE


What are your initial ideas about how you can help your learners achieve the enduring outcomes of the

course you are redesigning?

ASSESSMENTHow will you know learners have learned what you want them to learn?


We cannot measure learning directly. Instead we must make inferences

from evidence. Sound inferences depend on

assessments that are aligned with content.

Objectives of Assessment What is the Purpose of the Assessment?

Key questions to ask yourself about assessment

1. What should I be assessing?2. What is the best assessment to use?3. How can I be sure I am consistently

interpreting the results of the assessments?


1. What should I be assessing?

• Assessments should be aligned with your curricular priorities. – Enduring outcomes should ALWAYS be assessed.– Important to know outcomes are usually assessed.– Good to be familiar with information might not be

assessed.


2. What is the best assessment to use? Assessments Aligned to Curricular Priorities

Adapted from: Wiggins, G. & McTighe, J. (1997). Understanding by Design. Alexandria, VA: ASCD

Traditional Quizzes and Tests• Selected Response

Closed-ended, convergent Well-structured Scored via “answer key”

Academic Prompts• Constructed Response

Open-ended, ill-structured, divergent Academic conditions (e.g. exams, drills) Requires analysis, evaluation, and/or synthesis Judgment-based scoring (e.g., rubrics)

Performance Tasks • Papers, projects, design tasks, etc.

Open-ended, complex, ill-structured, divergent Approximation of practice, specific audience Higher autonomy, more personalized Judgment-based scoring (e.g., rubrics)

Familiar w

ith

Importa

nt to kn

ow

Enduring

Understanding

3. How can I be sure I am consistently interpreting the results of the assessments?

• For close-ended and well-structured assessment prompts.– Construct an accurate grading key– Determine what level of performance is acceptable for

demonstrating “mastery” [i.e. “pass”]• For more open-ended prompts and for performance tasks

– Construct a matrix what will link key features with how those features will be demonstrated to show mastery. This is called a RUBRIC.

– Determine what level of performance is acceptable for demonstrating “mastery.”


Some steps for creating a rubric

• Look at work you feel is of varying levels of quality.• What characteristics make some work “better” than

others?• What are markers within the work point to better

performance?

• Exercise: Work together to create a rubric for assessing– A memo– A weld


LEARNING OBJECTIVES AND TAXONOMIES

Learning objectives are the bridge between what you want learners to learn and how you know they learned it.

27Streveler and Smith

Constructing Learning Objectives Learning Activities vs. Learning Objectives


Constructing Learning Objectives Using Verb-Noun Format


Mapping Learning Objectives to Curricular Priorities

Annotated Example: Ruth Wertz - High Level Weekly Planning (Annotated Example).docx


• Bloom’s taxonomy of educational objectives: Cognitive Domain (Bloom & Krathwohl, 1956)

• A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives (Anderson & Krathwohl, 2001).

• Taxonomy of significant learning (Fink, 2003)• Evaluating the quality of learning: The SOLO

taxonomy (Biggs & Collis, 1982)• Facets of understanding (Wiggins & McTighe, 1998)

31

Taxonomies

Streveler and Smith

Revised Bloom’s Learning Taxonomy

Source: http://www.celt.iastate.edu/pdfs-docs/teaching/RevisedBloomsHandout.pdfStreveler and Smith 32


Source: http://www.celt.iastate.edu/pdfs-docs/teaching/RevisedBloomsHandout.pdf


Mapping Learning ObjectivesRange of Low-Order & High-Order Objectives

Annotated Example: Ruth Wertz - Map of Weekly Learning Objectives (Annotated Example).docx


Exercise

• For the course you are (re)designing:– Starting with your enduring outcomes – write

learning objectives for each of your 5 most important curricular priorities.

– Place those learning objectives in Bloom’s revised taxonomy.


37

Session 3-4 Preview• Pedagogies of Engagement – Cooperative Learning and

Challenge Based Learning– Informal – Bookends on a Class Session– Formal Cooperative Learning

• Key Resource– Review Smith, Sheppard, Johnson & Johnson, “Pedagogies of

engagement.”

KINDS OF ASSESSMENT


We cannot measure learning directly. Instead we must make inferences from evidence. Sound inferences depend on assessments that are aligned with content.

Objectives of Assessment What is the Purpose of the Assessment?

Formative Summative

Provide Quantitative Feedback (e.g.,

score or grade)

Evaluate Completion of Learning Objectives

Summarize Learning Achievement

Provide Qualitative Feedback

Evaluate Learning Progress

Improve/Redirect Learning Activities

Assessment ExamplesFormative and Summative Assessments

Formative Summative

Tests & Quizzes Self-test Quiz

Written Exam

Academic Prompts

Group Discussion Reflection Paper

Peer Review

Written ExamWritten Assignment

Practical Exam

Performance Tasks

Oral ExamReflection Paper

Practice lab practical with feedback

Design ProjectResearch Paper

Case StudyGraded lab practical

Observation & Dialogue

Oral ExamInformal Check for Understanding

Oral ExamPerformance Evaluation

Examples above are not exhaustive.

Glossary of important terms in summative assessment

• Criterion model of assessment– Determine a level of learning considered

acceptable and then measure every learner against that criterion.

• All could pass, none could pass.

• Normative model of assessment– Learners are measured against each other.

• “Grading on a curve.”

• Rubrics


EXAMPLES OF FORMATIVE FEEDBACK


Quick Thinks

• Reorder the steps• Paraphrase the idea• Correct the error• Support a statement• Select the response

Johnston, S. & Cooper,J. 1997. Quick thinks: Active- thinking in lecture classes and televised instruction. Cooperative learning and college teaching, 8(1), 2-7


Minute Paper

• What was the most useful or meaningful thing you learned during this session?

• What question(s) remain uppermost in your mind as we end this session?

• What was the “muddiest” point in this session?

• Give an example or application

• Explain in your own words . . .

Angelo, T.A. & Cross, K.P. 1993. Classroom assessment techniques: A handbook for college teachers. San Francisco: Jossey Bass. Streveler and Smith 44

Session Summary (Minute Paper)

Reflect on the session

1. Most interesting, valuable, useful thing you learned.2. Things that helped you learn.3. Question, comments, suggestions.4. Pace: Too slow 1 . . . . 5 Too fast5. Relevance: Little 1 . . . 5 Lots6. Instructional Format: Ugh 1 . . . 5 Ah


MOT 8221 – Spring 2009 – Session 1

Q4 – Pace: Too slow 1 . . . . 5 Too fast (3.3)Q5 – Relevance: Little 1 . . . 5 Lots (4.2)Q6 – Format: Ugh 1 . . . 5 Ah (4.4)

0

5

10

15

20

25

Q4 Q5 Q6

1

2

3

4

5


Exercise

• For your 5 most important learning objectives:– What kind of assessments might you use to

measure learners’ attained the level of mastery of the learning objectives?

– Where can you insert formative assessment to check learners’ progress?


49

Taxonomies of Types of LearningBloom’s taxonomy of educational objectives: Cognitive Domain

(Bloom & Krathwohl, 1956)

A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives (Anderson & Krathwohl, 2001).

Facets of understanding (Wiggins & McTighe, 1998)

Taxonomy of significant learning (Fink, 2003)

Evaluating the quality of learning: The SOLO taxonomy (Biggs & Collis, 1982)

50

The Six Major Levels of Bloom's Taxonomy of the Cognitive Domain(with representative behaviors and sample objectives)

Knowledge. Remembering information Define, identify, label, state, list, match Identify the standard peripheral components of a computer Write the equation for the Ideal Gas Law

Comprehension. Explaining the meaning of information Describe, generalize, paraphrase, summarize, estimate

In one sentence explain the main idea of a written passage Describe in prose what is shown in graph form

Application. Using abstractions in concrete situations Determine, chart, implement, prepare, solve, use, develop

Using principles of operant conditioning, train a rate to press a bar Derive a kinetic model from experimental data

Analysis. Breaking down a whole into component parts Points out, differentiate, distinguish, discriminate, compare

Identify supporting evidence to support the interpretation of a literary passage Analyze an oscillator circuit and determine the frequency of oscillation

Synthesis. Putting parts together to form a new and integrated whole Create, design, plan, organize, generate, write

Write a logically organized essay in favor of euthanasia Develop an individualized nutrition program for a diabetic patient

Evaluation. Making judgments about the merits of ideas, materials, or phenomena Appraise, critique, judge, weigh, evaluate, select

Assess the appropriateness of an author's conclusions based on the evidence given Select the best proposal for a proposed water treatment plant

51

Remember Understand Apply Analyze Evaluate Create

Factual Knowledge – The basic elements that students must know to be acquainted with a discipline or solve problems in it.a. Knowledge of terminologyb. Knowledge of specific details and elements

Conceptual Knowledge – The interrelationships among the basic elements within a larger structure that enable them to function together.a. Knowledge of classifications and categoriesb. Knowledge of principles and generalizationsc. Knowledge of theories, models, and structures

Procedural Knowledge – How to do something; methods of inquiry, and criteria for using skills, algorithms, techniques, and methods.a. Knowledge of subject-specific skills and algorithmsb. Knowledge of subject-specific techniques and methodsc. Knowledge of criteria for determining when to use appropriate procedures

Metacognitive Knowledge – Knowledge of cognition in general as well as awareness and knowledge of one’s own cognition.a. Strategic knowledgeb. Knowledge about cognitive tasks, including appropriate contextual and conditional knowledgec. Self-knowledge

The Cognitive Process Dimension

The Know

ledge Dim

ension

(Anderson & Krathwohl, 2001).

52http://www.uwsp.edu/education/lwilson/curric/newtaxonomy.htm

53http://www.celt.iastate.edu/pdfs-docs/teaching/RevisedBloomsHandout.pdf

Dee Fink – Creating Significant Learning Experiences A TAXONOMY OF SIGNIFICANT LEARNING

1. Foundational Knowledge • "Understand and remember" learning

For example: facts, terms, formulae, concepts, principles, etc. 2. Application

Thinking: critical, creative, practical (problem-solving, decision-making) Other skills

For example: communication, technology, foreign language Managing complex projects

3. Integration Making "connections" (i.e., finding similarities or interactions) . . .

Among: ideas, subjects, people 4. Human Dimensions

Learning about and changing one's SELF Understanding and interacting with OTHERS

5. Caring Identifying/changing one's feelings, interests, values

6. Learning How to Learn Becoming a better student Learning how to ask and answer questions Becoming a self-directed learner

Cognitive

Affective

Meta

SOLO Taxonomy • The Structure of Observed Learning Outcome (SOLO)

model consists of 5 levels of understanding– Pre-structural - The task is not attacked appropriately; the student

hasn’t really understood the point and uses too simple a way of going about it.

– Uni-structural - The student's response only focuses on one relevant aspect.

– Multi-structural - The student's response focuses on several relevant aspects but they are treated independently and additively. Assessment of this level is primarily quantitative.

– Relational - The different aspects have become integrated into a coherent whole. This level is what is normally meant by an adequate understanding of some topic.

– Extended abstract - The previous integrated whole may be conceptualised at a higher level of abstraction and generalised to a new topic or area.

http://en.wikipedia.org/wiki/Structure_of_Observed_Learning_Outcome

Teaching Teaching and Understanding Understanding

• Biggs SOLO taxonomy

• http://www.youtube.com/watch?v=iMZA80XpP6Y

identification and articulaton of enduring student learning outcomes

Documents

learning activities

matters of understanding

evidence of learners

design wiggins mctighe

needed knowledge

minerals design

big ideasstreveler

smith pilotte