problem solving assessment wendy adams bringing education to life
TRANSCRIPT
![Page 1: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/1.jpg)
Problem Solving Assessment
Wendy Adams
Bringing Education to Life
![Page 2: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/2.jpg)
Carl Wieman, Stanford University
University of Colorado, Boulder
![Page 3: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/3.jpg)
Introduction
• Problem solving is arguably the most important skill a physicist can have
• In spite of extensive research, very little progress has been made on how best to measure it and teach it.
• Other aspects of physics learning has seen great advances in the teaching and the measurement of learning.
What makes a good problem solver?
![Page 4: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/4.jpg)
Problem Solving:
“Problem solving is cognitive processing directed at achieving a goal when no solution method is obvious to the problem solver.” (Mayer, 1992)
Definition based on the solver.
![Page 5: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/5.jpg)
Hypothesis
• Difference between the complexity of the subject and the simplicity of the measurement tools.
– Usually all that can be measured is whether the person can solve the problem or not.
• Provides
– no insight as to what learner needs to do to improve, and
– no guidance to the teacher as to how to help them.
![Page 6: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/6.jpg)
• Delineate the skills that a person uses when solving problems.
• Measure each “sub-skill”
Simple enough….
Hypothesis
![Page 7: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/7.jpg)
Physics gets in the way!
• Maybe someone is a superb problem solver but they’ve never learned about angular momentum.
• Maybe they’ve learned an algorithm for solving similar problems and simply follow it.
![Page 8: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/8.jpg)
Expert Problem Solving?
1. Draw picture, 2. Describe physics principles, 3. Identify equation, 4. Do math, and 5. Check answer
“Problem solving is cognitive processing directed at achieving a goal when no solution method is obvious to the problem solver.”
(Mayer, 1992)
Only when the solver already knows how to do that type of “problem” – so it’s not Problem Solving!
![Page 9: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/9.jpg)
Expert Problem Solving?
• Schoenfeld (1992) - math professor vs. students• Wineburg (1998)- history professor vs. students• Sing (2005) – When physical intuition fails
What did they do?
• Visualize the problem• Monitor their own progress• Connect steps and pieces• Estimation• Content knowledge
![Page 10: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/10.jpg)
Problem Solving Sub-skills
Anything that can affect the subject’s ability to solve the problem.
‘Addition’ ‘Connects steps and Pieces’ ‘Wants to succeed on “test”’
![Page 11: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/11.jpg)
Problem Solving Assessment
• How can physics - free measurements be made?
• Study a wide range of people, solving complex problems – Specific failures show specific sub-skills.
![Page 12: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/12.jpg)
PSSA - Problem Solving skill Spectra Analysis
Base problem from the Jasper Woodbury Series*
• Story opens with Larry teaching Emily how to fly his ultra-light plane.
• Jasper tells Emily and Larry about camping trip at Boone’s Meadow 5 miles in.
• Jasper finds wounded Eagle• Emily has complicated problem of planning best transportation
considering route, payload, mileage and timing.• The Solver is required to help Emily plan the rescue.
*http://peabody.vanderbilt.edu/projects/funded/jasper/
![Page 13: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/13.jpg)
PSSA – Problem Solving skill Spectra Analysis
Two scripted “interns” solve the problem while the subject solves the problem. (adds task of analyzing interns as well)
![Page 14: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/14.jpg)
Sample
7. How confident are you about your answer? a. Positive b. Pretty sure c. Think it’s close d. Not sure at all
Do you think that means we have to count the gas in the tanks as part
of the payload or maybe it only counts extra fuel that you take along?
Sara considers this for awhile. That seems strange but he did say the payload includes
fuel so I think we should count that. What does a full tank of gas weigh?
How should I know? It held 5 gallons, does that help?
8. Do you know what the gas weighs or how they could figure this out?
Jasmine
Jasmine
Jasmine
Jasmine
Sara
Sara
![Page 15: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/15.jpg)
• Written instrument takes about 1 hour• It is a “problem” regardless of your content expertise.• Embedded data design - Free of discipline specific
content.• Identifies a student’s problem solving fingerprint or
spectra of sub-skills. – Strengths as well as weaknesses.
PSSA – Problem Solving skill Spectra Analysis
![Page 16: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/16.jpg)
Scripted Problem Solving
• Questions have been placed at crucial points asking the solver to provide
– their opinion on how the problem solving process is progressing
– bits of factual knowledge, planning, procedures, calculations, reflections and analysis of the two scripted intern’s problem solving skills.
![Page 17: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/17.jpg)
Sample
7. How confident are you about your answer? a. Positive b. Pretty sure c. Think it’s close d. Not sure at all
Do you think that means we have to count the gas in the tanks as part
of the payload or maybe it only counts extra fuel that you take along?
Sara considers this for awhile. That seems strange but he did say the payload includes
fuel so I think we should count that. What does a full tank of gas weigh?
How should I know? It held 5 gallons, does that help?
8. Do you know what the gas weighs or how they could figure this out?
Jasmine
Jasmine
Jasmine
Jasmine
Sara
Sara
![Page 18: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/18.jpg)
Scripted Problem Solving
• Unravels the sub-skills from the whole – reveals a person’s Spectrum of problem solving sub-skills
![Page 19: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/19.jpg)
Methodology
• 30 individual interviews• 14 paper and pencil versions graded
• List of 44 sub-skills identified in 3 major categories
![Page 20: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/20.jpg)
Categorization of Sub-skills
Knowledge – have*
Beliefs, Expectations & Motivation
Processes – do*
• Wants to solve the problem for self
• Careful/Thorough
• Planning (What)
• Connects steps and Pieces
• Meta-processing
• Addition/ Subtraction
• Algebra
• Number Sense
• Knowledge of own strengths
![Page 21: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/21.jpg)
How do we know PSSA tells us anything about how students solve science problems?
…Physics in particular?
Is it Valid?
![Page 22: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/22.jpg)
Physics Instructor
• Compared PSSA interview results to faculty evaluation of students strengths and weaknesses
• Found evaluation matched with PSSA results providing more detail and explanation for observed student behavior.
![Page 23: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/23.jpg)
Mechanics problem solving compared to PSSA paper and pencil results. - 5 students.
Mechanics Problem
University of Maryland Physics Education Research Group (2002). Activity-Based Physics (ABP) Alternative Homework Assignments. http://www.physics.umd.edu/perg/abp/aha/index.html
Given height of the pyramid, dimensions of a block, horsepower of a man, time allotted to build,
How many men were required to build the Great Pyramid of Giza?
![Page 24: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/24.jpg)
Results
All students easily identified and matched up!
• Pencil and paper PSSA graded blind• 3 point scale used for grading.
![Page 25: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/25.jpg)
Pyramid interview results
3 – point scale
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280
1
2
3
Karin
Paper and Pencil Pyramid Interview
![Page 26: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/26.jpg)
Pyramid interview w/ PSSA paper and pencil
3-point scale
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280
1
2
3
Karin
Paper and Pencil Pyramid Interview
![Page 27: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/27.jpg)
Pyramid of Giza ComparisonSpectrum Pyramid
Previously known facts Math - Geometry
Adaptability What should know inhibits effective solving
Ability to analyze interns
Enjoyed Analyzing Interns
Acquires information 1st time
![Page 28: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/28.jpg)
Quantum Mechanics problem solving compared to PSSA results for 5 students.
• Independent interviewer scored physics and Engineering students while solving a series of Quantum Mechanics’ problems biweekly interviews conducted all semester.
• I interviewed these same students solving the PSSA 6 months to a 1 year after the course.
Quantum Mechanics
![Page 29: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/29.jpg)
Suppose you are shooting photons at a screen one at a time and you see a dot appear on the screen as in the picture to the right. Where was that photon the instant before it hit the screen?
Quantum Mechanics
PhET.colorado.edu
![Page 30: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/30.jpg)
Example Data
Problem Solving Interviewer:
“Always had a knee jerk response which was not always good but then on her own she considers carefully and comes up with the right answer. …she’ll consider whatever is thrown out there, decisions are based on the most logical answer. If a suggestion does not make sense after careful consideration, she holds onto her beliefs. Very reliable”
Quantum Mechanics Interviewer:
“she seemed to view learning [as] how to accept every weird thing we told her… she thought the first step was to accept things and the second step was to try to understand them. She always rethought her ideas when another student suggested something, although she maintained enough skepticism to recognize that other students were often wrong.”
![Page 31: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/31.jpg)
Results
5-point scale
0
1
2
3
4
5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Guy
PSSA
PSSAonly
![Page 32: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/32.jpg)
Results
5-point scale
0
1
2
3
4
5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Guy
PSSA QM
PSSA withQuantum Mechanics
![Page 33: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/33.jpg)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 310
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
All Students (5)
Spectrum Quantum
PSSA interview compared to Quantum Interview88.5 % of scores agreed within 1 and 98.4% agreed within 2.
Acquires information 1st time through
![Page 34: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/34.jpg)
Quantum Mechanics comp.
Spectrum Quantum
Spatial – mapping Did not score skills missing from the rubric.
Previously known facts
Number Sense
Estimation
Interested in the context of the problemEnjoyed Analyzing Interns
Ability to analyze interns
Careful/ThoroughJudgment of importance of number values (is it material)
![Page 35: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/35.jpg)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 300
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Guy
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 250
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Rick
Very different spectra of sub-skills
![Page 36: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/36.jpg)
Physicists vs. Elementary Ed MajorsCommon Strengths Common
WeaknessPhysics &Engineering Students
Math – equation formation (4.3), Math - number sense (4), planning (4), creativity (3.8) and judgment (3.8)
Metacognitive Skills (4) and Meta-processing (2.7), Careful and Thorough (2.5)
Pre-Service Elementary Teachers
Metacognitive Skills (5), many wereCareful and thorough (3.8) and a few engaged inMeta-processing (3.8)
Creativity (1.8)Varied but serious (ie. Planning-big picture, -what and -how, reading comprehension, acquires information 1st time through)
![Page 37: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/37.jpg)
Typical ProblemsTypical homework problems require:
– Knowledge of the content (Newton’s Laws, algebra, …)
– Remember what s/he has calculated or reasoned
– Wants to solve the problem (for self, for grade, …)
…do not require:– Acquiring information 1st time through
• quite important in real life but almost never required in school.
– Visualize the Problem • Told to draw a picture• meaningless exercise when not required for solution
![Page 38: Problem Solving Assessment Wendy Adams Bringing Education to Life](https://reader035.vdocuments.us/reader035/viewer/2022081515/56649e045503460f94af0a59/html5/thumbnails/38.jpg)
Results• Itemization of sub-skills: knowledge, processes and beliefs,
expectations & motivations• Students bring the same skill set to very different types of
problem solving. • Trip Planning• Physics problems in general• A mechanics Problem• Quantum Mechanics Problems
• Problem Solving skill Spectra Analysis - PSSA
Complete listing of sub-skills: http://www.unco.edu/nhs/physics/faculty/adams/Research.htm