model-based inquiry: epistemology, modeling skills, assessment, & research
DESCRIPTION
Model-based Inquiry: Epistemology, Modeling Skills, Assessment, & Research. Janice Gobert The Concord Consortium mac.concord.org mtv.concord.org Based on work from 1) Making Thinking Visible (NSF #9980600) and 2) Modeling Across the Curriculum (IERI #0115699) . - PowerPoint PPT PresentationTRANSCRIPT
Model-based Inquiry: Epistemology, Modeling Skills, Assessment, & Research
Janice GobertThe Concord Consortium
mac.concord.org mtv.concord.org
Based on work from 1) Making Thinking Visible (NSF #9980600) and 2) Modeling Across the Curriculum (IERI #0115699).
All opinions expressed are those of the author and do not
necessarily reflect the views of the granting agencies.
How are you defining “scientific practice” in your design and empirical work? The Scientific Practice is Modeling, this includes model-based
reasoning, model-based inquiry, etc.
• MBL is a theory of science learning that integrates research in cognitive psychology and science education (Gobert & Buckley, 2000).
• Its tenets are that understanding requires the construction of mental models and all subsequent problem-solving, inferencing, or reasoning are done by means of manipulating or ‘running’ these mental models (Johnson-Laird, 1983).
• Model-based reasoning also involves the testing, and subsequent reinforcement, revision, or rejection of mental models.
• Modeling research at the Concord Consortium organizes learning activities, assessment, and research around model-based learning.
MBR Involves both internal and external
models
External Models,i.e., hypermodels
Mental model
cognitive processes act on mental model
Assumes students’ epistemologies influences model-based reasoning;Gobert & Discenna, 1997; Gobert & Pallant, 2004).
Other research literature….
In addition to students’ pre-instruction models in designing the unit, we (J. Gobert, Jim Slotta, Amy Pallant) drew on current findings from:
• causal models (White, 1993; Schauble et al, 1991; Raghavan & Glaser, 1995),
• model-based teaching and learning (Gilbert, S., 1991; Gilbert, J. 1993);
• model revising (Clement, 1989; 1993; Stewart & Hafner, 1991);
• diagram generation and comprehension (Gobert, 1994; Gobert & Frederiksen, 1988; Kindfield, 1993; Larkin & Simon, 1987; Lowe, 1989; 1993),
• the integration of text and diagrams (Hegarty & Just, 1993), and
• text comprehension (van Dijk & Kintsch, 1983; Kintsch, 1998).
How is it being supported?from Making Thinking Visible Project
(mtv.concord.org)
• Scaffold drawing of their own models of plate tectonics phenomena based on progressive model-building principles (model pieces acquisition).
• Scaffold on-line “field trip” to explore differences between the East and West coast in terms of earthquakes, volcanoes, mountains (beginning with the most salient differences to support knowledge building around the driving question; model-pieces acquisition).
• Posing a question about their current model (to model pieces integration and model-building).
• Learn about location of earth’s plates (to scaffold relationship between plate boundaries anf plate tectonic phenomena as model pieces integration).
• Reify important spatial and dynamic knowledge (model pieces integration) about transform, divergent, collisional, and convergent boundaries.
• Learn about causal mechanisms involved in plate tectonics, i.e., convection & subduction (scaffolded by reflection activities to integrate spatial, causal, dynamic, and temporal aspects of the domain- model pieces integration).
Pedagogical support (cont’d) from Making Thinking Visible Project
(mtv.concord.org)• Students are scaffolded to critique learning partners’ models using prompts in
WISE (reconstruct, reify, & reflect). Prompts include:1. Are the most important features in terms of what causes this geologic process
depicted in this model? 2. Would this model be useful to teach someone who had never studied this geologic
process before? 3. What important features are included in this model? Explain why you gave the model
this rating.4. What do you think should be added to this model in order to make it better for
someone who had never studied this geologic process before?
• Reflect on how their model was changed and what it now helps explain (reconstruct, reify, & reflect), e.g.,: prompts include:
“I changed my original model of.... because it did not explain or include....” “My model is now more useful for someone to learn from because it now includes….”
• Transfer what they have learned in the unit to answer intriguing points (reconstruct, reify, & reflect):
Why are there mountains on the East coast when there is no plate boundary there? How will the coast of California look in the future?
How do you know when you see it?
• Examples to follow….
Comments on example 1….
• Original model- focus on crustal layer, no causal mechanisms for what causes mountain formation.
• W. coast partners’ critique requested labels.
• Revised model-includes labels, and a cut away view of the interior of the earth which includes convection in the mantle.
Comments on example 2…
• Original model- cross section, no causal mechanisms for what causes mountain formation.
• W. coast partners’ critique requested information about direction of plate movement.
• Revised model-includes a cross section with plate movement, added the mantle as an interior layer.
Does it effect students’
epistemologies? *• Data to follow….
* Acknowledging the problem with assessing epistemologies with surveys.
WISE Period 1 - sig. Epistemological gains
02468
10121416
Cell
Mea
n
preMtot postMtotCell
TSA
Interaction Bar Plot for modelgain Effect: Category for modelgain * teacher
2 22.442 11.221 .692 .5044 1.384 .15761 988.926 16.212
1 115.697 115.697 16.046 .0002 16.046 .9872 83.882 41.941 5.817 .0049 11.633 .866
61 439.837 7.210
DF Sum of Squares Mean Square F-Value P-Value Lambda PowerteacherSubject(Group)Category for modelgainCategory for modelgain * teacherCategory for modelgain * Subject(Group)
ANOVA Table for modelgain
1.012 1.571 .2047.511 1.543 .5139
-.502 1.739 .5692
Mean Diff. Crit. Diff. P-ValueA, SA, TS, T
Fisher's PLSD for modelgain Effect: teacher Significance Level: 5 %
WISE Period 2 - sig. Epistemological gains
0
2
4
6
8
10
12
14
Cell
Mea
n
preMtot postMtotCell
TSA
Interaction Bar Plot for modelgain Effect: Category for modelgain * teacher
2 2.335 1.167 .079 .9244 .158 .06159 874.504 14.822
1 311.401 311.401 40.945 <.0001 40.945 1.0002 56.782 28.391 3.733 .0297 7.466 .659
59 448.710 7.605
DF Sum of Squares Mean Square F-Value P-Value Lambda PowerteacherSubject(Group)Category for modelgainCategory for modelgain * teacherCategory for modelgain * Subject(Group)
ANOVA Table for modelgain
.064 1.632 .9380-.289 1.632 .7268-.353 1.827 .7028
Mean Diff. Crit. Diff. P-ValueA, SA, TS, T
Fisher's PLSD for modelgain Effect: teacher Significance Level: 5 %
WISE Period 3 - sig. Epistemological gains
0
2
4
6
8
10
12
14
Cell
Mea
n
preMtot postMtotCell
TSA
Interaction Bar Plot for modelchange Effect: Category for modelchange * teacher
2 47.195 23.597 1.433 .2464 2.866 .28562 1021.132 16.470
1 366.531 366.531 54.803 <.0001 54.803 1.0002 106.362 53.181 7.952 .0008 15.903 .958
62 414.665 6.688
DF Sum of Squares Mean Square F-Value P-Value Lambda PowerteacherSubject(Group)Category for modelchangeCategory for modelchange * teacherCategory for modelchange * Subject(Group)
ANOVA Table for modelchange
-.809 1.684 .3437.833 1.654 .3207
1.642 1.876 .0857
Mean Diff. Crit. Diff. P-ValueA, SA, TS, T
Fisher's PLSD for modelchange Effect: teacher Significance Level: 5 %
WISE Period 4 - sig. Epistemological gains
0
2
4
6
8
10
12
14
Cell
Mea
n
preMtot postMtotCell
TSA
Interaction Bar Plot for modelchange Effect: Category for modelchange * teacher
2 63.678 31.839 2.807 .0681 5.614 .52362 703.214 11.342
1 190.437 190.437 35.768 <.0001 35.768 1.0002 65.833 32.917 6.182 .0036 12.365 .889
62 330.098 5.324
DF Sum of Squares Mean Square F-Value P-Value Lambda PowerteacherSubject(Group)Category for modelchangeCategory for modelchange * teacherCategory for modelchange * Subject(Group)
ANOVA Table for modelchange
-.073 1.392 .9180-1.589 1.367 .0231 S-1.516 1.551 .0552
Mean Diff. Crit. Diff. P-ValueA, SA, TS, T
Fisher's PLSD for modelchange Effect: teacher Significance Level: 5 %
WISE Period 5 - sig. Epistemological gains
0
2
4
6
8
10
12
14
Cell
Mea
n
preMtot postMtotCell
TSA
Interaction Bar Plot for modelchange Effect: Category for modelchange * teacher
2 26.202 13.101 .840 .4368 1.680 .18160 936.016 15.600
1 444.676 444.676 75.513 <.0001 75.513 1.0002 90.227 45.113 7.661 .0011 15.322 .950
60 353.325 5.889
DF Sum of Squares Mean Square F-Value P-Value Lambda PowerteacherSubject(Group)Category for modelchangeCategory for modelchange * teacherCategory for modelchange * Subject(Group)
ANOVA Table for modelchange
.701 1.631 .3970-.531 1.758 .5510
-1.232 1.909 .2040
Mean Diff. Crit. Diff. P-ValueA, SA, TS, T
Fisher's PLSD for modelchange Effect: teacher Significance Level: 5 %
Modeling Across the Curriculum Team
Principal & Co-Principal InvestigatorsPaul Horwitz, Concord Consortium, Principal InvestigatorJanice Gobert, Concord Consortium, Co-PI & Research DirectorRobert Tinker, Concord Consortium, Co-PIUri Wilensky, Northwestern University, Co-PI
Other senior personnel Barbara Buckley, Concord Consortium Chris Dede, Harvard University Ken Bell, Concord Consortium Sharona Levy, University of HaifaTrudi Lord, Concord Consortium Jaclyn Scobo (intern), Northeastern University
mac.concord.org; IERI #0115699
www.concord.org
http://ccl.northwestern.edu
Design of Activities, Scaffolding, & Research are based on…
Model-based learning (Gobert & Buckley, 2000) as well as other literature….
• cognitive and perceptual affordances of learning with technology-based representations (Gobert, 2005; Larkin & Simon, 1987)
• progressive model-building (White & Frederiksen, 1990; Raghavan & Glaser, 1995)
• students’ difficulties in learning with models (Sweller, et al, 1990; Gobert, 1994; Lowe, 1989; Head, 1984).
Thus, scaffolding is designed to… • guide search, supports perceptual cues, and inference-making from
perceptual cues (Larkin & Simon, 1987). • elicit prior knowledge, support integration with new knowledge, and
support reification & reflection of knowledge. Theory driving our analyses is based on…• expert problem-solving for estimating solutions (Paige & Simon, 1966)• experts vs. novices search and knowledge acquisition strategies (Gobert,
1994, 1999; Thorndyke & Stasz,1980).
Model-Based Learning in situ
Intrinsic Learner Factors
Epistemology of models(SUMS, Treagust et al, 2002)
….because students’ epistemologies influence both knowledge integration (Songer & Linn, 1991) and model-based reasoning (Gobert & Discenna, 1997),
Intrinsic Teacher Factors
Epistemology of models(adapted from
Grosslight et al, 1991)
Teaching experienceBackground
(adapted from Fishman, 1999)
Classroom FactorsImplementation of MAC activity use
(logged)Teacher practices
(reported via Classroom Communique)
Hypermodels*simulationsdiagrams
explanationsinstructionsdata tables
graphs
model reinforcementmodel revision
model rejection
Learner'sMentalModels
model evaluation
prior knowledge new information
model formationInteracting with
understandingreasoninggenerating
Phenomenaexperiencesexperiments
model use
+ MetacognitiveSelectingDirecting Monitoring
What is the model for the pedagogical support of the practice? What kinds
of designs put this model into effect?
Scaffolds from the MAC project include:
• Representational Competence: view and understand a representation or representational features of the domain.
• Model pieces acquisition: understand & reason with pieces of models (spatial, causal, functional, temporal).
• Model pieces integration: combine model components in order to come to a deeper understanding of how they work together as a causal system.
• Model based reasoning: reasoning with models or pieces of models. • Reconstruct, Reify, & Reflect: reify knowledge and transfer it to
another context or level of understanding.
Technology & Affordances for Research & Assessment with
ModelsTechnology:
Log files on students’ interactions with models capturing students’…
– Data on duration and sequence – Actions and choices with models– What info or help they seek– Responses to questions
Embedded & Performance Assessments with models & questions…
– Generate profile for students at
“pivotal” points in curriculum– Responses to questions
AffordancesImplementation data -- which activities
were used, pattern of use (consecutive or intermittent days) at classroom level & student level.
Finer-grained log data can be used for– Measure of students’
systematicity and inquiry skills.
– Test for interactions with prior knowledge & epistemology.
These data are used to derive student reports….
¯ Formative assessments ¯ Summative assessments¯ Performance assessments
Drill down to performance assessment
with logsCurrently we are focusing on log files as indices of:1) Domain-specific model based reasoning by investigating
“hot spots”2) Domain- General Inquiry skills (DoGI spots, similar to
NSES inquiry strands).3) This allows us to assess inquiry development
– both within (hot spots) and across domains (DoGI spots). – assess transfer from one domain to another– assess how a student’s inquiry skills are progressing
“independent” of content learning. Since our activities are enacted over multiple days and in
three domains, we avoid the problems faced by earlier studies of inquiry in which there were not enough data to get at students’ inquiry skills (Shavelson et al, 1999).
Inquiry “Hot Spots”Tasks or parts of tasks that contain multiple
components of model-based inquiry these, by definition, require deep reasoning.
MAC supports 5 strands of model-based inquiry. These are more specific than the NSES (1996) inquiry standards which were are not specific to current technology-based learning nor are the NSES strands specific to modeling tasks.
• Representational Competence: view and understand a representation or features of the domain.
• Model pieces acquisition: understand & reason with pieces of models (spatial, causal, functional, temporal).
• Model pieces integration: combine model components in order to come to a deeper understanding of how they work together as a causal system.
• Model based reasoning: reasoning with models or pieces of models. • Reconstruct, Reify, & Reflect: reify knowledge and transfer it to another
context or level of understanding.Fine-grained analysis, one hot spot at a time, is necessary
in order for us to code the various process variables we plan to aggregate and focus on.
Hot spot from Collisions task 5: Student sets mass of two balls
• The challenge: adjust the masses of the two balls to make the orange ball move as fast as possible after the collision.
Strategies for InquiryPreliminary analysis based on human
coding identified 2 different inquiry patterns:
1. haphazard2. systematic(Also, there are students who got it correct on first
trial, sometimes with explicit test).
These are consistent with literature: ~ experts vs. novices search and knowledge acquisition
strategies (Thorndyke & Stasz,1980; Gobert, 1994, 1999).~ expert problem-solving for estimating solutions (Paige &
Simon, 1966).
Examples …
Haphazard Strategy- this student obtained the correct answer (11.0; 1.0) on trials 2,10,(& 15) but did
not know it!Student 12116 made 15 trials:Blue Ball Orange ball
11.0 11.0 11.0 1.011.0 3.011.0 4.0 1.0 1.0 1.0 11.0 8.0 7.0 11.0 2.0 11.0 11.0 11.0 1.011.0 5.03.0 5.01.0 5.01.0 8.011.0 1.0
Systematic Strategy, e.g.,vary one ball at a time (a good strategy in the absence of prior knowledge).
Student 18115 had a plan:Blue Ball Orange ball
11.0 11.0 5.0 11.0
10.0 11.011.0 1.0
Another Hot Spot from Dynamica: “What settings cause the blue ball to stop when
it collides with the orange ball?”
Input sliders
Numerical data from runConstructed
text response
• Track students’ iterations of this as index of systematicity in inquiry.
TL3 time
TL3 RdTsk
T3 trials T3 values T3 Vx
T3 #rtPr
T3 success Q10A
T3 %vary1
T3 %rpt
T3 #eqPr
T3 #extrem
PrT3
%clgT3
%frg
T3 %goal Flips
T3 CAT
2.5 73 2 2.0 v 5.0 5.0 v 5.0
-1.7, 0.0,
1 1 that they must have have equal masses
1 0 1 0 1 0 0 B1
2.9 34 8 2.0 v 5.0, 4.0 v 11.0, 1.0 v 4.0,
11.0 v 11.0, 6.0 v 7.0, 5.0 v 7.0, 3.0 v 7.0, 7.0 v 7.0,
-1.71, -1.87, -2.4, 0.0, -0.31, -0.67, -1.6, 0.0,
2 1 they have to both have to be the same size
0.43 0 2 0 0.29 0.71 0.43 B2
2.7 130 1 5.0 v 5.0, 0.0, 1 1 match the orange 0 0 1 0 0 0 0 A
2 13 10 2.0 v 5.0, 1.0 v 5.0,
11.0 v 5.0, 8.0 v 5.0,
8.0 v 11.0, 7.0 v 11.0, 6.0 v 11.0, 7.0 v 10.0, 11.0 v 10.0, 10.0 v 10.0,
-1.71, -2.67, 1.5,
0.92, -0.63, -0.89, -1.18, -0.71, 0.19, 0.0,
1 1 The mass must be almost as big as the other ball
0.89 0 1 0 0.67 0.33 0.33 B2
CC’s approach for Task 3- Additional Categories for coding & 4
students’ data.
Additional categories (in addition to CMU) are % of trials in which~ set the masses as equal~ set the masses as extremes~ closer the the goal, further from the goal,~ goal flips.
Hot Spot from BioLogica: Monohybrid (Task 3):
produce only 2-legged offspring
Arrow toolCross toolSnip tool
Chromosome tool
Dragon genome chartPunnett square pad
The task
Requires changing both Legs alleles of one parent
Monohybrid Task 3 Subtasks & Data
collected• Predict whether a pair of dragons can have only 2-legged
offspring– Multiple choice question
• Describe the necessary parental genotypes.– Full text response
• Change alleles of one parent to homozygous recessive
• Cross parents– Success = making right cross– Number of crosses made– List of crosses made
Data Monohybrid performances
• Student performance is scored by computer based on– Prediction– Success– Number of attempts– Whether they repeated any crosses (an indication of haphazard behavior).
• Performances can be grouped into – Systematic & correct– Systematic & incorrect– Haphazard & correct– Haphazard & incorrect
Systematic vs. haphazard performance and Pre/Post
gainsDependent Var iable: Total Score PostT3CATSYS2NUM Mean Std. Error 95% Confidence Interval
Lower Bound Upper BoundHaphazard 19.980 .740 18.518 21.442Systematic 22.868 .651 21.582 24.154
Table below indicates that Pre-test covariate is significant, as is the two-category predictorvariable (S, H). Together, the covariate and this variable account for 25.2% of the variance in thePost-test scores.
Data based on 649 students in 10 member schools; (54.2%) in ‘regular’ classes.
ANCOVA with pre test score as covariate indicates pre-test is significant (p ≤ .001), as is the two-level predictor variable (Systematic versus Haphazard).
Together, the pretest scores and the systematicity variable account for 25.2% of the variance in the Post-test scores.
Students who are systematic at this task outperform students on the post-test who are not, irrespective of whether they succeeded at the inquiry task.
Thus the systematic inquiry is facilitating knowledge building (as measured by the post-test).
Overview of Data Analysis with Hot spots
We are aggregating hot spots and testing their relationship:
• conceptual learning measurements, i.e., pre-post content tests• measures of students’ epistemologies of models and views of science since
students’ epistemologies influence learning (Songer & Linn, 1991; Gobert & Discenna, 1997).
With these data, we can:• track students’ systematicity in learning with models as one important facet
of inquiry skills and conceptual learning. To us, inquiry skills co-evolve with content learning but each can be measured separately (sort of).
• test for development of inquiry strategies across time and across domains ~ complicated by task difficulty increasing over time ~ complicated by the co-evolution of the development between domain-knowledge and inquiry strategies ~ complicated by the likelihood that students build knowledge in small, conceptual pieces, I.e., about acceleration or velocity).
In the future, using log files we seek to identify at risk students- i.e., students whose inquiry strategies are buggy.
Domain-General Inquiry Spots (“DoGI”
spots)1- Making predictions with models2- Interpreting data from a representation (i.e.,
model/graph, pedigree, etc). 3- Making explanations (about models, etc)4- Mathematizing with models- Filling in an
equation/solve an equation; reasoning with an equation.
5- Designing and/or conducting an experiment with models.
Thus, if a student can do these types of tasks, they are doing model-based inquiry.