What evidence will we accept?

Disciplinary

Research

Strategies for

Assessing

Understanding

Diane Ebert-MayDepartment of Plant Biology

ebertmay@msu.edu

Anonymous (Change Magazine, 2001):

Anonymous (Change Magazine, 2001):

“I believe we would all agree that the absolute best teaching learning-assessment model is the one-on-one Socratic apprenticeship model with unlimited time with the student. But ever since Socrates took on two students rather than only one (to double his income), teachers have had to make compromises in teaching.”

“I believe we would all agree that the absolute best teaching learning-assessment model is the one-on-one Socratic apprenticeship model with unlimited time with the student. But ever since Socrates took on two students rather than only one (to double his income), teachers have had to make compromises in teaching.”

Larry Spence, Penn State

Larry Spence, Penn State

“Today’s graduates cannot meet the demands of workplace or without several more years of learning on the job. They cannot formulate and solve messy real-world problems, work well with others in high-stress team situations, write and speak forcefully and persuasively, or improve their own performance.”

“Today’s graduates cannot meet the demands of workplace or without several more years of learning on the job. They cannot formulate and solve messy real-world problems, work well with others in high-stress team situations, write and speak forcefully and persuasively, or improve their own performance.”

Question 1

How important is it to use multiple kinds of data to assess student learning?

Please respond on a scale if 0-100 in increments of 10:

How important is it to use multiple forms of data to assess student learning?

%

Relative Importance n=127

Question 2

How often do you use data to make instructional decisions?

Please respond on a scale of 0 - 100 in increments of 10:

How often do you use data to make instructional decisions?

n=127Frequency

%

True or False?

Assessing student learning in science parallels what scientists do as researchers.

1.Description:

What is happening?

2.Cause:

Does ‘x’ (teaching strategy) affect ‘y’ (understanding)?

3.Process or mechanism:

Why or how does ‘x’ cause ‘y’?

Parallel: ask questions

We collect data to find out what our students know.

Data helps us understand student thinking about concepts and content.

We use data to guide decisions about course/curriculum/innovative instruction

Parallel: collect data

Quantitative data - statistical analysis

Qualitative data

break into manageable units and define coding categories

search for patterns, quantify

interpret and synthesize

Valid and repeatable measures

Parallel: analyze data

Ideas and results are peer reviewed - formally and/or informally.

Parallel: peer review

Graduate Education

Often excellent at preparing individuals to design and carry out disciplinary research.

Graduate Education

Often inadequate and haphazard in preparing future faculty/professionals to take on the increasingly complex demands of the professoriate.

Teaching is not mentored, peer reviewed, or based on accumulated knowledge.

Solution: IRD model Intergenerational research teams (IRDs)

in cooperative academic environments» Who: senior faculty, junior faculty,

postdoctoral and graduate students. » What: scholarship of science teaching

and learning is fully integrated into the professional culture along with discipline-based activities.

Assessment is critical to both practices.

IRD Team at MSUJanet Batzli - Plant Biology [U of Wisconsin]

Doug Luckie - PhysiologyScott Harrison - Microbiology (grad student)Tammy Long - Plant BiologyJim Smith - ZoologyDeb Linton - Plant Biology (postdoc)

Heejun Lim - Chemistry EducationDuncan Sibley - Geology*National Science Foundation

Recognizing and Rewarding

Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics (2003)

» National Research Council» www.nap.edu/catalog/10024.html

What is assessment?

Data collection with the purpose of answering questions about…

students’ understanding

students’ attitudes

students’ skills

instructional design and implementation

curricular reform (at multiple grainsizes)

Formative - feedback to inform the student performance or program

Summative - judgements about the quality of the program

“When the cook tastes the soup that’s formative; when the guests taste the soup, that’s summative.” (Bob Stake)

Assessment Planning

Why do assessment?

Improve student learning and development.

Provides students substantive feedback about their understanding.

Challenge to use disciplinary research strategies to assess learning.

Research Methods

Data collection

approaches

Multiple Choice … … Concept Maps … … Essay … … Interview

high Ease of Assessment low

low Potential for Assessment of Learning high

Theoretical Framework• Ausubel 1968; meaningful learning• Novak 1998; visual representations• King and Kitchner 1994; reflective judgment• National Research Council 1999; theoretical frameworks for assessment

Assessment Gradient

What are central questions about learning and

understanding?1. What do we want our students to know and be able to do?2. What knowledge or misconceptions do our students bring to the course?3. What evidence will we accept that students know and can do?4. How does our instruction help understanding?

What Type of Learning?

Bloom (1956)

6 major categories in the Cognitive Domain of Educational Objectives

Condense to 4 - realistic to work with

Cognitive Levels

Knowledge - rememberUnderstanding and Application - grasp meaning, use, interpretCritical Analysis - original thinking, open-ended answers, whole to parts, parts to whole, evaluationSynthesis - make connections

If assessment is....

...data collection with a purpose of answering Qs.....

Then we...

gather data about students’ learning/understanding

make certain we gather the appropriate data to answer the question

use tools like Bloom’s taxonomy to ‘calibrate’ data

OR.....

Carbon Cycle = Rich Problem Why?

Integrates many biological concepts at multiple scales.Instruction can return to elements intrinsic in the carbon cycle - bioenergetics, metabolism. Several documented student misconceptions associated with the carbon cycle. Real-world applied consequences if students continue to misunderstand.

Some Common Misconceptions about Photosynthesis & Respiration

Concept 1: Matter disappears during decomposition of organisms in the soil.

Concept 2: Photosynthesis as Energy: Photosynthesis provides energy for uptake of nutrients through roots which builds biomass. No biomass built through photosynthesis alone.

Concept 3: Thin Air: CO2 and O2 are gases therefore, do not have

mass and therefore, can not add or take away mass from an organism.

Concept 4: Plant Altruism: CO2 is converted to O2 in plant leaves so

that all organisms can ‘breathe’.

Concept 5: All Green: Plants have chloroplasts instead of mitochondria so they can not respire.

Multiple choice question (pre-

post)

The majority of actual weight (dry biomass) gained by plants as they progress from seed to adult plant comes from which one of the following substances?

a. Particle substances in soil that are take up by plant roots. (15%).

b. Molecules in the air that enter through holes in the plant leaves (4%).

c. Substances dissolved in water taken up directly by plant roots. (28%).

d. Energy from the sun (29%).

n=138

Radish Problem (formative)Experimental Setup:

Weighed out 3 batches of radish seeds each weighing 1.5 g.Experimental treatments

Seeds placed on moistened paper towels in LIGHTSeeds placed on moistened paper towels in DARKSeeds not moistened (left DRY) placed in light

Radish problem (2)

After 1 week, all plant material was dried in an oven overnight (no water left) and plant biomass was measured in grams. Predict the biomass of the plant material in the various treatments. 1.Water, light2.Water, dark

3.No water, light

Results: Weight of Radish Seedlings

1.46 g 1.63 g 1.20 g

Write an explanation about the results.

Minke Whale Problem Two fundamental concepts in ecology are “energy flows” and “matter cycles”. In an Antarctic ecosystem with the food web given above, how could a carbon atom in the blubber of the Minke whale become part of a crabeater seal? Note: crabeater seals do not eat Minke whales. In your response include a drawing with arrows showing the movement of the C atom.

In addition to your drawing, provide a written description of the steps the carbon atom must take through each component of the ecosystem Describe which biological processes are involved in the carbon cycle.

Grandma Johnson Problem Hypothetical scenario: Grandma Johnson had very sentimental feelings toward Johnson Canyon, Utah, where she and her late husband had honeymooned long ago. Her feelings toward this spot were such that upon her death she requested to be buried under a creosote bush overlooking the canyon. Trace the path of a carbon atom from Grandma Johnson’s remains to where it could become part of a coyote. NOTE: the coyote will not dig up Grandma Johnson and consume any of her remains.

Spider Monkey Problem Deep within a remote forest of Guatemala, the remains

of a spider monkey have been buried under an enormous mahogany tree. Although rare, jaguars have been spotted in this forest by local farmers. Use coherently written sentences and clearly labeled drawings to explain how a carbon atom in glucose contained within muscle cells of the spider monkey might become part of a cell within the stomach lining of a jaguar. (Note:The jaguar does not dig up the monkey and eat the remains!) Include in your answer descriptions of the key features (not complete biochemical pathways!) of the organismal and cellular processes that explain how the carbon atom of the monkey’s corpse could become a part of the jaguar’s body.

Analysis of Responses

Used same scoring rubric for all three problems - calibrated by adding additional criteria when necessary, rescoring:

Examined two major concepts: Concept 1: Decomposers respire CO2

Concept 2: Plants uptake of CO2

Explanations categorized into two groups:Organisms (trophic levels)Processes (metabolic)

Code Organisms Code Processes and pathways 1 Decomposers IA Cellular Respiration IB Release CO2 2 IIA Pathway of Carbon

Primary producers IIA _1: through Air IIA _2 : through Root IIA _3 : no mention about pathway IIB Make Glucose IIC Photosynthesis 3 Herbivore III Respiration

(glycolysis, Kreb cycle) 4 Carnivore IV Respiration

(glycolysis, Kreb cycle)

Coding Scheme

Does active, inquiry-based instructional design influence students’ understanding of evolution and natural selection?

Pre-Posttest Analysis

•Changes in a population occur through a gradual change in individual members of a population.

•New traits in species are developed in response to need.

•All members of a population are genetically equivalent, variation and fitness are not considered.

•Traits acquired during an individual’s lifetime will be inherited by offspring.

Alternative Conceptions: Natural Selection

Instructional Design

Cooperative groups in class:

Guppy Problem: sexual vs. natural selection

•PBS film•Simulation•Analyze data•Written explanation

(AAAS 1999)

Explain the changes that occurred in the tree and animal. Use your current understanding of evolution by natural selection.

How do we develop rubrics?

Describe the goals for the activity, problem, taskSelect the assessment tasks aligned with goalsDevelop performance standardsDifferentiate levels of responses based on clearly described criteriaRate (assign value) the categories

Level of Achievement General Approach ComprehensionExemplary(5 pts)

• Addresses thequestion.• States a relevant,justifiable answer.• Presents arguments ina logical order.• Uses acceptable styleand grammar (noerrors).

• Demonstrates an accurate andcomplete understanding of thequestion.• Backs conclusions with dataand warrants.• Uses 2 or more ideas,examples and/or arguments thatsupport the answer.

Adequate(3 pts)

• Does not address thequestion explicitly,although does sotangentially.• States a relevant andjustifiable answer.• Presents arguments ina logical order.• Uses acceptable styleand grammar (oneerror).

• Demonstrates accurate but onlyadequate understanding ofquestion because does not backconclusions with warrants anddata.• Uses only one idea to supportthe answer.• Less thorough than above.

Needs Improvement(1 pt)

• Does not address thequestion.• States no relevantanswers• indicatesmisconceptions.• Is not clearly orlogically organized.• Fails to use acceptablestyle and grammar (twoor more errors).

• Does not demonstrate accurateunderstanding of the question.• Does not provide evidence tosupport their answer to thequestion.

No Answer (0 pts)

Scoring Rubric for Quizzes and Homework

Advantages of Scoring Rubrics

Improve the reliability of scoring written assignments and oral presentationsConvey goals and performance expectations of students in an unambiguous wayConvey “grading standards” or “point values” and relate them to performance goalsEngage students in critical evaluation of their own performance Save time but spend it well

Limitations of Scoring Rubrics

Problem of criteriaProblem of practice and regular use

Scoring Rubric website:http://www.wcer.wisc.edu/nise/cl1/flag/

Sample Rubrics for Organismal Biologywww.first2.orgwww.msu.edu/~ebertmay/isb202/home.html

Misconception: individuals evolve new traitsMisconception: individuals evolve new traits

% o

f S

tud

ents

n=80

Misconception: evolution is driven by needMisconception: evolution is driven by need

% o

f S

tud

ents

n=80

In guppy populations, what are the primary changes that occur gradually over time?

In guppy populations, what are the primary changes that occur gradually over time?

a.The traits of each individual guppy within a population gradually change.

b.The proportions of guppies having different traits within a population change.

c.Successful behaviors learned by certain guppies are passed on to offspring.

d.Mutations occur to meet the needs of the guppies as the environment changes.

Anderson et al 2002

Posttest: Student responses to mcPosttest: Student responses to mc

% o

f S

tud

ents

n=171

*

Animal/Tree Posttest: Gain in student understanding of fitnessAnimal/Tree Posttest: Gain in student understanding of fitness

% o

f S

tud

ents

n=80

What is the question?

What research and instructional designs?

What data collection methods?

How to analyze and interpret data?

Are findings valid and generalizable?

What are the next questions?

WHO?

What evidence will we accept?