evaluating the impact of change in curriculum and teaching bob delmas, elizabeth fry, laura le and...

Evaluating the Impact of Change in Curriculum and Teaching

Bob delMas, Elizabeth Fry, Laura Le and Anelise Sabbag

Funded by NSF DUE-1043141

Session Overview

• Introduce three assessment instruments• Participants examine subsets of items

• Background on assessment instruments

• Participants explore possible use of the assessment instruments in their curriculums

• Wrap-Up: recommendations for using the instruments

But First, Some Light Entertainment

Three Assessment Instruments• GOALS: 27 forced-choice items

Study design Reasoning about variability Sampling and sampling variability Interpreting confidence intervals and p-values Statistical inference Modeling and simulation

• MOST: Measure of statistical thinking 4 real-world contexts open-ended and forced-choice items

• Affect survey: attitudes and perceptions

Review Assessment Items

• GOALS items 15, 16, 17 and 18

• MOST items 7, 9, 10 and 11

• Affect items 1, 3, 4, 5 and 9

• Answer these questions for each item

– What do you think the item is trying to assess?

– How do you think students would answer?

– Are there options that you think students would choose/not choose?

Small Groups

Form a group with one or two other people and discuss your perspectives about the items

• What do you think the item is trying to assess?

• How do you think students would answer?

• Are there options that you think students would choose/not choose?

GOALS ItemsInterpret Confidence Interval Context

High School Class

Wants to estimate average wt. of

chocolate in Chocolate Chip cookies

Rinsed away

dough & weighed

chocolate

Estimated 95% Confidence Interval

GOALS Items 15 through 18Interpreting Confidence Intervals

For questions 15 to 18, indicate whether the interpretation of the interval provided is valid or invalid.

15. With 95% confidence, we can infer that each cookie for this generic brand has approximately 5.65 to 6.35 grams of chocolate.

Valid Invalid

16. With 95% confidence, we can infer that the average weight of chocolate per cookie in this generic brand is 6 grams plus or minus 0.35 grams.

Valid Invalid

17. We can infer that 95% of all cookies from this generic brand will have between 5.65 to 6.35 grams of chocolate. Valid Invalid

18. With 95% confidence, we can infer that the interval of 5.65 to 6.35 grams includes the true average weight of chocolate per cookie. Valid Invalid

MOST Item: Comparing Groups Context

20 students

Random assignment

10 students: Strategy A

10 students: Strategy B

Mean for A = 5 points higher than mean for B

MOST Item: Comparing Groups Context

7. Explain how the researcher could determine whether the difference in means of 5 points is large enough to claim that one preparation strategy is better than the other. (Be sure to give enough detail that someone else could easily follow your explanation in order to implement your proposed analysis and draw an appropriate inference (conclusion).)

100 students: Strategy A

100 students: Strategy B

MOST Item: 2nd Comparing Groups

9. Imagine that a p-value was produced for the situation described in Questions 7 and 8 where there were 10 students in each group. Would you expect to get the same p-value in this situation where there are 100 students in each group, or would you expect a larger or smaller p-value? Circle the letter of the best response to the question.

• Smaller p-value• Same p-value • Larger p-value

MOST Item: 2nd Comparing Groups

10. Circle the letter of the best explanation for your answer in Question 9.

a) Because a larger sample size produces a more accurate p-value.

b) Because a larger sample size decreases the amount of variation in the distribution of statistics.

c) Because a larger sample size allows us to generalize better to a whole population.

d) Because a larger sample size will give us more variation in the results.

e) Because the observed mean difference and null model didn’t change.

f) Other_______________________________________

MOST Item: 2nd Comparing Groups

MOST Item: Estimation Context

11. Explain how you could help them come up with a good estimate of how many hours a student at that school typically works at outside jobs based on their sample data. (Be sure to give enough detail that someone else could easily follow your explanation in order to implement your proposed analysis and draw an appropriate inference (conclusion).)

MOST Item: Estimation

Affect Items

1. This course helped me understand statistical information I hear or read about in the media.

3. This course helped me learn how to make better decisions when faced with uncertain outcomes.

4. This course helped me realize that how data are produced or collected has an impact on the scope of conclusions that can be made.

8. Learning to (use software/create models in TinkerPlots) helped me learn to think statistically.

9. I believe I am well prepared for future classes that require an understanding of statistics.

RESPONSE SCALE

Strongly Disagree Disagree Agree

Strongly Agree

Instrument Background

• General background on CATALST

• For each instrument– Assessment focus

– Design features

• Use of instruments in evaluation of CATALST curriculum

CATALST Project

• 4-year NSF-funded project (DUE-0814433)

• Radically different intro statistics course

– No t-tests; Use of probability for simulation and modeling

– Coherent curriculum that builds ideas of models, chance, simulated data, inference from first day

– Immersion of students in statistical thinking

– Activities based on real problems, real data

GOALS (Goals and Outcomes Associated with Learning Statistics)

• Focus on types of reasoning to be developed in a first statistics class

• 27 forced-choice items

• 2 versions, same except for items 19-22– One for students in course that teaches

randomization methods for inference– One for students in a course that teaches

traditional methods of inference

MOST (Models of Statistical Thinking)

• Focus on how students think about problems that involve statistical inference

• 4 real-life contexts designed to elicit statistical thinking

• 11 items– 4 open-ended questions, one for each context– 7 forced-choice questions to provide more detail

• 1 version– Can be used in traditional or randomization-based courses

Affect Survey

• Focus on attitudes and beliefs at the end of an introductory statistics class

• 12 items, each using 4-point scale:Strongly Disagree – Disagree – Agree – Strongly Agree

– 4 items about the course– 4 items about using software– 4 items about statistics

• 2 versions– One for classes (e.g, CATALST) that use TinkerPlots™ software– One for classes that use any other software

General Purpose and Use of Instruments• Use of instruments

– To evaluate important student outcomes: reasoning, thinking and attitudes/beliefs

• In CATALST, 3 groups were compared:– CATALST class at U of M (Spring 2012)

• 3 sections all taught the same way, with final version of materials

• Similar students enrolled in all sections

– CATALST classes at 4 other schools (Spring 2012) • Different types of students, requirements

– Non-CATALST: from 6 schools (Fall 2011 and Spring 2012)• Served as a type of control group

GOALS: Bootstrap Confidence Intervals of difference in Mean Proportion Correct for Each Item

(CATALST: n = 289; non-CATALST: n = 440)

Comparison: Confidence Intervals ItemsMean Percent Correct

Comparison: Confidence Intervals ItemsMean Total Number Correct

N MEAN SD

UofM CATALST

138 2.67 0.80

Non-UofM CATALST

151 2.31 0.85

Non-CATALST

440 2.10 0.97

Analyzing MOST Responses

• Statistical Thinking Checklist– Wild and Pfannkuch’s (1999) framework of statistical

thinking– Thinking like an expert from expert-novice literature

Students in Randomization-based Courses

Students in Traditional Course

Modeling Modeling

Integration of statistical and contextual

Integration of statistical and contextual

Domain-specific knowledge Domain-specific knowledge

Statistical Models

MOST Results:Exam Strategy, Description Item

CATALST UofM students

(n = 138)

CATALST non-UofM students (n = 120)

ModelingStudents described a Null Model 39.49% 21.25%

Integration of Statistical and ContextualStudents described the Context within Model

48.19% 32.92%

ModelingStudents provided a Description of the Simulation 43.84% 35.42%Domain-Specific KnowledgeStudents knew how to Make Conclusions based off of the (possible) statistical results

38.77% 24.58%

Integration of Statistical and ContextualStudents described the Context within Conclusion

28.62% 14.17%

MOST Results:Exam Strategy, Description Item

non-CATALST students (n = 187)

Modeling Students described a Null Hypothesis 4.55%Modeling Students described an Alternative Hypothesis 1.87%Integration of Statistical and ContextualStudents described the Context within Hypotheses

3.21%

Domain-Specific KnowledgeStudents described needing to Check Assumptions about the Data

0.80%

Statistical ModelsStudents Stated type of test (z, t, etc.) that would be appropriate for the problem

17.11%

Domain-Specific KnowledgeStudents knew how to Make Conclusions based off of the (possible) statistical results

17.11%

Integration of Statistical and ContextualStudents described the Context within Conclusion

5.35%

MOST Results:Exam Strategy, M-C items

Group

Students who chose “smaller p-value” and correct reason

CATALST UofM students (n= 138)

38.41%

CATALST non-UofM students (n = 120)

15.83%

non-CATALST students(n = 187)

18.18%

MOST Results:Study hours, interval estimate item

CATALST UofM students (n = 138)

CATALST non-UofM students (n = 120)

ModelingStudents described using the Data to find an Interval Estimate

30.07% 15.42%

Integration of Statistical and ContextualStudents described the Context within Model 23.19% 12.92%ModelingStudents provided a Description of the Simulation 36.23% 17.92%VariationStudents mentioned the Need for a variability measure for the CI calculation

48.91% 17.92%

Statistical ModelsStudents knew the CI calculation 40.22% 12.50%Domain-Specific KnowledgeStudents knew how to Make Conclusions based off of the (possible) statistical results and Interpret the interval

22.83% 7.92%

Integration of Statistical and ContextualStudents described the Context within Conclusion 28.99% 13.33%

MOST Results:Study hours, interval estimate item 

non-CATALST students (n = 187)

Statistical ModelsStudents Stated type of test (z, t, etc.) that would be appropriate for the problem

3.74%

Domain-Specific KnowledgeStudents described needing to Check Assumptions about the Data

0.27%

VariationStudents mentioned the Need for a variability measure for the CI calculation

4.81%

Statistical ModelsStudents knew the CI calculation 1.60%

Domain-Specific KnowledgeStudents knew how to Make Conclusions based off of the (possible) statistical results and Interpret the interval

3.48%

Integration of Statistical and ContextualStudents described the Context within Conclusion 7.49%

Affect Survey Results

Evaluating the Impact of Change

• Look at the full instruments!

• Questions to think about:– Would these provide useful information to you about your

students if you knew how they performed on these assessments?

– What could you learn from data that you would gather from your students using these assessments?

– What kind of data would you want to compare your student data to? (e.g., national sample, other statistics courses at your institution, other future statistics courses)

Summary and Final Thoughts

• Project Goal: Develop good assessment instruments that provide useful information for curriculum change and development

• Purpose of these instruments: research and evaluation projects

Summary and Final Thoughts

• Instrument development is time consuming, taking several years of testing and revision

• Still under development

Summary and Final Thoughts

• Not designed to replace other kinds of assessments in course (e.g., quizzes, minute papers, projects)

• We want people to use the instruments• Or, you can use some of the items from

an instrument• BUT…

If you use only some items from an instrument

DON'T use them for program evaluations or for research projects

DO use them as part of assessments in your own course

Thank You for your Participation

If you have comments, issues, ideas of what is missing in the instruments, please send one of the PIs an email:

Joan Garfield: jbg@umn.eduBob delMas: delma001@umn.eduAndy Zieffler: zief0002@umn.edu