assessment: designs and approaches
TRANSCRIPT
![Page 1: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/1.jpg)
Assessment:
Designs and Approaches
Gail Matthews-DeNatale
Northeastern University
Dave Consiglio
Bryn Mawr College
![Page 2: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/2.jpg)
The Cycle
Identify
Scope
Interpret
Data
Share &
Act Upon
Findings
Form
Research
Question
Design &
Implement
ResearchYou Are
Here
![Page 3: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/3.jpg)
On a Dime
http://www.flickr.com/photos/jdhancock/3544855402
![Page 4: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/4.jpg)
Gold Standard
http://www.flickr.com/photos/bullionvault/3591745555
![Page 5: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/5.jpg)
Three Scenarios
Activity
![Page 6: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/6.jpg)
Online Enrollment
Your Dean wants to raise the cap
on enrollment in online courses.
The cap limit will be revisited on an
annual basis. What research can
you perform with your students
and your faculty at your institution
to inform these decisions?
![Page 7: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/7.jpg)
Clickers
Your institution has been using
classroom response systems
(a.k.a. “clickers”) for several years.
Usage is robust. You’re leading a
task force to assess the impact of
clickers on learning.
![Page 8: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/8.jpg)
Blended Learning
A pilot initiative is underway at
your school to integrate blended
courses across the curriculum.
What will be the research
component for the pilot, how will
the research be performed, and
how will it be used?
![Page 9: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/9.jpg)
North Star Questions
What do you want to know/learn?
What’s the purpose?
What’s the population?
Who will receive the results?
Who could/should be involved?
![Page 10: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/10.jpg)
Scenario Planning
20Minutes
What do you want to know?
What’s the population?
What’s the purpose?
Who will be involved?
Who will receive the results?
![Page 11: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/11.jpg)
Gold Standard
http://www.flickr.com/photos/bullionvault/3591745555
![Page 12: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/12.jpg)
Gold Standard
![Page 13: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/13.jpg)
Reliability
and Validity
Gold Standard
![Page 14: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/14.jpg)
Controls
Gold Standard
![Page 15: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/15.jpg)
Example: Try Science
![Page 16: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/16.jpg)
Qualitative Methods (Mostly)
Background information (years teaching, education, prior OL)
Activity and work products(discussion texts, projects, reports)
Formative feedback responses
Teacher follow-up
interviewsProgram retention rates
Pre-post thought experiments
Self-reported increase in confidence
Change in classroom practice (pre- post -observation/video)
Tool
NVIVO
![Page 17: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/17.jpg)
Interviews
I thought I knew everything
about density. When I did my
physics unit in class, I thought I
was all set. As long as you know
the formula, you know density --
that's the way I taught it before.
Try Science blew my mind. At first I was a little bit angry
because I found out that for the three years I taught
density, in some cases, I know truly believe that I was
teaching it incorrectly. The assumptions that I made from
reading it instead of doing it were wrong.
![Page 18: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/18.jpg)
Turning Qual into Quant
96% Program Retention Rate
![Page 19: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/19.jpg)
Cost: $600,000
Report: 121 pages
Some Numbers
![Page 20: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/20.jpg)
Quantitative Example: CATS
Goal: Study effect of interactive instruction on student learning
Tools: Standardized concept inventory, demographic and
attitudinal survey of students, teaching practice survey of instructors
Population: Undergraduate students taking Astronomy 101
Study group: Thousands of students of dozens of higher
education institutions from across the country
Process: Pre and post tests using concept inventory, student
survey demographic and attitudinal survey, measure classroom and
institutional variables
Results: Statistical analysis of individual and classroom
influences on learning gains
![Page 21: Assessment: Designs and Approaches](https://reader030.vdocuments.us/reader030/viewer/2022032616/55a6006c1a28abc4498b45a5/html5/thumbnails/21.jpg)
Your Task Over Lunch
What would these “gold standard”
methodologies would look like if
they were done “on a dime”?
How might you adapt them for use
in situations that are under-
resourced and pressed for time?