gwynn mettetal. discuss different ways to assess outcomes help you decide which methods would be...

Gwynn Mettetal

Discuss different ways to assess outcomes Help you decide which methods would be

best for your project

Who are you? What sort of Vision 2020 project are you

planning? (the two sentence version)

Assessment—evidence that your project is making a difference

You MUST assess the effectiveness of your Vision 2020 grant to get continued funding!

Lots of strategies possible Depends on your goals Depends on your situation

Quantitative (numbers) Grades, attendance, ratings on a scale, retention rate

Qualitative (words) Interviews, essays, open ended survey questions

Both are fine, just different

Existing data (easiest, already there) ◦ Student records◦ Archival data◦ Student work in course

Conventional sources (easy, but must generate)◦ Behavioral data—journals, library usage◦ Perceptual data—surveys, focus groups, interviews

Inventive sources (difficult)◦ Products or performances

Must treat students respectfully Must protect privacy Must “do no harm”

Collecting new data (not coursework) from your own students?◦ Have someone else collect and hold until grades are in◦ Can’t force them to participate◦ Can’t take up too much instruction time

Institutional Review Board (IRB)◦ If planning to publish

Add power--compare groups!◦ Before and after◦ Different course units◦ This semester and last◦ Two sections with different methods◦ Your class to that of another instructor

Be realistic--start small

Validity—does your evidence (data) mean what you think it means?

Example test scores = deep learning? What if just rote memory? What if students cheated?

Reliability—would you get the same evidence if you collected it again? Or was this just a fluke?

Example: Test scores = deep learning? What if you gave again next week and scores were very

different?

In general, hard to have both.

Real life is messy (valid, not as reliable) Experiments are controlled (reliable, not as

valid)

Solution is . . .

Get several different types of data Different sources:

◦ Instructors, students, advisors, records Different methods:

◦ Surveys, observations, student work samples Different times:

◦ Start and end of semester, two different classes, two different semesters

Course evaluations

final project rubric

Comparison to last semester’s class

What data could YOU collect?

Qualitative analyses: look for themes in words and behaviors

Theme 1: Students understood more abstract concepts after group discussion. (Follow with quotes from student exams, other evidence.)

Quantitative analyses: simple graphs, tables Simple statistics: means, correlations, t-tests

Focus on practical significance, more than statistical significance

What would convince YOU?

If evidence was good, keep your old strategy If evidence was weak, tinker to improve your strategy Plan to assess again, after working with a new group of

students You will need to show how you used your data to get

continued Vision 2020 funding!