program evaluation: one approachprogram evaluation: one ... · assessment of their learning gains....

Program Evaluation: One ApproachProgram Evaluation: One ApproachBIO REU URSSA SALG

Alan D Christian and Robyn E HanniganAlan D. Christian and Robyn E. Hannigan

University of Massachusetts BostonUniversity of Massachusetts Boston

Coastal Research in Environmental ScienceCoastal Research in Environmental Science and Technology (CREST) REU

Student Assessment of Learning Gains (SALG)(SALG)

D l d i 1997 b El i S (UC B ld )▸ Developed in 1997 by Elaine Seymour (UC-Boulder)

▸ Co-evaluator for 2 NSF funded chemistry consortia (ChemLinksy (and ModularCHEM)▸ 1000 instructors in 3000 classes and >65,000 students

▸ Revised by Stephen Carroll, Elaine Seymour, and Tim Weston in 2007 for use outside of Chemistry


I t t’ f▸ Instrument’s focus▸ Degree a course enabled student learning

▸ Students assess and report on ▸ Own learning▸ Degree specific aspects of course contributed to learning

▸ Led to 5 overarching questions1. How much did the following aspects of the course help you in your

learning? 2. As a result of your work in this class, what gains did you make in

your understanding of each of the following?3. As a result of your work in this class, what gains did you make in

the following skills? 4. As a result of your work in this class, what gains did you make in

the following? 5. As a result of your work in this class, what gains did you make in

integrating the following?


SALG W b i ( l i )▸ SALG Website (www.salgsite.org) ▸ Research team for UW-Madison, Santa Clara University, UC-

Boulder, Pacific Metrics, Clotho Advanced Media, and EquilibriousLLC

▸ NSF Supported▸ DUE 0920801DUE 0920801


▸ Online Formative Assessment▸ Online Formative Assessment▸ Improve course▸ Accountability

▸ Can be customized▸ Can be customized▸ Reflects learning activities and objectives of individual courses

▸ FeedbackQ tit ti ti▸ Quantitative ratings

▸ Written responses▸ Focus

L i ti iti▸ Learning activities▸ Content▸ Materials

I t t▸ Instructors▸ Learn if students, on average, gained more in one area than

anotherInformed adjustments to design and teaching▸ Informed adjustments to design and teaching

▸ Documentation for tenure, merit, and promotion as a supplement or substitute to end of course evaluation


▸ DataU ( t ) f lf t d i▸ Use as a group measure (aggregate) of self reported gains▸ Anonymous responses

▸ Not able link to individuals▸ More likely to get honest responses

▸ Validity▸ SALG gathered data from 139 instructors use▸ SALG gathered data from 139 instructors use

▸ 79% used for course redesign versus accountability▸ Changes

▸ 60% modified class activities (lecture, discussion, hands-on)▸ 54% student learning activities▸ 43% course content43% course content▸ 33% information given

▸ 85% reported SALG provided qualitatively different and more useful feedback than traditional student evaluationsuseful feedback than traditional student evaluations

My Use of SALG

S d i i Bi l d E i l S i▸ Started using in Biology and Environmental Science Courses in 2005▸ Instruments

▸ Baseline SALG▸ End of course SALG

▸ Adjust course delivery activities and content based on▸ Adjust course delivery, activities, and content based on student aggregate data

▸ CREST-REU▸ Summer 2011

▸ NSF BIO REU 2011 SALG▸ NSF BIO REU 2011 SALG

Background of URSSA BIO REU SALG

▸ URSSA▸ URSSA▸ Undergraduate Research Student Self Assessment survey

▸ Self report instrument used for undergraduate research program administrators for program-level evaluation of student outcomes p g(Hunter et al 2009)▸ Well suited for REU programs

▸ NSF Supported ▸ CHE 0548488▸ CHE 0548488

▸ Weston et al 2011▸ BIO REU community used a modified URSA SALG to pilot▸ BIO REU community used a modified URSA SALG to pilot

assessment in summer 2010▸ Groups

▸ 232 students▸ 28 NSF funded BIO REU sites▸ 23 Institutions

▸ Supported by NSF▸ DBI 1052683 (Weston and Laursen)▸ DBI 1052683 (Weston and Laursen)▸ DBI 1052724 (Branchaw)▸ DBI 1052667 (Soto)

Components of BIO REU URSSA SALG (135 questions; 45-60 minutes)

1. Thinking and Working Like a Scientist▸ 8 questions

6. Research Program▸ 12 questions

7 T i i S i▸ 8 questions2. Personal Gains Related

to Research Work

7. Training Sessions▸ 6 questions

8 Background andto Research Work▸ 9 questions

3. Gains in Skills

8. Background and Motivation▸ 20 questions3 Ga s S s

▸ 13 questions4. Overall Experience and

20 questions9. Graduate School and

Career ActivitiespChanges in Attitudes▸ 8 questions

▸ 15 questions10.Demographics

5. Research Experience▸ 33 questions

▸ 10 questions

How to Use the BIO REU URSSA SALG

▸ Web site▸ Web site▸ www.salgsite.org

▸ Register as a new user (if don’t have an account)

▸ Click on “Returning User login”▸ Log in▸ Log in

▸ Your SALG home page


▸ Click on “Create a new instrument using the wizard to guide me”using the wizard to guide me

▸ In the “Wizard: Course description information” dialog window▸ Fill in information and click “Next”▸ Fill in information and click “Next”

▸ In the “Wizard: Validating Student Identities” dialog window

Ch ith “A th ti t ID”▸ Choose either “Authenticate ID” (requires emails of participants) or “Open Enrollment” (set a password so only your participants can takeso only your participants can take survey”

▸ Click next▸ In the “Wizard: Sharing your▸ In the Wizard: Sharing your

instrument” dialog window▸ Choose “Public, Anonymous, or

Private”Private▸ Click next


▸ In the “Wizard: Choose Baseline or SALG” dialog window▸ Choose “SALG”

▸ In the “Wizard: Choose SALG Source” dialog window▸ Choose “Reuse someone else’s instrument”

▸ In the “Wizard: Pick Public SALG to Reuse” dialog window▸ In the search box, type in “BIO REU

URSSA”URSSA▸ Look for one that has the following info

▸ Course: BIO REU URSSA▸ Semester: Summer Fall 2011▸ Description: Official BIO REU▸ Description: Official BIO REU

Survey▸ Instructor: Tim Weston

▸ Click on “Preview”▸ Click on “Next”▸ Click on Next

▸ In the “Wizard: choose Copy Options” dialog window▸ Choose “No, skip this step”


▸ In the “Wizard: Edit Questions” dialog windowg▸ You can edit any of the questions

or click on “Next”▸ In the “Wizard: Set Start and End

Dates” dialog windowDates dialog window▸ Choose your start and end dates

▸ Can do this later▸ Click nextClick next

▸ In the “Wizard: Announce to Students” dialog window▸ Choose an announcement ▸ Click “Done”

▸ In the Wizard: Completed” dialog window

Cli k “D ”▸ Click “Done”▸ Takes you back to your SALG home

page ▸ Should be able to see your▸ Should be able to see your

instrument


▸ Takes you back to your SALG y yhome page ▸ Should be able to see your

instrument under “Yourinstrument under Your Instruments:”

▸ To view and make changes to your instrumentyour instrument▸ Click on instrument number

in the “Open Control Panel column”column”


▸ Make Changes under the “Instrument Control Panel”

1. View or edit course name, description and/or other information about your course.

2 View or edit the student list (Use2. View or edit the student list. (Use this option to see who has completed the survey.)

3. Copy questions from another instrument into your active i t tinstrument.

4. Edit the questions5. Edit the start and end dates for your

survey.6 Announce the survey to students6. Announce the survey to students.7. Analyze and/or download the results

of your survey.▸ Reuse/adapt this instrument for a

new survey. (This instrument will not be affected.)

▸ View the ancestry of your instrument. (Shows the sources your instrument came from.)

▸ Delete this instrument and all▸ Delete this instrument and all associated data.

Data Output and Comparison Data

R G i▸ Report Generation▸ Online▸ Exports to Excelp

▸ Comparison DataA t D t f Bi l REU P▸ Aggregate Data for Biology REU Programs▸ www.bioreu.org

▸ Click on “Resources” tab on topp▸ Click on “Assessment and Evaluation”

▸ Analysis of all Biology REU program SALG data▸ GEO REU community can use as a comparison▸ GEO REU community can use as a comparison▸ However, your data will not be part of the BIO REU

aggregateM b i k i h URSSA d▸ May be an opportunity to work with URSSA to do something similar in GEO REU community

Additional Items

A di I▸ Appendix I▸ References

▸ Appendix IIAppendix II▸ Components of BIO REU URSSA SALG

Appendix IReferences

Reference

Hunter, A.B., T.J. Weston, S.L. Laursen, and H. Thiry 2009. URSSA: Evaluation student gains from undergraduateURSSA: Evaluation student gains from undergraduate research in science education. Council on Undergraduate Research Quarterly. 29(3):15-19

S E Wi D H t A & D ffi d S M (MSeymour, E., Wiese, D., Hunter, A. & Daffinrud, S.M. (Marzodel 2000). Creating a Better Mousetrap: On-line Student Assessment of their Learning Gains. Paper presentation at the National Meeting of the American Chemicalat the National Meeting of the American Chemical Society, San Francisco, CA.

Weston, T., J. Branchaw, J. Soto and S. Laursen 2011. 2010 BIO REU URSSA Pil t A t R t R t tBIO REU URSSA Pilot Assessment Report. Report to the National Science Foundation. 14 pages (www.bioreu.org; Resources tab; Assessment and Evaluation )Evaluation )

Anonymous. 2011. 2011BioREU_URRSSA_Instructions. 1 page (www.bioreu.org; Resources tab; Assessment and Evaluation )Evaluation )

Appendix IIComponents of BIO REU SALG

Components of BIO REU URSSA SALGp

Components of BIO REU URSSA SALG

program evaluation: one approachprogram evaluation: one ... · assessment of their learning gains....

Documents