using measurable outcomes to evaluate tutor programs jan norton, presenter
TRANSCRIPT
![Page 1: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/1.jpg)
Using Measurable Outcomes to Evaluate Tutor Programs
Jan Norton, Presenter
![Page 2: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/2.jpg)
IntroductionJan NortonEmail: [email protected]: (319) 499-1372Master’s in Educational Research& Psychology
25+ years as director of multi-discipline tutoring centers with CRLA certifications
Former reviewer for CRLA tutor training certification; current NADE certification reviewer for tutoring, SI, and developmental education programs
NCLCA Lifetime Learning Center Leadership certification
![Page 3: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/3.jpg)
Webinar ContentsTerminologyThe Black BoxTimetablesData/MeasurementData/Measurement CautionsProgram Outcomes & ExamplesStudent Outcomes & ExamplesClosing
![Page 4: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/4.jpg)
TerminologyVisionsMissionsValuesGoalsObjectivesOutcomesProgram vs. Student
OutcomesLearning Outcomes
![Page 5: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/5.jpg)
The Black Box
Something
Happens
![Page 6: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/6.jpg)
The Black Box
Something
Happens
What is going on in there? How do we measure it?
![Page 7: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/7.jpg)
TimetablesScope of assessment
◦≥ 6 years◦2 years◦1 year◦1 semester◦1 course◦1 assignment◦1 tutoring session
![Page 8: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/8.jpg)
TimetablesPlanning for Assessment
◦Continual processes◦Multiple times per
semester◦Once or twice per
semester◦Two or three times per year◦Once per year◦Every two years◦Every 4 years
![Page 9: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/9.jpg)
Data/MeasurementWhy Measure
What to measure◦Before (Benchmarks)◦During◦After (Outcomes)
![Page 10: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/10.jpg)
Data/MeasurementHow to measure
◦Program-generated data◦Course-specific data◦Campus data◦Campus surveys◦Standardized tools◦Locally developed tools
![Page 11: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/11.jpg)
Data/MeasurementMore about locally developed
tools◦Surveys◦Likert-type scales◦Explicated scales◦Behavioral rubrics◦Self assessment by student◦Tutor assessment◦Observer assessment
![Page 12: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/12.jpg)
Data/Measurement Cautions
Direct vs. Indirect MeasuresData access and consistencyRealistic limitationsCooperationThe Baseline ChallengeTime & money
(and staff and comfort and ……)
![Page 13: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/13.jpg)
Chat Time for Questions & Contributions
Finished the foundation: what we can and can’t measure, when & how to measure, and what data we will gain
Moving on to examples of program-level outcomes
![Page 14: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/14.jpg)
Program OutcomesPossible Areas of Assessment
Focus◦Establishing baselines◦Usage◦Satisfaction or quality◦Level/variety of services◦Staff development◦Reaching certification or meeting
standards
![Page 15: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/15.jpg)
Program Outcomes ExamplesOutcome Statement
◦As a result of the new marketing strategy, the number of students attending group tutoring sessions will increase by 10% in Spring 2013.
Timetable◦One semester
Starting Measure (Benchmark)◦Average of past attendance
Ending Measure ◦Workshop attendance records
![Page 16: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/16.jpg)
Program Outcomes ExamplesOutcome Statement
◦During 2012-13, the number of tutors who earn CRLA Level I certification will increase by 5%.
Timetable◦One academic year
Starting Measure (Benchmark)◦Number of tutors achieving Level I in
previous year(s)Ending Measure
◦Number of tutors achieving Level I in 2012-13
![Page 17: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/17.jpg)
Program Outcomes ExamplesOutcome Statement
◦Outside reviewers using the NADE self-study process will score the tutoring program at 85% or above on all sections.
Timetable◦One year
Starting Measure (Benchmark)◦ Internal review & self-study
Ending Measure◦External review
![Page 18: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/18.jpg)
Program Outcomes ExamplesOutcome Statement
◦As a result of tutor training sessions, tutors will demonstrate an improved understanding of Bloom’s Taxonomy.
Timetable◦One semester or year, depending on
training scheduleStarting Measure (Benchmark)
◦Pre-testEnding Measure
◦Post-test
![Page 19: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/19.jpg)
Chat Time for Questions & Contributions
Finished program-level outcomes
Moving on to examples of student outcomes
![Page 20: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/20.jpg)
Student Outcomes: MeasuresQualitative
◦Interviews & focus groups◦Opinion surveys◦Observations or self reports
Quantitative◦Assignment & course grades◦Instrument scores◦Observation rubrics
![Page 21: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/21.jpg)
Student Outcomes: TypesAffective
◦Build confidence◦Improve attitude◦Reduce anxiety
Cognitive◦Mastery of material◦Conscious strategies use
![Page 22: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/22.jpg)
Student Outcomes ExamplesOutcome Statement
◦After being tutored for at least one hour, clients will be more confident in their ability to accomplish the academic task they focused on in tutoring.
Timetable◦One tutoring session
Starting Measure (Benchmark)◦Client self-report
Ending Measure ◦Client self report
![Page 23: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/23.jpg)
Student Outcomes ExamplesOutcome Statement
◦ Students who work with the study skills tutors will demonstrate an increased ability to use memorization strategies.
Timetable◦One tutoring session
Starting Measure (Benchmark)◦Tutor or observer checklist
Ending Measure◦Tutor or observer checklist
![Page 24: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/24.jpg)
Student Outcomes ExamplesOutcome Statement
◦75% of the students who work with a reading tutor for at least two sessions will score at least 75% on the main ideas reading test.
Timetable◦Within three weeks
Starting Measure (Benchmark)◦Reading quiz at start of tutoring
Ending Measure◦Chapter reading test in class
![Page 25: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/25.jpg)
Student Outcomes ExamplesOutcome Statement
◦Students who attend at least 10 group tutoring sessions for Algebra Review 090 will earn an average grade of B or above in the course.
Timetable◦One semester, length of course
Starting Measure (Benchmark)◦Placement score into course
Ending Measure◦Course grade
![Page 26: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/26.jpg)
Student Outcomes ExamplesOutcome Statement
◦After attending mandatory tutoring for all 5 of their developmental education courses, students will be retained until the start of their junior year at a rate at least 25% higher than developmental students for whom tutoring was not required.
Timetable◦Two years
Starting Measure (Benchmark)◦Developmental course placement
Ending Measure◦Enrollment level
![Page 27: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/27.jpg)
Student Outcomes ExamplesOutcome Statement
◦Students attending SI for World Religions will “[apply] previously understood information, concepts, and experiences to a new situation or setting.” (CAS)
Timetable◦Within one semester
Starting Measure (Benchmark)◦Pre-survey
Ending Measure ◦Post-survey
![Page 28: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/28.jpg)
ClosingCheck with your institutional
research or assessment officeReview early drafts of
outcomes statements with your supervisor
Pre-test any locally created measurement tool
Be prepared for outcomes that are not met or wildly exceeded
![Page 29: Using Measurable Outcomes to Evaluate Tutor Programs Jan Norton, Presenter](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649d895503460f94a6f938/html5/thumbnails/29.jpg)
Closing
Remember the black box: we do not own or control the box, and our ability to see into it is very limited
Questions & Contributions?