Look! Our ‘Stuff’ is Making Look! Our ‘Stuff’ is Making an Impact:an Impact:

Using Meaningful Using Meaningful Assessment to Demonstrate Assessment to Demonstrate

the Effectiveness of the Effectiveness of Academic Support ServicesAcademic Support Services

Michael Anne GreerMichael Anne Greer

Director, Academic Success CenterDirector, Academic Success Center

Amanda NovakAmanda Novak

Director, Advising and New Student ServicesDirector, Advising and New Student Services

Texas Wesleyan UniversityTexas Wesleyan University

The Same Old Question:The Same Old Question:Why Assess?Why Assess?

We have toWe have to AccreditationAccreditation Reaffirm our rolesReaffirm our roles Demonstrate department’s Demonstrate department’s

contributions to institutional goalscontributions to institutional goals SEE IF OUR STUFF IS WORKINGSEE IF OUR STUFF IS WORKING

Who Should Be Assessing?Who Should Be Assessing?

Academic DepartmentsAcademic Departments• Experience creating, writing, and Experience creating, writing, and

evaluating learning outcomesevaluating learning outcomes Administrative DepartmentsAdministrative Departments

• Why it can be dauntingWhy it can be daunting• Which areas assess?Which areas assess?

Internal, within the departmentInternal, within the department Centralized Institutional Research Centralized Institutional Research

Let’s Look at Our ‘Stuff’Let’s Look at Our ‘Stuff’ Texas Wesleyan- brief statisticsTexas Wesleyan- brief statistics

• 3,204 students3,204 students• 1,794 undergraduates1,794 undergraduates• PrivatePrivate

Academic Success CenterAcademic Success Center• HistoryHistory• ServicesServices• RolesRoles

Advising & New Student Services Advising & New Student Services • HistoryHistory• ServicesServices• RolesRoles

Common PitfallsCommon Pitfalls

Avoiding assessmentAvoiding assessment• It’s not that bad…after 1 or 2 cycles, It’s not that bad…after 1 or 2 cycles,

much less daunting!much less daunting!• Don’t be afraid of the dataDon’t be afraid of the data• Much easier and more useful if kept up Much easier and more useful if kept up

regularlyregularly Staying on trackStaying on track

• Evaluation Grid• Advising Assessment Matrices

(Redmond & Babenchuk, 2011)(Redmond & Babenchuk, 2011)(Robbins, 2011)(Robbins, 2011)

Common PitfallsCommon Pitfalls

Alignment with UniversityAlignment with University• Crucial to demonstrate how your Crucial to demonstrate how your

department is contributing to department is contributing to institutional goals and missioninstitutional goals and mission

• Demonstrate needs (funding, staff, Demonstrate needs (funding, staff, resources)resources)

• Informed decisions for improvementInformed decisions for improvement Alignment with department missionAlignment with department mission

• Make sure goals support current missionMake sure goals support current mission

(Bresciani, 2011)(Bresciani, 2011)

Common PitfallsCommon Pitfalls Alignment with UniversityAlignment with University

• ““Our mission at Texas Wesleyan University is to develop Our mission at Texas Wesleyan University is to develop students to their full potential as individuals and as students to their full potential as individuals and as members of the world community.”members of the world community.”

• ““The Academic Success Center’s mission is to serve as a The Academic Success Center’s mission is to serve as a critical link between students and a fully successful critical link between students and a fully successful academic experience by providing personal support academic experience by providing personal support services that remove barriers to academic excellence services that remove barriers to academic excellence and assist in the development of critical thinking, and assist in the development of critical thinking, analytical reasoning, and problem solving skills.”analytical reasoning, and problem solving skills.”

• ““Academic advising at Texas Wesleyan University is a Academic advising at Texas Wesleyan University is a teaching and learning process dedicated to student teaching and learning process dedicated to student success. Consistent with the university mission, success. Consistent with the university mission, academic advising engages students in developing a academic advising engages students in developing a plan to realize their educational, career and life goals.” plan to realize their educational, career and life goals.”

(Bresciani, 2011)(Bresciani, 2011)

Common PitfallsCommon Pitfalls

Unreasonable goals and objectivesUnreasonable goals and objectives• Keep goals manageable and attainableKeep goals manageable and attainable• Be sure goals are relevant and Be sure goals are relevant and

meaningful to stakeholdersmeaningful to stakeholders• Goals should be broad and reflect what Goals should be broad and reflect what

your program would like to be, look your program would like to be, look like, dolike, do

(Bresciani, 2011)(Bresciani, 2011)

Common PitfallsCommon Pitfalls

Ineffective measuresIneffective measures• Are we measuring actual outcomes?Are we measuring actual outcomes?• Tracking visits does not measure outcomesTracking visits does not measure outcomes..

Lack of documentation/ trackingLack of documentation/ tracking• Are we asking for/ collecting/ pulling the Are we asking for/ collecting/ pulling the

data we need at the right time?data we need at the right time?• We’ve been doing all of these great things, We’ve been doing all of these great things,

but have no evidencebut have no evidence.. One survey is not assessment!One survey is not assessment!

Common PitfallsCommon Pitfalls

Invalid dataInvalid data• Does the data give us a valid measure Does the data give us a valid measure

of our outcomes?of our outcomes? Incorrect interpretation of dataIncorrect interpretation of data

• Increased GPA does not imply retentionIncreased GPA does not imply retention Ineffective use of data- not closing Ineffective use of data- not closing

the loop!the loop!

‘‘Backwards Thinking’Backwards Thinking’

Some struggle with how to think Some struggle with how to think about assessmentabout assessment

Let’s think backwards!Let’s think backwards! What are we doing?What are we doing?

• 85 students come in for writing tutoring85 students come in for writing tutoring• Held 12 academic skills workshopsHeld 12 academic skills workshops• 150 out of 200 freshmen register on the 150 out of 200 freshmen register on the

first day of early registration first day of early registration • Write 2 or 3 of your servicesWrite 2 or 3 of your services

‘‘Backwards Thinking’Backwards Thinking’

Why are we doing it?Why are we doing it?• ““Students need/ request it”- not specific Students need/ request it”- not specific

enoughenough• Need to demonstrate why spending the Need to demonstrate why spending the

time, effort, funds to support servicestime, effort, funds to support services

OutcomesOutcomes

What are we attempting to achieve?What are we attempting to achieve?• Outcomes- what the end user should Outcomes- what the end user should

know, have, be able to do as the result know, have, be able to do as the result of the serviceof the service

Must be specificMust be specific Must be measurableMust be measurable

(Bussell, 2011)(Bussell, 2011)

Writing OutcomesWriting Outcomes

““80% of undergraduates will utilize 80% of undergraduates will utilize tutoring services”- not an outcometutoring services”- not an outcome

““Students who utilize tutoring services Students who utilize tutoring services will succeed”- not specific enoughwill succeed”- not specific enough

““Students will benefit from workshops”- Students will benefit from workshops”- not measurablenot measurable

Students will receive adequate advising – Students will receive adequate advising – not easily measured, too broad not easily measured, too broad

Writing OutcomesWriting Outcomes

““Students who receive writing assistance Students who receive writing assistance will pass writing intensive courses with a will pass writing intensive courses with a C or better”C or better”

““Students who attend academic Students who attend academic workshops will gain effective note-taking workshops will gain effective note-taking methods”methods”

““Freshman Students will be able to Freshman Students will be able to calculate their GPA”calculate their GPA”

Write 2-3 service outcomesWrite 2-3 service outcomes

Measuring OutcomesMeasuring Outcomes

Is it working?Is it working? How do we know?How do we know?

• Measures- should assess actual Measures- should assess actual outcome, not another processoutcome, not another process

Direct Measures- assess actual performance Direct Measures- assess actual performance or productor product

Indirect Measures- measure perceptions or Indirect Measures- measure perceptions or opinionsopinions

MeasuresMeasures

Direct or Indirect?Direct or Indirect?• Practice test at end of math tutoring Practice test at end of math tutoring

sessionsession• Survey to workshop attendeesSurvey to workshop attendees• Focus group opinions of tutorsFocus group opinions of tutors• Demonstration exercise during workshopDemonstration exercise during workshop• Enrollment data during early registrationEnrollment data during early registration

Write 1 direct and 1 indirect for your Write 1 direct and 1 indirect for your outcomesoutcomes

Gathering the DataGathering the Data

What do the measures tell us?What do the measures tell us?• Data Collection ExamplesData Collection Examples

Survey responsesSurvey responses Focus group feedbackFocus group feedback Retention reportsRetention reports Grade reportsGrade reports GPAGPA Self Study Audits Self Study Audits Individual observation of students performing Individual observation of students performing

a task (especially useful in advising)a task (especially useful in advising)

Assessing the DataAssessing the Data

Are we doing well?Are we doing well?• Did we meet our goals?Did we meet our goals?• Do we need to increase/ decrease Do we need to increase/ decrease

goals?goals?• What are appropriate benchmarks?What are appropriate benchmarks?

CAS StandardsCAS Standards Professional association standardsProfessional association standards Peer Institutions Peer Institutions

Using the DataUsing the Data

Now what?Now what?• Closing the loopClosing the loop

What goals were met/ not met?What goals were met/ not met? Why or why not?Why or why not?

• Needs- resources, staff, fundingNeeds- resources, staff, funding How to improveHow to improve AffirmationsAffirmations Recommendations/ plans going forwardRecommendations/ plans going forward

ReferencesReferences Bresciani, M.J. (2011, June). Enhancing outcomes-based Bresciani, M.J. (2011, June). Enhancing outcomes-based

assessment for student affairs. assessment for student affairs. Webinar. Webinar. Lecture Lecture conducted from Innovative Educators, Thornton, CO.conducted from Innovative Educators, Thornton, CO.

Bussell, K.H. (2011, May). Strategic planning & outcomes Bussell, K.H. (2011, May). Strategic planning & outcomes assessment. assessment. Academic Student Support Services Council Academic Student Support Services Council retreat.retreat. Lecture conducted from Texas Wesleyan Lecture conducted from Texas Wesleyan University, Fort Worth, TX.University, Fort Worth, TX.

Redmond, J. & Babenchuk, I. (2011, September). Data-Redmond, J. & Babenchuk, I. (2011, September). Data-driven decision-making. driven decision-making. Learning centers: At the Learning centers: At the crossroads of student success. crossroads of student success. Lecture conducted from Lecture conducted from National College Learning Center Association 26National College Learning Center Association 26thth Annual Annual Conference, Indianapolis, IN.Conference, Indianapolis, IN.