Evaluating “By Appointment” Programs

Pamela J. Way, Ph.D.

“By appointment” programs Are designed to provide

individualized, one-on-one interactions

Are usually one hour in length

May be peer to peer

Before the evaluation….

Write mission statements for your agency and your programsThese statements define

anticipated, broadly-stated goals and outcomes

Inform the daily “doing business” process

Inform the evaluation path

Building your assessment How do you decide what to

evaluate?Choose the program

Choose one of the program’s components

Choose your target population

What do you want to know? Customer usage/satisfaction

This is baseline information; start here before designing a more complex evaluation

Learning outcomes – tutors?

Learning outcomes – tutees?

A word about definitions

Part of deciding what you want to know is knowing what – exactly – that is!

What’s your definition of “learning outcomes?”

Work in your groups to come up with some possibilities….

Assessing resources

Time

Money

Space/equipment

Human resources

What kinds of data are there? Quantitative

Qualitative

“Mixed” methods

Doing the legwork

Review current literature in the field

Find, evaluate, and select from existing instruments

Write your own questions

The IRB

An “IRB” – Institutional Review Board – review is required of all projects involving human participants

Minimize the risk to human subjects Ensure fully informed consent Promote equity in human research

When you’re ready to roll… Pilot the survey

Go “live”

Review incoming data

Data analysis

Descriptive

T-tests

ANOVA

Correlation/Regression

Categorical analyses

Going public

Although an evaluation effort will serve you and your program well, publishing or otherwise disseminating your results (conferences, etc.) will lend credibility to your program and your institution.

References – quantitative statistics

Statistics (Hays). A very dense but very complete textbook on designing studies as well as analyzing data.

Statistics (Freedman, Pisani, & Purves). Not as dense or complete as Hayes, but somewhat easier to navigate.

PDQ Statistics (Norman & Streiner). Down and dirty stats. Great for beginners.

Mixed Methodology: Combining Qualitative and Quantitative Approaches (Tashakkori & Teddlie).

References – qualitative statistics

Collecting and Interpreting Qualitative Materials (Denzin & Lincoln).

Survey Research Methods (Babbie). This is often considered the qualitative research “Bible.”

References - instruments

Developing and Using Tests Effectively (Jacobs & Chase). More geared to faculty, but many useful suggestions; easy to read and process

Psychological Testing: Design, Analysis, and Use (Friedenberg). Includes sources of information about tests.

Mental Measurements Yearbook. Full text info and reviews of many standardized tests in the educational field.

References - other

Publication Manual of the American Psychological Association. Essential if

you want to publish your results.On-line data bases: Academic Search

Premier, PsycARTICLES, PsycINFO. Google Scholar is also quite good.

SPSS – reasonably priced quantitative analysis software. Relatively easy to use.

Discussion Board

Please go to the Discussion board and respond to the prompt you will find for this module.

Thank you.