TAH Project EvaluationData Collection

Sun Associates

Jeff Sun jsun@sun-associates.com www.sun-associates.com/tah

We’ve evaluated 11 TAH projects over the past 4 years

Program evaluators for TAH and non-TAH projects Instructional Technology STEM Creating evaluation plans as part of strategic

planning initiatives

Data in Isolation is No Data at All Data collection must be rooted in performance

indicators that are in-turn aligned with project goals

What are we going to do? How do we know when we’ve done it/are doing it? What data supports this conclusion?

Data needs are defined by the indicators An example…

Question 1 Our first question is largely quantitative

Question 2 It turns out that Question 2 is much more

qualitative But this in fact address more of the project’s

goals

Data collection is linked to indicators Projects need quantitative as well as

qualitative data to respond to their indicators Quantitative data is pretty straight-forward

Tests (teacher and student) Counts (enrollments, participation, etc.)

Qualitative data requires more work and more specialized skills to collect and analyze

Qualitative Methods Rubrics and Checklists

Essentially like “grading criteria” Lists/examples of behaviors that constitute

acceptable performance Historical Thinking Skill benchmarks Other “effective teaching” rubrics – created by

project staff Rubrics for lesson plans/participant work products

Bottom line is that work and/or performance is assessed against these rubrics…and the results are data that go toward assessment of the project indicators

Classroom observations Basic data collection – “What Actually Happened?” Teacher de-brief/interview Recording observations later on a indicator-aligned

template Meeting observations

We consider staff meetings, board meetings, planning meetings, etc. as legitimate data collection events

As evaluators, we need to observe everything that goes into operating this project

Focus groups Engaging participants in a focused discussion

around questions that are aligned with the project indicators

Seeking normative statements as well as evidence of a disparity of opinion

But we’re looking for opinions Occur several times throughout the project year…

at minimum at the beginning and end of each participant cohort

Selection issues (who selects the participants) need to be considered

Can also be conceptualized as 1-1 interviews

Surveys The basic “workshop evaluation form” Given our typical sample size (20 or so

participants per session) we look for open-ended responses versus numerically scaled responses.

Online surveys allow for easy use of text responses

We also use surveys for background needs assessment – which can be used to substantiate later claims that the project has “met participants’ needs”.

Control Groups Not all of our evaluations make use of control

group methodology This is not appropriate – or practical – for many

project evaluations (and we would argue the former is true for most TAH projects)

When necessary, we have operated control groups for quantitative data (tests of teacher/student knowledge)

Control groups can also generate qualitative data, but extra work and expense is of little practical benefit in most TAH projects

Pulling it All Together Quantitative data provides quick facts that can be

input into the ED 524B form … But these “facts” can be easily misinterpreted or

misleading Quantitative data alone doesn’t provide any context for

understanding what the numbers may mean Qualitative data provides context and deep

description of project actions and outcomes Qualitative data provides considerable text for

written evaluation reports…and reflective, formative, discussion of project performance in meeting its indicators Which we see as the primary value of project evaluation

Jeff Sun jsun@sun-associates.com www.sun-associates.com/tah