designing and conducting an evaluation of your project ...€¦ · designing your evaluation 1....
TRANSCRIPT
March 6, 2014
Clemencia Cosentino (Mathematica Policy Research)
Catherine Mobley (Clemson University)
Catherine Brawner (Research Triangle Educational Consultants)
Matthew Ohland and Russell Long (Purdue University)
2014 Annual Grantees’ Meeting
National Science Foundation
Science, Technology, Engineering and Math Talent Expansion Program (STEP)
Research Triangle
Educational Consultants
Designing and Conducting an Evaluation
of Your Project that Meets NSF Expectations
Part 1: Describe the steps to design a project
evaluation (40 minutes)
Part 2: Answer questions (10 minutes)
Part 3: Design a project evaluation (small
group activity, 40 minutes)
Workshop Objectives
2
Research Triangle
Educational Consultants
Why Evaluate?
Questions to Ask Before You Start Designing Your
Evaluation
What to Consider In Designing Your Evaluation
Steps to Design Your Evaluation
An Example
Part 1: Topics to be Covered
3
Research Triangle
Educational Consultants
Mathematica® is a registered trademark of Mathematica Policy Research.
Why Evaluate?
To comply with funder requirements
– NSF, Your Institution, and other funders
To document project outcomes and impacts
– To use in other proposals, to get buy-in for reforms
at your institution, to justify expenditures, to make a
case for additional resources
To improve practice
– In your courses, department, institution, or wider
community
To contribute to knowledge
– In your field through publications and other
dissemination activities
4
Research Triangle
Educational Consultants
Mathematica® is a registered trademark of Mathematica Policy Research.
What to Ask Yourself Before Starting
What is the purpose of your evaluation?
Who is your audience?
What is your project trying to accomplish? How?
Do you have questions you want to answer?
What is your evaluation budget?
How much time do you have?
Do you have access to staff with the right skills?
5
Research Triangle
Educational Consultants
What to Consider in Designing Your Evaluation?
Focus
– Implementation (what was done and how?)
– Outputs (who was served?)
– Outcomes (what was produced?)
– Impacts (what was produced by the project?)
Type
– Formative (how can we improve?)
– Summative (how did we do?)
Method
– Qualitative or quantitative
– Multiple (using different methods)
– Mixed (integrating qualitative and quantitative methods)
6
Research Triangle
Educational Consultants
Alignment
– Evaluation questions and research methods
Rigor in quantitative research
– Experimental Designs (RCTs)
– Quasi-experimental Designs (matched comparison
groups)
– Other Designs (such as pre-post and descriptive studies)
Rigor in qualitative research
– Appropriate qualitative methods
– Sample selection
– Triangulation
What to Consider in Designing Your Evaluation?
7
Research Triangle
Educational Consultants
Designing Your Evaluation
1. Create a logic model
2. Draft evaluation questions
3. Select an appropriate design
4. Identify and recruit staff needed
5. Plan the budget, schedule, products
6. Establish a mechanism to monitor progress on
the evaluation
8
Research Triangle
Educational Consultants
An Example
9
Research Triangle
Educational Consultants
The problem
– Engineering statics course curriculum is dated; it does not
include active learning or use new technology and software
to engage students and prepare them for high-tech jobs
– Faculty suspect that this is affecting student performance
and their employment prospects
What is your project trying to accomplish?
– Increase student learning
– Better prepare students for future jobs
My New NSF Undergraduate Project
10
Research Triangle
Educational Consultants
How?
– Revise the curriculum in the engineering statics course to
incorporate active learning activities as well as the use of
new technologies and software
– Provide faculty with professional development on new
pedagogical approach and new technology and software
– Purchase new software, hardware, and other materials
needed
What is the purpose of your evaluation?
– To document whether the revised pedagogical approach
adopted was implemented in the engineering statics course
– If adopted, does it have a positive impact on student
learning and job prospects
– If it has a positive impact, to use findings to lobby for
similar changes in other courses
My New NSF Undergraduate Project (continued)
11
Research Triangle
Educational Consultants
What to Ask Yourself Before Starting
Who is your audience?
– Faculty and institutional administrators
– Foundations and alumni that may support future
expansion to other courses
– Other institutions that may want to adopt revisions
What will you do in your project?
– The curriculum will be revised by a team of faculty with
help from an expert consultant to adopt more active
learning (such as hands-on activities)
– New software and hardware will be purchased (and other
material as needed) for use in and out of the classroom
– Faculty will be trained to adopt use of active learning and
new technology
12
Research Triangle
Educational Consultants
Designing Your Evaluation
1. Create a logic model
2. Draft evaluation questions
3. Select an appropriate design
4. Identify and recruit staff needed
5. Plan the budget, schedule, products
6. Establish a mechanism to monitor progress
13
Research Triangle
Educational Consultants
Problem
Engineering statics curriculum is dated; does not engage students and prepare them for high-tech jobs
Inputs (resources
and partners)
- Purchase new software and hardware
- Partner with a technology company and an education consultant
Outputs (activities
and products)
- Provide faculty PD
- Incorporate active learning activities and new technologies into the curriculum
Outcomes(progress
toward goals)
- Increased student
participation
- Students report greater satisfaction with course
Impact (cause
and effect)
- Students earn better grades in
subsequent course
- Students who take
course have better job placement
rates
Step 1: Create A Logic Model
14
Research Triangle
Educational Consultants
Problem
Engineering statics curriculum is dated; does not engage students and prepare them for high-tech jobs
Inputs (resources
and partners)
- Purchase new software and hardware
- Partner with a technology company and an education consultant
Outputs (activities
and products)
- Provide faculty PD
- Incorporate active learning activities and new technologies into the curriculum
Outcomes(progress
toward goals)
- Increased student
participation
- Students report greater satisfaction with course
Impact (cause
and effect)
- Students earn better grades in
subsequent course
- Students who take
course have better job placement
rates
Step 1: Create A Logic Model
Implementation Study Outcomes
Study Impact
Study
Accountability Effectiveness
15
Research Triangle
Educational Consultants
Designing Your Evaluation
1. Create a logic model
2. Draft evaluation questions
3. Select an appropriate design
4. Identify and recruit staff needed
5. Plan the budget, schedule, products
6. Establish a mechanism to monitor progress
16
Research Triangle
Educational Consultants
Evaluation Questions?
Do faculty adopt the new pedagogical approach?
Does the revised approach have a positive impact
on student learning in the statics course?
Are students who take the revised course more
successful in the job market?
17
Research Triangle
Educational Consultants
Designing Your Evaluation
1. Create a logic model
2. Draft evaluation questions
3. Select an appropriate design
4. Identify and recruit staff needed
5. Plan the budget, schedule, products
6. Establish a mechanism to monitor progress
18
Research Triangle
Educational Consultants
Evaluation Designs?
Do faculty adopt the new pedagogical approach?
• Implementation study
Does the revised approach have a positive impact
on student learning in the statics course?
• Impact study (experimental or quasi-experimental)
Are students who take the revised course more
successful in the job market?
• Impact study (experimental or quasi-experimental)
19
Research Triangle
Educational Consultants
Designing Your Evaluation
1. Create a logic model
2. Draft evaluation questions
3. Select an appropriate design
4. Identify and recruit staff needed
5. Plan the budget, schedule, products
6. Establish a mechanism to monitor progress
20
Research Triangle
Educational Consultants
What to Ask Yourself Before Starting
Do you have staff with the evaluation expertise?
– Sociology faculty do implementation studies
– Statistics faculty and graduate students can do
quantitative analysis
– Need expertise in evaluation design to set up the study,
provide limited technical assistance, and review the
results/report to provide feedback
What is your evaluation budget?
– $50,000 per year (10 percent of the project’s budget)
How much time do you have?
– 3 years
21
Research Triangle
Educational Consultants
Can you tackle your evaluation questions?
Do faculty adopt the new pedagogical approach?
• Implementation study
• You conclude that you can do this in house with help from
sociology faculty
Does the revised approach have an impact on student learning in
the statics course?
• Impact study (experimental or quasi-experimental)
• You decide to revise some but not all sections of the statics
course (and assign faculty that received PD to the revised
sections). You will randomly assign students to the revised
(treatment) or not revised (control) sections (to be offered at
the same time)
Are students who take the revised course more successful in the
job market?
• Impact study (experimental or quasi-experimental)
• Cannot study this given that the project ends in 3 years
22
Research Triangle
Educational Consultants
Q & A
23
Research Triangle
Educational Consultants
Small Group Activity
Select a topic for your project
– Retention (of students)
– Partnerships (between 2- and 4-year institutions)
– Outreach (to K-12)
Join a table that is working on that topic
Design your project evaluation
– Use the handouts to guide your team’s work
Report back
– 2-minute presentation from each team
24
Research Triangle
Educational Consultants
Resources on STEP Central
Go to:
http://stepcentral.net/groups/posts/942/
25
Research Triangle
Educational Consultants
Thank You
Clemencia Cosentino
Catherine Mobley
Catherine Brawner
Matthew Ohland
Russell Long
26
Research Triangle
Educational Consultants