w w w. c e s. c l e m s o n. e d u / g e / planning engineering education research facilitator:...
TRANSCRIPT
W W W . C E S . C L E M S O N . E D U / G E /
Planning Engineering Education Research
Facilitator: Matthew W. Ohland
W W W . C E S . C L E M S O N . E D U / G E /
Workshop Objective
Describe steps in the planning and assessment of engineering education research
Give examples of each step
Small-group discussion to implement the step using a common example
W W W . C E S . C L E M S O N . E D U / G E /
Workshop Agenda
4:00-4:05 Introduction4:05-4:20 Defining the Purpose of the Evaluation
(What questions are we asking?)4:20-4:40 Clarify Project Objectives (What do we expect?)4:40-5:00 Create a Model of Change
(How will our efforts lead to the objectives?)5:00-5:15 Select Criteria and Indicators
(What data do we need to measure our progress?)5:15-5:25 Identify Data Sources (Where will we get that data?)5:25-5:45 Design Evaluation Research
(How will we analyze that data?)5:45-5:50 Monitor and Evaluate (What actually happened?)
Use and Report Results (Who needs to know?)5:50-6:00 Wrap-up and evaluations
W W W . C E S . C L E M S O N . E D U / G E /
Step 1: Purpose of the Evaluation
FormativeSummative
EfficacyEffectivenessCost
Examples• Is there a benefit to doing
this?• Can we improve it?• Does that benefit lead to
better retention / grades?• Do the benefits justify the
program cost?• Can we achieve the same
benefits on a larger scale?
W W W . C E S . C L E M S O N . E D U / G E /
Step 1 Action Steps
Decide purpose(s)
Primary questions? Order?
Primary audience(s)?
How will the results be used?
W W W . C E S . C L E M S O N . E D U / G E /
Step 1: Purpose of the Evaluation
Do we have a learning community?
Is it leading to better course retention / grades?
Can we achieve the same benefits cost-effectively on a larger scale?
W W W . C E S . C L E M S O N . E D U / G E /
Step 2: Project Objectives
Impact
Outcome
Process
Example• Improve graduation rate
by some %
• Increase course passing rate by some %
• Student or faculty opinions / behaviors change
• Reserve any facilities needed
W W W . C E S . C L E M S O N . E D U / G E /
Step 2 Action Steps
Write down project objectives• Impact• Outcome• Process
W W W . C E S . C L E M S O N . E D U / G E /
Step 2: Project Objectives
Increase graduation rate from 50% to 75%
Increase first-time Calculus passing rate from 50% to 70%
Students have positive opinion of group work
Secure dormitory and classroom space
W W W . C E S . C L E M S O N . E D U / G E /
Step 3: Create a Model of Change
Identify assumptions you can assessChoose relationships to test based on• Resources• Where you anticipate
problems• Where you have
control / can make improvements
Link what you do to what you expect to happenExample• Common residence /
classes • in between, we need a
sound theory of why this will happen
more will graduate
W W W . C E S . C L E M S O N . E D U / G E /
Step 3 Action Steps
Model of change• as specific / complete as needed
Review model assumptions
Use criteria to prioritize• Resources• Relevance• Control
W W W . C E S . C L E M S O N . E D U / G E /
Step 3: Create a Model of Change
Common residence / classes affiliation
Affiliation support
Support performance
Performance future performance
Future performance more will graduate
W W W . C E S . C L E M S O N . E D U / G E /
Step 4: Criteria and Indicators
Validity
Reliability
Sensitivity
Ease of interpretation
Usefulness
Example: define• Graduation rate
• Course pass rate
• Affiliation
• Life-long learning
• Enrollment
W W W . C E S . C L E M S O N . E D U / G E /
Step 4 Action Steps
Define a set of indicators and criteria • Impact• Outcome • Process
W W W . C E S . C L E M S O N . E D U / G E /
Step 4: Criteria and Indicators
Graduation rate• # graduated / # in original cohort
Calculus pass rate• # A, B, C / # in original cohort• D’s are no good – must be retaken
Affiliation • # in study group at end of sophomore year /
# students
Number of students enrolled in program
W W W . C E S . C L E M S O N . E D U / G E /
Step 5: Data Sources
ExamsSurveysObservationsStudent recordsSAT scores
Frequency• Resources• Guidance• Change
Example• Institutional Research
office (annual)• Course records (each
semester)• Registrar• Admissions• Assessment office• Process - monitor until
achieved
W W W . C E S . C L E M S O N . E D U / G E /
Step 5 Action Steps
Define data sources
Define frequency of measurement
W W W . C E S . C L E M S O N . E D U / G E /
Step 5: Data Sources
Grad / retention rate – Inst. Res. – AnnualCalculus performance – Math dept. – includes course grades and common exam results –end of semesterDorm space allocated – Housing – monitor until achievedSurvey of program participantsObserver evaluations of class interactionSAT scores—Admissions
W W W . C E S . C L E M S O N . E D U / G E /
Step 6: Design Evaluation Research
Selection
Mortality
Placebo is not an issue
Qualitative• Interviews
• Focus groups
• Systematic observation
Quantitative• Non-experimental
- Posttest only
- Pretest-Posttest
• Quasi-experimental- Time Series
- Nonequivalent control
• Experimental- Pretest-Posttest Control
- Multiple Intervention
W W W . C E S . C L E M S O N . E D U / G E /
Step 6 Action Steps
Design evaluation research studies for key questions
W W W . C E S . C L E M S O N . E D U / G E /
Step 6: Example Designs
Example-Interviews• Student expectations—
invitation process modified
• Resource utilization—approaches to motivate attendance
Example-Efficacy• Non-equivalent control
group
• Test for selection bias using baseline measures
• Matched pairs / groups
• Calculus grades and overall GPR
W W W . C E S . C L E M S O N . E D U / G E /
Step 7: Monitor and Evaluate
Establish project information system
Budget for evaluation
Evaluation meetings
Review and revise evaluation plan
Carry out studies
W W W . C E S . C L E M S O N . E D U / G E /
Step 8: Use and Report Results
Report results – to everyone
Use results to make improvements
W W W . C E S . C L E M S O N . E D U / G E /
Conclusions
Planning and assessment is essential
Start small if resources are limited
Develop a plan before starting
NSF and other agencies support well-designed educational research• CCLI—EMD / A&I, ASA, and other programs
Seek appropriate partners from education, psychology, sociology, statistics, etc.