1 assessment and evaluation diane m. bunce chemistry department the catholic university of america...
Post on 25-Dec-2015
225 Views
Preview:
TRANSCRIPT
11
Assessment and Assessment and EvaluationEvaluation
Diane M. BunceDiane M. BunceChemistry DepartmentChemistry Department
The Catholic University of AmericaThe Catholic University of AmericaWashington, DC 2064Washington, DC 2064
Bunce@cua.eduBunce@cua.edu
22
Project EvaluationProject EvaluationNSF 1997NSF 1997
What is it?What is it? Systematic investigation of the worth of a projectSystematic investigation of the worth of a project
• Answers the questions:Answers the questions: What have we accomplished?What have we accomplished? Are the project goals being met?Are the project goals being met? How effectively are the individual parts of the project How effectively are the individual parts of the project
working?working?
• Provides information to communicate to stakeholdersProvides information to communicate to stakeholders NSFNSF Administrators Administrators Faculty Faculty StudentsStudents Others not involved in the projectOthers not involved in the project
• Faculty peersFaculty peers• Other studentsOther students
General AudienceGeneral Audience
33
Role of Evaluation in ProjectRole of Evaluation in Project
Evaluation should be an integral partEvaluation should be an integral part• Address each goal of project.Address each goal of project.• Provide information to help improve project.Provide information to help improve project.• Offer necessary information to different groups Offer necessary information to different groups
of stakeholders.of stakeholders.• Need not be adversarial.Need not be adversarial.
Evaluation should include the criteria of proof Evaluation should include the criteria of proof accepted by each group of stakeholders.accepted by each group of stakeholders.
Funded at a level consistent with the work Funded at a level consistent with the work that needs to be accomplished.that needs to be accomplished.
44
Two Stages of EvaluationTwo Stages of Evaluation
Formative EvaluationFormative Evaluation• Opportunity to measure progress and Opportunity to measure progress and
suggest modifications suggest modifications duringduring the project the project Implementation EvaluationImplementation Evaluation
• Is the project being conducted as planned?Is the project being conducted as planned? Progress EvaluationProgress Evaluation
• Measures impact of project components at Measures impact of project components at various stages.various stages.
Summative EvaluationSummative Evaluation• Assesses quality and impact of the full Assesses quality and impact of the full
project project at the completion of project.at the completion of project.
55
StakeholdersStakeholders Who are the people who have an interest in the results of your Who are the people who have an interest in the results of your
project?project?• Funding AgencyFunding Agency
Judge the success of your individual project.Judge the success of your individual project. Judge the impact your project results have on the broader Judge the impact your project results have on the broader
goals of the program.goals of the program.• Administration of the participating institutionsAdministration of the participating institutions
Are the results compelling enough to make changes in the Are the results compelling enough to make changes in the staffing, scheduling or facilities of the institution?staffing, scheduling or facilities of the institution?
• FacultyFaculty Are the results compelling enough to change the way they Are the results compelling enough to change the way they
teach and the curricula materials they choose?teach and the curricula materials they choose?• StudentsStudents
Does the experience support/challenge their Does the experience support/challenge their academic/career choice?academic/career choice?
• Are more students attracted to science, mathematics, Are more students attracted to science, mathematics, engineering majors?engineering majors?
• Professional PeersProfessional Peers Will your project affect the way others teach?Will your project affect the way others teach?
66
Evaluation QuestionsEvaluation QuestionsQuestions must address the overall goals of Questions must address the overall goals of
the project.the project.
Goal: To provide authentic research Goal: To provide authentic research experience for beginning undergraduatesexperience for beginning undergraduates• Evaluation Question: Evaluation Question:
What constitutes authentic research?What constitutes authentic research? Do students experience authentic Do students experience authentic
research in this project as judged by research in this project as judged by science experts, faculty, science experts, faculty, administrators, and students administrators, and students themselves?themselves?
77
Evaluation QuestionsEvaluation Questions Goal: To increase the number of students who Goal: To increase the number of students who
choose to major in science, mathematics, choose to major in science, mathematics, engineering, and technology.engineering, and technology.• Evaluation Questions: Evaluation Questions:
How many students declare/change their How many students declare/change their majors from non-SMET to SMET?majors from non-SMET to SMET?
Does the % of students who successfully Does the % of students who successfully graduate with SMET majors increase as a graduate with SMET majors increase as a result of this project?result of this project?
Does the % of students who go onto SMET Does the % of students who go onto SMET careers increase as a result of the project?careers increase as a result of the project?
88
Evaluation QuestionsEvaluation Questions
Goal: To change the way introductory Goal: To change the way introductory chemistry is taught to students.chemistry is taught to students.• Evaluation QuestionEvaluation Question
How have faculty changed the How have faculty changed the process or curricula of these process or curricula of these courses as a result of this courses as a result of this project?project?
99
Measurement and Results of Measurement and Results of Evaluation Questions Evaluation Questions
What proof is needed to answer the questions What proof is needed to answer the questions asked?asked?• Quantitative: StatisticsQuantitative: Statistics
Significant Difference between groupsSignificant Difference between groups• Difference unlikely to occur by chanceDifference unlikely to occur by chance
Effect Size Effect Size • Change measured against standard deviationChange measured against standard deviation
• QualitativeQualitative Observation of specific behavior of groupsObservation of specific behavior of groups Perceptions of StakeholdersPerceptions of Stakeholders
• CombinationCombination Can results be Can results be appliedapplied to policy or program that to policy or program that
motivated the study?motivated the study? Will the study have direct implications for Will the study have direct implications for actionaction??
1010
Matching Evaluation Methodology Matching Evaluation Methodology to Evaluation Questionsto Evaluation Questions
What, How or Why something occurs?What, How or Why something occurs?• QualitativeQualitative
Does what occurs make a difference?Does what occurs make a difference?• QuantitativeQuantitative
How important is it?How important is it?• Mixed MethodsMixed Methods
What occurs and why?What occurs and why? What does it mean?What does it mean? How is it valued by stakeholders?How is it valued by stakeholders?
1111
Comparison of Qualitative and Comparison of Qualitative and Quantitative Evaluation DesignsQuantitative Evaluation Designs
QualitativeQualitative AssumptionsAssumptions
• Variables are complex, Variables are complex, interwoven and difficult to interwoven and difficult to measuremeasure
PurposePurpose• Interpretation of reality of Interpretation of reality of
situationsituation ApproachApproach
• Ends with hypothesisEnds with hypothesis• Researcher is the toolResearcher is the tool
QuantitativeQuantitative AssumptionsAssumptions
• Variables can be Variables can be identified and identified and relationships measuredrelationships measured
PurposePurpose• GeneralizabilityGeneralizability• PredictionPrediction
ApproachApproach• Starts with hypothesisStarts with hypothesis• Researcher manipulates Researcher manipulates
formal toolsformal tools
1212
Qualitative Evaluation DesignQualitative Evaluation Design Use when investigating “What”, “How”, or “Why” Use when investigating “What”, “How”, or “Why”
• Results can be used Results can be used To answer the question directly or To answer the question directly or To help develop a quantitative study.To help develop a quantitative study.
Types of Qualitative StudiesTypes of Qualitative Studies• Observer Study: observations of participants Observer Study: observations of participants
augmented by opportunities to talk with augmented by opportunities to talk with stakeholders. stakeholders.
• Case Study: In-depth analysis of a limited Case Study: In-depth analysis of a limited number of participants.number of participants.
1313
Analysis of Qualitative DataAnalysis of Qualitative Data (Miles and Huberman 1994)(Miles and Huberman 1994)
Data ReductionData Reduction• Process of selecting, focusing and transforming Process of selecting, focusing and transforming
datadata• Initial categorizations shaped by pre-Initial categorizations shaped by pre-
established evaluation questions.established evaluation questions. Data DisplayData Display
• Diagram, Chart or Matrix used to organize data Diagram, Chart or Matrix used to organize data for purpose of facilitating construction of for purpose of facilitating construction of conclusions.conclusions.
Conclusion Drawing and VerificationConclusion Drawing and Verification• Consideration of what data mean.Consideration of what data mean.• Revisiting data to verify emerging conclusions.Revisiting data to verify emerging conclusions.
1414
Quantitative Evaluation DesignQuantitative Evaluation Design
Quantitative studies report on aggregate Quantitative studies report on aggregate (average of students in sample)(average of students in sample)• (Qualitative studies report on a limited number (Qualitative studies report on a limited number
of specific cases)of specific cases) Types of Quantitative DesignsTypes of Quantitative Designs
• Pure experimental design: assume Pure experimental design: assume randomization of subjects to treatment and randomization of subjects to treatment and controlcontrol
• Quasi experimental design: randomization not Quasi experimental design: randomization not assumedassumed
1515
Pure Experimental DesignPure Experimental Design(Abraham & Cracolice, 1993-94)(Abraham & Cracolice, 1993-94)
Pretest-Post TestPretest-Post Test• Each student randomly assigned to treatment Each student randomly assigned to treatment
or control groups.or control groups.• Use pretest to measure variables of interest.Use pretest to measure variables of interest.• Implement treatment.Implement treatment.• Use post test (same as or equivalent to Use post test (same as or equivalent to
pretest).pretest). ProblemsProblems
• Possible interaction between pre and post Possible interaction between pre and post tests.tests.
1616
Pure Experimental DesignPure Experimental Design
Post-test only DesignPost-test only Design• Randomly assign students to treatment or control Randomly assign students to treatment or control
groups.groups.• Present treatment.Present treatment.• Administer post test on variables of interest.Administer post test on variables of interest.
Potential problemsPotential problems• Treatment and control groups may not be equivalent Treatment and control groups may not be equivalent
even though randomly assigned.even though randomly assigned.• If there is a differential drop out from treatment vs. If there is a differential drop out from treatment vs.
control groups, a biased sample may result.control groups, a biased sample may result.• Without pretest, cannot analyze students according to Without pretest, cannot analyze students according to
subgroup (low vs. high achieving students).subgroup (low vs. high achieving students).
1717
Quasi-Experimental DesignQuasi-Experimental Design
Students are not randomly assigned to Students are not randomly assigned to groups.groups.
Identification and measurement of Identification and measurement of pertinent covariables are compared pertinent covariables are compared between treatment and control groups between treatment and control groups to assure equality of groups.to assure equality of groups.• Gender, age, mathematical aptitude, Gender, age, mathematical aptitude,
logical reasoning ability, academic logical reasoning ability, academic accomplishment, science vs. nonscience accomplishment, science vs. nonscience major, etc.major, etc.
1818
Quasi-Experimental DesignQuasi-Experimental Design(Jaeger. Pg. 592)(Jaeger. Pg. 592)
Interrupted Time Series Interrupted Time Series • Data recorded for pertinent variable for many Data recorded for pertinent variable for many
consecutive points in time both before and consecutive points in time both before and after treatment is introduced.after treatment is introduced.
• Pattern of graph is studied before and after Pattern of graph is studied before and after treatment.treatment.
If graph shows an abrupt shift in level or direction at If graph shows an abrupt shift in level or direction at point of treatment, then treatment is cause.point of treatment, then treatment is cause.
AdvantageAdvantage• Useful with a small number of subjectsUseful with a small number of subjects
ProblemProblem• Separating out other possible causes of the Separating out other possible causes of the
shift in graph can be difficult.shift in graph can be difficult.
1919
Quasi-Experimental DesignQuasi-Experimental Design
One classroom designOne classroom design• Covariables of interest identified and Covariables of interest identified and
measured.measured. Ex. Logical reasoning ability, gender, Ex. Logical reasoning ability, gender,
academic aptitudeacademic aptitude
• Treatment applied.Treatment applied.• Achievement/Attitude measures Achievement/Attitude measures
administered.administered.• AnalysisAnalysis
Differences in achievement/attitude of Differences in achievement/attitude of subgroups, as identified through use of subgroups, as identified through use of covariables, are compared.covariables, are compared.
2020
Mixed Evaluation MethodsMixed Evaluation Methods(NSF 1997)(NSF 1997)
Approach combining both qualitative and Approach combining both qualitative and quantitative methodsquantitative methods
AdvantagesAdvantages• TriangulationTriangulation
validity of results strengthened by using more than validity of results strengthened by using more than one method to investigate question.one method to investigate question.
• Improved instrumentationImproved instrumentation Qualitative open-ended survey results used to design Qualitative open-ended survey results used to design
Likert scale survey.Likert scale survey.
• Mixed results may lead investigators to modify Mixed results may lead investigators to modify or expand research design.or expand research design.
2121
Mixed Methods:Mixed Methods:SurveysSurveys
Open-ended question surveys Open-ended question surveys (qualitative).(qualitative).• Analysis of responses can lead to Analysis of responses can lead to
development of a validated Likert Scale development of a validated Likert Scale Survey. Survey.
Likert Scale Survey (quantitative)Likert Scale Survey (quantitative)• Subjects respond to a series of Subjects respond to a series of
questions/statements by indicating the questions/statements by indicating the amount of agreement/disagreement on a amount of agreement/disagreement on a scale of 1scale of 15.5.
2222
Mixed Methods:Mixed Methods:InterviewsInterviews
Structured interviewsStructured interviews• Fixed sequence of pre-determined questionsFixed sequence of pre-determined questions• Interviewer does not deviate from script.Interviewer does not deviate from script.
Semi-structured interviewsSemi-structured interviews• Interviewer is free to ask spontaneous questions of Interviewer is free to ask spontaneous questions of
intervieweeinterviewee Think Aloud interviewThink Aloud interview
• Present subjects with a demonstration or Present subjects with a demonstration or problem. problem.
• Interviewee solves and thinks aloud.Interviewee solves and thinks aloud.• Interviewer asks questions such as Interviewer asks questions such as
“ “ What are you thinking?”What are you thinking?”
2323
Mixed Methods:Mixed Methods:InterviewsInterviews
Unstructured InterviewsUnstructured Interviews• Interviewer asks subject questions as Interviewer asks subject questions as
they arise. they arise. Not all subjects are asked the same Not all subjects are asked the same
questions.questions.
2424
SummarySummary Project EvaluationProject Evaluation
• Tied to project goalsTied to project goals• Includes responses of stakeholders in projectIncludes responses of stakeholders in project• Includes both formative and summative evaluationIncludes both formative and summative evaluation
MethodologyMethodology• Match methodology to evaluation questionsMatch methodology to evaluation questions
Mixed Methods uses both qualitative and quantitative Mixed Methods uses both qualitative and quantitative methods.methods.
• Validity of evaluation strengthened by using multiple methods Validity of evaluation strengthened by using multiple methods (triangulation).(triangulation).
ResultsResults• Inform stakeholdersInform stakeholders• Lead to actionLead to action
2525
ReferencesReferences Abraham, M. R. and M. S. Cracolice. (1993-94). Doing Research on Abraham, M. R. and M. S. Cracolice. (1993-94). Doing Research on
College Science Instruction: Realistic Research Designs. Journal of College Science Instruction: Realistic Research Designs. Journal of College Science Teaching. Dec.93/Jan.94, pp.150-153.College Science Teaching. Dec.93/Jan.94, pp.150-153.
Jaeger, R. M. (Ed.) (1988). Complementary Methods for Research Jaeger, R. M. (Ed.) (1988). Complementary Methods for Research in Education. 2in Education. 2ndnd Ed. Washington, DC: American Educational Ed. Washington, DC: American Educational Research Association.Research Association.
Miles, M. B. and Huberman, A. M. (1994). Qualitative Data Miles, M. B. and Huberman, A. M. (1994). Qualitative Data Analysis, 2Analysis, 2ndnd Ed. Newbury Park, CA: Sage. Ed. Newbury Park, CA: Sage.
The 2002 User-Friendly Handbook for Project Evaluation. (2002). The 2002 User-Friendly Handbook for Project Evaluation. (2002). Washington, DC: National Science Foundation (02-057).Washington, DC: National Science Foundation (02-057).
User Friendly Handbook for Mixed Methods Evaluations. (1997). User Friendly Handbook for Mixed Methods Evaluations. (1997). Washington, DC: National Science Foundation (97-153).Washington, DC: National Science Foundation (97-153).
top related