1 evaluation of education development projects ccli pi meeting august 15, 2008
Post on 25-Dec-2015
213 Views
Preview:
TRANSCRIPT
1
Evaluation of Education Evaluation of Education Development ProjectsDevelopment Projects
CCLI PI MeetingCCLI PI MeetingAugust 15, 2008August 15, 2008
2
Caution
The information in these slides The information in these slides represents the opinions of the represents the opinions of the individual program directors and not individual program directors and not an official NSF position.an official NSF position.
3
Session Goals
The session willThe session will
Increase your understanding of evaluation Increase your understanding of evaluation
Enable you to collaborate more effectively Enable you to collaborate more effectively with evaluation experts in preparing and with evaluation experts in preparing and performing effective project evaluation plansperforming effective project evaluation plans
It will It will NOT NOT make you an evaluation expertmake you an evaluation expert
4
Perspective
Ideas here applicable to your project Ideas here applicable to your project whether in Week 5 or Year 5whether in Week 5 or Year 5
Consider your evaluation in light of what Consider your evaluation in light of what you discover hereyou discover here
Make adjustments to try to get the best Make adjustments to try to get the best evaluation data possibleevaluation data possible
5
Definitions AssessmentAssessment
Measurement of something (e. g., students’ Measurement of something (e. g., students’ ability to solve problemsability to solve problems
Usually does not carry a value judgmentUsually does not carry a value judgment Assessment instrument (tool)Assessment instrument (tool)
The method or device for measuring (e. g., a The method or device for measuring (e. g., a test; survey; portfolio)test; survey; portfolio)
Evaluation (e-VALUE-ation)Evaluation (e-VALUE-ation) A judgment of something based upon results A judgment of something based upon results
from one or more assessmentsfrom one or more assessments Frequently compared to an expected outcomeFrequently compared to an expected outcome
6
Evaluation and Assessment Evaluation (assessment) is used in many waysEvaluation (assessment) is used in many ways
Individual’s Individual’s performance (grading)performance (grading) Program’s Program’s effectiveness (ABET; regional effectiveness (ABET; regional
accreditation)accreditation) Project’s Project’s progress & accomplishmentsprogress & accomplishments (monitoring & (monitoring &
validating)validating)
Session addresses Session addresses project evaluationproject evaluation May involve evaluating individual and group May involve evaluating individual and group
performance – but in theperformance – but in the context of the project context of the project
Project evaluation Project evaluation Formative Formative – monitoring progress– monitoring progress Summative Summative – characterizing final accomplishments– characterizing final accomplishments
7
Evaluation and Project Goals & Outcomes
Evaluation starts with carefully defined project goals Evaluation starts with carefully defined project goals & expected (measurable) outcomes& expected (measurable) outcomes
Goals & expected outcomes related to:Goals & expected outcomes related to: Project managementProject management outcomes outcomes
Initiating or completing an activityInitiating or completing an activityFinishing a “product”Finishing a “product”
Expected sExpected student outcomestudent outcomesModifying a learning outcomeModifying a learning outcomeModifying attitudes or perceptionsModifying attitudes or perceptions
Workshop focuses on student outcomesWorkshop focuses on student outcomes
9
Developing Student Behavior Goals & Outcomes
Start with one or more overarching statements Start with one or more overarching statements of project intentionof project intention Each statement is a Each statement is a GOALGOAL
What is your overall ambition? What do you What is your overall ambition? What do you hope to achieve?hope to achieve?
Convert each goal into one or more specific Convert each goal into one or more specific expected measurable resultsexpected measurable results Each result is an Each result is an EXPECTED OUTCOMEEXPECTED OUTCOME
How will achieving your “intention” reflect in How will achieving your “intention” reflect in student behavior?student behavior?
10
Goals – Objectives – Outcomes -- Questions
Converting goals to expected outcomes Converting goals to expected outcomes may involve intermediate stepsmay involve intermediate steps Intermediate steps maybe called Intermediate steps maybe called objectivesobjectives
More specific, more measurable than goalsMore specific, more measurable than goalsLess specific, less measurable than Less specific, less measurable than
outcomesoutcomes Expected outcomes lead to Expected outcomes lead to questionsquestions
These form the basis of the evaluationThese form the basis of the evaluation Evaluation process Evaluation process collects and interpretscollects and interprets
data to answer evaluation questionsdata to answer evaluation questions
11
Exercise
Identification of Goals and Expected Outcomes
Read the abstract Read the abstract Note - Goal statement removedNote - Goal statement removed
Suggest two plausible goalsSuggest two plausible goals One focused on a change in learning One focused on a change in learning One focused on a change in some other One focused on a change in some other
aspect of student behavioraspect of student behavior
Use student not instructor focusUse student not instructor focus
12
Abstract The goal of the project is …… The project is developing computer-The goal of the project is …… The project is developing computer-
based instructional modules for statics and mechanics of materials. based instructional modules for statics and mechanics of materials. The project uses 3D rendering and animation software, in which The project uses 3D rendering and animation software, in which the user manipulates virtual 3D objects in much the same manner the user manipulates virtual 3D objects in much the same manner as they would physical objects. Tools being developed enable as they would physical objects. Tools being developed enable instructors to realistically include external forces and internal instructors to realistically include external forces and internal reactions on 3D objects as topics are being explained during reactions on 3D objects as topics are being explained during lectures. Exercises are being developed for students to be able to lectures. Exercises are being developed for students to be able to communicate with peers and instructors through real-time voice communicate with peers and instructors through real-time voice and text interactions. The project is being evaluated by … The and text interactions. The project is being evaluated by … The project is being disseminated through … The broader impacts of project is being disseminated through … The broader impacts of the project are …the project are …
Substitute sampleSubstitute sample““organic chemistry” for “statics and mechanics of materials”organic chemistry” for “statics and mechanics of materials”““Interactions” for “external forces and internal reactions”Interactions” for “external forces and internal reactions”““molecules” for “objects”molecules” for “objects”
Focus student not instructor perspectiveFocus student not instructor perspective
13
PD’s Response - Goals Goals may focus onGoals may focus on
Cognitive changesCognitive changes Knowledge-driven Knowledge-driven Skill-driven Skill-driven
AAffective (or affective (or attitudinalttitudinal)) changes changes Success changesSuccess changes Diversity ChangesDiversity Changes
14
PD’s Response
Goals - Cognitive Change
GOALGOAL: Improve understanding or : Improve understanding or skillsskills
In the context ofIn the context of the course the course Describe verbally the effect of external Describe verbally the effect of external
forces on a solid objectforces on a solid object Solve textbook problems Solve textbook problems
In application In application beyond coursebeyond course Solve out-of-context problems Solve out-of-context problems Visualize 3-D problemsVisualize 3-D problems Communicate technical problems orallyCommunicate technical problems orally
15
PD’s Response Goals on Attitudinal Changes
GOALGOAL: Improve: Improve
Interest in the courseInterest in the course Attitude about Attitude about
ProfessionProfession CurriculumCurriculum DepartmentDepartment
Self- confidenceSelf- confidence Intellectual developmentIntellectual development
16
PD’s Response Goals on Success Changes
Goals: Improve Goals: Improve Recruitment ratesRecruitment rates Retention or persistence ratesRetention or persistence rates Graduation ratesGraduation rates
17
PD’s Response Goals on Diversity
““Broaden the participation of Broaden the participation of underrepresented groups”underrepresented groups”
GOAL:GOAL: To change a target group’s To change a target group’s Cognitive abilities (understanding Cognitive abilities (understanding
or skills)or skills)
AttitudesAttitudes
Success ratesSuccess rates
18
Exercise
Transforming Goals into Expected Outcomes
Write one expected measurable outcome Write one expected measurable outcome for each of the following goals:for each of the following goals:
1.1. Increase the students’ understanding Increase the students’ understanding of the concepts in staticsof the concepts in statics
1.1. Improve the students’ attitude about Improve the students’ attitude about engineering as a careerengineering as a career
19
PD’s Response Expected Outcomes
Conceptual understandingConceptual understanding Students will be better able to solve Students will be better able to solve
conceptual problems that don’t require the conceptual problems that don’t require the use of formulas or calculationsuse of formulas or calculations
Students will be better able to solve out-of-Students will be better able to solve out-of-context problems.context problems.
AttitudinalAttitudinal Students will be more likely to describe Students will be more likely to describe
engineering as an exciting careerengineering as an exciting career The percentage of students who transfer out The percentage of students who transfer out
of engineering after the statics course will of engineering after the statics course will decrease. decrease.
20
ExerciseTransforming Expected Outcomes
into Questions
Write a question for these expected Write a question for these expected measurable outcomes:measurable outcomes:
1.1. Students will be better able to solve Students will be better able to solve conceptual problems that do not require the conceptual problems that do not require the use of formulas or calculationsuse of formulas or calculations
2.2. In informal discussions, students will be In informal discussions, students will be more likely to describe engineering as an more likely to describe engineering as an exciting careerexciting career
21
PD’s Response
Questions
Conceptual understandingConceptual understanding
Did the students’ ability to solve Did the students’ ability to solve conceptual problems increase ?conceptual problems increase ?
Did the students’ ability to solve Did the students’ ability to solve conceptual problems increase conceptual problems increase because because of the use of the 3D rendering and of the use of the 3D rendering and animation softwareanimation software??
22
PD’s Response - Questions
AttitudinalAttitudinal Did the students discussions indicate Did the students discussions indicate
more excitement about engineering as more excitement about engineering as a career?a career?
Did the students discussions indicate Did the students discussions indicate more excitement, about engineering as more excitement, about engineering as a career a career because of the use of the 3D because of the use of the 3D rendering and animation software?rendering and animation software?
23
Reflection
What is the most surprising idea What is the most surprising idea
you heard in this session?you heard in this session?
25
FLAG (Field-Tested Learning Assessment Guide)
A Primer of Assessment & EvaluationA Primer of Assessment & Evaluation
Classroom Assessment Techniques (CATS)Classroom Assessment Techniques (CATS) QualitativeQualitative QuantitativeQuantitative
Searchable & Down-Loadable ToolsSearchable & Down-Loadable Tools
Resources in AssessmentResources in Assessment
http://www.flaguide.orghttp://www.flaguide.org
26
FLAG Classroom Assessment Techniques (CATS)
Attitudinal SurveyAttitudinal Survey Concept TestsConcept Tests Concept MappingConcept Mapping Conceptual Diagnostic TestsConceptual Diagnostic Tests InterviewsInterviews Performance Assessment PortfolioPerformance Assessment Portfolio Scoring RubricsScoring Rubrics Weekly ReportsWeekly Reports
http://www.flaguide.orghttp://www.flaguide.org
27
SALG (Student Assessment of Learning Gains)
Assesses perceived degree of “gain” students Assesses perceived degree of “gain” students made in specific aspects of the classmade in specific aspects of the class
Spotlights course elements that best support Spotlights course elements that best support student learning and those needing improvementstudent learning and those needing improvement
Web-based instrument requiring 10-15 minutes to Web-based instrument requiring 10-15 minutes to useuse
Easily modified by the instructorEasily modified by the instructor Provides instant statistical analysis of resultsProvides instant statistical analysis of results Facilitates formative evaluation throughout course Facilitates formative evaluation throughout course
http://www.salgsite.orghttp://www.salgsite.org
29
Introduction to CIs A tool that measures A tool that measures conceptual conceptual
understandingunderstanding Series of multiple choice questionsSeries of multiple choice questions
Questions involve Questions involve single conceptsingle concept Formulas, calculations, or problem solving not Formulas, calculations, or problem solving not
requiredrequired
Possible answers include Possible answers include ““detractorsdetractors”” Common errors Common errors Reflect commonReflect common “ “misconceptions”misconceptions”
Force Concept Inventory (FCI) is the prototypeForce Concept Inventory (FCI) is the prototype
30
Exercise
Evaluating a CI Tool
Suppose you where considering an Suppose you where considering an existing CI for use in your project’s existing CI for use in your project’s evaluationevaluation
What questions would you consider in What questions would you consider in deciding if the tool is appropriate?deciding if the tool is appropriate?
31
PD’s Response
Evaluating a CI Tool Nature of the toolNature of the tool
Is the tool relevant to what was taught? Is the tool relevant to what was taught? Is the tool competency based? Is the tool competency based? Is the tool conceptual or procedural?Is the tool conceptual or procedural?
Prior validation of the toolPrior validation of the tool Has the tool been tested? Has the tool been tested? Is there information on reliability and validity? Is there information on reliability and validity? Has it been compared to other tools? Has it been compared to other tools? Is it sensitive? Does it discriminate novice and expert?Is it sensitive? Does it discriminate novice and expert?
Experience of others with the toolExperience of others with the tool Has the tool been used by others besides the Has the tool been used by others besides the
developer? At other sites? With other populations? developer? At other sites? With other populations? Is there normative data?Is there normative data?
32
Decision Factors for Other Tools
Would these questions be different Would these questions be different for another tool?for another tool?
34
Hypothetical Concept Inventory Data
Pre Post Pre Post1 35 32 29% 23%2 35 32 34% 65%3 35 32 74% 85%
- - - - -
Quest
35
Exercise
Alternate Explanation For Change
Data suggests that the understanding Data suggests that the understanding of Concept #2 increasedof Concept #2 increased
One interpretation is that the One interpretation is that the intervention caused the changeintervention caused the change
List some List some alternative explanationsalternative explanations Confounding factorsConfounding factors Other factors that could explain the Other factors that could explain the
changechange
36
PD's Response
Alternate Explanation For Change
Students learned concept out of class (e. g., in Students learned concept out of class (e. g., in another course or in study groups with students another course or in study groups with students not in the course)not in the course)
Students answered with what the instructor wanted Students answered with what the instructor wanted rather than what they believed or “knew” rather than what they believed or “knew”
An external event (big test in previous period or a An external event (big test in previous period or a “bad-hair day”) distorted pre- or post-test data“bad-hair day”) distorted pre- or post-test data
Instrument was unreliableInstrument was unreliable Other changes in course and not the intervention Other changes in course and not the intervention
caused improvementcaused improvement Students not representative groupsStudents not representative groups
37
Exercise
Alternate Explanation for Lack of Change
Data suggests that the understanding Data suggests that the understanding of Concept #1 did not increase of Concept #1 did not increase
One interpretation is that the One interpretation is that the intervention did cause a change but it intervention did cause a change but it was masked by other factorswas masked by other factors
List some List some confounding factorsconfounding factors that that could have masked a real changecould have masked a real change
38
PD's Response
Alternate Explanations for Lack of Effect
An external event (big test in previous period An external event (big test in previous period or a “bad-hair day”) distorted pre- or post-or a “bad-hair day”) distorted pre- or post-test datatest data
The instrument was unreliableThe instrument was unreliable Implementation of the intervention was poorImplementation of the intervention was poor Population too smallPopulation too small One or both student groups not One or both student groups not
representative representative Formats were different on pre and post testsFormats were different on pre and post tests
39
Reflection
What is the most surprising idea What is the most surprising idea
you heard in this session so far?you heard in this session so far?
41
Exercise
Evaluation Plan
Read the evaluation plan Read the evaluation plan and suggest improvementsand suggest improvements
42
Exercise - Evaluation Plan The results of the project will be evaluated in several ways. The results of the project will be evaluated in several ways.
First, students will be surveyed at the end of the semester on First, students will be surveyed at the end of the semester on the content, organization, continuity of the topics, and the the content, organization, continuity of the topics, and the level of difficulty. Since the course can be taken as dual level of difficulty. Since the course can be taken as dual enrollment by local high school students, it is expected that enrollment by local high school students, it is expected that the course will attract more students to the college. Due to the course will attract more students to the college. Due to the urban setting of Grant College, a majority of these the urban setting of Grant College, a majority of these students will be working part time, and they will be asked students will be working part time, and they will be asked about the appropriateness of the content and its relevancy in about the appropriateness of the content and its relevancy in their jobs.their jobs.
Second, the professors teaching the subsequent advanced Second, the professors teaching the subsequent advanced bioinformatics lecture courses will be asked to judge the bioinformatics lecture courses will be asked to judge the students’ ability to apply bioinformatics. While the projects students’ ability to apply bioinformatics. While the projects in these upper level classes focus on different application in these upper level classes focus on different application areas, the problems frequently involve concepts initially areas, the problems frequently involve concepts initially learned in the new bioinformatics laboratory. The current learned in the new bioinformatics laboratory. The current advanced bioinformatics instructors have taught the course advanced bioinformatics instructors have taught the course for several years and are qualified to compare the future for several years and are qualified to compare the future students’ abilities with their previous students.students’ abilities with their previous students.
43
PD’s Responses - Evaluation Plan
Tie evaluation to expected outcomesTie evaluation to expected outcomes Include measures of student learningInclude measures of student learning Include capturing the demographics of Include capturing the demographics of
the population the population Use an “external” evaluator for objectivityUse an “external” evaluator for objectivity Describe processes for formative Describe processes for formative
evaluation and summative evaluationevaluation and summative evaluation Consider including beta test at one or Consider including beta test at one or
more other sitesmore other sites Include impact statementInclude impact statement
44
References NSF’s User Friendly Handbook for Project NSF’s User Friendly Handbook for Project
EvaluationEvaluation http://www.nsf.gov/pubs/2002/nsf02057/start.htmhttp://www.nsf.gov/pubs/2002/nsf02057/start.htm
Field-Tested Learning Assessment Guide (FLAG)Field-Tested Learning Assessment Guide (FLAG) http://www.flaguide.orghttp://www.flaguide.org
Online Evaluation Resource Library (OERL)Online Evaluation Resource Library (OERL) http://oerl.sri.com/http://oerl.sri.com/
CCLI Evaluation Planning WebinarCCLI Evaluation Planning Webinar http://oerl.sri.com/ccli_resources.htmlhttp://oerl.sri.com/ccli_resources.html
SALG (Student Assessment of Learning GainsSALG (Student Assessment of Learning Gains http://http://www.salgsite.orgwww.salgsite.org
top related