project evaluation webinar 3 of the transforming undergraduate education in science, technology,...

61
Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis Terpenny [email protected] [email protected] April 6, 2011 Sue Fitzgerald & Louis Everett [email protected] [email protected] April 7, 2011

Upload: posy-carson

Post on 25-Dec-2015

217 views

Category:

Documents


4 download

TRANSCRIPT

Page 1: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Project EvaluationWebinar 3 of the

Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series

Scott Grissom & Janis Terpenny [email protected] [email protected]

April 6, 2011

Sue Fitzgerald & Louis Everett [email protected] [email protected]

April 7, 2011

Page 2: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Workshop presenters

2

Janis TerpennyVirginia Tech

Mechanical Engineering &Engineering Education

Scott GrissomGrand Valley State University

Computer Science & Info Systems

Sue FitzgeraldMetropolitan State University

Computer Science

Louis EverettUniv. Texas El Paso

Mechanical Engineering

Page 3: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Most of the information presented in this workshop represents the presenters’ opinions and not an official NSF position

Local facilitators will provide the link to the workshop slides at the completion of the webinar.

Participants may ask questions by “raising their virtual hand” during a question session. We will call on selected sites and enable their microphone so that the question can be asked.

Responses will be collected from a few sites at the end of each Exercise. At the start of the Exercise, we will identify these sites in the Chat Box and then call on them one at a time to provide their responses.

3

Important Notes

Page 4: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Learning must build on prior knowledge ◦ Some knowledge correct ◦ Some knowledge incorrect – Misconceptions

Learning is ◦ Connecting new knowledge to prior knowledge◦ Correcting misconceptions

Learning requires engagement◦ Actively recalling prior knowledge◦ Sharing new knowledge◦ Forming a new understanding

Framework for the Session

Page 5: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Effective learning activities ◦ Recall prior knowledge -- actively, explicitly◦ Connect new concepts to existing ones◦ Challenge and alter misconceptions

Active & collaborative processes◦ Think individually◦ Share with partner◦ Report to local and virtual groups ◦ Learn from program directors’ responses

5

Preliminary Comments

Active & Collaborative Learning

Page 6: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Coordinate the local activities

Watch the time◦Allow for think, share, and report phases◦Reconvene on time -- 1 min warning slide◦ With one minute warning, check Chat Box to see if you will be

asked for a response

Ensure the individual think phase is devoted to thinking and not talking

Coordinate the asking of questions by local participants and reporting local responses to exercises

6

Facilitator’s Duties

Page 7: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

The session will enable you to collaborate more effectively with evaluation experts in preparing credible and comprehensive project evaluation plans…. it will not make you an evaluation expert.

Goal for Project Evaluation Session

Page 8: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

After the session, participants should be able to: Discuss the importance of goals, outcomes, and

questions in the evaluation process◦ Cognitive and affective outcomes

Describe several types of evaluation tools◦ Advantages, limitations, and appropriateness

Discuss data interpretation issues◦ Variability, alternative explanations

Develop an evaluation plan in collaboration with an evaluator◦ Outline a first draft of an evaluation plan

Session Outcomes

Page 9: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

The terms evaluation and assessment have many meanings◦ One definition

Assessment is gathering evidence Evaluation is interpreting data and making value judgments

Examples of evaluation and assessment◦ Individual’s performance (grading)◦ Program’s effectiveness (ABET and regional accreditation)◦ Project’s progress and success (monitoring and validating)

Session addresses project evaluation◦ May involve evaluating individual and group performance – but in the

context of the project Project evaluation

◦ Formative – monitoring progress to improve approach◦ Summative – characterizing and documenting final accomplishments

Evaluation and Assessment

Page 10: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Think about your favorite course. What types of in class activities could be called:◦ Assessment versus Evaluation◦ Formative versus Summative Evaluation

Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min

Watch time and reconvene after 6 min Use THINK time to think – no discussion, Selected local facilitators

report to virtual group

With one minute warning, check Chat Box to see if you will be asked for a response

10

Activity

Evaluation vs AssessmentFormative vs Summative

Page 11: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

One MinuteLook at Chat Box to see if you will be asked to respond

Page 12: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Think about your favorite course. What types of in class activities could be called:◦ Assessment versus Evaluation◦ Formative versus Summative Evaluation

Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min

Watch time and reconvene after 6 min Use THINK time to think – no discussion, Selected local facilitators

report to virtual group

With one minute warning, check Chat Box to see if you will be asked for a response

12

Activity

Evaluation vs AssessmentFormative vs Summative

Page 13: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

An in class quiz could be called assessment. Using the in class quiz results to realize the

students do not understand how to … because they did not … is an evaluation.

Using the evaluation to change the activity you used to teach … makes it a formative evaluation.

Aha! You find your new activity produces great assessment results makes it a summative evaluation.

13

PD Response

Evaluation vs AssessmentFormative vs Summative

Page 14: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Project Goals, Expected Outcomes, and Evaluation Questions

Page 15: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Effective evaluation starts with carefully defined project goals and expected outcomes

Goals and expected outcomes related to:◦Project management

Initiating or completing an activity Finishing a “product”

◦Student behavior Modifying a learning outcome Modifying an attitude or a perception

Evaluation and Project Goals/Outcomes

Page 16: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Goals provide overarching statements of project intention

What is your overall ambition? What do you hope to achieve?

Expected outcomes identify specific observable or measureable results for each goal

How will achieving your “intention” be reflected by changes in student behavior?How will it change their learning and their attitudes?

Learning Goals and Outcomes

Page 17: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Goals → Expected outcomes

Expected outcomes → Evaluation questions

Questions form the basis of the evaluation process

The evaluation process consists of the collection and interpretation of data to answer evaluation questions

Goals, Expected Outcomes, and Evaluation Questions

Page 18: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Read the abstract -- Goal statement removed Suggest two plausible goals

◦ One on student learning Cognitive behavior

◦ One on some other aspect of student behavior Affective behavior

Focus on what will happen to the students ◦ Do not focus on what the instructor will do

Long Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min

Watch time and reconvene after 6 min Use THINK time to think – no discussion, Selected local facilitators

report to virtual group

Activity

Identification of Goals/Outcomes

Page 19: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

The goal of the project is …… The project is developing computer-based instructional modules for statics and mechanics of materials. The project uses 3D rendering and animation software, in which the user manipulates virtual 3D objects in much the same manner as they would physical objects. Tools being developed enable instructors to realistically include external forces and internal reactions on 3D objects as topics are being explained during lectures. Exercises are being developed for students to be able to communicate with peers and instructors through real-time voice and text interactions. The project is being evaluated by … The project is being disseminated through … The broader impacts of the project are …

Two goals: one for student learning and one for student behavior

Non engineers should substitute: “Organic chemistry” for “statics and mechanics of materials” “Interactions” for “external forces and internal reactions”

Abstract

Page 20: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

One MinuteLook at Chat Box to see if you will be asked to respond

Page 21: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

The goal of the project is …… The project is developing computer-based instructional modules for statics and mechanics of materials. The project uses 3D rendering and animation software, in which the user manipulates virtual 3D objects in much the same manner as they would physical objects. Tools being developed enable instructors to realistically include external forces and internal reactions on 3D objects as topics are being explained during lectures. Exercises are being developed for students to be able to communicate with peers and instructors through real-time voice and text interactions. The project is being evaluated by … The project is being disseminated through … The broader impacts of the project are …

Two goals: one for student learning and one for student behavior

Non engineers should substitute: “Organic chemistry” for “statics and mechanics of materials” “Interactions” for “external forces and internal reactions”

Abstract

Page 22: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

GOAL: To improve conceptual understanding and processing skills

In the context of course◦ Draw free-body diagrams for textbook problems◦ Solve 3-D textbook problems ◦ Describe the effect(s) of external forces on a solid object orally

In a broader context◦ Solve out-of-context problems ◦ Visualize 3-D problems◦ Communicate technical problems orally◦ Improve critical thinking skills◦ Enhance intellectual development

PDs’ Response: Goals on Cognitive Behavior

Page 23: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

GOAL: To improve◦ Self- confidence◦ Attitude about engineering as a career

PDs’ Response: Goals on Affective Behavior

Page 24: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Write SMART outcomes for the goals Specific Measurable Attainable Realistic Timely

If I achieve the outcomes, I have expectation that I am closer to my goal

Goal: My students will be life long learners Outcomes – Some of these lack SMART components

Number of books read this year is larger than last year Seven non-fiction books are read over spring break Number of visits to the campus library is large

Outcomes provide evidence that goals are being achieved

Transforming Goals into Outcomes

Page 25: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Write expected measurable outcomes for each of the following goals:

◦ Improve the students’ understanding of the fundamental concepts in statics (cognitive)

◦ Improve the students’ self confidence (affective)

Long Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min

Watch time and reconvene after 6 min Use THINK time to think – no discussion, Selected local facilitators report to

virtual group

Activity

Transforming Goals into Outcomes

Non engineers may substitute: “Organic chemistry” for “statics”

Page 26: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

One MinuteLook at Chat Box to see if you will be asked to respond

Page 27: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Write expected measurable outcomes for each of the following goals:

◦ Improve the students’ understanding of the fundamental concepts in statics (cognitive)

◦ Improve the students’ self confidence (affective)

Long Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min

Watch time and reconvene after 6 min Use THINK time to think – no discussion, Selected local facilitators report to

virtual group

Activity

Transforming Goals into Outcomes

Non engineers may substitute: “Organic chemistry” for “statics”

Page 28: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

PDs’ Response: Expected Outcomes

Understanding of the fundamentals◦ By the end of the class, 70% of the students will be able to:

Correctly draw freebody diagrams of 2D truss structures Correctly write Newton’s laws when given a FBD Describe the effects on member force when one angle in a 2D

truss is changed

Self-Confidence◦ By the end of the semester:

30% of the class is willing to show the solution to any homework problem on the board

Self reported test anxiety reduces to 50% of the initial amount 80% will say the class was easier than they expected it would be 50% report they are excited about taking the follow-on course

Page 29: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Evaluation Questions

Understanding of the fundamentals◦ Are the students better able to describe the effects of changing

some variable in a simple problem◦ Are the students better able to describe the effects of changing

some variable in a simple problem as a result of the intervention

Self-Confidence◦ Do the students express more confidence in their solutions◦ Do the students express more confidence in their solutions as a

result of the intervention

Page 30: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Questions

“Hold-up your virtual hand” and you will be called upon after we unmute your mike.

30

Page 31: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

BREAK15 min

31

Page 32: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

BREAK1 min

32

Page 33: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Tools for Evaluating Learning Outcomes

Page 34: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Surveys ◦ Forced choice or open-ended responses

Concept Inventories◦ Multiple-choice questions to measure conceptual understanding

Rubrics for analyzing student products◦ Guides for scoring student reports, tests, etc.

Interviews◦ Structured (fixed questions) or in-depth (free flowing)

Focus groups◦ Like interviews but with group interaction

Observations◦ Actually monitor and evaluate behavior

Olds et al, JEE 94:13, 2005

NSF’s Evaluation Handbook

Examples of Tools for Evaluating Learning Outcomes 

Page 35: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Comparing Surveys and Observations

Surveys Efficient Accuracy depends on

subject’s honesty Difficult to develop reliable

and valid survey Low response rate

threatens reliability, validity & interpretation

Observations Time & labor intensive Inter-rater reliability must

be established Captures behavior that

subjects are unlikely to report

Useful for observable behavior

Olds et al, JEE 94:13, 2005

Page 36: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Example – Appropriateness of Interviews

Use interviews to answer these questions:◦What does program look and feel like?◦What do stakeholders know about the project?◦What are stakeholders’ and participants’ expectations?◦What features are most salient?◦What changes do participants perceive in themselves?

The 2002 User Friendly Handbook for Project Evaluation, NSF publication REC 99-

12175

Page 37: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Originated in physics -- Force Concept Inventory (FCI) Several are being developed in engineering fields Series of multiple choice questions◦ Questions involve single concept

Formulas, calculations or problem solving skills not required

◦ Possible answers include detractors Common errors -- misconceptions

Developing CI is involved◦ Identify misconceptions and detractors◦ Develop, test, and refine questions◦ Establish validity and reliability of tool ◦ Language is a major issue

Tool for Measuring Conceptual Understanding – Concept Inventory

Page 38: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Tool for Assessing Attitude

Pittsburgh Freshman Engineering Survey◦ Questions about perception

Confidence in their skills in chemistry, communications, engineering, etc.

Impressions about engineering as a precise science, as a lucrative profession, etc.

Validated using alternate approaches: ◦ Item analysis◦ Verbal protocol elicitation◦ Factor analysis

Compared results for students who stayed in engineering to those who left

Besterfield-Sacre et al , JEE 86:37, 1997

Page 39: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Tools for Characterizing Intellectual Development

Levels of Intellectual Development◦Students see knowledge, beliefs, and authority in

different ways “ Knowledge is absolute” versus “Knowledge is

contextual” Tools ◦Measure of Intellectual Development (MID)◦Measure of Epistemological Reflection (MER)◦ Learning Environment Preferences (LEP)

Felder et al, JEE 94:57, 2005

Page 40: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Suppose you were considering an existing tool (e. g., a concept inventory) for use in your project’s evaluation of learning outcomes

What questions would you consider in deciding if the tool is appropriate?

Long Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min

Watch time and reconvene after 6 min Use THINK time to think – no discussion Selected local facilitators report to virtual group

Activity

Considering an Existing Tool

Page 41: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

One MinuteLook at Chat Box to see if you will be asked to respond

Page 42: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Suppose you were considering an existing tool (e. g., a concept inventory) for use in your project’s evaluation of learning outcomes

What questions would you consider in deciding if the tool is appropriate?

Long Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min

Watch time and reconvene after 6 min Use THINK time to think – no discussion Selected local facilitators report to virtual group

Activity

Considering an Existing Tool

Page 43: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Nature of the tool◦ Is the tool relevant to what was taught? ◦ Is the tool competency based? ◦ Is the tool conceptual or procedural?

Prior validation of the tool◦ Has the tool been tested? ◦ Is there information concerning its reliability and validity? ◦ Has it been compared to other tools? ◦ Is it sensitive? Does it discriminate between a novice and an expert?

Experience of others with the tool◦ Has the tool been used by others besides the developer? At other

sites? With other populations? ◦ Is there normative data?

PDs’ Response

Evaluating an Existing Tool

Page 44: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Questions

Hold up your “virtual hand” to ask a question.

Page 45: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Interpreting Evaluation Data

Page 46: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Interpreting Evaluation Data

Comparision Group

Experimental Group

Comparision Group

Experimental Group

1 25 30 29% 23%2 24 32 34% 65%3 25 31 74% 85%

- - - - -

Question or

Concept

Percent w ith Correct AnswerNo. of Students

Data suggest that the understanding of Concept #2 increased

Page 47: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Comparision Group

Experimental Group

Comparision Group

Experimental Group

1 25 30 29% 23%2 24 32 34% 65%3 25 31 74% 85%

- - - - -

Question or

Concept

Percent with Correct AnswerNo. of Students

Data suggest that the understanding of Concept #2 increased

One interpretation is that the intervention caused the change

List some alternative explanations ◦ Confounding factors◦ Other factors that could explain the change

Long Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min

Watch time and reconvene after 6 min Use THINK time to think – no discussion Selected local facilitators report

to virtual group

Activity

Alternative Explanations For Change

Page 48: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

One MinuteLook at Chat Box to see if you will be asked to respond

Page 49: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Comparision Group

Experimental Group

Comparision Group

Experimental Group

1 25 30 29% 23%2 24 32 34% 65%3 25 31 74% 85%

- - - - -

Question or

Concept

Percent with Correct AnswerNo. of Students

Data suggest that the understanding of Concept #2 increased

One interpretation is that the intervention caused the change

List some alternative explanations ◦ Confounding factors◦ Other factors that could explain the change

Long Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min

Watch time and reconvene after 6 min Use THINK time to think – no discussion Selected local facilitators report

to virtual group

Activity

Alternative Explanations For Change

Page 50: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Students learned the concept out of class (e. g., in another course or in study groups with students not in the course)

Students answered with what they thought the instructor wanted rather than what they believed or “knew”

An external event distorted the pretest data The instrument was unreliable Other changes in the course and not the intervention was

responsible for the improvement The characteristics of groups were not similar

PDs’ Response

Alternative Explanations For Change

Page 51: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Data suggest that the understanding of the concept tested by Q1 did not improve

One interpretation is that the intervention did cause a change that was masked by other factors

Think about alternative explanations How would these alternative explanations

(confounding factors) differ from the previous list?

Alternative Explanations for Lack of Change

Comparision Group

Experimental Group

Comparision Group

Experimental Group

1 25 30 29% 23%2 24 32 34% 65%3 25 31 74% 85%

- - - - -

Question or

Concept

Percent w ith Correct AnswerNo. of Students

Page 52: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Evaluation Plan

Page 53: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

List the topics that need to be addressed in the evaluation plan (a.k.a. summarize)

Long Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min

Watch time and reconvene after 6 min Use THINK time to think – no discussion Selected local facilitators report to virtual group

Activity

Evaluation Plan

Page 54: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

One MinuteLook at Chat Box to see if you will be asked to respond

Page 55: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

List the topics that need to be addressed in the evaluation plan (a.k.a. summarize)

Long Exercise ---- 6 min◦ Think individually -------- ~2 min◦ Share with a partner ----- ~2 min ◦ Report in local group ---- ~2 min

Watch time and reconvene after 6 min Use THINK time to think – no discussion Selected local facilitators report to virtual group

Activity

Evaluation Plan

Page 56: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Name & qualifications of the evaluation expert◦ Get the evaluator involved early in the proposal development phase

Goals, outcomes, and evaluation questions Instruments for evaluating each outcome Protocols defining when and how data will be collected Analysis & interpretation procedures Confounding factors & approaches for minimizing their

impact Formative evaluation techniques for monitoring and

improving the project as it evolves Summative evaluation techniques for characterizing

the accomplishments of the completed project.

PDs’ Response

Evaluation Plan

Page 57: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Workshop on Evaluation of Educational Development Projects◦ http://www.nsf.gov/events/event_summ.jsp?cntn_id=108142&org=NSF

NSF’s User Friendly Handbook for Project Evaluation ◦ http://www.nsf.gov/pubs/2002/nsf02057/start.htm

Online Evaluation Resource Library (OERL)◦ http://oerl.sri.com/

Field-Tested Learning Assessment Guide (FLAG)◦ http://www.wcer.wisc.edu/archive/cl1/flag/default.asp

Student Assessment of Their Learning Gains (SALG)◦ http://www.salgsite.org/

Science education literature

Other Sources

Page 58: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Identify the most interesting, important, or surprising ideas you encountered in the workshop on dealing with project evaluation

Take ---- 4 min◦ Think individually -------- ~2 min◦ Report in local group ---- ~2 min

Watch time and reconvene after 4 min

Use THINK time to think – no discussion, Selected local facilitators report to virtual group

Activity

Final Reflection

Page 59: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

One MinuteLook at Chat Box to see if you will be asked to respond

Page 60: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

Questions

Hold up your “virtual hand” to ask a question.

Proposal & Award Policies & Procedures Guidehttp://www.nsf.gov/pubs/policydocs/pappguide/nsf11001/

Page 61: Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis

To download a copy of the presentation- go to:http://www.step.eng.lsu.edu/nsf/participants/

Please complete the assessment survey-go to: http://www.step.eng.lsu.edu/nsf/participants/

Thanks for your participation!