1

Monitoring & Evaluation

Plan Bea

iSTEP 2009

2

Monitoring – Why?• Are we on the right track?

• Do we need corrective action?

• What are the problems?

• Are we wasting resources/time on something?

• What are we missing?

3

Monitoring – What?

Activity/Project

What to monitor

Needs Assessment End user reactions / feedback

Cell Phone ProjectTechnology progress & compatibility with needs

Literacy ProjectTechnology progress & compatibility with needs

Braille Tutor ProjectTechnology progress & compatibility with needs

Documentation / Dissemination

Updates to blogs, journals and other media

iSTEP Student Learning

Lessons learned so far – introspection & group discussion

4

Monitoring – How?

• Weekly group “huddles”

• Plan vs. Actual comparison

• Feedback from partner (Eric Beda) – possibly weekly

• End user interactions – as many as possible

• Reality checks

5

Evaluation – Why?• My thesis depends on it

• Understand whether we are making an impact

• Measure significance of impact

• Identify areas for improvement

• Produce transparent results to stakeholders

• Make necessary adjustments

6

Evaluation – When?• Phase 1: Pre-Intervention Measurements

– What is the current status?

• Phase 2: Intermediate Assessment– Are we making progress toward our goals?

• Phase 3: Final Evaluation– Were we successful?

• Final Phase: iSTEP Overall Assessment– Did iSTEP students gain from the experience?

7

Evaluation – What?Activity/Project

Metric(s)

Cell Phone ProjectNumber of new entries into database, before, during and after intervention

Literacy ProjectStudent test scores before, during and after intervention: tests may include spelling, reading and writing

Braille Tutor ProjectStudent test scores before, during and after intervention: tests may include writing and reading in Braille language

CampaignNumber of comments on FB page, Hits on website, Amount of blog readership, Twitter activity, etc.

8

Evaluation – How?• Collect data!

• Questionnaires/Surveys to end users

• Field testing of technology

• Continuous interviews/conversations with partners and end users

• Determine, discover, and redefine metrics as necessary

9

Potential Difficulties• Data may be scarce and difficult to obtain• Evaluation settings will not be ideal

– Lots of confounding factors! – Statistical testing might be difficult

• Participation in surveys and questionnaires might be low• Language and/or cultural barriers• Time management will be tough – setting things up will be time-consuming• May not obtain accurate responses from end-users• Traveling to and from project locations could be costly

– Might limit interactions with end-users

• Unexpected delays and issues may throw off evaluation

10

Questions? Comments?

Thank you for listening

Suggestions Welcome and Appreciated!