Welcome2013 HIV/AIDS Community Information Outreach Program Evaluation Webinar

Online HIV/AIDS Resourceshttp://sis.nlm.nih.gov/outreach/training_outreach.html

Presenters

Arletha Williams-Livingston, Ph.D., MPHEvaluation and Organizational Capacity-building ConsultantTransforming Communities, Inc.transformingcomm@gmail.com

LaFrancis McMurray, MPHHealth Education SpecialistOak Ridge Associated Universitieslafrancis.mcmurray@orau.org

PurposeThe purpose of this webinar is to build organizational evaluation capacity, skills and knowledge, to discuss program evaluation requirements and to connect participants to evaluation technical assistance.

ObjectivesUnderstand the importance of program evaluation

Effectively evaluate ACIO projects

Access evaluation technical assistance as needed

Today’s TopicsWhat is Evaluation?Why Evaluate?Building Organizational Evaluation CapacityACIO Evaluation Requirements, Resources and Technical Assistance

What is Program Evaluation?Program

– Your campaign, action, training, interventionEvaluation

– Is the systemic investigation of the merit, worth, or significance of your work

Program Evaluation – Is a way to evaluate the specific projects and activities

of community groups and organizations

In other words…Program Evaluation is a combination of:

– Good Reporting– Good Detective Work– Good Historical Documentation and Storytelling

Program Evaluation helps organizations see how and in what ways they are really helping the community, OR how and if an organization is achieving its goals and objectives.

Three Main Types of Program Evaluation

Process EvaluationOutcome Evaluation Impact Evaluation

There are many more…

Process EvaluationDescribing and documenting the course of a program For Example:

– Who are the Members or Participants?– What activities were conducted?– How many people were in attendance?– When and Where are events and meetings being held?– What staff was there?– Who was involved in the planning?– Who was involved in the implementation?

Outcome EvaluationWhat were the immediate effects of the program?Changes in skills, knowledge, attitudes, and behavior For example:

– Did the participants learn anything?– Do the participants have a different attitude?– Do the participants do something differently or do

something new?– Was there a campaign victory?

Impact Evaluation What were the ultimate effects on the community?For example:

– How was the community affected? What were the effects?

Three types of Impact evaluation– Immediate (more like an outcome)

• Knowledge of housing shortage issue– Intermediate

• Advocacy work and heightened policy making and community awareness of housing issue

– Ultimate • Decrease in percentage of the target population who are in need of housing assistance

Why Evaluate?

Evaluation Is Used for:Program Development Mass Campaigns Policy FormationsIssue AdvocacyOrganizing Communities Funding Applications

Program Evaluation is used to…Determine if goals were met

Show effectiveness and impact of a program

Provide evidence of the process and outcomes

Questions to Ask When Developing an Evaluation Plan

Who is the Evaluation for?

What do we need to find out?

Why do we wantto find that out?

Where should we gather the information?

When will the findings be needed?

How will the results be used?

Building Evaluation Capacity

Capacity Building Values

Interdependence

Cultural diversity

Contextualized interpretation

Empowerment

ACIO Information Access Categories Evaluation Information

MeasuresQuestionsToolsResources

Information Access Category I Information Retrieval

Information RetrievalTo provide or improve access to electronic AIDS- and other health-related information resources by the organization or by the clients they serve and to implement access to state-of-the-art AIDS-related information tools via the Internet and/or mobile phone technologies. This may require the purchase of hardware (computer equipment), software, or contractual relationships necessary to obtain a connection to the Internet for the purpose of accessing AIDS-related information or sharing AIDS-related information.

Project ExamplesOnline living library databaseInformation retrieval skills development training for health outreach workers and local community based organizationsIncrease access to critical information for at risk youth using a multimedia hub-spoke model of online, social media and mobile platformsMobile-Internet-based HIV/AIDS Education: Linking the Public to SciencePurchase of computer equipment/software for cyber lab

MeasuresAwareness

Marketing/awareness of informationCommunity partnerships/community engagement

AccessRetrieval skills Access to computers/mobile/Internet

Proof of access/number of visits/types of resources

Evaluation QuestionsAwareness

In what ways is information advertised to potential consumers/clients?How are community partnerships engaged to help provide information access to residents?

AccessHow often are clients retrieving information?How are clients accessing information via computers, Internet and mobile?How many times & which types of resources clients are accessing?

Sample Evaluation ToolsAwarenessMarketing plan objectives, outcomesNumber of MOAs/MOUsNumber & quality of community partnership/community advisory board meetingsObservations of community partner engagementAccessSocial Media monitoring tools i.e. Tweetdeck, Website analytics i.e. Google Analytics

ResourcesSocial Media/ Health communications planning with evaluation componenthttps://www.healthcommworks.org/Account/LogOn?signInArea=SocialMediaWorks

US Department of Education Evaluating Online Learningwww.ed.gov/admins/lead/academic/evalonline/evalonline.pdf

Evaluation strategy for mobile device learninghttp://www.salt.org/salt.asp?pn=procabs&ss=l&key=153320&pid=2406

Information Access Category IISkills Development

Skills DevelopmentTraining to develop skills in accessing or using HIV/AIDS-related information including the use of NLM online databases and mobile resources as well as general AIDS information resources in order to improve services and benefit the community. This may include training for the staff of the agency, development of training programs for clients and patrons, training programs for other agencies, etc.

Project ExamplesCapacity-building training for youth serving professionals; and peer-to-peer educationInformation retrieval skills development training for health outreach workers and local community based organizationsTraining classes and workshops on NLMs online resources and social media sitesTraining for new HIV positive peer educators/counselors as online instructors and facilitate new training classes featuring health literacy

MeasuresTrainingDevelopment of trainings/types of: access/utilization/ general computer skills/database trainingTraining target populations: staff/clients/ patrons/agencies/special populationsNumber of trainings/trained usersTraining satisfactionTraining outcomes: knowledge/skills

Measures (cont’d)UtilizationTypes/amount of resources/information utilized: Internet/social media/email/mobile/brochures/factsheets/face-to-faceUtilization outcomes: Knowledge/skills/empowerment/efficacy/service improvement/community outcomes

Evaluation QuestionsTrainingWhat types of trainings are being developed?Who is the target training population?How many people are being trained? (Attendance/completion)Are consumers satisfied with trainings?What are the training outcomes/skills developed?UtilizationWhat type of information consumers are utilizing?How is that information being utilized?What are the outcomes associated with consumer utilization of information resources?

Sample Evaluation ToolsTrainingTraining Development TemplateSign-In sheetsPre/Post assessmentsSatisfaction surveysUtilizationSocial Media monitoring tools i.e. Tweetdeck, Website analytics i.e. Google AnalyticsQuestionnaires, Surveys, Qualitative analysis of responses

ResourcesTraining development templateSample pre/post skills assessmentsWriting measurable learning outcomes

http://www.gavilan.edu/research/spd/Writing-Measurable-Learning-Outcomes.pdf

Evaluating Training Programs-Kirkpatrick’s 4 levelshttp://www.wa.gov/esd/training/toolbox/tg_kirkpatrick.htm

Information Access Categories III and IVResource DevelopmentEquipment Acquisition

Resource DevelopmentTo support an organization to meet the needs of their clients or communities by developing specific educational or informational materials such as culturally or language specific fact sheets, bibliographies, guides or other products.

Equipment AcquisitionTo improve access to electronic AIDS and other health-related information resources which may require the purchase of hardware (computer equipment), software, or contractual relationships necessary to obtain a connection to the Internet for the purpose of accessing AIDS-related information or sharing AIDS-related information.

Project ExamplesProduce and syndicate culturally and linguistically appropriate messagesReady Reference Guides to person with HIV/AIDS or at high risk for HIVCulturally competent and age appropriate information in English and Spanish emphasizing prevention, risk reduction and skills buildingDevelop and facilitate access to factsheets on the treatment of HIV/AIDS and related information written in nontechnical language and translated into several languages

MeasuresResource needs Target population needs assessment/types of resources neededResource developmentResource materials created: type such as cultural specific, bibliography Resource materials utilized/sustained/maintained/updatedPurchases of hardware/software to develop resourcesUtilization of consultants/contractual relationships to develop resources AccessPurchases of hardware/software to access and share informationUtilization of consultants/contractual relationships to access and share information

Evaluation QuestionsResource needs Has a resource needs assessment been conducted for the target population?If so, what were the outcomes of that needs assessment?Resource developmentWhat types of resource materials are being developed?How are the resource materials being utilized/sustained/maintained/ updated?Was hardware/soft purchased to develop resource material? If so, what kind and how was it used?Were consultants/contractual relationships used to develop resource material? If so, who and what work did they complete?

Evaluation Questions (cont’d)AccessWas hardware/soft purchased to help agencies provide access/share materials? If so, what kind and how was it used?Were consultants/contractual relationships used to help agencies provide access/share materials? If so, who and what work did they complete?

Sample Evaluation ToolsResource needs Outcomes of Community/resource needs assessmentResource developmentResource development templateResource material tracking (How utilized/sustained/maintained/updated)Comparison checklist for equipment/software purchasesConsultant/contractual reportsAccessEquipment/software inventoryUtilization logConsultant/contractual reports

ResourcesTeamLab-University of Southern California

Designing materials for particular audiences/understanding culture http://teamlab.usc.edu/learn/roadmap/audience.html

CenterLink-The Community of LBT CentersConducting Needs assessmenthttp://www.lgbtcenters.org/how-to-do-a-community-needs-assessment.aspx

Resources (cont’d)Harvard Family Research Project Evaluation Exchange, Understanding Cost-Benefit and Cost-Effectiveness Evaluationhttp://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/methodology-15/at-what-price-benefit-cost-analysis-and-cost-effectiveness-analysis-in-program-evaluation

TechSoup.orgWebsite for Non-Profits and technology. Planning and how to articles

http://www.techsoup.org/support/articles-and-how-tos/articles-by-subject?subjectid=14

ACIOP Evaluation RequirementsQuarterly reportsFinal reportsIndividual agency evaluation conference callsEvaluation technical assistance

What’s next?Individual agency calls to discuss:

– Agency Evaluation capacity– Project evaluation plans– Technical assistance

Mid Year Webinar – Milestones– Evaluation needs of agencies– Final reports

Q & A

Contact Information

Arletha D. Livingston, Ph.D., MPH, CHESEmail: transformingcomm@gmail.comPhone: 404.368.9252