lingkungan 3

8/3/2019 lingkungan 3

1/12

Using evaluation to adapt health information outreach tothe complex environments of community-basedorganizations*!By Cynthia A. Olney, [email protected] ConsultantP.O. Box 4891Greensboro , North C arolina 27404

Objective: After arguing that most community-based organizations(CBOs) function as complex ada ptive system s, this white pape rdescribes the evaluation goals, questions, indicators, and methods mostimportant at different stages of community-based health informationoutreach.Main Points: This paper presents the basic characteristics of complexadaptive systems and argues that the typical CBO can be consideredthis type of system. It then presents evaluation as a tool for helpingoutreach teams adapt their outreach efforts to the CBO environmentand thus maximize success. Finally, it describes the goals, questions,indicators, and methods most important or helpful at each stage ofevaluation (community assessment, needs assessment and planning,process evaluation, and outcomes assessment).Literature: Literature from complex adaptive systems as applied tohealth care, business, and evaluation settings is presented. Evaluationmodels and applications, particularly those based on participatoryapproaches, are presented as methods for maximizing the effectivenessof evaluation in dynamic CBO environments.Conclusion: If one accepts that CBOs function as complex adaptivesystemscharacterized by dynamic relationships among many agents,influences, and forcesthen effective evaluation at the stages ofcommunity assessment, needs assessment and planrung, processevaluation, and outcomes assessment is critical to outreadi success.

For several months, an outreach team from a SouthTexas medical school library carefully planned a com-munity outreach training session and focus group tobe held at a community center serving a low-incomeHispanic colonia (community) in the Lower Rio GrandeValley. The outreach team developed relationshipswith community center staff and engaged their assis-tance in the planning. The team offered preliminarytraining to residents of the com munity, generating en-thusiasm and engaging som e of them as outreach part-ners. These partners, in tum, personally invited othercommunity members to the training session and focus* The outreach projects discussed in this paper were carried out in2001-2003 and were funded by the National Library of Medicineunder contract (#NO-1-LM-1-3515) with the Houston Academy ofMedicine-Texas Medical Center Library.t This paper is based on a presentation at the "Symposium on Com-munity-based Health Information Outreach"; National Library ofMedicine, Bethesda, Maryland; December 3, 2004.

group. The team p urchased gift cards from a local gro-cery store as incentives to attract participants andhired an experienced focus group facilitator fromSouth Texas to conduct the session in Spanish.The final session was scheduled for ten o'clock on aweekday morning in mid-March. When the outreachteam arrived at the site, commuruty center staffwarned that attendance probably would be very lowbecause of bad weather: the temperature had dippedto thirty-seven degrees. Outreach team members, par-ticularly those who had grown up in middle-classneighborhoods in northem climates, could not imag-ine how plans could fall apart over such weatheradry day, with temperatures above freezing. However,the commuruty staff predictions were accurate. Per-hap s because the families lacked adequate clothing orhousehold heat, this weather was bad enough to keepthe children home from school and mothers from at-tending the training session. It was, in fact, a slow dayfor outreach.


2/12

Olney

Most would call this an example of Murphy's Law:what can go wrong, will go wrong. I t a lso demons tra tedthe unpredictable nature of the environments of com-munity-based organizations (CBOs). Funding sourcescome and go. The turnover of low-paid and volunteerworkers tends to be high. Changing politics lead tochanging priorities. The involvement of and relation-ship between stakeholders can make decisions andprogress difficult. For instance, Scherrer outlined thechallenges faced in an environmental public healthoutreach project in Chicago: lack of institutional sup-port of Intemet technology, participants unwilling toshow lack of knowledge until they knew and devel-oped trust in the outreach coordinator, difficulty inbuilding cooperative relationships among volimteers,and unique culture of each organization [1].Such challenges make outreach, as well as evalua-tion, quite difficult. Evaluation methodologies toutedas yielding the most credible findings rely on outreachactivities that progress in a predictable, linear fashion,with carefully executed interventions (independentvariables), explicitly defined outcomes (dependent var-iables), and tightly controlled extraneous variables.Many consider experimental and quasi-experimentalresearch methodologies, control groups, randomizedsamples, and standard experience across groups nec-essary elements of good research design, yet a typicalCBO outreach project seldom can meet all or any ofthe ideal research conditions. Furthermore, the focusednature of such methods often misses some of the un-expected positive outcomes. On the other hand, whenpositive results are not obtained, such me thods seldomhelp explain why.The first step in conducting better evaluations ofCBO-based information outreadi is to understand andaccept the nature of the environment in which out-reach activities are conducted. This paper presents thebasic characteristics of complex adaptive systems, aconcept rooted in complexity theory, and argues thatmost CBOs can be considered complex adaptive sys-tems. It then argues that evaluation is a tool for help-ing teams adapt their outreach to the system and thusto maximize their success. Finally, it describes thegoals, questions, indicators, and methods that mayprove most helpful at each stage of evaluation: com-munity assessment, needs assessment and planning,process or formative evaluation, and outcomes evalu-ation.BACKGROUND

The impetu s for this pap er came from the author's ex-periences as an evaluation specialist with the libraryoutreach staff at the University of Texas Health ScienceCenter at San Antonio (UTHSCSA) Libraries locatedin San Antonio and Harlingen, Texas, on the TexasLower Rio Grande Valley Health Information HispanicOutreach (HI HO) Project [2, 3]. In September 2001,the UTHSCSA Library received a contract from theNational Library of Medicine (NLM) to develop anoutreach prog ram in the medically underserved Lower

Rio G rande Valley. NLM was p articularly interested inoutreach to communities in the valley because it wasabout to launch a new version of MedlinePlus, featur-ing Spanish-language consumer health care informa-tion. The Lower Rio Grande Valley, with its high pro-portion of Spanish or bilingual residents, seemed anideal community of users for MedlinePlus en espafiol[2, 3] .The outreach team, consisting of UTHSCSA librar-ians and an evaluation specialist, conducted an exten-sive community assessment of the Lower Rio GrandeValley community and settled on four sites for con-ducting outreach: a health careers high school, thecommuruty center mentioned above, and two medicalclinics with a patient base of primarily low-incomeHispanic families. At the high school, the team devel-oped a peer tutor program in partnership with theschool library, with four stu dents as the primary train-ers of MedlinePlus. At the clinics, we targeted our out-reach efforts toward diabetes educators, assuming thatthey would use the databases to access health infor-

mation about diabetes as part of their instructional re-sponsibilities, particularly for their Spanish-speakingclients. Our key partn ers at the comm unity center werepromotoras, residents of the community (all women inthis case) who had received training about local healthand social services. The promotoras are the point per-sons in their community for helping residents get as-sistance and information. The promotoras in this out-reach project had been trained through the TexasA&M University Center for Housing and Urban De-velopment (TAMU-CHUD), which coordinates thepromotora program as part of a broader colonia deveopment effort along the Texas-Mexico border The out-reach staff used four criteria to evaluate outreach suc-cess: (a) members of the community who served ateach site spontaneously used MedlinePlus either forwork or for personal use; (b) people who participatedin CBO activities used MedlinePlus in the absence ofmembers of the library outreach team; (c) CBO partic-ipants who became our key contacts, like promotorasand peer tutors, generated ideas for introducingMedlinePlus into their organization or ideas for teach-ing more members of the community to use the da-tabase; and (d) key contacts at the sites reported thatthey were teaching others to use MedlinePlus. Usingthese criteria, the outreach team judged two of thesites (the high school and the community center) ashighly successful but saw the other two sites as lesssuccessful.

We have since received funding to expand our out-reach efforts. In 2003, we received NLM contract mon -ies to extend our work w ith Hispan ic comm unities likeCameron Park in the Texas-Mexico border areas ofHarlingen-Brownsville, Laredo, and El Paso. In Sep-tember 2004, new NLM -funded projects w erelaunched to extend the high school outreach project toother Texas schools in the Lower Rio Grande Valley,San Antonio, and Laredo. We leamed fundamental les-sons on our first project that have guided our currentproject evaluations. These lessons helped us improve


3/12

Complex enviroments of CBOs

our community and needs assessment, process andformative evaluation, and outcomes assessment. Wehave gotten better at outreach, in part because w e haveleamed to adapt to the complex environments of ourCBOs using evaluation methods.COMMUNITY-BASED ORGANIZATIONS (CBOS)AS COMPLEX ADAPTIVE SYSTEMSThe concept of complex adaptive systemswhichcomes from a body of literature known as complexityscience, chaos theory, or network scienceis being ap-plied in areas of study as diverse as physics, engi-neering, terrorist network activity, and the spread ofinfectious diseases [4]. The concept of complex adap-tive systems has become increasingly popular in theorganizational literature as a basis for understanding,coping with, and succeeding in the chaos of businessenvironments [5-7]. In health care, complex adaptivesystem theory has been applied to describe patterns ofbehavior in clinical practices [8, 9]. The literature gen-erally describes the following key traits of complexadaptive systems. Complex adaptive systems feature an entangled xveb ofrelationships among many agents and forces, both internaland external. These influences cause constant change, ad-aptation, and evolution of the system in an unpredictable,nonlinear manner. This interconnectedness of the ele-ments in the system can lead to unpredictable reac-tions systemwide, with small events sometimes havingbig impacts, while large, planned efforts may have lit-tle effect at all. Complex adaptive systems are self-organizing. Pat tem sof behavior are not created from top-down policy.They emerge through a complicated system of rela-tionships, influences, and feedback loops inside andoutside the system. Thus, change in a system cannotbe forced; it must be shaped. Complex adaptive systems do not move predictably towardan end goal. Timelines for outreach activities must beflexible, because bursts of activity around a given pro-ject may be followed by periods of dormancy. Unex-pected developments can either enhance or thwartplans. In fact, the evolutionary nature of a complexadaptive system means that there is no end goal. Whatprogress may have been accomplished in a given out-readi project may disintegrate rapidly as conditionschange. Communication is heaviest at the boundaries of a system.Boundaries exist between two different parts of a sys-tem that must adjust to and with each another. For ex-ample, in our community center projects, boundariesexisted between community center staff hired by coun-ties, the promotoras, and TAMU-CHUD coordinatingstaff Boundaries are not good or bad; they are simplyparts of the system that generate communication. Forthis reason, they often are an excellent source of infor-mation about where outreach activities might be effec-tive in the system. Systemwide patterns of behaviors ca n be observed. A par-adox of complex adaptive systems is that, while they

can be dynamic and changeable, they also can dem-onstrate systemwide pattems of behavior, generatedby variables known as attractors, which will be re-peated at many levels of the system and can be diffi-cult to alter. Attractors may be written organizationalpriorities, seasonal events like year-end fiscal budgets,or more erratic variables like the day-to-day needs ofthe clients served by a CBO. Outreach staff are likelyto design effective outreach projects if they can rec-ognize the recurring pattems and work with them. Feedback loops are the mechanisms for change in a sys-tem. Feedback loops carry information, material, andenergy among agents in the system. If feedback loopsare well designed, they facilitate change and adapta-tion of the system. Patterns of behavior may repeat themselves at different levels of a system and across systems. In a complex adapt ivesystem, different attractors stimulate similar responsesand those responses will be observed at different levelsand parts of the system. These patterns are known asfractals [10]. While behavior is never exactly the same,similarities will occur across parts of the system,across systems, and across time.DIFFUSION OF INNOVATION: A USEFULFRACTALWhile complex adaptive systems are characterized bytheir uniqueness, clynamism, and changing nature,fractals emphasize similarities in system behaviorsthat can be observed and used. Because these pattemsexist, published reports of outreach projects becomecrucial to all who are trying to conduct similar out-reach projects, because they may provide useful back-ground information for project planning.While fractals will be somewhat system specific, thewell-documented diffusion of innovation theory de-scribes one practical fractal observed when commu-nities adopt new innovations. This fractal can be veryuseful for health information outreach [11]. (See Bur-roughs and Wood for a discussion of applying diffu-sion of innovation and other social theories to outreachplanning [12].) Charles and Henner [13] give an ex-ample of how a project used this theory as the basisof a training program to enhance health professionals'use of Intemet health resources. The project recruitedand trained innovators in the use of Intemet resources,treating them as potential change agents for how otherhealth professionals in the state would eventually ac-cess public health information.

In our four sites, diffusion of innovation did a re-markably good job of describing the pattems of adop-tion of MedlinePlus or lack thereof Table 1 comparesthe key features influencing diffusion of innovation atour four sites. For example, at our high school site, ourinnovators were school librarians and students, be-cause the relative advantage of MedlinePlus was superioto other search methods, they used to access healthinformation (e.g., Google, print resources). Complexitywas low because the stakeholders had Intemet expe-rience, and compatibility was high, because they often


4/12

Olney

Table 1A comparison of community-based organization (CBO) sites' readiness for MediinePlus using diffusion of innovation criteria

Criterion High school Community center CiinicsInnovators (first wave of users to adopt an innovation)Relative advantage of online resourcesComplexity of using online resourcesCompatibility of using online resources with other values, habits, needs,

and experiencesTrialability of online resources forexperimentation before commitment toadopt

Observability of resources to provide tangible results

IdentifiedSuperiorLowHighHighHigh

IdentifiedSuperiorHighModerately lowModerateHigh

Not successfully identifiedLess convenientHighLowLowLow

searched the Intemet for information as part of theirday-to-day activities. They had good opportunity towork with MedlinePlus {trialability) because of thetechnology available in the school, and, once they re-alized teachers wo uld accept MedlinePlus materiajfs forresearch projects, both librarians and students ob-served rewards (locating acceptable information quick-ly either for school clients or for research).Diffusion of innovation also explains the comm unitycenter promotoras' interest in MedlinePlus. First, theyhad no other convenient access to health information(relative advantage). Their role was to provide individ-ualized assistance to community residents, so accessto such an expansive health database was compatiblewith their values, needs, and habits. They saw tangiblerewards when using MedlinePlus (observability) in thatthey could quickly provide help for residents' healthconcems and sometimes relieve residents' anxiety.However, two barriers made our work with the prom-otoras more challenging than our work with students.First, the trialability was lower at the community cen-

ter, because Intemet access was unreliable. This prob-lem was resolved later in the project when promotorassecured laptops and Intemet wireless cards through agrant. Second, the complexity of Intemet usage washigher for promotoras than students because promotorashad less experience with computers.Analysis of our evaluation data at the end of theproject indicated that our clinic sites were at differentlevels of "readiness" for introduction of MedlinePlus.The diabetes educators, whom we thought would beinnovators, actually showed little motivation to leamMedlinePlus or show it to patients. They did not ap-pear to see a relative advantage of MedlinePlus overthe educational materials they already po ssessed. Nei-ther diabetes educator at either clinic seemed comfort-able with computers nor used computers or the Inter-net, so the process of using MedlinePlus was incom-patible with their habits andcomplex for them to un-derstand. While they had access to computers at theclinics to try out the online resources, their lack ofcomfort with computers made them unlikely to ex-plore the database. They saw no visible rewards forusing the Website. In fact, their own lack of computerexperience and comfort and their perception that theirpatients would share their reticence made them believethe online resources had greater cost than benefit.Consequently, MedlinePlus never "diffused" exten-

sively into these particular clinics during the projecttime frame.We leamed that diffusion of innovation was so de-scriptive of the patterns of adoption at our sites, weneeded touse it to ge t a better u nderstanding of com-mu nities' readiness for innovation before starting otheroutreach projects. We designed the guide sho wn in theappendix to help us with community assessment inother projects.CBOS AND PROGRAM EVALUATIONIf one assumes that CBOs often have characteristics ofcomplex adaptive systems, evaluation methods for as-sessing the complex environments of community-based organizations must be efficient and effective.They must be useful to those implementing the out-readi program and flexible enoug h to adapt to thechanging CBO environment. The methods must beparticipatory so that local stakeholders can help inte-grate outreach efforts into their orgaruzations. Finally,evaluation must answer the questions and goals spe-cific to each of the four stages of evaluation: commu-nity assessment, plarming and needs assessment, pro-cess evaluation, and outcomes evaluation.Community assessmentIf CBOs often function as complex adaptive systems,then the convmunity assessment stage is critical. Table2 presents the key questions to be addressed duringcommunity assessment and methods for doing so. Par-ticipatory methodologies are essential because stake-holders from different parts of the system are broughtinto thediscussion. Rapid rural appraisal (RRA, alsoknown as rapid appraisal) comprisesflexible, nexpen-sive methods that provide a great deal of informationabout a comm unity (see the USAID's publication Con-ducting a Participatory Evaluation for descr ip t ions of r a p -id appraisal methods [14]). RRA has been used effec-tively in situations where collection of quantitativedata leads to inissing orunreliable data [15]. While themethods vary, they commonly involve unstructuredinterviews with key informants representing variousstakeholder groups.Because of the quick, loosely controlled procedures,rapid appraisals m ay lack credibility in some researchand evaluation circles. However, a study comparing


5/12


Table 2Evaluation during community assessmentGoals Methods Questions to ask or observations to make

Identify innovators and opinion leaders

Assess boundaries: how weli do dif-ferent parts of the system coordi-nate and where does communica-tion or function breakdown?

Identify impo rtant stakeholder groups

Assess environmental "readiness"

Present the innovation and in-terview stakeholders either in-dividually or in focus groupsWho suggests ways to appiy the innovation to personal concems or pro-fessionai responsibiiities?Who taiks about how the innovation will benefit the overall community?

What are the issues among different stakeholder groups (i.e., at theboundaries)? Can the innovation resolve those issues? Will the issue af-fect how your outreach is conducted?Whom do the innovators and opinion leaders see as key groups that wiilbenefit from the innovation? How do they propose that it be used?How "ready" is the community to leam about the innovation?

the accuracy of rapid appraisal w ith conventional sur-veys in identifying people v^ith disabilities in a SouthIndian community showed that both approaches wereequally accurate in identifying commimity memberswith disabilities [16]. RRA, however, provided a moredescriptive view of how com munity m embers talk and\hmk about disability. RRA also permitted the re-searchers to educate the respondents throughout thedata collection process.For evaluation projects oriented toward introducinginnovations into communities, community assessmentmust be designed to identify the innovators in thecommimity and their relationship to others. The prom-otoras, for instance, immediately saw the potential forMedlinePlus to help them meet a need. Before leaminghow to access MedlinePlus, the promotoras had limitedresources for helping residents who came to them w ithhealth problems. Their options before MedlinePlustraining were to make appointments for communityresidents at clinics or to contact a local agency insearch of printed information. MedlinePlus gave themimmediate access to information for community resi-dents with health care concems.Evaluation also must be designed to bring stake-holders from different parts of the system together, sothat interaction of CBO community members acrossboundaries can be observed. As mentioned above,boundaries separate different parts of a system, andcommunication occurs across boundaries to help co-ordinate the system. Thus, group discussion formatsare also likely to identify stresses, tensions, and prob-lems that the innovation can relieve. For instance, abou ndary existed between studen ts and teachers at ourhigh school site. Students preferred the convenience ofaccessing information through the Intemet, but teach-ers did not trust Intemet sources and feared that stu-dents could more easily plagiarize online materialsthan printed ones. This tension, though subtle, was ob-served through teacher and student focus groups.Thus, we knew our training would be most rapidlyadopted by students, but we would have to train, andreassure, teachers to realize the potential ofMedlinePlus in the school community.It is also important to assess the readiness of stake-holders and of the CBO itself in terms of the charac-

teristics that facilitate diffusion of innovation into com-mimities. As we planned outreach, we consideredMedlinePlus as a tool prim arily for CBO staff and theirclients to gather information. Given that, we needed toknow how they currently get information and ifMedlinePlus offers advantages over their usual strat-egies. With the trialability dimension, we had to besure that, once we demonstrated MedlinePlus ortrained people in its use, they had access to technologythat would allow them to use their new skills andknowledge. We struggled with cormectivity problemsat our community center (both from the HI HO projectand subsequent projects), especially after the stateeliminated the technology infrastructure funds thathad provided computer technology and online accessfor many CBOs. At the clinics, computers were avail-able bu t were no t on the desk top s of all staff mem bersuntil later in the project.

Finally, we needed to understand how, and howquickly, people we trained might be able to describethe rewards of using the innovation. In early demon-strations at our sites, the high school and commurutycenter groups immediately talked about howMedlinePlus would help their organizations. At theclinics, staff were less able to imagine ways thatMedlinePlus would help them with their responsibil-ities toward patients. They were quite willing to allowus to work directly with patients in their waitingrooms, but they believed their patients would need ex-tensive assistance in using co mputers and they did notfeel they had the staff to provide that support. In otherwords, the staff did not see immediate observable re-wards in adopting the use of MedlinePlus. Conse-quently, MedlinePlus was never integrated into theclinics as it was at the school and community centersites.Because our funding for outreach is limited, wemust choose our sites wisely. We believe our most cost-effective outreach will occur at sites where we suc-cessfully locate community partners (innovators) whowill pass along their knowledge to others in the com-munity or help others access online resources. Ourstandard method now for assessing the community isfirst to m eet the people who run the orgaruzations andthen ask them to assemble people they think would be


6/12

Olney

Table 3Evaluation during needs assessment and pianningGoals Methods Products

Determine community needs asseen by stakeholders Deveiopment of iogic models

Assess shortcomings or tensions Empowerm ent evaluation methodswithin the system to see if theinnovation can resolve or reducethose problems

Stakeholders and outreach staff togefherwill develop a logic model showingthe connection between resources, activities, outputs, and outcomes relatedto the outreach effortsStakeholders and outreach staff will identify their criteria for evaluating successof their program and will identify the community's baseline standing in terms

of meeting those goais

interested in a short MedlinePlus training. We offerdemonstrations to different groups, asking questionsof participants and carefully observing their reactionsand enthusiasm. By developing a conimunity profileusing the guide, we can assess the readiness of theCBO to introduce MedlinePlus into the community itserves. We ask those attending the demonstrations toidentify the ongoing activities of the organizationwhere training sessions or one-to-one help can be pre-sented. This methodology helps us discem the inno-vators, who will become our partners in presentingMedlinePlus to the community. If those attending ourdemonstration show little interest in becoming activepartners in outreach, we are reluctant to use that CBOas a site because we believe the community membersare not yet ready to leam about and integrate the in-novation.Needs assessment and planningWe have leamed that our outreach efforts will workmuch better if the community members tell us how wecan meet their needs. Table 3 shows the goals of needsassessment and planning, along with the methods forconducting this stage of evaluation. Ideally, we woulduse participatory approaches in the needs assessmentand planning stages of outreach (see, for instance, theguid e pre sente d at the W. K. Kellogg Foundation Web-site [17] or Fetterman's empowerment evaluation mod-el [18]). For either of these methods, stakeholders as agroup set objectives, determine criteria for success,and, in the case of empowerment evaluation, deter-mine how to evaluate it.In real life, participatory methods like this take a lotof organization and are difficult to sustain over time.So we use rudimentary versions of these techniquesthat take less time from community members. We al-ways try to schedule discussion sessions at our initialdemonstrations, asking questions like "How can thisproduct help the residents?" "What classes are youcurrently teaching that could incorporate this tool?" or"How would you know whether or not the communitywas benefiting from this product?" We then organizethe plan based on their comments. The comm unity as-sessment guide (Appendix) is useful during the needsassessment and plarming phase as well.We also ask conununity members how they willknow that MedlinePlus has helped the participants inthe community or organization. This question is de-signed to help us identify indicators and methods for

outcome evaluation that our stakeholders will findcredible.Process evaluationStakeholders' initial enthusiasm for MedlinePlus is notsufficient to ensure project success. Outreach teamsneed to have metho ds for mo nitoring the changing en-vironment of CBOs so they can adapt their plan asneeded. For instance, just as our outreach staff had de-veloped a functional schedule for one-to-one outreachactivities with patients in one of the clinic waitingrooms, the clinic sudden ly introduced a new computersystem into its previously low-tech environment andshut do wn for weeks to train staff. We needed to adaptto incorporate this sudden change in plan. Table 4identifies the goals and methods for process evalua-tion.In monitoring our CBO environments, we kept rec-ords of contacts and attendance at different sites.Along with record ing num bers of interactions that ouroutreach team had with participants during each visit,our outreadi staff also recorded descriptions of partic-ipants' reactions to MedlinePlus. This allowed us toassess psychological barriers patients had toward theIntemet and computers.Our other key monitoring methods included sitevisits and interviewing. The evaluator interviewedmem bers of stakeholder g roups as well as members ofthe outreach team frequently and informally. Themonitoring interviews sought to answer two basicquestions: "How are you and the community usingMedlinePlus?" and "What barriers are you facingwhen trying to use it or teach it?" This method, forinstance, revealed the difficulty the promotoras had infinding the "en espan ol" button on the monitor screen.The outreach librarian also leamed that one promotoradid not have a concept of portals. Once the promotoraleft the MedlinePlus Website to go to another site, shedid not know how to navigate back to MedlinePlus, soshe would turn off the computer and start over again.The important point here is that the whole outreachstaff was instrumental in collecting information tohelp monitor and document progress and barriers.The key was to give them two simple evaluation ques-tions to monitor the site: "How are the stakeholdersusing MedlinePlus?" and "What barriers are gettingin the way of our outreach efforts?" They then keptjoumals of their contacts with stakeholders to captureinformation regarding these two questions. If outreach


7/12


Table 4Process evaluationGoals Methods Issues to address

Determine if the innovation is being adoptedwithin the system

Observe if those trained arestarting to trainothers or help others

Determine the barriers that have arisen, theconsequences for outreach plans, and therevisions that may be necessary

Coliection of usage data (e.g., Web traffic re-ports, activity reports, attendance countsfrom training sessions, staff journals of out-reach events, or contacts at outreach sites)Reports from or interviews with innovators and

early adopters

Revisit and revise logic model, preferably withstakeholdersRevisit criteria for success andassess thecommunity at midpoint

Lack of activity on Web traffic reports

Low attendance countsOutreach staff accounts showing resistance fromoutreach recipients, barriers, etc.Reports from innovators and adopters about what

is working and what is not with outreachIncorrect assumptions or environmental barriersthat require changes in activities, outputs, orexpected outcomesLack of progress in the stakeholders' perceptionsof goal attainment

staff have very simple evaluation goals and ways torecord their observations, the collection of formativeinformation can become second nature to them. Intum, they will discover subtle barriers that can befixed with minimal training or assistance.

In hindsight, we did a much better job of monitoringour high-yield pilot projects (the high school and thecommunity center) than our low-yield ones (the clin-ics). We naturally were attracted to the sites where weobserved enthusiasm for our product. However, on ourfinal interviews at the clinics, we discovered some un-expected innovators who had been present at our dem-onstrations and had picked up some MedlinePlussearch skills. They reported using MedlinePlus to re-search personal health concems. If we had identifiedthese innovators earlier, our success at these sitesmight have been enhanced.Outcomes assessmentOutcomes assessment can be one of the most chal-lenging aspects of outreach. Many evaluation methodsare directed toward tracking changes in individualparticipants. We faced two major problems with thisapproach. First, we provided very individualized as-sistance to participants, so outcomes are unique toeach person and, therefore, difficult to quantify com-pared to settings where leaming outcomes are stan-dardized across trainees. Second, our contacts withCBO clients or community residents are often tran-sient, such as brief interactions at health fairs or a one-time session during a community center visit. Locatingparticipants for follow-up surveys can be almost im-possible. Therefore, monitoring changes in individualoutreach clients may not be the best way to documentsuccess.

In complex adaptive systems, an analysis of systemchanges may yield a better understanding of outreachoutcomes. A combination of quantitative and qualita-tive methods will yield the most complete picture ofchange in the system. The quantitative methods arenecessary to see the extent the innovation has affectedthe system. The open-ended nature of qualitativemethods is likely to uncover unexpected outcomes.

both positive and negative, that are typical in complexadaptive systems.

Unfortunately, the detail in qualitative methods canquickly become overwhelming. To simplify analysis,qualitative methods can be semi-structured. An excel-lent method is the story-based approach, where stake-holders are asked what important changes have comeabout because of their knowledge and use of the in-novation [19]. Figure 1 illustrates a tool we designedto help promotoras at our community centers recordhow they helped residents and the results of such ef-forts. However, it is best to supplement the story-basedapproach with other methods (focus groups, key in-formant interviews, usage data, surveys) to validatefindings. For example, we interviewed the promotorasmonthly to get more stories, got updates on the resi-dents they wrote about, and listened to their adviceabout how to increase usage of MedlinePlus in thecommimity.

Listed below are some broad questions that outcomemethods could answer using examples from our storysheets, surveys, and key informant interviews. 7s the library resource being adopted by the CBO? Ex-ample: Cameron Park promotoras reported that resi-dents come to their homes day and night to accessMedlinePlus health information. Example: Apromotorawho works with a senior program used MedlinePlusto help seniors develop questions to ask doctors whenthey go to appointments. Has introduction of the innovation helped the comm unityempower the community members? In the hea l th p rom o-tion literature, empowerment is related to increasing thecontrol of individuals over their own lives. Community-based organizations can empower people by providingmeans that can help them make better decisions, criti-cally assess their health situations, and strengthen re-lationships with other individuals and organizations inthe community that can benefit them [20]. Example:Promotoras were particularly happy to have access tdrug information in Sparush to help residents better un-derstand side effects, safety for use with children, dos-age schedules, and other prescription-related issues. Ex-ample: One grandmother at a training session re-


8/12

Olney

Figure 1"Story sheet": a simple method for collecting ongoing stories of howoutreach efforts may be affecting the community

Recorder's initialsWeek ofWeekly Report

Location

How many people did you teach or help with MedlinePlus this week(In classes or one-to-one)Please tell us any stories of how residents benefited from M edlinePlus that you learned about this week. There isspace for two stories, but feel free towrite more on another form. If you had only one example, please write thatstory only. If you do not have any stories, please write that in the first blank box. Be sure to circle the gender andage of the person in the story, but do not add anyone's name!

Person 1 Gender: Female Male Age: Youth Teen Adult ElderWhat type of information did the person get from MedlinePlus?

How did the information help the person? What did the person do as a result of gettingth e MedlinePlus information?

Person 2 Gender: Female Male Age: Youth Teen Adult ElderWhat type of information did the person get from MedlinePius?

How did the information help the person? What did the person do as a result of gettingthe MedlinePlus information?

searched information about juvenile diabetes so shecould care for her granddaughter; she then printed in-formation for the child's parents and her other grand-parents, all of whom also cared for the child. Did usage of the innovation spread beyond the targetgroup? Example: High school students usedMedlinePlus to access health information for them-selves and their friends and family almost as often as

they used it to do research for school projects. Exam-ple: Promotoras directed us to other community leaderwho often assisted people in the community, such aslocal church leaders. Chie of our librarians helped achurch leader put together MedlinePlus informationfor a church-based meeting. Does the system incorporate the innovation as the orga-nization changes and evolves? Example: When Texas


9/12


Table 5Evaluation of system-based outcomesOutcome Indicators (methods in parentheses)

Has the resource or innovation been adopted bythe system?

Has introduction of your innovation helped thecommunity empower stai


10/12

Olneyf o r t yp i c a l he a l t h i n f o r m a t i on ou t r e a c h budge t s . How-ever, if the ge ne r a l goa l is to u n d e r s t a n d the p a t t e r n so f be ha v i o r in a sys t e m and adapt out reach ac t iv i t i esto the sys t e m , t he le s s p r e c i se m e a sur e s p r e se n t e d he r eare l ikely to be a d e q u a t e .In fac t , such methods are essent i a l for suc c e ss in theC B O e n v i r o n m e n t . As por t r a y e d he r e , e va l ua ti on is nol onge r j u s t a r e q u i r e m e n t by e x t e r na l f und i ng a ge nc i e st o ga uge the effect iveness of an i n f o r m a t i on ou t r e a c hprojec t . Ra ther , eva lua t ion is a ne c e ssa r y t oo l for pro-viding e f fec t ive out reach in thed y n a m i c e n v i r o n m e n t so f c om m u ni t y - ba se d o r ga n i z a t i ons .ACKNOWLEDGMENTS

The author recognizes and thanks the entire outreachteam from the libraries at the Harlingen and San An-tonio campuses of the University of Texas Health Sci-ence Center at San Antonio: Debra Warner, VirginiaBowden, FMLA, Evelyn Olivier, Jonquil Feldman, Gra-ciela Reyna, and Mary Jo Dwyer Thanks also to allour community partners at all four outreach sites whoparticipated in these outreach activities. Special thanksto Elliot Siegel and Frederick Wood of NLM's HealthInformation Programs Development and to CatherineBurroughs of the Outreach Evaluation Resource Cen-ter, Pacific Northwest Regional Medical Library, Uni-versity of Washington, for the guidance and manage-ment support they provided for our outreach projects.REFERENCES

1. ScHERRER CS. Outreach to community organizations: thenext consumer health frontier. J Med Libr Assoc 2002 Jul;90(3):285-9.2. WARNER DG , OLNEY CA, W O O D FB, H A N S E N L, BOWDENVM. High school peer tutors teach MedlinePlus: a model forHispanic outreach. J Med Libr Assoc 2005 Apr;93(2):243-52.3. THE UNIVERSITY OF TEXAS HEALTH SCIENCE CENTER ATSAN ANTONIO LIBRARIES. Final report: Texas Lower RioGrande Valley Health Information Hispanic Outreach Pro-ject. [Web document] . Report prepared for the Office ofHealth Information Programs Development of the NationalLibrary of Medicine, 2004 Feb. [cited 8 Feb 2005]. .4. W A T T S D J. Six degrees: the science of a connected age.New York, NY: Norton, 2003.5. WHEATLEY MJ. Leadership and the new science: discov-ering order in a chaotic world. 2nd ed. SanFrancisco, CA:Berrett-Koehler, 1999.6. EOYANG GH. Coping with chaos: seven simple rules.Cheyenne, WY: Lagum o, 1997.7. EOYANG GH , BERKAS T H . Evaluation in a complex adap-tive system. [Web document] . 1998. [cited 8 Feb 2005]. .8. MILLER WL , CRABTREE BF, MCD ANIHL R, STANCE KC. Un-derstanding change in primary care practice using complex-ity theory. J Fam Pract 1998 May;46(5):369-76.9. MILLER WL, M C D A N I E L RR, CRABTREE BF, STANCE KC.Practice jazz: un derstan ding variation in family practices us-ing complexity science. J Fam Pract 2001 Oct;50(10):872-8.10. JACKSON WJ . Heaven's fractal net: retrieving lost visions

in the humanities. Bloomington, IN: Indiana UniversityPress, 2003.11 . RocERS EM. Diffusion of innovations. 4th ed. New York,NY: Free Press, 1995.12. BURROUGHS CM , W O O D FB. Measuring the difference:guide to planning and evaluation health information out-reach. Seattle, WA: National Network of Libraries of Medi-cine, Pacific Northwest Region; Bethesda, MD: National Li-brary of Medicine, 2000.13. CHARLES P, H E N N E R T Evaluation from start to finish:incorporating comprehensive assessment into a training pro-gram for public health professionals. Health Prom Prac 2004Oct;5(4):362-71.14. US AGENCY FOR INTERNATIONAL DEVELOPMENT, CENTERFOR DEVELOPMENT INFORMATION AND EVALUATION. Con-ducting a participatory evaluation. [Web document]. Perfor-mance Mon itoring and Evaluation (TIPS 1996 #1), The A gen-cy, [cited 8 Feb 2005] . .15. SoBO EJ, SiMMES D R, LANDSVERK JA, KURTIN PS. Rapidassessment with qualitative telephone interviews: lessonsfrom an evaluation of California's Healthy Families programs& M edi-Cal for Children. A m J Eval 2003 Sep;24(3):399^08.16. KuRuviLLA S, JOSEPH A. Identifying disability: compar-ing house-to-house survey and rapid rural appraisal. HealthPolicy Plan 1999 Jun;14(2):182-90.17. WK KELLOGG FOUNDATION. Logic model developmentguide: using logic models to bring together planning, evalua-tion, and action (item #1209). [Web document]. Battle Creek,MI, 1 Dec 2001. [rev Jan 2004; cited 8 e h 2005]. .18. FETTERMAN DM . Foundations of empowerment evalua-tion. Tho usand Oaks , CA: Sage, 2001.19. DART J, DAVIES RA. Dialogical story-based evaluationtool: themost significant change technique. Am J Eval 2003Jun;24(2):137-55.20. LAVERACK G, LABONTE R. A planning framework forcommunity empowerment goals wi thin heal th promotion.Health Policy and Plan 2000 Sep;15(3):255-62.21 . PATTON MQ. Utilization-focused evaluation: the newcentury text. 3rd ed. Thousa nd Oak s, CA: Sage, 1997.Received April 2005; accepted May 2005

APPENDIXCommunity assessment question guideThe following guide presents the type of informationthe outreach team should gather when doing com-munity assessment of potential outreach sites. This in-formation can be gathered through a variety of meth-ods, including surveys, interviews, focus groups, toursof community-based organization (CBO) sites, sur-veys, and review of organizational records.1. Who are the primary contacts at the communitycenter?2. Describe the environment, i.e., anything that mayaffect the project. Anything that captures your atten-tion should be noted.3. Innovators and early adopters Who in the community has the most interest incomputers?


11/12


Who seems to be responsible for getting health careinformation to community members?4. Relative advantage: How are the innovators and early adopters gettinghealth information now? Will MedlinePlus be easier orharder to use than their current approaches? How are other community members getting healthinformation?5. Complexity: What group s in the community have experience us-ing the Internet? What groups will have a difficult time using thecomputer or the Internet6. Compatibility: What groups in the community are now learning touse the computer? Who in the community is involved in getting healthinformation for family and other community mem-bers?

How do peop le feel about the health care informa-tion available to them?7. Trialability: Wh ere can residents get comp uter access? Describe the community's public technology re-sources: num ber of stations, hours of access, reliabilityof Internet service. Is any type of training or assistance available to res-idents who want to leam to use the computers? Are there any activities or classes in which outreachlibrarians can demonstrate or teach health informationdatabases?8. Observability: What frustrates com munity members about gettinghealth care information for them selves, family, or com-munity? After examining the health information databaseswe dem onstrated, how do staff at the CBO or membersof the community think they could use them?


12/12

lingkungan 3

Documents