health policy-makers perceptions of their use of evidence

8/19/2019 Health Policy-makers Perceptions of Their Use of Evidence

1/15

Health policy-makers’ perceptions of their useof evidence: a systematic review

Simon Innv×r, Gunn Vist, Mari Trommald, Andrew OxmanHealth Services Research Unit, National Institute of Public Health, Oslo, Norway

Objectives: The empirical basis for theories and common wisdom regarding how to improve appropriate use of research evidence in policy decisions is unclear. One source of empirical evidence is interview studies with policy-makers. The aim of this systematic review was to summarise the evidence from interview studies of facilitators of,and barriers to, the use of research evidence by health policy-makers.

Methods: We searched multiple databases, including Medline, Embase, Socio le, PsychLit, PAIS, IBSS, IPSA andHealthStar in June 2000, hand-searched key journals and personally contacted investigators. We included interview studies with health policy-makers that covered their perceptions of the use of research evidence in health policy

decisions at a national, regional or organisational level. Two reviewers independently assessed the relevance of retrieved articles, described the methods of included studies and extracted data that were summarised in tables andanalysed qualitatively.

Results: We identi ed 24 studies that met our inclusion criteria. These studies included a total of 2041 interviews with health policy-makers. Assessments of the use of evidence were largely descriptive and qualitative, focusing onhypothetical scenarios or retrospective perceptions of the use of evidence in relation to speci c cases. Perceivedfacilitators of, and barriers to, the use of evidence varied. The most commonly reported facilitators were personalcontact (13/24), timely relevance (13/24), and the inclusion of summaries with policy recommendations (11/24).The most commonly reported barriers were absence of personal contact (11/24), lack of timeliness or relevance of research (9/24), mutual mistrust (8/24) and power and budget struggles (7/24).

Conclusions: Interview studies with health policy-makers provide only limited support for commonly held beliefsabout facilitators of, and barriers to, their use of evidence, and raise questions about commonsense proposals forimproving the use of research for policy decisions. Two-way personal communication, the most common suggestion,may improve the appropriate use of research evidence, but it might also promote selective (inappropriate) use of research evidence.

Journal of Health Services Research & P olicy Vol 7 No 4, 2002: 239–244 # The Royal Society of Medicine Press Ltd 200

Introduction

Weiss and Weiss argue that, ‘both decision-makers andsocial scientists behave as though social science researchmakes a genuine contribution to public policy’. 1 At thesame time, many researchers are sceptical about theextent to which research is used and many policy-makersare sceptical about the usefulness of research. AsLindblom and Cohen suggest: ‘ in public policy making, many suppliers and users of social research aredissatis ed, the former because they are not listened to,the latter because they do not hear much they want tolisten to’. 2

Several theories have been put forward to explain therole of research in policy-making and common wisdom

about how to improve the appropriate use of research isnot hard to nd, although empirical evidence tosupport it is dif cult to nd. 3 One source of such

evidence is the perceptions of those involved in policy-making. The objective of this systematic review was tosummarise the evidence from interview studies withhealth policy decision-makers that cover their use of research evidence in health policy decisions. We wereparticularly interested in identifying facilitators of, andbarriers to, the use of evidence by health policy decision-makers as a basis for developing strategies to improvethe appropriate use of research.

We found three previous non-systematic reviews that addressed policy-makers’ use of evidence. 4–6 The rsevaluated evidence regarding research utilisation in

relationship to the ‘two-communities’ metaphor and sixmodels of research utilisation. 3 The second reviewincluded 27 empirical studies with data relevant to the

Review article

Simon Innvær PhD , Researcher, Gunn E Vist PhD , Researcher, MariTrommald MD , Researcher, Andrew D Oxman MD , Head of Section,


2/15


3/15

use rather than actual use of research evidence. Five of the included studies did not report how many policy-makers were interviewed. The remaining 19 studiesinterviewed a total of 2041 policy-makers (median 58,range 16–479).

Study methodsThe included studies were all limited with respect to thegeneralisability of their results. Three studies intervieweda representative sample of health policy decision-makersdrawn from a clearly described sampling frame. 17,18,34Most studies reported how many decision-makers wereincluded and their positions, but the relation betweenthe included policy-makers and the system from whichthey were drawn was rarely described. Three of the 24included studies met all of our methodologicalcriteria: 17,18,34 i.e. clearly described the sampling frame

and the relationship of the included policy-makers tothis; had a response rate of at least 60%; and clearly described how the use of evidence and the determinantsof how evidence was used were measured or assessed(Table 2).Sevenstudiespartially met our criteria (Table 2,last column). The other 14 studies did not adequately describe the methods that were used.

Measurement or assessment of the use and determi-nants of use of evidence was mostly descriptive. Most studies included a list of what the investigators perceivedto be barriers to, or facilitators of, the use of evidence. It is unclear which barriers and facilitators the decisions-

makers found most important and which barriers andfacilitators were considered. Six of the 24 studies rated variables that appeared to determine the use of evidence(Table 2). 9,14,17,18,35

Facilitators and barriers

Many of the studies used open-ended questions. Therespondents and investigators used different words todescribe facilitators of, and barriers to, the use of research evidence. However, several common factors

were described across studies. These are summarised inTable 3. A detailed version of this table, including theexact words used in each of the included studies, isavailable from the authors. The most commonly mentioned facilitators of the use of research evidencein policy-making were:

. Personal contact between researchers and policy-makers (13/24).

. Timeliness and relevance of the research (13/24).

. Research that included a summary with clear recom-mendations (11/24).

. Good quality research (6/24).

. Research that con rmed current policy or endorsedself-interest (6/24)

The most commonly mentioned barriers to the use of research evidence in policy-making were:

. Absence of personal contact between researchers andpolicy-makers (11/24).

. Lack of timeliness or relevance of research (9/24).

. Mutual mistrust, including perceived political naivety of scientists and scienti c naivety of policy-makers(8/24).

. Power and budget struggles (7/24).

. Poor quality of research (6/24).

. Political instability or high turnover of policy-makingstaff (5/24).

Based on the ndings of these studies, personal two- way communication between researchers and decision-makers should be used to facilitate the use of research.This can reduce mutual mistrust and promote a betterunderstanding of policy-making by researchers andresearch by policy-makers. It can inform researchersabout what the decision-makers consider timely, relevant questions and policy-makers about how to obtain validanswers to these questions. However, the frequently identi ed facilitators, including personal two-way commu-nication, may not be easy to establish – for example,because of political instability or high turnover of policy-making staff.

Discussion

The strengths of this review include an extensive andsystematic literature search, explicit inclusion criteriaand a systematic and transparent approach to collectingand presenting data from the included studies. Every included study was read and appraised by at least two of the authors. The limitations of our review largely re ect the limitations of the literature we reviewed.

Because much research in the social sciences is poorly indexed in electronic databases, we may have missedrelevant studies. Personal communication with investiga-tors in the eld yielded only three additional studies and,although we attempted to contact the authors of all of theincluded studies, only three responded and no additional

references were identi ed through these contacts.The limitations of the included studies with respect tosampling and generalisability are not surprising in light of the fact that most of the studies were qualitative and were not necessarily intended to include representativesamples. Moreover, given the diversity of contexts in which health policy decision-making occurs, it is not possible to obtain a generalisable sample. Nonetheless,inadequate descriptions of the participants and contextsfor many of the included studies make it dif cult tointerpret the results.

No factor was mentioned in more than 13 of the 24

studiesas a facilitatoror barrier,evenwhen similar factors were grouped together. This variation can in part beexplained by the fact that some of the studies focused

Health policy-makers’ perceptions of their use of evidence Review article


4/15

on speci c cases and factors were considered only inrelationship to those cases. For example, in a study focusing on the use of an economic evaluation, it ismore likely that nancial constraints would be consid-ered as a barrier. In a developing country study, it is morelikely that international support would be considered as afacilitator. In a study focusing on an issue on which

interest groups have a strong position, it would be morelikely that decision-makers would consider alliances withinterest groups as a facilitator or a barrier.

Two theoretical perspectives were dominant in theliterature. The rst perspective draws analogies betweenthe relationship of researchers and policy-makers, andthe relationship between the natural sciences and thehumanities. From this perspective, problems with com-munication between researchers and policy-makers arelarge and dif cult to overcome. 9,36 This is often referredto as the ‘two-communities thesis’ or the ‘two cultures’.The other perspective focuses on the concept of the ‘useof research’. It addresses how the word ‘use’ may havefundamentally different meanings.

The `two-communities thesis’

The ‘two-communities thesis’ postulates the existence of two camps that lack the ability to take into account therealities or perspectives of one another. Caplan et alfound that social scientists see themselves as rational,objective and open to new ideas; 9 they see decision-makers as action- and interest-oriented, indifferent toevidence and new ideas. Decision-makers, on the otherhand, see themselves as responsible, action-oriented andpragmatic; they see scientists as naive, jargon-ridden andirresponsible in relationship to practical realities. Two- way communication between the two camps can facilitatea mutual understanding of a policy question and thekind of knowledge that is needed. This requires that researchers and decision-makers agree on which ques-tions can be answered on the basis of research evidenceand which require political judgement. This does not imply that research alone can answer policy questions,since these will always require some political judgement.

The aim of two-way communication is to help ensurethat research appropriately informs judgements for which policy-makers are accountable, not for researchersto assume the role of policy-makers.

The results of this reviewsupport the ‘two communitiesthesis’, since the most commonly identi ed facilitatorof the use of evidence was personal contact betweenresearchers and decision-makers. It is further supportedby the most commonly mentioned barriers to the use of evidence, absence of personal contact and mutualmistrust between decision-makers and researchers. Personal two- way communication may also be a necessary precondition

for other facilitators. For example, without personaltwo-way communication it may be dif cult for researchersto understand what decision-makers regard as timely

makers. It is not surprising that decision-makers nd it easier to use evidence that they have had an opportunity to in uence through two-way communication with theresearchers. Additionally, the included studies do not clarify whether decision-makers use only the researchthey want to use. Decision-makers may use evidence that supports their own ideology or their own political

programme.The ‘two-communities’ thesis sheds light on animportant possible paradox inherent in the results: thefactors that facilitate use of research may not necessarily be factors that researchers should seek to enhance. If what is required for research to be used is that researchersdo what the policy-maker wants themto do, then researchmay fail to ful l one of its most important functions,namely to be objective, reliable and unbiased.

What is `use’ of evidence?

Along with the ‘two-communit ies’ thesis, the question of what is meant by the concept of the ‘use’ of evidence is themost commonly discussed theoretical issue in the litera-ture on knowledge utilisation. The most frequent categor-isation of different types of use in this review is direct (‘instrumental’ or ‘engineering’), selective (‘symbolic’ or‘legitimating’) and enlightening (‘conceptual’) use of evidence. Direct use of evidence refers to speci c use of research results. It indicates that, if research results arerelevant for a solution, the results should directly affect the solution without much adjustment. Enlighteninguse of evidence refers to research that helps to ‘establishnew goals and bench marks of the attainable’ 37 and helpsto ‘enrich and deepen understanding of the complexity of problems and the unintended consequences of action’. 38 Selective use is strategic, involving use ‘tolegitimate and sustain predetermined positions’. 5

The different de nitions of ‘use’ contribute to thedif culty of interpretingthe results of the studiesincludedin this review.A health policy decision-makerwho de nesuse as direct use is likely to report less use of researchevidence than decision-makers who also include selectiveor enlightening use of evidence. Many of the studies have

addressed different kinds of use and found that healthpolicy decision-makers refer to use in all three of theabove ways.

The degree to which evidence is used directly, in anenlightening way or selectively may vary in relation to:

. Different types of decision-makers: upper, middle andlower level.

. Different types of policy questions: vague andcomplex, or focused and simple.

. Different issues: adoption versus implementation, ordecision versus action.

The degree to which evidence was directly used variedacross studies. One study found that 40% of the use of evidence was direct de ned as the ‘primary source in

Review ar ticle Health policy-makers’ perceptions of their use of evidence


5/15

had been commissioned. The study with only 7% direct use did not cover examples of commissioned research.This suggests that expecting direct use frequently may be unreasonable, particularly for research that is not commissioned. A third study, not included in thisreview, found that local administrators, close to pro-gramme operations, reported more direct use of

evidence than federal-level decision-makers.39

Enlightening use of evidence is more dif cult to assessthan direct use. Decision-makers may interpret this asno more than a question of whether they considerresearch to be of value. One study found that 60% of decision-makers ‘used’ evidence in thisway. 9 Self-reportedselective use of research, on the other hand, may bedif cult to elicit in interviews because it puts decision-makers in a bad light, although it is commonly perceivedas a problem by researchers and others. Research that con rms current policy or supports community pressuremay facilitate selective use. Personal contact may alsofacilitate selective use of research.

Conclusions

From the results of this review of 24 studies that met ourinclusion criteria, based on 2041 interviews with policy-makers, researchers who wish to increase the use of theresults of their research should: have personal and closetwo-way communication with decision-makers; providedecision-makers with a brief summary of their research with clear policy recommendations; ensure that theirresearch is perceived as timely, relevant and of highquality; include effectiveness data; argue that the resultsof their research are relevant to current policy anddemands from the community.They should avoid gettinginvolved in power and budget struggles and be aware of the high turnover of policy-making staff. Of course, if theaim is to increase appropriate (direct or enlightening)use of research rather than selective use, these strategies will often not be effective, as well as being dif cult toimplement.

If the results from the interview studies with decision-makers are taken literally, use of evidence will be

facilitated by timely and relevant research that givesdecision-makers the answers that they want. However,they may not always be able to get what they want, sinceresearchers should, and it is to be hoped will, insist onusing methods that protect against bias. If decision-makers and researchers start to talk together about thebarriers and facilitators found in this review, a morere ective, appropriate and cooperative way of workingtogether, which values both relevance and validity, may evolve.

The studies included in this review address decision-makers’perceptions of their use of evidence. Somestudies

suggested document analysis as a way to check whetherthese perceptions correspond to what was done, 12,15,19,21,33but none of the included studies did this system-

The methods that were used in the included studiesand their diversity limit the extent to which any rmconclusion can be drawn from this review. In fact, one of the main conclusions of this review is that there is, at best, only limited support for any of the many opinionsput forward in the literature on the use of researchevidence by policy-makers. At the same time, given the

diversity of the included studies, the policy-makers who were interviewed and the contexts in which they worked,it is striking that some factors were identi ed as fre-quently as they were.

Advice about how to improve the use of evidence by policy-makers is typically based on personal experience,supported by anecdotes. The results of this review providea context in which to consider such advice. They providea menu of factors to consider, but not a basis for any strong recommendations. Similarly, the two theoriesthat are commonly used in these studies provide usefulinsights for understanding the perceptionssummarised inthis review and other opinions about the use of evidenceby policy-makers, but there are limited empirical data tosupport these or other theories.

Acknowledgement We thank Janette Boynton for help with literature searches.

References1. Weiss JA, Weiss CH. Social scientists and decision maker

look at the usefulness of mental health research. In:Lorion RP et al, eds. Psychology and public policybalancing public service and professional need. Washington, DC: American Psychological Association,1996: 165–181

2. Lindblom CE, Cohen DK. Usable knowledge: socialscience and social problem solving. New Haven, CT: YaleUniversity Press, 1979

3. Granados A, Jonsson E, Banta HD, Bero L, Bonair ACochet C et al. EUR-Assess Project Subgroup Report onDissemination and Impact. International Journal of Technology Assessment in Health Care 1997; 13: 220–286

4. Dunn WN. The two-communities metaphor and models of knowledge use: an explanatory case study. Knowledge:

Creation, Diffusion, Utilization 1980; 1: 515–5365. Beyer JM, Trice HM. The utilization process: a conceptualframework and synthesis of empirical ndings. Adminis-trative Science Quarterly 1982; 27: 591–622

6. Nelson CE, Roberts J, Maederer CM, Wertheimer B, Johnson B. The utilization of social science informationby policy makers. American Behavioral Scientist 1987; 30:569–577

7. van de Vall M, Bolas C. Using social policy research foreducing social problems: an empirical analysis of struc-ture and functions. Journal of Applied Behavioral Science1982; 18: 49–67

8. Glaser EM, Taylor SH. Factors in uencing the success ofapplied research. American Psychologist 1973; 28: 140–146

9. Caplan N, Morrison A, Stambaugh RJ. The use of socialscience knowledge in policy decisions at the national level:a report to respondents. Ann Arbor, MI: The University of

Health policy-makers’ perceptions of their use of evidence Review article


6/15

research in public policy making. Toronto: LexingtonBooks, 1977: 183–198

11. Patton MQ, Grimes PS, Guthrie KM, Breanan NJ et al. Insearch for impact: an analysis of the utilization of federalhealth evaluation research. In: Weiss CH, ed. Using socialresearch in public policy making. Toronto: LexingtonBooks, 1977: 141–164

12. Weiss CH, Bucuvalas MJ. Truth test and utility test:decision makers’ frames of reference for social scienceresearch. American Sociological Review 1980; 45: 302–313

13. Weiss JA. Coping with complexity: An experimental study of public policy decision-making. In: Kinder DD, Palfrey TR, eds. Experimental foundations of political science. Ann Arbor, MI: University of Michigan Press, 1993: 185–208

14. McNeece CA, DiNitto DM, Johnson PJ. The utility of evaluation research for administrative decision-making. Administration in Social Work 1983; 7: 77–87

15. Sunesson S, Nilsson K. Explaining research utilization:beyond ‘functions’. Knowledge: Creation, Diffusion, Util-ization 1989; 10: 140–155

16. Florio E, DeMartini JR. The use of information by policymakers at the local community level. Knowledge1993; 15: 106–123

17. Boyer JF, Langbein LI. Factors in uencing the use of health evaluation research in congress. Evaluation Review 1991; 15: 507–532

18. Ibbotson SL, Long AF, Sheldon TA, Mason J. An initialevaluation of Effective Health Care bulletins as instru-ments of effective dissemination. Journal of Management in Medicine 1993; 7: 48–57

19. Soumerai SB, Ross-Degnan D, Fortess EE, Walser BL.Determinants of change in Medicaid pharmaceutical cost sharing: does evidence affect policy? Milbank Quarterly 1997; 75: 11–34

20. Harries U, Elliott H, Higgins A. Evidence-based policy-making in the NHS: exploring the interface betweenresearch and the commissioning process. Journal of PublicHealth Medicine 1999; 21: 29–36

21. Elliott H, Popay J. How are policy makers using evidence?Models of research utilisation and local NHS policy making. Journal of Epidemiology and Community Health 2000; 54: 461–468

22. Trostle J, Bronfman M, Langer A. How do researchersin uence decision-makers? Case studies of Mexicanpolicies. Health Policy and Planning 1999; 14: 103–114

23. Ross J. The use of economic evaluation in health care: Australian decision makers’ perceptions. Health Policy 1995; 31: 103–110

24. Oh CH. Explaining the impact of policy information onpolicy-making. Knowledge and Policy: The International Journal of Knowledge Transfer and Utilization 1997; 10:22–55

25. Oh CH, Rich RF. Explaining use of information in publicpolicymaking. Knowledge and Policy: The International Journal of Knowledge Transfer and Utilization 1996; 9: 3–35

26. Hilderbrand M, Simon J, Hyder A. The role of research inchild health policy and programs inPakistan. Geneva: TheCouncil of Health Research and Development (COHRED) Working Group on Research to Action and Policy, 2000:77–85

27. Rudat K. Evaluation of the Effective Health Care Bulletin.London: Of ce For Public Management, 1998: 1–46

28. Burns A, Charlwood P, Darling H, Fox DM, Green eld L,

Hamlyn L et al. Better information, better outcomes? Theuse of health technology assessment and clinical effective-ness data in health care purchasing decisions in theUnited Kingdom and the United States. New York:Milbank Memorial Fund, 2000: 1–26

29. Eyles J, Stoddart G, Lavis J, Pranger T, Molyneaux-Smith LMcMullan C. Making resource shifts supportive of thebroad determinants of health: The P .E.I. experience.Hamilton: McMaster Institute of Environment and Health,2000: 3–37

30. Gerhardus A, Kielmann K, Sanou A. Lessons in research toaction and policy: case studies from seven different countries. Geneva: The Council of Health Research andDevelopment (COHRED) Working Group on Research to Action and Policy, 2000: 19–27

31. Moodley J, Jacobs M. Research to action and policy:combating vitamin a de ciencies in South Africa. Geneva:The Council of Health Research and Development (COHRED) Working Group on Research to Action andPolicy, 2000: 54–66

32. Lavis JN, Farrant MSR, Stoddard GL. Barriers to employ-ment-related healthy public policy. Toronto: McMasterUniversity Centre for Health Economics and Policy Analysis, 2000: 00–03, 1–28

33. Lavis JN, Ross SE, Hohenadel J, Hurley J, Stoddard GL Woodward C et al. The role of health services research inCanadian provincial policy-making. CHSRF Report. 1997-

021. Hamilton: McMaster University, 2000: 1–4434. McNeece CA, DiNitto DM, Johnson PJ. The utility ofevaluation research for administrative decision-making. Administration in Social Work 1983; 7: 77–87

35. van de Vall M, Bolas C. Using social policy research forreducing social problems: an empirical analysis of struc-ture and functions. Journal of Applied Behavioral Science1982; 18: 49–67

36. Snow CP. The two cultures and the scienti c revolution.Cambridge: Cambridge University Press, 1959

37. Merton RK. The role of applied social science in theformation of policy: a research memorandum. Philosophy of Science 1949; 16: 161–181

38. Aaron HJ. P olitics and the professors: the great society in

perspective. Washington, DC: Brookings Institution, 197839. Alkin MC, Kosecoff J, Fitzgibbon C, Seligman R. Evalua

tion and decision-making: the title vii experience. Los Angeles: Center for the Study of Evaluation, University of California, 1974

Review ar ticle Health policy-makers’ perceptions of their use of evidence


7/15

Review article Health policy-makers’ perceptions of their use of evidence

C h a r a c t e

r i s t i c s o f i n c l u d e d s t u d i e s

y e a r c o n d u c t e d

b l i s h e d ) ,

O b j e c t i v e

S t u d y d e s i g n

P a r t i c i p a n t s / l e v e l o f

p o l i c y - m a k i n g

T y p e s o f e v i d e n c e

( f r o m w h i c h d e c i s i o n s

w e r e m a d e )

T y p e s o f d e c i s i o n s

( h y p o t h e t i c a l , p

e r c e i v e d

o r a c t u a l ) / t y p e s o f p o l i c y

C o m m e n t s

r l a n d s

1 7 *

T h e c o n d i t i o n s a n d

f u n c t i o n s i n f l u e n c i n g

u t i l i s a t i o n o f s o c i a l p o l i c y

r e s e a r c h ( p

. 4 9 )

S e m i - s t r u c t u r e d s u r v e y s

( p . 5

0 )

R e s e a r c h e r s a n d p o l i c y -

m a k e r s t h a t w e r e

r e s p o n s i b l e f o r

i n c o r p o r a t i n g t h e r e s e a r c h

r e s u l t s ( p

. 5 0 )

. N ˆ n o t

s t a t e d

1 2 0 r e s e a r c h p r o j e c t s o n

` c l i e n t - o r i e n t e d ’ s o c i a l

r e s e a r c h w i t h i n i n d u s t r i a l

a n d l a b o u r r e l a t i o n s ,

r e g i o n a l a n d u r b a n

p l a n n i n g , s

o c i a l w e l f a r e a n d

p u b l i c h e a l t h ( p

. 5 0 ) * *

P e r c e i v e d ( r e t r o s p e c t i v e )

d e c i s i o n s ( p

. 5 0 )

* R e v i s e d

v e r s i o n o f p a p e r

p r e s e n t e d i n 1 9 7 9 ( p

. 4 9 )

* * 4 0 o u t o f t h e 1 2 0 p r o j e c t s w e r e o n

s o c i a l w e l f a r e a n d p u b l i c h e a l t h

8 *

W h a t w a s d o n e , w

h a t

h e l p e d a n d w h a t h i n d e r e d

t h e s u c c e s s f o r t e n

r e s e a r c h p r o j e c t s u n d e r t h e

N a t i o n a l I n s t i t u t e o f M e n t a l

H e a l t h * * ( p

. 1 4 1 )

P e r s o n a l i n t e r v i e w s ( p

. 1 4 1 ) P r i n c i p a l i n v e s t i g a t o r , o

t h e r

s t a f f m e m b e r s

,

a d m i n i s t r a t o r s

a n d p e r s o n s

m o s t l i k e l y t o b e u t i l i s e r s

( p . 1

4 1 ) . N

ˆ n o t g i v e n

F i v e p r o j e c t s r a t e d l o w a n d

f i v e p r o j e c t s r a t e d h i g h f o r

` s u c c e s s ’ * * ( p

. 1 4 0 )

P e r c e i v e d u s e

* T h e i n t e r v i e w s w e r e d o n e i n e i t h e r

1 9 6 7 o r 1 9 6 8 ( p p . 1 4 0 ^ 1 4 1 )

* * S u c c e s s d e f i n e d a s p r o j e c t t h a t

w a s d i r e c t e d t o t h e o r i g i n a l

o b j e c t i v e s , e m b o d i e d i n a c l e a r

a n d c o g e n t r e p o r t , c a n d i d e n o u g h

t o b e r e p l i c a b l e

, a n d t h a t t h e

f i n d i n g s w e r e d i s s e m i n a t e d

a d e q u a t e l y ( p

. 1 4 0 )

^ 1 9 7 4 9 ( p

. v i i )

1 9 7 3 ^ 1

9 7 4 1 0

S o c i a l s c i e n c e r e s e a r c h

u t i l i s a t i o n a n d p o l i c y

f o r m a t i o n ( p

. v i i )

. A w a r e n e s s

o f r e s e a r c h , k

n o w l e d g e o f

s o u r c e s , i n f o r m a t i o n

r e t r i e v a l p r a c t i c e , i n t e r e s t i n

s o c i a l i n d i c a t o r s , e

v a l u a t i o n

o f w o r t h a n d o b j e c t i v i t y

,

m e t h o d s a n d m e a s u r e m e n t

p r o c e d u r e s , d

e g r e e o f

c o n f i d e n c e o n f i n d i n g s ,

a t t i t u d e s a b o u t u s e , f

a c t o r s

t h a t i n f l u e n c e u s e ( p

. i x )

P e r s o n a l i n t e r v i e w s , o p e n -

e n d e d a n d s t r u c t u r e d

( p p . v , i x )

U p p e r - l e v e l d e c i s i o n -

m a k e r s i n t h e e x e c u t i v e

b r a n c h o f t h e U S f e d e r a l

g o v e r n m e n t ( p p . v , v

i i i ) : 3 1

d e p u t y u n d e r s e c r e t a r i e s ,

5 4 i n s t i t u t e d i r e c t o r s , 6

7

d e p u t y a s s i s t a n t d i r e c t o r s ,

5 2 a g e n c y p e r s o n n e l .

N ˆ

2 0 4 ( p

. v )

5 7 5 r e p o r t e d i n s t a n c e s o f

u s e , w h e r e 3 8 5 i n v o l v e d

p r i m a r y r e s e a r c h s o u r c e s

( p . 8

) ; 9 4 % w e r e e i t h e r

f u n d e d b y t h e g o v e r n m e n t ,

c o n d u c t e d b y t h e

g o v e r n m e n t o r b o t h ( p

. 8 )

P e r c e i v e d ( s e l f - r e p o r t e d )

u s e w i t h s p e c i f i c f o l l o w - u p

q u e s t i o n s w h e n u s e w a s

r e p o r t e d ( p

. 1 ) . 1 5 % o f t h e

u s e o f k n o w l e d g e w a s o n

h e a l t h o r w e l f a r e p r o b l e m s

( p . 5

)

` M o s t o f t h e e x a m p l e s r e s p o n d e n t s

o f f e r e d t o i l l u s t r a t e k n o w l e d g e

a p p l i c a t i o n s r e a l l y i n v o l v e d t h e

a p p l i c a t i o n o f s e c o n d a r y s o u r c e

i n f o r m a t i o n ’ ( p

. 1 8 7 )

# 1 1

` T o e x a m i n e t h e n a t u r e a n d

d e g r e e o f u t i l i z a t i o n o f

f e d e r a l e v a l u a t i o n r e s e a r c h ’

( p . 1

4 3 )

O p e n - e n d e d i n t e r v i e w s ,

w i t h d i f f e r e n t q u e s t i o n s f o r

t h e d e c i s i o n - m a k e r a n d t h e

e v a l u a t o r ( p

. 1 4 3 )

T h r e e k e y i n f o r m a n t s f r o m

e a c h o f t h e 2 0 s t u d i e s w e r e

i n t e r v i e w e d : a p r o j e c t

o f f i c e r , t

h e d e c i s i o n - m a k e r

( g o v e r n m e n t l e v e l ) a n d t h e

e v a l u a t o r ( p . 1 4 3 ) . N

ˆ 6 0

( p . 1

4 3 )

2 0 n a t i o n a l h e a l t h c a r e

e v a l u a t i o n s ( c a s e s t u d i e s )

w i t h s o m e s y s t e m a t i c d a t a

c o l l e c t i o n , m a d e b e t w e e n

1 9 7 1 a n d 1 9 7 4

, b a s e d o n

1 1 f a c t o r s i d e n t i f i e d b y t h e

a u t h o r s ( p p . 1 4 2 ^ 1 4 3 )

P e r c e i v e d a n d s e l f - d e f i n e d

u s e ( p

. 1 4 4 ) . H e a l t h t r a i n i n g ,

l a b o r a t o r y p r o f i c i e n c y ,

n e i g h b o u r h o o d h e a l t h

c e n t r e s , h e a l t h s e r v i c e s

d e l i v e r y s y s t e m s ,

a l c o h o l i s m , l

o a n

f o r g i v e n e s s a n d t r a i n i n g

w o r k s h o p ( p p . 1 4 2 ^ 1 4 3 )

2 * a n d U S A

P o l i c y - m a k e r s ’ p e r c e p t i o n s

o f t h e u s e f u l n e s s o f s o c i a l

s c i e n c e s r e s e a r c h ( p p . 1 6 5 ,

3 0 2 )

P e r s o n a l i n t e r v i e w s , b o t h

o p e n - e n d e d a n d s t r u c t u r e d

( p p . 1 6 8 , 1 7 1 , 3 0 3 )

5 1 f e d e r a l , 5 2 s t a t e ( f r o m

t e n s t a t e s ) a n d 5 2 l o c a l

o f f i c i a l d e c i s i o n - m a k e r s i n

u p p e r - l e v e l p o s i t i o n s

( p p . 1 6 7 , 3 0 3 ) . N

ˆ 1 5 5

E a c h d e c i s i o n - m a k e r r e a d

t w o a b s t r a c t s ( t w o - p a g e r s )

o u t o f 5 0 f o r a c t u a l s o c i a l

s c i e n c e r e s e a r c h r e p o r t s * *

( p p . 1 6 8 , 3 0 3 )

H y p o t h e t i c a l d e c i s i o n s ( i . e .

` r e l e v a n t t o t h e i s s u e s y o u r

o f f i c e d e a l s w i t h ’ ) o n m e n t a l

h e a l t h

, a l c o h o l a n d d r u g

a b u s e * * ( p p . 1

6 6 , 3

0 3 )

* N o t s t a t e d , b

u t p u b l i s h e d A p r i l

1 9 8 0

. R e p r i n t e d f r o m

A m e r i c a n

P s y c h o l o g i s t 1 9 8 1 . B a s e d o n t h e

s a m e r e s e a r c h p r o j e c t a s U S A

1 9 8 0

1 2 ( b u t p a r t l y o n o t h e r p a r t s o f

t h e d a t a )

* * T h e d e c i s i o n - m a k e r s w e r e g i v e n

d i f f e r e n t ` e m p i r i c a l ’ i n t e r v e n t i o n s

( i . e . t w o - p a g e r s ) a n d a r e t h e r e f o r e

d i f f i c u l t t o c o m p a r e o n t h e e f f e c t


8/15


i n u e

d

r c o n d u c t e d

h e d ) ,

O b j e c t i v e






w e r e m a d e )



e r c e i v e d


C o m m e n t s

T o o b s e r v e t h e i m p a c t t h a t

d e c i s i o n c o m p l e x i t y

, a n d

a m o u n t o f i n f o r m a t i o n

a v a i l a b l e , h a d o n t h e q u a l i t y

o f t h e d e c i s i o n p r o c e s s

( p . 1

8 7 )

S t r u c t u r e d , s e m i - s t r u c t u r e d

a n d o p e n - e n d e d q u e s t i o n -

n a i r e s c o m b i n e d w i t h a

c o n t r o l l e d e x p e r i m e n t .

C o n s e q u e n c e s o f

c o m p l e x i t y w e r e e x a m i n e d

b y e f f e c t s o f i n f o r m a t i o n

l o a d a n d n u m b e r o f

d e c i s i o n a l t e r n a t i v e s

( p p . 1 8 7 , 1 9 3 , 1 9 4 ) * *

E x p e r i e n c e d p u b l i c a n d

p r i v a t e s e c t o r m a n a g e r s

f r o m s i x m i d - c a r e e r

p r o g r a m m e s a t H a r v a r d ,

M I T

, S t a n f o r d a n d t h e

W a s h i n g t o n P u b l i c A f f a i r s

C e n t e r ( p p . 1

8 7 , 1 8 8 )

.

N ˆ

1 3 2 ( p

. 1 8 8 )

E a c h d e c i s i o n - m a k e r

r e c e i v e d a l o n g

m e m o r a n d u m d e s c r i b i n g a

p r o b l e m , p

r e s e n t i n g

e v i d e n c e u s e f u l i n

a n a l y s i n g t h e p r o b l e m , a

n d

p o s e d a d e c i s i o n t o b e

m a d e b a s e d o n r e a l p e o p l e

a n d e v e n t s ( p

. 1 8 8 )

T w o h y p o t h e t i c a l d e c i s i o n s :

o n e o n t h e s t a t e ’ s D e p a r t -

m e n t o f H e a l t h a n d R e h a b i -

l i t a t i v e s e r v i c e s a n d o n e o n

a m e t h a d o n e m a i n t e n a n c e

p r o g r a m m e i n N e w Y o r k

C i t y ( p p . 1 8 8 ^ 1 8 9 )

.

D e c i s i o n h a d t o b e m a d e

f r o m a v a i l a b l e i n f o r m a t i o n

w i t h i n 1 h o u r 4 5 m i n u t e s

( p . 1

9 3 )

* R e p r i n t e d f r o m t h e

J o u r n a l o f

P o l i c y

A n a l y s

i s a n d M a n a g e m e n t

2 : 6 6 ^ 8

7 ( p

. 1 8 5 )

* * T h e e x p e r i m e n t w a s b a s e d o n s i x

d i f f e r e n t v e r s i o n s o f e a c h o f t h e t w o

c a s e s . T h e s i x d i f f e r e n t v e r s i o n s o f

t h e c a s e s

v a r i e d i n i n f o r m a t i o n

c o m p l e x i t y

. E a c h d e c i s i o n - m a k e r

h a d o n l y o n e d e c i s i o n t o m a k e a t

o n l y o n e l e v e l o f c o m p l e x i t y

( p p . 1 8 9 ^ 1 9 0 , 1 9 3 )

T o d e t e r m i n e s o m e o f t h e

f a c t o r s t h a t i n f l u e n c e

a d m i n i s t r a t o r s ’ u s e o f

e v a l u a t i v e d a t a i n d e c i s i o n -

m a k i n g , a n d t o e x a m i n e t h e

t y p e s o f d e c i s i o n s ( p

. 7 9 )

S t r u c t u r e d a n d o p e n - e n d e d

p o s t a l q u e s t i o n n a i r e s

( p p . 7 9 ^ 8

0 )

D i r e c t o r s i n c o m m u n i t y

m e n t a l h e a l t h c e n t r e s i n

F l o r i d a a n d N o r t h C a r o l i n a

( p . 7

9 ) . N

ˆ 4 2

C o n d u c t e d s t u d i e s o n

p r o g r a m m e e f f e c t i v e n e s s

o r i m p a c t d u r i n g t h e l a s t 1 2 -

m o n t h p e r i o d ( p

. 8 1 )

P e r c e i v e d ( i . e . r e p o r t e d )

u s e ( p p . 8 0 ^ 8

1 )

1̂ 9 8 5 1 5

D i f f e r e n t w a y s o f u s i n g

s o c i a l r e s e a r c h :

i n s t r u m e n t a l , p o l i t i c a l /

c o n f l i c t , e n l i g h t e n m e n t ,

i n t e r a c t i v e , a n d t a c t i c a l

( p p . 1 4 1 , 1 4 3 ^ 1 4 4 )

I n t e r v i e w s ( p

. 1 5 2 )

S o c i a l w o r k e r s ,

m a n a g e m e n t d i r e c t o r s ,

l o c a l a n d n a t i o n a l p o l i t i c i a n s

i n 1 5 S w e d i s h c i t i e s a n d

c o m m u n e s ( p

. 1 5 2 ) . N

ˆ 9 0

( p . 1

5 2 )

D o c u m e n t s t h a t t h e

r e s p o n d e n t s r e f e r r e d t o

.

T h e s e d o c u m e n t s w e r e a l s o

a n a l y s e d b y t h e a u t h o r s

( p . 1

5 2 )

P e r c e i v e d / r e t r o s p e c t i v e

d e c i s i o n s ( p

. 1 5 2 )

T o d e t e r m i n e w h a t t y p e o f

i n f o r m a t i o n d e c i s i o n -

m a k e r s u s e , a n d h o w

i d e o l o g y a n d i n t e r e s t s

i n f l u e n c e t h e u s e o f

i n f o r m a t i o n ( p

. 1 0 9 )

P e r s o n a l , s e m i - s t r u c t u r e d

i n t e r v i e w s ( p

. 1 1 2 )

L o c a l p l a n n i n g c o m m i t t e e s

a n d g o v e r n m e n t e m p l o y e e s

i n t w o r u r a l c o m m

u n i t i e s i n

t h e P a c i f i c N o r t h w e s t

( p . 1

1 0 ) . N

ˆ 2 7 ( p

. 1 1 0 )

S o u r c e s o f i n f o r m a t i o n u s e d

i n c o m m u n i t y h e a l t h c a r e

p l a n n i n g , a

n d t h e

a s s e s s m e n t o f t h e s t r a t e g i c

p l a n n i n g p r o c e s s ( p

. 1 1 2 )

P e r c e i v e d u s e ( p

. 1 1 2 )

W h a t f a c t o r s i n f l u e n c e t h e

u s e o f e v a l u a t i o n r e s e a r c h

( o n h e a l t h a n d h u m a n

s e r v i c e s ) i n c o n g r e s s

( p . 5

0 8 ) *

S t r u c t u r e d t e l e p h o n e


. 5 1 3 ) * *

C o n g r e s s m e m b e r s a n d

t h e i r s t a f f ( p

. 5 1 2 ) . N

ˆ 1 0 0

E a c h d e c i s i o n - m a k e r c h o s e

o n e s t u d y t h e y w e r e f a m i l i a r

w i t h ( p

. 5 1 5 )


. 5 1 5 ) .

E v a l u a t i o n s t u d i e s f r o m t h e

D e p a r t m e n t o f H e a l t h a n d

H u m a n S e r v i c e s t h a t d e a l

w i t h n a t i o n a l h e a l t h

p r o g r a m m e s ( p p . 5 0 7 , 5 1 5 )

* U s e i s d e f i n e d h e r e a s b o t h

i n s t r u m e n t a l ( d i r e c t

i m p l e m e n t a t i o n ) , c

o n c e p t u a l ( i t

c h a n g e d t h e w a y o f t h i n k i n g ) a n d

p e r s u a s i v e ( u s e d t o c o n v i n c e

o t h e r s , p .

5 1 6 )

* * T h e i n t e r v i e w s l a s t e d 1 5 ^ 4

0

m i n u t e s ( p . 5 1 4 )

U s e a n d r e a s o n s f o r u s e / n o t

u s e o f t h e

E ¡ e c t i v e H e a l t h

C a r e B u l

l e t i n ( p p . 5 1 ^ 5

4 )

S t r u c t u r e d a n d o p e n - e n d e d

p o s t a l q u e s t i o n n a i r e s

( p p . 5 0 ^ 5

1 )

D i s t r i c t g e n e r a l m

a n a g e r s ,

d i r e c t o r s o f p u b l i c h e a l t h

a n d d i r e c t o r s o f p l a n n i n g /

c o m m i s s i o n i n g i n U K

d i s t r i c t h e a l t h a u t h o r i t i e s o r

c o m m i s s i o n i n g c o n s o r t i a

( p . 5

0 ) . N

ˆ 1 7 2 ( p

. 5 1 )

E ¡ e c t i v e H e a l t h C a r e

B u l

l e t i n N o . 1 , 1 9 9 1

, o n

s c r e e n i n g f o r o s t e o p o r o s i s

t o p r e v e n t f r a c t u r e s ( p

. 5 0 ) *

P e r c e i v e d u s e o f i n f o r m a t i o n

a n d h y p o t h e t i c a l d e c i s i o n s ,

c h a n g e o f o p i n i o n a n d

c o n f i d e n c e t o a c t o n

i n f o r m a t i o n i n t h e b u l l e t i n

( p p . 5 2 ^ 5

4 )

* R e s p o n d e n t s w e r e q u e s t i o n e d

a b o u t a b u l l e t i n t h a t h a d b e e n

p r e v i o u s l y p o s t e d t o t h e m

( p p . 5 1

, 5 2 )

c o n t i n u e

d


9/15


10/15


i n u e

d

r c o n d u c t e d

h e d ) ,

O b j e c t i v e






w e r e m a d e )



e r c e i v e d


C o m m e n t s

9 9 8 #

2 8 *

H o w U K a n d U S p u r c h a s e r s

v a l u e a n d u s e r e s e a r c h

a s s e s s i n g h e a l t h

t e c h n o l o g y a n d c l i n i c a l

e f f e c t i v e n e s s ( p p . v , 1 )

S e m i - s

t r u c t u r e d t e l e p h o n e


. 4 )

U K : 1 4 n o n - m e d i c a l h e a l t h

a u t h o r i t y p u r c h a s e r s , s

e v e n

p u b l i c h e a l t h p h y s i c i a n s ,

n i n e G P s

. U S A : 1 3 p u b l i c

o f f i c i a l s f r o m 1 1 s t a t e s , n

i n e

p r i v a t e s e c t o r p u r c h a s e r s ,

t h r e e c o n s u l t a n t s .

N ˆ

5 5

( p . 4

)

A c c e s s t o a n d u s e o f h e a l t h

t e c h n o l o g y a s s e s s m e n t /

c l i n i c a l e f f e c t i v e n e s s

i n f o r m a t i o n ( p p . 4 , 1 1 )


. 1 )

* S t a t e d 1 9 9 9 ( p

. 3 )

1̂ 9 9 9 2 9

A c a s e s t u d y o f c r o s s -

s e c t o r a l r e s o u r c e a l l o c a t i o n

i n t h e h u m a n s e r v i c e s

s y s t e m i n P r i n c e E d w a r d

I s l a n d ( p

. 4 )

S e m i - s

t r u c t u r e d a n d o p e n -

e n d e d i n t e r v i e w s a n d f o c u s

g r o u p s ( p p . 1 3 ^ 1

6 ,

A p p e n d i x 1 )

K e y i n f o r m a n t s h o l d i n g

d i f f e r e n t p o s i t i o n s w i t h i n t h e

f i v e h e a l t h r e g i o n s , t h e

D e p a r t m e n t o f S o c i a l

S e r v i c e s a n d v a r i o u s

s e c t o r s ( p

. 4 ) , a l l r e g i o n s , a

l l

s e c t o r s a n d m a n a g e m e n t

l e v e l s ( p

. 1 3 )

. N ˆ

5 8 ( p

. 9 )

N o t s t a t e d

P e r c e i v e d d e c i s i o n s o n

c r o s s - s e c t o r a l r e s o u r c e

a l l o c a t i o n s ( p

. 5 )

19 9 9 3 0

A c a s e s t u d y o n h o w

r e s e a r c h p l a y e d a r o l e i n

d e c i s i o n - m a k i n g a b o u t

` S h a r e d C a r e ’ * f o r

c h i l d h o o d i l l n e s s ( p p . 1 9 ^

2 0 )

S e m i - s

t r u c t u r e d i n t e r v i e w s

( p . 2 0

)

M i n i s t r y o f H e a l t h

o f f i c i a l s

( p . 2

2 ) . N

ˆ n o t g i v e n

W o r k s h o p s a n d r e s e a r c h

p a p e r s t h a t t h e a u t h o r s h a d

p r o v i d e d M i n i s t r y o f H e a l t h

o f f i c i a l s w i t h d u r i n g

p r e v i o u s y e a r s ( p

. 2 1 )

P e r c e i v e d u s e o f t h e

p r o v i d e d s c i e n t i f i c

i n f o r m a t i o n i n d e c i s i o n -

m a k i n g ( p

. 2 0 )

* ` S h a r e d c a r e ’ i s a b o u t i m p r o v e d

c o n t a c t a n d c o l l a b o r a t i o n b e t w e e n

h e a l t h c e n t r e s t a f f a n d m o t h e r s

( p . 2

0 )

0 0 0 3

1

B a r r i e r s a g a i n s t a n d

m e c h a n i s m s f o r i m p r o v i n g

t h e v i t a m i n A r e s e a r c h ^

p o l i c y ^ a c t i o n c o n n e c t i o n

( p . 5

5 )

S e m i - s

t r u c t u r e d i n t e r v i e w s

( p . 5 5

)

K e y i n f o r m a n t s t h a t w e r e

r e l e v a n t r o l e - p l a y e r s ( p

. 5 5 )

.

N ˆ n o t s t a t e d

N o t s t a t e d


. 5 5 )

2 *

T o a s s e s s t h e i n f l u e n c e o f

i n f o r m a t i o n a b o u t t h e h e a l t h

c o n s e q u e n c e s o f

u n e m p l o y m e n t a n d

i n s e c u r i t y o n p o l i c y - m a k i n g

( p . 5

)

S t r u c t u r e d , s e m i - s t r u c t u r e d

a n d o p e n - e n d e d t e l e p h o n e

i n t e r v i e w s ( p p . 5 , 9 )

P o l i c y - m a k e r s i n t h e h e a l t h

a n d e m p l o y m e n t s e c t o r s o f

a l l t h r e e l e v e l s o f C a n a d i a n

g o v e r n m e n t , a n d

e x e c u t i v e

d i r e c t o r s o f t e n C a n a d i a n

n o n - g o v e r n m e n t a l o r g a n -

i s a t i o n s ( p

. 5 ) . N

ˆ 4 8 ( p

. 5 )

N o t s t a t e d

P e r c e i v e d / r e t r o s p e c t i v e

u s e f u l n e s s o f p a s t r e s e a r c h

i n f o r m a t i o n a b o u t t h e h e a l t h

c o n s e q u e n c e s o f

u n e m p l o y m e n t a n d

i n s e c u r i t y o n p o l i c y - m a k i n g

( p . 5

)

* S t a t e d d a t e f o r w o r k i n g p a p e r i s

M a r c h 2 0 0 0

3

T o u n d e r s t a n d w h e t h e r ,

h o w , a

n d u n d e r w h a t

c o n d i t i o n s h e a l t h s e r v i c e s

r e s e a r c h p l a y s a r o l e i n

p r o v i n c i a l p o l i c y - m a k i n g

( p . i

)

B o t h p e r s o n a l , s

e m i -

s t r u c t u r e d i n t e r v i e w s a n d

s u r v e y s ( p p . 8 , 1 1 )

K e y i n f o r m a n t s w i t h i n a

v a r i e t y o f k e y h e a l t h p o l i c i e s

i n t w o C a n a d i a n p r o v i n c e s

( O n t a r i o a n d

S a s k a t c h e w a n ) ( p

. i i ) .

N ˆ

2 2 ( p

. 1 8 )

N o t s t a t e d *

A c t u a l a n d p e r c e i v e d

d e c i s i o n s w i t h i n t h e f i e l d s o f

h e a l t h s e r v i c e s j u r i s d i c t i o n ,

f i n a n c i a l a r r a n g e m e n t s

,

p r o g r a m m e d e l i v e r y

a r r a n g e m e n t s a n d

p r o g r a m m e c o n t e n t

( p p . 7 , 1 3 )

* I t i s n o t s t a t e d w h a t k i n d o f

e v i d e n c e w a s u s e d , b

u t r e f e r e n c e s

t o d i r e c t u s e o f e v i d e n c e i n

d e c i s i o n r e p o r t s , r

e p o r t i n g o n

a c t u a l u s e o f r e s e a r c h , i

m p l i e s t h a t

e v i d e n c e w a s u s e d ( p p . 1 3

, 1 8 ^ 1 9 )

r a c t i t i o n e r s ; M I T

, M a s s a c h u s e t t s I n s t i t u t e o f T e c h n o l o g y ; N H S , N a t i o n a l H e a l t h S e r v i c e ; R & D

, r e s e a r c h a n d d e v e l o p m e n t .


11/15


S t u d y m e t h o d s

fe r e n c e

S a m p l e f r a m e

( S F ) a a n d s a m p l e

s t r a t e g y ( S S ) b

R e s p o n s e

r a t e

M e a s u r e m e n t o f u s e o f e v i d e n c e

M e a s u r e m e n t o f

d e t e r m i n a n t s o f u s e

S u m m a r y o f d e s c r i p t i o n o f m e t h o d s ( c o m

p l e t e , p a r t i a l , o r n o t

a d e q u a t e l y d e s c r i b e d ) a n d c o m m e n t s

r l a n d s

1 7

S F : n o t s t a t e d

S S : n o t s t a t e d

N o t s t a t e d

A q u a n t i t a t i v e 4 - p o i n t s c o r e w e i g h t e d a c c o r d i n g t o

t h r e e d e c i s i o n l e v e l s

, w i t h s u m m e d s c o r e r a n g i n g

f r o m 0 t o 1 8 ( p

. 5 1 )

T w o - w a y A N O V A ( p

. 6 6 ) N o t a d e q u a t e l y d e s c r i b e d

8



N o t s t a t e d

D e s c r i p t i v e

N o s t a t i s t i c a l t e c h n i q u e

N o t a d e q u a t e l y d e s c r i b e d

^ 1 9 7 4 9

1 9 7 3 ^

v i i )

S F : p a r t i a l ( p . v i i i )

S S : n o t m e n t i o n e d ,

p a r t l y s n o w b a l l e d

( p . v

i i i )

9 5 % ( p

. 2 4 )

B o t h q u a n t i t a t i v e l y r a t e d a n d w e i g h t e d ( p p . 2 5

, 2 8 ,

3 3 , 3

5 , 3 6 ) a n d q u a l i t a t i v e l y d e s c r i b e d ( p p . 1 ^ 4 )

D e s c r i p t i v e l i s t i n g o f

r a t i n g s ( p p . 2 8

, 3 3 , 3 5

,

3 6 ) a n d M C A

( m u l t i v a r i a t e r e g r e s s i o n )

a n a l y s i s ( p

. 2 7 )

P a r t i a l

I t i s s t a t e d t h a t ` t h i s r e p o r t r e p r e s e n t o n l y a f i r s t s u m m a r y o f s o m e o f t h e

h i g h l i g h t s a n d m o r e i m p o r t a n t f i n d i n g s ’ ( p

. v i ) . S

u b s e q u e n t r e p o r t s n o t

f o u n d

1 1 *

S F : p a r t i a l ( p . 1 4 3 )

S S : s n o w b a l l e d ,

p a r t l y r a n d o m

( p . 1

4 3 )

N o t s t a t e d

D e s c r i p t i v e

N o s t a t i s t i c a l t e c h n i q u e

P a r t i a l

* T h e i n t e r v i e w s a p p e a r t o h a v e b e e n m a d e b e t w e e n 1 9 7 3 a n d 1 9 7 6

2 a n d U S A S F : p a r t i a l ( p . 1 6 7 )


9 0 % ( p

. 1 6 7 )

B o t h q u a n t i t a t i v e d e s c r i p t i o n s , r

a t e d a n d w e i g h t e d

a n d q u a l i t a t i v e * ( p p . 1 6 9 f f , 3 0 3 )

D e s c r i p t i v e l i s t i n g o f

r a t i n g s ( p p . 1 7 0 ^ 1 7 2 ,

1 7 4 ) a n d 2 5 v a r i a b l e s

a n a l y s e d u s i n g f a c t o r

a n a l y s i s t o d e v e l o p f i v e

d i m e n s i o n s * * ( p . 3 0 4 )

P a r t i a l

* B o t h a 4 - p o i n t ( p . 1 6 8 ) a n d a 5 - p o i n t w e i g h t e d r a t i n g s c a l e ( p p . 1 7 3 ,

3 0 3 ) . T h e q u a l i t a t i v e r a t i n g c o m e s f r o m l i s t i n g o f m o s t m e n t i o n e d

c h a r a c t e r i s t i c s o f u s e f u l n e s s

* * T h e f i v e d i m e n s i o n s a r e l i s t e d i n T a b l e 3 u n d e r f a c i l i t a t o r s

1 3

S F : P a r t i a l

( p p . 1 8 7 ^ 1 8 8 )

S S : r a n d o m s a m p l e *

( p p . 1 8 7 ^ 1 8 8 )

.

V o l u n t e e r e d

( p . 1

8 8 )

Q u a n t i t a t i v e l i s t i n g o f t h e d e c i s i o n q u a l i t y o f

e v i d e n c e ^ u s e o n s i x d i f f e r e n t l e v e l s o f c o m p l e x i t y

( p p . 1 9 5 ^ 2 0 8 )

L i s t i n g o f f r e q u e n c i e s

( p p . 1 9 6 ^ 1 9 8 )

P a r t i a l

* T h e 1 3 2 d e c i s i o n - m a k e r s w e r e r a n d o m l y d i s t r i b u t e d t o t w o o f 5 0 c a s e

s t u d i e s a n d s i x d i f f e r e n t l e v e l s ( p

. 1 9 4 )

1 4


( p . 7

9 ) *


( p . 7

9 ) *

6 5 % ( p

. 7 9 )

Q u a n t i t a t i v e l i s t i n g a n d r a t i n g s ( p p . 8 2 ^ 8 6

)

F r e q u e n c i e s a n d r a t e d

i m p o r t a n c e ( a s v e r y ,

m o d e r a t e l y o r n o t

i m p o r t a n t ) ( p . 8 2 )

* S F a n d S S c o m p l e t e i f t h e s t u d y ’ s f o c u s c o n s i d e r e d o n l y d i r e c t o r s i n

F l o r i d a a n d N o r t h C a r o l i n a , a n d i f t h e r e w e r e o n l y o n e d i r e c t o r i n e a c h

C o m m u n i t y M e n t a l H e a l t h C e n t e r

9 8 4 ^



N o t s t a t e d

Q u a l i t a t i v e a n d d e s c r i p t i v e ( p p . 1 4 6 ^ 1 5 0 )

L i s t i n g o f f r e q u e n c i e s

( p p . 1 4 5 , 1 4 9 )


T h e

a u t h o r s m a i n l y d i s c u s s a n d d e m o n s t r a t e d i f f e r e n t w a y s o f u t i l i s i n g

r e s e a r c h , a s e i t h e r : i n s t r u m e n t a l , p o l i t i c a l / c o n f l i c t , e

n l i g h t e n m e n t ,

i n t e r a c t i v e , o r t a c t i c a l ( p

. 1 4 4 ) . ` .

. . u s i n g r e s e a r c h ` ` i n s t r u m e n t a l l y ’ ’ o r f o r

` ` e n l i g h t e n m e n t ’ ’ p u r p o s e s m e a n t d i f f e r e n t t h i n g s i n d i f f e r e n t s e t t i n g s a n d

d i f f e r e n t o r g a n i z a t i o n a l p o s i t i o n s ’ ( p

. 1 4 5 )

1 6

S F : p a r t i a l ( p . 1 1 0 )

S S : c o m p l e t e *

( p . 1

1 0 )

1 0 0 % ( p

. 1 1 0 )

Q u a l i t a t i v e a n d d e s c r i p t i v e ( p

. 1 1 2 )

Q u a l i t a t i v e


* C o m p l e t e r e f e r s t o t h e 2 7 r e p r e s e n t a t i v e s i n t h e t w o p l a n n i n g

c o m

m i t t e e s i n t h e t w o c o m m u n i t i

health policy-makers perceptions of their use of evidence

Documents