disclaimer - unodc.org
TRANSCRIPT
Disclaimer
This document contains collective views of an international group of experts and does not necessarily repre-sent the views of the WHO, UNDCP or EMCDDA.
© World Health Organization, 2000
This document is not a formal publication of the World Health Organization (WHO) and all rights are reservedby the Organization. The document may, however, be freely reviewed, abstracted, reproduced and translated,in part or in whole, but not for sale or for use in conjunction with commercial purposes
International guidelines for the evaluation of treatment services and systems
for psychoactive substance use disorders
John MarsdenInstitute of Psychiatry, King’s College London, United Kingdom
Alan OgborneAlan Ogborne and Associates, London, Ontario, Canada
Michael FarrellInstitute of Psychiatry, King’s College London, United Kingdom
Brian RushCentre for Addiction and Mental Health, Toronto, Ontario
WHO UNDCP EMCDDA
World Health United Nations European Monitoring Organization International Drug Centre for Drugs
Control Programme and Drug Addiction
WHO/MSD/MSB/00.5Original: English
Distribution: General
Maria Lucia O. Souza Formigoni, Ph.D.
University of Sao PauloEscola Paulista de Medicina, Brazil
Professor Christine Godfrey
University of York, United Kingdom
Professor Wayne Hall
Executive DirectorNational Drug and Alcohol Research CentreUniversity of New South Wales, Australia
Jennifer Hillebrand
Management of Substance DependenceDepartment of Mental Health and SubstanceDependence World Health Organization
Dr Alexander Kantchelov
National Center for AddictionSofia, Bulgaria
Petra Paula Merino
European Monitoring Centre for Drugs and DrugAddiction (EMCDDA)
Dr Maristela Monteiro
Management of Substance DependenceDepartment of Mental Health and SubstanceDependence World Health Organization
Margareta Nilson
European Monitoring Centre for Drugs and DrugAddiction (EMCDDA)
Ms Christina Gynna Oguz
Chief, Demand Reduction Section,United Nations International Drug ControlProgramme (UNDCP)
Acknowledgements
2
Dr Charles Parry
Mental Health and Substance AbuseNational Urbanisation & Health ResearchProgrammeMedical Research Council, South Africa
Dr Vichai Poshyachinda
Chief of Drug Dependence Research CenterInstitute of Health ResearchChulalongkorn University, Thailand
Dr Vladimir Poznyak
Management of Substance DependenceDepartment of Mental Health and SubstanceDependence World Health Organization
Dr Edle Ranvndal
National Institute for Alcohol & Drug Research,Norway
Dr Margret Rihs-Middel
Federal Office of Public HealthMain Unit Dependency and AidsSection Policy and Research, Switzerland
Dr Juana Tomás Rosselló
Drug Abuse Treatment Adviser, Demand Reduction SectionUnited Nations International Drug ControlProgramme (UNDCP)
Dr Maree Teesson
National Drug and Alcohol Research CentreUniversity of New South Wales, Australia
This project has been partly developed under thedirectorship of Dr Mary Jansen, former WHOSubstance Abuse Department.
We would like to thank to the following reviewers who provided valuable commentaries on various drafts ofthe guidelines:
This publication is the result of an exemplary collabo-ration between the World Health Organisation(WHO), the United Nations International DrugControl Programme (UNDCP) and the EuropeanMonitoring Centre for Drugs and Drug Abuse (EMCD-DA). Already in 1996 UNDCP and WHO initiated jointwork towards the development of tools for pro-gramme managers to carry out evaluation of theirexisting treatment approaches and to help themdecide on the allocation of resources and access toeffective treatment to their patients. A series of work-books on evaluation of treatment were prepared andfield tested in several countries, and should be seen ascomplimentary to the present guidelines, for trainingand planning purposes.
The association of the EMCDDA with this projectstarted in 1997 in Athens with a seminar organisedjointly by the EMCDDA, the Greek National FocalPoint and WHO. The seminar aimed at reaching con-sensus among key European scientists, networks andpolicy makers in the area of drug abuse treatmentevaluation and the role of the EMCDDA in this field.Since then, the EMCDDA became a partner of WHOand UNDCP in this project, and will now contribute tothe diffusion of the guidelines and the associatedworkbooks across the EU.
Since the inception of the project, further internationaldevelopments have occurred and need mention. TheDeclaration on the Guiding Principles of Drug DemandReduction and the Action Plan for its implementationadopted by United Nations Member States in 1998and 1999 respectively, highlight the importance andneed for evidence-based practices as well as of learn-ing from experiences. These are key pillars for thedevelopment and implementation of enhanced drugdemand reduction strategies and programmes, a tar-get to be reached by Member States by 2003 inaccordance with the 1998 UN General AssemblyPolitical Declaration on the Drug Problem. TheUNDCP is committed to providing guidance and assis-tance to Member States in reaching this target, andviews these guidelines for treatment evaluation andthe workbook series as key instruments to encouragethe assessment of the effectiveness of drug abusetreatment strategies and activities, and the exchangeof evaluation results.
The Helsinki European Summit endorsed the EU DrugStrategy 2000-2004 in December 1999. The Strategyprovides a framework for all drug-related activities inthe EU over the next five years. The European Union
3
Action Plan on Drugs (2000-2004) set out an agreedprogramme for the implementation of the principlesand targets set in the Strategy. The Action Plan statesthat The European Monitoring Centre on Drugs andDrug Addiction will establish guidelines for the evalua-tion of drug policies in Europe.
The World Health Organisation decided to make 2001the year for Mental Health. One of the major objec-tives of the coming World Health Assembly 2001,World Health Day 2001, and the World Health Reportin 2001 will be the control of mental health problems.The promotion of this objective through the WHO willalso include recommendations for effective servicedelivery in the field of substance dependence. Thepresent guidelines will serve as one step towardsimproved treatment for psychoactive substance disor-ders internationally.
In the past, evaluation of treatment services has notbeen recognised as a means to enhance the efficacy ofexisting services. However, due to recent developmentsin health and social care systems, and budget con-straints, the need for more systematic evaluations hasincreased. Evaluations not only help to justify financialsupport for treatment services; they are also essentialto increase and maintain best clinical practice. The suc-cess of future evaluations will partly depend on theavailability of information on how to implement andplan evaluations. For this reason, practical and compre-hensive guidance on evaluation is necessary. This publi-cation is especially addressed to policy makers, pro-gramme managers and service providers in the field ofdrug demand reduction who will find the guidelines aunique source of reference. The guidelines will helpdecisions to be made about critical aspects, especiallyat the moment of conceptualising, planning and com-missioning the evaluation of treatment services. Theyshould also serve as a valuable reference in designingevaluation and to make optimal use of availableresources. In order to improve drug demand reductionpolicies in the area of assistance to drug abusers, theguidelines are also aimed at disseminating critical, sci-entifically supported knowledge in this regard.
We would like to thank the authors of the guidelinesfor providing a compact and succinct overview of dif-ferent types of evaluation. We would also like toexpress our thanks to all experts and policy makerswho helped to review the guidelines by providing theirvaluable comments on earlier drafts.
WHO/UNDCP/EMCDDA
Foreword to the WHO/UNDCP/EMCDDAguidelines for treatment evaluation
Contents
4
Acknowledgements
Foreword
List of boxes
Preface
Background
Structure of the guidelines
Section 1: Setting the stage for evaluation
Communication with interested partiesResources Setting objectivesGetting specialist adviceEthical issues
Section 2: The main types of evaluation
Needs assessmentProcess evaluationClient satisfaction studiesOutcome evaluationNaturalistic, observation studiesExperimental, controlled designsEconomic evaluationsCost analysisCost-effectiveness analysisCost-utility and cost benefit analysis
Section 3: Special issues
Reliability, validity and sensitivity of measuresReliability of self-report dataTime periods for measuresAssessing non-treatment factors that may affect outcomeSubject recruitment and follow-upSampling and sample sizeStatistical testing and confidence intervals
Section 4: Reporting results
Increasing the impact of evaluation
Summary and conclusions
References
2
3
5
6
7
7
8
899
1212
13
13151617181819191920
21
21212122222323
24
24
26
27
5
List of boxes
9
10
11
13
14
15
16
17
18
25
Box 1
Components of a treatment programme
Box 2
Logic model for a hypothetical detoxification centre
Box 3
Logic model for hypothetical youth services programme
Box 4
The five major types of evaluation
Box 5
Major types of activities in need assessments
Box 6
Questions and issues addressed in process evaluation
Box 7
Client characteristics considered in process and/or outcome studies
Box 8
Major activities in process evaluation
Box 9
Questions and issues addressed in client satisfaction studies
Box 10
Structure and content of a formal report from an outcome evaluation
6
We have tried to prepare the guidelines using anon-technical language. Nevertheless, some evalua-tion issues of a more technical nature have beencovered in Section 3: Special Issues. Those who wishto read a more general account of evaluation meth-ods and issues should skip this section. Finally, wedo encourage those planning an evaluation toobtain expert advice wherever feasible. Gettingappropriate advice early on in the planning of anevaluation is very important and can ensure thatappropriate resources are identified, that the correctevaluation design is selected, and the data is analy-ses and correctly presented.
We hope to find the guidelines useful and warmlywelcome feedback and suggestions.
Please write to:
World Health Organization (WHO)
Management of Substance Dependence
Department of Mental Health and Substance Dependence
Avenue Appia 201211 Geneva 27Switzerland
These guidelines describe methods for the evalua-tion of treatment services and systems for substanceuse disorders. The guides are intended to be a com-panion resource to the World Health Organization(WHO), United Nations Drug Control Program(UNDCP) and European Monitoring Centre onDrugs and Drug Addiction (EMCDDA) workbookseries on the evaluation of costs and effects of treat-ment for substance use disorders, co-ordinated byDr. Maristela Monteiro and Dr Brian Rush (WHO,2000).
The main audience is policy makers, commissionersof treatment services and treatment agency person-nel who want to know more about research evalua-tion and commission or undertake evaluation stud-ies. We hope that there will be much to interest theinternational research community as well.
The guidelines are intended to be of particular inter-est for countries where research and evaluation inthis area is not widely developed. Our aim is to offera concise description of the main evaluation meth-ods and to help the reader to select the best type ofstudy to answer specific questions. The goal is toencourage readers to the use of evaluation to helpdevelop and maintain effective and efficient treat-ment services and treatment systems.
These guidelines are not intended to be a detailed,step-by-step instruction manual on how to doresearch. The interested reader should obtain therelated series of workbooks which cover this indetail (WHO/UNDCP/EMCDDA, 2000). We havereferred to particular workbooks in several places forfurther reading and this document is a natural com-panion to the series. There are also other valuablesources of information which can be accessed viathe internet. For example, valuable information canbe found at the Cochrane Collaboration site forinformation on treatment effectiveness (seehttp://www.cochrane.org); an evaluation instrumentbank can be accessed at the ECMDDA site (seehttp://www.emcdda.org); and a series of best prac-tice guides on treatment and evaluation issues canbe obtained from the US Center for SubstanceAbuse Treatment (see http://text.nlm.nih.gov).
Preface
7
The misuse of illicit drugs, prescription drugs or alco-hol is a world-wide problem and there is an interna-tional commitment to providing prevention andtreatment services. Estimates have suggested thatthe annual global population prevalence of illicitdrug use is between three and four percent; preva-lence estimates for heroin and cocaine use areapproximately 8 million and 13.3 million adultsrespectively (United Nations International DrugControl Programme [UNDCP], 1997). Prescriptiondrug abuse is known to be high in some countries(Adams et al., 1993) and alcohol-related problemscan far exceed those involving all other drugs interm of prevalence and costs to society (Single et al,1996). Several different types of services and treat-ments have been developed for substance abuse dis-orders. These can be broadly categorised as follows:
■ “Open access” services:■ advice, education, information, early interven-
tion programmes;■ needle/syringe exchange programmes (in some
nations/regions)
■ “Structured” services: ■ prescribing interventions (inpatient and outpa-
tient/community settings);■ community-based psychosocial counselling and
relapse prevention;■ residential rehabilitation programmes
These five categories are really only part of the pic-ture. Social reintegration and support services maysometimes be available to support treatment gainsfrom the above services. Space does not permit usto give a full account of the various modalities oftreatment and the array of different providers. Theseconcern the specific approach or philosophy of atreatment – such as maintenance or reductionregimes in substitution treatment for opiate users, orthe style of psychosocial counselling (e.g. cognitivebehavioural), or the philosophy of residential reha-bilitation (e.g. 12-step; therapeutic community).
It is quite common for a treatment programme tocontain several different therapeutic componentslinked together (e.g. methadone maintenance withcognitive behavioural psychotherapy). It is also quitecommon for a person with substance-related diffi-culties to receive health and social care services fromseveral service providers during the course of their
treatment. Support and assistance may also comefrom the family and friends and other support net-works. These practical aspects of treatment haveimplications for how evaluation research in this fieldis constructed and the questions addressed.
There is reason for optimism about investing in thetreatment of people with substance-related prob-lems. Many careful studies have shown that treat-ment can result in short, medium, and sometimeslong-term reductions in substance use, improve-ments in health and reduced demands placed onhealth and social services among those treated (seereviews by Landry 1995; Miller and Hester, 1986;WHO Expert Committee on Drug Abuse, 1996;Roberts and Ogborne, 1999; National Institute onDrug Abuse, 1999). There is also good evidencethat some types of treatment can reduce criminalityand the associated costs of law enforcement amongsubstance abusers who maintain their habits bycrime (Tims, 1995). The economic benefits of treat-ment can also exceed treatment costs (Holder andBlose, 1992; Holder et al., 1991; Centre forSubstance Abuse Treatment, 1999) and by the samecriteria, treatment compares favourably with inter-diction and other law enforcement efforts directedat individual drug users (Rydell and Everingham,1994).
Structure of the Guidelines
The guidelines have four sections. Section 1 sets thestage for evaluation research and describes theimportance of assessing the feasibility of a study interms of the questions to be addressed, theresources required, ethical considerations and consul-tation with interested parties. Section 2 summarisesthe key features of the five main types of researchevaluation: needs assessment, process evaluation,client satisfaction studies, outcome evaluation andeconomic evaluations. Section 3 describes seven spe-cial issues of a more technical nature: the reliability,validity and sensitivity of measures, reliability of self-report data, time periods for measures, assessingnon-treatment factors that may affect outcome, sub-ject recruitment and follow-up, sampling and samplesize, and statistical analysis and testing. Section 4offers guidance of report writing and ensuring resultsfrom evaluation studies are effectively presented tokey audiences.
Background
An evaluation is done when one wants to assess thecosts, effects or impact of a treatment or treatmentsystem. Evaluation therefore provides feedback tokey audiences to help them with various types ofdecisions. A healthy culture for evaluation is onethat encourages the routine collection and dissemi-nation of information to help improve how servicesare delivered. Nevertheless, in many countries, theevaluation of treatments for substance use disordersis in development. In fact, evaluation may well havehad a minor role in influencing what treatment serv-ices have been developed in a country. Some policymakers and planners have relied on personal experi-ence, opinions and testimonials rather than researchdata or published studies. This has led to disparitiesin the development and management of treatmentservices and to the support of interventions ofuncertain effectiveness or efficiency. As we shallshow, evaluation research can help identify whichtreatment services are needed as well as assess theefficiency and effectiveness of existing services.
Prior to starting any type of evaluation, it is a goodidea to assess the feasibility of the desired research(see Wholey, 1994). A feasibility assessment shouldexamine the following:■ results from previous research; ■ the specific questions to be addressed; ■ the size and scope of the study;■ the human resources which will be needed;■ the skills needed by the research team;■ the financial resources needed;■ the timetable for the study
8
Communication with interested parties
Discussions with interested groups – such as serviceproviders, clients of treatment programmes, andtreatment funding agencies – can be an invaluablemeans of learning about the extent of support forthe study, the direction it should take and practicalissues concerning its implementation. In most situa-tions those with the greatest interest in evaluation(the stakeholders) will be treatment programme per-sonnel, representatives from the community, fund-ing bodies and government. It may be very helpfulfor some stakeholders to serve as members of anadvisory committee for the study.
Good communication with key stakeholdersthroughout the implementation of an evaluation isvital and they should be involved in an early discus-sion of findings and implications. It is also importantthat a clear understanding of the informationrequirements and interests of the funding body issecured. Discussions with all relevant stakeholdersshould be held at the outset and their views andconcerns sought throughout the study. These dis-cussions will help to formulate the central questionsto be addressed. The aim is to clarify who wants toknow what, by when, with what degree of preci-sion, and at what cost. Each stakeholder may haveunique experiences and perspectives that can con-tribute to the overall understanding of the issuesand to the design and implementation of usefulevaluations.
Different groups may, of course, have different ideasor emphases on what to evaluate. For example, pol-icy makers and service purchasers may be mostinterested in costs and efficiency, while serviceprovider staff may be more interested in assessingthe benefits of a new treatment. Naturally, the num-ber of questions which are worth looking at mayover-stretch the time and resources available. If thisis the case, it is essential that the evaluation teamlook at the questions that have the highest priority.
Setting the Stage for EvaluationSection 1
9
Setting objectives
When thinking about the objectives for an evalua-tion of a treatment programme, it is important to beclear about how it operates and what it is expectedto achieve. A “logic model” can be of help in repre-senting these elements as a diagram (see Rush andOgborne, 1991). A logic model describes the differ-ent components of a treatment programme andwhat these are designed to achieve (see definitionsbox 1). When fully developed, logic models can leadnaturally to the design of management informationsystems for the monitoring of services. A logicmodel can also be useful when inducting new staffand for communicating with funding bodies, boardmembers, clients and others with an interest in theprogramme. Two examples of logic models areshown in boxes 2 and 3.
Resources
Some evaluations can be done quite quickly and donot require major resources. In fact, relatively simpleand modest studies can be of great value in learningmore about how services operate and how they canbe improved (Rossi and Freeman, 1982). Othertypes of evaluation are complex and may need asubstantial investment of funds to be done well. It isalso important to ensure that those to be involvedwill have enough time to devote to critical tasks andthat the timetable for the project is realistic. In somecases resources are allocated and set aside when atreatment programme is in the planning phase. Thisis the ideal situation and policy makers are encour-aged to support evaluation by earmarking adequateresources for the evaluation of all new treatment ini-tiatives. More commonly, evaluations require specialgrants from governments and other domestic andinternational funding bodies.
ELEMENTS DESCRIPTION
Programme or system components Activities which are directed to the attainment of specific goals
Implementation objectives What a programme or system seeks to achieve
Short-term outcome objectives Changes which the programme or system seeks to induce in its target clients in the immediate or short term.
Long-term outcome objectives Changes which the programme or system seeks to induce in its target clients in the longer term.
Components of a treatment programme Box 1
10
Logic model for a hypothetical detoxification centre Box 2
Long-term Outcome Objectives
ASSESSMENT& INTAKE
DIAGNOSIS &TREATMENTPLANNING
WITHDRAWALMANAGEMENT& TREATMENT
REFERRAL
To determine eligibility for
service■
To assess motivation for
treatment■
To determine individual needs
■
To obtain standard somatic, mental
and psychological information
■
To form therapeutic
alliance with client and family
■
To provide information on
programme
To provide information
about treatment options
■
To motivate clients to seek
further treatment
To monitor withdrawal symptoms
■
To prescribe standard
medication■
To conduct laboratory and
other tests■
To educate family and clients
■
To motivate clients to
complete the programme
■
To manage withdrawal from
alcohol
To obtain an ICD-10 diagnosis
■
To develop a treatment plan
■
To sign a therapeutic
contract
Main Components
ImplementationObjectives
Short-termOutcomeObjectives
To increase client’s motivation for detoxification
■
To identify and refer clients needing
medical care
Complete withdrawal from alcohol
■
Stabilisation of mental and physical status
To increase client awareness of treatment
options■
To increase changes that clients will accept a
referral for treatment
Reduced risk of relapse to alcohol abuse■
Increased social integration■
Reduced public drunkenness
11
Logic model for hypothetical youth services programme Box 3
Long-term Outcome Objectives
Main Components
ImplementationObjectives
Short-termOutcomeObjectives
ASSESSMENT& INTAKE
PARENTPROGRAMME
COUNSELLING EDUCATION STABILISATION
To gather basic
information on clients
■
To administer standard
assessment instruments
■
To develop a treatment plan
with youth and others
To teach appropriate
parenting skills and techniques
■
To provide a forum for the
exchangeof mutual
support for parents in
a group format■
To educate parents about consequences of PS use on
the youth and the family in a group format
To provide decision focussed groups
■
To provide individual and group counselling
■
To implement treatment
plans
To teach risk avoidance
■
To provide information
on the consequences of drug use
To provide crisis inter-ventions to adolescents
■
To help clients find accom-modation
■
To teach basic life skills
To increase client's
self-esteem
To increase self-efficacy in avoiding high risk situations
■
To increase knowledge of
the impact and conse-quences
of PSU
To stabilise clients and
their situations■
To improve life skills
To reduce psychoactive drug use
To reduce the negative consequences of psychoactive
drug use
To improve the general well being of the client and his/her family system
To improve family
relationships
Getting specialist advice
On occasions, assistance should be sought fromspecialists from countries where similar evaluationshave been conducted. A certain amount of technicalexpertise is also required. Particularly by the follow-ing areas:■ Economic analysis;■ Effectiveness and cost-effectiveness;■ Treatment system performance;■ Statistical analysis;■ Study planning and conceptualisation;
Although readily available in some countries, othercountries may have limited expertise in some ofthese areas. However, there are usually researchinstitutions and universities whose staff can assist inthe planning or conducting evaluations and prepar-ing. Also students from social science, social workand other relevant courses may be willing to assistin data collection and analysis to gain work experi-ence in the evaluation field. It is, however, impor-tant to recognise that the involvement of universityor college faculty and students may have implica-tions for the timing of evaluation.
12
Ethical issues
Evaluation can raise ethical concerns. It is critical toensure that everyone involved understands the pur-poses of the evaluation and gives informed consentto participate. The concerns of potential participantsshould be addressed at this stage, particularly issuesrelating to confidentiality. Anyone asked to provideinformation for an evaluation or to undergo otherprocedures for research purposes (e.g. randomassignment to a new form of treatment) should beasked to sign an informed consent form if they wishto participate, following adequate information andresolution of any questions. Confidentiality of infor-mation should be assured and respected. If there arecircumstances under which confidentiality could bebreached (e.g. records could be subpoenaed by thecourts) then this must be clearly explained inadvance. An appropriate ethics review body shouldconsider all studies that involve human subjects.
13
Needs assessment
The goal of a needs assessment is to estimate thecurrent and/or future nature, prevalence and inci-dence of substance use disorders in a specific popu-lation. This information is then used to help to guidethe development and operation of treatment andsupport services. Ideally, a needs assessment shouldbe undertaken before a treatment or network oftreatment services is planned and implemented.
In this section, we describe five main types of evalu-ation: (a) needs assessment (b) process evaluation(c) client satisfaction (d) outcome evaluation and (e)economic evaluation. These may overlap to greateror lesser degrees. As indicated in Box 4, each typeof evaluation aims to answer certain types of ques-tions and the decision to undertake a particular typeof evaluation will be determined by the questions ofinterest.
TYPE OF EVALUATION EXAMPLES OF QUESTIONS ADDRESSED
Needs assessment What is the nature and extent of substance use disorders in the community?
How well do current services address the needs of the population at risk?
Process evaluation What is the programme/service/system expected to accomplish?
To what extent is the treatment implemented as planned (programme fidelity)?
Are services of good quality?
What are the views and perceptions of the programme staff on the value and quality of the service?
What are the characteristics of the clients of the service?
Which clients do not complete treatment?
Outcome evaluation What proportion of clients complete treatment?
What proportion of clients reduce their drug use after treatment?
Are clients better off than before treatment?
Is our treatment better than no treatment or better than other treatments?
Why does the programme achieve the outcomes that it does?
Client satisfaction Do clients feel that the treatment met their needs?
What did clients like least or find unhelpful?
Why do some people drop out?
Economic evaluation
Cost studies What is the average cost per client?
Cost effectiveness How do two or more treatments compare in terms of cost per unit of outcome?
Cost utility Given their relative costs, how do two or more treatments compare with respect to the quality and quantity of the life of their clients after treatment?
Cost benefit Does the programme yield more benefits than costs valuing everything in money terms?
The five major types of evaluation
The Main Types of EvaluationSection 2
Box 4
14
There is no universally agreed definition of 'need' orhow one should assess it. From the perspective ofan individual, “felt need” is what the person wants(Bradshaw, 1972) and from an epidemiological per-spective, need can be defined by the ability of thatpopulation to benefit from effective health careservices (Stevens and Raftery, 1994). Sometimes'need' is used to refer to what people in a commu-nity believe should be provided or what problemsprofessionals think should be treated. A needsassessment can address the gap between existingservices and those needed to adequately treat ormanage known problems in the target population(Marsden et al., forthcoming). This view of needsexplicitly links the research evidence for particularinterventions into health and social care planningand delivery.
Some needs assessments use statistical methods toestimate the size of the population in need of treat-ment from survey, treatment utilisation and/or otherindicator data (see DeWit and Rush, 1996). Needsassessments can also draw on public and profession-al opinions about what works and what should bedone and they are often commissioned by fundingagencies wanting guidance on how to allocate exist-ing or new funds. The main types of activities
involved in needs assessments are summarised inbox 5.
Ideally, needs assessment should be an ongoing andflexible process that is sensitive to changes in thelevels and types of problems within the areas ofconcern. Change may be seen in the followingareas: ■ New trends in substance use and consumption
patterns in the target population;■ Movements in the geographical distribution and
concentration of psychoactive substance use; ■ Fluctuations in the demand for services; ■ New barriers to accessing treatment encountered
by special groups;■ The relationship between substance use and other
conditions (notably HIV infection, and bloodborne viral hepatitis);
■ National, regional and local policies in response todrugs;
■ The organisation of health and social care services;■ Changes in the resources available for treatment;
and ■ New evidence for the effectiveness of treatment
interventions.
■ Defining the geographic boundaries of the population(s) whose needs are to be considered (local,regional or national)
■ Use of existing population data on the prevalence and incidence of specific problems in the target popu-lation and in sub-groups
■ Use of indicators to provide indirect estimates of the prevalence and incidence of specific problems inthe target population and in sub-groups
■ Special population surveys that focus on the issues of concern ■ Focus group discussion with key stakeholders (commissioners, clinicians, treatment providers and service
users) to explore what they want from services■ Reviews of service arrangement in other, similar jurisdictions■ Reviews of the literature on the cost-effectiveness of particular types of interventions ■ Identifying current treatment services and auditing their capacity, use and waiting lists■ Comparing existing arrangements with generally agreed-upon needs or with different perspectives on
existing needs ■ Preparation of recommendations for increasing treatment coverage, purchasing efficiency and service
effectiveness.
Major types of activities in need assessments Box 5
15
Process evaluation
Process evaluations look at how a treatment or aprogramme operates. Issues such as the efficiencyand quality of treatment services are addressed andan evaluation made of the extent to which a treat-ment has been implemented as intended (fidelityassessment). Process evaluations also try to identifyareas where improvements can be made. Processevaluations are not directly concerned with changesin clients that may result from the treatmentsreceived. They focus on how treatment services orsystems operate and the ways in which resourcesare used to produce outputs (e.g. number of clientsassessed; numbers treated). Specific questions for aprocess evaluation can include the following: ■ The extent to which a service or system is concep-
tually or theoretically sound;■ Whether the clients served are those intended;
■ Whether clients experience difficulty in accessingthe programme;
■ The extent to which the programme staff are ade-quately trained and the extent to which they aresatisfied with their work.
Process evaluations may also consider internal andexternal coordination, organisational and servicemanagement issues and case management andrecord keeping (see box 6 and 7). The steps thatmay be taken when conducting a process evaluationare shown in box 8:
Questions and issues addressed in process evaluation
■ Focus on single programme, service or agency■ Number and characteristics of clients served (see box 7)■ Nature and extent of treatment(s) provided■ Ease of access to target groups■ Costs of service■ Programme logic■ Staff qualifications■ Quality of records■ Discharge planning and aftercare arrangements■ Case management ■ Internal and external co-ordination■ Compliance with established standards■ Characteristics of environment that impact programme delivery■ Focus on a network or system of services or agencies■ Characteristics of clients served by different agencies (see box 7)■ Referral and cross-referral patterns■ Case management across services■ Intra-agency co-ordination in planning for direct client service ■ Characteristics of broader environment that impact the function on the system
Box 6
Process evaluations are closely related to pro-gramme audits and continuous quality improvement(CQI) initiatives. CQI focuses on customer satisfac-tion, accurate measurement of activities, ongoingimprovement of services and operational processes,and meaningful involvement of people at all levelsin the organizational process. Other ingredients,including programme logic models, integrated carepathways, client satisfaction surveys and routineoutcome monitoring, can all be part of CQI. Theresults of process evaluations may have importantimplications for individual staff and managers.
Space limitations preclude a discussion of routinetreatment utilisation and audit systems. These pro-vide valuable information about the operation of atreatment system. Setting up monitoring systemsthat routinely provide information on the operationof a treatment service or system can also reduce thecosts of process evaluations.
Client satisfaction studies
The assessment of client satisfaction can add animportant “consumer” perspective to evaluations oftreatment services and systems (Lebour, 1983).Without such a perspective, problems such as highdrop out or low success rates might be related onlyto client factors when, in reality, these problemsmay reflect shortcomings in the services provided orin the treatment system. However, it is important torecognise that evidence of client satisfaction is not,in itself, sufficient to establish the effectiveness of atreatment. Clients with no base for comparison maybe satisfied with services that more objective meas-ures show are of poor quality. In the short-term,clients may also indicate high satisfaction with serv-ices that are well run yet quite ineffective. Likewise,clients may express dissatisfaction with services thatare successful nonetheless in reducing their sub-stance use.
Client satisfaction surveys can be used for routine orperiodic “check-ups” on the quality of services fromthe clients' perspectives. They can also be used toassess client reactions to changes in service delivery.For example, a service may have planned changesto increase its efficiency but be concerned that thesecould lead to decreased client satisfaction. Clientsatisfaction surveys are most useful when they are
16
designed to meet specific objectives and when theyuse appropriate methods and measures. Samplingprocedures, timing, cultural acceptability and sensi-tivity of the questions to various levels of satisfac-tion must all be considered carefully.
The most common method for assessing client satis-faction is to use a confidential self-administeredquestionnaire. This can be given to clients at dis-charge or follow-up. In some cases, confidential per-sonal or telephone interviews, or focus groups maybe more appropriate. If interviews or focus groupsare used, it is preferable to have them conducted bysomeone who is not directly connected with the
Client characteristics considered in process and/or outcome studies
■ Number of clients served■ Age, gender (intake)■ Source of referral (intake)■ Treatment history (intake)■ Motivation for treatment and readiness for
change (intake)■ Post-intake treatment from index treatment
and elsewhere■ Marital/partnership status■ Education■ Work situation/income levels■ Accommodation status and stability■ Involvement with justice system■ Health and social service use■ Alcohol and drug use levels and patterns■ Level of alcohol/drug dependence■ Adverse consequences of alcohol/drug use■ Mental health problems■ Physical health problems■ Self-esteem■ Coping skills■ Social relationships
Box 7
17
service. This may be an independent evaluator, vol-unteer or former client trained to take on this role.In all cases clients should be assured that theirresponses would not in any way affect their presentor future treatment.
It is important to take cultural differences intoaccount regarding expectations of feedback on pub-lic and private services. In jurisdictions where con-sumerism is firmly established, frank verbal or writ-ten feedback may be freely given. However, directnegative feedback in some cultures may be consid-ered impolite and complaints may only be sharedwith intimate acquaintances. Direct and challengingquestions may also be culturally inappropriate.Experiences with (and attitudes toward) the use ofquestionnaires, interviews, focus groups and othermethods of enquiry also differ between cultures.Methods for soliciting client feedback must take intoaccount the prevailing cultural norms and seek toensure the use of appropriate methods that assessclient beliefs and opinions. In addition to using con-fidential survey methods there is increasing interestin establishing channels through which service userscan have an active input into service delivery. Someof the specific issues that could be addressed inclient satisfaction studies are indicated in box 9.
Outcome evaluations
Outcome evaluations look at whether clients of atreatment service or system have changed over timeand how much of this can be attributed to the carereceived. Although such evaluations sometimesinvolve single clients, usually one or more groups ofclients that have received a certain treatment withinthe system are involved. In the past, most outcomestudies of people treated with substance use prob-lems used post-treatment abstinence from sub-stances as a primary indicator of treatment success.However, modern studies focus more on patternsand levels of post-treatment substance use and alsoconsider post-treatment functioning in other lifeareas. Three other problem domains are usuallyassessed: health risk behaviours, health problemsand various aspects of personal/social functioning,including employment, family and other personalrelationship problems and criminal behaviour. Box 7summarises some of the specific issues commonlyconsidered. Further details and information on spe-cific measurements and procedures are provided inthe WHO/UNDCP/EMCDDA workbook on out-come evaluation. We recommend that expert adviceis sought in the selection of measures for particularstudies, especially in cases where new measures areto be developed. Particularly attention should bepaid to issues of reliability and validity (seeSection 3: Special issues).
Major activities in process evaluations
■ Review of original planning documents concerning the need for the service/system, the objectives, thespecific treatment methods used and the resources that were thought be to required
■ Focus groups or interviews with programme participants or others such as community planners, man-agers, staff, community agency representatives and families
■ Development and critique of programme logic models ■ Chart audits■ Analysis of information from built-in management information systems■ Observation of the programme(s) in action, for example, to rate the extent and quality of implementa-
tion and integration with other programmes;■ Surveys of the population for whom the programme or treatment system was intended and of actual
programme participants.
Box 8
18
achieved and to identify which clients change mostor least. They can also show how outcomes varywith the amount or type of treatment received.However, observational designs cannot definitivelyconclude that any observed changes were caused bythe treatment(s) of interest. This is because they donot control for many of the other factors that mightexplain the changes observed (e.g. non-treatmentfactors, maturation, cyclical changes).
In some circumstances it may be feasible to obtainone or more comparison groups of cases that aresimilar to the clients of a target treatment but whohave not been treated or have been treated bysome alternative means. If the groups are initiallyvery similar, then it can be assumed that any differ-ences over time are at least partly due to differencesin treatment. Suitable comparison groups might becases treated in a different programme or casesincluded in studies reported in the literature.However, it is important to ensure that the treat-ment and comparison groups are as similar as possi-ble and not specifically selected to give them anadvantage in any one case (e.g. screened for moti-vation or social stability).
Experimental, controlled designs
Experimental designs involve the allocation of clientsto one or more treatments or to a no-treatment orwaiting list (delayed treatment condition). In ran-domised controlled trials, a simple randomisation orgroup balancing procedure is used to allocate clientsto the experimental or comparison treatment (usual-ly the “standard” form of care which is usuallydelivered) condition. In substance use treatment,these designs are often conducted in the context ofthe day-to-day operation of a treatment programmeand are known as randomised clinical trials or ran-domised field experiments (see Dennis 1994).
When properly conducted, experimental designs canproduce the most convincing evidence that meas-ured outcomes were due to the treatment studied.However the validity of experimental designs can becompromised unless care is taken to adhere to thestudy procedure and the treatments are implement-ed as planned. A common problem is that it may bedifficult to ensure that those in different treatmentconditions actually receive the same amount of
When conducting an outcome evaluation, it isimportant to develop a theory or at least a frame-work for accounting for why a treatment processachieves a measured outcome (Chen, 1990). Processevaluations are the logical partner to outcome eval-uation and good quality outcome evaluation shouldalways have a process component. There are severaldifferent designs that can be used in outcome evalu-ations.
Naturalistic, observation studies
In naturalistic or observational studies clients areassessed on a set of measures at several points intime – usually before, during and after they com-plete a treatment programme of interest. Thesestudies are useful for the study of treatment systemswhere clients engage in multiple treatments of vary-ing intensities and duration. There have been severalmajor multi-site naturalistic post studies conductedincluding the Drug Abuse Treatment OutcomeStudy (DATOS) in the United States and theNational Treatment Outcome Study (NTORS) in theUnited Kingdom (http://www.datos.org;http://www.ntors.org.uk). These sorts of study canbe used to show if the desired client outcomes are
Questions and issues addressed in client satisfaction studies
■ Physical setting of treatment■ Helpfulness of staff■ Costs of treatment■ Accessibility of services■ Family involvement■ Waiting times■ Information■ Amount of service■ Effectiveness■ Case management and co-ordination■ Confidentiality
Box 9
attention when two treatment methods are beingcompared. For example, clients and therapists maynot be equally enthusiastic about the treatmentsbeing compared. If not, the outcomes for sometreatments may be deflated because no onebelieved they would work or made much effort todeliver them as planned. As with comparison groupstudies there are technical and logistical problemswith the design and conduct of experimental evalu-ations. Consultation with an experienced evaluatoris strongly recommended if an experimental designis being considered.
Economic evaluations
Economic evaluations assess the resources requiredto provide treatment and the resulting benefits. Acentral question posed by many economic evalua-tions is whether the treatment or treatment systemstudied is an efficient use of resources (seeDrummond et al., 1987). Treatment funding agen-cies are increasingly demanding that treatment serv-ices and systems be evaluated from an economicperspective. However, the economic evaluation oftreatment is still an emerging discipline and thereare only a few published studies that can be charac-terised as full economic evaluations.
It is important to be aware that economic evalua-tions are undertaken from a particular perspectiveand that the results can vary with the perspectivetaken. Generally, the perspective taken is that ofsociety as a whole or that of the agency providingfunds for treatment (usually the government).However, economic evaluations can also be con-ducted from the perspectives of clients or familymembers. Overall, economic evaluations can becomplex and time consuming; since there are manytechnical and logistical issues to consider we recom-mend that consultation with an expert is well worth-while. There are three types of economic evaluation:
19
Cost analysis
A cost analysis determines the overall cost of theresources used to provide a treatment activity, serv-ice or system. These may include both direct costs(such as clinical staff salaries) as well as overheadssuch as building and maintenance costs, administra-tive salaries, equipment and supplies. The amount ofdirect resources used for specific activities can bemeasured and the indirect resources apportioned.The results are then used to compute various indica-tors of economic performance (e.g. costs of residen-tial treatment per client per week or episode; costper counselling visit). This type of economic evalua-tion is, however, quite limited and it is important torecognise that the relative costs of treatment pro-grammes may not reflect their relative effectiveness.Sometimes lower cost programmes may be as effec-tive as those costing a lot more (Holder et al.,1991). However, for people who also have multiplesocial and mental health problems in addition totheir substance-related difficulties, comprehensive,and usually higher cost interventions are likely to bemore effective than more basic and lower cost inter-ventions (McLellan et al. 1998).
Cost-effectiveness analysis
Cost-effectiveness analysis (CEA) compares the costsand outcomes of two or more different treatmentswith similar objectives. Examples of outcomes usedfor a CEA include: “number of cases that achieve acertain outcome or degree of success” or “durationof abstinence of rehabilitation following discharge”or perhaps a “reduction in substance-related prob-lems”. Multiple outcomes are sometimes consideredand the results may vary with different outcomes.For example, one treatment may be more cost effec-tive than another at reducing drug use but less costeffective at increasing employment. The first stage inCEA is usually to estimate the costs per patient out-come achieved (via a decision analysis). The secondstage involves comparison of the cost-effectivenessratios for each of the treatments studied.
Cost-utility and cost benefit analysis
Cost utility analysis (CUA) uses ratings of the qualityof life and life expectancy and survival of clientswho have been treated as primary outcome meas-ures. Often these ratings are used to compute ameasure of 'quality adjusted life years' (QALYs) thatare used to indicate a degree of discrepancybetween a client's actual outcome status and anideal status from a client's own perspective. CUA isincreasingly viewed as more appropriate for healthcase evaluation than cost-effectiveness analysisbecause the latter does not take intangible benefitsor the quality of outcomes into consideration.
In cost benefit analysis (CBA) the costs of deliveringtreatment and the outcomes from treatment areboth measured in monetary units. For example, thecosts of health or social services used or theincreased tax revenues paid by employed drugusers. These types of studies are uncommon andthere is some criticism of the appropriateness ofmeasuring outcomes solely in monetary terms. Inaddition to other technical matters, these studiesrequire attention to the issues of discounting. Thisoccurs because the economic benefits of a pro-gramme can occur long after the programme costsare used up. The 'future values’ of these benefitstherefore need to be assessed in terms of presentday realities and the lost opportunities to useresources in other ways. Comprehensive economicevaluations require two or more suitable alternativesto be evaluated. The most desirable design todemonstrate robust effects is the randomised con-trolled trial. However, the choice of design for aneconomic evaluation should be guided by the mainquestions being addressed.
20
In this section, we discuss a number of importantissues that are of a more technical nature. Theyapply to all types of evaluations – but there severalspecific matters that are of particular concern in out-come studies. Guidelines on data analysis arebeyond the scope of this document and space limi-tations also prohibit discussion of the distinctionsbetween qualitative and quantitative evaluationmethods. Basic information on data collection andanalysis and on other technical issues can be foundin the WHO/UNDCP/EMCDDA workbooks. Theissues are also covered in numerous basic andadvanced textbooks on evaluation.
Reliability, validity and sensitivity ofmeasures
Measures are observations or ratings of behaviour,attitudes, beliefs and the like. The principal require-ments of a measuring device (such as a question-naire) are that it should measure what it is supposedto measure (validity) and do so consistently (reliabili-ty). There are several ways to assess validity andreliability that allow different inferences to be made.The reliability of measures used in evaluationresearch can be assessed by examining consistencyacross time (test-retest-reliability) and consistencyacross observers (inter-rater reliability). Internal con-sistency is a useful measure of reliability for multi-item scales. This indicates the extent to which allitems measure a single construct.
Several kinds of validity can be used for the meas-ures used in programme evaluation. Face validityrefers to whether the contents of specific questionsor measures reflect what the indicator is supposedto measure. Assessments of validity of a measureare usually undertaken in three ways:
■ coverage of all the salient features of the intendedconstruct (content validity);
■ by ensuring that the instrument does not includephenomena germane to other constructs (con-struct validity); and
■ there is a strong positive correlation betweenscores on the instrument and other related instru-ments designed to measure the same construct(concurrent validity).
21
Special Issues
Ideally, a chosen measure must also be free fromresponse bias and be non-reactive (insensitive) toextraneous factors (including physical setting, clientexpectation, staff behaviour and accountability pres-sures). Measures must also be sensitive to changesin the phenomena studied. A measure may be validyet not sufficiently sensitive to measure subtle butimportant changes. Generally, the sensitivity of ameasure increases with the range of possible scoresit has. Thus a dichotomous response item – (e.g. yesor no) – will probably be less sensitive than a scalemade up of several items.
Reliability of self-report data
Self-report is the most commonly used data sourcein evaluation studies. There is sometimes concernabout the reliably and validity of such reports. Oftenthese concerns are overstated. The validity of self-report information cannot be considered independ-ently from the method used to obtain the reportand will therefore vary across measurement con-texts. The extent to which self-report is reliable,depends on a set of dynamic factors (see Babor,Brown and Del Boca, 1990). These include the fol-lowing: ■ the characteristics of the client (e.g. physical and
psychological state; motivation and fatigue); ■ contextual and task variables (e.g. context and
setting for the data collection; confidentialityassurances; nature of the questions and answerformat); and
■ cognitive factors (e.g. comprehension and memo-ry and retrieval processes)
In addition to client self-report there may be anopportunity to gather other collateral informationincluding biological data. Biological measures suchas urine, blood or even hair can be collected andanalysed for drugs. There are advantages and disad-vantages with each type of sample and expertadvice is therefore recommended in the design, useand interpretation of biological data of this sort(Wolff et al., 1999).
Time periods for measures
An important decision concerns the time period overwhich outcomes will be assessed. For example,although a follow-up period may be six months in
Section 3
22
duration, a decision needs to be reached about thetime period over which clients will be asked to recallthe key outcome measures. The same time periodshould be chosen for both baseline assessment andthe follow-up interview. For most indicators, the rec-ommended recall periods are 30 days and 90 days.
Assessing non-treatment factors thatmay affect outcome
When people with substance use disorders engagewith treatment they are almost always exposed toother influences in their personal and social lives.These influences can exert either a positive or nega-tive influence on the client’s ability and willingness tochange behaviour (see for example, Moos, Cronkiteand Finney, 1994). For example, a client may havecontact with friends who are problematic substanceusers or experience a sudden social problem (suchhomelessness) and this may increase the risk ofrelapse. These influences are important and evalua-tors should try to measure the nature and extent ofthese factors when assessing treatment outcome.
There are also several non-treatment factors thatmay operate when assessing the impact of a treat-ment programme on a cohort of clients. Firstly,some clients may not really fully engage with theprogramme but simply 'mature-out' of drug use.This is particularly likely to be the case with youngersubstance users. Secondly, some clients whoapproach treatment services may have periods ofrelative stability or instability that occur independ-ently of any treatment received. Thirdly, there is arather confusing statistical artefact called 'regressionto the mean' which can influence change scores forgroups on measures that are not completely corre-lated and differ from a theoretical “true” score (astatistician should be consulted to review this issue).Fourthly, the data collection and evaluation processitself may influence clients. They may, for example,be motivated to respond in a positive manner to thefollow-up interviewer. A follow-up interviewer whois in contact with clients on several occasions mayalso motivate them to change by asking questionsand showing an interest in their situations. Variousresearch designs can be used to control for theinfluence of non-treatment factors on client out-come. No one method is universally effective and no
single study is ever definitive. It is the convergenceof research evidence from different study designsundertaken by different investigators in differentcountries that enables conclusions to be drawnabout the role of treatment and non-treatment fac-tors in patient outcomes.
Subject recruitment and follow-up
For outcome studies, an unbiased sample of subjectsshould be recruited at entry to a treatment pro-gramme. Baseline evaluation information (pre-treat-ment assessment) should be collected as early as pos-sible in the intake/assessment process. At entry totreatment some clients may be unable to participatein a study due to physical and psychological healthcomplications. Recall periods should be timed fromentry into treatment in these situations (and the timefrom intake to baseline data collection noted).
An “intent-to-treat”follow-up strategy is also rec-ommended for all outcome evaluations. This meansthat all clients recruited are sought for follow-upand not only those who completed treatment. Thereis a general consensus that at least 70% of subjectsrecruited to an outcome study should be followed-up for the results to be reliable. The timing of fol-low-up points is also important and published stud-ies report on information obtained at many differentpoints in time. Some evaluations report on clientchanges at the time of discharge or after a particularperiod of contact. Other evaluations have followedsamples of clients for several years. Most commonare reports of outcomes assessed at three, six, or 12months after treatment has ended. The timing offollow-up and the intervals considered will have asignificant impact on the results and conclusionsdrawn from a study. Short-term follow-up studiesare likely to show better results than longer termones. This is because between 60% and 80% of'relapses' to substance use occurs in the first threeto four months following discharge. Results willdepend on the type of outcome being assessed. Forexample, changes in drinking and drug use may notbe stable, but long-term overall improvements infamily functioning may occur. Overall, the selectionof follow-up periods and intervals must be consis-tent with the objectives of the evaluation if it is toproduce useful results.
23
Sampling and sample size
A sampling strategy refers to the methods used toselect the representative “units” (such as treatmentagencies or treatment participants) to be included ina study. A sampling procedure that assures thateach element in the population has an equal chanceof being selected is referred to as “simple randomsampling”. Sometimes a systematic random sam-pling method can be used where a starting point ischosen (e.g. sampling every 20th school in a popu-lation or every 20th class member). An alternativeto simple random sampling is stratified random sam-pling. This method is normally used where: ■ It is not possible to generate a list of all the mem-
bers of a population; ■ where the members of a population are too geo-
graphically dispersed to be contacted; and ■ where the population contains important sub-
groups that are either of variable sizes and/orgeographically dispersed.
In contrast to simple random sampling and stratifiedsampling, (where single subjects are selected fromthe population), in “cluster sampling” the subjectsare selected in groups or clusters. Cluster sampling iscommonly used when the population is very geo-graphically dispersed and may already form naturalgroups.
When considering the size of a sample that isrequired, this is to some extent determined by thenature of the study, its objectives and the measuresto be used. In outcome studies that aim at compar-ing outcomes for two or more groups of clients, thesample size depends on the magnitude of the differ-ence on a specified outcome measure that isrequired to detect change between the groups. Astatistician can assist in calculating the required sam-ple size based on the following information:■ the degree of confidence one wants in the results
obtained; ■ the relative sizes of groups to be compared; ■ the expected frequency of the behaviour in one
group; and■ the magnitude of the difference that is to be
detected between the groups to be compared
When determining the number of people to be fol-lowed-up in an outcome evaluation, allowanceshould be made for clients who cannot be contactedand for whom outcome information will be missing.It is reasonable to expect that up to 30% of caseschosen for follow-up cannot be traced. The sampleselected for follow-up should therefore be increasedby 30%.
Sample size is also important in needs assessmentsurveys. In general, the required sample size in suchcases is smaller than commonly believed. However,it depends on the number of subgroups that may beof interest and the expected accuracy of results.Consultation with a statistician is recommended forthe design and analysis of studies involving surveysamples.
Statistical testing and confidenceintervals
Almost all outcome evaluations involve an elementof statistical testing. In a randomised controlled trial,for example, researchers will ask about the “statisti-cal significance” of differences in response measuresbetween the control and experimental group(s) atintake and follow-up and between these two points.The conventional way of tackling this question is touse statistical hypothesis testing. This usually startswith the assumption that there is no differencebetween groups (the so-called null hypothesis). Astatistical test is then used to see if any observeddifference can be considered genuine (at the popu-lation level) or due to chance. It is quite commonfor researchers to present the values of a statisticaltest and to indicate the probability that theobserved difference is due to chance. However, thisonly indirectly tells the reader how much confidenceto have in the results. It is therefore recommendedthat evaluation studies also report the magnitude ofdifferences over time and between groups inabsolute value where the measure is meaningful(e.g. a mortality rate) and in standard deviationunits when the outcome is measured in an essential-ly arbitrary way (e.g. a score on an attitude scale).The upper and lower limits of a confidence intervalaround this estimate should also be provided (the95% bound is commonly used).
In this section, we consider the reporting of theresults of an evaluation and how to apply theresults. Each of the types of evaluation discussed inthis document will lead to a report with someunique features. As an example, only reports for anoutcome evaluation will be considered here. Box 10shows one possible structure for an outcome evalu-ation report. This has been informed by the profes-sional consensus statement published in the Journalof the American Medical Association on key infor-mation to report from clinical trials (see Begg, et al.,1996). The box presents a more generic overview ofthe sorts of material to include in a formal report.An executive summary of the evaluation should beprepared and this is a valuable means of describingthe key findings and significance of a study to awide audience.
Increasing the impact of evaluation
It is unfortunately the case that evaluations of treat-ment for substance abuse do not always have animpact on decisions about what treatment pro-grammes will be supported and how treatment willbe delivered (Miller, 1987; Ogborne, 1988). Thereare many reasons for this including the vested inter-ests of some treatment providers, the poor qualityof some evaluations and, in some instances, the lackof clear-cut results. However, the impact of evalua-tion may also be limited by poor communicationbetween evaluators and decision-makers.
The importance of involving key decision makers inthe earliest discussions of evaluation will increasethe chances that evaluations will generate timelyand useful information. Good communication goeswell beyond simply providing funding bodies copiesof evaluation reports. These reports are oftenlengthy and sometimes quite technical and may failto hold readers’ attention. Also reports in them-selves may not be sufficient to convince decisionmakers that particular programmes should be sup-ported. This is especially so when evaluations sug-gest that new programmes would have to be fund-ed at the expense of existing programmes or thatexisting programs need to be radically altered. Inthese cases there is likely to be opposition to changefrom those with a vested interest in the status quo.Resistance to change can also occur when there is a
24
lack of skills and resources in existing delivery sys-tems or lack of understanding of the need forchange and a lack of appreciation of the benefits ofchange. In some cases prevailing belief and valuesystems can impede the adoption of new treatmentsand in others cases resistance to change reflectsfears for loss of jobs or status among serviceproviders.
Several strategies have been proposed for increasingthe impact of evaluation and there is also a large lit-erature on the diffusion of innovation that has someimportant lessons for those seeking to promoteevaluation and the adoption of treatments for sub-stance abuse (Davis, 1978; Eveland, 1986; Backer,1991). It is recommended that, where possible, pri-ority should be given to ways that clearly demon-strate the advantages of evidence-based treatments.This might include the implementation of specialdemonstration projects to be visited by those inter-ested in using the same methods. The demonstra-tion programmes can also provide on-site trainingopportunities. As well it would be useful to developclear guidelines or “how to” manuals for using thenew methods.
Change agents – people who aim to bridge the gapbetween knowledge and practice by actively pro-moting evidence – based practices – have proved tobe helpful in many situations. These should be peo-ple who have, or are able to earn the respect ofthose they seek to influence and who have a goodunderstanding of new methods. They need stronginterpersonal skills and the ability to find creativeways to promote change in different contexts.
Reporting ResultsSection 4
25
HEADING SUBHEADING DESCRIPTION
Title Description of study and the research team and source of any funding support
Abstract Structured format (e.g. aims and objectives; study design; treatment(s) studied and their settings; participants; measurements; methods and procedures; findings; conclusions)
Introduction Background to the study and summary of relevant previous research
State prospectively defined hypothesis, clinical objectives, and planned subgroup or covariate analysis
Methods Protocol Subject population studied
Participant inclusion/exclusion criteria
Interventions studied (modality; intensity and duration)
Primary and secondary outcome measure(s)
Describe how sample size was calculated (power calculation)
Plan for statistical analysis (main and subsidiary analyses)
Procedure Subject recruitment methods
Unit of randomisation (e.g. individual, cluster, geographic) – if used
Randomisation methods
Number and timing of follow-up points
Subject tracking procedures and interview methods (e.g. face-to-face; telephone; self-completion)
Results Descriptive Study profile (figure) summarising number of participants through the study
Numbers of subjects in treatment over time and numbers interviewed and lost to the study at follow-up
Baseline subject characteristics
Inferential Effect of programme on primary and secondary outcome measures
Discussion/ Summary of main findingscommentary Extent of bias and/or design weaknesses and study limitations
Implications of the findings
Recommendations
Structure and content of a formal report from an outcome evaluation Box 10
Summary and Conclusions
26
There is still much to be learned about the mosteffective, efficient and humane ways to help peoplewith substance use disorders and to reduce theassociated harms to users themselves and to others.There are still many treatments that have not beenrigorously evaluated and some treatments, foundeffective in some context or cultures, have not alsobeen tested in others. The emergence of new drugs,new types of drug users and new patterns of usemay also require the development of new treat-ments that need to be evaluated. Evaluation shouldtherefore be a feature of treatment systems andtreatment policy. Evaluation will help to (a) identifytreatment needs (b) plan needs-based interventions,(c) show if these interventions are consistent withneeds and plans and (d) show if theses interventionsare effective and efficient.
These guidelines summarise the main features ofevaluation and we hope that they will motivate thereader to support and champion the evaluation ofsubstance abuse treatment within their countriesand communities. It is also hoped that readers willbe motivated to learn more about evaluation and beinterested in consulting the WHO/UNDCP/EMCD-DA workbooks.
The guidelines have not attempted to conceal thefact that some types of evaluation are quite com-plex and require special skills. This would be a dis-service to the cause of evaluation and to the manypeople who have worked hard to develop the eval-uation methodologies that are helping to improvetreatment in some contexts. We have, however, alsoattempted to show that evaluation does not have tobe daunting. All readers are strongly encouraged toconsider supporting or undertaking basic evaluationseven if more complex alternatives are not at presentfeasible. These initial efforts can later be extendedas more skills are acquired and more resourcesbecome available. Evaluations and related activitiesthat do not require highly special skills or extensiveresources are:
■ Need assessments that use existing quantitativedata and/or key informant interviews;
■ Use of logic models to plan new services and toclarify the assumptions and objectives of existingprogrammes or systems;
■ Development and implementation of basic, rou-tine or periodic monitoring systems to show (1)what services are being provided, to what clientsand at what cost (2) amount of treatmentreceived by different clients (3) modes of dis-charge (completed program, drop out, dischargedfor cause etc); and
■ Periodic use of simple self-completion question-naires or confidential interviews to assess clientssatisfaction
Examples of evaluations of these types are given inthe WHO/UNDCP/EMCDDA workbooks and in anevaluation casebook published by the AddictionResearch Foundation of Ontario (Graham et al.,1994).
Adams E.H, Kopstein A.N The non-medical use of*prescription* drugs in the United States. In:
Cooper J.R., Czechowicz D.J, Molinari S.P., PetersenR.C. (Eds.) Impact of Prescription Drug DiversionControl Systems on Medical Practice and PatientCare. NIDA Research Monograph 131. RockvilleMD: National Institute on Drug Abuse, 1993.
Babor, T.F., Brown, J., & Del Boca, F.K. Validity ofself-reports in applied research on addictive behav-iours: fact or fiction? Behavioral Assessment,1990,12: 5-31.
Backer, T.E. Drug abuse technology transfer. NIDAmonograph serious # 1555; Rockville, MD; NationalInstitute on Drug Abuse, 1991.
Begg, C., Mildred, C., Eastwood, S., Horton, R.,Moher, D., Olkin, I., Pitkin, R., Rennie, D., Schulz,K.F., Simel, D. & Stroup, D.F. Improving the qualityof reporting of randomised controlled trials. Journalof the American Medical Association, 1996,276:637-639.
Bradshaw, J.S. A taxonomy of social need. In:McLachlan, G. (Ed.) Problems and Progress inMedical Care: essays on current research. 7th series.Oxford: Oxford University Press, 1972.
Centre for Substance Abuse Treatment The Costsand Benefits of Substance Abuse Treatment.
Available online at [http://neds.calib.com/prod-ucts/pdfs/costben.pdf], 1999.
Chen, H.-T. Theory-driven evaluations. NewburyPark: Sage, 1990.
Davis, H.A. Management of change and innovationin mental health services. Hospital and CommunityPsychiatry, 1978, 29: 649-658.
Dennis, M.L. Ethical and practical randomised fieldexperiments. In: Wholey, J.S., Hatry, H.P.&Newcomer, K.E. (Eds.) Handbook of PracticalProgram Evaluation. San Francisco: Jossey Bass,1994.
DeWit, D.J., & Rush, B.R. Assessing the need forsubstance abuse services: A critical review of needsassessment models. Evaluation and ProgramPlanning, 1996, 19: 41-64.
27
Drummond, M.F., Stoddart, G.L. & Torrance, G.W.Methods for the economic evaluation of health careprogrammes. Oxford: Oxford University Press,1987.
Eveland, J.D. Diffusion, technology transfer andinnovation. Thinking and talking about change.Knowledge: Creation, Diffusion and Utilization,1986, 8: 303-322.
Graham, K, Woo, G.A. Smythe, C. and others TheEvaluation Casebook: Using Evaluation to EnhanceProgram Quality in Addictions. Toronto: AddictionResearch Foundation. (Now part of the Centre forAddiction and Mental Health), 1994.
Holder, H., Longabaugh, Miller, Rubonis. The costeffectiveness of treatment for alcoholism firstapproximation. Journal of Studies on Alcohol, 1991,52: 517-525.
Holder, H.D. & Blose, J.O. The reduction of healthcare costs associated with alcoholism treatment: A14 year longitudinal study. Journal of Studies onAlcohol, 1992, 53: 293-302.
Landry, M. Overview of Addiction TreatmentEffectiveness, Office of Applied Studies.116.Rockville, MD: SAMHSA, 1995.
Lebour, D.L. Similarities and differences betweenmental health and health care evaluations studiesassessing consumer satisfaction. Evaluation andProgramme Planning, 1983, 2:197-245.
Marsden, J., Strang, J. with Lavoie, D., Abdulrahim,D., Hickman, M & Scott, S. Epidemiologically-basedneeds assessment: Drug misuse. In: Stevens, A. &Raftery, J. (Eds.) Health Care Needs Assessment.Oxford: Radcliffe, forthcoming.
McLellan, A.T., Hagan, T.A., Levine, M., Gould, F.,Meyers, K., Bencivengo, M., and Durell, J.Supplemental social services improve outcomes inpublic addiction treatment. Addiction, 1998,93:1489-1499.
Miller, W.E. and Hester, R.K. The effectiveness oftreatment techniques: what works and what does-n't, in Treating addictive behaviours: processes ofchange, W.E. Miller and N. Heather, Editors, PlenumPress: New York, 1986.
References
28
Miller, W.R., Behavioral alcohol treatment researchadvances: barriers to utilization. Advances inBehavior Research and Therapy, 1987, 9: 145-164.
Moos, R.H., Finney, J.W. & Cronkite, R.C.Alcoholism treatment. Context, Process andOutcome. New Yor: Oxford University, 1990.
National Institute on Drug Abuse, Principles of DrugAddiction Treatment: A research-based guide,National Institutes of Health: Washington D.C,1999.
Ogborne, A.C. Bridging the gap between the twocultures of alcoholism research and treatment.British Journal of Addiction, 1988, 83: 729-733.
Roberts, G. and Ogborne,A.C., Best Practices:Substance Abuse Treatment and Rehabilitation,Office of Alcohol, Drugs and Dependency Issues,Health Canada: Ottawa, 1999.
Rossi, P.H., & Freeman, H.E. Evaluation: A systemat-ic approach (2nd Ed.). Beverly Hills, California: SagePublications, 1982.
Rush, B.R., & Ogborne, A.C. Program logic models:Expanding their role and structure for planning andevaluation. Canadian Journal of ProgramEvaluation, 1991,6(2): 95-106.
Rydell, C.P. and S.S. Everingham, ControllingCocaine: Supply versus Demand Programs, Rand:Santa Monica, CA, 1994.
Single, E., Robson, L., Xie, X., Rehm, J. The costs ofsubstance abuse in Canada. Ottawa. The CanadianCentre on Substance Abuse, 1996.
Stevens, A. & Raftery, J. Health Care NeedsAssessment. The Epdemiologically based needsassessment reviews. Oxford: Radcliffe MedicalPress, 1994.
Tims, F. Drug Abuse Treatment Effectiveness andCost-Effectiveness, Rockville, Maryland: NationalInstitute on Drug Abuse, 1995.
United Nations International Drug ControlProgramme World Drug Report. Oxford UniversityPress, New York, 1997.
Wholey, J.S. Assessing the feasibility and likely use-fulness of evaluation. In: J.S. Wholey, J.S., Hatry,
H.P. & Newcomer, K.E.(Eds.) Handbook of PracticalProgram Evaluation. San Francisco: Jossey Bass,1994.
Wolff, K., Farrell, M., Marsden, J., Monteiro, M.G.,Ali, R, Welch, S. & Strang, J. A Review of BiologicalIndicators of illicit drug use: practical considerationsand clinical usefulness. Addiction, 1999, 94: 1279-1298.
World Health Organization, United Nations DrugControl Program and European Monitoring Centreon Drugs and Drug Addiction Evaluation ofPsychoactive Substance Use Disorder Treatment.Facilitators Manual and Workbooks 1-8. WHOGeneva, 2000.
WHO Expert Committee on Drug Dependence,30th Report, World Health Organization: Geneva,1996.
