Vol. 42 No. 4 October 2011 Journal of Pain and Symptom Management 493

Special Section: The PRISMA Symposium

The PRISMA Symposium 1: Outcome ToolUse. Disharmony in European OutcomesResearch for Palliative and AdvancedDisease Care: Too Many Tools in PracticeRichard Harding, BSc, MSc, DipSW, PhD, Steffen T. Simon, MSc, MD,Hamid Benalia, BSc, MA, Julia Downing, RN, PhD,Barbara A. Daveson, BMus (Mus Thy), Dip HSM, PhD,Irene J. Higginson, BMedSci, BM BS, FFPHM, FRCP, PhD,and Claudia Bausewein, MSc, MD, PhD, on behalf of PRISMADepartment of Palliative Care, Policy and Rehabilitation and The Cicely Saunders Institute of

Palliative Care (R.H., H.B., B.A.D., I.J.H., C.B.), King’s College London, London, United Kingdom;

Department of Palliative Medicine and Clinical Trials Center (S.T.S.), University of Cologne, Cologne,

Germany; and Makerere University (J.D.), Kampala, Uganda

Abstract

Context. As the European population ages and the number of cancer deaths

annually increases, there is an urgent requirement to provide high-quality,effective care. The measurement of outcomes in advanced disease is complex, andto conduct comparative research and meta-analyses, appropriate tool selection isessential.

Objectives. This study aimed to identify the outcome tools currently in use inend-of-life care (both clinically and for research) across Europe and investigatethe preferred features of outcome tools from the perspective of those who selectand apply them.

Methods. A pan-European Internet-based survey of tool users was conductedin research and clinical populations. Respondents were asked to identify thetools they are using and describe ideal features of the measures. The study wasconducted in accordance with guidance for best practice in web-basedresearch.

Results. Of the 311 participants who completed a survey, 99 tools in clinical careand audit, and 94 in research, were cited by less than 10 participants. Further datarevealed that respondents require the number of potential tools to be rationalizedand that brief tools are favored.

Conclusion. The selection of valid and appropriate tools for palliative carepopulations requires expert guidance and support to ensure that clinicians andresearchers are collecting data that have validity and potential for comparison

Address correspondence to: Richard Harding, BSc, MSc,DipSW, PhD,King’s CollegeLondon,Cicely SaundersInstitute, Department of Palliative Care, Policy and

Rehabilitation, Bessemer Road, London SE5 9PJ,United Kingdom. E-mail: richard.harding@kcl.ac.uk

Accepted for publication: June 23, 2011.

� 2011 U.S. Cancer Pain Relief CommitteePublished by Elsevier Inc. All rights reserved.

0885-3924/$ - see front matterdoi:10.1016/j.jpainsymman.2011.06.008

494 Vol. 42 No. 4 October 2011Harding et al.

within and between populations and countries. J Pain Symptom Manage2011;42:493e500. � 2011 U.S. Cancer Pain Relief Committee. Published by ElsevierInc. All rights reserved.

Key Words

Outcome assessment, palliative care, terminal care, end-of-life care, research, Europe

IntroductionWith an annual 1.7 million deaths from can-

cer in Europe and an aging population,1e4

there is an urgent need to improve care at theend of life for patients and families. However,end-of-life research is currently underresourcedand underdeveloped across Europe. TheWorldHealth Organization Europe’s guidance ‘‘Palli-ative Care: The Solid Facts’’ shows that, inmany countries, less than 0.5% of researchspending in cancer is allocated to end-of-lifeand palliative care5 and recommendscollaboration in end-of-life research to enhancethe evidence base.

There are barriers to the conduct of high-quality research in end-of-life cancer care.6

One central issue in the conduct of this re-search is themeasurement of effects on andout-comes for patients. This is important for qualityimprovement, needs assessment, and evalua-tion of specific treatments or interventions,whether in practice or through specific trials.The lack of standardized measurement meansthat there are differences in the interpretationof study results; meta-analyses are often limitedbecause no core measure is used, and, impor-tantly, some studies fail because they haveused inappropriate measures without adequatesensitivity.7e9 There may be differences amongpatients, carers, and health professionals intheir views of the priorities at the end of life10

and even between professionals.11

Populations toward the end of life are hetero-geneous, and clinical interests include physio-logical, psychological, and social aspects of thephenomenon in question. The focus is notonly the patient, but also the family. This het-erogeneity represents a major challenge for re-search and especially when the populationapproaches the final year of the disease trajec-tory.12 There also are challenges in undertakingmeasurement among patients who are in thelast year of life and very ill, whomay be confused

or cognitively impaired, and in capturing emo-tional, social, spiritual, and physical compo-nents. Some measures are, however, in morecommon use in these populations. The com-mon uses could be enhanced and expanded,providing better guidance on the selectionand use of measures through Europeancollaboration.The quantity of end-of-life research seems to

be steadily increasing. However, the quality canbe disputed.13 A systematic literature review ofexisting potential scales identified a numberof measures with varying scientific validity,14

and few tools appear to meet the required stan-dards of outcome measure development andvalidity.15,16 The selection of appropriate toolsalso should reflect the World Health Organiza-tion’s acknowledgment that health care shouldbe responsive to public preferences and expec-tations as one of the three intrinsic goals of anyhealth system, alongside health and fairness.17

Preferences and priorities have always beencentral in end-of-life care, under the auspicesof a person-centered and individualized ap-proach to care.The European Association for Palliative Care

Research Network has conducted a preliminarysurvey of end-of-life and palliative care centersin Europe and has identified 143 active centersfrom 22 different countries.18 This suggests thatthere is a major potential for collaborative, co-ordinated research in Europe. However, thereis a lack of coordination of measurement inEurope. PRISMA is a three-year project fundedby the European Union with a special focus onoutcome measurement in palliative care.19,20

One of the goals of PRISMA is to map and har-monize European approaches and experiencesin end-of-life care measurement. To achievethis, one work package focused on best practicein outcome measurement. In an online survey,professionals working in palliative care inEurope and Africa were asked about their

Vol. 42 No. 4 October 2011 495PRISMA 1: Outcome Tool Use

experiences and use of outcome measures inpalliative care. This survey showed that profes-sionals across Europe, both in clinical careand in research, are, for the most part, usingand positive about using outcome measures,but the survey also revealed that they are in ur-gent need of help (e.g., training and resourcesin the selection, implementation, and analysisof outcome tools).21

In this paper, we present data from thisPRISMA study, which also aimed to identifythe outcome tools currently in use in end-of-life care (both clinically and for research) acrossEurope, and investigate the preferred featuresof outcomes tools from the perspective of thosewho select and apply them.

MethodsThe study used a web-based, cross-sectional

survey design. Details of the methods are de-scribed elsewhere.21

Survey Development and AdministrationThe specifically developed questionnaire was

designed within the multiprofessional PRISMAwork package, and the electronic version waspiloted in seven European countries (Austria,Germany, Italy, The Netherlands, Norway,Portugal, and the United Kingdom), with 20professionals working inpalliative care. The sur-vey was conducted in English because of re-stricted resources; however, participants wereinvited to respond to open questions either inEnglish or in their native language. Special at-tention was given to a user-friendly design, on-line support features during completion of thesurvey, such as alerts for unanswered questionsand ‘‘back buttons’’ for reviewing answers, anddata protection to ensure confidentiality andanonymity. Adaptive questioning was used,where certain questions were only displayedbased on the responses to other items to reducethenumber and complexity of thequestions. Toimprove the completion rate, only one questionwas displayed per screen page. Overall, thequestionnaire contained a maximum 59 pages(screens) for people who used outcome mea-sures in both clinical care and research.

Survey TopicsThefinalquestionnaireaddressed the following

areas of enquiry: first, respondent characteristics

(age, gender, and whether active in clinicaluse of tools, research use of tools, or both),clinical field and research field, and homecountry. Second, the outcome measures cur-rently or ever used in the clinical care or re-search of palliative/end-of-life populationswere identified (using both a list of commontools and a space for respondents to addother tools). Third, respondents were askedto identify their preferred features of an out-come measure (number of items, ranking ofthe possible domains, and whether patient/staff/carer versions are preferred). Lastly,respondents were asked their views of whatthey see as the required properties whenselecting an outcome measure.

SamplingThe inclusion criteria for the survey sample

were professionals working in palliative care(e.g., physicians, nurses, other allied profes-sionals, and academics) in clinical care, audit,or research in Europe.

The following institutions and databases wereused to sample potential participants for the sur-vey: 1) national palliative care associations werecontacted in eight countries of the PRISMA col-laboration (Belgium, Germany, Italy, The Neth-erlands, Norway, Portugal, Spain, and theUnited Kingdom). To adjust for the differentsizes of countries and associations, a weightedrandom sample of 20% of each group wasdrawn if the total number of clinicians (physi-cians, nurses, and other) in the association wasmore than 500 members and a sample of 33%if the total number was less than 500 members;2) an existing database of outcome tool userswas used (211 users); 3) authors of studies usingcommon outcome tools were invited; 4) a list of61 European chairpersons and researchers inpalliative care and known to PRISMA wasestablished. All contacts on this list were invitedto participate in the survey; 5) all members ofthe U.K. Palliative Nursing Group of the RoyalCollege of Nursing and the 28 known nationalnurse consultants were invited; and 6) all PRIS-MA members were invited (38 individuals).

Data CollectionThe European online survey was conducted

in October and November 2009. Invitatione-mails with information about the contentand purpose of the survey, and the estimated

Table 1Tools Used in Clinical Care/Audita

Tools Used and ReferenceClinical Care/

Audit (n)

Karnofsky Performance Status (KPS)Scale24

187

Edmonton Symptom AssessmentScale (ESAS)25

120

PPS26 70Palliative care Outcome Scale(POS)27,28

68

SDS29 40Support Team Assessment Schedule(STAS)30

40

Palliative Care Assessment Tool31 29European Organization for Research and 21

496 Vol. 42 No. 4 October 2011Harding et al.

time for completion, were sent out to potentialparticipants either by the palliative care associ-ations or directly to named individuals. The in-vitation e-mail included a link to the web-basedsurvey. In total, 2000 e-mails were sent, fol-lowed by two reminder e-mails.

AnalysisFollowing The Checklist for Reporting Re-

sults of Internet E-Surveys guidelines,22 the par-ticipation rate was calculated by dividing thenumber of replies to the first question by thenumber of unique site visitors (defined as thosevisiting the first page of the online survey), andthe completion rate was calculated by the num-ber of participants answering the first questiondivided by the number of people submittingthe last question. Descriptive analyses of allquestions were conducted using frequenciesfor categorical data andmeans and standard de-viations (SD) for continuous variables. Answersfrom open-ended questions were analyzed us-ing content analysis. First, the answer optionswere collated and then coded, searching forcommon themes.23 SPSS version 17.0 (SPSS,Inc., Chicago, IL)was used for quantitative anal-ysis and NVivo (QSR International, Southport,UK) to assist in qualitative data analysis.

EthicsEthic approval was obtained from the Re-

search Ethics Committee at King’s CollegeLondon (BDM/08/09-102). Ethics approvalfrom other countries was not needed becausethis survey did not include patients but onlyprofessionals.

Treatment of Cancer Quality of LifeQuestionnaire-C30 (EORTC QLQ-C30)32

Measuring the Quality of Life of SeriouslyIll Patients Questionnaire (QUAL-E)33

19

Barthel Index34 17McGill Quality of Life Questionnaire(MQOL)35

16

Unspecified Visual Analogue Scale 16European Organization for Research andTreatment of Cancer Quality of LifeQuestionnaire-C15-PAL (EORTCQLQ-C15-PAL)36

15

Hospital Anxiety and Depression Scale(HADS)37

15

Mini-Mental State Examination38 13Therapy Impact Questionnaire39 11Memorial Symptom Assessment Scale40 10Cited less than 10 times 99

Total number of tools 116

aMultiple answers possible.

ResultsSample Response and Characteristics

The total number of participants who beganthe questionnaire was 495 of 1291, and 311 of495 completed the survey. The participationrate was 38%, and the completion rate was63%. Respondents’ mean age was 45.8 years(SD 8.93), and 187 (63.6%) were female. Themajority were clinicians of varying professions(n¼ 213, 71.5%), with 27 identifying as re-searchers (9.1%) and around one-fifth activelyboth clinicians and researchers (n¼ 58,19.5%). Respondents had the following yearsof experience in palliative care: 0e5 years,

n¼ 106 (35.6%); 6e10 years, n¼ 82 (27.5%),and more than 10 years, n¼ 109 (36.6%).Clinician responders were primarily medical

(n¼ 164, 60.3%), followed by nursing (n¼ 69,25.4%), with the remaining professions return-ing 39 responses (14.3%). Researcher respon-dents also were primarily medical (n¼ 47,55.3%), followed by nursing (n¼ 12, 14.1%),psychology (n¼ 8, 9.4%), social science(n¼ 6, 7.1%), and others (n¼ 12, 14.1%).The greatest number of responses came fromthe United Kingdom (95), followed by Italy(37), Germany (29), Portugal (20), Spain(19), Norway (12), Belgium (5), Ireland (3),Greece (2), The Netherlands (2), and oneeach from Austria, Denmark, France, Hungary,and Poland. Five were ‘‘Europe’’ unspecifiedand 14 ‘‘other.’’

Tools UsedThe tools used by respondents are de-

scribed in Tables 1 and 2.24e42 The five most

Table 2Tools Used in Researcha

Tools Used and Reference Research (n)

KPS24 69ESAS25 53European Organization for Research and

Treatment of Cancer Quality of LifeQuestionnaire-C30 (EORTC QLQ-C30)32

48

Palliative care Outcome Scale (POSor African POS)27,28

36

PPS26 30European Organization for Research and

Treatment of Cancer Quality of LifeQuestionnaire-C15-PAL (EORTCQLQ-C15-PAL)32

18

STAS30 18MQOL35 15Functional Assessment of Cancer

Therapy-General4114

HADS37 14SDS29 11Schedule for the Evaluation of Individual

Quality of Life4210

Cited less than 10 times 94

Total number of tools 106

aMultiple answers possible.

Vol. 42 No. 4 October 2011 497PRISMA 1: Outcome Tool Use

commonly used tools by clinicians in clinicalcare and audit were (in descending fre-quency) the Karnofsky Performance StatusScale, the Edmonton Symptom AssessmentScale, the Palliative Performance Scale, thePalliative Outcome Scale, and the SymptomDistress Scale. The most commonly cited toolsby researchers were the Karnofsky Perfor-mance Status Scale, the Edmonton SymptomAssessment Scale, the European Organizationfor Research and Treatment of Cancer Qualityof Life Questionnaire-C30, the Palliative Out-come Scale, and the Palliative PerformanceScale. It is noteworthy that for clinical careand audit, 99 tools were cited less than 10times, and for research, 94 were cited lessthan 10 times.

Respondent Views on Preferred Tool PropertiesThe quantitative and qualitative data identi-

fied the ideal properties of an outcome tool inpalliative and end-of-life care. With respect tothe number of items, the preferred rangewas be-tween six and 10 items. Responses were as fol-lows: between one and five, n¼ 43 (14.0%);between six and 10, n¼ 154 (50.0%); between11 and 15, n¼ 79 (25.6%); between 16 and 20,n¼ 21 (6.8%); andmore than20,n¼ 11 (4.6%).

A number of common themes emergedfrom the open-ended data. First respondents

commented that too many tools are in exis-tence: ‘‘There are too many, they need to berationalised so that everywhere is using thesame one(s).’’ (Clinician, U.K.). ‘‘Would it bepossible to REDUCE the number of tools?!’’(Researcher, Italy).

A priority for collaboration was identified,using appropriate tools in common:

Yes - we need multicentre collaborationswithin countries to test the reliability, valid-ity, utility, etc. of existing tools before westart inventing more! (Clinician and re-searcher, U.K.).

Cooperation of all associations worldwide.Maybe in this way, asking people which dothe work and use actual tools. (Clinician,Germany).

However, this collaboration and shared useof tools must include best science in selectingand adapting tools to local settings:

A reference group (expert inPC tools) to con-tact when tools are to be chosen for researchor clinical aims would really be useful. It canprevent researchers to reinvent the wheel byreviewing the literature at any time. (Clini-cian and researcher, unknown country).

Local country organisations should evaluatethe available tools and discuss in workshopshow they fit with the daily practice, enablingtheir continuity. (Clinician, Portugal).

Further data described the preferred contentand structure of an outcome tool. Not onlyshould the tool include those matters of great-est importance to both carers and families butto those responsible for service design:

Include domains generated by patients andcarers concerns. Ensure domains are con-gruent with the information [of] strate-gists/purchasers of palliative care servicesto save effort. (Clinician, U.K.).

To ensure that tools meet the multiple out-comes of interest in palliative and end-of-lifecare, the provision of modules that can be in-cluded as necessary was advocated:

Yes. It would be very useful to have a toolkitwhich worked together, and had been testedand validated as such. So for instance, symp-tom tool/QoL tool/functional status tool

498 Vol. 42 No. 4 October 2011Harding et al.

which followed a similar format, and couldbe used together or separately, and had con-sistency of time frame and format. (Clinicianand researcher, U.K.).

Lastly, with respect to tool format, respon-dents called for simplicity: ‘‘They should beshort and easy to understand for patients.’’ (Cli-nician, Germany). ‘‘They should be short, notcomplicated. Easy to take.’’ (Clinician, Spain).

DiscussionThis is the first pan-European survey describ-

ing the views of professionals on the use andpreferred features of outcome measures in pal-liative care. The data demonstrate a very clearchallenge for collaboration and the advance-ment of clinical practice and evidence genera-tion. Among our sample, over 100 tools werecited that had been used, and nearly one hun-dred were only cited less than 10 times. A chal-lenge to palliative and end-of-life care is thatwehavemultiple endpoints across very differentdomains and that patient populations (anddiagnoses) are heterogeneous. Therefore, itwould be inappropriate to propose that all clin-ical services and studies use a specified set oftools without the possibility of introducingadditional tools as needed. However, it is clearthat there is an urgent need to rationalize coretools to those that have been developed and val-idated within our population and have soundpsychometric properties. The data from thisstudy propose the use of core and additionalbatteries of items, and this may address theproblems of specific services and studies need-ing to further investigate and measure specificoutcomes of interest.

The need to provide effective and appropri-ate care for patients facing life-limiting andlife-threatening disease is growing and if this isto be achieved, then larger, robust studies areneeded to overcome the historic methodologi-cal failures of experimental studies in palliativecare.43 This requires the use of appropriatetools, collaborative studies, and meta-analysesof existing data sets. None of these require-ments can be achieved if common, robust toolsare not selected and applied.

There are a number of limitations to ourstudy. Although we followed best practice

guidance for web-based surveys, a number ofmethodological limitations have been noted,including possible participation bias (in ourcase, those with an interest in outcome tools)and the potential for individuals to completemultiple submissions.44 The requirement toparticipate in the English language also mayhave reduced participation. However, our par-ticipation and completions rates are acceptablyhigh.If outcomes for patients and families facing

life-limiting and life-threatening diseases areto be measured and improved in Europe (andbeyond), then those selecting tools must be en-abled tomake informed choices about themostappropriate tool for their aims and the field. Itcannot be expected that individuals can easilyaccess the necessary volume of information tobe able to identify, locate, and appraise poten-tial tools. Conversely, those developing toolsshould ensure that the tools being recommen-ded meet the requirements for brevity, validitythat enables collaboration, and the provisionof modules that meet individual service/studyneeds. Lastly, the data presented here suggestthat new tools are not required but that judi-cious selection and refinement of existing toolswith appropriate scientific properties should beadvocated.

Disclosures and AcknowledgmentsPRISMA is funded by the European Commis-

sion’s Seventh Framework Programme (con-tract number: Health-F2-2008-201655), withthe overall aim to coordinate high-quality inter-national research into end-of-life cancer care.PRISMA aims to provide evidence and guidanceon best practice to ensure that research canmeasure and improve outcomes for patientsand families. PRISMA activities aim to reflectthe preferences and cultural diversities of citi-zens, the clinical priorities of clinicians, andappropriately measure multidimensional out-comes across settings where end-of-life careis delivered. Principal Investigator: RichardHarding; Scientific Director: Irene J Higginson;Work Package 4 Lead: Claudia Bausewein;PRISMA Members: Gwenda Albers, BarbaraAntunes, Ana Barros Pinto, Claudia Bausewein,Dorothee Bechinger-English, Hamid Benalia,Lucy Bradley, Lucas Ceulemans, Barbara A.

Vol. 42 No. 4 October 2011 499PRISMA 1: Outcome Tool Use

Daveson, LucDeliens,No€elDerycke,MartinedeVlieger, Let Dillen, Julia Downing, Michael Ech-teld, Natalie Evans, Dagny Faksv�ag Haugen,Lindsay Flood, Nancy Gikaara, Barbara Gomes,Marjolein Gysels, Sue Hall, Richard Harding,Irene J. Higginson, Stein Kaasa, JonathanKoffman, Pedro Lopes Ferreira, Johan Menten,Natalia Monteiro Calanzani, Fliss Murtagh,Bregje Onwuteaka-Philipsen, H. Roeline W.Pasman, Francesca Pettenati, Robert Pool,Tony Powell, Miel Ribbe, Katrin Sigurdardottir,Steffen Simon, Franco Toscani, Bart van denEynden, Jenny van der Steen, Paul VandenBerghe, and Trudie van Iersel.

The authors thank all participants for theirtime and engagement in the workshop. Thecontribution of all WP4 PRISMA membersalso is kindly acknowledged.

References1. Boyle P, Ferlay J. Cancer incidence and mortal-ity in Europe, 2004. Ann Oncol 2005;16:481e488.

2. International Agency for Research on Cancer.New European cancer figures - World CancerAgency says major efforts needed toward preventionin Europe. Lyon, France: World Health Organiza-tion International Agency for Research on Cancer,2007. Available from http://www.iarc.fr/ENG/Press_Releases/pr174a.html. Accessed May 2, 2011.

3. Gomes B, Higginson IJ. Where people die(1974e2030): past trends, future projections andimplications for care. Palliat Med 2008;22:33e41.

4. Anderson GF, Hussey PS. Health and populationageing: a multinational comparison. 1999. Availablefrom http://www.commonwealthfund.org/publications/publications_show.htm?doc_id¼221235. Accessed May 2,2011.

5. World Health Organization. Palliative Care: TheSolid Facts. Copenhagen, Denmark: World HealthOrganization, 2004.

6. Higginson IJ. It would be NICE to have moreevidence? Palliat Med 2004;18:85e86.

7. Harding R, Higginson IJ. What is the best way tohelp caregivers in cancer and palliative care? A sys-tematic literature review of interventions and theireffectiveness. Palliat Med 2003;17:63e74.

8. Higginson IJ, Finlay IG, Goodwin DM, et al. Isthere evidence that palliative care teams alter end-of-life experiences of patients and their caregivers?J Pain Symptom Manage 2003;25:150e168.

9. Solano JP, Gomes B, Higginson IJ. A comparisonof symptom prevalence in far advanced cancer, AIDS,heart disease, chronic obstructive pulmonary disease

and renal disease. J Pain Symptom Manage 2006;31:58e69.

10. McIlfatrick S. Assessing palliative care needs:views of patients, informal carers and healthcareprofessionals. J Adv Nurs 2007;57:77e86.

11. Schneider N, Buser K, Amelung VE. Discrep-ancies in the viewpoints of different German healthcare providers on palliative care. Eval Health Prof2007;30:96e109.

12. Christakis NA. Advances in palliative care re-search methodology. Palliat Med 2006;20:725e726.

13. Kaasa S, Hjermstad MJ, Loge JH. Methodologi-cal and structural challenges in palliative care re-search: how have we fared in the last decades?Palliat Med 2006;20:727e734.

14. Hearn J, Higginson IJ. Outcome measures inpalliative care for advanced cancer patients: a review.J Public Health Med 1997;19:193e199.

15. Terwee CB, Bot SDM, de Boer MR, et al. Qualitycriteria were proposed for measurement propertiesof health status questionnaires. J Clin Epidemiol2007;60:34e42.

16. Mokkink LB, Terwee CB, Patrick DL, et al. TheCOSMIN checklist for assessing the methodologicalquality of studies on measurement properties ofhealth status measurement instruments: an interna-tional Delphi study. Qual Life Res 2010;19:539e549.

17. World Health Organization. World HealthReport 2000: Health Systems: Improving perfor-mance. Geneva, Switzerland: World Health Organi-zation, 2000.

18. Kaasa S, Torvik K, ChernyN,HanksG, deConno F.Patient demographics and centre description inEuropean palliative care units. Palliat Med 2007;21:15e22.

19. PRISMA KCL. PRISMAWebsite. 2011. Availablefrom www.prismafp7.eu. Accessed May 1, 2011.

20. Harding R, Higginson IJ. PRISMA: a pan-European co-ordinating action to advance the sci-ence in end-of-life cancer care. Eur J Cancer 2010;46:1493e1501.

21. Bausewein C, Simon ST, Benalia H, et al. Imple-menting Patient Reported Outcome Measures(PROMs) in palliative care - users’ cry for help.Health Qual Life Outcomes 2011;9:27.

22. Eysenbach G. Improving the quality of web sur-veys: the Checklist for Reporting Results of InternetE-Surveys (CHERRIES). J Med Internet Res 2004;6:e34.

23. Garcia J, Evans J, Reshaw M. ‘‘Is there anythingelse you would like to tell us’’ - methodological is-sues in the use of free-text comments from postalsurveys. Qual Quant 2004;38:113e125.

24. Karnofsky DA, Abelmann WH, Craver LF,Burchenal JH. The use of the nitrogen mustardsin the palliative treatment of carcinoma. With

500 Vol. 42 No. 4 October 2011Harding et al.

particular reference to bronchogenic carcinoma.Cancer 1948;1:634e656.

25. Bruera E, Kuehn N, Miller MJ, Selmser P,Macmillan K. The Edmonton Symptom AssessmentSystem (ESAS): a simple method for the assessmentof palliative care patients. J Palliat Care 1991;7:6e9.

26. Anderson F, Downing GM, Hill J, Casorso L,Lerch N. Palliative Performance Scale (PPS):a new tool. J Palliat Care 1996;12:5e11.

27. Hearn J, Higginson IJ. Development and valida-tion of a core outcome measure for palliative care:the palliative care outcome scale. Qual Saf HealthCare 1999;8:219e227.

28. Harding R, Selman L, Agupio G, et al. Valida-tion of a core outcome measure for palliative carein Africa: the APCA African Palliative OutcomeScale. Health Qual Life Outcomes 2010;8:10.

29. McCorkle R, Young K. Development of a symp-tom distress scale. Cancer Nurs 1978;1:373e378.

30. Higginson IJ, McCarthy M. Validity of the sup-port team assessment schedule: do staffs’ ratings re-flect those made by patients or their families? PalliatMed 1993;7:219e228.

31. Ellershaw JE, Peat SJ, Boys LC. Assessing the ef-fectiveness of a hospital palliative care team. PalliatMed 1995;9:145e152.

32. Aaronson NK, Ahmedzai S, Bergman B, et al.The European Organization for Research andTreatment of Cancer QLQ-C30: a quality-of-life in-strument for use in international clinical trials inoncology. J Natl Cancer Inst 1993;85:365e376.

33. Steinhauser KE, Clipp EC, Bosworth HB, et al.Measuring quality of life at the end of life: validationof the QUAL-E. Palliat Support Care 2004;2:3e14.

34. Wade DT, Collin C. The Barthel ADL Index:a standard measure of physical disability? Int DisabilStud 1988;10:64e67.

35. Cohen SR, Mount BM, Strobel MG, Bui F. TheMcGill Quality of Life Questionnaire: a measure of

quality of life appropriate for people with advanceddisease. A preliminary study of validity and accept-ability. Palliat Med 1995;9:207e219.

36. Groenvold M, Petersen MA, Aaronson NK, et al.The development of the EORTC QLQ-C15-PAL:a shortened questionnaire for cancer patients inpalliative care. Eur J Cancer 2006;42:55e64.

37. Zigmond AS, Snaith RP. The hospital anxietyand depression scale. Acta Psychiatr Scand 1983;67:361e370.

38. Folstein MF, Folstein SE, McHugh PR. ‘‘Mini-mental state’’. A practical method for grading thecognitive state of patients for the clinician.J Psychiatr Res 1975;12:189e198.

39. Tamburini M, Rosso S, Gamba A, et al.A therapy impact questionnaire for quality-of-life as-sessment in advanced cancer research. Ann Oncol1992;3:565e570.

40. Portenoy RK, Thaler HT, Kornblith AB, et al.The Memorial Symptom Assessment Scale: an in-strument for the evaluation of symptom prevalence,characteristics and distress. Eur J Cancer 1994;30A:1326e1336.

41. Cella DF, Tulsky DS, Gray G, et al. The Func-tional Assessment of Cancer Therapy scale: develop-ment and validation of the general measure. J ClinOncol 1993;11:570e579.

42. O’Boyle C, McGee H, Hickey A, McGee H,Joyce CRB. The Schedule for the Evaluation of Indi-vidual Quaility of Life (SEIQoL). Administrationmanual. Dublin, Ireland: Royal College of Surgeonsin Ireland, 1993.

43. Grande GE, Todd CE. Why are trials in palliativecare so difficult? Palliat Med 2000;14:69e74.

44. Riva G, Teruzzi T, Anolli L. The use of the inter-net in psychological research: comparison of onlineand offline questionnaires. Cyberpsychol Behav2003;6:73e80.