validation of the rhinoplasty outcomes evaluation (roe) questionnaire adapted to brazilian...

6
Validation of the Rhinoplasty Outcomes Evaluation (ROE) questionnaire adapted to Brazilian Portuguese Suemy Cioffi Izu Eduardo Macoto Kosugi Alessandra Stanquini Lopes Karen Vitols Branda ˜o Leonardo Bomediano Garcia Sousa Vinı ´cius Magalha ˜es Suguri Luis Carlos Grego ´rio Accepted: 18 September 2013 / Published online: 1 October 2013 Ó Springer Science+Business Media Dordrecht 2013 Abstract Purpose The aim of this study was to validate the Rhi- noplasty Outcomes Evaluation (ROE) questionnaire adap- ted to the Brazilian Portuguese. Method A prospective study was conducted with ROE administration to 56 patients submitted to rhinoplasty (preoperatively, and then 15-day and 90-day postopera- tively) and 100 volunteers without the need or desire of cosmetic or functional nasal surgery. Reliability (internal consistency and test–retest reproducibility), validity, responsiveness and clinical interpretability were assessed. Results Rhinoplasty patients’ mean preoperative score was 7.14, 15 days post-op 17.73 and 90 days post-op 20.50, while controls presented 17.94 points (p \ 0.0001), showing the questionnaire’s validity and responsiveness. Internal consistency was 0.86. Inter- and intra-examiner test–retest reproducibility was 0.90 and 0.94, respectively. The effect size caused by the surgery was considered large (15 days post-op compared to the preoperative score: effect size = 3.22; 90 days post-op compared to preoperative score: effect size = 4.06). The minimally important dif- ference was 8.67 points, so changes smaller than 9 points in ROE might not be perceived by the patient as an improvement or worsening. Conclusion The Brazilian Portuguese version of ROE is a valid instrument to assess results in rhinoplasty patients. Keywords Rhinoplasty Quality of life Questionnaires Introduction Physicians’ concept of a good surgical outcome could be very different from patients’ thoughts. Most likely, the professional experience of surgeons leads to different expectations compared to patients [1]. A given surgeon may be satisfied with his or her results, but if patients themselves are not similarly pleased, then the intervention cannot totally be considered a success [2]. However, patient satisfaction could be influenced by many variables, such as availability and convenience of health care, the ‘‘bedside manner’’ of the doctor, affability of the extended team and perceived cleanliness of the hospital, which can complicate evaluation of clinical outcome [3, 4]. In this context, the quality of life questionnaires are very suitable tools that allow quantitative assessment of other- wise subjective results, such as patient satisfaction and, consequently, surgery success [3]. Quality of life can be defined as ‘‘individuals’ perception of their position in life in the context of the culture and value systems in which they live and in relation to their goals, expectations, stan- dards and concerns. It is a broad ranging concept affected in a complex way by the persons’ physical health, psy- chological state, level of independence, social relationships and their relationship to salient features of their environ- ment’’ [5]. In aesthetic interventions, more than any other aspect of rhinology, patient satisfaction and quality of life must be the measure against which successful procedure should be judged [3]. Based on such philosophy, Alsarraf developed four new outcome instruments to assess the results of cosmetic facial procedures: Rhinoplasty Outcomes Evaluation (ROE), Facelift Outcomes Evaluation (FOE), Blepharoplasty Outcomes Evaluation and Skin Rejuvenation Outcomes Evaluation (SROE) [2, 6]. In order to measure outcomes S. C. Izu E. M. Kosugi (&) A. S. Lopes K. V. Branda ˜o L. B. G. Sousa V. M. Suguri L. C. Grego ´rio Department of Otorhinolaryngology and Head and Neck Surgery, UNIFESP-EPM, Sao Paulo, SP, Brazil e-mail: [email protected] 123 Qual Life Res (2014) 23:953–958 DOI 10.1007/s11136-013-0539-x

Upload: luis-carlos

Post on 21-Jan-2017

216 views

Category:

Documents


1 download

TRANSCRIPT

Validation of the Rhinoplasty Outcomes Evaluation (ROE)questionnaire adapted to Brazilian Portuguese

Suemy Cioffi Izu • Eduardo Macoto Kosugi • Alessandra Stanquini Lopes •

Karen Vitols Brandao • Leonardo Bomediano Garcia Sousa • Vinıcius Magalhaes Suguri •

Luis Carlos Gregorio

Accepted: 18 September 2013 / Published online: 1 October 2013

� Springer Science+Business Media Dordrecht 2013

Abstract

Purpose The aim of this study was to validate the Rhi-

noplasty Outcomes Evaluation (ROE) questionnaire adap-

ted to the Brazilian Portuguese.

Method A prospective study was conducted with ROE

administration to 56 patients submitted to rhinoplasty

(preoperatively, and then 15-day and 90-day postopera-

tively) and 100 volunteers without the need or desire of

cosmetic or functional nasal surgery. Reliability (internal

consistency and test–retest reproducibility), validity,

responsiveness and clinical interpretability were assessed.

Results Rhinoplasty patients’ mean preoperative score

was 7.14, 15 days post-op 17.73 and 90 days post-op

20.50, while controls presented 17.94 points (p \ 0.0001),

showing the questionnaire’s validity and responsiveness.

Internal consistency was 0.86. Inter- and intra-examiner

test–retest reproducibility was 0.90 and 0.94, respectively.

The effect size caused by the surgery was considered large

(15 days post-op compared to the preoperative score: effect

size = 3.22; 90 days post-op compared to preoperative

score: effect size = 4.06). The minimally important dif-

ference was 8.67 points, so changes smaller than 9 points in

ROE might not be perceived by the patient as an

improvement or worsening.

Conclusion The Brazilian Portuguese version of ROE is a

valid instrument to assess results in rhinoplasty patients.

Keywords Rhinoplasty � Quality of life �Questionnaires

Introduction

Physicians’ concept of a good surgical outcome could be

very different from patients’ thoughts. Most likely, the

professional experience of surgeons leads to different

expectations compared to patients [1]. A given surgeon

may be satisfied with his or her results, but if patients

themselves are not similarly pleased, then the intervention

cannot totally be considered a success [2]. However,

patient satisfaction could be influenced by many variables,

such as availability and convenience of health care, the

‘‘bedside manner’’ of the doctor, affability of the extended

team and perceived cleanliness of the hospital, which can

complicate evaluation of clinical outcome [3, 4].

In this context, the quality of life questionnaires are very

suitable tools that allow quantitative assessment of other-

wise subjective results, such as patient satisfaction and,

consequently, surgery success [3]. Quality of life can be

defined as ‘‘individuals’ perception of their position in life

in the context of the culture and value systems in which

they live and in relation to their goals, expectations, stan-

dards and concerns. It is a broad ranging concept affected

in a complex way by the persons’ physical health, psy-

chological state, level of independence, social relationships

and their relationship to salient features of their environ-

ment’’ [5]. In aesthetic interventions, more than any other

aspect of rhinology, patient satisfaction and quality of life

must be the measure against which successful procedure

should be judged [3].

Based on such philosophy, Alsarraf developed four new

outcome instruments to assess the results of cosmetic facial

procedures: Rhinoplasty Outcomes Evaluation (ROE),

Facelift Outcomes Evaluation (FOE), Blepharoplasty

Outcomes Evaluation and Skin Rejuvenation Outcomes

Evaluation (SROE) [2, 6]. In order to measure outcomes

S. C. Izu � E. M. Kosugi (&) � A. S. Lopes �K. V. Brandao � L. B. G. Sousa � V. M. Suguri � L. C. Gregorio

Department of Otorhinolaryngology and Head and Neck

Surgery, UNIFESP-EPM, Sao Paulo, SP, Brazil

e-mail: [email protected]

123

Qual Life Res (2014) 23:953–958

DOI 10.1007/s11136-013-0539-x

such as patient satisfaction and quality of life in the facial

plastic surgery patient, he identified three key aspects that

constituted such satisfaction for each treatment modality of

interest: physical, emotional and social factor. Emotional

and social factors were similar among each of the specific

interventions assessed by Alsarraf [2], while physical fac-

tors presented very specific characteristics depending on

the surgical option.

ROE was translated and cross-culturally adapted to

Brazilian Portuguese [7], according to Guillemin et al. [8].

However, validation process was not carried out. The

objective of this study is to validate the Brazilian Portu-

guese version of the ROE.

Method

This research was approved by the Ethics in Research

Committee of the institution, under protocol number CEP

1791/11, and all participants were volunteers and signed

the informed consent form.

The Brazilian Portuguese ROE questionnaire [7] is

shown in Fig. 1. It is comprised of six questions; each of

the six items was scored on a 0–4 scale, with 0 repre-

senting the most negative response and 4 representing the

most positive response. Therefore, the total score can vary

from 0 to 24. In order to facilitate the comprehension of

the results, the total score can be divided by 24 and

multiplied by 100, so that the score can vary from 0 to

100 %. So, 24 points or 100 % means the most patient

satisfaction [2].

Recruitment of patients

Volunteer patients that desired cosmetic nasal surgery

(rhinoplasty) with or without functional surgery (septo-

plasty and/or turbinectomy) were recruited from a rhinol-

ogy unit to be part of rhinoplasty group. Exclusion criteria

comprised the desire not to participate in the study or

revision cases. Brazilian Portuguese ROE questionnaire

was filled in by participants in the preoperative (pre-op)

and in 15 and 90 days postoperative periods (PO15d and

PO90d, respectively). In the preoperative period, the

questionnaire was administered by two of the authors

(A.S.L. and K.V.B.) and was repeated by one of them after

15 days to check reproducibility. Afterward, the patients

were assessed by one of the authors (A.S.L. or K.V.B.) in

the PO15d and in the PO90d. In the postoperative evalu-

ations, the patients also answered whether they felt: much

better, a little better, about the same, a little worse or much

worse, than the period before the intervention.

The control group was made up of healthy volunteers

with no desire or need for aesthetic and/or functional nasal

surgery, who were recruited from the clinic, employees and

students or relatives from our institution.

Analysis of data

Reliability was analyzed in two ways: internal consistency

and test–retest reproducibility. Internal consistency refer-

red to the way individual items relate to each other, in order

to provide homogeneity among them, and was measured

using Cronbach’s alpha [9]. The minimum acceptable score

Fig. 1 Brazilian Portuguese

version of Rhinoplasty

Outcomes Evaluation

questionnaire

954 Qual Life Res (2014) 23:953–958

123

for Cronbach’s alpha is 0.7 [10–12]. Test–retest repro-

ducibility measured the stability of an instrument over time

after repeated testing [10, 11]. The questionnaire was

applied twice in 15 days by the same examiner (intra-

examiner test–retest reproducibility) and twice in the same

day by two different examiners (inter-examiner test–retest

reproducibility) [10]. The correlation between the scores,

measured by intra-class correlation coefficient (ICC), must

be of, at least, 0.7 [10, 11].

Measure’s validity is the capacity of the questionnaire to

reflect differences among known groups (diseased vs. non-

diseased) using the unpaired t test or Mann–Whitney

U test, depending on samples’ homogeneity of variance

measured by Levene’s test and normal distribution mea-

sured by Kolmogorov–Smirnov test. Basically, Brazilian

Portuguese ROE should be able to distinguish the group of

patients that wish/need rhinoplasty from the group of

subjects that did not [10, 11].

Responsiveness is the ability of the questionnaire to

detect clinical differences over time. Pre- and postoperative

scores were compared using paired t test or Wilcoxon

signed-rank test, depending on samples’ homogeneity of

variance measured by Levene’s test and normal distribu-

tion measured by Kolmogorov–Smirnov test. Also,

responsiveness was assessed by measuring the effect size:

the mean change score divided by baseline standard devi-

ation. By convention, an effect size of [0.2 is considered

small, [0.5 moderate and [0.8 a large improvement in

health quality of life [10, 11].

Clinical interpretability can be calculated by the mini-

mally important difference (MID), the smallest change in

scores that a group of patients can detect as a real

improvement [13]. In order to do this, patients were divi-

ded into groups according to a reported transition rating

scale comparing pre- and postoperative health: much bet-

ter, a little better, about the same, a little worse or much

worse. Then, the mean change in score for those who

reported to be ‘‘a little better’’ minus the mean change in

score for those who reported to be ‘‘about the same’’ pro-

duced the MID [11, 13].

For the statistical tests, results of p \ 0.05 were con-

sidered significant.

Results

Patient characteristics

Fifty-six patients were evaluated in the pre-op, PO15d and

PO90d of rhinoplasty (rhinoplasty group), whereas a hun-

dred volunteers were included in control group, as shown

in Table 1. There was no difference in gender or age dis-

tribution between both groups.

Data obtained

Internal consistency of the questionnaire, measured by the

Cronbach’s alpha coefficient, was high (0.86). Eliminating

one question at time, Cronbach’s alpha scores varied from

0.79 to 0.87, showing homogeneity among questions.

Test–retest reproducibility was evaluated with all

patients in the pre-op. Inter-examiner and intra-examiner

ICC were 0.90 and 0.94, respectively, indicating good

reliability.

Controls, pre-op and PO15d ROE scores were normally

distributed according to Kolmogorov–Smirnov test

(p = 0.13, p = 0.12 and p = 0.23, respectively). How-

ever, PO90d ROE scores were not (p = 0.01).

Validity of the instrument was shown in Table 2. Le-

vene’s test showed heterogeneity of variance between

patients and controls, so Mann–Whitney U test was per-

formed. Brazilian Portuguese ROE questionnaire could

distinguish patients from controls.

Responsiveness of the instrument was noticed even in a

very recent postoperative follow-up such as 15 days, as

presented in Table 3. Moreover, the questionnaire was

sufficiently sensitive for detecting changes in patients’

satisfaction from PO15d to PO90d. As PO90d ROE scores

were not normally distributed, Wilcoxon signed-rank test

was used.

Table 1 Characteristics of the sample

Characteristics Rhinoplasty Controls

Women Men Total Women Man Total

Number 33 23 56 56 44 100

Percentage 58.93 % 41.07 % 100 % 56 % 44 % 100 %

Mean age (years) 27.57 31.16 29.65 30.02 31.39 30.79

SD of age (years) 9.47 10.00 9.86 7.03 10.42 9.07

Range of age (years) 14–53 15–50 14–53 21–66 18–65 18–66

t test: female patients versus male patients, p = 0.18; female controls versus male controls, p = 0.44; total women versus total men, p = 0.48

SD standard deviation

Qual Life Res (2014) 23:953–958 955

123

The evolution of Brazilian Portuguese ROE scores

according to the time of the surgery can be seen in Table 4.

Preoperative patients’ scores were much lower than

controls’ scores. At PO15d, patients have already presented

improvement in their quality of life, which placed them on

the same level as control group. At PO90d, rhinoplasty was

responsible for a greater improvement in patients’ quality

of life that led them even better than controls.

Rhinoplasty effect size on quality of life (measured by

the mean change score divided by baseline standard devi-

ation) detected by Brazilian Portuguese ROE was consid-

ered large (PO15d in relation to pre-op = 3.22; PO90d in

relation to pre-op = 4.06). Effect size between PO15d and

PO90d was moderate (0.71).

The mean score for each question in each group could

be seen in Table 5. Most of questions have presented sta-

tistically significant difference among groups, but between

PO15d and controls, as presented in Table 5.

To calculate the MID, clinical condition classification

from both PO15d and PO90d was obtained. Mean changes

in scores from postoperative to preoperative for each of the

symptom transition rating groups were showed in Table 6.

The minimally important difference was of 8.67 points.

That means that a variation of less than 9 points in the

Brazilian Portuguese ROE might not be perceived as

worsening or improvement by the patient.

Discussion

When elaborating the ROE, Alsarraf was concerned about

the ease and simplicity of instrument administration [2].

After translation and cross-cultural adaptation [7], despite

difficulties in text reading and comprehension by part of

the population seen in our service, this characteristic was

not lost. In this study, questions were read to the patients

Table 2 Validation of ROE

ROE score Groups

Rhinoplasty (pre-op) Controls

Participants 56 100

Mean score 7.14 17.94

SD score 3.29 3.91

Median score 7 18

Levene’s test: patients versus controls p = 0.04*; Mann–Whitney

U test: patients versus controls p \ 0.0001*; effect size (95 %

CI) = -2.92 (-3.37 to -2.45)

ROE Rhinoplasty Outcome Evaluation; pre-op preoperative; SD

standard deviation; CI confidence interval

Table 3 Responsiveness of ROE

ROE score Rhinoplasty

Pre-op PO15d PO90d

Average 7.14 17.73 20.50

SD 3.29 3.88 3.51

Median 7 18 21

Levene’s test: p = 0.27; KS test: pre-op p = 0.12, PO15d p = 0.23,

PO90d p = 0.01*; t test: pre-op versus PO15d p \ 0.00001*; Wil-

coxon test pre-op versus PO90d p \ 0.000001*; Wilcoxon test

PO15d versus PO90d p \ 0.0001*; ES (95 % CI): pre-op versus

PO15d 2.94 (2.39–3.46); pre-op versus PO90d 3.93 (3.27–4.53);

PO15d versus PO90d 0.75 (0.36–1.13)

ROE Rhinoplasty Outcome Evaluation; pre-op preoperative; PO15d

15 days postoperative; PO90d 90 days postoperative; ES effect size;

CI confidence interval

Table 4 Variation of the patients’ scores in comparison with the controls

ROE score Pre-op Controls Variation ROE Levene’s test Mann–Whitney U test Effect size (95 % CI)

Mean 7.14 17.94 -10.80 p = 0.04* p \ 0.0001* -2.92 (-3.36 to -2.44)

SD 3.29 3.91

Median 7 18

ROE score PO15d Controls Variation ROE Levene’s test t test Effect size (95 % CI)

Mean 17.73 17.94 -0.21 p = 0.96 p = 0.75 -0.05 (-0.38 to 0.27)

SD 3.88 3.91

Median 18 18

ROE score PO90d Controls Variation ROE Levene’s test Mann–Whitney U test Effect size (95 % CI)

Mean 20.50 17.94 2.56 p = 0.25 p \ 0.0001* 0.68 (0.34–1.01)

SD 3.51 3.91

Median 21 18

Kolmogorov–Smirnov test: controls p = 0.13, pre-op p = 0.12, PO15d p = 0.23, PO90d p = 0.01*

ROE Rhinoplasty Outcome Evaluation; pre-op preoperative; PO15d 15 days postoperative; PO90d 90 days postoperative; CI confidence interval

956 Qual Life Res (2014) 23:953–958

123

instead of self-administration, like others Brazilian ques-

tionnaires studies [7, 10, 14, 15]. Probably, the adminis-

tration of the questionnaire to patients did not alter its

purpose, since the reading was done ipsis litteris, without

any explanation of the questions. Moreover, this way of

administration had some advantages over self-administra-

tion, such as faster filling out time, lower rate of missing

data and interviewees’ preference [16]. However, an

interviewer can enhance motivation to respond, creating

worse results in preoperative answers and better results in

postoperative ones [17]. Nevertheless, a study with stroke

patients showed that modes of administration of question-

naires could be used interchangeably [18].

Brazilian Portuguese version of ROE questionnaire

showed high internal consistency like the original one, with

Cronbach’s alpha coefficient of 0.86 and 0.84, respectively

[6]. The test–retest reproducibility was assessed in different

ways. In the original study, patients filled out the ques-

tionnaire (self-administration) two times: first, during ini-

tial consult and then on the day of surgery (0.5–9.5 weeks

after the initial consult; mean = 3.5 weeks), with intra-

class correlation coefficient of 0.83 [6]. Alsarraf et al. did

not assess inter-examiner test–retest reproducibility, due to

self-administration of questionnaires. Despite differences

in the administration of questionnaire, high correlation

coefficients have been achieved by both forms.

Regarding the validity of the questionnaire, Brazilian

Portuguese ROE had an optimum performance. A great

difference in scores was noted when comparing subjects

that needed or desired rhinoplasty (mean = 7.14 or

29.75 %) with subjects that did not (mean = 17.94 or

74.75 %), and these results were in agreement with Izu

et al. [7], who obtained a cutoff of 12, separating ‘‘dis-

eased’’ from ‘‘non-diseased’’ subjects.

According to other studies that used ROE questionnaire

as an outcome measure for rhinoplasty [6, 7, 19–23],

postoperative scores were much better than preoperative

ones. In original ROE validation, Alsarraf et al. [6] showed

a mean gain of 44.5 % in ROE scores after a five-month

follow-up, whereas the present study reached 55.66 % of

mean gain after three-month follow-up. PO3 m Group

scores were statistically significant better than control

group scores, indicating that postoperative patients’ satis-

faction could have exceeded that of control group. This

demonstrated the responsiveness of the test, which pre-

sented high effect size too.

Follow-up time varied in the literature [6, 7, 19–23]. In

the present study, differences in satisfaction were noted

between 15 and 90 days follow-up, probably due to

important edema and other complaints presented by

patients at PO15d, which usually improved at PO90d. After

6 months from surgery, less improvement usually occurs,

so that Arima et al. [23] with follow-up time varying from

6 months to 10 years, did not find any statistically signif-

icant difference in quality of life related to the follow-up

time.

When analyzing preoperative scores for each question,

nasal aesthetic issues specifically (questions 1, 5 and 6)

presented worse scores. Question 2, about nasal obstruc-

tion, also had low scores, but better than nasal aesthetic

questions. However, all questions presented good postop-

erative scores, showing that even if primary purpose was

only aesthetic, great attention should be paid to obstructive

factors during rhinoplasty, since this would promote great

improvement in quality of life too.

Although this study presented improvements in patients’

quality of life after cosmetic rhinoplasty, the purpose of

this study was not to evaluate this procedure specifically,

but to demonstrate the ability of Brazilian version of ROE

questionnaire to accurately distinguish these modifications:

Could ROE discern satisfied versus unsatisfied postopera-

tive patients? Satisfied patients are as pleased as people

who do not deserve rhinoplasty? So, comparing pre- and

postoperative patients with healthy controls could be the

best way to measure the exact impairment in quality of life

caused by disease in preoperative point and the gain after

surgery. However, the choice of control group always

Table 5 Mean scores per question per group

Question ROE mean scores

Pre-op PO15d PO90d Controls

1. 0.76 2.88 3.55 2.79

2. 1.53 2.75 3.45 3.03

3. 1.54 3.11 3.61 2.89

4. 2.75 3.52 3.91 3.84

5. 0.46 2.82 3.41 2.36

6. 0.05 2.55 2.48 2.95

t test: significant for all the comparisons between groups for each

question, but for questions 1, 2, 3 and 6 between PO15d and controls;

questions 4 and 6 between PO90d and controls; and question 6

between PO15d and PO90d

ROE Rhinoplasty Outcome Evaluation; pre-op preoperative; PO15d

15 days postoperative; PO90d 90 days postoperative

Table 6 Post–preoperative scores variation per transition group

Symptom transition rating group N Mean variation SD

Much better 94 13.04 4.42

A little better 15 8.67 2.66

The same 1 0 X

A little worse 2 0.5 4.95

Much worse 0 X X

N number; SD standard deviation

Qual Life Res (2014) 23:953–958 957

123

evokes discussion. We would like to choose volunteers

who really do not want or need nasal surgery, so that we

could compare these satisfied volunteers with satisfied

postoperative patients. Now, with a validated instrument,

many settings should be compared, beyond pure cosmetic

rhinoplasty, as malformation and post-trauma reconstruc-

tive rhinoplasty, for instance.

Conclusion

Brazilian Portuguese version of the ROE questionnaire is a

valid instrument to assess results in rhinoplasty patients,

presenting good internal consistency, reproducibility,

validity and responsiveness.

References

1. Moolenburgh, S. E., Mureau, M. A., & Hofer, S. O. (2008).

Aesthetic outcome after nasal reconstruction: Patient versus panel

perception. Journal of Plastic, Reconstructive & Aesthetic Sur-

gery, 61(12), 1459–1464.

2. Alsarraf, R. (2000). Outcomes research in facial plastic surgery:

A review and new directions. Aesthetic Plastic Surgery, 23(3),

192–197.

3. Hopkins, C. (2009). Patient reported outcome measures in rhi-

nology. Rhinology, 47(1), 10–17.

4. Fitzpatrick, R. (1991). Surveys of patient satisfaction : I-impor-

tant general considerations. British Medical Journal, 302(6781),

887–889.

5. The WHOQOL Group. (1998). The World Health Organization

Quality of Life Assessment (WHOQOL): Development and

general psychometric properties. Social Science and Medicine,

46(12), 1569–1585.

6. Alsarraf, R., Larrabee, W. F., Anderson, S., Murakami, C. S., &

Johnson, C. S, Jr. (2001). Measuring cosmetic facial plastic

surgery outcomes: A pilot study. Archives of Facial Plastic

Surgery, 3(3), 198–201.

7. Izu, S. C., Kosugi, E. M., Brandao, K. V., Lopes, A. S., Garcia, L.

B., Suguri, V. M., et al. (2012). Normal values for the Rhino-

plasty Outcome Evaluation (ROE) questionnaire. Brazilian

Journal of Otorhinolaryngology, 78(4), 76–79.

8. Guillemin, F., Bombardier, C., & Beaton, D. (1993). Cross-cul-

tural adaptation of health-related quality of life measures: Liter-

ature review and proposed guidelines. Journal of Clinical

Epidemiology, 46(12), 1417–1432.

9. Cronbach, L. J. (1951). Coefficient alpha and the internal struc-

ture of tests. Psychometrika, 16(3), 297–334.

10. Kosugi, E. M., Chen, V. G., Fonseca, V. M., Cursino, M. M.,

Mendes-Neto, J. A., & Gregorio, L. C. (2011). Translation, cross-

cultural adaptation and validation of SinoNasal Outcome Test

(SNOT): 22 to Brazilian Portuguese. Brazilian Journal of Oto-

rhinolaryngology, 77(5), 663–669.

11. Hopkins, C., Gillett, S., Slack, R., Lund, V. J., & Browne, J. P.

(2009). Psychometric validity of the 22-item Sinonasal Outcome

Test. Clinical Otolaryngology, 34(5), 447–454.

12. Scientific Advisory Committee of the Medical Outcomes Trust.

(2002). Assessing health status and quality-of-life instruments :

Attributes and review criteria. Quality of Life Research, 11(3),

193–205.

13. Jaeschke, R., Singer, J., & Guyatt, G. H. (1989). Measurement of

health status. Ascertaining the minimal clinically important dif-

ference. Controlled Clinical Trials, 10(4), 407–415.

14. Klatchoian, D. A., Len, C. A., Terreri, M. T., Silva, M., Itamoto,

C. H., Ciconelli, R. M., et al. (2008). Quality of life of children

and adolescents from Sao Paulo: reliability and validity of the

Brazilian version of the Pediatric Quality of Life Inventory ver-

sion 4.0 Generic Core Scales. Jornal de Pediatria, 84(4),

308–315.

15. Fonseca, E. S., Camargo, A. L., Castro, R. A., Sartori, M. G.,

Fonseca, M. C., Lima, G. R., et al. (2005). Validation of a quality

of life questionnaire (King0s Health Questionnaire) in Brazilian

women with urinary incontinence. Revista Brasileira de Ginec-

ologia e Obstetrıcia, 27(5), 235–242.

16. Weinberger, M., Oddone, E. Z., Samsa, G. P., & Landsman, P. B.

(1996). Are health-related quality-of-life measures affected by

the mode of administration? Journal of Clinical Epidemiology,

49(2), 135–140.

17. Bowling, A. (2005). Mode of questionnaire administration can

have serious effects on data quality. Journal of Public Health

(Oxford, England), 27(3), 281–291.

18. Sutton, C. J., Watkins, C. L., Cook, N., Leathley, M. J., McAdam,

J., & Dey, P. (2013). Postal and face-to-face administration of

stroke outcome measures: Can mixed modes be used? Stroke,

44(1), 217–219.

19. Hellings, P. W., & Nolst Trenite, G. J. (2007). Long-term patient

satisfaction after revision rhinoplasty. The Laryngoscope, 117(6),

985–989.

20. Arima, L. M., Velasco, L. C., & Tiago, R. S. (2012). Influence of

age on rhinoplasty outcomes evaluation: A preliminary study.

Aesthetic Plastic Surgery, 36(2), 248–253.

21. Faidiga, G. B., Carenzi, L. R., Yassuda, C. C., Silveira, F., Lago,

T. D., Leite, M. G., et al. (2010). Long-term evaluation in aes-

thetic rhinoplasty in an academic referral center. Brazilian

Journal of Otorhinolaryngology, 76(4), 437–441.

22. Meningaud, J. P., Lantieri, L., & Bertrand, J. C. (2008). Rhino-

plasty: An outcome research. Plastic and Reconstructive Surgery,

121(1), 251–257.

23. Arima, L. M., Velasco, L. C., & Tiago, R. S. (2011). Crooked

nose: Outcome evaluations in rhinoplasty. Brazilian Journal of

Otorhinolaryngology, 77(4), 510–515.

958 Qual Life Res (2014) 23:953–958

123