Voice of the Users A Demographic Study ofSoftware Feedback Behaviour

James Tizard1 Tim Rietz2 Kelly Blincoe1

1University of Auckland New Zealand2Karlsruhe Institute of Technology Germany

jtiz003aucklanduniacnz timrietzkitedu kblincoeaucklandacnz

AbstractmdashUser feedback on mobile app stores product fo-rums and on social media can contain product developmentinsights There has been a lot of recent research studying thisfeedback and developing methods to automatically extractrequirement-related information This feedback is generallyconsidered to be the ldquovoice of the users however onlya subset of software users provide online feedback If thedemographics of the online feedback givers are not repre-sentative of the user base this introduces the possibility ofdeveloping software that does not meet the needs of all usersIt is therefore important to understand who provides onlinefeedback to ensure the needs of underrepresented groups arenot being missed

In this work we directly survey 1040 software users abouttheir feedback habits software use and demographic infor-mation Their responses indicate that there are statisticallysignificant differences in who gives feedback on each onlinechannel with respect to traditional demographics (genderage etc) We also identify key differences in what motivatessoftware users to engage with each of the three channels Ourfindings provide valuable context for requirements elicitedfrom online feedback and show that considering informationfrom all channels will provide a more comprehensive viewof user needs

I INTRODUCTION

Software users give large volumes of feedback onlineabout the products they are using [1] [2] Previous researchhas found that software feedback in app stores socialmedia and on product forums contains valuable productdevelopment insights that can be used to guide the evolutionof the software being discussed [2] [3] [4] This feedbackis generally considered to be the ldquovoice of the users [5]There has been a recent surge of requirements engineeringresearch developing ways to make this feedback moreaccessible to product development teams (eg [3] [6] [7][8])

However only a subset of software users provide onlinefeedback If online feedback is being used to drive productdevelopment decisions the concerns and desires of onlythe vocal users are being considered If the demographicsof the vocal users are not representative of the overallset of users this introduces the possibility of developingbiased software that does not meet the needs of all usersTherefore itrsquos important to understand which softwareusers do give online feedback and in doing so identifygroups whose views may be underrepresented

Yet very little research has investigated who is givingonline feedback for software products with respect to thedemographics of the users This may be due to the factthat demographic information of feedback givers is notreadily available On some feedback channels even the fullname of the person providing the feedback is unavailable

Some preliminary studies have investigated the gender andgeographic location of users providing feedback on appstores [5] [9] These studies found that men were morelikely than women to provide feedback on the Apple appstore However these results are obtained by approximatinggender based on usernames since actual gender identityof the feedback givers is not available on app stores

In this work we overcome the online data sparsityproblem by directly surveying 1040 software users abouttheir software use feedback habits and their demographicinformation We collect demographic information of feed-back givers on three popular channels app stores productforums and social media By directly collecting demo-graphic information we were able to examine feedbackhabits across a wide range of demographics includinggender age education and ethnicity We also investigatedwhat motivates feedback givers and if their software usagehabits relates to their feedback giving habits

Our study was guided by the following research ques-tions

RQ1 What are the demographics of software users whoreport to give online written feedback

RQ2 What motivates software users to give onlinefeedback and are there differences across demographics

RQ3 Does the likelihood of giving online writtenfeedback vary based on the type of software used andthe duration of software usage

The contributions of this paper are insights about whogives online feedback and what motivates them Specifi-cally (1) We show that there are differences in the feedbackhabits of software users based on traditional demographicsFor gender men reported giving more written feedbackthan women With age distinct patterns emerged withrespondents between 35 and 45 reporting to give the mostwritten feedback on all channels (2) We show that usergroups have different motivations to give feedback andthese motivations vary across each of the three feedbackchannels Respondents also reported differences in thesuccess of in-app prompts between eliciting app ratingsand written feedback and differences in the frequencyindividual feedback givers write on app stores productforums and social media (3) We present evidence thatsoftware users feedback habits also vary with respect tothe way they use software Respondents who spend morehours each day on their phone or computer report givingmore written feedback about the software theyrsquore usingThe software platform being used also has a relationship towritten feedback rates with Linux (computer) and Android

(phone) users reporting to give more feedback than thoseusing other platforms

Our findings give valuable insights into which softwareusers give online feedback and what motivates them todo so Additionally the findings emphasise the need tomine all three feedback channels in order to get themost representative requirements from the user base whenleveraging online user feedback

The paper is structured as follows Section II reviews therelated work that informed our research In section III wepresent our research methodology The results are presentedin section IV and discussed in section V including adiscussion on the threats to validity Finally section VIconcludes the paper

II RELATED WORK

A User feedback in requirements engineering

Researchers have found requirements relevant informa-tion in feedback on several prominent online channelsincluding app stores social media and product forums[2] [3] [4] These channels can contain large volumes ofvaluable information Pagano and Maalej [2] found thatapproximately a third of user reviews on app stores containinformation related to software requirements User feedbackthat contains bug reports or feature requests (and more)can be used by developers to address the needs and desiresof their users which is critical to the ongoing success oftheir software

Manually eliciting software requirements from onlinefeedback can be extremely time intensive due to thelarge volumes and varying quality of text language thatcomes from highly distributed user bases [10] Much recentresearch has investigated methods to automatically minerequirements in user feedback on app stores Twitter andproduct support forums [1] [3] [6] [7] [8] [11] [12]

B Demographics of software user who give feedback

There has been limited research to understand whichsoftware users give online feedback and what motivatesthem Guzman et al [5] looked at the difference betweenmen and women who give feedback on the Apple app storeThey manually approximated the gender of each personleaving a written review based on their username Theyfound a slight majority (57) of the reviews were writtenby men However there were differences in this ratio whengeographic region was considered For example in India83 of feedback givers were men In Australia womenwrote the majority (67) of the reviews They did not findany statistically significant differences in review sentimentcontent and rating between genders

Another study investigated differences in feedback fromthe Apple app stores of eight countries [9] This study foundthat feedback characteristics such as sentiment contentrating and length significantly varied between the countries

These studies were both limited to the Apple app store Inaddition since demographic information like gender is notavailable for app store users gender was only approximatedand other demographics like age could not be studiedThis study provides a more thorough investigation intodemographics of feedback givers across three prominent

types of online feedback channels (app stores productforums and social media)

III METHODOLOGY

To better understand which software users give onlinefeedback and what motivates them we ran an onlinequestionnaire This enabled us to survey a large numberof software users (1040) to determine if reported feedbackhabits and motivations differed across demographics

A Survey Design

The survey consisted of 24 multiple-choice questionsshown in Table I The survey consisted of five mainsets of questions The first three sets of questions askedabout the feedback the participant provides in the threefeedback channels under investigation app stores (Q1-5)social media (Q6-9) and product forums (Q10-13) Theremaining two sets of questions collect software usageinformation (Q15-18) and demographic information (Q19-24) Descriptions of what was meant by app store andproduct forum feedback were given within the survey tohelp participants understand the question context Questionseliciting details on feedback habits were asked beforesoftware usage and demographic questions to highlight thepropose of the study and to maintain participant interest

The sets of questions on the three feedback channelseach follow the same general format First the participantis asked if they have given feedback on that channel Nextif applicable they are asked how frequently they givefeedback the type of feedback given (eg reporting abug) and their motivation for providing feedback on thischannel These questions were all multiple choice Theanswer options for the type of feedback provided and themotivation for providing feedback were based on findingsfrom recent research studies on each of these feedbackchannels The participants were also asked about theirperceptions on the impact of their feedback on influencingchanges in the software products (Q14) For some questionsparticipants could select more than one answer choice (egmotivation for giving feedback) The full list of questionsand answer choices is shown in Table I Abbreviatedanswers for each question are given in the table anunabbreviated copy of the survey can be found on Zenodo1

The software usage questions asked participants howthey interact with software products including the types ofdevices they use the types of software they use and thehours spent on devices each day The answer choices forthe types of software were obtained from the categories ofapps on popular app stores

The demographic questions collected information onthe participants age gender ethnicity education andemployment These questions and their associated answerchoices were informed by traditional marketing demo-graphic categories [13] as well as the New Zealand census(2018) [14]

B Ethics Approval

The survey had ethics approval from the University ofAucklandrsquos Human Participants Ethics Committee

1httpszenodoorgrecord3674076XkxNFygzZPY

(phone) users reporting to give more feedback than thoseusing other platforms

Our findings give valuable insights into which softwareusers give online feedback and what motivates them todo so Additionally the findings emphasise the need tomine all three feedback channels in order to get themost representative requirements from the user base whenleveraging online user feedback

The paper is structured as follows Section II reviews therelated work that informed our research In section III wepresent our research methodology The results are presentedin section IV and discussed in section V including adiscussion on the threats to validity Finally section VIconcludes the paper

II RELATED WORK

A User feedback in requirements engineering

Researchers have found requirements relevant informa-tion in feedback on several prominent online channelsincluding app stores social media and product forums[2] [3] [4] These channels can contain large volumes ofvaluable information Pagano and Maalej [2] found thatapproximately a third of user reviews on app stores containinformation related to software requirements User feedbackthat contains bug reports or feature requests (and more)can be used by developers to address the needs and desiresof their users which is critical to the ongoing success oftheir software

Manually eliciting software requirements from onlinefeedback can be extremely time intensive due to thelarge volumes and varying quality of text language thatcomes from highly distributed user bases [10] Much recentresearch has investigated methods to automatically minerequirements in user feedback on app stores Twitter andproduct support forums [1] [3] [6] [7] [8] [11] [12]

B Demographics of software user who give feedback

There has been limited research to understand whichsoftware users give online feedback and what motivatesthem Guzman et al [5] looked at the difference betweenmen and women who give feedback on the Apple app storeThey manually approximated the gender of each personleaving a written review based on their username Theyfound a slight majority (57) of the reviews were writtenby men However there were differences in this ratio whengeographic region was considered For example in India83 of feedback givers were men In Australia womenwrote the majority (67) of the reviews They did not findany statistically significant differences in review sentimentcontent and rating between genders

Another study investigated differences in feedback fromthe Apple app stores of eight countries [9] This study foundthat feedback characteristics such as sentiment contentrating and length significantly varied between the countries

These studies were both limited to the Apple app store Inaddition since demographic information like gender is notavailable for app store users gender was only approximatedand other demographics like age could not be studiedThis study provides a more thorough investigation intodemographics of feedback givers across three prominent

types of online feedback channels (app stores productforums and social media)

III METHODOLOGY

To better understand which software users give onlinefeedback and what motivates them we ran an onlinequestionnaire This enabled us to survey a large numberof software users (1040) to determine if reported feedbackhabits and motivations differed across demographics

A Survey Design

The survey consisted of 24 multiple-choice questionsshown in Table I The survey consisted of five mainsets of questions The first three sets of questions askedabout the feedback the participant provides in the threefeedback channels under investigation app stores (Q1-5)social media (Q6-9) and product forums (Q10-13) Theremaining two sets of questions collect software usageinformation (Q15-18) and demographic information (Q19-24) Descriptions of what was meant by app store andproduct forum feedback were given within the survey tohelp participants understand the question context Questionseliciting details on feedback habits were asked beforesoftware usage and demographic questions to highlight thepropose of the study and to maintain participant interest

The sets of questions on the three feedback channelseach follow the same general format First the participantis asked if they have given feedback on that channel Nextif applicable they are asked how frequently they givefeedback the type of feedback given (eg reporting abug) and their motivation for providing feedback on thischannel These questions were all multiple choice Theanswer options for the type of feedback provided and themotivation for providing feedback were based on findingsfrom recent research studies on each of these feedbackchannels The participants were also asked about theirperceptions on the impact of their feedback on influencingchanges in the software products (Q14) For some questionsparticipants could select more than one answer choice (egmotivation for giving feedback) The full list of questionsand answer choices is shown in Table I Abbreviatedanswers for each question are given in the table anunabbreviated copy of the survey can be found on Zenodo1

The software usage questions asked participants howthey interact with software products including the types ofdevices they use the types of software they use and thehours spent on devices each day The answer choices forthe types of software were obtained from the categories ofapps on popular app stores

The demographic questions collected information onthe participants age gender ethnicity education andemployment These questions and their associated answerchoices were informed by traditional marketing demo-graphic categories [13] as well as the New Zealand census(2018) [14]

B Ethics Approval

The survey had ethics approval from the University ofAucklandrsquos Human Participants Ethics Committee

1httpszenodoorgrecord3674076XkxNFygzZPY

TABLE ISURVEY QUESTIONS

Question RQ Topic Question AnswerSource

Q1 All App store What review types have you given to mobile apps in the past(choose all that apply)(None Prompted rating Prompted written review Direct rating Direct written review)

-

Q2 RQ2 App store How many times have you given mobile apps you use a star rating in the last year(None 1-4 times 5-12 times 13-26 times 27-52 times 53 or more times)

-

Q3 7 11 All App store (Q3)Product forum (Q7)Social media (Q11)

How many times have you written (or given a review) on this channel in the last year(None 1-4 times 5-12 times 13-26 times 27-52 times 53 or more times)

-

Q4 8 12 RQ1 App store (Q4)Product forum (Q8)Social media (Q12)

What types of posts (or reviews) have you written about software (or apps)(choose all that apply)(Praise (all channels) Report bug (all channels) Request feature (all channels)) Ask a question(all channels) Recommend to others (app stores social media) Dissuade others (app stores socialmedia) Discuss shortcoming (app stores social media) Dispraise or criticise (app store productforum) Discuss a helpful situation (app stores) Discuss specific feature (app stores) Assist others(product forums) Other please specify (all channels))

Q4[2]Q8[3]Q12[4]

Q5 9 13 RQ2 App store (Q5)Product forum (Q9)Social media (Q13)

What was your motivation(s) to write on this channel in the past(choose all that apply)(Show appreciation Show dissatisfaction Influence improvement Recommend Discourage others Connect or socialise about software No specific motivation Other please specify)

Q5[2]Q9[3]Q13[4]

Q6 All Product forums How have you used software product forums in the past(choose all that apply)(I havenrsquot Reading and viewing Written posts)

-

Q10 All Social media Have you used social media (Eg Twitter Facebook) to discuss software products you are using(choose all that apply)(I havenrsquot Reading and viewing Written posts)

-

Q14 RQ2 App storeProduct forumSocial media

How likely do you think it is for an appsoftware product to change based on your online reviews(Definitely will Probably will Might or might not Probably wonrsquot Definitely wonrsquot)

[15]

Q15 RQ3 Software usage What type of mobile phone do you currently use(choose all that apply)(iPhone Android (Eg Samsung Pixel) I donrsquot use a mobile phone Other please specify)

-

Q16 RQ3 Software usage What type of computer do you currently use(choose all that apply)(Windows Mac (Apple) Linux I donrsquot use a computer Other please specify)

-

Q17 RQ3 Software usage How many hours per day do you use your phone(Less than 1 hour 1-4 hours 4-8 hours More than 8 hours)

-

Q18 RQ3 Software usage How many hours per day do you use your computer(Less than 1 hour 1-4 hours 4-8 hours More than 8 hours)

-

Q19 RQ1 Demographics Do you work or have you previously worked in the software industry(No I work or have worked in software Other please specify)

-

Q20 RQ1 Demographics How old are you(Under 18 years old 18-24 years old 25-34 years old 35-44 years old 45-54 years old Over55 years old)

[14]

Q21 RQ1 Demographics What is your gender(Man Woman Prefer not to say Prefer to self-specify (please specify))

[14]

Q22 RQ1 Demographics What is your ethnicity(White (European) Asian Pacific people African Middle Eastern Latin American Otherplease specify)

[14]

Q23 RQ1 Demographics What is your highest level of education completed(Secondary school Post secondary Vocational training 1-2 year tertiary education Bachelordegree (3-4 years) Master degree (postgraduate) Doctoral (postgraduate) Other please specify)

[16]

Q24 RQ1 Demographics What is your current employment status(Employed full-time (gt 40 hours) Employed part-time (lt 40 hours) Currently unemployed Student Retired Self-employed Unable to work Other please specify)

[14]

TABLE IIRESPONDENT DEMOGRAPHICS

DemographicType Group Number of

RespondentsPercentage ofRespondents

Gender Men 571 5490

Women 454 4365

Other 16 154

Age Under 18 years 61 587

18 - 24 years 571 5490

25 - 34 years 285 2740

35 - 44 years 50 481

45 - 54 years 29 279

Over 55 44 423

Ethnicity WhiteEuropean 790 7596

Asian 149 1433

Middle Eastern 26 250

Latin American 24 231

Pacific and Maori 18 173

African 7 067

Other 27 260

Education Secondary school 411 3952

Vocational Training 14 135

1-2 year Tertiary 62 592

Bachelor degree 390 3750

Master degree 129 1240

Doctoral degree 25 240

Other 9 087

Employment Full time (gt 40 hours) 215 2067

Part time (lt 40 hours) 78 750

Student 644 6192

Self-employed 28 269

Currently unemployed 39 375

Retired 15 144

Unable to work 4 038

Other 18 173

C Recruiting Participants

To recruit participants convenience sampling was used[17] This was chosen as the best method to engage a goodnumber of survey participants in a reasonable time periodThe possible sources of bias from our sampling method-ology are discussed in section V-A Survey participationwas advertised via several avenues (described below) andincentivised with the chance to win a $200e120 cash prizeThe survey was primarily made available online throughthe Qualtics survey platform [18]

A link to the Qualtics survey was shared on Facebookand Twitter by the authors (and their colleagues) Inaddition we recruited from a pool of university participantsusing the hroot software [19] The pool includes nearly3500 participants who registered online to be invited toand participate in scientific studies either on-site or onlineThis pool is mainly advertised at the Karlsruhe Institute ofTechnology so the pool contains primarily students betweenthe ages of 18 and 30 Through hroot 2570 participantswere invited Hardcopies of the survey were also distributedin public areas of Auckland city during December 2019The completed hardcopy survey responses were manuallyconsolidated with the online survey responses The surveywas open to anyone 16 years or older

TABLE IIIUSER FEEDBACK WITH AGE SIGNIFICANCE TESTS

App Store Product Forums Social MediaChi2 p Chi2 p Chi2 p

Under 18 lt 18 - 24 496 0026 579 0016 0017 0896Under 18 lt 25 - 34 3600 0058 11419 0001 0165 0685Under 18 lt 35 - 44 11760 0001 17087 lt 0001 8264 000418 - 24 lt 25 - 34 027 0603 904 0003 0936 033318 - 24 lt 35 - 44 5603 0018 12183 lt 0001 26509 lt 000125 - 34 lt 35 - 44 6570 001 2169 0141 14214 lt 000135 - 44 gt 45 - 54 0997 0318 0049 0825 190 016845 - 54 gt Over 55 0571 0450 0437 0508 0010 091935 - 44 gt Over 55 5487 0019 0770 0380 4815 0028

Note statistically significant results are bolded

D Survey Participants

Across all collection channels 1040 participants fullycompleted the survey All respondents reported having usedsoftware on computer or mobile therefore all respondentsare software users The make up of the survey respondentswith respect to gender age ethnicity education andemployment are shown in Table II

Regarding the highest level of education obtainedwe noticed that many respondents reported secondaryschool (411) and bachelor degree (390) Given the hrootsoftware recruited from a pool of university participants wesuspected education level could be associated with the ageof the participants We saw that 9002 of secondary schooleducated reported to be under 25 compared to only 4161of those who have higher education After controlling forage we did not see any significant differences in feedbackhabits in regard to education level Thus we do not reportresults considering education level

E Survey Analysis

To answer our research questions we analysed the ratioof respondents in each user group (based on demographicsor software usage) that reported a particular behaviour eggiving feedback on a particular feedback channel or havinga certain motivation Chi-squared tests which tests fordifferences in proportion between two groups [20] wereused to find if differences in reported behaviours betweenuser groups are statistically significant

IV RESULTS

A Demographics

RQ1 What are the demographics of software users whoreport to give online written feedback

In this section we present the percentage of writtenfeedback givers in each demographic group

Feedback across online channels Overall 3096of survey respondents reported having written feedbackon any of the three online channels The most surveyrespondents reported having written feedback on app stores(1816) then on product forums (1345) and least onsocial media (711) The majority of feedback givingrespondents gave feedback to only one channel (7764)1957 had written on two channels with 280 writingon all three (Fig 1) A Chi-squared test showed the higherrate of respondents using only one feedback channel overmultiple channels is statistically significant (plt0001)

Age Under 18rsquos reported to have given the leastfeedback of all ages across all channels (app store 66

Fig 1 Overview of the proportion of feedback givers who use each online channel

Fig 2 User feedback with age

TABLE IVCOMPARING APP STORE AND FORUM FEEDBACK WITH AGE

App Store () Product Forums ()Under 25 years old 1772 93425 years old and over 1887 1985

TABLE VUSER FEEDBACK WITH GENDER

Number ofRespondents

App Store()

Product Forums()

Social Media(amp)

Men 571 2032 1804 823Women 454 1454 815 573

forums 00 social 49) (Fig 2) Conversely 35-45year oldrsquos (50 respondents) reported to give the mostfeedback across all channels (app store 340 forums280 social 260) Chi-squared tests show there arestatistically significant differences between ages (shown inTable III)

Also of note respondents under 25 preferred to givefeedback to the app store over product forums shown inbold in table IV Under 25rsquos preference for app stores wasshown to be statistically significant using a chi-squaredtest (p lt 001) Respondents 25 and over used app storesand forums more equally with those over 44 reportingmore forum use However the differences in channel usefor those 25 and above was not found to be significant

Gender Men reported to give more feedback thanwoman across all channels shown in table V On apps

TABLE VIUSER FEEDBACK TYPE WITH GENDER

App Store () Forums () Social Media ()

FeedbackType

Men Women Men Women Men Women

Praise 4138 5000 2039 1081 3830 3077

Reportbug

4052 4848 7379 5676 4681 4231

Requestfeature

2672 1818 3204 2162 2766 3846

Askquestion

259 606 8835 9459 6809 6538

Recommendto others

1296 1667 NA 3617 1923

Dissuadeothers

1034 606 NA 851 1154

Discussshortcomings

4741 3636 NA 4681 3462

Dispraise orcriticise

1810 1515 1650 811 NA

Helpfulsituation

3621 2727 NA NA

Discussfeature

2155 2273 NA NA

Assistothers

NA 5534 2162 NA

stores 203 of men and 145 of women reported togive feedback On product forums 180 of men and81 of women reported to give feedback On socialmedia the difference was the smallest with 82 and57 respectively reporting to give feedback Chi-squaredtests showed that the difference between men and womenrespondents was statistically significant for app stores(p=002) and product forums (plt0001)

Men and women respondents reported some differencesin the types of feedback they give on all three feedbackchannels shown in Table VI Differences of note areindicated in bold in the table On app stores more womenfeedback givers reported praising apps than feedbackgiving men (w 50 4138) and also reported givingbug reports More men reported describing a situation anapp was helpful reported an app short coming and requesta new features

On product forums both genders were very likely to ask

TABLE VIIUSER FEEDBACK WITH EMPLOYMENT TYPE

Number ofRespondents

Forums()

Under25 years

()Full-time 215 2140 2093

Part-time 78 1282 6667

Student 644 1087 7857

Full-time (no under 25rsquos) 170 2294 000

Part-time (no under 25rsquos) 26 1154 000

Students (no under 25rsquos) 138 1884 000

TABLE VIIIFEEDBACK OF SOFTWARE PROFESSIONALS

Number ofRespondents

App Store()

Forums()

SocialMedia

()Software Professionals 171 2749 1988 1287

Other Respondents 869 1632 1218 598

a question about software with 8835 of men feedbackgivers and 9459 of women Men feedback givers weremore likely to give other types of feedback includingreport a problem request a feature give praise givecriticism and assist others On social media more menreported recommending software to others and discussingshort comings More women reported requesting newfeatures

Employment Respondents working full time reportedusing product forums at a higher rate than those workingpart time and students (Table VII) However there isa strong association between employment level and ageas 7857 of students are also under 25 In the bottomhalf of table VII all under 25 year old respondents wereremoved from the analysis showing the difference betweenemployment levels is not as large when consideringonly older respondents The feedback differences betweenemployment groups were not found to be statisticallysignificant using chi-squared tests after the exclusion ofthe under 25 year old respondents

Software professionals Respondents who work or haveworked in software (software professionals) reported tohave given feedback at a higher rate than those who havenot worked in software on all channels (Table VIII) Chi-squared tests showed that the feedback rate differencebetween software professionals and other respondents wassignificant on all channels (app stores p=0001 productforums p=001 social media p=0002)

Ethnicity The majority of survey respondents wereeither Caucasian (790) or Asian (149) which limited ourfindings with respect to ethnicity However the ethnicdemographics of the respondents are representative for astudy based in New Zealand and Germany Only the dif-ference between Caucasian and Asian feedback rate couldbe investigated and this difference was not statisticallysignificant on any channel

Fig 3 Impact of in app prompts

Answer to RQ1 There are statistically significantdifferences in the amount of written feedbackgiven by software users with respect to traditionaldemographics For gender men reported givingmore feedback than women on all three feedbackchannels The types of feedback men and womenreported to give also varied in notable ways Withage distinct patterns emerged with respondentsbetween 35 and 45 reporting to give the most feed-back and under 18rsquos reporting to give the least onall channels Additionally software professionalsreported giving significantly more feedback thanother respondents

B Motivations

RQ2 What motivates software users to give onlinefeedback and are there differences across demographics

The section presents our findings with respect to whatmotivates users to give online feedback The difference inmotivations across the three channels and between groupsare given

Overall The reported motivations to give feedback onapp stores product forums and social media are given intable IX The motivations are given as a percentage of allusers who give written feedback on each channel Multiplemotivations could be given by each respondent As canbe seen the motivations vary across feedback channelsShow appreciation for software was the most commoncited motivation on app stores (6515) and Social media(5676) Get help with software was the top motivation topost on product forums (7037) Influencing improvementwas also a prominent motivation being the third most citedon all channels

Mobile app prompts (Fig 3) 5245 of all surveyrespondents reported having previously given a star ratingto an app Of those who have given a star rating 6575only gave the rating when prompted within the app neverdirectly on the app store 1816 of respondents reportedhaving given a written review to an app Of those whohave given a written review 3175 only gave a writtenreview when prompted to within the app

Gender Some differences in motivations to give feed-back were reported between men and women The per-centage of men and woman feedback givers who cited

TABLE IXMOTIVATIONS TO GIVE FEEDBACK

App Store () Product Forum () Social Media ()

1 Show appreciation 6515 1 Get help 7037 1 Show appreciation 5676

2 Influence improvement 5202 2 Influence improvement 4429 2 Influence improvement 5135

3 Show dissatisfaction 3485 3 Show appreciation 2643 3 Show dissatisfaction 3784

4 Recommendto others 2980 4 Recommend to others 1786 4 Connect or socialise 3514

5 Discourage others 1263 5 Show dissatisfaction 1643 5 Recommend to others 3243

6 Get help 920 6 Connect or socialise 1572 6 Get help 2273

7 No specific motivation 505 7 No specific motivation 786 7 Discourage others 1486

8 Connect or socialise 152 8 Discourage others 357 8 No specific motivation 811

TABLE XMOTIVATIONS TO GIVE FEEDBACK WITH GENDER

App Store () Product Forums () Social Media ()

Motivation Men Women Men Women Men Women

Showappreciation

6724 7273 2816 1892 5745 5385

Showdissatisfaction

3621 3636 1359 2162 3191 4615

Influenceimprovement

5776 5000 4951 2703 5532 4231

Recommend 2979 3462 3276 3030 1845 1351

Discourage 1638 606 194 541 1064 1923

Connectsocialise

431 758 1359 1892 2766 4231

Get help 1014 588 7111 7778 4118 000No specificmotivation

086 303 971 000 426 1154

Fig 4 Feedback given by individual users each year on each channel

each motivation are shown in table X where notabledifferences are indicated in bold On app stores men weremore motivated to discourage others from using a dislikedapp On product forums more men cited influencing animprovement in the software as a motivation On socialmedia more women were motivated to show dissatisfactionand connect or socialise about a software product Also onsocial media more men cited influence improvement andget help These results are bolded in Table X

Feedback frequency The majority of feedback giversreported having given feedback between 0 and 4 times inthe last year across all channels (Fig 4) App stores hadthe least respondents reporting to give more than 4 piecesof feedback product forums had the most respondentsgiving feedback more than 4 times

Perception of influencing developers Survey re-spondents who believed that software developers woulddefinitely not be influenced by online feedback were lesslikely to give feedback than those who believed influence

TABLE XIUSER FEEDBACK WITH PERCEPTION OF INFLUENCING DEVELOPERS

Number ofRespondents

App Store()

Forums()

Socialmedia()

Definitely will 83 1446 1807 723

Probably will 265 1925 1396 792

Might or might not 416 1875 1490 649

Probably will not 248 1895 968 806

Definitely will not 27 370 741 000

was more likely on all channels However chi-squared testsshowed these differences werenrsquot statistically significantFeedback rates with perception of influencing developersare shown in table XI the lower definitely not values areindicated in bold

Answer to RQ2 Showing appreciation was the topmotivation given to write feedback on app storesand social media On product forums getting helpwas the most commonly cited motivation (table IX)Differences in the motivations of men and womento give written feedback on each channel were alsoreportedIn-app prompts were reported to be very effectiveat motivating app users to give star ratings but lesseffective at eliciting written feedback Individualsurvey respondents reported engaging with eachfeedback channel at different frequencies writingon product forums the most times a year and leaston app stores

C Type of software and duration of use

RQ3 Does the likelihood of giving online writtenfeedback vary based on the type of software used andthe duration of software usage

iPhoneAndroid Android users reported giving feed-back to the app store at a higher rate than iPhone users(Table XII) 1348 of iPhone users reported having givenwritten feedback on app stores compared to 2184 ofAndroid users A chi-squared test showed this difference tobe statistically significant given in table XIII The higherfeedback rate of Android users on app stores has beenindicate in bold in table XII

WindowsMacLinux Linux users reported giving writ-ten feedback on app stores and product forums at a higher

TABLE XIIUSER FEEDBACK WITH DEVICE TYPE

Device Number ofRespondents

App Store()

ProductForum

()

SocialMedia

()Android 618 2184 1375 680

iPhone 423 1348 1277 733

Linux 94 3191 2660 1064

Windows 759 1910 146 646

Mac 275 1673 1236 1018

TABLE XIIIUSER FEEDBACK WITH DEVICE TYPE SIGNIFICANCE TESTS

App Store Product Forums

Chi2 p Chi2 p

Android gt iPhone 11144 0001 0087 0769

Linux gt Windows 7651 0006 8073 0004Linux gt Mac 8974 0003 9531 0002

Statistically significant results are bolded

TABLE XIVUSER FEEDBACK WITH DAILY COMPUTER USE

Daily Computer Use Number ofRespondents

App Store()

Forums()

SocialMedia

()Less than 1 hour 109 1835 1009 642

1 - 4 hours 436 1514 940 596

4 - 8 hours 363 2066 1708 799

More than 8 hours 110 2182 2364 909

rate than Windows and Mac users (Table XII) Chi-squaredtests showed these differences to both be statisticallysignificant (Table XIII) The difference between Windowsand Mac users feedback was not statistically significantThe statistically significant higher feedback rates of Linuxusers are indicated in bold in table XII

Hours of computer use Respondents who reported ahigher daily computer use (hours) were more likely to givefeedback to product forums indicated in bold in table XIVThe least forum feedback was given by respondents usingtheir computer less then 1 hour or between 1 and 4 hoursa day Those using their computer between 4 and 8 hoursgave more feedback and those using their computer morethan 8 hours a day gave at the highest rate Chi-squaredtests showed that the feedback rate differences between 1 -4 hours and 4 - 8 hours and between 1- 4 hours and over8 hours were statistically significant (Table XV)

Hours of phone use Respondents who reported ahigher daily phone use (hours) were more likely togive feedback to social media indicated in bold in tableXVI However chi-squared tests showed these differenceswerenrsquot statistically significant

TABLE XVCOMPUTER DAILY USE SIGNIFICANCE TESTS (PRODUCT FORUMS)

Daily Computer Use Chi2 p

Less than 1 hour lt 1-4 hours 0001 0971

Less than 1 hour lt 4-8 hours 2619 0106

1 - 4 lt Over 8 hours 15233 lt 0001Less than 1 lt Over 8 hours 6221 00131-4 hours lt 4-8 hours 9722 00024-8 hours lt Over 8 hours 1983 0159

Statistically significant results are bolded

TABLE XVIUSER FEEDBACK WITH DAILY PHONE USE

Daily Phone Use Number ofRespondents

App Store()

Forums()

SocialMedia

()Less than 1 hour 52 1538 1538 3851 - 4 hours 664 1657 1370 6634 - 8 hours 266 2293 1165 789More than 8 hours 51 1765 1765 1373

Answer to RQ3 Statistically significant differ-ences were reported in the amount of writtenfeedback given based on the type of software usedand the duration of daily use Respondents whospend more hours each day on their computerreported giving more written feedback to productforums Those using the Linux OS gave morewritten feedback to app stores and product forumsthan those using Windows and Mac Android usersreported giving more written feedback to app storesthan iPhone users

V DISCUSSION

In this section we first discuss the threats to validity ofour study and then describe the implications and potentialavenues for future work

A Threats to validity

Convenience sampling used in this study to elicit surveyparticipants is a non-probabilistic sampling method anda possible source of bias [17] The target population ofthis study are users of software and mobile applicationsSurvey participants were engaged via Facebook Twitterthe Karlsruhe Institute of Technology survey pool and inAuckland cities public areas therefore only a subset ofthe target population had the opportunity to participateAdditionally all respondents who completed the surveywere self-selected and their feedback habits may notgeneralise to all software users

To mitigate this bias we collected data from a largenumber of software users (1040 participants) Recruitmentwas done through three channels social media publicspaces and the hroot survey pool to increase the oddsof recruiting a diverse set of respondents However wecannot claim that our results generalise outside of oursample Future studies can replicate our survey to validateour findings

Our participants are not representative across all demo-graphics The demographics of our respondents are listedin Table II The majority of participants are whiteEuropeanand many are students When presenting our resultswe present proportions based on the total number ofrespondents in each demographic group We also used chi-squared tests to determine significance between differentdemographic groups which accounts for the sample size ofeach population being compared Therefore findings foundto be statistically significant had a sufficient number of re-spondents in each demographic to satisfy the test Howeverdue to a low number of participants in some demographicgroups not all demographics could be analysed Futurestudies should replicate this survey to enable analysis ofadditional demographics

In addition the majority of participants identified asmen or women We did give participants the option toself-specify gender but very few participants chose toself-specify Thus our analysis was limited to only thedifferences between participants who identified as men andwomen Again future studies should replicate the surveyto enable analysis beyond these binary genders

Our results are based only on self-reported feedbackhabits Demographic information of feedback givers is notreadily available on the feedback channels we investigated(often the real name of the writer isnrsquot even given) Thisdata sparsity problem means our findings canrsquot be directlyvalidated against actual feedback data One previous studyby Guzman et al [5] approximated the gender of feedbackgivers on app stores from their usernames Using theseapproximations they found that men were more likely thanwomen to provide feedback on the Apple app store whichis in line with what our respondents reported and supportsour findings

B Implications and Future Work

Implication 1 The findings presented in this papersuggest that when leveraging online user feedback to getthe most representative user views and desires feedbackfrom multiple feedback channels should be consideredWe found statistically significant differences in the userswho reported to give feedback on app stores productforums and social media with respect to traditionaldemographics and software usage habits For exampleolder respondents prefer product forums to app storeswhile younger respondents prefer app stores

Importantly a majority of feedback giving respondentsreported only engaging with one of these three feedbackchannels This indicates that considering multiple channelswill enable feedback from a more diverse set of users

We also found key differences in what motivates softwareusers to engage with each of the three channels The mostcited motivation on app stores and social media was to showappreciation for the appsoftware Whereas on productforums showing appreciation was much less of a motivatingfactor instead getting help was the top cited motivationShowing dissatisfaction recommending and discouragingothers were also significantly more cited on app stores andsocial media On social media connecting with other userswas reported to be a more common motivation than on theother channels

These motivation differences suggest that the feedbackon each online channel is likely to contain different productdevelopment insights For example feedback on productforums contain users trying to get help and therefore likelydescribes ways the software is unintuitive or difficult to useOn app stores and social media users are more motivatedto communicate how they feel about the softwareapp tothe developers and other users These differences againemphasis the benefit to considering feedback from allchannels as each channel may provide unique insights

Implication 2 How to elicit feedback from under-represented groups needs further study We saw somedemographics were less likely to give feedback than othersFor example respondents 35-44 years old report to providethe most feedback on all three feedback channels whileboth older and younger respondents gave less feedback(Fig 2) Also men reported giving feedback at a higher ratethan women across all three channels This is in line withthe results of Guzman et al which found that the Apple appstore had more feedback from men [5] Future work shouldinvestigate why some user groups give more feedback thanothers to understand how to better elicit feedback fromthese underrepresented groups This work could surveyunderrepresented user groups on why they donrsquot givefeedback so that these factors can be directly addressedChanges to the feedback channels themselves may evenbe considered to make the tool more inclusive Recentresearch found that most software has gender inclusivityissues [21] so it is possible that similar inclusivity issuesexist in the software that collects online feedback

Additionally studies could be performed to see ifother means of collecting feedback that augments writtenfeedback on the feedback channels we studied could beused to help provide a more comprehensive set of feedbackacross demographics For example recent research pro-posed using a conversational agent (ladderbot) to conductshort requirements interviews to elicit user feedback [22]Future research could evaluate whether some demographicswould prefer other user feedback collection methods (likeladderbot)

Implication 3 Feedback prompts are effective at elicit-ing feedback for app stores and may be effective if appliedmore widely in computer software However more workis needed to understand how to prompt users to givedetailed feedback Mobile apps widely use prompts to elicitfeedback More survey respondents reported giving writtenfeedback on app stores than on any other channel Much ofthis feedback is prompted The number of respondents whohave provided unprompted app store feedback (1239) isvery similar to the number who report to have written postson product forums (1345) This suggests that the promptsare successful in eliciting additional feedback givers Theprompts are even more effective at eliciting app ratingswhich take less time to provide than written feedbackFuture research could study whether prompts could besuccessful in collecting other types of user feedback andhow they could be integrated into other feedback channels

Implication 4 The types of software devices respondentsuse also has an association to feedback habits Investigatingwhy users of some devices give more feedback may giveinsights into how to motivate and facilitate feedback

On phones more Android users give written feedbackthan iPhone users It is not clear why there are differencesin feedback across devices but it may be influenced bydifferences in prompt rates app quality app store usabilityor even those who choose to use each phone type iOSdevelopers could benefit in understanding these factors inorder to encourage more feedback from their users

On computers respondents who use the Linux OS morecommonly had given written feedback to app stores andproduct forums than those who do not use Linux Thefeedback habits of Mac and Windows users were relativelysimilar across all feedback channels The higher feedbackrates of Linux users may be related to the prevalence ofsoftware developers using it In fact 43 of respondentsusing Linux also reported working in the software industrycompared to only 16 of all respondents Our resultsshowed that software professionals are more likely toprovide online feedback possibly because they understandhow that feedback will be used by development teamsFuture research can investigate more thoroughly the reasonsfor differences across devices

Other avenues for future work Investigate otherfeedback channels Our study was limited to app storesproduct forums and social media Future work couldperform a similar investigation considering other feedbackchannels like issue trackers

Replicate survey in other countries Our survey re-spondents were mostly from two countries New Zealandand Germany Future work could replicate our survey byeliciting responses in additional countries This would alsoenable analysis at the ethnicity level if more ethnic diversityin the participants was achieved

Understand gender differences in product forum engage-ment In addition to men being more likely than women topost on product forums men also reported using productsforums for different reasons While men and women bothprimarily used forums to ask software related questionsmen also reported higher rates of giving other types offeedback on product forums including reporting problemsrequesting features praising and criticising the softwareas well as assisting others Further research is neededto understand the gender difference in engagement withproduct forums

Making missing demographics more transparent Cur-rently it is difficult for product development teams toknow whether the feedback collected from online feed-back channels is biased and misses the voices of someunderrepresented groups Future research could deviseways to make this more transparent to enable softwaredevelopment teams to more proactively consider the needsof the underrepresented groups and produce more inclusivesoftware

VI CONCLUSION

The online user feedback written on app stores productforums and social media is a valuable source of require-ments for software developers and has been a focus ofrequirements engineering researchers However limitedstudies have been done to understand which softwareusers give this feedback and what motivates them In thiswork we directly surveyed 1040 software users about

their feedback habits software use and demographicinformation

The responses indicate significant differences in thedemographics of software users who give feedback oneach online channel For gender men reported giving morefeedback than women and with age respondents between35 and 45 reporting to give the most feedback acrossall channels We also found strong evidence that youngersoftware users (under 25) prefer to engage with app storeswhereas older software users use product forums at equal(to app stores) and sometimes higher rates

We identified key differences in what motivates softwareusers to engage with each of the three channels Comparingchannels respondents reported the top motivation to givefeedback on app stores and social media was to showappreciation whereas on forums the most cited motivationwas to get help with software products Differences betweenthe motivations of men and women to give feedbackwere also reported for each of the channels Respondentsreported in app prompts to be significantly more effective inmotivating them to give app ratings over written feedbackAdditionally individual feedback givers reported to engagemore times a year on product forums than on with appstores

Differences in feedback habits were also reported withthe ways respondents use software Those who spendmore hours each day on their phone or computer reportedgiving more feedback about the software theyrsquore using Thesoftware platform being used also presented a relationshipto feedback rates with more Linux (computer) and Android(phone) users reporting to give feedback than those whouse the alternatives

The findings presented in this paper give meaningfulinsights into which software users give online feedbackand the motivations they have to give it We foundnotable differences in those who give feedback to eachonline channel which emphasises the need to mine allthree feedback channels to get the most representativerequirements from software users when leveraging onlinefeedback

REFERENCES

[1] E Guzman R Alkadhi and N Seyff ldquoA needle in a haystackWhat do twitter users say about softwarerdquo in 2016 IEEE 24thInternational Requirements Engineering Conference (RE) Sep2016 pp 96ndash105

[2] D Pagano and W Maalej ldquoUser feedback in the appstore Anempirical studyrdquo in 2013 21st IEEE International RequirementsEngineering Conference (RE) July 2013 pp 125ndash134

[3] J Tizard H Wang L Yohannes and K Blincoe ldquoCan a con-versation paint a picture mining requirements in software fo-rumsrdquo in 2019 IEEE 27th International Requirements EngineeringConference (RE) IEEE 2019 pp 17ndash27

[4] E Guzman R Alkadhi and N Seyff ldquoAn exploratory studyof twitter messages about software applicationsrdquo RequirementsEngineering 07 2017

[5] E Guzman and A P Rojas ldquoGender and user feedback Anexploratory studyrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 381ndash385

[6] E Guzman M Ibrahim and M Glinz ldquoA little bird told me miningtweets for requirements and software evolutionrdquo in 2017 IEEE 25thInternational Requirements Engineering Conference (RE) IEEE2017 pp 11ndash20

[7] W Maalej and H Nabil ldquoBug report feature request orsimply praise on automatically classifying app reviewsrdquo in 2015IEEE 23rd International Requirements Engineering Conference

(RE) vol 00 Aug 2015 pp 116ndash125 [Online] Availabledoiieeecomputersocietyorg101109RE20157320414

[8] A D Sorbo S Panichella C V Alexandru C A Visaggio andG Canfora ldquoSurf Summarizer of user reviews feedbackrdquo in 2017IEEEACM 39th International Conference on Software EngineeringCompanion (ICSE-C) May 2017 pp 55ndash58

[9] E Guzman L Oliveira Y Steiner L C Wagner and M GlinzldquoUser feedback in the app store a cross-cultural studyrdquo in 2018IEEEACM 40th International Conference on Software EngineeringSoftware Engineering in Society (ICSE-SEIS) IEEE 2018 pp13ndash22

[10] E C Groen N Seyff R Ali F Dalpiaz J Doerr E GuzmanM Hosseini J Marco M Oriol A Perini et al ldquoThe crowd inrequirements engineering The landscape and challengesrdquo IEEEsoftware vol 34 no 2 pp 44ndash52 2017

[11] S Panichella A Di Sorbo E Guzman C A Visaggio G Canforaand H C Gall ldquoArdoc App reviews development orientedclassifierrdquo in Proceedings of the 2016 24th ACM SIGSOFTInternational Symposium on Foundations of Software Engineeringser FSE 2016 New York NY USA ACM 2016 pp 1023ndash1027[Online] Available httpdoiacmorg10114529502902983938

[12] N Chen J Lin S C H Hoi X Xiao and B ZhangldquoAr-miner Mining informative reviews for developers frommobile app marketplacerdquo in Proceedings of the 36th InternationalConference on Software Engineering ser ICSE 2014 NewYork NY USA ACM 2014 pp 767ndash778 [Online] Availablehttpdoiacmorg10114525682252568263

[13] N Papadopoulos O M Martiacuten M Cleveland and M Laroche

ldquoIdentity demographics and consumer behaviorsrdquo InternationalMarketing Review 2011

[14] ldquoNew zealand census 2018rdquo httpswwwstatsgovtnzinformation-releases2018-census-population-and-dwelling-countsaccessed December 2019

[15] R Likert ldquoA technique for the measurement of attitudesrdquo Archivesof psychology 1932

[16] U ISCED ldquoInternational standard classification of education 2011rdquo2012

[17] I Etikan ldquoComparison of convenience sampling and purposivesamplingrdquo American Journal of Theoretical and Applied Statisticsvol 5 p 1 01 2016

[18] ldquoQualtrics survey platformrdquo httpswwwqualtricscom accessedDecember 2019

[19] O Bock A Nicklisch and I Baetge ldquohroot Hamburg registrationand organization online toolrdquo in WiSo-HH Working Paper Series2012

[20] M L McHugh ldquoThe chi-square test of independencerdquo Biochemiamedica Biochemia medica vol 23 no 2 pp 143ndash149 2013

[21] M Burnett S Stumpf J Macbeth S Makri L BeckwithI Kwan A Peters and W Jernigan ldquoGendermag A methodfor evaluating softwarersquos gender inclusivenessrdquo Interacting withComputers vol 28 no 6 pp 760ndash787 2016

[22] T Rietz and A Maedche ldquoLadderbot A requirements self-elicitation systemrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 357ndash362

TABLE IIRESPONDENT DEMOGRAPHICS

DemographicType Group Number of

RespondentsPercentage ofRespondents

Gender Men 571 5490

Women 454 4365

Other 16 154

Age Under 18 years 61 587

18 - 24 years 571 5490

25 - 34 years 285 2740

35 - 44 years 50 481

45 - 54 years 29 279

Over 55 44 423

Ethnicity WhiteEuropean 790 7596

Asian 149 1433

Middle Eastern 26 250

Latin American 24 231

Pacific and Maori 18 173

African 7 067

Other 27 260

Education Secondary school 411 3952

Vocational Training 14 135

1-2 year Tertiary 62 592

Bachelor degree 390 3750

Master degree 129 1240

Doctoral degree 25 240

Other 9 087

Employment Full time (gt 40 hours) 215 2067

Part time (lt 40 hours) 78 750

Student 644 6192

Self-employed 28 269

Currently unemployed 39 375

Retired 15 144

Unable to work 4 038

Other 18 173

C Recruiting Participants

To recruit participants convenience sampling was used[17] This was chosen as the best method to engage a goodnumber of survey participants in a reasonable time periodThe possible sources of bias from our sampling method-ology are discussed in section V-A Survey participationwas advertised via several avenues (described below) andincentivised with the chance to win a $200e120 cash prizeThe survey was primarily made available online throughthe Qualtics survey platform [18]

A link to the Qualtics survey was shared on Facebookand Twitter by the authors (and their colleagues) Inaddition we recruited from a pool of university participantsusing the hroot software [19] The pool includes nearly3500 participants who registered online to be invited toand participate in scientific studies either on-site or onlineThis pool is mainly advertised at the Karlsruhe Institute ofTechnology so the pool contains primarily students betweenthe ages of 18 and 30 Through hroot 2570 participantswere invited Hardcopies of the survey were also distributedin public areas of Auckland city during December 2019The completed hardcopy survey responses were manuallyconsolidated with the online survey responses The surveywas open to anyone 16 years or older

TABLE IIIUSER FEEDBACK WITH AGE SIGNIFICANCE TESTS

App Store Product Forums Social MediaChi2 p Chi2 p Chi2 p

Under 18 lt 18 - 24 496 0026 579 0016 0017 0896Under 18 lt 25 - 34 3600 0058 11419 0001 0165 0685Under 18 lt 35 - 44 11760 0001 17087 lt 0001 8264 000418 - 24 lt 25 - 34 027 0603 904 0003 0936 033318 - 24 lt 35 - 44 5603 0018 12183 lt 0001 26509 lt 000125 - 34 lt 35 - 44 6570 001 2169 0141 14214 lt 000135 - 44 gt 45 - 54 0997 0318 0049 0825 190 016845 - 54 gt Over 55 0571 0450 0437 0508 0010 091935 - 44 gt Over 55 5487 0019 0770 0380 4815 0028

Note statistically significant results are bolded

D Survey Participants

Across all collection channels 1040 participants fullycompleted the survey All respondents reported having usedsoftware on computer or mobile therefore all respondentsare software users The make up of the survey respondentswith respect to gender age ethnicity education andemployment are shown in Table II

Regarding the highest level of education obtainedwe noticed that many respondents reported secondaryschool (411) and bachelor degree (390) Given the hrootsoftware recruited from a pool of university participants wesuspected education level could be associated with the ageof the participants We saw that 9002 of secondary schooleducated reported to be under 25 compared to only 4161of those who have higher education After controlling forage we did not see any significant differences in feedbackhabits in regard to education level Thus we do not reportresults considering education level

E Survey Analysis

To answer our research questions we analysed the ratioof respondents in each user group (based on demographicsor software usage) that reported a particular behaviour eggiving feedback on a particular feedback channel or havinga certain motivation Chi-squared tests which tests fordifferences in proportion between two groups [20] wereused to find if differences in reported behaviours betweenuser groups are statistically significant

IV RESULTS

A Demographics

RQ1 What are the demographics of software users whoreport to give online written feedback

In this section we present the percentage of writtenfeedback givers in each demographic group

Feedback across online channels Overall 3096of survey respondents reported having written feedbackon any of the three online channels The most surveyrespondents reported having written feedback on app stores(1816) then on product forums (1345) and least onsocial media (711) The majority of feedback givingrespondents gave feedback to only one channel (7764)1957 had written on two channels with 280 writingon all three (Fig 1) A Chi-squared test showed the higherrate of respondents using only one feedback channel overmultiple channels is statistically significant (plt0001)

Age Under 18rsquos reported to have given the leastfeedback of all ages across all channels (app store 66

Fig 1 Overview of the proportion of feedback givers who use each online channel

Fig 2 User feedback with age

TABLE IVCOMPARING APP STORE AND FORUM FEEDBACK WITH AGE

App Store () Product Forums ()Under 25 years old 1772 93425 years old and over 1887 1985

TABLE VUSER FEEDBACK WITH GENDER

Number ofRespondents

App Store()

Product Forums()

Social Media(amp)

Men 571 2032 1804 823Women 454 1454 815 573

forums 00 social 49) (Fig 2) Conversely 35-45year oldrsquos (50 respondents) reported to give the mostfeedback across all channels (app store 340 forums280 social 260) Chi-squared tests show there arestatistically significant differences between ages (shown inTable III)

Also of note respondents under 25 preferred to givefeedback to the app store over product forums shown inbold in table IV Under 25rsquos preference for app stores wasshown to be statistically significant using a chi-squaredtest (p lt 001) Respondents 25 and over used app storesand forums more equally with those over 44 reportingmore forum use However the differences in channel usefor those 25 and above was not found to be significant

Gender Men reported to give more feedback thanwoman across all channels shown in table V On apps

TABLE VIUSER FEEDBACK TYPE WITH GENDER

App Store () Forums () Social Media ()

FeedbackType

Men Women Men Women Men Women

Praise 4138 5000 2039 1081 3830 3077

Reportbug

4052 4848 7379 5676 4681 4231

Requestfeature

2672 1818 3204 2162 2766 3846

Askquestion

259 606 8835 9459 6809 6538

Recommendto others

1296 1667 NA 3617 1923

Dissuadeothers

1034 606 NA 851 1154

Discussshortcomings

4741 3636 NA 4681 3462

Dispraise orcriticise

1810 1515 1650 811 NA

Helpfulsituation

3621 2727 NA NA

Discussfeature

2155 2273 NA NA

Assistothers

NA 5534 2162 NA

stores 203 of men and 145 of women reported togive feedback On product forums 180 of men and81 of women reported to give feedback On socialmedia the difference was the smallest with 82 and57 respectively reporting to give feedback Chi-squaredtests showed that the difference between men and womenrespondents was statistically significant for app stores(p=002) and product forums (plt0001)

Men and women respondents reported some differencesin the types of feedback they give on all three feedbackchannels shown in Table VI Differences of note areindicated in bold in the table On app stores more womenfeedback givers reported praising apps than feedbackgiving men (w 50 4138) and also reported givingbug reports More men reported describing a situation anapp was helpful reported an app short coming and requesta new features

On product forums both genders were very likely to ask

TABLE VIIUSER FEEDBACK WITH EMPLOYMENT TYPE

Number ofRespondents

Forums()

Under25 years

()Full-time 215 2140 2093

Part-time 78 1282 6667

Student 644 1087 7857

Full-time (no under 25rsquos) 170 2294 000

Part-time (no under 25rsquos) 26 1154 000

Students (no under 25rsquos) 138 1884 000

TABLE VIIIFEEDBACK OF SOFTWARE PROFESSIONALS

Number ofRespondents

App Store()

Forums()

SocialMedia

()Software Professionals 171 2749 1988 1287

Other Respondents 869 1632 1218 598

a question about software with 8835 of men feedbackgivers and 9459 of women Men feedback givers weremore likely to give other types of feedback includingreport a problem request a feature give praise givecriticism and assist others On social media more menreported recommending software to others and discussingshort comings More women reported requesting newfeatures

Employment Respondents working full time reportedusing product forums at a higher rate than those workingpart time and students (Table VII) However there isa strong association between employment level and ageas 7857 of students are also under 25 In the bottomhalf of table VII all under 25 year old respondents wereremoved from the analysis showing the difference betweenemployment levels is not as large when consideringonly older respondents The feedback differences betweenemployment groups were not found to be statisticallysignificant using chi-squared tests after the exclusion ofthe under 25 year old respondents

Software professionals Respondents who work or haveworked in software (software professionals) reported tohave given feedback at a higher rate than those who havenot worked in software on all channels (Table VIII) Chi-squared tests showed that the feedback rate differencebetween software professionals and other respondents wassignificant on all channels (app stores p=0001 productforums p=001 social media p=0002)

Ethnicity The majority of survey respondents wereeither Caucasian (790) or Asian (149) which limited ourfindings with respect to ethnicity However the ethnicdemographics of the respondents are representative for astudy based in New Zealand and Germany Only the dif-ference between Caucasian and Asian feedback rate couldbe investigated and this difference was not statisticallysignificant on any channel

Fig 3 Impact of in app prompts

Answer to RQ1 There are statistically significantdifferences in the amount of written feedbackgiven by software users with respect to traditionaldemographics For gender men reported givingmore feedback than women on all three feedbackchannels The types of feedback men and womenreported to give also varied in notable ways Withage distinct patterns emerged with respondentsbetween 35 and 45 reporting to give the most feed-back and under 18rsquos reporting to give the least onall channels Additionally software professionalsreported giving significantly more feedback thanother respondents

B Motivations

RQ2 What motivates software users to give onlinefeedback and are there differences across demographics

The section presents our findings with respect to whatmotivates users to give online feedback The difference inmotivations across the three channels and between groupsare given

Overall The reported motivations to give feedback onapp stores product forums and social media are given intable IX The motivations are given as a percentage of allusers who give written feedback on each channel Multiplemotivations could be given by each respondent As canbe seen the motivations vary across feedback channelsShow appreciation for software was the most commoncited motivation on app stores (6515) and Social media(5676) Get help with software was the top motivation topost on product forums (7037) Influencing improvementwas also a prominent motivation being the third most citedon all channels

Mobile app prompts (Fig 3) 5245 of all surveyrespondents reported having previously given a star ratingto an app Of those who have given a star rating 6575only gave the rating when prompted within the app neverdirectly on the app store 1816 of respondents reportedhaving given a written review to an app Of those whohave given a written review 3175 only gave a writtenreview when prompted to within the app

Gender Some differences in motivations to give feed-back were reported between men and women The per-centage of men and woman feedback givers who cited

TABLE IXMOTIVATIONS TO GIVE FEEDBACK

App Store () Product Forum () Social Media ()

1 Show appreciation 6515 1 Get help 7037 1 Show appreciation 5676

2 Influence improvement 5202 2 Influence improvement 4429 2 Influence improvement 5135

3 Show dissatisfaction 3485 3 Show appreciation 2643 3 Show dissatisfaction 3784

4 Recommendto others 2980 4 Recommend to others 1786 4 Connect or socialise 3514

5 Discourage others 1263 5 Show dissatisfaction 1643 5 Recommend to others 3243

6 Get help 920 6 Connect or socialise 1572 6 Get help 2273

7 No specific motivation 505 7 No specific motivation 786 7 Discourage others 1486

8 Connect or socialise 152 8 Discourage others 357 8 No specific motivation 811

TABLE XMOTIVATIONS TO GIVE FEEDBACK WITH GENDER

App Store () Product Forums () Social Media ()

Motivation Men Women Men Women Men Women

Showappreciation

6724 7273 2816 1892 5745 5385

Showdissatisfaction

3621 3636 1359 2162 3191 4615

Influenceimprovement

5776 5000 4951 2703 5532 4231

Recommend 2979 3462 3276 3030 1845 1351

Discourage 1638 606 194 541 1064 1923

Connectsocialise

431 758 1359 1892 2766 4231

Get help 1014 588 7111 7778 4118 000No specificmotivation

086 303 971 000 426 1154

Fig 4 Feedback given by individual users each year on each channel

each motivation are shown in table X where notabledifferences are indicated in bold On app stores men weremore motivated to discourage others from using a dislikedapp On product forums more men cited influencing animprovement in the software as a motivation On socialmedia more women were motivated to show dissatisfactionand connect or socialise about a software product Also onsocial media more men cited influence improvement andget help These results are bolded in Table X

Feedback frequency The majority of feedback giversreported having given feedback between 0 and 4 times inthe last year across all channels (Fig 4) App stores hadthe least respondents reporting to give more than 4 piecesof feedback product forums had the most respondentsgiving feedback more than 4 times

Perception of influencing developers Survey re-spondents who believed that software developers woulddefinitely not be influenced by online feedback were lesslikely to give feedback than those who believed influence

TABLE XIUSER FEEDBACK WITH PERCEPTION OF INFLUENCING DEVELOPERS

Number ofRespondents

App Store()

Forums()

Socialmedia()

Definitely will 83 1446 1807 723

Probably will 265 1925 1396 792

Might or might not 416 1875 1490 649

Probably will not 248 1895 968 806

Definitely will not 27 370 741 000

was more likely on all channels However chi-squared testsshowed these differences werenrsquot statistically significantFeedback rates with perception of influencing developersare shown in table XI the lower definitely not values areindicated in bold

Answer to RQ2 Showing appreciation was the topmotivation given to write feedback on app storesand social media On product forums getting helpwas the most commonly cited motivation (table IX)Differences in the motivations of men and womento give written feedback on each channel were alsoreportedIn-app prompts were reported to be very effectiveat motivating app users to give star ratings but lesseffective at eliciting written feedback Individualsurvey respondents reported engaging with eachfeedback channel at different frequencies writingon product forums the most times a year and leaston app stores

C Type of software and duration of use

RQ3 Does the likelihood of giving online writtenfeedback vary based on the type of software used andthe duration of software usage

iPhoneAndroid Android users reported giving feed-back to the app store at a higher rate than iPhone users(Table XII) 1348 of iPhone users reported having givenwritten feedback on app stores compared to 2184 ofAndroid users A chi-squared test showed this difference tobe statistically significant given in table XIII The higherfeedback rate of Android users on app stores has beenindicate in bold in table XII

WindowsMacLinux Linux users reported giving writ-ten feedback on app stores and product forums at a higher

TABLE XIIUSER FEEDBACK WITH DEVICE TYPE

Device Number ofRespondents

App Store()

ProductForum

()

SocialMedia

()Android 618 2184 1375 680

iPhone 423 1348 1277 733

Linux 94 3191 2660 1064

Windows 759 1910 146 646

Mac 275 1673 1236 1018

TABLE XIIIUSER FEEDBACK WITH DEVICE TYPE SIGNIFICANCE TESTS

App Store Product Forums

Chi2 p Chi2 p

Android gt iPhone 11144 0001 0087 0769

Linux gt Windows 7651 0006 8073 0004Linux gt Mac 8974 0003 9531 0002

Statistically significant results are bolded

TABLE XIVUSER FEEDBACK WITH DAILY COMPUTER USE

Daily Computer Use Number ofRespondents

App Store()

Forums()

SocialMedia

()Less than 1 hour 109 1835 1009 642

1 - 4 hours 436 1514 940 596

4 - 8 hours 363 2066 1708 799

More than 8 hours 110 2182 2364 909

rate than Windows and Mac users (Table XII) Chi-squaredtests showed these differences to both be statisticallysignificant (Table XIII) The difference between Windowsand Mac users feedback was not statistically significantThe statistically significant higher feedback rates of Linuxusers are indicated in bold in table XII

Hours of computer use Respondents who reported ahigher daily computer use (hours) were more likely to givefeedback to product forums indicated in bold in table XIVThe least forum feedback was given by respondents usingtheir computer less then 1 hour or between 1 and 4 hoursa day Those using their computer between 4 and 8 hoursgave more feedback and those using their computer morethan 8 hours a day gave at the highest rate Chi-squaredtests showed that the feedback rate differences between 1 -4 hours and 4 - 8 hours and between 1- 4 hours and over8 hours were statistically significant (Table XV)

Hours of phone use Respondents who reported ahigher daily phone use (hours) were more likely togive feedback to social media indicated in bold in tableXVI However chi-squared tests showed these differenceswerenrsquot statistically significant

TABLE XVCOMPUTER DAILY USE SIGNIFICANCE TESTS (PRODUCT FORUMS)

Daily Computer Use Chi2 p

Less than 1 hour lt 1-4 hours 0001 0971

Less than 1 hour lt 4-8 hours 2619 0106

1 - 4 lt Over 8 hours 15233 lt 0001Less than 1 lt Over 8 hours 6221 00131-4 hours lt 4-8 hours 9722 00024-8 hours lt Over 8 hours 1983 0159

Statistically significant results are bolded

TABLE XVIUSER FEEDBACK WITH DAILY PHONE USE

Daily Phone Use Number ofRespondents

App Store()

Forums()

SocialMedia

()Less than 1 hour 52 1538 1538 3851 - 4 hours 664 1657 1370 6634 - 8 hours 266 2293 1165 789More than 8 hours 51 1765 1765 1373

Answer to RQ3 Statistically significant differ-ences were reported in the amount of writtenfeedback given based on the type of software usedand the duration of daily use Respondents whospend more hours each day on their computerreported giving more written feedback to productforums Those using the Linux OS gave morewritten feedback to app stores and product forumsthan those using Windows and Mac Android usersreported giving more written feedback to app storesthan iPhone users

V DISCUSSION

In this section we first discuss the threats to validity ofour study and then describe the implications and potentialavenues for future work

A Threats to validity

Convenience sampling used in this study to elicit surveyparticipants is a non-probabilistic sampling method anda possible source of bias [17] The target population ofthis study are users of software and mobile applicationsSurvey participants were engaged via Facebook Twitterthe Karlsruhe Institute of Technology survey pool and inAuckland cities public areas therefore only a subset ofthe target population had the opportunity to participateAdditionally all respondents who completed the surveywere self-selected and their feedback habits may notgeneralise to all software users

To mitigate this bias we collected data from a largenumber of software users (1040 participants) Recruitmentwas done through three channels social media publicspaces and the hroot survey pool to increase the oddsof recruiting a diverse set of respondents However wecannot claim that our results generalise outside of oursample Future studies can replicate our survey to validateour findings

Our participants are not representative across all demo-graphics The demographics of our respondents are listedin Table II The majority of participants are whiteEuropeanand many are students When presenting our resultswe present proportions based on the total number ofrespondents in each demographic group We also used chi-squared tests to determine significance between differentdemographic groups which accounts for the sample size ofeach population being compared Therefore findings foundto be statistically significant had a sufficient number of re-spondents in each demographic to satisfy the test Howeverdue to a low number of participants in some demographicgroups not all demographics could be analysed Futurestudies should replicate this survey to enable analysis ofadditional demographics

In addition the majority of participants identified asmen or women We did give participants the option toself-specify gender but very few participants chose toself-specify Thus our analysis was limited to only thedifferences between participants who identified as men andwomen Again future studies should replicate the surveyto enable analysis beyond these binary genders

Our results are based only on self-reported feedbackhabits Demographic information of feedback givers is notreadily available on the feedback channels we investigated(often the real name of the writer isnrsquot even given) Thisdata sparsity problem means our findings canrsquot be directlyvalidated against actual feedback data One previous studyby Guzman et al [5] approximated the gender of feedbackgivers on app stores from their usernames Using theseapproximations they found that men were more likely thanwomen to provide feedback on the Apple app store whichis in line with what our respondents reported and supportsour findings

B Implications and Future Work

Implication 1 The findings presented in this papersuggest that when leveraging online user feedback to getthe most representative user views and desires feedbackfrom multiple feedback channels should be consideredWe found statistically significant differences in the userswho reported to give feedback on app stores productforums and social media with respect to traditionaldemographics and software usage habits For exampleolder respondents prefer product forums to app storeswhile younger respondents prefer app stores

Importantly a majority of feedback giving respondentsreported only engaging with one of these three feedbackchannels This indicates that considering multiple channelswill enable feedback from a more diverse set of users

We also found key differences in what motivates softwareusers to engage with each of the three channels The mostcited motivation on app stores and social media was to showappreciation for the appsoftware Whereas on productforums showing appreciation was much less of a motivatingfactor instead getting help was the top cited motivationShowing dissatisfaction recommending and discouragingothers were also significantly more cited on app stores andsocial media On social media connecting with other userswas reported to be a more common motivation than on theother channels

These motivation differences suggest that the feedbackon each online channel is likely to contain different productdevelopment insights For example feedback on productforums contain users trying to get help and therefore likelydescribes ways the software is unintuitive or difficult to useOn app stores and social media users are more motivatedto communicate how they feel about the softwareapp tothe developers and other users These differences againemphasis the benefit to considering feedback from allchannels as each channel may provide unique insights

Implication 2 How to elicit feedback from under-represented groups needs further study We saw somedemographics were less likely to give feedback than othersFor example respondents 35-44 years old report to providethe most feedback on all three feedback channels whileboth older and younger respondents gave less feedback(Fig 2) Also men reported giving feedback at a higher ratethan women across all three channels This is in line withthe results of Guzman et al which found that the Apple appstore had more feedback from men [5] Future work shouldinvestigate why some user groups give more feedback thanothers to understand how to better elicit feedback fromthese underrepresented groups This work could surveyunderrepresented user groups on why they donrsquot givefeedback so that these factors can be directly addressedChanges to the feedback channels themselves may evenbe considered to make the tool more inclusive Recentresearch found that most software has gender inclusivityissues [21] so it is possible that similar inclusivity issuesexist in the software that collects online feedback

Additionally studies could be performed to see ifother means of collecting feedback that augments writtenfeedback on the feedback channels we studied could beused to help provide a more comprehensive set of feedbackacross demographics For example recent research pro-posed using a conversational agent (ladderbot) to conductshort requirements interviews to elicit user feedback [22]Future research could evaluate whether some demographicswould prefer other user feedback collection methods (likeladderbot)

Implication 3 Feedback prompts are effective at elicit-ing feedback for app stores and may be effective if appliedmore widely in computer software However more workis needed to understand how to prompt users to givedetailed feedback Mobile apps widely use prompts to elicitfeedback More survey respondents reported giving writtenfeedback on app stores than on any other channel Much ofthis feedback is prompted The number of respondents whohave provided unprompted app store feedback (1239) isvery similar to the number who report to have written postson product forums (1345) This suggests that the promptsare successful in eliciting additional feedback givers Theprompts are even more effective at eliciting app ratingswhich take less time to provide than written feedbackFuture research could study whether prompts could besuccessful in collecting other types of user feedback andhow they could be integrated into other feedback channels

Implication 4 The types of software devices respondentsuse also has an association to feedback habits Investigatingwhy users of some devices give more feedback may giveinsights into how to motivate and facilitate feedback

On phones more Android users give written feedbackthan iPhone users It is not clear why there are differencesin feedback across devices but it may be influenced bydifferences in prompt rates app quality app store usabilityor even those who choose to use each phone type iOSdevelopers could benefit in understanding these factors inorder to encourage more feedback from their users

On computers respondents who use the Linux OS morecommonly had given written feedback to app stores andproduct forums than those who do not use Linux Thefeedback habits of Mac and Windows users were relativelysimilar across all feedback channels The higher feedbackrates of Linux users may be related to the prevalence ofsoftware developers using it In fact 43 of respondentsusing Linux also reported working in the software industrycompared to only 16 of all respondents Our resultsshowed that software professionals are more likely toprovide online feedback possibly because they understandhow that feedback will be used by development teamsFuture research can investigate more thoroughly the reasonsfor differences across devices

Other avenues for future work Investigate otherfeedback channels Our study was limited to app storesproduct forums and social media Future work couldperform a similar investigation considering other feedbackchannels like issue trackers

Replicate survey in other countries Our survey re-spondents were mostly from two countries New Zealandand Germany Future work could replicate our survey byeliciting responses in additional countries This would alsoenable analysis at the ethnicity level if more ethnic diversityin the participants was achieved

Understand gender differences in product forum engage-ment In addition to men being more likely than women topost on product forums men also reported using productsforums for different reasons While men and women bothprimarily used forums to ask software related questionsmen also reported higher rates of giving other types offeedback on product forums including reporting problemsrequesting features praising and criticising the softwareas well as assisting others Further research is neededto understand the gender difference in engagement withproduct forums

Making missing demographics more transparent Cur-rently it is difficult for product development teams toknow whether the feedback collected from online feed-back channels is biased and misses the voices of someunderrepresented groups Future research could deviseways to make this more transparent to enable softwaredevelopment teams to more proactively consider the needsof the underrepresented groups and produce more inclusivesoftware

VI CONCLUSION

The online user feedback written on app stores productforums and social media is a valuable source of require-ments for software developers and has been a focus ofrequirements engineering researchers However limitedstudies have been done to understand which softwareusers give this feedback and what motivates them In thiswork we directly surveyed 1040 software users about

their feedback habits software use and demographicinformation

The responses indicate significant differences in thedemographics of software users who give feedback oneach online channel For gender men reported giving morefeedback than women and with age respondents between35 and 45 reporting to give the most feedback acrossall channels We also found strong evidence that youngersoftware users (under 25) prefer to engage with app storeswhereas older software users use product forums at equal(to app stores) and sometimes higher rates

We identified key differences in what motivates softwareusers to engage with each of the three channels Comparingchannels respondents reported the top motivation to givefeedback on app stores and social media was to showappreciation whereas on forums the most cited motivationwas to get help with software products Differences betweenthe motivations of men and women to give feedbackwere also reported for each of the channels Respondentsreported in app prompts to be significantly more effective inmotivating them to give app ratings over written feedbackAdditionally individual feedback givers reported to engagemore times a year on product forums than on with appstores

Differences in feedback habits were also reported withthe ways respondents use software Those who spendmore hours each day on their phone or computer reportedgiving more feedback about the software theyrsquore using Thesoftware platform being used also presented a relationshipto feedback rates with more Linux (computer) and Android(phone) users reporting to give feedback than those whouse the alternatives

The findings presented in this paper give meaningfulinsights into which software users give online feedbackand the motivations they have to give it We foundnotable differences in those who give feedback to eachonline channel which emphasises the need to mine allthree feedback channels to get the most representativerequirements from software users when leveraging onlinefeedback

REFERENCES

[1] E Guzman R Alkadhi and N Seyff ldquoA needle in a haystackWhat do twitter users say about softwarerdquo in 2016 IEEE 24thInternational Requirements Engineering Conference (RE) Sep2016 pp 96ndash105

[2] D Pagano and W Maalej ldquoUser feedback in the appstore Anempirical studyrdquo in 2013 21st IEEE International RequirementsEngineering Conference (RE) July 2013 pp 125ndash134

[3] J Tizard H Wang L Yohannes and K Blincoe ldquoCan a con-versation paint a picture mining requirements in software fo-rumsrdquo in 2019 IEEE 27th International Requirements EngineeringConference (RE) IEEE 2019 pp 17ndash27

[4] E Guzman R Alkadhi and N Seyff ldquoAn exploratory studyof twitter messages about software applicationsrdquo RequirementsEngineering 07 2017

[5] E Guzman and A P Rojas ldquoGender and user feedback Anexploratory studyrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 381ndash385

[6] E Guzman M Ibrahim and M Glinz ldquoA little bird told me miningtweets for requirements and software evolutionrdquo in 2017 IEEE 25thInternational Requirements Engineering Conference (RE) IEEE2017 pp 11ndash20

[7] W Maalej and H Nabil ldquoBug report feature request orsimply praise on automatically classifying app reviewsrdquo in 2015IEEE 23rd International Requirements Engineering Conference

(RE) vol 00 Aug 2015 pp 116ndash125 [Online] Availabledoiieeecomputersocietyorg101109RE20157320414

[8] A D Sorbo S Panichella C V Alexandru C A Visaggio andG Canfora ldquoSurf Summarizer of user reviews feedbackrdquo in 2017IEEEACM 39th International Conference on Software EngineeringCompanion (ICSE-C) May 2017 pp 55ndash58

[9] E Guzman L Oliveira Y Steiner L C Wagner and M GlinzldquoUser feedback in the app store a cross-cultural studyrdquo in 2018IEEEACM 40th International Conference on Software EngineeringSoftware Engineering in Society (ICSE-SEIS) IEEE 2018 pp13ndash22

[10] E C Groen N Seyff R Ali F Dalpiaz J Doerr E GuzmanM Hosseini J Marco M Oriol A Perini et al ldquoThe crowd inrequirements engineering The landscape and challengesrdquo IEEEsoftware vol 34 no 2 pp 44ndash52 2017

[11] S Panichella A Di Sorbo E Guzman C A Visaggio G Canforaand H C Gall ldquoArdoc App reviews development orientedclassifierrdquo in Proceedings of the 2016 24th ACM SIGSOFTInternational Symposium on Foundations of Software Engineeringser FSE 2016 New York NY USA ACM 2016 pp 1023ndash1027[Online] Available httpdoiacmorg10114529502902983938

[12] N Chen J Lin S C H Hoi X Xiao and B ZhangldquoAr-miner Mining informative reviews for developers frommobile app marketplacerdquo in Proceedings of the 36th InternationalConference on Software Engineering ser ICSE 2014 NewYork NY USA ACM 2014 pp 767ndash778 [Online] Availablehttpdoiacmorg10114525682252568263

[13] N Papadopoulos O M Martiacuten M Cleveland and M Laroche

ldquoIdentity demographics and consumer behaviorsrdquo InternationalMarketing Review 2011

[14] ldquoNew zealand census 2018rdquo httpswwwstatsgovtnzinformation-releases2018-census-population-and-dwelling-countsaccessed December 2019

[15] R Likert ldquoA technique for the measurement of attitudesrdquo Archivesof psychology 1932

[16] U ISCED ldquoInternational standard classification of education 2011rdquo2012

[17] I Etikan ldquoComparison of convenience sampling and purposivesamplingrdquo American Journal of Theoretical and Applied Statisticsvol 5 p 1 01 2016

[18] ldquoQualtrics survey platformrdquo httpswwwqualtricscom accessedDecember 2019

[19] O Bock A Nicklisch and I Baetge ldquohroot Hamburg registrationand organization online toolrdquo in WiSo-HH Working Paper Series2012

[20] M L McHugh ldquoThe chi-square test of independencerdquo Biochemiamedica Biochemia medica vol 23 no 2 pp 143ndash149 2013

[21] M Burnett S Stumpf J Macbeth S Makri L BeckwithI Kwan A Peters and W Jernigan ldquoGendermag A methodfor evaluating softwarersquos gender inclusivenessrdquo Interacting withComputers vol 28 no 6 pp 760ndash787 2016

[22] T Rietz and A Maedche ldquoLadderbot A requirements self-elicitation systemrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 357ndash362

Fig 1 Overview of the proportion of feedback givers who use each online channel

Fig 2 User feedback with age

TABLE IVCOMPARING APP STORE AND FORUM FEEDBACK WITH AGE

App Store () Product Forums ()Under 25 years old 1772 93425 years old and over 1887 1985

TABLE VUSER FEEDBACK WITH GENDER

Number ofRespondents

App Store()

Product Forums()

Social Media(amp)

Men 571 2032 1804 823Women 454 1454 815 573

forums 00 social 49) (Fig 2) Conversely 35-45year oldrsquos (50 respondents) reported to give the mostfeedback across all channels (app store 340 forums280 social 260) Chi-squared tests show there arestatistically significant differences between ages (shown inTable III)

Also of note respondents under 25 preferred to givefeedback to the app store over product forums shown inbold in table IV Under 25rsquos preference for app stores wasshown to be statistically significant using a chi-squaredtest (p lt 001) Respondents 25 and over used app storesand forums more equally with those over 44 reportingmore forum use However the differences in channel usefor those 25 and above was not found to be significant

Gender Men reported to give more feedback thanwoman across all channels shown in table V On apps

TABLE VIUSER FEEDBACK TYPE WITH GENDER

App Store () Forums () Social Media ()

FeedbackType

Men Women Men Women Men Women

Praise 4138 5000 2039 1081 3830 3077

Reportbug

4052 4848 7379 5676 4681 4231

Requestfeature

2672 1818 3204 2162 2766 3846

Askquestion

259 606 8835 9459 6809 6538

Recommendto others

1296 1667 NA 3617 1923

Dissuadeothers

1034 606 NA 851 1154

Discussshortcomings

4741 3636 NA 4681 3462

Dispraise orcriticise

1810 1515 1650 811 NA

Helpfulsituation

3621 2727 NA NA

Discussfeature

2155 2273 NA NA

Assistothers

NA 5534 2162 NA

stores 203 of men and 145 of women reported togive feedback On product forums 180 of men and81 of women reported to give feedback On socialmedia the difference was the smallest with 82 and57 respectively reporting to give feedback Chi-squaredtests showed that the difference between men and womenrespondents was statistically significant for app stores(p=002) and product forums (plt0001)

Men and women respondents reported some differencesin the types of feedback they give on all three feedbackchannels shown in Table VI Differences of note areindicated in bold in the table On app stores more womenfeedback givers reported praising apps than feedbackgiving men (w 50 4138) and also reported givingbug reports More men reported describing a situation anapp was helpful reported an app short coming and requesta new features

On product forums both genders were very likely to ask

TABLE VIIUSER FEEDBACK WITH EMPLOYMENT TYPE

Number ofRespondents

Forums()

Under25 years

()Full-time 215 2140 2093

Part-time 78 1282 6667

Student 644 1087 7857

Full-time (no under 25rsquos) 170 2294 000

Part-time (no under 25rsquos) 26 1154 000

Students (no under 25rsquos) 138 1884 000

TABLE VIIIFEEDBACK OF SOFTWARE PROFESSIONALS

Number ofRespondents

App Store()

Forums()

SocialMedia

()Software Professionals 171 2749 1988 1287

Other Respondents 869 1632 1218 598

a question about software with 8835 of men feedbackgivers and 9459 of women Men feedback givers weremore likely to give other types of feedback includingreport a problem request a feature give praise givecriticism and assist others On social media more menreported recommending software to others and discussingshort comings More women reported requesting newfeatures

Employment Respondents working full time reportedusing product forums at a higher rate than those workingpart time and students (Table VII) However there isa strong association between employment level and ageas 7857 of students are also under 25 In the bottomhalf of table VII all under 25 year old respondents wereremoved from the analysis showing the difference betweenemployment levels is not as large when consideringonly older respondents The feedback differences betweenemployment groups were not found to be statisticallysignificant using chi-squared tests after the exclusion ofthe under 25 year old respondents

Software professionals Respondents who work or haveworked in software (software professionals) reported tohave given feedback at a higher rate than those who havenot worked in software on all channels (Table VIII) Chi-squared tests showed that the feedback rate differencebetween software professionals and other respondents wassignificant on all channels (app stores p=0001 productforums p=001 social media p=0002)

Ethnicity The majority of survey respondents wereeither Caucasian (790) or Asian (149) which limited ourfindings with respect to ethnicity However the ethnicdemographics of the respondents are representative for astudy based in New Zealand and Germany Only the dif-ference between Caucasian and Asian feedback rate couldbe investigated and this difference was not statisticallysignificant on any channel

Fig 3 Impact of in app prompts

Answer to RQ1 There are statistically significantdifferences in the amount of written feedbackgiven by software users with respect to traditionaldemographics For gender men reported givingmore feedback than women on all three feedbackchannels The types of feedback men and womenreported to give also varied in notable ways Withage distinct patterns emerged with respondentsbetween 35 and 45 reporting to give the most feed-back and under 18rsquos reporting to give the least onall channels Additionally software professionalsreported giving significantly more feedback thanother respondents

B Motivations

RQ2 What motivates software users to give onlinefeedback and are there differences across demographics

The section presents our findings with respect to whatmotivates users to give online feedback The difference inmotivations across the three channels and between groupsare given

Overall The reported motivations to give feedback onapp stores product forums and social media are given intable IX The motivations are given as a percentage of allusers who give written feedback on each channel Multiplemotivations could be given by each respondent As canbe seen the motivations vary across feedback channelsShow appreciation for software was the most commoncited motivation on app stores (6515) and Social media(5676) Get help with software was the top motivation topost on product forums (7037) Influencing improvementwas also a prominent motivation being the third most citedon all channels

Mobile app prompts (Fig 3) 5245 of all surveyrespondents reported having previously given a star ratingto an app Of those who have given a star rating 6575only gave the rating when prompted within the app neverdirectly on the app store 1816 of respondents reportedhaving given a written review to an app Of those whohave given a written review 3175 only gave a writtenreview when prompted to within the app

Gender Some differences in motivations to give feed-back were reported between men and women The per-centage of men and woman feedback givers who cited

TABLE IXMOTIVATIONS TO GIVE FEEDBACK

App Store () Product Forum () Social Media ()

1 Show appreciation 6515 1 Get help 7037 1 Show appreciation 5676

2 Influence improvement 5202 2 Influence improvement 4429 2 Influence improvement 5135

3 Show dissatisfaction 3485 3 Show appreciation 2643 3 Show dissatisfaction 3784

4 Recommendto others 2980 4 Recommend to others 1786 4 Connect or socialise 3514

5 Discourage others 1263 5 Show dissatisfaction 1643 5 Recommend to others 3243

6 Get help 920 6 Connect or socialise 1572 6 Get help 2273

7 No specific motivation 505 7 No specific motivation 786 7 Discourage others 1486

8 Connect or socialise 152 8 Discourage others 357 8 No specific motivation 811

TABLE XMOTIVATIONS TO GIVE FEEDBACK WITH GENDER

App Store () Product Forums () Social Media ()

Motivation Men Women Men Women Men Women

Showappreciation

6724 7273 2816 1892 5745 5385

Showdissatisfaction

3621 3636 1359 2162 3191 4615

Influenceimprovement

5776 5000 4951 2703 5532 4231

Recommend 2979 3462 3276 3030 1845 1351

Discourage 1638 606 194 541 1064 1923

Connectsocialise

431 758 1359 1892 2766 4231

Get help 1014 588 7111 7778 4118 000No specificmotivation

086 303 971 000 426 1154

Fig 4 Feedback given by individual users each year on each channel

each motivation are shown in table X where notabledifferences are indicated in bold On app stores men weremore motivated to discourage others from using a dislikedapp On product forums more men cited influencing animprovement in the software as a motivation On socialmedia more women were motivated to show dissatisfactionand connect or socialise about a software product Also onsocial media more men cited influence improvement andget help These results are bolded in Table X

Feedback frequency The majority of feedback giversreported having given feedback between 0 and 4 times inthe last year across all channels (Fig 4) App stores hadthe least respondents reporting to give more than 4 piecesof feedback product forums had the most respondentsgiving feedback more than 4 times

Perception of influencing developers Survey re-spondents who believed that software developers woulddefinitely not be influenced by online feedback were lesslikely to give feedback than those who believed influence

TABLE XIUSER FEEDBACK WITH PERCEPTION OF INFLUENCING DEVELOPERS

Number ofRespondents

App Store()

Forums()

Socialmedia()

Definitely will 83 1446 1807 723

Probably will 265 1925 1396 792

Might or might not 416 1875 1490 649

Probably will not 248 1895 968 806

Definitely will not 27 370 741 000

was more likely on all channels However chi-squared testsshowed these differences werenrsquot statistically significantFeedback rates with perception of influencing developersare shown in table XI the lower definitely not values areindicated in bold

Answer to RQ2 Showing appreciation was the topmotivation given to write feedback on app storesand social media On product forums getting helpwas the most commonly cited motivation (table IX)Differences in the motivations of men and womento give written feedback on each channel were alsoreportedIn-app prompts were reported to be very effectiveat motivating app users to give star ratings but lesseffective at eliciting written feedback Individualsurvey respondents reported engaging with eachfeedback channel at different frequencies writingon product forums the most times a year and leaston app stores

C Type of software and duration of use

RQ3 Does the likelihood of giving online writtenfeedback vary based on the type of software used andthe duration of software usage

iPhoneAndroid Android users reported giving feed-back to the app store at a higher rate than iPhone users(Table XII) 1348 of iPhone users reported having givenwritten feedback on app stores compared to 2184 ofAndroid users A chi-squared test showed this difference tobe statistically significant given in table XIII The higherfeedback rate of Android users on app stores has beenindicate in bold in table XII

WindowsMacLinux Linux users reported giving writ-ten feedback on app stores and product forums at a higher

TABLE XIIUSER FEEDBACK WITH DEVICE TYPE

Device Number ofRespondents

App Store()

ProductForum

()

SocialMedia

()Android 618 2184 1375 680

iPhone 423 1348 1277 733

Linux 94 3191 2660 1064

Windows 759 1910 146 646

Mac 275 1673 1236 1018

TABLE XIIIUSER FEEDBACK WITH DEVICE TYPE SIGNIFICANCE TESTS

App Store Product Forums

Chi2 p Chi2 p

Android gt iPhone 11144 0001 0087 0769

Linux gt Windows 7651 0006 8073 0004Linux gt Mac 8974 0003 9531 0002

Statistically significant results are bolded

TABLE XIVUSER FEEDBACK WITH DAILY COMPUTER USE

Daily Computer Use Number ofRespondents

App Store()

Forums()

SocialMedia

()Less than 1 hour 109 1835 1009 642

1 - 4 hours 436 1514 940 596

4 - 8 hours 363 2066 1708 799

More than 8 hours 110 2182 2364 909

rate than Windows and Mac users (Table XII) Chi-squaredtests showed these differences to both be statisticallysignificant (Table XIII) The difference between Windowsand Mac users feedback was not statistically significantThe statistically significant higher feedback rates of Linuxusers are indicated in bold in table XII

Hours of computer use Respondents who reported ahigher daily computer use (hours) were more likely to givefeedback to product forums indicated in bold in table XIVThe least forum feedback was given by respondents usingtheir computer less then 1 hour or between 1 and 4 hoursa day Those using their computer between 4 and 8 hoursgave more feedback and those using their computer morethan 8 hours a day gave at the highest rate Chi-squaredtests showed that the feedback rate differences between 1 -4 hours and 4 - 8 hours and between 1- 4 hours and over8 hours were statistically significant (Table XV)

Hours of phone use Respondents who reported ahigher daily phone use (hours) were more likely togive feedback to social media indicated in bold in tableXVI However chi-squared tests showed these differenceswerenrsquot statistically significant

TABLE XVCOMPUTER DAILY USE SIGNIFICANCE TESTS (PRODUCT FORUMS)

Daily Computer Use Chi2 p

Less than 1 hour lt 1-4 hours 0001 0971

Less than 1 hour lt 4-8 hours 2619 0106

1 - 4 lt Over 8 hours 15233 lt 0001Less than 1 lt Over 8 hours 6221 00131-4 hours lt 4-8 hours 9722 00024-8 hours lt Over 8 hours 1983 0159

Statistically significant results are bolded

TABLE XVIUSER FEEDBACK WITH DAILY PHONE USE

Daily Phone Use Number ofRespondents

App Store()

Forums()

SocialMedia

()Less than 1 hour 52 1538 1538 3851 - 4 hours 664 1657 1370 6634 - 8 hours 266 2293 1165 789More than 8 hours 51 1765 1765 1373

Answer to RQ3 Statistically significant differ-ences were reported in the amount of writtenfeedback given based on the type of software usedand the duration of daily use Respondents whospend more hours each day on their computerreported giving more written feedback to productforums Those using the Linux OS gave morewritten feedback to app stores and product forumsthan those using Windows and Mac Android usersreported giving more written feedback to app storesthan iPhone users

V DISCUSSION

In this section we first discuss the threats to validity ofour study and then describe the implications and potentialavenues for future work

A Threats to validity

Convenience sampling used in this study to elicit surveyparticipants is a non-probabilistic sampling method anda possible source of bias [17] The target population ofthis study are users of software and mobile applicationsSurvey participants were engaged via Facebook Twitterthe Karlsruhe Institute of Technology survey pool and inAuckland cities public areas therefore only a subset ofthe target population had the opportunity to participateAdditionally all respondents who completed the surveywere self-selected and their feedback habits may notgeneralise to all software users

To mitigate this bias we collected data from a largenumber of software users (1040 participants) Recruitmentwas done through three channels social media publicspaces and the hroot survey pool to increase the oddsof recruiting a diverse set of respondents However wecannot claim that our results generalise outside of oursample Future studies can replicate our survey to validateour findings

Our participants are not representative across all demo-graphics The demographics of our respondents are listedin Table II The majority of participants are whiteEuropeanand many are students When presenting our resultswe present proportions based on the total number ofrespondents in each demographic group We also used chi-squared tests to determine significance between differentdemographic groups which accounts for the sample size ofeach population being compared Therefore findings foundto be statistically significant had a sufficient number of re-spondents in each demographic to satisfy the test Howeverdue to a low number of participants in some demographicgroups not all demographics could be analysed Futurestudies should replicate this survey to enable analysis ofadditional demographics

In addition the majority of participants identified asmen or women We did give participants the option toself-specify gender but very few participants chose toself-specify Thus our analysis was limited to only thedifferences between participants who identified as men andwomen Again future studies should replicate the surveyto enable analysis beyond these binary genders

Our results are based only on self-reported feedbackhabits Demographic information of feedback givers is notreadily available on the feedback channels we investigated(often the real name of the writer isnrsquot even given) Thisdata sparsity problem means our findings canrsquot be directlyvalidated against actual feedback data One previous studyby Guzman et al [5] approximated the gender of feedbackgivers on app stores from their usernames Using theseapproximations they found that men were more likely thanwomen to provide feedback on the Apple app store whichis in line with what our respondents reported and supportsour findings

B Implications and Future Work

Implication 1 The findings presented in this papersuggest that when leveraging online user feedback to getthe most representative user views and desires feedbackfrom multiple feedback channels should be consideredWe found statistically significant differences in the userswho reported to give feedback on app stores productforums and social media with respect to traditionaldemographics and software usage habits For exampleolder respondents prefer product forums to app storeswhile younger respondents prefer app stores

Importantly a majority of feedback giving respondentsreported only engaging with one of these three feedbackchannels This indicates that considering multiple channelswill enable feedback from a more diverse set of users

We also found key differences in what motivates softwareusers to engage with each of the three channels The mostcited motivation on app stores and social media was to showappreciation for the appsoftware Whereas on productforums showing appreciation was much less of a motivatingfactor instead getting help was the top cited motivationShowing dissatisfaction recommending and discouragingothers were also significantly more cited on app stores andsocial media On social media connecting with other userswas reported to be a more common motivation than on theother channels

These motivation differences suggest that the feedbackon each online channel is likely to contain different productdevelopment insights For example feedback on productforums contain users trying to get help and therefore likelydescribes ways the software is unintuitive or difficult to useOn app stores and social media users are more motivatedto communicate how they feel about the softwareapp tothe developers and other users These differences againemphasis the benefit to considering feedback from allchannels as each channel may provide unique insights

Implication 2 How to elicit feedback from under-represented groups needs further study We saw somedemographics were less likely to give feedback than othersFor example respondents 35-44 years old report to providethe most feedback on all three feedback channels whileboth older and younger respondents gave less feedback(Fig 2) Also men reported giving feedback at a higher ratethan women across all three channels This is in line withthe results of Guzman et al which found that the Apple appstore had more feedback from men [5] Future work shouldinvestigate why some user groups give more feedback thanothers to understand how to better elicit feedback fromthese underrepresented groups This work could surveyunderrepresented user groups on why they donrsquot givefeedback so that these factors can be directly addressedChanges to the feedback channels themselves may evenbe considered to make the tool more inclusive Recentresearch found that most software has gender inclusivityissues [21] so it is possible that similar inclusivity issuesexist in the software that collects online feedback

Additionally studies could be performed to see ifother means of collecting feedback that augments writtenfeedback on the feedback channels we studied could beused to help provide a more comprehensive set of feedbackacross demographics For example recent research pro-posed using a conversational agent (ladderbot) to conductshort requirements interviews to elicit user feedback [22]Future research could evaluate whether some demographicswould prefer other user feedback collection methods (likeladderbot)

Implication 3 Feedback prompts are effective at elicit-ing feedback for app stores and may be effective if appliedmore widely in computer software However more workis needed to understand how to prompt users to givedetailed feedback Mobile apps widely use prompts to elicitfeedback More survey respondents reported giving writtenfeedback on app stores than on any other channel Much ofthis feedback is prompted The number of respondents whohave provided unprompted app store feedback (1239) isvery similar to the number who report to have written postson product forums (1345) This suggests that the promptsare successful in eliciting additional feedback givers Theprompts are even more effective at eliciting app ratingswhich take less time to provide than written feedbackFuture research could study whether prompts could besuccessful in collecting other types of user feedback andhow they could be integrated into other feedback channels

Implication 4 The types of software devices respondentsuse also has an association to feedback habits Investigatingwhy users of some devices give more feedback may giveinsights into how to motivate and facilitate feedback

On phones more Android users give written feedbackthan iPhone users It is not clear why there are differencesin feedback across devices but it may be influenced bydifferences in prompt rates app quality app store usabilityor even those who choose to use each phone type iOSdevelopers could benefit in understanding these factors inorder to encourage more feedback from their users

On computers respondents who use the Linux OS morecommonly had given written feedback to app stores andproduct forums than those who do not use Linux Thefeedback habits of Mac and Windows users were relativelysimilar across all feedback channels The higher feedbackrates of Linux users may be related to the prevalence ofsoftware developers using it In fact 43 of respondentsusing Linux also reported working in the software industrycompared to only 16 of all respondents Our resultsshowed that software professionals are more likely toprovide online feedback possibly because they understandhow that feedback will be used by development teamsFuture research can investigate more thoroughly the reasonsfor differences across devices

Other avenues for future work Investigate otherfeedback channels Our study was limited to app storesproduct forums and social media Future work couldperform a similar investigation considering other feedbackchannels like issue trackers

Replicate survey in other countries Our survey re-spondents were mostly from two countries New Zealandand Germany Future work could replicate our survey byeliciting responses in additional countries This would alsoenable analysis at the ethnicity level if more ethnic diversityin the participants was achieved

Understand gender differences in product forum engage-ment In addition to men being more likely than women topost on product forums men also reported using productsforums for different reasons While men and women bothprimarily used forums to ask software related questionsmen also reported higher rates of giving other types offeedback on product forums including reporting problemsrequesting features praising and criticising the softwareas well as assisting others Further research is neededto understand the gender difference in engagement withproduct forums

Making missing demographics more transparent Cur-rently it is difficult for product development teams toknow whether the feedback collected from online feed-back channels is biased and misses the voices of someunderrepresented groups Future research could deviseways to make this more transparent to enable softwaredevelopment teams to more proactively consider the needsof the underrepresented groups and produce more inclusivesoftware

VI CONCLUSION

The online user feedback written on app stores productforums and social media is a valuable source of require-ments for software developers and has been a focus ofrequirements engineering researchers However limitedstudies have been done to understand which softwareusers give this feedback and what motivates them In thiswork we directly surveyed 1040 software users about

their feedback habits software use and demographicinformation

The responses indicate significant differences in thedemographics of software users who give feedback oneach online channel For gender men reported giving morefeedback than women and with age respondents between35 and 45 reporting to give the most feedback acrossall channels We also found strong evidence that youngersoftware users (under 25) prefer to engage with app storeswhereas older software users use product forums at equal(to app stores) and sometimes higher rates

We identified key differences in what motivates softwareusers to engage with each of the three channels Comparingchannels respondents reported the top motivation to givefeedback on app stores and social media was to showappreciation whereas on forums the most cited motivationwas to get help with software products Differences betweenthe motivations of men and women to give feedbackwere also reported for each of the channels Respondentsreported in app prompts to be significantly more effective inmotivating them to give app ratings over written feedbackAdditionally individual feedback givers reported to engagemore times a year on product forums than on with appstores

Differences in feedback habits were also reported withthe ways respondents use software Those who spendmore hours each day on their phone or computer reportedgiving more feedback about the software theyrsquore using Thesoftware platform being used also presented a relationshipto feedback rates with more Linux (computer) and Android(phone) users reporting to give feedback than those whouse the alternatives

The findings presented in this paper give meaningfulinsights into which software users give online feedbackand the motivations they have to give it We foundnotable differences in those who give feedback to eachonline channel which emphasises the need to mine allthree feedback channels to get the most representativerequirements from software users when leveraging onlinefeedback

REFERENCES

[1] E Guzman R Alkadhi and N Seyff ldquoA needle in a haystackWhat do twitter users say about softwarerdquo in 2016 IEEE 24thInternational Requirements Engineering Conference (RE) Sep2016 pp 96ndash105

[2] D Pagano and W Maalej ldquoUser feedback in the appstore Anempirical studyrdquo in 2013 21st IEEE International RequirementsEngineering Conference (RE) July 2013 pp 125ndash134

[3] J Tizard H Wang L Yohannes and K Blincoe ldquoCan a con-versation paint a picture mining requirements in software fo-rumsrdquo in 2019 IEEE 27th International Requirements EngineeringConference (RE) IEEE 2019 pp 17ndash27

[4] E Guzman R Alkadhi and N Seyff ldquoAn exploratory studyof twitter messages about software applicationsrdquo RequirementsEngineering 07 2017

[5] E Guzman and A P Rojas ldquoGender and user feedback Anexploratory studyrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 381ndash385

[6] E Guzman M Ibrahim and M Glinz ldquoA little bird told me miningtweets for requirements and software evolutionrdquo in 2017 IEEE 25thInternational Requirements Engineering Conference (RE) IEEE2017 pp 11ndash20

[7] W Maalej and H Nabil ldquoBug report feature request orsimply praise on automatically classifying app reviewsrdquo in 2015IEEE 23rd International Requirements Engineering Conference

(RE) vol 00 Aug 2015 pp 116ndash125 [Online] Availabledoiieeecomputersocietyorg101109RE20157320414

[8] A D Sorbo S Panichella C V Alexandru C A Visaggio andG Canfora ldquoSurf Summarizer of user reviews feedbackrdquo in 2017IEEEACM 39th International Conference on Software EngineeringCompanion (ICSE-C) May 2017 pp 55ndash58

[9] E Guzman L Oliveira Y Steiner L C Wagner and M GlinzldquoUser feedback in the app store a cross-cultural studyrdquo in 2018IEEEACM 40th International Conference on Software EngineeringSoftware Engineering in Society (ICSE-SEIS) IEEE 2018 pp13ndash22

[10] E C Groen N Seyff R Ali F Dalpiaz J Doerr E GuzmanM Hosseini J Marco M Oriol A Perini et al ldquoThe crowd inrequirements engineering The landscape and challengesrdquo IEEEsoftware vol 34 no 2 pp 44ndash52 2017

[11] S Panichella A Di Sorbo E Guzman C A Visaggio G Canforaand H C Gall ldquoArdoc App reviews development orientedclassifierrdquo in Proceedings of the 2016 24th ACM SIGSOFTInternational Symposium on Foundations of Software Engineeringser FSE 2016 New York NY USA ACM 2016 pp 1023ndash1027[Online] Available httpdoiacmorg10114529502902983938

[12] N Chen J Lin S C H Hoi X Xiao and B ZhangldquoAr-miner Mining informative reviews for developers frommobile app marketplacerdquo in Proceedings of the 36th InternationalConference on Software Engineering ser ICSE 2014 NewYork NY USA ACM 2014 pp 767ndash778 [Online] Availablehttpdoiacmorg10114525682252568263

[13] N Papadopoulos O M Martiacuten M Cleveland and M Laroche

ldquoIdentity demographics and consumer behaviorsrdquo InternationalMarketing Review 2011

[14] ldquoNew zealand census 2018rdquo httpswwwstatsgovtnzinformation-releases2018-census-population-and-dwelling-countsaccessed December 2019

[15] R Likert ldquoA technique for the measurement of attitudesrdquo Archivesof psychology 1932

[16] U ISCED ldquoInternational standard classification of education 2011rdquo2012

[17] I Etikan ldquoComparison of convenience sampling and purposivesamplingrdquo American Journal of Theoretical and Applied Statisticsvol 5 p 1 01 2016

[18] ldquoQualtrics survey platformrdquo httpswwwqualtricscom accessedDecember 2019

[19] O Bock A Nicklisch and I Baetge ldquohroot Hamburg registrationand organization online toolrdquo in WiSo-HH Working Paper Series2012

[20] M L McHugh ldquoThe chi-square test of independencerdquo Biochemiamedica Biochemia medica vol 23 no 2 pp 143ndash149 2013

[21] M Burnett S Stumpf J Macbeth S Makri L BeckwithI Kwan A Peters and W Jernigan ldquoGendermag A methodfor evaluating softwarersquos gender inclusivenessrdquo Interacting withComputers vol 28 no 6 pp 760ndash787 2016

[22] T Rietz and A Maedche ldquoLadderbot A requirements self-elicitation systemrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 357ndash362

TABLE VIIUSER FEEDBACK WITH EMPLOYMENT TYPE

Number ofRespondents

Forums()

Under25 years

()Full-time 215 2140 2093

Part-time 78 1282 6667

Student 644 1087 7857

Full-time (no under 25rsquos) 170 2294 000

Part-time (no under 25rsquos) 26 1154 000

Students (no under 25rsquos) 138 1884 000

TABLE VIIIFEEDBACK OF SOFTWARE PROFESSIONALS

Number ofRespondents

App Store()

Forums()

SocialMedia

()Software Professionals 171 2749 1988 1287

Other Respondents 869 1632 1218 598

a question about software with 8835 of men feedbackgivers and 9459 of women Men feedback givers weremore likely to give other types of feedback includingreport a problem request a feature give praise givecriticism and assist others On social media more menreported recommending software to others and discussingshort comings More women reported requesting newfeatures

Employment Respondents working full time reportedusing product forums at a higher rate than those workingpart time and students (Table VII) However there isa strong association between employment level and ageas 7857 of students are also under 25 In the bottomhalf of table VII all under 25 year old respondents wereremoved from the analysis showing the difference betweenemployment levels is not as large when consideringonly older respondents The feedback differences betweenemployment groups were not found to be statisticallysignificant using chi-squared tests after the exclusion ofthe under 25 year old respondents

Software professionals Respondents who work or haveworked in software (software professionals) reported tohave given feedback at a higher rate than those who havenot worked in software on all channels (Table VIII) Chi-squared tests showed that the feedback rate differencebetween software professionals and other respondents wassignificant on all channels (app stores p=0001 productforums p=001 social media p=0002)

Ethnicity The majority of survey respondents wereeither Caucasian (790) or Asian (149) which limited ourfindings with respect to ethnicity However the ethnicdemographics of the respondents are representative for astudy based in New Zealand and Germany Only the dif-ference between Caucasian and Asian feedback rate couldbe investigated and this difference was not statisticallysignificant on any channel

Fig 3 Impact of in app prompts

Answer to RQ1 There are statistically significantdifferences in the amount of written feedbackgiven by software users with respect to traditionaldemographics For gender men reported givingmore feedback than women on all three feedbackchannels The types of feedback men and womenreported to give also varied in notable ways Withage distinct patterns emerged with respondentsbetween 35 and 45 reporting to give the most feed-back and under 18rsquos reporting to give the least onall channels Additionally software professionalsreported giving significantly more feedback thanother respondents

B Motivations

RQ2 What motivates software users to give onlinefeedback and are there differences across demographics

The section presents our findings with respect to whatmotivates users to give online feedback The difference inmotivations across the three channels and between groupsare given

Overall The reported motivations to give feedback onapp stores product forums and social media are given intable IX The motivations are given as a percentage of allusers who give written feedback on each channel Multiplemotivations could be given by each respondent As canbe seen the motivations vary across feedback channelsShow appreciation for software was the most commoncited motivation on app stores (6515) and Social media(5676) Get help with software was the top motivation topost on product forums (7037) Influencing improvementwas also a prominent motivation being the third most citedon all channels

Mobile app prompts (Fig 3) 5245 of all surveyrespondents reported having previously given a star ratingto an app Of those who have given a star rating 6575only gave the rating when prompted within the app neverdirectly on the app store 1816 of respondents reportedhaving given a written review to an app Of those whohave given a written review 3175 only gave a writtenreview when prompted to within the app

Gender Some differences in motivations to give feed-back were reported between men and women The per-centage of men and woman feedback givers who cited

TABLE IXMOTIVATIONS TO GIVE FEEDBACK

App Store () Product Forum () Social Media ()

1 Show appreciation 6515 1 Get help 7037 1 Show appreciation 5676

2 Influence improvement 5202 2 Influence improvement 4429 2 Influence improvement 5135

3 Show dissatisfaction 3485 3 Show appreciation 2643 3 Show dissatisfaction 3784

4 Recommendto others 2980 4 Recommend to others 1786 4 Connect or socialise 3514

5 Discourage others 1263 5 Show dissatisfaction 1643 5 Recommend to others 3243

6 Get help 920 6 Connect or socialise 1572 6 Get help 2273

7 No specific motivation 505 7 No specific motivation 786 7 Discourage others 1486

8 Connect or socialise 152 8 Discourage others 357 8 No specific motivation 811

TABLE XMOTIVATIONS TO GIVE FEEDBACK WITH GENDER

App Store () Product Forums () Social Media ()

Motivation Men Women Men Women Men Women

Showappreciation

6724 7273 2816 1892 5745 5385

Showdissatisfaction

3621 3636 1359 2162 3191 4615

Influenceimprovement

5776 5000 4951 2703 5532 4231

Recommend 2979 3462 3276 3030 1845 1351

Discourage 1638 606 194 541 1064 1923

Connectsocialise

431 758 1359 1892 2766 4231

Get help 1014 588 7111 7778 4118 000No specificmotivation

086 303 971 000 426 1154

Fig 4 Feedback given by individual users each year on each channel

each motivation are shown in table X where notabledifferences are indicated in bold On app stores men weremore motivated to discourage others from using a dislikedapp On product forums more men cited influencing animprovement in the software as a motivation On socialmedia more women were motivated to show dissatisfactionand connect or socialise about a software product Also onsocial media more men cited influence improvement andget help These results are bolded in Table X

Feedback frequency The majority of feedback giversreported having given feedback between 0 and 4 times inthe last year across all channels (Fig 4) App stores hadthe least respondents reporting to give more than 4 piecesof feedback product forums had the most respondentsgiving feedback more than 4 times

Perception of influencing developers Survey re-spondents who believed that software developers woulddefinitely not be influenced by online feedback were lesslikely to give feedback than those who believed influence

TABLE XIUSER FEEDBACK WITH PERCEPTION OF INFLUENCING DEVELOPERS

Number ofRespondents

App Store()

Forums()

Socialmedia()

Definitely will 83 1446 1807 723

Probably will 265 1925 1396 792

Might or might not 416 1875 1490 649

Probably will not 248 1895 968 806

Definitely will not 27 370 741 000

was more likely on all channels However chi-squared testsshowed these differences werenrsquot statistically significantFeedback rates with perception of influencing developersare shown in table XI the lower definitely not values areindicated in bold

Answer to RQ2 Showing appreciation was the topmotivation given to write feedback on app storesand social media On product forums getting helpwas the most commonly cited motivation (table IX)Differences in the motivations of men and womento give written feedback on each channel were alsoreportedIn-app prompts were reported to be very effectiveat motivating app users to give star ratings but lesseffective at eliciting written feedback Individualsurvey respondents reported engaging with eachfeedback channel at different frequencies writingon product forums the most times a year and leaston app stores

C Type of software and duration of use

RQ3 Does the likelihood of giving online writtenfeedback vary based on the type of software used andthe duration of software usage

iPhoneAndroid Android users reported giving feed-back to the app store at a higher rate than iPhone users(Table XII) 1348 of iPhone users reported having givenwritten feedback on app stores compared to 2184 ofAndroid users A chi-squared test showed this difference tobe statistically significant given in table XIII The higherfeedback rate of Android users on app stores has beenindicate in bold in table XII

WindowsMacLinux Linux users reported giving writ-ten feedback on app stores and product forums at a higher

TABLE XIIUSER FEEDBACK WITH DEVICE TYPE

Device Number ofRespondents

App Store()

ProductForum

()

SocialMedia

()Android 618 2184 1375 680

iPhone 423 1348 1277 733

Linux 94 3191 2660 1064

Windows 759 1910 146 646

Mac 275 1673 1236 1018

TABLE XIIIUSER FEEDBACK WITH DEVICE TYPE SIGNIFICANCE TESTS

App Store Product Forums

Chi2 p Chi2 p

Android gt iPhone 11144 0001 0087 0769

Linux gt Windows 7651 0006 8073 0004Linux gt Mac 8974 0003 9531 0002

Statistically significant results are bolded

TABLE XIVUSER FEEDBACK WITH DAILY COMPUTER USE

Daily Computer Use Number ofRespondents

App Store()

Forums()

SocialMedia

()Less than 1 hour 109 1835 1009 642

1 - 4 hours 436 1514 940 596

4 - 8 hours 363 2066 1708 799

More than 8 hours 110 2182 2364 909

rate than Windows and Mac users (Table XII) Chi-squaredtests showed these differences to both be statisticallysignificant (Table XIII) The difference between Windowsand Mac users feedback was not statistically significantThe statistically significant higher feedback rates of Linuxusers are indicated in bold in table XII

Hours of computer use Respondents who reported ahigher daily computer use (hours) were more likely to givefeedback to product forums indicated in bold in table XIVThe least forum feedback was given by respondents usingtheir computer less then 1 hour or between 1 and 4 hoursa day Those using their computer between 4 and 8 hoursgave more feedback and those using their computer morethan 8 hours a day gave at the highest rate Chi-squaredtests showed that the feedback rate differences between 1 -4 hours and 4 - 8 hours and between 1- 4 hours and over8 hours were statistically significant (Table XV)

Hours of phone use Respondents who reported ahigher daily phone use (hours) were more likely togive feedback to social media indicated in bold in tableXVI However chi-squared tests showed these differenceswerenrsquot statistically significant

TABLE XVCOMPUTER DAILY USE SIGNIFICANCE TESTS (PRODUCT FORUMS)

Daily Computer Use Chi2 p

Less than 1 hour lt 1-4 hours 0001 0971

Less than 1 hour lt 4-8 hours 2619 0106

1 - 4 lt Over 8 hours 15233 lt 0001Less than 1 lt Over 8 hours 6221 00131-4 hours lt 4-8 hours 9722 00024-8 hours lt Over 8 hours 1983 0159

Statistically significant results are bolded

TABLE XVIUSER FEEDBACK WITH DAILY PHONE USE

Daily Phone Use Number ofRespondents

App Store()

Forums()

SocialMedia

()Less than 1 hour 52 1538 1538 3851 - 4 hours 664 1657 1370 6634 - 8 hours 266 2293 1165 789More than 8 hours 51 1765 1765 1373

Answer to RQ3 Statistically significant differ-ences were reported in the amount of writtenfeedback given based on the type of software usedand the duration of daily use Respondents whospend more hours each day on their computerreported giving more written feedback to productforums Those using the Linux OS gave morewritten feedback to app stores and product forumsthan those using Windows and Mac Android usersreported giving more written feedback to app storesthan iPhone users

V DISCUSSION

In this section we first discuss the threats to validity ofour study and then describe the implications and potentialavenues for future work

A Threats to validity

Convenience sampling used in this study to elicit surveyparticipants is a non-probabilistic sampling method anda possible source of bias [17] The target population ofthis study are users of software and mobile applicationsSurvey participants were engaged via Facebook Twitterthe Karlsruhe Institute of Technology survey pool and inAuckland cities public areas therefore only a subset ofthe target population had the opportunity to participateAdditionally all respondents who completed the surveywere self-selected and their feedback habits may notgeneralise to all software users

To mitigate this bias we collected data from a largenumber of software users (1040 participants) Recruitmentwas done through three channels social media publicspaces and the hroot survey pool to increase the oddsof recruiting a diverse set of respondents However wecannot claim that our results generalise outside of oursample Future studies can replicate our survey to validateour findings

Our participants are not representative across all demo-graphics The demographics of our respondents are listedin Table II The majority of participants are whiteEuropeanand many are students When presenting our resultswe present proportions based on the total number ofrespondents in each demographic group We also used chi-squared tests to determine significance between differentdemographic groups which accounts for the sample size ofeach population being compared Therefore findings foundto be statistically significant had a sufficient number of re-spondents in each demographic to satisfy the test Howeverdue to a low number of participants in some demographicgroups not all demographics could be analysed Futurestudies should replicate this survey to enable analysis ofadditional demographics

In addition the majority of participants identified asmen or women We did give participants the option toself-specify gender but very few participants chose toself-specify Thus our analysis was limited to only thedifferences between participants who identified as men andwomen Again future studies should replicate the surveyto enable analysis beyond these binary genders

Our results are based only on self-reported feedbackhabits Demographic information of feedback givers is notreadily available on the feedback channels we investigated(often the real name of the writer isnrsquot even given) Thisdata sparsity problem means our findings canrsquot be directlyvalidated against actual feedback data One previous studyby Guzman et al [5] approximated the gender of feedbackgivers on app stores from their usernames Using theseapproximations they found that men were more likely thanwomen to provide feedback on the Apple app store whichis in line with what our respondents reported and supportsour findings

B Implications and Future Work

Implication 1 The findings presented in this papersuggest that when leveraging online user feedback to getthe most representative user views and desires feedbackfrom multiple feedback channels should be consideredWe found statistically significant differences in the userswho reported to give feedback on app stores productforums and social media with respect to traditionaldemographics and software usage habits For exampleolder respondents prefer product forums to app storeswhile younger respondents prefer app stores

Importantly a majority of feedback giving respondentsreported only engaging with one of these three feedbackchannels This indicates that considering multiple channelswill enable feedback from a more diverse set of users

We also found key differences in what motivates softwareusers to engage with each of the three channels The mostcited motivation on app stores and social media was to showappreciation for the appsoftware Whereas on productforums showing appreciation was much less of a motivatingfactor instead getting help was the top cited motivationShowing dissatisfaction recommending and discouragingothers were also significantly more cited on app stores andsocial media On social media connecting with other userswas reported to be a more common motivation than on theother channels

These motivation differences suggest that the feedbackon each online channel is likely to contain different productdevelopment insights For example feedback on productforums contain users trying to get help and therefore likelydescribes ways the software is unintuitive or difficult to useOn app stores and social media users are more motivatedto communicate how they feel about the softwareapp tothe developers and other users These differences againemphasis the benefit to considering feedback from allchannels as each channel may provide unique insights

Implication 2 How to elicit feedback from under-represented groups needs further study We saw somedemographics were less likely to give feedback than othersFor example respondents 35-44 years old report to providethe most feedback on all three feedback channels whileboth older and younger respondents gave less feedback(Fig 2) Also men reported giving feedback at a higher ratethan women across all three channels This is in line withthe results of Guzman et al which found that the Apple appstore had more feedback from men [5] Future work shouldinvestigate why some user groups give more feedback thanothers to understand how to better elicit feedback fromthese underrepresented groups This work could surveyunderrepresented user groups on why they donrsquot givefeedback so that these factors can be directly addressedChanges to the feedback channels themselves may evenbe considered to make the tool more inclusive Recentresearch found that most software has gender inclusivityissues [21] so it is possible that similar inclusivity issuesexist in the software that collects online feedback

Additionally studies could be performed to see ifother means of collecting feedback that augments writtenfeedback on the feedback channels we studied could beused to help provide a more comprehensive set of feedbackacross demographics For example recent research pro-posed using a conversational agent (ladderbot) to conductshort requirements interviews to elicit user feedback [22]Future research could evaluate whether some demographicswould prefer other user feedback collection methods (likeladderbot)

Implication 3 Feedback prompts are effective at elicit-ing feedback for app stores and may be effective if appliedmore widely in computer software However more workis needed to understand how to prompt users to givedetailed feedback Mobile apps widely use prompts to elicitfeedback More survey respondents reported giving writtenfeedback on app stores than on any other channel Much ofthis feedback is prompted The number of respondents whohave provided unprompted app store feedback (1239) isvery similar to the number who report to have written postson product forums (1345) This suggests that the promptsare successful in eliciting additional feedback givers Theprompts are even more effective at eliciting app ratingswhich take less time to provide than written feedbackFuture research could study whether prompts could besuccessful in collecting other types of user feedback andhow they could be integrated into other feedback channels

Implication 4 The types of software devices respondentsuse also has an association to feedback habits Investigatingwhy users of some devices give more feedback may giveinsights into how to motivate and facilitate feedback

On phones more Android users give written feedbackthan iPhone users It is not clear why there are differencesin feedback across devices but it may be influenced bydifferences in prompt rates app quality app store usabilityor even those who choose to use each phone type iOSdevelopers could benefit in understanding these factors inorder to encourage more feedback from their users

On computers respondents who use the Linux OS morecommonly had given written feedback to app stores andproduct forums than those who do not use Linux Thefeedback habits of Mac and Windows users were relativelysimilar across all feedback channels The higher feedbackrates of Linux users may be related to the prevalence ofsoftware developers using it In fact 43 of respondentsusing Linux also reported working in the software industrycompared to only 16 of all respondents Our resultsshowed that software professionals are more likely toprovide online feedback possibly because they understandhow that feedback will be used by development teamsFuture research can investigate more thoroughly the reasonsfor differences across devices

Other avenues for future work Investigate otherfeedback channels Our study was limited to app storesproduct forums and social media Future work couldperform a similar investigation considering other feedbackchannels like issue trackers

Replicate survey in other countries Our survey re-spondents were mostly from two countries New Zealandand Germany Future work could replicate our survey byeliciting responses in additional countries This would alsoenable analysis at the ethnicity level if more ethnic diversityin the participants was achieved

Understand gender differences in product forum engage-ment In addition to men being more likely than women topost on product forums men also reported using productsforums for different reasons While men and women bothprimarily used forums to ask software related questionsmen also reported higher rates of giving other types offeedback on product forums including reporting problemsrequesting features praising and criticising the softwareas well as assisting others Further research is neededto understand the gender difference in engagement withproduct forums

Making missing demographics more transparent Cur-rently it is difficult for product development teams toknow whether the feedback collected from online feed-back channels is biased and misses the voices of someunderrepresented groups Future research could deviseways to make this more transparent to enable softwaredevelopment teams to more proactively consider the needsof the underrepresented groups and produce more inclusivesoftware

VI CONCLUSION

The online user feedback written on app stores productforums and social media is a valuable source of require-ments for software developers and has been a focus ofrequirements engineering researchers However limitedstudies have been done to understand which softwareusers give this feedback and what motivates them In thiswork we directly surveyed 1040 software users about

their feedback habits software use and demographicinformation

The responses indicate significant differences in thedemographics of software users who give feedback oneach online channel For gender men reported giving morefeedback than women and with age respondents between35 and 45 reporting to give the most feedback acrossall channels We also found strong evidence that youngersoftware users (under 25) prefer to engage with app storeswhereas older software users use product forums at equal(to app stores) and sometimes higher rates

We identified key differences in what motivates softwareusers to engage with each of the three channels Comparingchannels respondents reported the top motivation to givefeedback on app stores and social media was to showappreciation whereas on forums the most cited motivationwas to get help with software products Differences betweenthe motivations of men and women to give feedbackwere also reported for each of the channels Respondentsreported in app prompts to be significantly more effective inmotivating them to give app ratings over written feedbackAdditionally individual feedback givers reported to engagemore times a year on product forums than on with appstores

Differences in feedback habits were also reported withthe ways respondents use software Those who spendmore hours each day on their phone or computer reportedgiving more feedback about the software theyrsquore using Thesoftware platform being used also presented a relationshipto feedback rates with more Linux (computer) and Android(phone) users reporting to give feedback than those whouse the alternatives

The findings presented in this paper give meaningfulinsights into which software users give online feedbackand the motivations they have to give it We foundnotable differences in those who give feedback to eachonline channel which emphasises the need to mine allthree feedback channels to get the most representativerequirements from software users when leveraging onlinefeedback

REFERENCES

[1] E Guzman R Alkadhi and N Seyff ldquoA needle in a haystackWhat do twitter users say about softwarerdquo in 2016 IEEE 24thInternational Requirements Engineering Conference (RE) Sep2016 pp 96ndash105

[2] D Pagano and W Maalej ldquoUser feedback in the appstore Anempirical studyrdquo in 2013 21st IEEE International RequirementsEngineering Conference (RE) July 2013 pp 125ndash134

[3] J Tizard H Wang L Yohannes and K Blincoe ldquoCan a con-versation paint a picture mining requirements in software fo-rumsrdquo in 2019 IEEE 27th International Requirements EngineeringConference (RE) IEEE 2019 pp 17ndash27

[4] E Guzman R Alkadhi and N Seyff ldquoAn exploratory studyof twitter messages about software applicationsrdquo RequirementsEngineering 07 2017

[5] E Guzman and A P Rojas ldquoGender and user feedback Anexploratory studyrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 381ndash385

[6] E Guzman M Ibrahim and M Glinz ldquoA little bird told me miningtweets for requirements and software evolutionrdquo in 2017 IEEE 25thInternational Requirements Engineering Conference (RE) IEEE2017 pp 11ndash20

[7] W Maalej and H Nabil ldquoBug report feature request orsimply praise on automatically classifying app reviewsrdquo in 2015IEEE 23rd International Requirements Engineering Conference

(RE) vol 00 Aug 2015 pp 116ndash125 [Online] Availabledoiieeecomputersocietyorg101109RE20157320414

[8] A D Sorbo S Panichella C V Alexandru C A Visaggio andG Canfora ldquoSurf Summarizer of user reviews feedbackrdquo in 2017IEEEACM 39th International Conference on Software EngineeringCompanion (ICSE-C) May 2017 pp 55ndash58

[9] E Guzman L Oliveira Y Steiner L C Wagner and M GlinzldquoUser feedback in the app store a cross-cultural studyrdquo in 2018IEEEACM 40th International Conference on Software EngineeringSoftware Engineering in Society (ICSE-SEIS) IEEE 2018 pp13ndash22

[10] E C Groen N Seyff R Ali F Dalpiaz J Doerr E GuzmanM Hosseini J Marco M Oriol A Perini et al ldquoThe crowd inrequirements engineering The landscape and challengesrdquo IEEEsoftware vol 34 no 2 pp 44ndash52 2017

[11] S Panichella A Di Sorbo E Guzman C A Visaggio G Canforaand H C Gall ldquoArdoc App reviews development orientedclassifierrdquo in Proceedings of the 2016 24th ACM SIGSOFTInternational Symposium on Foundations of Software Engineeringser FSE 2016 New York NY USA ACM 2016 pp 1023ndash1027[Online] Available httpdoiacmorg10114529502902983938

[12] N Chen J Lin S C H Hoi X Xiao and B ZhangldquoAr-miner Mining informative reviews for developers frommobile app marketplacerdquo in Proceedings of the 36th InternationalConference on Software Engineering ser ICSE 2014 NewYork NY USA ACM 2014 pp 767ndash778 [Online] Availablehttpdoiacmorg10114525682252568263

[13] N Papadopoulos O M Martiacuten M Cleveland and M Laroche

ldquoIdentity demographics and consumer behaviorsrdquo InternationalMarketing Review 2011

[14] ldquoNew zealand census 2018rdquo httpswwwstatsgovtnzinformation-releases2018-census-population-and-dwelling-countsaccessed December 2019

[15] R Likert ldquoA technique for the measurement of attitudesrdquo Archivesof psychology 1932

[16] U ISCED ldquoInternational standard classification of education 2011rdquo2012

[17] I Etikan ldquoComparison of convenience sampling and purposivesamplingrdquo American Journal of Theoretical and Applied Statisticsvol 5 p 1 01 2016

[18] ldquoQualtrics survey platformrdquo httpswwwqualtricscom accessedDecember 2019

[19] O Bock A Nicklisch and I Baetge ldquohroot Hamburg registrationand organization online toolrdquo in WiSo-HH Working Paper Series2012

[20] M L McHugh ldquoThe chi-square test of independencerdquo Biochemiamedica Biochemia medica vol 23 no 2 pp 143ndash149 2013

[21] M Burnett S Stumpf J Macbeth S Makri L BeckwithI Kwan A Peters and W Jernigan ldquoGendermag A methodfor evaluating softwarersquos gender inclusivenessrdquo Interacting withComputers vol 28 no 6 pp 760ndash787 2016

[22] T Rietz and A Maedche ldquoLadderbot A requirements self-elicitation systemrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 357ndash362

TABLE IXMOTIVATIONS TO GIVE FEEDBACK

App Store () Product Forum () Social Media ()

1 Show appreciation 6515 1 Get help 7037 1 Show appreciation 5676

2 Influence improvement 5202 2 Influence improvement 4429 2 Influence improvement 5135

3 Show dissatisfaction 3485 3 Show appreciation 2643 3 Show dissatisfaction 3784

4 Recommendto others 2980 4 Recommend to others 1786 4 Connect or socialise 3514

5 Discourage others 1263 5 Show dissatisfaction 1643 5 Recommend to others 3243

6 Get help 920 6 Connect or socialise 1572 6 Get help 2273

7 No specific motivation 505 7 No specific motivation 786 7 Discourage others 1486

8 Connect or socialise 152 8 Discourage others 357 8 No specific motivation 811

TABLE XMOTIVATIONS TO GIVE FEEDBACK WITH GENDER

App Store () Product Forums () Social Media ()

Motivation Men Women Men Women Men Women

Showappreciation

6724 7273 2816 1892 5745 5385

Showdissatisfaction

3621 3636 1359 2162 3191 4615

Influenceimprovement

5776 5000 4951 2703 5532 4231

Recommend 2979 3462 3276 3030 1845 1351

Discourage 1638 606 194 541 1064 1923

Connectsocialise

431 758 1359 1892 2766 4231

Get help 1014 588 7111 7778 4118 000No specificmotivation

086 303 971 000 426 1154

Fig 4 Feedback given by individual users each year on each channel

each motivation are shown in table X where notabledifferences are indicated in bold On app stores men weremore motivated to discourage others from using a dislikedapp On product forums more men cited influencing animprovement in the software as a motivation On socialmedia more women were motivated to show dissatisfactionand connect or socialise about a software product Also onsocial media more men cited influence improvement andget help These results are bolded in Table X

Feedback frequency The majority of feedback giversreported having given feedback between 0 and 4 times inthe last year across all channels (Fig 4) App stores hadthe least respondents reporting to give more than 4 piecesof feedback product forums had the most respondentsgiving feedback more than 4 times

Perception of influencing developers Survey re-spondents who believed that software developers woulddefinitely not be influenced by online feedback were lesslikely to give feedback than those who believed influence

TABLE XIUSER FEEDBACK WITH PERCEPTION OF INFLUENCING DEVELOPERS

Number ofRespondents

App Store()

Forums()

Socialmedia()

Definitely will 83 1446 1807 723

Probably will 265 1925 1396 792

Might or might not 416 1875 1490 649

Probably will not 248 1895 968 806

Definitely will not 27 370 741 000

was more likely on all channels However chi-squared testsshowed these differences werenrsquot statistically significantFeedback rates with perception of influencing developersare shown in table XI the lower definitely not values areindicated in bold

Answer to RQ2 Showing appreciation was the topmotivation given to write feedback on app storesand social media On product forums getting helpwas the most commonly cited motivation (table IX)Differences in the motivations of men and womento give written feedback on each channel were alsoreportedIn-app prompts were reported to be very effectiveat motivating app users to give star ratings but lesseffective at eliciting written feedback Individualsurvey respondents reported engaging with eachfeedback channel at different frequencies writingon product forums the most times a year and leaston app stores

C Type of software and duration of use

RQ3 Does the likelihood of giving online writtenfeedback vary based on the type of software used andthe duration of software usage

iPhoneAndroid Android users reported giving feed-back to the app store at a higher rate than iPhone users(Table XII) 1348 of iPhone users reported having givenwritten feedback on app stores compared to 2184 ofAndroid users A chi-squared test showed this difference tobe statistically significant given in table XIII The higherfeedback rate of Android users on app stores has beenindicate in bold in table XII

WindowsMacLinux Linux users reported giving writ-ten feedback on app stores and product forums at a higher

TABLE XIIUSER FEEDBACK WITH DEVICE TYPE

Device Number ofRespondents

App Store()

ProductForum

()

SocialMedia

()Android 618 2184 1375 680

iPhone 423 1348 1277 733

Linux 94 3191 2660 1064

Windows 759 1910 146 646

Mac 275 1673 1236 1018

TABLE XIIIUSER FEEDBACK WITH DEVICE TYPE SIGNIFICANCE TESTS

App Store Product Forums

Chi2 p Chi2 p

Android gt iPhone 11144 0001 0087 0769

Linux gt Windows 7651 0006 8073 0004Linux gt Mac 8974 0003 9531 0002

Statistically significant results are bolded

TABLE XIVUSER FEEDBACK WITH DAILY COMPUTER USE

Daily Computer Use Number ofRespondents

App Store()

Forums()

SocialMedia

()Less than 1 hour 109 1835 1009 642

1 - 4 hours 436 1514 940 596

4 - 8 hours 363 2066 1708 799

More than 8 hours 110 2182 2364 909

rate than Windows and Mac users (Table XII) Chi-squaredtests showed these differences to both be statisticallysignificant (Table XIII) The difference between Windowsand Mac users feedback was not statistically significantThe statistically significant higher feedback rates of Linuxusers are indicated in bold in table XII

Hours of computer use Respondents who reported ahigher daily computer use (hours) were more likely to givefeedback to product forums indicated in bold in table XIVThe least forum feedback was given by respondents usingtheir computer less then 1 hour or between 1 and 4 hoursa day Those using their computer between 4 and 8 hoursgave more feedback and those using their computer morethan 8 hours a day gave at the highest rate Chi-squaredtests showed that the feedback rate differences between 1 -4 hours and 4 - 8 hours and between 1- 4 hours and over8 hours were statistically significant (Table XV)

Hours of phone use Respondents who reported ahigher daily phone use (hours) were more likely togive feedback to social media indicated in bold in tableXVI However chi-squared tests showed these differenceswerenrsquot statistically significant

TABLE XVCOMPUTER DAILY USE SIGNIFICANCE TESTS (PRODUCT FORUMS)

Daily Computer Use Chi2 p

Less than 1 hour lt 1-4 hours 0001 0971

Less than 1 hour lt 4-8 hours 2619 0106

1 - 4 lt Over 8 hours 15233 lt 0001Less than 1 lt Over 8 hours 6221 00131-4 hours lt 4-8 hours 9722 00024-8 hours lt Over 8 hours 1983 0159

Statistically significant results are bolded

TABLE XVIUSER FEEDBACK WITH DAILY PHONE USE

Daily Phone Use Number ofRespondents

App Store()

Forums()

SocialMedia

()Less than 1 hour 52 1538 1538 3851 - 4 hours 664 1657 1370 6634 - 8 hours 266 2293 1165 789More than 8 hours 51 1765 1765 1373

Answer to RQ3 Statistically significant differ-ences were reported in the amount of writtenfeedback given based on the type of software usedand the duration of daily use Respondents whospend more hours each day on their computerreported giving more written feedback to productforums Those using the Linux OS gave morewritten feedback to app stores and product forumsthan those using Windows and Mac Android usersreported giving more written feedback to app storesthan iPhone users

V DISCUSSION

In this section we first discuss the threats to validity ofour study and then describe the implications and potentialavenues for future work

A Threats to validity

Convenience sampling used in this study to elicit surveyparticipants is a non-probabilistic sampling method anda possible source of bias [17] The target population ofthis study are users of software and mobile applicationsSurvey participants were engaged via Facebook Twitterthe Karlsruhe Institute of Technology survey pool and inAuckland cities public areas therefore only a subset ofthe target population had the opportunity to participateAdditionally all respondents who completed the surveywere self-selected and their feedback habits may notgeneralise to all software users

To mitigate this bias we collected data from a largenumber of software users (1040 participants) Recruitmentwas done through three channels social media publicspaces and the hroot survey pool to increase the oddsof recruiting a diverse set of respondents However wecannot claim that our results generalise outside of oursample Future studies can replicate our survey to validateour findings

Our participants are not representative across all demo-graphics The demographics of our respondents are listedin Table II The majority of participants are whiteEuropeanand many are students When presenting our resultswe present proportions based on the total number ofrespondents in each demographic group We also used chi-squared tests to determine significance between differentdemographic groups which accounts for the sample size ofeach population being compared Therefore findings foundto be statistically significant had a sufficient number of re-spondents in each demographic to satisfy the test Howeverdue to a low number of participants in some demographicgroups not all demographics could be analysed Futurestudies should replicate this survey to enable analysis ofadditional demographics

In addition the majority of participants identified asmen or women We did give participants the option toself-specify gender but very few participants chose toself-specify Thus our analysis was limited to only thedifferences between participants who identified as men andwomen Again future studies should replicate the surveyto enable analysis beyond these binary genders

Our results are based only on self-reported feedbackhabits Demographic information of feedback givers is notreadily available on the feedback channels we investigated(often the real name of the writer isnrsquot even given) Thisdata sparsity problem means our findings canrsquot be directlyvalidated against actual feedback data One previous studyby Guzman et al [5] approximated the gender of feedbackgivers on app stores from their usernames Using theseapproximations they found that men were more likely thanwomen to provide feedback on the Apple app store whichis in line with what our respondents reported and supportsour findings

B Implications and Future Work

Implication 1 The findings presented in this papersuggest that when leveraging online user feedback to getthe most representative user views and desires feedbackfrom multiple feedback channels should be consideredWe found statistically significant differences in the userswho reported to give feedback on app stores productforums and social media with respect to traditionaldemographics and software usage habits For exampleolder respondents prefer product forums to app storeswhile younger respondents prefer app stores

Importantly a majority of feedback giving respondentsreported only engaging with one of these three feedbackchannels This indicates that considering multiple channelswill enable feedback from a more diverse set of users

We also found key differences in what motivates softwareusers to engage with each of the three channels The mostcited motivation on app stores and social media was to showappreciation for the appsoftware Whereas on productforums showing appreciation was much less of a motivatingfactor instead getting help was the top cited motivationShowing dissatisfaction recommending and discouragingothers were also significantly more cited on app stores andsocial media On social media connecting with other userswas reported to be a more common motivation than on theother channels

These motivation differences suggest that the feedbackon each online channel is likely to contain different productdevelopment insights For example feedback on productforums contain users trying to get help and therefore likelydescribes ways the software is unintuitive or difficult to useOn app stores and social media users are more motivatedto communicate how they feel about the softwareapp tothe developers and other users These differences againemphasis the benefit to considering feedback from allchannels as each channel may provide unique insights

Implication 2 How to elicit feedback from under-represented groups needs further study We saw somedemographics were less likely to give feedback than othersFor example respondents 35-44 years old report to providethe most feedback on all three feedback channels whileboth older and younger respondents gave less feedback(Fig 2) Also men reported giving feedback at a higher ratethan women across all three channels This is in line withthe results of Guzman et al which found that the Apple appstore had more feedback from men [5] Future work shouldinvestigate why some user groups give more feedback thanothers to understand how to better elicit feedback fromthese underrepresented groups This work could surveyunderrepresented user groups on why they donrsquot givefeedback so that these factors can be directly addressedChanges to the feedback channels themselves may evenbe considered to make the tool more inclusive Recentresearch found that most software has gender inclusivityissues [21] so it is possible that similar inclusivity issuesexist in the software that collects online feedback

Additionally studies could be performed to see ifother means of collecting feedback that augments writtenfeedback on the feedback channels we studied could beused to help provide a more comprehensive set of feedbackacross demographics For example recent research pro-posed using a conversational agent (ladderbot) to conductshort requirements interviews to elicit user feedback [22]Future research could evaluate whether some demographicswould prefer other user feedback collection methods (likeladderbot)

Implication 3 Feedback prompts are effective at elicit-ing feedback for app stores and may be effective if appliedmore widely in computer software However more workis needed to understand how to prompt users to givedetailed feedback Mobile apps widely use prompts to elicitfeedback More survey respondents reported giving writtenfeedback on app stores than on any other channel Much ofthis feedback is prompted The number of respondents whohave provided unprompted app store feedback (1239) isvery similar to the number who report to have written postson product forums (1345) This suggests that the promptsare successful in eliciting additional feedback givers Theprompts are even more effective at eliciting app ratingswhich take less time to provide than written feedbackFuture research could study whether prompts could besuccessful in collecting other types of user feedback andhow they could be integrated into other feedback channels

Implication 4 The types of software devices respondentsuse also has an association to feedback habits Investigatingwhy users of some devices give more feedback may giveinsights into how to motivate and facilitate feedback

On phones more Android users give written feedbackthan iPhone users It is not clear why there are differencesin feedback across devices but it may be influenced bydifferences in prompt rates app quality app store usabilityor even those who choose to use each phone type iOSdevelopers could benefit in understanding these factors inorder to encourage more feedback from their users

On computers respondents who use the Linux OS morecommonly had given written feedback to app stores andproduct forums than those who do not use Linux Thefeedback habits of Mac and Windows users were relativelysimilar across all feedback channels The higher feedbackrates of Linux users may be related to the prevalence ofsoftware developers using it In fact 43 of respondentsusing Linux also reported working in the software industrycompared to only 16 of all respondents Our resultsshowed that software professionals are more likely toprovide online feedback possibly because they understandhow that feedback will be used by development teamsFuture research can investigate more thoroughly the reasonsfor differences across devices

Other avenues for future work Investigate otherfeedback channels Our study was limited to app storesproduct forums and social media Future work couldperform a similar investigation considering other feedbackchannels like issue trackers

Replicate survey in other countries Our survey re-spondents were mostly from two countries New Zealandand Germany Future work could replicate our survey byeliciting responses in additional countries This would alsoenable analysis at the ethnicity level if more ethnic diversityin the participants was achieved

Understand gender differences in product forum engage-ment In addition to men being more likely than women topost on product forums men also reported using productsforums for different reasons While men and women bothprimarily used forums to ask software related questionsmen also reported higher rates of giving other types offeedback on product forums including reporting problemsrequesting features praising and criticising the softwareas well as assisting others Further research is neededto understand the gender difference in engagement withproduct forums

Making missing demographics more transparent Cur-rently it is difficult for product development teams toknow whether the feedback collected from online feed-back channels is biased and misses the voices of someunderrepresented groups Future research could deviseways to make this more transparent to enable softwaredevelopment teams to more proactively consider the needsof the underrepresented groups and produce more inclusivesoftware

VI CONCLUSION

The online user feedback written on app stores productforums and social media is a valuable source of require-ments for software developers and has been a focus ofrequirements engineering researchers However limitedstudies have been done to understand which softwareusers give this feedback and what motivates them In thiswork we directly surveyed 1040 software users about

their feedback habits software use and demographicinformation

The responses indicate significant differences in thedemographics of software users who give feedback oneach online channel For gender men reported giving morefeedback than women and with age respondents between35 and 45 reporting to give the most feedback acrossall channels We also found strong evidence that youngersoftware users (under 25) prefer to engage with app storeswhereas older software users use product forums at equal(to app stores) and sometimes higher rates

We identified key differences in what motivates softwareusers to engage with each of the three channels Comparingchannels respondents reported the top motivation to givefeedback on app stores and social media was to showappreciation whereas on forums the most cited motivationwas to get help with software products Differences betweenthe motivations of men and women to give feedbackwere also reported for each of the channels Respondentsreported in app prompts to be significantly more effective inmotivating them to give app ratings over written feedbackAdditionally individual feedback givers reported to engagemore times a year on product forums than on with appstores

Differences in feedback habits were also reported withthe ways respondents use software Those who spendmore hours each day on their phone or computer reportedgiving more feedback about the software theyrsquore using Thesoftware platform being used also presented a relationshipto feedback rates with more Linux (computer) and Android(phone) users reporting to give feedback than those whouse the alternatives

The findings presented in this paper give meaningfulinsights into which software users give online feedbackand the motivations they have to give it We foundnotable differences in those who give feedback to eachonline channel which emphasises the need to mine allthree feedback channels to get the most representativerequirements from software users when leveraging onlinefeedback

REFERENCES

[1] E Guzman R Alkadhi and N Seyff ldquoA needle in a haystackWhat do twitter users say about softwarerdquo in 2016 IEEE 24thInternational Requirements Engineering Conference (RE) Sep2016 pp 96ndash105

[2] D Pagano and W Maalej ldquoUser feedback in the appstore Anempirical studyrdquo in 2013 21st IEEE International RequirementsEngineering Conference (RE) July 2013 pp 125ndash134

[3] J Tizard H Wang L Yohannes and K Blincoe ldquoCan a con-versation paint a picture mining requirements in software fo-rumsrdquo in 2019 IEEE 27th International Requirements EngineeringConference (RE) IEEE 2019 pp 17ndash27

[4] E Guzman R Alkadhi and N Seyff ldquoAn exploratory studyof twitter messages about software applicationsrdquo RequirementsEngineering 07 2017

[5] E Guzman and A P Rojas ldquoGender and user feedback Anexploratory studyrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 381ndash385

[6] E Guzman M Ibrahim and M Glinz ldquoA little bird told me miningtweets for requirements and software evolutionrdquo in 2017 IEEE 25thInternational Requirements Engineering Conference (RE) IEEE2017 pp 11ndash20

[7] W Maalej and H Nabil ldquoBug report feature request orsimply praise on automatically classifying app reviewsrdquo in 2015IEEE 23rd International Requirements Engineering Conference

(RE) vol 00 Aug 2015 pp 116ndash125 [Online] Availabledoiieeecomputersocietyorg101109RE20157320414

[8] A D Sorbo S Panichella C V Alexandru C A Visaggio andG Canfora ldquoSurf Summarizer of user reviews feedbackrdquo in 2017IEEEACM 39th International Conference on Software EngineeringCompanion (ICSE-C) May 2017 pp 55ndash58

[9] E Guzman L Oliveira Y Steiner L C Wagner and M GlinzldquoUser feedback in the app store a cross-cultural studyrdquo in 2018IEEEACM 40th International Conference on Software EngineeringSoftware Engineering in Society (ICSE-SEIS) IEEE 2018 pp13ndash22

[10] E C Groen N Seyff R Ali F Dalpiaz J Doerr E GuzmanM Hosseini J Marco M Oriol A Perini et al ldquoThe crowd inrequirements engineering The landscape and challengesrdquo IEEEsoftware vol 34 no 2 pp 44ndash52 2017

[11] S Panichella A Di Sorbo E Guzman C A Visaggio G Canforaand H C Gall ldquoArdoc App reviews development orientedclassifierrdquo in Proceedings of the 2016 24th ACM SIGSOFTInternational Symposium on Foundations of Software Engineeringser FSE 2016 New York NY USA ACM 2016 pp 1023ndash1027[Online] Available httpdoiacmorg10114529502902983938

[12] N Chen J Lin S C H Hoi X Xiao and B ZhangldquoAr-miner Mining informative reviews for developers frommobile app marketplacerdquo in Proceedings of the 36th InternationalConference on Software Engineering ser ICSE 2014 NewYork NY USA ACM 2014 pp 767ndash778 [Online] Availablehttpdoiacmorg10114525682252568263

[13] N Papadopoulos O M Martiacuten M Cleveland and M Laroche

ldquoIdentity demographics and consumer behaviorsrdquo InternationalMarketing Review 2011

[14] ldquoNew zealand census 2018rdquo httpswwwstatsgovtnzinformation-releases2018-census-population-and-dwelling-countsaccessed December 2019

[15] R Likert ldquoA technique for the measurement of attitudesrdquo Archivesof psychology 1932

[16] U ISCED ldquoInternational standard classification of education 2011rdquo2012

[17] I Etikan ldquoComparison of convenience sampling and purposivesamplingrdquo American Journal of Theoretical and Applied Statisticsvol 5 p 1 01 2016

[18] ldquoQualtrics survey platformrdquo httpswwwqualtricscom accessedDecember 2019

[19] O Bock A Nicklisch and I Baetge ldquohroot Hamburg registrationand organization online toolrdquo in WiSo-HH Working Paper Series2012

[20] M L McHugh ldquoThe chi-square test of independencerdquo Biochemiamedica Biochemia medica vol 23 no 2 pp 143ndash149 2013

[21] M Burnett S Stumpf J Macbeth S Makri L BeckwithI Kwan A Peters and W Jernigan ldquoGendermag A methodfor evaluating softwarersquos gender inclusivenessrdquo Interacting withComputers vol 28 no 6 pp 760ndash787 2016

[22] T Rietz and A Maedche ldquoLadderbot A requirements self-elicitation systemrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 357ndash362

TABLE XIIUSER FEEDBACK WITH DEVICE TYPE

Device Number ofRespondents

App Store()

ProductForum

()

SocialMedia

()Android 618 2184 1375 680

iPhone 423 1348 1277 733

Linux 94 3191 2660 1064

Windows 759 1910 146 646

Mac 275 1673 1236 1018

TABLE XIIIUSER FEEDBACK WITH DEVICE TYPE SIGNIFICANCE TESTS

App Store Product Forums

Chi2 p Chi2 p

Android gt iPhone 11144 0001 0087 0769

Linux gt Windows 7651 0006 8073 0004Linux gt Mac 8974 0003 9531 0002

Statistically significant results are bolded

TABLE XIVUSER FEEDBACK WITH DAILY COMPUTER USE

Daily Computer Use Number ofRespondents

App Store()

Forums()

SocialMedia

()Less than 1 hour 109 1835 1009 642

1 - 4 hours 436 1514 940 596

4 - 8 hours 363 2066 1708 799

More than 8 hours 110 2182 2364 909

rate than Windows and Mac users (Table XII) Chi-squaredtests showed these differences to both be statisticallysignificant (Table XIII) The difference between Windowsand Mac users feedback was not statistically significantThe statistically significant higher feedback rates of Linuxusers are indicated in bold in table XII

Hours of computer use Respondents who reported ahigher daily computer use (hours) were more likely to givefeedback to product forums indicated in bold in table XIVThe least forum feedback was given by respondents usingtheir computer less then 1 hour or between 1 and 4 hoursa day Those using their computer between 4 and 8 hoursgave more feedback and those using their computer morethan 8 hours a day gave at the highest rate Chi-squaredtests showed that the feedback rate differences between 1 -4 hours and 4 - 8 hours and between 1- 4 hours and over8 hours were statistically significant (Table XV)

Hours of phone use Respondents who reported ahigher daily phone use (hours) were more likely togive feedback to social media indicated in bold in tableXVI However chi-squared tests showed these differenceswerenrsquot statistically significant

TABLE XVCOMPUTER DAILY USE SIGNIFICANCE TESTS (PRODUCT FORUMS)

Daily Computer Use Chi2 p

Less than 1 hour lt 1-4 hours 0001 0971

Less than 1 hour lt 4-8 hours 2619 0106

1 - 4 lt Over 8 hours 15233 lt 0001Less than 1 lt Over 8 hours 6221 00131-4 hours lt 4-8 hours 9722 00024-8 hours lt Over 8 hours 1983 0159

Statistically significant results are bolded

TABLE XVIUSER FEEDBACK WITH DAILY PHONE USE

Daily Phone Use Number ofRespondents

App Store()

Forums()

SocialMedia

()Less than 1 hour 52 1538 1538 3851 - 4 hours 664 1657 1370 6634 - 8 hours 266 2293 1165 789More than 8 hours 51 1765 1765 1373

Answer to RQ3 Statistically significant differ-ences were reported in the amount of writtenfeedback given based on the type of software usedand the duration of daily use Respondents whospend more hours each day on their computerreported giving more written feedback to productforums Those using the Linux OS gave morewritten feedback to app stores and product forumsthan those using Windows and Mac Android usersreported giving more written feedback to app storesthan iPhone users

V DISCUSSION

In this section we first discuss the threats to validity ofour study and then describe the implications and potentialavenues for future work

A Threats to validity

Convenience sampling used in this study to elicit surveyparticipants is a non-probabilistic sampling method anda possible source of bias [17] The target population ofthis study are users of software and mobile applicationsSurvey participants were engaged via Facebook Twitterthe Karlsruhe Institute of Technology survey pool and inAuckland cities public areas therefore only a subset ofthe target population had the opportunity to participateAdditionally all respondents who completed the surveywere self-selected and their feedback habits may notgeneralise to all software users

To mitigate this bias we collected data from a largenumber of software users (1040 participants) Recruitmentwas done through three channels social media publicspaces and the hroot survey pool to increase the oddsof recruiting a diverse set of respondents However wecannot claim that our results generalise outside of oursample Future studies can replicate our survey to validateour findings

Our participants are not representative across all demo-graphics The demographics of our respondents are listedin Table II The majority of participants are whiteEuropeanand many are students When presenting our resultswe present proportions based on the total number ofrespondents in each demographic group We also used chi-squared tests to determine significance between differentdemographic groups which accounts for the sample size ofeach population being compared Therefore findings foundto be statistically significant had a sufficient number of re-spondents in each demographic to satisfy the test Howeverdue to a low number of participants in some demographicgroups not all demographics could be analysed Futurestudies should replicate this survey to enable analysis ofadditional demographics

In addition the majority of participants identified asmen or women We did give participants the option toself-specify gender but very few participants chose toself-specify Thus our analysis was limited to only thedifferences between participants who identified as men andwomen Again future studies should replicate the surveyto enable analysis beyond these binary genders

Our results are based only on self-reported feedbackhabits Demographic information of feedback givers is notreadily available on the feedback channels we investigated(often the real name of the writer isnrsquot even given) Thisdata sparsity problem means our findings canrsquot be directlyvalidated against actual feedback data One previous studyby Guzman et al [5] approximated the gender of feedbackgivers on app stores from their usernames Using theseapproximations they found that men were more likely thanwomen to provide feedback on the Apple app store whichis in line with what our respondents reported and supportsour findings

B Implications and Future Work

Implication 1 The findings presented in this papersuggest that when leveraging online user feedback to getthe most representative user views and desires feedbackfrom multiple feedback channels should be consideredWe found statistically significant differences in the userswho reported to give feedback on app stores productforums and social media with respect to traditionaldemographics and software usage habits For exampleolder respondents prefer product forums to app storeswhile younger respondents prefer app stores

Importantly a majority of feedback giving respondentsreported only engaging with one of these three feedbackchannels This indicates that considering multiple channelswill enable feedback from a more diverse set of users

We also found key differences in what motivates softwareusers to engage with each of the three channels The mostcited motivation on app stores and social media was to showappreciation for the appsoftware Whereas on productforums showing appreciation was much less of a motivatingfactor instead getting help was the top cited motivationShowing dissatisfaction recommending and discouragingothers were also significantly more cited on app stores andsocial media On social media connecting with other userswas reported to be a more common motivation than on theother channels

These motivation differences suggest that the feedbackon each online channel is likely to contain different productdevelopment insights For example feedback on productforums contain users trying to get help and therefore likelydescribes ways the software is unintuitive or difficult to useOn app stores and social media users are more motivatedto communicate how they feel about the softwareapp tothe developers and other users These differences againemphasis the benefit to considering feedback from allchannels as each channel may provide unique insights

Implication 2 How to elicit feedback from under-represented groups needs further study We saw somedemographics were less likely to give feedback than othersFor example respondents 35-44 years old report to providethe most feedback on all three feedback channels whileboth older and younger respondents gave less feedback(Fig 2) Also men reported giving feedback at a higher ratethan women across all three channels This is in line withthe results of Guzman et al which found that the Apple appstore had more feedback from men [5] Future work shouldinvestigate why some user groups give more feedback thanothers to understand how to better elicit feedback fromthese underrepresented groups This work could surveyunderrepresented user groups on why they donrsquot givefeedback so that these factors can be directly addressedChanges to the feedback channels themselves may evenbe considered to make the tool more inclusive Recentresearch found that most software has gender inclusivityissues [21] so it is possible that similar inclusivity issuesexist in the software that collects online feedback

Additionally studies could be performed to see ifother means of collecting feedback that augments writtenfeedback on the feedback channels we studied could beused to help provide a more comprehensive set of feedbackacross demographics For example recent research pro-posed using a conversational agent (ladderbot) to conductshort requirements interviews to elicit user feedback [22]Future research could evaluate whether some demographicswould prefer other user feedback collection methods (likeladderbot)

Implication 3 Feedback prompts are effective at elicit-ing feedback for app stores and may be effective if appliedmore widely in computer software However more workis needed to understand how to prompt users to givedetailed feedback Mobile apps widely use prompts to elicitfeedback More survey respondents reported giving writtenfeedback on app stores than on any other channel Much ofthis feedback is prompted The number of respondents whohave provided unprompted app store feedback (1239) isvery similar to the number who report to have written postson product forums (1345) This suggests that the promptsare successful in eliciting additional feedback givers Theprompts are even more effective at eliciting app ratingswhich take less time to provide than written feedbackFuture research could study whether prompts could besuccessful in collecting other types of user feedback andhow they could be integrated into other feedback channels

Implication 4 The types of software devices respondentsuse also has an association to feedback habits Investigatingwhy users of some devices give more feedback may giveinsights into how to motivate and facilitate feedback

On phones more Android users give written feedbackthan iPhone users It is not clear why there are differencesin feedback across devices but it may be influenced bydifferences in prompt rates app quality app store usabilityor even those who choose to use each phone type iOSdevelopers could benefit in understanding these factors inorder to encourage more feedback from their users

On computers respondents who use the Linux OS morecommonly had given written feedback to app stores andproduct forums than those who do not use Linux Thefeedback habits of Mac and Windows users were relativelysimilar across all feedback channels The higher feedbackrates of Linux users may be related to the prevalence ofsoftware developers using it In fact 43 of respondentsusing Linux also reported working in the software industrycompared to only 16 of all respondents Our resultsshowed that software professionals are more likely toprovide online feedback possibly because they understandhow that feedback will be used by development teamsFuture research can investigate more thoroughly the reasonsfor differences across devices

Other avenues for future work Investigate otherfeedback channels Our study was limited to app storesproduct forums and social media Future work couldperform a similar investigation considering other feedbackchannels like issue trackers

Replicate survey in other countries Our survey re-spondents were mostly from two countries New Zealandand Germany Future work could replicate our survey byeliciting responses in additional countries This would alsoenable analysis at the ethnicity level if more ethnic diversityin the participants was achieved

Understand gender differences in product forum engage-ment In addition to men being more likely than women topost on product forums men also reported using productsforums for different reasons While men and women bothprimarily used forums to ask software related questionsmen also reported higher rates of giving other types offeedback on product forums including reporting problemsrequesting features praising and criticising the softwareas well as assisting others Further research is neededto understand the gender difference in engagement withproduct forums

Making missing demographics more transparent Cur-rently it is difficult for product development teams toknow whether the feedback collected from online feed-back channels is biased and misses the voices of someunderrepresented groups Future research could deviseways to make this more transparent to enable softwaredevelopment teams to more proactively consider the needsof the underrepresented groups and produce more inclusivesoftware

VI CONCLUSION

The online user feedback written on app stores productforums and social media is a valuable source of require-ments for software developers and has been a focus ofrequirements engineering researchers However limitedstudies have been done to understand which softwareusers give this feedback and what motivates them In thiswork we directly surveyed 1040 software users about

their feedback habits software use and demographicinformation

The responses indicate significant differences in thedemographics of software users who give feedback oneach online channel For gender men reported giving morefeedback than women and with age respondents between35 and 45 reporting to give the most feedback acrossall channels We also found strong evidence that youngersoftware users (under 25) prefer to engage with app storeswhereas older software users use product forums at equal(to app stores) and sometimes higher rates

We identified key differences in what motivates softwareusers to engage with each of the three channels Comparingchannels respondents reported the top motivation to givefeedback on app stores and social media was to showappreciation whereas on forums the most cited motivationwas to get help with software products Differences betweenthe motivations of men and women to give feedbackwere also reported for each of the channels Respondentsreported in app prompts to be significantly more effective inmotivating them to give app ratings over written feedbackAdditionally individual feedback givers reported to engagemore times a year on product forums than on with appstores

Differences in feedback habits were also reported withthe ways respondents use software Those who spendmore hours each day on their phone or computer reportedgiving more feedback about the software theyrsquore using Thesoftware platform being used also presented a relationshipto feedback rates with more Linux (computer) and Android(phone) users reporting to give feedback than those whouse the alternatives

The findings presented in this paper give meaningfulinsights into which software users give online feedbackand the motivations they have to give it We foundnotable differences in those who give feedback to eachonline channel which emphasises the need to mine allthree feedback channels to get the most representativerequirements from software users when leveraging onlinefeedback

REFERENCES

[1] E Guzman R Alkadhi and N Seyff ldquoA needle in a haystackWhat do twitter users say about softwarerdquo in 2016 IEEE 24thInternational Requirements Engineering Conference (RE) Sep2016 pp 96ndash105

[2] D Pagano and W Maalej ldquoUser feedback in the appstore Anempirical studyrdquo in 2013 21st IEEE International RequirementsEngineering Conference (RE) July 2013 pp 125ndash134

[3] J Tizard H Wang L Yohannes and K Blincoe ldquoCan a con-versation paint a picture mining requirements in software fo-rumsrdquo in 2019 IEEE 27th International Requirements EngineeringConference (RE) IEEE 2019 pp 17ndash27

[4] E Guzman R Alkadhi and N Seyff ldquoAn exploratory studyof twitter messages about software applicationsrdquo RequirementsEngineering 07 2017

[5] E Guzman and A P Rojas ldquoGender and user feedback Anexploratory studyrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 381ndash385

[6] E Guzman M Ibrahim and M Glinz ldquoA little bird told me miningtweets for requirements and software evolutionrdquo in 2017 IEEE 25thInternational Requirements Engineering Conference (RE) IEEE2017 pp 11ndash20

[7] W Maalej and H Nabil ldquoBug report feature request orsimply praise on automatically classifying app reviewsrdquo in 2015IEEE 23rd International Requirements Engineering Conference

(RE) vol 00 Aug 2015 pp 116ndash125 [Online] Availabledoiieeecomputersocietyorg101109RE20157320414

[8] A D Sorbo S Panichella C V Alexandru C A Visaggio andG Canfora ldquoSurf Summarizer of user reviews feedbackrdquo in 2017IEEEACM 39th International Conference on Software EngineeringCompanion (ICSE-C) May 2017 pp 55ndash58

[9] E Guzman L Oliveira Y Steiner L C Wagner and M GlinzldquoUser feedback in the app store a cross-cultural studyrdquo in 2018IEEEACM 40th International Conference on Software EngineeringSoftware Engineering in Society (ICSE-SEIS) IEEE 2018 pp13ndash22

[10] E C Groen N Seyff R Ali F Dalpiaz J Doerr E GuzmanM Hosseini J Marco M Oriol A Perini et al ldquoThe crowd inrequirements engineering The landscape and challengesrdquo IEEEsoftware vol 34 no 2 pp 44ndash52 2017

[11] S Panichella A Di Sorbo E Guzman C A Visaggio G Canforaand H C Gall ldquoArdoc App reviews development orientedclassifierrdquo in Proceedings of the 2016 24th ACM SIGSOFTInternational Symposium on Foundations of Software Engineeringser FSE 2016 New York NY USA ACM 2016 pp 1023ndash1027[Online] Available httpdoiacmorg10114529502902983938

[12] N Chen J Lin S C H Hoi X Xiao and B ZhangldquoAr-miner Mining informative reviews for developers frommobile app marketplacerdquo in Proceedings of the 36th InternationalConference on Software Engineering ser ICSE 2014 NewYork NY USA ACM 2014 pp 767ndash778 [Online] Availablehttpdoiacmorg10114525682252568263

[13] N Papadopoulos O M Martiacuten M Cleveland and M Laroche

ldquoIdentity demographics and consumer behaviorsrdquo InternationalMarketing Review 2011

[14] ldquoNew zealand census 2018rdquo httpswwwstatsgovtnzinformation-releases2018-census-population-and-dwelling-countsaccessed December 2019

[15] R Likert ldquoA technique for the measurement of attitudesrdquo Archivesof psychology 1932

[16] U ISCED ldquoInternational standard classification of education 2011rdquo2012

[17] I Etikan ldquoComparison of convenience sampling and purposivesamplingrdquo American Journal of Theoretical and Applied Statisticsvol 5 p 1 01 2016

[18] ldquoQualtrics survey platformrdquo httpswwwqualtricscom accessedDecember 2019

[19] O Bock A Nicklisch and I Baetge ldquohroot Hamburg registrationand organization online toolrdquo in WiSo-HH Working Paper Series2012

[20] M L McHugh ldquoThe chi-square test of independencerdquo Biochemiamedica Biochemia medica vol 23 no 2 pp 143ndash149 2013

[21] M Burnett S Stumpf J Macbeth S Makri L BeckwithI Kwan A Peters and W Jernigan ldquoGendermag A methodfor evaluating softwarersquos gender inclusivenessrdquo Interacting withComputers vol 28 no 6 pp 760ndash787 2016

[22] T Rietz and A Maedche ldquoLadderbot A requirements self-elicitation systemrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 357ndash362

Our participants are not representative across all demo-graphics The demographics of our respondents are listedin Table II The majority of participants are whiteEuropeanand many are students When presenting our resultswe present proportions based on the total number ofrespondents in each demographic group We also used chi-squared tests to determine significance between differentdemographic groups which accounts for the sample size ofeach population being compared Therefore findings foundto be statistically significant had a sufficient number of re-spondents in each demographic to satisfy the test Howeverdue to a low number of participants in some demographicgroups not all demographics could be analysed Futurestudies should replicate this survey to enable analysis ofadditional demographics

In addition the majority of participants identified asmen or women We did give participants the option toself-specify gender but very few participants chose toself-specify Thus our analysis was limited to only thedifferences between participants who identified as men andwomen Again future studies should replicate the surveyto enable analysis beyond these binary genders

Our results are based only on self-reported feedbackhabits Demographic information of feedback givers is notreadily available on the feedback channels we investigated(often the real name of the writer isnrsquot even given) Thisdata sparsity problem means our findings canrsquot be directlyvalidated against actual feedback data One previous studyby Guzman et al [5] approximated the gender of feedbackgivers on app stores from their usernames Using theseapproximations they found that men were more likely thanwomen to provide feedback on the Apple app store whichis in line with what our respondents reported and supportsour findings

B Implications and Future Work

Implication 1 The findings presented in this papersuggest that when leveraging online user feedback to getthe most representative user views and desires feedbackfrom multiple feedback channels should be consideredWe found statistically significant differences in the userswho reported to give feedback on app stores productforums and social media with respect to traditionaldemographics and software usage habits For exampleolder respondents prefer product forums to app storeswhile younger respondents prefer app stores

Importantly a majority of feedback giving respondentsreported only engaging with one of these three feedbackchannels This indicates that considering multiple channelswill enable feedback from a more diverse set of users

We also found key differences in what motivates softwareusers to engage with each of the three channels The mostcited motivation on app stores and social media was to showappreciation for the appsoftware Whereas on productforums showing appreciation was much less of a motivatingfactor instead getting help was the top cited motivationShowing dissatisfaction recommending and discouragingothers were also significantly more cited on app stores andsocial media On social media connecting with other userswas reported to be a more common motivation than on theother channels

These motivation differences suggest that the feedbackon each online channel is likely to contain different productdevelopment insights For example feedback on productforums contain users trying to get help and therefore likelydescribes ways the software is unintuitive or difficult to useOn app stores and social media users are more motivatedto communicate how they feel about the softwareapp tothe developers and other users These differences againemphasis the benefit to considering feedback from allchannels as each channel may provide unique insights

Implication 2 How to elicit feedback from under-represented groups needs further study We saw somedemographics were less likely to give feedback than othersFor example respondents 35-44 years old report to providethe most feedback on all three feedback channels whileboth older and younger respondents gave less feedback(Fig 2) Also men reported giving feedback at a higher ratethan women across all three channels This is in line withthe results of Guzman et al which found that the Apple appstore had more feedback from men [5] Future work shouldinvestigate why some user groups give more feedback thanothers to understand how to better elicit feedback fromthese underrepresented groups This work could surveyunderrepresented user groups on why they donrsquot givefeedback so that these factors can be directly addressedChanges to the feedback channels themselves may evenbe considered to make the tool more inclusive Recentresearch found that most software has gender inclusivityissues [21] so it is possible that similar inclusivity issuesexist in the software that collects online feedback

Additionally studies could be performed to see ifother means of collecting feedback that augments writtenfeedback on the feedback channels we studied could beused to help provide a more comprehensive set of feedbackacross demographics For example recent research pro-posed using a conversational agent (ladderbot) to conductshort requirements interviews to elicit user feedback [22]Future research could evaluate whether some demographicswould prefer other user feedback collection methods (likeladderbot)

Implication 3 Feedback prompts are effective at elicit-ing feedback for app stores and may be effective if appliedmore widely in computer software However more workis needed to understand how to prompt users to givedetailed feedback Mobile apps widely use prompts to elicitfeedback More survey respondents reported giving writtenfeedback on app stores than on any other channel Much ofthis feedback is prompted The number of respondents whohave provided unprompted app store feedback (1239) isvery similar to the number who report to have written postson product forums (1345) This suggests that the promptsare successful in eliciting additional feedback givers Theprompts are even more effective at eliciting app ratingswhich take less time to provide than written feedbackFuture research could study whether prompts could besuccessful in collecting other types of user feedback andhow they could be integrated into other feedback channels

Implication 4 The types of software devices respondentsuse also has an association to feedback habits Investigatingwhy users of some devices give more feedback may giveinsights into how to motivate and facilitate feedback

On phones more Android users give written feedbackthan iPhone users It is not clear why there are differencesin feedback across devices but it may be influenced bydifferences in prompt rates app quality app store usabilityor even those who choose to use each phone type iOSdevelopers could benefit in understanding these factors inorder to encourage more feedback from their users

On computers respondents who use the Linux OS morecommonly had given written feedback to app stores andproduct forums than those who do not use Linux Thefeedback habits of Mac and Windows users were relativelysimilar across all feedback channels The higher feedbackrates of Linux users may be related to the prevalence ofsoftware developers using it In fact 43 of respondentsusing Linux also reported working in the software industrycompared to only 16 of all respondents Our resultsshowed that software professionals are more likely toprovide online feedback possibly because they understandhow that feedback will be used by development teamsFuture research can investigate more thoroughly the reasonsfor differences across devices

Other avenues for future work Investigate otherfeedback channels Our study was limited to app storesproduct forums and social media Future work couldperform a similar investigation considering other feedbackchannels like issue trackers

Replicate survey in other countries Our survey re-spondents were mostly from two countries New Zealandand Germany Future work could replicate our survey byeliciting responses in additional countries This would alsoenable analysis at the ethnicity level if more ethnic diversityin the participants was achieved

Understand gender differences in product forum engage-ment In addition to men being more likely than women topost on product forums men also reported using productsforums for different reasons While men and women bothprimarily used forums to ask software related questionsmen also reported higher rates of giving other types offeedback on product forums including reporting problemsrequesting features praising and criticising the softwareas well as assisting others Further research is neededto understand the gender difference in engagement withproduct forums

Making missing demographics more transparent Cur-rently it is difficult for product development teams toknow whether the feedback collected from online feed-back channels is biased and misses the voices of someunderrepresented groups Future research could deviseways to make this more transparent to enable softwaredevelopment teams to more proactively consider the needsof the underrepresented groups and produce more inclusivesoftware

VI CONCLUSION

The online user feedback written on app stores productforums and social media is a valuable source of require-ments for software developers and has been a focus ofrequirements engineering researchers However limitedstudies have been done to understand which softwareusers give this feedback and what motivates them In thiswork we directly surveyed 1040 software users about

their feedback habits software use and demographicinformation

The responses indicate significant differences in thedemographics of software users who give feedback oneach online channel For gender men reported giving morefeedback than women and with age respondents between35 and 45 reporting to give the most feedback acrossall channels We also found strong evidence that youngersoftware users (under 25) prefer to engage with app storeswhereas older software users use product forums at equal(to app stores) and sometimes higher rates

We identified key differences in what motivates softwareusers to engage with each of the three channels Comparingchannels respondents reported the top motivation to givefeedback on app stores and social media was to showappreciation whereas on forums the most cited motivationwas to get help with software products Differences betweenthe motivations of men and women to give feedbackwere also reported for each of the channels Respondentsreported in app prompts to be significantly more effective inmotivating them to give app ratings over written feedbackAdditionally individual feedback givers reported to engagemore times a year on product forums than on with appstores

Differences in feedback habits were also reported withthe ways respondents use software Those who spendmore hours each day on their phone or computer reportedgiving more feedback about the software theyrsquore using Thesoftware platform being used also presented a relationshipto feedback rates with more Linux (computer) and Android(phone) users reporting to give feedback than those whouse the alternatives

The findings presented in this paper give meaningfulinsights into which software users give online feedbackand the motivations they have to give it We foundnotable differences in those who give feedback to eachonline channel which emphasises the need to mine allthree feedback channels to get the most representativerequirements from software users when leveraging onlinefeedback

REFERENCES

[1] E Guzman R Alkadhi and N Seyff ldquoA needle in a haystackWhat do twitter users say about softwarerdquo in 2016 IEEE 24thInternational Requirements Engineering Conference (RE) Sep2016 pp 96ndash105

[2] D Pagano and W Maalej ldquoUser feedback in the appstore Anempirical studyrdquo in 2013 21st IEEE International RequirementsEngineering Conference (RE) July 2013 pp 125ndash134

[3] J Tizard H Wang L Yohannes and K Blincoe ldquoCan a con-versation paint a picture mining requirements in software fo-rumsrdquo in 2019 IEEE 27th International Requirements EngineeringConference (RE) IEEE 2019 pp 17ndash27

[4] E Guzman R Alkadhi and N Seyff ldquoAn exploratory studyof twitter messages about software applicationsrdquo RequirementsEngineering 07 2017

[5] E Guzman and A P Rojas ldquoGender and user feedback Anexploratory studyrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 381ndash385

[6] E Guzman M Ibrahim and M Glinz ldquoA little bird told me miningtweets for requirements and software evolutionrdquo in 2017 IEEE 25thInternational Requirements Engineering Conference (RE) IEEE2017 pp 11ndash20

[7] W Maalej and H Nabil ldquoBug report feature request orsimply praise on automatically classifying app reviewsrdquo in 2015IEEE 23rd International Requirements Engineering Conference

(RE) vol 00 Aug 2015 pp 116ndash125 [Online] Availabledoiieeecomputersocietyorg101109RE20157320414

[8] A D Sorbo S Panichella C V Alexandru C A Visaggio andG Canfora ldquoSurf Summarizer of user reviews feedbackrdquo in 2017IEEEACM 39th International Conference on Software EngineeringCompanion (ICSE-C) May 2017 pp 55ndash58

[9] E Guzman L Oliveira Y Steiner L C Wagner and M GlinzldquoUser feedback in the app store a cross-cultural studyrdquo in 2018IEEEACM 40th International Conference on Software EngineeringSoftware Engineering in Society (ICSE-SEIS) IEEE 2018 pp13ndash22

[10] E C Groen N Seyff R Ali F Dalpiaz J Doerr E GuzmanM Hosseini J Marco M Oriol A Perini et al ldquoThe crowd inrequirements engineering The landscape and challengesrdquo IEEEsoftware vol 34 no 2 pp 44ndash52 2017

[11] S Panichella A Di Sorbo E Guzman C A Visaggio G Canforaand H C Gall ldquoArdoc App reviews development orientedclassifierrdquo in Proceedings of the 2016 24th ACM SIGSOFTInternational Symposium on Foundations of Software Engineeringser FSE 2016 New York NY USA ACM 2016 pp 1023ndash1027[Online] Available httpdoiacmorg10114529502902983938

[12] N Chen J Lin S C H Hoi X Xiao and B ZhangldquoAr-miner Mining informative reviews for developers frommobile app marketplacerdquo in Proceedings of the 36th InternationalConference on Software Engineering ser ICSE 2014 NewYork NY USA ACM 2014 pp 767ndash778 [Online] Availablehttpdoiacmorg10114525682252568263

[13] N Papadopoulos O M Martiacuten M Cleveland and M Laroche

ldquoIdentity demographics and consumer behaviorsrdquo InternationalMarketing Review 2011

[14] ldquoNew zealand census 2018rdquo httpswwwstatsgovtnzinformation-releases2018-census-population-and-dwelling-countsaccessed December 2019

[15] R Likert ldquoA technique for the measurement of attitudesrdquo Archivesof psychology 1932

[16] U ISCED ldquoInternational standard classification of education 2011rdquo2012

[17] I Etikan ldquoComparison of convenience sampling and purposivesamplingrdquo American Journal of Theoretical and Applied Statisticsvol 5 p 1 01 2016

[18] ldquoQualtrics survey platformrdquo httpswwwqualtricscom accessedDecember 2019

[19] O Bock A Nicklisch and I Baetge ldquohroot Hamburg registrationand organization online toolrdquo in WiSo-HH Working Paper Series2012

[20] M L McHugh ldquoThe chi-square test of independencerdquo Biochemiamedica Biochemia medica vol 23 no 2 pp 143ndash149 2013

[21] M Burnett S Stumpf J Macbeth S Makri L BeckwithI Kwan A Peters and W Jernigan ldquoGendermag A methodfor evaluating softwarersquos gender inclusivenessrdquo Interacting withComputers vol 28 no 6 pp 760ndash787 2016

[22] T Rietz and A Maedche ldquoLadderbot A requirements self-elicitation systemrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 357ndash362

On phones more Android users give written feedbackthan iPhone users It is not clear why there are differencesin feedback across devices but it may be influenced bydifferences in prompt rates app quality app store usabilityor even those who choose to use each phone type iOSdevelopers could benefit in understanding these factors inorder to encourage more feedback from their users

On computers respondents who use the Linux OS morecommonly had given written feedback to app stores andproduct forums than those who do not use Linux Thefeedback habits of Mac and Windows users were relativelysimilar across all feedback channels The higher feedbackrates of Linux users may be related to the prevalence ofsoftware developers using it In fact 43 of respondentsusing Linux also reported working in the software industrycompared to only 16 of all respondents Our resultsshowed that software professionals are more likely toprovide online feedback possibly because they understandhow that feedback will be used by development teamsFuture research can investigate more thoroughly the reasonsfor differences across devices

Other avenues for future work Investigate otherfeedback channels Our study was limited to app storesproduct forums and social media Future work couldperform a similar investigation considering other feedbackchannels like issue trackers

Replicate survey in other countries Our survey re-spondents were mostly from two countries New Zealandand Germany Future work could replicate our survey byeliciting responses in additional countries This would alsoenable analysis at the ethnicity level if more ethnic diversityin the participants was achieved

Understand gender differences in product forum engage-ment In addition to men being more likely than women topost on product forums men also reported using productsforums for different reasons While men and women bothprimarily used forums to ask software related questionsmen also reported higher rates of giving other types offeedback on product forums including reporting problemsrequesting features praising and criticising the softwareas well as assisting others Further research is neededto understand the gender difference in engagement withproduct forums

Making missing demographics more transparent Cur-rently it is difficult for product development teams toknow whether the feedback collected from online feed-back channels is biased and misses the voices of someunderrepresented groups Future research could deviseways to make this more transparent to enable softwaredevelopment teams to more proactively consider the needsof the underrepresented groups and produce more inclusivesoftware

VI CONCLUSION

The online user feedback written on app stores productforums and social media is a valuable source of require-ments for software developers and has been a focus ofrequirements engineering researchers However limitedstudies have been done to understand which softwareusers give this feedback and what motivates them In thiswork we directly surveyed 1040 software users about

their feedback habits software use and demographicinformation

The responses indicate significant differences in thedemographics of software users who give feedback oneach online channel For gender men reported giving morefeedback than women and with age respondents between35 and 45 reporting to give the most feedback acrossall channels We also found strong evidence that youngersoftware users (under 25) prefer to engage with app storeswhereas older software users use product forums at equal(to app stores) and sometimes higher rates

We identified key differences in what motivates softwareusers to engage with each of the three channels Comparingchannels respondents reported the top motivation to givefeedback on app stores and social media was to showappreciation whereas on forums the most cited motivationwas to get help with software products Differences betweenthe motivations of men and women to give feedbackwere also reported for each of the channels Respondentsreported in app prompts to be significantly more effective inmotivating them to give app ratings over written feedbackAdditionally individual feedback givers reported to engagemore times a year on product forums than on with appstores

Differences in feedback habits were also reported withthe ways respondents use software Those who spendmore hours each day on their phone or computer reportedgiving more feedback about the software theyrsquore using Thesoftware platform being used also presented a relationshipto feedback rates with more Linux (computer) and Android(phone) users reporting to give feedback than those whouse the alternatives

The findings presented in this paper give meaningfulinsights into which software users give online feedbackand the motivations they have to give it We foundnotable differences in those who give feedback to eachonline channel which emphasises the need to mine allthree feedback channels to get the most representativerequirements from software users when leveraging onlinefeedback

REFERENCES

[1] E Guzman R Alkadhi and N Seyff ldquoA needle in a haystackWhat do twitter users say about softwarerdquo in 2016 IEEE 24thInternational Requirements Engineering Conference (RE) Sep2016 pp 96ndash105

[2] D Pagano and W Maalej ldquoUser feedback in the appstore Anempirical studyrdquo in 2013 21st IEEE International RequirementsEngineering Conference (RE) July 2013 pp 125ndash134

[3] J Tizard H Wang L Yohannes and K Blincoe ldquoCan a con-versation paint a picture mining requirements in software fo-rumsrdquo in 2019 IEEE 27th International Requirements EngineeringConference (RE) IEEE 2019 pp 17ndash27

[4] E Guzman R Alkadhi and N Seyff ldquoAn exploratory studyof twitter messages about software applicationsrdquo RequirementsEngineering 07 2017

[5] E Guzman and A P Rojas ldquoGender and user feedback Anexploratory studyrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 381ndash385

[6] E Guzman M Ibrahim and M Glinz ldquoA little bird told me miningtweets for requirements and software evolutionrdquo in 2017 IEEE 25thInternational Requirements Engineering Conference (RE) IEEE2017 pp 11ndash20

[7] W Maalej and H Nabil ldquoBug report feature request orsimply praise on automatically classifying app reviewsrdquo in 2015IEEE 23rd International Requirements Engineering Conference

(RE) vol 00 Aug 2015 pp 116ndash125 [Online] Availabledoiieeecomputersocietyorg101109RE20157320414

[8] A D Sorbo S Panichella C V Alexandru C A Visaggio andG Canfora ldquoSurf Summarizer of user reviews feedbackrdquo in 2017IEEEACM 39th International Conference on Software EngineeringCompanion (ICSE-C) May 2017 pp 55ndash58

[9] E Guzman L Oliveira Y Steiner L C Wagner and M GlinzldquoUser feedback in the app store a cross-cultural studyrdquo in 2018IEEEACM 40th International Conference on Software EngineeringSoftware Engineering in Society (ICSE-SEIS) IEEE 2018 pp13ndash22

[10] E C Groen N Seyff R Ali F Dalpiaz J Doerr E GuzmanM Hosseini J Marco M Oriol A Perini et al ldquoThe crowd inrequirements engineering The landscape and challengesrdquo IEEEsoftware vol 34 no 2 pp 44ndash52 2017

[11] S Panichella A Di Sorbo E Guzman C A Visaggio G Canforaand H C Gall ldquoArdoc App reviews development orientedclassifierrdquo in Proceedings of the 2016 24th ACM SIGSOFTInternational Symposium on Foundations of Software Engineeringser FSE 2016 New York NY USA ACM 2016 pp 1023ndash1027[Online] Available httpdoiacmorg10114529502902983938

[12] N Chen J Lin S C H Hoi X Xiao and B ZhangldquoAr-miner Mining informative reviews for developers frommobile app marketplacerdquo in Proceedings of the 36th InternationalConference on Software Engineering ser ICSE 2014 NewYork NY USA ACM 2014 pp 767ndash778 [Online] Availablehttpdoiacmorg10114525682252568263

[13] N Papadopoulos O M Martiacuten M Cleveland and M Laroche

ldquoIdentity demographics and consumer behaviorsrdquo InternationalMarketing Review 2011

[14] ldquoNew zealand census 2018rdquo httpswwwstatsgovtnzinformation-releases2018-census-population-and-dwelling-countsaccessed December 2019

[15] R Likert ldquoA technique for the measurement of attitudesrdquo Archivesof psychology 1932

[16] U ISCED ldquoInternational standard classification of education 2011rdquo2012

[17] I Etikan ldquoComparison of convenience sampling and purposivesamplingrdquo American Journal of Theoretical and Applied Statisticsvol 5 p 1 01 2016

[18] ldquoQualtrics survey platformrdquo httpswwwqualtricscom accessedDecember 2019

[19] O Bock A Nicklisch and I Baetge ldquohroot Hamburg registrationand organization online toolrdquo in WiSo-HH Working Paper Series2012

[20] M L McHugh ldquoThe chi-square test of independencerdquo Biochemiamedica Biochemia medica vol 23 no 2 pp 143ndash149 2013

[21] M Burnett S Stumpf J Macbeth S Makri L BeckwithI Kwan A Peters and W Jernigan ldquoGendermag A methodfor evaluating softwarersquos gender inclusivenessrdquo Interacting withComputers vol 28 no 6 pp 760ndash787 2016

[22] T Rietz and A Maedche ldquoLadderbot A requirements self-elicitation systemrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 357ndash362

(RE) vol 00 Aug 2015 pp 116ndash125 [Online] Availabledoiieeecomputersocietyorg101109RE20157320414

[8] A D Sorbo S Panichella C V Alexandru C A Visaggio andG Canfora ldquoSurf Summarizer of user reviews feedbackrdquo in 2017IEEEACM 39th International Conference on Software EngineeringCompanion (ICSE-C) May 2017 pp 55ndash58

[9] E Guzman L Oliveira Y Steiner L C Wagner and M GlinzldquoUser feedback in the app store a cross-cultural studyrdquo in 2018IEEEACM 40th International Conference on Software EngineeringSoftware Engineering in Society (ICSE-SEIS) IEEE 2018 pp13ndash22

[10] E C Groen N Seyff R Ali F Dalpiaz J Doerr E GuzmanM Hosseini J Marco M Oriol A Perini et al ldquoThe crowd inrequirements engineering The landscape and challengesrdquo IEEEsoftware vol 34 no 2 pp 44ndash52 2017

[11] S Panichella A Di Sorbo E Guzman C A Visaggio G Canforaand H C Gall ldquoArdoc App reviews development orientedclassifierrdquo in Proceedings of the 2016 24th ACM SIGSOFTInternational Symposium on Foundations of Software Engineeringser FSE 2016 New York NY USA ACM 2016 pp 1023ndash1027[Online] Available httpdoiacmorg10114529502902983938

[12] N Chen J Lin S C H Hoi X Xiao and B ZhangldquoAr-miner Mining informative reviews for developers frommobile app marketplacerdquo in Proceedings of the 36th InternationalConference on Software Engineering ser ICSE 2014 NewYork NY USA ACM 2014 pp 767ndash778 [Online] Availablehttpdoiacmorg10114525682252568263

[13] N Papadopoulos O M Martiacuten M Cleveland and M Laroche

ldquoIdentity demographics and consumer behaviorsrdquo InternationalMarketing Review 2011

[14] ldquoNew zealand census 2018rdquo httpswwwstatsgovtnzinformation-releases2018-census-population-and-dwelling-countsaccessed December 2019

[15] R Likert ldquoA technique for the measurement of attitudesrdquo Archivesof psychology 1932

[16] U ISCED ldquoInternational standard classification of education 2011rdquo2012

[17] I Etikan ldquoComparison of convenience sampling and purposivesamplingrdquo American Journal of Theoretical and Applied Statisticsvol 5 p 1 01 2016

[18] ldquoQualtrics survey platformrdquo httpswwwqualtricscom accessedDecember 2019

[19] O Bock A Nicklisch and I Baetge ldquohroot Hamburg registrationand organization online toolrdquo in WiSo-HH Working Paper Series2012

[20] M L McHugh ldquoThe chi-square test of independencerdquo Biochemiamedica Biochemia medica vol 23 no 2 pp 143ndash149 2013

[21] M Burnett S Stumpf J Macbeth S Makri L BeckwithI Kwan A Peters and W Jernigan ldquoGendermag A methodfor evaluating softwarersquos gender inclusivenessrdquo Interacting withComputers vol 28 no 6 pp 760ndash787 2016

[22] T Rietz and A Maedche ldquoLadderbot A requirements self-elicitation systemrdquo in 2019 IEEE 27th International RequirementsEngineering Conference (RE) IEEE 2019 pp 357ndash362