icse 2016 - overcoming open source project entry barriers with a portal for newcomers

Overcoming Open Source Project Entry Barriers with a Portal for

Newcomers

Igor Steinmacher

Tayana Conte

Christoph Treude

Marco A. Gerosa

“Newcomers are explorers who must orient themselves within an unfamiliar landscape...”

[Dagenais et al. 2010]

B. Dagenais, H. Ossher, R.K.E Bellamy, M.P. Robillard and J.P. de Vries, Moving into a new software project landscape, in ICSE 2010.

Context “I received the project that I need to contribute to. I don’t know how to start.”

“The information I found in the project website are long and confusing. I felt really lost and concerned.”

2

82% dropped out [Steinmacher et al., CHASE 2013]

49% performed a single contribution

[Pinto et al., SANER 2016]

Bar

riers

Mod

el3

Steinmacher, I., Conte, T., Gerosa, M.A., Redmiles, D.F. ”Social Barriers Faced by Newcomers Placing Their First Contribution in Open Source Software Projects, In: ACM CSCW 2015

Bar

riers

Mod

el4

Steinmacher, I., Conte, T., Gerosa, M.A., Redmiles, D.F. ”Social Barriers Faced by Newcomers Placing Their First Contribution in Open Source Software Projects, In: ACM CSCW 2015

Goal

Propose and evaluate FLOSScoach, a web portal created to support the first contribution of newcomers to OSS

projects

5

Mapping the barriers model6

• Q1. How do newcomers use the portal to overcome contribution barriers?

• Q2. Does the use of the portal impact newcomers’ self-efficacy?

• Q3. What is the perceived usefulness, ease of use, and likely future use?

Research Questions

7

• 65 students (from 2 universities)▫ Case and Control groups

• Q1. How do newcomers use the portal to overcome contribution barriers?▫ Diary study

• Q2. Does the use of the portal impact newcomers’ self-efficacy?▫ Self-efficacy survey

• Q3. What is the perceived usefulness, ease of use, and likely future use?▫ Technology Acceptance Model (TAM)

Methodology

8

• Diary Study

▫ Access to everyday behavior, which affords access to the experience’s immediacy and provides accounts of phenomena over time [Symon 2004]

▫ Similar to voluntary OSS contribution setting Work whenever desired At any time and place

▫ Interaction with participants via comments Constant feedback

Q1. How do newcomers use the portal to overcome contribution barriers?

9

Symon, G. 2004. Qualitative research diaries. Essential Guide to Qualitative Methods in Organizational Research. SAGE publications. 98–113

• The initial feeling of the control group participants was uncertainty and doubt on how to proceed

“I don’t know how to start” “I am feeling the necessity of finding something that will be my guide during this process” • The participants who used FLOSScoach felt more oriented and

confident“…the tool helped me a lot, because it gave me an outstanding guidance… [FLOSScoach]

made me spend less time and made me more confident”

• Spread documentation and information overload made newcomers feel lost

• FLOSScoach organization facilitated finding correct information“For me, the task was facilitated mainly by two factors: 1 - Presentation of necessary

information only; 2 - organization of information” – P2-01


Contribution Process

10

• Overall, content of FLOSScoach obviated the need to talk to the community

“I did not need to talk to them. The tool was very clear. It is very easy, very good.”

• Providing a message template was also beneficial

“I liked the message template, showing how to introduce myself and to present the problems I am facing. Even having proficiency in English, I did not know the more polite way of asking for help. This example helped to be clear,

concise to present the message objective, and also to reduce the shyness”


Social Interactions

11


• Understanding the architecture/code structure, understanding the code, finding the correct artifact to fix an issue, local environment setup were recurrently reported by both groups▫ Frustration, irritation, and demotivation were co-reported with the

barriers

“I was expecting to move forward, because so far I did not had time to look at the source code… It is frustrating”

“The issues with the dependency are still there, so I decided to clone the repository again. I am feeling tired and frustrated”

• Technical issues were the main reason why most students were unable to deliver…

“FLOSScoach is really interesting… It was a good starting point, that helped me learn the etiquette and the process, but it did not help me with the technical

development problems”

Technical issues

12

Q2. Does the use of the portal impact newcomers’ self-efficacy?

• Self-efficacy is a measure of the confidence in the participants’ perceived ability to perform a task, which can impact one’s actual ability to complete a task [Bandura 1986]

▫ 10 questions related to self-perceived ability to perform OSS related activities adapted from related work [Davidson et al. 2014]

▫ Pre- and post- questionnaires with the same set of questions

▫ Difference between pre- and post- results can indicate success or failure/problem

13

Bandura, A. 1986. Social foundations of thought and action: a social cognitive theory. Prentice-Hall

Davidson, J.L., Mannan, U.A., Naik, R., Dua, I. and Jensen, C. 2014. Older Adults and Free/Open Source Software: A Diary Study of First-Time Contributors. Proceedings of The International Symposium on Open Collaboration (2014)

Self-efficacy survey: before and after

14

Q5 (pro-cess of

con-tribut-ing)

Q6(con-tribut-ing to

OSS is

inter-esting

)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Before - Portal group After - Portal groupBefore - Control group After - Control group

Q2. Does the use of the portal impact newcomers’ self-efficacy?

• Case participants▫ 16 increased self-efficacy score▫ 8 decreased self-efficacy score

• Control participants▫ 7 increased their self-efficacy score ▫ 14 decreased self-efficacy score

• Total self-efficacy score significantly decreased for the control group participants (p=0.005)▫ no significant difference in the portal group

• Total self-efficacy after the assignment was greater for the portal group than for the control group (p=0.013)▫ no significant difference before the assignment

15

Q3. What is this web portal’s perceived usefulness, ease of use, and likely future use?

• We applied the Technology Acceptance Model (TAM)

▫ Assess user perception about a technology’s usefulness, ease of use, and predicted future use

▫ All case group participants were asked to answer to the TAM questionnaire after contributing

▫ 14 questions (Likert-scale) 6 questions regarding usefulness 6 questions related to ease of use 2 questions about future use

16

Usefulness

Q3. What is this web portal’s perceived usefulness, ease of use, and likely future use?

17

Ease of Use

Future Use

Threats to validity• The findings are not generalizable to all projects

• Participants can write their diary entries in an unfiltered way

• Most of our participants were novices to software development in real settings

• Students may have felt that they needed to provide positive feedback in the surveys

• Subjectivity of the qualitative diary analysis

18

Conclusion• FLOSSCoach: A newcomer-specific portal focused on the

first contribution

• Empirical evidence that organizing the information available in a project by means of a web portal reduces process-related barriers and the need of communication, but technical hurdles still persists.

• FLOSScoach improved newcomers’ experiences regarding the contribution process▫ Newcomers felt more confident and oriented▫ FLOSScoach positively influenced newcomers’ self-efficacy▫ Newcomers perceived the portal as useful and easy to use, and would use

it to support future contributions

19

Thanks!

Questions?

Igor [email protected]

Outsider

Onboarding Contributing

Newcomer Developer Core Developer

Motivation

Attractiveness RetentionOnboarding Barriers

Scope and Definitions• Community-based OSS projects• Newcomers• First code contribution• Contribution = source code

Onboarding

Onboarding Barriers

22

Features for newcomers• Clear contribution flow helps newcomers gain

confidence about what to do and in what order

• Newcomer specific page containing only the resources they need and not flood them

• The identification of tasks that are considered easy or simple for new contributors helps them finding their way

• A message template facilitates and encourages newcomers to interact with the community

23