refine a gamifiedplatform for participatory requirements engineering

REfine: A Gamified Platform for

Participatory Requirements Engineering

R. Snijders, F. Dalpiaz, S. Brinkkemper (Utrecht University, NL)

Mahmood Hosseini, Raian Ali (Bournemouth University, UK)

Atilla Özüm (KPMG Technology Advisory, NL)

Agenda

1. Stakeholders involvement in RE

2. The potential: crowdsourcing, gamification

3. The REfine platform

4. Results from a case study

5. Conclusions and Future Work

REfine: a gamified platform for participatory RE


Software projects keep failing, or being challenged

Results from the 2014 Chaos Standish Report

16.2

31.1

52.7

Successful Cancelled Challenged



What are the key determinants for success/failure?

Taken from the 2014

Standish Chaos Report



But what can stakeholder/user involvement deliver?

More accurate requirements

Expensive and unnecessary features avoided

Improved acceptance (Kujala, 2003)

Higher quality of requirements

Higher chances of project success (Kujala et al., 2005)

Improved loyalty

Broadened market (Kabbedijk et al., 2009)


2. The potential:

crowdsourcing, gamification



Involving the crowd! Relying on a crowd of humans to

get things done!

Feature requests (Win10)

Crowdwork (Mturk)



Gamify it! The trend is manifest

4Square, Stackoverflow, LinkedIn, etc.

Gamification: “the use of game design elements in non-

game contexts” (Deterding et al., 2011)

Intrinsic kinds of motivation

Personal gratification

Social status



An interactive platform to elicit and refine requirements

Crowd-centric: all stakeholders are welcome to participate

Gamified: incentives through game elements



Discussing needs via REfine (perform actions)

Agree/disagree with needs

Add comments

“Liking” comments

Branching needs

Actions have a cost

Money obtained

when others find

one’s contributions

useful



Leaderboards to determine the most useful participants

General leaderboard (overall points)

Per type of contribution: ideator, assessor, commenter

Per need



The CCRE method

From feasibility to focus group

Fits in agile development (sprints)

Highly iterative, despite figure

Main characteristics

Determines if crowd involvement

has a potential

Uses an interactive platform like REfine

Rewards the most active stakeholders

by inviting them to the focus groups



The Qubus software product

A governance, risk, and compliance tool developed by KPMG

Under the hood, it hosts a workflow management system

Object of our study: needs for version 7 beta



Crowdsourcing preparation

Three-steps invitation: internal stakeholders, clients & users,

off-stage actors

Small crowd involved (19 people)

Crowd involvement

21 needs, 37 comments, 130 votes

Little moderation

One-month period



Stakeholder type N Active Needs Comments Votes Points

Avg. Total Avg. Total Avg. Total Avg. Total

Community

manager

1 - 2 2 8 8 21 21 41 41

Product

management

2 67% 0 0 3 6 6 12 13.5 27

Development team 4 100% 0.8 3 3.8 15 14.8 59 26.8 107

Experts 4 100% 1 4 1.3 5 2.5 10 16 64

Client 1 100% 0 0 0 0 0 0 3 3

Users 1 6% 1 1 0 0 3 3 11 11

Off-stage actors 6 86% 1.8 11 0.5 3 4.2 25 20 120

Total 19 50% 1.1 21 1.9 37 6.8 130 19.6 373



Questionnaire: comparing with previous experiences in

communicating requirements

0% 20% 40% 60% 80% 100%

Difficulty

Usefulness

Engagement

1 2 3 4 5

3.88 (more engaging)

4.19 (more useful)

2.69 (as difficult)



Questionnaire: perceived experience/usefulness of Refine

3.47

2.82

4.12

1.92

4.71

4.41

2.94



Evaluation with experts from the industry (generality)

1 = strong disagreement

7 = strong agreement



User involvement is crucial for project success

Potential: crowdsourcing and gamification

The REfine tool

Gamification to maximize stakeholder involvement

Mixed results from a case study

Gamification improves the experience

It is difficult to attract a large crowd



Large-scale studies with more stakeholders

Long-term incentives (real-life gamification)

How to assess the actual effect of gamification?

Ongoing work using user stories


Thanks for listening!

Questions?

[email protected]

[email protected]


References

S. Deterding, D. Dixon, R. Khaled, and L. Nacke, “From Game Design Elements to

Gamefulness: Defining Gamification,” in Proc. of MindTrek: Envisioning Future Media

Environments, 2011, pp. 9–15

J. Kabbedijk, S. Brinkkemper, S. Jansen, and B. van der Veldt, “Customer Involvement

in Requirements Management: Lessons from Mass Market Software Development,”

in Proc. of RE, Aug. 2009, pp. 281–286

S. Kujala, “User Involvement: A Review of the Benefits and Challenges,” Behaviour &

Information Technology, 2003

S. Kujala, M. Kauppinen, L. Lehtola, and T. Kojo, “The Role of User Involvement in

Requirements Quality and Project Success,” in Proc. of RE, 2005, pp. 75–84

R. Snijders, F. Dalpiaz, M. Hosseini, A. Shahri, and R. Ali, “Crowd-Centric

Requirements Engineering,” in Proc. of the 2nd International Workshop on

Crowdsourcing and Gamification in the Cloud, 2014
