collaborative sketching in crowdsourcing design: a new method for idea generation

19
Collaborative sketching in crowdsourcing design: a new method for idea generation Lingyun Sun Wei Xiang Shi Chen Zhiyuan Yang Accepted: 18 August 2014 Ó Springer Science+Business Media Dordrecht 2014 Abstract Design integrates concepts and solves problems. Crowdsourcing design imports vast knowledge and produces creative ideas. It publishes design tasks, collects dozens of contributors’ ideas and reward the best. Contributors in crowdsourcing design work individually when generating ideas. However, those who collaborate could make better use of crowd’s knowledge, which might produce ideas of higher quality. By ana- lyzing the advantages and disadvantages of crowdsourcing design, this article proposes a collaborative crowdsourcing design method that integrates crowd’s sketching processes. This method uses a tree to arrange crowd’s ideas and enable flexible modifications of prior ideas. A demonstration system named Sketchfans is developed, and the results of a crowdsourcing sketching experiment using this system are presented. The experimental results validate that this method is effective; participants rely heavily on the idea tree for inspiration, and the best ideas appear around the ends of branches on the idea tree. Moreover, participants displayed unique development patterns. They often developed high- quality ideas from initial ideas that were regarded as poor quality. The demonstration system Sketchfans, supports this method well. Finally, we optimize Sketchfans based on analysis of participants’ activities and feedback. Keywords Idea generation Á Conceptual design Á Collaborative sketching Á Crowdsourcing L. Sun Á W. Xiang Á S. Chen (&) Á Z. Yang Modern Industrial Design Institute, Zhejiang University, Hangzhou, China e-mail: [email protected] L. Sun e-mail: [email protected] W. Xiang e-mail: [email protected] Z. Yang e-mail: [email protected] 123 Int J Technol Des Educ DOI 10.1007/s10798-014-9283-y

Upload: zhiyuan

Post on 07-Apr-2017

217 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Collaborative sketching in crowdsourcing design: a new method for idea generation

Collaborative sketching in crowdsourcing design: a newmethod for idea generation

Lingyun Sun • Wei Xiang • Shi Chen • Zhiyuan Yang

Accepted: 18 August 2014� Springer Science+Business Media Dordrecht 2014

Abstract Design integrates concepts and solves problems. Crowdsourcing design

imports vast knowledge and produces creative ideas. It publishes design tasks, collects

dozens of contributors’ ideas and reward the best. Contributors in crowdsourcing design

work individually when generating ideas. However, those who collaborate could make

better use of crowd’s knowledge, which might produce ideas of higher quality. By ana-

lyzing the advantages and disadvantages of crowdsourcing design, this article proposes a

collaborative crowdsourcing design method that integrates crowd’s sketching processes.

This method uses a tree to arrange crowd’s ideas and enable flexible modifications of prior

ideas. A demonstration system named Sketchfans is developed, and the results of a

crowdsourcing sketching experiment using this system are presented. The experimental

results validate that this method is effective; participants rely heavily on the idea tree for

inspiration, and the best ideas appear around the ends of branches on the idea tree.

Moreover, participants displayed unique development patterns. They often developed high-

quality ideas from initial ideas that were regarded as poor quality. The demonstration

system Sketchfans, supports this method well. Finally, we optimize Sketchfans based on

analysis of participants’ activities and feedback.

Keywords Idea generation � Conceptual design � Collaborative sketching �Crowdsourcing

L. Sun � W. Xiang � S. Chen (&) � Z. YangModern Industrial Design Institute, Zhejiang University, Hangzhou, Chinae-mail: [email protected]

L. Sune-mail: [email protected]

W. Xiange-mail: [email protected]

Z. Yange-mail: [email protected]

123

Int J Technol Des EducDOI 10.1007/s10798-014-9283-y

Page 2: Collaborative sketching in crowdsourcing design: a new method for idea generation

Introduction

The internet enables people around the world to cooperate and collaborate easily.

Crowdsourcing design recruits contributors from various backgrounds to submit solutions

for design tasks, and makes use of the vast knowledge and experiences residing in crowd at

a low cost. 99Designs as a typical crowdsourcing design website holds small design

contests that collect on average 86 ideas for a task (Wooten and Ulrich 2011), while only

rewarding the best.

Researchers have proposed multiple methods to improve the quality of ideas in

crowdsourcing design (Wooten and Ulrich 2011; Morris et al. 2013), but they have not

changed the way of idea generation. Contributors still work individually to produce ideas.

Design is an iterative process during which ideas are constantly evaluated and reformulated

(Restrepo and Christiaans 2004); the crowd’s knowledge and perspectives are beneficial to

contributors’ idea evaluation and idea generation processes. Studies of group design also

propose methods that integrate group members’ design processes and enhance the design

quality (Shah et al. 2001). Therefore, people who collaborate to design might produce high

quality ideas. Besides, crowd who collaborate could learn from others’ design processes

and improve their design ability, and this enables sustainable crowd work (Kittur et al.

2013).

This article integrates people’s sketching processes (thus constituting the crowd in

crowdsourcing) to explore a new method of crowdsourcing design. It proposed a collab-

orative sketching method that employed a tree structure to arrange crowd’s sketches,

developed a demonstration system named Sketchfans, and conducted an experiment to

examine the quality of ideas produced by collaborative sketching. Sketchfans consisted of

a task module to publish and participate in tasks, and a sketching module to support

sketching and arrange people’s ideas into idea trees. People in different locations could use

Sketchfans to scan existing idea trees and go on sketching. Sketchfans thus enabled distant

and unsynchronized collaborative sketching, which might be beneficial for crowdsourcing

design. This experiment instructed participants to undertake design tasks using Sketchfans,

and recorded their activities. Participants’ ideas were assessed, and their suggestions for

this collaborative sketching process were transcribed. We compared the usefulness and the

originality of the ideas generated by multiple periods of crowdsourced sketching, and

summarized the development patterns between ideas and their antecedents to examine how

the ideas evolve. Sketchfans is then optimized on account of participants’ activity patterns

and suggestions.

Previous studies

Design as an evolution process

Design involves a unique way of thinking. It starts from vague problems and aims to fulfill

ambiguous goals. Designers need to gather a lot of information to structure problems and

goals, and find paths that connect them (Goldschmidt 1997). They achieve this by con-

tinually refining their solutions. These solutions import new concepts, reveal design situ-

ations, and enrich designers’ understanding of current problems (Sim and Duffy 2003).

Problems are thus re-formulated and new design constraints are added, inducing solutions

that are more suitable than before (Restrepo and Christiaans 2004). The evolution of

solutions distinguishes design from other domains (Cross 2006).

L. Sun et al.

123

Page 3: Collaborative sketching in crowdsourcing design: a new method for idea generation

A typical industrial design process involves the four stages of conceptual sketching,

drawing, modelling, and prototyping (Pei 2009). Sketching is the main method of idea

generation. As a quick visual representation, sketches comprise both form and function

(Temple 1994; Taura et al. 2012). They provide external memories of designers’ thoughts,

enabling functional issues to be recalled at any time and communicated with others (Suwa

et al. 1998). Moreover, sketches are more ambiguous than drawings. Because designers’

understanding of problems and goals changes during the sketching process, ambiguous

sketches induce new interpretations of existing configurations, and these are the driving

force behind the discovery of further solutions (Pache 2005; Suwa et al. 2000). Therefore,

sketching largely supports the evolution of solutions in the design process.

Crowdsourcing design

Crowdsourcing represents the act of assembling large numbers of people via an open call

to complete a task (Howe 2006a, b). Four key components constitute a crowdsourcing

process: pre-selection of contributors, accessibility of peer contributions, aggregation of

contributions, and remuneration for contributors (Doan et al. 2011; Geiger et al. 2011).

Malone et al. also identified the goal, participant, process, and incentive as core organi-

zational genes (2009).

The advantage of crowdsourcing derives from the knowledge and experiences residing

in contributors that were previously unknown and unavailable (Bogers and West 2012;

Malone et al. 2009). Contributors provide varied perspectives and content, and work in a

divide-and-conquer format, supporting a wide and quick exploration of problems (Erickson

2011; Geiger et al. 2012). The variety of contributors also brings challenges. Crowd-

sourcing initiators must offer effective incentives, propose proper tasks, manage multiple

submissions of varying quality, and handle unpredictable actions (Malone et al. 2009; Jain

2010). Another challenge is that varied contributors usually participated individually at

different times during crowdsourcing. Crowdsourcing is thus an unsynchronized and dis-

tant collaborative process, which might reduce the quality of collaborative works.

Crowdsourcing supports design. The vast knowledge imported by crowds facilitates a

wide exploration of design ideas. Companies built communities to collect feedback and

creative ideas for their products (Bayus 2013); they also post design tasks and choose

solutions from crowd’s submissions. Design contest is the main form of crowdsourcing

design. Websites including Designcrowd (http://www.designcrowd.com/) and 99designs

(http://99designs.com/) hold multiple small-scale design contests, recruit contributors to

submit ideas, and reward the best. Apart from the best idea, other submitted ones provide

initiators with fresh insights and commitment toward the central design problem (Tidball

et al. 2011).

Researchers have employed various methods to support crowdsourcing design. Giving

direct feedback to contributors enhances the quality and number of submissions (Wooten

and Ulrich 2011). Dontcheva et al. (2011) recommended that work group support,

incentives align with desired behavior, appropriate wording, and monitor use of other

sources fostered creativity.

These methods enhanced the quality of ideas, but didn’t change the way of idea gen-

eration in design contests; contributors still work individually to generate and revise their

ideas. This individual work has two drawbacks. First, design requires problem exploration

and thorough consideration of alternatives, and so hastily generated ideas submitted for

relatively low chances of reward may not meet the quality standards required for a mature

design. Second, these forms of crowdsourcing design only collect the crowd’s final ideas.

Collaborative sketching in crowdsourcing design

123

Page 4: Collaborative sketching in crowdsourcing design: a new method for idea generation

Designers have their specific ‘‘lenses’’ to interpret the design task and refine ideas (Lehoux

et al. 2011). A collaborative crowdsourcing design method that aggregates these ‘‘lenses’’

should be more appropriate.

Studies on group design provide valuable references to crowdsourcing design. Memory

is an associative network, once a concept is activated, its closely related concepts are

retrieved more easily than weakly connected concepts (Raaijmakers and Shiffrin 1981).

The ideas proposed by a member in design group activated others’ weakly connected

concepts that were hard to retrieve, resulting in better design (Linsey et al. 2011). Group

design methods were developed accordingly. The method called C-sketch passes sketches

sequentially through the design group so that designers could add and modify others’ ideas

(Shah et al. 2001); Gallery Method sets times for group discussion of ideas (Pahl et al.

2007); Filter Mediated Design filters the modified part of design ideas and integrates them

in a cohesive way for further modification (Haymaker et al. 2000). These methods facilitate

the reinterpretation and improve the quality of ideas.

Group design methods suggest that integrating group members’ design processes is

helpful. However, the unsynchronized and distributed crowd in crowdsourcing design do

not meet the features of design groups. During collaborative crowdsourcing design, each

contributor design for a short time, and then leaves. Contributors face multiple ideas

generated by unfamiliar contributors from the beginning of the design process. They must

understand these ideas and find valuable information to support their explorations.

Moreover, contributors are not familiar with each other, their actions are unpredictable,

and the quality of their ideas are varied. Simply applying these group design methods in

crowdsourcing design process might result in a mess.

Nickerson and Sakamoto (2010) and Yu et al. (2011) have explored a collaborative

crowdsourcing design method that split the design process into basic steps. Using this

method, ideas are first submitted, and better ideas are chosen as stimulation and given to

another group, inducing next-generation ideas. Every step calls for an individual crowd-

sourcing process. This method improves design quality. Even when the next-generation

ideas are a simple combination of prior ideas, their quality scores still increase.

Another possible method is arranging crowd’s sketching processes effectively. Sketch

records designers’ train of thoughts and helps idea generation; crowd facing well-organized

sketches of prior contributors can interpret these sketches to understand ideas and generate

new ones. Moreover, well organized sketching processes enable crowds to continue

sketching from any stages of prior sketching process, which mitigate the effect of low

quality ideas. Sun et al. (2014) used a tree to organize the ideas and proved to be useful in

individual sketching. The branches on the tree represent the design paths during sketching

processes. This article proposed a method that employed this tree structure to arrange

crowd’s ideas and display the design paths, thus supporting collaborative sketching in

crowdsourcing design.

A method of collaborative sketching

The collaborative sketching method involved a sketching module and a task module

(Fig. 1). In sketching module, contributors studied the prior ideas organized as an idea tree,

sketched new ideas which were recognized as new nodes of the tree. Initiators published

their tasks in the task module; contributors searched through design tasks in this module,

downloaded the latest idea trees, and submitted their ideas back to this module. The task

module then integrated these ideas into the idea tree. During these processes, incentives

L. Sun et al.

123

Page 5: Collaborative sketching in crowdsourcing design: a new method for idea generation

encouraged contributors to participate in tasks and develop ideas cooperatively. A col-

laborative sketching demonstration system, named Sketchfans, was then developed fol-

lowing the framework of this method.

Sketching module

The sketching module supported contributors’ idea generation processes and arranged their

ideas into an idea tree. The display of the idea tree was optimized to highlight inspirational

ideas.

Idea tree

An idea tree that showed the connections among ideas was developed referring to the tree

structure proposed by Sun et al. (2014). The first node on the tree is a description of the

design task, and following nodes are ideas that present solutions. Every branch is an

evolving path on which following ideas are developed from prior ones. The tree grows via

the sketching process, with branches extending as contributors develop ideas on the

branches. When designers develop new ideas that solve design tasks from new perspec-

tives, new branches linked to the task description node are established. Therefore, an idea

tree sorts piles of ideas into several branches that clearly display the evolution process.

Subsequent contributors could then choose idea nodes on one branch and scan sketches

drawn during the growth of this branch, which might enhance their understanding of ideas.

The sketching module generated the idea tree from contributors’ sketching processes.

An algorithm has been proposed that recognizes creative ideas (Sun et al. 2014), but the

unpredictability of the sketching activities in crowdsourcing results in low recognition

precision. The sketching module thus encouraged contributors to mark their creative ideas

after contributors have sketched them. Explorations that had not generated an idea formed

temporary nodes on the idea tree.

A typical sketching process for a task is shown in Fig. 2. The task was to design

communal facilities for elderly people’s recreation, and this had produced a tree containing

twelve ideas. During this process, the contributor scanned three branches involving cal-

ligraphy (idea 1), recreational projection (idea 2, idea 3, idea 5, idea 7, idea 8, and idea 9),

and a treadmill (idea 4, idea 6, idea 10, idea 11, and idea 12). When choosing idea 9, which

used funny shadows, the contributor found a new way of producing shadows with sunlight

(idea 13). Idea 1, based on writing brushes, also attracted his attention, inducing the floor of

calligraphy (idea 14); the elderly could teach their grandchildren to write while strolling.

Next, the contributor explored idea 14 for a while (temporary node), checked the treadmill

branch back and forth, and chose idea 11 to develop the cooperative treadmill into a

competitive one (idea 15). Idea 11 then have two developments.

Fig. 1 Collaborative sketching method. (Color figure online)

Collaborative sketching in crowdsourcing design

123

Page 6: Collaborative sketching in crowdsourcing design: a new method for idea generation

Display optimization of idea tree

Crowdsourcing sketching often collects lots of ideas while contributors may only scan a

few, and so there is a large probability that valuable ideas will be neglected. Therefore, two

rules were set to optimize the display of the idea tree accordingly.

The first rule is ‘‘Ideas having multiple following ideas are enlarged.’’ This rule pre-

supposes that those ideas that have induced multiple following ideas are inspirational and

easily stimulate new thoughts in the continuing collaborative sketching process. The

second rule is ‘‘Ideas on long branches are enlarged.’’ This rule emphasizes the evolution

of ideas; those on long branches have been widely endorsed and consistently enhanced,

making them more likely to trigger new and better ideas.

The size of the idea nodes is adjusted according to these two rules. Ideas having

multiple following ideas and those on longer branches are bigger. We use a weight to

balance the effect of the two rules. The idea tree in Fig. 2 could thus be optimized as

shown in Fig. 3.

Task module

The task module displayed tasks and integrated the ideas that had been submitted. During

collaborative sketching, multiple contributors downloaded the same idea tree, and then

Fig. 2 Idea tree grows during a contributor’s sketching process (numbers display the sequences of ideas,grey rectangle represents a temporary node, and orange borders mark the ideas sketched by this contributor.Ideas 3, 5, 6, 7, 8, and 10 are omitted). (Color figure online)

L. Sun et al.

123

Page 7: Collaborative sketching in crowdsourcing design: a new method for idea generation

submitted their ideas at different times. All new idea nodes and branches were linked to

prior nodes. The task module thus received new idea nodes and corresponding sketches,

and linked these new nodes to the idea tree. In this way, contributors could view the idea

tree with all submitted ideas.

Sketchfans

This study developed Sketchfans for the iPad. Compared with a professional sketching

environment involving a computer and tablet, the iPad has a wide range of users, and offers

portable hardware that allows contributors to sketch freely. This demonstration system

contained an online task module and an offline sketching module. The task module dis-

played thumbnails of design tasks and idea trees, and the sketching module provided basic

sketching functions, arranged contributors’ ideas into idea trees, and uploaded new ideas to

the task module. The default size of the biggest idea node on the tree is set to 25 9 25

pixels to display the full structure of the tree, and it was possible to zoom in or out to view

details of the tree.

Contributors used Sketchfans to participate in collaborative sketching. As shown in

Fig. 4a, b, a contributor could check the tasks in which he had participated and search for

other tasks. In this example, he chose the already-participated in task to examine the latest

Fig. 3 Display optimization of the idea tree in Fig. 2. (Color figure online)

Collaborative sketching in crowdsourcing design

123

Page 8: Collaborative sketching in crowdsourcing design: a new method for idea generation

idea tree (Fig. 4c). By zooming in on the idea tree and clicking relevant nodes, the

contributor selected some exciting ideas and downloaded the tree to work on improving

them. Using the sketching module of Sketchfans, the contributor clicked idea nodes that he

was interested in, sketched new ideas, and pressed the ‘‘add’’ button to mark these ideas

(Fig. 5). The sketching module then linked these marked ideas as new nodes in the idea

tree. After sketching, the contributor submitted the updated tree to the task module by

clicking the ‘‘upload’’ button.

Initiators also used Sketchfans to publish collaborative sketching tasks. An initiator

clicked the ‘‘add’’ button in the sketching module to build a new idea tree for the design

task, described the task in the first node, and uploaded this one-node tree. The new task was

then published in the task module to await others’ participation.

A test of collaborative sketching method

Collaborative sketching integrates crowd’s thoughts and perspectives, which might be

beneficial to the generation of ideas. This section reports a collaborative sketching

experiment, and evaluates the generated ideas to further explore the effectiveness of col-

laborative sketching in crowdsourcing design.

Hypotheses

The first hypothesis is that the originality and the usefulness of ideas increase during

collaborative sketching. Subsequent participants scan a lot of ideas and generate more

original and useful ideas.

The second hypothesis is that the originality of ideas increases during the growth of

branches on the tree. Following contributors retrieve weak connected concepts when

interpreting the prior ideas proposed by ‘‘strangers,’’ thus producing more original ideas.

The third hypothesis is that the best ideas appeared near the end of branches. The

originality and the usefulness of ideas improve with the modifications. Therefore, ideas

near the end of long branches would be better.

The fourth hypothesis is that the ideas having multiple following ideas and ideas

inducing long branches are of higher originality. Participants choose inspirational ideas and

develop them. Therefore, these ideas are more original than ideas with only one

development.

Fig. 4 Interface of task module: a tasks in which the contributor has participated, b all tasks, and c previewof the idea tree. (Color figure online)

L. Sun et al.

123

Page 9: Collaborative sketching in crowdsourcing design: a new method for idea generation

Participants and procedure

A sketching experiment was conducted. Twenty-three participants (14 female, 9 male)

involved in this experiment. Among these participants, twenty students had an average of

3.6 years’ learning experience in industrial design, one had learned marketing for 3 years,

one had learned environmental design for 3 years, and one had worked as an industrial

designer for 2 years. People in crowdsourcing are usually physically separated and do not

know each other. To simulate this, participants in this experiment came and completed the

task separately, and did not know each other during the whole experiment. Seven of the 23

participants were randomly chosen, and sketched twice at different times to simulate the

repeated participation during actual crowdsourcing design process. The others only sket-

ched once during the experiment. Participants used the new iPad and a stylus to draw

sketches (Fig. 6). They were instructed to design communal facilities for elderly people’s

recreation, among which the best idea would be rewarded with 100 yuan (approximately

US$16), and ideas on the path to the best idea would share 50 yuan. Participants in design

experiments usually got 15–20 yuan, and so the much higher rewards of this experiment

ensured participants’ substantial effort.

The entire experiment consisted of three stages:

1. Introduction of sketching process. The participant was introduced to the experiment,

and allowed to familiarize himself/herself with the Sketchfans software and iPad.

2. Individual sketching process. The participant first downloaded and examined the idea

tree. Initially, the idea tree only contained one node that described the design task.

After examination, the participant was instructed to sketch three further ideas using

Sketchfans. She/he pressed a button each time she/he finished sketching an idea to add

this idea on the idea tree. The scanning and sketching time was not limited. The

participant uploaded the three new ideas when finished.

Fig. 5 Interface of sketching module (left column contains functional buttons, right column displays theidea tree). (Color figure online)

Collaborative sketching in crowdsourcing design

123

Page 10: Collaborative sketching in crowdsourcing design: a new method for idea generation

3. Idea explanation. After sketching, the participant explained her/his ideas and those that

she/he chose to develop, reported any obstacles during the process, and gave some

suggestions about Sketchfans. These reports were transcribed, and suggestions were

categorized.

Data analysis

The ideas participants sketched are numbered according to their chronological order. The

majority of participants spent less than 20 min to scan the idea tree and less than 20 min to

sketch three ideas. Participants generated 90 ideas and produced a tree with 20 branches.

The number of ideas is typical for design contests hold by crowdsourcing websites like

99designs and CrowdSpring (Wooten and Ulrich 2011).

Each idea on a branch changes parts of the prior idea. An idea consists of four hier-

archies involving the goal, function, structure, and detail. The goals of ideas refer to the

benefits they can bring to people, the functions refer to what the facilities do to meet goals,

the structures refer to what the facilities use to implement functions, and the details refer to

components of structures. Ideas having new goals built new branches. The developments of

ideas on a branch were categorized into three categories: physical principles, working

principles, and embodiment according to the article of Shah et al. (2003). Physical prin-

ciple developments refer to new functions used to satisfy one goal, for example, using

shadows (function: produce shadows) rather than videos to guide exercise (goal: guide

exercise). Working principle developments are new structures used to satisfy one function,

for instance, producing shadows (function: produce shadows) using the sun (structure:

hollowed plate) rather than a lamp. Embodiment developments refer to new details, such as

new figures of shadows (detail: shape of the hollow). The idea 15 in Fig. 2 is thus working

principle development because it uses a new structure, i.e., the opposite treadmill, to meet

the run function. Two researchers with 4 years’ learning experience in industrial design got

familiar with the categories, coded these developments individually, and settled eight

inconsistencies by discussion.

Fig. 6 Participant sketching on an iPad. (Color figure online)

L. Sun et al.

123

Page 11: Collaborative sketching in crowdsourcing design: a new method for idea generation

Sketch quality influences the perception of idea quality (Kudrowitz et al. 2012). Par-

ticipants’ varied sketching skills might influence the idea evaluation. To analyze the

changes in idea originality and usefulness more accurately, we exclude such effects by

redrawing all sketches. A graduate student with 4 years’ industrial design experience

redraws all ideas after all sketching processes. He copied these sketches and smoothed the

lines according to participants’ explanation of their ideas. As shown in Fig. 7, the graduate

student connected the detached lines of the chair while keeping the relative location of

these lines. Therefore, the re-drawing adjusted the qualities of two left sketches in Fig. 7 to

the same and did not change the ideas.

Existing methods of idea evaluation mainly focus on two aspects: whether the idea is

unusual and original, and whether the idea could be implemented and have practical

benefits (Nelson et al. 2009; Dean et al. 2006; Oman et al. 2013; Kudrowitz and Wallace

2013). We thus evaluated the originality and the usefulness of ideas, and selected the best

ideas according to the sum of these two averaged scores. The original ideas are those that

are unusual and rare. The useful ideas are those actually solve problems and are easy to

implement. For example, the idea involving four balls and four holes offers new recreation

method (the lower pictures in Fig. 7), but whether the elderly people likes it, and whether it

is easy to maintain in public space are questionable. Therefore, it would get a high orig-

inality score and a medium usefulness score. A relaxing chair is easy to implement and

offer a place to sit on, but there have been plenty of similar relaxing chairs; thus it is

unoriginal but useful.

Shah et al. evaluated idea originality based on the frequency of ideas; the more times

one idea was proposed by participants, the less original the idea was (2003). However,

during the collaborative sketching experiment, one original idea proposed by a participant

might be appreciated and developed by subsequent participants, thus lowering the origi-

nality score of this idea. Therefore, this article employed judges to evaluate these ideas. Six

graduate students with 4.8 years’ industrial design experience participate in idea evalua-

tion. These ideas are randomly sorted and evaluated in terms of their usefulness and their

originality using five-point scales, where five denoted highly useful and very original, and

one represented useless and unoriginal. The inter-rater agreement is acceptable with a

Kendall’s W of 0.623 for originality score and 0.665 for usefulness score. The originality

score and the usefulness score of each idea are then averaged, and the best idea and ideas

along the path to this idea are rewarded.

Results

The distribution of ideas on the tree was polarized (Fig. 8). Ten branches had only one idea

each, whereas the longest branch had 14 ideas. When viewing the tree, participants tended

to develop ideas on longer branches, and they also attended to other inspirational ideas.

The first idea proposed in the experiment was not developed any further until the 33rd idea.

Another branch containing 12 ideas was built in the middle of experiment, and grew fairly

quickly.

Participants could build a new branch and develop the existing branches during the

experiment. The proportion of ideas that build new branches was not affected by time.

Twelve of the first half of the 90 ideas built new branches, and eight of the second half built

new branches.

Participants employed more embodiment developments and less working principles in

the second half of 90 ideas (v2 = 8.34, p = 0.015). Seven ideas in the first half of 90 ideas

Collaborative sketching in crowdsourcing design

123

Page 12: Collaborative sketching in crowdsourcing design: a new method for idea generation

employed embodiment developments, while the amount is eighteen in the second half of

90 ideas. The proportion of ideas with physical principles reminded the same.

Originality and usefulness of ideas

The ideas received a mean score of 2.74 (SD = 0.59) for originality and 2.84 (SD = 0.61)

for usefulness. The best idea received an overall score of 8 (originality score = 3.5,

Fig. 7 Examples of the sketches before and after re-drawing. (Color figure online)

Fig. 8 Distribution of thenumber of idea nodes on abranch. (Color figure online)

L. Sun et al.

123

Page 13: Collaborative sketching in crowdsourcing design: a new method for idea generation

usefulness score = 4.5), and the worst received only 3.83 (originality score = 2.33, use-

fulness score = 1.5). Of the 90 ideas, the latter 45 were more useful than the former

(t = 2.06, p = 0.043). Their originality scores were not significantly higher.

The originality and usefulness of ideas on the first half of a branch were compared with

those on the second half. Each half of a branch had half the quantity of ideas. Ideas on the

second halves were more original than those on the first (Z = 3.99, p = 0.003), with an

average originality score of 2.96 compared to 2.50. The average usefulness of ideas on

each half did not differ. The change of originality and usefulness scores of ideas as the

branches grew was further analyzed. As shown in Fig. 9, the originality scores increase,

whereas the usefulness scores reach their peak around the 5th and 9th idea, and then

decrease.

The ideas were studied to determine the reasons for this change in usefulness scores.

Developments before the middle of a branch concentrated on fixing flaws with the initial

ideas, with typical examples being ‘‘ergonomic support for the elderly’’ and ‘‘circular table

with chessboard for multiple people.’’ On the second half of branches, developments

brought in new functions and features, such as ‘‘running competition’’ and ‘‘vertical

chessboard.’’ Participants explored new possibilities after improving the initial functions,

thus scoring lower for usefulness and higher for originality. This could also be confirmed

by the 13th idea, where further fixation on new ideas improved the usefulness score

(Fig. 9). Therefore, usefulness fluctuated, whereas originality increased.

The best ideas and the worst ideas

The features of the four best ideas, the four most original ideas, and the four most useful

ideas were analyzed. These ideas appeared in long branches after several rounds of

development (Fig. 10). The left best idea that appeared in the second branch from the left

was proposed late in the experiment (the 81st of the 90 ideas), which might have not

enough time to be developed. This article also analyzed the features of the four worst ideas,

the four least original ideas, and the four least useful ideas. The worst ideas appeared on

long branches while they appeared during the early developments. Ideas that were not quite

good attracted lots of developments. It was surprising that one branch could have both the

best ideas and the worst ideas. The third branch from the left had three least original ideas

and two most original ideas. The second branch from the right had one best idea and one

worst idea, and one most useful idea and one least useful idea.

Features of ideas inducing long branches and those having multiple following ideas

The originality and usefulness scores of the first ideas on long branches did not differ from

those on short branches (originality: t = -0.96, p = 0.351; usefulness: t = -0.11,

p = 0.916). Ideas on long branches had the same originality and usefulness as those on

short branches (originality: t = -0.64, p = 0.528; usefulness: t = 0.32, p = 0.754).

However, the ideas on long branches had more varied originality scores and more stable

usefulness scores than those on short branches. They had a standard deviation of 0.61 on

originality and 0.58 on usefulness, while the ideas on short branches had a standard

deviation of 0.54 on originality and 0.74 on usefulness.

Participants did not choose high-quality ideas to develop. Rather, they chose to develop

ideas that were not so good. Ideas having multiple (more than two) following ideas had the

same usefulness as other ideas, and slightly lower originality (t = -1.78, p = 0.080).

Collaborative sketching in crowdsourcing design

123

Page 14: Collaborative sketching in crowdsourcing design: a new method for idea generation

They were followed by larger modifications than ideas with few following ideas

(v2 = 9.73, p = 0.008). Indeed, 17 out of 38 following ideas made physical principle

developments, whereas for ideas having one or two following ideas, only five out of 32

following ideas were made on physical principles.

Usability of Sketchfans

All participants successfully finished the sketching task and reported that the other sketches

were helpful. As sketches involved both text and drawings, participants could easily report

the exact meaning of the ideas that they chose to develop. They reported that the tree

structure assisted the understanding of ideas and stimulated new thoughts, but also men-

tioned that some sketches were messy and difficult to understand. Participants explored the

idea tree when searching for inspiration. They would generally view ideas, and relied

heavily on the thumbnails on the idea tree to search for what they wanted. Therefore, a

large proportion of suggestions were related to the display of the idea tree. Ten participants

felt that the default size of the idea tree was too small to view thumbnails and click on idea

nodes, and six participants had difficulty finding ideas that they had scanned before. Other

Fig. 9 Originality andusefulness of ideas during thegrowth of branches. (Color figureonline)

Fig. 10 Positions of the ideas on branches (other branches have been omitted. The lower an idea on abranch, the later it was proposed during the sketching process). (Color figure online)

L. Sun et al.

123

Page 15: Collaborative sketching in crowdsourcing design: a new method for idea generation

suggestions called for a more fluid loading of the idea tree and a clearer understanding of

others’ ideas.

Discussion

Collaborative sketching allows a balanced exploration of design tasks. Participants propose

a broad range of ideas, and develop several of them thoroughly. They easily examine and

develop ideas on long branches, without fixating on them. Among the seven branches that

had more than five ideas, two contain only one idea until the middle of experiment.

Participants collaborate well during the exploration. Even though they can not contact each

other, participants implement prior ideas step-by-step and developed some great ideas.

Test of collaborative sketching method

The collaborative sketching process method is effective. Hypothesis one and hypothesis

two are partly supported by the results. Subsequent ideas tend to be more useful than prior

ideas during collaborative sketching; while they have the same originality. Originality

constantly increases along the branches, and usefulness increases when participants fix

flaws in the prior ideas on the branches. Collaborative sketching performs two kinds of idea

improvements. Scanning many ideas enables a thorough consideration of the design tasks,

producing more useful ideas; while for ideas on the same branch, constant exploration on

the design path supports further breakthroughs and induces more original ideas modified

from prior solutions.

Hypothesis three is supported by the results. During collaborative sketching, high-

quality ideas still come from the modification rather than a burst of inspiration. The best

ideas appear on the end of branches and the worst ideas appear on the former part of

branches. Participants might come up with all sorts of ideas at the beginning of a branch,

but following developments effectively enhance these ideas, as shown in Fig. 10, even the

best ideas could be produced from the worst ones.

Hypothesis four is contrary to the results. It is interesting that participants are not

inspired by highly original ideas, and would rather develop ideas of lower originality. Two

factors may contribute to this. First, because collaborative sketching focuses on the con-

ceptual stage of design, the original ideas might be too original for other participants to

build a connection from these concepts to their own knowledge. Therefore, participants

will find it hard to generate new ideas from these original ideas. Second, participants know

little about each other during collaborative sketching. They only scan others’ ideas for a

limited time, and have no opportunity for group discussion or debate. The time duration is

too short for an unfamiliar original idea to be fully understood and stimulate new thoughts.

The less original ideas, on the contrary, are ‘‘usable new thoughts’’ that offer a starting

point for participants’ development. Originality enhancements are achieved by exploring

from a familiar zone, i.e., significant developments of less original ideas, rather than

developing original ideas in a new zone.

Sketchfans supports the collaborative sketching process. Participants are easily able to

find thumbnails of the experimental task, understand relations among ideas on the idea tree,

and click on idea nodes. The idea tree enables more functions than we first thought. It not

only displays relations among ideas, but also acts as a quick index that helps participants

search and recall the ideas they have viewed. Another notable feature is that participants

Collaborative sketching in crowdsourcing design

123

Page 16: Collaborative sketching in crowdsourcing design: a new method for idea generation

prefer to choose clear ideas to develop. As Sketchfans only offers free drawing function,

ideas produced by participants with relatively poor drawing skills might be ignored, nar-

rowing the scope of suitable contributors.

This study tests the effectiveness of the collaborative sketching method in crowd-

sourcing design, rather than the effects of collaborative sketching components. Thus, two

minor components of collaborative sketching were adjusted to observe the process more

clearly. Each participant sketched three ideas; participants sketched in a controlled envi-

ronment for this experiment, rather than in their familiar workspace. At the same time, we

kept the key features of collaborative sketching: participants were recruited and rewarded;

they sketched on others’ sketches and did not know the identities of other participants; their

sketching activities and the time spent on sketching were not restricted, and the number of

ideas was typical for a crowdsourcing design contest. These key features ensure that the

experiment is a reliable simulation of the real collaborative sketching process.

Optimization of sketchfans

Participants frequently used the idea tree during sketching. The display of the idea tree was

modified and new functions were added. As participants examined ideas on branches,

Fig. 11 Participants could mark one idea for later use: a the button to mark an idea, b the marked idea, andc the button to unmark an idea. (Color figure online)

Fig. 12 Sketching interface with marked ideas. (Color figure online)

L. Sun et al.

123

Page 17: Collaborative sketching in crowdsourcing design: a new method for idea generation

rather than the entire structure of the idea tree, we enlarged the default size of the idea

nodes and shortened the distance between them to make it easier to preview and click.

Following a brief test using three participants, the default node size was set to 40 9 40

pixels, and the distance between the centers of two adjacent ideas was set to 60 pixels.

A ‘‘favorite’’ function was added to the idea tree. This function marks ideas for later

use. As shown in Fig. 11, when an idea node was pressed for 3 s, a button would pop up to

turn this idea node into a circle. Circles among squares clearly display participants’

favorite ideas on the idea tree (Fig. 12), offering a quick search and recall experience.

Participants could only see their own marks during the sketching process. These marked

idea nodes denote those that caught the participant’s interest during the sketching process,

and offer valuable data for adjusting the size of the idea nodes.

Conclusion

This article proposes a new crowdsourcing design method that recruits crowds to collab-

oratively sketch for design tasks. This method arranges contributors’ ideas in an idea tree,

showing their relations and helping subsequent contributors to develop those ideas. A

demonstration system named Sketchfans is developed, and this method is examined

experimentally. Finally, the demonstration system is optimized according to participants’

sketching activities and feedback.

The experimental results reveal that the collaborative sketching method is effective in

enhancing the quality of contributors’ ideas. Subsequent ideas during the sketching pro-

cesses are more useful than prior ideas, and ideas on the second half of branches got higher

originality scores than those on the first half of branches. The best ideas appear around the

end of branches, and some of them are even developed from the worst ideas. Participants

make balanced explorations. They largely develop familiar ideas and gradually implement

ideas during collaborative sketching. This method supports the participants’ sketching

well; participants seldom feel hindered, and rely heavily on the idea tree for inspiration.

The method also displays some shortcomings. Insufficient communication and discus-

sion make it hard to design for tasks that have complex constraints and details. Since

participants collaborate in this method and interactions between participants are significant

for collaboration, a module that supports unsynchronized communication should be

included in this method. This article presents an initial exploration of collaborative

sketching in crowdsourcing design. Further studies will optimize the arrangement and

display of ideas to improve design quality, and develop a collaborative sketching system

that enables communication and discussion.

Acknowledgments This paper is supported by the National Natural Science Foundation of China(61004116) and the Zhejiang Provincial Natural Science Foundation of China (LY13E050005,LY13F030002).

References

Bayus, B. L. (2013). Crowdsourcing new product ideas over time: An analysis of the Dell IdeaStormcommunity. Management Science, 59(1), 226–244.

Bogers, M., & West, J. (2012). Managing distributed innovation: Strategic utilization of open and userinnovation. Creativity and Innovation Management, 21(1), 61–75.

Cross, N. (2006). Designerly ways of knowing (1st ed.). London: Springer.

Collaborative sketching in crowdsourcing design

123

Page 18: Collaborative sketching in crowdsourcing design: a new method for idea generation

Dean, D. L., Hender, J. M., Rodgers, T. L., & Santanen, E. L. (2006). Identifying quality, novel, and creativeIdeas: Constructs and scales for idea evaluation. Journal of the Association for Information Systems,7(10), 646–698.

Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011). Crowdsourcing systems on the world-wide web.Communications of the ACM, 54(4), 86–96.

Dontcheva, M., Gerber, E., & Lewis, S. (2011). Crowdsourcing and creativity. In The ACM CHI conferenceon human factors in computing systems: Workshop on crowdsourcing and human computation,Vancouver, Canada, May 7–12 2011.

Erickson, T. (2011) Some thoughts on a framework for crowdsourcing. In The ACM CHI conference onhuman factors in computing systems: Workshop on crowdsourcing and human computation, Van-couver, Canada, May 7–12 2011.

Geiger, D., Rosemann, M., Fielt, E., & Schader, M. (2012) Crowdsourcing information systems-definition,typology, and design. In International conference on information systems, Orlando, USA December16–19 2012 (Paper 53).

Geiger, D., Seedorf, S., Schulze, T., Nickerson, R. C., & Schader, M. (2011) Managing the crowd: Towardsa taxonomy of crowdsourcing processes. In Americas conference on information systems, Detroit,Michigan, August 4–8 2011 (Paper 430).

Goldschmidt, G. (1997). Capturing indeterminism: Representation in the design problem space. DesignStudies, 18(4), 441–455. doi:10.1016/S0142-694X(97)00011-2.

Haymaker, J., Keel, P., Ackermann, E., & Porter, W. (2000). Filter mediated design: Generating coherencein collaborative design. Design Studies, 21(2), 205–220.

Howe, J. (2006a). Crowdsourcing: A definition. Crowdsourcing: Tracking the rise of the amateur. http://crowdsourcing.typepad.com/cs/2006/06/crowdsourcing_a.html.

Howe, J. (2006b). The rise of crowdsourcing. Wired Magazine, 14(6), 1–4.Jain, R. (2010). Investigation of governance mechanisms for crowdsourcing initiatives. In Americas con-

ference on information systems, Lima, Peru, August 12–15 2010 (Paper 557).Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., et al. (2013). The future of

crowd work. Paper presented at the the 16th ACM conference on computer supported cooperative workand social computing, San Antonio, USA, February 23–27.

Kudrowitz, B., Te, P., & Wallace, D. (2012). The influence of sketch quality on perception of product-ideacreativity. AI EDAM, 26(Special Issue 03), 267–279. doi:10.1017/S0890060412000145.

Kudrowitz, B., & Wallace, D. (2013). Assessing the quality of ideas from prolific, early-stage productideation. Journal of Engineering Design, 24(2), 120–139.

Lehoux, P., Hivon, M., Williams-Jones, B., & Urbach, D. (2011). The worlds and modalities of engagementof design participants: A qualitative case study of three medical innovations. Design Studies, 32(4),313–332.

Linsey, J. S., Clauss, E. F., Kurtoglu, T., Murphy, J. T., Wood, K. L., & Markman, A. B. (2011). Anexperimental study of group idea generation techniques: Understanding the roles of idea representationand viewing methods. Journal of Mechanical Design, 133(3). doi:10.1115/1.4003498.

Malone, T., Laubacher, R., & Dellarocas, C. (2009). Harnessing crowds: Mapping the genome of collectiveintelligence. Research Paper No. 4732-09, MIT.

Morris, R. R., Dontcheva, M., Finkelstein, A., & Gerber, E. (2013). Affect and creative performance oncrowdsourcing platforms. In Affective computing and intelligent interaction (ACII), 2013 humaineassociation conference on, 2013 (pp. 67–72), IEEE.

Nelson, B. A., Wilson, J. O., Rosen, D., & Yen, J. (2009). Refined metrics for measuring ideation effec-tiveness. Design Studies, 30(6), 737–743. doi:10.1016/j.destud.2009.07.002.

Nickerson, J., & Sakamoto, Y. (2010). Crowdsourcing creativity: Combining ideas in networks. In Work-shop on information in networks, New York, USA, September 24–25 2010.

Oman, S. K., Tumer, I. Y., Wood, K., & Seepersad, C. (2013). A comparison of creativity and innovationmetrics and sample validation through in-class design projects. Research in Engineering Design, 24(1),65–92.

Pache, M. W. (2005). Sketching for conceptual design. Empirical results and future tools. Munich: Tech-nischen Universitat Munchen.

Pahl, G., Wallace, K., & Blessing, L. T. M. (2007). Engineering design: A systematic approach. Berlin:Springer.

Pei, E. (2009). Building a common language of design representations for industrial designers & engi-neering designers. Loughborough: Loughborough University.

Raaijmakers, J. G., & Shiffrin, R. M. (1981). Search of associative memory. Psychological Review, 88(2),93–134. doi:10.1037/0033-295X.88.2.93.

L. Sun et al.

123

Page 19: Collaborative sketching in crowdsourcing design: a new method for idea generation

Restrepo, J., & Christiaans, H. (2004). Problem structuring and information access in design. Journal ofDesign Research, 4(2), 1551–1569.

Shah, J. J., Smith, S. M., & Vargas-Hernandez, N. (2003). Metrics for measuring ideation effectiveness.Design Studies, 24(2), 111–134. doi:10.1016/S0142-694X(02)00034-0.

Shah, J. J., Vargas-Hernandez, N., Summers, J. D., & Kulkarni, S. (2001). Collaborative sketching (C-Sketch)—An idea generation technique for engineering design. The Journal of Creative Behavior,35(3), 168–198.

Sim, S. K., & Duffy, A. H. (2003). Towards an ontology of generic engineering design activities. Researchin Engineering Design, 14(4), 200–223.

Sun, L., Xiang, W., Chai, C., Wang, C., & Huang, Q. (2014). Creative segment: A descriptive theory appliedto computer-aided sketching. Design Studies, 35(1), 54–79. doi:10.1016/j.destud.2013.10.003.

Suwa, M., Gero, J. S., & Purcell, T. A. (1998). The roles of sketches in early conceptual design processes InProceedings of the twentieth annual conference of the Cognitive Science Society.

Suwa, M., Gero, J., & Purcell, T. (2000). Unexpected discoveries and S-invention of design requirements:Important vehicles for a design process. Design Studies, 21(6), 539–567.

Taura, T., Yamamoto, E., Fasiha, M. Y. N., Goka, M., Mukai, F., Nagai, Y., et al. (2012). Constructivesimulation of creative concept generation process in design: A research method for difficult-to-observedesign-thinking processes. Journal of Engineering Design, 23(4), 297–321.

Temple, S. (1994). Thought made visible: The value of sketching. Co-Design Journal, 1, 16–25.Tidball, B., Mulder, I., & Stappers, P. J. (2011). Online design contests: A network of inspiration for

designers. Paper presented at the proceedings of the 4th world conference on design research, IASDR,2011, Delft, The Netherlands, October 31–November 4.

Wooten, J., & Ulrich, K. (2011). Idea generation and the role of feedback: Evidence from field experimentswith innovation tournaments. Available at SSRN 1838733.

Yu, L., & Nickerson, J. V. (2011). Cooks or cobblers? Crowd creativity through combination. In Pro-ceedings of the SIGCHI conference on human factors in computing systems, 2011 (pp. 1393–1402),ACM.

Collaborative sketching in crowdsourcing design

123