a theoretical and empirical investigation of feedback in ideation … · 2020-01-29 ·...

A Theoretical and Empirical Investigation of Feedbackin Ideation Contests

Juncai JiangPamplin School of Business, Virginia Tech, Blacksburg, Virginia 24061, USA, [email protected]

Yu Wang*College of Business, California State University, Long Beach, Long Beach, California 90840, USA, [email protected]

I deation contests are commonly used across public and private sectors to generate new ideas for solving problems, cre-ating designs, and improving products or processes. In such a contest, a firm or an organization (the seeker) outsources

an ideation task online to a distributed population of independent agents (solvers) in the form of an open call. Solverscompete to exert efforts and the one with the best solution wins a bounty. In evaluating solutions, the seeker typically hassubjective taste that is unobservable to solvers. In practice, the seeker often provides solvers with feedback, which dis-closes useful information about her private taste. In this study, we develop a game-theoretic model of feedback in unblindideation contests, where solvers’ solutions and the seeker’s feedback are publicly visible by all. We show that feedbackplays an informative role in mitigating the information asymmetry between the seeker and solvers, thereby inducing sol-vers to exert more efforts in the contest. We also show that some key contest and solver characteristics (CSC, includingcontest reward, contest duration, solver expertise, and solver population) have a direct effect on solver effort. Interestingly,by endogenizing the seeker’s feedback decision, we find that the optimal feedback volume increases with contest reward,contest duration, solver expertise, but decreases with solver population. Thus, CSC elements also have an indirect effect onsolvers’ effort level, with feedback volume mediating this effect. Employing a dataset from Zhubajie.com, a leading onlineideation platform in China, we find empirical evidence that is consistent with these theoretical predictions.

Key words: ideation contest; feedback; information asymmetry; crowdsourcing; contest and solver characteristicsHistory: Received: January 2019; Accepted: October 2019 by Fred Feinberg, after 2 revisions.

1. Introduction

In today’s world, businesses across various industriesneed to innovate and evolve faster than ever before tostay ahead of the competition. Researchers and practi-tioners have placed an increasing focus on ideation—the process of generating new ideas and solutions forbusiness problems, creating designs, and improvinggoods, services or processes. In the past, employeesand contractors were the principal sources of newideas. However, in-house ideation can be costly, sub-ject to free riding, and limited by the number andscope of ideas (Luo and Toubia 2015, Toubia 2006).Recent technological advances have enabled firmsand organizations (i.e., seekers) to outsource ideationprojects on the Internet to a distributed group ofexperts and enthusiasts (i.e., solvers) in the form of anopen call, or “crowdsourcing” (Bayus 2013, Boudreauand Lakhani 2013, Howe 2006). In particular, ideationcontest has gained massive popularity where thecrowdsourcing process is organized as an open con-test with a bounty for the best ideas (Prpi�c et al. 2015,Terwiesch and Ulrich 2009). As compared to tradi-tional methods, ideation contest can be exceptionally

effective in generating more and better solutions at alower cost from a large number of solvers withdiverse knowledge and expertise (Gupta et al. 2019,Jiang et al. 2018, Poetz and Schreier 2012).For marketing professionals, ideation contests offer

an easy, practical and cost-effective method to tap intolarge online communities for creative ideas and out-of-the-box solutions. They are able to crowdsourceconcepts for the next-generation products, designs ofnew landmarks and buildings, logo designs forbrands and events, and many more. For instance, inits annual contest “Crash the Super Bowl,” Frito-Layinvites consumers to create commercials for Doritosand promises at least one entry to be aired during theSuper Bowl. From 2006 to 2016, Frito-Lay receivedover 36,000 entries, which helped the company winthe first place on the USA Today Ad Meter Poll anumber of times. AT&T, Coca-Cola, and AmericanExpress all sponsor online creativity contests to seekinnovative ideas for the company (Rathi 2014). Asanother example, Threadless has built its entire busi-ness model on ideation contests. Every week, itreceives about a thousand designs from its onlinecommunity of artists. Ten of them are selected as

Please Cite this article in press as: Jiang, J., Y. Wang. A Theoretical and Empirical Investigation of Feedback in Ideation Contests. Pro-duction and Operations Management (2019), https://doi.org/10.1111/poms.13127

Vol. 0, No. 0, xxxx–xxxx 2019, pp. 1–20 DOI 10.1111/poms.13127ISSN 1059-1478|EISSN 1937-5956|19|00|0001 © 2019 Production and Operations Management Society

http://Zhubajie.com

https://doi.org/10.1111/poms.13127

winning designs, which are then used for makingclothing and other products. The winning artistsreceive a reward as well as royalties for sales. Whereaslarge companies are able to host ideation contestsindependently, smaller firms and organizations oftenresort to established online ideation platforms, such asInnoCentive, 99Designs, Zhubajie, and IdeaConnec-tion, that help connect seekers with millions of poten-tial solvers.Despite its popularity, ideation contest suffers from

information asymmetry between the seeker and sol-vers. This is because ideation contests typicallyinvolve broad and non-detailed tasks where relevantparties have their own private and subjective taste forpotential solutions (Boudreau et al. 2011, Prpi�c et al.2015, Terwiesch and Xu 2008). Although the seekerdetermines the winning solution based on her prefer-ences, solvers do not have full information aboutthem. Consider a company running an ideation con-test for a new logo. The seeker—the top executive or agroup of decision-makers in the company—may havepreferences for the color, font, style, shape, and size ofthe logo, or any combination of the above. Yet, solversparticipating in the contest may not fully comprehendthe subtlety in the logo design. “Does the seeker like amodern or retro feel, a dynamic or static image, amasculine or feminine look?” More broadly speaking,logos subtly convey brand personality and position-ing—something the seeker understands but cannotfully articulate.Interestingly, during the course of the contest, the

seeker often provides solvers with feedback com-ments that contain useful information about her pri-vate taste. To the best of our knowledge, there is scantresearch on seekers’ feedback even though it is a com-monly seen feature across many online ideation plat-forms. To fill this gap, we ask three importantresearch questions in the context of ideation contests.First, how does the seeker’s feedback volume affectsolvers’ effort level? Second, what is the impact ofcontest and solver characteristics (CSC) on solverefforts? In this study, we focus our attention on fourCSC elements: contest reward, contest duration, sol-ver expertise, and solver population. Last but not theleast, by endogenizing the feedback volume decisionof the seeker, we are able to examine how CSC ele-ments influence feedback volume. Specifically, we areinterested in understanding whether feedback func-tions as a mediating variable in the relationshipbetween CSC and solvers’ effort level. The answer tothe last research question is central to understandingthe underlying process through which CSC elementsaffect solvers’ effort level and has important implica-tions on the design of ideation contests.To address these research questions, we develop a

game-theoretic model of ideation contest with

feedback, in which a number of solvers compete towin a pre-specified amount of reward. The seekerselects the winner based on solvers’ performance,which in turn depends on their effort and the matchvalue between the seeker and solvers. This matchvalue term captures the information asymmetry dueto the private taste of the seeker. That is, while the see-ker has perfect information about her own taste, sol-vers do not but have some belief about it. The seekercan provide feedback that discloses information onher taste. Feedback is modeled as a noisy signal thatallows solvers to update their belief following Baye-sian learning, thus resulting in greater match value. Inthis model, while solvers choose how much effort toexpend, the seeker decides the optimal feedback vol-ume. Consistent with our empirical context, ideationcontest is “unblind” in the sense that both solutionssubmitted by solvers and feedback provided by theseeker are publicly visible to all (Bockstedt et al. 2016,Wooten and Ulrich 2015, 2017).Our theoretical results reveal an informative role of

feedback: feedback mitigates the information asym-metry between the seeker and solvers, thereby induc-ing solvers to expend greater effort. We also identify adirect effect of CSC in that solvers’ optimal effort levelincreases with contest reward, contest duration andsolver expertise, but decreases with solver popula-tion. By endogenizing the seeker’s feedback strategy,we show that the seeker’s optimal feedback volumedepends on CSC. Specifically, the optimal feedbackvolume increases with contest reward, contest dura-tion, and solver expertise, but decreases with solverpopulation. Thus, our analysis uncovers an indirecteffect CSC elements have on solvers’ effort throughfeedback volume, indicating that the seeker’s feed-back plays a mediating role in the relationship betweenCSC and solvers’ effort. We collect data from 9771logo design contests hosted on Zhubajie.com, theleading ideation platform in China. We find empiricalevidence that is consistent with our theoretical predic-tions in the context of logo design contests.In addition, our key theoretical findings are invari-

ant when model assumptions are relaxed to allow forendogenous solver entry and pre-feedback solverefforts. Various robustness tests also indicate that ourempirical results are insensitive to alternative defini-tions and operationalization of key variables.Our contribution to the literature is fourfold. First,

we contribute to the marketing literature on new pro-duct innovation and idea generation (e.g., Luo andToubia 2015, Stephen et al. 2016, Toubia 2006) bymodeling and empirically analyzing online ideationcontests with feedback, a relatively new yet promis-ing idea generation process that has been understud-ied in the marketing field. Second, while informationasymmetry between the seeker and solvers is an

Jiang and Wang: The Role of Feedback in Ideation Contest2 Production and Operations Management 0(0), pp. 1–20, © 2019 Production and Operations Management Society


http://Zhubajie.com


inherent feature of ideation contests, to the best of ourknowledge, we make the first attempt to investigatethe use of feedback to mitigate information asymme-try. Our analytical model provides theoreticalgrounds for the informative role of feedback, which isvalidated empirically using the dataset we collectedfrom Zhubajie.com. Thus, our theory and empiricalinvestigation complement prior literature by provid-ing a new explanation for the use of feedback in idea-tion contests.1 Third, this study complements theprevious literature on the optimal design of ideationcontests by simultaneously examining four CSC vari-ables. Although prior research has investigated con-test reward (Gupta et al. 2019, Liu et al. 2014, Slot2013) and solver population (Boudreau et al. 2011,Terwiesch and Xu 2008), contest duration and solverexpertise are largely ignored. Our work fills this gapin the literature. Lastly, we endogenize the seeker’sfeedback decision to reveal that the seeker choosesfeedback volume strategically depending on CSC.Thus, feedback plays a partial mediating role in therelationship between CSC and solver effort. This find-ing is further validated through empirical analysis.As such, we contribute to the previous studies ononline feedback since they unanimously take feed-back as exogenous (e.g., Wooten and Ulrich 2017).The remainder of this study is organized as follows.

Section 2 discusses the related literature. Section 3presents the theoretical model and solution, fromwhich we develop our hypotheses. Section 4 empiri-cally tests the hypotheses. In section 5, we examinethe robustness of our main results. We present theconclusions, limitations and future research directionsin section 6.

2. Literature Review

Our research draws on two streams of literature. Wesee our paper contributing to the marketing literatureon ideation. Marketing practitioners and researchershave long recognized the importance of ideation, as itis essential to the “design and marketing of new prod-ucts, to marketing strategy, and to the creation ofeffective advertising copy” (Toubia 2006). Earlyresearch on ideation mostly focuses on the in-houseidea generation and selection process (e.g., Burroughset al. 2011, Rossiter and Lilien 1994, Toubia 2006, Tou-bia and Flor�es 2007). Later works examine firm prac-tice that takes a direction to involve external ideatorson online platforms (see, e.g., Bayus 2013, Chan et al.2015, Luo and Toubia 2015, Stephen et al. 2016).Bayus (2013) investigates Dell’s IdeaStorm forumwhere users, who are mainly Dell customers, proposeideas and Dell decides whether to implement them.He finds that users with past success are less likely tocontinue proposing good ideas, while comments from

other users can effectively mitigate this negativeeffect. Using similar data collected from Dell’s Idea-Storm, Chan et al. (2015) reveal how peer-to-peer andpeer-to-firm interactions influence users’ subsequentidea generation. Luo and Toubia (2015) find that ononline idea generation platforms the task structurehas contrasting effects on users with different levelsof knowledge—high-knowledge users prefer abstractcues such as problem decomposition while low-knowledge users do better with concrete cues such asstimulus ideas. In a series of experiments, Stephenet al. (2016) show that high connectivity of users inonline ideation platforms can lead to low innovative-ness, as connected users tend to come up with similarideas and receive similar or redundant inspirationsfrom one another. The context of these studies is dif-ferent from ours because they focus on platformswhere non-contest ideation processes take place. Weextend this literature stream by explicitly modelingideation contests in which multiple solvers competeto provide the best solution.2,3 We also empiricallytest the role of feedback in these ideation contests.This study also draws on the emerging literature

that investigates various issues in ideation contests:information asymmetry, feedback, and contest andsolver characteristics. Terwiesch and Xu (2008) areamong the first to formally propose a model of idea-tion contests. They identify information asymmetry asthe key difference between ideation contests andother types of contests (e.g., expertise-based contestsand trial-and-error contests). They suggest that “theseeker’s taste, which is uncertain for the solver, playsan important role in determining what constitutes agood solution” and “the performance of a solutionhas a significant noise term that reflects heterogeneityof solvers’ solutions in matching the seeker’s taste.”Building on Terwiesch and Xu (2008), Boudreau et al.(2011) further show that information asymmetry cangreatly influence market outcomes using a datasetfrom Topcoder. However, neither paper explicitlyexplores the possibility that seekers can be strategic inusing feedback comments to reduce informationasymmetry, a question we address.Despite its prevalence, the use of feedback in the

context of ideation contests has largely been ignored.4

A notable exception is Wooten and Ulrich (2017) whoran a field experiment on performance feedback (inthe form of ratings) in ideation contests. Specifically,they compare three alternative feedback policies: a di-rected feedback policy that gives feedback related to sol-vers’ performance during the contest, a randomfeedback policy that leaves completely random feed-back, and a no feedback policy where no feedback isprovided at all. They show that, compared with theother two policies, the directed feedback policyinduces solver participation and reduce quality

Jiang and Wang: The Role of Feedback in Ideation ContestProduction and Operations Management 0(0), pp. 1–20, © 2019 Production and Operations Management Society 3


http://Zhubajie.com


variation as the contest progresses. Moreover, direc-ted feedback policy could result in lower top-endquality of ideation solutions, although the averagequality is unambiguously higher. Instead of ratings,we study text comments that reveal the seeker’s tastein ideation solutions. We show that feedback com-ments can effectively reduce the information asym-metry inherent in ideation contests and lower solvers’risk of investing efforts. In addition, Wooten andUlrich (2017) take feedback as an exogenous processthrough explicit manipulation in the experimentaldesign. In contrast, we allow the seeker to makestrategic decisions on feedback volume. We find boththeoretically and empirically that seekers are indeedstrategic in choosing feedback volume depending onthe CSC elements of the contests.Our study also adds to the nascent but growing

literature on the optimal design of ideation contests.This literature stream examines how contest charac-teristics, such as solver population and contestreward affect the output of ideation contests. In ananalytical model, Terwiesch and Xu (2008) showthat although a larger solver population can reducesolver effort by intensifying competition, open entrycan still be an optimal strategy for the seeker if theprimary concern is the maximum performance fromthe crowd, because a large solver population canincrease solver diversity and improve the output ofthe contest. Subsequently, Boudreau et al. (2011)analyze data from Topcoder and find that a largersolver population reduces solvers’ average perfor-mance but has no significant impact on the maxi-mum performance, providing empirical support forTerwiesch and Xu (2008). In a field experiment con-ducted on Taskcn.com, Liu et al. (2014) find that ahigher contest reward can induce solvers to submitmore solutions of higher quality. Hofstetter et al.(2018) demonstrate that winning a contest andreceiving a reward can positively influence solverparticipation in subsequent contests. Based on thisresult, they recommend that the contest platformshould encourage multiple reward levels in ideationcontests. Terwiesch and Xu (2008) reveal that a per-formance-contingent reward regime can more effec-tively motivate solvers to expend their effort than afixed-price reward regime. Jiang et al. (2018) sug-gest that instant cash rewards could lower the qual-ity of solutions by drawing a large number ofunqualified solvers into the contest. Gupta et al.(2019) cast doubt on the use of ideation contests inuser content generation by showing that monetaryreward may reduce the amount of non-monetaryincentives (such as “likes” and “upvotes”) awardedby peers for social recognition, due to the “taint”effect. Slot (2013) examines the size, number, andspread of rewards on the creativity (the degree of

originality, unusualness, and novelty) of solvers inonline idea generation contests. She finds that sol-vers’ creativity increases with the size and numberof rewards but decreases with the spread ofrewards. We extend this literature stream by simul-taneously examining the impact of CSC elements(solver population, contest reward, contest duration,and solver expertise) on solver effort, as well as themediating role of feedback in the relationshipbetween CSC and solvers’ effort level.

3. Theory and HypothesisDevelopment

Consider an unblind ideation contest initiated by aseeker in which there are N (N ≥ 2) homogenous sol-vers (indexed by i). Following the contests and tour-naments literature (e.g., Aoyagi 2010, Ederer 2010,Terwiesch and Xu 2008), solver performance can bemeasured in a one-dimensional space. We model theideation contest as a single-winner contest. The seekerselects the solver with the highest performance andoffers a pre-specified reward V, while the remainingsolvers receive zero payoff. This single-winner alloca-tion rule is the most popular format in ideation con-tests (Terwiesch and Xu 2008).The game unfolds in two stages. In the first stage,

the seeker determines her feedback volume. In thesecond stage, all solvers simultaneously choose howmuch effort to put into the contest. The sequence ofmoves reflects the fact that solvers’ effort decisionsare more flexible than the seeker’s feedback volumedecision, that is, solvers can always decide whetherto expend more effort after the seeker gives feed-back and the volume of feedback is known.5 Weemploy the subgame perfect Nash equilibrium andsolve the model by backward induction. Modelnotations are summarized in Table A1 in OnlineAppendix A.

3.1. Stage Two: Solvers’ Effort LevelSolvers are risk neutral and exogenously endowedwith expertise a. Solver i’s performance when expend-ing effort level ei is given by the following:

di ¼ aei þ li: ð1ÞThe first term takes the multiplicative form, whichimplies that the marginal return of effort on perfor-mance increases with solver expertise (Ederer 2010,Moldovanu and Sela 2001).6 The second term cap-tures the match value that is determined by howwell solver i’s solution appeals to the seeker’s taste.Before receiving feedback, solver i’s match value is arandom draw from a normal distribution with amean of lprior and a variance of s2:



http://Taskcn.com


li �N lprior; s2

� �: ð2Þ

Feedback comments are conceptualized as noisy sig-nals that contain information about the seeker’s privatetaste, which can help solvers improve the match valueof their solutions. Suppose that the seeker provides atotal of M feedback signals, X ¼ xmf gm¼1;...;M, whichare observed by all solvers because the contest is “un-blind” in nature. We assume that all feedback signalsare i.i.d. drawn from a normal distribution centeredaround lfeedback, given by the following:

xm �N lfeedback; r2

� �: ð3Þ

We assume lfeedback > lprior to capture the fact that feed-back contains useful information about the seeker’staste. r2 is the variance and 1=r2 measures the precisionof feedback signals. We assume that solvers followBayesian learning; that is, they learn from feedback sig-nals X ¼ xmf gm¼1;...;M such that their match values areupdated following the Bayes’ rule. This approach pro-vides a parsimonious yet flexible framework for model-ing the effect of feedback on solvers’ performance.Thus, upon receivingM feedback signals, solver i’s pos-terior match value follows the following distribution:

lijX�NMs2

Ms2 þ r2�xþ r2

Ms2 þ r2lprior;

s2r2

Ms2 þ r2

� �;

ð4Þ

where �x ¼ 1M

PMm¼1

xm is the sample mean. From Equa-

tion (4), we observe that feedback has two sepa-rate effects on solvers’ match value: a match-enhancing effect and an uncertainty-reduction effect. Thematch-enhancing effect occurs because the posterior

mean of the match value E Ms2

Ms2þr2�xþ r2

Ms2þr2 lpriorh i

¼Ms2

Ms2þr2 lfeedbackþ r2

Ms2þr2 lprior improves and increasingly

approaches lfeedback as feedback volume M increases.The uncertainty-reduction effect emerges as the vari-

ance of the posterior match value s2r2

Ms2þr2 decreases

with M, which implies that solvers’ posterior matchvalue becomes less dispersed when they receive morefeedback signals. In other words, solvers are morecertain about the seeker’s taste when feedback vol-ume is higher.Equation (4) indicates that, after receiving feedback

signals, solvers’ performance follows a normal distri-bution given by the following:

dijX�N aei þ Ms2

Ms2 þ r2�xþ r2

Ms2 þ r2lprior;

s2r2

Ms2 þ r2

� �:

ð5Þ

Thus, the difference in performance between solversi and k is also normally distributed:

di � dkjX�N a ei � ekð Þ; 2s2r2

Ms2 þ r2

� �: ð6Þ

When comparing two solvers’ performance, we findfrom Equation (6) that the match-enhancing effect offeedback is cancelled out because they received thesame feedback signals, but the uncertainty-reductioneffect remains. Consequently, the probability thatsolver i achieves a higher level of performance thansolver k is given by the following:

Pr di [ dkjXð Þ ¼ Ua ei � ekð Þ Ms2 þ r2

� �2s2r2

� �; ð7Þ

where U �ð Þ is the c.d.f. of a standard normal distribu-tion. Thus, solver i’s winning probability is given bythe following:

Pr winið Þ ¼Yk 6¼i

Pr di [ dkjXð Þ

¼Yk 6¼i

Ua ei � ekð Þ Ms2 þ r2

� �2s2r2

� �: ð8Þ

We assume that solvers have identical utility func-tions that are separable with respect to the benefitexpected from the contest and the cost of the effortexerted. Solver i’s expected benefit from participatingin the contest, V Pr winið Þ, increases with contestreward and winning probability. We assume a quad-ratic cost function of effort a Tð Þei2, where a Tð Þ is thecost sensitivity of effort and is a function of contestduration T. a Tð Þ is assumed to be sufficiently large toguarantee the concavity of the utility function, orequivalently, a Tð Þ[ �a, where �a is a function of modelparameters and is defined in Online Appendix B. Inline with the prior literature (e.g., Zhang 2016), weassume that the marginal cost of effort decreases withcontest duration, a0 Tð Þ\ 0. The intuition of thisassumption is that when the contest duration islonger, solvers are less likely to face time pressureand time constraints, so the marginal cost of effortbecomes lower.As such, in stage two, each solver trades off his

effort for the likelihood of winning the reward bychoosing the effort level that maximizes the expectedutility:

maxei

U eið Þ ¼ V Pr winið Þ � a Tð Þei2: ð9Þ

We focus on symmetric equilibrium in the sense thatsymmetric solvers choose the same effort level. Thesolvers’ optimal effort level is characterized inProposition 1.




PROPOSITION 1. There is a unique symmetric pure-strat-egy Nash equilibrium in solvers’ effort level:

e� ¼ aV N � 1ð Þ Ms2 þ r2� �

2N�12ffiffiffip

pa Tð Þs2r2 : ð10Þ

The equilibrium effort level increases with feedback vol-ume, contest reward, contest duration, and solver exper-tise, but decreases with solver population.

PROOF. See Online Appendix B.

Proposition 1 indicates that solvers expend greaterefforts when they receive more feedback commentsfrom the seeker. This result can be attributed to theuncertainty-reduction effect of feedback. As feedbackvolume increases, solvers become more certain aboutthe seeker’s taste and the extent of information asym-metry between the seeker and solvers is mitigated.With more precise information, solvers can betterassess their performance relative to those of othersand hence their likelihood of winning. As a result, sol-vers strategically invest greater efforts into the con-test. We call this the informative role of feedback.Proposition 1 also identifies a direct effect of CSC onsolvers’ effort level. The benefit of exerting effort ishigher when contest reward increases. Therefore, sol-vers expend more effort when reward is higher.Longer contest duration reduces the marginal cost ofeffort and hence incentivizes solvers to exert moreeffort. Higher solver expertise boosts the marginalreturn of effort on performance and thus induces sol-vers to expend greater effort. In addition, an increasein solver population intensifies competition amongsolvers, thereby decreasing their winning probability.Thus, solvers strategically reduce effort when solverpopulation increases.

3.2. Stage One: Seeker’s Feedback StrategyIn this subsection, we return to stage one, in whichthe seeker rationally anticipates solvers’ behaviorwhen choosing her feedback volume to maximizeprofit given by:

maxM

P ¼ E maxi

dif g

� V � bM: ð11Þ

The first term is the expected maximum perfor-mance of all solvers, which is the expected benefitthat the seeker obtains from the contest. At thesame time, the seeker incurs costs from two sources.First, the seeker needs to pay the winner a contestreward of V. Second, composing and sending feed-back is costly. We assume a linear cost function ofbM. To guarantee the existence of an interior

solution, we innocuously assume that �b [ b [ b,where both �b and b are functions of the modelparameters specified in Online Appendix B. InProposition 2, we characterize the seeker’s optimalfeedback volume.

PROPOSITION 2. The seeker’s optimal feedback volume isgiven by:

M� ¼ rs

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffilfeedback � lprior � s2

R1�1 zdU zð ÞN

b� a2V N�1ð Þ2Nþ1

2ffiffip

pa Tð Þr2

vuuut � r2

s2: ð12Þ

The equilibrium feedback volume increases with contestreward, contest duration, and solver expertise, butdecreases with solver population.


This proposition can be explained by how CSCelements impact the uncertainty-reduction effect offeedback. Higher contest reward magnifies theuncertainty-reduction effect and, as a result, themarginal benefit of providing feedback comments isgreater. Recognizing this, the seeker strategicallyincreases feedback volume. Similarly, the uncer-tainty-reduction effect and thus the marginal benefitof offering feedback are enhanced when contestduration is longer, solver expertise is higher, andsolver population is smaller. Propositions 1 and 2together imply CSC’s indirect effect—through feed-back volume—on solvers’ effort level. In otherwords, the relationship between CSC and solvers’effort level is partially mediated by feedbackvolume. Table 1 summarizes our key model predic-tions from Propositions 1 and 2.

3.3. Hypotheses DevelopmentOur theoretical framework generates three sets ofhypotheses that are amenable to empirical testing.Drawn from Proposition 1, Hypothesis 1 is concernedwith how feedback volume affects solvers’ effortlevel; Hypothesis 2 lays out the direct effect of CSC onsolvers’ effort level. Hypothesis 3 is derived from Pro-position 2 and addresses how CSC elements affectfeedback volume.

Table 1 Summary of Key Model Predictions

Variable Solvers’ effort level The Seeker’s feedback volume

Feedback Volume ↑ n/aContest Reward ↑ ↑Contest Duration ↑ ↑Solver Expertise ↑ ↑Solver Population ↓ ↓




HYPOTHESIS 1. (H1). The Effect of Feedback Volume onEffort: Solvers’ efforts are higher in ideation contests withgreater feedback volume.

HYPOTHESIS 2. (H2). The Direct Effects of Contest andSolver Characteristics on Effort:

H2(a). Solvers’ efforts are higher in ideation contestswith higher rewards.H2(b). Solvers’ efforts are higher in ideation contestswith longer duration.H2(c). Solvers’ efforts are higher in ideation contestswith higher solver expertise.H2(d). Solvers’ efforts are lower in ideation contestswith larger solver population.

HYPOTHESIS 3. (H3). The Effects of Contest and SolverCharacteristics on Feedback Volume:

H3(a). The seeker provides more feedback when contestreward is higher.H3(b). The seeker provides more feedback when con-test duration is longer.H3(c). The seeker provides more feedback when solverexpertise is higher.

H3(d). The seeker provides less feedback when the sol-ver population is larger.

Hypotheses 1–3 are summarized in Figure 1.7

4. Empirical Investigation

4.1. Research ContextOur empirical setting is Zhubajie.com, a leading idea-tion contest website in China founded in 2006. As ofDecember 2015, Zhubajie had hosted over 5 millioncontests, attracted 13.6 million solvers, and given outnearly US$3 billion in rewards. It is the world’s lar-gest online marketplace for ideation contests in termsof the size of the solver pool.8 Registration on Zhuba-jie is free for both seekers and solvers, and there is nofee for solvers to participate in contests. Zhubajie col-lects 20% of contest reward as its commission and dis-tributes the remaining 80% to the winner—a revenuemodel similar to those of other platforms.Zhubajie offers services in many categories. Of

these, graphic/logo design, marketing/sales, andweb design are the top three in terms of popularity.We focus on the logo design category for the follow-ing reasons. First, because logo design is a type ofideation project, information asymmetry arising fromthe seeker’s private taste is prominent (Terwiesch and

FeedbackVolume Effort

Solver Expertise

Solver Population

-Hypothesis 2(d)

+Hypothesis 2(c)

+Hypothesis 1

+Hypothesis 3(c)

-Hypothesis 3(d)

Contest and Solver Characteristics (CSC)

Contest Duration Hypothesis 2(b)

+Hypothesis 3(b)

+

Contest Reward Hypothesis 2(a)

+Hypothesis 3(a)

+

Figure 1 Model Framework and Summary of Hypotheses



http://Zhubajie.com


Xu 2008). Second, the company or brand logo is oneof the most important business assets lying at the coreof business branding strategies. Third, logo designrequires considerable effort from solvers, so they tendto strategically choose their effort level when facingcompetition. Lastly, logo design contests display largevariations in CSC, hence providing a rich context forempirical analysis.A logo design contest begins when a seeker posts

an announcement of the task on the platform.Figure 2 provides an example from Witmart.com,Zhubajie’s US portal. AArealty, the seeker in thisexample, created a logo design contest with a US$200reward, a 30-day duration, and a description statingthe task’s purpose, requirements and basic informa-tion about the company. Then, solvers could registerand begin submitting solutions. During the time per-iod covered by our data (between March 2007 andMarch 2010), Zhubajie adopted an unblind formatwhere all submitted logo designs and seeker feedbackcomments were viewable by all contestants. Thispractice is consistent with our theoretical modelsetup. Figure 3 shows an example of AArealty’s feed-back about a solution, which can be accessed by allsolvers. At the conclusion of the contest, the seekerselected the winner who then received the reward.

4.2. Data DescriptionWe collected information relevant to our study fromall 11,584 logo design contests hosted by Zhubajiebetween March 2007 and March 2010. We first

dropped incomplete contests and those with morethan one winner, which comprised nearly 8% of thecontests in our sample. The major reason for multiplewinners was that when a seeker failed to select a win-ner within the allotted time frame, the platformwould step in and evenly distribute the reward to allparticipating solvers. Next, we eliminated contests inwhich only one solver participated because the solvermight not have acted strategically when there was nocompetition. Lastly, we removed contests with areward below 50 Chinese yuan (CNY).9 Most of thesecontests were launched as test runs and thus failed toattract solvers. The resulting dataset included a sam-ple of 9771 logo design contests in which 415,779 sol-vers submitted a total of 572,046 logo designs andwon nearly 3.4 million CNY in rewards. An averagelogo contest in our data offered a reward of 346.55CNY, had a duration of 10.72 days, attracted 54.51solvers, included 6.70 feedback comments, andreceived 57.24 solutions.

4.3. VariablesEffort. On the solver side, our key dependent variableis the effort level. Since solver effort is not directlyobservable, we measure it by Number of Solutions,defined as the total number of logo designs a solversubmitted in the course of the contest. In general, alarger number of solutions indicate higher effort levelexerted by the solver.10 Number of Solutions is thenaveraged over all solvers in the contest to obtain con-test-level measure of solver effort.

Figure 2 Example of a Logo Design Contest [Color figure can be viewed at wileyonlinelibrary.com]



http://Witmart.com

www.wileyonlinelibrary.com


Feedback Volume. We define feedback as the text com-ments sent to solvers by the seeker regarding her taste(such as preferences for the colors, style, font, and sizeof the logo). Two independent coders read each com-ment and judged whether it contained informationabout the seeker’s taste. Intercoder agreement was92%, suggesting high consistency in identifying see-ker feedback. Where disagreements existed, theywere resolved by the authors. Feedback Volume foreach contest is the total number of text commentsfrom the seeker that contained taste information.Contest Reward. Contest reward is the monetary

incentive offered by the seeker to the winner. Weexpress the reward amount in CNY.Contest Duration. The duration of the contest is cal-

culated as the difference between its end and startdates expressed in the number of days.11

Solver Population. Solver population is the totalnumber of solvers who participated in the contest.Solver Expertise. Prior research used various para-

digms to assess expertise, including prior knowl-edge, task-specific skills, and acquired experience(Winch 2010). We employed Average Income as the

measure of solver expertise, because a solver’strack record is a strong indicator of his knowledge,skills and experience. Specifically, Average Income isobtained by dividing a solver’s cumulative incomeearned from the ideation platform (in CNY) by thetotal number of contests in which he participated.Then, we compute the average over all solvers inthe contest to obtain the contest-level measure ofsolver expertise.Feedback-Seeking Volume. The feedback literature in

psychology and organizational behavior has longrecognized that recipients do not passively wait forfeedback. Instead, they proactively request feedbackfrom the provider, thereby engaging in the so-called“feedback-seeking behavior” (for a review, see Ash-ford et al. 2003). In our research context, solversseek feedback on Zhubajie through text messageswith explicit requests. Two independent coders readthe solvers’ text messages to identify feedback-seek-ing behavior. Intercoder agreement was 88%, sug-gesting high consistency in identifying solvers’feedback-seeking behavior. Where there were dis-agreements, the authors resolved them. For each

Figure 3 Example of Feedback Related to Logo Design [Color figure can be viewed at wileyonlinelibrary.com]



www.wileyonlinelibrary.com


contest, feedback-seeking volume was measured bythe total number of feedback-seeking text messagesfrom solvers. Although feedback-seeking volumewas not considered in our analytical model, weused it as an instrument variable in the empiricalanalysis.Description Length. Description Length is the total

number of words (i.e., Chinese characters) in the taskdescription. We use it as an instrumental variable totackle the potential endogeneity problem.Table 2 summarizes the descriptive statistics and

the correlation matrix of these variables.

4.4. Empirical AnalysisWe now turn to the empirical framework to test thethree sets of hypotheses established in section 3. Sinceempirical analysis is conducted at the contest level,we use subscript j for contests. We propose the fol-lowing specification to assess how feedback and CSCaffect the solvers’ effort level:

ln Effortj þ 1� � ¼ b0 þ b1 ln Feedback Volumej þ 1

� �þ b2 ln Contest Rewardj

� �þ b3 ln Solver Expertisej þ 1

� �þ b4 ln Solver Populationj

� �þ b5 ln Contest Durationj

� �þ ej:

ð13ÞSimilarly, to assess the effect of CSC on feedbackvolume, we develop the following econometricmodel:

ln Feedback Volumejþ 1� �¼ b6þb7 ln Contest Rewardj

� �þb8 ln Solver Expertisejþ 1

� �þb9 ln Solver Populationj

� �þb10 ln Contest Durationj

� �þ tj:

ð14Þ

We add one to their value and then take the naturallog for variables that could take the value of zero(e.g., Effort, Solver Expertise and Feedback Volume),while performing the log-transformation for othervariables. This method facilitates the interpretationof our results. In Equations (13) and (14), ɛj and υjinclude all unobservable factors that affect solvers’effort level in contest j and seeker j’s feedback vol-ume, respectively. Additionally, both ɛj and υj arenormally distributed.If ɛj and υj were independent of each other, we

would be able to separately estimate the two regres-sion models in Equations (13) and (14). However,there may be unobservable heterogeneity among con-tests in the error terms that simultaneously affects thesolvers’ choices of effort and the seeker’s decisionabout feedback volume. In other words, ɛj and υj arelikely to be correlated. Consequently, we mustaddress two econometric issues in our specifications.A trivial issue is that separate estimations of Equa-tions (13) and (14) lead to efficiency loss in parameterestimates, which can be easily solved by using seem-ingly unrelated regression (SUR). Moreover, weshould also account for a simultaneity problem inEquation (13) because Feedback Volume is endoge-nously determined, leading to biased estimates(Greene 2000).12

We pursue the instrumental variable (IV) approachto correct the endogeneity bias. Valid instrumentsshould correlate with the seeker’s decision on feed-back volume and affect the solvers’ effort levelthrough feedback volume only. Our first instrumentis Feedback-Seeking Volume. It has long been shownthat feedback-seeking behavior has a direct and posi-tive impact on feedback volume (Ashford and Cum-mings 1983, Ashford and Tsui 1991). At the sametime, feedback-seeking behavior is unlikely to bedirectly correlated with solvers’ effort level, sincethere is no obvious link between them. Our secondinstrument is Description Length. Contest description

Table 2 Summary Statistics and Correlation Matrix

Variable Mean SD Min Max (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11)

(1) Number of Solutions 1.05 0.25 0.13 5.92 1(2) Elapsed Time 3.69 6.03 0.02 299.76 0.13 1(3) Feedback Volume 6.70 13.31 0 409 0.31 0.07 1(4) Contest Reward 346.55 464.43 50 10,000 0.05 0.20 0.19 1(5) Contest Duration 10.72 12.46 1 376 0.07 0.87 0.09 0.28 1(6) Solver Population 54.51 47.51 2 1060 0.12 0.17 0.24 0.63 0.24 1(7) Average Income 95.29 54.36 0 428.96 0.09 0.13 0.08 0.35 0.10 0.21 1(8) Tenure 3.49 1.73 0 12.57 0.03 0.04 0.01 0.03 �0.03 �0.02 0.69 1(9) Experience 274.37 237.93 0 3647 �0.12 �0.05 �0.05 �0.12 �0.07 �0.14 0.21 0.58 1(10) Feedback-SeekingVolume

7.18 18.02 0 930 0.45 0.17 0.44 0.38 0.18 0.57 0.08 �0.05 �0.13 1

(11) Description Length 63.98 26.92 0 173 �0.02 �0.09 �0.09 0.50 0.20 0.36 0.31 0.05 �0.08 0.12 1




is another way for the seeker to provide contest-related information, including her private taste. Weexpect that a lengthy (vs. short) contest descriptioncontains more information about the seeker’s prefer-ences; therefore, there is less need for utilizing feed-back. Stated differently, Description Length isnegatively related to Feedback Volume. At the sametime, Description Length has no direct effect on solvers’effort level. Thus, these two instruments can help uscontrol for unobserved contest and solver characteris-tics that affect solver effort and feedback volume atthe same time. Therefore, Feedback-Seeking Volume andDescription Length satisfy the exclusion restriction andare appropriate instruments. We add Feedback-SeekingVolume and Description Length to Equation (14) asindependent variables, as follows:

ln FeedbackVolumejþ1� �¼ b6þb7 ln ContestRewardj

� �þb8 ln ContestDurationj

� �þb9 ln SolverExpertisejþ1

� �þb10 ln SolverPopulationj

� �þb11 ln Feedback-SeekingVolumejþ1

� �þb12 ln DescriptionLengthjþ1

� �þxj:

ð15ÞWe run three-stage least square (3SLS) regressionanalysis and deal with both the simultaneity and theinefficiency issues by jointly estimating Equa-tions (13) and (15). We report the 3SLS estimationresults in Table 3. A Durbin–Wu–Hausman test onthe full model in column (3–4) indicates that endo-geneity is indeed a serious concern in our dataset(v2 = 53.8 and p ≤ 0.01).Column (3-1) presents the results for a 3SLS regres-

sion model in which only Feedback Volume and IVsare included. Our results show a positive and statisti-cally significant coefficient of Feedback Volume, thussupporting H1 (b1 = 0.209, SE = 0.007, p < 0.001).Particularly, for a 1% increase in feedback volume,the number of logo designs submitted by each solverincreases by 0.21%. With an average solver popula-tion of 54.51, this amounts to 11.45% increase in thetotal number of solutions in the contest. Thus, wefind strong evidence for the informative role offeedback in mitigating the information asymmetryinherent in ideation contests, thereby inducingsolvers to invest more effort in the contest. Addition-ally, the coefficients of both IVs are statistically sig-nificant at the 0.001 level, their signs are as expected,and their values are consistent across other modelspecifications. These results indicate that feedback-seeking behavior encourages the seeker to providemore feedback comments, while a detailed taskdescription reduces the need for the seeker to offerfeedback comments.

Column (3-2) includes CSC in the solver effortmodel, allowing us to examine the direct effect ofCSC on solvers’ effort, as proposed in H2. H2(a) sug-gests that an increase in contest reward can directlymotivate higher effort exerted by solvers. As pre-dicted in H2(a), the coefficient of ln(Contest Reward) ispositive and significant at the 0.001 level (b2 = 0.025,SE = 0.001, p < 0.001). This finding is consistent withthat of the field experiment by Liu et al. (2014), whomanipulated the contest reward at two levels, 100CNY and 300 CNY, and found that solvers submittedmore solutions in the high (vs. low) reward condition.The magnitude of the effect in our study is also com-parable to theirs. As expected in H2(b), our resultssuggest a positive effect of contest duration on thenumber of solutions (b3 = 0.028, SE = 0.001,p < 0.001). Our results also indicate a strong correla-tion between solver effort and expertise, thus sup-porting H2(c) (b4 = 0.019, SE = 0.001, p < 0.001). Thecoefficient of Solver Population is negative and statisti-cally significant (b5 = �0.018, SE = 0.001, p < 0.001),thus lending support to H2(d). This finding is consis-tent with that of a similar analysis by Boudreau et al.(2011), who show that solvers’ average performanceon Topcoder is lowered when the number of solversincreases. In summary, H2 is supported, confirmingthe direct effect of CSC on solvers’ effort level.H3(a-d) present hypotheses about how the seeker

strategically chooses feedback volume, depending onCSC. To examine H3, we include CSC in the seekerfeedback model and report the results in column(3-3). We find that contest reward is positively associ-ated with feedback volume and is consistent with ourpredictions in H3(a) (b6 = 0.145, SE = 0.008,p < 0.001). The coefficient of contest duration is posi-tive and significantly different from zero, thus con-firming H3(b) (b7 = 0.134, SE = 0.008, p < 0.001). H3(c) states that solver expertise has a positive effect onfeedback volume, which is supported by our empiri-cal results (b8 = 0.085, SE = 0.007, p < 0.001). In addi-tion, we find negative and statistically significantcoefficients for solver population, supporting H3(d)(b9 = �0.089, SE = 0.008, p < 0.001). Hence, H3 is alsostrongly supported. H3 also implies the indirect effectthat CSC elements have on solvers’ effort level throughfeedback volume. H1–3 collectively suggests thatfeedback volume plays a partial mediating role in therelationship between CSC and solvers’ effort level.In column (3-4), we present the full model by

including CSC on both sides. We find that most coeffi-cients are still statistically significant at the 0.001 level,and the signs of the coefficients are consistent withour hypotheses, except that H2(a), H2(d), and H3(c)are left unsupported (b2, b5, and b8 are all insignifi-cant at the 0.05 level). This could be due to variabledefinitions and operationalization, which we




investigate further in section 5.1. Another potentialreason is data aggregation (to the contest level), whichadds more noise to the data. Based on the full modelestimates in column (3-4), we are able to compute thetotal effect of CSC on solvers’ effort level by combin-ing both the direct and indirect effects. When the see-ker increases the reward by 1%, the number ofsolutions submitted by each solver increases by 0.03%(=b2 9 1% + b1 9 b7 9 1%). Similarly, when theduration of the contest is extended by 1%, the numberof solutions submitted by each solver increases by0.03% (=b3 9 1% + b1 9 b8 9 1%). When each sol-ver’s average income increases by 1%, the number ofsolutions submitted by each solver increases by 0.02%(=b4 9 1% + b1 9 b9 9 1%). When the solver popu-lation increases by 1%, the number of solutions sub-mitted by each solver decreases by 0.02%(=b5 9 1% + b1 9 b10 9 1%). In light of the largenumber of participants in each contest (the averagebeing 54.51), these effects are statistically significantas well as economically meaningful.As a comparison, we also run the OLS, SUR, and

2SLS regression and present the estimation results inTable 4. Column (4-1) reports the OLS results. Asexpected, OLS estimates of the seeker feedback modelare overall consistent with H3 because they are unbi-ased. Nonetheless, the solver effort model suffers

from the simultaneity issue, leading to biased esti-mates. Particularly, contrary to the predictions of ourtheoretical model and the prior literature, we woulderroneously conclude that solver effort decreases withcontest reward but increases with solver population.This highlights the value of endogenizing seeker’sfeedback decision and taking account of the endo-geneity issue that arises from endogenous feedbackvolume. Then we run the SUR regression with coeffi-cients reported in column (4-2). Compared with theOLS estimates in column (4-1), the efficiency of SURcoefficients has improved, but they are still biasedand may have the wrong sign. As shown in column(4-3), 2SLS corrects the endogeneity bias but does notresolve the inefficiency issue.

5. Robustness and Model Extensions

In this section, we relax some assumptions of the the-oretical model and perform various empirical checksto establish the robustness of our results.

5.1. Alternative Variable Definitions andOperationalizationWe first investigate whether our empirical results aresensitive to variable definitions and operationalization.Two variables that are measured by surrogates and

Table 3 Three-Stage Least Square Regression Results (Using Number of Solutions to Measure Solver Effort and Average Income to Measure SolverExpertise)

3-1 3-2 3-3 3-4 Hypothesized sign

Solver effort model Dependent variable: ln(Number of Solutions+1)ln(Feedback Volume+1) 0.209*** 0.208*** 0.209*** 0.208*** +

(0.007) (0.007) (0.008) (0.007)ln(Contest Reward) 0.025*** �0.005 +

(0.001) (0.003)ln(Contest Duration) 0.028*** 0.011** +

(0.001) (0.004)ln(Average Income+1) 0.019*** 0.016*** +

(0.001) (0.004)ln(Solver Population) �0.018*** �0.006 �

(0.002) (0.004)

Seeker feedback model Dependent variable: ln(Feedback Volume+1)ln(Contest Reward) 0.145*** 0.167*** +

(0.008) (0.017)ln(Contest Duration) 0.134*** 0.088*** +

(0.008) (0.018)ln(Average Income+1) 0.085*** 0.015 +

(0.007) (0.017)ln(Solver Population) �0.089*** �0.064*** �

(0.008) (0.019)ln(Feedback-Seeking Volume+1) 0.269*** 0.254*** 0.248*** 0.249*** +

(0.008) (0.008) (0.008) (0.008)ln(Description Length+1) �0.073*** �0.184*** �0.213*** �0.213*** –

(0.010) (0.010) (0.011) (0.011)Number of observations 9771 9771 9771 9771System-weighted R2 22.70% 24.17% 24.05% 24.27%

Note: Standard errors are reported in parentheses; **p < 0.01; ***p < 0.001.




thus may suffer from this issue are solver effort andexpertise. As such, we employ an alternative measureof solvers’ effort, Elapsed Time, defined as the amountof time solvers spent on the contest. It is calculated asthe difference between the registration date and thelast submission date in the unit of days. Then, ElapsedTime is averaged across all solvers in the contest toobtain the contest-level measure. This method is con-sistent with the project management literature inwhich effort is usually measured as the amount oftime required to complete a task (e.g., workdays)(Kerzner 1995).13 Additionally, Elapsed Time is unli-kely to be correlated with Feedback-Seeking Volume (theIV for Feedback Volume). This is because the time ittakes solvers to write text messages to request feed-back is negligible compared with the time they spentin the contest. Similarly, we do not see an obvious linkbetween Elapsed Time and Description Length.Two alternative measures of solver expertise are

Tenure and Experience. Tenure is defined as the num-ber of months elapsed since the solver participatedin his first contest, while Experience is defined as thetotal number of contests in which a solver has

participated before joining the contest. It is easy tosee that both Tenure and Experience contribute to theaccumulation of knowledge, skills, and experience.Both measures are averaged, respectively, over allsolvers in the contest to obtain contest-level solverexpertise.We replicate our analysis using these alternative

measures of solver effort and solver expertise. Estima-tion results are reported in Table 5. Columns (5-1)and (5-2) show the robustness of our main specifica-tion using Tenure and Experience as alternative mea-sures of solver expertise. We find that, in addition toall the empirically supported hypotheses in sec-tion 4.4, the effect of solver expertise on feedback vol-ume becomes significant at the 0.01 level in bothmodels, therefore supporting H3(c). Columns (5-3),(5-4), and (5-5) show that all three hypotheses (H1–3)are strongly supported when Elapsed Time is used tomeasure solver effort and Average Income, Tenure, andExperience are used to measure solver expertise,respectively. These estimation results suggest that ourempirical results are robust, offering strong supportto our theory.

Table 4 OLS, SUR, and 2SLS Regression Results (Using Number of Solutions to Measure Solver Effort and Average Income to Measure SolverExpertise)

4-1 4-2 4-3Hypothesized

signOLS SUR 2SLS

Solver effortmodel

Dependent variable: ln(Number of Solutions+1)

ln(Feedback Volume+1) 0.035*** 0.048*** 0.208*** +(0.001) (0.001) (0.007)

ln(Contest Reward) �0.020*** �0.022*** �0.005 +(0.002) (0.002) (0.004)

ln(Contest Duration) 0.013*** 0.011*** 0.011** +(0.002) (0.002) (0.004)

ln(Average Income+1) 0.018*** 0.018*** 0.016*** +(0.002) (0.002) (0.004)

ln(Solver Population) 0.018*** 0.017*** �0.006 –(0.002) (0.002) (0.004)

Adjusted R2 14.40% – 9.33%

Seeker feedback model Dependent variable: ln(Feedback Volume+1)ln(Contest Reward) 0.107*** 0.206*** 0.212*** +

(0.018) (0.018) (0.018)ln(Contest Duration) 0.118*** 0.081*** 0.089*** +

(0.019) (0.018) (0.018)ln(Average Income+1) 0.028⌐ 0.011 0.013 +

(0.018) (0.017) (0.017)ln(Solver Population) �0.057** �0.091*** �0.058** –

(0.020) (0.019) (0.019)ln(Feedback-Seeking Volume+1) – – 0.227*** +

(0.009)ln(Description Length+1) – – �0.361*** –

(0.022)Adjusted R2 2.61% – 11.68%System-weighted R2 – 20.58% –Number of observations 9771 9771 9771

Note: Standard errors are reported in parentheses; ⌐p < 0.1; **p < 0.01; ***p < 0.001.




Recall that, before taking the logarithm, we add oneto variables that may take the value of zero:ln Variablej þ 1� �

. This includes variables such asNum-ber of Solutions, Average Income, Feedback Volume, Feed-back-Seeking Volume, and Description Length. Toexamine whether this approach drives our results, weemploy the following variable operationalization:

ln Variablej þ X� �

; ð16Þ

where X > 0. We re-estimate the main model (Num-ber of Solutions and Average Income as measures ofsolver effort and expertise, respectively) by varyingthe value of X: X = 0.01, X = 0.1, X = 0.5, and X = 2.Table A2 in Online Appendix A summarizes theresults. The case of X = 1 is shown in column (A2-4)for ease of comparison.Overall, variable operationalization may influence

the significance and effect size of the model coeffi-cients but not their signs. A closer examinationreveals that b4 is negative and statistically significantat the 0.05 level for all values of X except for X = 1,lending additional support to H2(d) that larger sol-ver population decreases solver effort; b9 becomesstatistically significant as X decreases, providing

some support to H3(c) that higher solver expertiseleads to higher feedback volume; however, b2 is stillinsignificant under various values of X. All othersignificant estimates in the main model remainunchanged.To summarize, all hypotheses are supported by the

main model (see Table 3) except for H2(a), H2(d), andH3(c). Further exploration uncovers that the estimatesof these three effects appear to be sensitive to variabledefinitions and operationalization, while all otherresults remain extremely robust. In particular, whenElapsed Time is used as the measure of solver effort, allhypothesized effects—including H2(a), H2(d) and H3(c)—are supported empirically (see Table 5). Thismay suggest that solvers tend to devote significantlymore time working on the logo design in response tohigher contest reward and smaller solver populationwithout increasing submissions. Furthermore, whensolver effort is measured by Number of Solutions, theeffect of solver expertise, measured by Tenure or Expe-rience, on feedback volume is significant, lending sup-port to H3(c) (see Table 5). The effect of SolverPopulation on Solver Effort appears to be sensitive to Xin ln Variablej þ X

� �, with H2(d) being supported for

all values of X except for X = 1 (see Table A2). Taken

Table 5 Three-Stage Least Square Regression Results (Using Alternative Measures of Solver Effort and Solver Expertise)

5-1 5-2 5-3 5-4 5-5

Measure of solver effort Number of Solutions Number of Solutions Elapsed Time Elapsed Time Elapsed Time

Hypothesized signMeasure of solver expertise Tenure Experience Average Income Tenure Experience

Solver effort model Dependent variable: ln(Effort+1)ln(Feedback Volume+1) 0.209*** 0.209*** 0.172*** 0.176*** 0.179*** +

(0.007) (0.007) (0.010) (0.010) (0.010)ln(Contest Reward) �0.001 0.001 0.008⌐ 0.027*** 0.042*** +

(0.003) (0.003) (0.005) (0.005) (0.005)ln(Contest Duration) 0.011** 0.010** 0.804*** 0.807*** 0.801*** +

(0.004) (0.004) (0.005) (0.005) (0.006)ln(Solver Expertise+1) 0.022*** 0.000 0.092*** 0.155*** 0.011*** +

(0.006) (0.002) (0.005) (0.008) (0.03)ln(Solver Population) �0.004 �0.005 �0.023*** �0.014* �0.019*** –

(0.004) (0.004) (0.006) (0.006) (0.006)

Seeker feedback model Dependent variable: ln(Feedback Volume+1)ln(Contest Reward) 0.165*** 0.171*** 0.187*** 0.183*** 0.191*** +

(0.016) (0.016) (0.018) (0.017) (0.017)ln(Contest Duration) 0.090*** 0.091*** 0.088*** 0.090*** 0.091*** +

(0.018) (0.018) (0.018) (0.018) (0.018)ln(Solver Expertise+1) 0.067** 0.048*** 0.014 0.065* 0.046*** +

(0.026) (0.010) (0.017) (0.026) (0.010)ln(Solver Population) �0.062*** �0.067*** �0.062*** �0.060** �0.064*** –

(0.019) (0.019) (0.019) (0.019) (0.019)ln(Feedback-Seeking Volume+1) 0.249*** 0.253*** 0.239*** 0.241*** 0.243*** +

(0.008) (0.008) (0.009) (0.009) (0.009)ln(Description Length+1) �0.212*** �0.212*** �0.277*** �0.270*** �0.276*** –

(0.011) (0.011) (0.017) (0.017) (0.017)Number of observations 9771 9771 9771 9771 9771System-weighted R2 24.08% 23.97% 70.76% 70.78% 69.46%

Note: Standard errors are reported in parentheses; ⌐p < 0.1; *p < 0.05; **p < 0.01; ***p < 0.001.




together, most hypothesized effects are robust to vari-able definitions and operationalization. Thus, the the-oretical predictions are largely supported by ourempirical analysis.

5.2. Heterogeneous SolversOur analytical model assumes that all solvers in thesame contest are homogeneous. This assumptiongreatly simplifies the analysis and is necessary forderiving the closed-form solutions. However, in prac-tice, ideation contests naturally attract contestantsfrom a heterogeneous pool of solvers. As demon-strated in our data, solvers competing in the samecontest generally have different levels of expertise.This creates a slight disconnect between the homo-geneity assumption and the actual data, even thoughwe take average of solver expertise across all solversin the contest to obtain the contest-level measure. Inthis subsection, we test the robustness of our findingsby varying the level of solver heterogeneity.To this end, for each contest we calculate the

standard deviation of solver expertise measured byAverage Income. Then we divide all contests into twohalves by performing a median split based on thestandard deviation of solver expertise. In this way,we obtain two subsamples with the “above-med-ian” (“below-median”) sample consisting of contestswith relatively heterogeneous (homogeneous) sol-vers. We run 3SLS regression models separatelywith both subsamples using Average Income as themeasure of solver expertise and alternative mea-sures of solver effort. We also follow the same pro-cedure for the other two measures of solverexpertise: Tenure and Experience. Estimation resultsare summarized in Tables A3 and A4 in OnlineAppendix A. We find that the two subsamples pro-duce qualitatively the same results as the mainmodel. This suggests that, despite the homogeneityassumption in the theoretical model, analysis usingsubsamples of more (or less) heterogeneous solversdoes not affect our main results.

5.3. Endogenous Solver EntryIn the main model, we assume that solvers’ entry intothe contest is exogenous (i.e., solver population N isexogenously determined). Now we relax this assump-tion by allowing solvers to decide whether to enterthe contest. Let �N ( �N is sufficiently large) be the totalnumber of solvers who consider whether to partici-pate. The game sequence is as follows: the seekerchooses feedback volume in stage one; solvers simul-taneously decide whether to enter the contest in stagetwo; and solvers expend effort in stage three. Wesolve the model by backward induction. It is easy tosee that the solution of stage three remains the sameso we obtain from Equation (10) the equilibrium

effort level e� N Mð Þ; Mð Þ as a function of feedback vol-umeM and solver population N Mð Þ.Then we go back to stage two and characterize sol-

vers’ entry decision. An increase in solver populationhas two effects on solvers. The expected benefit ofentering the contest decreases because a larger solverpopulation reduces the likelihood of winning. This istermed benefit-reduction effect of entry. We also identifya cost-reduction effect when endogenizing solvers’effort level: the cost of entering the contest decreasesbecause a larger solver population lowers solvers’effort and the associated cost. To obtain an interiorsolution, we assume that a Tð Þ is sufficiently large, thatis, a Tð Þ[ ��a Tð Þ where ��a Tð Þ is defined in OnlineAppendix B. With this assumption, the benefit-reduc-tion effect outweighs the cost-reduction effect. Thenet effect is such that solvers’ utility from entering thecontest decreases with solver population. Solvers’equilibrium entry strategy is formally presented inthe following proposition.

PROPOSITION 3. There exists a unique mixed-strategyNash equilibrium in solvers’ entry into the contest. Theprobability that each solver enters the contest is N�

�Nwhere

N*is the equilibrium solver population and is the uniquesolution of

2N�þ2

N� � 1ð Þ2 ¼a2V Ms2 þ r2

� �2pa Tð Þs4r4 : ð17Þ

N* increases with feedback volume, contest reward, con-test duration, and solver expertise.


Since solvers are worse off when more competi-tors join the contest, it is never optimal for all ofthem to enter the contest in equilibrium becausethey will obtain negative utility by doing so. Thus,solvers adopt a mixed strategy where some of thementer the contest while others do not. In equili-brium, all solvers obtain an expected utility of zero,that is, they are indifferent to entering the contest ornot. We find that the equilibrium solver populationincreases with feedback volume, contest reward,contest duration, and solver expertise. This resultcan be attributed to their impacts on the benefit-and cost-reduction effects. An increase in feedbackvolume, contest reward, and solver expertise magni-fies both benefit- and cost-reduction effects. Sincethe cost-reduction effect increases as a faster pacethan the benefit-reduction effect, the equilibriumsolver population increases with these variables.Similarly, an increase in contest duration weakensboth effects but the cost-reduction effect decreases




more slowly than the benefit-reduction effect. As aresult, the equilibrium solver population increaseswith contest duration.14

5.4. Pre-Feedback Solver EffortThe sequence of decisions in the main theoreticmodel is intuitive as it reflects the relative flexibilityin the decision-making process: solvers have a lot offlexibility in exerting more effort after receiving theseeker’s feedback comments. Nonetheless, feedbackis usually given based on submitted solutions, indi-cating that solvers must have already invested someeffort before any feedback is given. To allow for thispossibility, we now consider a three-stage model inwhich solvers can exert effort both before and afterthe seeker’s feedback decision is made. More speci-fically, solvers expend pre-feedback effort in stageone, the seeker decides on feedback volume in stagetwo, and solvers choose the post-feedback effort instage three. We assume that the seeker offers feed-back to solvers at some time when the contest isrunning. Without loss of generality, let the time atwhich feedback is provided be hT where h 2 0; 1ð Þand T is the contest druation. With this assumption,the pre-feedback duration is hT and the post-feed-back duration is 1� hð ÞT. We also restrict our atten-tion to the symmetric Nash equilibrium in which allsolvers share the same pre- and post-feedbackefforts. Detailed proof is provided in OnlineAppendix B.In this model, solver i’s performance depends on

both pre- and post-feedback effort:

di ¼ a eprei þ e

posti

� �þ li: ð18Þ

We solve the model by backward induction, begin-ning with solving the stage-three decision while tak-ing pre-feedback efforts and feedback volume asgiven. Solvers’ utility maximization problem is writ-ten as

maxepost

i

U eposti jeprei ;M

� �¼VPr winið Þ�a 1�hð ÞTð Þepost2i ; ð19Þ

where Pr winið Þ is solver i’s likelihood of winning thecontest, given by

Pr winið Þ¼Yk 6¼i

Pr di[dkjXð Þ

¼Yk 6¼i

Ua e

posti þe

prei �e

postk �e

prek

� �Ms2þr2� �

2s2r2

0@

1A:

ð20Þ

By invoking symmetry eprei ¼ e

prek , we obtain the equi-

librium post-feedback effort:

epost� ¼ aV N � 1ð Þ Ms2 þ r2� �

2Nþ12ffiffiffip

pa 1� hð ÞTð Þs2r2 : ð21Þ

It is apparent that the post-feedback effort levelincreases with feedback volume, contest reward,contest duration, and solver expertise, and decreaseswith solver population. Then we go back to stagetwo and analyze the seeker’s decision on feedbackvolume. The profit of the seeker is the same as inEquation (11). The seeker maximizes her profit bychoosing feedback volume M. By solving this profitmaximization problem, we obtain

M� ¼ rs

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffilfeedback � lprior � s2

R1�1 zdU zð ÞN

b� aepre � a2V N�1ð Þ2Nþ1

2ffiffip

pa 1�hð ÞTð Þr2

vuuut � r2

s2: ð22Þ

It can be shown that feedback volume increaseswith pre-feedback effort, contest reward, contestduration, solver expertise, and decreases with solverpopulation. Finally, we go back to stage one andevaluate solvers’ pre-feedback efforts. We assumethat solvers have rational expectations on their post-feedback effort level, with utility

U eprei

� �¼VYk 6¼i

Ua e

prei þe

posti �e

prek �e

postk

� �M�s2þr2� �

2s2r2

0@

1A

�a hTð Þepre2i :

ð23Þ

Thus, we have eposti ¼ e

postk ¼ epost�. All solvers simul-

taneously choose their pre-feedback effort level bymaximizing their utility function, which can berewritten as follows:

maxepre

i

U eprei

� � ¼ VYk 6¼i

Ua e

prei � e

prek

� �M�s2 þ r2� �

2s2r2

!

� a hTð Þepre2i :

ð24ÞWe find that there exists an optimal pre-feedbackeffort e

pre�i that satisfies the following equation:

epre�2 b� a2V N� 1ð Þ2Nþ1

2ffiffiffip

pa 1� hð ÞTð Þr2� aepre�

!

�a2V2 N� 1ð Þ2 lfeedback�lprior� s2

R1�1 zdU zð ÞN

� �2Nþ1

2ps2r2a2 hTð Þ ¼ 0:

ð25ÞFurthermore, epre� increases with contest reward,contest duration, and solver expertise, and decreases




with solver population. (Detailed proof is providedin Online Appendix B.) From the above analysis, weconclude that all three sets of hypotheses in sec-tion 3.3 still hold in this three-stage model.

6. Implications and Conclusions

Advances in information technology have fundamen-tally transformed how companies engage in ideationprocesses. In today’s business world, many companiesrely on ideation contests to complete tasks that werepreviously performed by employees or contractors.Harnessing the power of the Internet, ideation contestsallow companies to tap into “the wisdom of the crowd”by soliciting a large number of solutions at minimalcosts. Although ideation contests are widely adoptedby businesses, little is known about how feedback isused and how it affects solvers’ effort level.In this study, we build a theoretical model of idea-

tion contest that incorporates the feedback mechanismthrough which seekers reveal their private taste to sol-vers. Ideation contests are characterized by informationasymmetry—the seekers’ private information abouttheir taste is imperfectly known to solvers. Informationasymmetry leads to market inefficiency and solvers’suboptimal behavior. We find that the solvers’ effortlevel increases as seekers provide more feedbackbecause it can help reduce solvers’ uncertainty aboutthe seekers’ taste. We also find a direct effect of contestand solver characteristics (CSC) on solvers’ effort level.The solvers’ equilibrium effort level increases with con-test reward, contest duration, and solver expertise butdecreases with solver population. By endogenizing theseeker’s feedback volume, we find that feedback par-tially mediates the relationship between CSC and thesolvers’ effort level. Thus, CSC elements also have anindirect effect on solvers’ effort level through the see-ker’s feedback.Theoretical hypotheses are tested using data from

9771 contests hosted by Zhubajie.com. The empiricalanalysis provides strong support for our theory. Vari-ous robustness checks and model extensions are con-ducted to corroborate the robustness of our findings.Our empirical results are invariant when employingalternative variable definitions and operationaliza-tion, or using a relatively homogeneous or heteroge-neous subsample. In addition, we extend the basemodel to endogenize solvers’ entry decision andexpand to a three-stage game that allows solvers toexpend effort before feedback is given. These modelextensions lend further support to our main analyticalresults and propositions.

6.1. Theoretical and Managerial ImplicationsWe investigate the role of seeker’s feedback in idea-tion contests using a combination of theoretical and

empirical analyses. The informative role of feedbacksuggests that seekers should be strongly encouragedto provide comments on solvers’ submissions. Thiscan be achieved by educating seekers regarding theimportance of giving feedback, which is consistentwith the practice of several ideation contest platforms(e.g., 99Designs and DesignContest), although someother platforms remain silent and may therefore suf-fer from an undersupply of feedback. In addition,ideation contest platforms can focus on increasing thebenefits and reducing the costs of feedback provision.This can be done by awarding seekers who activelyleave feedback with extra bonuses and by investing ineasy-to-use and efficient feedback systems. Platformsshould also encourage solvers to engage in feedback-seeking behaviors. For example, Zhubajie recom-mends to solvers, “Frequent communication with theemployer (seeker) during the work process will helpyou to deliver better results.”15

One challenge that seekers encounter in ideationcontests involves the design and management of CSC.Using a dataset with substantial variation in contestdesign, we examine four CSC elements concurrentlyin this study. Firstly, we show that higher contestrewards incentivize greater solver effort. AlthoughLiu et al. (2014) find a similar result in a field experi-ment on Taskcn.com, they only consider two rewardlevels: 100 CNY and 300 CNY. Our result is more gen-eral and suggests that firms seeking ideation solutionsvia contests should carefully select appropriatereward amounts to achieve optimal outcomes. Thismonetary aspect also has practical implications forplatforms as their primary revenue source comprisesthe transaction fee charged at a certain percentage ofthe contest rewards. Secondly, our study is the first toshow that contest duration can positively affect sol-vers’ effort. Therefore, seekers must balance theirrequirements for high-quality solutions with theirdesire for rapid delivery time. Thirdly, we demon-strate that solvers with greater expertise tend to spendmore time working on the task and are more likely todeliver solutions in larger quantity and with higherquality. Therefore, it is important for seekers to keeptrack of solver profiles and to make sure that capableand experienced solvers can participate. To do so,seekers can strategically invite the most promisingsolvers to join the contests (Mo et al. 2018). For exam-ple, 99Designs and DesignCrowd allow seekers tobrowse through the pool of solvers and send invita-tions directly to individual solvers. Moreover, it is inthe platforms’ best interest to build a large pool ofhighly-skilled solvers and encourage their active par-ticipation using various incentives. This strategy iscritical to seekers’ satisfaction and the healthy growthof the platform. Lastly, our theory predicts thatincreased competition among solvers tends to



http://Zhubajie.com

http://Taskcn.com


dampen their effort level, a finding that is consistentwith previous literature (e.g., Boudreau et al. 2011,Terwiesch and Xu 2008). However, as Terwiesch andXu (2008) point out, a larger solver population is notnecessarily bad for seekers, especially when they arelooking for a single best solution.Another important finding of this study is the

mediating role of feedback in the relationshipbetween CSC and the solvers’ effort level, which hasprofound theoretical and managerial implications.Previous literature usually takes the seekers’ feedbackstrategy as exogenous, while our paper takes the firststep toward proposing and validating that seekers arestrategic in determining whether to provide feedbackand the feedback volume. Therefore, when assessingthe total effect of CSC on solvers’ effort level, firmsshould consider both the direct effect and the indirecteffect through feedback volume. Otherwise, thedesign of CSC could be suboptimal. We also recog-nize that the solver-side effort model suffers from theendogeneity bias if the seekers’ strategic decisionsregarding feedback volume are not taken intoaccount. Ideation contests designed without this con-sideration could be counterproductive. To the best ofour knowledge, this study is the first to investigatefeedback as a critical mediator in the CSC–effort rela-tionship.

6.2. Limitations and Directions for FutureResearchOur research is not exempt from limitations, whichsuggest future research directions. Our analyticalmodel features an ideation contest with symmetricsolvers. Although this assumption greatly simplifiesour analysis, it may not reflect the reality of theideation contest markets. More specifically, thereexists substantial heterogeneity in solvers’ expertiseand capability of extracting information from feed-back. Consequently, solvers are not symmetric as inthe base model. It would also be interesting forfuture studies to further investigate whether solverswith varying levels of expertise would respond dif-ferently to seekers’ feedback. If so, the effectivenessof feedback in motivating solver effort might bemoderated by solver expertise. Another interestingtopic is to study how feedback affects the distribu-tion of solver expertise in a contest, when solverentry is endogenous. Additionally, solvers with var-ious types of expertise might respond differently toCSC. For example, it would be interesting to investi-gate whether rewards would be more useful indriving solvers with high or low expertise toexpend greater efforts.We have assumed that solvers’ match values with

the seeker follow an identical distribution. In reality,some solvers might have previous experience of

working for certain seekers and would be moreknowledgeable about these seekers’ private taste thanothers. These more experienced solvers might startwith a different distribution of match values, andmay also interpret feedback differently than thosewith little prior knowledge. However, we note thatnumerical analysis in this case is necessary as there isno closed-form solution in Bayesian learning frame-work. Future research should examine how solvers’prior information would affect their strategic behav-ior in ideation contests.The limitations of our dataset should also be noted.

Although we use instrumental variables such as Feed-back-Seeking Volume to correct potential endogeneitybias, we lack instruments or control variables to fullyestablish causality, although our empirical results areconsistent with theoretical predictions. Futureresearch can investigate a richer context with moreoptions of instrumental variables and control vari-ables.Finally, since it is important to demonstrate the

generalizability of these results, corroborating ourfindings in subsequent research would be quite use-ful, especially across platforms and across service cat-egories. It is possible that both the strength of theinformative and mediating roles of feedback are con-tingent on the nature of the platform and the servicecategory, as the level of information asymmetry isessential in driving our theoretical predictions.

Acknowledgments

We are immensely grateful to the Department Editor, theSenior Editor, and the three anonymous reviewers for theirconstructive comments that greatly helped us improve thepaper. We thank the seminar participants at Virginia Tech,our audience at the 37th Annual ISMS Marketing Scienceconference, and Dr. Sam Min at California State University,Long Beach for their insightful feedback on earlier drafts ofthe manuscript.

Notes

1Indeed, ideation contest platforms such as 99Designs andDesignContest have recognized the importance of feed-back in ideation contests and actively encourage it. Forinstance, 99Designs recommends to seekers, “Leaving acomment is the most important and productive way ofproviding feedback, as it allows you to interact with thedesigners and provide guidance. Tell designers if theirdesign is going in the right direction, and how they canimprove their submission.” DesignContest emphasizes,“Evaluating entries and communicating with designershelp you to get exactly the design you’re looking for.”2It is worth noting that our work is related to but differentfrom the freelance literature. Freelancers compete in mar-ketplaces to take on jobs such as web development, pro-gramming, writing, and translation. This involves an




auction-like procedure, ending with a contract signed bythe buyer and the winning freelancer. Selection of the win-ner is based on participating freelancers’ initial bids andother observable characteristics such as reputation. Theselected freelancer starts his work after he wins the con-tract. This implicates that the buyer’s main challenge liesin the information asymmetry caused by a potential “le-mon” problem (see, e.g., Gandini 2016, Yoganarasimhan2013, 2015). Since the buyer only receives the work(solution) only from the winning freelancer, she does notbenefit from the “wisdom of the crowd,” which is in starkcontrast to the ideation contests we study.3Ideation contest bears considerable similarity with salescontest that has received much attention in the market-ing literature (see Chen and Lim 2013, 2017, Chen et al.2011, Kalra and Shi 2001, Lim 2010, Lim et al. 2009, toname a few). However, they are different in two impor-tant ways: (i) The seeker in an ideation contest usuallyseeks and pays for the favorite solution among all sub-missions, whereas the purpose of sales contests is toincrease the overall output of all contestants; (ii) In idea-tion contests solvers act on a completely voluntary basis,whereas salespersons are often obliged to participate insales contests. Thus, characteristics of these contests arefundamentally different, as is evident in our model insection 3.4There is a large body of research on performance feedbackin contests and tournaments in the economics and busi-ness literature (see Aoyagi 2010, Ederer 2010, Moldovanuand Sela 2001, to name a few), which unanimously exami-nes feedback that contains information about contestants’performance relative to others. Such performance feedback istreated as a signal of contestants’ hidden effort. In con-trast, we study feedback comments that reveal the see-ker’s private taste. It is also noteworthy that in ideationcontests, solvers’ effort level (or its proxy) is publiclyknown. Thus, prior research findings on performancefeedback cannot be generalized to the context of ideationcontests.5In section 5.4, we extend the model by allowing solversto exert effort before feedback is given.6If solvers’ performance is assumed to take an additiveform of solver expertise and effort, that is, di = a + ei + li,solver expertise will be cancelled out when comparingtwo different solvers’ performance and hence does notaffect the equilibrium effort level.7In this model, solvers are assumed to be homogenous. InOnline Appendix C, we extend the theoretical model toallow for solver heterogeneity. Simulation analysis sug-gests that the main findings remain unchanged and, more-over, solvers with higher expertise expend greater effortsthan those with lower expertise. We also empiricallyinvestigate the implications of solver heterogeneity in sec-tion 5.2.8Source: http://www.witmart.com/ and http://techcrunch.com/2012/12/08/asias-secret-crowdsourcing-boom/ (ac-cessed date July 1, 2019).9Our results are almost invariant when we include con-tests with reward below 50 CNY.10Submission of a revised logo design is counted asanother solution.

11In practice, the seeker can terminate the contest prior tothe initially proposed end date. Thus, the actual contestduration could be different (shorter) than the proposedcontest duration. Our empirical results are similar whenthe proposed contest duration is used.12In Equation (13) FeedbackVolumei is correlated with ɛibecause FeedbackVolumei is a function of υi as seen inEquation (14), thus leading to the simultaneity issue.Notably, Equation (14) suffers from inefficiency yet hasno endogeneity issue.13A potential concern for using elapsed time to measureeffort is that if solvers intentionally postpone submittingtheir designs until the last minute, then elapsed time inac-curately reflects the amount of time they actually spend onthe contests. However, this is unlikely in our context. In ourdata, over 80% of solvers registered on the first day of thecontest, and only 5% of solutions are submitted on the lastday of the contest. Moreover, the mean of Elapsed Time is3.69 days, which is about 34% of contest duration, indicat-ing that procrastination is an uncommon practice in ourcontext. There are at least two reasons for this. First, Zhuba-jie allows the seeker to terminate the contest earlier than theproposed ending time without prior notice once a satisfac-tory logo design has been received. Indeed, 26.7% of thecontests ended prior to the proposed closing time. As aresult, solvers may lose their opportunity for the reward bydelaying submissions. Second, since many ideation contestsare simultaneously held, solvers are better off completinglogo designs in one contest and then shifting to the nextone. Delaying makes it more difficult for solvers to keeptrack of all contests in which they are participating. There-fore, Elapsed Time is a valid measure of solver effort.14The seeker’s decision on feedback volume in the firststage is not reported due to space constraints. But it isavailable upon request.15Source: http://en.zhubajie.com/help/index_f_1_r_1(accessed date July 1, 2019).

ReferencesAoyagi, M. 2010. Information feedback in a dynamic tournament.

Games Econ. Behav. 70(2): 242–260.

Ashford, S. J., L. L. Cummings. 1983. Feedback as an individualresource: Personal strategies of creating information. Org.Behav. Hum. Decis. Process 32(3): 370–398.

Ashford, S. J., A. S. Tsui. 1991. Self-regulation for managerialeffectiveness: The role of active feedback seeking. Acad.Manag. J. 34(2): 251–280.

Ashford, S. J., R. Blatt, D. VandeWalle. 2003. Reflections on thelooking glass: A review of research on feedback-seekingbehavior in organizations. J. Manage. 29(6): 773–799.

Bayus, B. L. 2013. Crowdsourcing new product ideas over time:An analysis of the Dell IdeaStorm Community. ManagementSci. 59(1): 226–244.

Bockstedt, J., C. Druehl, A. Mishra. 2016. Heterogeneous submissionbehavior and its implications for success in innovation contestswith public submissions. Prod. Oper. Manag. 25(7): 1157–1176.

Boudreau, K. J., K. R. Lakhani. 2013. Using the crowd as an inno-vation partner. Harv. Bus. Rev. 57(5): 843–863.

Boudreau, K. J., N. Lacetera, K. R. Lakhani. 2011. Incentives andproblem uncertainty in innovation contests: An empiricalanalysis. Management Sci. 57(5): 843–863.



http://www.witmart.com/

http://techcrunch.com/2012/12/08/asias-secret-crowdsourcing-boom/

http://techcrunch.com/2012/12/08/asias-secret-crowdsourcing-boom/

http://en.zhubajie.com/help/index_f_1_r_1


Burroughs, J. E., D. W. Dahl, C. P. Moreau, A. Chattopadhyay, G.J. Gorn. 2011. Facilitating and rewarding creativity duringnew product development. J. Market. 75(4): 53–67.

Chan, K. W., S. Y. Li, J. J. Zhu. 2015. Fostering customer ideationin crowdsourcing community: The role of peer-to-peer andpeer-to-firm interactions. J. Interact. Market. 31: 42–62.

Chen, H., N. Lim. 2013. Should managers use team-based con-tests? Management Sci. 59(12): 2823–2836.

Chen, H., N. Lim. 2017. How does team composition affect effortin contests? A theoretical and experimental analysis. J. Mark.Res. 54(1): 44–60.

Chen, H., S. H. Ham, N. Lim. 2011. Designing multiperson tour-naments with asymmetric contestants: An experimental study.Management Sci. 57(5): 864–883.

Ederer, F. 2010. Feedback and motivation in dynamic tourna-ments. J. Econ. Manage. Strat. 19(3): 733–769.

Gandini, A. 2016. Digital work: Self-branding and social capital in thefreelance knowledge economy.Market. Theory 16(1): 123–141.

Greene, W. H. 2000. Econometric Analysis, 6th edn. Prentice Hall,New York, NY.

Gupta, D., J. Jiang, Y. Xie, D. Chakravarti. 2019. Incentivizinguser-generated content: A double-edged sword—Evidencefrom a natural experiment and a lab study. Working paper,Virginia Tech, Blacksburg, VA.

Hofstetter, R., J. Z. Zhang, A. Herrmann. 2018. Successive openinnovation contests and incentives: Winner-take-all or multi-ple prizes? J. Prod. Innov. Manag. 35(4): 492–517.

Howe, J. 2006. The rise of crowdsourcing. Wired Magazine. 14(6):1–4.

Jiang, Y., J. Ni, X. J. Chen. 2018. Is this just about now? Short-runversus long-run rewards in crowdsourcing markets. Workingpaper, Johns Hopkins University, Baltimore, MD.

Kalra, A., M. Shi. 2001. Designing optimal sales contests: A theo-retical perspective. Market. Sci. 20(2): 170–193.

Kerzner, H. 1995. Project Management: A Systems Approach to Plan-ning, Scheduling, and Controlling. Van Nostrand-Reinhold,New York.

Lim, N. 2010. Social loss aversion and optimal contest design.J. Mark. Res. 47(4): 777–787.

Lim, N., M. J. Ahearne, S. H. Ham. 2009. Designing sales con-tests: Does the prize structure matter? J. Mark. Res. 46(3): 356–371.

Liu, T. X., J. Yang, L. A. Adamic, Y. Chen. 2014. Crowdsourcingwith all-pay auctions: A field experiment on Taskcn. Manage-ment Sci. 60(8): 2020–2037.

Luo, L., O. Toubia. 2015. Improving online idea generation platformsand customizing the task structure on the basis of consumers’domain-specific knowledge. J. Market. 79(5): 100–114.

Mo, J., S. Sarkar, S. Menon. 2018. Know when to run: Recommen-dations in crowdsourcing contests. MIS Q. 42(3): 919–944.

Moldovanu, B., A. Sela. 2001. The optimal allocation of rewardsin contests. Am. Econ. Rev. 91(3): 542–558.

Poetz, M. K., M. Schreier. 2012. The value of crowdsourcing: Canusers really compete with professionals in generating newproduct ideas? J. Prod. Innov. Manag. 29(2): 245–256.

Prpi�c, J., P. P. Shukla, J. H. Kietzmann, I. P. McCarthy. 2015. Howto work a crowd: Developing crowd capital through crowd-sourcing. Bus. Horiz. 58(1): 77–85.

Rathi, A. 2014. To encourage innovation, make it a competition.Available at https://hbr.org/2014/11/to-encourage-innovation-make-it-a-competition (accessed date July 1, 2019).

Rossiter, J. R., G. L. Lilien. 1994. New “brainstorming” principles.Aust. J. Manag. 19(1): 61–72.

Slot, J. H. 2013. Crossing Boundaries: Involving External Parties inInnovation. CentER, Tilburg.

Stephen, A. T., P. P. Zubcsek, J. Goldenberg. 2016. Lower connec-tivity is better: The effects of network structure on redun-dancy of ideas and customer innovativeness ininterdependent ideation tasks. J. Mark. Res. 53(2): 263–279.

Terwiesch, C., K. T. Ulrich. 2009. Innovation Tournaments: Creatingand Selecting Exceptional Opportunities. Harvard BusinessSchool Press, Boston, MA.

Terwiesch, C., Y. Xu. 2008. Innovation contests, open innovation,and multi-agent problem solving. Management Sci. 54(9):1529–1543.

Toubia, O. 2006. Idea generation, creativity, and incentives. Mar-ket. Sci. 25(5): 411–425.

Toubia, O., L. Flor�es. 2007. Adaptive idea screening using con-sumers. Market. Sci. 26(3): 342–360.

Winch, C. 2010. Dimensions of Expertise: A Conceptual Exploration ofVocational Knowledge. Continuum, London.

Wooten, J. O., K. T. Ulrich. 2015. The impact of visibility in inno-vation tournaments: Evidence from field experiments. Work-ing paper, University of South Carolina, Columbia, SC.

Wooten, J. O., K. T. Ulrich. 2017. Idea generation and the role offeedback: Evidence from field experiments with innovationtournaments. Prod. Oper. Manag. 26(1): 80–99.

Yoganarasimhan, H. 2013. The value of reputation in an onlinefreelance marketplace. Market. Sci. 32(6): 860–891.

Yoganarasimhan, H. 2015. Estimation of beauty contest auctions.Market. Sci. 35(1): 27–54.

Zhang, J. 2016. Deadlines in product development. ManagementSci. 62(11): 3310–3326.

Supporting InformationAdditional supporting information may be found onlinein the Supporting Information section at the end of thearticle.

Appendix A: Additional Tables.Appendix B: Proof of Propositions.Appendix C: Additional Analysis.



https://hbr.org/2014/11/to-encourage-innovation-make-it-a-competition

https://hbr.org/2014/11/to-encourage-innovation-make-it-a-competition


a theoretical and empirical investigation of feedback in ideation … · 2020-01-29 ·...

Documents