Expert Crowdsourcing with Flash TeamsDaniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Michael Bernstein

{dretelny, robi, ato1120, msb}@cs.stanford.edu

Crowdsourcing is Plateauing in ComplexityCrowd workers struggle to complete open ended and highly interdependent tasks.

Despite good intentions, teams of crowd experts are typically neither effective nor efficient at these tasks.

Expert crowdsourcing lacks generalizable techniques for guiding crowds through complex tasks.

STANFORD HCI GROUP

Flash TeamsWe introduce flash teams, modular team structures that computationally guide a team of crowd experts reproducibly and at scale. These modular structures enable crowd expert teams to grow, adapt, and recombine into larger organizations.

Defining attributes of flash teams:

MODULARITY: self-contained and replicable team structures composed of crowd experts that are computationally guided.

ELASTICITY: parameterizable team growth on demand, allowing teams to grow and shrink as expertise are needed.

PIPELINING: parallelism enabled by defining interdependencies and handoffs.

Methods:

To explore the feasibility and scope of flash teams, and understand the effectiveness of modularity, elasticity and pipelining, we gathered thirteen different flash teams to complete complex tasks ranging from design to education.

We recruit experts through oDesk, a crowdsourcing platform for expert work that allows requesters to post a request under a specific expertise category (e.g., Python development, sound production, or visual design).

Flash Team ExamplesOur evaluation of flash teams to date includes the successful completion of eight different flash team examples, utilizing different team structures and workflows. The flash teams varied in size, complexity and expertise and they produced outputs that ranged in completion time and quality. The images below depict the team structures, workflows and output from three napkin sketch, animation and MOOC flash team examples we completed.

Foundry, a Flash Team Authoring PlatformTo encapsulate the foundations of the flash teams formalism and lessen the threshold to creating flash teams, we created Foundry, a web platform that allows requesters to create, reuse and recombine flash team structures and workflows and allows team members to coordinate tasks and track progress. The image to the right shows a screenshot of Foundry.

http://hci.stanford.edu