crowdsourcing and markets in the social media age

11 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Thomas Sandholm—Research Scientist, HP Labs, Palo Alto

December 17, 2009

Crowdsourcing and Markets in the Social Media Age

22 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Information Abundance and Attention Scarcity

– Information overflow on mobile devices•Small visual real-estate•Limited bandwidth•Limited input capabilities

– How can crowdsourcing, markets, and context improve the mobile Web experience?

33 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Agenda

–Crowdsourcing

–Markets

–Social Media

–Examples

–SCL Projects

–HP Gloe

44 © Copyright 2009 Hewlett-Packard Development Company, L.P.

CROWDSOURCING

55 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Wisdom of Crowds

– Surowiecki: The Wisdom of Crowds (2004)•Skepticism about dumb mobs•Crowds can be smarter than smartest individual if:

1.Diverse (otherwise no new ideas)

2.Independent (otherwise groupthink)

3.Decentralized (otherwise specialization is hard)

4.Aggregation (otherwise local knowledge is lost)

66 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Crowdsourcing or Peer Production:Specialization and reputation drive long tail content production

– Business model•Open call•Parallel problem solving•Vetting of solutions

– Phenomenon•Historically: few publishers and millions of readers•Now: millions of publishers with few readers

– Examples•Wikipedia, YouTube, Linux, Digg

77 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Dynamics of Peer ProductionWhy do people contribute to social media?

– Lack of attention reason to stop contributing

– Feedback loop of attention leads to power laws

– Negligible part of content produced by majority of users

– Negligible number of users produce majority of content

– Hubs of well-connected users vital to dissemination

– Persistence paradoxically inversely correlated to success

88 © Copyright 2009 Hewlett-Packard Development Company, L.P.

MARKETS

99 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Market PropertiesSurowiecki’s fourth condition

– Aggregation of decentralized local knowledge

– Determine value of goods•Price varies until supply and demand are in equillibrium•Self-adjusting and decentralized

– Walras’ Law in General Equilibrium Theory•Sum of excess demand across all markets equals zero

– Market Mechanism:•Incentive compatible, individually rational, strategy proof•Incentive to participate truthfully, non-gameable•Selfish optimization leads to welfare •Effective high-quality, truthful information signals

1010 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Prediction or Information MarketsFocus on information aggregation and prediction

– Voting with costs and payoffs

– 1988 Forsythe, Nelson, Neumann: Iowa Electronic Markets•Predict outcome of political races•Polling less accurate due to phrasing of question•Real money forces truthful behavior

– 2003 Hanson: DoD PAM ―Terrorism Futures‖•Public outcry – unethical to bet on terrorism attacks•Publicity – Crowdcast, Inkling, HubDub, Bet2Give•Same games, some real for-profit markets

– Yahoo Tech Buzz, Google internal information flow markets•Cube neighbors influence betting behavior

1111 © Copyright 2009 Hewlett-Packard Development Company, L.P.

SOCIAL MEDIA

1212 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Web 2.0 Social Media AppsDrove content production paradigm shift

– Long tail content •Unpopular content gets published

– Grass root reputation•meritocracy•open APIs for mashups

– Collaborative filtering•Amazon book recommendations

– The network effect •More users more quality opposite to Web 1.0

– Social networks are leveraged to disseminate information •Viral marketing•Friends determine what you see

– Flickr, MySpace, YouTube, Facebook, Twitter•Not technology leaders but grabbed critical social mass•Innovators in how people interact

1313 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Context-aware applicationsLocation, Social, Time, Weather Conditions

–Mobile devices more personal•Already used to interact with friends

–Sensors give more information to applications•No explicit user involvement

– iPhone paved the way for the mobile Web•Full-featured Web browsers on smartphones

–Affordable unlimited 3G data plans–Privacy opt-in

•Intrusive apps more accepted if they have a social value

1414 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Location Based ServicesPull or push content based on sensed location

– Geography-aware service on mobile device

– GIS for mobile phones

– GPS + Internet on smartphones

– Find what’s around you

– Map based, AR based or Distance/Popularity Listings

– Google Navigation/Maps, Wikitude.Me, Layar, Sherpa, Where, GeoVector, Loopt

1515 © Copyright 2009 Hewlett-Packard Development Company, L.P.

EXAMPLES

1616 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Digg – a crowdsourced newspaper

– Crowd-rendered front page

– Novelty and popularity tradeoff

– Update frequency unmatched

1717 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Delicious – social bookmarking

– Share bookmarks with friends

– Find popular bookmarks

– Search by tags and tag clouds

1818 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Google AdWords – bidding for page prominence

– Market mechanisms used to leverage attention economy

– Key word auctions

– Non-intrusive, relevant presentation

– Virtually all of Google’s revenue

1919 © Copyright 2009 Hewlett-Packard Development Company, L.P.

SOCIAL COMPUTING LAB PROJECTS

2020 © Copyright 2009 Hewlett-Packard Development Company, L.P.

BRAIN

– Prediction of events in small groups

– Forecasting in teams biased towards ―loudest‖ member

– When members put a bet on their forecast they predict more carefully and accurately

– Risk attitudes of members matter, when normalized away the weighted prediction becomes more accurate

2121 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Tycoon

– Market based computational resource allocation

– Resources capacity allocated proportional to individual bids and inversely proportional to aggregatebids

– q = b/(b+y)

– Variable pricing, self adjusting to demand

– Similar properties to second bid auction but converges quicker and has no discrete clearing times

2222 © Copyright 2009 Hewlett-Packard Development Company, L.P.

HP Gloe

2323 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Idea

– Popularity search by radius, aggregation by URL

– Social filtering

– Folksonomy of hierarchical channels

– Cyberspace notes

– Votes governed by market economy

Vote on web page to location mappings

2424 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Popularity Ranking and Social Filtering

2525 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Folksonomy of hierarchical channels

Pizza

Restaurants

Burgers

Restaurants

Pizza

Restaurants

Diners

Restaurants

2626 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Cyberspace notes

2727 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Votes governed by market economy

– Limited recommendation budget•American Idol type voting

– Contributions controlled by currency•Earn, pay, penalties, income streams

– Optimal budget allocation •Across regions in marketing campaigns•Across hierarchical key words

– Voting minimal effort but truth-telling enforced•Restaurant services forcing you to leave comments to vote

– Maximize information obtained from crowd•Value distribution beyond 5 star ratings

2828 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Demo

– Launched on Android market Dec 1st

– ~6 million recommendations, ~8000 channels, ~150 users

– Data from Wikipedia, TripAdvisor, Panoramio, WikiTravel, POI Plaza, IMDb, DBPedia, Geonames, Google, Twitter, Bing

2929 © Copyright 2009 Hewlett-Packard Development Company, L.P.

3030 © Copyright 2009 Hewlett-Packard Development Company, L.P.

3131 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Research Problems

– Design a market that gives incentive to users to contribute valuable mappings

– Visualization, control and optimization techniques to

trade off popularity (clicks and recs), novelty, and

distance

– Dynamic content provisioning, recommendation decay

– Optimized channel and location-based bidding strategies

– Data analytics, e.g. behavioral patterns in Web page recommendations

3232 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Summary

–Crowdsourcing can be a very effective technique to offer long tail content if certain conditions are met

–Markets can be used to control the quality of contributions and to aggregate local information concisely

–Your social network provides a way to effectively filter and discover new information

3333 © Copyright 2009 Hewlett-Packard Development Company, L.P.

Q & A