crowdsourcing 102: mining real-time data

CROWD SOURCING 102Real-Time Data Mining and Curation with SwiftRiver

Personal Democracy Forum 2011jg@swiftly.org

@jongos @swiftriver

Tuesday, June 7, 2011

About UshahidiUshahidi is a free, open-source platform used for crowdsourcing and visualizing data geospatially. It was born out of the 2008 election unrest when founders Juliana Rotish, Erik Hersman, Ory Okolloh and David Kobia wanted to allow Kenyan citizens a way to SMS reports of incident to know what was occurring around them. This was one of the earliest uses of crowdsourcing for crisis response.

Notable UsesUshahidi has been deployed in major global crisis scenarios, allowing organizations to draw situational awareness from the crowd. To date it ’s been downloaded over 15,000 times.

Some of the more notable deployments include recently in Egypt, the Haiti earthquakes, the fires in Russia, the Queensland floods in Australia.

The ChallengeA s t h e a m o u n t s o f d a t a aggregated by Ushahidi users grows, they face a common problem. How do they effectively manage this realtime data? How can we help them discover credible and actionable info from the deluge of reports they’ll get from the public? The SwiftRiver initiative was created to begin to answer some of these questions for Ushahidi deployers.

Tuesday, June 7, 2011

“It’s not information overload. It’s filter failure.”

- Clay Shirky

Tuesday, June 7, 2011

PLATFORM GOALS

Consider the context, relevance defined by the user

Offer an opt-in global database of trust and authority

Algorithms augment, but not define, human decision making

Work across media channels (Twitter, Email, Feeds, SMS)

Be accessible (offline/online/mobile)

Index massive amounts of the mobile/social web

Tuesday, June 7, 2011

THIS IS A DATA PROBLEM

Tuesday, June 7, 2011

DATA IS AGNOSTIC

Tuesday, June 7, 2011

STRATEGY

Operate more like a startup than activists

Test humanitarian solutions in a business context

Journalists, Private Sector, NGOs & Government Agencies

Build internal engineering capacity

Tuesday, June 7, 2011

PROGRESS

7,000+ downloads in 6 months

7,000+ API Users

100,000+ Lines of code

5 APIs and 2 Apps

Data Items Processed - 70,000,000 (liberal extrapolation)

Tuesday, June 7, 2011

Tuesday, June 7, 2011

Tuesday, June 7, 2011

Tuesday, June 7, 2011

ABC Australia Deployment

Tuesday, June 7, 2011

Sweeper - User Interface

Tuesday, June 7, 2011

CROWD SOURCING 102Real-Time Data Mining and Curation with SwiftRiver

jg@swiftly.org@jongos @swiftriver

Tuesday, June 7, 2011