crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · topics • crowdfunding...

CrowfordCrowd Investment Data Portal

Group 13 LABBE, Kevin Patrick Joseph 

MARTYNAVA, Karina  THOMPSON, Julien Edward

Topics

• Crowdfunding basics

• Schema Mapping / ER (Gathering Data)

• Data Fusion (Data Analysis)

• Data Portal

Crowdfunding

! Crowdfunding

! Schema mapping / ER

! Data fusion

! Data Portal

Crowdfunding• Fund a project by a large number of people

• Start-up, Video games, charity…

! Crowdfunding

! Schema mapping / ER

! Data fusion

! Data Portal

Crowdfunding

• over 2 billions $

• 100K projects

• 10 milion contributors

! Crowdfunding

! Schema mapping / ER

! Data fusion

! Data Portal

Crowford• Gather projects from different sources

• Predict if a project will be successful or not

! Crowdfunding

! Schema mapping / ER

! Data fusion

! Data Portal

Schema Mapping / ER

! Crowdfunding

! Schema mapping / ER

! Data Fusion

! Data Portal

Gathering data

! Crowdfunding

! Schema mapping / ER

! Data Fusion

! Data PortalData Sources

• Same structure

• Same theme (fund projects)

! Crowdfunding

! Schema mapping / ER

! Data Fusion

! Data PortalData Sources

Crowdfund data

Project

Idea (Pen, Video game, Product…)

Packages / Perks What you get / Money

Author(s)

Crawling! Crowdfunding

! Schema mapping / ER

! Data Fusion

! Data Portal

Crawling

• Use JavaScript to generate project page

• Private API that generate JSON

• Python script w/ http2 that generate requests

! Crowdfunding

! Schema mapping / ER

! Data Fusion

! Data Portal

https://www.indiegogo.com/private_api/explore?filter_funding=&filter_percent_funded=&filter_q

uick=new&filter_status=&pg_num=2

Crawling

• Have to extract data from websites

• 2 Spider bots (crawlers) using Scrapy

! Crowdfunding

! Schema mapping / ER

! Data Fusion

! Data Portal

Crawling w/ Scrapy

• Python Framework for extracting data

• Write Spider (crawling bots)

• Parse data and extract with xpath

• Export data (Schema mapping)

• Initialize the spider

• Download and extract data

• Export the item

• Export the item

Data scheme

Author

Project Perks / Packages

! Crowdfunding

! Schema mapping / ER

! Data Fusion

! Data Portal

Project

Author

Project Summary Perks / Packages

! Crowdfunding

! Schema mapping / ER

! Data Fusion

! Data PortalData scheme

Data scheme

• Project / Author

• Project / Perks

• Recommendation : Project / Related_Project

• ER for multiple authors / perks / projects…

! Crowdfunding

! Schema mapping / ER

! Data Fusion

! Data Portal

Results

• Our working set

• 36 000 projects

• 65 000 authors

• over 230 000 perks

! Crowdfunding

! Schema mapping / ER

! Data Fusion

! Data Portal

Data fusion

! Crowdfunding

! Schema mapping / ER

! Data Fusion

! Data Portal

And other data analysis

Recommendation• Use buzzwords in project description

• Use n-grams (word combination)

• Similarity measures using Pairwise metrics

• Linear kernels

• Can be used for data fusion

! Crowdfunding

! Schema mapping / ER

! Data Fusion

! Data Portal

Success Prediction

• How much money has been collected

• How much time

• The average pledge

• …

! Crowdfunding

! Schema mapping / ER

! Data Fusion

! Data Portal

Success PredictionRandom Forests Logistic regression

RESULT 1 RESULT 2

COMPARISON

RESULT

! Crowdfunding

! Schema mapping / ER

! Data Fusion

! Data Portal

Data Portal

! Crowdfunding

! Schema mapping / ER

! Data fusion

! Data Portal

Goal

Browse successful projects

! Crowdfunding

! Schema mapping

! Data fusion

! Data Portal

DataBase

PostgreSQL

DataBase

PostgreSQL

Web Interface

Django

DataBase

PostgreSQL

Web Interface

Django

Project List Project Info

Web Interface

• Project list

• Filter

• Access project page

• Allows you to download the datasets

! Crowdfunding

! Schema mapping

! Data fusion

! Data Portal

Demo