Group 9CAO HengruiCHEN JiahangLAM Wai Kit

Pop Music Portal2015 US Music Year-End Report (Nielsen)

Data Sources

Web CrawlerLanguage Used: JavaScript, node.js

AdvantagesAJAX – Crawling multiple pages at the same time

node.js, jQuery, JSON

DatabasePostgreSQL

AdvantagesRelational Database

NoSQL

Functions/Operators for JSON

Database SchemaER diagram

LimitationsQuality

Inconsistency

Incompleteness

Precision

QuantityLarge volume

Complex JSON structure

Web InterfacePHP, HTML based

http://164.132.194.29:3098/page/index.php

Key featuresFull tables of information

Statistics shown in charts

Search function

Database adminPostgreSQL admin page

http://164.132.194.29:3069/

Key features:Interface for engineers

SQL queries

RSwoosh – IntroJava based library

User determined threshold

Key procedures:Divide data into subsets (R, R’)

Retrieve and compare records from subsets

Assign to closest entity set

Repeat until R is empty

Ref: http://ilpubs.stanford.edu:8090/859/1/2008-7.pdf

RSwoosh – ResultBefore – 61878 records

After – 46772 records

Reduced by 24.4%

Dedupe – IntroPython based library

Key procedures:Manual grouping

Training (Supervised learning)

Blocking map

Pairwise matching within block

Ref: http://dedupe.readthedocs.io/

Dedupe – BlockingBlocking map

ProblemsList of Arabic song names

sourate al munafiqun

sourat al dhuha

sourate athariyat - english translation

sourate al hujurat - hafs muratal

sourat al jumua

sourat al fajr

sourate al hashr

sourate al qalam - warch

ProblemsList of classical music names

24 Préludes, Op.28: 11. in B major

24 Préludes, Op.28: 13. in F sharp major

24 Préludes, Op.28: 13. in F sharp major

24 Préludes, Op.28: 15. in D flat major ("Raindrop")

24 Préludes, Op.28: 15. in D flat major ("Raindrop")

24 Préludes, Op.28: 17. in A flat major

MethodologySimple voting

Term frequency (merged track_name)

Aggregation

Spotify > last.fm > DiscogsSpotify Echonest – over 30m known songs, accessible by Spotify player

Last.fm – Many duplicate entries

Discogs – Inaccurate information but most completed

Group 9CAO HengruiCHEN JiahangLAM Wai Kit