hurricane sandy data analytics han dong shujia zhou iab meeting 2013

Hurricane Sandy Data Analytics

Han DongShujia Zhou

IAB Meeting 2013

Outline

• System Overview• Data Collection and Cleaning• Bag-Of-Words Model• Topical Model Visualization• Twitter Activity Graph• Heat Map• Conclusions• Future Work

System Overview

Data Collection and Filtering

Location Size of Data (MB)Florida 100

South and North Carolina 200

Georgia 80

Virginia 100

Maryland / Washington DC 60

New York City 40

New York 100

Massachusetts/Rhode Island 50

~360,000 unique Twitter comments~600 Mbytes of data

Bag-Of-Words Model

Topical Model Visualization

Twitter Activity

26-Oct-12 27-Oct-12 28-Oct-12 29-Oct-12 30-Oct-120

500

1000

1500

2000

2500

3000

3500

4000

4500

EvacuateNot Evacuate

Num

ber o

f Tw

eets

26-Oct-12 27-Oct-12 28-Oct-12 29-Oct-12 30-Oct-120

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

EvacuateNot EvacuateRa

tio

Heat Map

Conclusion

• Implemented a system to automate social media data extraction, processing and visualization.

Future Work

• Apply the current data and system in another major hurricane this year.

This work was funded by NSF CHMPR through NOAA. We thank Ben Kyger for helpful

discussions.