tools for google data
TRANSCRIPT
![Page 1: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/1.jpg)
1
Tools for Google Data
Hal VarianKauffman Blogger Conference
April 2013
![Page 2: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/2.jpg)
2
Google Trends
Google Correlate
Google Consumer Surveys
![Page 3: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/3.jpg)
Searches for [hangover]
Which day of the week are there the most searches for [hangover]?
1: Sunday
2: Monday
3: Tuesday
4: Wednesday
5: Thursday
6: Friday
7: Saturday
![Page 4: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/4.jpg)
Search index for [hangover]
![Page 5: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/5.jpg)
Hangover geo
![Page 6: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/6.jpg)
Hangover-vodka time series
![Page 7: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/7.jpg)
Searches for [civil war]
![Page 8: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/8.jpg)
Searches for [term paper]
![Page 9: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/9.jpg)
Gift for boyfriend v Gift for girlfriend
Forboyfriend
For girlfriend
![Page 10: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/10.jpg)
Gift for husband v Gift for wife
Forhusband
For wife
![Page 11: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/11.jpg)
11
Google Trends
Google Correlate
Google Consumer Surveys
![Page 12: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/12.jpg)
Searches correlated with [weight loss]
![Page 13: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/13.jpg)
Plot of [weight loss] and [best vacation spots]
New Year
![Page 14: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/14.jpg)
Correlated with [weight loss] 3 weeks later
![Page 15: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/15.jpg)
15
Unemployment
![Page 16: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/16.jpg)
Initial claims: good leading indicator for recessions
Grey bars indicate recessions
![Page 17: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/17.jpg)
Google Correlate with initial claims data
![Page 18: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/18.jpg)
Initial claims and [unemployment filing]
![Page 19: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/19.jpg)
Regression models
Baseline model yt = a yt-1 + c + et gives an in-sample MAE of 3.1%
Adding the “filing for unemployment” query yt = a yt-1 + b qt + c + et gives an in-sample MAE of 3.0%
Train using t weeks, forecast t+1 (rolling window forecast)MAE of baseline = 3.2%, MAE with query = 3.2%, 0% improvement
During recession MAE of baseline = 3.7%, MAE with query = 3.3%, 8.7% improvement
![Page 20: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/20.jpg)
Weekly pattern for [unemployment office]
![Page 21: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/21.jpg)
Gun sales
![Page 22: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/22.jpg)
FBI gun sales background check
![Page 23: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/23.jpg)
NICS time series
[stack on] has highest raw correlation[gun shops] is chosen by statistical model
![Page 24: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/24.jpg)
Trend
![Page 25: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/25.jpg)
Seasonal
![Page 26: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/26.jpg)
[gun shops]
![Page 27: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/27.jpg)
Searches on [gun shop]
![Page 28: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/28.jpg)
28
Google Trends
Google Correlate
Google Consumer Surveys
![Page 29: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/29.jpg)
29
How it works
![Page 30: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/30.jpg)
30
![Page 31: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/31.jpg)
31
![Page 32: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/32.jpg)
32
![Page 33: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/33.jpg)
33
b
![Page 34: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/34.jpg)
34
How this changes surveys
Anyone can do them
The cost is dramatically lower
Results come back in a few hours
Surveys can be replicated … or not
You can measure sensitivity wording
![Page 35: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/35.jpg)
35
Consumer Sentiment
![Page 36: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/36.jpg)
Consumer sentiment and retail sales
![Page 37: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/37.jpg)
Methodology
Time series = trend + seasonal + regression + error
Find the best predictors for regression
Fat regression problem (aka p > n)
Sometimes get good fit by chance alone
Want predictors that work out of sample
Does regression provide incremental value over trend+seasonal?
![Page 38: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/38.jpg)
BSTS: Bayesian Structural Time Series
Estimating time series: use Kalman filter techniquesExpress time series as trend + seasonal + noise (“basic structural model”)
Forecast univariate model using Kalman filter
Advantages: flexibility, adaptive, interpretable, handles non-stationarity well
Model selection using “spike and slab” Bayesian regressionSpike: prior probability that coefficient is included in regression
Slab: diffuse prior for coefficient, conditional on inclusion
Estimate a posterior probability that variable is in model
Combines well with Kalman techniques
Final forecast is weighted average of many models, with weights given by posterior probabilities (Bayesian model averaging)
Example of “ensemble estimation”
Agnostic with respect to “true model”
Tends to avoid overfitting by avoiding choice of “best” single model
![Page 39: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/39.jpg)
Probability of inclusion of predictor (n=98, k= 195)
White: procyclicalBlack: countercyclical
![Page 40: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/40.jpg)
Start with “trend”
![Page 41: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/41.jpg)
Start with “trend”
![Page 42: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/42.jpg)
Add “financial planning”
![Page 43: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/43.jpg)
Add “Investing”
![Page 44: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/44.jpg)
Add “Business News”
![Page 45: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/45.jpg)
Add “Search Engines”
![Page 46: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/46.jpg)
Add Energy and Utilities
![Page 47: Tools for Google Data](https://reader036.vdocuments.us/reader036/viewer/2022071600/613d2f1f736caf36b75a503f/html5/thumbnails/47.jpg)
Challenges for the future
Private sector has high-frequency, real time data and a lot of it!
Visa, Mastercard, American Express
UPS and FedEx
Wal-Mart, Target, etc
Supermarket scanner data
Search engines
Government agencies
Long historical series, but usually low frequency
Carefully constructed but labor intensive, with delayed release and periodic revisions
How to combine the public and private data?
How to integrate massive amounts of private sector real-time information with traditional government statistics