strengthening reproducibility in network science ... · 10. provide public access to scripts, runs,...
TRANSCRIPT
![Page 1: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/1.jpg)
Reproducibility & Generalizability @ TwitterStrengthening Reproducibility in Network Science workshopNetSci 2017
Brandon Roy@bcroygbivJune 19, 2017
![Page 2: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/2.jpg)
Twitter is a real-time information network – it’s what’s happening right now
What is Twitter?
![Page 3: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/3.jpg)
Twitter is a real-time information network – it’s what’s happening right now
I choose other users to followAll tweets by those users render into my timeline
A tweet can be retweeted
If some users I follow in turn follow me, it’s a mutual follow
What is Twitter?
![Page 4: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/4.jpg)
Health, Usage and BehaviorHUB team
- Define and model user “health” at individual and population level- Identify causal factors for health and usage- Characterize user interests- Translate insights into experiments and build prototype systems- ...
![Page 5: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/5.jpg)
- Analytics & Machine Learning- Machine learning infrastructure / platforms- User metrics and revenue modeling- Content understanding (text, images, video)- Data services and integration- User modeling- …
(and friends)Twitter Science
![Page 6: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/6.jpg)
The systematic study of the structure and behavior of the physical and natural world through observation and experiment
Newton, observing apple falling from a tree develops a theory:- Apples are attracted toward the Earth?- Fruit is attracted toward the Earth?- Unobserved force attracts all masses to one another
Develops Law of Universal Gravitation
Depends on a minimal set of conditions
Scientific findings are reproducible under appropriate conditions
Assumption: laws of physics are stable
Science
![Page 7: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/7.jpg)
Developmental psychology – how do children learn words?
Study through observation and experiment
Observational study preserves natural system. Can correlate features of objects & environment (e.g. shape, color, salience) with words learned
Experimental study can isolate and test factors, may be more easily repeatable. But may also lose important aspects of system under analysis
Assumption: human nature / behavior is relatively stable
Science
Medina et. al., 2011
![Page 8: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/8.jpg)
Studying TwitterTwitter is both social and a technical system. Parts are simple, but system is complex
Consists of millions of users producing, sharing, and consuming content
![Page 9: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/9.jpg)
Awareness
Trial
Regular use
Studying TwitterTwitter is both social and a technical system. Parts are simple, but system is complex
Consists of millions of users producing, sharing, and consuming content
How can we make Twitter better? How can we grow the platform?
![Page 10: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/10.jpg)
Awareness
Trial
Regular use
Twitter is both social and a technical system. Parts are simple, but system is complex
Consists of millions of users producing, sharing, and consuming content
How can we make Twitter better? How can we grow the platform?
Studying Twitter
![Page 11: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/11.jpg)
Awareness
Trial
Regular useRegular use
Studying TwitterTwitter is both social and a technical system. Parts are simple, but system is complex
Consists of millions of users producing, sharing, and consuming content
How can we make Twitter better? How can we grow the platform?
![Page 12: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/12.jpg)
Randomly assign users into control / treatment groups
Record key metrics and look for stat. sig. differences between groups
Product experimentationA/B testing
![Page 13: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/13.jpg)
Randomly assign users into control / treatment groups
Record key metrics and look for stat. sig. differences between groups
In this example, we learn green button is “better” than blue…
But we don’t necessarily have a “theory” of button color
Product experimentationA/B testing
If we are able to replicate on other websites, with other text, with potentially other background colors, we’ll start to feel more confident about green buttons
![Page 14: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/14.jpg)
We’ve been experimenting with account recommendations to new users
Change recommendation algorithm for subset of new users and compare to control group
Feel confident finding would be valid (on avg) for all users due to random sampling strategy
If things looks good we expect reproducibility and will “ship it” to all users
Caveat: other parts of system may change, could affect these findings!
Product experimentationDDG = “Duck Duck Goose”
![Page 15: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/15.jpg)
Graph stateConsumptionPassive engagementsRich media
Graph actionsProductionActive engagementsSocial interaction
Many questions we would like to answer but cannot (easily) manipulate through experiment
But we can try to study these questions using other methods
Example: what makes a user “healthy”?
Observational data analysis
![Page 16: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/16.jpg)
Graph stateConsumptionPassive engagementsRich media
Graph actionsProductionActive engagementsSocial interaction
Many questions we would like to answer but cannot (easily) manipulate through experiment
But we can try to study these questions using other methods
Example: what makes a user “healthy”?
Observational data analysis
![Page 17: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/17.jpg)
Characterizing graph state
Link type
0 - 6061 - 500
501 - 3,0003,001 - 25,000
25,001 - 200,000200,001 - 2,000,000
2,000,000+
B’s # followers
Near zeroVery lightLightMedium Non-TweeterMedium TweeterHeavy Non-TweeterHeavy Tweeter
B’s usage state
![Page 18: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/18.jpg)
Characterizing graph state
![Page 19: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/19.jpg)
HypothesisUser’s graph supports their activity, and only certain types of links are important for driving heavy usage
AnalysisMatch users with same covariates except variable in question
Compare matched users who differ on variable in question
For example, find pair of users who have same graph summary counts except for # of small, heavy tweeter accounts followed and look for different health outcomes
Analysis
![Page 20: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/20.jpg)
![Page 21: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/21.jpg)
![Page 22: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/22.jpg)
![Page 23: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/23.jpg)
Very excited when we first got this result
Was intuitive, suggests ingredients for a great Twitter experience
But would be more convincing if we could reproduce analysis with different data.
Better yet, reproduce effect with controlled experiment.
But how to implement this change?
Observational data analysis
![Page 24: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/24.jpg)
1. For every result, keep track of how it was produced2. Avoid manual data manipulation steps3. Archive the exact versions of all external programs used4. Version control all custom scripts5. Record all intermediate results, when possible in standardized formats6. For analyses that include randomness, note underlying random seeds7. Always store raw data behind plots8. Generate hierarchical analysis output, allowing layers of increasing detail to be inspected9. Connect textual statements to underlying results
10. Provide public access to scripts, runs, and results
Reproducibility recommendations
Sandve et. al., 2013
from Sandve et. al., 2013
![Page 25: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/25.jpg)
1. For every result, keep track of how it was produced2. Avoid manual data manipulation steps3. Archive the exact versions of all external programs used4. Version control all custom scripts5. Record all intermediate results, when possible in standardized formats6. For analyses that include randomness, note underlying random seeds7. Always store raw data behind plots8. Generate hierarchical analysis output, allowing layers of increasing detail to be inspected9. Connect textual statements to underlying results
10. Provide public access to scripts, runs, and results
Reproducibility recommendationsfrom Sandve et. al., 2013
Sandve et. al., 2013
Great to reproduce analysis… even better to reproduce the effect!
![Page 26: Strengthening Reproducibility in Network Science ... · 10. Provide public access to scripts, runs, and results Reproducibility recommendations from Sandve et. al., 2013 Sandve et](https://reader033.vdocuments.us/reader033/viewer/2022051810/6018eabc35b9364d95480865/html5/thumbnails/26.jpg)
Sandve GK, Nekrutenko A, Taylor J, Hovig E (2013) Ten Simple Rules for Reproducible Computational Research. PLoS Comput Biol 9(10): e1003285. https://doi.org/10.1371/journal.pcbi.1003285
Medina, T., Snedeker, J., Trueswell, J., & Gleitman, L (2011). How words can and cannot be learned by observation. Proceedings of the National Academy of Sciences, 108(22), 9014.
References