3 nips papers we loved - persagen consulting1705.07874.pdf · vp data science at instacart,...

13
3 NIPS Papers We Loved tech.instacart.com/3-nips-papers-we-loved-befb39a75ec2 Jeremy Stanley VP Data Science at Instacart, conquering the world one at a time. Dec 14, 2017 Know your model’s limits, interpret it’s behavior and learn from variable length sets. At NIPS 2017 what surprised me the most was not the size of the crowds (they were huge), the extravagance of the parties (I sleep early) or the controversy of the “rigor police ” debate (it was entertaining). No, what surprised me the most was the number of papers I saw that (when combined with talks and posters) were both relatively easy to understand and of immediate practical use. In this post, I will briefly explain three of our favorites: 1. Knowing your model’s limits Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles Lakshminarayanan et. al 2017, paper & video (1:00:10) 2. Interpreting model behavior A Unified Approach to Interpreting Model Predictions Lundberg et al. 2017, paper , video (17:45) & github 3. Learning from variable length sets Deep Sets Zaheer et al. 2017, paper & video (16:00) Interpreting model behavior Lundberg et al. 2017, paper , video (17:45) & github Most complex machine learning models are black boxes — we simply cannot fully understand how they work. However, we can gain deeper insight locally into the predictions that they make, and through this insight can better understand our data and models. This understanding can be used to: 1/5

Upload: others

Post on 09-Jun-2020

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: 3 NIPS Papers We Loved - Persagen Consulting1705.07874.pdf · VP Data Science at Instacart, conquering the world one at a time. Dec 14, 2017 Know your model’s limits, interpret

3 NIPS Papers We Lovedtech.instacart.com/3-nips-papers-we-loved-befb39a75ec2

Jeremy StanleyVP Data Science at Instacart, conquering the world one at a time.

Dec 14, 2017

Know your model’s limits, interpret it’s behavior and learn from variable length sets.

At NIPS 2017 what surprised me the most was not the size of the crowds (they were huge), the extravagance of the parties (I sleep early) orthe controversy of the “rigor police” debate (it was entertaining).

No, what surprised me the most was the number of papers I saw that (when combined with talks and posters) were both relatively easy tounderstand and of immediate practical use.

In this post, I will briefly explain three of our favorites:

1. Knowing your model’s limitsSimple and Scalable Predictive Uncertainty Estimation using Deep EnsemblesLakshminarayanan et. al 2017, paper & video (1:00:10)

2. Interpreting model behaviorA Unified Approach to Interpreting Model PredictionsLundberg et al. 2017, paper, video (17:45) & github

3. Learning from variable length setsDeep SetsZaheer et al. 2017, paper & video (16:00)

Interpreting model behavior

Lundberg et al. 2017, paper, video (17:45) & github

Most complex machine learning models are black boxes — we simply cannot fully understand how they work. However, we can gain deeperinsight locally into the predictions that they make, and through this insight can better understand our data and models.

This understanding can be used to:

1/5

victoria
Typewriter
https://tech.instacart.com/3-nips-papers-we-loved-befb39a75ec2 Discusses: https://arxiv.org/abs/1705.07874 https://github.com/slundberg/shap https://www.facebook.com/nipsfoundation/videos/1553537531404147/
Page 2: 3 NIPS Papers We Loved - Persagen Consulting1705.07874.pdf · VP Data Science at Instacart, conquering the world one at a time. Dec 14, 2017 Know your model’s limits, interpret

Build intuition for how our algorithms behaveAlter end user experiences to provide more context for predictionsDebug model building issues arising from data quality, model fit or generalization abilityMeasure the value of different features in a model, and inform decisions for future data collection and engineering

At Instacart, we often want to deeply understand models we build such as:

The expected time until a user places their next order, as a function of their past order, delivery, site and rating behaviorWhat product pairs are good replacements for each-other in case we cannot find what the customer originally requestedHow our customers react to limited delivery availability options or busy pricing

The SHAP (SHapley Additive exPlanations) paper and package provides an elegant way to decompose a model’s predictions into additiveeffects, which can then be easily visualized.

For example, here is a visualization that explains a Light GBM prediction of the chance a household earns $50k or more from a UCI censusdataset:

Lundberg et al. 2017 (github)

In this case, the log-odds likelihood of high income is -1.94, and the largest factor depressing this chance is young age (blue), and the largestfactor increasing income is marital status (red).

Furthermore, you can visualize the aggregate impact of features on model predictions over an entire dataset with visualizations like these:

Lundberg et al. 2017 (github)

Here they find that Age is most predictive, but really because there is a group (young) which is separated and low income. Capital Gain is thenext most predictive, in part because of both very high and very low contributions.

2/5

Page 3: 3 NIPS Papers We Loved - Persagen Consulting1705.07874.pdf · VP Data Science at Instacart, conquering the world one at a time. Dec 14, 2017 Know your model’s limits, interpret

This is a huge improvement over the typical information gain based variable importance visualizations commonly used with packages likeXGBoost and LightGBM, which only show the relative importance of each feature:

R XGBoost Vignette

The package can also provide rich partial dependence plots which show the range of impact that a feature has across the training datasetpopulation:

Lundberg et al. 2017 (github)

Note that the vertical spread of values in the above plot represent interaction effects between Age and other variables (the effect of Agechanges with other variables). This is in contrast to traditional partial dependence plots which show only the effect of varying Age in isolation.

To understand how the SHAP algorithm works, consider this example for a single observation:

3/5

Page 4: 3 NIPS Papers We Loved - Persagen Consulting1705.07874.pdf · VP Data Science at Instacart, conquering the world one at a time. Dec 14, 2017 Know your model’s limits, interpret

Lundberg et al. 2017 (video)

Their model is predicting the chance of high income, and on average predicts a base rate of 20% for the entire population, denoted by E[f(x)].For this specific example (named John in the talk), they predict a 55% probability, denoted by f(x).

The SHAP values answer the question of how they got from 20% to 50% for John.

Lundberg et al. 2017 (video)

They begin by ordering the features randomly, perhaps starting with Age, and ask how much the average prediction of 20% changes for userswhose age is the same as John’s, denoted E[f(x) | x₁]. This can be found by integrating f(x) over all other features besides x₁ in the trainingdataset (a process that can be done efficiently in trees).

Suppose that they find that the prediction goes up to 35%, and so this gives them an estimate for the effect of Age, ϕ₁=15%. They theniteratively repeat this process through the remaining variables (concluding with marital status), to estimate ϕ₂, ϕ₃ and ϕ₄ for each of the otherthree features in this example:

Lundberg et al. 2017 (video)

However, unless a model is purely additive, the estimates for ϕ will vary with the ordering of features chosen. The SHAP algorithm solves thisby averaging over all possible 2ᴺ orderings. The computational burden of computing all such orderings is alleviated by sampling M of themand using a regression model to attribute the impact from the samples to each feature.

The paper justifies the above approach using game theory, and further shows that this theory unifies other interpretation methodologies suchas LIME and DeepLIFT:

4/5

Page 5: 3 NIPS Papers We Loved - Persagen Consulting1705.07874.pdf · VP Data Science at Instacart, conquering the world one at a time. Dec 14, 2017 Know your model’s limits, interpret

Lundberg et al. 2017 (video)

And finally, because no NIPS paper would be complete without an MNIST example, they show that the SHAP algorithm does a better job atexplaining what part of an 8 represents the essence of an 8 (as opposed to a 3):

Lundberg et al. 2017 (paper)

5/5

Page 6: 3 NIPS Papers We Loved - Persagen Consulting1705.07874.pdf · VP Data Science at Instacart, conquering the world one at a time. Dec 14, 2017 Know your model’s limits, interpret

May 11, 2018

Demystifying Black-Box Models with SHAP Value Analysismedium.com/civis-analytics/demystifying-black-box-models-with-shap-value-analysis-3e20b536fc80

As an Applied Data Scientist at Civis, I implement the latest data science research to solve real-world problems. We recently worked with aglobal tool manufacturing company to reduce churn among their most loyal customers. A newly proposed tool, called SHAP (SHapley AdditiveexPlanation) values, allowed us to build a complex time-series XGBoost model capable of making highly accurate predictions for whichcustomers were at risk, while still allowing for an individual-level interpretation of the factors that made each of these customers more or lesslikely to churn.

To understand why this is important, we need to take a closer look at the concepts of model accuracy and interpretability. Until recently, wealways had to choose between an accurate model that was hard to interpret, or a simple model that was easy to explain but sacrificed someaccuracy. Classic methods like logistic regression or a simple decision tree make it easy to explain why we assign a person to the positive ornegative class, but there is only so much predictive power we can squeeze out of these basic models. To improve accuracy, more complexmodels may use thousands of these decision trees and then combine their results with yet another model or ensemble rule (e.g. majorityvote). On the other end of the complexity spectrum, deep learning uses neural networks with multiple interconnected layers, each layerlooking at a higher level of abstraction of the underlying data. This added complexity gives these models more flexibility, allowing them toreach high accuracy levels that cannot be obtained by simple models, but at the expense of our ability to comprehend why the model madethe predictions it did. Even the people who designed and trained the model can no longer explain what led one person to get assigned to oneclass over another. For the work we do at Civis (where our models have to generate insights and recommendations for actions), getting thetrade-off between accuracy and interpretability just right can be a difficult balancing act. With SHAP values, we are finally able to get both!

The SHAP values technique was proposed in recent papers by Scott M. Lundberg from the University of Washington [ 1, 2]. It is based onShapley values, a technique used in game theory to determine how much each player in a collaborative game has contributed to its success.In our case, each SHAP value measures how much each feature in our model contributes, either positively or negatively, to a customer’spredicted churn risk score (see Figure 1). This is a similar idea to feature importance in logistic regression, where we can determine theimpact of each feature by looking at the magnitude of its coefficient. However, SHAP values offer two important benefits. First, SHAP valuescan be calculated for any tree-based model, so instead of being restricted to simple, linear — and therefore less accurate — logistic regressionmodels, we can build complex, non-linear and more accurate models. Second, each individual customer will have their own set of SHAPvalues. Traditional feature importance algorithms will tell us which features are most important across the entire population, but this one-size-fits-all approach doesn’t always apply to each individual customer. A factor that is an important driver for one customer may be a non-factor foranother. By looking only at the global trends, these individual variations can get lost, with only the most common denominators remaining. Withindividual-level SHAP values, we can pinpoint which factors are most impactful for each customer, allowing us to customize our next actionsaccordingly.

1/2

victoria
Typewriter
https://medium.com/civis-analytics/demystifying-black-box-models-with-shap-value-analysis-3e20b536fc80
Page 7: 3 NIPS Papers We Loved - Persagen Consulting1705.07874.pdf · VP Data Science at Instacart, conquering the world one at a time. Dec 14, 2017 Know your model’s limits, interpret

Figure 1. SHAP values measure the impact of each variable on a customer’s Engagement score (measuring their likelihood toremain a loyal customer in the next month or year). For each individual customer, this allows us to identify the biggest risk factors(red arrows) and protective factors (blue arrows), and recommend a tailored intervention plan.

While SHAP values can be a great tool, they do have shortcomings (although they are common in calculating feature importance usingobservational data). For one, SHAP values are sensitive to high correlations among different features. When features are correlated, theirimpact on the model score can be split among them in an infinite number of ways. This means that the SHAP values will be lower than if allbut one of the correlated feature(s) had been removed from the model. The risk is that dividing impacts this way makes them look lessimportant than if their impacts remained undivided. To be fair, all known feature importance methods have this problem. A second shortcomingis that SHAP values represent a descriptive approximation of the predictive model. For example, SHAP values can tell us that for a givencustomer, a low number of sales visits has the largest negative impact on their risk score, so we may decide to schedule more sales visits inthe upcoming month. However, we cannot determine based on the SHAP values alone what the impact of this intervention will be. Again, thisis a fundamental limitation to data science. There is only so much we can do with observational data. To accurately estimate the impact ofdifferent churn prevention techniques, we will need to conduct a randomized controlled trial (RCT).

We think there’s a lot of promise in SHAP values. Instead of having to choose between accuracy and interpretability, we finally have a tool thatlets us push the envelope in terms of model complexity and accuracy, while still allowing us to derive intuitive explanations for each individualprediction. SHAP values have been added to the XGBoost library in Python, so the tool is available to anyone. Scott Lundberg, the author ofthe SHAP values method, has expressed interest in expanding the method to a broader selection of models, beyond tree-based algorithms.As we continue to test this out further, we’ll report back with our experience!

References:

Lundberg SM, Lee SI (2017), “Consistent feature attribution for tree ensembles”, presented at the 2017 ICML Workshop on HumanInterpretability in Machine Learning (WHI 2017), Sydney, NSW, Australia (https://arxiv.org/abs/1706.06060)Lundberg SM, Lee SI (2017), “A Unified Approach to Interpreting Model Predictions”, Neural Information Processing Systems (NIPS)2017 (https://arxiv.org/abs/1705.07874)

2/2

Page 8: 3 NIPS Papers We Loved - Persagen Consulting1705.07874.pdf · VP Data Science at Instacart, conquering the world one at a time. Dec 14, 2017 Know your model’s limits, interpret

�✁✂✄☎✆☎✄✂✝✁✞ ✟✠✡✆☛✄☞ ✌✠✍✄☛✎ ✏✝✂✑ ✒✓✔✕

✖✗ ✘✙✚✛✜✢✣✗✢ ✤✥✦✧✢✘✛✗ ✘✗ ✢★✦ ✩✘✦✪✫ ✛✩ ✙✣✬★✘✗✦ ✪✦✣✜✗✘✗✭ ✘✧ ✮★✯ ✣✗ ✣✪✭✛✜✘✢★✙✙✣✫✦ ✣ ✬✦✜✢✣✘✗ ✫✦✬✘✧✘✛✗✰

✱★✘✧ ✘✧ ✘✙✚✛✜✢✣✗✢ ✩✛✜ ✣ ✲✣✜✘✦✢✯ ✛✩ ✜✦✣✧✛✗✧✰ ✖✧ ✣✗ ✦✗✫ ✥✧✦✜✳ ✴ ✣✙✙✛✜✦ ✪✘✵✦✪✯ ✢✛ ✢✜✥✧✢ ✣ ✜✦✬✛✙✙✦✗✫✣✢✘✛✗ ✘✩ ✴ ✥✗✫✦✜✧✢✣✗✫✮★✯ ✘✢✮✣✧ ✦✶✚✛✧✦✫ ✢✛ ✙✦✰ ✖✧ ✣✗ ✛✜✭✣✗✘✷✣✢✘✛✗✳ ✥✗✫✦✜✧✢✣✗✫✘✗✭ ✢★✣✢ ✬✥✧✢✛✙✦✜✧✙✣✫✦ ✣ ✚✥✜✬★✣✧✦ ✸✦✬✣✥✧✦ ✢★✘✧ ✬✣✙✚✣✘✭✗ ✮✣✧✚✣✜✢✘✬✥✪✣✜✪✯ ✦✹✦✬✢✘✲✦ ✬✣✗ ✣✪✪✛✮✙✦ ✢✛ ✢✣✘✪✛✜ ✙✯ ✩✥✢✥✜✦ ✛✥✢✜✦✣✬★ ✦✹✛✜✢✧✰

✺✛✮✦✲✦✜✳ ✢★✘✧ ✘✧ ✣ ✬★✣✪✪✦✗✭✘✗✭ ✣✗✫ ✧✢✘✪✪ ✫✦✲✦✪✛✚✘✗✭ ✩✘✦✪✫ ✘✗ ✙✣✬★✘✗✦ ✪✦✣✜✗✘✗✭✰ ✴✗ ✢★✘✧ ✚✛✧✢✳ ✴ ✣✙ ✭✛✘✗✭ ✢✛ ✫✘✧✬✥✧✧ ✦✶✣✬✢✪✯ ✮★✣✢ ✘✢✙✦✣✗✧ ✢✛ ✘✗✢✦✜✚✜✦✢ ✣ ✙✛✫✦✪✳ ✣✗✫ ✦✶✚✪✛✜✦ ✣ ✗✛✲✦✪ ✢✦✬★✗✘✤✥✦ ✬✣✪✪✦✫ ✻✺✖✼ ✽★✢✢✚✧✾✿✿✭✘✢★✥✸✰✬✛✙✿✧✪✥✗✫✸✦✜✭✿✧★✣✚❀ ✮★✘✬★ ✘✧✚✣✜✢✘✬✥✪✣✜✪✯ ✦✹✦✬✢✘✲✦ ✣✢ ✣✪✪✛✮✘✗✭ ✥✧ ✢✛ ✢✣✵✦ ✢★✦ ★✛✛✫ ✛✹ ✬✛✙✚✪✦✶ ✣✪✭✛✜✘✢★✙✧✰

❁✛✗✢✦✗✢✧✾

❂★✣✢ ✫✛✦✧ ✘✢ ✙✦✣✗ ✢✛ ✘✗✢✦✜✚✜✦✢ ✣ ✙✛✫✦✪❃✱★✦ ✬✛✙✚✪✦✶✘✢✯ ❄ ✘✗✢✦✜✚✜✦✢✣✸✘✪✘✢✯ ✢✜✣✫✦ ✛✹❅✰✺✛✮ ✬✣✗ ✮✦ ✘✗✢✦✜✚✜✦✢ ✬✛✙✚✪✦✶ ✙✛✫✦✪✧❃❆✰

❅✰

✻★✣✚✪✦✯ ✲✣✪✥✦✧✖✗ ✛✲✦✜✲✘✦✮❅✰❁✣✪✬✥✪✣✢✘✗✭ ✣ ✻★✣✚✪✦✯ ✲✣✪✥✦❆✰

❆✰

✻✺✖✼ ✲✣✪✥✦✧❇✰

❈❉❊❋ ●❍■❏ ❑❋ ▲■❊▼ ❋❍ ❑▼❋■◆❖◆■❋ ❊ ▲❍●■P ◗❊▼● ❘❉❙ ❑❏ ❑❋ ❏❍ ❉❊◆●❚❯

❱✦✢❲✧ ✧✢✣✜✢ ✸✯ ✫✦✩✘✗✘✗✭ ✦✶✣✬✢✪✯ ✮★✣✢ ✘✢ ✙✦✣✗✧ ✢✛ ✘✗✢✦✜✚✜✦✢ ✣ ✙✛✫✦✪✰ ✖✢ ✣ ✲✦✜✯ ★✘✭★ ✪✦✲✦✪✳ ✴ ✮✣✗✢ ✢✛ ✥✗✫✦✜✧✢✣✗✫✮★✣✢ ✙✛✢✘✲✣✢✦✫ ✣✬✦✜✢✣✘✗ ✚✜✦✫✘✬✢✘✛✗✰

❳✛✜ ✘✗✧✢✣✗✬✦✳ ✪✦✢✧ ✜✦✥✧✦ ✢★✦ ✚✜✛✸✪✦✙ ✩✜✛✙ ✢★✦ ❨❩❬✛✛✧✢ ✫✛✬✥✙✦✗✢✣✢✘✛✗✳ ✮★✦✜✦ ✭✘✲✦✗ ✢★✦ ✣✭✦✳ ✭✦✗✫✦✜ ✣✗✫ ✛✬✬✥✚✣✢✘✛✗ ✛✩ ✣✗✘✗✫✘✲✘✫✥✣✪✳ ✴ ✮✣✗✢ ✢✛ ✚✜✦✫✘✬✢ ✮★✦✢★✦✜ ✛✜ ✗✛✢ ✢★✦✯ ✮✘✪✪ ✪✘✵✦ ✬✛✙✚✥✢✦✜ ✭✣✙✦✧✾

✴✗ ✢★✘✧ ✬✣✧✦✳ ✙✯ ✘✗✚✥✢ ✩✦✣✢✥✜✦✧ ✣✜✦ ✣✭✦✳ ✭✦✗✫✦✜ ✣✗✫ ✛✬✬✥✚✣✢✘✛✗✰ ✴ ✮✣✗✢ ✢✛ ✵✗✛✮ ★✛✮ ✢★✦✧✦ ✩✦✣✢✥✜✦✧ ✘✙✚✣✬✢✦✫ ✢★✦✙✛✫✦✪❲✧✚✜✦✫✘✬✢✘✛✗ ✢★✣✢ ✧✛✙✦✛✗✦✮✛✥✪✫ ✪✘✵✦ ✬✛✙✚✥✢✦✜ ✭✣✙✦✧✰

✺✛✮✦✲✦✜✳ ✢★✦✜✦ ✣✜✦ ✢✮✛ ✫✘✹✦✜✦✗✢ ✮✣✯✧ ✢✛ ✘✗✢✦✜✚✜✦✢ ✢★✘✧✾

❭✗ ✣ ❪❫❴❵❛❫ ❱✛✛✵✘✗✭ ✣✢ ✢★✦ ✦✗✢✘✜✦ ✫✣✢✣✧✦✢✳ ✮★✘✬★ ✩✦✣✢✥✜✦✧ ✫✘✫ ✢★✦ ✣✪✭✛✜✘✢★✙ ✩✘✗✫✙✛✧✢ ✚✜✦✫✘✬✢✘✲✦❃ ❨❩❬✛✛✧✢❲✧ ❜✭✦✢❝✧✬✛✜✦✽❀❲✩✥✗✬✢✘✛✗ ❄ ✮★✘✬★ ✬✛✥✗✢✧ ★✛✮✙✣✗✯ ✢✘✙✦✧ ✣ ✩✦✣✢✥✜✦ ✮✣✧ ✥✧✦✫ ✢✛ ✧✚✪✘✢ ✢★✦ ✫✣✢✣ ❄ ✘✧ ✣✗ ✦✶✣✙✚✪✦ ✛✩ ✬✛✗✧✘✫✦✜✘✗✭ ✭✪✛✸✣✪✩✦✣✢✥✜✦ ✘✙✚✛✜✢✣✗✬✦✳ ✧✘✗✬✦ ✘✢ ✪✛✛✵✧ ✣✢ ✮★✣✢ ✮✣✧ ✪✦✣✜✗✦✫ ✩✜✛✙ ✣✪✪ ✢★✦ ✫✣✢✣✰

❞ ❡❢ ❣

❤✐✐❥❦❧♠♠♥♦♣❡❥qr♦s❦t♥❡✉♠✈✇❦❡①✈♥✇❦♠②♣✐✇✈❥✈✇✐②♣③④♥❡✉❥r✇⑤④✉❡⑥✇rttt ⑦♠⑧⑨♠❞⑩❶ ❞❞❧❷❸ ❹❺

victoria
Typewriter
https://canopylabs.com/resources/interpreting-complex-models-with-shap/
Page 9: 3 NIPS Papers We Loved - Persagen Consulting1705.07874.pdf · VP Data Science at Instacart, conquering the world one at a time. Dec 14, 2017 Know your model’s limits, interpret

�✁✂

❭✗ ✣ ❫❴✄❛❫☎✣✯✸✦✳ ✣✬✜✛✧✧ ✣✪✪ ✘✗✫✘✲✘✫✥✣✪✧✳ ✣✭✦ ✮✣✧ ✢★✦✙✛✧✢ ✘✙✚✛✜✢✣✗✢ ✩✦✣✢✥✜✦✳ ✣✗✫ ✯✛✥✗✭✦✜ ✚✦✛✚✪✦ ✣✜✦✙✥✬★✙✛✜✦ ✪✘✵✦✪✯✢✛ ✪✘✵✦ ✬✛✙✚✥✢✦✜ ✭✣✙✦✧✰ ❬✥✢ ✘✩ ❳✜✣✗✵ ✘✧ ✣ ✆✝✞✯✦✣✜✞✛✪✫ ✮★✛✮✛✜✵✧ ✣✧ ✣ ✲✘✫✦✛ ✭✣✙✦ ✢✦✧✢✦✜✳ ✘✢❲✧ ✪✘✵✦✪✯ ✢★✣✢ ★✘✧ ✛✬✬✥✚✣✢✘✛✗ ✘✧✭✛✘✗✭ ✢✛ ✸✦✙✥✬★✙✛✜✦ ✧✘✭✗✘✩✘✬✣✗✢ ✢★✣✗ ★✘✧ ✣✭✦ ✘✗ ✫✦✢✦✜✙✘✗✘✗✭ ✮★✦✢★✦✜ ★✦ ✪✘✵✦✧ ✬✛✙✚✥✢✦✜ ✭✣✙✦✧✰ ✴✫✦✗✢✘✩✯✘✗✭ ✮★✘✬★✩✦✣✢✥✜✦✧ ✮✦✜✦✙✛✧✢ ✘✙✚✛✜✢✣✗✢ ✩✛✜ ❳✜✣✗✵ ✧✚✦✬✘✩✘✬✣✪✪✯ ✘✗✲✛✪✲✦✧ ✩✘✗✫✘✗✭ ✩✦✣✢✥✜✦ ✘✙✚✛✜✢✣✗✬✦✧ ✛✗ ✣ ❜✪✛✬✣✪❲ ❄ ✘✗✫✘✲✘✫✥✣✪ ❄ ✪✦✲✦✪✰

❂✘✢★ ✢★✘✧ ✫✦✩✘✗✘✢✘✛✗ ✛✥✢ ✛✩ ✢★✦ ✮✣✯✳ ✪✦✢❲✧ ✙✛✲✦ ✛✗ ✢✛ ✛✗✦ ✛✩ ✢★✦ ✸✘✭ ✬★✣✪✪✦✗✭✦✧ ✘✗ ✙✛✫✦✪ ✘✗✢✦✜✚✜✦✢✣✸✘✪✘✢✯✾

✟✠✡☛☞✌✍ ✎✏ ✑✒✓✔✒✒✌ ☞✌✓✒✠✕✠✒✓✡✑☞✖☞✓✗ ✡✌☛ ✘✎✙✕✖✒✚☞✓✗

❱✦✢❲✧ ✬✛✗✧✘✫✦✜ ✣ ✲✦✜✯ ✧✘✙✚✪✦ ✙✛✫✦✪✾ ✣ ✪✘✗✦✣✜ ✜✦✭✜✦✧✧✘✛✗✰ ✱★✦ ✛✥✢✚✥✢ ✛✩ ✢★✦✙✛✫✦✪ ✘✧

✴✗ ✢★✦ ✪✘✗✦✣✜ ✜✦✭✜✦✧✧✘✛✗ ✙✛✫✦✪ ✣✸✛✲✦✳ ✴ ✣✧✧✘✭✗ ✦✣✬★ ✛✩ ✙✯ ✩✦✣✢✥✜✦✧ ✣ ✬✛✦✹✘✬✘✦✗✢ ✣✗✫ ✣✫✫ ✦✲✦✜✯✢★✘✗✭ ✥✚ ✢✛ ✭✦✢ ✙✯ ✛✥✢✚✥✢✰✴✗ ✢★✦ ✬✣✧✦ ✛✩ ✙✯ ✬✛✙✚✥✢✦✜ ✭✣✙✦✧ ✚✜✛✸✪✦✙✳✙✯ ✘✗✚✥✢ ✩✦✣✢✥✜✦✧ ✮✛✥✪✫ ✸✦ ✽ ❀✰

✴✗ ✢★✘✧ ✬✣✧✦✳ ✘✢✧ ✧✥✚✦✜ ✦✣✧✯ ✢✛ ✩✘✗✫ ✢★✦ ✘✙✚✛✜✢✣✗✬✦ ✛✩ ✣ ✩✦✣✢✥✜✦✛ ✘✩ ★✣✧ ✣ ✪✣✜✭✦ ✣✸✧✛✪✥✢✦ ✲✣✪✥✦✳ ✢★✦✗ ✩✦✣✢✥✜✦ ★✣✫ ✣ ✸✘✭ ✘✙✚✣✬✢✛✗ ✢★✦ ✩✘✗✣✪ ✛✥✢✬✛✙✦ ✽✦✰✭✰ ✘✩ ✘✧ ✪✣✜✭✦✳ ✢★✦✗ ✣✭✦ ✮✣✧ ✣✗ ✘✙✚✛✜✢✣✗✢ ✩✦✣✢✥✜✦❀✰ ✺✛✮✦✲✦✜✳ ✢★✦✜✦ ✘✧ ✣✪✧✛ ✣ ✫✜✣✮✸✣✬✵✳ ✮★✘✬★ ✘✧ ✢★✣✢✢★✘✧ ✙✛✫✦✪ ✘✧ ✧✛ ✧✘✙✚✪✦ ✢★✣✢ ✘✢ ✬✣✗ ✛✗✪✯ ✥✗✬✛✲✦✜ ✪✘✗✦✣✜ ✜✦✪✣✢✘✛✗✧★✘✚✧✰

❳✛✜ ✘✗✧✢✣✗✬✦✳ ✙✣✯✸✦ ✣✭✦ ✘✧ ✣✗ ✘✙✚✛✜✢✣✗✢ ✩✦✣✢✥✜✦✳ ✣✗✫ ✘✩ ✯✛✥❲✜✦ ✸✦✢✮✦✦✗ ❅❆ ✣✗✫ ❅✜ ✯✛✥❲✜✦✙✥✬★✙✛✜✦ ✪✘✵✦✪✯ ✢✛ ✪✘✵✦ ✬✛✙✚✥✢✦✜✭✣✙✦✧ ✢★✣✗ ✣✢ ✣✗✯ ✛✢★✦✜ ✣✭✦✛ ✧✘✗✬✦ ✢★✘✧ ✘✧ ✣ ✗✛✗✞✪✘✗✦✣✜ ✜✦✪✣✢✘✛✗✧★✘✚✳ ✣ ✪✘✗✦✣✜ ✜✦✭✜✦✧✧✘✛✗ ✮✛✥✪✫✗❲✢ ✸✦ ✣✸✪✦ ✢✛ ✥✗✬✛✲✦✜ ✘✢✰

✴✗ ✛✜✫✦✜ ✢✛ ✥✗✬✛✲✦✜ ✢★✘✧ ✙✛✜✦ ✬✛✙✚✪✘✬✣✢✦✫ ✜✦✪✣✢✘✛✗✧★✘✚✳ ✴❲✪✪ ✗✦✦✫ ✣✙✛✜✦ ✬✛✙✚✪✘✬✣✢✦✫✙✛✫✦✪✰

✺✛✮✦✲✦✜✳ ✣✧ ✧✛✛✗ ✣✧ ✴ ✧✢✣✜✢ ✥✧✘✗✭ ✙✛✜✦ ✬✛✙✚✪✘✬✣✢✦✫✙✛✫✦✪✧✳ ✴ ✪✛✧✦ ✢★✦ ✦✣✧✦ ✛✩ ✘✗✢✦✜✚✜✦✢✣✸✘✪✘✢✯ ✮★✘✬★ ✴ ✭✛✢ ✮✘✢★ ✢★✘✧ ✪✘✗✦✣✜✙✛✫✦✪✰ ✴✗ ✩✣✬✢✳ ✣✧ ✧✛✛✗ ✣✧ ✴ ✢✜✯ ✢✛ ✧✢✣✜✢ ✥✗✬✛✲✦✜✘✗✭ ✗✛✗✞✪✘✗✦✣✜✳ ✛✜ ✦✲✦✗ ✘✗✢✦✜✮✛✲✦✗ ✜✦✪✣✢✘✛✗✧★✘✚✧ ❄ ✦✰✭✰ ✮★✣✢ ✘✩ ✣✭✦ ✘✧ ✘✙✚✛✜✢✣✗✢✫✦✚✦✗✫✘✗✭ ✛✗ ✯✛✥✜ ✭✦✗✫✦✜❃ ❄ ✢★✦✗ ✘✢ ✸✦✬✛✙✦✧ ✲✦✜✯ ✢✜✘✬✵✯ ✢✛ ✘✗✢✦✜✚✜✦✢ ✢★✦✙✛✫✦✪✰

✱★✘✧ ✫✦✬✘✧✘✛✗ ❄ ✸✦✢✮✦✦✗ ✣✗ ✦✣✧✯ ✢✛ ✘✗✢✦✜✚✜✦✢✙✛✫✦✪ ✮★✘✬★ ✬✣✗ ✛✗✪✯ ✥✗✬✛✲✦✜ ✧✘✙✚✪✦ ✜✦✪✣✢✘✛✗✧★✘✚✧✳ ✛✜ ✬✛✙✚✪✦✶✙✛✫✦✪✧ ✮★✘✬★✬✣✗ ✩✘✗✫ ✲✦✜✯ ✘✗✢✦✜✦✧✢✘✗✭ ✚✣✢✢✦✜✗✧ ✢★✣✢✙✣✯ ✸✦ ✫✘✹✘✬✥✪✢ ✢✛ ✘✗✢✦✜✚✜✦✢ ❄ ✘✧ ✢★✦ ✢✜✣✫✦ ✛✹ ✸✦✢✮✦✦✗ ✘✗✢✦✜✚✜✦✢✣✸✘✪✘✢✯ ✣✗✫ ✬✛✙✚✪✦✶✘✢✯✰

✱★✘✧ ✘✧ ✣✫✫✘✢✘✛✗✣✪✪✯ ✬✛✙✚✪✘✬✣✢✦✫ ✸✯ ✢★✦ ✩✣✬✢ ✢★✣✢ ✴ ✙✘✭★✢ ✸✦ ✘✗✢✦✜✚✜✦✢✘✗✭ ✣ ✙✛✫✦✪ ✸✦✬✣✥✧✦ ✴❲✙ ★✛✚✘✗✭ ✢✛ ✪✦✣✜✗ ✧✛✙✦✢★✘✗✭ ✗✦✮✣✗✫ ✘✗✢✦✜✦✧✢✘✗✭ ✣✸✛✥✢ ✢★✦ ✫✣✢✣✰ ✴✩ ✢★✘✧ ✘✧ ✢★✦ ✬✣✧✦✳ ✣ ✪✘✗✦✣✜ ✙✛✫✦✪ ✙✣✯ ✗✛✢ ✬✥✢ ✘✢✳ ✧✘✗✬✦ ✴ ✙✣✯ ✣✪✜✦✣✫✯ ✸✦ ✩✣✙✘✪✘✣✜ ✮✘✢★ ✢★✦✜✦✪✣✢✘✛✗✧★✘✚✧ ✘✢ ✮✛✥✪✫ ✥✗✬✛✲✦✜✰

✱★✦ ✘✫✦✣✪ ✬✣✧✦ ✮✛✥✪✫ ✢★✦✜✦✩✛✜✦ ✸✦ ✢✛ ★✣✲✦ ✣ ✬✛✙✚✪✦✶✙✛✫✦✪ ✮★✘✬★ ✴ ✬✣✗ ✣✪✧✛ ✘✗✢✦✜✚✜✦✢✰

✢✎✔ ✘✡✌ ✔✒ ☞✌✓✒✠✕✠✒✓ ✘✎✙✕✖✒✚✙✎☛✒✖✣✤

✱★✘✗✵✘✗✭ ✣✸✛✥✢ ✪✘✗✦✣✜ ✜✦✭✜✦✧✧✘✛✗✧ ★✣✧ ✯✘✦✪✫✦✫ ✣ ✭✛✛✫✮✣✯ ✛✩ ✢★✘✗✵✘✗✭ ✣✸✛✥✢✙✛✫✦✪ ✘✗✢✦✜✚✜✦✢✣✢✘✛✗✧✾

✴❲✪✪ ✣✧✧✘✭✗ ✢✛ ✦✣✬★ ✩✦✣✢✥✜✦ ✣ ✬✛✦✹✘✬✘✦✗✢ ✮★✘✬★ ✫✦✧✬✜✘✸✦✧ ❄ ✪✘✗✦✣✜✪✯ ❄ ★✛✮ ✢★✦ ✩✦✣✢✥✜✦ ✣✹✦✬✢✧ ✢★✦ ✛✥✢✚✥✢ ✛✩ ✢★✦✙✛✫✦✪✰ ❂✦❲✲✦✣✪✜✦✣✫✯ ✫✘✧✬✥✧✧✦✫ ✢★✦ ✧★✛✜✢✬✛✙✘✗✭✧ ✛✩ ✢★✘✧ ✙✛✫✦✪✳ ✸✥✢ ✸✦✣✜ ✮✘✢★✙✦✾

✖✬✜✛✧✧✙✣✗✯ ✫✣✢✣ ✚✛✘✗✢✧✳ ✢★✦ ✬✛✦✹✘✬✘✦✗✢✧ ✮✘✪✪ ✩✣✘✪ ✢✛ ✬✣✚✢✥✜✦ ✬✛✙✚✪✦✶ ✜✦✪✣✢✘✛✗✧★✘✚✧✰ ❬✥✢ ✛✗ ✣✗ ✘✗✫✘✲✘✫✥✣✪ ✪✦✲✦✪✳ ✢★✦✗ ✢★✦✯❲✪✪ ✫✛✩✘✗✦✳ ✧✘✗✬✦ ✩✛✜ ✣ ✧✘✗✭✪✦ ✚✜✦✫✘✬✢✘✛✗✳ ✦✣✬★ ✲✣✜✘✣✸✪✦ ✮✘✪✪ ✢✜✥✪✯ ★✣✲✦ ✘✙✚✣✬✢✦✫ ✢★✦✙✛✫✦✪❲✧ ✚✜✦✫✘✬✢✘✛✗ ✸✯ ✣ ✬✛✗✧✢✣✗✢ ✲✣✪✥✦✰

❳✛✜ ✘✗✧✢✣✗✬✦✳ ✬✛✗✧✘✫✦✜ ✢★✦ ✬✣✧✦ ✛✩ ❳✜✣✗✵✳ ✢★✦ ✆✝✞✯✦✣✜✞✛✪✫ ✲✘✫✦✛ ✭✣✙✦ ✢✦✧✢✦✜ ✮★✛ ✪✛✲✦✧ ✬✛✙✚✥✢✦✜ ✭✣✙✦✧✰ ❳✛✜ ★✘✙✳ ✮✘✪✪ ✸✦★✘✭★✳ ✣✗✫ ✮✘✪✪ ✸✦ ✪✛✮✰

❬✥✢ ✢★✦✗✳ ✩✛✜ ❬✛✸✸✯✳ ✣ ❅✥✞✯✦✣✜✞✛✪✫✳ ✮✘✪✪ ✸✦ ★✘✭★ ✧✘✗✬✦ ✢★✦✙✛✫✦✪ ★✣✧ ✧✦✦ ✢★✣✢ ❅✥✞✯✦✣✜ ✛✪✫✧ ✢✦✗✫ ✪✛✲✦ ✬✛✙✚✥✢✦✜ ✭✣✙✦✧❵✦✄❛✧★✦ ✢★✦✯ ✣✜✦ ❅✥ ✯✦✣✜✧ ✛✪✫✰

❂★✣✢ ✮✦❲✲✦ ✫✛✗✦ ★✦✜✦ ✘✧ ✢✣✵✦ ✣ ✬✛✙✚✪✦✶✙✛✫✦✪✳ ✮★✘✬★ ★✣✧ ✪✦✣✜✗✢ ✗✛✗✞✪✘✗✦✣✜ ✚✣✢✢✦✜✗✧ ✘✗ ✢★✦ ✫✣✢✣✳ ✣✗✫ ✸✜✛✵✦✗ ✘✢ ✫✛✮✗ ✘✗✢✛ ✪✛✢✧

✩ ❡❢ ❣

❤✐✐❥❦❧♠♠♥♦♣❡❥qr♦s❦t♥❡✉♠✈✇❦❡①✈♥✇❦♠②♣✐✇✈❥✈✇✐②♣③④♥❡✉❥r✇⑤④✉❡⑥✇rttt ⑦♠⑧⑨♠❞⑩❶ ❞❞❧❷❸ ❹❺

Page 10: 3 NIPS Papers We Loved - Persagen Consulting1705.07874.pdf · VP Data Science at Instacart, conquering the world one at a time. Dec 14, 2017 Know your model’s limits, interpret

��✂

�✁✂

✛✩ ✪✘✗✦✣✜ ✙✛✫✦✪✧ ✮★✘✬★ ✫✦✧✬✜✘✸✦ ✂✄☎✂✆✂☎✧❛❫ ☎❛✝❛ ✞❴✂✄✝★✟ ✴✢✧ ✘✙✚✛✜✢✣✗✢ ✢✛ ✗✛✢✦ ✢★✣✢ ✢★✦✧✦ ✦✶✚✪✣✗✣✢✘✛✗ ✬✛✦✹✘✬✘✦✗✢✧ ✣✜✦ ✗✛✢ ✢★✦✛✥✢✚✥✢ ✛✩ ✢★✦✙✛✫✦✪✳ ✸✥✢ ✜✣✢★✦✜ ✮★✣✢ ✮✦ ✣✜✦ ✥✧✘✗✭ ✢✛ ✘✗✢✦✜✚✜✦✢ ✢★✘✧ ✙✛✫✦✪✰ ❬✯ ✣✭✭✜✦✭✣✢✘✗✭ ✣✪✪ ✛✩ ✢★✦✧✦ ✧✘✙✚✪✦✳ ✘✗✫✘✲✘✫✥✣✪✙✛✫✦✪✧ ✢✛✭✦✢★✦✜✳ ✮✦ ✬✣✗ ✥✗✫✦✜✧✢✣✗✫ ★✛✮ ✢★✦✙✛✫✦✪ ✸✦★✣✲✦✧ ✣✬✜✛✧✧ ✣✪✪ ✢★✦ ✬✥✧✢✛✙✦✜✧✰

✻✛✳ ✢✛ ✧✥✙ ✥✚✾

✴✗✧✢✦✣✫ ✛✩ ✢✜✯✘✗✭ ✢✛ ✦✶✚✪✣✘✗ ✢★✦ ✮★✛✪✦ ✬✛✙✚✪✦✶✙✛✫✦✪✳ ✴ ✣✙ ✠✥✧✢ ✭✛✘✗✭ ✢✛ ✢✜✯ ✣✗✫ ✦✶✚✪✣✘✗ ★✛✮ ✢★✦ ✬✛✙✚✪✦✶✙✛✫✦✪ ✸✦★✣✲✦✫ ✩✛✜✛✗✦ ✫✣✢✣ ✚✛✘✗✢✰ ✴❲✪✪ ✫✛ ✢★✘✧ ✥✧✘✗✭ ✣ ✪✘✗✦✣✜ ✦✶✚✪✣✗✣✢✘✛✗ ✙✛✫✦✪✛ ✪✦✢❲✧ ✬✣✪✪ ✘✢ ✭✰

✴✗ ✣✫✫✘✢✘✛✗✳ ✢✛ ✩✥✜✢★✦✜ ✧✘✙✚✪✘✩✯ ✙✯ ✧✘✙✚✪✦ ✙✛✫✦✪✳ ✴ ✮✛✗❲✢ ✙✥✪✢✘✚✪✯ ✢★✦ ✬✛✦✹✘✬✘✦✗✢✧ ✸✯ ✢★✦ ✛✜✘✭✘✗✣✪ ✩✦✣✢✥✜✦ ✲✣✪✥✦✳ ✰ ✴✗✧✢✦✣✫✳ ✴❲✪✪✙✥✪✢✘✚✪✯ ✘✢ ✸✯ ❅ ✘✩ ✢★✦ ✩✦✣✢✥✜✦ ✘✧ ✚✜✦✧✦✗✢✳ ✣✗✫ ✝ ✘✩ ✘✢ ✘✧ ✗✛✢✰

✴✗ ✢★✦ ✬✣✧✦ ✛✩ ✚✜✦✫✘✬✢✘✗✭ ✮★✛ ✪✛✲✦✧ ✬✛✙✚✥✢✦✜ ✭✣✙✦✧✳ ✮★✣✢ ✴ ✢★✦✜✦✩✛✜✦ ✭✦✢ ✘✧ ✢★✦ ✩✛✪✪✛✮✘✗✭✾

✮★✦✜✦ ✳ ✢★✦ ✛✜✘✭✘✗✣✪ ✚✜✦✫✘✬✢✘✛✗ ✛✩ ✢★✦✙✛✫✦✪ ✩✛✜ ❳✜✣✗✵✰

✡✛✢✦ ✢★✣✢ ✢★✦ ✬✛✦✹✘✬✘✦✗✢✧ ✣✚✚✪✯ ✛✗✪✯ ✢✛ ❳✜✣✗✵✛ ✘✩ ✴ ✮✣✗✢ ✢✛ ✩✘✗✫ ★✛✮ ✢★✦✙✛✫✦✪ ✸✦★✣✲✦✫ ✩✛✜ ❬✛✸✸✯✳ ✴❲✪✪ ✗✦✦✫ ✢✛ ✩✘✗✫ ✣ ✗✦✮ ✧✦✢ ✛✩✬✛✦✹✘✬✘✦✗✢✧✰ ✴✗ ✣✫✫✘✢✘✛✗✳ ✧✘✗✬✦ ❬✛✸✸✯ ✫✛✦✧✗❲✢ ★✣✲✦ ✣ ✠✛✸✳ ✴ ✙✥✪✢✘✚✪✘✦✫ ✸✯ ✝ ✽✧✘✗✬✦ ✢★✦✜✦ ✘✧✗❲✢ ✣✗ ❀✰ ✺✘✧ ✧✘✙✚✪✦✙✛✫✦✪ ✮✘✪✪ ✢★✦✜✦✩✛✜✦ ✸✦

✴❲✪✪ ✫✛ ✢★✘✧ ✩✛✜ ✣✪✪ ✢★✦ ✫✣✢✣ ✚✛✘✗✢✧ ✣✗✫ ✣✭✭✜✦✭✣✢✦ ✘✢ ✢✛ ✭✦✢ ✣✗ ✘✫✦✣ ✛✩ ★✛✮✙✯✙✛✫✦✪ ✮✛✜✵✦✫ ✭✪✛✸✣✪✪✯✰

✡✛✮ ✢★✣✢ ✴ ★✣✲✦ ✢★✘✧ ✩✜✣✙✦✮✛✜✵ ✮✘✢★✘✗ ✮★✘✬★ ✢✛ ✘✗✢✦✜✚✜✦✢ ✬✛✙✚✪✦✶✙✛✫✦✪✧✳ ✴ ✗✦✦✫ ✢✛ ✢★✘✗✵ ✣✸✛✥✢ ✦✶✣✬✢✪✯ ✮★✣✢ ✚✜✛✚✦✜✢✘✦✧ ✴✮✣✗✢ ✢✛ ✬✣✚✢✥✜✦ ✢✛ ✸✦ ✥✧✦✩✥✪✰

☛❉❊❖P■❙ ☞❊P✌■❏

✍❴✎✏ ✑❴✒ ✄❛✄ ✓ ✄❛❫✄✧❫❛✝✦ ✔✕

✱★✦ ✧✛✪✥✢✘✛✗ ✢✛ ✩✘✗✫✘✗✭ ✢★✦ ✲✣✪✥✦✧ ✛✩ ✚✜✦✫✣✢✦✧✙✣✬★✘✗✦ ✪✦✣✜✗✘✗✭✰ ✴✗ ✩✣✬✢✳ ✘✢ ★✣✧ ✘✢✧ ✩✛✥✗✫✣✢✘✛✗✧ ✘✗ ✭✣✙✦ ✢★✦✛✜✯✰

❁✛✗✧✘✫✦✜ ✢★✦ ✩✛✪✪✛✮✘✗✭ ✧✬✦✗✣✜✘✛✾ ✣ ✭✜✛✥✚ ✛✩ ✚✦✛✚✪✦ ✣✜✦ ✚✪✣✯✘✗✭ ✣ ✭✣✙✦✰ ✖✧ ✣ ✜✦✧✥✪✢ ✛✩ ✚✪✣✯✘✗✭ ✢★✘✧ ✭✣✙✦✳ ✢★✦✯ ✜✦✬✦✘✲✦ ✣ ✬✦✜✢✣✘✗✜✦✮✣✜✫✛ ★✛✮ ✬✣✗ ✢★✦✯ ✫✘✲✘✫✦ ✢★✘✧ ✜✦✮✣✜✫ ✸✦✢✮✦✦✗ ✢★✦✙✧✦✪✲✦✧ ✘✗ ✣ ✮✣✯ ✮★✘✬★ ✜✦✩✪✦✬✢✧ ✦✣✬★ ✛✩ ✢★✦✘✜ ✬✛✗✢✜✘✸✥✢✘✛✗✧❃

✱★✦✜✦ ✣✜✦ ✣ ✩✦✮ ✢★✘✗✭✧ ✮★✘✬★ ✦✲✦✜✯✛✗✦ ✬✣✗ ✣✭✜✦✦ ✛✗✛✙✦✦✢✘✗✭ ✢★✦ ✩✛✪✪✛✮✘✗✭ ✬✛✗✫✘✢✘✛✗✧ ✮✘✪✪ ✙✦✣✗ ✢★✦ ✭✣✙✦ ✘✧ ❜✩✣✘✜❲ ✣✬✬✛✜✫✘✗✭✢✛ ✻★✣✚✪✦✯ ✲✣✪✥✦✧✾

✱★✦ ✧✥✙ ✛✩ ✮★✣✢ ✦✲✦✜✯✛✗✦ ✜✦✬✦✘✲✦✧ ✧★✛✥✪✫ ✦✤✥✣✪ ✢★✦ ✢✛✢✣✪ ✜✦✮✣✜✫❅✰✴✩ ✢✮✛ ✚✦✛✚✪✦ ✬✛✗✢✜✘✸✥✢✦✫ ✢★✦ ✧✣✙✦ ✲✣✪✥✦✳ ✢★✦✗ ✢★✦✯ ✧★✛✥✪✫ ✜✦✬✦✘✲✦ ✢★✦ ✧✣✙✦ ✣✙✛✥✗✢ ✩✜✛✙ ✢★✦ ✜✦✮✣✜✫❆✰✻✛✙✦✛✗✦✮★✛ ✬✛✗✢✜✘✸✥✢✦✫ ✗✛ ✲✣✪✥✦ ✧★✛✥✪✫ ✜✦✬✦✘✲✦ ✗✛✢★✘✗✭❇✰✴✩ ✢★✦ ✭✜✛✥✚ ✚✪✣✯✧ ✢✮✛ ✭✣✙✦✧✳ ✢★✦✗ ✣✗ ✘✗✫✘✲✘✫✥✣✪❲✧ ✜✦✮✣✜✫ ✩✜✛✙ ✸✛✢★ ✭✣✙✦✧ ✧★✛✥✪✫ ✦✤✥✣✪ ✢★✦✘✜ ✜✦✮✣✜✫ ✩✜✛✙ ✢★✦✘✜ ✩✘✜✧✢✭✣✙✦ ✚✪✥✧ ✢★✦✘✜ ✜✦✮✣✜✫ ✩✜✛✙ ✢★✦ ✧✦✬✛✗✫ ✭✣✙✦

✥✰

✱★✦✧✦ ✣✜✦ ✩✣✘✜✪✯ ✘✗✢✥✘✢✘✲✦ ✜✥✪✦✧ ✢✛ ★✣✲✦ ✮★✦✗ ✫✘✲✘✫✘✗✭ ✣ ✜✦✮✣✜✫✳ ✣✗✫ ✢★✦✯ ✢✜✣✗✧✪✣✢✦ ✗✘✬✦✪✯ ✢✛ ✢★✦✙✣✬★✘✗✦ ✪✦✣✜✗✘✗✭ ✚✜✛✸✪✦✙✮✦✣✜✦ ✢✜✯✘✗✭ ✢✛ ✧✛✪✲✦✰ ✴✗ ✣ ✙✣✬★✘✗✦ ✪✦✣✜✗✘✗✭ ✚✜✛✸✪✦✙✳ ✢★✦ ✜✦✮✣✜✫ ✘✧ ✢★✦ ✩✘✗✣✪ ✚✜✦✫✘✬✢✘✛✗ ✛✩ ✢★✦ ✬✛✙✚✪✦✶✙✛✫✦✪✳ ✣✗✫ ✢★✦✚✣✜✢✘✬✘✚✣✗✢✧ ✘✗ ✢★✦ ✭✣✙✦ ✣✜✦ ✩✦✣✢✥✜✦✧✰ ✱✜✣✗✧✪✣✢✘✗✭ ✢★✦✧✦ ✜✥✪✦✧ ✘✗✢✛ ✛✥✜ ✚✜✦✲✘✛✥✧ ✗✛✢✣✢✘✛✗✾

✧★✛✥✪✫ ✸✦ ✦✤✥✣✪ ✢✛ ✳ ✢★✦ ✚✜✛✸✣✸✘✪✘✢✯ ✢★✦ ✬✛✙✚✪✦✶ ✙✛✫✦✪ ✣✧✧✘✭✗✦✫ ✢✛ ❳✜✣✗✵ ✛✩ ✪✘✵✘✗✭ ✬✛✙✚✥✢✦✜ ✭✣✙✦✧❅✰✴✩ ✢✮✛ ✩✦✣✢✥✜✦✧ ✬✛✗✢✜✘✸✥✢✦✫ ✢★✦ ✧✣✙✦ ✲✣✪✥✦ ✢✛ ✢★✦ ✩✘✗✣✪ ✚✜✦✫✘✬✢✘✛✗✳ ✢★✦✗ ✢★✦✘✜ ✬✛✦✹✘✬✘✦✗✢✧ ✧★✛✥✪✫ ★✣✲✦ ✢★✦ ✧✣✙✦ ✲✣✪✥✦❆✰✴✩ ✣ ✩✦✣✢✥✜✦ ✬✛✗✢✜✘✸✥✢✦✫ ✗✛✢★✘✗✭ ✢✛ ✢★✦ ✩✘✗✣✪ ✚✜✦✫✘✬✢✘✛✗ ✽✛✜ ✘✩ ✘✢ ✘✧ ✙✘✧✧✘✗✭❀✳ ✢★✦✗ ✘✢✧ ✬✛✗✢✜✘✸✥✢✘✛✗ ✢✛ ✧★✛✥✪✫ ✸✦ ✝❇✰

⑧ ❡❢ ❣

❤✐✐❥❦❧♠♠♥♦♣❡❥qr♦s❦t♥❡✉♠✈✇❦❡①✈♥✇❦♠②♣✐✇✈❥✈✇✐②♣③④♥❡✉❥r✇⑤④✉❡⑥✇rttt ⑦♠⑧⑨♠❞⑩❶ ❞❞❧❷❸ ❹❺

Page 11: 3 NIPS Papers We Loved - Persagen Consulting1705.07874.pdf · VP Data Science at Instacart, conquering the world one at a time. Dec 14, 2017 Know your model’s limits, interpret

��✂

�✁✂

✴✩ ✴ ✣✫✫ ✥✚ ✳ ✢★✦✗ ✢★✘✧ ✧★✛✥✪✫ ✸✦ ✦✤✥✣✪ ✢✛ ✰✥✰

✴✢❲✧ ✮✛✜✢★ ✗✛✢✘✗✭ ✢★✣✢ ✧✛ ✩✣✜✳ ✛✥✜ ✧✘✙✚✪✦✙✛✫✦✪ ✸✯ ✫✦✩✣✥✪✢ ✜✦✧✚✦✬✢✧ ✜✥✪✦✧ ❇ ✣✗✫ ✥✰

✴✢ ✢✥✜✗✧ ✛✥✢ ✢★✣✢ ✢★✦✜✦ ✘✧ ✛✗✪✯ ✛✗✦✙✦✢★✛✫ ✛✩ ✬✣✪✬✥✪✣✢✘✗✭ ✧✛ ✢★✣✢ ✘✢ ✮✘✪✪ ✣✪✧✛ ✜✦✧✚✦✬✢ ✜✥✪✦✧ ❅ ✣✗✫ ❆✰ ❱✪✛✯✫ ✻★✣✚✪✦✯ ✘✗✢✜✛✫✥✬✦✫✢★✘✧ ✙✦✢★✛✫ ✘✗ ❅✂✆❇ ✽✮★✘✬★ ✘✧ ✮★✯ ✲✣✪✥✦✧ ✛✩ ✬✣✪✬✥✪✣✢✦✫ ✘✗ ✢★✘✧ ✮✣✯ ✣✜✦ ✵✗✛✮✗ ✣✧ ✻★✣✚✪✦✯ ✲✣✪✥✦✧❀✰

✱★✦ ✻★✣✚✪✦✯ ✲✣✪✥✦ ✩✛✜ ✣ ✬✦✜✢✣✘✗ ✩✦✣✢✥✜✦ ✽✛✥✢ ✛✩ ✢✛✢✣✪ ✩✦✣✢✥✜✦✧❀✳ ✭✘✲✦✗ ✣ ✚✜✦✫✘✬✢✘✛✗ ✽✢★✘✧ ✘✧ ✢★✦ ✚✜✦✫✘✬✢✘✛✗ ✸✯ ✢★✦ ✬✛✙✚✪✦✶✙✛✫✦✪❀ ✘✧

✱★✦✜✦❲✧ ✣ ✸✘✢ ✢✛ ✥✗✚✣✬✵ ★✦✜✦✳ ✸✥✢ ✢★✘✧ ✘✧ ✣✪✧✛✙✥✬★✙✛✜✦ ✘✗✢✥✘✢✘✲✦ ✢★✣✗ ✘✢ ✪✛✛✵✧✰ ✖✢ ✣ ✲✦✜✯ ★✘✭★ ✪✦✲✦✪✳ ✮★✣✢ ✢★✘✧ ✦✤✥✣✢✘✛✗ ✫✛✦✧ ✘✧✬✣✪✬✥✪✣✢✦ ✮★✣✢ ✢★✦ ✚✜✦✫✘✬✢✘✛✗ ✛✩ ✢★✦✙✛✫✦✪ ✮✛✥✪✫ ✸✦ ✮✘✢★✛✥✢ ✩✦✣✢✥✜✦ ✳ ✬✣✪✬✥✪✣✢✦ ✢★✦ ✚✜✦✫✘✬✢✘✛✗ ✛✩ ✢★✦✙✛✫✦✪ ✮✘✢★ ✩✦✣✢✥✜✦ ✳✣✗✫ ✢★✦✗ ✬✣✪✬✥✪✣✢✦ ✢★✦ ✫✘✹✦✜✦✗✬✦✾

✱★✘✧ ✘✧ ✘✗✢✥✘✢✘✲✦✛ ✴ ✬✣✗ ✠✥✧✢ ✣✫✫ ✩✦✣✢✥✜✦✧ ✣✗✫ ✧✦✦ ★✛✮ ✢★✦✙✛✫✦✪❲✧ ✚✜✦✫✘✬✢✘✛✗ ✬★✣✗✭✦✧ ✣✧ ✘✢ ✧✦✦✧ ✗✦✮ ✩✦✣✢✥✜✦✧✰ ✱★✦ ✬★✣✗✭✦ ✘✗ ✢★✦✙✛✫✦✪❲✧ ✚✜✦✫✘✬✢✘✛✗ ✘✧ ✦✧✧✦✗✢✘✣✪✪✯ ✢★✦ ✦✹✦✬✢ ✛✩ ✢★✦ ✩✦✣✢✥✜✦✰

✺✛✮✦✲✦✜✳ ✢★✦ ✛✜✫✦✜ ✘✗ ✮★✘✬★ ✯✛✥ ✣✫✫ ✩✦✣✢✥✜✦✧ ✘✧ ✘✙✚✛✜✢✣✗✢ ✢✛ ★✛✮ ✯✛✥ ✣✧✧✘✭✗ ✢★✦✘✜ ✲✣✪✥✦✧✰ ❱✦✢❲✧ ✬✛✗✧✘✫✦✜ ❬✛✸✸✯❲✧ ✦✶✣✙✚✪✦ ✢✛✥✗✫✦✜✧✢✣✗✫✮★✯✛ ✘✢❲✧ ✢★✦ ✩✣✬✢ ✢★✣✢ ★✦ ✘✧ ❵❴✝✑ ❅✥ ✣✗✫✙✣✪✦ ✢★✣✢✙✦✣✗✧ ★✦ ★✣✧ ✣ ★✘✭★ ✬★✣✗✬✦ ✛✩ ✪✘✵✘✗✭ ✬✛✙✚✥✢✦✜ ✭✣✙✦✧✰ ✱★✘✧✙✦✣✗✧ ✢★✣✢ ✮★✘✬★✦✲✦✜ ✩✦✣✢✥✜✦ ✮✦ ✣✫✫ ✧✦✬✛✗✫✮✘✪✪ ✭✦✢ ✣ ✫✘✧✚✜✛✚✛✜✢✘✛✗✣✢✦✪✯ ★✘✭★ ✮✦✘✭★✢✘✗✭✳ ✧✘✗✬✦ ✢★✦✙✛✫✦✪ ✮✘✪✪ ✧✦✦ ✢★✣✢❬✛✸✸✯ ✘✧ ✣ ✜✦✣✪✪✯ ✪✘✵✦✪✯ ✬✣✗✫✘✫✣✢✦ ✩✛✜ ✪✘✵✘✗✭ ✬✛✙✚✥✢✦✜ ✭✣✙✦✧ ✛✗✪✯ ✮★✦✗ ✘✢ ★✣✧ ✸✛✢★ ✚✘✦✬✦✧ ✛✩ ✘✗✩✛✜✙✣✢✘✛✗✰

✱✛ ✸✦✢✢✦✜ ✘✪✪✥✧✢✜✣✢✦ ✢★✘✧✳ ✪✦✢✧ ✘✙✣✭✘✗✦ ✢★✣✢ ✮✦ ✣✜✦ ✢✜✯✘✗✭ ✢✛ ✣✧✧✘✭✗ ✩✦✣✢✥✜✦ ✲✣✪✥✦✧ ✢✛ ✢★✦ ✫✦✬✘✧✘✛✗ ✢✜✦✦ ✩✜✛✙ ✢★✦ ❨❩❬✛✛✧✢✫✛✬✥✙✦✗✢✣✢✘✛✗✰ ✄✘✹✦✜✦✗✢ ✘✙✚✪✦✙✦✗✢✣✢✘✛✗✧ ✛✩ ✫✦✬✘✧✘✛✗ ✢✜✦✦✧ ★✣✲✦ ✫✘✹✦✜✦✗✢ ✮✣✯✧ ✛✩ ✫✦✣✪✘✗✭ ✮✘✢★✙✘✧✧✘✗✭ ✲✣✪✥✦✧✳ ✸✥✢ ✩✛✜ ✢★✘✧✢✛✯ ✦✶✣✙✚✪✦✳ ✪✦✢✧ ✧✣✯ ✢★✣✢ ✘✩ ✣ ✲✣✪✥✦ ✢★✦ ✢✜✦✦ ✧✚✪✘✢✧ ✛✗ ✘✧ ✙✘✧✧✘✗✭✳ ✘✢ ✬✣✪✬✥✪✣✢✦✧ ✢★✦ ✣✲✦✜✣✭✦ ✛✩ ✢★✦ ✪✦✣✲✦✧ ✸✦✪✛✮ ✘✢✰

✖✧ ✣ ✜✦✙✘✗✫✦✜✳ ★✦✜✦ ✘✧ ✢★✦ ✫✦✬✘✧✘✛✗ ✢✜✦✦ ✽✮✘✢★ ❬✛✸✸✯ ✪✣✸✦✪✪✦✫❀✾

☎✂✎★✝✏ ✒✦✆❫❫ ★✦✦ ✝❴❵❵✞✆★ ❛❪✦✏ ❛✄☎ ✝✑✦✄ ✑✂★ ❪✦✄☎✦✎✟

❸ ❡❢ ❣

❤✐✐❥❦❧♠♠♥♦♣❡❥qr♦s❦t♥❡✉♠✈✇❦❡①✈♥✇❦♠②♣✐✇✈❥✈✇✐②♣③④♥❡✉❥r✇⑤④✉❡⑥✇rttt ⑦♠⑧⑨♠❞⑩❶ ❞❞❧❷❸ ❹❺

Page 12: 3 NIPS Papers We Loved - Persagen Consulting1705.07874.pdf · VP Data Science at Instacart, conquering the world one at a time. Dec 14, 2017 Know your model’s limits, interpret

��✂

�✁✂

�✂✂

❂★✦✗ ✢★✦✙✛✫✦✪ ✧✦✦✧ ❬✛✸✸✯❲✧ ✣✭✦✳ ✘✢ ✮✘✪✪ ✢✣✵✦ ★✘✙ ✪✦✄ ✛✗ ✢★✦ ✩✘✜✧✢ ✧✚✪✘✢✰ ✱★✦✗✳ ✧✘✗✬✦ ✘✢ ✫✛✦✧✗❲✢ ★✣✲✦ ✣ ✭✦✗✫✦✜ ✯✦✢✳ ✘✢ ✮✘✪✪ ✣✧✧✘✭✗★✘✙ ✢★✦ ✣✲✦✜✣✭✦ ✛✩ ✢★✦ ✪✦✣✲✦✧ ✸✦✪✛✮✳ ✛✜ ✰ ✻✛ ✢★✦ ✦✹✦✬✢ ✛✩ ✢★✦ ✣✭✦ ✩✦✣✢✥✜✦ ✘✧ ❅✰✝✆✰

✱★✦✗✳ ✮★✦✗ ✢★✦✙✛✫✦✪ ✪✦✣✜✗✧ ★✦ ✘✧ ✙✣✪✦✳ ✘✢ ✮✘✪✪ ✭✘✲✦ ★✘✙ ✣ ✧✬✛✜✦ ✛✩ ❆✰ ✱★✦ ✦✹✦✬✢ ✛✩ ✢★✦ ✭✦✗✫✦✜ ✩✦✣✢✥✜✦ ✘✧ ✢★✦✜✦✩✛✜✦✰

✻✛ ✘✗ ✢★✘✧ ✧✬✦✗✣✜✘✛✳ ✳ ✣✗✫ ✰

☎✦✆✝✏ ❫✦✝★ ★❛✞✒✦ ★✦✦ ✑✂★ ❪✦✄☎✦✎✏ ❛✄☎ ✝✑✦✄ ✑✂★ ❛❪✦✟

✴✗ ✢★✦ ✬✣✧✦ ✮★✦✜✦ ✮✦ ✛✗✪✯ ★✣✲✦ ✣ ✭✦✗✫✦✜✳ ✢★✦✙✛✫✦✪ ✫✛✦✧✗❲✢ ★✣✲✦ ✣✗ ✣✭✦ ✢✛ ✧✚✪✘✢ ✛✗✰ ✴✢ ✢★✦✜✦✩✛✜✦ ★✣✧ ✢✛ ✢✣✵✦ ✣✗ ✣✲✦✜✣✭✦ ✛✩ ✣✪✪✢★✦ ✪✦✣✲✦✧ ✸✦✪✛✮ ✢★✦ ✜✛✛✢✰

❳✘✜✧✢✳ ✢★✦ ✣✲✦✜✣✭✦ ✛✩ ✢★✦ ✫✦✚✢★ ❆ ✪✦✣✲✦✧✾ ✰ ✱★✘✧ ✜✦✧✥✪✢ ✘✧ ✢★✦✗ ✣✲✦✜✣✭✦✫ ✮✘✢★ ✢★✦ ✛✢★✦✜ ✫✦✚✢★ ❅ ✪✦✣✩✾ ✰ ✻✛✳✢★✦ ✦✹✦✬✢ ✛✩ ✢★✦ ✭✦✗✫✦✜ ✩✦✣✢✥✜✦ ✘✧ ✝✰✝❆✆✰

✱★✦✗✳ ✮★✦✗ ✢★✦✙✛✫✦✪ ✪✦✣✜✗✧ ★✦ ✘✧ ❅✥✳ ✘✢ ✭✘✲✦✧ ★✘✙ ✣ ✧✬✛✜✦ ✛✩ ❆✰ ✱★✦ ✦✹✦✬✢ ✛✩ ✢★✦ ✣✭✦ ✩✦✣✢✥✜✦ ✘✧ ✢★✦✗ ✰

✻✛ ✘✗ ✢★✘✧ ✧✬✦✗✣✜✘✛✳ ✳ ✣✗✫ ✰

❂★✘✬★ ✲✣✪✥✦ ✧★✛✥✪✫ ✮✦ ✣✧✧✘✭✗ ❃ ✴✩ ✮✦ ✣✧✧✘✭✗ ✣ ✲✣✪✥✦ ✛✩ ❅✰✂✝✆✳ ✫✛✦✧ ✢★✘✧ ✙✦✣✗ ✮✦ ✣✧✧✘✭✗ ✣ ✲✣✪✥✦ ✛✩✝✰✝❆✆ ✽✧✘✗✬✦✳ ✸✯ ✜✥✪✦ ❅ ✛✩ ✻★✣✚✪✦✯ ✩✣✘✜✗✦✧✧✳ ✢★✦ ✢✛✢✣✪ ✬✛✦✹✘✬✘✦✗✢✧ ✙✥✧✢ ✦✤✥✣✪ ✢★✦ ✩✘✗✣✪ ✚✜✦✫✘✬✢✘✛✗ ✛✩ ✢★✦✙✛✫✦✪ ✩✛✜ ❬✛✸✸✯✳ ✘✗ ✢★✘✧✬✣✧✦ ❆❀❃

✱★✘✧ ✘✧ ✩✣✜ ✩✜✛✙ ✘✫✦✣✪✳ ✧✘✗✬✦ ✘✢ ✘✭✗✛✜✦✧ ✢★✦ ✩✘✜✧✢ ✧✦✤✥✦✗✬✦✳ ✘✗ ✮★✘✬★ ✮✛✥✪✫ ✭✦✢ ✝✰✂✆ ✣✗✫ ✮✛✥✪✫ ✭✦✢ ❅✰✝✆✰

❂★✣✢ ✣ ✻★✣✚✪✦✯ ✲✣✪✥✦ ✫✛✦✧ ✘✧ ✬✛✗✧✘✫✦✜ ✸✛✢★ ✲✣✪✥✦✧✳ ✬✣✪✬✥✪✣✢✘✗✭ ✣ ✮✦✘✭★✢✦✫ ✧✥✙ ✢✛ ✩✘✗✫ ✢★✦ ✩✘✗✣✪ ✲✣✪✥✦✰ ✱★✘✧ ✘✧ ✮★✯ ✢★✦ ✦✤✥✣✢✘✛✗✩✛✜ ✙✥✧✢ ✚✦✜✙✥✢✦ ✛✲✦✜ ✣✪✪ ✚✛✧✧✘✸✪✦ ✧✦✢✧ ✛✩ ✩✦✣✢✥✜✦ ✭✜✛✥✚✘✗✭✧ ✽✙✘✗✥✧ ✢★✦ ✩✦✣✢✥✜✦ ✮✦ ✣✜✦ ✘✗✢✦✜✦✧✢✦✫ ✘✗❀✰ ✱★✘✧ ✘✧✫✦✧✬✜✘✸✦✫ ✘✗ ✢★✦ ✸✦✪✛✮ ✢★✦ ✧✥✙✙✣✢✘✛✗✳ ✮★✦✜✦ ✘✧ ✣✪✪ ✢★✦ ✩✦✣✢✥✜✦✧✰

✺✛✮ ✣✜✦ ✢★✦ ✮✦✘✭★✢✧ ✣✧✧✘✭✗✦✫ ✢✛ ✦✣✬★ ✬✛✙✚✛✗✦✗✢ ✛✩ ✢★✦ ✧✥✙❃ ✴✢ ✸✣✧✘✬✣✪✪✯ ✬✛✗✧✘✫✦✜✧ ★✛✮✙✣✗✯ ✫✘✹✦✜✦✗✢ ✚✦✜✙✥✢✣✢✘✛✗✧ ✛✩ ✢★✦✧✦✢✧ ✦✶✘✧✢✳ ✬✛✗✧✘✫✦✜✘✗✭ ✸✛✢★ ✢★✦ ✩✦✣✢✥✜✦✧ ✮★✘✬★ ✣✜✦ ✘✗ ✢★✦ ✧✦✢ ✽✢★✘✧ ✘✧ ✫✛✗✦ ✸✯ ✢★✦ ❀✳ ✣✧ ✮✦✪✪ ✣✧ ✢★✦ ✩✦✣✢✥✜✦✧ ✮★✘✬★ ★✣✲✦ ✯✦✢✢✛ ✸✦ ✣✫✫✦✫ ✽✢★✘✧ ✘✧ ✫✛✗✦ ✸✯ ✢★✦ ❀✰ ❳✘✗✣✪✪✯✳ ✦✲✦✜✯✢★✘✗✭ ✘✧ ✗✛✜✙✣✪✘✷✦✫ ✸✯ ✢★✦ ✩✦✣✢✥✜✦✧ ✮✦ ★✣✲✦ ✘✗ ✢✛✢✣✪✰

✞✡✖✘✟✖✡✓☞✌✍ ✡ ✠✡✡✕✖✒✗ ☛✡✖✟✒

❳✛✜ ❬✛✸✸✯✳ ✮★✣✢ ✮✛✥✪✫ ✢★✦ ✻★✣✚✪✦✯ ✲✣✪✥✦ ✸✦ ✩✛✜ ★✘✧ ✣✭✦❃

❳✘✜✧✢✳ ✴ ✗✦✦✫ ✢✛ ✬✛✗✧✢✜✥✬✢✙✯ ✧✦✢✧ ✻✰ ✱★✦✧✦ ✣✜✦ ✣✪✪ ✚✛✧✧✘✸✪✦ ✬✛✙✸✘✗✣✢✘✛✗✧ ✛✩ ❬✛✸✸✯❲✧ ✩✦✣✢✥✜✦✧✳ ✦✶✬✪✥✫✘✗✭ ★✘✧ ✣✭✦✰ ✻✘✗✬✦ ★✦ ✛✗✪✯★✣✧ ✛✗✦ ✛✢★✦✜ ✩✦✣✢✥✜✦ ❄ ★✘✧ ✭✦✗✫✦✜ ❄ ✢★✘✧ ✯✘✦✪✫✧ ✢✮✛ ✧✦✢✧✾ ☞ ✌✳ ✣✗✫ ✣✗ ✦✙✚✢✯ ✧✦✢ ☞✌✰

✡✦✶✢✳ ✴ ✗✦✦✫ ✢✛ ✬✣✪✬✥✪✣✢✦ ✩✛✜ ✦✣✬★ ✛✩ ✢★✦✧✦ ✧✦✢✧✳ ✻✰ ✡✛✢✦ ✢★✣✗ ✣✧ ✴ ★✣✲✦ ❆ ✩✦✣✢✥✜✦✧✳ ✰

✓✄ ✝✑✦ ✄❛★✦✒✑✦✎✦ ✍

✱★✦ ✚✜✦✫✘✬✢✘✛✗ ✛✩ ✢★✦✙✛✫✦✪ ✮★✦✗ ✘✢ ✧✦✦✧ ✗✛ ✩✦✣✢✥✜✦✧ ✘✧ ✢★✦ ✣✲✦✜✣✭✦ ✛✩ ✣✪✪ ✢★✦ ✪✦✣✲✦✧✳ ✮★✘✬★ ✮✦ ★✣✲✦ ✬✣✪✬✥✪✣✢✦✫ ✢✛ ✸✦ ✝✰✝❆✆✰❂✦❲✲✦ ✣✪✧✛ ✬✣✪✬✥✪✣✢✦✫ ✢★✣✢ ✮★✦✗ ✘✢ ✧✦✦✧ ✛✗✪✯ ✢★✦ ✣✭✦✳ ✘✢ ✘✧ ❅✰✝✆✳ ✧✛

✱★✘✧ ✯✘✦✪✫✧

❷ ❡❢ ❣

❤✐✐❥❦❧♠♠♥♦♣❡❥qr♦s❦t♥❡✉♠✈✇❦❡①✈♥✇❦♠②♣✐✇✈❥✈✇✐②♣③④♥❡✉❥r✇⑤④✉❡⑥✇rttt ⑦♠⑧⑨♠❞⑩❶ ❞❞❧❷❸ ❹❺

Page 13: 3 NIPS Papers We Loved - Persagen Consulting1705.07874.pdf · VP Data Science at Instacart, conquering the world one at a time. Dec 14, 2017 Know your model’s limits, interpret

��✂

�✁✁✂

�✁✁✂

�✁�✂

✓✄ ✝✑✦ ✄❛★✦✒✑✦✎✦ ✍

❂✦❲✲✦ ✬✣✪✬✥✪✣✢✦✫ ✢★✣✢ ✢★✦ ✚✜✦✫✘✬✢✘✛✗ ✛✩ ✢★✦✙✛✫✦✪ ✮✘✢★ ✛✗✪✯ ✢★✦ ✭✦✗✫✦✜ ✘✧ ✝✰✝❆✆✳ ✣✗✫ ✢★✦✗ ✮★✦✗ ✘✢ ✧✦✦✧ ✸✛✢★ ★✘✧ ✣✭✦ ✣✗✫ ★✘✧✭✦✗✫✦✜ ✘✧ ❆✳ ✧✛

✻✛

✖✫✫✘✗✭ ✢★✦✧✦ ✢✮✛ ✲✣✪✥✦✧ ✢✛✭✦✢★✦✜ ✯✘✦✪✫✧

✡✛✢✦ ✢★✣✢ ✢★✘✧ ✲✣✪✥✦✙✣✵✦✧ ✧✦✗✧✦✛ ✘✢✧ ✜✘✭★✢ ✘✗ ✢★✦✙✘✫✫✪✦ ✛✩ ✮★✣✢ ✮✦ ✬✣✪✬✥✪✣✢✦✫ ✮★✦✗ ✮✦ ✬✣✪✬✥✪✣✢✦✫ ✩✦✣✢✥✜✦ ✘✙✚✛✜✢✣✗✬✦ ✠✥✧✢ ✸✯✣✫✫✘✗✭ ✩✦✣✢✥✜✦✧ ✛✗✦ ✸✯ ✛✗✦✰

✴✗ ✧✥✙✙✣✜✯✳ ✻★✣✚✪✦✯ ✲✣✪✥✦✧ ✬✣✪✬✥✪✣✢✦ ✢★✦ ✘✙✚✛✜✢✣✗✬✦ ✛✩ ✣ ✩✦✣✢✥✜✦ ✸✯ ✬✛✙✚✣✜✘✗✭ ✮★✣✢ ✣ ✙✛✫✦✪ ✚✜✦✫✘✬✢✧ ✮✘✢★ ✣✗✫ ✮✘✢★✛✥✢ ✢★✦✩✦✣✢✥✜✦✰ ✺✛✮✦✲✦✜✳ ✧✘✗✬✦ ✢★✦ ✛✜✫✦✜ ✘✗ ✮★✘✬★ ✣ ✙✛✫✦✪ ✧✦✦✧ ✩✦✣✢✥✜✦✧ ✬✣✗ ✣✹✦✬✢ ✘✢✧ ✚✜✦✫✘✬✢✘✛✗✧✳ ✢★✘✧ ✘✧ ✫✛✗✦ ✘✗ ✦✲✦✜✯ ✚✛✧✧✘✸✪✦ ✛✜✫✦✜✳✧✛ ✢★✣✢ ✢★✦ ✩✦✣✢✥✜✦✧ ✣✜✦ ✩✣✘✜✪✯ ✬✛✙✚✣✜✦✫✰

☛✂✄☎ ☞❊P✌■❏

✆✗✩✛✜✢✥✗✣✢✦✪✯✳ ✭✛✘✗✭ ✢★✜✛✥✭★ ✣✪✪ ✚✛✧✧✘✸✪✦ ✬✛✙✸✘✗✣✢✘✛✗✧ ✛✩ ✩✦✣✢✥✜✦✧ ✤✥✘✬✵✪✯ ✸✦✬✛✙✦✧ ✬✛✙✚✥✢✣✢✘✛✗✣✪✪✯ ✥✗✩✦✣✧✘✸✪✦✰

❱✥✬✵✘✪✯✳ ✢★✦ ✻✺✖✼ ✪✘✸✜✣✜✯ ✘✗✢✜✛✫✥✬✦✧ ✛✚✢✘✙✘✷✣✢✘✛✗✧ ✮★✘✬★ ✣✪✪✛✮ ✻★✣✚✪✦✯ ✲✣✪✥✦✧ ✢✛ ✸✦ ✥✧✦✫ ✘✗ ✚✜✣✬✢✘✬✦✰ ✴✢ ✫✛✦✧ ✢★✘✧ ✸✯✫✦✲✦✪✛✚✘✗✭ ✙✛✫✦✪ ✧✚✦✬✘✩✘✬ ✣✪✭✛✜✘✢★✙✧✳ ✮★✘✬★ ✢✣✵✦ ✣✫✲✣✗✢✣✭✦ ✛✩ ✫✘✹✦✜✦✗✢ ✙✛✫✦✪❲✧ ✧✢✜✥✬✢✥✜✦✧✰ ❳✛✜ ✘✗✧✢✣✗✬✦✳ ✻✺✖✼❲✧ ✘✗✢✦✭✜✣✢✘✛✗✮✘✢★ ✭✜✣✫✘✦✗✢ ✸✛✛✧✢✦✫ ✫✦✬✘✧✘✛✗ ✢✜✦✦✧ ✢✣✵✦✧ ✣✫✲✣✗✢✣✭✦ ✛✩ ✢★✦ ★✘✦✜✣✜✬★✯ ✘✗ ✣ ✫✦✬✘✧✘✛✗ ✢✜✦✦❲✧ ✩✦✣✢✥✜✦✧ ✢✛ ✬✣✪✬✥✪✣✢✦ ✢★✦ ✻✺✖✼✲✣✪✥✦✧✰

✱★✘✧ ✣✪✪✛✮✧ ✢★✦ ✻✺✖✼ ✪✘✸✜✣✜✯ ✢✛ ✬✣✪✬✥✪✣✢✦ ✻★✣✚✪✦✯ ✲✣✪✥✦✧ ✧✘✭✗✘✩✘✬✣✗✢✪✯ ✩✣✧✢✦✜ ✢★✣✗ ✘✩ ✣ ✙✛✫✦✪ ✚✜✦✫✘✬✢✘✛✗ ★✣✫ ✢✛ ✸✦ ✬✣✪✬✥✪✣✢✦✫ ✩✛✜✦✲✦✜✯ ✚✛✧✧✘✸✪✦ ✬✛✙✸✘✗✣✢✘✛✗ ✛✩ ✩✦✣✢✥✜✦✧✰

✝❍▼✞P✌❏❑❍▼

✻★✣✚✪✦✯ ✲✣✪✥✦✧✳ ✣✗✫ ✢★✦ ✻✺✖✼ ✪✘✸✜✣✜✯✳ ✣✜✦ ✚✛✮✦✜✩✥✪ ✢✛✛✪✧ ✢✛ ✥✗✬✛✲✦✜✘✗✭ ✢★✦ ✚✣✢✢✦✜✗✧ ✣ ✙✣✬★✘✗✦ ✪✦✣✜✗✘✗✭ ✣✪✭✛✜✘✢★✙ ★✣✧✘✫✦✗✢✘✩✘✦✫✰

✴✗ ✚✣✜✢✘✬✥✪✣✜✳ ✸✯ ✬✛✗✧✘✫✦✜✘✗✭ ✢★✦ ✦✹✦✬✢✧ ✛✩ ✩✦✣✢✥✜✦✧ ✘✗ ✘✗✫✘✲✘✫✥✣✪ ✫✣✢✣✚✛✘✗✢✧✳ ✘✗✧✢✦✣✫ ✛✩ ✛✗ ✢★✦ ✮★✛✪✦ ✫✣✢✣✧✦✢ ✽✣✗✫ ✢★✦✗✣✭✭✜✦✭✣✢✘✗✭ ✢★✦ ✜✦✧✥✪✢✧❀✳ ✢★✦ ✘✗✢✦✜✚✪✣✯ ✛✩ ✬✛✙✸✘✗✣✢✘✛✗✧ ✛✩ ✩✦✣✢✥✜✦✧ ✬✣✗ ✸✦ ✥✗✬✛✲✦✜✦✫✰ ✱★✘✧ ✣✪✪✛✮✧ ✩✣✜ ✙✛✜✦ ✚✛✮✦✜✩✥✪ ✘✗✧✘✭★✢✧ ✢✛✸✦ ✭✦✗✦✜✣✢✦✫ ✢★✣✗ ✮✘✢★ ✭✪✛✸✣✪ ✩✦✣✢✥✜✦ ✘✙✚✛✜✢✣✗✬✦✙✦✢★✛✫✧✰

✠✎✟✠✘✒✣

✻✰ ❱✥✗✫✸✦✜✭✳ ✻ ❱✦✦✳ ✖ ✆✗✘✩✘✦✫ ✖✚✚✜✛✣✬★ ✢✛ ✴✗✢✦✜✚✜✦✢✘✗✭☎✛✫✦✪ ✼✜✦✫✘✬✢✘✛✗✧✳ ❆✝❅✝

❣ ❡❢ ❣

❤✐✐❥❦❧♠♠♥♦♣❡❥qr♦s❦t♥❡✉♠✈✇❦❡①✈♥✇❦♠②♣✐✇✈❥✈✇✐②♣③④♥❡✉❥r✇⑤④✉❡⑥✇rttt ⑦♠⑧⑨♠❞⑩❶ ❞❞❧❷❸ ❹❺