off the beaten path - usenix · off the beaten path author: mohit suley created date: 11/1/2019...
TRANSCRIPT
![Page 1: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/1.jpg)
Off The Beaten PathObservability Focus: From Service, To Customer
Mohit Suley (he/him)Bing Engineering
@mohitsuleyhttps://www.linkedin.com/in/mohitsuley/
![Page 2: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/2.jpg)
You’re Probably Not Alone
https://medium.com/@copyconstruct/monitoring-and-observability-8417d1952e1c Recommended Reading
![Page 3: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/3.jpg)
Observability, for the next 45 minutes, in this room
• Monitoring• Alerting
• Debugging• Distributed Tracing
• Log Aggregation
• Testing
• Insights
• And more…
Observability Monitoring
https://en.wikipedia.org/wiki/Hubble_Space_Telescope
![Page 4: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/4.jpg)
Observability Focus
OBSERVERCUSTOMER
APP STACK
OBSERVED
![Page 5: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/5.jpg)
Observability Focus – A Shift
OBSERVERCUSTOMER
APP STACK
OBSERVEDOBSERVED
![Page 7: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/7.jpg)
Hearing or Listening?
![Page 8: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/8.jpg)
App MetricsOur Comfort Zone
METRICS
![Page 9: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/9.jpg)
Availability: Less Than What You Think It Is
Total Addressable Market
Customers With Fast Internet
Customers Who Could Connect
Customers Typical metrics-based focus
Thinking Beyond 9’s: https://www.usenix.org/conference/srecon19asia/presentation/srinivasamurthy
![Page 10: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/10.jpg)
Network Error Logging
https://w3c.github.io/network-error-logging/
![Page 11: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/11.jpg)
Performance: More Than A Numbers Game
5th 25th 50th 75th 90th Percentiles
Load
Tim
e
what we measurebroken pages
slower networks
Focus on Overall Distribution, Not Just One Point
![Page 12: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/12.jpg)
UsabilityIs There A Way To Measure Behavior?
METRICSUSABILITY
![Page 13: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/13.jpg)
How do you debug usability issues?
![Page 14: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/14.jpg)
What if you could see what your user saw?
![Page 15: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/15.jpg)
And, had the data to solve the usability issue
Root cause:
Search criteria:
- Ads are present- Ads background color not default
![Page 16: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/16.jpg)
Measuring Usability @ Scale
Session Replay Behavioral Analytics
https://github.com/microsoft/clarity-js
Structured Data
![Page 17: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/17.jpg)
Customer FeedbackRead ‘The Feels’
METRICSUSABILITYFEEDBACK
![Page 18: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/18.jpg)
The Ugly
If you were satisfied with your experience I would appreciate if you could complete
feedback.
If you could give ME a 10 it would really help me out. This is my scorecard and
determines how I get paid.
![Page 19: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/19.jpg)
The Bad
How Do You Feel On a Scale of 1 to 5?
![Page 20: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/20.jpg)
The Better
Structured, Context-Aware
![Page 21: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/21.jpg)
The Best
“Email address appears to be incorrect. Looks like the name of the business has changed as well. You might want to change this.”
“I didn’t expect you could get scores this fast!”
Start - Stanford CoreNLP http://corenlp.run/
![Page 22: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/22.jpg)
A Good Feedback System Is …
• Context-Aware
• Accessible
• Proactive
• Capable of Sentiment Extraction
• Actionable
![Page 23: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/23.jpg)
Insider ProgramsExtend, Embrace … Engage
METRICSUSABILITYFEEDBACKINSIDERS
![Page 24: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/24.jpg)
People Want To Be Cool Too
Work *With* Customers, Not Just *For* Customers
![Page 25: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/25.jpg)
Going Beyond Experimentation
Engaged Customers Can Tell You What Your Numbers Can’t
https://www.bing.com/search?q=democratic+candidates+2020
![Page 26: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/26.jpg)
Isn’t That Dogfood?
• Dogfooding doesn’t mean you’re an Insider
• You might use your own product, but you’re not a customer
Dogfooder ≠ Insider
![Page 27: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/27.jpg)
Listening ToursZero Distance To Customers
METRICSUSABILITYFEEDBACKINSIDERSLISTENERS
![Page 28: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/28.jpg)
Customer, Meet Engineer
CC: http://gadgetsin.com/uploads/2011/03/limited_edition_bad_robot_collectible_figure_1.jpgCC: https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/CashRegister.svg/670px-CashRegister.svg.png
Engineers Are People Too
![Page 29: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/29.jpg)
How Does It Help Engineering?
• You realize its not all ‘bots’.
• Talk To Product Fans
• Understand Diversity of Location/People
![Page 30: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/30.jpg)
Finishing NotesWhy This Audience?
![Page 31: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/31.jpg)
The Feature You Work On Is ‘The Site’
• You Run Telemetry Platforms
• You Are The First Line of ‘Help!’
• You Know Ground Reality of Systems *and* the 30K feet view
You Are The Most Appropriate Customer Advocate
![Page 32: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/32.jpg)
Observability: The Collective Path
APPMETRICS
PRODUCTUSABILITY
SENTIMENTFEEDBACK
INSIDER PROGRAMS
LISTENING TOURS
![Page 33: Off The Beaten Path - USENIX · Off The Beaten Path Author: Mohit Suley Created Date: 11/1/2019 10:18:52 AM](https://reader033.vdocuments.us/reader033/viewer/2022060302/5f08765b7e708231d42221a3/html5/thumbnails/33.jpg)
Thank You!Deck Link
Mohit Suley (he/him)Bing Engineering
@mohitsuleyhttps://www.linkedin.com/in/mohitsuley/