your path to data-driven quality
DESCRIPTION
Your Path To Data-Driven Quality. Seth Eliot Principal Knowledge Engineer, Test Excellence. About Seth. Digital Media Services. A/B Testing of Services. Petabytes Processed. Services and Cloud and Data-Driven Quality and. Roadmap. …how to create your “roadmap” to DDQ. At destination. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/1.jpg)
Sponsored by
![Page 2: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/2.jpg)
Your Path To Data-Driven Quality
Seth EliotPrincipal Knowledge Engineer, Test Excellence
![Page 3: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/3.jpg)
Services andCloud andData-Driven Quality and...
A/B Testing of Services
Digital Media Services
Petabytes Processed
About Seth
![Page 4: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/4.jpg)
Roadmap
![Page 5: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/5.jpg)
In This Session, You’ll Learn……how to create your “roadmap” to DDQRoadmap Road, Car, Gas At destination
Roadmap DDQ StrategyYou and your...
Application (service or product)EnvironmentEngineering Processes
![Page 6: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/6.jpg)
What is it? Why is it important?
Data-Driven Quality
![Page 7: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/7.jpg)
Data, where have we been?The HiPPO
Highest Paid Person’s Opinion
Engineering DataTest pass/fail resultsBug countsDelivery cadenceCode coverageCode Churn
![Page 8: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/8.jpg)
Data, where are we going?Engineering dataTest results Scoring engines using Bayesian
analysisProduction-quality dataYou may have heard of TiPNot as difficult as you might
thinkLots of solutions for lots of application
types
Real Users
Production Environments
![Page 9: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/9.jpg)
Determine your questionsDesign for production-quality dataSelect your data sourcesUse the right data toolsGet answers to your questionsLearn new questionsRepeat
The roadmap
![Page 10: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/10.jpg)
AvailabilityPerformance
Usage? GQM - 1994
Start at the beginning
“Big Data” is hot, let’s start with that“Big Data” is hot, let’s start with that
![Page 11: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/11.jpg)
Prioritize answers you seekDetermine your questions
![Page 12: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/12.jpg)
Why not just get data and look for answers?
Do night-lights cause near-sightedness in children?
Quinn, et al, 1999
Does sunscreen increase chance of drowning?
![Page 13: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/13.jpg)
What questions does EXO ask? Monthly
Uptime Percentage
Service Credit
< 99.9% 25% < 99% 50%< 95% 100%
“Downtime” is defined as any period of time when users are unable to send or receive email via all supported mailbox access
![Page 14: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/14.jpg)
Is it available?Often a Pri 0
Critical work stoppage for user“Dialtone”
Is the application (product, service) there for the user?
![Page 15: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/15.jpg)
Application works, but feature does notOccasionally does not render/load properly
違った言語
How do users perceive availability?
![Page 16: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/16.jpg)
Huge ImpactHow is performance?
Power of Production Data
Real usersMultiple environments
End to endScale & geo-diversity
![Page 17: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/17.jpg)
What do users do?
Customer Experience Improvement Program (CEIP)
![Page 18: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/18.jpg)
Your turn - Determine your questions
What type of answer are you looking for?Availability, performance, usage?Prioritize
When does availability NOT come first?
![Page 19: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/19.jpg)
Your turn - Determine your questions
What are the key scenarios for the type you selected?
What is high priority?
SharePoint
![Page 20: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/20.jpg)
Get data from or near production
Design for production-quality data
![Page 21: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/21.jpg)
Two types of data to acquireActive: syntheticPassive: real (RUM)
Active data in prodFor services only?Client: is the service
there?
![Page 22: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/22.jpg)
Staged data acquisition mitigates riskService
Product (client, on-prem server)Stage 1: Partial
or whole product team
Stage 2: Dogfood
Stage 3: Technology Adoption
Programs (TAP)
Stage 4: Some clients in
productionStage 5: All Customers
Stage 1: In prod, no users
Stage 2: Dogfood
Stage 3: Some servers in prod
Stage 4: Some more servers in
prod
Stage 5 : World-wide
prod
Deployment Validation
Service Validation Scale Validation
Real-time service quality
![Page 23: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/23.jpg)
Staged data acquisition - Netflix
1B API requests per day
Canary Deployment
![Page 24: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/24.jpg)
Staged Data Acquisition - Facebook
DogfoodIn prod, no users (except internal ones)Some servers in ProductionWorld-wide deploymentFeature light-up
![Page 25: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/25.jpg)
Speed of deployment
Deploy
Detect
Usually easy for servicesClient apps may have a deployment capabilityOr may make use of feature light-up
EaaSy - Everything is now connected and thus updatable
![Page 26: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/26.jpg)
Staged data acquisition - OutlookFiltering and aggregation at clientBe kind to the clientPipeline to collect and process dataMake it easyStaged Data AcquisitionFeature
CrewOutlook
TeamMS Office
Team Microsoft Customers
Scale Validation
![Page 27: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/27.jpg)
![Page 28: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/28.jpg)
Your turn - Design for production-quality data
What might be your stages for risk mitigated data acquisition?Role of active and passive monitoring?How can you engineer for EaaSy deployment?
![Page 29: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/29.jpg)
Determine the data necessary to answer your questions
Select your data sources
![Page 30: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/30.jpg)
CPU Memory
Storage Network
Infrastructure data
![Page 31: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/31.jpg)
Application data
New and Open
Download Template
Failure Rate
Average Time
![Page 32: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/32.jpg)
Hang and crash dataSpecialized application dataMost frequently encountered conditionsBucket the Big Data and find the offendersGet offending code and function calls (stack)But it worked in DogfoodCulprit was a old version add-inHarden against this
![Page 33: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/33.jpg)
Usage dataClient-side instrumentationProprietaryJavascript: clicks, hovers [web apps]Get 1x1 GIF: Page Views [web apps]
Combine into more complex scenariosHow did user get to shopping cart checkout?
100%Users
50%Users
50%Users
Control:Existing System
Treatment:Existing System with Feature X
Users interactions instrumented, analyzed & compared
Analyze at the end of the experiment
![Page 34: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/34.jpg)
Feedback data
![Page 35: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/35.jpg)
Feedback data
![Page 36: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/36.jpg)
Feedback data also includes…
Xbox Kinect
Social Media Mining
Customer Support Data
![Page 37: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/37.jpg)
Active monitoring dataData Center X
Data Center YAM2
Service
AM3AM1
Dependency Service(s)
Client
![Page 38: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/38.jpg)
Ease of detection
Deploy
Detect
EaaSy – Rich near real-time telemetry
Again services have it easyMany clients are always connectedOn prem servers (enterprise) require partnerships
![Page 39: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/39.jpg)
Data handling - privacyTransparency and ControlCollection and RetentionDepends on Type anonymous Datapseudonymouspersonally identifiable info (PII)sensitive PII
Depends on Purposeprovide the serviceimprove the current serviceimprove a future version serviceimprove non-associated servicescontent personalizationad targeting
![Page 40: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/40.jpg)
Data handling – non-service productsClient and on-prem server considerationsUser owned resources: bandwidthbatterydisk, cpu, etc…
Correlationsend-to-end across clients and servicesby user , by session
![Page 41: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/41.jpg)
Your turn - Select your data sources
Infrastructure dataApplication dataHang and crash dataUsage dataFeedback dataActive monitoring dataDesign to handle your data
![Page 42: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/42.jpg)
A 50,000 foot viewUse the right data tools
![Page 43: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/43.jpg)
Data storage and processing systemsDatabase Table storage
(SQL)Optimized for CRUD – (create, read, update, delete) of single records
Data Warehouse
Table storage (SQL)
Table structure optimized for queries and bulk insert
OLAP Cubes(Online Analytical Processing)
Multidimensional Cube (MDX)
Aggregations of measures for multiple dimensions
Hadoop / Map-Reduce
Distributed File System (HDFS)
Big Data TB-PB-EBUnstructured Data
![Page 44: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/44.jpg)
Hadoop in 60 Seconds
Hadoop HDFS
ADDFCAECC
ADD FCA
ECCADD
ADDADD
FCAFCA
FCA
ECCECC
ECC
Map-Reduce1xA1xC1xF
1xA2xD
1xE2xC
2xA0xB3xC2xD1xE1xF
![Page 45: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/45.jpg)
A common data flowPower BI in
ExcelCubeTelemetr
ySystem
SQL DW
Programmatic accessOther
Visualizations
Real-timeMonitoring
![Page 46: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/46.jpg)
Web Apps integrated with OneDrive, FB, web mail, etc
Real-time monitoring
![Page 47: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/47.jpg)
TwitterXbox Kinect
Feedback
![Page 48: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/48.jpg)
Your turn - Use the right data processing tools
DB, DW, Cube, Big-Data platformsPut it all together
Do you need real-time monitoring?…sentiment analysis?
![Page 49: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/49.jpg)
and learn new questions Get answers to your questions
![Page 50: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/50.jpg)
Outlook.com prioritizes performance
500 Million measurements per month
JSIJavaScript Instrumentation
View Inbox – Page Load Time (PLT) by BrowserAs experienced by actual users
![Page 51: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/51.jpg)
Prioritizes availability
Predict 75% of dips 24 hours ahead of time
Avai
labi
lity
Time
![Page 52: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/52.jpg)
52
Netflix prioritizes perceived availability & performance
![Page 53: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/53.jpg)
Yammer prioritizes usage
What happens to new user retention when you shorten the signup flow?
It goes down!
Don’t ship that feature
![Page 54: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/54.jpg)
Find new questions and repeat
Xbox recco
Try algoCollect dataAdjust algoCollect datarepeat
![Page 55: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/55.jpg)
Find new questions and repeat
In Visual Studio 2012 we added asynchronous loading for solutions
We also added Telemetry
Results look good
![Page 56: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/56.jpg)
Your turn – Get Answers, more questions
Where are the answers to your quality assessment questions?
![Page 57: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/57.jpg)
SummaryDetermine your questionsDesign for production-quality dataSelect your data sourcesUse the right data toolsGet answers to your questionsLearn new questionsRepeatThe most recent version of this deck can be found at http://setheliot.com
![Page 58: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/58.jpg)
Special thanks to these folksRavi VedulaAndrea JesseBill HodgheadDanny ThayerJoseph SefairKitty ThomasAmanda ReinkeHeather Lader
David BrooksMike TholfsenJodie DraperBrian MuellerTara RothDror CohenNathan HalsteadLori Oviatt Thanks!
Monica CatundaLynette SkinnerJoe SchumacherDonny LuuJohn HoeggerAlain Anyouzoa
Any questions?
![Page 59: Your Path To Data-Driven Quality](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816935550346895de09084/html5/thumbnails/59.jpg)
Thank you for joining us!