sponsored by. your path to data-driven quality seth eliot principal knowledge engineer, test...
TRANSCRIPT
![Page 1: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/1.jpg)
Sponsored by
![Page 2: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/2.jpg)
Your Path To Data-Driven Quality
Seth EliotPrincipal Knowledge Engineer, Test Excellence
![Page 3: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/3.jpg)
Services andCloud andData-Driven Quality and...
A/B Testing of Services
Digital Media Services
Petabytes Processed
About Seth
![Page 4: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/4.jpg)
Roadmap
![Page 5: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/5.jpg)
In This Session, You’ll Learn……how to create your “roadmap” to DDQ
Roadmap Road, Car, Gas At destination
Roadmap DDQ StrategyYou and your...
Application (service or product)EnvironmentEngineering Processes
![Page 6: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/6.jpg)
What is it? Why is it important?
Data-Driven Quality
![Page 7: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/7.jpg)
Data, where have we been?The HiPPO
Highest Paid Person’s Opinion
Engineering DataTest pass/fail resultsBug countsDelivery cadenceCode coverageCode Churn
![Page 8: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/8.jpg)
Data, where are we going?Engineering dataTest results Scoring engines using Bayesian
analysisProduction-quality dataYou may have heard of TiPNot as difficult as you might
thinkLots of solutions for lots of application
types
Real Users
Production Environments
![Page 9: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/9.jpg)
Determine your questionsDesign for production-quality dataSelect your data sourcesUse the right data toolsGet answers to your questionsLearn new questionsRepeat
The roadmap
![Page 10: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/10.jpg)
AvailabilityPerformance
Usage? GQM - 1994
Start at the beginning
“Big Data” is hot, let’s start with that“Big Data” is hot, let’s start with that
![Page 11: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/11.jpg)
Prioritize answers you seek
Determine your questions
![Page 12: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/12.jpg)
Why not just get data and look for answers?
Do night-lights cause near-sightedness in children?
Quinn, et al, 1999
Does sunscreen increase chance of drowning?
![Page 13: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/13.jpg)
What questions does EXO ask? Monthly
Uptime Percentage
Service Credit
< 99.9% 25% < 99% 50%< 95% 100%
“Downtime” is defined as any period of time when users are unable to send or receive email via all supported mailbox access
![Page 14: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/14.jpg)
Is it available?Often a Pri 0
Critical work stoppage for user“Dialtone”
Is the application (product, service) there for the user?
![Page 15: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/15.jpg)
Application works, but feature does notOccasionally does not render/load properly
違った言語
How do users perceive availability?
![Page 16: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/16.jpg)
Huge Impact
How is performance?Power of Production Data
Real usersMultiple environments
End to endScale & geo-diversity
![Page 17: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/17.jpg)
What do users do?
Customer Experience Improvement Program (CEIP)
![Page 18: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/18.jpg)
Your turn - Determine your questions
What type of answer are you looking for?Availability, performance, usage?Prioritize
When does availability NOT come first?
![Page 19: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/19.jpg)
Your turn - Determine your questions
What are the key scenarios for the type you selected?
What is high priority?
SharePoint
![Page 20: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/20.jpg)
Get data from or near production
Design for production-quality data
![Page 21: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/21.jpg)
Two types of data to acquire
Active: syntheticPassive: real (RUM)
Active data in prodFor services only?Client: is the service
there?
![Page 22: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/22.jpg)
Staged data acquisition mitigates riskService
Product (client, on-prem server)Stage 1: Partial
or whole product team
Stage 2: Dogfood
Stage 3: Technology Adoption
Programs (TAP)
Stage 4: Some clients in
production
Stage 5: All Customers
Stage 1: In prod, no users
Stage 2: Dogfood
Stage 3: Some servers in prod
Stage 4: Some more servers in
prod
Stage 5 : World-wide
prod
Deployment Validation
Service Validation Scale Validation
Real-time service quality
![Page 23: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/23.jpg)
Staged data acquisition - Netflix
1B API requests per day
Canary Deployment
![Page 24: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/24.jpg)
Staged Data Acquisition - Facebook
DogfoodIn prod, no users (except internal ones)Some servers in ProductionWorld-wide deploymentFeature light-up
![Page 25: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/25.jpg)
Speed of deployment
Deploy
Detect
Usually easy for services
Client apps may have a deployment capability
Or may make use of feature light-up
EaaSy - Everything is now connected and thus updatable
![Page 26: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/26.jpg)
Staged data acquisition - OutlookFiltering and aggregation at clientBe kind to the client
Pipeline to collect and process dataMake it easy
Staged Data AcquisitionFeature
CrewOutlook
TeamMS Office
Team Microsoft Customers
Scale Validation
![Page 27: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/27.jpg)
![Page 28: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/28.jpg)
Your turn - Design for production-quality data
What might be your stages for risk mitigated data acquisition?
Role of active and passive monitoring?
How can you engineer for EaaSy deployment?
![Page 29: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/29.jpg)
Determine the data necessary to answer your questions
Select your data sources
![Page 30: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/30.jpg)
CPU Memory
Storage Network
Infrastructure data
![Page 31: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/31.jpg)
Application data
New and Open
Download Template
Failure Rate
Average Time
![Page 32: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/32.jpg)
Hang and crash dataSpecialized application dataMost frequently encountered conditionsBucket the Big Data and find the offendersGet offending code and function calls (stack)
But it worked in DogfoodCulprit was a old version add-inHarden against this
![Page 33: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/33.jpg)
Usage dataClient-side instrumentationProprietaryJavascript: clicks, hovers [web apps]Get 1x1 GIF: Page Views [web apps]
Combine into more complex scenariosHow did user get to shopping cart checkout?
100%Users
50%Users
50%Users
Control:Existing System
Treatment:Existing System with Feature X
Users interactions instrumented, analyzed & compared
Analyze at the end of the experiment
![Page 34: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/34.jpg)
Feedback data
![Page 35: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/35.jpg)
Feedback data
![Page 36: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/36.jpg)
Feedback data also includes…
Xbox Kinect
Social Media Mining
Customer Support Data
![Page 37: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/37.jpg)
Active monitoring dataData Center X
Data Center YAM2
Service
AM3AM1
Dependency Service(s)
Client
![Page 38: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/38.jpg)
Ease of detection
Deploy
Detect
EaaSy – Rich near real-time telemetry
Again services have it easy
Many clients are always connected
On prem servers (enterprise) require partnerships
![Page 39: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/39.jpg)
Data handling - privacyTransparency and ControlCollection and RetentionDepends on Type anonymous Datapseudonymouspersonally identifiable info (PII)sensitive PII
Depends on Purposeprovide the serviceimprove the current serviceimprove a future version serviceimprove non-associated servicescontent personalizationad targeting
![Page 40: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/40.jpg)
Data handling – non-service productsClient and on-prem server considerations
User owned resources: bandwidth
battery
disk, cpu, etc…
Correlations
end-to-end across clients and services
by user , by session
![Page 41: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/41.jpg)
Your turn - Select your data sources
Infrastructure dataApplication dataHang and crash dataUsage dataFeedback dataActive monitoring data
Design to handle your data
![Page 42: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/42.jpg)
A 50,000 foot view
Use the right data tools
![Page 43: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/43.jpg)
Data storage and processing systemsDatabase Table storage
(SQL)Optimized for CRUD – (create, read, update, delete) of single records
Data Warehouse
Table storage (SQL)
Table structure optimized for queries and bulk insert
OLAP Cubes(Online Analytical Processing)
Multidimensional Cube (MDX)
Aggregations of measures for multiple dimensions
Hadoop / Map-Reduce
Distributed File System (HDFS)
Big Data TB-PB-EBUnstructured Data
![Page 44: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/44.jpg)
Hadoop in 60 Seconds
Hadoop HDFS
ADDFCAECC
ADD FCA
ECC
ADDADD
ADD
FCAFCA
FCA
ECCECC
ECC
Map-Reduce
1xA1xC1xF
1xA2xD
1xE2xC
2xA0xB3xC2xD1xE1xF
![Page 45: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/45.jpg)
A common data flowPower BI in
ExcelCube
Telemetry
System
SQL DW
Programmatic accessOther
Visualizations
Real-timeMonitoring
![Page 46: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/46.jpg)
Web Apps integrated with OneDrive, FB, web mail, etc
Real-time monitoring
![Page 47: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/47.jpg)
TwitterXbox Kinect
Feedback
![Page 48: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/48.jpg)
Your turn - Use the right data processing tools
DB, DW, Cube, Big-Data platformsPut it all together
Do you need real-time monitoring?
…sentiment analysis?
![Page 49: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/49.jpg)
and learn new questions
Get answers to your questions
![Page 50: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/50.jpg)
Outlook.com prioritizes performance
500 Million measurements per month
JSIJavaScript Instrumentation
View Inbox – Page Load Time (PLT) by BrowserAs experienced by actual users
![Page 51: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/51.jpg)
Prioritizes availability
Predict 75% of dips 24 hours ahead of time
Avai
labi
lity
Time
![Page 52: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/52.jpg)
52
Netflix prioritizes perceived availability & performance
![Page 53: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/53.jpg)
Yammer prioritizes usage
What happens to new user retention when you shorten the signup flow?
It goes down!
Don’t ship that feature
![Page 54: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/54.jpg)
Find new questions and repeat
Xbox recco
Try algoCollect dataAdjust algoCollect datarepeat
![Page 55: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/55.jpg)
Find new questions and repeat
In Visual Studio 2012 we added asynchronous loading for solutions
We also added Telemetry
Results look good
![Page 56: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/56.jpg)
Your turn – Get Answers, more questions
Where are the answers to your quality assessment questions?
![Page 57: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/57.jpg)
SummaryDetermine your questionsDesign for production-quality dataSelect your data sourcesUse the right data toolsGet answers to your questionsLearn new questionsRepeatThe most recent version of this deck can be found at http://setheliot.com
![Page 58: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/58.jpg)
Special thanks to these folksRavi VedulaAndrea JesseBill HodgheadDanny ThayerJoseph SefairKitty ThomasAmanda ReinkeHeather Lader
David BrooksMike TholfsenJodie DraperBrian MuellerTara RothDror CohenNathan HalsteadLori Oviatt Thanks!
Monica CatundaLynette SkinnerJoe SchumacherDonny LuuJohn HoeggerAlain Anyouzoa
Any questions?
![Page 59: Sponsored by. Your Path To Data-Driven Quality Seth Eliot Principal Knowledge Engineer, Test Excellence](https://reader030.vdocuments.us/reader030/viewer/2022032707/56649e215503460f94b0e29a/html5/thumbnails/59.jpg)
Thank you for joining us!