test-driven data analysis - eagle.co.nz driven... · test-driven data analysis do you believe your...
TRANSCRIPT
![Page 1: TEST-DRIVEN DATA ANALYSIS - eagle.co.nz Driven... · TEST-DRIVEN DATA ANALYSIS Do you believe your analytical results? ... Titanic: Machine Learning from Disaster. Build a Model Get](https://reader031.vdocuments.us/reader031/viewer/2022030423/5aaab0297f8b9a9a188e88ca/html5/thumbnails/1.jpg)
TEST-DRIVEN DATA ANALYSISDo you believe your analytical results?
Thanks to:Nicholas Radcliffehttp://[email protected] of Mathematics, University of Edinburgh
![Page 2: TEST-DRIVEN DATA ANALYSIS - eagle.co.nz Driven... · TEST-DRIVEN DATA ANALYSIS Do you believe your analytical results? ... Titanic: Machine Learning from Disaster. Build a Model Get](https://reader031.vdocuments.us/reader031/viewer/2022030423/5aaab0297f8b9a9a188e88ca/html5/thumbnails/2.jpg)
![Page 3: TEST-DRIVEN DATA ANALYSIS - eagle.co.nz Driven... · TEST-DRIVEN DATA ANALYSIS Do you believe your analytical results? ... Titanic: Machine Learning from Disaster. Build a Model Get](https://reader031.vdocuments.us/reader031/viewer/2022030423/5aaab0297f8b9a9a188e88ca/html5/thumbnails/3.jpg)
The Big Idea
Transfer the ideas of test-driven development
from software development
to data analysis
![Page 4: TEST-DRIVEN DATA ANALYSIS - eagle.co.nz Driven... · TEST-DRIVEN DATA ANALYSIS Do you believe your analytical results? ... Titanic: Machine Learning from Disaster. Build a Model Get](https://reader031.vdocuments.us/reader031/viewer/2022030423/5aaab0297f8b9a9a188e88ca/html5/thumbnails/4.jpg)
![Page 5: TEST-DRIVEN DATA ANALYSIS - eagle.co.nz Driven... · TEST-DRIVEN DATA ANALYSIS Do you believe your analytical results? ... Titanic: Machine Learning from Disaster. Build a Model Get](https://reader031.vdocuments.us/reader031/viewer/2022030423/5aaab0297f8b9a9a188e88ca/html5/thumbnails/5.jpg)
How
models are really developed
![Page 6: TEST-DRIVEN DATA ANALYSIS - eagle.co.nz Driven... · TEST-DRIVEN DATA ANALYSIS Do you believe your analytical results? ... Titanic: Machine Learning from Disaster. Build a Model Get](https://reader031.vdocuments.us/reader031/viewer/2022030423/5aaab0297f8b9a9a188e88ca/html5/thumbnails/6.jpg)
![Page 7: TEST-DRIVEN DATA ANALYSIS - eagle.co.nz Driven... · TEST-DRIVEN DATA ANALYSIS Do you believe your analytical results? ... Titanic: Machine Learning from Disaster. Build a Model Get](https://reader031.vdocuments.us/reader031/viewer/2022030423/5aaab0297f8b9a9a188e88ca/html5/thumbnails/7.jpg)
![Page 8: TEST-DRIVEN DATA ANALYSIS - eagle.co.nz Driven... · TEST-DRIVEN DATA ANALYSIS Do you believe your analytical results? ... Titanic: Machine Learning from Disaster. Build a Model Get](https://reader031.vdocuments.us/reader031/viewer/2022030423/5aaab0297f8b9a9a188e88ca/html5/thumbnails/8.jpg)
![Page 9: TEST-DRIVEN DATA ANALYSIS - eagle.co.nz Driven... · TEST-DRIVEN DATA ANALYSIS Do you believe your analytical results? ... Titanic: Machine Learning from Disaster. Build a Model Get](https://reader031.vdocuments.us/reader031/viewer/2022030423/5aaab0297f8b9a9a188e88ca/html5/thumbnails/9.jpg)
![Page 10: TEST-DRIVEN DATA ANALYSIS - eagle.co.nz Driven... · TEST-DRIVEN DATA ANALYSIS Do you believe your analytical results? ... Titanic: Machine Learning from Disaster. Build a Model Get](https://reader031.vdocuments.us/reader031/viewer/2022030423/5aaab0297f8b9a9a188e88ca/html5/thumbnails/10.jpg)
![Page 11: TEST-DRIVEN DATA ANALYSIS - eagle.co.nz Driven... · TEST-DRIVEN DATA ANALYSIS Do you believe your analytical results? ... Titanic: Machine Learning from Disaster. Build a Model Get](https://reader031.vdocuments.us/reader031/viewer/2022030423/5aaab0297f8b9a9a188e88ca/html5/thumbnails/11.jpg)
tdda level1: CONSTRAINTS
Look before you leap!
Checking the data conforms to your assumptions before you start.
This is tedious to generalise so there are tools to help...
l Not just the obvious input, but also intermediate and output sets
![Page 12: TEST-DRIVEN DATA ANALYSIS - eagle.co.nz Driven... · TEST-DRIVEN DATA ANALYSIS Do you believe your analytical results? ... Titanic: Machine Learning from Disaster. Build a Model Get](https://reader031.vdocuments.us/reader031/viewer/2022030423/5aaab0297f8b9a9a188e88ca/html5/thumbnails/12.jpg)
Tools to automate L1 constraint testsTool Read input Rules Action Flexibility,
Complexity
ArcGIS Input Dialog Existence, schema
Block from starting
Low
RDBMS Table schema
Field constraints, triggers
Rejection Med
TDDA python module
Pandas framework
Regular expression generator
Report Med
FME Attribute Validatortransformer
Choose from built-in rules, custom tests
Report, repair or filter
High
![Page 13: TEST-DRIVEN DATA ANALYSIS - eagle.co.nz Driven... · TEST-DRIVEN DATA ANALYSIS Do you believe your analytical results? ... Titanic: Machine Learning from Disaster. Build a Model Get](https://reader031.vdocuments.us/reader031/viewer/2022030423/5aaab0297f8b9a9a188e88ca/html5/thumbnails/13.jpg)
![Page 14: TEST-DRIVEN DATA ANALYSIS - eagle.co.nz Driven... · TEST-DRIVEN DATA ANALYSIS Do you believe your analytical results? ... Titanic: Machine Learning from Disaster. Build a Model Get](https://reader031.vdocuments.us/reader031/viewer/2022030423/5aaab0297f8b9a9a188e88ca/html5/thumbnails/14.jpg)
TEST-DRIVEN DATA ANALYSIS