Using
Data Warehouse Automation
to enable
Agile Projects
Douglas Barrett(WhereScape)
How to use data warehouse automation to deliver agile analytics with the business without bypassing IT.
WhereScape
Model Automation Build Automation
discover design
document
build, refine, document, promote, operate, support
and enhance
Traditional analytics environment
Sales
Billing
The Data Warehouse is not dead!
Governed / Monitored / Reliable
One version of the truth!
Services
Financials
Data Warehouse
• IT is too slow
• Overcomplicated
• Hard to change
• Slow
REQUIREMENTS
DW framework
Profile
Logical Model
Physical Model
DB Architecture
Storage Mgmt
Index Mgmt
OLAP Design
ETL Mapping
ETL Dev
Versioning
Testing
Scheduling
Deployment
Word/Excel
Mainly in-house
solutions
BI Tools
Trillium
DataStage
PowerDesigner
Modelling
JIRA
SSMS
SSIS
SSAS
TFS
SQL Agent
Change tool
Change tool
Change tool
Change tool
Change tool
Change tool
Informatica
ErWIN
DB Management
SSIS Profile
Viewer
Documentation
Word/Excel
TFS
ETL
DW Traditional Approach
Business can not define accurate requirements
Business wants fast delivery cycles
Business wants to change requirements
often
Source data changes / New sources
Agile is Everything
Just give me access, I don’t need a data warehouse
Direct access in PowerBI / Tableau / Excel will give me what I need
See the rise of Desk top Visualization / Data Prep tools
Business goes round IT
Sales
Business goes round IT!
Sales
Desktop tools are good, but:
• No data persistence / history• Limited re-use• No IT Management
• Monitoring• Tuning• Operations• Standards• Back up
Business goes round IT!
Desktop tools are good, but:
• Quickly grows
Sales
Billing
Services
Financials
A vacuum is emerging between the business and IT
Symptom:Tableau, Qlik, PowerBI bought and managed by the business.
Business goes round IT!
IT and ETLBusiness and desktop tools
DW Framework
Profile
Physical Model
DB Architecture
Storage Mgmt
Index Mgmt
OLAP Design
ETL Mapping
ETL Dev
Version Control
Workflow
Deployment
Maintenance
Logical Model
Documentation
REQUIREMENTS
Blends development tasks1 development task instead of 6Efficient changeAgile projects
DW Better Approach - Automate
WhereScape 3DWhereScape RED
Today
1960s
1 person
1920s
3 people one car
Robots
Automation
• Productivity• Consistency
2000s
1990s
More Programmers
1980s
Programmers
Even More Programmers!
DW Automation
• Productivity?• Consistency????
12
USA I UK I NZ I Australia I South Africa
DW Architecture
Metadata
13
USA I UK I NZ I Australia I South Africa
Demo
3D• Discovery• Design
RED• Build• Document• Operate
14
USA I UK I NZ I Australia I South Africa
3D Demo
Source Target
15
USA I UK I NZ I Australia I South Africa
RED Demo