lean data architecture: minimize investment, maximize value brandon byars [email protected]...
TRANSCRIPT
Lean Data Architecture:Minimize Investment,
Maximize Value
Brandon [email protected]@BrandonByars
Manvir [email protected]@ManvirGrewal
Brave New World
Every 2 days we create as much informationas we did from the dawn of civilization to 2003
- Eric Schmidt, 2011
Big Data• Volume• Velocity• Variety
NoSQL•Schemaless•Non-relational
Future, Backwards
Data Platform
Support variety of analytics/users
Ubiquitous access
Logicwolf.com
Evolves as business changes
Sound good…So what's stopping you?
Companies lose $8.2 Million annually due to bad data – Gartner
Study estimated total cost to US economy of data quality problems at over US$600 billion per annum - Wayne Eckerson
SpreadMarts
Didn’t test well enough…
Upfront and Inflexible design
Vk.com
Bring everything, we may need it
Manual Testing is good for…
Not good for data or software
Spreadsheet comparisons…over and over!
Where are your bottlenecks?
The Data Focus Problem
"Data warehousing and BI is one of the largest expenses in CIO budgets, and to-date has had limited value to the business."
- Jeff Smith, CEO Shared Services, Suncorp
Gartner estimates that between 70-80% of BI projects are failures
How do we get to the future?
Wallchan.com
App Analysis Modeling DBA ETL Testing Report
2 weeks4 weeks
2 weeks4 weeks
2 weeks4 weeks
2 weeks4 weeks
2 weeks4 weeks
2 weeks4 weeks
2 weeks
Customer
Value Streams
App Analysis Modeling
Systems Thinking
How long would it take you to add a single column to a report?
From Idea to Prod…
Muda
Eight Wastes
Logismarket.de
BI/DW: Status Quo
Data
Lean BI: Value First
value
Cost & Time
100%
100%
MVP
Sometimes we accidentally build something that nobody wants, in which case it doesn’t matter if we do it on time and on budget
Prototype reportDashboard populated with static dataRudimentary scoring model
Reduce Batch Size
Dev2ops.org
Reduce WIP
Work In Progress
L = λW
Simplify Architecture – Message Based
ReportsListenerBus
DB
DB
DB DB
Automated Testing
How do we test?
Test data set based on profilingVolume is not necessaryVariety is necessaryVariability/Complexity is necessaryVelocity is necessary
System testingPlay transactions from sources in UAT/pre-prod environment
Developer Sandboxes
Automated Build
Continuous Integration
References
Data Explosion is coming!