extreme scoping 2006 cincinnati - swoc dama · data warehouse role in bi enterprise data warehouse...

Copyright 2004-2006, Larissa T. Moss, Method Focus, Inc.

“Extreme Scoping”What iterative development really means

presented by

Larissa T. MossPresident, Method Focus, Inc.

DAMA – CincinnatiOctober 25, 2006


OutlineWhy traditional project management (PM) does not work on DW projectsCommon DW failures and PM challengesSpiral DW methodologySoftware release conceptSelf-organizing project teamsDifferent project planning processBI program/portfolio managementBest practices/critical success factorsOrganizational culture change


Business Intelligence …

…is a cross-organizational disciplineand an enterprise architecturefor an integrated collection of

operational as well as decision support applications and databases,

which provide the business communityeasy access to their business data, and

allows them to make accurate business decisions.

… is not a system or a product


From Chaos to Architecture

DataMart

DataMart

DataMartOperational

Data Store Enterprise

Data Warehouse

OperMart

OperMart

OperMart

CRMAnal

ExplorationWarehouse

BI Access & Delivery


Data Warehouse Role in BI

EnterpriseEnterpriseDataData

WarehouseWarehouse

OperationalOperationalDataDataStoreStore

Dat

a M

anag

emen

t

Dat

a D

eliv

ery

Order Entry

ERPBack Office

SalesAutomation

CRMFront Office

AccountsReceivable

OtherLegacy

&Historical

CustomerDemogr.

Web&

External

ExplorationExplorationWarehouseWarehouse

MarketingMarketingData MartData Mart

Data Mining

CRM Analytical

OperationalOperationalMartMart

Other BI apps.


DW Goals and Objectives

1. Data Management• data integration• data cleansing

Get control over theexisting data chaos

2. Data Delivery• data access• data manipulation

Provide intuitive accessto business information

Business Intelligence Framework

80% 20%


Traditional Development MethodOperational

Systems(Legacy)

Decision SupportSystems Business Units

L

L

L

L

FinanceFA

EngineeringEN

Product SalesPS

Customer SupportCS

MarketingMK


Uncoordinated Development

Business Technology

paired up

BusinessManager

BusinessManager

ITManager

ITManager

... ...

... ...

ChiefExecutive

Officer

ChiefOperating

Officer

ChiefInformation

Officer

Business Units

Marketing

Financial

Product

Custom

er

Distribution

Inventory

Sales

KW KW KW KW KW KW KW

IT IT IT IT IT IT IT

Information Technology Units

Do You Know Your Business ?

• data redundancy• process redundancy• dirty data

“swim lane development”


Waterfall Methodologies

Post-MortemReview

Business Need

Project Plan

FunctionalRequirements

SystemAnalysis

Development

Testing

ImplementationProductionSystem Maintenance

ClientApproval

System Design

External Design

Internal Design

“Silos”


Using the Waterfall Approach

Mistakes are less expensive to fix early in the development process!

Requirements are well definedScope is manageableScope and users are confined to one business unit

(department, business function) Technology infrastructure is known and provenData volumes are relatively smallDevelopment activities are the same on every projectDevelopment activities are not cross-organizationalProject schedules are relatively easy to estimate


Industrial-Age Mental Model

TIMESCOPE

BUDGET

PEOPLE

QUALITY

1 2 3 4 5

highest to lowest priorityPr

ojec

t Con

stra

ints

• Cheaper, faster, better • Automate as quickly as possible

Priority

Cost-based value proposition

“Scrap and rework”

(Larry English)


Time versus QualityEveryone wants quality, but rarely is the extra time given or taken to achieve it.

Quality and time are polarized constraints.

The higher the quality the more time it takes to deliver.

Companies are driven by shorter and shorter schedules.

SCOPE

TIMEYAH DDD


The Time Trap

Redundancy increases

Developmenttime increases

Employeeburnoutincreases

Quality decreases

Usersatisfaction decreases

Maintenance increases

Need for morestaff increases

Business value decreases

Productivity decreases

Inability to support business drivers


Proliferation of RedundancyLegacy Data Warehouses

Data Marts

Marketing

Finance

Product Sales

Engineering

Users

L

L

L

L

DM

DM

EDW

DM

DM

transformation?cleansing? Customer Support

BI ?BI ?

Framework?


Adding to the Data Chaos

DW andData Marts

DM

DM

EDW

DM

DM

transformation?cleansing?

DMDM

DM

DMEDW


What’s different about DW projectsData Warehouse is not a system or productDatabases are structured differentlyUsers are more directly involvedDirty data becomes more visibleMore integration is requiredStandards are neededDifficult to justify costTimeframes are tighterPossibility of failure is increased


Over budgetSlipped scheduleFunctions not implementedUnhappy usersPoor performancePoor availabilityPoor or no scalabilityBenefits never materializedCost exceeded benefits

Common DW Failures


Too complicated for users Poor quality data/reportsDirty data was underestimatedDW development was treated like any other system development projectTraditional methodology did not workRAD meant skip analysis and design

Common DW Failures


Common DW Failures

DW techniques were not understood or deployedETL process did not fit into batch windowMeta data is not available to usersDirty data continues to existData is still not integratedEveryone is building stovepipe “Lega” Marts


Unrealistic scheduleUnrealistic user expectationsBudget too smallNo management commitmentNo user involvementLost or changed sponsorLost or changed staff (high turnover)Untrained or unavailable staff

Project Management Challenges



Disjointed DW administration (no BI Program Management)Changing business prioritiesUnclear or changing requirements (scope creep) New technology not understoodNo resources for data standardizationNo dispute resolution procedure


The Lesson?You cannot keep doing

what you have always doneand expect the results to be different.

“That wouldn’t be logical”Spock, Star Trek


Information-Age Mental Model

QUALITYTIME

PEOPLE

BUDGET

SCOPE

1 2 3 4 5

Proj

ect C

onst

rain

ts

Priority

highest to lowest priority

• Reassemble the entire enterprise • Reuse assets from inventory

Investment-basedvalue proposition

“Reassemble reusable components”

(John Zachman)


Coordinated Development

ODSDM

Discover, Communicate, Integrate, Document, Coordinate

Operational Environment

EDWOM

BI Databases

Decision Support Environment

Operational Systems

“cross-organizational development”

Business Units

Marketing

Financial

Product

Custom

er

Distribution

Inventory

Sales

KW KW KW KW KW KW KW

IT IT IT IT IT IT IT

Information Technology Units

EnterpriseInformationManagement


Cross-Organizational DevelopmentCommitment to data integration embedded in the methodologyCross-organizational program managementEnterprise information management groupStandards that include a common information architecture (enterprise data model)Coordinating the development/ETL processes

Disallowing stovepipe developmentExtracting and cleansing source data only onceReconciling data transformations and storing the reconciliation totals as meta data

< principles

< governance

< resources

< policies

< enforcement


Fundamental Changes1. Stop working in isolated “swim lanes”

Standardize and “share” data2. Centrally manage data like a business asset

(Enterprise Information Management [EIM])Assemble data as needed from the data inventory (enterprise data model and meta data)Standardize and reconcile data transformations Standardize business metrics for BI applications

3. Scale down project scopes to incorporate EIM activitiesUse the software release concept

4. Modify your methodology to be based on cross-organizational disciplines


Engineering Stages

Deployment

Justification Readiness

Planning Preparation

BusinessAnalysis Integration

Construction Solution

Design ProductIterative

Flexibleentry points


Development Steps

1Business CaseAssessment

Justification

Business Analysis

4Requirements

Definition

5Data

Analysis

6ApplicationPrototyping

7Meta Data RepositoryAnalysis

2Enterprise

InfrastructureEvaluation

3Project

Planning

Planning

Design

10Meta Data Repository

Design

8Database

Design

9ETL Design

Construction

14Meta DataRepositoryDevelopmt

13Data Mining

12ApplicationDevelopmt

11ETL

Developmt

15Implementatn

16Release

Evaluation

Deployment


Business Case Assessment

Business problem or business opportunity is definedReadiness and feasibility of a BI solution is assessedThe proposed BI application is cost-justifiedRamifications of not addressing the business problem or business opportunity are documentedBusiness sponsorship is obtained

Justification

Business needsBusiness needs


Enterprise Infrastructure Evaluation

Technical infrastructure gap analysis is performedTechnology platform is upgraded– Hardware, middleware, network components– Software, operating systems, tools

Non-technical infrastructure gap analysis is performedNon-technical infrastructure components are enhanced– Standards, procedures, business rules– Enterprise architecture, methodology– Scope management (change control)– Roles and responsibilities, team structure

Planning

Technology gapsTechnology gaps

Procedural gapsProcedural gaps


Project PlanningHigh-level BI project requirements defined for– Data, Functionality– Infrastructure components (technical and non-technical) Roles and responsibilities are defined and assignedTeam structures are createdProject constraints, assumptions, and risks are identifiedThe project charter is writtenThe project plan is developed– Source data is analyzed to get realistic ETL estimates– Cross-organizational activities are taken

into consideration when estimating tasks

Planning

Game plan Game plan


Project Requirements DefinitionProject requirements are compared to project constraintsThe BI application is separated into multiple releases for iterative deployment (scope management)An application requirements document is written, describing the project scope in terms of – Infrastructure requirements– Access and analysis requirements– Data requirements– Meta data requirements– Security requirements

BusinessAnalysis

Product definitionProduct definition


Data AnalysisExtensive businessbusiness analysis of data is performedA normalized logical data model (business view of data) is created (not database design!)Business meta data is defined and capturedData-related business rules are defined and capturedSource data is analyzed during modeling sessions Project-specific data model is merged into the enterprise data model (enterprise information architecture)Data definitions and data domains are standardizedData disputes are resolved with data owners and information consumers

BusinessAnalysis

Information architectureInformation architecture


Application PrototypingUsed to be called systemssystems analysisFunctional project deliverables are studiedImplementation solutions are tried and tested– Critical or difficult deliverables are partially

designed, developed, and tested (prototyped)– Technology and tools are tested– User interfaces are testedBI project requirements are again compared to project constraintsBI project requirements are revised

BusinessAnalysis

FeasibilityFeasibility


Meta Data Repository Analysis

Business meta data components are identifiedTechnical meta data components are identifiedSources of meta data (DBMS and tool dictionaries) are identified and analyzedA meta model of required meta data components is created Meta-meta data is created

BusinessAnalysis

ContextContext


Database DesignCommonly called “logical database design” (not to be confused with logical data modeling!)Appropriate design schema for BI target databases are selected (relational or multi-dimensional)A denormalized physical data model (database view) is created, either entity-relationship or star-schemaTechnical meta data is defined and capturedPhysical database design is performed– Indexing strategy– Dataset and index placement– Partitioning– Clustering

Design

Data storage Data storage


ETL DesignSource data is mapped into BI target databasesThe process flow is designed for extracting, sorting, merging, cleansing, transforming, and loading dataAn ETL process flow diagram is createdThe ETL staging area is defined or enhanced (can be centralized or decentralized, but all ETL activities must be coordinated)Common reconciliation and error handling routines are designedData transformation and cleansing requirements are designed

Design

MigrationMigration


Meta Data Repository DesignDecision to build or buy a meta data repository is madeIf the decision is to build, its components are designed – Meta data repository database (E-R or OO)– Meta data migration (load) process– Meta data application (reports, user interface)– Meta data online help functionIf the decision is to buy, a meta data repository product has to be evaluated and selected– Products have to be evaluated– Vendors have to be evaluated– RFP or RFQ is issued

Design

Context storageContext storage


ETL Development

Three ETL processes are developed – Initial load– Historical load– Incremental loadETL processes are tested– Integration or regression tested– Performance or stress tested– QA and acceptance testedETL technical meta data is defined and captured

Construction

SourcingSourcing


Application Development

Can be final iteration of an “operational prototype”Access and analysis deliverables are finalized for this BI project releaseApplication programs are tested– Integration or regression tested– Performance or stress tested– QA and acceptance testedAggregation, summarization, and other algorithmic technical meta data is defined, standardized, and captured

Construction

DeliveryDelivery


Data Mining

Data mining tool is installed and testedData mining databases are designed and createdAnalytical data models are developed and “trained” Source data is prepared (consolidated, cleansed, filtered, aggregated)Data mining databases are loadedData mining operations are selected and runData mining results are compared to industry models

Construction

Knowledge discoveryKnowledge discovery


Meta Data Repository Development

The meta data repository database is built (or enhanced)The meta data repository programs are developed and tested – Integration or regression tested– QA and acceptance testedETL-specific meta data programs are included in the ETL job schedule– Data quality statistics– ETL reconciliation totals– ETL errors and rejections

Construction

Context deliveryContext delivery


ImplementationCreate BI target databases in productionCreate the meta data repository database in productionCreate production program libraries (ETL, application, meta data repository)Move all programs into production librariesApply appropriate security measuresLoad BI target databasesLoad the meta data repositoryConduct training

Deployment

SolutionSolution


Release Evaluation

A post-implementation review (PIR) is organized– PIRs are conducted about 2 months after

implementation– All project team members are invited to participate– Meeting should be held off-site– Meeting should be facilitated by third-party– Meeting should be documented by third-party scribeAfter the PIR, action items are followed up onSome action items may be included as requirements in future BI application releases

Deployment

OptimizationOptimization


Business Integration Activities

Cross-organizationalCross-organizationalCross-organizationalProject-specificProject-specificCross-organizationalProject-specificCross-organizationalCross-organizationalCross-organizationalCross-organizationalCross-organizationalProject-specificCross-organizationalCross-organizationalProject-specificCross-organizational

Development Steps1. Business Case Assessment ........................2.A Technical Infrastructure Evaluation ...........2.B Non-Technical Infrastructure Evaluation ...3. Project Planning ..........................................4. Requirements Definition ..............................5. Data Analysis ...............................................6. Application Prototyping ...............................7. Meta Data Repository Analysis ...................8. Database Design .........................................9. ETL Design .......................................….......10. Meta Data Repository Design ....................11. ETL Development .....................................12. Application Development .........................13. Data Mining ..............................................14. Meta Data Repository Development ........15. Implementation .........................................16. Release Evaluation ...................................

Enterprise InformationManagementTouch Points


Parallel Development Activities1

Bus. CaseAssess.Justification

Go/No-Go decision Project kick-off2

EnterpriseInfrastr.

3Project

Planning

4RequiremtDefinition

Planning(Analysis)

5Data

Analysis

7MDR

Analysis

10MDR

Design

9ETL

Design

14MDR

Develop.

13Data

Mining

12ApplicationDevelop.

11ETL

Develop.

6Prototyping

8DB Design

BackEndETL

FrontEnd Appl.

MetaDataRep.

Analysis

Design

Construction

15Implement.

16Release

Eval.Deployment


Project DynamicsBusiness CaseAssessment Planning and

Requirements

Data Analysis

ApplicationPrototyping

Meta DataRepository Analysis

ETLDesign


Meta Data RepositoryDesign

ETLDevelopment Application

Prototyping

Meta DataRepositoryDevelopment

Database Design

Data Mining

ApplicationDevelopment

Release Implementation

ETLTesting

Meta DataRepositoryTesting

Application Testing

BI/DW Project

ETL Design

< parallel< parallel

< iterative< iterative

< dynamic< dynamic


Software Release Concept

Project = ApplicationProject = Application//

“Refactoring”- Kent Beck

“Extreme scoping”- Larissa Moss

“feels like prototyping”

SecondRelease

FirstRelease

FourthRelease

Reusable &Expanding

FinalRelease

BI Application

FifthRelease

ThirdRelease

Projects


Using the Software Release Approach

Mistakes are less expensive to fix early in the development process!

Unstable requirements can be tested and enhanced in small incrementsScope is very small and manageableTechnology infrastructure can be tested and provenData volumes (per release) are relatively smallProject schedules are easier to estimate because the scope is very smallDevelopment activities can be iteratively refined, honed, and adapted


Using the Software Release Approach

And the quality of the release deliverables (and ultimatelythe quality of the DW applications) will be higher!

And the development process will get faster and faster!

Unstable requirements can be tested and enhanced in small incrementsScope is very small and manageableTechnology infrastructure can be tested and provenData volumes (per release) are relatively smallProject schedules are easier to estimate because the scope is very smallDevelopment activities can be iteratively refined, honed, and adapted


Software Release Guidelines

Deliver every three to six months (first release will take longer)Strictly control the scope and keep it very smallKeep expectations realisticThe enterprise infrastructure must be robust (technical and non-technical)Meta data must be an integral part of each release; otherwise, the releases will not be manageableDesigns, programs, and tools must be flexible

SecondRelease

FirstRelease

FourthRelease

FinalRelease

BI Application

FifthRelease

ThirdRelease


Team Organization

EXTENDED

TechnicalServices

StrategicArchitect

Auditor

SecurityOfficer

End userSupport

BusinessSponsor

Operations

ProjectManagement

TechnicalSkills

EIMSkills

COREBusiness

Participation

ETLTrack

Appl.Track

MDRTrack

Programmers

TechnicalAdvisoryBoard

BusinessAdvisoryBoard


Project Team StructureProject core team

Self-organizing SWAT teamOptimum team size: 3 – 5 people; never more than 7Full-time involvement on the project from beginning to endMeet daily to control project activities

Development track teamSelf-organizing XP (extreme programming) teamOptimum team size: 2-3 people; never more than 5Full-time involvement on their development trackMeet daily to control development activities

Extended teamInvolvement on an as-needed basisSupport roles


Project Team StaffingProject core team

One hands-on project managerOne business representative with some decision-making authorityOne business analyst from IT (data administrator or business liaison)One technical lead person from IT (senior programmer or senior systems analyst)

Development track teamOne senior programmerOne programmer or programmer/analyst

Extended team Business sponsor Technical support teamOperationsEtc.


Core Team RolesApplication lead developerBI infrastructure architectBusiness representative Data administrator (EIM)Data quality analyst Database administrator/designer ETL lead developerMeta data administrator (EIM) Project managerSubject matter expert


Application Lead Developer

Designs and leads the development of the BI application’s access and analysis componentsMentors the “front-end” application developersWrites and tests the more complex queriesSupervises data access analysis and prototypingactivities of the data delivery componentsPerforms stress testing of the BI applicationUpdates the technical meta data in the repository


BI Infrastructure Architect

Identifies the technology requirements to support the BI goals and objectivesArchitects, installs, and maintains the technical infrastructure componentsSets technology priorities based on the strategic direction of the organizationParticipates in tool evaluation and selectionMonitors BI application use, trends, performance


Business RepresentativeHelps with project planning Determines deliverablesParticipates in tool evaluation and selectionIdentifies data and functional requirementsParticipates in data and process modeling sessionsParticipates in prototyping sessionsProvides data definitionsParticipates in testing activitiesValidates the quality of dataValidates the accuracy of reports and queriesIdentifies/validates transformation rules for source dataResolves disputes among business units


Data AdministratorPerforms cross-organizational data analysisCreates and manages the logical data modelEnsures the quality and integrity of the data modelManages the integrated enterprise data modelCaptures the business meta dataDevelops data standards Administers, manages and controls the CASE toolsAssists data quality analyst with dirty data discoveriesHelps the meta data administrator with populating themeta data repositoryParticipates with DBA in logical database design activities


Data Quality Analyst

Analyzes the source files and identifies dirty dataPerforms source to target data mapping (ETL specs)Creates specifications for ETL reconciliation totalsLooks for opportunities to re-use and integrate dataProvides system verification of data definitionsProvides system verification of business rulesHelps with capturing meta data into the repositoryHelps with training business people


Database Administrator/Designer

Designs the BI target databasesCreates physical database structuresResponsible for database maintenance(Monitoring, tuning, reorganization, backup and recovery)

Participates in extracting and loading datafrom the source systems to the DW databasesImplements database securityProvides internal support for DBMS softwareAssisting and enhancing the use of query toolsto access the DW databases

DatabaseDesign

DBMS environment

Applicationsupport


ETL Lead Developer

Designs and leads the development of the ETL process Mentors the “back-end” ETL developersHelps to evaluate, select, install, and test the ETL toolCreates and maintains the process flow diagram for the ETL processHelps in the selection of which source files to use when there are multiple sourcesPerforms stress testing of the ETL processUpdates the technical meta data in the repository


Meta Data Administrator

Creates the meta model for the meta data repositoryBuilds or selects a meta data repositoryImplements the meta data repositoryProvides access to the meta data repositoryAdministers, manages and controls the meta data repositoryMerges and loads meta data from the other tools into the meta data repositoryParticipates with DA in establishing data standards


Project Manager

Overall responsibility for the BI projectDefines, plans, controls, reviews the project activitiesTracks and reports on project progressStaffs the project with the right peopleEstimates and measures the costs and benefitsActs as a quality reviewer throughout the projectNegotiates with tool vendors, business representatives, and the business sponsorResolves technical issuesRaises disputes to the advisory boards


Subject Matter Expert

Provides business knowledge for a subject areaMakes suggestions on the required format of the data

(detailed, summarized, aggregated, derived and historical)

Writes test cases and participates in all testing activitiesActively participates in application prototypingHelps establish criteria for query tool selectionTrains business people in the use of query toolsIdentifies source data quality problems and establishes processes to resolve them


Extended Team RolesApplication developersBI support staff (helpdesk)Business sponsorData MinerETL developersIT auditor or QA analystMeta data repository developerNetwork servicesOperationsSecurity officerStakeholders

Technical services TestersTool administratorWeb developersWebmaster


Software Release Planning Process1. Estimate development effort for entire BI application1. Estimate development effort for entire BI application

592 Write query script specifications 587 593 Write access interface programming specifications 588 594 Write online help function programming specifications 589 595 Step 11: Extract/Transform/Load Development 596 Build and unit test the ETL process 542, 549 597 Code the ETL programs 598 If using an ETL tool, write instructions for ETL tool modules 599 Capture the ETL technical metadata for the metadata repository 600 Write code to produce reconciliation totals, quality metrics, and load statistics 601 Unit test each individual program module 597, 600 602 If using an ETL tool, unit test each ETL tool module 598, 600 603 Write the scripts to execute the ETL programs and the sort, merge, and load utilities 601, 602 604 Integration or regression test the ETL process 596 605 Create a test plan with test cases for the ETL process 606 Create test data for the ETL programs 607 Integration or regression test the entire ETL process 606, 606 608 Log the actual test results and document any test issues

Methodology

from Business Intelligence Roadmap (Moss & Atre)

• activities• dependencies• deliverables• roles

Work Breakdown Structure (WBS)

describes

• tasks• sequence• schedule• resources• milestones• checkpoints

Traditional Project Plan

defines

refines

Software Releases & Project Activities• number of releases• day-to-day work activities• progress tracking and status reporting• change control and impact analysis

drives

Based on source: TDWI

overall effort,resources,

cost


Software Release Planning Process1. Estimate development effort for entire BI application1. Estimate development effort for entire BI application

2. Break BI application into software releases2. Break BI application into software releases

SecondRelease

FirstRelease

FourthRelease

FinalRelease

BI Application

NthRelease

ThirdRelease

20062006


Software Release Planning Process

3. Create milestones from DDD to YAH for first release 3. Create milestones from DDD to YAH for first release

1. Estimate development effort for entire BI application1. Estimate development effort for entire BI application


FirstRelease

Milestone

Activity

Task/SubtaskTask

Deliverable

PROJECT

Highest level of projectpartitioning

Major work effort of 1-3 weeks consisting of one or more tasks

A unit of work that may be measured in days or hours produces

produces MilestoneDeliverable

reported

managed



4. Organize and assign parallel development tracks4. Organize and assign parallel development tracks




1Bus. Case

Assess.

2Enterprise

Infrastr.

3Project

Planning


5Data

Analysis

7MDR

Analysis

10MDR

Design

9ETL

Design

14MDR

Develop.

13Data

Mining


11ETL

Develop.

6Prototyping

8DB Design

BackEndETL

FrontEnd Appl.

MetaDataRep.

15Implement.

16Release

Eval.



5. Create detailed work assignments for all milestones5. Create detailed work assignments for all milestones







6. Create Gantt chart showing milestones6. Create Gantt chart showing milestones






ID Milestone Name DurationDetermine problem/opportunity

Analyze cost/benefit of new projectDetermine critical success factorsAnalyze resources available

Approval of project



6. Create Gantt chart showing milestones6. Create Gantt chart showing milestones



3. Create milestones from DDD to YAH for next release 3. Create milestones from DDD to YAH for next release

1. Review development effort for entire BI application1. Review development effort for entire BI application

2. Review and adjust BI application software releases2. Review and adjust BI application software releases

SecondRelease

FourthRelease

FinalRelease

NthRelease

ThirdRelease


Iterative BI Application DevelopmentPlanning

Requiremts& Data Analysis

Requiremts& ApplicationPrototyping


ETLDesign


Meta DataRepository Design

ETLDevelopment


Data Analysis

Data Mining


ETLTesting


ApplicationTesting

ETL Design


Release Implementatn

BusinessCase

Assessment

Post-Impl.Review

Planning




ETLDesign



ETLDevelopment


Data Analysis

Data Mining


ETLTesting


ApplicationTesting

ETL Design



BusinessCase

Assessment

Post-Impl.Review

Planning




ETLDesign



ETLDevelopment


Data Analysis

Data Mining


ETLTesting


ApplicationTesting

ETL Design



BusinessCase

Assessment

Post-Impl.Review

Planning




ETLDesign



ETLDevelopment


Data Analysis

Data Mining


ETLTesting


ApplicationTesting

ETL Design



BusinessCase

Assessment

Post-Impl.Review

Planning




ETLDesign



ETLDevelopment


Data Analysis

Data Mining


ETLTesting


ApplicationTesting

ETL Design



BusinessCase

Assessment

Post-Impl.Review

Planning




ETLDesign



ETLDevelopment


Data Analysis

Data Mining


ETLTesting


ApplicationTesting

ETL Design



BusinessCase

Assessment

Post-Impl.Review

Release 1

Release 2

Release 3

Release 4

Release 5

Release 6

BIApplication


Flexible Entry and Exit Points

Third release of a Marketing data martDeliverables:

20 new data elements from a new internal source 2 new reports

Data will be added to the existing BI databaseExisting report writer will be used (no new tool)Meta data repository is not in scope (separate project)Online help function is not in scope (last release)Data mining is not part of the requirements at allBudget and resources are not an issueNo post-implementation review necessary after this minor release

Example:Example: SecondRelease

FirstRelease

FourthRelease

FinalRelease

BI Application

FifthRelease

ThirdRelease


Entry and Exit Criteria

1Business CaseAssessment

Justification

Business Analysis

4Requirements

Definition

5Data

Analysis

6ApplicationPrototyping

7Meta Data RepositoryAnalysis

2Enterprise

InfrastructureEvaluation

3Project

Planning

Planning

Design

10Meta Data Repository

Design

8Database

Design

9ETL Design

Construction

14Meta DataRepositoryDevelopmt

13Data Mining

12ApplicationDevelopmt

11ETL

Developmt

15Implementatn

16Release

Evaluation

Deployment


Project Organization

3Project

Planning

9ETL

Design

11ETL

Develop.

8DB Design

BackEndETL

Front-End Applicatn.

15Implmnt.

4RequiremtDefinition5

DataAnalysis


6Prototyping


Planning(Analysis)

Analysis

Design

Construction

Deployment

ThirdRelease

Milestone

Activity

Task/SubtaskTask

Deliverable

PROJECT

Highest level of projectpartitioning

Major work effort of 1-3 weeks consisting of one or more tasks

A unit of work that may be measured in days or hours

produces

produces

MilestoneDeliverable

reported

managed

ID

Milestone Name

Duration

Determine problem/opportunityAnalyze cost/benefit of new project


BI Program/Portfolio Management

• Common Development Methodology• Common Enterprise Standards • Common Ratified Business Rules• Common Enterprise Infrastructure

• Coordinated ETL Processes • Common Meta Data• Common Enterprise

Information Architecture

ProjectManagement

TechnicalSkills

EIM Skills

COREEXTENDED

TechnicalServices

StrategicArchitect

Auditor

SecurityOfficer

End userSupport

BusinessParticipation

BusinessSponsor

ETLTrack

Appl.Track

MDRTrack

Programmers

Operations

ProjectManagement

TechnicalSkills

EIM Skills

COREEXTENDED

TechnicalServices

StrategicArchitect

Auditor

SecurityOfficer

End userSupport


BusinessSponsor

ETLTrack

Appl.Track

MDRTrack

Programmers

Operations

ProjectManagement

TechnicalSkills

EIM Skills

COREEXTENDED

TechnicalServices

StrategicArchitect

Auditor

SecurityOfficer

End userSupport


BusinessSponsor

ETLTrack

Appl.Track

MDRTrack

Programmers

Operations


BI Program Management ComponentsBI steering committee made up of business executives and senior business managers Collective sponsorship from business executivesCommunicate enterprise-wide data integration principles to all lines of business (LOB) [ongoing]Create a BI strategy appropriate for the organizationCreate an enterprise information management groupIdentify data owners and data stewards in all LOBAssign business representatives to CORE teams


BI Program Management ComponentsHeaded by a BI program manager or BI directorPerform periodic readiness assessments

Identify new information needsAscertain user satisfaction

Create and enforce a common non-technical infrastructure (“Core Competencies”)Prioritize DW projects in alignment with strategic goals (BI application management)Determine DW project interdependenciesCoordinate DW project resources and activities (Software release management)


BI Maturity Model

1. Prenatal 2. Infant 3. Child 4. Teenager 5. Adult 6. Sage

“Production Reporting”

“Spreadmarts”

“Data Marts”

“Data Warehouses”

“Enterprise DW”

“Analytic Services”

GULF CHASM

Source: Wayne Eckerson, TDWI


Best Practices/Critical Success FactorsStop building silo point solutions for individual usersCreate a BI Program with executive business sponsorsNever sacrifice quality for time or scopeProfile your source data before committing to a project delivery date (deadline)Involve enterprise information management groupUse a spiral, cross-organizational DW methodologyPrototype extensivelyBuild and deliver each BI application in multiple 90/120-day software releases Organize the project work into parallel (simultaneous) tracksCreate small self-organizing SWAT teamsInsist on daily user participation on project activities


Over budgetSlipped scheduleFunctions not implementedUnhappy usersPoor performancePoor availabilityPoor or no scalabilityBenefits never materializedCost exceeded benefits

Common DW Failures

Software releases

User participation on CORE team

Spiral methodology

BI Program Management

Software releases


Too complicated for users Poor quality data/reportsDirty data was underestimatedDW development was treated like any other system development projectTraditional methodology did not workRAD meant skip analysis and design

Common DW Failures


Spiral methodology

Enterprise Information Management


Common DW Failures

DW techniques were not understood or deployedETL process did not fit into batch windowMeta data is not available to usersDirty data continues to existData is still not integratedEveryone is building stovepipe “Lega” Marts

Software releasesand prototyping


Spiral methodology

Enterprise Information Management


Unrealistic scheduleUnrealistic user expectationsBudget too smallNo management commitmentNo user involvementLost or changed sponsorLost or changed staff (high turnover)Untrained or unavailable staff



BI Program Management with collective sponsorship

Self-organizing SWAT team



Disjointed DW administrationChanging business prioritiesUnclear or changing requirements (scope creep) New technology not understoodNo resources for data standardizationNo dispute resolution procedure


Software releases and user participation on CORE team


Prototyping


Organizational ImpactNo BI program management in placeCross-organizational tasks and responsibilities are not well definedNew project dynamics are not understoodTeams are still organized the traditional wayBusiness people still don’t participate on project activitiesData quality responsibility is not clear or ignoredDW projects are usually cost justified using the industrial-age mental model (no ROA)No reward for data sharingResistance to change


Inevitable Culture Shift

Business and IT collaboration (“partnership”)Business and business collaboration (“partnership”)IT and IT collaboration (“partnership”)Shifted roles and responsibilitiesCommon architecture Enterprise standardsCross-organizational activitiesSoftware release concept


Power Shift

Business people must play an increased role and assume responsibilities for:

Data ownership – Information ArchitectureEnforcement of standardization – IncentivesIntegration and standardization across allbusiness units – Cross-Organizational DisciplinesDispute resolution across all BI projects – LeadershipGovernance – BI Program Management


Charge-Back PolicySystem ownership – by business unit

Automate a business process from scratchCost-based value propositionBusiness units pay for their own systemsSystem = programs coupled with data

old

new Information ownership – by the enterpriseReassemble the entire enterprise quicklyInvestment-based value propositionOrganization pays for enterprise dataBusiness units pool to pay for shared BI applicationsBI application = BI tools and programs decoupled from data


New Incentive Policy

Applicable to the business side as well as IT.

Management and staff should be evaluated on:Team work and collaboration (IT and business)Data sharing (business)Data quality (business)Adherence to standards (IT and business)Adherence to cross-organizational development approach (IT and business)

Rewards (salary increases and bonuses) should be tied to above performance measures


New LeadershipCEO

CIOCKOCOO

DG ...EA,EIM,

MDM,DW,

BI,CRM,

CDI,PIM,

…

LOB Execs IT Execs

ChiefKnowledge

Officer

CFOcollaboration collaboration


Thank You

Larissa T. MossMethod Focus, [email protected]

ISBN 0-201-61635-1

ISBN 0-201-78420-3

ISBN 0-201-76033-9

ISBN 0-321-24099-5

extreme scoping 2006 cincinnati - swoc dama · data warehouse role in bi enterprise data warehouse...

Documents