BIOPRESENTATION

Better Software Conference & EXPOSeptember 27-30, 2004

San Jose, CA USA

W6

September 29, 2004 3:00 PM

CUSTOMER FOCUSED BUSINESSMETRICS THROUGHOUT THE

SDLC

Steve WrennLiberty Mutual Insurance Information Systems

STEPHEN WRENNStephen Wrenn is currently the Sr. Director of IT Service & Quality Assurance at LibertyMutual Insurance Group. There he is responsible for bringing people, process andtechnology together to solve insurance business problems on a global basis.

Prior to Liberty Mutual, Stephen was the VP of Global Service & Support for EnterasysNetworks, where he led a team of service professionals in delivering industry leadingnetwork quality initiatives, and global expansion of Enterasys professional services.

Stephen has over 20 years experience in IT company’s, holding leadership positions atHewlett-Packard, Cabletron, and Cisco Systems. He has built technical supportorganizations in both the US and abroad, most recently encompassing offices in London,Singapore and Hong Kong.

Stephen holds a B.S. in Electrical Engineering from the University of Massachusetts andan MBA from the Whittemore School of Business at the University of New Hampshire.

Customer Focused Business Metrics Throughout the SDLC

Better Software Conference & Expo ’04Measuring and Reporting Track

Steve Wrenn, Director IT Service & Quality AssuranceLiberty Mutual Insurance Group, Information Systems

Introduction

A quick overview of Liberty Mutual IS and it’s culture.What we set out to do.Our process or journey

ConstraintsSDLC vs. PLCQuality FrameworkDevelopment Focus

OutcomesWhat we learned

Where We Were at Liberty

First of all you have to understand a few things about Liberty Mutual

Insurance Business is inherently Risk AdversePrivate Company with, until recently, mostly internal customer’s for ITCurrently profitable in a very stable and long-life business

After that very Much Like a Lot of IT Development Shops

Not enough time to do everythingLack of controls…after all, Coding is an art right?

What We Set Out To do

Run IT Like a Business….Increase Level of Service to our customers by delivering high quality products and services on time and on budget.Get it right the first time… One and Done!

Our Process to“Run IT Like A business”

First decide where we want to go.Strategic objectives

Business objectivesCompetitive advantageApplication Development Focus

Getting the Voice of the Customer (VOC).Who’s the customer?What do customers want/need

Operational FocusTQM, Six-Sigma, CMMI Level ?

Customer Focus

Faster Better

Cheaper

CUSTOMERDELIGHT!

Creativity, Flexibility and

Responsivenessare the Keys!

Our Process…….continued

Then evaluate where we currently stand. (Nothing new here, a simple GAP analysis should suffice nicely…yeah right!)

Baseline metrics….what metrics?Our work is so unique you can’t measure it

Evaluation of current processesThings are working fine, we’re making money, why do we want to change thingsProcesses were all different, so any metrics collected could not be compared.

All hard to do without knowing current output (systems thinking).

We finally settled on quality control data gathering for 6 months.

Our Constraints

The old project management triple constraint of scope, schedule and budget, and also a mindset that said get it done on time, at all costs.

“What do you think was the main cause of this?”

It All Starts With Requirements

Or does it?

SDLC vs. PLC Mindset

The software development life cycle of most companies are very well thought out and focused on getting a project done on time, on schedule and on scope.How many of them are actually focused on delivering the customer what they want and need to be successful?

How do they know if they are?

The PLC Feedback-Loop

Think customer input, which in most companies comes from the users of your current products and services. (VOC)How many of you proactively use this information to feedback to your development teams the fruits of their labors, or to design your next product?

“The past is a pretty good predictor of the future”

LMIG IT QualityFramework / Methodology

Product Life Cycle (PLC)

Plan Build Run Retire

Software DevelopmentLife Cycle (SDLC) Production

Customer(End-User)

Voice of the Customer (Requests/Concerns)

Change Management (For New & Production)

Problem Management (Restore & Eliminate)

Tools to enable Effectiveness & EfficiencyMetrics for Success and Adjustments (Targets)

B.U.

Release/AcceptancePre-Production

PDCA PDCA PDCA

•Requirements•SQA/Test Plan•QA Deliverables

•Stress Testing•Final SLA•System Burn-in

•SLA Monitoring•Support Training•EOL Strategy

Quality Assurance

Our Development Focus

Plan and BuildDefine, Design, Develop & TestBefore you jump into this, there is a little education that is extremely valuable.

“How do you keep the team focused on getting the customer’s what they need to be successful?”

A Focused Development Team

Help them start to think like business managersTake the time to explain to them what the fruits of their labors are.

They are not developing code, they are building pieces to a product that fit into a system or systems that somehow make money for the company!Show them how their applications contribute, they may surprise you with their ideas here.Show them how quality, both high and low, effects the outcome

From the Quality/Test Perspective

Worked with the development team to put metrics in place that constantly let them know how they and their processes are doing.

Plan, Do, Check, Act….make it self-managing by building a metrics program or dashboardClosed-loop, systems thinking and trainingThere can be a lot of cultural shock here.

Liberty ITContinuous Process Improvement

People Process

Technology

Optimum Productivity!

Knowledge Sharing,Customer Focus

andLeadershipare Key!

Our Outcomes

We developed a quality process and measurement program that collected few, but very specific measurements at each stage of our development cycle.We improved our first release quality by 40% in the first 6 months of our initiative

How did we do it?

LMGIT - SDLC Key Quality Deliverables - Worksheet

RequirementsDefinition Architecture &Scope

DefinitionRequirements

Definition Analysis Architecture &Design

Construction &Test Implementation

PMO RequiredRequest for Project FundingReportCreate Project Charter *Create Initial IT Project Plan(006) with Key Milestones*PMWB ForecastRisk /Issues LogChange ManagementProcess

QA Best PracticesBISAT/ OpportunityEvaluation (100) * /**Create Initial SQA Plan*Review with Technology MgtBoard (TMB) *Review with EnterprisePortfolio Review Board(EPRB) *Obtain Key StakeholderDecision for “Go / No Go” forproceeding with Bus.Requirements *Statement of Work /Contract (if applicable)Sarbanes Oxley

PMO RequiredFirst cut SLA Performance *Create Initial Test StrategyDocument

QA Best PracticesBusiness Alternatives (125)*Bus Objectives (130)*Proposed Workflow (132) *System Context Diagram -“To Be” (140) *Requirements ManagementPlanGlossary of Terms

QA Best Practices UpdatedUpdated SQA PlanUpdated IT Project Plan

PMO RequiredFreeze Detailed BusinessRequirements (135s) - OOProjects = Use CaseScenarios, Class Diagrams,Activity Diagrams, etc.* (all changes now gothrough the ChangeManagement Process)***Technical ArchitecturePreliminary Review Report *Logical Data Model *Completed DisasterRecovery Requirements *

QA Best PracticesSecurity Requirements *System InterfaceRequirements *Create Test PlanCreate Test Cases -Functional, System,Integration, & UATBaseline Schedule/ BudgetBusiness RulesData Conversion RulesArchive Purge RequirementsUpdated Glossary of Terms

QA Best Practices UpdatedUpdated / Test Strategy -Finalized *Baselined IT Project Plan

PMO RequiredFunctional Specs (490) *Infrastructure RecoveryRequirements (IRR) - 1st Cut

QA Best PracticesTechnical Alternatives (260)Detailed Technical &Architec. Design Review *User Interface Specifications

System Storyboard/ Flow(250/255)

HTML / Word Page Mock-Ups (495)Non Functional /Supplementary Specs -Performance Reqmnts,Archive /Purge, Security,Disaster Recovery, etc.OO / Baselined Object ModelFinalizd Funct’l Data Model *Desktop QA Plan *Program Specs (570)End User Training StrategyImplementation Strategy

QA Best PracticesUpdatedFinalized Test Cases -Functional, SystemsIntegration, and UATFinalized Test Plan *Updated Project Plan

PMO RequiredUser Acceptance Testing

QA Best PracticesSource Code Mgt Plan/Configuration MgtService Center ChangeTicketDefine/Create TestingEnvironmentsSoftware Code CreatedConduct Peer Code ReviewsUnit TestingFunctional TestingSystems Integrated TestingPerformance TestingVolume TestingStress TestingDesktop QA CertificationNotificationUAT Testing Sign-Off * / ****Risk Assessment to Prod *Software Code Frozen * (allcoding changes now go thruChange Mgt Process)Baselined Phys Data ModelTest Communication Plan forTeams/ Management

QA Best PracticesUpdatedFinalized IRR Document *Finalized SLA *Updated IT Project Plan

PMO RequiredImplementation PlanIncluding ImplementationChecklist (back out plan andSignoff (800)Lessons Learned and ActionPlan Document (060)

QA Best PracticesResolved RequirementsChange Management IssueProduction Support Plans *On-line Help System: UserGuide (700); Help Desk(701); Training (710);Installation Instructions (720)Release Notes - BusinessEnd User and IS ReleaseNotesPerform Post ImplementationVerification TestingObtain Client Sign Off for“Go/ No Go” to deploy toProduction *IT - QA Sign Off per SQAPlanMonitor System Performancefor 90 DaysPublished Re-Usable Assetsin Component Re-UseSystem Maintenance Support Plans

David H. Maurer / June 15, 2004

* Key Stakeholder Sign-Offs

** (NNN) Refers to theLMSDP ProcessArtifacts

IT Project PlanPMWB (Project Managers Workbench)

Risk/ Issues LogChange Management Process/ Log (Mantis) -> ServiceCenter

SQA Plan

Project Management

LMGIT - SDLC - Key Quality Deliverables Worksheet - V1.03

Project Name: ______________________

Project Code: ______________

IT Project Mgr : _______________________

Bus Project Mgr : ______________________

*** Date Business Requirements Frozen: _____/_____/____Proj Mgr: __________________ QA Mgr ___________________

**** Date User Acceptance Testing Completed : _____/____ _/_____Proj Mgr ___________________ QA Mgr _______________________

SDLC Quality Gates / Metrics

Initiation / Scope DefinitionInstituted a rigorous project qualifying review

ROI or competitive advantageStrategic fit…or else!

Requirements DefinitionDo we know what this requirement is for?Has it been verified for testability and signed of by the Test or QA manager?Have the Requirements been signed off by all parties?

Requirements DefinitionCustomer Needs / Wants Analysis

Requirements DefinitionQuality Requirements

5 Major Issues Preventing High Quality Requirements (and thus setting the project up for failure from the start)

Requirements not completeRequirements not testable (to gray)Requirements developed without end-user inputRequirements thought to be non-negotiableRequirements not signed of by all parties.

SDLC Quality Gates / Metrics

AnalysisTest plans completedRequirements freeze in place (change mgt)

Architecture and DesignTest cases completed (100%)

Requirements Freeze in place

SDLC Quality Metrics Gates

Construction and TestChanges in this stage require higher level of sign-offSLA developed

ImplementationSLA signed-off by all parties30, 60, 90 day monitoring of qualityProject estimation chart completed

What We Learned

Culture change is more difficult than any technical challenge you will encounterThe more time spent in planning the less time spent in other phases or “An ounce of planning is worth a pound of development”.Signatures are very powerful in setting and keeping expectationsWe very often take too much for granted when it comes to business / technical understanding.Metrics need to be SMART

Liberty IT Monthly or Quarterly Report Card

FY’04 Q1 Deliverables

Key Monthly Accomplishments

Strategy #1: Engage proactively and regularly with customers, partners and sales organizations, and technology development stakeholders, to ensure delivery of consistent, high quality, and relevant service and support worldwide.–>/= 4.6 out of 5.0 on WW customer satisfaction survey–>15% Increase in Productivity –<99% of budgeted expenses

Strategy #2: Grow current service offerings in order to enable the business groups to deliver better service to their customers, and thus further differentiate the total Enterasys value proposition from the competition.–>110% WW Bookings targets (15% growth)–>65% Contribution Margin (cost down 10%, rev up 15%)Strategy #3: Develop and deliver new services that add value to our products and solutions, and differentiate them in the marketplace.–>50% of services developed in last 3 years

Status

• Measurable accomplishments towards your business plans•Productivity, costs, customer satisfaction, etc….

• Examples…….•Completed Design Stage on project XX•% of projects in development that are on time•Change management measures•Quality before release measures•Quality in production measures

CONCERNED

COMPLETE

ON-TARGET

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

FY'01 Q

1 Q2 Q3 Q4 FY'02

Q1

Q2 Fcs

t

Q3 Fcs

t

Q4 Fcs

t

US Canada LA EMEA AP

Customer Satisfaction Rating

Customer Satisfaction Rating

$0$5,000

$10,000$15,000$20,000$25,000$30,000$35,000$40,000$45,000$50,000

FY'01 Q

1Q2 Q3 Q4

FY'02 Q

1Q2 Q3 Q4

Bkgs Actual F'cst

SDLC Quality Gates / Metrics Data vs. Information

Data is IMPORTANT!…Without Data …………..…..Opinion PrevailsData + Analysis = IntelligentInformationData by itself is the library without a librarian.

The Importance of Proper Measurement

Positive or Negative?

The Importance of Proper Measurement

Positive or Negative?

The Importance of Proper Measurement

Positive or Negative?

LMIG IT QualityFramework / Methodology

Product Life Cycle (PLC)

Plan Build Run Retire

Software DevelopmentLife Cycle (SDLC) Production

Customer(End-User)

Voice of the Customer (Requests/Concerns)

Change Management (For New & Production Apps)

Problem Management (Restore & Eliminate)

Tools to enable Effectiveness & EfficiencyMetrics for Success and Adjustments (Targets)

B.U.

Release/AcceptancePre-Production

Thoughts …

“Don’t Tell Me … Prove It!”“Close-The-Loop … Systems Thinking”

PDCA PDCA PDCA

•Requirements•SQA/Test Plan•QA Deliverables

•Stress Testing•Final SLA•System Burn-in

•SLA Monitoring•Support Training•EOL Strategy

Quality Assurance