Terry LeeperTerry LeeperDirector Platform StrategyDirector Platform StrategyEMEAEMEA

The Microsoft The Microsoft Development ProcessDevelopment Process

Overview of TopicsOverview of Topics What to ship?What to ship? Team Structure, RolesTeam Structure, Roles How a milestone worksHow a milestone works Keeping builds stableKeeping builds stable Bugs!Bugs! End GamesEnd Games

Not CoveredNot Covered How MS recruits teamsHow MS recruits teams Product SupportProduct Support Shipping delaysShipping delays

Software BasicsSoftware Basics

Shipping TimelinesShipping Timelines

Multi-Month

Multi-Year

Real-Time, Daily, Weekly, Monthly, Quarterly Webzines, subscription fulfillment, open source

Web apps, Desktop software

Complex rich client apps, Operating Systems, Server Applications

General Microsoft General Microsoft PhilosophyPhilosophy

Common Microsoft PitfallsCommon Microsoft Pitfalls Team starts coding before the product Team starts coding before the product

vision is clearly defined, communicated, vision is clearly defined, communicated, and bought-in by peopleand bought-in by people

Vision is technology focused instead of Vision is technology focused instead of customer focusedcustomer focused

Verbal consensus on the vision, but Verbal consensus on the vision, but nothing written downnothing written down

Code reuse across companyCode reuse across company

Figuring Out What To BuildFiguring Out What To Build Most important and most dangerous part of a productMost important and most dangerous part of a product

Importance/danger grows exponentially with team Importance/danger grows exponentially with team sizesize

Failure to get crisp on product vision always produces Failure to get crisp on product vision always produces problemsproblems The release date will slip, and the feature-set won’t The release date will slip, and the feature-set won’t

be perfectbe perfect Clear product plan document provides team focus on Clear product plan document provides team focus on

the product visionthe product vision Helps prioritize goals, product focus, and overall Helps prioritize goals, product focus, and overall

feature-setfeature-set Helps provide road-map for a large team as to how Helps provide road-map for a large team as to how

everything fits togethereverything fits together Helps partners understand what you are buildingHelps partners understand what you are building

Product PlanningProduct Planning

Market NeedsMarket Needs Company Strategy and PartnershipsCompany Strategy and Partnerships

Supporting architectures, platformsSupporting architectures, platforms Supporting new technologiesSupporting new technologies

Customers’ Requests, ComplaintsCustomers’ Requests, Complaints Product Team VisionProduct Team Vision The Core FundamentalsThe Core Fundamentals

Security, Stability, Productivity, Usability, Security, Stability, Productivity, Usability, Performance Performance

Product PlanningProduct Planning

Before M0Before M0 Steering GroupSteering Group

Cross Functions, All LevelsCross Functions, All Levels Focus Groups, Surveys, Much MoreFocus Groups, Surveys, Much More

Answers Answers Where are we going?Where are we going? Map Product onto Long-term VisionMap Product onto Long-term Vision Define Release Specific VisionDefine Release Specific Vision Establish Product Release PillarsEstablish Product Release Pillars

Planning for DevelopmentPlanning for Development

Post-Mortem Previous Product CyclePost-Mortem Previous Product Cycle Development TimelinesDevelopment Timelines

CodingCoding StabilizationStabilization IntegrationsIntegrations Milestone Exit CriteriaMilestone Exit Criteria

Build and Release StrategyBuild and Release Strategy Checking in processesChecking in processes Build processesBuild processes Daily release processesDaily release processes

Cross-Coordination Cross-Coordination StrategyStrategy Working with Product TeamsWorking with Product Teams

Division TeamsDivision Teams Other Microsoft TeamsOther Microsoft Teams

Working with Vendors, CustomersWorking with Vendors, Customers DependenciesDependencies DeliverablesDeliverables

Product StructureProduct Structure GM, VPGM, VP Product GroupsProduct Groups

VS Core, VB, C#, C++, etc.VS Core, VB, C#, C++, etc. Office Core, Word, Excel,Office Core, Word, Excel, BaseOS, GDI, etcBaseOS, GDI, etc..

Team StructureTeam Structure

Products built at Microsoft by “Product Units”Products built at Microsoft by “Product Units” Contains 3 focused disciplines (Dev, Test, PM)Contains 3 focused disciplines (Dev, Test, PM) Product Units are run and report to Product Unit ManagersProduct Units are run and report to Product Unit Managers

Program ManagementProgram Management Responsible for product feature-set and feature designResponsible for product feature-set and feature design Customer advocate within teamCustomer advocate within team Communicators to other groups, customers, managementCommunicators to other groups, customers, management Box and Feature PMsBox and Feature PMs

DevelopmentDevelopment Responsible for implementation Responsible for implementation Responsible for architectureResponsible for architecture

TestingTesting Responsible for quality assuranceResponsible for quality assurance Responsible for writing customer scenariosResponsible for writing customer scenarios

Product Team StructureProduct Team Structure

Product UnitProduct Unit

ManagerManager

DeveloperDeveloper

ManagerManagerTestingTesting

ManagerManager

GroupGroup

ProgramProgram

ManagerManager

Dev LeadsDev Leads

DevelopersDevelopers

Test LeadsTest Leads

TestersTesters

PM LeadsPM Leads

Program MgrsProgram Mgrs

Other DisciplinesOther Disciplines

Build team Build team Product ReleaseProduct Release MarketingMarketing Product DesignProduct Design User EducationUser Education UsabilityUsability

CommunityCommunity LocalizationLocalization Product Support Product Support

ServicesServices Sustaining Sustaining

EngineeringEngineering

Microsoft’s Product Cycle Microsoft’s Product Cycle Model Model

Microsoft Product Cycle Microsoft Product Cycle ModelModel All teams follow this processAll teams follow this process

Implementation of the process variesImplementation of the process varies

Phases sometimes overlapPhases sometimes overlap Could have multiple cycles, in Could have multiple cycles, in

different stages, going in paralleldifferent stages, going in parallel Ideally All Teams Enter Cycle TogetherIdeally All Teams Enter Cycle Together Reality is That They Often Don’tReality is That They Often Don’t

SchedulingScheduling Milestone = unit of product cycle progressMilestone = unit of product cycle progress

Goal: Enable flexible Goal: Enable flexible planning/tracking/stabilization/feedbackplanning/tracking/stabilization/feedback

Milestone schedule: M0, M1…, Beta1, Beta2, RTMMilestone schedule: M0, M1…, Beta1, Beta2, RTM Enable accurate view of progress and distance leftEnable accurate view of progress and distance left

Intended to ensure that team is never “flying blind”Intended to ensure that team is never “flying blind” Features are scheduled in milestones in priority orderFeatures are scheduled in milestones in priority order

Enables schedule flexibility to respond to feedback Enables schedule flexibility to respond to feedback in later milestonesin later milestones

Milestone only “done” when quality “exit criteria” is Milestone only “done” when quality “exit criteria” is metmet Forcing function so team doesn’t go fast and looseForcing function so team doesn’t go fast and loose Ensure very stable, usable, complete product at Ensure very stable, usable, complete product at

milestone exitsmilestone exits

Development MilestonesDevelopment Milestones

Each is a Mini-ReleaseEach is a Mini-Release CodingCoding

Fixed Amount of TimeFixed Amount of Time Code Complete DateCode Complete Date

IntegrationsIntegrations Team code changesTeam code changes External team changesExternal team changes

A Linear View of the PCMA Linear View of the PCM

Plan, Design M1 M2 M3 Stabilize

KickoffCode

Complete RTM/RTWImplementation

Starts

Key Feature Milestone EventsKey Feature Milestone Events

Milestone EventMilestone Event DefinitionDefinition

Spec CompleteSpec Complete Date design specs for milestone features written & Date design specs for milestone features written & reviewedreviewed

Feature CodingFeature Coding Typically 8-9 weeks in length for feature milestonesTypically 8-9 weeks in length for feature milestones

Code Complete (CC)Code Complete (CC) Date all features scheduled for the milestone are Date all features scheduled for the milestone are finishedfinished

Test Plan Complete Test Plan Complete Test plans for all milestone features written reviewedTest plans for all milestone features written reviewed

ZBB Test Pass (ZBB TP)ZBB Test Pass (ZBB TP) All existing functional tests are run on current buildsAll existing functional tests are run on current builds

Zero Bug Bounce (ZBB)Zero Bug Bounce (ZBB) # of milestone bugs older than 48 hours = 0# of milestone bugs older than 48 hours = 0

Zero Resolved Bugs (ZRB)Zero Resolved Bugs (ZRB) # of resolved milestone bugs to be verified before # of resolved milestone bugs to be verified before closing = 0closing = 0

Test Sign-OffTest Sign-Off Final verification and media sign-off on the milestone Final verification and media sign-off on the milestone buildbuild

How do we define a How do we define a feature?feature?

Must be scheduled and testable!Must be scheduled and testable! Specing is reasonably balancedSpecing is reasonably balanced

PMs usually write design specPMs usually write design spec Devs write implementation specsDevs write implementation specs QA writes testing specQA writes testing spec

Performance goalsPerformance goals

Design SpecsDesign Specs Never write production code without designing up-Never write production code without designing up-

frontfront Excellent rule when working on a project just by yourselfExcellent rule when working on a project just by yourself Absolutely mission critical in a team projectAbsolutely mission critical in a team project

Feature-set driven by Program Managers at MicrosoftFeature-set driven by Program Managers at Microsoft Own the customer experience and make right thing happenOwn the customer experience and make right thing happen Drive leading feature team (PM, Dev, Test) during design Drive leading feature team (PM, Dev, Test) during design

processprocess Own writing the final design specification for each featureOwn writing the final design specification for each feature

A good feature design specification is:A good feature design specification is: Clear about the goals and non-goals of a featureClear about the goals and non-goals of a feature Clear about how feature is used by customers and partnersClear about how feature is used by customers and partners Precise about object model and architecture design of featurePrecise about object model and architecture design of feature Clear enough for separate developer, test, documentation and Clear enough for separate developer, test, documentation and

localization team resources to deliver it together without localization team resources to deliver it together without ambiguityambiguity

Example Design Spec Example Design Spec

CodingCoding Developers at MS use a variety of different tools for Developers at MS use a variety of different tools for

codingcoding Visual Studio, Emacs, VI, Source Insight, etc.Visual Studio, Emacs, VI, Source Insight, etc. No tool-specific files or code-injection allowed to be checked-inNo tool-specific files or code-injection allowed to be checked-in

Maintain single source tree for entire product lineMaintain single source tree for entire product line Can type “build -c” from root to build CLR, ASP.NET, .NET FX, Can type “build -c” from root to build CLR, ASP.NET, .NET FX,

Visual Studio and setup programs from scratch (takes 9+ Visual Studio and setup programs from scratch (takes 9+ hours)hours)

Can type “build -c” in any sub-directory of source tree to only Can type “build -c” in any sub-directory of source tree to only build/re-build a sub-portion of product (99% scenario)build/re-build a sub-portion of product (99% scenario)

Each team maintains a separate private “branch” off Each team maintains a separate private “branch” off main treemain tree Use internal source control system optimized for Use internal source control system optimized for

branching/mergingbranching/merging Minimizes check-ins into main tree and avoids build breaksMinimizes check-ins into main tree and avoids build breaks

CodingCoding C++ and C# the two common languages used C++ and C# the two common languages used

in Microsoftin Microsoft Emphasize efficient, clean, maintainable codeEmphasize efficient, clean, maintainable code

Avoid hacks, messy tricks, and stupid Avoid hacks, messy tricks, and stupid optimizationsoptimizations

Code we ship lives a minimum of 10 years Code we ship lives a minimum of 10 years (guaranteed support)(guaranteed support)

Documentation stored externally from source Documentation stored externally from source codecode Avoid documentation writers colliding with Avoid documentation writers colliding with

developersdevelopers All resource strings stored externally from All resource strings stored externally from

source codesource code Visual Studio localized into 8 languagesVisual Studio localized into 8 languages

CodingCoding All code check-ins must be code-reviewed by another All code check-ins must be code-reviewed by another

developer on the team prior to submitting any developer on the team prior to submitting any changes to the source treechanges to the source tree Applies to both the most junior and most senior Applies to both the most junior and most senior

member of the dev teammember of the dev team Dev responsible for check-in suites to exercise Dev responsible for check-in suites to exercise

implemented featuresimplemented features Moving to model where features can’t be checked-Moving to model where features can’t be checked-

in without 60% block-level code-coverage of all new in without 60% block-level code-coverage of all new code from developer written unit-testscode from developer written unit-tests

All product check-in suites must run and pass 100% All product check-in suites must run and pass 100% before any check-in can be submitted into source tree before any check-in can be submitted into source tree (2-3 hour process)(2-3 hour process) Applies both to code you’ve touched, and code you Applies both to code you’ve touched, and code you

haven’t touchedhaven’t touched ““Check-in mail” email sent to team after each submit Check-in mail” email sent to team after each submit

from the buddy machine summarizing the changes from the buddy machine summarizing the changes made, bugs fixed, and files modifiedmade, bugs fixed, and files modified

Producing Daily BuildsProducing Daily Builds A new “build” of the product is built and released every dayA new “build” of the product is built and released every day

Force engineering discipline, critical in the product end-gameForce engineering discipline, critical in the product end-game Build number indicates build date (40730 = July 30Build number indicates build date (40730 = July 30thth, 40810 = August , 40810 = August

1010thth)) Central build lab produces daily builds for entire division (~2000 Central build lab produces daily builds for entire division (~2000

people)people) Staffed 24 hours a day throughout the weekStaffed 24 hours a day throughout the week Each product team must have a build facilitation developer (BFD) on Each product team must have a build facilitation developer (BFD) on

call with beeper to help with any build problems with that productcall with beeper to help with any build problems with that product Build process:Build process:

Build team syncs source code tree (~midnight)Build team syncs source code tree (~midnight) Kick off end-to-end build (~4:00am)Kick off end-to-end build (~4:00am) Build complete (~1:00pm)Build complete (~1:00pm) Perform BVT (Build Verification Tests) run to verify build works Perform BVT (Build Verification Tests) run to verify build works

(~4:00pm)(~4:00pm) Pick up hot-fixes from BFD coordinator and re-build for BVT failuresPick up hot-fixes from BFD coordinator and re-build for BVT failures Repeat hotfix/BVT cycle until the build passes clean Repeat hotfix/BVT cycle until the build passes clean

What a build process!What a build process!

Lab Automation MachinesLab Automation Machines

TestingTesting Test team staffed by devs who are responsible for designing test plans, Test team staffed by devs who are responsible for designing test plans,

writing automated tests, and building the test infrastructurewriting automated tests, and building the test infrastructure Focus on driving up quality, preventing regressions, and enabling Focus on driving up quality, preventing regressions, and enabling

rapid analysis of different builds, variations, and language releasesrapid analysis of different builds, variations, and language releases Current Whidbey Test Status:Current Whidbey Test Status:

102,000 Functional Test Cases102,000 Functional Test Cases 505,000 Functional Test Scenarios505,000 Functional Test Scenarios 71 Stress Mix Variations71 Stress Mix Variations ~1000 servers in test lab to run all of this in an automated way~1000 servers in test lab to run all of this in an automated way

Internal test management system stores and manages test plans/runsInternal test management system stores and manages test plans/runs Allows user to easily add/remove/analyze test plans and casesAllows user to easily add/remove/analyze test plans and cases Allows user to remotely re-image machines within the labAllows user to remotely re-image machines within the lab Allows user to remotely kick off a test-run on a suite of lab Allows user to remotely kick off a test-run on a suite of lab

machines machines Allows user to remotely analyze results of test-runs and publish Allows user to remotely analyze results of test-runs and publish

resultsresults

Maintain ControlMaintain Control

TestingTesting Test plans are first step of the QA processTest plans are first step of the QA process

Documents completed before milestone exit Documents completed before milestone exit detailing all testing variations needed to cover a detailing all testing variations needed to cover a feature areafeature area

Target goal with tests is to reach > 85% product Target goal with tests is to reach > 85% product code coverage by end of the product cyclecode coverage by end of the product cycle We measure this statistic throughout the productWe measure this statistic throughout the product

Always on the lookout for “test holes”Always on the lookout for “test holes” When a bug is opened, the tester for that area When a bug is opened, the tester for that area

always makes sure that there is a corresponding always makes sure that there is a corresponding test-case to prevent it in the futuretest-case to prevent it in the future

Regular automated test runs catch regressionsRegular automated test runs catch regressions Rough metric is 1 regression per 3-6% of bug Rough metric is 1 regression per 3-6% of bug

fixesfixes

Bug TrackingBug Tracking Bugs and work-items tracked within internal bug-Bugs and work-items tracked within internal bug-

systemsystem Enables rich reporting and change history tracking on each Enables rich reporting and change history tracking on each

itemitem All MS products in same bug system (enables linking bugs)All MS products in same bug system (enables linking bugs)

Bugs “triaged” by feature leads and assigned priorityBugs “triaged” by feature leads and assigned priority Priority 0, 1, 2, 3, 4Priority 0, 1, 2, 3, 4 Bugs fixed in priority orderBugs fixed in priority order

Daily status mails sent to entire division tracking bug Daily status mails sent to entire division tracking bug statusstatus Key metrics to watch: incoming and resolve numbers + bugs Key metrics to watch: incoming and resolve numbers + bugs

per devper dev

After the final feature-milestone is completed, life for After the final feature-milestone is completed, life for the product team revolves around driving the numbers the product team revolves around driving the numbers to zeroto zero

Bug QueryBug Query

Bug DetailBug Detail

Ship Mode Glide PathShip Mode Glide Path Large projects “glide in” as opposed to end suddenlyLarge projects “glide in” as opposed to end suddenly

Like a super-tanker -- you can’t dock the ship abruptlyLike a super-tanker -- you can’t dock the ship abruptly Critical to get real-world feedback on product quality earlyCritical to get real-world feedback on product quality early

MS teams typically hold two public betas prior to RTM releaseMS teams typically hold two public betas prior to RTM release Many builds given to key partnersMany builds given to key partners

Key set of steps along the “glide path” to the “end-game”:Key set of steps along the “glide path” to the “end-game”: 1) Lock the feature-set and stop adding/changing design of 1) Lock the feature-set and stop adding/changing design of

featuresfeatures 2) Complete a full test pass to find all bugs in the locked 2) Complete a full test pass to find all bugs in the locked

designdesign 3) Push for a zero bug bounce (ZBB) on the locked design3) Push for a zero bug bounce (ZBB) on the locked design 4) Absorb bug-bounce over few weeks from ZBB4) Absorb bug-bounce over few weeks from ZBB 5) Triage all non-must fix bugs out of the system5) Triage all non-must fix bugs out of the system 6) Enter the “end-game” mode and start minimizing code 6) Enter the “end-game” mode and start minimizing code

churnchurn

StabilizationStabilization

Major Feature Work CompleteMajor Feature Work Complete Dependencies SatisfiedDependencies Satisfied

More Complex = More Time to StabilizeMore Complex = More Time to Stabilize

Focus on Fixing BugsFocus on Fixing Bugs Design Change Process in PlaceDesign Change Process in Place Justify Breaking ChangesJustify Breaking Changes Track Bug TrendsTrack Bug Trends

Incoming and Resolve RatesIncoming and Resolve Rates Trends Towards ZBB, ZRBTrends Towards ZBB, ZRB

ValidationValidation

Don’t “Bounce” Too HighDon’t “Bounce” Too High Focus is on Running Through Test CasesFocus is on Running Through Test Cases

Major Comprehensive “Test Passes”Major Comprehensive “Test Passes” Regression TestingRegression Testing Compatibility TestingCompatibility Testing

Visibility of Prerelease to CustomersVisibility of Prerelease to Customers Beta Customer ListsBeta Customer Lists Beta KitsBeta Kits Support for Beta CustomersSupport for Beta Customers If Final Release, RC Sign-OffsIf Final Release, RC Sign-Offs

End-Game ModeEnd-Game Mode As a product enters the “end-game” mode its “war team” takes overAs a product enters the “end-game” mode its “war team” takes over

War Team is made up of most senior team members – all with ship War Team is made up of most senior team members – all with ship experienceexperience

War Team makes final end-game calls and guides the ship in into portWar Team makes final end-game calls and guides the ship in into port War Team meets at least once a day – twice a day (or more) at the very endWar Team meets at least once a day – twice a day (or more) at the very end

Over the span of the final weeks before a release, the war team will Over the span of the final weeks before a release, the war team will steadily raise the “triage bar” and towards the end only allow steadily raise the “triage bar” and towards the end only allow “showstopper” bugs to be fixed“showstopper” bugs to be fixed If product is ready to ship, the # of showstopper bugs will slowly decline as If product is ready to ship, the # of showstopper bugs will slowly decline as

check-in rates dropcheck-in rates drop This is where shipping becomes less of a science and more of an art – This is where shipping becomes less of a science and more of an art –

the senior leaders need to “feel” that the product is right, have the senior leaders need to “feel” that the product is right, have confidence in the test coverage, the customer scenarios, and the confidence in the test coverage, the customer scenarios, and the overall teamoverall team

Once the rate of incoming bugs that stick gets to zero, we put the build Once the rate of incoming bugs that stick gets to zero, we put the build in “escrow” for a few days and wait and see. If it holds, ship it. If not, in “escrow” for a few days and wait and see. If it holds, ship it. If not, repeat until finally there.repeat until finally there.

Golden Rule of Commercial Software:Golden Rule of Commercial Software: ““2 years from now customers will not remember if you shipped a good 2 years from now customers will not remember if you shipped a good

product 2 months late. But they will absolutely remember if you shipped a product 2 months late. But they will absolutely remember if you shipped a poor quality product 2 months early.”poor quality product 2 months early.”

End GameEnd Game

Allow “Bake Time” After ValidationAllow “Bake Time” After Validation Focus on Finding “Ship-Stoppers”Focus on Finding “Ship-Stoppers”

Daily MeetingsDaily Meetings ““Ship-room”, “Tactics”, or “War-room”Ship-room”, “Tactics”, or “War-room” At Product Team LevelAt Product Team Level At Division LevelAt Division Level

Triage Late Incoming Critical IssuesTriage Late Incoming Critical Issues ““Bug Bars”Bug Bars” Manage Risk – Minimize ChurnManage Risk – Minimize Churn

Declare Release CandidatesDeclare Release Candidates Drive Product Sign-offDrive Product Sign-off

When to Sign-OffWhen to Sign-Off No Active “Ship-Stoppers”No Active “Ship-Stoppers” Meet Exit CriteriaMeet Exit Criteria

Key Scenarios Meet Quality ExpectationsKey Scenarios Meet Quality Expectations Product Features Meet Quality ExpectationsProduct Features Meet Quality Expectations Product Meets Release RequirementsProduct Meets Release Requirements

Security, Branding, Legal, Globalization, Security, Branding, Legal, Globalization, Logo, Etc.Logo, Etc.

Complete Test Coverage RequirementsComplete Test Coverage Requirements Automated and Manual TestingAutomated and Manual Testing All Supported PlatformsAll Supported Platforms All Supported LanguagesAll Supported Languages

Product Pre-ReleasesProduct Pre-Releases ““Practice” ShippingPractice” Shipping

Surface Any Late Integration IssuesSurface Any Late Integration Issues Shutting Down the ProductShutting Down the Product

““Practice” ServicingPractice” Servicing Setup Technology ChangesSetup Technology Changes Another Area of TestingAnother Area of Testing

Getting the Product in Customers’ Getting the Product in Customers’ HandsHands Proof of ConceptProof of Concept Real World CodeReal World Code User FeedbackUser Feedback

© 2003-2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.