1

Developing NuGetJEFF HANDLEY | DEVELOPMENT LEAD | NUGET | MICROSOFT

@JEFFHANDLEYJEFFHANDLEY.COM

Source: www.imdb.com 2

Airborne (1993)

3

4

5

6

NuGet LandscapeWHAT MAKES UP THE NUGET PROJECT

7

NuGet Project Open Source (owned by Outercurve Foundation)

◦ Releases shipped by Microsoft

Included in all Visual Studio installations

Millions of users (some knowingly, some not) around the world

Dozens of high-profile dependent projects at Microsoft

International contributors◦ Only a few external contributors are in the US

Very large backlog of issues

Conflicting requirements

Backward compatibility

Mixture of hard and soft deadlines

Extremely costly test passes

8

Outercurve and MicrosoftNUGET CLIENT

Outercurve’s NuGet Project

Microsoft’s NuGet-Based Package

Manager

In the Box with Visual Studio

NUGET GALLERY

Outercurve’s NuGet Gallery Projectwww.nuget.org

Azure Hosting provided by

Microsoft

Operated by Microsoft on behalf

of Outercurve

9

NuGet’s Core TeamNUGET CLIENT

Visual Studio Extension

WebMatrix Extension

NuGet.exe Command-Line

NuGet Packages

3 Developers

2 Testers

NUGET GALLERY

www.nuget.org

staging.nuget.org

docs.nuget.org

build.nuget.org

3 Developers

2 Testers

Always 1 member on ops/support rotation

10

NuGet’s Core TeamNUGET CLIENT

Visual Studio Extension

WebMatrix Extension

NuGet.exe Command-Line

NuGet Packages

3 Developers

2 Testers

NUGET GALLERY

www.nuget.org

staging.nuget.org

docs.nuget.org

build.nuget.org

3 Developers

2 Testers

Always 1 member on ops/support rotation

11

Release ManagementNUGET CLIENT

12

NuGet Releases

New release every 10 weeks since NuGet 1.5

2/12/20111.1

3/30/20111.2

4/25/20111.3

6/17/20111.48/30/2011

1.5

12/13/20111.6 4/4/2012

1.7

5/23/20121.8 6/19/2012

2.0

10/4/20122.1 12/12/2012

2.2

2/15/20132.2.1 4/25/2013

2.5

6/26/20132.6 8/22/2013

2.7

10/7/20132.7.1

1/14/20111.0

13

NuGet in Visual Studio Releases

2/12/20111.1

3/30/20111.2

4/25/20111.3

6/17/20111.48/30/2011

1.5

12/13/20111.6

Introducedwith

ASP.NETMVC 3and

Web Pages v1

4/4/20121.7

5/23/20121.8 6/19/2012

2.0

10/4/20122.1 12/12/2012

2.2

2/15/20132.2.1 4/25/2013

2.5

6/26/20132.6

8/22/2012Visual Studio2012 RTM

4/4/2013Visual Studio

2012 Update 26/26/2013Visual Studio2013 Preview

9/9/2013Visual Studio

2013 RC

1/14/20111.0

5/31/2012Visual Studio

2012 RC

2/28/2012Visual Studio2012 Beta

8/22/20132.7

10/7/20132.7.1

Visual Studio2013 RTM

14

Release Tiers CI build from Outercurve Foundation always available at http://build.nuget.org

◦ Usually 100 or fewer users◦ Build created from every commit

Beta/RC builds from Microsoft occasionally available at http://nuget.codeplex.com◦ 1,000 – 2,500 users◦ Published after we hit code-complete◦ Real-world testing of newly introduced features or significant design changes

Final builds from Microsoft published to the Visual Studio Extension Gallery◦ 500,000 – 750,000 users◦ Released after our test team signs off, a few weeks before Visual Studio locks down for its next release◦ Last stage real-world testing from users that are knowingly using NuGet and are a bit more forgiving

In the box with Visual Studio◦ Millions of users◦ Many of these users don’t know what NuGet is and will file Visual Studio bugs if we break them

15

Backlog CodePlex: 858 open issues as of 10/4/2013

Microsoft Internal Issues◦ TFS: Fluctuates from 0 to 25

◦ Dr. Watson reports◦ Microsoft Connect◦ Microsoft shipping requirements◦ Future Visual Studio release issues

Strategic Product Features◦ Schedule alignment for release within Visual Studio and Visual Studio Update

16

Strategic Product Features Package Manager for the Microsoft Development Platform

◦ Not just ASP.NET◦ Not just .NET◦ Includes C++, Windows Store, and Windows Phone◦ Cross-platform support with Command-Line, Xamarin Studio, MonoDevelop, and SharpDevelop

Improving Package Discovery (through collaboration with the NuGet Gallery crew)

Avoiding Package Version Hell

Platform Multi-Targeting and Re-Targeting

New Usage Workflows for Improved Application Maintenance and Ecosystem Growth

Helping Improve Gallery Reliability

17

Triaging Issues1. Is it a high-priority issue (crash, hang, regression)?

◦ Include it in the current release

2. Is the bug fix or feature needed for the current release?◦ Include it in the current release

3. Is it a high value item that fits in the next release?◦ Include it in the next release

4. Do we agree with the concept?◦ Put it in the “Up For Grabs” release◦ Ideally picked up by the community◦ High-voted items are reviewed when planning releases and some items are pulled in

5. Do we disagree with the concept?◦ Close it

18

Priorities1. Strategic roadmap that integrates with the Microsoft development platform

◦ Features planned for 2-3 releases at a time◦ Align with Visual Studio releases: Preview, Beta, RC, RTM, Visual Studio Update

2. Community-reported high-value bug fixes or features◦ Including those that come with external pull requests

3. Highly-voted community feature requests

4. Community-reported high-value bug fixes

5. Microsoft partner team bug reports

6. Microsoft partner team feature requests

7. Community-reported low-value bug fixes

19

Philosophy Small and easy bug fixes are low priority

◦ Otherwise, we’d spend 100% of our time making small changes that affect only a handful of users

Don’t prematurely act upon cool ideas◦ Does the idea spark new strategic concepts?◦ Do other new ideas relate?◦ Can the feature be implemented with backwards compatibility?◦ Could this feature become obsolete based on other potential strategic directions?

Favor community-requested features over partner team features◦ Avoids scenario-specific features that conflict with broad usage◦ Encourages partner teams to find workarounds with existing features◦ Those workarounds often surface pain points others might also be experiencing

We’d rather delight 1,000,000 external users than gain 1 more internal partner team

20

Conflicting Requirements Microsoft’s Customers vs. NuGet’s Ecosystem

◦ Group Policy control over package sources

Bold New Features vs. Backwards Compatibility

◦ Package Restore◦ Build-time reference resolution

NuGet User Experience vs. Microsoft’s shipping requirements

◦ Online consent◦ Loading only Microsoft-signed assemblies◦ Modified content files

Favor user desires as much as possible

Work within the constraints of Microsoft’s shipping requirements

Negotiate and compromise with privacy, security, and legal representatives to arrive at acceptable implementations

21

Source ControlNUGET CLIENT

22

master

mer

ge

FeatureA

FeatureB

Branch: ͞S2.7͞T

Branch: ͞S2.7.1͞T

mer

gem

erge

Branch: ͞S2.8͞T

Tag: ͞S2.7͞T

Tag: ͞S2.7.1͞T

Git

Branch per feature Branch per version Tag for release

23

Continuous Integration Outercurve Builds

◦ TeamCity – running on Virtual Machines in Windows Azure◦ 1 VM build agent with .NET 4.0 for building the VS Extension for VS 2010 and VS 2012◦ 1 VM build agent with .NET 4.5 for building the VS Extension for VS 2013

◦ Hosted at http://build.nuget.org◦ Produces non-localized, unsigned builds with Outercurve branding and licensing◦ “Developer Branches” build configuration builds all “dev-” prefixed branches

Microsoft Builds◦ TeamCity – running on Virtual Machines in a group lab

◦ 1 VM build agent with .NET 4.0 for building the VS Extension for VS 2010 and VS 2012◦ 1 VM build agent with .NET 4.5 for building the VS Extension for VS 2013

◦ Only accessible on the corporate network◦ Produces localized, signed builds with Microsoft branding and licensing

24

Going Dark Microsoft-confidential work cannot go to CodePlex

◦ Integration with new versions of Visual Studio◦ Support for new platforms (e.g. Windows Phone 8)

Team Foundation Service private project◦ Fork the NuGet Git repo into that project◦ Move confidential development into that fork◦ Keep all other development on CodePlex

Merge commits into TFS daily

Configure our Internal Builds to use the TFS fork instead of CodePlex

25

TestingNUGET CLIENT

26

Release Timing Release sign-off takes about 4 weeks after code-complete

◦ 40% of our 10-week release cadence is spent testing and fixing bugs

Snap releases to the Visual Studio schedule

Release to the gallery first, and then integrate into Visual Studio’s build

Between Visual Studio Beta and RTM, we have 2 or 3 tight quality-driven releases◦ Meet Microsoft’s release requirements◦ Address any partner team blocking issues◦ Focus on extremely broad use of the product

After Visual Studio hits RTM, we go big with features and aim for a 3-month release

27

Test Matrix Dimensions Visual Studio SKUS

◦ VS 2010◦ Express for Web◦ Express for Phone◦ Pro+

◦ VS 2012+ - All SKUs

Project Types◦ Virtually all Microsoft Project Types

◦ Lightswitch◦ Windows Store◦ Windows Phone◦ Portable Class Libraries

Languages: C#, VB, F#, JavaScript

Operating Systems◦ Windows XP◦ Windows Server 2003+◦ Windows Vista and Windows 7◦ Windows 8◦ Windows 8.1

Different types of NuGet Packages

Different types of NuGet Package Sources◦ Gallery◦ NuGet.Server◦ Local Disk◦ UNC Share

Source Control Integration◦ TFS◦ Git, Mercurial◦ None

NuGet Clients◦ VS Extension◦ WebMatrix Extension◦ NuGet.exe Command-Line

Plus, all of the combinations of actual features

28

Achieving Release Sign-Off Unit Tests managed and run by developers

End-to-end Functional tests managed and run by developers and through automation◦ Based on a PowerShell test system included within our project◦ Runs within Visual Studio, automating VS through the DTE

QA Tests for interactive testing◦ Mostly automated, but exploratory testing always finds more bugs◦ Run across a farm of machines in our group’s test lab, covering the machine configuration matrix◦ A representative set of combinations give us confidence◦ Uses an internal VS automation library to simulate user actions in VS

Group-wide bug bashes

Company-wide partner team bug bashes

Public Beta testing from CodePlex builds

Reach 0 bugs in CodePlex for the current release

29

Throughput109 Weeks•Total time working on those 10 releases

10 Releases•816 work items have been completed

3 Developers•Averaging 27 work items per release

2 Testers•Each testing 3.7 bug fixes or features per week

30

Success Never defined by throughput

No regressions

No minor updates necessary

Users delighted by new features

Took steps toward our strategic roadmap

Addressed highly-voted community bugs and features

Accepted a level of failure in smaller user base releases

31

Success Rate NuGet 1.6

◦ Required a 1.6.1 update◦ Regressions◦ Missed bugs in new features

NuGet 1.7◦ Success

NuGet 1.8◦ Success

NuGet 2.0◦ Success

NuGet 2.1◦ Success

NuGet 2.2◦ Success

NuGet 2.5◦ Success

NuGet 2.6◦ Success

NuGet 2.7◦ Required a 2.7.1 update◦ Regression◦ Missed bugs in new features

32

Tiered Releases are Key

Our tiered release approach worked for 2.7

2.7.1 update in place in time for integrating into Visual Studio

Thousands of users affected instead of millions

As Visual Studio’s broad user base jumps from NuGet 2.2 to NuGet 2.7, they’ll have no idea we ever introduced regressions

You wouldn’t have noticed the camera in Airborne if I hadn’t pointed it out to you

33

Q & A

Get a NuGet Sticker!

Twitter: @jeffhandley

Website: jeffhandley.com