netflix oss meetup season 4 episode 4
TRANSCRIPT
Netflix Open SourceSeason 4, Episode 4
Introduction
Why does Netflix Open Source?
Improve Engineering● Great feedback from wider community● Collaborate through open code
Recruit new and retain engineering talent● Hard problems are openly worked on
Industry Alignment
Why does Netflix Open Source?
Netflix movesto cloud
2008
2013
2016
Netflix Github -http://netflix.github.io
Netflix’s approach to open source
Form a small cross-functional team working group that centralizes OSS competence, assisting decentralized teams working with OSS spend less time focusing on the administrative aspects (legal, tooling, branding, monitoring, and community promotion).
Open source enabler - OSS Interest Group
● Internal mailing list● Meets once per month● Topics from developers● Help each other with
common problems
Agenda
Assisting open source at Netflix● Github management and security● Build, CI and release tools
Engaging in open source● Transparency & OSS maturity● Fostering communities
@rusmeshenberg
@SonOfGarr
@rfletcherEW
mikegrima
Agenda
Open Source Offices and the TODO Group
Nithya A. RuffDirector, Open Source Office, Western Digital
@nithyaruff
Github Management & SecurityMike Grima
Security Tools - Sensitive Data Leakage
● We scan source code for○ Access keys, passwords, tokens,
hostnames
● We scan code automatically and frequently
Scumblr
Security Tools - User Management
● Provide tools and automation for user access○ Adding / Removing users○ Performing privileged tasks
● We aim for self-service as much as possible!
Github Organizational Management
● Management must be easy○ Otherwise, teams will go it alone (BAD)
● Less is more: fewer orgs = Good
Github Organizational ManagementBYOGID:● User links to internal ID● All tools then can
associate identity
Two Factor Auth Enforcement● Automation to boot users who don’t● Be careful - education on recovery!
Github Organizational Management
● Owners○ Limited group - due to power○ Broker owner actions via ChatOps
● Netflixer group○ Full write permissions on all repos
● Outside collaborators○ Added by netflixers, validated over time
ChatOps for GitHub Management
ChatOps for GitHub Management
ChatOps for GitHub Management
More advanced commands & DUO!
Building Netflix OSSMike McGarr
OSS builds needs...
● Consistent build automation● Continuous integration● Release software versions● Publish to JCenter and Maven central● Validate license files● Simple and consistent
http://nebula-plugins.github.io
build.gradle file
● Add contacts to build.gradle● Bundle build metadata into .jar ● Publish .jar files to Bintray● Git tag to build release versions● Optional: lock dependencies
Reduce boilerplate
● Jenkins on Cloudbees● Setup builds with the Job DSL plugin
Continuous integration
● What we didn’t like?○ Lacked declarative builds○ Config not in source○ Complex setup○ Not as common in OSS community
Travis CI
All commits to master./gradlew -Prelease.travisci=true build snapshot
Releases w/ Nebula + TravisCI
Every pull request./gradlew build
Commit tagged with vX.Y.Z-rc#./gradlew -Prelease.travisci=true candidate
Commit tagged with vX.Y.Z./gradlew -Prelease.travisci=true final
https://jcenter.bintray.com
Distributing the binaries
What happens when...
Backup source Netflix OSS
● Backup Github repos to internal Git
git remote add internal \ ssh://[email protected]/foo/bar.git
Transparency and OSS MaturityRuslan Meshenberg@rusmeshenberg
4 seasons of NetflixOSS
Many OSS Projects
In 4 years
All wildly successful?
Some - yes
Some… not so much
What are some of the challenges?
Lack of OSS transparency / direction
What are some of the challenges?
Internal / OSS Divergence
What are some of the challenges?
Maturing and EOL of projects
What are some of the challenges?
Separating ideas from code
What are some of the challenges?
All leading to variable levels of support
What are we doing about it?
Data to the rescue!
Org Health TrackingProject Health TrackingBacklog of PRs and Issues
Overall Org Health Tracking
Metrics we track
● Issues○ open, closed, TTC
● Pull Requests○ open, closed, TTC
● Last commit timing● Stars/forks● Num contributors
Project Health Tracking
github.com/Netflix/
OSSTracker
Transparency about project lifecycle
OSSMETADATA file:
● Active● Maintenance● Archived
Project Ownership
All projects have● Development lead, Management lead● Shepherd from OSS function areaOnly projects with active leads stay active!
Transparency about project evolution
Transparency about project evolution
Converging internal and OSS
Less maintenance for us
You get exactly what we use
Fostering communitiesRob Fletcher
What’s in it for us?Tangible contributions
● Leverage enhancements made for other cloud providers
● Titus integration made easier● Role-based authentication
Intangible benefits
● Influencing the conversation● Validation of concept & implementation● Recruitment● Retention
Spinnaker contributions — Clouddriver
Spinnaker contributions — Orca
Encouraging engagementContributors
● Public roadmap● “no” > ignoring people● We can’t do everything — encourage
contributions ● Review community PRs & issues regularly● Make the 1st step easy● Don’t let docs mislead
Open tools
● GitHub● Slack● Readme.io● Travis CI
You gotta do it every day…that’s the hard part
TODO GroupNithya A. Ruff
• Many of us who ran open source program offices shared a private mailing list to commiserate…
• It was an avenue to discuss issues in private and even find ways to collaborate on open source projects…
• Focused on Silicon Valley companies initially
• In 2014 we had an idea of scaling and opening up the the community more…
• Announced the TODO Group @Scale 2014 conference!
• TODO Group is a group of companies who want to– collaborate on best practices on running open source program
offices– share open source policies and training material– codify quality criteria for well-run open source projects and
communities– build and share tools to maintain those quality standards
• As we scaled our open source programs, we realized we all built similar tools for the purposes of corporate scale open source…
• What is corporate scale open source?
• Corporate participants in open source have a number of unique concerns:– scale (i.e., Google and Microsoft have many open source projects)– insights– cultural– legal / governance
• Companies doing open source generally want to be good community citizens, to be open and inclusive, to operate meritocracies. They also need to run a business and be aware of responsibilities to their employees, shareholders and the broader community.
• To establish the TODO Group as a legitimate legal entity, we partnered with the Linux Foundation to make the TODO Group an official collaborative project!
• The LF helps with legal paperwork, running events and gives the TODO Group access to its 650+ members
Western Digital and Open Source
• Started the Open Source office at SanDisk to engage with the community and go past consumption and compliance
• Branded it Open @ SanDisk and became a visible supporter of events and communities
• Increased contribution and competency inside the company around open source development models
• With the acquisition by Western Digital, created a single office across WD, HGST and SanDisk
Demo Stations
Conductor(workflow)
Questions?