Delivering Developer Tools at ScaleJoe Levy (@jodoglevy) – Principal Technical Program Manager

Mike Ross – Senior Software Engineer

Oracle Bare Metal Cloud Services, Developer Experience

Oracle Bare Metal Cloud Services

• Infrastructure as a service (IaaS) Compute (physical and virtual machines)

Block Storage

Virtual Networking

Identity

Load Balancing

Object Storage

Audit

Database

• Built by ex-cloud folks

• More info: https://cloud.oracle.com/en_US/bare-metal

DemoBare Metal Cloud Services

March 2015

Developer Experience:Support devs where they are (languages)

Developer Experience:Support devs where they are (tools)

https://www.linux.com/news/top-open-source-cloud-projects-2014

Developer Experience (aka fill all the boxes)Compute Block

Storage

Virtual

Networking

Load

Balancing

Object

Storage

Audit Database

Java

SDK

Python

SDK

Ruby

SDK

CLI

HDFS

Connector

Chef Knife

Plugin

Terraform

Provider

Developer Experience: Progress to dateCompute Block

Storage

Virtual

Networking

Load

Balancing

Object

Storage

Audit Database

Java

SDK

In preview

Python

SDK

N/A

Ruby

SDK

In preview N/A

CLI N/A In preview

HDFS

Connector

N/A N/A N/A N/A N/A N/A

Chef Knife

Plugin

N/A N/A N/A N/A

Terraform

Provider

N/A

Developer Experience: Progress to date

Boxes: 7 dev tools X 7 services = 49

Team: 6 devs

Developer Experience: The futureCompute Block

Storage

Virtual

Networking

Load

Balancing

Object

Storage

Audit Database

Java

SDK

In preview

Python

SDK

N/A

Ruby

SDK

In preview N/A

CLI N/A In preview

HDFS

Connector

N/A N/A N/A N/A N/A N/A

Chef Knife

Plugin

N/A N/A N/A N/A

Terraform

Provider

N/A

Fortran

SDK

Logo

SDK

…

Developer Experience: The future

Boxes: N dev tools X 7 services = 7N

Team: 6 devs

Developer Experience: The future (for real)Compute Block

Storage

Virtual

Networking

Load

Balancing

Object

Storage

Audit Database Puppies

as a

service

Haircuts

as a

service

…

Java

SDK

In preview

Python

SDK

N/A

Ruby

SDK

In

preview

N/A

CLI N/A In preview

HDFS

Connector

N/A N/A N/A N/A N/A N/A

Chef Knife

Plugin

N/A N/A N/A N/A

Terraform

Provider

N/A

Fortran

SDK

Logo

SDK

…

Developer Experience: The future (for real)

Boxes: N dev tools X M services

Team: 6 devs

Developer Experience: The future(for real, for real)

• Every service is actually a collection of many resources

• More resources added to each service over time, all needing dev tools support

• VCN service resources:

• Cpe

• CrossConnect

• CrossConnectGroup

• CrossConnectMapping

• CrossConnectLocation

• CrossConnectPortShapeSpeed

• CrossConnectStatus

• DhcpOption

• Drg

• DrgAttachment

• EgressSecurityRule

• FastConnectProviderService

• IcmpOptions

• IngressSecurityRule

• InternetGateway

• IpSecConnection

• PortRange

• RouteRule

• RouteTable

• SecurityList

• Subnet

• TcpOptions

• TunnelConfig

• TunnelStatus

• UdpOptions

• Vcn

• VirtualCircuit

• VirtualCircuitBandwithShape

• Vnic

• VnicAttachment

Developer Experience: The future(for real, for real)

Compute Block

Storage

Virtual

Networking

Load

Balancing

Object

Storage

Audit Database Puppies

as a

service

Haircuts

as a

service

…

Java

SDK

In preview

Python

SDK

N/A

Ruby

SDK

In

preview

N/A

CLI N/A In preview

HDFS

Connector

N/A N/A N/A N/A N/A N/A

Chef Knife

Plugin

N/A N/A N/A N/A

Terraform

Provider

N/A

Fortran

SDK

Logo

SDK

…

Developer Experience: The future(for real, for real)

Boxes: N dev tools X M services X Z features

Team: 6 devs

Developer Experience: The future(for real, for real, for real)

• Pace of innovation rapidly increasing

• 1 year ago org size: 300

• 1 year ago Developer Experience team size: 4 devs

• Today org size: 850 (+185%)

• Today Developer Experience team size: 6 devs (+50%)

• 1 year from now org size: 1,500 (+80%)

• 1 year from now Developer Experience team size: 9 devs (+50%)

• We must maintain feature parity, despite team size not scaling at same pace as org at large

Developer Experience: The future(for real, for real, for real)

Boxes: N dev tools X M services X Z features X Y pace of innovation

Team: 6 devs

Developer Experience: The future(for real, for real, for real, for real)

• Different release scopes:

• Internal-only

• Our services call each other

• For their own infrastructure management

• To enable some cross-service customer use cases

• Some of the APIs / parameters they use are not meant for public consumption

• Internal-preview

• Let Oracle and special customers try features before we’re ready to publicize them

• Public-preview

• Let public customers try out new features before we certify them as production-ready

• Public

• Release a feature as production-ready, to the public

Developer Experience: The future(for real, for real, for real, for real)

Boxes: N dev tools X M services X Z features X Y pace of innovation X W release scopes

Team: 6 devs

The problem:How to keep up with work of this scale!?

Boxes: N dev tools X M services X Z features X Y pace of innovation X W release scopes

Team: 6 devs

The solution• Swagger

• Open Sourcing

• API Consistency

Swagger• https://swagger.io

• Declaratively describe REST APIs in a machine-readable way

• Manage your services’ interface specifications as you would your code

• Open Source

• Useful for: Generating REST API documentation

Code reviewing REST APIs (to ensure good REST use, and consistency across BMCS APIs)

Automated & manual

Creating automated pipelines around service specifications

Service (scaffolding) generation

Client generation! (open source generators exist for many languages)

We generate:

Java SDK

Ruby SDK

Python SDK

CLI

Generated Terraform provider (work in progress)

Swagger spec: Launch instance

Results when generated

• REST API docs

• Java SDK

• Python SDK

• Ruby SDK

• CLI: bmcs compute instance launch

DemoSwagger and Bare Metal Cloud Services

Swagger: Cons & Gotchas

• Making modifications / polish to generated code

• Keeping backwards compatibility as the Swagger spec changes

• Contributions require understanding of metaprogramming and Swagger codegen, not just language the dev tool was “written” in

• Non-language-related build tooling needed to contribute

Ex: Maven (Java) required to build Python SDK

Open Sourcing• Get the developer community involved

Finding and fixing issues

Adding new features

Answering questions

Improving documentation

• Create advocates for your products

• Find potential hires who are passionate about the area you work in

• Be a good developer citizen -- let others use your code as a basis for their projects / learning

• Other useful parts of GitHub: Archived releases

Subscribe-able release notification channel

Metrics

• Open sourced so far: Java SDK: https://github.com/oracle/bmcs-java-sdk

Python SDK: https://github.com/oracle/bmcs-python-sdk

Terraform Provider: https://github.com/oracle/terraform-provider-baremetal

Chef BMCS Knife Plugin: https://github.com/oracle/knife-bmcs

CLI (soon!)

Open Sourcing: Cons & Gotchas

• Syncing changes between GitHub repository and internal repository

• Keeping private things, private

• Accepting pull requests on generated code (since we codegen our code)

API Consistency• Useful to the user

• Can build higher-level functionality once per dev tool (instead of once per resource), since resources do things same way Ex: pagination

BMCS list APIs return opc-next-page header, and take in that header’s value via query string: ?page=<token>

Ex: waiters BMCS resources have a lifecycleState field to indicate creation / updating / deleting status

• Reduces need to special case generated code Ex: serializing OR filter semantics into an API call

/cars?make=subaru&make=toyota

• Makes testing easier When resources are managed similarly, test cases are similar too

Less investigation / writing of new code, more porting existing tests for a new resource type

• Swagger makes API consistency much easier! Swagger = service interface specifications as code

Diffs

Code review tools

Automated review

One problem area left:Multiple release scopes

N dev tools X M services X Z features X Y pace of innovation X W release scopes

• Internal-only

• Our services call each other

• For their own infrastructure management

• To enable some cross-service customer use cases

• Some of the APIs / parameters they use are not meant for public consumption

• Internal-preview

• Let Oracle and special customers try features before we’re ready to publicize them

• Public-preview

• Let public customers try out new features before we certify them as production-ready

• Public

• Release a feature as production-ready, to the public

Addressing the Problem:Multiple release scopes

release

preview

some-feature-to-go-public

internal

some-never-public-feature

Swagger spec branching

Addressing the Problem:Multiple release scopes

Swagger spec custom fields

CreatePuppyDetails:

properties:

name:

maxLength: 255

minLength: 1

type: string

ownerId:

maxLength: 255

minLength: 1

type: string

isPottyTrained:

type: boolean

x-obmcs-preview-only: true

dogType:

type: string

canBreatheFire:

type: boolean

x-obmcs-internal-only: true

• x-obmcs-preview-only

• Filtered out in public dev tool

generation

• Gets a preview disclaimer in its

description in preview dev tools

• x-obmcs-internal-only

• Filtered out in preview and public dev

tool generation

Q/AP.S. – we’re hiring! Interested in this kind of thing?

Email me at joe.levy@oracle.com or come find me afterwards