version your cloud: using perforce to manage your object storage

#

Daniel AnolikSolutions Architect, Jeppesen

Version Your Cloud

#

Daniel AnolikSolutions Architect, Airborne Navigation SystemsJeppesen (A Boeing Company)• 8 years with Jeppesen

• 15 years of direct SCM experience, 8 with Perforce

• Software developer, solutions architect, Scrum master, bluegrass mandolin picker, wilderness survival enthusiast…

#

Daniel AnolikSolutions Architect, Airborne Navigation SystemsJeppesen (A Boeing Company)• 8 years with Jeppesen

• 15 years of direct SCM experience, 8 with Perforce

• Software developer, solutions architect, Scrum master, bluegrass mandolin picker, wilderness survival enthusiast,

and lucky father of this baby girl:

#

Jeppesen at a Glance• 80-year strong company founded in aviation safety• Wide array of software supporting:

– airborne navigation systems– flight planning services– pilot training – airline operations management

#

Supplier of Critical Aviation Data

#

• Distribute many data types to many applications running on many platforms

• Support frequent data updates for hundreds of thousands of customers around the world

• Transfer hundreds of terabytes of data each month• Don’t be a bottleneck!

The Challenge

#

Initial Plan

Data Brokering Magic

Jeppesen DataProduction

Magic

JeppesenInfrastructure CDM

Cloud Object

Storage

Workflow Magic

Data PublishingInterface

Boring File Server

#

The system must support:• Fine read/write access controls within storage tree

• Atomic change-sets for publishing files

• Detailed history tracking

• Individual file restores

• Data lifecycle management

Hmm…It’s starting to sound like we need a real CMS system in there

Pesky Implementation Constraints

#

Improved Plan

Data Brokering Magic

Jeppesen DataProduction

Magic

JeppesenInfrastructure CDM

Cloud Object

Storage

Workflow Magic

Data PublishingInterface

#

• Supported all our implementation constraints with minimal customization

• Leverage existing in-house expertise

• P4D had very stable track record at Jeppesen

• Efficient, scalable blob storage

Why Perforce vs Other CMS?

#

• Keeping changes atomic:– File sets are submitted to P4 as single changelist– Data is processed by the changelist– Data is published by the changelist

• Files are never ‘available’ to our applications until all files in a given changelist are available

Leveraging Perforce: Changelists

#

• Changelists tell us:– What files were published,– when,– and by whom

• Spec depots!• Soft deletes of files minimize ‘clutter’ while still

supporting data retention requirements

Leveraging Perforce: History

#

• Custom counters– are defined for each data type

– track last successful changelist processed

– provide a central, durable mechanism for state tracking

across different work nodes in the system

Leveraging Perforce: Counters

#

• Job queue are simple files in p4

• In process jobs are ‘locked’

• Completed jobs are ‘deleted’

• Full job history is retained

Leveraging Perforce: Locks

#

Leveraging Perforce: Branching

Private “Staged” Data

Public “Promoted” Data

p4 integrate

Depot paths represent the promotional model of the data

#

• P4 protections restrict access by:– depot path/data library– pattern matching/file extensions – IP addresses

• Service accounts granted read and write access to only what they need

Leveraging Perforce: Security

#

• 100% of interaction with Perforce fully automated through the P4Java API

• Custom utilities used to:– Read and write data in depot

– Manage branch integrations

– Read and update counters

– Read changelist details

– ‘Expire’ content

Leveraging Perforce: Java API

#

All the great capabilities of cloud based object storage, plus true configuration management:

– Detailed history tracking and extreme auditability

– Reproducibility: roll back any part of your ‘cloud’ to any point in time

– Atomic changelist management of file sets

The Result: Object Storage++

#

Any Questions?

#

##

Thank you!And have a safe flight home.

Daniel Anolikdaniel.anolik@jeppesen.com