version your cloud: using perforce to manage your object storage
DESCRIPTION
Jeppesen uses a combination of their own data centers and a variety of 3rd party cloud providers to support their distribution of navigation data to most of the world’s pilots. But before any data leaves the building it goes through Perforce. Learn why Jeppesen chose to put Perforce at the heart of their global data distribution system, and how you too could leverage Perforce to help provide atomic changesets and true history tracking to your cloud-scale object stores.TRANSCRIPT
![Page 1: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/1.jpg)
#
Daniel AnolikSolutions Architect, Jeppesen
Version Your Cloud
![Page 2: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/2.jpg)
#
Daniel AnolikSolutions Architect, Airborne Navigation SystemsJeppesen (A Boeing Company)• 8 years with Jeppesen
• 15 years of direct SCM experience, 8 with Perforce
• Software developer, solutions architect, Scrum master, bluegrass mandolin picker, wilderness survival enthusiast…
![Page 3: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/3.jpg)
#
Daniel AnolikSolutions Architect, Airborne Navigation SystemsJeppesen (A Boeing Company)• 8 years with Jeppesen
• 15 years of direct SCM experience, 8 with Perforce
• Software developer, solutions architect, Scrum master, bluegrass mandolin picker, wilderness survival enthusiast,
and lucky father of this baby girl:
![Page 4: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/4.jpg)
#
Jeppesen at a Glance• 80-year strong company founded in aviation safety• Wide array of software supporting:
– airborne navigation systems– flight planning services– pilot training – airline operations management
![Page 5: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/5.jpg)
#
Supplier of Critical Aviation Data
![Page 6: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/6.jpg)
#
• Distribute many data types to many applications running on many platforms
• Support frequent data updates for hundreds of thousands of customers around the world
• Transfer hundreds of terabytes of data each month• Don’t be a bottleneck!
The Challenge
![Page 7: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/7.jpg)
#
Initial Plan
Data Brokering Magic
Jeppesen DataProduction
Magic
JeppesenInfrastructure CDM
Cloud Object
Storage
Workflow Magic
Data PublishingInterface
Boring File Server
![Page 8: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/8.jpg)
#
The system must support:• Fine read/write access controls within storage tree
• Atomic change-sets for publishing files
• Detailed history tracking
• Individual file restores
• Data lifecycle management
Hmm…It’s starting to sound like we need a real CMS system in there
Pesky Implementation Constraints
![Page 9: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/9.jpg)
#
Improved Plan
Data Brokering Magic
Jeppesen DataProduction
Magic
JeppesenInfrastructure CDM
Cloud Object
Storage
Workflow Magic
Data PublishingInterface
![Page 10: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/10.jpg)
#
• Supported all our implementation constraints with minimal customization
• Leverage existing in-house expertise
• P4D had very stable track record at Jeppesen
• Efficient, scalable blob storage
Why Perforce vs Other CMS?
![Page 11: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/11.jpg)
#
• Keeping changes atomic:– File sets are submitted to P4 as single changelist– Data is processed by the changelist– Data is published by the changelist
• Files are never ‘available’ to our applications until all files in a given changelist are available
Leveraging Perforce: Changelists
![Page 12: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/12.jpg)
#
• Changelists tell us:– What files were published,– when,– and by whom
• Spec depots!• Soft deletes of files minimize ‘clutter’ while still
supporting data retention requirements
Leveraging Perforce: History
![Page 13: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/13.jpg)
#
• Custom counters– are defined for each data type
– track last successful changelist processed
– provide a central, durable mechanism for state tracking
across different work nodes in the system
Leveraging Perforce: Counters
![Page 14: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/14.jpg)
#
• Job queue are simple files in p4
• In process jobs are ‘locked’
• Completed jobs are ‘deleted’
• Full job history is retained
Leveraging Perforce: Locks
![Page 15: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/15.jpg)
#
Leveraging Perforce: Branching
Private “Staged” Data
Public “Promoted” Data
p4 integrate
Depot paths represent the promotional model of the data
![Page 16: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/16.jpg)
#
• P4 protections restrict access by:– depot path/data library– pattern matching/file extensions – IP addresses
• Service accounts granted read and write access to only what they need
Leveraging Perforce: Security
![Page 17: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/17.jpg)
#
• 100% of interaction with Perforce fully automated through the P4Java API
• Custom utilities used to:– Read and write data in depot
– Manage branch integrations
– Read and update counters
– Read changelist details
– ‘Expire’ content
Leveraging Perforce: Java API
![Page 18: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/18.jpg)
#
All the great capabilities of cloud based object storage, plus true configuration management:
– Detailed history tracking and extreme auditability
– Reproducibility: roll back any part of your ‘cloud’ to any point in time
– Atomic changelist management of file sets
The Result: Object Storage++
![Page 19: Version Your Cloud: Using Perforce to Manage Your Object Storage](https://reader033.vdocuments.us/reader033/viewer/2022061201/5469b9e5af7959cb768b53aa/html5/thumbnails/19.jpg)
#
Any Questions?
#