architectural changes between dmr and ci 18-jun-2013

Architectural Changes between DMR and Ci

18-Jun-2013

Database Changes

• Migrating from Oracle to MongoDB– All new data stores have been created in MongoDB

• Auth• Accounts• Inbox/Notification Framework• Audit• CPS• Asset Relationships

– Next Up• Collection Based Security• Assets Details/Files with metadata• Product/Title Hierarchy• Projects

API Strategy• Goal = Simplify

– Reduce number of calls required to get things done– Reduce need for detailed understanding of the back end– Enable 3rd party developers to work un-encumbered– Clear documentation with samples in various languages

• From SOAP to REST– All new APIs are REST based

• Auth, Customer Accounts, User Profile, Inbox, Audit, CPS, Folders• Beta: Ingest, Retrieve, Preview

– Next Up• Share• Search• Get Details• User Admin

• POC currently underway with Apigee and 3scale– Evaluating developer portal, OATH2, traffic management and analytics

Search

• Moved from Oracle Text to CloudSearch – Powers smugmug.com (over 1Billion photos)

Underlying technology powers amazon.com

– Moving to production in the next release

Ingest• Ported legacy windows services to asynchronous SWF to handle long running

ingests

• Implemented ability to execute steps in parallel

• Auto-scaling for long running tasks (CPS)

• Ability to re-submit ingest requests

• Ability to create to-be-made/virtual assets

• Next Up– Ingest in place– Ultra-fast transcode

Export Services

• Modified export service to allow multiple instances to run simultaneously via SQS

• Eliminated need to move assets prior to exporting without transcode

• Next Up– Migrating to SWF

Other Areas Improvement

• Improvements– Multi-tenancy– Distributed Administration– SSO via SAML token support– 2FA

• Simplification (future)– Metadata setup and configuration– Remove Asset Type hierarchy– Re-consider Metadata Based Security

• Determine if a Search Layer can meet the requirements

Build/Deployment• Sprints are managed in Jira

• Source control is BitBucket (Git managed service)

• Automated build/deploy using Bamboo ( CI and Release managed service)– All 3 systems are integrated to enable efficient tracking of deployables back

to source back to tasks

• Deployment cycles have been significantly reduced. Currently tracking towards weekly deploys– The process is still too onerous and is being iterated on to get us to daily

deploys

Example of our approach

• MCS is adding a specialized product to the application portfolio that leverages the best parts of the current stack while refactoring problem areas: ie, Security, Files.

• These refactored pieces will be iterated over to meet MediaBox requirements and enable us to replace the poor performers in the stack.

Mediabox Auth Admin Splunk Aspera

DMR Auth CPS Audit Notification

Workflow Replay CPS Datama

rt Aspera MkProxy

Oracle MongoDB MS SQL

S3 CloudFront

ElastiCache EC2 VPC

Hotfolders

Master SuiteUIs

APIs

Workers

Data stores

Infrastructure

Splunk

Route 53

architectural changes between dmr and ci 18-jun-2013

Documents