private cloud: database-as-a-service with enterprise manager : oow 16

DBaaS Comcast OOW16

Hybrid DBaaS with Oracle Enterprise Manager: Comcast's 7 Goals for the CloudSeptember 20, 2016

Bala Kuchibhotla, Vice President Engineering, OracleJayson Hurd Database Platform Architect, ComcastTejas Gohil DBA Manager, Comcast

1

AgendaPresentation title (optional)2

Superior Service with DBaaSComcast Company & Department Vision &futureCurrent State and HistoryDirection, Solutions & Architecture

2



3

Complete Cloud Control

Optimized, Efficient | |Integrated Cloud & On-premise Stack ManagementAgile, AutomatedComplete Cloud & On-premise Lifecycle Management

Scalable, SecureSuperior Enterprise-Grade Management

Copyright 2016, Oracle and/or its affiliates. All rights reserved. |

4

The Communication Hub of Oracle

Single pane of glass for hardware and software management across cloud & on-premiseCentralized communication & collaboration for diagnostics and troubleshootingDesigned to share critical information while maintaining sandboxesIntegrated named credentials & auditing

5


5

Database as a Service: Agility in DevOpsGive Everyone What They Want Most

DBaaS

Finance

Sales

Manufacturing

Marketing

HRUsers WantEasy self-service Access to data storePerformance optimized for service levels

IT WantsAgility to spin up new instances & provision dataSimplified deployment on standardized platformsLeverage resource optimization techniques like storage snapshotting to save on CAPEX.Less maintenance and better support.


6

IntegrationAPI integration with 3rd party tools , change requests, dev workflows

EfficiencyConsolidated, shared platform maintained with automation

ElasticityAgile scale up, scale down, relocation of services

AccessibilityAutomated request channel via self-service Databaseas-a-Service

AccountabilityMeter usage, quota, compliance and governance

Oracle Database as a ServiceFully functional Database with any Data on any infrastructure

Dedicated Database

Pluggable Database

Database Schema

DatabaseDataInfrastructure

Full Clone

EmptySnap clones

ORORPhysical, Virtual, or Engineered Systems(Hybrid Cloud)On Premise & On Oracle Cloud


7

Agenda

Superior Service with DBaaSComcast Company & Department Vision &futureCurrent State and HistoryDirection, Solutions & ArchitecturePresentation title (optional)8

8

Im sure most of you know Comcast as another Cable company like the others. And that is true, to a point. In many ways we are 9

Vision: Company Strategy22M subscribers$74B revenueNBC UniversalContent/ChannelsComcast ventures

Comcast BusinessXfinity InternetXfnity TVX1 On Demand/PPVXfinity VoiceXfnity home securityNationwide hotspots

Competition is Google & AmazonTechnology focusSilicon Valley, Denver, Philadephia Product supplier

Presentation title (optional)10Spirit of InnovationQuality of Service Delivery

CompanyProductsInnovation

10

Vision: 7 Principles of Database Management

Automated

Self-Serviced

Controlled Delegation

MTTR Reduction

Agile Provisioning

Common Interface

ConsolidatedPresentation title (optional)11

So when we pulled ourselves up to 30000 feet and let the shackles go from our ankles, we narrowed our scope, our mission statement, down to 7 principles.

This is the title of our presentation because it is the very philosophy that drives what we are doing.

We need to be absolutely as automated as possible. Everyone says this, but it is really true. We cannot manage to scale with manual processes. Manual processes are both time consuming and prone to error. Therefore we must commit to develop rather than do so much. It is very easy to start doing instead of thinking how to automate for repetition. This requires commitment to change the very way we think as engineers and operations people.Our environment needs to be come self service. This has many drivers. In our organization we have devops as you may know from your organization. Our job is to provide a shared platform, a cloud, that can be consumed by developers and application support. And the quicker they can consume that, the better. That means cutting out ticketing procedures, tying duties to roles in Oracle Enterprise Manager. That means everything from provisioning, to patching, to maintenance must be self service. It can be limited to our DBAs to use as well, but it needs to be fast and self-service. Getting rid of the cycle of pain, for ticketing and hands offs, is the goal. The more we build in process and sticky procedures to getting things done, the more reviled we become. The more reviled we become, the more we cost the organization, and the more we slow down progress and continuous delivery.Controlled delegation I hinted at this in the previous point. We should be able to delegate the automated and self service solutions to either devops or other DBAs. Our efforts in engineering and architecture should be enabling them and giving the right duties to the right people, packaged, tested and ready to go against predictable and stable platforms.MTTR reduction all this must have the effect of reducing MTTR. That means we provision consistently, we patch regularly and predictably and we resolve issues much faster as a result. MTTR reduction is our key charter for 2016 and beyond from our CIO. This platform must, and will meet that demand admirably.

Agile provisioning We cannot be taking days, weeks or months to provision services. Anything from small development environments to full blown hardware RAC/Dataguard needs to be provisioned in hours. This is a must. We can no longer be a roadblock to our organization.We must have a common interface to all of this. This is obviously achieved with OEM. OEM needs to be a central point, a tier 1 application in its own right, that is the center of all database operations, provisioning, etc. This extends to mean that most back end duties are also done through OEM. We cannot have random manual work being done on a large scale as this risks the stability and viability of our platform.Finally, we must consolidate. 12c multitenant and resource manager bring us past the aforementioned either-or choice between risky schema consolidation and explosions of hardware and oracle homes. Oracle has truly enabled cloud management for us to meet this demand.11



12

Current State:ChallengesComcast - Oracle OpenWorld 201613

13

Presentation title (optional)142000-2009: Big IronEasy to ManageShared codeConsolidated

High riskResource controlsLess modular

BulkheadedRedundantScaled

UnmanageableExpensive, wastefulSilo-driven

2009-2015: Explosion

History

Lets go backwards a few years, again to understand our history and how we have arrived where we are, and why we are charting our innovative future.

2000-2009 think Big Iron. This was yesteryears way to manage a lot of applications in a smaller infrastructure footprint. Most of our application groups were schemas in large 9i and 10g databases running on very powerful Sun hardware. While this did offer capacity and simplicity of management, it did mean we had too many eggs in one basket. When one application sneezed, many others caught a cold. This meant some very long nights and high pressure conference calls.

It also meant that brining in new applications and code became very risky ventures the antithesis of our modern day continuous delivery. Being so inflexible meant we could not innovate.

And so the pendulum swung the other way14

Current State:Oracle Footprint

Presentation title (optional)15

IT Department - APS

Provisioning ServicesField AutomationOracle across the organization700+ applications supported

Oracle Footprint

20000 managed targets2200+ databases1300 virtual machines1800 hosts

Within Comcast we belong to the largest IT branch known as APS - Application Platform Services. Our division handles all provisioning and field services application. Hence we are the most customer facing of all IT departments. Our Oracle services, however, services the entire Comcast organization, or roughly 80-90% of all Oracle databases at Comcast.

This accounts in terms of our OEM targets, 20000 managed targets, 2000+ databases including 1300 virtual machines and 1800 hosts.

Our environment is almost entirely Linux x86 RAC with Dataguard, and most applications are 3-node RAC with a corresponding dataguard. Very little, if any, is currently consolidated. 90%+ of the environment is 11.2, with our preferred version of 11.2.0.4.

Of course, all this is quickly becoming legacy15



16

Direction

Comcast - Oracle OpenWorld 201617

17

Solutions


18


Self Service DBaaS PortalOracle OEM Cloud Pack PortalProvisioningCDB/PDB creationCloning OEM Service Catalog Active DirectoryPatching and UpgradesConfiguration ChangesTroubleshootingPerformance Tuning Metrics Reporting

OEM Automation & Monitoring3-Tier DeploymentMultitenant

Pluggable DB

Maximum DensityLower EnvironmentsSilverDedicated DB

RAC/DataGuard

Resource Controls

Highly AvailableProduction SharedGoldDedicated HW

RAC/DataGuard

RAC/Golden GateDedicated ProductionGold/PlatinumDensityPerformanceSelf Service L2 & DevOpsUsage TrackingHigh AvailabilityFault ToleranceArchitecture

From these principles we have developed a master framework for managing Oracle through Enterprise Manager and the Cloud Pack.

As I mentioned, and from the top. A common interface, tied into our LDAP Active Directly, accesses through the cloud pack portal. All routine duties for provisioning and patching etc. are done through a controlled service catalog no manual work. All troubleshooting and performance metrics are done through the OEM interface or fed to other internal interfaces through the API a central point source-of-truth for all Oracle related operations.

Underlying this is a 3-tier deployment model, a truly efficient balance between density and performance, meeting the demands of our customers while also not creating silos of resources that cost us millions.

All VMs are converted to multitenant self-service pluggable databases sitting on performant 3-node RAC. All production that is not of heavy utilization will be a CDB not PDB due to the Dataguard requirements of being able to fail over individual applications and all will be RAC/DataGuard. Effectively, you should be choosing dedicated hardware due to RTO/RPO, or business requirements. All our shared environments offer the highest level of availability. You should be selecting dedicated vs. cloud/shared hardware based on your load and that is it. As we are able to consolidate, we have resource manager on our CDBs to cage resources and guarantee service levels.Finally for high load applications we can still use dedicated hardware. We can stage hardware in datacenters throughout our footpring so as to deliver to the customer in a matter of hours.

As all this is tied together we can track resource utilization by active directory group, cloud zones and pools. This means we can tell our senior management and accounting people, who is using what and where. And of course, they can self-service this data for themselves!

19

Build and Migration


Migrating to the cloud how?

Let me summarize

First we must build out a new architecture framework.

The framework must be easy to stand up and then built gradually in order to minimize up front hardware costs.

Then we have to migrate

Target is to migrate 1300 VMs to our Oracle private cloud for all our lower environments dev/qa to between 9 and 18 hosts (3 6 clusters)Create criteria for migrating lesser used production DBs to shared environments (1:10 consolidation ratio) since we were 100% RAC/DataGuard for nearly every schema (many applications had multiple rac/dataguard cluster combinations. This makes my our results look fantastic! Easy pickings to densifyEVERYTHING to be managed by DBaaS including platforms with dedicated hardware (zones)

20

Implementation ApproachEasy Entry

All this theory is great, and Im sure the architects and directors in the room have enjoyed this. But our job is to then take this and implement into reality.

Again, as I said earlier, this may not be as hard as you think.

We build a framework. We were able to build a framework, from planning to readiness, in under 5 months. Now, our OEM footprint had already been built out and needed only some modifications to meet this. But keep in mind, the duty is to build out a framework first, then move to the model as your time and budget fits.Next we must engineer standards so that as the proverbial bathtub is filled, things just gradually fall into line over the coming yearsResource management and subscription limits on shared clusters must be strictly enforced in order to maintain the integrity of the systemService catalog items using OEM must be the only point of provisioning, patching and other routine maintenance. Anything outside of this threatens the ease of use and stability of the cloudThings like naming conventions and clear/concise end user documentation must be available. The key is that end users do not need to come to DBAs to request that things be done. As confusion is encountered with devops engineers, documentation and communications must be tightened and enhancedHardware zones must be built out strategically. Comcast has 5 major datacenters with various network zones (firewall areas). We had to strategically plan the rollout and migrations while at the same time accommodating net new buildouts. This is difficult to do without spending $millions up front in hardware. Think of just in time inventory and supply chain management that is what we are effectively doing.Build a clear and concise user intake process. Users must easily obtain access to the self service platform21

Solution:Implementation ApproachOEM Level 4 HA/DR OEM ArchitecturePresentation title (optional)22Oracle Enterprise ManagerCloud PackLifecycle ManagementPublic Cloud Gateway

There were two major parts to our framework buildout:

Making OEM into the tier 1 platform it has to be to support our 700 applications, operations and BAU people.Underlying hardware footprint architecture and buildout.

Let me first discuss the Level 4 HA DR architecture here. OEM was put in two datacenters, 3 servers each. Underlying that is RAC/dataguard 12c across the same datacenters.

Then Active Directory was enabled so that we can leverage AD groups for role based access to OEM. This means that application teams control who gets access once we assign the groups. More self service!

We added the cloud pack and lifecycle management to our oracle licensed product portfolio to manage the environment and provide self service to our customers.

This architecture is the highest level of OEM architecture as prescribed by Oracle. Leverage their documentation and sales/service expertise for help here.22


65 TB Storage SavingsMax CPU/Memory90%+ Dev Host Count Reduction10%+ Prod Host Count Reduction

Architecture:Initial Hardware Zones12c Multitenant

Talking about the three consolidation models I laid out in the high level architecture..

Our lower environments will likely see the greatest benefit because we are able to use multitenant to its fullest extent.

In effect we are looking currently at migrating our 1300 VMs to just 3 RACs!

Granted, these are much heavier hardware footprints:

HP DL 560 machines (our XL hosts), beefed up each to 512GB memory and full 10Gbps network cards to handle interconnect, backup and end user traffic.

If we need more capacity we will simply add more RAC clusters as new cloud zones.

And the savings as I mentioned are HUGE. We can accurately save roughly 65TB of usable, high and SAN storage, fully consolidate CPU and memory usage.

And patching becomes MUCH easier as we will create new CDBs and migrate pluggable databases this way, one by one. 23

Initial Hardware Zones Lower Environments12c Multitenant

1300+ VMs to 9 HostsPresentation title (optional)24

Here is just a graphical demonstration of how we plan to migrate literally hundreds of PDBs into given RAC clusters. Multitenant allows us to do this!24

Shared Production CDB/PDB RAC/DataGuardPresentation title (optional)25

Using the same hardware for ease of provisioning, we are building out shared production zones for lower utilization applications.

Instead of using PDBs, we will deploy individual CDBs per application. Now, those CDBs may contain multiple PDBs but the architecture would ensure each CDB is treated as a failure group meaning it could be failed over or switched over using DataGuard as one unit. Each CDB would be deployed as a service catalog option most likely by our own internal DBAs. Self service to devops teams might be possible in the future if we are able to put some controls in for subscription levels in particular zones.

In effect we need to limit utilization in two ways aggregate CDBs on each cluster and then how much each CDB uses. We will do this by carefully managing the subscription levels and having the right resource management plans attached to each CDB. We also plan to undersubscribe these clusters in order to have buffer capacity if databases grow in usage. Again, as we need, we simply scale out by adding more zones and clusters.

Let me also for a moment just touch on the topic of dedicated hardware. Many of our applications will have dedicated hardware, no doubt. The concept here however will be the same in terms of cloud management. Rather than shared zones, we have dedicated zones that is all. When we manage through OEM cloud pack, we manage EVERYTHING. In our organization there is no maybe we move to the cloud, maybe we dont. This model encompasses ALL our oracle footpring, period.

25

Migration PathsComcast - Oracle OpenWorld 201626

Golden Gate

RMAN

Expdp/Impdp

26

Hybrid Cloud (2017)Presentation title (optional)27

As we move to this in house private cloud we are also going to begin experimenting with a hybrid model

Through this same OEM footprint we will be adding access to Oracles Public Cloud. We can then make these resources available to our end users transparently.

We will be exploring the business justification for this and shifting resources between private and public accordingly.

Your choice between underlying private/hybrid/public cloud resources is really going to be driven by your own economic, scale and business requirements.

But in either case, the model we laid out is the same, and the philosophy and principles equally applicable.

27

Future RoadmapOEM PlatformEliminate VMsNet new to DBaaS2-3 year process2000+ DatabasesAutomation through OEM

13cShowbackSnap ClonesRelocations (plug/unplug)Devops patchingDevops full deploymentCapacity planning (IT Analytics)Comcast - Oracle OpenWorld 201628Migrations

28

ConclusionThere is no other future!Comcast - Oracle OpenWorld 201629

29

30

Dont Miss the SessionsTitleDayTimeExpanding DBaaS Beyond Data Centers: Hybrid Cloud Onboarding via Oracle Enterprise Manager [CON6985]Wednesday11:00 AM *Deep Dive: Snap Clone and Data Refresh Solutions using Oracle Enterprise Manager [CON6982]Wednesday1:30 PM *Best Practices for Upgrading to Oracle Enterprise Manager 13c: Learn from Customers [CON6989]Thursday1:15 PM *

Location: Hotel Nikko - Peninsula (25th Floor)DayTimeRelief from Chronic Patching PainOracle Enterprise Manager Fleet Maintenance [HOL7632]Tuesday11:30 AMHarnessing the Value of Hybrid CloudComplete Management of Cloud Services [HOL7631]Wednesday1:15 PM

Do it Yourself! (Hands on Lab)* Moscone South - 305


31

Additional ResourcesRelated VideosYoutube.com/OracleEnterpriseMgr

Join the Conversation

Twitter.com/oracle_em

Oracle.com/newsletters

Facebook.com/oracleenterprisemanager

Blogs.Oracle.com/OEMEnterprise Manager Page on O.com

Database as a Service Page on OTN

Cloud Administration Guide (Documentation)

MOS Note 1549855.1 : Enterprise Manager Cloud Control Recommended Plug-Ins and Patches for Database as a Service (DBaaS)


32