1

Building a Dynamic Building a Dynamic Datacenter with Oracle Datacenter with Oracle over NFS and NetAppover NFS and NetApp

Blackboard Managed HostingBlackboard Managed Hosting(sm)(sm)

Peter WatermanSenior Manager of Technology and Innovationpete.waterman@blackboard.com

2

Blackboard Managed HostingBlackboard Managed Hosting

Blackboard Inc. is a world leader in e-Education software - our online learning application, the Blackboard Learning System, is the most widely-adopted course management system among U.S. postsecondary institutions.

Blackboard Managed Hosting is a business unit within Blackboard which provides fully managed hosting services for all Blackboard software suites to a global client base of over 700 institutions. Our service includes management of every layer from the server metal to software administration and internet presence.

Our clients demand cost effective reliability and performance – we require efficiency and agility to provide it.

3

• Six datacenter facilitiesSix datacenter facilities on three continents

• 1.2+ petabytes1.2+ petabytes of NetApp enterprise storage

• 2200+ Dell servers2200+ Dell servers (including 1400+ blades)

• 30+ million users30+ million users across all installations

• 1040+ Oracle databases1040+ Oracle databases

• 1350+ application servers1350+ application servers providing web content

• 350+ million hits350+ million hits per day across all installations

• 15+ terabytes15+ terabytes of data transferred daily

At a Glance…At a Glance…

4

Our Typical Installation (mostly single Our Typical Installation (mostly single tenant):tenant):

Hosted Application Stack

NFS

5

Rebuilding the BoxRebuilding the Box

In late 2006, Blackboard Managed Hosting recognized that our yearly growth projections were so extreme that we would need to perform a core redesign of our methodology towards the following goals:

•Reduce complexity of system management

•Massively speed up backup and recovery

•Provide near-instant deployment and capacity expansion

•Improve system performance

•Reduce licensing and infrastructure cost using virtualization

•Avoid locking into a single current hypervisor

6

Blackboard Advanced Hosting PlatformBlackboard Advanced Hosting Platform

• Linux nfsroot: moves the OS to NetApp filers and allows migration between physical and virtual (P2V/V2P/V2V)

• Virtualization (RHEL5 Xen): allows effective performance upgrades for smaller clients and reduces future infrastructure and licensing costs

• NetApp Flexible Volumes: provides near-instant provisioning, recovery, cloning, and backups of NFS data

• Scalent V/OE: enterprise software for managing network booting, physical hardware, and virtual machines

• Oracle over NFS: moves commodity databases to NetApp filers, increasing performance and simplifying backup/recovery

7

Oracle over NFS: Tier 1Oracle over NFS: Tier 1

“Old” standard deployed by Blackboard for 2+ years

Follows standard Oracle/NetApp best practices whitepapers

High cost offering for more critical clients

Typically includes Oracle RAC

Hardest piece to manage is disk IO availability and balancing load on filer heads

8

ONFS Tier 2: “You are doing WHAT?!”ONFS Tier 2: “You are doing WHAT?!”

For most of our clients, a “pure” ONFS implementation is not cost effective…

Thus we built a solution that makes DBA’s cry:

Entire /home/oracle is on a single qtree

No archive logs (archivelogmode is off)

All snapshots are HOT – inconsistent

Stop throwing things and I’ll tell you why this actually makes sense!

9

Oracle over NFS without Archive Logs….Oracle over NFS without Archive Logs….

Easy: One mount. One location. Snapmirror a master oracle qtree and you have a “new” database in moments.

Cheap: Considerably less storage used without archive logs (snapshot < archive logs)

Most Manageable: Clone a running database and start up the clone in seconds with a couple commands. Recover from any snapshot in seconds, whether rollback or clone, without having to match up archive logs.

10

Not for Everyone though…Not for Everyone though…

Minor loss of data is a given in recovery/rollback/cloning when DB is hot

Any PITR not available – can only recover to snapshots

Minor performance hit during startup of inconsistent state database after recovery/rollback/cloning

Some chance of planetary alignment causing complete loss of data (haven’t seen it yet, but we’ve tried pretty hard)

11

Educate. Innovate. Everywhere.Educate. Innovate. Everywhere.

Okay, now you can throw things…Okay, now you can throw things…

Follow up questions:Follow up questions:pete.waterman@blackboard.compete.waterman@blackboard.com

Blackboard Managed HostingBlackboard Managed Hosting