data center operating system (dcos) ibm platform solutionsfiles.meetup.com/18744996/platform and...
TRANSCRIPT
© 2014 IBM Corporation2
Agenda
• Market Context
• DCOS Definitions
• IBM Platform Overview
• DCOS Adoption in IBM
• Spark on EGO
• EGO-Mesos Integration
© 2014 IBM Corporation3
Market Context
IBM Systems and Cloud
IaaS
Frameworks
Emerging Layer in the Stack
IBM Non IBM
Client Requirement developing for
cross framework resource
management, service management
and life cycle management in a
shared cloud environment
1. Sea Change: CAMSS workloads are transforming both infrastructure and applications
2. Islands of application frameworks developing creating problems for IT (sprawl, utilization)
3. Emergence of Open Source projects (YARN, Mesos, Kubernetes , Docker) to address
Watson
BI
YARN
Swarm/ComposeMagnum
Diego
EGO
HDFS
Kubernetes
© 2014 IBM Corporation4
Data Center Operating System
A Data Center Operating System (DCOS) is a technology foundation for
Software Defined Infrastructure products/solutions, providing:
•Resource aggregation across the data center(s)
•Multi-tenancy
•Policy-based sharing
•Integration with multiple workload types
•Supports enterprise capabilities (Reporting, GUI, HA, Security)
And delivering the following values:
•Improved infrastructure utilization
•Application performance and SLA
© 2014 IBM Corporation5
What is DCOS: Node Operating System Revisited
Device Drivers
read/write
Hardware
Application
Middleware
OS Installer & Management
Tools
File System
Memory Manager
Process Manager
Services Manager
malloc/free
fork/exec
service start|stop
insmod/rmmod
System
Servicescron,nfsd,etc
Web server,
app server,
database,etc
Applications
Setup & Configure
C API
An operating system
exists to support
applications but needs to
be installed, configured
and managed
© 2014 IBM Corporation6
Node OS To Data Center OS
Hardware
Node OS
Hardware
Node OS
Hardware
Node OS
Hardware
Node OS
Node Agent Node Agent Node Agent Node Agent
Distributed File/Block/Object System
Resource Manager
Remote Execution & Container
Management
Distributed Services Manager
Device Drivers for Nodes
Virtual / Physical Hardware
Patterns & REST API
Manage long-running services
lifecycle
Aggregate & share resources across
multiple frameworks
Manage execution of containers
(discovery, clustering, load-balancing)
Persistent storage for applications
and services supporting multiple
protocols
Nodes become the resources managed by Data Center OS. Specialized hardware
(storage, network switches, routers) become software services on commodity hardware.
IBM Confidential
© 2014 IBM Corporation7
How Data Center Operating System is used
Hardware
Node OS
Hardware
Node OS
Hardware
Node OS
Hardware
Node OS
Node Agent Node Agent Node Agent Node Agent
Distributed File/Block/Object System
Resource Manager
Remote Process Execution & Container
Management
Services Manager
System Services (eg Storage
Protocol Gateways, NFV)
Application Frameworks, PaaS
(Hadoop, CloudFoundry,Symphony)
IaaS
Setup ,
Configure and
Manage
Patterns & REST API
ApplicationsRun-Time
Execution &
Workload
Management
Device Drivers for Nodes
Virtual Hardware
Manage long-running
services lifecycle
Aggregate & share
resources across multiple
frameworks
Manage execution of
containers (discovery,
clustering, load-balancing)
Persistent storage for
applications
and services
IBM Confidential
© 2014 IBM Corporation10
Example: IBM Platform EGO
EGO master EGO standby master EGO Slave EGO Slave EGO Slave
EGO Agents EGO Agents EGO Agents EGO Agents EGO Agents
EGO core daemon
(vemkd)
EGO service
controller (initd)
EGO Kernel
EGO Kernel APIs
Platform Symphony
Platform Symphony
MapReduce
Platform LSF
Platform Cluster
Manager
IBM Cloud Manager
3rd Party Applications
Platform
Application
Service Controller
EGO components
Applications on EGO
© 2014 IBM Corporation12
IBM Platform Computing Infrastructure software for high performance applications
– 20 years managing distributed scale-out systems with 2000+ customers in many industries
– Market leading workload, resource and cluster management
– Unmatched scalability (small clusters to global grids) and enterprise production-proven reliability
– Heterogeneous environments – x86 and Power plus 3rd party systems, virtual and bare metal, accelerators / GPU, cloud, etc.
– Data-aware with multiple Elastic Storage integrations
– Shared services for both compute and data intensive workloads
23 of 30 largest
commercial
enterprises
Over 5M
CPUs under
management
60% of top financial
services companies
© 2014 IBM Corporation13
Platform Computing As Part of IBM Software Defined Infrastructure
IBM Platform Resource Manager (EGO/DCOS)
Software
Defined
Compute
MapReduceSymphony
Application
Service
Controller
LSF
High Performance
Analytics (Low Latency Parallel)
Hadoop /
Big Data
Application
Frameworks(Long Running Services)
High Performance
Computing(Batch, Serial, MPI, Workflow)
Example
Applications
&
Application
Frameworks
HomegrownHomegrown
Spectrum Scale (w/ LTFS) - XIV / “Purple” - SAN Volume Controller
Virtual Storage Center - Tivoli Storage Manager
Software
Defined
Storage
On-premises, On-Cloud, Hybrid
Physical InfrastructureHypervisor
x86 Linux on z
Software Defined
Infrastructure
Management
IBM Platform
Cluster Manager
Bare Metal Provisioning Virtual Machine Provisioning SoftLayer APIs & Services
IBM Cloud Manager
with OpenStack
IBM Platform Computing
Cloud Service
Other
Compute
Management
Software
Traditional
Commercial
Applications
© 2014 IBM Corporation15
Technology Foundation: Platform EGOResource Metrics Collection
• CPU utilization
• Number of cores
• Memory
• I/O
• Disk space
• Network
• User defined
Platform
EGO
Network Costs
DC
1
DC
2
DC3
DC
1
DC
2
DC
3Output
- Initial Placement
- Runtime Management
- Defragmentation & Migration
Offering 1
300
Offering 2
200
Offering 3
400
Jan Dec
Contract #123 Contract #444
Contract
#677
Contract #677
Contract #888 Contract # 888
#999
Contract # 78934
Contract # 768689
Contract
#55Contract
# 889
Reservations & Quotas
Supply: Resource Group Hierarchy
Rack
Group
Rack
Group
Rack
Group
DC
/
Rack Rack Rack Rack Rack Rack
DC
Demand: Consumer Tree
DB
AppServer
AppServer
AppServer
WebServer
WebServer
Rac
K1
Rac
k2
Rac
k3
R2
R2
R3
© 2014 IBM Corporation16
EGO Components
• Four major components
– LIM – Load Information Manager
– PEM – Process Execution Manager
– VEMKD – VEM Kernel Daemon
– EGOSC – EGO Service Controller
WS Interface
PEM
LIM
VEMKDLIM PEM
PEM
LIM
PEM
LIM
APIs
Master
Agents . . .
Clients
EGOSC
© 2014 IBM Corporation17
Illustration of three shared-resource models
A combination of all three models can be managed within a single grid at
the same time!
EGO Resource Sharing Policies
© 2014 IBM Corporation19
EGO Resource Policies (Cont’d)
• EGO Scheduling Policies
– Ownership
– Borrow/Lend
– Dynamic share
– Hybrid
– Multiple Dimension Scheduling (Improved DRF)
– Exclusive allocation
– Standby Service
– Smart Reclaim
– Resource Group Preference
– Topology Aware Scheduling
© 2014 IBM Corporation20
APIs
• Pseudo code of a sample client program that does ask EGO for some resource and start some “work”.
handle = vem_open(…)
vem_logon(handle, user, password) # authenticate client to EGO
vem_register(handle, …) # register client and callbacks
allocationID = vem_alloc(handle, allocationSpec…) # asks for some resource
containerID = vem_startcontainer(handle, allocationID, host, containerSpec, …)
vem_allocfree(handle, allocationID) # free allocation
vem_unregister(handle, …) # unregister
vem_logoff(handle)
vem_close(handle)
© 2014 IBM Corporation21
A Service Controller for complex long
running services
• Service and Application definition
• Service life cycle management
• Complex service dependency
• HA, Persistency, virtual IP
• Elastic service pool
• Auto-scaling
• Multiple triggers for grow/shrink
• Dynamic services deployment
• Unified resource management
• Resource sharing among long running services and tasks/jobs
• Stateful vs. stateless services
• API & scriptable interface
DCOS for Platform: Application Service Controller( ASC)
Examples:
• App servers
• Big Insights instance, Streams, Hbase, Oozie,
Native SQL apps, Mongo DB, Cassandra
IBM Confidential
© 2014 IBM Corporation22
DCOS for OpenStack: Platform Resource Scheduler
• Provides dynamic resource management for IBM OpenStack clouds• Automated management
• Reduce Infrastructure costs
• Improved application performance and high availability
• Higher quality of service
• More flexible resource selection
• Intelligent placement – automated, runtime resource optimization
• Included as optional scheduler and optimization service in ICM 4.2
• Included as a chargeable add-on product for IBM SmartCloud Orchestrator 2.4
• Full compatibility with the Nova APIs and fits seamlessly into OpenStack environments
• Part of IBM SDE portfolio
© 2014 IBM Corporation23
DCOS for Watson
Watson QA Resources Watson Ingestion Resources
Application Service
Controller (ASC)
EGO Agents EGO Agents
Bluemix
EGO
Docker Registry
Watson IngestionWatson QA Zuul
QA REST API
ZuulZuul
QA REST API
Symphony SOAM
EGO
Admin GUI
Classifier
Service
Training
Service
Tenant #N
Classifier
ServiceClassifier
Runtime
Classifier
Training
Ingestion Front-End
ServiceIngestion Front-End
ServiceAlchemy CSF
Classifier
Service
Training
Service
Tenant #2
Classifier
ServiceClassifier
Runtime
Classifier
Training
Classifier
Service
Training
Service
Tenant #1
Classifier
ServiceClassifier
Runtime
Classifier
Training
Value Watson QA:
• High availability and intelligent scheduling of long-
running QA services
• Improved multi-tenancy and higher utilization
Value Watson Ingestion:
• Better application performance through low-latency
task scheduling of ETL
• Improved multi-tenancy and higher utilization
Spectrum Scale (Future)Spectrum Scale (Future)
Admin GUI
IBM Confidential
© 2014 IBM Corporation26
DCOS for Spark
EGO Resource Orchestrator
Spark On EGO
Zeppeline
GUI Engine
ExecutorsExecutors
Analysis portal
Scala Notesbook
Shared within tenants
Spark Cloud Tenant -1
Spark On EGO
Zeppeline
GUI Engine
ExecutorsExecutors
Spark Cloud Tenant -2
Spark Engine
Share Spark Context
And scheduling Job
Within Spark EGO
Resource Management
Fine-grained scheduling
Reclaim and Share
Among tenants
Self-Service Portal
Tenants Creating and Provision
Spark Cloud Tenants
Spark Cloud Management Portal ( ASC)
© 2014 IBM Corporation32
Mesos / EGO• Motivation• Platform EGO, efficient enterprise-strength technology• Mesos much less mature than EGO
• Mesos supports a single simple scheduling policy (DRF – dominant resource fairness)”Unlike universities, we don’t care about fairness. We have a business to run”John Wilkes, Google Omega
• One Mesos framework can grab all resources by running long tasks, even with DRF (when other frameworks are idle)
• New ‘dynamic reservation mechanism’ in Mesos makes this even worse• Basically, no centralized administration of cluster policies• No support for organizational hierarchy, ownership/lending/borrowing, time-based
resource planning, pre-emption, priority, load-balancing, packing/striping, etc, etc.• However, Mesos provides plug-in mechanism for replacement of “resource allocator”
• EGO policy mechanism allows for• Performance / QoS protection for important workloads, service jobs, time-sensitive
interactive tasks• Ownership / reservation, lending with reclaim, • Balanced job distribution, packing, striping,
• Support for organizational policies• Consumer trees, parent policies,
• Flexible resource management• Lending / borrowing, shared pools,• Time-based resource plans• Resource rank
© 2014 IBM Corporation33
Mesos master
RM plugin
EGO Kernel(vemkd)
Platform Management Console
Mesos / EGO: Integration outline
Mesos slave
Mesos slave
Mesos slave
Mesos slave
Framework Framework