Introduction to Achieving Performance and Scale on Large OpenStack Clouds
Alec Hothan, Principal Engineer
DEVNET-1693
• OpenStack Performance and Scale: challenges and existing tools
• A new generation of fully integrated tools to measure tenant network and storage at scale
• Design of these tools
• What they measure and how
• How to read results
• Free and open-source: install and use the tools yourself
Agenda
OpenStack Performance and ScaleTools and challenges
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 4DEVNET-1693
OpenStack Performance Metrics
Control Plane Metrics
Resource operations
Storage/Data PlaneTraffic metrics
Host system metrics
Networking
Compute Storage
tenants, userscpu, memory… networks
routers…
volumesimages…
Bare Metal
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5DEVNET-1693
Measuring these metrics is challenging
Abundance of tools
Complex measuring conditions
Automation is key
ps, top, strace, ptrace, perf…
iperf, netperf, nuttcp…
fio, jmeter, siege, wrk, cb
OpenStack rally, tempest
Cloudbench, PerfKitBenchmarker…
OpenStack variants/integration
Cloud scale
Failure conditions
Repeatable
Consistent
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 6DEVNET-1693
Example: web services and user experience
Density
Throughput
Latency
HTTP servers per compute node
HTTP requests per second
Latency percentile requirements:
User perception…
90th %ile 99th %ile Max
Fast
< 0.1 sec
Noticeable
< 1 sec
Slow
< 10 sec
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 7DEVNET-1693
Latency histogram (each traffic generator)
Latency
histogram
min
90th %ile
99th %ile
max
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 8DEVNET-1693
Latency histogram aggregation
+N x latency histograms
(distributed)
Overall system latency
histograms
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 9DEVNET-1693
Latency histograms at cloud scale
Number of HTTP servers per cloud 1000
Number of users per HTTP server x 1000
Number of requests per user/sec x 1
Test duration in seconds x 100
Total number of requests 100M
Number of latency histograms to consolidate
(1000 users/HTTP traffic generator)1000
A new generation of performance/scale tools
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 11DEVNET-1693
Tools characteristics
Fully integrated
End to end
Accurate and precise
Tight integration with OpenStack
Automated
VM and user level metrics
All batteries included
< msec precision
Correct consolidation of metrics
Simple and AccessibleInstallation
Usage
Result interpretation
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 12DEVNET-1693
OpenStack tools for performance/scale
Control Plane API-level
Performance
Single Flow Performance
KloudBuster
Networking
Compute Storage
Rally
High Scale Performance
VMTP
✅
✅
✅ ✅
✅ ✅
(coming soon)NFV Data Plane✅
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 13DEVNET-1693
Neutron Tenant Network Flows
VMs can run on same or
different compute nodes
North-South flows (NAT)
East-West flows
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Networking
Compute Storage
14DEVNET-1693
VMTP
• Single flow VM-level performance
• TCP
• UDP unicast and multicast
• ICMP
• Tenant network flows
• East-West and North-South
• Intra-node and inter-node
Validation and Performance Measurement of Network Flows
• Encapsulation and Neutron implementation agnostic
• Also supports
• Provider network
• SR-IOV NICs
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 15DEVNET-1693
VMTP Throughput Chart
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 16DEVNET-1693
KloudBuster
• Configure:
• Number of HTTP servers (1 per VM)
• Number of HTTP clients per server
• HTTP requests/sec
• VM placement policy
• Stages and orchestrates everything
• Simple runs
• Progression runs (HTTP servers)
• Real time latency monitoring
OpenStack Data Plane Performance at Scale
• Aggregates and consolidates all results
• Total throughput
• Micro-sec precision latency histograms• HDR (High Dynamic Range) histogram
library
Networking
Compute Storage
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VM VM VM
VM
proxy
VMVM
KB
17DEVNET-1693
Data Plane Scale Overview (East-West 1 cloud)
N x http servers
N x (http traffic generators (m users))
East-West L3 traffic
N x m flows
staging
orchestration
results aggregation
Internal network
tenant
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
proxy VM VM VM VM
VM VMVM
VM
KB
18DEVNET-1693
Data Plane Scale Overview (North-South 2 clouds)
N x http servers
N x (http traffic generators (m users))
North-South traffic
N x m flows
staging
orchestration
results aggregation
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 19DEVNET-1693
Latency Chart: 1K to 20K HTTP Connections
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 20DEVNET-1693
Real-Time Latency Monitoring (vPC failure)One protected
interface going down
Latency stacked
iso-percentile
scrolling layers
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 21DEVNET-1693
KloudBuster
• Configure:
• VM count
• Cinder volumes (block storage) or ephemeral disks
• Storage workload definition
• Stages and orchestrates everything
• Progression runs (VM count)
OpenStack Storage Performance at Scale
• Aggregates and consolidates all results
• Total throughput
• micro-second precision latency histograms (HDR)
Networking
Compute Storage
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 22DEVNET-1693
KloudBuster Storage Scale Test Overview
ephemeral
or block storagestorage network
tenant networkstaging
orchestration
results aggregation
N x Fio testers
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 23DEVNET-1693
Generated Charts (NFVi profile/Ceph backend)
progressions
concurrent VMs
Latency iso-percentile
line charts
workload results tabs
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 24DEVNET-1693
High IOPs profile (61K IOPs random read Cinder)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 25DEVNET-1693
High Throughput profile (3,743 MB/s seq read Cinder)
Installation and User Interfaces
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Installation and User InterfacesVMTP KloudBuster
Docker Container docker pull berrypatch/vmtp docker pull berrypatch/kloudbuster
PyPI installation (pip) pip install vmtp pip install kloudbuster
VM Image (REST/Web server) http://storage.apps.openstack.org/images/kloudbuster_v6.qcow2
git clone git clone https://github.com/openstack/vmtp.git git clone https://github.com/openstack/kloudbuster.git
CLI ☑️ ☑️
REST ☑️
Web UI ☑️
27DEVNET-1693
Q & A
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What’s next
• Meet us at the Cisco Live! DevNet OpenStack Demo Zone
• Video demos of VMTP and KloudBuster
• Contact: [email protected]
• VMTP: https://github.com/openstack/vmtp
• KloudBuster: https://github.com/openstack/kloudbuster
29DEVNET-1693
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.
• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.
30DEVNET-1693
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
31DEVNET-1693
Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services
Thursday, July 14th, 2016
11:30 am - 12:30pm, In the Oceanside A room
What to expect from this innovation talk
• Insights on market trends and forecasts
• Preview of key technologies and capabilities
• Innovative demonstrations of the latest and greatest products
• Better understanding of how Cisco can help you succeed
Register to attend the session live now or
watch the broadcast on cisco.com
Thank you