introduction to achieving performance and scale on...
TRANSCRIPT
![Page 1: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/1.jpg)
Introduction to Achieving Performance and Scale on Large OpenStack Clouds
Alec Hothan, Principal Engineer
DEVNET-1693
![Page 2: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/2.jpg)
• OpenStack Performance and Scale: challenges and existing tools
• A new generation of fully integrated tools to measure tenant network and storage at scale
• Design of these tools
• What they measure and how
• How to read results
• Free and open-source: install and use the tools yourself
Agenda
![Page 3: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/3.jpg)
OpenStack Performance and ScaleTools and challenges
![Page 4: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/4.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 4DEVNET-1693
OpenStack Performance Metrics
Control Plane Metrics
Resource operations
Storage/Data PlaneTraffic metrics
Host system metrics
Networking
Compute Storage
tenants, userscpu, memory… networks
routers…
volumesimages…
Bare Metal
![Page 5: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/5.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5DEVNET-1693
Measuring these metrics is challenging
Abundance of tools
Complex measuring conditions
Automation is key
ps, top, strace, ptrace, perf…
iperf, netperf, nuttcp…
fio, jmeter, siege, wrk, cb
OpenStack rally, tempest
Cloudbench, PerfKitBenchmarker…
OpenStack variants/integration
Cloud scale
Failure conditions
Repeatable
Consistent
![Page 6: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/6.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 6DEVNET-1693
Example: web services and user experience
Density
Throughput
Latency
HTTP servers per compute node
HTTP requests per second
Latency percentile requirements:
User perception…
90th %ile 99th %ile Max
Fast
< 0.1 sec
Noticeable
< 1 sec
Slow
< 10 sec
![Page 7: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/7.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 7DEVNET-1693
Latency histogram (each traffic generator)
Latency
histogram
min
90th %ile
99th %ile
max
![Page 8: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/8.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 8DEVNET-1693
Latency histogram aggregation
+N x latency histograms
(distributed)
Overall system latency
histograms
![Page 9: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/9.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 9DEVNET-1693
Latency histograms at cloud scale
Number of HTTP servers per cloud 1000
Number of users per HTTP server x 1000
Number of requests per user/sec x 1
Test duration in seconds x 100
Total number of requests 100M
Number of latency histograms to consolidate
(1000 users/HTTP traffic generator)1000
![Page 10: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/10.jpg)
A new generation of performance/scale tools
![Page 11: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/11.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 11DEVNET-1693
Tools characteristics
Fully integrated
End to end
Accurate and precise
Tight integration with OpenStack
Automated
VM and user level metrics
All batteries included
< msec precision
Correct consolidation of metrics
Simple and AccessibleInstallation
Usage
Result interpretation
![Page 12: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/12.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 12DEVNET-1693
OpenStack tools for performance/scale
Control Plane API-level
Performance
Single Flow Performance
KloudBuster
Networking
Compute Storage
Rally
High Scale Performance
VMTP
✅
✅
✅ ✅
✅ ✅
(coming soon)NFV Data Plane✅
![Page 13: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/13.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 13DEVNET-1693
Neutron Tenant Network Flows
VMs can run on same or
different compute nodes
North-South flows (NAT)
East-West flows
![Page 14: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/14.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Networking
Compute Storage
14DEVNET-1693
VMTP
• Single flow VM-level performance
• TCP
• UDP unicast and multicast
• ICMP
• Tenant network flows
• East-West and North-South
• Intra-node and inter-node
Validation and Performance Measurement of Network Flows
• Encapsulation and Neutron implementation agnostic
• Also supports
• Provider network
• SR-IOV NICs
![Page 15: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/15.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 15DEVNET-1693
VMTP Throughput Chart
![Page 16: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/16.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 16DEVNET-1693
KloudBuster
• Configure:
• Number of HTTP servers (1 per VM)
• Number of HTTP clients per server
• HTTP requests/sec
• VM placement policy
• Stages and orchestrates everything
• Simple runs
• Progression runs (HTTP servers)
• Real time latency monitoring
OpenStack Data Plane Performance at Scale
• Aggregates and consolidates all results
• Total throughput
• Micro-sec precision latency histograms• HDR (High Dynamic Range) histogram
library
Networking
Compute Storage
![Page 17: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/17.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VM VM VM
VM
proxy
VMVM
KB
17DEVNET-1693
Data Plane Scale Overview (East-West 1 cloud)
N x http servers
N x (http traffic generators (m users))
East-West L3 traffic
N x m flows
staging
orchestration
results aggregation
Internal network
tenant
![Page 18: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/18.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
proxy VM VM VM VM
VM VMVM
VM
KB
18DEVNET-1693
Data Plane Scale Overview (North-South 2 clouds)
N x http servers
N x (http traffic generators (m users))
North-South traffic
N x m flows
staging
orchestration
results aggregation
![Page 19: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/19.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 19DEVNET-1693
Latency Chart: 1K to 20K HTTP Connections
![Page 20: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/20.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 20DEVNET-1693
Real-Time Latency Monitoring (vPC failure)One protected
interface going down
Latency stacked
iso-percentile
scrolling layers
![Page 21: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/21.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 21DEVNET-1693
KloudBuster
• Configure:
• VM count
• Cinder volumes (block storage) or ephemeral disks
• Storage workload definition
• Stages and orchestrates everything
• Progression runs (VM count)
OpenStack Storage Performance at Scale
• Aggregates and consolidates all results
• Total throughput
• micro-second precision latency histograms (HDR)
Networking
Compute Storage
![Page 22: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/22.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 22DEVNET-1693
KloudBuster Storage Scale Test Overview
ephemeral
or block storagestorage network
tenant networkstaging
orchestration
results aggregation
N x Fio testers
![Page 23: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/23.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 23DEVNET-1693
Generated Charts (NFVi profile/Ceph backend)
progressions
concurrent VMs
Latency iso-percentile
line charts
workload results tabs
![Page 24: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/24.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 24DEVNET-1693
High IOPs profile (61K IOPs random read Cinder)
![Page 25: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/25.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 25DEVNET-1693
High Throughput profile (3,743 MB/s seq read Cinder)
![Page 26: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/26.jpg)
Installation and User Interfaces
![Page 27: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/27.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Installation and User InterfacesVMTP KloudBuster
Docker Container docker pull berrypatch/vmtp docker pull berrypatch/kloudbuster
PyPI installation (pip) pip install vmtp pip install kloudbuster
VM Image (REST/Web server) http://storage.apps.openstack.org/images/kloudbuster_v6.qcow2
git clone git clone https://github.com/openstack/vmtp.git git clone https://github.com/openstack/kloudbuster.git
CLI ☑️ ☑️
REST ☑️
Web UI ☑️
27DEVNET-1693
![Page 28: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/28.jpg)
Q & A
![Page 29: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/29.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What’s next
• Meet us at the Cisco Live! DevNet OpenStack Demo Zone
• Video demos of VMTP and KloudBuster
• Contact: [email protected]
• VMTP: https://github.com/openstack/vmtp
• KloudBuster: https://github.com/openstack/kloudbuster
29DEVNET-1693
![Page 30: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/30.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.
• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.
30DEVNET-1693
![Page 31: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/31.jpg)
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
31DEVNET-1693
![Page 32: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/32.jpg)
Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services
Thursday, July 14th, 2016
11:30 am - 12:30pm, In the Oceanside A room
What to expect from this innovation talk
• Insights on market trends and forecasts
• Preview of key technologies and capabilities
• Innovative demonstrations of the latest and greatest products
• Better understanding of how Cisco can help you succeed
Register to attend the session live now or
watch the broadcast on cisco.com
![Page 33: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/33.jpg)
Thank you
![Page 34: Introduction to Achieving Performance and Scale on …d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/DEVNET-1693.pdf · Introduction to Achieving Performance and Scale on ... jmeter,](https://reader033.vdocuments.us/reader033/viewer/2022051010/5aa99e397f8b9a81188d080e/html5/thumbnails/34.jpg)