performance testing in virtualized...
TRANSCRIPT
Performance Testing in Virtualized
Environments Emily Apsey
Product Engineer
Introduction
• Product Engineer on the Performance Engineering Team
• Overview of team
- Specialty in Virtualization
- Citrix, VMWare, Microsoft
• Dedicated to Product Virtualization
- Configuration
- Scalability/Density
- Provide feedback loop across all development teams
• Vendors
- Active participation by vendors
- Hands on configuration and tuning
- NVIDIA
- Hardware
- SuperMicro and Dell r720
- K1 and K2 cards
- Driver (on hypervisor and VM)
• Microsoft
- Issaquah labs - Nov 13
- RemoteFX team
- EEC lab – Nov 14
• VMWare
- vSGA – Jan 14
- vGPU – Oct 14
- Future Reference Architecture Doc
• Citrix
- Majority of our Users
Collaborative Testing
Presentation Overview
• Why and how we tested using Nvidia GRID cards
- Test methodology for virtualized test environments
• Lesson Learned
• Future Testing
Virtualization – Esri
• ArcGIS in virtualized environments
- Users began virtualizing our ArcMap software
- Improve understanding of performance and scalability of ArcGIS in virtualized
environments
- Improve information, internally and externally, related to:
- Virtualization environments
- What to expect
- Performance
- Scalability
- New technologies
- NVIDIA K1 and K2 cards
What and Why?
• ArcGIS Pro
- New addition to ArcGIS Desktop
suite
- Desktop- analysis, publication,
visualization
- New rendering engine
- DirectX11 and OpenGL
- Currently most heavily used app
- ArcMap, based on GDI+
- Heavily used in Citrix XenApp
- Must virtualize well
- Provide great UX
Testing Goals
• Show if a shareable GPU is needed (ArcMap + Pro)
- ArcMap – single threaded, GDI+ rendering
- Pro – multi threaded, DX or OpenGL rendering
• GPU needed for both 2D and 3D workflows?
- How to monitor and communicate performance/ scalability
• Optimal VM configurations
- vCPU, Memory, GPU Profile for optimal user experience
- What and how to communicate information to users, IT administrators
- How many VMs can a single GRID card accommodate?
• Administration
- Know a little before you go on-site or create a demo of ArcGIS Desktop in virtualized
environments
Graphics Options in Virtualization
GRID vGPU Profiles
GPU Profile
• Citrix and VMWare vGPU
- Amount of Frame Buffer
- Application requires at least 1 GB of FB
- Identify Optimal profile for both VMWare and Citrix Hypervisors
- Dependent on data and workflow
- Effects density
• Microsoft Hyper-V
- Does not use GPU Profiles
- Some control by screen resolution
Data and Workflows
• 2D Denver
- Local government – Vector data model
- Database- Points, line and polygons
• 3D Virtual City
- Philly – Raster data, Z values extruded, building textures
• Zoom/Pan to a series of standardized bookmarks within App
- Time rendering (Sum of all bookmarks)
- Frames Per Second (FPS)
- Average – Sample every 100 milliseconds
- Minimum – Indicates jitter, poor user experience
Testing Tools
• Visual Studio vs. Light-weight application.
- VS -performance overhead on each VM: disk, startup, GPU
- UI maps maintenance overhead
• Login VSI
- Trail period, can be scriptable
• Accurate Performance Metrics within Application
- Internally developed ArcGIS Pro Add-In
- Exposure of valuable metrics
- Propagating events into an API method.
- Very accurate performance metrics
- Used by developers
- Light weight
ArcGIS Pro Add-In
• Specifically designed for performance
• Script and reproducible
• Self contained
• Packaging data and test package to business partners
• Great for Demos!
ArcGIS Pro Add-In UI
Automate Test Execution
• Test Controller
- Tests are executed via command line
- Calls upon a python script which creates windows scheduled tasks
- Opens ArcGIS Pro application and project
- Rendering events logged
- Automatically closes ArcGIS Pro when test completes
• Scalability – across multiple VM’s
• 1, 4, 8, 12, 16 VM’s
- TestList.csv
- Test Cleanup
- Collect logs from VM’s to File Share or SAN
- Unique Names – VM Name
Collecting Data
• ArcGIS ProAddIn Log
- Draw Times, FPS – Avg and Min
- Named Metrics
- Licensing Time
- Open Project Time
• ArcMon (ArcGIS Pro) – Internal Use only
- Hung Threads
• Nvidia SMI
- GPU Utilization, GPU Memory Utilization
• ESXiTop (VMWare)
- CPU Utilization (Host and VM)
• Python based parsers
- Export to log which is useable for reporting purposes
- Drawback - No automated UI to show system metrics
ArcGIS Pro Add-In Log
2D SumBkAvg StDev FPSAvg FPSMin
1 92.51 66.65 42.06
4 87.78 4.44 66.60 42.93
8 100.62 6.75 65.99 37.54
12 106.86 4.13 69.30 37.54
16 111.95 4.95 74.11 23.70
3D SumBkAvg StDev FPSAvg FPSMin
1 81.00 54.02 13.74
4 79.66 4.35 53.41 14.65
8 86.95 4.33 47.38 10.71
12 92.00 6.19 43.23 9.64
16 103.42 8.00 40.91 7.66
Python
based
parser
Additional Hands-On Testing
• Further validate automated test findings
• Usability tests were executed manually by team members
• Further qualifying the UX
- Validating rendering quality to human eye
- Overall responsiveness of the application
User Experience Ranking
1 to 5 Unacceptably jerky; Poor to annoying user experience
6 Jerky, but useable
7 Moderately smooth, moderate jerkiness
8 Smooth, minor jerkiness
9 to 10 Very smooth; little to no difference between a non-VDI solution. No apparent jerkiness, lag/tearing, or texturing delays.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Host CPU Utilization Scalability of 240Q Profile - 3D Data
16 VMs:Host CPU
14 VMs:Host CPU
12 VMs:Host CPU
10 VMs:Host CPU
0
10
20
30
40
50
60
70
80
90
100
16
:13
:02
16
:13
:18
16
:13
:33
16
:13
:49
16
:14
:04
16
:14
:20
16
:14
:35
16
:14
:51
16
:15
:07
16
:15
:22
16
:15
:38
16
:15
:54
16
:16
:09
16
:16
:25
16
:16
:40
16
:16
:56
16
:17
:11
16
:17
:27
16
:17
:43
16
:17
:58
16
:18
:14
16
:18
:29
16
:18
:45
16
:19
:01
16
:19
:16
16
:19
:32
16
:19
:47
16
:20
:03
16
:20
:18
16
:20
:34
16
:20
:50
16
:21
:05
16
:21
:21
16
:21
:36
16
:21
:52
16
:22
:08
16
:22
:23
16
:22
:39
16
:22
:54
16
:23
:10
16
:23
:25
16
:23
:41
Host Utilization Rates 16VMs - 6vCPU: 3D data
GPU1 %Util
GPU2 %Util
GPU3 %Util
GPU4 %Util
Cpu(Total)\% Util Time
Answers obtained from testing
• ArcGIS Pro
- GPU is required for 3D data; K2 card preferred
- GPU aids 2D data delivery by relieving CPU; considered recommended.
GPU can be configured to be shareable (vGPU)
- Exception XenApp; requires Pass Through GPU
- Optimal VM configuration can vary by HyperVisor (encoding/decoding)
GPU Utilization
ArcGIS Pro – XenApp 7.6
0
10
20
30
40
50
60
70
80
90
100
XenApp 7.6 - 4 Users 3D 16vCPU, 16 GB of Memory
XenApp76\% Processor Time XenApp76\% GPU Usage XenApp76\% GPU Memory Usage
Requires GPU Pass-Through
- K2 GPU Memory = 4GB
OS does not do well splitting
GPU resources across
different terminal sessions
XenDesktop recommended
- Dedicated resources
- vGPU profile manages FB
- Dictates VM density
Esri collaborating with Nvidia
and Citrix
Recommended GPU Profiles
3D Data
2D Data
2D Data
• Dependent on user data and workflows
Answers obtained from testing (continued)
• ArcMap
• Rendering engine does not
utilize GPU
- GPU is not heavily utilized
- Only when user explicitly
executes hardware
acceleration operations
Map docs
Rendering Sum Times (mins)
Lesson Learned
• Online data introducing network latency
- Eliminate ArcGIS Online data for initial benchmarks
• Licensing
- License Server vs. Application licensing
- Eliminates logins into Application
• FPS sampling – Avg and Min
• Think time (5 seconds vs 10 seconds)
- Aggressively intensive, further refine to closely model user workflows
• Nvidia Drivers – HyperVisor and VM, same release
Future testing
• Mixed workflows
- Initial testing focused on rendering pipeline
- Add analytical operations (CPU intensive)
- How GPU Utilization and density is affected when users are simultaneously running a CPU
intensive geoprocessing task?
• Mixed Virtualization Vendor Solutions
- VMWare ESXi XenDesktop
• Storage – SAN
- More realistic – large infrastructure don’t use SSDs
• Cloud based VDI (DaaS)
- Increasing questions and interest
- Distributors – Europe, NZ, US
- Esri Redlands – Ed Services, Customer Holistic Testing
