slb perf automation
Post on 11-Apr-2015
42 Views
Preview:
DESCRIPTION
TRANSCRIPT
SLB PERF AUTOMATION
Agenda
• Perf Tenants Overview • Perf Test cases• Perf Setup• Performance Test Metrics• Investigation Perf Bugs
Perf Tenants• Client Tenants ( Generates Inbound Traffic )
• Queries Xstore configuration table (SLBPerfClientConfigTable) for TenantName mentioned in its service definition file periodically.
• Parses the Xml configuration see below and starts the http/udp/tcp traffic • Xml configuration consists of
– Servers endpoints to send traffic– No of clients ( threads) that will send traffic– MessageSize,frequency,url to target, sendtime, resttime .. Etc
• Log client traffic data ( pass/fail request ) to the Xstore table mentioned in the configuration
• Server Tenants ( Generates Outbound Traffic )• Based on Http/Tcp/Udp, listens on the endpoint for incoming requests from the client• Same as Client tenant but reads (SLBPerfServerConfigTable) for TenantName mentioned in its service definition
file periodically. • Parses the Xml configuration see below and starts the http/udp/tcp outbound traffic
Sr Tenant Type Artifact No Of Tenants
No Of DIPS
No OfEndpoints
No Of Direct Ports
Total Enpoints
1 Http tenant sh25EP6DP1DIP 150 1 25 6 4650
2 Http tenant sh25EP6DP2DIP 200 2 25 6 7400
3 Http tenant sh25EP6DP20DIP 10 20 25 6 1450
4 Http tenant sh25EP6DP50DIP 1 50 25 6 325
5 Http tenant sh25EP6DP200DIP 1 200 25 6 1225
6 Http tenant sh10EP0DP400DIP 1 400 10 0 10
7 Http tenant Private25EP100DP10DIP 1 10 25 100 1025
10 Http tenant WebServer400EP1DIPXS_1 1 1 400 0 400
Total Endpoints: 16485
Server Cluster (CO1CNFPerf01) -with 3 replica• Deployed with SLB to be tested • Deployed with Canary tenant (for inbound failures /Outbound ) • Deployed with ~400- 500 server tenants of varying configurations ( varying
endpoints, dips, protocols ( udp/http ) ). Client Cluster
• Deployed with 20 clients (http/udp
Server cluster load• Mux Throughput ~800 Mbits/sec• HP SNAT per sec = 8 • No of endpoints in the cluster ( see tenant configurations) - ~15,000
Perf Setup
Performance Testing Overview Process
XStore
Server ClusterClient Cluster
Client tenant_2
Client tenant_n
Client tenant_1
Read configurations and upload results
Servertenant_1
Servertenant_2
Servertenant_n
Read configurations and upload results
Outbound traffic
In-bound traffic
CanaryTenant
• Data Plane
– Measure Inbound and outbound Connectivity (Success and Failure rate)– Measure Latency
Sr. no Scenario From => to ( time in UTC) Connectivity data
1 SLBM update
2/25/2013 7:12:10 AM => 2/25/2013 7:43:01 AM Inbound: Success Percentage: 100% Average Latency time: 0.1 Seconds
Outbound: Success Percentage: 100% Average Latency time: 8.66 Seconds
2 MUX update
2/25/2013 7:43:08 AM => 2/25/2013 8:19:21 AM Inbound: Success Percentage: 100% Average Latency time: 0.01 Seconds
Outbound: Success Percentage: 99.85% Average Latency time: 9.22 Seconds
3 SLB HP update:
2/25/2013 8:20:33 AM => 2/25/2013 8:50:14 AM Inbound: Success Percentage: 100% Average Latency time: 0.29 Seconds
Outbound: Success Percentage: 99.99% Average Latency time: 5.31 Seconds
Performance Test Metrics
• Control Plane
– Programming time • SLBM only failure• LBE failure• SLB + LBE failure
– Deployment of various tenants
Performance Test Metrics ( Cont …)
Serial No. Test-Actions. Programming Time (SLB 9) Result Comments
1 SLBM only < 1 min Pass
2 LB-Engine restart < 1 min Pass No Goal states resent
3LB-Engine restart + SLBM failover
597.77 seconds. Pass
• SLBM Counters• No Failover of slbm for 3 days • No memory build up• CPU hovering around 30% (expected for 80 K snat slots)• HP’s connected to SLBM stable
• Mux Counters• No Mux failures in the last 3 days• No memory build up• CPU on the 1 core (data path) is 28%
• Canary failures– https://test1.diagnostics.monitoring.core.windows.net/content/
search/search.html?table=FcCanaryMeasurementEtwTableVer1v0&start=2013-02-24+05%3a53%3a29Z&end=2013-02-25+08%3a01%3a35Z&query=%22cluster%3d%3d%22%22CO1CNFPerf01%22%22+and+IsError%3d%3dTrue%22%3b+groupby+Analyzer+&useindex=True&utc=True
Performance Test Metrics ( Cont …)
Perf test cases• http://team/sites/cnftest/_layouts/xlviewer.aspx?id=/sites/cnftest/Shared%2
0Documents/SLBPerfTestcases.xlsx
Perf Improvement Plans• Currently setting pre-conditions for Mux Throughput and HP SNAT are achieved is
semi-manual process ( feed back loop) ( Xstore settings changes ). – Need tools for above
• More granular analysis and measurements– Checking failure on every clients– Latency for different sizes of upload and download.
Tools / Resources
• Tcping64• SLB Perf Test Automation path
o ..\rd_cloudnet_n_slb\services\cloudnet\test\SLBPerfTestAutomation
• SLB Perf Tenantso ..\rd_cloudnet_n_slb\services\cloudnet\test\
SLBPerfTestAutomation\SLBPerfTenants
Thank you!!!
top related