SLB PERF AUTOMATION

Agenda
Perf Tenants Overview Perf Test cases Perf Setup Performance Test Metrics Investigation Perf Bugs

Perf Tenants
Client Tenants ( Generates Inbound Traffic )
Queries Xstore configuration table (SLBPerfClientConfigTable) for TenantName mentioned in its service definition file periodically. Parses the Xml configuration see below and starts the http/udp/tcp traffic Xml configuration consists of
Servers endpoints to send traffic No of clients ( threads) that will send traffic MessageSize,frequency,url to target, sendtime, resttime .. Etc

Log client traffic data ( pass/fail request ) to the Xstore table mentioned in the configuration

Server Tenants ( Generates Outbound Traffic )

Based on Http/Tcp/Udp, listens on the endpoint for incoming requests from the client Same as Client tenant but reads (SLBPerfServerConfigTable) for TenantName mentioned in its service definition file periodically. Parses the Xml configuration see below and starts the http/udp/tcp outbound traffic

Perf Setup
Server Cluster (CO1CNFPerf01) -with 3 replica Deployed with SLB to be tested Deployed with Canary tenant (for inbound failures /Outbound ) Deployed with ~400- 500 server tenants of varying configurations ( varying endpoints, dips, protocols ( udp/http ) ). Client Cluster Deployed with 20 clients (http/udp Server cluster load Mux Throughput ~800 Mbits/sec HP SNAT per sec = 8 No of endpoints in the cluster ( see tenant configurations) - ~15,000
Sr 1 2 3 4 5 6 7 10 Tenant Type Http tenant Http tenant Http tenant Http tenant Http tenant Http tenant Http tenant Http tenant Artifact sh25EP6DP1DIP sh25EP6DP2DIP sh25EP6DP20DIP sh25EP6DP50DIP sh25EP6DP200DIP sh10EP0DP400DIP Private25EP100DP10DIP WebServer400EP1DIPXS_1 No Of Tenants 150 200 10 1 1 1 1 1 No Of DIPS 1 2 20 50 200 400 10 1 No OfEndpoints 25 25 25 25 25 10 25 400 No Of Direct Ports 6 6 6 6 6 0 100 0 Total Endpoints: Total Enpoints 4650 7400 1450 325 1225 10 1025 400 16485

Performance Testing Overview Process

XStore
Read configurations and upload results Read configurations and upload results

Client tenant_1

Servertenant_1 Servertenant_2

Outbound traffic

Client tenant_2

In-bound traffic
Client Cluster
Server Cluster

Client tenant_n

Servertenant_n

CanaryTenant

Performance Test Metrics

Data Plane
Measure Inbound and outbound Connectivity (Success and Failure rate) Measure Latency
Sr. no Scenario From => to ( time in UTC) Connectivity data

SLBM update

2/25/2013 7:12:10 AM => 2/25/2013 7:43:01 AM

Inbound: Success Percentage: 100% Average Latency time: 0.1 Seconds Outbound: Success Percentage: 100% Average Latency time: 8.66 Seconds

MUX update

2/25/2013 7:43:08 AM => 2/25/2013 8:19:21 AM

Inbound: Success Percentage: 100% Average Latency time: 0.01 Seconds Outbound: Success Percentage: 99.85% Average Latency time: 9.22 Seconds

SLB HP update:

2/25/2013 8:20:33 AM => 2/25/2013 8:50:14 AM

Inbound: Success Percentage: 100% Average Latency time: 0.29 Seconds Outbound: Success Percentage: 99.99% Average Latency time: 5.31 Seconds

Performance Test Metrics ( Cont )

Control Plane
Programming time
SLBM only failure LBE failure SLB + LBE failure

Deployment of various tenants

TestActions.

Serial No.

Programming Time (SLB 9)

Result

Comments

1 2

SLBM only LB-Engine restart LB-Engine restart + SLBM failover

< 1 min < 1 min

Pass Pass No Goal states resent

597.77 seconds.

Pass

Performance Test Metrics ( Cont )

SLBM Counters
No Failover of slbm for 3 days No memory build up CPU hovering around 30% (expected for 80 K snat slots) HPs connected to SLBM stable

Mux Counters
No Mux failures in the last 3 days No memory build up CPU on the 1 core (data path) is 28%

Canary failures
https://test1.diagnostics.monitoring.core.windows.net/content/s earch/search.html?table=FcCanaryMeasurementEtwTableVer1v0 &start=2013-02-24+05%3a53%3a29Z&end=2013-0225+08%3a01%3a35Z&query=%22cluster%3d%3d%22%22CO1CNF Perf01%22%22+and+IsError%3d%3dTrue%22%3b+groupby+Analy zer+&useindex=True&utc=True

Perf test cases

http://team/sites/cnftest/_layouts/xlviewer.aspx?id=/sites/cnftest/Shared%2 0Documents/SLBPerfTestcases.xlsx

Perf Improvement Plans

Currently setting pre-conditions for Mux Throughput and HP SNAT are achieved is semi-manual process ( feed back loop) ( Xstore settings changes ).
Need tools for above

More granular analysis and measurements

Checking failure on every clients Latency for different sizes of upload and download.

Tools / Resources
Tcping64 SLB Perf Test Automation path
o ..\rd_cloudnet_n_slb\services\cloudnet\test\SLBPerfTestAutoma tion

SLB Perf Tenants

o ..\rd_cloudnet_n_slb\services\cloudnet\test\SLBPerfTestAutoma tion\SLBPerfTenants

Thank you!!!