ProxySG Performance Webcast

PROXYSG PERFORMANCE
WEBCAST
PAUL KAO
Director Product Management
paul.kao@bluecoat.com
December 16, 2014
Blue Coat Confidential
Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.
AGENDA
ProxySG Overview
Architecture (SGOS, CW, SW, Policy checkpoints)
System resources/metrics
Performance Model
Factors Impacting Performance
Authentication, ICAP, Policy, SSL, misc.
Critical Resource Monitoring

CPU, Memory, CW, network
Troubleshooting Performance Problems

Baseline, CPU monitor, Policy trace, Sysinfo
PROXYSG OVERVIEW
Copyright 2014
2013 Blue Coat Systems Inc. All Rights Reserved.
SGOS OVERVIEW
SGOS is a secure, hardened and proprietary OS developed

by Blue Coat to be robust and scalable at the highest levels
of performance
It is unlike other operating systems
Microkernel, message pass architecture using admin and worker
model for processes
Run to completion semantics
Uses an object store (cache engine/cache admin), no file system, no
directory structure
Policy is deeply integrated into SGOS

Checkpoints at entry/exit of proxy traffic flow to evaluate policy
transaction
SGOS ARCHITECTURE
Client Worker (CW) Processes HTTP session between SG and client

Server Worker (SW) Processes HTTP session between SG and OCS
Retrieval Worker (RW) Pipeline and keeps the content of the cache fresh
Specialized Worker Handles a specific protocol, like streaming, CIFS, etc.
POLICY CHECKPOINTS
server_url.domain=
client.address=
http.response.apparent_data_type=
set(response.header.Set-Cookie, x")
Workers provide available information to policy

Policy transaction re-evaluated at each check point
Policy decisions are stored a policy ticket
PROXYSG APPLIANCE
PHYSICAL RESOURCES
Core appliance resources are:
CPU, Memory, Disk, Network Interface
CPU
No CPU throttling - continue to handle more load until appliance is at
CPU limit (assuming other resources are available). At this point,
requests take longer to process, with longer transaction times.
Memory
Threshold Monitor (TM) engages at 80% memory pressure, goes into
regulation, which limits HTTP acceptance to reduce rate of processing
new incoming connections.
Disk
At high disk utilization, back off mechanisms will engage to maintain
throughput at the expense of cache efficiency (disk read/writes)
Network Interface
Will trigger event log if network interface is saturated (TCP livelock)
PROXYSG APPLIANCE METRIC

USER COUNT & CLIENT WORKER
Appliance has fixed CPU/Memory/Disk/Network resources
One additional metric Licensed Client IP
From a sizing perspective, Licensed Client IP is the maximum
unique IPs that a given SG appliance should handle
Usually, Client IP is synonymous with user/employee
Licensed Client IP
A soft limit on HW appliances
A hard limit on Virtual appliances
Performance of appliances constrained by available number of
HTTP/TCP-Tunnel Client Workers (CW) for processing
Each appliance model has its own CW limit
CW limit does not limit any other TCP session on SG

CW limit is only a count of active client side sessions
PERFORMANCE MODEL
Copyright 2014
2013 Blue Coat Systems Inc. All Rights Reserved.
PERFORMANCE MODEL
10
FACTORS IMPACTING PERFORMANCE
1.
2.
Network deployment
3.
Authentication mode
4.
DNS, Content Filtering
5.
ICAP REQMOD (DLP)
6.
7.
Client
ICAP RESPMOD (CAS)

System services, logging
8.
9.
Policy
SSL
11
PERFORMANCE FACTORS
1. CLIENT
12
1. CLIENT SIDE
Client to SG connection (client side)
Limited by HTTP/TCP-Tunnel CW
User (client IP) is not an enforced metric. User is a model for sizing
CW limit does not include other TCP sessions (auth, ICAP, bypass,..)
Dont confuse TCP-Tunnel proxy CW as the TCP connection limit!!!
S-Series hardware
S-series models 5 connections/per user (user = unique client IP)
S200-10 S200-20 S200-30 S200-40 S400-20 S400-30 S400-40 S500-10 S500-20
Users
Max CW
400
1,200
2,600
5,000
6,000
2,000
6,000
13,000
25,000
30,000
14,000
25,000
30,000
50,000
70,000 125,000 150,000 250,000
Examples:
Financial trader, 50 conns per user
Kiosk, 1 connection per user
13
PERFORMANCE FACTORS
2. NETWORK DEPLOYMENT
14
2. NETWORK DEPLOYMENT
Network 101
Link/duplex settings
WCCP
GRE vs L2
Set MTU appropriately to avoid fragmentation with GRE
Physically Inline (bridging)

Good for smaller sites
Larger sites with significant non web (bypass) traffic that can
consume network resources
15
PERFORMANCE FACTORS
3. AUTHENTICATION MODE
16
3. AUTHENTICATION
Evaluated at CI
Choice of Authentication mode can impact performance
Explicit proxy with NTLM: SG issues a 407 challenge for each
connection
IP Surrogate: After initial authentication, will use authentication cache
Kerberos: credentials validated without need to contact DC
NTLM does not scale well

NTLM credential cannot be cached, and must be validated by DC
Default Windows configuration processes only one request at a time
via Schannel
Exacerbated by latency and load on DC (SG-DC or SG-BCAA-DC)
Kerberos preferred for scalability

17
PERFORMANCE FACTORS
4. DNS, CONTENT FILTERING
18
4. DNS, CONTENT FILTER
DNS
Not a high consumer of CPU, but can be cause of latency
If external DNS servers are slow/overloaded, Proxy will amplify the
problem
Use caution for policies/logging that trigger RDNS lookups
Content Filtering (evaluated at Client In)

BCWF
Efficient categorization for high performance
Settings for lower memory footprint appliances
Web Pulse DRTR

Minimal overhead
19
PERFORMANCE FACTORS
5. ICAP REQMOD (DLP)
20
5. ICAP GENERAL & ICAP REQMOD
ICAP Internet Content Adaptation Protocol

Used to vector both REQuest and RESPonse traffic for scanning
ICAP General Performance considerations
Persistent connection with re-use

Sufficient ICAP connections to handle throughput or queuing will occur
Relatively expensive content must be sent over ICAP
Policy dictates how much content is sent (ICAP best practices)
Worst case is all content sent to ICAP
ICAP REQMOD evaluated at CI (before Server Out)

Scan data on outbound request
Scanning POST body data
Incremental cost due to low volume of data (POST body data)

21
PERFORMANCE FACTORS
6. ICAP RESPMOD (CAS/AV)
22
6. ICAP RESPMOD
(CONTENT ANALYSIS)
Evaluated at Server In (SI)
Higher cost due to volume of incoming request data
For ICAP RESPMOD, cache to disk for performance (no
need to return payload when response is 204 No
Modification)
Infinite Streams
ICAP deferred connections
ICAP mirroring (SG6.5)
Secure ICAP
SSL cost in initial connection setup
SSL overhead of bulk encryption low
23
PERFORMANCE FACTORS
7. SYSTEM SERVICES
24
7. SYSTEM SERVICES
Access logging
Log entry written when connection is complete
A few percent overhead when enabled
Obviously more overhead if multiple log facilities in use
Health Checks
SNMP
Attack Detection
Failover, SGRP (VRRP)
Connection Forwarding
Scripts, polling of local policy
Snapshots, Debug logs
25
PERFORMANCE FACTORS
8. POLICY
26
8. POLICY AND CPU
Policy impact can range from minimal

to majority of CPU cost on SG
Look for policy best practices
Avoid regexes, order rules most likely to
match first, group rules, etc.
A point of reference
Policy used for SWG/ICAP/SSL consumes
about 15% of total CPU
Scale appropriately for higher/lower policy
usage
Variation across platforms
Only use as a rule of thumb
Not guaranteed to be exact
May change in the future
27
PERFORMANCE FACTORS
9. SSL
28
9. SSL INTERCEPT
29
CERTIFICATE EMULATION STATISTICS

(SG6.5.5.1)
SSL Statistics (in Sysinfo and SSL/Statistics URL)
https://SG_IP:8082/SSL/statistics
Certificate Emulation
SPS51
Total certificates emulated
2,264
SPS52
Total RSA 2048 bit key certificates emulated
2,250
SPS53
Current cached emulated server certificates
1,078
SPS54
Total emulated server certificates added to cache
1,390
SPS55
Total emulated server certificates removed from cache due to timeout
SPS56
Total emulated server certificates removed from cache due to maxsize
SPS57
Total emulated server certificates removed from cache due to signature mismatch
312
SPS58
Total emulated server certificates removed from cache due to config changes
SPS59
Total emulated server certificates add to cache failures
874
SPS61
Total server certificate cache successful lookups
42,109
SPS62
Total proxy certificates emulated
SPS63
Total certificate emulation failures
% certificate emulation change = SPS51 / (SPS51 + SPS61)

In steady state, % of new emulations should be very small
31
SSL & WILDCARD CERTIFICATES

Wildcard certificates (e.g., *.google.com and others)
Google and other properties starting to use wildcard certificates

Wildcards allow certs with the same CN to appear on multiple servers.
Different servers have different certs (different expiration, keys, extensions, etc.)
SGs emulated certificates are cached using CN as the key value
SG is seeing these different certs all with the same CN, causing a collision in the
certificate cache and forcing SG to re-emulate certificate
This can lead to high CPU on all SG6.x versions (6.2 through to 6.5)
Future certificate cache enhancement planned, use policy resolution below
Wildcard certificates Resolution

Install the following policy (creates a unique instance for each certificate)
<ssl-intercept>
ssl.forward_proxy(https) ssl.forward_proxy.splash_text("$(xrs-certificate-serial-number)$(x-rs-certificate-validfrom)$(x-rs-certificate-valid-to)")
Monitor efficacy using % certificate emulations (=SPS51 / (SPS51 + SPS61))
32
SSL PROXY CERTIFICATE CACHE
Advanced URL https://SG_IP:8082/sslproxy/certcache

SSL Proxy Certificate Cache
URL_Path /sslproxy/certcache
<PRE>Certificate Cache Contents
Number of cache entries: 1078

Common Name, Splash Text, Splash URL, Server Keyring
rtax.criteo.com,, $(x-rs-certificate-serial-number)
cloudfront.net,,
www.bgov.com,,
s3.wpc.edgecastcdn.net,,
www.palottery.state.pa.us,,
beacon.walmart.com,,
$(x-rs-certificate-valid-from)
$(x-rs-certificate-valid-to)
*.linkedin.com, 020000000001456FAAB168CFFE4A Apr 17 12:30:30 2014 GMT Apr 17 12:30:30 2015 GMT,
beis.cc.iup.edu,,
www.syncaccess.net,,
*.widget.custhelp.com,062306473BAC372720E3496C661336F0Feb 28 00:00:00 2014 GMTMar 30 23:59:59 2015 GMT,
ads.dotomi.com,02F7CASep 3 03:33:55 2014 GMTNov 5 14:50:00 2015 GMT,
*.wer.microsoft.com,28DB34EB000100005898Apr 4 17:56:38 2013 GMTApr 4 17:56:38 2015 GMT,
*.ebay.com,,
*.googleusercontent.com,,
*.reson8.com,D3C03378DC74A2ABF36132E69E273C45Jun 2 00:00:00 2014 GMTJul 21 23:59:59 2015 GMT,
stage.tracker.springserve.com,,
services.addons.mozilla.org,,
*.tapad.com,024906Jun 2 08:10:18 2013 GMTSep 3 03:30:13 2016 GMT,
*.dropbox.com,,
33
WILDCARD CERTIFICATE RESOLUTION

VPM
From VPM, edit SSL-Intercept layer
Click on "Splash Text" and paste

the below text in the box:
$(x-rs-certificate-serialnumber)$(x-rs-certificate-validfrom)$(x-rs-certificate-valid-to)
34
CRITICAL RESOURCE MONITORING &

TROUBLESHOOTING PERFORMANCE
35
CRITICAL RESOURCE MONITORING

What key metrics should be monitored?
CPU Utilization
Memory Pressure
Network Throughput
Client side HTTP connections (CWs)
Response time through ProxySG (and DNS response time)
Establish a Baseline and Peak utilization
Beware trend averages over long time intervals that flatten peaks
Identify true peak CPU utilization in busy hour
Peak CPU typically correlates with memory and connections
Baseline CPU distribution across components with CPU monitor
SNMP MIBs
See BLUECOAT-SG-PROXY-MIB.txt for resource monitoring
Also BLUECOAT-SG-ICAP-MIB.txt has been added in SG6.5
See Critical Resource Monitoring of the ProxySG on BTO

Has the connection limit for each platform
36
Common performance issues

High CPU
Slowness
Easier to troubleshoot if you have already established a point of

reference (baseline)
Issue repeatable?
Time of occurrence
Over a long period of time?
Over a short period of time?
Intermittent?
Tools
CPU Monitor
Sysinfo snapshots
Policy trace
37
HIGH CPU
External Network Factors
Typically not going to be cause of high CPU on SG
Dependent Factors
Problem with Authentication server or Auth configuration (Kerberos falling back to
NTLM)
Internal factors
Audit config changes to SG complex policy/regexes?

Loops authentication, forwarding loops
Upgrade of SG version/bug?
Undersized?
Self inflicted - enabling snapshots/debug logs too frequently?
Traffic patterns that change SG resource utilization
Change in traffic pattern resulted in lack of available resources

Change in traffic pattern hitting expensive policy
Under attack? Viruses, rogue apps, open proxy
Bug?
38
HIGH CPU
Data collection
Enable CPU monitor
Create and enable 5 min snapshots
Dont change the existing daily or hourly snapshot values
Is high CPU constant, randomly spiking or just at peak busy

hour?
39
HIGH CPU EXAMPLE 1
Example-1
>>>> CPU is high for Policy evaluation
CPU Monitor
Lots of regex rules in policy

Very complex policy (lots of rules)
Authentication problem
High number of transection per sec
CPU 0
97%
Policy evaluation - HTTP
81%
HTTP and FTP
5%
Object Store
5%
Access Logging
2%
Miscellaneous
1%
CPU 1
94%
75%
TCPIP
11%
HTTP and FTP
5%
DNS service
1%
40
HIGH CPU EXAMPLE 2
Example-2
>>>> CPU is high in Object Store
CPU Monitor
System had hard time to read or

write anything to disk.
Indicate problem with Disk.
CPU 0
100%
Object Store
ce_admin
Access Logging
CPU 1
98%
97%
1%
19%
TCPIP
8%
tcpip
HTTP and FTP
7%
6%
http
1%
kernel
1%

policy_enforcement
3%
1%
41
HIGH CPU EXAMPLE 3
Example-3
>>>> CPU is high across multiple components
CPU Monitor:
Configured interval duration:
5 seconds
Current interval complete in:
2 seconds
CPU is almost evenly distributed

between
HTTP and FTP
TCPIP
Object store
Policy evaluation
Load/sizing issue
CPU 0
77%
TCPIP
31%
HTTP and FTP
17%
Object Store
13%
7%
DNS service
1%
Access Logging
1%
Miscellaneous
1%
42
HIGH CPU EXAMPLE 4
Example-4
>>>> CPU is high in TCP
Configured interval duration:
5 seconds
Current interval complete in:
0 seconds
Too much bypass traffic.

Too many TCP connections.
May be a TCP attack.
Too many entries in time wait
state.
CPU 0
35%
Object Store
14%
HTTP and FTP
13%
3%
Miscellaneous
2%
CPU 1
100%
TCPIP
HTTP and FTP
5%
1%
DNS service
1%
90%
43
SLOWNESS
Can be difficult to troubleshoot, especially if intermittent
External Network Factors
audit change requests to (upstream) network (over last week)
E.g., new FW installed last weekend
Network: Packet loss, retransmissions, asymmetric routing
Dependent Factors
DNS, Authentication, 3rd party ICAP servers
Internal factors
Audit config changes to SG, starting with most recent (work
backwards to last 2-3 days if intermittent problem)
Traffic patterns that change SG resource utilization

SSL ciphers
Attack/bot
44
SLOWNESS
Data collection
May require multiple rounds of troubleshooting (PCAP & Sysinfo snapshots)
Easiest to target specific client or server to test
May need to test with different configurations and capture with different filter to
narrow down the issue
Important to analyze Snapshots.

Check if resource load are high (e.g. CPU, memory, HTTP worker and etc.)
Check on any priority 1 events & health check occurred during the time of the issue.
Check on the trend of the issue (how frequent it occurs and any correlation with other
components or stats)
45
SUMMARY
ProxySG Architecture
Appliance resources, CW limit
Performance Model
Factors Impacting Performance
ICAP (built into sizing model/guide)
Policy (sky is the limit)
SSL (SSL traffic mix amount of SSL decryption)
Resource and Health Monitoring

Critical resource monitoring
Health monitoring
Troubleshooting
Importance of establishing a performance baseline
Tools to troubleshoot performance
46
47
THANK YOU FOR JOINING TODAY!
Please provide feedback on this webcast and suggestions

for future webcasts to:
john.dyer@bluecoat.com
Webcast replay and slide deck found here

within 48 hours:
https://bto.bluecoat.com/training/customersupport-technical-webcasts
(Requires BTO log-in)
48
BLUE COAT CUSTOMER FORUMS
Community where you can learn from and

share your valuable knowledge and experience
with other Blue Coat customers
Research, post and reply to topics relevant to
you at your own convenience
Blue Coat Moderator Team ready to offer
guidance, answer questions, and help get you
on the right track
Access at forums.bluecoat.com and register
for an account today!
49
QUICK SURVEY
We are truly committed to continuous improvement for these

Technical Webcasts. At the end of the event you will be redirected to a very short survey about satisfaction with this
Program. Please help us out by taking two minutes to
complete it. Thank you!
Questions for Paul?

50
Questions?
51
PROXYSG PERFORMANCE WEBCAST

QUESTIONS
Q1:Is a Client Worker (CW) created for every HTTP
connection? or a single CW can handle multiple HTTP
connections?
Q2: The cost with the wildcard certificates -- does that
generate a lower "cost" in a reverse proxy model where the
wildcard cert is on the proxy, not on the OCS/Internet?
Q3: How does Licensed Client IP correlate to Concurrent
users?
Q4: Is it possible to monitor the number of client
connections per IP in the SG?
52

QUESTIONS
Q5: So I notice that BC has recommended the S-series to
replace many 510/810/300/600 ProxySG's - does this mean
the S-series is exactly the same or are they truly an
improvement in performance and connection numbers?
Q6: In "Critical Resource Monitoring" Guide talks about
Connection limit per device... that's the same than CW that
has that particular model?
Q7: What about multiple users on a single Citrix server?
Licenses
Q8: Client side, how will you typically handle the reach of
TCP limit (65,000) for a specific IP in the larger models that
could handle a lot more connections?
53

QUESTIONS
Q9: Note: IP surrogate won't work in a NATed load balancer
configuration. About IP surrogate, do you have advice
about implementation. Especially about the possibility that
one IP can be shared between users (hiding IP or Citrix
users)?
Q10: we are currently logging the category of URLs. What
kind of impact can we expect if we add the application field
in our logs for BCWF?
Q11: where can we view utilization on the proxy like the pie
graph like the BC SWG policy pie chart?
Q12: Can you talk about ECDHE, from a performance
standpoint, what should the default policy be set to?
54

QUESTIONS
Q13: for ICAP, will the proxy perform better if a dedicated
interface is assigned for ICAP communication versus the
same interface for all other user traffic?
Q14: Today's sizing guides assumes 15% of SSL traffic.
That's not realistic. At least 60% of Web browsing is SSL. Is
there any sizing guide that assumes higher SSL percentage
use? We're having serious problems sizing the right SG to
our customers.
Q15: Will be sysinfo "reader" (tool that support rep uses)
available for channels?
Q16: For troubleshooting we saw recommendations to add
snapshots every 5 minutes. How much free CPU resources
(%) should be free to enable this without generating a new
problem ?
55

QUESTIONS
Q17: When the ProxySG is in high CPU/high memory panic
mode.. is there anything we can do to bring that down other
than reboot the device?
Q18: Regarding the CW limit: We've long seen it as our
primary bottleneck. Does Bluecoat publish the CW figures
publically yet, or do we have to ask our VAR to get the
figures on the proxy models at purchase time?
Q19: does rebooting the proxy impact performance from a
caching standpoint?
Q20: Hi, regarding memory pressure - do I understand
correctly that while the proxy is in the regulation state, it
just regulates NEW client connections but keeps
processing the active ones?
56

QUESTIONS
Q21: Is it common to see a small amount of traffic bound
for blocked URLs on our outside sensors? Is this part of the
handshake process before the block is implemented?
Q22: Good morning. Regarding the licensed client IP... Is
there a way for us to identify the "soft" limit on the
ProxySG's GUI or CLI?
Q23: Is there a way to monitor the number of CWs in use?
Q24: What is the cost of running Trace layers (80 and 443)
in the VPM?
Q25: What might it indicate if the memory utilization is
significantly higher than the cpu utilization on average?
57

QUESTIONS
Q26: For bandwidth performance issues. Is there a way to
see who is downloading what in real-time?
Q27: If the network throughput is above the recommended
threshold by bluecoat but CPU is still normal, will this
cause any issue on performance?
Q28: From a performance standpoint. What are the
recommendations around attack detection and delete on
abandonment?
58

ProxySG Performance Webcast

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

ProxySG Performance Webcast

Încărcat de

Drepturi de autor:

Formate disponibile

PROXYSG PERFORMANCE

Blue Coat Confidential

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Critical Resource Monitoring

Troubleshooting Performance Problems

Blue Coat Confidential

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Blue Coat Confidential

SGOS is a secure, hardened and proprietary OS developed

Policy is deeply integrated into SGOS

Blue Coat Confidential

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Client Worker (CW) Processes HTTP session between SG and client

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Workers provide available information to policy

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

PROXYSG APPLIANCE METRIC

CW limit does not limit any other TCP session on SG

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Blue Coat Confidential

Blue Coat Confidential

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

FACTORS IMPACTING PERFORMANCE

DNS, Content Filtering

ICAP REQMOD (DLP)

ICAP RESPMOD (CAS)

Blue Coat Confidential

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Blue Coat Confidential

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

70,000 125,000 150,000 250,000

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Blue Coat Confidential

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Physically Inline (bridging)

Blue Coat Confidential

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Blue Coat Confidential

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

NTLM does not scale well

Kerberos preferred for scalability

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Blue Coat Confidential

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

4. DNS, CONTENT FILTER

Content Filtering (evaluated at Client In)

Web Pulse DRTR

Blue Coat Confidential

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Blue Coat Confidential

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

5. ICAP GENERAL & ICAP REQMOD

ICAP Internet Content Adaptation Protocol

ICAP General Performance considerations

Persistent connection with re-use

ICAP REQMOD evaluated at CI (before Server Out)

Incremental cost due to low volume of data (POST body data)

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Blue Coat Confidential

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.

Blue Coat Confidential

Copyright 2014 Blue Coat Systems Inc. All Rights Reserved.