Psts v111 Student Us WMK

?
PRZTcBWP_TaCa^dQ[TbW^^cX]V
ETabX^]
BcdST]cCTgcQ^^Z
Property of Blue Touch Training Services.

NOT for Distribution. For Reference Purposes Only.
BlueTouch Training Services — PacketShaper Troubleshooting Course v 1.1.1
Contact Information
Blue Coat Systems Inc.

420 North Mary Avenue
Sunnyvale, California 94085
North America (USA) Toll Free: +1.866.302.2628 (866.30.BCOAT)

North America Direct (USA): +1.408.220.2200
Asia Pacific Rim (Hong Kong): +852.2166.8121
Europe, Middle East, and Africa (United Kingdom): +44 (0) 1276 854 100
training@bluecoat.com
training.books@bluecoat.com
www.bluecoat.com
Copyright© 1999-2010 Blue Coat Systems, Inc. All rights reserved worldwide. No part of this document may be
reproduced by any means nor modified, decompiled, disassembled, published or distributed, in whole or in part, or
translated to any electronic medium or other means without the written consent of Blue Coat Systems, Inc. All right, title
and interest in and to the Software and documentation are and shall remain the exclusive property of Blue Coat Systems,
Inc. and its licensors. BluePlanet™, CacheFlow™, DRTR™, ProxyAV™, ProxyClient™, ProxyRA Connector™, ProxyRA
Manager™, SGOS™, and WebPulse™ are trademarks of Blue Coat Systems, Inc. Blue Coat®, BlueSource®, BlueTouch®,
Control Is Yours®, K9®, IntelligenceCenter®, PacketShaper®, ProxySG®, Permeo®, the Permeo logo, and the Blue Coat
logo are registered trademarks of Blue Coat Systems, Inc. All other trademarks contained in this document and in the
Software are the property of their respective owners.
BLUE COAT SYSTEMS, INC. DISCLAIMS ALL WARRANTIES, CONDITIONS OR OTHER TERMS, EXPRESS OR
IMPLIED, STATUTORY OR OTHERWISE, ON SOFTWARE AND DOCUMENTATION FURNISHED HEREUNDER
INCLUDING WITHOUT LIMITATION THE WARRANTIES OF DESIGN, MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL BLUE COAT SYSTEMS, INC., ITS
SUPPLIERS OR ITS LICENSORS BE LIABLE FOR ANY DAMAGES, WHETHER ARISING IN TORT, CONTRACT OR
ANY OTHER LEGAL THEORY EVEN IF BLUE COAT SYSTEMS, INC. HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
ii Property of Blue Touch Training Services.

Table of Contents
Course Introduction .....................................................................................1
Chapter 1: Troubleshooting Methodology ...................................................3
Chapter 2: BlueTouch Online ....................................................................15
Chapter 3: Hardware Overview ................................................................. 23
Chapter 4: PacketShaper Boot Process ....................................................37
Chapter 5: Understanding PacketShaper Commands ...............................47
Chapter 6: Analyzing PacketShaper Logs .................................................63
Chapter 7: Hardware Troubleshooting .......................................................73
Chapter 8: Hardware Failure Case Studies ............................................... 87
Chapter 9: Configuration Issues ................................................................99
Chapter 10: Configuration Case Study ....................................................113
Chapter 11: Classification Case Study ....................................................119
Chapter 12: Troubleshooting PacketWise Software ................................129
Chapter 13: Software Image Case Study ................................................141
Chapter 14: Access and Performance Issues .........................................147
Chapter 15: Performance Case Study .....................................................159
Chapter 16: PacketShaper Traffic Flows .................................................167
Chapter 17: PacketShaper Report Issues ............................................... 185
Property of Blue Touch Training Services. iii

iv Property of Blue Touch Training Services.

Course Introduction
The PacketShaper ®Troubleshooting course is intended for support engineers who wish to deliver
high-quality service.
After studying this course, you will understand:
• How to approach the troubleshooting process.
• Specific tools for diagnosing issues with deployments of the Blue Coat® PacketShaper.
• Details about specific PacketShaper hardware and software components.
• How to analyze and resolve PacketShaper hardware and software issues.
This course assumes that you have completed the following PacketShaper training courses or
have equivalent PacketShaper experience:
• Blue Coat Certified PacketShaper Administrator (BCPSA).
• Blue Coat Certified PacketShaper Professional (BCPSP).
In many cases, step-by-step instructions for simple processes are not provided. Students should be
familiar with basic networking concepts, such as local-area networks, the Internet, monitoring,
TCP/IP concepts, and packet inspection methods.
Applicable Software Versions

This course is based on version 8.4 of the PacketWise® operating system that is used on the
PacketShaper. In earlier versions of PacketWise, some features described in this course might not
work as described here, and the appearance and functionality of screens, menus, commands, and
displays might be different from what you see here.
Typographic Conventions
• In this book, text appearing in this font generally is text that is part of a graphical user
interface. This includes text in labels, names of buttons and menus, and Web page addresses
that you type into a Web browser.
• Text appearing in this font generally is text that is part of a command-line interface. This
includes prompts, user input, and responses. This font also is used to show the content of
some communication protocols, such as headers, commands, and data between a client and a
server.
• In both cases, text that appears in italics like this or like this represents text that you should
replace with text specific to your deployment. For example, the URL
https://applianceIPaddr:8082 appears often in this book. In this example, the text
applainceIPaddr should be replaced with the actual four-octet numeric IP address of your
PacketShaper .
Property of Blue Touch Training Services. 1

2 Property of Blue Touch Training Services.

Chapter 1: Troubleshooting Methodology
When a customer’s network security and performance are on the line, it is imperative that the
customer have access to qualified technical resources. Prompt, efficient, capable response to
customer issues helps ensure customer satisfaction.
This chapter discusses how to approach the solution to a technical issue. By doing so, you can
deliver a methodology around a customer service request using a consistent and scientific
approach that will lead from information gathering to solution delivery. The central idea is to
impart a uniform approach to arriving at a solution that any other support engineer on the team
would have reached using the same strategy and methodology.
The idea is to gather information in the most appropriate way and ask the customer for the most
necessary information that is needed to arrive at the solution.
After studying this chapter, you will understand:
• The three phases of troubleshooting.
• How to define an issue and gather relevant information.
• How to use diagrams to identify potential causes.
• How to identify the most likely causes and investigate them.
• How to choose the best solution.
• How to implement the solution, communicate it to the customer, and document it for future
reference.
Finally, a case study takes a typical problem report and walks through diagnosis and solution of
the problem.

Troubleshooting Methodology
Divide problem solving into three phases
– Research
– Analysis
– Solution
Use Ishikawa diagram

– Brainstorm on potential issues
– Root cause analysis
© Blue Coa t Systems, In c. 2009 . All Rights Reserve d.
6OLGH2YHUYLHZ
The basic troubleshooting methodology described here involves dividing the problem-solving
technique into three phases. It starts with the customer making a contact either by phone or e-mail
to technical support. Then, support engineers work to deliver a solution.
This process needs is divided into three parts:
• Research: The solution to any problem starts with researching the problem itself. Questions
such as “What is the real cause of the problem?” can be helpful at this point. Such questions
will help in understanding the cause behind the problem, rather than trying to arrive at a
solution at this stage.
• Analysis: To better understand any problem, it is worthwhile to analyze the problem in depth.
Use any or all of the available information received from the customer to do an in-depth
analysis of the problem. Use the available data to replicate the problem to identify the root
cause.
• Solution: Implement a proposed solution based on the analysis in the previous phase. Try to
implement the solution in a small lab environment to ensure that it can be implemented in a
production environment. Communicate the solution to the customer, verify that the solution is
satisfactory, and document it for future reference.
One problem-solving tool is the Ishikawa diagram, also known as a fishbone diagram. This diagram
helps you brainstorm the potential issues and correlate those issues to an cause-and-effect
situation, ultimately leading to the solution that is required to solve the problem. Ishikawa
diagrams are discussed in detail later in this chapter.

Troubleshooting Phases
6OLGH7URXEOHVKRRWLQJSKDVHV
Three phases are involved in the troubleshooting methodology.
Phase 1: Research
Defining the issue is the most important task in the research phase. Make sure that you
understand what the real issue is. Ensure that you understand the primary need of the customer
rather than the root cause of the issue. Why is the customer facing this issue? What was the
customer trying to achieve when the problem arose? At this point, avoid adding any notes about
what the real cause might be. This stage is primarily an information-gathering stage, not an
attempt to determine a possible cause.
Next, describe any associated symptoms as mentioned by the customer. Document any changes
that might occur as a result of the symptoms experienced.
Important: Make sure you document all changes before and after the occurrence of the
specific problem. In instances involving large production environments, many
administrators, and different policy implementations, documenting these
changes can be extremely difficult.
Most times, the person reporting such issues might not have seen the problem occur personally
but just has the responsibility to report it. In such cases, some information might unintentionally
be left out. This leads to a situation where a support engineer does not receive all of the necessary
information from the customer. It is always good to probe the customer in a professional manner
to obtain all necessary information for problem-solving.
Phase 2: Analysis
In this important phase, the Ishikawa diagram comes into play. The support engineer must do a
thorough analysis of the possible causes of the problem. List all possible potential and probable
causes, and then investigate the most likely causes of the problem.

Phase 3: Solution
Once you have identified the most likely cause of the problem, try developing a solution to the
problem. Consider all possible variables that might hinder the proper execution of the solution at
this stage. If the solution you have come up with is reproducible, test it several times before
implementing it and communicating it to the customer. The last step in this process is to document
the solution. Documenting the solution helps provide the solution more efficiently to a different
customer with the same problem. That way, the service request count is reduced considerably.

Phase 1-1: Define the Issues

What is the abnormal behavior?
– The answer should not contain the symptoms
– State the facts about the issue without making any
judgment
What is the normal behavior?

– Define what the customer is trying to achieve
– Determine whether the goal is achievable
6OLGH'HILQHWKHLVVXHV
The first part of troubleshooting is to properly define the issues. Start by asking the customer to
describe the abnormal behavior observed. Document what is not working as designed or
expected. Make sure the answer does not contain any symptoms.
Important: Avoid arriving at any conclusions based on any preconceived judgments. State
the available facts without making any judgments.
Also, try to collect as much information from the customer about what was trying to be achieved
when the issue arose. Sometimes, it might be possible that the customer is trying to achieve
something that a product is not designed to do.
For example, the customer might experience issues where some HTTPS sites are denied access and
users receive a message that a page cannot be displayed. Ask the customer to describe the
expected normal behavior. Do not jump to conclusions at this stage, assuming that this could be a
possible reason for the issue. Collect as much information as possible from the customer.

Phase 1-2: Describe the Symptoms
6OLGH6\PSWRPVGHVFULSWLRQ
Next, explore in detail the symptoms observed during the occurrence of the issue. The broad
categories of symptoms include time of incident, diagnostic information, and identification of
operational or non-operational components. Describing each of these symptoms in detail will
make it easier to gather more information and document them appropriately, which is discussed
in the following steps.
• Time of incident: This could further be broken down into asking the customer when the
incident first occurred, whether the behavior is continuing to occur, or how long has the
problem has been occurring.
• Diagnostic information: It is wise to get as much statistical information from the customer that
is close to the time of occurrence of the incident. The support engineer can additionally ask for
memory dump and packet capture details if the information provided by the customer is not
adequate for diagnosis.
• Operational and non-operational parts: The support engineer needs to judiciously gather
information regarding what components or operational parts are still working after
occurrence of the incident. Any non-operational or dysfunctional parts can help identify
possible failures.

Phase 1-2: Gather Information

P roblem Diagnos tic Log File s C onfigura tio Tr fl Output Pack et Trace CLI
Files n Files Comm ands
C ompre ssion X( Le gacy) X X
C lassification X X X X X X
S ha ping X X X X
P erform ance X X X
B ooting X X X
N etwork X X X X
c onnec tivity
is sues
R eporting X X
is sues
6OLGH,QIRUPDWLRQJDWKHULQJ
At this point, the problem usually is somewhat vague and requires more definition. This is where
the fact-gathering step of the troubleshooting methodology is used. Fact gathering is the process
of using diagnostic tools to collect information specific to the network and network devices that
are involved in a problem. Additional information should include data that excludes other
possibilities and helps pinpoint the actual problem.
It is important to gain as much information as possible to actually define the problem while in the
problem-definition phase of the troubleshooting methodology. Without a proper and specific
definition of what the problem is, it will be much harder to isolate and resolve. Information that is
useful for defining a problem is listed in the table above.
The above table shows the most information you might need in several common troubleshooting
categories on the Blue Coat® PacketShaper®.
Ask the customer to provide the least amount of information necessary for you to diagnose and
solve the case, and avoid asking for information that is not relevant to the problem at hand.
Important: Diag files, config files and CLI commands are the most commonly used
troubleshooting tools. For this reason, you should be very familiar with all of
them. Most of these tools are covered in detail later in this course.

Phase 1-3: Document the Changes

Changes around the time of the incident
– New firewall policies
– New routes
– New anti-virus software running on desktop
Compare to working units

– Are there other devices with same configuration?
– Are the other devices operating as expected?
6OLGH'RFXPHQWFKDQJHV
After defining the issue, ask what changes happened before and after the occurrence of the
problem. Investigate possible interactions with external devices.
For example, any of these could cause a problem:
• Changes to firewall policies or configuration.
• Changes to router configuration.
• Installation of anti-virus software on user desktops.
For instance, consider a situation where company A acquires company B, not realizing that
company B had the same private subnet addressing space. This caused part of the company
network to stop working when incorporating the routes for company B as part of one big cloud for
WAN IP settings. In such a situation, the issue was not with the PacketShaper but with the
redundancy in the private subnet addressing scheme.
Suggest that the customer compare similar working units throughout the customer’s organization.
Many of the Blue Coat products deployed in a single organization might have the same
configuration; if so, check whether they experienced similar issues. Document these findings, and
compare them to other devices working properly in the system. Interacting with other devices
might make it appear that the PacketShaper is not performing correctly when the problem actually
is elsewhere.
By this point, you should have a good set of research, and you should be armed with intelligence
about the problem, the nature of the problem, and the diagnostic info that you need. Now you can
move on to the analysis phase.

Phase 2-1: Ishikawa diagram
6OLGH,VKLNDZDGLDJUDP
The second phase of the methodology involves analysis of the situation. Start by listing potential
and probable causes.
The Ishikawa diagram is designed to analyze the problem by using the available resources and
information to generate, singly or in combination, a feasible solution. By a process of elimination,
one should be able to arrive at finite causes and, therefore, a solution to the problem.
Note the following numbered points on the above graph:
1. Identify the event: In this example, we will use the diagram to analyze a situation where the
customer reports slow performances issues on the PacketShaper . Several factors might be the
cause.
2. List general causes: Internal issues on the PacketShaper, such as high CPU utilization and
memory pressure, might be potential causes of performance and access issues. Also,
unresponsive WUI and unreachable appliance might be potential causes. The horizontal lines
parallel to each general cause represent the potential root causes for the issue. High CPU
might be an effect, but it cannot be attributed as a real cause of the issue. One might even say
that the high CPU and memory utilization can cause slowness performance issues, but these
two factors are not the real causes. So, it is better to analyze what might be other causes for the
issue.
3. Suggest possible root causes: High CPU utilization could be caused by an undersize
PacketShaper, poorly performing policies, or high traffic through the appliance. Similarly,
high memory pressure could be caused by memory allocation failure or memory counters
being affected.
You do not need to understand the specific technical issues raised in this analysis. They are
discussed in depth later in this course.

Phase 2-2: Research Likely Causes
6OLGH5HVHDUFKOLNHO\FDXVHV
After interpreting the Ishikawa diagram, select the most likely cause of the issue. Focus only on
the selected cause, and eliminate the less likely causes of the issue.
Also, correlate the data to identify similar trends between various symptoms, and then highlight
the most likely cause.
This step of the troubleshooting model is used to investigate the possible causes of the failure. It is
quite easy to create a very long list of possible causes. That is why it is so important to gather as
much relevant information as you can and to create an accurate problem statement. By defining
the problem and assigning the corresponding boundaries, the resulting list of possible causes
diminishes because the entries in the list will be focused on the actual problem and not on possible
problems. These are only possible causes; the engineer will still have to create an action plan,
implement it, and observe to determine whether the changes made were effective. When the list of
possible problems is long, more iteration is required to actually solve the problem; the engineer
must check each of these possibilities and fix them if they are the cause of the problem.
At this stage of the troubleshooting methodology, start investigating the most likely causes of the
issue. Develop a hypothesis by attempting to replicate the issue in your own environment. Apply
the same policies and configurations that the customer did.
Attempt to verify your hypothesis about the cause of the problem; if necessary, ask the customer to
perform a specific, well-defined test to identify the cause, but make sure that the tests do not
interfere with the customer’s production environment.
If your tests confirm that the suspected cause of the problem is the actual cause, then move on to
the solution phase. Otherwise, consider the next most likely cause of the problem, and repeat this
phase.

Phase 3: Solution
6OLGH6ROXWLRQSKDVH
The final phase of the troubleshooting process is the solution — developing it, implementing it,
and documenting it. After defining the initial symptoms as clearly as possible and identifying the
root causes, follow these steps:
1. Develop the solution:
a. Propose a solution. Document the proposal without any bias. Do not operate on
incomplete facts to develop a solution that the customer cannot reasonably implement
in their production environment.
b. Implement the solution on an internal system, and verify that the proposed solution
appears to fix the issue entirely or reduce its severity.
Important: Always verify the possible solution on an internal system. Under no

circumstances should you ask the customer to test a sample solution in
their production environment.
c. Perform regression testing, a process to verify that the solution does not cause other
unexpected behaviors or failures. Although it often is not feasible to verify every
feature of every component during regression testing, special attention should be
given to verifying critical behaviors, behaviors related to the problem component, and
the functionality of other recent changes.
2. Implement the solution:
a. Provide a solution that the customer can implement in their environment. Depending
on the problem, this can take one of many forms, such as an update to operating
software, changes to one or more configuration files, or simply some instructions on
how to use the product.
b. Verify that the solution indeed fixes the issue in its entirety or helps reduce the
severity of the issue. Also, ensure that the solution in no way negatively impacts the
customer’s production environment.

3. Document the solution — a vital step that must not be omitted:

a. If the solution requires writing code, ensure that the code is updated for the correct
version of operating software and is properly entered in the call-tracking system and
technical support knowledgebase.
b. If a similar service request comes up later and another support engineer needs to
handle it, the second engineer can go back and query the support databases for past
reports. This reduces the time spent resolving similar service requests in the future.

Chapter 2: BlueTouch Online
BlueTouch™ Online is the online customer support service for Blue Coat Systems.
To get a BlueTouch Online login, go to https://support.bluecoat.com, click Need a login?, and then
follow the instructions given. You will receive a confirmation e-mail that allows you to begin using
BlueTouch Online immediately.
Logins are created only for individuals and not groups. An individual login, however, allows a
user to see all of their company’s cases. Creating logins for individuals versus groups allows Blue
Coat to identify who is creating or modifying records for a company, and control who in the
customer’s company has access to BlueTouch Online records. Blue Coat deactivates individual
logins when notified that users no longer work for a company or should no longer have access.
• The main functions available in BlueTouch Online.
• How service requests are prioritized and resolved.
• How to use BlueTouch Online for other service-related functions.

Overview
Secure Web access to Blue Coat resources
– Knowledge base
– Software downloads
– Technical briefs
– Product documentation
Technical Support access
– 24/7 access
– Create and review service requests
– RMA services
6OLGH2YHUYLHZ
BlueTouch Online is a service that provides secure access to Blue Coat information and resources
24 hours a day, seven days a week.
Benefits include:
• Secure online access to Blue Coat resources such as the knowledge base of product support
information, downloads of new software releases, technical briefs, and product
documentation.
• The ability to create, modify, and update technical support requests (called SRs) and manage
return material authorizations (RMAs).
BlueTouch Online is available to Blue Coat partners and customers with products actively covered
under the one-year warranty or a service contract.

Manage Service Requests
6OLGH0DQDJHVHUYLFHUHTXHVWV
Blue Coat manages all customer related technical questions for products under a valid support
contract through service requests (SRs). This is typically done by contacting a Blue Coat global
support center, or by opening a service request through BlueTouch Online.
As shown above:
1. To see a list of open service requests, select Service Management > Service Requests.
2. To open a new service request, click Open New SR.
3. To view details on a specific service request, click the individual SR number. You can then
provide more information about the issue or upload additional diagnostic information.
As SRs are opened, technical information about the product, environment, and customer site is
collected, and a service priority level is assigned for each case. A duty manager is on call as a
resource to assist customers who might feel that the priority of their issue has not been accurately
characterized, or that the response has not been within stated commitments.

Service Request Priorities

Prio rit y 1 Priority 2 Priority 3 Priority 4
(Critical) (High ) (Med ium) ( Low)
Telephone Telephone Online submission with Online submission with

technical support technical support BlueTouch Online BlueTouch Online
9Network or 9 Operational aspect of 9Performance of the 9 Operational issues
application outage; no networ k or application network or application with no impact to
workaround is severely degraded is impaired with limited business operations
9 Critical customer 9 Continuous or impact to business and no loss of
business oper ation is frequent instabilities operations functionality
fully impair ed by affecting customer 9 Functional, stress, or 9 General how-to
inadequate business or networ k perfor mance failure questions
performance oper ations with a workar ound 9 Documentation
9 Impaired 9 Inability to deploy a 9 Successful issues
functionality, critically featur e, function, or workaround in place for 9 Process issues
impacting customer’s capability a P2 issue
business oper ations 9 Successful
workar ound in place for
a P1 issue
6OLGH6HUYLFHUHTXHVWSULRULWLHV
The priority of a service request is determined by evaluating the problem type and technical
impact. The priority plays an important role by setting the initial response time, update frequency,
and guidance for the time to escalate issues to a higher level. By setting priority levels, Blue Coat is
able to balance its resources for all customers, and to allow timely resolution of technical issues.
• A priority 4, low-priority service request is appropriate in cases such as these: operational
issues for certain features and capabilities with no impact to business operations and no loss
of functionality; general how-to questions; or documentation and process issues.
• A priority 3, medium-priority service request is appropriate in cases such as these:
performance of the network or application is impaired with limited impact to business
operations; a functional, stress, or performance failure with a workaround; or a successful
workaround is in place for a priority 2 issue.
• A priority 2, high-priority service request is appropriate for issues such as these: an
operational aspect of the network or an application is severely degraded; continuous or
frequent instabilities affecting customer business or network operations; inability to deploy a
feature, function, or capability; or a successful workaround is in place for a priority 1 issue.
• A priority 1, critical-priority service request is appropriate for issues such as these: a network
or application outage where no workaround is available; critical customer business operation
is fully impaired by inadequate performance; or impaired functionality is critically impacting
the customer’s business operations.
Priority 3 is the highest level that can be submitted through BlueTouch Online. Priority 2 and
priority 1 requests should be opened directly through telephone contact.
While a service request is being handled, it is serviced by a team of Blue Coat support personnel
that can be expanded to include development engineers if it cannot be resolved by technical
support staff.

Service Request Status

Status Descript ion
CREATED Default state of an SR wh en it is first opened .

ASSIGNED SR ha s been a ssigned to a support en gineer.
OPEN SR is open, and the issue has not rea ched a re solution.
Any S Rs shown in thi s state are owned by you.
CL OSED SR ha s been closed.
RE SOLVED A solution to the problem has b een pro vided to the customer.
SR is still ope n.
INFO REQ SR is awaiting information from the customer.
ESCALATED SR ha s been e scalated to development support.
RMA Issue is a ha rdware RMA.
6OLGH6HUYLFHUHTXHVWVWDWXV
A service request always has one or more statuses associated with it. During the life cycle of a
service request, it can progress through some, many, or most of these statuses.
• Created: The SR has been created but not assigned to a support engineer.
• Assigned: The SR has been assigned to a support engineer.
• Open: The SR has not been resolved. Any open SR that you can view in BlueTouch Online is
one that you own.
• Closed: The SR has been closed.
• Resolved: A solution to the problem has been provided to the customer, but the SR still is open
pending confirmation that the solution is acceptable to the customer.
• Information requested: The customer has been asked to provide information to assist in
diagnosis.
• Escalated: The SR has been escalated to development support.
• RMA: The problem was caused by defective hardware that will be replaced through the RMA
process.

Other Services
6OLGH2WKHUVHUYLFHV
BlueTouch Online also offers access to other customer-support services. Some of these include, as
shown in the above examples:
1. Access to a comprehensive knowledge base that provides customers a one-stop source to
search the entirety of Blue Coat technical support information.
2. The ability to download updates to system software for products that the customer is licensed
to use.
3. Access to the Blue Coat Licensing Portal, which allows centralized management of all of a
customer’s Blue Coat licenses.
4. A complete set of Blue Coat product documentation.

Blue Coat Licensing Portal

Product Available funct ions
Conten t fil terin g Activate l icen se
ProxyS G ProxyClient / SSL activati on; acti vate upg rade; licensi ng
page; revert upgrade
ProxyAV Activate l icen se; antivirus seria l number ; downloa d
licen se; upgrad e cold standby; swap licen ses
ProxyRA Activate l icen se; download license; swap licenses
Blue Coat Reporter Activate l icen se
PacketShap er Download licen se; activate upg rade; revert upgra de
Intel ligenceCenter / Get license; upg rade; revert upgra de
Poli cyCen ter
NetCache Activate l icen se
Appl iance certificate Birth certificate valid ati on
verifica tion
© Blue Co at Syst ems, I nc. 20 09. All Rig hts Reser ved.
6OLGH%OXH&RDW/LFHQVLQJ3RUWDO
The Blue Coat Licensing Portal provides access to license-related functions for Blue Coat products.
To access the licensing portal from the BlueTouch Online homepage, select Licensing. Then, select
License a Proxy to perform licensing functions for a ProxySG®, or select License Others to perform
other licensing functions. When an organization purchases hardware or software licenses, e-mail
containing activation codes is sent to the e-mail address the organization specified at purchase
time. To activate licenses, you need to have the codes from that e-mail. Other license-related
functions at the Blue Coat Licensing Portal include:
• Content filtering: This feature of the ProxySG requires a separate license. To enable it, select
this option and enter the activation code.
• ProxySG: Four functions are available: SSL license activation, ProxySG upgrade, ProxySG
licensing, and the ability to revert to a previous upgrade.
• ProxyAV™: Five functions are available: license activation for systems at version 3.1 or later,
license activation for systems older than version 3.1, downloading anti-virus license for
systems at version 3.1 or later, upgrading a cold-standby appliance, and swapping a version
3.1 or later license from one appliance to another.
• ProxyRA™: Three functions are available: activate, download, and swap licenses.
• Blue Coat Reporter: To enable this application, select this option and type the activation code.
• PacketShaper®: Three functions are available: download a license, upgrade, and revert
upgrade.
• IntelligenceCenter® / PolicyCenter: Three functions are available: get a license, upgrade, and
revert upgrade.
• NetCache: To activate licenses for legacy NetCache equipment, select this and enter the code.
• Appliance certificate verification: Enter a hardware serial number to determine whether that
ProxySG supports Blue Coat appliance certificates.


Chapter 3: Hardware Overview
Understanding the hardware that is used in Blue Coat products is an essential part of supporting
these products. After studying this chapter, you will understand:
• Blue Coat® PacketShaper® hardware:
❐ Available series and models
❐ Components inside each series and model
• When hardware can be reconfigured:
❐ Available field upgrades
❐ Which parts are user-serviceable and which are not
• Configuration limits for each member of the PacketShaper product family:
❐ Supported bandwidth limits
❐ Concurrent TCP flows
This chapter will not teach you:
• How to install or upgrade Blue Coat hardware. Refer to the appropriate installation guide that
is shipped with the hardware.
• How to diagnose and respond to hardware failures. This topic is covered in the Hardware
Troubleshooting chapter.
• How to plan deployment of Blue Coat products and allocate PolicyCenter appliances on a
network. These topics are covered in sales and channel partner training.
Note: The hardware specifications in this chapter are current as of November 2009, but they
are subject to change. If you are supporting older deployments in the field, you might
encounter older versions of Blue Coat hardware.
Important: When handling any hardware, make sure that proper health and safety
procedures are followed, even if you are not the person actually working with
the hardware. Refer to the installation guide for each product for details on
procedures to follow to ensure safe handling. Failure to follow proper
procedures might damage equipment and pose a health hazard.

Overview
PacketShaper product family
Options and serviceability
Configuration limits
2 © Blue Co at Syst ems, I nc. 20 09. All Rig hts Reser ved.
6OLGH2YHUYLHZ
Blue Coat PacketShaper is part of the Application Delivery Network (ADN), an infrastructure that
provides complete application visibility, acceleration and security. PacketShaper delivers
integrated visibility, control and compression capabilities in a single appliance. With
PacketShaper, an IT organization can identify all the applications on the network and monitor
response times and utilization at the application level.
PacketShaper supports ADN with unmatched visibility to optimize WAN performance. With
PacketShaper, customers can automatically classify and measure network applications, provide
quality-of-service (QoS) provisioning to control traffic and increase WAN capacity with
compression capabilities.

Product Family
3 © Blue Coa t Systems, In c. 2009 . All Rights Reserve d.
6OLGH3URGXFWIDPLO\
There are five members of the PacketShaper product family:

• PacketShaper 900: Used for deploying on your network’s edge. Supports 256 traffic classes
and shaping policies and comes with a back up interface. This hardware model can handle
link speed up to 2Mbps.
• PacketShaper 1700: This model can support up to 512 traffic classes and shaping policies. Also
available are capabilities to handle link speeds from 2 to10Mbps.
• PacketShaper 3500: Deployed at the network edge for large meshed environments. This model
can support up to 1024 traffic classes and shaping policies. Link speeds from 2 to 45Mbps are
supported. Also available with an additional LAN Expansion Module (LEM) to extend
deployment options.
• PacketShaper 7500: This is the most effective model available for large edge sites (with WAN
link rates of 10 Mbps or higher), as well as small core deployments. Capabilities are the same
as for the PacketShaper 3500, except that the PacketShaper 7500 can handle link speeds up to
200Mbps.
• PacketShaper 10000: This model is best suited for large core deployments. The appliance can
support more than 2,048 traffic classes and shaping policies. The product can best manage
deployments requiring link speeds up to 1Gbps. Also available with additional LEM
hardware extensions.

PacketShaper 900
6OLGH3DFNHW6KDSHU
The PacketShaper 900 is the entry-level member of the PacketShaper product family, designed for
small offices and branches. The PacketShaper 900 is recommended for deployment at the network
edge. PacketShaper 900 supports PacketWise versions 8.2.2 or later. The above photo shows the
main user-serviceable hardware components of the PacketShaper 900:
1. One disk drive
2. CPU cooler
3. 512MB RAM
4. One compact flash (CF) card
5. Network interfaces, copper 10/100 Mbps
6. Chassis fan

PacketShaper 1700
6OLGH3DFNHW6KDSHU
The PacketShaper 1700 is a mid-level member of the PacketShaper product family. With increased
processor and memory scalability to accommodate future features and functional enhancements,
the PacketShaper 1700 is a full-featured branch office solution. The PacketShaper 1700 supports
PacketWise version 7.3 or higher. The above photo shows the main user-serviceable hardware
components of the PacketShaper 1700:
1. One 80GB hard disk drive
2. A 1.3G Celeron CPU core, covered by its heat sink in this photo
3. 512MB RAM
4. Two compact flash cards, 64MB each
5. Network interfaces, copper 10/100/1000Mbps
6. One power supply
7. Fan assembly

PacketShaper 3500 / 7500
6OLGH3DFNHW6KDSHUVHULHV
The PacketShaper 3500 is equally at home in the branch office or in the enterprise core. A key
difference between the PacketShaper 3500 and the Packetshaper 1700 is the capacity for two extra
LEMs. This lets the PacketShaper 3500 look at multiple LANs with the same device, making it
better suited for data centers.
The PacketShaper 7500 is designed for data centers that now need or will need to shape up to
200Mbps of traffic, will be using several smaller compression-enabled PacketShaper appliances at
branch locations or have enough hosts to require far more IP flows or classes than either the 3500
or 1700 can handle. Both PacketShaper3500 and PacketShaper7500 models support PacketWise
versions 7.3 and higher. The above photo shows the main user-serviceable hardware components
of the PacketShaper 3500 and 7500 line of products:
1. One 80GB hard disk drive
2. CPU
a. PacketShaper 3500: 1.3G Celeron
b. PacketShaper 7500: 1.8G Pentium M
3. RAM: 1GB on the PacketShaper 3500, 2GB on the PacketShaper 7500
4. Two compact flash drives, 64 MB each
5. Network interfaces, copper 10/100/1000Mbps
6. PCI LEM slots, copper 10/100/1000 Mbps, fiber small form-factor pluggable (SFP)
7. Fan assembly
8. Power supply units: one on the PacketShaper 3500, two on thePacketShaper 7500

PacketShaper 10000
6OLGH3DFNHW6KDSHU
Designed for deployment in locations where space is limited, the PacketShaper 10000 delivers the
tools needed to manage and optimize performance at the network edge. PacketShaper 10000 has
dual removable power supplies. These power supplies can be hot swapped; that is, replaced while
an appliance has power and is operating. Hardware revisions G and later of the PacketShaper
10000 are compliant with the Restriction of Hazardous Substances (RoHS) directive. They use a
new power supply that is not compatible with earlier revisions. The above photo shows the
user-serviceable hardware components of the PacketShaper 10000:
1. One hard disk drive: 40GB on the PacketShaper 10000, 80GB on the PacketShaper
10000(RoHS)
2. CPU: 3.2GHz Intel Xeon on the PacketShaper 10000, 4.6GHz Intel Xeon on the PacketShaper
10000 (RoHS)
3. RAM, 2048MB
4. One compact vertical flash card, 64 MB
5. Network interfaces, copper 10/100/1000 Mbps, fiber 1000 Mbps
6. LEMs, copper 10/100/1000 Mbps, fiber SFP
7. Fan assembly
8. Power supply, two units

Optional Upgrades – LEM (Copper)
6OLGH&RSSHU/$1([WHQVLRQ0RGXOH
PacketShaper LEMs provide additional network interfaces to the PacketShaper, creating flexible
deployment options for complex topologies. The 3500, 7500 and 10000 series of appliances have
slots for up to two LEMs (each LEM has two ports added to the standard interfaces), providing
PacketShaper appliances with up to three pairs of network interfaces. LEMs are critical for
topologies such as:
• Multiple LAN segments feeding a single WAN or Internet link
• A DMZ where the firewall/router splits the network into two separate LANs (protected and
DMZ)
• Routers configured for redundancy with multiple physical paths
• PacketShaper Direct Standby for state redundancy on monitoring and shaping multiple WAN
links
• PacketShaper watch mode, non-inline installations with taps, span and mirror ports for
monitoring and compression estimation
• Networks desiring out-of-band management for additional physical and/or logical
management paths.
The above photo displays the features of the copper LEM:
1. Inside port of the LEM
2. Outside port of the LEM
3. Jumpers that need to be removed when using the standby option on the PacketShaper .

Optional Upgrades – LEM (Fiber)
6OLGH)LEHU/(0
Blue Coat’s fiber optic gigabit Ethernet LEM, offers advanced features not available in earlier LEM
models. This LEM utilizes LC-type connectors and smaller, modular SFP transceivers that can be
easily exchanged, if desired. The LEM also has an RJ-11 port that can control an optional external
optical bypass switch, enabling failover bypass in the event the PacketShaper is reset or powered
off.
The above photo of the fiber optic LEM displays the following:
1. One of the exchangeable SFP transceivers
2. SFP transceiver slot
3. RJ-11 port that can control an optional bypass switch.

Optional Upgrades – Fiber Bypass Switch
6OLGH)LEHUE\SDVVVZLWFK
The fiber bypass switch is an external BlueCoat device that connects a fiber optic network’s inside
and outside interfaces. When the bypass switch is connected to a PacketShaper appliance, the
switch prevents the appliance from interrupting network access when the appliance is not
powered on or is in bypass mode. If the PacketShaper appliance is operating normally, it supplies
power to the fiber bypass switch through a control cable. When the bypass switch is powered on,
it pivots internal mirrors to reflect optical signals to the appropriate fiber optic connector,
connecting the Inside and Outside ports on the bypass switch to the Inside and Outside ports on
the PacketShaper appliance.
If the PacketShaper appliance is not powered on or is in bypass mode, the fiber bypass switch’s
relay mirrors revert to their default bypass (power off) state. When the switch is powered off, the
mirrors connect the bypass switch’s Outside port directly to its Inside port, allowing the fiber optic
signal to completely bypass the PacketShaper appliance.
Though Blue Coat has released more than one model of Fiber Optic LEM, the Fiber Bypass Switch
is only compatible with newer LEMs which use SFP transceivers and have control ports. With a
compatible LEM installed in an expansion slot, the fiber bypass switch can be used with the
PacketShaper 3500 and 7500 series. The above photo shows how a fiber bypass switch is connected
to a PacketShaper and the network:
1. Inside port of the fiber bypass switch connects to the network switch.
2. Outside port of the fiber bypass switch connects to the network router.
3. Connect the PacketShaper appliance’s Inside port to port on the fiber bypass switch.
4. Connect the PacketShaper appliance’s Outside port to port on the fiber bypass switch.
5. Using a control cable, connect the control cable’s larger connector (RJ-45) to the control port on
the fiber bypass switch and the smaller connector (RJ-11) to the control port on the
PacketShaper appliance.

Hardware Serviceability
Customer-serviceable
– Hard disk drive
– LEM
– Power supply
– Cooling units
– Main board battery
Not customer-serviceable
– RAM
– CPU
– Compact flash card
6OLGH+DUGZDUHVHUYLFHDELOLW\
The PacketShaper product family is designed with a mean time between failure (MTBF) ranging
from 50,000 to 230,000 hours, depending on the hardware model. But hardware failures can
happen. Therefore, many of the components inside the PacketShaper are customer-serviceable if
they fail in the field:
• Disk drives: All models of the PacketShaper have field-replaceable hard drives that can be
ordered from Blue Coat. Hard drives cannot be hot-swapped. You must turn off and unplug
the appliance before removing the defective hard drive. After the hard drive is replaced, the
measurement engine is reset and all measurement data on the appliance will be lost.
• LEM: PacketShaper 3500, 7500 and 10000 models support up to two field-replaceable LEMs
that can be ordered from Blue Coat. You can choose from gigabit copper and fiber-optic LEM
options.
• Power supply: PacketShaper 1700, 3500, 7500 and 10000 models have removable power
supplies. PacketShaper 1700 and 3500 models have one power supply module while
PacketShaper 7500 and 10000 models have two hot-swappable power supplies.
RoHS-compliant versions of the PacketShaper 10000 (Revision G and later) use a new power
supply that is not compatible with older PacketShaper 10000 models. The two power supplies
on the RoHS-compliant PacketShaper 10000 models are stacked vertically; the power supplies
on older PacketShaper 10000 models are installed side-by-side.
• Cooling units: PacketShaper 3500 and 7500 models have field-replaceable, hot-swappable
cooling units. If you are hot-swapping a cooling unit as part of scheduled maintenance, the
new cooling unit should be installed within a minute of removing the old one, to prevent
overheating of system components. PacketShaper 10000 does not have a hot-swappable
cooling unit. You must turn off and unplug the appliance before removing the cooling unit.
• Main board battery: PacketShaper 1700, 3500 and 7500 models have a system battery that can
be replaced in the field.
Some of the hardware components in the PacketShaper cannot be replaced in the field:

• RAM: Although customers can install additional memory as part of an upgrade kit, defective
memory cannot be replaced in the field. The entire unit must be returned. RAM from sources
other than Blue Coat is not supported.
• CPU: The CPU can be upgraded in the field as part of an upgrade kit, but a defective CPU or
motherboard must be returned.
• Compact flash card: PacketShaper compact flash card cannot be upgraded to a larger capacity.
Defective flash cards can be returned to Blue Coat for replacement.

Configuration Limits
6OLGH&RQILJXUDWLRQOLPLWV
Typically, two occasions prompt an evaluation of the most suitable PacketShaper model for a
particular network environment. Most commonly, the need surfaces when trying to decide which
model to order for an initial purchase. Occasionally, the same task becomes relevant again after
network growth or topology changes impact the original assumptions used to select an appliance.
The PacketShaper product family is designed to meet the performance needs of organizations of
all sizes. But it is important that each deployment of PacketShaper appliances be properly sized to
match customer requirements. If a deployment is improperly sized, performance might be
degraded, and customers might suspect that their hardware is failing when instead it is being
asked to do more than it is capable of doing.
When sizing a new PacketShaper deployment, consider the intended deployment mode. It is
important to know where you will install the PacketShaper appliances in your network. The
locations depend on a number of factors, such as the type of network topology and what traffic
you want the appliance to monitor and manage.
When deciding which series and model of PacketShaper to use, performance metrics depend on
the type of deployment that will be used. Always adhere to the Blue Coat recommended
configuration limits for various hardware models.
The above diagram represents only a part of the sizing recommendations for various
PacketShaper hardware models. The chart in the lower segment shows the maximum number of
concurrent TCP flows for each model of the PacketShaper. The upper chart shows the maximum
bandwidth that is supported by the same models.


Chapter 4: PacketShaper Boot Process
The term boot means to start the operating system. It derives from the phrase pulling yourself up by
your bootstraps and aptly describes the procedure by which a small program, permanently stored
on the hard disk, executes automatically to load the operating system into main memory
(sometimes called system RAM) and begin its initialization. The first time your system boots is
during installation. The operating system can be booted again (often referred to as rebooting).
Understanding the boot process in the Blue Coat® PacketShaper® line of products can prove
beneficial while troubleshooting booting-related issues. This chapter examines the stages of the
PacketShaper hardware boot sequence, including how the PacketShaper locates and loads the
PacketWise operating system. A PacketShaper appliance cannot operate without the PacketWise
software. Each PacketShaper has a predetermined boot sequence for locating and loading
PacketWise. This chapter describes the stages and importance of this boot sequence.
• The basic boot sequence in the PacketShaper.
• What happens during the Power-On-Self-Test process.
• How to identify potential causes of failure in the PacketShaper during the boot process.
• How to interrupt the boot process and the different stages involved there in.

Overview
Boot sequence
Power-On-Self-Test (POST)
Boot monitor
Interrupt boot process
6OLGH,QWURGXFWLRQ
Each time a hardware appliance boots up, it goes through an initial series of processes. This
sequence of events is named a boot sequence. During the boot sequence, the appliance activates the
necessary hardware components and loads the appropriate software so that a user can interact
with the appliance.The boot sequence can take anywhere from a few seconds to several minutes,
depending on the appliance’s configuration. If your appliance was turned off unexpectedly, the
boot time might increase since the system may perform some additional checks to make sure
everything is OK.
This chapter describes the boot sequence in the Blue Coat PacketShaper. As the boot sequence
takes effect, the PacketShaper undergoes a sub process called Power-On-Self-Test (POST), which is
a pre-boot sequence procedure. POST involves executing a defined series of tests to determine if
the hardware is working properly. Any errors found during the POST process are stored or
reported through the appliance’s LCD. Once the POST sequence completes, execution is handed
over to the normal boot sequence which typically runs a boot loader or the PacketWise software.
Boot monitor mode, an important aspect of the boot sequence, is also discussed later in the
chapter. Details on what happens when a boot sequence is interrupted is covered toward the end
of the chapter.

Boot Sequence
6OLGH%RRWVHTXHQFH
The PacketShaper BIOS is what starts the appliance running when you turn it on. The above slide
describes the high level view of the PacketShaper boot sequence. A typical boot sequence
involves:
1. Once the PacketShaper appliance is powered-on, the BIOS loads into RAM.
2. The first thing that the BIOS does is to perform the Power-On Self-Test (POST). The POST is a
built-in diagnostic program that checks your hardware to ensure that everything is present
and functioning properly, before the BIOS begins the actual boot. As part of the POST
procedure, the input output device status is verified.
3. Next, the BIOS begins to search for the boot drive within the appliance. The existence of the
boot drive indicates the existence of all input/output devices in the appliance. In the
PacketShaper BIOS, only the B: drive is used for booting.
4. The boot monitor loads. The boot monitor is a set of application files that resides in the boot
drive B: of the PacketShaper. The boot drive is one of the supported ROM chips that is on the
motherboard of the appliance. During this process, the PacketShaper LCD or the serial console
displays the message Booting....., to indicate the booting process.
5. The operating system and Packet Wise software load. Once the PacketWise image file is
executed, the PacketWise OS is loaded, and this brings up the PacketShaper Web User
Interface (WUI) on your monitor. The corresponding LCD or serial console display shows
Loading...... and signifies the successful loading of the PacketWise image.
Note: To interrupt the booting sequence at any point of time, press Control+Y, Control+A
or Control+B. These commands and their resultant activities are described later in
the chapter.

POST
6OLGH3RZHU2Q6HOI7HVW
The diagnostic routine built into the BIOS is called POST. It ensures that the hardware installed in
a system is present and functional as expected. The POST operation verifies that the system is
ready to initialize the boot sequence.
POST is the first thing that the PacketShaper BIOS does when it boots the appliance. The POST
runs very quickly, and you will normally not even notice that it is happening—unless it finds a
problem. POST is a purely hardware-focussed diagnostic check; if there are unrecoverable
problems during the process, it ends with the PacketShaper shutting itself down. This can happen
due to one of these reasons:
• The CPU temperature is very high. This can be fixed by checking the air flow vent and
clearing any blocks that might exists.
• The CPU is not functioning as expected. This can be rectified by:
❐ Reseating the CPU in its socket. Follow all safety precautions mentioned in the product
documentation to avoid any injury.
❐ Checking the CPU chassis cooling fan. When the PacketShaper is powered on, make sure
you hear the sound of the CPU fan, which can indicate if the component is working
properly.
• A hardware problem is encountered. Under such circumstances, the PacketShaper displays
specific error codes, that can be used to diagnose the exact problem. One commonly
encountered error is Error 03, which can be traced to problems with the RAM and indicates
either the memory is loose or it is bad. Most of the time, the error message goes away when
you take the memory out and reseat it properly. If this does not solve the problem, call Blue
Coat Customer Support to request an RMA for the appliance.
Sometimes, the PacketShaper LCD does not show any error codes and displays a blank screen.
In such cases, shut down the appliance and check the CPU and or RAM. Remove and reseat
the CPU and or RAM and power cycle the appliance again. If the problem persists, escalate
the problem to Blue Coat Customer Support and RMA the appliance.

Boot Monitor
6OLGH%RRWPRQLWRU
A boot monitor is a small low-level program typically hard-wired into a system.The boot monitor
is an executable file and resides at the start of ROM.It permits very basic memory operation
functions, including transfers, to be completed. A boot monitor permits software to be
downloaded to the memory. It also provides a simple terminal interface which allows basic
memory operations.
Boot monitor differs with the PacketWise OS used in your appliance. Pressing Ctrl+Y on your
keyboard via the serial console reveals the version of the boot monitor used in that appliance.
After the completion of the POST process, the boot monitor searches for the PacketWise image
files in the 9.256/BIN directory. Normally, the image.zoo file is searched for and if the boot monitor
is unsuccessful in locating that, it attempts to locate the backup.zoo file. Once a valid PacketWise
image is identified, the boot monitor loads that image. Otherwise, it enters the boot monitor mode.
Additionally, pressing Ctrl +Y at any stage during the process instructs the PacketShaper to
enter the boot monitor mode directly.

Interrupting Boot Process

Boot monitor mode
– Ctrl + Y
– During appliance booting or loading
Safe mode
– Ctrl + A
– During appliance loading only
Revert to backup image

– Ctrl + B
– During appliance loading only
6OLGH,QWHUUXSWLQJWKHERRWSURFHVV
The PacketWise software is booted automatically when you turn on your appliance or reboot your
system. However, you can also interrupt the boot sequence and initiate an interactive session. This
procedure is useful when you want to redefine system parameters, such as the amount of
available physical memory. There are three options to interrupt the PacketShaper boot process:
• To enter boot monitor mode press Ctrl + Y during the booting or loading sequence.
Slide 4-6 describes the boot monitor mode with the help of serial console display images.
• To enter safe mode press Ctrl +A during the appliance loading sequence. The serial console
when accessed at this stage displays Loading....and implies that the PacketShaper is
attempting to the load the PacketWise image files at that point of time. Once this command is
executed, the PacketShaper enters the safe mode. Slide 4-7 describes the safe mode in detail.
• To revert to a backup PacketWise image use Ctrl+B. This command is applicable only during
the loading sequence. Use of this command is recommended during instances when the
PacketShaper continuously reboots itself or when an administrator would like to manually
revert back to a previously loaded image of PacketWise.
In the case of the PacketShaper initiated reboots, pressing Ctrl + B automatically reverts to
the backup version of the PacketWise image.

Boot Monitor Mode
6OLGH,QWHUUXSWLQJERRWSURFHVV³ERRWPRQLWRUPRGH
The above slide describes the boot monitor mode while accessing the PacketShaper using the
serial console. As shown above, you can enter boot monitor mode:
1. During booting.
2. During loading.
The following description is common to both the booting and loading sequences. In either case,
the touch password is required for access to the boot monitor. In the above examples, the
PacketShaper is running version 3.0 of the M30 boot monitor.
Note: Different hardware models might host different versions of the boot monitor.
The Boot Monitor Menu has the following options:

❐ A. Display System Configuration: View the IP address assigned to your
appliance. You can also assign an IP address if it has not been previously assigned to your
PacketShaper using this option.
❐ B. Browse Directory Contents: Browse the contents of the flash disk and the hard
drives on the PacketShaper.If the PacketShaper enters boot monitor mode, use this
option to check for files with.bad extension in the 9.256/BIN directory.
❐ C. Browse PacketWise Images: Look at different PacketWise image versions
residing on the PacketShaper.
❐ D. Execute PacketWise Image and E. Download File via LAN (TFTP GET):
Manage PacketWise image files.

Note: In appliances where option D is not available, the administrator needs to download
the neccesary PacketWise image file from http://bto.bluecoat.com and rename it as
image.zoo and save it in the 9.256/BIN directory using the TFTP GET command.
This procedure permits the PacketShaper to locate the image.zoo file during a
reboot.
❐ F. Upload a File via LAN (TFTP PUT): Back up the configuration files on the
PacketShaper. In cases where the image.zoo files are corrupted and the config files are
not impacted, using this option helps to back up all neccesary config files on your
PacketShaper.
❐ G. Upgrade Boot Firmware: Upgrade the existing firmware, such as the boot
monitor, on the PacketShaper. A good example of firmware could be the boot monitor for
your hardware platform.
Important: It is important to note that Blue Coat no longer distributes firmware to end
customers.
❐ H. Recover Flash Disk(9.256):Recover your PacketShaper ‘s flash disk. The

process involves deleting all files stored in the flash disk and restoring only the factory
default files on the appliance after formatting the disk. All PacketWise image and backup
files are lost during the recovery process.
❐ I. Perform Hardware Reset: Restart the appliance. This is identical to performing
the reset command in the CLI.

Safe Mode
6OLGH,QWHUUXSWLQJERRWSURFHVV³VDIHPRGH
The PacketShaper appliance has a built-in safeguard that enables recovery from a corrupted
software image. When the appliance detects a bad image (after repeated crashes), it reboots into
safe mode. PacketShaper can also enter into safe mode, if the configuration files are corrupted or
missing. The basic.cfg files are appliance specific and cannot be ported between two
appliances and if done, can corrupt the configuration file.
The above slide describes what occurs when the PacketShaper goes into the safe mode:
1. The appliance cannot be accessed via the standard WUI and the address bar of the WUI reads
as http://psIPaddr/corrupt.htm. Safe mode turns shaping off and prohibits any configuration
access, such as the traffic, measure, setup shaping, and class commands. Safe mode
is reported in the login banner of the WUI. Access to the appliance is allowed only via Telnet,
FTP or serial console; all secure access to the appliance is blocked. This can be confirmed by
accessing the CLI and typing the banner show command. The output reveals that the SSH
daemon is stopped.
Note: If you are connected to the PacketShaper through the serial console, you can
repeatedly press Ctrl-A during the boot process to enter safe mode manually. This
avoids the delay involved with the automatic safe mode process.
To recover from this error condition, you can revert to the last image using the image
revert or FTP a new software image file to the appliance and load the new image using the
image load command.
2. The screen capture displays what was described earlier as seen from the CLI. Observe that the
appliance entered safe mode during the loading sequence and HTTPS service failed to start on
port 443.

Revert to Backup Image
6OLGH5HYHUWWREDFNXSLPDJH
If you have attempted to load a version of PacketWise that is not supported by your hardware
platform, such as version 7.3 or 8.0 on a PacketShaper 1400, your PacketShaper will not boot.
Instead, the message Loading... will remain on the LCD panel.
To recover the appliance, revert to the backup image of PacketWise, which is the image previously
installed on the appliance before you loaded the unsupported image. The recovery procedure
must be performed from a serial console:
1. Power cycle the appliance.
2. The message Loading... appears on the LCD panel, as the appliance tries to boot.
3. Press Ctrl+B.
4. The PacketShaper reboots using its backup image.

Chapter 5: Understanding PacketShaper Commands
The command line interface (CLI) is another user interface used for configuring, monitoring, and
maintaining PacketShaper appliances. This user interface provides direct and simple execution of
the PacketWise commands. All functions available via the browser interface are also accessible
with CLI commands. In addition, a number of CLI commands support special features and
diagnostic tasks that are not incorporated in the browser interface.This chapter describes some of
the important CLI commands used in troubleshooting PacketShaper appliances. For general
PacketShaper CLI commands, refer to the PacketShaper CLI Commands in Print documentation for
your version of the PacketWise.
Important: Many of the troubleshooting techniques described in this course depend upon
the availability of undocumented and unsupported features and commands.
Although Blue Coat provides this information in troubleshooting courses as an
aid to students attending these courses, Blue Coat does not commit to continuing
to make these features and commands available, in similar or different formats,
in future versions. Moreover, Blue Coat does not commit to resolving service
reports related to the use of undocumented features and commands.
After completing this chapter, you will understand:

• Basic features of the CLI commands
• Command categories supported in the PacketWise OS
• Sub-groups under the general commands category
• Different shaper commands that can be used to analyze traffic-related issues
• Some of the important debugging commands used in troubleshooting the PacketShaper.

Overview
Features
Accessibility
Command categories
Debugging commands
6OLGH,QWURGXFWLRQ
A command-line interface (CLI) is a mechanism for interacting with a computer operating system
or software by typing commands to perform specific tasks. The PacketShaper CLI can be accessed
in many different ways: serial console, remote access, Web User Interface (WUI) command
interpreter, and the Quick Commands utility.
From a troubleshooting point of view, PacketShaper CLI commands are divided into general,
system level diagnostic and debugging commands. These commands are organized hierarchically
and perform specific tasks on the PacketShaper.
PacketWise’s default settings are appropriate for most configurations. However, you can adjust
the system variables if your situation warrants it. Always use discretion when modifying these
variables. Some of the CLI commands that help you modify these system variable settings are
discussed in this chapter.

Features
Unix-like interface
Supports
– Abbreviations
– Command syntax verification
– Execution of multiple commands
– Recall of historical commands
6OLGH&/,IHDWXUHV
The PacketShaper CLI can be run in a text terminal or in a terminal emulator such as PuTTY. Upon
completion, the command usually returns output to the user in the form of text lines on the CLI.
This output may be an answer if the command was a question, or otherwise a summary of the
operation.
Typing a complete command name is not always required for the command to execute. The
PacketShaper CLI recognizes an abbreviated command when the abbreviation contains enough
characters to uniquely identify the command. For example, the version verbose command can
be abbreviated as ver ver. However, for example, pa is not an acceptable abbreviation because
it could stand for either partition or packetcapture.
PacketShaper CLI also supports the execution of multiple commands separated by a semicolon(;).
For example, you can type the following at the command prompt: set shaping; link show;
ver ver. The CLI displays the outputs from all the above mentioned commands simultaneously.
The command history feature saves, the commands that you enter during a session. The default
number of saved commands is 10, but the number is configurable within the range of 0 to 256. This
command history feature is particularly useful for recalling long or complex commands.
To recall commands from the history buffer, use the following methods:
• Press the Up Arrow key— Recalls commands beginning with the most recent command.
Repeat the key sequence to recall successively older commands.
Note: The use of the up arrow key does not work when access to the PacketShaper is via the
serial console. Typing !! at the command prompt repeats any previous commands.
• Press the Down Arrow key— Recalls the most recent commands in the history buffer after
they have been recalled using the Up Arrow key. Repeat the key sequence to recall
successively more recent commands.

• history command — Displays the last 20 commands that were entered into the CLI; each
command is prefixed by a number. Any command on the history list can be executed by using
the !<n> command, where <n> is the number next to the command on the history list.

Accessibility
6OLGH'LIIHUHQWDFFHVVPHWKRGV
You can access the PacketShaper CLI by any of these methods:

1. Serial console: If remote-access methods do not work due to network or configuration issues,
you can access the appliance directly with a null-modem cable. Another situation in which
you would use a console connection is when you forget the touch password; you can access
the appliance to reset the touch password.
Connect your workstation to the PacketShaper ‘s console port using a null-modem cable. The
specifications for the serial port are 9,600 bits per second, 8 bits of data, 1 stop bit, no parity,
and hardware flow control. To activate the serial console after physically connecting to the
serial port, press the Enter key several times. When you successfully connect, you will see the
appliance’s command-line prompt, PacketShaper#.
2. Remote login: The CLI can be accessed via remote login using any utility available for your
operating system. Log into the PacketShaper using its IP address and provide the necessary
details when prompted. When you successfully connect, you will see the appliance’s
command-line prompt, PacketShaper#.
3. Web browser: If you are using the PacketWise browser interface and need to enter a few
commands from the command line, you can launch the Command Interpreter in your browser
window. Enter http://psIPaddr/cli.htm, where psIPaddr is the IP address of your PacketShaper.
Note: The above option is not tested and all commands via the Command Interpreter might
not work successfully.
4. Quick commands: The Multi-Class Quick CLI Commands utility allows you to execute CLI
commands for multiple classes in one operation from within the WUI. To use this feature, go
to the Info tab and click quick commands. In addition, the utility provides a command line into
which you can enter CLI commands without having to access the serial port of the
PacketShaper. The following command types can be executed from the Multi-Class Quick CLI
utility: policy, class, compression, misc and partition.

Command Categories
6OLGH&RPPDQGFDWHJRULHV
PacketShaper commands can be divided into four main categories: general, system-level
diagnostic, debugging and boot interrupt commands. The CLI commands are organized in a
hierarchical fashion, with commands that perform a similar function grouped together under the
same level. For example, all general commands that display information about the setup, utility
and miscellaneous information are grouped under the general command and all commands that
display information about the system-level diagnostics are under the same category. The above
slide illustrates a portion of the PacketShaper command hierarchy.
The general commands include all the default commands that are supported by the CLI. This
includes the following groups of commands to carry out specific tasks:
setup,utility,miscellaneous,shaper and diagnostic.
System-level diagnostic commands help you to collect the log and diagnostic files on your
PacketShaper. You can also use this group of commands to view the system event monitoring
process. Use the system-level commands to check if there is a memory outage on the
PacketShaper.
Debugging commands are hidden by default and need to be enabled before execution. These
commands can be enabled by turning on the showdebugCommands sys set command.
Boot interrupt commands are specific to the boot process in the PacketShaper and are discussed in
the Understanding PacketShaper Boot Process chapter.

General Commands
6OLGH*HQHUDO&/,FRPPDQGV
Some of the general commands that are supported by the PacketShaper can be accessed by typing
help at the command prompt. This displays a list of available group of commands on your
appliance.
• diagnostic commands: This command group is used primarily in troubleshooting different
problems in the PacketShaper. For example, the arp command is used to display and update
the Address Resolution Protocol (ARP) table. The arp command is used while
troubleshooting a network problem and to manipulate the table contents. You will need touch
access to your appliance to execute the arp command.
The dns set of commands helps you manage DNS names in the traffic class matching rules
and converts them to IP addresses. The subset of commands include: lookup, rlookup,
names, refresh,servers and trace.
Use the ping command to test connectivity with another device on the network. If the device
answers the pings from the PacketShaper, one of these messages displays:
x.x.x.x is alive or x packets transmitted
x packets received
If PacketWise cannot connect with the device, one of these messages displays:
no answer from x.x.x.x
0 packets received.
System-level diagnostic commands can be accessed by typing in sys at the command prompt.
These commands are to be used exclusively for diagnostic and troubleshooting purposes by
authorized Technical Support personnel.
If you would like to display the time since last PacketShaper reset, use the uptime command.
• miscellaneous commands: Some of the well known commands in this group comprise of:
banner,exit,reset,radius,unit,zip,watch and version.

• setup commands: Use the following commands to configure and view the basic settings on
your PacketShaper: access, capture, compression, ipaddress, keys, link,
modem, nic, portal, reset and variable.
• shaper commands: Use this group of commands to configure and view all traffic shaping
features in your PacketShaper. Some of the prominent commands in this group are: agent,
event,frame,hostdb,policy,rtm,ipfilter and traffic and are discussed in
Slide 5-6.
• utility commands: Use these commands to for carrying out simple utility based tasks on
your PacketShaper. Some of the important commands in this group are: cd,cat,ftpget,
ftpput,mkdir,tail and rmdir.

Shaper Commands (1)
6OLGH6KDSHUFRPPDQGVIRUWURXEOHVKRRWLQJSDUW
The above slide describes the shaper commands in detail. As mentioned in the previous slide,
this group of commands are used to configure and view all traffic shaping features in your
PacketShaper. Some of the sub categories under the shaper commands are:
1. Traffic classification in the PacketShaper is hierarchical and is structured like a tree. The
class command lets you manipulate the traffic class tree, test out the tree to see how it
classifies traffic and store the traffic shaping related configurations to non-volatile storage. For
example, class rule lets you add or delete a matching rule. class criteria lets you
show, track or view application-specific criteria. class licenses enforces a license count
on the maximum number of active flows permitted on the appliance. class services
displays the available services on the appliance. Use class user-services to create,
delete or display user defined services.
class group commands are new CLI comamnds introduced in PacketWise 8.5 and are used
to define custom service groups and their properties. You can delete, move, create new,
override, reinherit or reset service groups in the specific class.
2. The class show command displays the different traffic classes available on the appliance
and different flags and partitions it is associated with. Class flags can take different values
depending on whether they are auto created, discovered, exception, inherited, policy, cache-
able or a dedicated link.
The class load command loads a new traffic configuration file. This command loads the
traffic tree and everything related to the classes in the tree, such as policies and partitions. This
feature can be used to share configurations with other appliances. You can FTP a saved
configuration to the flash disk root (9.256/) of another PacketShaper and then activate it with
the class load command.
Note: Issuing the class load command will revert an appliance in shared mode back to
local mode.

You can load traffic.cfg, an.ldi file (such as config.ldi), or both. If both files are specified, one
file must be a traffic.cfg and the other must be a config.ldi.
class reset reverts the traffic class tree to its factory default configuration. All other
classes, policies, and partitions are removed.
3. The partition apply command can be used to create a static partition for a traffic class.
Always ensure that traffic shaping is enabled for partitions to take effect. Use the partition
remove command to remove a static partition from a traffic class. The bandwidth allocated to
this traffic class is returned to the parent class.
To create a dynamic per-user partition for a traffic class, use the partition dynamic
apply command. You must create a static partition for the class using the partition apply
command mentioned above.
To display current static partition usage on the PacketShaper, use the partition show. This
output lists both minimum and maximum partition size settings. It also lists the rate of
priority traffic. In addition, it prints an asterisk (*) next to any minimum or maximum value
that is not pure, that is, if the programmed value was adjusted due to oversubscription or the
use of the strings fixed or none. The adjusted values, not the programmed values, are listed,
followed by an asterisk.
4. policy commands are used to apply, remove or test policies on the PacketShaper. In order to
limit the rate of new flows to or from a unique host, use the policy flowlimit
command.This command can be used to detect and control a SYN flood or similar
denial-of-service attack directed at a particular host or if the attack is from a specific IP
address. Flows exceeding the rate are blocked from passing through the appliance.
To test a policy to determine what rate will be allocated, use the policy test command.
5. The traffic commands can be used to mitigate issues related to traffic flow. traffic flow is
used to display summary information about some or all currently active TCP and /or UDP
connections.
To display recent traffic flows for a specific hots or traffic class, use the traffic history
command. The output includes the date, time, IP address, port number, and URL of each flow
in the specified class.
To show current license usage for classes that have had the number of TCP flows limited, use
the traffic licenses command.
6. To display the current, maximum, and possible number of sessions for TCP, UDP, and Legacy
traffic types, use the traffic active command. This command is a valuable tool for
determining how close the unit is to reaching its capacity. It also gives a histogram of the
number of host entries in various time buckets (based on idle time). Use the traffic
bandwidth command to display bandwidth utilization for a partition.
To display guaranteed rate utilization for a specific traffic class sub tree, use the traffic
guaranteed command.
7. The traffic tree command provides detailed information about how often classes and
their associated policies are accessed by the PacketWise classification process, along with rate
information for each class.
To re-examine existing flows to see if they can successfully be classified based on PacketWise’s
knowledge of new flows that have started since the appliance booted, use the traffic
reclassify command. The PacketShaper automatically performs this function once every
15 minutes, so you normally should not need to use this command.

Shaper Commands (2)
6OLGH6KDSHUFRPPDQGVFRQWLQXHG
Some more shaper commands that are used during troubleshooting are:
1. The host database is a record of all hosts that have active connections through the unit. Once a
host closes its connection, the host will be purged from the database. In addition, the
PacketShaper will clear host entries if they are not active for approximately 10 minutes. Thus,
the hostdb is a real-time list of hosts.
To display the host IP address, estimated access speed, number of speed changes, the number
of TCP and UDP flows that a specified host has processed, the amount of time the host has
been idle, the status of the match rule cache, and compression status, use the hostdb show
command.
To display the host IP address, average and current connections, current guaranteed and
excess bandwidth, and throughput information, use the host info command.
To display another set of options to manage the host database, use the hostdb side
command. Available sub commands are: auto,default,manual,reset and rm.
To determine which users are consuming the most bandwidth, use the hostdb topusers
command. This command is applicable only for TCP/IP traffic because PacketWise does not
track sessions or hosts for non-IP traffic.
2. To display the programmed link speeds with current link statistics for your PacketShaper, use
the link show command.The output includes details on the current rate, one-minute
average and peak rate values for all the network interfaces.
3. To check the measurement engine status, to see whether the measurement engine needs to be
reset, or to display the details for a specific measurement type, use the measure show
command.
To stop or pause the measurement engine, use the measure stop command. The
measurement engine is automatically stopped before backing up or restoring measurement
data and automatically restarted after the operation is complete.

To restart the measurement engine after it has been stopped with the measure stop
command, use the measure start command. To restore measurement data files to the
measure directory on the unit’s hard drive (9.258/ measure), use the measure restore
command to restore data in case data became corrupted after the last backup or to copy
measurement data to another appliance.
4. To create an IP filter that configures a PacketShaper to filter traffic based on IP address, use the
ipfilter set of commands. You can create up to 2,000 IP filters on a PacketShaper; filter
entries are saved in the PacketShaper configuration file.
A subset of commands include clear, which can be used to remove all IP filters created so far
and show, which displays all configured IP filters.

System Level Diagnostic Commands
6OLGH6\VWHPOHYHOFRPPDQGV
PacketShaper’s system commands can be accessed by typing help sys at the command prompt.
This results in displaying another sub group of system commands that can be executed
seperately.The above slide shows some of the important system commands available via the CLI:
• collectlog: This command collects all the log and diagnostic files on the appliance.
• debuginfo: This command displays all the system debugging information and results in a
single file that contains literally all the diagnostic information on the PacketShaper. The entire
process might take anywhere between 5-10 minutes to complete and is a memory intensive
process.
• diag: This command enables viewing all the diagnostic files on the PacketShaper in the
9.258/ DIAG directory. System information such as PacketShaper uptime, NIC information,
basic configuration, output from the setup show command, existing MIBs, memory output
from sys kmemory command and traffic details that hit the default class when the diagnostic
is captured are all recorded. The PacketShaper collects the above mentioned details once every
15 minutes for a maximum of 300 files.
• info: This command displays the following information for your PacketShaper: CPU brand,
CPU speed, memory available — both RAM and Flash, firmware and bootloader versions.
• kmemory: This command displays the memory buffers allocated on the PacketShaper. Two
important commands that are useful during troubleshooting are sys kmemory com and
sys kmemory pkt, which displays the current and allocated values for different memory
components.
• limits: This command displays the current, remaining and total values for the statistically
allocated and dynamically allocated objects on the PacketShaper.
• set: This command displays variable settings on the PacketShaper.

• verify: To troubleshoot hard or flash disk problems on the PacketShaper, use the sys
verify commands. sys verify hard command checks for file system problems on the
hard disk, if any are found, and displays an error message on the appliance’s WUI Info page.
The appliance is reset during the execution of this command. To fix any hard disk problem,
use the sys clean command.

Debugging Commands
Category Sa mple Com mand

System var iables sys set<variables>
Enabli ng debug ging commands sys set showdebugcommands 1
Mo dified system variable s sys set -nd
Packe tShap er MIBs mib<variables>
6OLGH'HEXJJLQJFRPPDQGV
PacketShaper debugging commands can help you troubleshoot problems with the appliance and
its network connections. Debugging commands can affect PacketShaper performance, so you
must first explicitly enable their use. To enable debugging commands, enter:
# sys set showdebugcommands 1
Common debugging commands include:

• sys set variables: Display or change PacketShaper system parameters, including
current, default, minimum, and maximum values.
Important: Blue Coat does not recommend that customers change these values.
The option -nd displays parameters whose values have been changed from their defaults.
• mib variables: Display the counters for the specified SNMP MIBs.
Note: Not all debugging commands are described in this course. Only the ones that are
widely used by Blue Coat Technical Support are covered here.
To see the different system commands that govern the appliance’s operation, type in sys set in
the command prompt. An entire set of system commands are displayed with numerical values
corresponding to the current, default, minimum and maximum values for the system variables on
your PacketShaper.
Important: Blue Coat does not recommend modifying the default values or settings
displayed by the sys set commands.
PacketShaper MIBs can be displayed only after the showdebugcommands are enabled. For
example, typing mib nic0 displays the MIB counters for the Inside port of your PacketShaper.

Enabling Debugging Commands
6OLGH(QDEOHGHEXJJLQJFRPPDQGV
The above slide explains what was described in Slide 5-9 regarding the showdebugcommands.
1. The screen capture shows the Current, Default, Min and Max values for the specific
system variable. The system variable under consideration has a value of zero against the
current and default fields. The minimum value that the variable can take is zero and the
maximum value is 1.
2. Typing the sys set showdebugcommands 1 command allows you to access the hidden
commands in the PacketShaper. Examples of hidden commands are the mib variable, sys
flow and sys bios commands.
3. Typing the sys set showdebugcommands without the numerical value displays the
current value of the system variable. In this case, the current value of the variable is 1, which
has changed from the default value of 0.

Chapter 6: Analyzing PacketShaper Logs
When solving any problem, having a complete picture of the situation is a good place to start. By
starting with general information and working downward, eliminate the things that are obviously
not the cause, and collect information and possible solutions. Additionally, a broad view of the
problem often can give the most information.
Important: Many of the troubleshooting techniques described in this course depend upon
the availability of undocumented and unsupported features and commands.
Although Blue Coat provides this information in troubleshooting courses as an
aid to students attending these courses, Blue Coat does not commit to continuing
to make these features and commands available, in similar or different formats,
in future versions. Moreover, Blue Coat does not commit to resolving service
reports related to the use of undocumented features and commands.
Like solving any problem, it is best to have as much information as possible when troubleshooting
the Blue Coat® PacketShaper®. Although many tools are available, most do not provide a broad
overview of the entire state of the PacketShaper. PacketShaper emits a message, called a heartbeat,
to a designated Blue Coat server on the following occasions:
• PacketShaper boot up (on warm reset or cold boot)
• Daily (every 24 hours of uptime)
• After a system failure (on boot up, following a system restart)
The heartbeat message provides Blue Coat support professionals with key information about the
PacketShaper, such as what modules are enabled, its memory allocation and usage, how long the
unit has been running since last reboot (uptime), and basic configuration settings. Boot up and
crash heartbeat messages append the boot log (a list of the versions that were booted on the
PacketShaper, including the date and time of each boot). Crash heartbeat messages include the
crash log.
Using the information contained in the heartbeat messages, Blue Coat is able to provide better,
faster support to its users. After studying this chapter, you will understand:
• What is contained in the bootlog and crash log files.
• What are the different diagnostic files available in the PacketShaper.
• How to collect the diagnostic files for further analysis.
• Different tools used for analyzing the diagnostic files.

Analyzing Bootlog
Bootlog
– Updated for every reboot
– Located in 9.256/LOG directory
– Contains date and time stamps of reboots, Pack etWise software
version and reboot annotations.
Possible Reboots
– Appliance initiated reboots
– Appliance reboots to safe mode
– Appliance continuously reboots
– Appliance enters boot monitor mode
6OLGH$QDO\]LQJERRWORJ
Log files in the PacketShaper can be found in the 9.256/LOG and 9.258/LOG directories. The
9.256/LOG directory contains log files that correspond to the boot related statistics. The
9.258/LOG directory contains all event notifications that are logged on the appliance’s hard drive.
One of the important files that resides in the 9.256/LOG directory is that of the bootlog, which
contains detailed historical records of booting in the PacketShaper. Detailed analysis of the
bootlog reveals the causes behind individual reboots in the appliance. PacketWise software
ensures that the bootlog contains all the necessary information to analyze with each and every
boot that happens in the PacketShaper. Each entry in the bootlog reveals the date and timestamps
of the reboot, corresponding PacketWise software version and a brief annotation on the condition
of the reboot.
Analysis of the bootlog contents reveals one of the following reboot conditions:
• Appliance initiated
• Appliance reboots to safe mode
• Appliance continuously reboots
• Appliance enters boo t monitor mode
The bootlog file is in plain text and can be analyzed with Note Pad or Word Pad. Details on
analyzing a bootlog can be found in the Hardware Failure Case Studies and Troubleshooting
PacketWise chapters, dealt earlier in the course.

Analyzing Crash Log

Crash log
– Created for every PacketShaper initiated reboot
– Located in the 9.256/LOG directory
– File format : mmddhhMM.SS(GMT)
– Contains information on system events captured before
reboot
Possible reboots
– Appliance initiated reboot only
6OLGH&UDVKORJ
In addition to the bootlog mentioned in the previous slide, the 9.256/LOG directory contains
another log file — crash log, that also helps in troubleshooting different issues on
thePacketShaper. Crash logs are created only when a PacketShaper initiated self- reboot occurs
and can be easily identified by the minus (-) sign at the end of the log entry. PacketWise captures
the last 300 line entries of the system events before such an appliance initiated reboot occurs.
Crash log entries contain output similar to the output from the sys event display CLI
command and some low-level information in hexadecimal format. The output has a file format
similar to MMDDhhmm.ss recorded in GMT. Analysis of this information reveals the exact module
in the appliance that caused the fatal error.
A sample crash log file is shown below. Crash log can be interpreted using a text editor such as
Note Pad or Word Pad.

• The PacketShaper under consideration experienced a crash on Thursday, May 21, 2009 at
08.01.56, Beijing local time. Corresponding time in GMT format also appears in the very next
line.
• The crash log reveals that the appliance is from the PacketShaper 900 family with shaping
feature turned off. Additional details such as appliance serial number and firmware version
are also seen in the crash log.
• Line entries beginning with 001 indicate the last 300 lines of system events just before the
appliance crashed. One of the important indicators for Support is the abbreviations that
appear in each of the 300 line entries. These abbreviations can have values such as Warning
(W), Information(I), Alert(A) or Fatal/Fault(F).

Understanding Diagnostic Files

Contains a series of diagnostic command output
– uptime, version verbose, set show, traffic
and host commands
– MIBs
– System variables and memory buffers
Located in 9.258/DIAG directory
Captured every 15 minutes
Maximum of 300 files
Command list and the interval can be edited
6OLGH'LDJQRVWLFILOHV
PacketWise supports numerous diagnostic commands that generate detailed output necessary to
troubleshoot different problems in the PacketShaper. Some of the details in such a diagnostic
command output include those from the uptime, ver ver, set show, traffic and host
CLI commands. Additionally, certain measurement engine and system variable information in the
form of MIBs are also included in the output.
All diagnostic files can be accessed from the 9.258/DIAG directory on the appliance and is
captured at a regular interval of every fifteen minutes for a maximum of 300 files. In the event of
real-time troubleshooting, the capture interval period can be modified up to five minutes.

Collecting Diagnostic Files

Web User Interface( WUI)
– File browser to zip LOG and DIAG folders
Command Line Interface(CLI)

– Using sys collectlog command
– FTP . zip file from 9.258/MYZIP directory
– Manually zip and upload files
6OLGH&ROOHFWLQJGLDJQRVWLFILOHV
Diagnostic files can be collected for further troubleshooting analysis both from the PacketShaper
WUI and CLI. These files can then be compressed and zipped using a built-in utility in the
PacketShaper. Due to the large size of some of these diagnostic files, Blue Coat recommends that
you attach these files to your support request and not as separate e-mail attachments.
In order to collect the log and diagnostic files through the PacketShaper WUI, open the File
Browser and go to the drive and directory where the files are located. Select the check box next to
each file or directory you want to include in the zip file.
Note: The PacketWise zipping utility uses the recursive parameter: if a directory is selected
to be zipped, the utility will automatically zip all of its contents, including any nested
directories.
To collect the diagnostic and log files through the PacketShaper CLI, use the sys collectlog
command. Executing this command automatically results in two zipped files, each from the
9.258/DIAG and 9.256/LOG directories. The output file have a file format that resembles like
9.258/MYZIP.zip.You can then send these to Blue Coat Support using the FTP option.
Additionally, you can also manually compress the diagnostic and log files using the zip - r
9.258/<file_name> 9.256/log 9.258/diag command. The resulting output can then be
manually sent for analysis using the FTP option.

Analyzing Tools
Parser tool
Blue Coat Internal Tool

– IC Service Center
Freeware Tool
– Parse-O-Matic
– TextHarvest
6OLGH$QDO\]LQJWRROV
The complex nature of the PacketShaper diagnostic files makes it impossible to be analyzed by
network administrators. Understanding and interpreting the diagnostic information contained in
more than 300 different files in a given instance requires highly experienced support personnel.
Blue Coat uses a combination of both proprietary and public tools to interpret the diagnostic
information collected fromPacketShaper appliances. This slide provides an overview of two such
tools — IC Service Center and Text Harvest. These tools use parsing technology to interpret the
series of diagnostic data to a more readable format.
IC Service Center is a Blue Coat proprietary tool used only by Blue Coat Support engineers. The
following slide describes a typical output from the IC Service Center tool.
Text Harvest is an integrated set of tools to gather, extract, organize, and search information across
the Internet. With modest effort users can tailor Text Harvest to digest information in many
different formats, and offer custom search services on the Internet. A key goal of the tool is to
provide a flexible system that can be configured in various ways to create many types of indexes.
Text Harvest also allows users to extract structured (attribute-value pair) information from many
different information formats and build indexes that allow these attributes to be referenced during
queries, for example, searching for all documents with a certain regular expression in the title
field.
An important advantage of Text Harvest is that it allows users to build indexes using either
manually constructed templates for maximum control over index content or automatically
extracted data constructed templates for easy coverage of large collections, or using a hybrid of the
two methods. Text Harvest is designed to make it easy to distribute the search system on a pool of
networked machines to handle higher load.

IC Service Center Output
6OLGH,&6HUYLFH&HQWHU
IC ServiceCenter helps you rapidly analyze diagnostic files from the PacketShaper. It downloads
diagnostic files from the /DIAG directory of PacketShaper and can be used to diagnose errors and
potential problems. IC ServiceCenter archives your files on a Blue Coat file server, and you can
return to view them at any time, or download data at a later point and then switch between
current and previous data.
Support engineers can easily download the necessary diagnostic files from the PacketShaper using
the Get Files option on the IC Service Center menu.Then, choose whether to use Smart Analysis to
rapidly identify possible issues, or Data Drilldown to isolate specific variables. You can select a
specific time range, print information, and save information locally using CSV (comma-separated)
format.
The desktop version of IC ServiceCenter is also available. You can use the desktop version to
analyze diagnostic files you have already downloaded to your local machine.
Data from different PacketShaper MIBs can be collected and interpreted to identify issues on the
appliance. The data can then be analyzed as either a CSV output or using a graphical format. For
example, the slide shows CPU utilization on a PacketShaper appliance is interpreted graphically
using the IC Service Center. As you can see, the CPU utilization data is captured over an 8-hour
time frame from 19.00 to 15.00 hours. Current and average utilizations statistics are represented
using color coded line graphs. Any variation in the current utilization variable can be easily
pinpointed to the exact time of occurrence and can lead to further analysis in troubleshooting
possible performance related issues in the PacketShaper.

TextHarvest Output
6OLGH7H[W+DUYHVW
Customer A reports significant impact on one of his critical applications in his network
environment. Blue Coat Support requests related diagnostic information to identify and pin-point
the exact cause of the above problem. One of the preliminary steps in the diagnosis is to check on
the status of traffic shaping in the PacketShaper. Turning shaping off could possibly be the cause
of the above problem. The above slide describes the steps involved in using the Text Harvest tool
to identify the exact occurrence of shaping being turned off on the PacketShaper.
1. Step 1 displays the entire set of diagnostic files downloaded to a local directory. As you can
see, there are more than 300 different files that are contained in the zipped file.
2. Each individual diagnostic file can then be opened using text editors such as Text pad or Word
Pad. A look at the data presented shows that traffic shaping is turned on.
3. Using the Text Harvest capabilities, all the 300 different diagnostic files can be analyzed to
identify when traffic shaping was turned off. Through the Input file option, provide the exact
location of the diagnostic files by navigating to the appropriate local directories.Use of wild
card features is permitted in the Text Harvest tool.
4. Specify keywords or variables such as date and packet shaping in the /Keep List field. In order
to optimize the resultant output, make sure to include the proper syntax for these keywords in
the appropriate field.
5. Click Start to initiate the query based on above criteria in all the 300 diagnostic files.
6. Resulting output shows all the instances involving the boolean combination of date and
shaping keywords. Using the File Viewer button in the output, one can easily narrow down to
specific keywords, in this case, to identify when shaping was turned off.
7. File Viewer output reveals that traffic shaping was turned off onTuesday, J an 08, 2008 at
14:55:52. Such detailed information can help in identifying exact causes of performance on the
PacketShaper.


Chapter 7: Hardware Troubleshooting
When problems with Blue Coat equipment are reported, it is important that support engineers
properly diagnose the cause of the problems before attempting to fix them. A problem can fall into
one of several categories:
• Hardware: One or more physical pieces of equipment has failed, causing either partial or total
inability to normally use the equipment.
• Software: One or more of the programs running on the hardware has failed to perform as
specified.
• User error: Some issues that are reported as hardware or software problems are actually the
result of a user’s incorrect usage of a product.
This chapter covers identification and resolution of hardware issues; other chapters in this course
deal with software issues. This chapter focuses on Blue Coat® PacketShaper® appliances, but the
troubleshooting techniques for other appliances are similar.
• The tools available to diagnose hardware problems.
• How to determine whether a unit is serviceable in the field or must be returned for repair or
replacement.
This chapter will not teach you:
• How to make the actual repairs to a defective unit. Refer to the installation instructions for
each particular piece of hardware.
• How to solve every possible hardware issue you may encounter in the field. Rather, this
chapter aims to provide a series of procedures you can follow to analyze incidents as they
occur, and then discusses some specific problems that have been reported and how to resolve
them.
When performing any of the hardware-related steps in this chapter, make sure that proper health
and safety procedures are followed, even if you are not the person actually working with the
hardware. Refer to the installation guide for each product for details on procedures to follow to
ensure safe handling. Failure to follow proper procedures might damage equipment and pose a
health hazard.

Overview
Define the pathology
– Hardware issues
– Configuration issues
– Other issues
Solutions to hardware issues

– Use PacketShaper recovery tools
– Replace serviceable parts
– RMA
6OLGH,QWURGXFWLRQ
One of the most important steps in hardware diagnosis is to determine the actual cause of the
problem. As many as 40% to 50% of reported hardware failures are not true hardware failures.
Instead, these are caused by such things as:
• Improper installation of hardware: If devices or boards are plugged into the wrong connector
or are not seated properly, they might be reported as non-functional.
• Improper installation of software, such as an operating system upgrade: If the proper upgrade
path is not followed, the appliance might not work as desired.
• Incorrect understanding of how the appliance is supposed to work: Reports of this type
generally fall into the read-the-manual category. Users might not understand a system’s boot
cycle, a device’s operation might not be fully understood, or a user might not understand a
message or display.
These cases still require support activity in order to resolve them to the user’s satisfaction. This
chapter focuses on hardware-related issues; for information on resolving other types of issues,
refer to other chapters in this course.
Another goal of hardware diagnosis is to ensure that customer equipment is returned, or RMA’d
(the letters stand for return merchandise authorization), only when necessary, that only the proper
equipment is RMA’d, and that the proper replacement equipment is sent. When a hardware
failure occurs, it is important to identify the failed component and determine whether the problem
can be corrected by sending a replacement component, possibly enabling the equipment to
operate in degraded mode until then.
But some hardware problems are not serviceable in the field and require shipment of replacement
units. Prompt diagnosis of such problems is important so that the replacement can be placed in
service as quickly as possible.
Hardware problems are, of course, unplanned, asynchronous events. The first report of a problem
can come in one of several ways:

• The PacketShaper might completely stop working. Although this might appear to be a serious
problem, there sometimes are simple causes (the unit was unplugged) and resolutions (plug it
back in) for such problems. If simple checks do not resolve the problem, however, a more
serious failure might have occurred.
• The PacketShaper might note a hardware-related issue by making an entry in the event log
and, optionally, sending an e-mail alert to the PacketShaper administrator. In these cases, the
PacketShaper continues to function, but possibly in a degraded mode, and a more serious
failure might be imminent if action is not taken.
• The PacketShaper administrator might receive a report from a user that something is not
working as expected. While such reports often point to software problems (or no actual
problem at all), a user’s report might be the first indication of a component failure.
Administrators should be able to analyze a situation enough to determine whether a problem
exists and, if so, whether it requires initiating an external support request.
The problem will have one of these resolutions:
• No trouble found.
• Trouble is not related to hardware.
• One or more components will be replaced.
• The entire unit will be replaced.
Sometimes, political considerations also are part of the troubleshooting process. There may be
cases where continuing to spend time diagnosing a problem is no longer desirable and a quick
solution — although not the most optimal solution — will be chosen. Such considerations are
beyond the scope of this course.
The art of troubleshooting consists of determining which resolution is most appropriate for each
particular problem. As with all troubleshooting, let the charts on the following pages serve as a
guide, not as firm rules.

Hardware Matrix
Pro du ct lin e / Featu res PS 900 PS 1700 PS 3500 PS 7500 PS 10000
Max throughput 2 Mbps 10 Mbps 45 Mbps 200 Mbps 10 Gbps
Max classes 256 512 1024 1024 2048
Max concur rent flows 7.5K 45 K 60 K 300 K 450 K
Compression 2 Mbps 10 Mbps 20 Mbps 45 Mbps 155 Mbps
Max compression 5 15 30 100 1000

tunnels
Links speeds with 512K, 2M 2M, 6M, 2M, 6M, 10M, 10M, 45M, 100M, 200M,
Shaping 10M 45M 100M, 200M 310M, 1G
Interface pairs 2 1 1+LEM option 1+LEM option 1+ LEM option
Interface type Copper Copper Fiber options F iber options Fiber options
Size Small form 1U rack 2U rack 2U rack 2U rack
6OLGH+DUGZDUHPDWUL[
PacketShaper hardware models can be categorized based on the LCD panel setup and comprise of
the following groups:
• PacketShaper 900
• PacketShaper 1700, 3500 and 7500
• PacketShaper 10000
Appliances in the same group share similar LCD panel messages. The above slide describes some
of the hardware and configurations details for the currently supported models in the
PacketShaper product line.

Front Panel Indicators
6OLGH)URQWSDQHOLQGLFDWRUV
Understanding the front panel displays of the PacketShaper is an essential first step in problem
diagnosis. This section describes the front panel displays of the four current models in the product
family, shown in the above photos.
All PacketShaper models

1. Power LED:
Illuminated when the appliance is plugged into an active power outlet and is turned on.
Status LED:
❐ Green: All links are up, shaping is on, site router address is detected by the PacketShaper
or set to none.
❐ Amber: If the above conditions are not met.
PacketShaper 900
1. Disk LED: Indicates flash memory and or hard disk activity.
2. Link LED:
❐ Amber: When link is up and flickers for network activity.
❐ Green: When the appliance is connected to a 100 Mbps link.
PacketShaper 1700
1. Fault LED: Illuminated when in safe or corrupted mode.
2. Link LED: Illuminated when the network cable is properly connected on both ends
Tx/Rx LED: Flickers when the appliance is transmitting and receiving data.
Speed LED: Indicates link speed

❐ Amber: 1 Gbps
❐ Green: 100 mbps
❐ Off: 10 Mbps
3. Link, Tx/Rx and Speed LEDs for the one gigabit management port.
4. LCD panel.
PacketShaper 3500 and 7500 models

1. Fault and Status LEDs
2. Link, Tx/Rx and Speed LEDs for the network ports.
3. Link, Tx/Rx and Speed LEDs for the one gigabit management port.
4. LCD panel
5. Two expansion slots to accommodate LAN Expansion Module(LEM).
PacketShaper 10000
1. Fault and Status LEDs
2. Two expansion slots to accommodate LEMs.
3. Network port with Small Form-factor Pluggable(SFP) transceivers.
4. LCD panel

Initial Visual Inspection
6OLGH9LVXDOLQVSHFWLRQ
If a PacketShaper is visibly malfunctioning or does not function at all, a few basic high-level
checks should be performed first. Even though most of these are common-sense troubleshooting
techniques, always verify that they have been tried before attempting to diagnose a more serious
problem.
1. Perform some basic power checks:
a. Is the power cord plugged in at both ends, at the PacketShaper and at the power
source?
b. Is the power source operating properly?
c. For models with a power switch, is the switch on?
d. For models with the front panel LCD, is the backlight on the LCD panel on?
e. Is there a noise coming from the fan?
Also, note that in some countries, power plugs have their own internal fuses, and a power
fault might be caused by a blown fuse in the plug.
2. If the PacketShaper is receiving power but does not respond, there might be a power supply
problem.
3. If the PacketShaper does not boot properly, determine what is happening during the attempt.
Refer to the Troubleshooting PacketWise Software chapter in this course.
4. If the PacketShaper boots properly, an individual hardware component (such as a disk) might
be failing. Gather the following information:
a. Bootlog statistics.
b. Diagnostic files
c. Related CLI output.

Diagnose Boot Problems
6OLGH%RRWSUREOHPV
If the PacketShaper is receiving power but does not boot properly, a corrupted or failed disk is a
common cause.To diagnose a PacketShaper that fails to boot, try these steps:
1. Apply power to the appliance and check the power LED for any activity. The power LED is
illuminated when appliance is plugged into an active power outlet and the unit is turned on.
If so:
a. The PacketShaper LCD displays Booting / Loading .... message.
b. Check if the PacketWise software is loaded successfully. If so, the appliance can be
accessed by the Web User Interface (WUI) or the Command LIne Interface(CLI).
c. If not, the appliance enters PacketWise failed-to-load state. Refer to the Troubleshooting
PacketWise Software chapter for troubleshooting this condition.
2. If the PacketShaper power LED does not turn on, perform basic checks on the appliance as
explained in Slide 7-4.
3. If the basic checks on the appliance is does not reveal any issues, the PacketShaper is probably
Dead-On-Arrival (DOA). Contact Blue Coat Customer Support to request a Return
Merchandise Authorization (RMA) on the hardware.

Diagnose Flash Disk Issues

Check available flash memory
Delete files with .bad extension
Execute flash disk recovery command via

– Telnet, SSH or Serial console
– Boot monitor mode
6OLGH)ODVKGLVNLVVXHV
PacketShaper stores the PacketWise software that is needed to boot the appliance, in its flash
memory. The PacketShaper’s flash disk (9.256/) contains the following directories: BIN,
CFG,LOG,CMD and PLG. If any one of these directories is corrupted, the PacketShaper will not
boot properly. Access the PacketShaper WUI and locate any error messages displayed under the
Info tab. The front panel LCD display is another location to look for any booting or loading issues
on the PacketShaper.
Access the flash disk through the file browser option in the PacketShaper WUI. One of the first
steps to identify flash disk issues is to look for.bad file extensions in the contents. If so, take the
following measures to rectify such issues:
• Delete files with the .bad extension in the flash memory. In order to delete such files, disable
write protection on these files by executing the sys set writeprotectfiles CLI
command. PacketShaper then tries to restore these files from the .SAV or .OLD files in the flash
disk.
• Execute flash recovery commands from the PacketShaper CLI. This can be achieved through
two different methods:
❐ Access the appliance through Telnet, SSH or serial console and execute the sys verify
flash CLI command to check for errors on the flash drive. Any errors detected in this
process can be rectified by executing the sys clean flash CLI command.
PacketShaper removes any file system problem from the flash disk. During the process,
PacketShaper stores the neccesary files in a temporary location and formats the flash disk.
The contents of the flash disk can then be restored.
❐ If the flash disk recovery using the sys clean flash was not successful, perform flash
disk recovery from the boot monitor mode. Select option H) Erase and reload PacketShaper
Compact Flash from the boot monitor menu options.

Diagnose Hard Disk Issues
6OLGH+DUGGLVNLVVXHV
The PacketShaper hard disk (9.258/) contains these directories: SAV, MEASURE, DIAG, LOG,
PKTLOG, TMP and AGENT. Any issues in the hard disk is displayed on the Info tab in the
PacketShaper WUI. Executing the banner show CLI command also brings up error messages
that indicate issues with the hard disk. The above flowchart describes some the steps needed to
diagnose hard disk problems:
1. Check the availability of the hard disk by either one of these options:
a. Through the WUI, change to the 9.258/ directory. Is the 9.258/ directory available? If
so, perform hard disk recovery. If not, proceed to
b. Through the CLI, execute the sys disk stat or sys dio stat commands to
check for hard disk errors. If the disks are detected, perform hard disk recovery on the
appliance.
2. Execute the sys eventdisplay CLI command

Diagnose NIC Issues
6OLGH1,&LVVXHV
The above flowchart shows some high-level steps to take when diagnosing problems with
network interface cards.
Before beginning problem analysis, you should have:
• Sysinfo data from the ProxySG.
• Statistics from the switch, hub, or router to which the NIC is connected.
• A serial console connection with crossover cable.
After you have collected these items, follow these steps:
1. Check the rear of the ProxySG to see whether the NIC’s link light is on. If it is, go to step 4.
2. Perform basic equipment checks:
a. Verify that the NIC is currently connected to the correct switch, hub, or router.
b. Reseat the card. A loosely fitted card might be the cause.
c. Check speed and duplex settings on the ProxySG.
d. Check speed and duplex settings on the switch, hub, or router at the other end of the
connection. Both devices must have the same settings.
e. Verify that the Ethernet cable is functional. Use another cable that is verified as
working.
f. Verify that the switch, hub, or router port is functional. Use another port that is
verified as working.
3. Check again to see whether the NIC’s link light is on. If it is not, the NIC is faulty and should
be replaced.
4. Check the statistics for the NIC by issuing these commands on the serial console:

% ping -t interface
% show arp
% show ip-route
Also, check for duplicate IP addresses in the same subnet.
5. If the problem still exists, check the release notes for the NIC and for the ProxySG, and escalate
the problem if necessary. Otherwise, continue to monitor the equipment or close the problem
report.

Other Components
CMOS battery
– Error message : Irregular voltage
– Check Info tab on the W UI and LCD panel
Redundant power supply

– Error message : Power 1 or 2 failed
– Check Info tab on the W UI and LCD panel
RAM
– Error message : FAILED, Error = 03
– Check serial console
6OLGH2WKHUKDUGZDUHFRPSRQHQWV
Other components of the PacketShaper can malfunction. This section lists some of these
components and some problems that have been reported.
CMOS Battery
The CMOS memory holds the BIOS settings that you define in the BIOS setup program. This
memory is powered using a battery so that the settings are not lost when you turn off the power to
the system. Many motherboards do not test the status of this battery and will just indicate
problems with the CMOS memory when the battery fails, but some motherboards can detect
when the battery is no longer functioning.
CMOS battery failure can be easily detected from the Info page on the PacketShaper WUI. Error
message highlighting irregular voltage are typically found on the WUI. Similar messages can also
be observed by issuing the banner show CLI command or from the PacketShaper LCD panel.
Once identified, the faulty CMOS battery can be easily replaced by requesting a replacement from
Blue Coat.
Redundant Power Supply

This is a power supply that actually includes two (or more) units within it, each of which is
capable of powering the entire system by itself. If for some reason there is a failure in one of the
units, the other one will seamlessly take over to prevent the loss of power to the PacketShaper.
Problem with redundant power supply is generally observed on the Info page of the PacketShaper
WUI or from the LCD panel. Typical error message reads as Power 1 failed or Power 2 failed . Faulty
power supplies are field replaceable and can be hot swapped in PacketShaper 7500 and
PacketShaper 10000 models.

RAM
Problems with PacketShaper RAM can be due to one of the following reasons — RAM not seated
properly or dislocated from its position . PacketShaper fails to boot under such circumstances and
boot error such as FAILED, Error =03 can be viewed from the PacketShaper serial console.
Reseat the RAM and power cycle the PacketShaper and see if the above error is rectified. RAM is
not field replaceable and can only be replaced by issuing a RMA to Blue Coat.

Chapter 8: Hardware Failure Case Studies
This chapter covers two Blue Coat® PacketShaper® case studies related to customer reports of
hardware failure.
This chapter is structured so that these case studies are analyzed based on the preliminary
information obtained from real service requests received from Blue Coat customers. It is highly
suggested that the support engineer ask for additional information from the customer only when
the available information at hand is inconclusive in troubleshooting the customer’s problem.
Asking for extra information might lead to customer frustration and does not help the support
process.
On the other hand, as these case studies show, information supplied by the customer is not always
complete. Alternate means of diagnosis sometimes need to be investigated.
These case studies are presented in a step-by-step manner that reflects the actual troubleshooting
steps that you would take if you were diagnosing a similar issue.
• How to create an archive of basic PacketShaper log and diagnostic files.
• How to use text-processing tools to simultaneously analyze multiple related incidents.
• How to analyze crash logs to determine high-level reasons for PacketShaper reboots.

Case 1 – NIC Problem
6OLGH&DVH³1,&SUREOHP
In this case study, a customer located in Perth, Australia, reported that their network had gone
down on March 2, 2009, between 9 a.m. and 10 a.m. local time.
The first step in analyzing such a problem is to learn as much as possible about the customer
configuration. These facts were discovered:
• A PacketShaper is deployed inline between a Layer 4 switch and a router.
• During initial configuration, straight-through cables were installed between the switch and
the Inside port of the PacketShaper, and between the Outside port and the router. Because the
PacketShaper supports automatic medium-dependent interface crossover (MDIX), crossover
cables are not required.
• Automatic negotiation of link speed has been enabled.
When a PacketShaper loses its connection to the network, a visual inspection of the LEDs on the
Inside and Outside ports often indicates a possible cause. As shown in the above diagram, when
connection to the network has been lost, the Inside and Outside port LEDs usually will display the
following:
1. LEDs for the Inside port:
❐ LINK: Illuminated, which means the inside link is up.
❐ Tx/Rx: Flickering, which means that traffic is passing between the PacketShaper and the
switch.
❐ SPEED: Green, which means that link speed has been negotiated at 100Mbps.
2. LEDs for the Outside port:
❐ LINK: Not illuminated, which means the outside link is down.
❐ Tx/Rx: Not flickering, which means that no traffic is passing between the PacketShaper
and the router.

❐ SPEED: Off, which means that no link speed has been negotiated.
However, in the case reported by this customer, the failure was detected by an external network-
management tool, and nobody on-site visually inspected the PacketShaper at the time of the
failure.
In the absence of a visual inspection, the log and diagnostic files collected by the PacketShaper can
help diagnose the problem and identify the root cause.

Create Diagnostic Archive

Collect output from the following commands on the
PacketShaper:
#
version verbose; date; uptime; set show; banner show
#
rm -f 9.256/cmd/ts-cmd.cmd; rm -f 9.258/ts.zip
setup capture portable ts-cmd.cmd
zip -q 9.258/ts.zip startup.cmd
zip -rq 9.258/ts.zip 9.258/diag
zip -rq 9.258/ts.zip 9.256/log
zip -q 9.258/ts.zip 9.256/cfg/config.ldi
zip -rq 9.258/ts.zip 9.256/cmd
#
Creates an archive called ts.zip
6OLGH&UHDWHGLDJQRVWLFDUFKLYH
The commands shown above are the standard commands that are used in PacketShaper
troubleshooting to collect log and diagnostic files. First, perform the following commands in the
CLI, and save the terminal session as a text file:
• version verbose: Information about the hardware and software on this PacketShaper.
• date: When this archive was created.
• uptime: How long since the last reboot of this PacketShaper.
• set show: Displays the setup configuration.
• banner show: Displays any current error messages that are part of the login banner.
Next, perform the following commands to create the log and diagnostic archives:
• Two rm commands: Remove any previously captured log and diagnostic files.
• setup capture: Captures a new portable configuration file in text-editable format.
• zip commands: Add the indicated files and directories into the ts.zip archive. The -r flag
(“recursive”) means to recursively traverse the entire specified directory, and the -q flag
(“quiet”) means that no status messages will be displayed while the archive is created.
The choice of files and directories to include in the archive is standard and is based upon
accumulated experience in troubleshooting the PacketShaper:
• The file startup.cmd controls how this PacketShaper behaves when it is booted.
• The directory 9.258/diag contains up to 300 diagnostic files recorded by the PacketShaper at
15-minute intervals.
• The directory 9.256/log includes boot logs that show details of reboots and crashes.
• The file 9.256/config/config.ldi contains the current configuration of this PacketShaper.
• The directory 9.256/cmd contains any command files that have been created.

Analyze Boot Log
6OLGH$QDO\]HERRWORJ
To continue your analysis after you have created or received the ts.zip archive, unzip the archive
into a directory on your workstation.
In this case study, the first file to analyze is the boot log, which contains a record of all
PacketShaper reboots and is contained in the file 9.256/log/BOOTLOG. This file is in plaintext, so
you can read it with a tool such as Notepad or WordPad.
As shown above, the most recent reboot of this PacketShaper occurred on February 2, 2009, at
00:58:11 GMT. Entries in the boot log always are recorded in GMT, so you need to mentally correct
for local time to reconcile it with customer problem reports. In this case, the time difference for
Perth is GMT+8, so the reboot took place at 08:58:11 local time.
However, the customer reported the problem on March 2 — not February 2 — so there is no reboot
logged at the time of the problem.
Also, the following can be concluded from the last entry in the boot log:
• This reboot was a hard reboot; there is no indicator next to the software version release date.
• There is no crash log; there is no minus sign at the end of the line.
Because this problem does not appear to be related to a reboot of this PacketShaper, continue your
analysis.

Analyze Diagnostic Files
6OLGH$QDO\]HGLDJQRVWLFILOHV
Next, analyze the diagnostic files in directory 9.258/diag. As many as 300 diagnostic files are kept
by the PacketShaper and collected once every 15 minutes, meaning that you could have a very
large volume of information to digest in order to diagnose a problem.
Rather than individually scan each file for relevant information, you should use a tool that looks
for similar information in multiple files. Blue Coat technical support uses an internal tool called IC
Service Center, you might have the legacy Unix text-search command grep available on your
workstation, or you can use a third-party text-parsing tool such as TextHarvest.
For consistency across multiple audiences and backgrounds, this course uses TextHarvest for
scanning multiple text files. TextHarvest is available at most popular freeware download sites.
The above example shows the output of TextHarvest scanning the 300 diagnostic files that were
collected in the ts.zip archive. For each file, TextHarvest was instructed to display lines that
contain any of the following text strings:
• Date, time, timezone
• Inside nic speed
• Outside nic speed
Spelling and spacing are significant when processing these files; for example, the two-word phrase
time zone would not match timezone.
Among all 300 of the files in this case, one of them is different. On March 2 at 9:53 a.m. local time,
the Outside link was reported down. Out of the 300 files, one anomaly has been quickly spotted,
and it corresponds to the time range of the initial problem report.

Case 1 – Solution
NIC cable loose or unplugged
Problems with the device connected to Outside port

– Down
– Failed to negotiate link speed
6OLGH&DVH³VROXWLRQ
What caused the problem? Some possibilities include the following:

• The NIC cable was not properly connected between the Outside port and the router.
• The router was down.
• The router failed to properly negotiate link speed with the PacketShaper.
The PacketShaper did not reboot during the time of the incident and probably is not the cause.
This leaves the NIC as the likely source of the problem. What could be the root cause?
• The cable was loose, or someone might have accidentally unplugged the cable and then
plugged it back in.
• The router was rebooted and failed to negotiate link speed. When routers are connected with
automatic link speed negotiation and renegotiation happens after a reboot, the NIC might
require some time to properly negotiate the link speed, and this could lead to a report of the
link being down. If this happens frequently at a customer site, recommend that the customer
hard-code the NIC link speeds instead of using automatic negotiation.
These possibilities were communicated to the customer, who was satisfied that the PacketShaper
was not to blame.
Although the customer’s use of straight-through cables was not a problem in this case, auto-MDIX
can cause problems in some scenarios. When a PacketShaper is connected to Layer 3 devices, Blue
Coat recommends the use of crossover cables because of intermittent problems that can occur
during negotiation. Again in this case, hard-coding link speeds can resolve such an issue.

Case 2 – Crash Issue
6OLGH&DVH³FUDVKLVVXH
In the second case study, a customer reported that the PacketShaper was rebooting approximately
once per hour. This in turn was causing the customer’s network to flap — a condition in which a
router advertises a destination network first via one path and then via another path.
A normally deployed PacketShaper cannot cause a customer’s network to go down. When a
PacketShaper reboots or is shut down, the bypass relay engages and allows traffic to pass through
the inoperative PacketShaper. (Some PacketShaper models can be configured as a direct standby.
In these cases, the PacketShaper should be a potential point of failure in the customer’s network, so
the bypass relay is normally disabled in such configurations.)
Network traffic continues to flow, but the PacketShaper might drop a few packets while the
bypass relays switch over. (This is the clicking sound that you might hear when a PacketShaper
goes down or comes back up.) This can cause a link-state change on the connected devices,
typically a switch and a router. These devices have two things to cope with:
• Renegotiation of the interfaces. This can take more than a couple of milliseconds on some
devices and might not complete correctly. To avoid this, hard-code all link speeds for
permanently connected devices such as routers, firewalls, and servers.
• Flapping interfaces trigger events such as routing updates and Spanning Tree Protocol (STP)
activity. These can block the interface for a longer period — from seconds to even a couple of
minutes.
In addition to these observations, log and diagnostic files on the customer’s PacketShaper can be
used to analyze the crashes and determine possible causes.

Analyze Boot Log
6OLGH$QDO\]HERRWORJ
Again in this case, a ts.zip archive was created and unzipped, and the first step is to examine the
boot log in the folder 9.256/log.
Two key points are apparent from the boot log:
1. All of the reboots are crashes. This is indicated by the minus sign at the end of each line in the
boot log.
2. The crashes took place at 10:46, 11:56, 13:06, 14:16, and so on — about once every 70 minutes.
The boot log suggests a recurring, serious problem with this PacketShaper, so additional analysis
is indicated.

Analyze Most Recent Crash
6OLGH$QDO\]HPRVWUHFHQWFUDVK
Every time a PacketShaper crashes, a crash log is created in the directory 9.256/log. To continue
analyzing this case, open the log corresponding to the most recent crash, as shown above.
As part of the crash log, the PacketShaper records up to the 300 most recent system events that
took place before the crash.
In the above examples of the system event log, note the following:
1. Event 298 shows that the non-maskable interrupt (NMI) or hardware watchdog has been
triggered, and event 300 suggests that the software installed on the PacketShaper encountered
a condition (either hardware or software) that it was not designed to handle. The F: identifier
shows that a fault has occurred.
2. Scrolling up in the system event log can reveal other events that took place before the
PacketShaper rebooted. Event 36 is a warning (marked with the W: identifier) that shows the
hard disk is thought to be out of space. In this series of events, the function prefix is me, which
suggests that the PacketShaper measurement engine is reporting the errors and is unable to
write its database to the hard disk.
The same series of messages occurs repeatedly before the crash. Because multiple crashes were
reported, the next step is to analyze another crash log.

Analyze Another Crash
6OLGH$QDO\]HDQRWKHUFUDVK
As shown above, a second crash log taken about 70 minutes before the most recent log shows the
same events, same faults, and same warnings:
1. The NMI and hardware watchdog is triggered.
2. Before that event, a series of warnings about disk space is generated from the measurement
engine.
In fact, when other crash logs from this case were analyzed, the same crashes were found in all of
them. (You can use a tool such as TextHarvest to search for the messages across all of these crash
logs.)
Checking additional crash logs is an important part of problem analysis because you should verify
that no other factors were contributing to the series of reboots that the customer reported.

Case 2 – Solution
Hard disk issue
– Volume might be full
– Errors were detected
– Delete all unnecessary files
Corrective actions
– Perform measure reset
– Perform sys clean hard
If problem persists, RMA disk or PacketShaper
6OLGH&DVH³VROXWLRQ
What caused the problem? Based on analysis of the crash logs, some possibilities include the
following:
• The hard disk might be full.
• The hard disk might be experiencing hardware faults.
The following commands were performed in the CLI on the PacketShaper:
• measure reset, to reset the measurement engine.
• sys clean hard, to format the hard disk and restore its factory default settings.
In this case, this fixed the customer’s issues, and the PacketShaper stopped rebooting.
However, if the problem had persisted after taking these actions, the next step would be to RMA
either the hard disk or the entire PacketShaper.

Chapter 9: Configuration Issues
Majority of the support requests relating to the PacketShaper can be easily identified as system
configuration mistakes. Configuration errors can appear when reverting to earlier PacketWise
software versions since they don’t support services or features you have configured on the Blue
Coat®PacketShaper®. You may also see configuration errors if you have the wrong version of a
plug-in installed or if a plug-in is missing.
After completing this chapter, you will understand:
• The different types of configuration errors and how to identify them.
• How wrong system settings can cause configuration errors.
• What types of configuration errors arise from unresolved ARP and DNS entries.
• What errors arise in the traffic class tree and matching rules within these classes.
• How a wrongly configured adaptive response agent can cause configuration error.
• How to resolve errors after restoring a PacketWise configuration.

Configuration Errors
6OLGH&RQILJXUDWLRQHUURUV
Configuration errors in the PacketShaper can be easily identified from both the Web User Interface
(WUI) and the Command Line Interface (CLI). Info tab on the WUI and the banner show CLI
command are the most widely used approaches to identify any configuration errors on the
PacketShaper. Errors thus displayed form the prerequisite for troubleshooting more complex
issues. The above slide describes some of the commonly encountered configuration errors on the
PacketShaper:
1. Configuration errors due to matching rule problem in a traffic class and a disabled Winny
class is observed in the Directory Services section of the Info tab.
2. Wrongly configured network gateway details can sometimes cause the appliance to route
transactions to non-local address. Setting wrong values for the NIC speed fields can also
create configuration errors and are displayed in the Info tab.
3. During a PacketWise upgrade, old plug-ins that are necessary for the functioning of the
PacketShaper become obsolete and are not loaded. Such instances can also cause
configuration errors to appear on the PacketShaper.
4. Configuration errors can also be identified by issuing the banner show command. the
output displays the messages that are initially shown after logging into PacketWise software.
You can use the banner show command to display all of the appliance’s configuration errors,
warning messages, and notices.
100Property of Blue Touch Training Services.

Initial Setup
6OLGH,QLWLDOVHWXS
When you deploy a PacketShaper on your network, you must configure settings, such as an IP
address, to enable communication with the appliance. These settings are configured during the
Guided Setup process. After viewing the error messages displayed on the Info tab, verifying the
basic settings on the appliance can help pinpoint any errors during the initial configuration
process. In the above slide:
1. Verify that the Gateway and SiteRouter fields have proper values in them. Gateway is the IP
address that the PacketShaper uses to reach other networks. Site router contains the IP address
of the access router to the link the appliance is managing. Sometimes, customers enter the
same IP address details for both the gateway and site router. If there is only one router
configured on the network, PacketShaper will detect traffic that is passing through that router
alone. This results in other traffic being ignored and will cause classification related problems.
As a best practice, always set the site router value to none.
DNS server needs to be properly configured for the functioning of SNTP server and for DNS
lookup of certain traffic classes.
In order to properly manage bandwidth, PacketWise must know the capacity of the access link
it is managing. These details are provided in the Inbound Rate and Outbound fields. It is also
imperative that the license keys for different modules is equal or higher than the WAN settings.
NIC mode settings under the LAN settings field needs to be properly set. In the above screen
capture, Inside Fast Ethernet NIC Mode is set to auto-negotiate and the Outside fast Ethernet NIC
Mode is set to 10BaseTfull-duplex setting. If set this way, this clearly sets the stage for a NIC
negotiation issue between the outside port and the router.
2. Basic configuration details as seen on the WUI can also be viewed through the CLI by issuing
the setup show command. The output is divided into non-sharable (local) and sharable
settings. The sharable settings are part of the configuration file (config.ldi). Use the output of
this command to verify if the subnet mask IP addresses are properly configured on the
PacketShaper.

a. The gateway setting shows 10.0.0.2 as the configured address with an error message
that says that the gateway is not found. This indicates that the customer did not
configure the gateway with proper details.
Make sure to properly configure the date, time and time zone fields to match the exact
physical location and time zone of the PacketShaper. If not, there is a high possibility that
the report generation process will reflect wrong details.
b. Site router has been configured to use the 10.0.0.1 address when it is recommended
that this field be set to none as best practice.
Use the values displayed from the setup show command output to analyze any
problems reported by the customer.

Address Resolution
6OLGH$GGUHVVUHVROXWLRQ
The watch mode feature in PacketShaper allows you to define routers by either an IP address or a
MAC address. If an IP address is specified, PacketWise will send an Address Resolution Protocol
(ARP) request through the management port in order to resolve the IP address to its MAC
address.
If the IP address of a router cannot be resolved to its corresponding MAC address, traffic to or
from that router will not be classified and watch mode will not work for that traffic. This can be
verified using the arp test CLI command. The above slide shows that the ARP request from the
PacketShaper was not successfully resolved. The same also can be verified using the arp show
command.
Sometimes, customers report that they cannot manage the PacketShaper through a LEM when the
main interface is unplugged. When PacketShaper receives an ARP request, it will broadcast
through all of its ports. It is not uncommon for a LEM interface’s MAC address to be associated
with the wrong interface. The workaround is to create a private ARP entry in the startup.cmd file
so the entry will be automatically created each time the appliance reboots. Use the arp drop and
arp privadd CLI commands for this purpose.

DNS Settings
6OLGH'166HWWLQJV
Customers sometimes report the following — host name resolution does not occur in the case of
time.nist.gov web site, despite the fact that SNTP feature is turned on. As a result of this, the time
and date option on the PacketShaper does not gets updated automatically. Possible causes for this
problem could be that the DNS servers are not properly configured during initial set up or that a
wrong DNS server has been configured.
1. To verify if the DNS server was properly configured, issue the dns server CLI command.
The output in the above screen capture shows that the server is unknown. This confirms that a
wrong IP address was configured as the DNS server. Confirm that the IP address provided
during the initial setup belongs to a DNS server by issuing a ping test.
2. To verify if all domain names and addresses on the PacketShaper are correctly configured,
issue a dns names CLI command. The output lists all the resolved domain names and their
addresses with error messages, if any. In the above example, the domain name time.nist.gov is
not properly resolved and has a corresponding error message that mentions that request to
that address is pending. This only implies that no DNS server was configured to resolve this
request and PacketShaper is continuously attempting to resolve the domain name for this
case.
3. To configure one or more DNS servers for PacketWise to access, use the setup dns CLI
command. Specify up to eight IP addresses, separating each with a space, or use none to clear
previously set addresses.

Traffic Class Errors
6OLGH7UDIILFFODVVHUURUV
Traffic class errors can be easily identified from the Info tab on the PacketShaper WUI. Navigate to
the Manage page to narrow down the exact details of the traffic class error. The traffic tree on the
Manage page displays prominent yellow signs that indicate that particular traffic class has
problems.
In the above slide, Winny and VOIP traffic classes have errors associated with them. Double
clicking the individual traffic classes brings up the detailed explanation of the errors. In the above
example, Winny traffic class exists in the class tree but is disabled by default. Changing the system
variable associated with this class or entirely deleting the traffic class can help resolve this issue.
PacketWise sometimes displays Error 1201 if the traffic class under consideration is not properly
classified. Possible cause for this error could be that there already exists another traffic class with
same name. Renaming the traffic classes with unique names helps resolve this issue.

Matching Rules
6OLGH0DWFKLQJUXOHHUURUV
Matching rules define the criteria used by PacketWise to identify traffic types. Every traffic class
must have at least one matching rule. When traffic discovery creates a traffic class, it creates one or
more matching rules to characterize the traffic type. In a similar fashion, when you create a class
manually, you must specify the matching rules that describe the application’s flows.
A traffic class can have multiple matching rules, which are treated as separate, distinct
specifications. When PacketWise tries to map a traffic flow to a class, it compares the flow with the
criteria in the class’ first matching rule. If PacketWise does not find a match, it continues through
the rules until a match is found or until it runs out of matching rules, in which case it moves on to
the next class in the tree. If a specific traffic class cannot be found for a flow, the traffic is classified
in the Default traffic class for the sub tree. The following are the widely observed errors relating to
matching rules on the PacketShaper:
• Error 3401- Out of object matching rules: This message indicates that the PacketShaper has
reached its configuration limits. Use the sys lim CLI command to verify PacketShaper's
configuration limits. If you have reached the maximum number of classes or matching rules
and need to create a custom class, you can delete unwanted classes, such as DiscoveredPort
classes. This will free up resources to allow the custom class to be placed in the tree.
Note: If you want to be notified automatically when your appliance is close to reaching its
object limits, use the adaptive response feature. The Unit Limits agent monitors the
number of classes, matching rules, partitions, and dynamic partitions on your
appliance and alerts you when the appliance is approaching its limits.
• attr iqosMatchingRule, Failed to add matching rule to traffic class configuration errors: As of
PacketWise 5.2.1, the number of matching rules available is shown using the sys limits
CLI command, should not be allowed to reach 0. Leaving at least two rules available is
required to avoid this type of configuration error. If your PacketShaper is reporting zero rules
available and you are seeing these configuration errors, remove rules and or classes until two
matching rules are available. Wait for at least 10 seconds for PacketShaper to update its
resource allocation.

You should also consider upgrading to a PacketShaper model that supports more traffic
classes and matching rules.
• Matching Rule incompatible with parent...: This error message is encountered while loading the
configuration from one PacketShaper to the other. Checking the matching rules of the parent
class and verifying the value for the Server Location field can sometimes give additional
information for the cause of such errors. Change the Server Location field value to Any.
Additionally, delete the auto-discovered child classes and let them re-discover traffic.
• Matching rule incorrectly defined: As the error message implies, the traffic class under
consideration does not have correct matching rules defined. To resolve this error, delete the
traffic class or modify the system variable values for this traffic class from off to on.

Adaptive Response
Customer experiences
– Slow traffic
– Adaptive response(AR) agent not working
Solutions
– Verify CPU health status
– Check if numerous AR agents are running at the same time
– Verify AR agent parameters
– Check if individual AR agents have been enabled
– Check if action file is properly configured
6OLGH$GDSWLYHUHVSRQVHHUURUV
Incorrect configuration of adaptive response parameters can result in configuration errors in the
PacketShaper. Often, one might observe one of the following errors on the PacketShaper:
• Configuration error in /default, object /_goals/XXXXXX, attrib iqosGoalTypeName = "XXXXXX", No
template "XXXXXX" found for agent: This error occurs when a template is not found for an
existing Adaptive Response agent. This may be due to the Adaptive Response plug-in having
been removed, during a software upgrade. Another cause may be that the configuration
references an agent that has become obsolete in the latest version of the Adaptive Response
plug-in.
If the latest Adaptive Response plug-in is loaded and you are still seeing configuration errors,
navigate to the Adaptive Response Settings page and find the agents that are flagged with
errors. These particular agents are obsolete and can be deleted with the agent delete CLI
command.
• No templates were loaded: This error message indicates that the adaptive response plug-in is
not installed on the PacketShaper. Make sure that the most current adaptive response plug-in,
such as AR710v1 for 7.1.0, is placed in the 9.256/PLG directory.
Note: You will need to reset the PacketShaper to load the plug-in.
• Action file xxx.cmd not found: Missing action files are another cause of configuration errors on
the PacketShaper. You might observe error messages on the Info page, as shown in the screen
capture below, as a result of this.

Solutions
Implementing some of the below mentioned procedures can sometimes help resolve adaptive
response configuration errors:
• Scheduling numerous adaptive response agents at the same time will cause PacketShaper to
process these requests in a serial fashion that causes extensive utilization of CPU resources.
Make sure to space out the schedule intervals on different agents to avoid such issues.
Disabling specific agents that cause slowness can also help fix this issue.
• Enable each of the individual adaptive response agents after turning on the feature globally.
These agents are disabled by default and need to be individually enabled for the adaptive
response feature to work properly. When disabled, the notification options through e-mail,
SNMP or syslog alerts are also disabled.
• Re-creating missing action files from the Adaptive Response Settings page can also sometimes
resolve above described configuration errors.
• Install any missing agent template by updating to the current version of the adaptive response
plug-in for your PacketWise software.

Restore Configuration
6OLGH5HVWRULQJFRQILJXUDWLRQ
PacketWise automatically stores your settings in a file named config.ldi. Use the config
save CLI command to create the config.ldi file. The config save and config load
commands are useful for experimenting with different configuration settings. For example, you
can save your current settings, make changes to the configuration (such as create new partitions or
policies), and then return to the original configuration if you prefer it. You can create as many
configurations as you like.
The config.ldi file contains the traffic tree configuration, including all classes, class IDs,
partitions, policies, host lists, and events, as well as all sharable configuration settings such as
packet shaping, traffic discovery, passwords, SNMP, e-mail, SNTP, compression, and Syslog. The
config.ldi file should be backed up on a regular basis, as it can be used to restore a
configuration if needed.
In addition, PacketWise offers a way to capture your traffic configuration and settings in an
executable command (.CMD) file. The backup.cmd file mentioned in the above flowchart can be
created using the setup capture CLI command. Restoring a configuration by running a CMD
file takes much longer, possibly hours than loading a config.ldi, which takes less than a
minute. However, Blue Coat recommends that you create and backup the CMD file as a safeguard
in case the config.ldi fails to load. PacketWise configuration can be restored using the
config.ldi file either by:
• Executing the config load CLI command that loads only the traffic tree configuration.
• Executing the class load CLI command that loads the sharable configuration settings, in
addition to the traffic tree configuration.
The above flowchart broadly describes some of the processes involved in troubleshooting
configuration errors after a PacketWise restore operation:
1. Verify if configuration errors are displayed on the Info page. If so, identify the type of error
message displayed. If matching rules error is displayed, proceed to identify and troubleshoot
the error as described in Slide 9-6.

2. If not, configuration errors can be identified as due to settings mismatch in the customer’s
network environment. Modify the network settings to eliminate any configuration errors.


Chapter 10:Configuration Case Study
This chapter presents another case study based on an actual service request filed by a Blue Coat®
PacketShaper® customer.
In this example, the customer says that the PacketShaper is not properly displaying the hostnames
of the top users of a critical service, preventing the customer from identifying those users.
Although the customer’s perception is that the PacketShaper is not performing correctly, the
reality turns out to be something else.
One of the challenges in providing technical support is telling customers that the cause of a
problem might not be what they first suspect. When the customer needs to configure other devices
or software in order to work properly across their network, the support engineer must use tact and
finesse to communicate this to the customer in a way such that the company’s business
relationship with the customer is not harmed.
• How to investigate DNS-related configuration settings on the PacketShaper.
• How to determine when DNS errors are occurring.
• The interaction between DNS servers and the PacketShaper.

Case Study – Overview
6OLGH&DVHVWXG\³RYHUYLHZ
In this case study, the customer is running a critical application on an internal CIFS server that is
accessed by internal hosts and remote locations. The customer wants to monitor who is using this
resource, but the PacketShaper is not reporting all of the hostnames of top CIFS users.
On the PacketShaper, the customer manually created a class called CIFS_TCP. In the Host Analysis
field for the class, Top Talkers and Top Listeners have been enabled. As shown above, however,
these displays show only IP addresses — not DNS-resolved hostnames.
The customer reports that, other than this issue, the CIFS server is working as expected.

Chapter 10: Configuration Case Study
Verify Settings
6OLGH9HULI\VHWWLQJV
As in other troubleshooting scenarios, the first step is to check basic settings on the PacketShaper.
1. Examine the output of the set show command, or look at the similar information in a ts.zip
file. Under the non-sharable settings, verify that:
❐ A gateway has been configured.
❐ A DNS server has been configured. If the DNS server is in a different subnet, then traffic
between the DNS server and the PacketShaper needs to be routed, which is a function of
the gateway.
2. Under the sharable settings of the set show output, verify that the Inside and Outside
interfaces are set to unsecure. If access to the interfaces is restricted by a list of IP addresses,
then verify that the IP address of the DNS server is in either the Inside or Outside list,
depending on the topology of that specific network.
3. In the output from the arp show command, verify that the gateway is available. If a MAC
address is shown for the IP address of the gateway, then the gateway usually is available.
4. In the output from the dns servers command, verify that the same server reported in set
show is listed here. In this case, the server is available and has been idle for only 12 seconds,
which suggests that it is active. A ping of the DNS server from the PacketShaper confirms that
it is alive and responding.
So far, nothing suggests a cause of the customer’s problem.

DNS Lookup
6OLGH'16ORRNXS
Next, perform some tests from the PacketShaper to the DNS server.
1. Try to look up one of the hostnames in the customer’s network. In this case, the hostname
pld-ad1 correctly resolves to an IP address.
2. Try to look up one of the IP addresses that originally was reported as not resolving. Here, no
hostname is returned for IP address 10.40.57.213.
3. Perform the dns names command, which lists all domain names and addresses that are
configured in traffic class matching rules. Here, IP addresses of external hosts correctly
resolve, but the third host in the list does not resolve because there is no response from the
DNS server.
Any host that is flagged with an error message such as this will not be updated in any
PacketShaper reports.
Now, a hint of the problem is starting to appear.

Chapter 10: Configuration Case Study
DNS MIBs
6OLGH'160,%V
Investigate whether any DNS-related errors are being reported by the PacketShaper. To do this,
examine the DNS-related Management Information Base (MIB) on the PacketShaper.
First, debugging commands must be enabled on the PacketShaper. In the CLI, enter the command
sys set showdebug 1
Important: Diagnostic commands that are visible only when showdebug has been enabled
are intended to be used only under the guidance of Blue Coat Customer Support.
Then, enter the command

mib dns
As shown above, the measurement variable RespNoError shows an increasing number of errors
over time, which is consistent with DNS queries failing to resolve.
With all of the information collected during the stages of this analysis, a conclusion about the
customer’s problem now can be made.

Case Study – Solution

Not all DNS queries are resolved
– Internet hosts are resolved successfully
– Internal hosts are not resolved
Customer’s DNS server needs to resolve internal hosts
– If not, information not available to PacketShaper
PacketShaper is not a DNS server
– It relies on information from DNS servers
6OLGH&DVHVWXG\³VROXWLRQ
To summarize what has been learned about DNS behavior on the customer’s PacketShaper:
• DNS queries from external hosts resolve successfully.
• DNS queries from internal hosts do not resolve.
An important point to remember is that the PacketShaper is not a DNS server. It relies on external
DNS servers to resolve IP addresses and hostnames. Only if the DNS server returns a response can
the PacketShaper include such information in its reports, such as top talkers and top listeners.
Many service requests are generated by customers who misunderstand the interaction between
DNS servers and the PacketShaper.
In many customer networks, DNS servers are configured to not resolve internal hosts. However,
such deployments mean that hostname information for internal hosts is not available to the
PacketShaper.
Here, the customer was informed of configuration requirements for their DNS server, as well as
the consequences for the PacketShaper if the DNS server configuration was not changed.

Chapter 11:Classification Case Study
In this example, a customer reports that their PacketShaper is classifying some network traffic as
belonging to Timbuktu® Pro, a remote-desktop application for personal computers. The customer
is not aware of any Timbuktu traffic on their network, so they want to know its source.
As is often the case in troubleshooting, the analysis of this problem covers many topics without
finding any root cause, or possibly finding problems other than the one that the customer
reported. The art of troubleshooting calls for the support engineer to make intelligent choices
about what areas to investigate and when to suspect that a support engineer might not be able to
correct the problem. Acquiring the skill of knowing when to escalate an issue is a key stage in
one’s development as a support engineer.
• How to identify traffic classes that the PacketShaper discovers automatically.
• How to use the traffic flow command to check traffic flows by class and host.
• When to suspect that an unexpected PacketShaper behavior might be a root cause.

In this case study, the customer reports that a class called Timbuktu has been automatically created
on their PacketShaper, even though the customer is using no such service and is not aware of any
clients who are.
The topology of the customer’s network is shown in the above diagram:
• The PacketShaper is deployed between the firewall and the router.
• The corporate LAN is connected to the firewall through a Layer 4 switch.
• A server farm is connected to the firewall and uses the 172.29.42.0/24 network segment.
• One of the servers in the farm is a Microsoft Exchange server at 172.29.42.24.
The customer wants to know why the Timbuktu class has been created and the source of the
network traffic that is causing this to happen.

Chapter 11: Classification Case Study
New Class Found
6OLGH1HZFODVVIRXQG
As shown in the above screen capture from the customer’s PacketShaper:

1. The customer created the class Inbound/To-42-Subnet with automatic discovery enabled.
2. The class Timbuktu has been discovered.
3. Clicking on the class name shows that it was automatically discovered.
This confirms the customer’s report of an unexpected class, but it does not yet identify the source
of the traffic that is causing it to appear.

Analyze TS File
6OLGH$QDO\]H76ILOH
Next, a ts.zip file was collected from the customer’s PacketShaper. Examine the following:
1. Output from the version verbose command: This is a PacketShaper 3500 and is running
software version 8.3.2.
2. Licenses and plug-ins: The compatibility key is installed, a link speed of up to 6Mbps is
licensed, and only one plug-in is loaded. This is adequate to perform traffic discovery.
3. Output from the set show command: No site router is configured, which means that all
traffic is classified. Discovery is on, shaping is on, and the link speed is 4.5Mbps.
4. Output from the banner show command: The measurement engine has not been reset. This
probably happened because the software image has been upgraded and the measurement
engine was not reset.
5. Memory allocation: Misclassification of traffic can happen if there are failures in memory
allocation. These are shown under the Fails column, but in this case there are none. (The lines
between 5 and 98 are not shown due to space considerations, but they all showed zero failures
as well.)
6. CPU utilization: The CPU currently is 99% idle, which means that utilization is only 1%. Any
non-zero values in the last 10 items would indicate latency that the PacketShaper is injecting
into the customer’s network, but that is not the case here.
So far, the only problem discovered is the measurement engine. Because part of troubleshooting
involves identifying problems of which the customer might not be aware, be sure to report this —
and be sure to warn the customer that resetting the measurement engine will cause the
PacketShaper to reboot and erase accumulated report data. The customer should be reminded to
back up any measurement data and existing reports that they wish to keep.
However, the measurement engine most likely is not the cause of the reported problem. The data
so far show no basic configuration problems, no license key problems, no memory allocation
failures, and no CPU utilization issues, so continue your analysis.

Check for Host-Sidedness
6OLGH&KHFNIRUKRVWVLGHGQHVV
Because the customer said that the Timbuktu class is created automatically, investigate where there
are any host-sidedness issues on this PacketShaper.
The customer enabled top talkers and top listeners on Timbuktu and learned that this traffic is
coming from the Exchange server at 172.29.42.24.
Before asking the customer for more information, use the 300 diagnostic files contained in ts.zip to
determine whether this IP address is found in any of the files. A text-parsing tool such as
TextHarvest makes this task simpler and is shown above:
1. Select all of the files in the diag subdirectory.
2. Display lines containing either the string Date, time, timezone or 172.29.42.24. Because the
header text string Date, time, timezone is found in every file being searched, this creates a
rolling list of timestamps in the output.
3. This reveals that 09:17 local time on January 20, IP address 172.29.42.24 is found on the Inside
port.

Check Host Info
6OLGH&KHFNKRVWLQIR
To further check host-sidedness, examine the output from the host info command, which also is
available in the ts.zip file. (Again, several intermediate lines in the output have been removed
from the above example.)
As shown above, all of the IP addresses in the 172.29.42.0/24 network segment belong to the
inside, which is correct and shows that when the PacketShaper learns about new hosts in this
network segment, it is assigning them to the correct side.
If this had not been the case, then this would need to be corrected on the customer’s system before
proceeding.
Because the traffic seems to be normal so far, the next step is to more deeply investigate the nature
of the traffic that is being misclassified.

Check Traffic Flow by Class
6OLGH&KHFNWUDIILFIORZE\FODVV
The traffic flow command displays information about currently active TCP connections or
UDP sessions that are passing through the PacketShaper. In the example shown above:
• The t option shows TCP connection.
• The u option shows non-TCP flows such as UDP.
• The p option shows port numbers.
• The x option shows full class names in the output (this option is used with the C option that
follows).
• The I option shows nonidle flows.
• The C option shows class names in the output.
• The c option takes the next argument (in this case, Inbound/To-42-Subnet/Timbuktu) as
the class name for which information is displayed.
The output from this command shows:
1. Source and destination hosts with the IP address and port number of each. Here, 172.29.42.24
port 1419 is talking to 172.29.82.108 port 1377. The source port is always 1419, and the
destination port varies, which is common in most TCP applications.
2. The name of the inbound class.
3. The name of the outbound class.
4. The name of the service, which here is Timbuktu-Snd.
This shows that all of the traffic in the Timbuktu class is hitting from the Exchange server. The next
step is to further investigate the traffic to and from that server.

Check Traffic Flow by Host
6OLGH&KHFNWUDIILFIORZE\KRVW
The Exchange server should be used by e-mail applications, so why is Timbuktu traffic being seen
on it? To investigate this, use the traffic flow command to show all of the traffic currently
flowing through the PacketShaper to and from that specific address.
As in the previous example, the traffic flow command takes several options. Here, the final
option differs: The A option takes the next argument (in this case, 172.29.42.24) as the IP
address for which information is displayed.
The output from this command shows:
1. In this flow, traffic to the Exchange server is being interpreted by the PacketShaper as
belonging to the Timbuktu service.
2. Now, traffic is hitting the SameSide class, but the service is MAPI. The traffic is considered
same-side because it originates and terminates on the LAN. (Refer back to the topology
diagram to see why this is the case.) The PacketShaper does not count same-side traffic in
WAN link-size calculations.
3. This is again same-side traffic to the Exchange server, but the service is Timbuktu. Some
same-side traffic appears to be from MAPI, and some appears to be from Timbuktu.
4. This shows a proper classification of traffic. The Exchange traffic shows a proper inbound
class, outbound class, and service.
5. Port 1419 is the correct port number for Exchange traffic, but this is the same port number that
previously is shown as generating Timbuktu traffic.
This suggests some type of confusion in the PacketShaper and that something might not be
performing according to design.


Summary of analysis
– Misclassification was due to faulty PacketShaper software
– Engineering was notified
– Measurement engine also needs to be reset
Solution
– Permanent fix scheduled for next software release
Workaround for this customer
– Include port 1419 in the matching rule for the class
MS-Exchange with these parameters:
Inside port: 1419
Inside host IP address: 172.29.42.24
This problem was escalated to Blue Coat Engineering, who concluded that the PacketShaper was
indeed misclassifying Exchange traffic as Timbuktu traffic at Layer 7. A fix to the PacketShaper
software was developed and scheduled for the next release.
Also, because the support engineer discovered that the measurement engine on this PacketShaper
had not been reset after a software upgrade, it also was recommended that the customer reset the
measurement engine, even though that was not related to the problem that had been reported.
To solve the immediate need of this customer, however, a workaround was suggested. Traffic on
port 1419 can be from either Exchange or Timbuktu, but in this case, the source of port 1419 traffic
is well known — the Exchange server.
In the Edit matching rule display of the WUI, the matching rule for the MS-Exchange class can be
modified to include the inside port number and host IP address of the Exchange server to force
traffic from that address to be classified correctly as Exchange traffic.
This workaround solved the customer’s issue until the next release of PacketShaper software was
distributed.


Chapter 12:Troubleshooting PacketWise Software
PacketWise is the software that consists of many complex modules that control the network
visibility, control, and Xpress features of the Blue Coat® PacketShaper®. It is of utmost importance
that for a PacketShaper to be operational, the PacketWise software be properly installed and
loaded.
With the user-friendly Web User Interface (WUI) and Command Line Interface (CLI), PacketWise
allows you to easily configure and manage the PacketShaper. The PacketWise configuration is
stored in the basic.cfg file. The 9.256/CFG directory in the file system contains the
appliance-specific settings such as IP address, DNS server, NIC information, and domain name.
Problems with the PacketWise software can cause the PacketShaper to either crash or reboot. In
this chapter, you will find basic information that will help you prepare for troubleshooting
PacketWise software issues. After studying this chapter, you will understand:
• The steps involved in the PacketWise loading process.
• Different types of appliance reboots encountered in the PacketShaper and how to troubleshoot
them.
• The contents of a PacketShaper boot log and how to interpret them to understand
PacketShaper reboots.
• The problems arising due to different plug-ins installed on the PacketShaper.

Overview
PacketWise loading process
PacketShaper reboots
– Appliance initiated
– Safe mode
– Continuous reboots
– Failed-to-load
Plug-in related issues
6OLGH&KDSWHURYHUYLHZ
There are many causes of PacketWise software issues, and it is the responsibility of the support
engineer to gather diagnostic information, investigate, and identify the root causes of these issues.
Software-related issues on the PacketShaper fall into these areas: problems with loading of the
software and appliance reboots that are caused by missing or incompatible plug-ins installed on
the PacketShaper.
PacketShaper reboots can be further categorized into:
• Ones that are appliance initiated.
• Ones that can arise because the appliance entered safe mode
• Ones that happen frequently within a short duration of time.
• Ones that occur because PacketWise software failed-to-load, that causes the appliance to enter
boot monitor mode.
A good place to start understanding why these reboots occur would be to look into the boot log
files that are generated. The PacketShaper boot log records significant occurrences that take place
during the operation of the appliance. Boot log files provide valuable information needed to
debug issues arising from such reboots.
PacketWise software issues can also be caused by not installing or upgrading the appropriate
plug-ins on the PacketShaper.

Chapter 12: Troubleshooting PacketWise Software
PacketWise Loading Process
6OLGH3DFNHW:LVHORDGLQJSURFHVV
Loading the PacketWise software comprises of a series of steps. The above flowchart shows a high
level view of the steps involved beginning with switching the PacketShaper on. The appliance
starts to locate a valid PacketWise image file. A default starting point for this would be the
9.256/BIN/image.zoo location in the PacketShaper flash disk. The following are the possible
scenarios that can be encountered during the process:
1. Can the PacketShaper locate a valid image in the 9.256/BIN/image.zoo location? If so, did the
image file get loaded properly? If not, the PacketShaper continues locating the backup image
file in the 9.256/BIN/backup.zoo location.
2. Can the PacketShaper locate the backup.zoo image file? If so, did the image file get loaded
properly? If not, the PacketShaper continues locating the image file in the
9.258/SAV/backup.zoo location. If so, did the image file get loaded properly? If so, the
PacketShaper WUI displays. If not, the PacketShaper enters boot monitor mode.
Note: The above mentioned steps can be interrupted at any time by pressing the Control
+ Y on your keyboard. As a result of this, the PacketShaper enters boot monitor
mode directly.

PacketShaper Reboots
User experiences
– Appliance initiated reboots
– Appliance reboots to safe mode
– Appliance continuously reboots
– Appliance enters boot monitor mode
Solution
– Analyze PacketShaper boot log
Located in 9.256/LOG directory
Contains date and time stamps of reboots, software version
and reboot annotations
Boot log updated for each reboot
6OLGH3DFNHW6KDSHUUHERRWV
One of the most important aspects of troubleshooting PacketWise related issues is to understand
the causes behind PacketShaper reboots. These can be categorized into:
• PacketShaper initiated reboot, which can be because of wrongly installed configuration files or
due to the expiration of the license keys. Such reboots do not occur frequently. This type of
reboot generates a crash log that can be used for further analysis.
• PacketShaper reboots to safe mode due to missing or corrupted configuration files. If the
appliance reboots frequently within a very short period of time, it suspects an error in the
configuration and enters safe mode.
• PacketShaper continuously reboots, which can be because of a corrupted image file or
configuration related issue. On an average, the PacketShaper reboots up to eight times and
tries to recover the backup.zoo file during the process.
• PacketShaper enters boot monitor mode, because it failed to load the image.zoo or the
backup.zoo file from the 9.256/ or 9.258/ locations.
The PacketShaper boot log records significant occurrences that take place during the operation of
the appliance. Boot log is located in the 9.256/LOG directory and contains the date and time stamps
of the various reboots on the appliance. Boot log also contains valuable information in the form of
annotations which can be interpreted during troubleshooting. Contents of the boot log are
updated with each reboot on the appliance. Analyzing the boot log can provide insights into some
of the above mentioned reboots. The following slide describes a boot log and how to interpret its
contents in detail.

PacketShaper Bootlog
6OLGH%RRWORJGHWDLOV
PacketShaper boot log can be viewed from the 9.256/LOG directory by issuing a cat or more CLI
command. Boot log can also be downloaded and later interpreted using a text editor. The above
example shows a sample of the boot log. Contents of the boot log are typically arranged column
wise and contain details of the timestamps (in mmddyy format) followed by the exact time of
reboot in GMT time zone and the PacketWise version running on the appliance during the
occurrence of the reboot. The last column indicates the engineering release date for that version of
PacketWise. In the above example, note the following:
1. The presence of a plus sign (+) indicates that this is a user initiated reboot caused by issuing
the reset command via the CLI. This type of reboot can also be carried out from the
PacketShaper WUI. The appliance is running version 8.4.4g1 of the PacketWise software.
2. The presence of a SAFE+ also indicates a user initiated reboot, except that the user pressed
Control -A during the reboot to enter the safe mode. The PacketShaper is running
PacketWise version 8.5.1b1 during the occurrence of the reboot.
3. The presence of a + revert indicates a user initiated reboot and the appliance reverted to the
backup version of the PacketWise software. Note that the user reverted to the previous version
of PacketWise as mentioned in Step 1.
4. The presence of a minus (-) sign indicates a PacketShaper initiated reboot and is accompanied
by a crash log.
5. The presence of the SAFE - indicates a PacketShaper initiated reboot and that the appliance
entered safe mode after the reboot.
6. The absence of any sign towards the end indicates a forced shutdown of the PacketShaper due
to a power outage.

Note: Under rare circumstances, one might also see an asterisk (*) sign towards the end of a
line in the boot log. This indicates that the PacketWise cannot attribute a specific
reason for the reboot of the appliance. No crash logs are created for such type of
reboots.

Safe Mode
Corrupted PacketWise image
Too many reboots detected in a short period
Appliance accessible via

– Serial console
– Telnet
– FTP
6OLGH6DIHPRGH
The PacketShaper has a built-in safeguard that enables recovery from a corrupted software image.
When the appliance detects a bad image (after repeated crashes), it reboots into safe mode. Safe
mode turns shaping off and prohibits any configuration access, such as the traffic, measure,
setup shaping, or class commands. Safe mode is reported in the login banner in the
PacketShaper WUI.
Note: If you are connected to the PacketShaper via serial console, you can repeatedly press
Control+A during the boot process to enter safe mode manually. This avoids the
delay involved with the automatic safe mode process.
During safe mode, access to the PacketShaper can be via the serial console, Telnet or FTP. To
recover from this error condition:
• Revert to the last image using the image revert command.
• FTP a new PacketWise software image file to the appliance and load the new image using the
image load command.

Troubleshooting Safe Mode
6OLGH7URXEOHVKRRWLQJVDIHPRGH
If a PacketShaper has entered safe mode, a few basic steps need to be carried out to troubleshoot
the situation. Blue Coat recommends to perform a backup at the very outset even before
troubleshooting. This step allows you to reload the PacketWise configuration at a later point of
time. Possible causes for the PacketShaper entering safe mode are:
1. Configuration issues:
a. Check to make sure that important configuration files such as basic.cfg and or
config.ldi are present. If either one of these two files are either missing or
non-readable, the backup process mentioned earlier cannot be successfully
performed.
• If the basic.cfg file is missing, the config.ldi file can be backed up separately. The
basic.cfg file can always be rebuilt by contacting Blue Coat Technical Support.
• The config.ldi file cannot be rebuilt if corrupted or missing and does not support the
backing up of the configuration files. Blue Coat recommends that config.ldi file be
backed up at regular intervals to avoid potential issues during the safe mode.
b. Assuming that the configuration files are available and backed up, the following are
the options to proceed with troubleshooting configuration related issues:
• Clear current PacketWise configuration and reboot the appliance. Try reloading the
configuration by either one of these methods — using only traffic tree, entire
configuration or restoring from the backup.cmd file.
• Restore the configuration files stored in the 9.256/CFG directory by copying it from the
.SAV files. Corrupted configuration files can be identified using a .BAD extension and
can be restored using the .SAV files in the same directory. A second option to restore
configuration files is to rebuild the basic.cfg files by calling Blue Coat Technical
Support.

Note: basic.cfg files are appliance specific and cannot be ported or copied between
appliances.
2. PacketWise image issues:

a. Perform an image revert operation to regain the original configuration.
b. Verify if the PacketShaper boots to safe mode. If so, install a fresh PacketWise image
using the TFTP option.

Continuous Reboots
User experiences
– Reboots up to a maximum of 8 times
– backup.zoo loaded after eighth reboot
– Appliance crashes due to
image.zoo file corruption
Memory related issues
NMI trigger
Solution
– Use Ctrl+B command to interrupt reboot and revert to
backup image
6OLGH&RQWLQXRXVUHERRWV
PacketShaper can sometimes experience continuous reboots within a short duration of time. Such
reboots can occur as many as eight times and the backup.zoo image file is loaded into the
appliance after the eight attempt. Possible causes for such continuous reboots are:
• Corrupted image.zoo file: PacketShaper will try to locate the backup.zoo file to fix this
problem.
• Memory related issues: PacketWise cannot identify the reason behind the memory issue and
causes continuous reboots.
• Non Maskable Interrupt (NMI) trigger: PacketShaper cannot isolate the reason behind the
NMI trigger and causes the reboots.
Pressing Control-B during the reboot causes the PacketShaper to interrupt the reboot process
and reverts it to the backup image. You can use the other interrupt commands — Control-A and
Control -Y depending on your requirements.

Failed–to–Load
6OLGH)DLOHGWRORDGVLWXDWLRQ
PacketWise software can sometimes fail-to-load when requests to execute the image.zoo or
backup.zoo files are given. Under such circumstances, the only feasible option to load the
PacketWise software is through the boot monitor mode. Some of the possible causes for the above
situation are:
1. PacketWise software: This situation could arise if the PacketWise software is not found or is
found corrupted in 9.256/BIN or /SAV directories. PacketShaper enters boot monitor mode as a
result of this. To recover from this state, use the TFTP option to load a new PacketWise image
to the appliance.
2. Configuration: This situation can arise if any of the following occur:
a. PacketShaper flash disk is corrupted. Perform flash disk recovery while in boot
monitor mode to rectify the situation.
b. If all of the configuration files in 9.256/CFG directory are missing, the appliance is
bound to enter boot monitor mode. Use the TFTP option to restore files to the
9.256/CFG directory. If there is no back up file available during the TFTP process,
escalate the case to Blue Coat Technical Support asking to rebuild the basic.cfg file.
3. Hardware: This can be attributed to the following factors:
a. RAM: If a PacketShaper does not meet the minimum RAM requirements for a specific
PacketWise version, there is a very high possibility that PacketWise will fail to load
and enters the boot monitor mode. Blue Coat recommends that you meet hardware
requirements as specified in the product Release Notes.
b. Corrupted flash disk: Under normal circumstances, PacketWise software loads from
the PacketShaper flash disk. If the flash disk is corrupted, PacketWise software will
fail-to-load and causes the PacketShaper to enter boot monitor mode. Flash disk
recovery can be performed under boot monitor mode and is dependent on the
PacketWise version installed on the appliance.

Plug-ins
User experiences
– Info page and plug-ins page error messages in WUI
– banner show error message in CLI
– Appliance reboot in rare circumstances
Solution
– Delete and reinstall all appropriate plug-ins
6OLGH,VVXHVZLWKSOXJLQV
Plug-ins are downloadable files that extend the functionality of an existing software release and
are stored in the 9.256/PLG directory on the PacketShaper. Most plug-in related problems arise
during the PacketWise upgrade process. In cases where a wrong or incompatible plug-in is
installed on the appliance, the following can be observed:
• Error messages on the PacketShaper WUI, indicating obsolete plug-ins for that version of
PacketWise software.
• Error messages that can be viewed by issuing the banner show CLI command.
• PacketShaper reboots due to wrong plug-ins being installed.
During a PacketWise software upgrade, Blue Coat recommends deleting any older versions of the
plug-ins and installing current and appropriate versions to match the PacketWise software.
Important: When Blue Coat creates a new plug-in, the file will be posted on the BlueTouch™
Online download page. You can view and download plug-in files compatible
with your software version through the PacketWise browser interface, by using
the Blue Touch Online download site at https://bto.bluecoat.com/download or
with the update command in the CLI.

Chapter 13:Software Image Case Study
In this example, the problem report is focused on an application running on the customer’s
network, but the cause of the problem turns out to be something different. An important skill to
develop in troubleshooting is learning to focus on the big picture; this helps you distinguish
between symptoms and causes.
Also, many problem reports that initially appear complicated actually can have simple causes and
simple solutions. Always search for simple resolutions before becoming involved in complex
analysis and solutions that would be unnecessarily disruptive to the customer.
• How to identify a PacketShaper that has entered safe mode.
• How the PacketShaper boot log reports problems with software images.
• How to identify and resolve corrupted PacketShaper image and configuration files.

In this case study, a PacketShaper has been deployed in a customer network that has two remote
locations. On Monday, when users at these remote offices arrived at work after the weekend, they
complained that checking their e-mail was much slower than normal.
The complaint comes only from users at the remote offices, not those at headquarters. From the
LAN, the Microsoft Exchange e-mail server works fine.
This suggests that there probably is not an issue with the Exchange server. Instead, the problem
likely is in one of these areas:
• PacketShaper configuration;
• PacketShaper hardware;
• PacketShaper software; or
• the WAN link.
To analyze this issue, start by determining the relevant PacketShaper configuration. In this case,
the customer says that Exchange traffic has been guaranteed to 2Mbps inbound, with no burstable
option.
Also, the customer says that only secure access (HTTPS and SSH) is allowed to the PacketShaper
management interfaces; the other access methods have been disabled.

Chapter 13: Software Image Case Study
PacketShaper Inaccessible
6OLGH3DFNHW6KDSHULQDFFHVVLEOH
The next step in diagnosis is to try accessing the PacketShaper.

1. Because, as learned in the previous step, only secure access is available, use the URL
https://psIPaddr, where psIPaddr is the IP address of the PacketShaper. In this case, however
the connection attempt times out, and the PacketShaper cannot be accessed even through
HTTPS.
2. Next, ping the PacketShaper to determine whether it is visible on the network. Here, the
PacketShaper responds to pings, which means that it is operating and is responding to
network requests. This suggests that something is wrong with the configuration of the
PacketShaper.
3. To learn more about the status of this PacketShaper, try to access it via HTTP even though only
secure access is allowed. If the PacketShaper is performing as expected, the connection will
time out. However, in this case, the HTTP request is redirected to the page corrupt.htm, and
the PacketShaper responds that it has reverted to safe mode.
Recall that a PacketShaper enters safe mode after detecting a bad software image following
repeated crashes, or when configuration files are corrupted or missing. In safe mode, shaping is
disabled, and configuration commands such as traffic, measure, setup shaping, and
class cannot be performed.
To resolve this problem, the cause of entering safe mode must be identified and corrected.

Analyze Boot Log
6OLGH$QDO\]HERRWORJ
Because safe mode was entered, the next place to look is the boot log.
Analysis of the boot log for this PacketShaper reveals the following:
1. The PacketShaper crashed several times in a short period — at 02:10, 02:12, 02:33, 03:48, 03:58,
and 04:00 local time. The minus sign at the end of each line indicates a crash.
2. Finally, after the crash at 04:00, the PacketShaper reached its threshold for crashes and entered
safe mode, indicated by -Corrupt (or, in some versions, SAFE-) at the end of the line.
Now that one or more corrupted files on the PacketShaper are suspected, the next step is to
identify those files.

Chapter 13: Software Image Case Study
Identify Corrupt Files
6OLGH,GHQWLI\FRUUXSWILOHV
To search for corrupted files on the PacketShaper, look in directories on the flash disk where such
files are likely to be found.
1. Directories 9.256/bin and 9.256/cfg might contain files with the extension .BAD. These could be
image files or configuration files. In this case, the file BACKUP.BAD exists and most likely
indicates a corrupted software image. This file was created when the PacketShaper repeatedly
tried to recover from a crash, gave up, reverted to the previous version, and renamed the
image BACKUP.ZOO to BACKUP.BAD.
2. This PacketShaper currently is running software version 8.2.3. This happened after, as shown
in the boot log, repeated crashes occurred while running version 8.4.3 and the PacketShaper
reverted to a backup image.
If other corrupted files are detected, the PacketShaper changes their file extensions to .BAD. For
example, if BASIC.CFG became corrupted, it would be renamed BASIC.BAD.
Because the only corrupted file found here is the backup software image, it is reasonable to
conclude that the corrupted software image is the cause of the PacketShaper reboots.


Remove corrupted image BACKUP.BAD
Upload latest software image to PacketShaper
Load latest software image on PacketShaper
Reboot PacketShaper
Verify proper operation
Based on the previous analysis, the following steps should correct the customer’s issue:
• Access the PacketShaper through the serial console or Telnet, and remove the corrupted
software image BACKUP.BAD. (If other corrupted files had been detected, those also would be
removed.)
• Open an FTP session to the PacketShaper, and upload the latest software image.
• On the PacketShaper, use the image load command to load the new image.
• Reboot the PacketShaper.
• Verify that users at remote locations are able to access e-mail with proper response times.
The customer reported that this fixed the problem.

Chapter 14:Access and Performance Issues
This chapter discusses Blue Coat® PacketShaper® performance and access and how to diagnose
issues related to both. Performance generally is related to three major factors with the
PacketShaper: high CPU utilization, high memory utilization and system limits. Rarely will you
come across issues truly related to high network utilization; such issues usually can be attributed
to high CPU utilization, high memory utilization, or both.
In general, the leading causes of performance issues on the PacketShaper involve the following:
• Configuration, different types of classes, partitions and policies currently on the appliance.
Traffic class tree can be a leading cause for performance issues.
• Traffic mix, too many new flows per minute on the PacketShaper can have an impact on
classification and in turn performance.
• Adaptive response (AR), configuring too many AR agents can cause high utilization of CPU.
Access to the PacketShaper can sometimes be delayed via the Web user Interface (WUI) and can
result in system time outs. High CPU utilization can sometimes cause such access issues. After
studying this chapter, you will understand:
• How to resolve a slow and unresponsive PacketShaper WUI occurrence.
• How to fix an unreachable PacketShaper.
• What are the common causes of performance issues on the PacketShaper .
• How to interpret important diagnostic information pertaining to performance issues.
• How to troubleshoot performance issues due to CPU utilization, memory and system limits.

Unresponsive WUI
6OLGH8QUHVSRQVLYH:8,
The common symptoms experienced when PacketShaper access issues occurs usually has to do
with slow access to resources, time outs, network errors, and inability to connect. Traffic load is
not the only influence on the speed of the PacketWise WUI. The number of traffic classes and the
way the traffic tree is configured influences the performance of the browser interface. In addition,
PacketWise uses Java and the files needed to monitor and manage can be large.
In an overload situation, high CPU utilization and high memory pressure are also observed. When
memory resources become scarce, the PacketShaper tries to compensate by delaying the
processing of new flows or delaying memory allocation for some existing flows until other flows
have a chance to complete and free their associated resources. Possible scenarios when a
PacketShaper exhibits an unresponsive WUI are:
• Graphs time out even before appearing on the screen. This can be because of high latency
conditions on the network. Increasing the graph time out variable on the PacketShaper,
available both via the WUI and CLI, helps fixing this issue. Modifying the sys set
graphTimeoutSeconds variable from a default value of 60 seconds to a desired number
enables the graphs to display quickly.
• Large traffic class tree takes time to display. Traffic tree and its configuration also influences
the performance of the PacketShaper WUI. Traffic classification is always in the top-to-bottom
fashion and can take a lot of time to locate a specific class in such a large tree. Pruning the
unwanted or less used traffic classes can be help manage the WUI performance. Redesigning
the traffic tree after pruning might prove beneficial in many cases. Adjusting the system
variable values for the DiscoveredPorts class can help prevent the ballooning of the traffic class
tree.
• Access to the PacketShaper WUI is continuously denied. Browser settings for cache or pop-up
windows might be the cause of such behavior. Private data such as saved passwords might
not be migrated after a password update and can cause problems with loading of the WUI.
Clear your browser cache and attempt to access the WUI. Modifying browser settings to allow
pop-up windows can help display some WUI features quickly.

Chapter 14: Access and Performance Issues
• CPU utilization statistics reveal a low value (in%) for system idleness. CPU utilization and
system idleness are inversely related and a low idleness value indicates a highly utilized CPU.
One of the possible symptoms of high CPU utilization is an unresponsive WUI.
Troubleshooting such phenomenon can be handled in many ways and is discussed in detail in
Slide 14-6.
• Access to the PacketShaper is restricted. When maximum limits for the system settings on the
PacketShaper are reached, user access is denied. Any error in the network and security
settings can also cause access issues. Ensure proper IP address is provided in the network and
security settings to avoid such issues. Troubleshooting an unreachable PacketShaper is
covered in detail in Slide 14-2.
• Lastly, accessing the PacketShaper from a remote branch office can lead to local host traffic
generation from an external LAN segment. Local host traffic needs to compete with existing
priority 6 traffic on the PacketShaper and assigns a lower priority for the local host traffic
class. This in turn, can cause delayed response to the WUI, when accessed from a remote
location. Allocating a small partition for the local host traffic can sometimes eliminate the
above experience.

Unreachable PacketShaper
6OLGH8QUHDFKDEOH3DFNHW6KDSHU
PacketShaper can sometimes become unreachable — both via the WUI and the CLI, for numerous
reasons. Recovering access to the appliance can be carried out in many ways. In the above
flowchart:
1. Access to the PacketShaper can be limited in many ways by security settings and securing the
interfaces. Disabling the below mentioned options might help resolve any access problems.
a. Security settings on the Setup tab on the PacketShaper WUI might have been
disabled. Access to the appliance — via HTTPS, SSH, FTP, HTTP, Telnet, SNMP and
TCP Echo options, is enabled by default. Depending on specific deployment
requirements, all non-secure access to the appliance might be disabled. This makes
the appliance accessible only via HTTPS and SSH methods. If the customer tries
accessing the appliance via HTTP, the WUI will not display per settings in the Setup
tab.
b. Enabling or disabling access to the PacketShaper over the inside and or outside
network interfaces can sometimes make the appliance unreachable. Access can be:
• Unsecure, which enables unlimited access over specified interface.
• Secure, which access blocks all access from the specified interface.
• List, which enables access to a list of eight IP addresses.
Important: When both the inside and outside interfaces are set to secure, access to the
PacketShaper is available only via the serial console. The browser interface
access is disabled.
2. Password issues such as typing the wrong password or losing an existing password can also
cause access issues with the PacketShaper.
❐ If the appliance has been configured using RADIUS or TACAS+ authentication, and
circumstances when RADIUS server access is limited, access to the PacketShaper is only
via the local host password.

❐ If you forget the touch password, you can use the password recovery method to access
the PacketShaper and then reset the password. To ensure that security is not
compromised, this feature works only when you are directly connected to the
PacketShaper via serial console. Typing touchpwd= in the serial console within thirty
seconds, helps you reset the password.
3. Gateway issues can be attributed to wrong IP settings on the network and can cause
PacketShaper access issues. Verifying the IP setting can help resolve the above issue.

Performance Issues: Symptoms
6OLGH0DMRUV\PSWRPVRISHUIRUPDQFHLVVXHV
The above slide describes some of the common causes and associated symptoms for performance
issues on the PacketShaper. CPU, memory and PacketShaper system limitations are the three
major causes of performance issues on the appliance. Some of the widely reported symptoms that
can be attributed to a performance issue on the PacketShaper are:
• General delay in accessing management traffic, which is heavily dependent on policies
applied to the localhost policies. Large traffic trees can also cause extensive load on the
PacketShaper CPU and cause potential performance problems.
• Too many packet retransmissions occur when router queues deepen and cause dropped
packets.
• Very high level of CPU utilization viewed from the sys health output.
• Misclassification reported due to the delay between a host detection and the computation of
its speed capabilities on the network.
The following slides describe the different options available to analyze and resolve performance
based issues on the PacketShaper.

System Health
6OLGH6\VWHPKHDOWKDQGFRQILJXUDWLRQOLPLWV
Understanding the PacketShaper’s system health parameters and configuration limits can be of
immense help to interpret performance based issues. PacketWise has several types of capacity
limits that can impact the ability to expand your traffic management solution. For example, if you
run out of traffic classes or matching rules, auto-discovery or adding any new types of traffic can
be a problem. It's a good idea to check PacketShaper’s available capacities every few months or
whenever you have made lots of configuration changes such as changes to your traffic class tree,
your policies and partitions, the amount of traffic your network supports, the number of units
with Xpress enabled, and so on.
Refer to the PacketGuide documentation for an estimate of system boundaries based on each
PacketShaper model.
You can also check the real-time system health and configuration limits with the net nic and
sys limits CLI commands. In the above slide:
1. The sys limits command output lists the maximum number of objects allowed, currently
used, and remaining for each object such as classes, partitions, TCP flows and policies. The
maximum number of supported TCP flows for this model of PacketShaper is 5120, while the
current usage is 4. This implies that the appliance can handle a remaining of 5116 flows
through it. Objects such as Other IP flows and Legacy flows also serve as indicators for different
configuration limits on the appliance.
2. The traffic active command is another important tool for determining how close the
PacketShaper is to reaching its capacity. The output displays the current, maximum, and
possible number of sessions for TCP, UDP, and Legacy traffic types. Flows are considered
current if they have had a packet within the last minute. The value listed for maximum flow is
the maximum number that has been displayed when the traffic active command has been
executed.The possible flows represent the PacketShaper’s maximum number of concurrent
flows allowed on the appliance.

Note: The highlighted values for different objects shown under the Total column in Step 1 is
the same as the values for Flows (Possible) row in Step 2. You can use either of the
above commands to verify the configuration limits on the PacketShaper.
3. The net nic command lets you view the network statistics such as packets transmitted and
discarded from the PacketShaper. Use this command to analyze performance related issues
when there are strong indications of high CPU utilization on the PacketShaper. Some of the
important counters from this output are:
a. rxQueued, indicates the number of packets in the queue that are waiting to be
processed.
b. RxDrops, indicates the number of packets dropped due to the buffer reaching its
limits.
c. RxErrors, indicates the different errors mentioned in counter numbers 15,17,19 and 21.
d. RxLateDrops, indicates the number of packets dropped because of NIC overload.
If you find that the PacketShaper is running out of capacity, there are a variety of remedies,
depending on which item and which hardware model you have. You can:
• Upgrade to a larger hardware model. For some models, such as the PacketShaper 3500, you
can upgrade by simply by purchasing a new software key that enables more classes.
• Consider trimming your traffic tree. Often, the traffic class tree gets full because
auto-discovery creates a lot of classes for traffic that passed only once. You can go through the
class tree and delete these unnecessary classes to free space.

Analyzing CPU Utilization
6OLGH&38XWLOL]DWLRQ
PacketWise’s adaptive response feature automatically monitors for conditions of interest, detects
potential problems, and optionally notifies administrators and or takes corrective actions if a
problem is detected. You can keep tabs on many aspects of PacketWise health at all times without
frequent investigations. If real-time monitoring of the PacketShaper health is needed,use the sys
health CLI command. In the above slide:
1. The amount of CPU utilized is represented in percentage values. The output contains values
for Current% Idle, Average% Idle and Minimum% Idle parameters. Higher values in the current
and average parameters indicate a lower CPU utilization. The output in the above slide shows
that only 2% of the PacketShaper CPU is currently being used.
2. Memory buffers are strong indicators of the PacketShaper CPU status. High values for these
buffers indicate CPU stress and can mean possible delay in packet transmission. This in turn
can lead to performance related issues on the appliance. In the above slide, the default value of
zero on these buffers indicate correct CPU utilization on the PacketShaper.

CPU Issues
6OLGH7URXEOHVKRRWLQJ&38LVVXHV
CPU utilization figures are valuable indicators of performance related issues on the PacketShaper.
Such diagnostic information is recorded every fifteen minutes on the PacketShaper and is used by
Blue Coat Technical Support while dealing with such issues. Trend analysis on the diagnostic
information can reveal CPU utilization patterns on the PacketShaper. In the above flowchart:
1. Does analyzing diagnostic information reveal memory or CPU issues on the PacketShaper ?
Memory failure indicators on these files need to be handled separately and are discussed in
Slide 14-7. If not, verify if the CPU utilization is high on the appliance.
a. If so, fault isolating some of the features on the appliance can sometimes help rectify
this issue. Adaptive response (AR) is one such feature which can cause potential
performance issues on the PacketShaper.
b. If not, proceed to investigate other causes that lead to performance issues.
2. Is AR is a cause for performance issues? If so, disabling the AR feature or troubleshooting the
specific AR agent can help solve this issue. If not, analyze to see if Xpress compression is a
possible cause for performance deterioration on the PacketShaper.
a. If so, enabling or disabling the Xpress feature and monitoring CPU utilization can
sometimes resolve the issue.
b. If not, proceed to verify if the Shaping feature influences CPU utilization statistics.
3. Is Shaping causing extensive use of CPU resources on the PacketShaper? Modify Shaping
options to either on,off, bypass or pass thru conditions and observing CPU utilization is the
next recommended step. Observe the occurrence of packet drop during the process.

a. Are packets are dropped when Shaping is turned off or in the passthru mode? If so,
the PacketShaper has reached is configuration limits. Use the sys limits command
to verify configuration limits and upgrade the hardware to resolve this issue.
b. If not, verify if bandwidth utilization is high and proceed to enable load shedding on
the PacketShaper. Load shedding is a built-in technique in the PacketShaper to
intelligently manage packet drop without impacting performance.
c. Is performance still impacted? If so, enable the system profiler option on the
PacketShaper and escalate to Blue Coat Technical Support for further resolution.
d. If not, performance issues on the PacketShaper have been successfully resolved using
one of the options mentioned in the above steps.

Memory Issues
6OLGHWURXEOHVKRRWLQJPHPRU\LVVXHV
Analyzing diagnostic information to identify performance issues on the PacketShaper is a key step
in identifying if CPU utilization or memory failures are the cause. Memory allocation failures
occur when the PacketShaper encounters numerous packets within a short duration of time and
the cannot allocate necessary memory resources to process these packets. The sys kmem pkt
type and sys kmem com type CLI commands provide an insight into the causes of the memory
allocation failures. In the above flowchart:
1. Are the diagnostic files revealing affected memory counters on thePacketShaper? If so, how
many such counters are impacted? If many counters are affected, reboot the appliance and
verify if the problem persists.
a. If so, free up memory resources on the PacketShaper by disabling either one of these
options:
• Response time measurement(RTM) feature
• Traffic history option
• SSL common name criterion classification
Verify if the problem persists and proceed to escalate to Blue Coat Technical Support for
further analysis.
b. If not, verify that the problem is solved and proceed to close the support request.
2. Does the diagnostic file reveal only one memory counter failure? If so, reboot the appliance
and enable the blurt option for that specific memory counter on the PacketShaper. Collect the
required statistics and escalate to Blue Coat Technical Support for further analysis.

Chapter 15:Performance Case Study
PacketShaper® customer, demonstrating one of the common performance-related issues when a
PacketShaper is dealing with many flows and high bandwidth.
In this example, a PacketShaper that is managing hundreds of users is experiencing problems with
traffic classification. Users are reporting slow Internet access, and the customer’s network
engineer is seeing PacketShaper statistics that suggest improper classification.
Even though the cause of this problem might be difficult to isolate, effective troubleshooting still
requires that a support engineer analyze the basic configuration and recent history of the
PacketShaper to determine whether the customer might not have noticed any fundamental issues.
This means that even when a solution is relatively quick to implement, the analysis can take a
while to complete.
• How to analyze class hits to determine whether the PacketShaper is correctly classifying
traffic.
• How to identify PacketShaper memory allocation failures and their possible causes.
• Some configuration changes that can be made to a PacketShaper that needs to process a large
volume of traffic.

In this case study, a PacketShaper has been deployed at a university with more than 500 users.
Some of these users are complaining that the Internet is very slow.
Some of these users are engaged in peer-to-peer file sharing, and most of the complaints come
specifically from students who are using BitTorrent®.
In an effort to identify the problem, the network engineer at the university has used the traffic
flow command on the PacketShaper to learn more about the traffic being generated by the
students who are complaining. The network engineer has discovered that some of the BitTorrent
traffic is being wrongly classified into a user-created class called Corbett, which is a class based on
IP addresses to identify a particular group of campus users.
What could cause this situation? Possibilities include:
• Issues with how partitions might have been configured on the PacketShaper.
• The customer’s WAN link might be full.
• The PacketShaper might not otherwise be performing as expected.
To analyze this problem, examine the ts.zip archive provided by the customer and the traffic flows
that the network engineer has observed.

Chapter 15: Performance Case Study
Initial Analysis
6OLGH,QLWLDODQDO\VLV
First, examine the usual information from the PacketShaper to get a picture of the basic
configuration and status.
As seen in the output shown above:
1. The version verbose command shows that this is a PacketShaper 10000 running software
version 8.3.3.
2. A valid license key is installed, and this PacketShaper is licensed for a WAN link size of up to
622Mbps.
3. All installed plug-ins are up to date.
4. A site router has been configured for this PacketShaper. This means that the PacketShaper
manages only traffic that passes through the specified gateway.
5. The banner show command reports that interfaces are down. However, given the
configuration shown on the previous page, this is normal because this PacketShaper is using
only LAN connections.
These observations suggest that the basic configuration of this PacketShaper is correct.

Analyze Class Hits
6OLGH$QDO\]HFODVVKLWV
Because the customer observed misclassification of traffic, the network engineer at the customer
site collected the above PacketShaper reports on traffic flow from two hosts that were generating
BitTorrent traffic on the network. The -A option of the traffic flow command takes the next
argument as the IP address on which to report.
As shown in the above output:
1. BitTorrent traffic from IP address 10.82.99.60 is hitting the correct classes both inbound and
outbound, and the traffic service is correctly identified as BT-Data. Also, non-BitTorrent traffic
also appears to be correctly classified.
2. However, traffic from IP address 10.82.90.71 is hitting the Corbett classes both inbound and
outbound, even though the traffic service is correctly identified as BT-Data. (For readability,
not all of the 153 current TCP flows are shown here.)
The network engineer also reported that this behavior was not consistent. BitTorrent traffic from
one host that was being correctly classified at one point would later be incorrectly classified as
Corbett, and vice versa.
This shows that similar traffic is hitting different classes and no pattern appears to be obvious, but
the reasons for this are not yet known.

Analyze TS Archive
6OLGH$QDO\]H76DUFKLYH
Next, analyze the contents of the ts.zip archive to search for any possible causes of the traffic
misclassification.
As shown in the above excerpts from the archive:
1. The boot log shows no recent crashes; no minus signs are next to any of the entries in the log.
2. Host-sidedness is not an issue; all of the 129.82 network segments are detected as I. (For
readability, many of the addresses are not shown here.) This indicates that all of the hosts are
learned correctly.
3. The number of flows is somewhat high but is not close to the limit for this PacketShaper; out
of 300,000 maximum TCP flows, only slightly more than 16,000 have been detected at any one
time.
4. Link utilization after shaping is 164Mbps, which is quite high but not at the WAN link size of
205Mbps that was seen earlier.
These parts of the ts.zip archive do not point to any cause for the misclassification of traffic.

Check Rules and Memory
6OLGH&KHFNUXOHVDQGPHPRU\
With no cause yet apparent, the analysis becomes deeper. Next, try to determine whether
matching rules or memory allocation might be responsible.
1. The class show command displays traffic class information for a specific class or the entire
traffic tree. Here, the class Inbound/FileSharing/BitTorrent appears to be good: The rules are
standard, no policy is defined, and there is a partition for the parent class, Inbound/FileSharing.
2. The sys kmem pkt type and sys kmem com type commands display statistics about
PacketShaper memory allocation. Here, we see a problem: The number of failed memory
allocations for type appness utils is extremely high. A failure on any one of the appness types
shows that a memory allocation failure is happening in the classification engine.
Memory allocation failures occur when the PacketShaper encounters numerous packets within a
short time and cannot allocate the necessary memory resources to process these packets. This,
coupled with the earlier discovery of high link utilization, leads to a likely conclusion that the
memory allocation failures are causing the traffic to be misclassified.


Cause
– Large amounts of inbound traffic
– Memory allocation failures
Resolution
– Reboot PacketShaper
If errors persist, free memory by disabling features
– Crumbs
sy s set crumbsPerClass Count 0
– IP classification accelerator cache

sy s set hosttspeccache 0
In summary, this PacketShaper is misclassifying traffic because a very large amount of traffic is
flowing through the appliance, leading to memory allocation failures. This is a normal
consequence of very large inbound traffic and is not an error in the PacketShaper.
The only way to clear these failures is to reboot the PacketShaper. Doing so usually corrects the
problems that led to traffic misclassification.
However, if memory allocation failures continue to happen after a reboot, the customer might
need to free some memory by disabling unused or nonessential features. The following features
often are candidates to be disabled:
• Crumbs: These are used with the traffic history recent command, which lists recent
flows for a specified traffic class can be useful for analyzing the type of traffic that is falling
into a Default class. To disable crumbs, enter this CLI command:
sys set crumbsPerClassCount 0
• IP classification accelerator cache: Controls IP address caching in the PacketShaper. When a
PacketShaper is managing a large number of hosts, as is happening in this case study,
disabling this cache can improve PacketShaper performance. To disable this cache, enter this
CLI command:
sys set hosttspeccache 0
These recommendations were communicated to the customer, who implemented them and
resolved their misclassification issues.


Chapter 16:PacketShaper Traffic Flows
The way PacketWise organizes traffic dictates how it can analyze and control that traffic. Traffic
tree is the tool that organizes your traffic, and its configuration is a crucial choice in determining
the features you have available. For example, if your traffic tree doesn’t distinguish SAP from
Oracle, then you can’t measure distinct response times for each. Or, if your tree doesn’t distinguish
traffic to your Paris office from traffic to your Oslo office, then you won’t be able to compare traffic
volumes or give prescribed amounts of bandwidth to each branch.
Application discovery and the creation of a corresponding traffic tree are prerequisites for
understanding application behavior and controlling its performance. PacketWise differentiates
one application from another by evaluating characteristics in traffic flows and organizing them
into classes. Each traffic class contains at least one matching rule, a set of characteristics that
identifies a specific traffic type.While many applications or devices can identify traffic on
well-known ports, PacketWise is application aware and goes beyond classification by port
number. Sometimes customers experience issues with traffic classification that can be attributed to
many different issues. This chapter discusses the possible causes for issues with traffic
classification in the PacketShaper. After studying this chapter, you will understand:
• Common issues with traffic classification in the PacketShaper
• How to identify traffic misclassification and steps to rectify the same
• How host analysis can help provide insight into host and flow activity on your network.
• What is host sidedness and how it can help resolve some of the traffic flow issues between
inside and outside hosts
• Different causes for policy related issues and how to troubleshoot them in the PacketShaper
• How to identify and troubleshoot partition issues
• Identify and control aggressive traffic and infected users in your network.

Common Issues
Traffic not hitting right class
Traffic hitting default bucket
Misclassification
Very few classes discovered
Obsolete plug-ins
6OLGH&RPPRQLVVXHV
The above slide discusses some of the widely reported issues relating to traffic classification on the
PacketShaper. One of the common classification issue that customers report is that their network
traffic does not hit the right class in the traffic tree. A good example would be a situation where
traffic needs to be classified based on IP addresses and the PacketShaper has it classified under a
totally different class. Traffic tree order in the PacketShaper could be a cause for such behavior and
re-ordering the traffic tree can sometimes help resolve this issue.
Customers have also widely reported network traffic hitting the default class in the traffic tree.
There might be many reasons for this behavior, one such reason being unknown traffic flowing
through the appliance and has traffic discovery turned off. The PacketShaper then classifies such
traffic under the default traffic bucket.
Another case of traffic classification issue is that of misclassification. PacketShaper appliances
have been reported to classify FTP traffic under a completely different layer 7 traffic class when it
needs to be classified under an existing class for FTP traffic. In other instances, the traffic tree
displays very few classes as discovered despite traffic discovery being turned on. Such behavior is
commonly observed when the PacketShaper is in a web proxy environment where traffic is not
completely sent through the PacketShaper.
Lastly, every version of the PacketWise software comes with the latest plug-ins needed to
effectively benefit from the capabilities of the classification engine in that version. Failing to
upgrade or install the required plug-ins to match the PacketWise version can also sometimes cause
traffic classification issues.

Chapter 16: PacketShaper Traffic Flows
Misclassification
6OLGH0LVFODVVLILFDWLRQLVVXHV
PacketShaper appliances have been reported to misclassify traffic in different network

environments. Customer A notices that all of his Citrix traffic, both applications and print traffic,
gets classified under Citrix-print traffic class. Customer B recently upgraded his PacketWise
software to the latest version and observed that SSH traffic is being classified under SkypeData
traffic class. Such instances of misclassification can be resolved by monitoring the class hits for that
particular traffic class. Class hits indicate the number of traffic flows match a class and occur only
at the beginning of a flow or session. Class hits can be analyzed by clicking on Monitor > Class Hits
on the PacketShaper WUI. A best practice measure would be to clear the existing Class Hits
statistics on the Monitor page to capture the most recent network flow data. In the above flowchart:
1. Observe if the concerned traffic hits the intended class in the traffic tree. If an increment in the
class hits data is observed, the traffic is properly classified. Proceed to verify if policy hits
occur for this traffic class. If not, the traffic is being misclassified by the PacketShaper. Perform
host analysis on the traffic class to identify why and or where the misclassification occurs.
Slide 16-3 discusses the host analysis process in detail.
2. If class hits data is incremental along with the policy hit statistics, verify if the policy hits are
for the correct class hits. Sometimes the default class displays enormous policy hits because of
no policy being defined for the concerned traffic class. This can possibly be a policy related
issue in the PacketShaper. Troubleshooting policy related issues can help resolve this issue
and is being discussed in detail in Slide 16-6 later in the chapter.

Host Analysis
6OLGH+RVWDQDO\VLV
The host analysis reporting tool allows you to gain insight into host and flow activity on your
network. With this tool, you can display information formerly available only in the command-line
interface using the hostdb info and traffic flow commands. From the PacketShaper WUI,
click Report > Tools > Host Analysis. Specifically, you can:
• List bandwidth utilization and flow information for hosts on the network. This list can consist
of inside hosts, outside hosts, active hosts, or all hosts, and can be sorted by bandwidth
utilization, new connections, failed connections, or host IP address.
• Find flows for a particular IP address, port number, or protocol. For each flow, the list
provides the IP addresses of the conversation pair, the class name into which PacketShaper
classified the flow, and the protocol of the flow.
• Drill-down to find out detailed flow information for a suspicious host (for example, one that is
using excessive bandwidth or creating an inordinate number of connections or failed
connections).
Using the hostdb info and traffic flow CLI commands, you can carry out detailed host
analysis for resolving traffic misclassification issues on the PacketShaper.
• hostdb info command displays the host IP address, average and current connections,
current guaranteed and excess bandwidth, and throughput information. The host database is
a record of all hosts that have active connections through the unit. Once a host closes its
connection, the host will be purged from the database. In addition, the appliance will clear
host entries if they aren't active for approximately ten minutes. Thus, the host database is a
real-time list of hosts.
• traffic flow command displays summary information about some or all currently active
TCP connections and/or UDP sessions.
In the above flowchart:

1. After the host analysis has been carried out, verify if the hosts have been properly identified
on thePacketShaper. Compare the host-sidedness with the display output from the host
analysis tables. If the hosts have been properly identified, proceed to the next step. If not,
resolve the host sidedness issue as described in Slide 16-4.
2. Verify if the inbound and outbound traffic classes match with the hosts identified in the
previous step. If there traffic classes do not match, then there is a clear case of traffic
misclassification that occurs in the PacketShaper. Try to carry out either of the following
measures to resolve the Misclassification issue.
a. Check the configuration to ensure the existence of correct matching rules in the traffic
class.
b. Update PacketWise software to support the recommended plug-ins for that software
release. This will help in properly classifying the traffic flowing through the
appliance.

Host Sidedness
6OLGH+RVWVLGHGQHVV
The above slide describes the necessary steps to resolve the host sidedness issue in the
PacketShaper. Using the hostdb show CLI command displays the host IP address, estimated
access speed, number of speed changes, the number of TCP and UDP flows that a specified host
has processed, the amount of time the host has been idle, the status of the match rule cache, and
compression status. In the above flowchart:
• Check if the inside and outside ports of the PacketShaper are connected properly to match the
network topology requirements. If not, reconnect the network cables to match the correct
ports. Use the hostdb side reset all CLI command to clear the side settings for a particular host
or all hosts.
• Check if the host parameters have been identified on the correct side. Wrong routing
configurations can also lead to the host being identified on the wrong side of the network.
• Using the serial console access, manually assign the hosts to the correct ports on the appliance.
By securing the ports in the process, the administrator prevents the PacketShaper from
updating the host database table.
• Flush or clear the host database table by issuing the host side reset all CLI
command.The next time PacketWise sees a flow from that address it will again try to figure
out whether the host is inside or outside. This might be necessary if a particular host is seen on
the wrong side — you can add the host to the proper side list (inside or outside) and then reset
the host so that PacketWise will rediscover the host and place it on the correct side. Create a
new host list to properly map the hosts on the appliance.
• Assign appropriate input and output ports to match the host list details. Unsecure the ports to
enable management traffic. It is recommended that you save the above configuration steps as
a startup. cmd file and use it to have a regular host sidedness table each time you reboot
the PacketShaper.

Causes for Policy Issues
License key expiration

Configuration issues
Safe mode
Misclassification
Inherited policy
WUI cosmetic errors
6OLGH3ROLF\LVVXHV
Policy related issues on the PacketShaper can be due to numerous factors, some of which include:
• License key expiration can lead to policy related issues. Use the set show, set key and ver
ver CLI commands to check the license key status on the appliance.
• Configuration settings such as shaping and site router fields also play an important role in
identifying policy related issues. Ensure that shaping option is turned on or enabled on the
appliance. If not, traffic classification will occur but policy will not be implemented with an
exception for ignore policies. All other types of policies — rate, priority, never admit and discard
need shaping to be enabled for effective implementation.
When you set the site router to none, the appliance manages all traffic passing through it,
regardless of whether the traffic is going to or from the site router. Most customers set the site
router to none; this is the recommended setting. When you set a site router IP address, the
appliance only monitors/manages Ethernet packets going to and from this router. All other
Ethernet packets, including multicast, are ignored.
• Policy enforcements do not happen when the PacketShaper operates under safe mode. Safe
mode turns shaping off and prohibits any configuration access, such as the traffic,
measure, setup shaping, or class commands. Safe mode is reported in the login banner.
• Misclassification is another cause for policy related issues. Slide 16-2 discusses this in detail.
• A traffic class inherits another class' policy if it has no policy of its own. For each passing
traffic flow, PacketWise traverses the traffic tree looking for a matching traffic class. When
found, PacketWise adjusts the metrics (such as class hits) for the class and applies any
associated partition and policies. If the class has no associated policy, PacketWise continues
searching until another matching class is found that does have a policy. The traffic flow
inherits the second class' policy. But the second class gets no performance metrics recorded
other than an additional policy hit.

PacketWise searches classes in a particular order as it looks for a qualifying inheritable class. It
searches down the traffic tree, first examining the original class’ successive siblings, then the
parent’s siblings, then the grandparent’s siblings, and so on, until a qualifying class is found.
• Rarely the PacketShaper WUI displays some cosmetic errors that differ from the
corresponding CLI output. In such cases, execute the me dump CLI command on policy hits
for the specific traffic class. Rebooting the appliance after this restores the correct policy to the
appropriate traffic class.

Troubleshooting Policy Issues
6OLGH7URXEOHVKRRWLQJSROLF\LVVXHV
The above slide describes the steps to troubleshoot various policy related issues on the
PacketShaper. In the above flowchart:
• The very first step in resolving policy related issue is to verify the status of traffic shaping on
the appliance. Shaping can be set to the following values — On, Off, Passthru or Bypass.
❐ When shaping is on, traffic is classified and measured, and policies and partitions actively
control bandwidth allocation.
❐ When shaping is off, traffic is classified and measured, but it is not actively managed.
❐ Bypass mode prevents both packet shaping and further network management access; it is
as if the appliance were removed, and cables connected around it.
❐ Passthru turns off all shaping, classification, and measurement.
• Identify the different behaviors displayed by the PacketShaper based on the above shaping
states. Some of which include:
❐ Slow LAN traffic when shaping is turned on
This scenario is probably attributed to the way PacketShaper is deployed in the network
which involves a DMZ segment. If the traffic from LAN to the DMZ is passing through
the appliance, such complaints are bound to occur. The best solution to such problem is to
ignore the such network traffic. Additionally, classifying the traffic by IP segments and
appropriately applying an ignore policy for such classes can help PacketShaper not
include these in the bandwidth calculations.
❐ Lack of connections
Two possible reasons arise when the above behavior is displayed — class license and
policy flow limits. Class licenses limit the number of TCP flows allowed simultaneously in
the given class, where the number of flows admitted to a class is based on a fixed number
instead of the available bandwidth. Increasing the flow limits using the class
licenses CLI command can sometimes resolve the above issue.

Policy flow limit controls the rate of new flows to or from a unique host. This command
can be used to detect and control a SYN Flood or similar denial-of-service attack directed
at a particular host or if the attack is from a specific IP address. Flows exceeding the rate
are blocked from passing through the unit.
Note: The limits are set to default values of 10,000 flows per minute on client hosts and
100,000 flows per minute on servers; depending on your network, you may need
to change these defaults for effective control of SYN floods.
Flow limits are automatically set on any classes that have a rate or priority policy assigned
to them, and PacketWise will automatically block any flows that exceed these limits.
Policy flow limits can be verified by using the hostdb info CLI command, which
displays the host IP address, average and current connections, current guaranteed and
excess bandwidth, and throughput information. The hostdb info command output can
be extended to show additional information such as:
• -sf: sort hosts by flow per minute in descending order
• -sf: sort hosts by rate of failed new TCP connections
• -sr: sort hosts by current rate in descending order
Monitoring the failed flows after this can help identify policy flow limits on the appliance.
Additionally, analyzing the PacketShaper MIB counters from the support diagnostic files
can also help resolving policy flow limit issues. Specifically, MIB counters 31 - 33 can help
shed additional light into troubleshooting policy related problems. Once the
corresponding traffic class is identified, either globally turning off or increasing the values
for the policy flow limit helps solve the issue.
❐ Slow traffic after applying policy
Use the graph options in the WUI to identify any instances of guaranteed rate failure for
that particular traffic class. Under such circumstances, PacketWise in unable to allocate
the guaranteed rate when applying a class’ rate policy. Verify if the correct policy is being
applied under such circumstances. if yes, proceed to manage admission control on that
traffic class. Admission control is a mechanism for a rate policy that determines what
happens when there isn't enough bandwidth to satisfy guaranteed rate requests (such as
refuse connections or give a trickle of bandwidth). Admission control lets you set a
minimum level of service — and even refuse connections — when congestion occurs or a
persistent backlog exists. If not, check excess bandwidth and policy guidelines and
proceed to troubleshoot partition related issues.

Partition Issues
6OLGH3DUWLWLRQLVVXHV
The above slide describes some of the widely reported problems that can be attributed to partition
related settings on the PacketShaper and possible solutions to overcome such issues. A best
practice would be to observe network behavior by changing the network shaping to either on, off,
bypass or pass thru settings. Common symptoms include:
1. Partition not getting configured reserved bandwidth.
This behavior is commonly observed in a traffic class that is configured with a specific
partition size and corresponding statistics on the WUI differ from the configured values.
Possible reasons include:
a. Partitions with burst able bandwidth: This situation can be explained using an
example where the partition size is 384 K. 256K is used at the moment and there are
two other partitions that have reached their fixed rate limit and both of them want to
use the spare 128K bandwidth. PacketShaper allocates excess bandwidth to partitions
depending on what the policies are for the individual classes. Even if all things are
equal (for example, every class had priority 3 policy), the excess is distributed on a
first come/first served basis and will not necessarily be distributed evenly. However,
by instituting guaranteed rate policies with varied burst priorities you can control
where the excess will go first.
In the above slide, bandwidth utilization for a partition can be identified using the traffic
bandwidth CLI command. The output displays the following:
❐ Programmed minimum bandwidth = 1M
❐ Adjusted bandwidth = 800K. Since this class is overcommitted, the PacketShaper has
adjusted the bandwidth to 800K.
Scrolling down the CLI command output, you will observe that, current guaranteed excess
rate is 1.2 M, but the actual demand for this traffic class is 1.4Mat priority level 3, which is the
default priority setting. The traffic class has a maximum limitation of 2.0M, the 800K is
considered guaranteed based on the above and 1.2M is considered as excess rate.

b. Overcommitted bandwidth: If you have an appliance with a link size of 10M and you
have guaranteed partitions which aggregates to 20M, you are over committing by
10M. In order to make the size of the guaranteed partitions equal to the specified link
size, PacketWise will dynamically scale down all the partition sizes so that total of all
the guaranteed partition sizes will be 10M — exactly equal to the specified link size. In
order to get around this, do not overcommit the bandwidth. Do not guarantee more
than you have specified on the link size. You can make the partitions burst able to a
higher limit if you want. The overcommittment applies only to the guaranteed
portion of the partition, not to the burst able portion.
Through the WUI, click on Manage >Partition Summary to get an over view of current
partitions. In the above slide, the presence of the (*) against the /Inbound/Critical/FTP traffic
class indicates that the partition is overcommitted by 29.9%. Under such circumstances, the
PacketShaper dynamically modifies the partition sizes to ensure the guaranteed partition
size for that traffic class.
2. Less connections.
This behavior could be attributed to many factors on the PacketShaper. One of the widely
observed cause could be that of full partition size. Analyzing the PacketShaper MIBs— mib
relay can explain the full utilization of the partitions. In the above slide, MIB counter
numbers 18,19, 20 and 21 provide indication of packets dropped due to complete partition
utilization. Any incremental value in the above MIB counters is a clear indication of less
connections.
3. Dynamic partition displays full status.
This behavior can be verified by checking on the system limitations for that particular model
of the PacketShaper. Issuing a sys limits CLI command reveals the current, remaining and
total available dynamic partitions for that appliance. Dynamic partition summary can be
analyzed both from the WUI and CLI.
In the above slide, the command output for the part dyn sum is used to analyze the
dynamic partition status. As you can see, the output shows all the configured dynamic
partitions and the number of users currently using each partition. Users that are not active can
be replaced by new users. There are five current users which implies five hosts are allowed for
that dynamic sub-partition. Further, amongst these five subpartitions, only one is active, none
idle, one Gone and three LongGone.
A subpartition is considered Idle if it has not been active for 300 seconds (5 minutes). Idle
subpartitions still have flows which are sending packets. A subpartition is considered Gone if
the flows associated with it have been gone 30 seconds or less, or LongGone if they have been
gone more than 30 seconds. When the dynamic partition cap has been reached, new
subpartitions are created from LongGone and Gone partitions.
4. Partition not utilized.
This condition is normally observed when a traffic class has allocated partitions but does not
display when utilization graphs are generated from the WUI. One of the possible causes for
this behavior could be incorrect WAN link setting during the initial set up. Ensure this setting
matches the correct WAN link size of your network. Another factor could be that of expired
license key for shaping or link size settings. Either providing the correct license key or
upgrading the same can help to restore shaping and proper application of the partitions for
that particular traffic class.

Aggressive Traffic
Identify aggressive hosts
– Top talkers and listeners
– host info CLI command
Control aggressive hosts

– Contain or limit bandwidth
– Use adaptive response feature
6OLGH$JJUHVVLYHWUDIILF
PacketWise keeps track of the hosts that generate the most traffic. You can configure PacketWise to
track the Top Talkers (hosts that initiate the most traffic) and Top Listeners (hosts that receive the
most traffic) for up to 12 different traffic classes. If traffic in one of the Default or numbered port
classes is regularly consuming a significant portion of total bandwidth (such as more than 10%),
you can use the Top Talkers/Listeners feature to create new classes, thereby reducing the amount
of unclassified traffic. You can turn on Top Talkers and/or Talk Listeners for a Default or
numbered port class, and after a few days, look at a report to see if any single host is using a
substantial percentage of the total (for example, more than 15%). You can then create a class for
this host.
Using the hostdb info CLI command displays the host IP address, average and current
connections, current guaranteed and excess bandwidth, and throughput information. Additional
information on the hostbd info command is provided in Slide 16-6. Additionally, traffic
active CLI command displays the current, maximum, and possible number of sessions for TCP,
UDP, and Legacy traffic types. This command is a valuable tool for determining how close the unit
is to reaching its capacity. It also gives a histogram of the number of host entries in various time
buckets (based on idle time).
Controlling such aggressive hosts can be achieved by containing bandwidth using partitions.
Make the partition burst able, and use the minimum and maximum from previous steps for the
partition’s field values. Remember to create partitions for your traffic on both your Inbound and
Outbound traffic trees.
Using class license CLI command limits the number of TCP flows allowed simultaneously
in the given class, where the number of flows admitted to a class is based on a fixed number
instead of the available bandwidth.
You can also automatically identify and control aggressive hosts using the adaptive response
feature in the PacketShaper. The hosts category of agents uses information from PacketShaper’s
host database. These agents are useful for identifying hosts that are using too much bandwidth or
that may be attacking your network, for example spoofing and SYN attacks.

The high bandwidth host agent monitors hosts, evaluating whether any one is using too much
bandwidth, and can help prevent any one host from consuming too much bandwidth. The High
Bandwidth Host agent will not return correct values if the appliance has more than one instance of
this agent type, or if the Quota Bandwidth Host agent is also enabled. The agent tracks hosts
sending and/or receiving an excessive amount of traffic, allowing you to find hosts that are
downloading data at high levels.

Infected Users
Symptoms
– Numerous flows by single host
– Numerous class hits
– Variations in TCP health graph
– Changes in flow limit counter
Solutions
– Use policy flow limit
– Use partitions to limit bandwidth
6OLGH,QIHFWHGXVHUV
Sometimes customers experience one or all of the following behaviors — numerous flows by
single host, too many class hits, marked variations in the TCP health graph and that of the flow
limit counters.
A high number of new flows per minute and or failed flows per minute can indicate an attack is
occurring. Adaptive response offers several agent templates that monitor flows to or from hosts to
help in the detection of SYN attacks and spoofing. After setting up these adaptive response agents,
you can be notified automatically when a host has excessive failed flows and or new flows per
minute, thus alerting you to a possible attack. In addition, you can automatically restrict the
bandwidth of these violating hosts so that you can minimize the problem while you do further
investigation.
Class hits let you know the number of times flows match a class. By monitoring class hits
regularly, you will know what constitutes a normal number of hits in a class and an abnormally
high number of hits will be apparent.
Two PacketWise measurement variables—tcp-conn-inits and tcp-conn-server-ignores
track the number of new TCP connections and the number of TCP connections for which the
server never responded. By studying the values of these variables for Inbound and Outbound
traffic, you can establish baselines for TCP connections in a normal, non-attack setting. Then,
using the adaptive response feature, you can be notified automatically when these values are
outside the normal range. The TCP Health graph gives you a comprehensive picture of TCP
connections for a particular link, partition, or class. It compares the number of TCP connections
that were started, aborted, ignored by the server and refused by the server during the specified
time period.
One of the effective measures to control infected users can be achieved by using policy flow limit
CLI command, which limits the rate of new flows to or from a unique host. This command can be
used to detect and control a SYN flood or similar denial-of-service attack directed at a particular
host or if the attack is from a specific IP address. Flows exceeding the rate are blocked from
passing through the PacketShaper.

Note: The limits are set to default values of 10,000 flows per minute on client hosts and
100,000 flows per minute on servers; depending on your network, you may need to
change these defaults for effective control of SYN floods. Flow limits are
automatically set on any classes that have a rate or priority policy assigned to them,
and PacketWise will automatically block any flows that exceed these limits.

Unknown Traffic
Symptoms
– Misclassification
– Traffic hitting default bucket
Analysis
– Host analysis
– traffic flow, traffic history CLI commands
– Packet capture
6OLGH8QNQRZQWUDIILF
Many customers view PacketWise auto-discovery, or automatic traffic classification, as their

favorite and most useful feature. Knowing the identity of traffic running over your network is a
big first step in managing and controlling the performance of network applications. When
PacketWise sees a traffic flow, it matches the flow’s characteristics to those of each class in the
traffic tree. If it finds a match, that class’ metrics are incremented accordingly, and the flow is
managed with the class’ policies. If it does not find a match, but PacketWise can identify the traffic,
and traffic discovery is enabled, PacketWise creates a corresponding new traffic class. There are
several reasons PacketWise might not create a class:
• Traffic discovery is disabled
• PacketWise can't identify the traffic
• There are so many classes in the traffic tree that more are not possible
• Insufficient number of flows have passed to prompt PacketWise to make a new class
(anywhere from one to 11 flows must pass before a traffic type gets its own class, depending
on the type of traffic)
• The PacketWise unit was recently plugged in and started monitoring long sessions that were
already in progress. Until a new flow starts, this traffic all counts as Default. In these cases, the
solution is just to wait another few hours until you examine the tree.
In any of these cases, PacketWise matches the flow with most appropriate Default class (usually
Inbound's or Outbound's Default class). Determining what traffic is in the Default class is usually
not a concern or priority. But if the amount of traffic in a Default class increases precipitously, or if
most of your traffic is classified in the Default class, you'll probably want to figure out what that
traffic is.
Some of the measures to identify and capture unknown traffic include:
• The host analysis reporting tool allows you to gain insight into host and flow activity on your
network. Using this tool you can:

❐ List bandwidth utilization and flow information for hosts on the network. This list can
consist of inside hosts, outside hosts, active hosts, or all hosts, and can be sorted by
bandwidth utilization, new connections, failed connections, or host IP address.
❐ Find flows for a particular IP address, port number, or protocol. For each flow, the list
provides the IP addresses of the conversation pair, the class name into which
PacketShaper classified the flow, and the protocol of the flow.
❐ Drill-down to find out detailed flow information for a suspicious host (for example, one
that is using excessive bandwidth or creating an inordinate number of connections or
failed connections).
• Explore the mysterious default or port-based class further with the traffic flow CLI
command. It has many options that display a variety of different information. If you enter
traffic flow without any additional parameters, you get a list of the command's options.
You can use the traffic flow command for a particular address that interests you (that you
got from Top Listeners, for example), a particular class or host, a certain number of flows, and
more.
• Use the traffic history recent <classname> CLI command for your class to see the
date, time, IP address, port number, and URL for each flow in the specified class. If there is an
IP address or DNS name that you'd like to explore further, do a traffic history find
<host>. This command allows you to see in which classes a host's flows hit, as well as the
number of flows and the protocol. If the protocol cannot be identified, a dash (—) appears.
• Have PacketWise capture a log of the class' traffic and then feed that log to a sniffer or third
party analyzing software such as EtherPeek.

Chapter 17:PacketShaper Report Issues
Blue Coat® PacketShaper® measures many characteristics of network traffic and stores associated
metrics onboard for up to two months, providing the capability to create pre-configured or custom
reports. Using PacketShaper’s extensive reporting capabilities, you can view graphs of link and
application usage and performance. In addition, PacketWise collects data for over 120 different
measurement variables; this data can be graphed in a variety of ways. All reports are accessible
through PacketShaper’s Web User Interface (WUI).
PacketWise's statistical reports are user customizable. You specify the object for which you want to
gather measurement data — a specific link, partition, or class — and choose which type of data
you want to analyze, such as class hits, number of TCP connections refused by the server, packet
count, or number of retransmissions. The PacketShaper measurement engine is a background
process that collects time-series and histogram data that is stored in the appliance’s hard drive.
Problems can arise with the data collected and stored by the measurement engine, and this can
lead to issues with reports based on this data.
This chapter covers some of the common issues related to PacketShaper reports. This chapter
assumes that:
• You are familiar with the different report options available on the PacketShaper.
• You are familiar with various options for creating graphs on the Report tab through the
PacketShaper WUI.
• Some of the common error messages during reporting and how to interpret them.
• Major causes of reporting issues and how to troubleshoot them.
• Common problems encountered during report data retrieval and how to resolve them.

Common Error Messages
Error Code s Possible Ca uses Solution

Error 4210 In sufficient me mory Re boot applian ce
Un wanted ad aptive Di sable unwa nted adaptive response
response agents agents
Large traffic class tree Re duce / prune traffic class tre e
Error 1711 Mea surement engi ne error me show / me start CLI
command
Trouble shoo t measurement engine
6OLGH&RPPRQHUURUPHVVDJHV
Two of the most commonly encountered error messages while working on PacketShaper reports
are:
• Error 4210: This error is observed when there are system-related issues on the PacketShaper,
such as insufficient memory for metric operations. Possible causes include:
❐ The PacketShaper does not have enough memory to run the requested reports, or the CPU
is too busy. Reboot the appliance to resolve these issues.
❐ Unwanted adaptive response agents can use memory resources on the PacketShaper and
sometimes lead to error 4210 being displayed. Determine whether there are any unwanted
adaptive response agents created on the PacketShaper. Disabling these agents can help
solve this problem. Navigate to the Setup tab on the PacketShaper WUI and select
adaptive response from the drop-down list. Locate the agent that is causing the above
issue and try disabling it.
❐ A large traffic class tree on the PacketShaper can cause the extensive use of memory
resources and prevent report generation. Reducing or pruning the traffic class tree can
resolve this condition.
• Error 1711: This error is primarily due to the measurement engine error on the PacketShaper.
Issuing the me show command via the CLI will bring out any measurement engine errors on
the appliance. If any the counters show the word ERROR, try resetting that measurement
group by issuing the CLI command measure reset group-name,where group-name is the
name of the group with the error. To reset all measurement groups, do not specify a group by
name.This will reboot the PacketShaper and clear the report data for that group.
Troubleshooting measurement engine errors are discussed in detail in Slide 17-2.

Chapter 17: PacketShaper Report Issues
Troubleshoot Measurement Engine
6OLGH7URXEOHVKRRWPHDVXUHPHQWHQJLQH
Measurement engine problems are possible causes of reporting errors on the PacketShaper and
understanding the troubleshooting process behind it is of utmost importance. The above
flowchart shows a high-level framework for troubleshooting measurement engine errors and
starts with executing the me show CLI command.
1. Is the PacketShaper measurement engine running properly? If not, restart the measurement
engine.
2. Are any errors found during the running of the measurement engine? If so, check the type of
error found — is it a specific type of measurement error or all categories of measurement
variables? To resolve this, enter the CLI command measure reset type where type is the
variable that is causing errors. To reset all measurement data, do not specify a type.
3. Are there any errors that occur after the issue of the me reset command? If so, this can be
due to problems on the PacketShaper hard drive. Formatting the hard drive using the sys
clean hard command usually helps solving this type of problem. If not, contact Blue Coat
Technical Support to request a Return Merchandise Authorization (RMA) for the
PacketShaper.
4. Check whether the PacketWise software on the PacketShaper has been recently updated. The
measure reset command needs to be performed after every major PacketWise update
because new measurement variables might not be compatible with previous versions and
might cause reporting errors..

Restore Measurement Data
6OLGH5HVWRUHPHDVXUHPHQWGDWD
PacketWise stores data on dozens of measurement variables, such as avg-round-trip-time and

class-hits. For classes and partitions, PacketShaper stores at least one day’s worth of per-minute
data samples and at least one month’s worth of per-hour data samples. For links, PacketShaper
stores seven days of per-minute data samples and six months of per-hour samples. Customers
sometimes want to transfer or restore measurement data from one PacketShaper to another.
Important: If you are restoring measurement data files to a different appliance from which
they were backed up, make sure that both appliances are on the same hardware
platform and use the same PacketWise software version.
The above flowchart describes the steps involved in restoring measurement data between two
PacketShaper appliances, PS1 and PS2:
1. Backup all measurement data on PS1 by issuing the measure backup all groups CLI
command. Resulting measurement data files have a default file name of XXXXXXXY.dat, where
X is the 7-digit serial number of the PS1 appliance and Y is the running number for each
measurement group.
2. Rename all the PS1 measurement data files with the serial number of PS2, keeping the
measurement group number as the last digit.
3. To restore the measurement data on PS2, enter the CLI command measure restore all
groups.
Note: Blue Coat recommends that you back up all the measurement data on PS2 before
using the measure restore command. Copy or transfer the config.ldi file from PS1
to PS2 after issuing the measure restore command.

Issues with Saved Reports
6OLGH6DYHUHSRUW
You might want to save a PacketShaper report for future reference, perhaps to make comparisons
of different shaping strategies at different times. Use the save report button on the PacketShaper
WUI for this purpose. Common issues with saved reports include:
1. The customer reports that no additional reports can be saved using save report. The number of
reports that can be saved on a PacketShaper is defined by the system variable
reportDefinesCnt and has a default value of 10. If this default value is exceeded even by
one additional report, the customer cannot save any new additional reports. Deleting any
existing saved reports or increasing the default values of the reportDefinesCnt variable
can help resolve this problem.
2. The customer reports that a previously saved report cannot be deleted. This typically happens
when a PacketShaper is attached to a PolicyCenter. Deleting the saved report from the
PolicyCenter can help resolve this problem.

Issues with Report Data Retrieval
6OLGH5HSRUWGDWDUHWULHYDOLVVXHV
Reports can be generated from the PacketShaper WUI from the Report, Manage or Top Ten tabs.
Under certain circumstances, reports are not generated because of data retrieval issues. Common
issues with data retrieval include:
1. The customer cannot create reports from the Report tab of the PacketShaper WUI. Report
generation from the Report tab is only possible when touch access to the appliance is
enabled. So, ensure that touch access is enabled for the customer under consideration.
Note: Report generation from the Top ten and Manage tabs are permitted when look access
to the PacketShaper is enabled.
2. The customer reports a system time-out message during report generation. This can be caused
by large amounts of traffic flowing through the PacketShaper, which gives a higher priority to
classification and measurement tasks. To allow more time for the PacketShaper to generate
reports, change the graph timeout value:
a. In the WUI: Go to Setup > system variables > Browser interface and enter a new value
for Graph Timeout.

b. In the CLI: Enter the command sys set graphTimeoutSeconds value, where
value is the timeout value in seconds, up to a maximum of 600 seconds (10 minutes).


Your comments, please
Thank you for taking this BlueTouch Training Services course. Your comments on this course are
appreciated and will help Blue Coat improve future versions of this course.
Course: PacketShaper Troubleshooting

Edition: Student textbook
Version: 1.1.1
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
______________________________________________________________________________________
Send your comments via postal mail to:

Blue Coat Systems Inc.
BlueTouch Training Services
410 North Mary Avenue
Sunnyvale, California USA 94085
Or you can send comments via e-mail to:

training.books@bluecoat.com
When e-mailing, please include the course name, edition, and version as shown above.
For information on other courses offered by BlueTouch Training Services, go online to:
http://bluecoat.com/support/training



Psts v111 Student Us WMK

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Psts v111 Student Us WMK

Încărcat de

Drepturi de autor:

Formate disponibile

?

Property of Blue Touch Training Services.

Blue Coat Systems Inc.

North America (USA) Toll Free: +1.866.302.2628 (866.30.BCOAT)

ii Property of Blue Touch Training Services.

Course Introduction .....................................................................................1

Chapter 1: Troubleshooting Methodology ...................................................3

Chapter 2: BlueTouch Online ....................................................................15

Chapter 3: Hardware Overview ................................................................. 23

Chapter 4: PacketShaper Boot Process ....................................................37

Chapter 5: Understanding PacketShaper Commands ...............................47

Chapter 6: Analyzing PacketShaper Logs .................................................63

Chapter 7: Hardware Troubleshooting .......................................................73

Chapter 8: Hardware Failure Case Studies ............................................... 87

Chapter 9: Configuration Issues ................................................................99

Chapter 10: Configuration Case Study ....................................................113

Chapter 11: Classification Case Study ....................................................119

Chapter 12: Troubleshooting PacketWise Software ................................129

Chapter 13: Software Image Case Study ................................................141

Chapter 14: Access and Performance Issues .........................................147

Chapter 15: Performance Case Study .....................................................159

Chapter 16: PacketShaper Traffic Flows .................................................167

Chapter 17: PacketShaper Report Issues ............................................... 185

Property of Blue Touch Training Services. iii

NOT for Distribution. For Reference Purposes Only.

iv Property of Blue Touch Training Services.

Applicable Software Versions

Property of Blue Touch Training Services. 1

NOT for Distribution. For Reference Purposes Only.

2 Property of Blue Touch Training Services.

Property of Blue Touch Training Services. 3

NOT for Distribution. For Reference Purposes Only.

 Use Ishikawa diagram

© Blue Coa t Systems, In c. 2009 . All Rights Reserve d.

4 Property of Blue Touch Training Services.

© Blue Coa t Systems, In c. 2009 . All Rights Reserve d.

Three phases are involved in the troubleshooting methodology.

Property of Blue Touch Training Services. 5

NOT for Distribution. For Reference Purposes Only.

6 Property of Blue Touch Training Services.

Phase 1-1: Define the Issues

 What is the normal behavior?

© Blue Coa t Systems, In c. 2009 . All Rights Reserve d.

Property of Blue Touch Training Services. 7

NOT for Distribution. For Reference Purposes Only.

Phase 1-2: Describe the Symptoms

© Blue Coa t Systems, In c. 2009 . All Rights Reserve d.

8 Property of Blue Touch Training Services.

Phase 1-2: Gather Information

© Blue Coa t Systems, In c. 2009 . All Rights Reserve d.

Property of Blue Touch Training Services. 9

NOT for Distribution. For Reference Purposes Only.

Phase 1-3: Document the Changes

 Compare to working units

© Blue Coa t Systems, In c. 2009 . All Rights Reserve d.

10 Property of Blue Touch Training Services.

Phase 2-1: Ishikawa diagram

© Blue Coa t Systems, In c. 2009 . All Rights Reserve d.

Property of Blue Touch Training Services. 11

NOT for Distribution. For Reference Purposes Only.

Phase 2-2: Research Likely Causes

© Blue Coa t Systems, In c. 2009 . All Rights Reserve d.

12 Property of Blue Touch Training Services.

Use Ishikawa diagram

What is the normal behavior?

Compare to working units

Options and serviceability

Interrupt boot process