Monitoring and Tuning TCP/IP Networking

Adrian.Cockcroft@sun.com

Adrian Cockcroft - Sun Microsystems 3/16/01

Monitoring and Tuning TCP/IP Networking Page 1

Contents Introduction Communications Principles TCP - A Simple Approach TCP Measurements TCP Tunables How Web Servers Use TCP TCP Behavior Plots

Adrian Cockcroft - Sun Microsystems 3/16/01

Monitoring and Tuning TCP/IP Networking Page 2

Introduction Keep it simple to start with Understand the basic high level measurements Look at common usage by web servers Focus on monitoring TCP activity ......then...... Look at the complexities of TCP in more detail See plots of TCP behaving and misbehaving Learn how to look for problems Learn how to tune TCP
Adrian Cockcroft - Sun Microsystems 3/16/01 Monitoring and Tuning TCP/IP Networking Page 3

Communications Principles How do you communicate remotely? Making phone calls to a person at a phone number. Phone calls are bidirectional connections. You keep trying until you connect to the right person and have your conversation. You know when the information arrives at its destination. TCP stands for Transmission Control Protocol, it is like the protocol you use to make a phone call.

Adrian Cockcroft - Sun Microsystems 3/16/01

Monitoring and Tuning TCP/IP Networking Page 4

Connection Based Communications A phone call has a complex calling sequence: Dial number This is Bill, is Jim there? Hi Jim, heres my message. Thanks, are we done? Good-bye. Hang up. Ringing, pick up Yes, this is Jim. OK Bill, heres my reply. Yes, good-bye. Hang up.

Adrian Cockcroft - Sun Microsystems 3/16/01

Monitoring and Tuning TCP/IP Networking Page 5

Sources and Destinations For phone calls there is a person at each location. Network locations are specied using Internet Protocol (IP) addresses - like 192.161.1.100 The IP address only species the location, there is an additional port number that identies who to talk to at that location - like 192.161.1.100:80 for port 80 (HTTP). Each TCP/IP connection is uniquely identied by two addresses and their port numbers. Special port numbers are reserved for well known services and higher level protocols Other port numbers are assigned when needed
Adrian Cockcroft - Sun Microsystems 3/16/01 Monitoring and Tuning TCP/IP Networking Page 6

Listening to the Hold Musak Lots of people phone into a switchboard There may not be enough lines to hold on so you could get an unobtainable signal You reach the operator and ask for someone You wait while the operator tries to nd the person You are on hold, in a listen queue with a very tinny version of Abbas greatest hits playing.... Hopefully your call will be accepted You may be rejected or give up waiting (time out)
Adrian Cockcroft - Sun Microsystems 3/16/01 Monitoring and Tuning TCP/IP Networking Page 7

The TCP Listen Queue TCP also has a listen queue! All incoming connections that have not yet completed the handshake and been accepted are in the queue. The listen queue size is xed by the application, a systemwide tunable limit sets the maximum of 1024.
The limit you set on Solaris is tcp_conn_req_max_q0 You need to tune your web server conguration le as well

You can calculate the average queue length, and set the maximum to at least 3 times the average (95%ile)
mean listen queue length = mean connect setup time * mean connect rate maximum listen queue length > 3 * mean connect setup time * mean connect rate Slow Internet users: 1024 > 3 * 0.5s * 600 conn/s Fast local users: 1024 > 3 * 0.001s * 300000 conn/s
Adrian Cockcroft - Sun Microsystems 3/16/01 Monitoring and Tuning TCP/IP Networking Page 8

Remembering and Rejecting Calls After you talk to someone you remember them for a while, and can decide whether to talk to them again. Caller-id tells you who is ringing your cellphone. You keep numbers for people you dont want to talk to in your phone so you can just ignore the call. TCP/IP remembers every connection for 1-4 minutes. If one end keeps transmitting when the other has closed down, then it will get sent reset packets to tell it that no-one wants to listen. With high connection rates, its hard to remember very many connections, and TCP needs to be tuned.
Adrian Cockcroft - Sun Microsystems 3/16/01 Monitoring and Tuning TCP/IP Networking Page 9

Start Slow - Then Speed Up We always exchange a short greeting, then maybe a longer sentence, before talking for a long time. The other person interrupts all the time saying OK, yes, uh-huh, which acknowledges receipt. If you dont get any acknowledgement or you get a query - sorry, could you repeat that? you repeat yourself in a short phrase and stop to check its OK. TCP/IP doesnt know how fast the link to the client is so it cant just send data at full speed immediately. TCP/IP sends one packet, waits for the ACK, then sends two packets, waits for the ACK and keeps doubling until it reaches full speed.
Adrian Cockcroft - Sun Microsystems 3/16/01 Monitoring and Tuning TCP/IP Networking Page 10

I Repeat Myself When Under Stress, I Repeat Myself When Under Stress, I Repeat Myself When Under Stress If you dont think the other person is listening, you repeat what you are saying until you get acknowledged. If there is noise on the line, or too many people talking at once you cant hear so you repeat yourself. TCP/IP resends packets if it doesnt get an ACK within its time-out period. The time-out varies adaptively, and is remembered for routes that have been used before. Unlike a phone line, the time taken to get packets over a connection varies a lot, depending on packet size, congestion and changes in the route. It is normal to retransmit over Internet connections
Adrian Cockcroft - Sun Microsystems 3/16/01 Monitoring and Tuning TCP/IP Networking Page 11

Stop! Youre Talking Too Fast! When you are listening and cant take in the information fast enough, you ask the other person to slow down or pause until you catch up, because you can only remember a few sentences at a time. The receiving end of a TCP/IP connection controls the ow of data into its buffer space by sending a sliding window size to the sender along with its ACKs. If the window is too small and latency is high then throughput is affected. Default 8KB needs to be tuned.
For sustained max speed: window size > mean bandwidth * mean latency 56Kbit modem is OK with typical latency: 8KB > 6KB/s * 1.3s 100MBit LAN is marginal: 8KB > 10000KB/s * 0.8ms (with routers 2-3ms) Set to at least 32KB for Gigabit ethernet and DSL or Cable Modem Internet
Adrian Cockcroft - Sun Microsystems 3/16/01 Monitoring and Tuning TCP/IP Networking Page 12

Denial of Service Attack Its as if someone is jamming your cellphone, and it cant get through to make a call. TCP can be (and has been) jammed as well. Special programs talk directly to the network card.
They construct badly formed TCP SYN packets. The destination is your machine, but the return address is corrupted so the reply does not go back to the sender. Your machine uses up a listen queue slot, replies and waits until it times out. When the listen queue is full no-one can talk to you. Solaris detects this strange pattern and throws out bad packets. Two queues are used, so completed connections are separated

Latest variation attempts to saturate the network.

Hackers take over lots of high bandwidth systems and use them to ood the victim with so much trafc its network saturates.
Adrian Cockcroft - Sun Microsystems 3/16/01 Monitoring and Tuning TCP/IP Networking Page 13

How Do Protocols Use TCP/IP? TCP/IP is used underneath HTTP for web servers with very high connection rates. TCP/IP is used underneath FTP for le transfers with moderate connection rates. Database client/server connections use TCP/IP underneath with low connection rates. NFSTM started off running over UDP/IP on reliable networks with 8KB transfers. NFS now defaults to TCP/IP between two SolarisTM machines and can work better over congested networks with 32KB transfers. The connection rate is low.
Adrian Cockcroft - Sun Microsystems 3/16/01 Monitoring and Tuning TCP/IP Networking Page 14

Measuring and Monitoring TCP Capacity and Throughput Metrics to Watch Connections
Current number of established connections New outgoing connection rate (active opens) Outgoing connection attempt failure rate New incoming connection rate (passive opens) Incoming connection attempt failure rate (resets)

Throughput
Input and output byte rates Input and output segment rates Output byte retransmit percentage Duplicate input byte percentage

Adrian Cockcroft - Sun Microsystems 3/16/01

Monitoring and Tuning TCP/IP Networking Page 15

TCP Measurements
Established Connections Throughput Measures

Outgoing Active Opens Attempt Failures Incoming Passive Opens Connection Rejected Reset sent

Adrian Cockcroft - Sun Microsystems 3/16/01

Monitoring and Tuning TCP/IP Networking Page 16

Obtaining Measurements Generic: get the TCP MIB via SNMP Unix: netstat -s shows the TCP counters Standard TCP metric names:
tcpCurrEstab: current number of established connections tcpActiveOpens: number of outgoing connections since boot tcpAttemptFails: number of outgoing failures since boot tcpPassiveOpens: number of incoming connections since boot tcpOutRsts: number of resets sent to reject connection tcpEstabResets: resets sent to terminate established connections (tcpOutRsts - tcpEstabResets): incoming connection failures tcpOutDataSegs, tcpInDataSegs: data transfer in segments tcpRetransSegs: retransmitted segments

Byte level throughput statistics are vendor specic

Adrian Cockcroft - Sun Microsystems 3/16/01 Monitoring and Tuning TCP/IP Networking Page 17

Check that you understand the basics Which protocol is like a phone call? What goes with an IP address to identify the service? What happens if you try to talk and no-one is listening? What are the two kinds of activity to monitor? What does a tcpAttemptFail mean? How do you know your message got through?

Adrian Cockcroft - Sun Microsystems 3/16/01

Monitoring and Tuning TCP/IP Networking Page 18

Internet Server Issues TCP Connections are expensive

TCP is optimized for reliable data on long lived connections Making a connection uses a lot more CPU than moving data Connection setup handshake involves several round trip delays Each open connection consumes about 1 KB plus data buffers Pending connections cause listen queue issues Each new connection goes through a slow start ramp up

Other TCP Issues

TCP windows can limit high latency high speed links Lost or delayed data causes time-outs and retransmissions HTTP persistent connections carry several ops on one connect

Look at behavior plots next to see TCP in action

Turn snoop data into plots with the packet shell tcp.analysis tool Get it from http://playground.sun.com/psh
Adrian Cockcroft - Sun Microsystems 3/16/01 Monitoring and Tuning TCP/IP Networking Page 19

Sequence Number

TCP in Pictures
Medium sized HTTP request and response with retransmit congestion

Window Advance

Request Packet

ACK Packets Retrans Packets

Time in seconds
Adrian Cockcroft - Sun Microsystems 3/16/01 Monitoring and Tuning TCP/IP Networking Page 20

TCP in Pictures
Clean transfer with slow start ramp up - Transfer with window close ow control

Ramp up packets

Window closed

client limited

ACK<min retrans

Network limited

Adrian Cockcroft - Sun Microsystems 3/16/01

Monitoring and Tuning TCP/IP Networking Page 21

TCP in Pictures
Problem with increased minimum retransmit time-out Persistent connections showing two small transfers and a large image over high speed net

retrans too slow Packet gets lost

Three transfers

Adrian Cockcroft - Sun Microsystems 3/16/01

Monitoring and Tuning TCP/IP Networking Page 22

Tuning TCP Conguration Parameters TCP Tunables (use Solaris ndd command)
tcp_close_wait_interval (actually tcp_time_wait_interval) default 240000 ms. Reduced to 60000 for SPECweb96 runs tcp_conn_req_max_q0 - incomplete listen connection limit default 1024 - no need to tune unless tcpListenDropQ0 seen tcp_conn_req_max_q - pending completed connection limit default 128 - no need to tune it unless tcpListenDrop seen tcp_slow_start_initial - number of initial packets to send default 1 - set to 2 to be the same as other vendors tcp_xmit_hiwat and tcp_recv_hiwat - window size default 8192 - set to 32768 (which is Windows and MacOS default)

TCP Connection Hash Table (add to /etc/system)

Increase from default of 256, must be power of 2, up to 262144 set tcp:tcp_conn_hash_size=32768
Adrian Cockcroft - Sun Microsystems 3/16/01 Monitoring and Tuning TCP/IP Networking Page 23

TCP Rules and the SE Toolkit Let SE watch your TCP stack for you
ignore everything if TCP throughput is less than 2 KB/s warn if retransmit rate is over 15%, problem if over 25% warn if listen drops are seen, problem if over 0.5/s increase the listen queue size until this problem stops warn of SYN service denial attack if tcpHalfOpenDrop over 2/s try to block the source of the attack at high levels warn if connection refused - sending reset packets at 0.5/s increase this threshold if it becomes annoying possible port scanner attack if resets at over 2/s this threshold is also too low, set higher if you get false alarms warn if attempted outgoing connection fails over 2/s try to nd the process on your system that is failing to connect warn if incoming duplicate packets over 15% problem if duplicates at 25% - remote is retransmitting at you
Monitoring and Tuning TCP/IP Networking Page 24

Adrian Cockcroft - Sun Microsystems 3/16/01

Tools that use the TCP rule

virtual_adrian.se - text log of problems seen zoom.se - GUI front end to rules percollator.se - long term web server data collector Also use tcp_monitor.se to check and tune TCP

Adrian Cockcroft - Sun Microsystems 3/16/01

Monitoring and Tuning TCP/IP Networking Page 25

Conclusion Think about TCP/IP behaving like the phone system Monitor the simple measurements Tune TCP parameters when needed Look at TCP packet ow diagrams to understand what is really going on Use TCP rules to automatically watch for problems, but customize the rule thresholds to your situation

Adrian Cockcroft - Sun Microsystems 3/16/01

Monitoring and Tuning TCP/IP Networking Page 26