UDP GSO Offload

Boris Pismenny and Yossi Kuperman

July 11th -13th , 2018

© 2018 Mellanox Technologies | Confidential 1

UDP Segmentation Motivation
 QUIC - A new network protocol to rule them all
• Combines TCP, TLS, and HTTP

 Deployed by Google and Akamai

 10% of internet traffic is over QUIC

 Solves Head-Of-The-Line blocking

 The QUIC transport runs over UDP

© 2018 Mellanox Technologies | Confidential 2

UDP Segmentation uAPI
• User sends a large UDP message
with the MSS as UDP_SEGMENT
metadata

• Kernel segments the large

message to MSS (+headers) sized
packets

• No encapsulations at the moment

Note: This code is part of tools/testing/selftests/net/udpgso_bench_tx
written by Willem de Bruijn

© 2018 Mellanox Technologies | Confidential 3

UDP Segmentation Offload
Problem: HW was not built with UDP LSO offload in mind 

Could we provide new offloads using existing hardware (e.g.

ConnectX-4 and Connetx-5)?

© 2018 Mellanox Technologies | Confidential 4

 HW – Existing Segmentation Support
 Building a Packet:
• Duplicate headers to all packets.
• Segment payload to multiple packets.

 Segmentation offload:
• Increment IP identification counter each packet.
• UDP Length calculation.
• Perform IP and UDP checksum.

© 2018 Mellanox Technologies | Confidential 5

 HW – Existing Segmentation Support
 Building a Packet:
• Duplicate headers to all packets.
• Segment payload to multiple packets.

 Segmentation offload:
• Increment IP identification counter each packet.
• UDP Length calculation.
• Perform IP and UDP checksum.

 Missing Segmentation offload:

• Calculating UDP checksum while original UDP
length is incorrect.
© 2018 Mellanox Technologies | Confidential 6
Workaround to missing HW feature
 Split Message to two different descriptors Segment 0

Segment 1
 Adjust headers
• Fix length Segment 2

• Adjust last segment IP ID

Segment 3 SendLSO

Segment 4

Segment 5

Segment 6

Segment 7 Send
© 2018 Mellanox Technologies | Confidential 7
Workaround to missing HW feature
 Split Message to two different descriptors Segment 0

Segment 1
 Adjust headers
• Fix length Segment 2

• Adjust last segment IP ID

Segment 3 SendLSO

 This functionality is already provided by Segment 4

GSO_PARTIAL
Segment 5
• Still need to adjust the UDP total length
Segment 6

Segment 7 Send
© 2018 Mellanox Technologies | Confidential 8
UDP Single Stream Performance
40 100
line rate
35 90
1.5X improvement

CPU Utilization (%)

80
Throughput Gb/s

30 with half the CPU

utilization 70
25 3X improvement 60
20 with comparable 50
CPU utilization
15 40
30
10
20
5 10
0 0
64 256 512 1024 1472
Maximum Segment Size
UDP CPU (TX) UDP SW Segmentation CPU (TX) UDP HW Segmentation CPU (TX)

UDP UDP SW Segmentation UDP HW Segmentation

© 2018 Mellanox Technologies | Confidential 9

Summary
 Old dog learns new trick – UDP Segmentation is supported with
ConnectX-4 and ConnectX-5.

 UDP Segmentation Offload improves performance by 1.5x-10x

 Full support (not GSO_PARTIAL) in future HW requires only a small

change

Timeline:
 Software offload introduced in April 14th
 RFC for Intel NICs May 3rd
 Patches for Mellanox NICs upstream since June 29th © 2018 Mellanox Technologies | Confidential 10
Discussion
 Maybe GSO partial UDP Segmentation should provide the segmented length and
not the unsegmented length?
• Currently the IP length and the UDP length of the partial segment do not match
(e.g. IP.total_length = 1500 UDP.length = 15K)

 What other offloads could improve UDP/QUIC performance?

• How about UDP De-Segmentation?
- What is the uAPI for that?
• QUIC RSS based on connection ID?
- What is the interface to configure it?

 Are there any plans to include some part of QUIC in the Linux kernel?
• For QUIC inline crypto offload the kernel must be involved
© 2018 Mellanox Technologies | Confidential 11