Listifyed (UDP) GRO

Listifyed (UDP) GRO

Steffen Klassert

secunet Security Networks AG

Dresden

Linux Netconf, Boston, June, 2019

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Generic Receive Offload (GRO)

I Software packet aggregation on Layer 2

I Merges packets of a flow into one big packet (up to 64 KB)

I Reduces the number of network stack traversals

I Good performance if a local socket can handle GRO

I Good performance on forwarding if the TX NIC can do LSO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Generic Receive Offload (GRO)

I Software packet aggregation on Layer 2

I Merges packets of a flow into one big packet (up to 64 KB)

I Reduces the number of network stack traversals

I Good performance if a local socket can handle GRO

I Good performance on forwarding if the TX NIC can do LSO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Generic Receive Offload (GRO)

I Software packet aggregation on Layer 2

I Merges packets of a flow into one big packet (up to 64 KB)

I Reduces the number of network stack traversals

I Good performance if a local socket can handle GRO

I Good performance on forwarding if the TX NIC can do LSO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Generic Receive Offload (GRO)

I Software packet aggregation on Layer 2

I Merges packets of a flow into one big packet (up to 64 KB)

I Reduces the number of network stack traversals

I Good performance if a local socket can handle GRO

I Good performance on forwarding if the TX NIC can do LSO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Generic Receive Offload (GRO)

I Software packet aggregation on Layer 2

I Merges packets of a flow into one big packet (up to 64 KB)

I Reduces the number of network stack traversals

I Good performance if a local socket can handle GRO

I Good performance on forwarding if the TX NIC can do LSO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Generic Receive Offload (GRO)

I Software packet aggregation on Layer 2

I Merges packets of a flow into one big packet (up to 64 KB)

I Reduces the number of network stack traversals

I Good performance if a local socket can handle GRO

I Good performance on forwarding if the TX NIC can do LSO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Generic Receive Offload (GRO)

I Software packet aggregation on Layer 2

I Merges packets of a flow into one big packet (up to 64 KB)

I Reduces the number of network stack traversals

I Good performance if a local socket can handle GRO

I Good performance on forwarding if the TX NIC can do LSO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Problematic GRO usecases

I Does not perform well if GSO is neededI GRO and GSO add overhead

I calculating checksumsI modifies headersI touches packet data

I UDP needs needs software GSO if NIC can’t do LSO

I IPsec needs software GSO if NIC can’t offload ESP

I Is there a better way to do GRO if software GSO is needed?

I Yes: Listifyed GRO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Problematic GRO usecases

I Does not perform well if GSO is neededI GRO and GSO add overhead

I calculating checksumsI modifies headersI touches packet data

I UDP needs needs software GSO if NIC can’t do LSO

I IPsec needs software GSO if NIC can’t offload ESP

I Is there a better way to do GRO if software GSO is needed?

I Yes: Listifyed GRO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Problematic GRO usecases

I Does not perform well if GSO is neededI GRO and GSO add overhead

I calculating checksumsI modifies headersI touches packet data

I UDP needs needs software GSO if NIC can’t do LSO

I IPsec needs software GSO if NIC can’t offload ESP

I Is there a better way to do GRO if software GSO is needed?

I Yes: Listifyed GRO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Problematic GRO usecases

I Does not perform well if GSO is neededI GRO and GSO add overhead

I calculating checksumsI modifies headersI touches packet data

I UDP needs needs software GSO if NIC can’t do LSO

I IPsec needs software GSO if NIC can’t offload ESP

I Is there a better way to do GRO if software GSO is needed?

I Yes: Listifyed GRO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Problematic GRO usecases

I Does not perform well if GSO is neededI GRO and GSO add overhead

I calculating checksumsI modifies headersI touches packet data

I UDP needs needs software GSO if NIC can’t do LSO

I IPsec needs software GSO if NIC can’t offload ESP

I Is there a better way to do GRO if software GSO is needed?

I Yes: Listifyed GRO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Problematic GRO usecases

I Does not perform well if GSO is neededI GRO and GSO add overhead

I calculating checksumsI modifies headersI touches packet data

I UDP needs needs software GSO if NIC can’t do LSO

I IPsec needs software GSO if NIC can’t offload ESP

I Is there a better way to do GRO if software GSO is needed?

I Yes: Listifyed GRO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Problematic GRO usecases

I Does not perform well if GSO is neededI GRO and GSO add overhead

I calculating checksumsI modifies headersI touches packet data

I UDP needs needs software GSO if NIC can’t do LSO

I IPsec needs software GSO if NIC can’t offload ESP

I Is there a better way to do GRO if software GSO is needed?

I Yes: Listifyed GRO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Problematic GRO usecases

I Does not perform well if GSO is neededI GRO and GSO add overhead

I calculating checksumsI modifies headersI touches packet data

I UDP needs needs software GSO if NIC can’t do LSO

I IPsec needs software GSO if NIC can’t offload ESP

I Is there a better way to do GRO if software GSO is needed?

I Yes: Listifyed GRO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Problematic GRO usecases

I Does not perform well if GSO is neededI GRO and GSO add overhead

I calculating checksumsI modifies headersI touches packet data

I UDP needs needs software GSO if NIC can’t do LSO

I IPsec needs software GSO if NIC can’t offload ESP

I Is there a better way to do GRO if software GSO is needed?

I Yes: Listifyed GRO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Problematic GRO usecases

I Does not perform well if GSO is neededI GRO and GSO add overhead

I calculating checksumsI modifies headersI touches packet data

I UDP needs needs software GSO if NIC can’t do LSO

I IPsec needs software GSO if NIC can’t offload ESP

I Is there a better way to do GRO if software GSO is needed?

I Yes: Listifyed GRO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Listifyed GRO

I Does not merge packets into a big packet

I Builds a linked list of packets

I Linked packets travel together though the stack

I GSO just needs to unlink the packets

I Listifyed GRO/GSO does not need to touch packet data

I Listifyed GRO/GSO performs better than standard GRO/GSO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Listifyed GRO

I Does not merge packets into a big packet

I Builds a linked list of packets

I Linked packets travel together though the stack

I GSO just needs to unlink the packets

I Listifyed GRO/GSO does not need to touch packet data

I Listifyed GRO/GSO performs better than standard GRO/GSO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Listifyed GRO

I Does not merge packets into a big packet

I Builds a linked list of packets

I Linked packets travel together though the stack

I GSO just needs to unlink the packets

I Listifyed GRO/GSO does not need to touch packet data

I Listifyed GRO/GSO performs better than standard GRO/GSO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Listifyed GRO

I Does not merge packets into a big packet

I Builds a linked list of packets

I Linked packets travel together though the stack

I GSO just needs to unlink the packets

I Listifyed GRO/GSO does not need to touch packet data

I Listifyed GRO/GSO performs better than standard GRO/GSO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Listifyed GRO

I Does not merge packets into a big packet

I Builds a linked list of packets

I Linked packets travel together though the stack

I GSO just needs to unlink the packets

I Listifyed GRO/GSO does not need to touch packet data

I Listifyed GRO/GSO performs better than standard GRO/GSO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Listifyed GRO

I Does not merge packets into a big packet

I Builds a linked list of packets

I Linked packets travel together though the stack

I GSO just needs to unlink the packets

I Listifyed GRO/GSO does not need to touch packet data

I Listifyed GRO/GSO performs better than standard GRO/GSO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Listifyed GRO

I Does not merge packets into a big packet

I Builds a linked list of packets

I Linked packets travel together though the stack

I GSO just needs to unlink the packets

I Listifyed GRO/GSO does not need to touch packet data

I Listifyed GRO/GSO performs better than standard GRO/GSO

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Max UDP Throughput in Mbps (single flow / 30 sec)

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Avg Latency at Max Throughput

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Min Latency at Max Throughput

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Max Latency at Max Throughput

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Status of listifyed GRO

I Implemented for UDP (RFC state)I TCP forwarding could benefit too if software GSO is needed

I Should listifyed GRO support TCP too?

I Problem: How to decide between standard and listifyed GRO?I Global knob to enable/disableI Do a route lookup at GRO layer

I Gives a good guess on output NIC and xfrmI But: Could be rerouted by netfilter etc.

I Add some TC/NF hook before GRO that can add route infosI NF flowtable?I eBPF programm?

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Status of listifyed GRO

I Implemented for UDP (RFC state)I TCP forwarding could benefit too if software GSO is needed

I Should listifyed GRO support TCP too?

I Problem: How to decide between standard and listifyed GRO?I Global knob to enable/disableI Do a route lookup at GRO layer

I Gives a good guess on output NIC and xfrmI But: Could be rerouted by netfilter etc.

I Add some TC/NF hook before GRO that can add route infosI NF flowtable?I eBPF programm?

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Status of listifyed GRO

I Implemented for UDP (RFC state)I TCP forwarding could benefit too if software GSO is needed

I Should listifyed GRO support TCP too?

I Problem: How to decide between standard and listifyed GRO?I Global knob to enable/disableI Do a route lookup at GRO layer

I Gives a good guess on output NIC and xfrmI But: Could be rerouted by netfilter etc.

I Add some TC/NF hook before GRO that can add route infosI NF flowtable?I eBPF programm?

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Status of listifyed GRO

I Implemented for UDP (RFC state)I TCP forwarding could benefit too if software GSO is needed

I Should listifyed GRO support TCP too?

I Problem: How to decide between standard and listifyed GRO?I Global knob to enable/disableI Do a route lookup at GRO layer

I Gives a good guess on output NIC and xfrmI But: Could be rerouted by netfilter etc.

I Add some TC/NF hook before GRO that can add route infosI NF flowtable?I eBPF programm?

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Status of listifyed GRO

I Implemented for UDP (RFC state)I TCP forwarding could benefit too if software GSO is needed

I Should listifyed GRO support TCP too?

I Problem: How to decide between standard and listifyed GRO?I Global knob to enable/disableI Do a route lookup at GRO layer

I Gives a good guess on output NIC and xfrmI But: Could be rerouted by netfilter etc.

I Add some TC/NF hook before GRO that can add route infosI NF flowtable?I eBPF programm?

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Status of listifyed GRO

I Implemented for UDP (RFC state)I TCP forwarding could benefit too if software GSO is needed

I Should listifyed GRO support TCP too?

I Problem: How to decide between standard and listifyed GRO?I Global knob to enable/disableI Do a route lookup at GRO layer

I Gives a good guess on output NIC and xfrmI But: Could be rerouted by netfilter etc.

I Add some TC/NF hook before GRO that can add route infosI NF flowtable?I eBPF programm?

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Status of listifyed GRO

I Implemented for UDP (RFC state)I TCP forwarding could benefit too if software GSO is needed

I Should listifyed GRO support TCP too?

I Problem: How to decide between standard and listifyed GRO?I Global knob to enable/disableI Do a route lookup at GRO layer

I Gives a good guess on output NIC and xfrmI But: Could be rerouted by netfilter etc.

I Add some TC/NF hook before GRO that can add route infosI NF flowtable?I eBPF programm?

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Status of listifyed GRO

I Implemented for UDP (RFC state)I TCP forwarding could benefit too if software GSO is needed

I Should listifyed GRO support TCP too?

I Problem: How to decide between standard and listifyed GRO?I Global knob to enable/disableI Do a route lookup at GRO layer

I Gives a good guess on output NIC and xfrmI But: Could be rerouted by netfilter etc.

I Add some TC/NF hook before GRO that can add route infosI NF flowtable?I eBPF programm?

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Status of listifyed GRO

I Implemented for UDP (RFC state)I TCP forwarding could benefit too if software GSO is needed

I Should listifyed GRO support TCP too?

I Problem: How to decide between standard and listifyed GRO?I Global knob to enable/disableI Do a route lookup at GRO layer

I Gives a good guess on output NIC and xfrmI But: Could be rerouted by netfilter etc.

I Add some TC/NF hook before GRO that can add route infosI NF flowtable?I eBPF programm?

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Status of listifyed GRO

I Implemented for UDP (RFC state)I TCP forwarding could benefit too if software GSO is needed

I Should listifyed GRO support TCP too?

I Problem: How to decide between standard and listifyed GRO?I Global knob to enable/disableI Do a route lookup at GRO layer

I Gives a good guess on output NIC and xfrmI But: Could be rerouted by netfilter etc.

I Add some TC/NF hook before GRO that can add route infosI NF flowtable?I eBPF programm?

Listifyed (UDP) GRO

Generic Receive Offload (GRO)

Status of listifyed GRO

I Implemented for UDP (RFC state)I TCP forwarding could benefit too if software GSO is needed

I Should listifyed GRO support TCP too?

I Problem: How to decide between standard and listifyed GRO?I Global knob to enable/disableI Do a route lookup at GRO layer

I Gives a good guess on output NIC and xfrmI But: Could be rerouted by netfilter etc.

I Add some TC/NF hook before GRO that can add route infosI NF flowtable?I eBPF programm?