ipq806x hardware acceleration
TRANSCRIPT
IPQ806x Hardware acceleration
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
• Features• Designed for Home Gateways (CPE)• Flow detection based “All-or-nothing”
offload• Acceleration supports:
• IPv4, IPv6, NAT, PPPoE, L2TP, VLAN, Qdisc
• Performance gain• Linux: 640k pps (bridged) – 220kpps (routed)• NSS: 7200k pps (bridged) – 7200kpps (routed)• 11x (bridged) – 32x (routed)
• Functional behavior• 0% cpu load seen in Linux• Keep Linux counters up to date• Does not require functional changes at an
upper level (user space)
NSS acceleration model
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
• Inspect all outgoing packets at POSTROUTING chain by registering post routing hooks.• Inspect conntrack and device events to destroy and regenerate connection.• Inspect NSS status and stats to update the connection state and statistics info in Linux and ECM DB.
ECM Front End Inspect Packets and Events
FRONT ENDNETFILTERPF_BRIDGE
POST ROUTING
L2 PKTs
L2 PKTs
NETFILTERIPvX
POST ROUTING L3 PKTs
L3 PKTs
INTERFACENOTIFIER
MTU / UP/ DOWN /
BOND NOTIFIER
Bond Link/ Release/Enslave
NSS Driver
NSS Status and Stats Sync
NSS Firmware
LINUX CONNTRACK
IPCT_DESTROY/ IPCT_MARK
CONNTRACK NOTIFIER
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
Example of IPv4 TCP rule creation
NSS Firmware
Linux
TCP Data
The Accelerated TCP flow
TCP: ACKOr Syn+ACK
Transmit the packet
Add the rule & reply the establish cmd
Match Any Accelerated
Rule?
No
ECM
Send the Packet to Linux
CreateIPv4 Rule
Packet
Linux net stack
NetfiterPrerouting
Routing or Bridging
NetfiterPostrouting
NSS Driver
Connection Established
Check TCP Connection can be
accel?
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
Example of IPv4 TCP rule destroy
NSS Firmware
Linux Kernel
TCP Data
The Accelerated TCP flow
TCP: FIN Packet
Send the Packet to Linux
Transmit The packet
DestroyIPv4 Rule
IPv4 RuleSync
Add the rule & reply the establish cmdIs it a
connected TCP Data?
No
Linux net stack
NetfiterPrerouting
Routing / Bridging
NetfiterPostrouting
ECM Remove the connection
NSS Driver
ConntrackConntrack destroy event
ConnectionDestroy ?
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
Example of IPv4 rule API• Common messaging interfaces
• Protocol type: IPv4, IPv6, PPP, LAG…
• Rule type: CREATE, DESTROY, CONN_STAT_SYNC , NODE_STAT_SYNC…
• Callback/args pointers: will be passed back in the FW ACK/NACK reply
• IPv4 rule create message structure example
struct nss_ipv4_rule_create_msg {/* Request */uint16_t valid_flags; /* Indicate which of the parameters below is filled-in
Indirectly says which operation to be done on the flow */uint16_t rule_flags; /* Bit flags associated with the rule */struct nss_ipv4_5tuple tuple; /* src_ip, dst_ip, src_port, dst_port, proto */struct nss_ipv4_connection_rule conn_rule; /* src_mac, dst_mac, src_iface, dst_iface, src_mtu, dst_mtu,
nat_src_ip, nat_dst_ip, nat_src_port, nat_dst_port */struct nss_ipv4_protocol_tcp_rule tcp_rule; /* TCP related accleration parameters */struct nss_ipv4_pppoe_rule pppoe_rule; /* flow_session_id, flow_remote_mac, ret_session_id, ret_remote_mac */struct nss_ipv4_qos_rule qos_rule; /* flow_qos_tag, ret_qos_tag, */struct nss_ipv4_dscp_rule dscp_rule; /* flow_dscp, ret_dstp */struct nss_ipv4_vlan_rule vlan_primary_rule; /* ingress_vlan_tag, egress_vlan_tag */struct nss_ipv4_vlan_rule vlan_secondary_rule; /* ingress_vlan_tag, egress_vlan_tag – for QinQ */
/* Response */uint32_t index; /*Slot ID for cache stats to host OS */
};
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
Linux
ECM
Interfaces & Connections statistics update• Stats updates sent
periodically from the Firmware• Per-interfaces stats
(update net_devices)
• Per-connections stats (update conntracks)
• Minor modifications to ppp/l2tp/ipsec… layers for iface look-up and stats update
NSS Driver
NODE_STATS_SYNCmessage
CONN_STATS_SYNCmessage
ppp0
ethN
tun0
br0
Look-up interface
NSS Firmware
Look-up conntrack
stats++
conntracksstats++
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
Qdisc acceleration
# tc qdisc add dev eth0 root handle 1: tbf rate 1000Mbit burst 100k limit 100# tc qdisc add dev eth0 parent 1: handle 10: prio bands 3# tc qdisc add dev eth0 parent 10:1 handle 100: tbf rate 2Mbit burst 10k limit 100# tc qdisc add dev eth0 parent 100: handle 1000: pfifo limit 100# tc qdisc add dev eth0 parent 10:2 handle 200: tbf rate 40Mbit burst 30k limit 100# tc qdisc add dev eth0 parent 200: handle 2000: red limit 100k min 30k max 80k \avpkt 1k burst 55 probability 0.20# tc qdisc add dev eth0 parent 10:3 handle 300: red limit 100k min 30k max 80k \avpkt 1k burst 55 probability 0.30
tbf
tbf tbf
r
e
d
prio
r
e
d
p
f
i
f
o
1:
10:
100: 200:
300:
1000: 2000:
10:3
10:210:1
nsstbf
nsstbl nsstbl
n
s
s
r
e
d
nssprio
n
s
s
r
e
d
n
s
s
p
f
i
f
o
1:
10:
100: 200:300:
1000: 2000:
10:310:210:1
# tc qdisc add dev eth0 root handle 1: nsstbl rate 1000Mbit burst 100k# tc qdisc add dev eth0 parent 1: handle 10: nssprio bands 3# tc qdisc add dev eth0 parent 10:1 handle 100: nsstbl rate 2Mbit burst 10k# tc qdisc add dev eth0 parent 100: handle 1000: nsspfifo limit 100# tc qdisc add dev eth0 parent 10:2 handle 200: nsstbl rate 40Mbit burst 30k# tc qdisc add dev eth0 parent 200: handle 2000: nssred limit 100k min 30k max 80k \avpkt 1k burst 55 probability 0.50# tc qdisc add dev eth0 parent 10:3 handle 300: nssred limit 100k min 30k max 80k \avpkt 1k burst 55 probability 0.50 set_default
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada