netmap presentation
TRANSCRIPT
FreeBSD Netmap
Amir [email protected]
Intelligent Communication and Information Systems LaboratoryFlorida Tech
What’s netmap?
• Allows line-rate speed on commodity hardware and NICs (14.88 Mpps on 10G 1.48 Mpps on 1G)
• Simple API but still allows to utilize hardware specific features (multi-queue NICs).
• No need to mess up with Kernel coding, Netmap applications can easily be ran in userland.
• Full control over host stack.
Conventional Operating Systems
Packet Journey in Kernel
ether_input(struct ifnet *ifp, struct ether_header *eh,
struct mbuf *m) {
m->m_pkthdr.rcvif = ifp;
eh = mtod(m, struct ether_header *);
m->m_data += sizeof(struct ether_header);
m->m_len -= sizeof(struct ether_header);
m->m_pkthdr.len = m->m_len;
ether_demux(ifp, eh, m);
}
void ether_demux(ifp, eh, m)
ether_type = ntohs(eh>ether_type);
switch (ether_type) {
case ETHERTYPE_IP:
. . .
break;
schednetisr(NETISR_IP);
inq = &ipintrq;
(void) IF_HANDOFF(inq, m, NULL);
Packet Journey in Kernel
//ISR for IPwhile(1) {
ip_input(m);s = splimp();IF_DEQUEUE(&ipintrq, m);splx(s);if (m == 0)
return;ip_input(m);
}
void ip_input(struct mbuf *m) {if (m->m_len < sizeof (struct ip) &&(m = m_pullup(m, sizeof (struct ip))) == 0) {
}ip = mtod(m, struct ip *);ipstat.ips_toosmall++;return;
Network I/O costs
• per-byte cost: amount of traffic - copying,checksum computation, encryption
• per-packet cost: comes from the manipulationof descriptors (allocation and destruction,metadata management) and the execution ofsystem calls, interrupts, and device-driverfunctions…
• The larger packets the less per-packet cost.
Sockets Bottlenecks
• IPC Communication “System Calls”.
• Parsing RAW Ethernet frames into different data structure at different layers, mbufs, skubuff.
• Memory Copy (inter layers)
• Memory Allocation/Deallocation
Design Decisions 30 years ago.
• memory was a scarce resource;
• links operated at low (by today’s standards) speeds;
• parallel processing was an advanced research topic; and the ability to work at line rate in
• all possible conditions was compromised by hardware limitations in the NIC
Packet Handling Costs
UDP
System Call
mbuf
Memory Copy
Hardware
TCP
What’s netmap
• “Netmap is a novel framework that employs some known techniques to reduce packet-processing costs. Its key feature, apart from performance, is that it integrates smoothly with existing operating-system internals and applications. This makes it possible to achieve great speedups with just a limited amount of new code, while building a robust and maintainable system.”
Netmap goals
• Reduce system calls
• Remove allocation costs
• Eliminate data copies.
The inspiration for data structure design comes from the way that network interface stores buffers.
Netmap Data Structure
Safe slots: [curr … curr + available – 1]
netmap API
• Access– open(‘’/dev/netmap’’);– ioctl(fd, NIOCREG, arg)– mmap(…, fd, 0) maps buffers and rings.
• Transmit– Fill to avail buffers, starting from slot cur.– IOCTL NIOCTXSYNC
• Receive– IOCTL NIOCRXSYNC– Process up to avail slots from slot cur
Example
Multi Queue NIC
• Multiple Descriptor Queues – queues can be associated with specific processor
cores
• Receive-Side Scaling (RSS) – To determine which receive queue to use for incoming
packets,
– specific flow for a given packet is determined by the calculation of a hash value derived from fields
• Extended Message-Signaled Interrupts (MSI-X) – Multiple interrupt vectors
Host Stack Access
• In netmap mode host stack is disconnected from NIC.
• But the operating system still can use NIC.• There’s two additional rings for incoming
outgoing traffic of host stack.• Packets destinated to the host stack are
queued netmap and then and passed to the host stack and vice versa.
• The approach provides an ideal solution for traffic filters, manipulators and etc.
Lightening fast packet forwarding
...
src = &src_nifp->slot[i]; /* locate src and dst slots */
dst = &dst_nifp->slot[j];
/* swap the buffers */
tmp = dst->buf_index;
dst->buf_index = src->buf_index;
src->buf_index = tmp;
/* update length and flags */
dst->len = src->len;
/* tell kernel to update addresses in the NIC rings */
dst->flags = src->flags = BUF_CHANGED;
...
/ conditional part
Comparsion