![Page 1: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/1.jpg)
Mick Tarsel [email protected] Networking Team, IBMMarch 2016
OpenVSwitch and Trace Points: Getting Started in Open Source
![Page 2: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/2.jpg)
Docker/OVS Project SetupWhere to startHelpful tools & tipsTrace eventsTracing packetsClosing thoughts
Agenda
![Page 3: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/3.jpg)
OVS and Docker on GNU/LinuxSporadic latency of ARP packets in slow path
Project Overview
OVS Kernel Module
ovs-vswitchdUser space
Kernel space
OVS Slow Path
![Page 4: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/4.jpg)
Read, a lot.Learn git
–git clone https://github.com/openvswitch/ovsDon't be overwhelmed
–Stick with small chunksLearn about tools
Where to Start?
![Page 5: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/5.jpg)
Knowing your environmentcscope
– http://cscope.sourceforge.net/
Narrowing it Down
cscope screenshot
![Page 6: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/6.jpg)
Tcpdump?– I want to ID a packet
SystemTaphttps://sourceware.org/systemtap/
– Events (upcall) and handlers (tag it)
– Kprobes – dynamic tracing
perf– Trace points – static event tracing
• /Documentation/trace/tracepoints.txt
Measuring Packet Latency?
![Page 7: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/7.jpg)
Record local variables & functions
Fast tracing
Requires some setup (not too simple)
Trace Points
![Page 8: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/8.jpg)
http://lwn.net/Articles/379903/– “To solve this issue of automating the tracepoints, the
TRACE_EVENT() macro was born.” - Rostedt
Trace event must:– Create trace point
– Create call back function
– Record data
– Parse data
Output goes to /sys/kernel/debug/tracing/
Usable from user space
Trace events
![Page 9: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/9.jpg)
My OVS Trace Event
![Page 10: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/10.jpg)
Calling the Trace Event
![Page 11: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/11.jpg)
Following Packets
OVS Kernel Module
ovs-vswitchd
User space
Kernel space
trace_upcall_end(packet, report_packet(packet));
trace_upcall_start(skb, tag_packet(skb));
![Page 12: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/12.jpg)
=== ARP Packets === id Time in Nano Seconds
Max: 5 4721362.0Min: 6 64742.0Average: 883686.666667Total ARP Packets: 6
=== IP Packets === id Time in Nano Seconds
Max: 3 95160.0Min: 8 44850.0
Average: 69619.0Total IP Packets: 4
=== All Captured Packets === id Time in Nano Seconds Protocol
Max: 5 4721362.0 0x806Min: 8 44850.0 0x800
Average: 558059.6Total Packets captured: 10
Output# perf record -e openvswitch:upcall_start -e openvswitch:upcall_end -a
![Page 13: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/13.jpg)
Results - ARPTime: COUNT MIN MAX AVERAGE-----------------------------------------------------------------------------RTT 1914 0 2001 1058K-2-OVS 1914 0 1995 1013OVS-2-K 1914 0 914 45-----------------------------------------------------------------------------
base-2 logarithmic histogram of round trip timesvalue |-------------------------------------------------- count 0 | 21 1 | 17 2 | 11 4 | 1 8 | 4 16 | 15 32 |@ 39 64 |@@@ 87 128 |@@@@@@ 132 256 |@@@@@@@@ 184 512 |@@@@@@@@@@@@@@ 325 1024 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1078 2048 | 0 4096 | 0
Count = number of packetsValue = time in Jiffies
![Page 14: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/14.jpg)
Results - IPv4Time: COUNT MIN MAX AVERAGE-----------------------------------------------------------------------------RTT 230 0 53 5K-2-OVS 230 0 44 2OVS-2-K 230 0 50 2-----------------------------------------------------------------------------
base-2 logarithmic histogram of round trip timesvalue |-------------------------------------------------- count 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 102 1 |@@@@@@@@@@@@@@@@@ 53 2 |@@@@@@ 20 4 |@@@@ 14 8 |@@@ 10 16 |@@@@@@ 18 32 |@@@@ 13 64 | 0 128 | 0
Count = number of packetsValue = time in Jiffies
![Page 15: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/15.jpg)
Extend packet tracing tools to other projects
ovs-benchmark tool
Future Plans
![Page 16: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/16.jpg)
IBM LTC
LTC Networking Team– Pradeep Satyanarayana
– Dave Wilder
Acknowledgments
![Page 17: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/17.jpg)
These are things that helped me....– Stay organized with git
– READ
– Don't be intimidated
– Subscribe to mailing lists and answer questions
– Use tools
Closing thoughts
![Page 18: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/18.jpg)
Questions?
Thank you!
![Page 19: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/19.jpg)
sk_buff, sk_buffer, skb
Socket Buffer
Source image: http://vger.kernel.org/~davem/skb_data.html
![Page 20: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/20.jpg)
tag_packet()
![Page 21: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/21.jpg)
report_packet()
![Page 22: OpenVSwitch and Trace Points: Getting Started in Open Source](https://reader034.vdocuments.us/reader034/viewer/2022042611/589d95e51a28aba9498bc7be/html5/thumbnails/22.jpg)
upcall_end trace point