socket direct protocol over pci express interconnect: design, implementation and evaluation
DESCRIPTION
Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation. By Ahmed Bukhamsin. Oct 2012. 1. Outline. PCI Express (PCIe) Overview PCIe Interconnect Proposed System Implementation Evaluation Conclusion. School of Computing Science / Ahmed Bukhamsin. 2. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/1.jpg)
Oct 2012
Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation
By Ahmed Bukhamsin
1
![Page 2: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/2.jpg)
School of Computing Science / Ahmed Bukhamsin
Outline
• PCI Express (PCIe) Overview• PCIe Interconnect• Proposed System• Implementation• Evaluation• Conclusion
2
![Page 3: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/3.jpg)
School of Computing Science / Ahmed Bukhamsin
PCI Express Overview 1
• PCI = Peripheral Component Interconnect
3
![Page 4: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/4.jpg)
School of Computing Science / Ahmed Bukhamsin
PCI Express Overview 2
4
![Page 5: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/5.jpg)
School of Computing Science / Ahmed Bukhamsin
PCI Express Overview 3
• PCIe Evolution• High throughput• Low latency• Low power consumption• PCIe over cable
* Aggregate bandwidth implies simultaneous traffic in both directions.
5
![Page 6: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/6.jpg)
School of Computing Science / Ahmed Bukhamsin
PCIe Interconnect 1
• Issues with connecting two root complexes• Bus ownership• Endpoints resource allocation
• Solution: Non-transparent bridge (NTB)
6
![Page 7: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/7.jpg)
School of Computing Science / Ahmed Bukhamsin
PCIe Interconnect 2
• NTB isolates the connected PCIe domains• NTB provides a communication mechanism between the connected PCIe
domains
7
![Page 8: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/8.jpg)
School of Computing Science / Ahmed Bukhamsin
PCIe Interconnect 3
• NTB communication mechanism • Acquires memory in both systems• Provides address translation to allow data exchange• Provides doorbell and scratchpad registers accessible from both sides
8
![Page 9: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/9.jpg)
School of Computing Science / Ahmed Bukhamsin
Proposed System 1
• We propose to develop the Socket Direct Protocol (SDP)
• SDP makes it possible to run TCP based socket applications without modifications
9
![Page 10: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/10.jpg)
School of Computing Science / Ahmed Bukhamsin
Proposed System 2
10
Applications
SDP Socket Preloaded Library
Protocol Switch
SDP
TCP
IP
Communication API
NTB Driver
User Space
Kernel Space
NIC Driver
PCIe NTB CardEthernet NIC
DMA Driver
![Page 11: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/11.jpg)
School of Computing Science / Ahmed Bukhamsin
Proposed System 2
• SDP Socket Preloaded Library• Rule based protocol switch
11
Applications
SDP Socket Preloaded Library
Protocol Switch
SDP
TCP
IP
Communication API
NTB Driver
User Space
Kernel Space
NIC Driver
PCIe NTB CardEthernet NIC
DMA Driver
![Page 12: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/12.jpg)
School of Computing Science / Ahmed Bukhamsin
Proposed System 2
• SDP Socket Preloaded Library• Rule based protocol switch
• SDP Kernel Module• The main component of the solution• Register itself as new address family• Implements all TCP functions
12
Applications
SDP Socket Preloaded Library
Protocol Switch
SDP
TCP
IP
Communication API
NTB Driver
User Space
Kernel Space
NIC Driver
PCIe NTB CardEthernet NIC
DMA Driver
![Page 13: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/13.jpg)
School of Computing Science / Ahmed Bukhamsin
Proposed System 2
• SDP Socket Preloaded Library• Rule based protocol switch
• SDP Kernel Module• The main component of the solution• Register itself as new address family• Implements all TCP functions
• Communication API• Group of functions exported by the
driver for SDP to access the driver
13
Applications
SDP Socket Preloaded Library
Protocol Switch
TCP
IP
Communication API
NTB Driver
User Space
Kernel Space
NIC Driver
PCIe NTB CardEthernet NIC
DMA Driver
SDP
![Page 14: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/14.jpg)
School of Computing Science / Ahmed Bukhamsin
Proposed System 2
14
Applications
SDP Socket Preloaded Library
Protocol Switch
TCP
IP
Communication API
NTB Driver
User Space
Kernel Space
NIC Driver
PCIe NTB CardEthernet NIC
DMA Driver
SDP
• SDP Socket Preloaded Library•Rule based protocol switch
• SDP Kernel Module• The main component of the solution•Register itself as new address family• Implements all TCP functions
• Communication API•Group of functions exported by the
driver for SDP to access the driver
• DMA and NTB Drivers•Cards manufacturer drivers
![Page 15: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/15.jpg)
School of Computing Science / Ahmed Bukhamsin
Implementation 1
• SDP Connection
Server Client
- Listen()
- accept()
Scratchpad registers
- Connect()
Interrupt
Create client Buffer
- Read registersServer BufferCreate server Buffer
IP and Port
Interrupt
Connection establishedInterrupt
Connection established
Server port
Ready
Conniction ID
client buf addr
Client BufferServer buff addr
-Read Reg.
15
![Page 16: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/16.jpg)
School of Computing Science / Ahmed Bukhamsin
Implementation 2
• Send and Receive
Data Source Data SinkTime
recv 1
Data Ready Signal
DMA Data Copy
Wa
itin
g
sendmsg 1
...
sendmsg 2sendmsg 3
sendmsg n
copy 1
recv 2
copy 2
recv 3
copy 3
...
recv n
copy n
SDP Buf SDP BufApplication Application
... ... ...
...
16
![Page 17: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/17.jpg)
School of Computing Science / Ahmed Bukhamsin
Evaluation 1
• Benchmarks• Throughput• Latency• CPU Utilization
• Application Testing• Apache Web Server
17
![Page 18: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/18.jpg)
School of Computing Science / Ahmed Bukhamsin
Evaluation 2
• Throughput
18
![Page 19: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/19.jpg)
School of Computing Science / Ahmed Bukhamsin
Evaluation 3
• Throughput
19
![Page 20: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/20.jpg)
School of Computing Science / Ahmed Bukhamsin
Evaluation 3
• Throughput
20
![Page 21: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/21.jpg)
School of Computing Science / Ahmed Bukhamsin
Evaluation 3
• Throughput
21
![Page 22: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/22.jpg)
School of Computing Science / Ahmed Bukhamsin
Evaluation 3
• Throughput
22
![Page 23: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/23.jpg)
School of Computing Science / Ahmed Bukhamsin
Evaluation 4
• Latency
23
![Page 24: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/24.jpg)
School of Computing Science / Ahmed Bukhamsin
Evaluation 5
• Application
24
![Page 25: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/25.jpg)
School of Computing Science / Ahmed Bukhamsin
Evaluation 5
• Application
25
![Page 26: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/26.jpg)
School of Computing Science / Ahmed Bukhamsin
Evaluation 5
• Application
26
![Page 27: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/27.jpg)
School of Computing Science / Ahmed Bukhamsin
Evaluation 5
• Application
27
![Page 28: Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation](https://reader035.vdocuments.us/reader035/viewer/2022062315/56815870550346895dc5d0a1/html5/thumbnails/28.jpg)
School of Computing Science / Ahmed Bukhamsin
Conclusion
• We design and implement SDP• We evaluate it and compare it to SDP over Infiniband• Our SDP offers comparable performance to Infiniband for concurrent
transfers of large message sizes• Application testing shows more u 10% improvement in performance
over Infiniband• PCIe can serve as an interconnect for communication-intensive
applications
28