a double-edged sword: security threats and opportunities ... · rsi [osdi ’16] fasst [atc ’16]...
TRANSCRIPT
A Double-Edged Sword: Security Threats and Opportunities
in One-Sided Network Communication
Shin-Yeh Tsai Yiying Zhang
CPU
Traditional (Two-sided communication)
�2
Memory
User A
Server
CPU
Traditional (Two-sided communication)
�2
Memory
User A GET Key-50
Send→ Server
CPU
Traditional (Two-sided communication)
�2
Memory
User A GET Key-50Key-50
Send→ Server
CPU
Traditional (Two-sided communication)
�2
Memory
User A GET Key-50Key-50
←Reply Server
CPU
Traditional (Two-sided communication)
�2
Memory
User A GET Key-50
SET Key-100
Key-50
User B
Send→ Server
CPU
Traditional (Two-sided communication)
�2
Memory
User A GET Key-50
SET Key-100
Key-50
Key-100User B
Send→ Server
CPU
Traditional (Two-sided communication)
�2
Memory
User A GET Key-50
SET Key-100
Key-50
Key-100User B
←Reply Server
CPU
Traditional (Two-sided communication)
�2
Memory
User A GET Key-50
SET Key-100
Key-50
Key-100User B
One-sided communication
MemoryCPU
Server
CPU
Traditional (Two-sided communication)
�2
Memory
User A GET Key-50
SET Key-100
Key-50
Key-100User B
One-sided communication
MemoryCPU
Server
CPU
Traditional (Two-sided communication)
�2
User A
Memory
User A GET Key-50
SET Key-100
Key-50
Key-100User B
One-sided communication
MemoryCPU
Server
CPU
Traditional (Two-sided communication)
�2
GET Key-50User A
Memory
User A GET Key-50
SET Key-100
Key-50
Key-100User B
One-sided communication
MemoryCPU
Server
Key-50
CPU
Traditional (Two-sided communication)
�2
GET Key-50User A
User B
Memory
User A GET Key-50
SET Key-100
Key-50
Key-100User B
One-sided communication
MemoryCPU
Server
Key-50
CPU
Traditional (Two-sided communication)
�2
SET Key-100
GET Key-50User A
User B
Memory
User A GET Key-50
SET Key-100
Key-50
Key-100User B
One-sided communication
MemoryCPU
Server
Key-50
Key-100
�3
RDMA
Omni-Path
NVMeOF
GPUDirect
Gen-Z
�3
[ATC ’13]Pilaf
[NSDI ’14]FaRM
[SIGCOMM ’14]HERD
[SOSP ’15]DrTM
[SoCC ’17]APUS
[SOSP ’15]FaRM + Xact
[ASPLOS ’15]Mojim
[EuroSys ’16]DrTM+R
[VLDB ’16]RSI
[OSDI ’16]FaSST
[ATC ’16]Cell
[OSDI ’16]Wukong
[SoCC ’17]Hotpot
[ATC ’17]Octopus
[VLDB ’17]NAM-DB
[OSDI ’18]DRTM+H
[SOSP ’17]LITE [SOSP ’17]
KV-Direct
[FAST ’19]Orion
[SYSTOR '19]Storm
RDMA
Omni-Path
NVMeOF
GPUDirect
Gen-Z
�3
[ATC ’13]Pilaf
[NSDI ’14]FaRM
[SIGCOMM ’14]HERD
[SOSP ’15]DrTM
[SoCC ’17]APUS
[SOSP ’15]FaRM + Xact
[ASPLOS ’15]Mojim
[EuroSys ’16]DrTM+R
[VLDB ’16]RSI
[OSDI ’16]FaSST
[ATC ’16]Cell
[OSDI ’16]Wukong
[SoCC ’17]Hotpot
[ATC ’17]Octopus
[VLDB ’17]NAM-DB
[OSDI ’18]DRTM+H
[SOSP ’17]LITE [SOSP ’17]
KV-Direct
[FAST ’19]Orion
Performance[SYSTOR '19]
Storm
RDMA
Omni-Path
NVMeOF
GPUDirect
Gen-Z
�3
[ATC ’13]Pilaf
[NSDI ’14]FaRM
[SIGCOMM ’14]HERD
[SOSP ’15]DrTM
[SoCC ’17]APUS
[SOSP ’15]FaRM + Xact
[ASPLOS ’15]Mojim
[EuroSys ’16]DrTM+R
[VLDB ’16]RSI
[OSDI ’16]FaSST
[ATC ’16]Cell
[OSDI ’16]Wukong
[SoCC ’17]Hotpot
[ATC ’17]Octopus
[VLDB ’17]NAM-DB
[OSDI ’18]DRTM+H
[SOSP ’17]LITE [SOSP ’17]
KV-Direct
[FAST ’19]Orion
Performance[SYSTOR '19]
Storm
Scalability
RDMA
Omni-Path
NVMeOF
GPUDirect
Gen-Z
�3
[ATC ’13]Pilaf
[NSDI ’14]FaRM
[SIGCOMM ’14]HERD
[SOSP ’15]DrTM
[SoCC ’17]APUS
[SOSP ’15]FaRM + Xact
[ASPLOS ’15]Mojim
[EuroSys ’16]DrTM+R
[VLDB ’16]RSI
[OSDI ’16]FaSST
[ATC ’16]Cell
[OSDI ’16]Wukong
[SoCC ’17]Hotpot
[ATC ’17]Octopus
[VLDB ’17]NAM-DB
[OSDI ’18]DRTM+H
[SOSP ’17]LITE [SOSP ’17]
KV-Direct
[FAST ’19]Orion
Performance[SYSTOR '19]
Storm
ScalabilityUsability
RDMA
Omni-Path
NVMeOF
GPUDirect
Gen-Z
�3
[ATC ’13]Pilaf
[NSDI ’14]FaRM
[SIGCOMM ’14]HERD
[SOSP ’15]DrTM
[SoCC ’17]APUS
[SOSP ’15]FaRM + Xact
[ASPLOS ’15]Mojim
[EuroSys ’16]DrTM+R
[VLDB ’16]RSI
[OSDI ’16]FaSST
[ATC ’16]Cell
[OSDI ’16]Wukong
[SoCC ’17]Hotpot
[ATC ’17]Octopus
[VLDB ’17]NAM-DB
[OSDI ’18]DRTM+H
[SOSP ’17]LITE [SOSP ’17]
KV-Direct
[FAST ’19]Orion
Performance
What about Security?
[SYSTOR '19]Storm
ScalabilityUsability
RDMA
Omni-Path
NVMeOF
GPUDirect
Gen-Z
Outline• Introduction and Background
• Vulnerabilities in One-Sided Communication
• Vulnerabilities in One-Sided Hardware
• Opportunities in One-Sided Communication
• Conclusion
�4
Vulnerability 1: Lack of Accountability
�5
Memory
User ACPU
• WRITE accountability
Vulnerability 1: Lack of Accountability
�5
Memory
User ACPU
SET Key-50
• WRITE accountability
Vulnerability 1: Lack of Accountability
�5
Memory
User ACPU
SET Key-50
Server: Who SET the (corrupted) record?
• WRITE accountability
Vulnerability 1: Lack of Accountability
�5
Memory
User ACPU
SET Key-50
Server: Who SET the (corrupted) record?
• WRITE accountability
• READ accountability
Vulnerability 1: Lack of Accountability
�5
Memory
User ACPU
GET Key-100
SET Key-50
Server: Who SET the (corrupted) record?
User B
• WRITE accountability
• READ accountability
Vulnerability 1: Lack of Accountability
�5
Memory
User ACPU
GET Key-100
SET Key-50
Server: Who SET the (corrupted) record?
Server: Who GET the (corrupted) record?User B
• WRITE accountability
• READ accountability
Vulnerability 2: Denial of Service
�6
Attack hardwareMetadata1 Metadata2
• Hard to trace attackers
• Can overload NICs easily
Attacker
Vulnerability 2: Denial of Service
�6
Attack hardwareMetadata1 Metadata2
• Hard to trace attackers
• Can overload NICs easily
Attacker
Vulnerability 2: Denial of Service
�6
Attack hardwareMetadata1 Metadata2
• Hard to trace attackers
• Can overload NICs easily
Attacker
Vulnerability 2: Denial of Service
�6
Attack hardwareMetadata1 Metadata2
• Hard to trace attackers
• Can overload NICs easily
Discussion and Defense• Adding intermediate layer at the sender side
�7Library NIC CPU
MemoryClient
Client NIC CPU
Memory
Intermediate Layer
Discussion and Defense• Adding intermediate layer at the sender side
• Enhancing SmartNIC at the receiver side
�7
SmartNIC
Client NIC CPU
Memory
Client NIC CPU
MemorySoC
Library NIC CPU
MemoryClient
Client NIC CPU
Memory
Intermediate Layer
Outline• Introduction and Background
• Vulnerabilities in One-Sided Communication
• Vulnerabilities in One-Sided Hardware
• Opportunities in One-Sided Communication
• Conclusion
�8
One- and Two-Sided Hardware
�9
Memory
BerkeleySocket
CPU User
Kernel
Two-Sided
One- and Two-Sided Hardware
�9
Memory
BerkeleySocket
CPU User
Kernel
Two-Sided
One- and Two-Sided Hardware
�9
Memory
BerkeleySocket
CPU User
Kernel
Two-Sided
One- and Two-Sided Hardware
�9
1. Address mapping 2. Permission checking 3. Resource isolation
Memory
BerkeleySocket
CPU User
Kernel
Two-Sided
One- and Two-Sided Hardware
�9
1. Address mapping 2. Permission checking 3. Resource isolation
Memory
CPU UserKernel
One-sided CommunicationOne-Sided
Memory
BerkeleySocket
CPU User
Kernel
Two-Sided
One- and Two-Sided Hardware
�9
1. Address mapping 2. Permission checking 3. Resource isolation
Memory
CPU UserKernel
One-sided CommunicationOne-Sided
Memory
BerkeleySocket
CPU User
Kernel
Two-Sided
One- and Two-Sided Hardware
�9
1. Address mapping 2. Permission checking 3. Resource isolation
Memory
CPU UserKernel
One-sided CommunicationOne-Sided
Memory
BerkeleySocket
CPU User
Kernel
Two-Sided
Memory Region 1. rkey/lkey 2. Address
Vulnerability 3 - Predictable Hardware Managed Keys
�10
Virtual addr + rkey
Physical addr
PTE Translation
rkey
/lkey
Val
ue
0
1M
2M
3M
nth-MemoryRegion Registered0 1000 2000 3000 4000 5000
ConnectX-3ConnectX-4ConnectX-5
Vulnerability 3 - Predictable Hardware Managed Keys
�10
Virtual addr + rkey
Physical addr
PTE Translation
rkey
/lkey
Val
ue
0
1M
2M
3M
nth-MemoryRegion Registered0 1000 2000 3000 4000 5000
ConnectX-3ConnectX-4ConnectX-5
Vulnerability 3 - Predictable Hardware Managed Keys
�10
Virtual addr + rkey
Physical addr
PTE Translation
rkey
/lkey
Val
ue
0
1M
2M
3M
nth-MemoryRegion Registered0 1000 2000 3000 4000 5000
ConnectX-3ConnectX-4ConnectX-5
Vulnerability 3 - Predictable Hardware Managed Keys
�10
Virtual addr + rkey
Physical addr
PTE Translation
rkey
/lkey
Val
ue
0
1M
2M
3M
nth-MemoryRegion Registered0 1000 2000 3000 4000 5000
ConnectX-3ConnectX-4ConnectX-5
Vulnerability 4 - Side Channel in NICs
�11
ConnectX-5, 1KB READ request latency
Virtual addr + rkey
Physical addr
PTE Translation Pe
rcen
tile
0
10
20
30
Latency (us)0 1 2 3 4 5 6 7 8
HitMiss-PageTableEntriesMiss-MemoryRegionInfo
Vulnerability 4 - Side Channel in NICs
�11
ConnectX-5, 1KB READ request latency
Virtual addr + rkey
Physical addr
PTE Translation Pe
rcen
tile
0
10
20
30
Latency (us)0 1 2 3 4 5 6 7 8
HitMiss-PageTableEntriesMiss-MemoryRegionInfo
Vulnerability 4 - Side Channel in NICs
�11
ConnectX-5, 1KB READ request latency
Virtual addr + rkey
Physical addr
PTE Translation Pe
rcen
tile
0
10
20
30
Latency (us)0 1 2 3 4 5 6 7 8
HitMiss-PageTableEntriesMiss-MemoryRegionInfo
Side-Channel Attacks in RDMA (Pythia, USENIX Sec ‘19)
�12
Server Machine
RDMA Network
RNIC SRAM
Main Memory CPU
Client Machine
Attacker
Client Machine
Victim
QP MR PTE
PCIePTEMRData Ac
cess
Pro
babi
lity
(%)
0
20
40
60
80
100
Timeline (ms)0 20 40 60 80 100 120 140
VictimAttacker
Discussion and Defense• Generate memory registration keys cryptographically
�13
Sequential to Random
Discussion and Defense• Generate memory registration keys cryptographically
• Isolate on-board resources for different clients
�13
On-boardResource
Sequential to Random
Discussion and Defense• Generate memory registration keys cryptographically
• Isolate on-board resources for different clients
• Enhancing SmartNIC at the receiver side
�13
On-boardResource
SmartNIC
Client NIC CPU
Memory
Client NIC CPU
MemorySoC
Sequential to Random
Outline• Introduction and Background
• Vulnerabilities in One-Sided Communication
• Vulnerabilities in One-Sided Hardware
• Opportunities in One-Sided Communication
• Conclusion
�14
Opportunity of One-sided Communication
�15
ORAM Access Server
Opportunity of One-sided Communication
�15
ORAM Access Server
ORAM READ/WRITE
Opportunity of One-sided Communication
�15
ORAM Access Server
ORAM READ/WRITE
Opportunity of One-sided Communication
�15
ORAM Access Server
ORAM READ/WRITE
Opportunity of One-sided Communication
�15
ORAM Access Server
ORAM READ/WRITE
Opportunity of One-sided Communication
�15
ORAM Access Server
ORAM READ/WRITE
Opportunity of One-sided Communication
�15
ORAM Access Server
One-sided READ
ORAM READ/WRITE
Opportunity of One-sided Communication
�15
ORAM Access Server
One-sided READ
ORAM READ/WRITE
Opportunity of One-sided Communication
�16
ORAM Access Server
K% One-sided READ
(1-K)% ORAM READ100% ORAM WRITE
Opportunity of One-sided Communication
�17
ORAM Access Server
X% One-sided READ
(1-X)% ORAM READ100% ORAM WRITE
Thro
ughp
ut (K
OPS
)
0
10
20
30
K% of One-Sided READ Operations0 25 50 75 100
Conclusion• Security concerns of one-sided communication
• Tradeoffs between Performance and Security
• Hardware Vendor, Software Developers, and Datacenter
�18
Conclusion• Security concerns of one-sided communication
• Tradeoffs between Performance and Security
• Hardware Vendor, Software Developers, and Datacenter
�18