minimizing the window of compromise …...1 2 3 ca node swarmkit’s implementation renew valid from...
TRANSCRIPT
PRACTICAL MTLSMINIMIZING THE WINDOW OF COMPROMISE
Ying Li @cyli
TYPICAL MICROSERVICE ARCHITECTURE
PROBLEM
S1
S1S2
S3
DB
VPC
VLAN-TASTIC MICROSERVICE ARCHITECTURE
PROBLEM
S1
S1S2
S3
DB
CORRECT MICROSERVICE ARCHITECTURE
PROBLEM
S1
S1S2
S3
DB
APPLICATION TLS LIFECYCLE
Bootstrap RevokeRenew
PROBLEM
• CSR ➡ CA • Configuration
BOOTSTRAP
PROBLEM
• Schedule
RENEW
PROBLEM
• Schedule • CSR ➡ CA • Configuration
RENEW
PROBLEM
• Schedule • CSR ➡ CA • Configuration • Restart
PROBLEM
RENEW
• CRL • OCSP [Stapling]
REVOKE
PROBLEM
AUTOMATE, AUTOMATE, AUTOMATE
• Promotes adoption of mTLS
PRINCIPLE
AUTOMATE, AUTOMATE, AUTOMATE
• Promotes adoption of mTLS • Single location for private key
PRINCIPLE
AUTOMATE, AUTOMATE, AUTOMATE
• Promotes adoption of mTLS • Single location for private key • Shorter certificate expiry
PRINCIPLE
SWARMKIT OVERVIEW
https://github.com/docker/swarmkit
SWARMKIT OVERVIEW
Worker
Manager
Manager ManagerWorker
Worker
Worker
WorkerWorker
CLUSTER
SWARMKIT OVERVIEW
Worker
Manager
Manager ManagerWorker
Worker
Worker
WorkerWorker
CLUSTER
NodeNode
Node
NodeNode
Node
SWARMKIT OVERVIEW
Worker
Manager
Manager ManagerWorker
Worker
Worker
WorkerWorker
raft store
CLUSTER
SWARMKIT OVERVIEW
Node
CA
CA CANode
Node
Node
NodeNode
raft store
CLUSTER
SWARMKIT’S IMPLEMENTATION
BOOTSTRAP
SWMTKN-1-mx8suomaom825bet6-cm6zts22rl4hly2Known Prefix
Token Version
Hash of Root CA
Random Secret
SWARMKIT’S IMPLEMENTATION
1. Retrieve, validate Root CA certificate.
BOOTSTRAP
1
CA
Node
SWARMKIT’S IMPLEMENTATION
1. Retrieve, validate Root CA certificate.
2. CSR + secret token ➡ CA. (TLS)
BOOTSTRAP
1 2
CA
Node
SWARMKIT’S IMPLEMENTATION
1. Retrieve and validate Root CA Public key material.
2. CSR + secret token ➡ CA. (TLS)
3. Get certificate. (TLS)
BOOTSTRAP
1 23
CA
Node
SWARMKIT’S IMPLEMENTATION
RENEW
Valid From
Valid Until
50% 80%
SWARMKIT’S IMPLEMENTATION
1. CSR + ➡ CA. (mTLS)
2.Get certificate. (mTLS)
RENEW
1 2
CA
Node
SWARMKIT’S IMPLEMENTATION
RENEW
Restart
1. Trigger extra leader election
2. Workers all need to reconnect to managers
3. Reschedule work
SWARMKIT’S IMPLEMENTATION
RENEW
SWARMKIT’S IMPLEMENTATION
RENEW
SWARMKIT’S IMPLEMENTATION
RENEW
Server
Existing connections
New connections
SWARMKIT’S IMPLEMENTATION
RENEW
Client
Existing connections
New connections
SWARMKIT’S IMPLEMENTATION
REVOKE
SWARMKIT’S IMPLEMENTATION
REVOKE
SWARMKIT’S IMPLEMENTATION
REMOVE
CRLS, OCSP [Stapling]
REMOVE
SWARMKIT’S IMPLEMENTATION
NODE BLACKLISTNode ID Certificate Expiry
a8h1vsk3k9o5nwea858ty9kma 2017-08-26 01:02:52 UTC
k80l2au3yq9f7x6r2oca13vwt 2017-07-15 11:35:23 UTC
n970d5be9ccgnreg4iti4jho3 2017-08-01 22:59:05 UTC
REMOVE
SWARMKIT’S IMPLEMENTATION
Worker/Manager Manager
Request
Validate node IDagainst blacklist
Authorize role
Perform work
Response
Worker/Manager Manager
REMOVE
SWARMKIT’S IMPLEMENTATION
BLACKLIST VS WHITELIST
REMOVE
SWARMKIT’S IMPLEMENTATION
Manager
Manager Manager
delayed join
REMOVE
SWARMKIT’S IMPLEMENTATION
Manager
Manager Manager
Rotate CA
PROBLEM
CA ROTATION
PROBLEM
• (conf.) All nodes: trust old and new CA • (wait.) Verify all nodes
1
CA ROTATION
PROBLEM
• (conf.) All nodes: trust old and new CA • (wait.) Verify all nodes • (conf.) All nodes: renew certificates • (wait.) Verify all nodes
1
2
CA ROTATION
PROBLEM
• (conf.) All nodes: trust old and new CA • (wait.) Verify all nodes • (conf.) All nodes: renew certificates • (wait.) Verify all nodes • (conf.) All nodes: trust new CA only • (wait.) Verify all nodes
1
2
3
CROSS-SIGNED INTERMEDIATE
RootA
Key Info: A Signed by: A
RootB
Key Info: B Signed by: B
RootB
X
Leaf cert: X Signed by: B
Root: B
PRINCIPLE
CROSS-SIGNED INTERMEDIATE
RootA
Key Info: A Signed by: A
RootB
Key Info: B Signed by: B
RootA
X
Leaf cert: X Signed by: B
Root: A
PRINCIPLE
CROSS-SIGNED INTERMEDIATE
RootA
Key Info: A DN: A Signed by: A
RootB
Key Info: B DN: B Signed by: B
Key Info: B DN: B Signed by: A
RootA
RootA
IntermediateB’
PRINCIPLE
Leaf cert: X Signed by: B’
Root: A
RootA
RootA
IntermediateB’
X
CROSS-SIGNED INTERMEDIATE
PRINCIPLE
RootA
RootA
IntermediateB’
RootB
Leaf cert: X Signed by: B
Root: B
X
CROSS-SIGNED INTERMEDIATE
PRINCIPLE
CA ROTATION
SWARMKIT’S IMPLEMENTATION
• (conf.) All nodes: trust old and new CA • (wait.) Verify all nodes • Generate cross-signed intermediate
CA ROTATION
SWARMKIT’S IMPLEMENTATION
• Generate cross-signed intermediate • (conf.) All nodes: renew certificates • (wait.) Verify all nodes
1
CA ROTATION
SWARMKIT’S IMPLEMENTATION
• Generate cross-signed intermediate • (conf.) All nodes: renew certificates • (wait.) Verify all nodes • (conf.) All nodes: trust new CA • (wait.) Verify all nodes • Throw away cross-signed intermediate
1
2
CA ROTATION: BEFORE ROTATION
SWARMKIT’S IMPLEMENTATION
Node Trust Root:
Node TLS Certificate:
Cluster Trust Root:
Cluster Cert Issuer:
RootA
RootA
RootA
RootA
RootAZ
CA ROTATION: START ROTATION
SWARMKIT’S IMPLEMENTATION
Node Trust Root:
Node TLS Certificate:
RootA
RootA
RootA
IntermediateB’
Cluster Trust Root:
Cluster Cert Issuer:
RootA
RootA
RootAZ
CA ROTATION: NODE CERT RENEWAL
SWARMKIT’S IMPLEMENTATION
Node Trust Root:
Node TLS Certificate:
RootA
RootA
RootA
IntermediateB’
Cluster Trust Root:
Cluster Cert Issuer:
RootA
RootA
RootA
IntermediateB
X
CA ROTATION: NODE CERT RENEWAL
SWARMKIT’S IMPLEMENTATION
Node1 Node2 Node3 Node4 Node5
Trust Root
TLS Certificate
RootA
RootA
Z
RootA
Ro AIntermediate
B
X
RootA
Ro AIntermediate
B
X
RootA
Ro AIntermediate
B
X
RootA
Z
RootA
RootA
RootA
RootA
CA ROTATION: ROTATE TRUST ROOT
SWARMKIT’S IMPLEMENTATION
Node Trust Root:
Node TLS Certificate:
RootB
Cluster Trust Root:
Cluster Cert Issuer:
RootB
RootA
RootA
IntermediateB
RootB
X
RootA
RootA
IntermediateB’
CA ROTATION: ROTATE TRUST ROOT
SWARMKIT’S IMPLEMENTATION
Node1 Node2 Node3 Node4 Node5
Trust Root
TLS Certificate
RootB
RootB
RootB
RootA
Ro AIntermediate
B
X
RootA
Ro AIntermediate
B
X
RootA
Ro AIntermediate
B
X
RootA
Ro AIntermediate
B
X
RootA
Ro AIntermediate
B
X
RootA
RootA
CA ROTATION: FINISH ROOT ROTATION
SWARMKIT’S IMPLEMENTATION
Node Trust Root:
Node TLS Certificate:
RootB
Cluster Trust Root:
Cluster Cert Issuer:
RootB
RootB
RootA
RootA
IntermediateB
RootB
X
DEMO
SUMMARY
MINIMIZING THE WINDOW OF COMPROMISE
SUMMARY
MINIMIZING THE WINDOW OF COMPROMISE• automatic bootstrap, renewal • short certificate expiry
SUMMARY
MINIMIZING THE WINDOW OF COMPROMISE• automatic bootstrap, renewal • short certificate expiry • certificate revocation • CA rotation
SUMMARY
MORE INFORMATION
https://github.com/docker/swarmkit
https://diogomonica.com/2017/01/11/hitless-tls-certificate-rotation-in-go/
https://github.com/cloudflare/cfssl
(@cyli)