minimizing the window of compromise …...1 2 3 ca node swarmkit’s implementation renew valid from...

PRACTICAL MTLSMINIMIZING THE WINDOW OF COMPROMISE

Ying Li @cyli

TYPICAL MICROSERVICE ARCHITECTURE

PROBLEM

S1

S1S2

S3

DB

VPC

VLAN-TASTIC MICROSERVICE ARCHITECTURE

PROBLEM

S1

S1S2

S3

DB

CORRECT MICROSERVICE ARCHITECTURE

PROBLEM

S1

S1S2

S3

DB

APPLICATION TLS LIFECYCLE

Bootstrap RevokeRenew

PROBLEM

• CSR ➡ CA • Configuration

BOOTSTRAP

PROBLEM

• Schedule

RENEW

PROBLEM

• Schedule • CSR ➡ CA • Configuration

RENEW

PROBLEM

• Schedule • CSR ➡ CA • Configuration • Restart

PROBLEM

RENEW

• CRL • OCSP [Stapling]

REVOKE

PROBLEM

AUTOMATE, AUTOMATE, AUTOMATE

• Promotes adoption of mTLS

PRINCIPLE

AUTOMATE, AUTOMATE, AUTOMATE

• Promotes adoption of mTLS • Single location for private key

PRINCIPLE

AUTOMATE, AUTOMATE, AUTOMATE

• Promotes adoption of mTLS • Single location for private key • Shorter certificate expiry

PRINCIPLE

SWARMKIT OVERVIEW

https://github.com/docker/swarmkit

SWARMKIT OVERVIEW

Worker

Manager

Manager ManagerWorker

Worker

Worker

WorkerWorker

CLUSTER

SWARMKIT OVERVIEW

Worker

Manager

Manager ManagerWorker

Worker

Worker

WorkerWorker

CLUSTER

NodeNode

Node

NodeNode

Node

SWARMKIT OVERVIEW

Worker

Manager

Manager ManagerWorker

Worker

Worker

WorkerWorker

raft store

CLUSTER

SWARMKIT OVERVIEW

Node

CA

CA CANode

Node

Node

NodeNode

raft store

CLUSTER

SWARMKIT’S IMPLEMENTATION

BOOTSTRAP

SWMTKN-1-mx8suomaom825bet6-cm6zts22rl4hly2Known Prefix

Token Version

Hash of Root CA

Random Secret

SWARMKIT’S IMPLEMENTATION

1. Retrieve, validate Root CA certificate.

BOOTSTRAP

1

CA

Node

SWARMKIT’S IMPLEMENTATION

1. Retrieve, validate Root CA certificate.

2. CSR + secret token ➡ CA. (TLS)

BOOTSTRAP

1 2

CA

Node

SWARMKIT’S IMPLEMENTATION

1. Retrieve and validate Root CA Public key material.

2. CSR + secret token ➡ CA. (TLS)

3. Get certificate. (TLS)

BOOTSTRAP

1 23

CA

Node

SWARMKIT’S IMPLEMENTATION

RENEW

Valid From

Valid Until

50% 80%

SWARMKIT’S IMPLEMENTATION

1. CSR + ➡ CA. (mTLS)

2.Get certificate. (mTLS)

RENEW

1 2

CA

Node

SWARMKIT’S IMPLEMENTATION

RENEW

Restart

1. Trigger extra leader election

2. Workers all need to reconnect to managers

3. Reschedule work

SWARMKIT’S IMPLEMENTATION

RENEW

SWARMKIT’S IMPLEMENTATION

RENEW

SWARMKIT’S IMPLEMENTATION

RENEW

Server

Existing connections

New connections

SWARMKIT’S IMPLEMENTATION

RENEW

Client

Existing connections

New connections

SWARMKIT’S IMPLEMENTATION

REVOKE

SWARMKIT’S IMPLEMENTATION

REVOKE

SWARMKIT’S IMPLEMENTATION

REMOVE

CRLS, OCSP [Stapling]

REMOVE

SWARMKIT’S IMPLEMENTATION

NODE BLACKLISTNode ID Certificate Expiry

a8h1vsk3k9o5nwea858ty9kma 2017-08-26 01:02:52 UTC

k80l2au3yq9f7x6r2oca13vwt 2017-07-15 11:35:23 UTC

n970d5be9ccgnreg4iti4jho3 2017-08-01 22:59:05 UTC

REMOVE

SWARMKIT’S IMPLEMENTATION

Worker/Manager Manager

Request

Validate node IDagainst blacklist

Authorize role

Perform work

Response

Worker/Manager Manager

REMOVE

SWARMKIT’S IMPLEMENTATION

BLACKLIST VS WHITELIST

REMOVE

SWARMKIT’S IMPLEMENTATION

Manager

Manager Manager

delayed join

REMOVE

SWARMKIT’S IMPLEMENTATION

Manager

Manager Manager

Rotate CA

PROBLEM

CA ROTATION

PROBLEM

• (conf.) All nodes: trust old and new CA • (wait.) Verify all nodes

1

CA ROTATION

PROBLEM

• (conf.) All nodes: trust old and new CA • (wait.) Verify all nodes • (conf.) All nodes: renew certificates • (wait.) Verify all nodes

1

2

CA ROTATION

PROBLEM

• (conf.) All nodes: trust old and new CA • (wait.) Verify all nodes • (conf.) All nodes: renew certificates • (wait.) Verify all nodes • (conf.) All nodes: trust new CA only • (wait.) Verify all nodes

1

2

3

CROSS-SIGNED INTERMEDIATE

RootA

Key Info: A Signed by: A

RootB

Key Info: B Signed by: B

RootB

X

Leaf cert: X Signed by: B

Root: B

PRINCIPLE

CROSS-SIGNED INTERMEDIATE

RootA

Key Info: A Signed by: A

RootB

Key Info: B Signed by: B

RootA

X

Leaf cert: X Signed by: B

Root: A

PRINCIPLE

CROSS-SIGNED INTERMEDIATE

RootA

Key Info: A DN: A Signed by: A

RootB

Key Info: B DN: B Signed by: B

Key Info: B DN: B Signed by: A

RootA

RootA

IntermediateB’

PRINCIPLE

Leaf cert: X Signed by: B’

Root: A

RootA

RootA

IntermediateB’

X

CROSS-SIGNED INTERMEDIATE

PRINCIPLE

RootA

RootA

IntermediateB’

RootB

Leaf cert: X Signed by: B

Root: B

X

CROSS-SIGNED INTERMEDIATE

PRINCIPLE

CA ROTATION

SWARMKIT’S IMPLEMENTATION

• (conf.) All nodes: trust old and new CA • (wait.) Verify all nodes • Generate cross-signed intermediate

CA ROTATION

SWARMKIT’S IMPLEMENTATION

• Generate cross-signed intermediate • (conf.) All nodes: renew certificates • (wait.) Verify all nodes

1

CA ROTATION

SWARMKIT’S IMPLEMENTATION

• Generate cross-signed intermediate • (conf.) All nodes: renew certificates • (wait.) Verify all nodes • (conf.) All nodes: trust new CA • (wait.) Verify all nodes • Throw away cross-signed intermediate

1

2

CA ROTATION: BEFORE ROTATION

SWARMKIT’S IMPLEMENTATION

Node Trust Root:

Node TLS Certificate:

Cluster Trust Root:

Cluster Cert Issuer:

RootA

RootA

RootA

RootA

RootAZ

CA ROTATION: START ROTATION

SWARMKIT’S IMPLEMENTATION

Node Trust Root:

Node TLS Certificate:

RootA

RootA

RootA

IntermediateB’

Cluster Trust Root:

Cluster Cert Issuer:

RootA

RootA

RootAZ

CA ROTATION: NODE CERT RENEWAL

SWARMKIT’S IMPLEMENTATION

Node Trust Root:

Node TLS Certificate:

RootA

RootA

RootA

IntermediateB’

Cluster Trust Root:

Cluster Cert Issuer:

RootA

RootA

RootA

IntermediateB

X

CA ROTATION: NODE CERT RENEWAL

SWARMKIT’S IMPLEMENTATION

Node1 Node2 Node3 Node4 Node5

Trust Root

TLS Certificate

RootA

RootA

Z

RootA

Ro AIntermediate

B

X

RootA

Ro AIntermediate

B

X

RootA

Ro AIntermediate

B

X

RootA

Z

RootA

RootA

RootA

RootA

CA ROTATION: ROTATE TRUST ROOT

SWARMKIT’S IMPLEMENTATION

Node Trust Root:

Node TLS Certificate:

RootB

Cluster Trust Root:

Cluster Cert Issuer:

RootB

RootA

RootA

IntermediateB

RootB

X

RootA

RootA

IntermediateB’

CA ROTATION: ROTATE TRUST ROOT

SWARMKIT’S IMPLEMENTATION

Node1 Node2 Node3 Node4 Node5

Trust Root

TLS Certificate

RootB

RootB

RootB

RootA

Ro AIntermediate

B

X

RootA

Ro AIntermediate

B

X

RootA

Ro AIntermediate

B

X

RootA

Ro AIntermediate

B

X

RootA

Ro AIntermediate

B

X

RootA

RootA

CA ROTATION: FINISH ROOT ROTATION

SWARMKIT’S IMPLEMENTATION

Node Trust Root:

Node TLS Certificate:

RootB

Cluster Trust Root:

Cluster Cert Issuer:

RootB

RootB

RootA

RootA

IntermediateB

RootB

X

DEMO

SUMMARY

MINIMIZING THE WINDOW OF COMPROMISE

SUMMARY

MINIMIZING THE WINDOW OF COMPROMISE• automatic bootstrap, renewal • short certificate expiry

SUMMARY

MINIMIZING THE WINDOW OF COMPROMISE• automatic bootstrap, renewal • short certificate expiry • certificate revocation • CA rotation

SUMMARY

MORE INFORMATION

https://github.com/docker/swarmkit

https://diogomonica.com/2017/01/11/hitless-tls-certificate-rotation-in-go/

https://github.com/cloudflare/cfssl

(@cyli)