acme - open source infra · let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e...
TRANSCRIPT
![Page 1: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/1.jpg)
ACMENot just for rockets anymore!
SCaLE 15x
Magnus Hagander [email protected]
Image: Kenneth Lu (flickr)
![Page 2: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/2.jpg)
ACMENew ways of blowing things up
Image: wikipedia
![Page 3: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/3.jpg)
Magnus HaganderRedpill Linpro
Infrastructure servicesPrincipal database consultant
PostgreSQLCore Team memberCommitterPostgreSQL Europe
![Page 4: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/4.jpg)
A small case study
![Page 5: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/5.jpg)
The environmentThe postgresql.org infrastructureAround 65 VMs
5 datacenters (4 countries)1 cloud (aws)
Around 0 staff(4-5 with 0 dedicated time, at best)
![Page 6: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/6.jpg)
The environmentDebian jessie
Has been lenny>squeeze>wheezy>Custom config management
Not puppet/chef/etcBecause they sucked at the timeAnd considering problem scope
(Almost) fully automated
![Page 7: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/7.jpg)
The challengeEncrypt everything
(well...)https everywhere the obviousAlso smtp, imap, pgsql, etc, etcBoth public and restricted
Certificate management
![Page 8: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/8.jpg)
The dark agesIndividual service certificates
Manual issuingManual renewal
Domain level wildcard certificateFor *.postgresql.org
Nothing for other domainsShared private keysStill manual
![Page 9: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/9.jpg)
Enter ACMEAutomatic Certificate Management EnvironmentBest known implementation: LetsEncrypt
![Page 10: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/10.jpg)
LetsEncryptIssues domain validated certificates
Same as we had beforeFully automated validationShort lifetime (90 days)
Requires automation
![Page 11: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/11.jpg)
certbotDefault client for LetsEncrypt
![Page 12: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/12.jpg)
certbotRequires exposed http servicesTries to auto-config webserver
SCARY
![Page 13: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/13.jpg)
ACMEIs a protocolNot a clientMultiple ways to verify exists
Just not in default client
![Page 14: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/14.jpg)
ACME dns-01Issue TXT records in DNSBetter suited for central management
DNS probably already is
![Page 15: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/15.jpg)
ACME dns-01
![Page 16: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/16.jpg)
![Page 17: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/17.jpg)
New set of problemsCentralized key distribution
Private keys in one placeNot good for security!
Or distributed access to DNSDoable with dynamic DNSAs long as it's controlled
![Page 18: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/18.jpg)
Back to postgresql.orgExisting simple config managementCentral APIClient certificate authenticatedCan be leveraged
![Page 19: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/19.jpg)
ACME in pginfra
![Page 20: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/20.jpg)
![Page 21: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/21.jpg)
ACME in pginfra
![Page 22: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/22.jpg)
ACME in pginfraOn the VM
... borka pginfra: Completed user and package checks.
... borka pginfra: Creating certificate request for 5-borka.postgresql.org
![Page 23: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/23.jpg)
ACME in pginfraOn central server
~$ ./letsencrypt_cron.py Getting challenges for 1 identifiersSetting up for 1 remaining challengesWaiting for 8 more records to show up in DNSWaiting for 8 more records to show up in DNSWaiting for 4 more records to show up in DNSWaiting for 2 more records to show up in DNSWaiting for 1 more records to show up in DNSAll records present in DNSWaiting for 1 challenges...Issued certificate for borka.postgresql.org
![Page 24: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/24.jpg)
ACME in pginfra
![Page 25: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/25.jpg)
ACME in pginfraBack on the VM
borka pginfra: Downloading certificate 5-borka.postgresql.orgborka pginfra: Replaced file /etc/lighttpd/certfiles/5-borka.postgresql.org.combined contentsborka pginfra: Replaced file /etc/lighttpd/certfiles/5-borka.postgresql.org.chain with /etc/ssl/certs/pginfra_public_5-borka.postgresql.org.chainborka pginfra: Replaced file /etc/lighttpd/conf-available/_pginfra_auto_ssl.conf contentsborka pginfra: Completed user and package checks.borka pginfra: Restarting service lighttpd
![Page 26: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/26.jpg)
ACME in pginfraKeys stay on VM
![Page 27: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/27.jpg)
![Page 28: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/28.jpg)
ACME in pginfraServices never exposed
![Page 29: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/29.jpg)
![Page 30: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/30.jpg)
ACME in pginfraAudit trail and certificates archived
![Page 31: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/31.jpg)
![Page 32: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/32.jpg)
What does it look like?Simple codeacme python module
DNS support not released yetUsing git head versionSame as certbot...
OpenSSL...
![Page 33: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/33.jpg)
Generating CSRdef sync_public_certificates(): ... for c in certdata: if c['csrneeded']: key = crypto.PKey() key.generate_key(crypto.TYPE_RSA, 4096) req = crypto.X509Req() req.get_subject().CN = hostname if c['secondary']: req.add_extensions([crypto.X509Extension( b'subjectAltName',critical=False, value=", ".join("DNS:%s" % d for d in c['secondary' req.set_version(2) req.set_pubkey(key) req.sign(key, "sha256") csrdata[c['name']] = crypto.dump_certificate_request( crypto.FILETYPE_PEM, req)
![Page 34: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/34.jpg)
Central integrationdef main(): dns = LetsencryptDnsManager() curs.execute("""SELECT c.id, primaryname, secondarynames, csrFROM letsencrypt_certificate cLEFT JOIN letsencrypt_issuedcertificate icON ic.basecert_id=c.id WHERE csr != ''GROUP BY c.id HAVING max(issuedat) < now()-'60 days'::intervalOR max(issuedat) IS NULL""") leissuers = [LetsencryptIssuer(*r) for r in curs.fetchall()]
if len(leissuers) == 0: sys.exit(0)
leclient = LetsencryptClient()
![Page 35: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/35.jpg)
Central integration# Get all possible identifiers (the same one might be used more than once, so make unique)identifiers = set(chain.from_iterable([i.get_all_identifiers() for i
leclient.get_challenges(identifiers)remaining = leclient.remaining_challenges()if remaining: for challenge in remaining: dns.add_challenge_record(challenge.get_dns_name(), challenge.get_dns_value())
# Update zone serials and commit dns.flush_challenges()
while True: n = dns.check_records() if n == 0: break time.sleep(30)
![Page 36: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/36.jpg)
Central integration# Trigger letsencrypt to checkfor challenge in remaining: challenge.answer_challenge()
# Wait for all challenges to be confirmedwhile True: remaining = leclient.remaining_challenges(True) if not remaining: break time.sleep(30)
for i in leissuers: (pemcert, pemchain, expires) = i.issue(leclient) curs.execute("INSERT INTO letsencrypt_issuedcertificate ...."
dns.cleanup()
![Page 37: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/37.jpg)
Certificate deploymentDepends on webserverAlready have plugin setupsNote order of certs, keys and chains!Don't forget to restart!
![Page 38: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/38.jpg)
Certificate renewalSame as reissueNo special handlingSeparate rate limit
![Page 39: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/39.jpg)
Rate limitsLetsencrypt has rate limits
20 new certs / domain / week100 names / cert5 duplicate certs / week500 registrations / ip / 3 hours300 pending authorization
We're nowhere near these limits
![Page 40: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/40.jpg)
ConclusionsMuch easier than before
Close to 0 work deployment0 work maintenance and renewal
Better securityNo shared keys
![Page 41: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/41.jpg)
ConclusionsDirect work with ACME is easy!Don't forget to monitor expiry!!
![Page 42: ACME - Open Source Infra · Let s e n cr y pt h as rate l imit s 2 0 n e w ce rt s / do main / we e k 1 0 0 n ame s / ce rt 5 du pl icate ce rt s / we e k 5 0 0 regis t rat io n s](https://reader036.vdocuments.us/reader036/viewer/2022081405/5f09af717e708231d4280617/html5/thumbnails/42.jpg)
Thank you!Magnus Hagander
[email protected] @magnushagander
http://www.hagander.net/talks/
This material is licensed