integrating keystone with large-scale centralized ... · the kerberos ticket-granting ticket can be...
TRANSCRIPT
![Page 1: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/1.jpg)
Integrating Keystone with Large-scale Centralized Authentication
Chris JaniszewskiOpenStack Solutions [email protected]: chrisj.cloud
Ken HoldenOpenStack Solutions [email protected]: holdenthecloud.com
![Page 2: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/2.jpg)
● Level set● Benefits● Issues and Bottlenecks● Best Practices● Tuning and Results
Agenda
![Page 3: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/3.jpg)
● Authentication and Authorization● Keystone● Tokens● LDAP● Domains● Kerberos and SSO
Level Set
![Page 4: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/4.jpg)
● Keystone (OpenStack Identity Service)● Methods of Authentication
● Authorization tokens● Transport Layer Security (TLS) with X.509
Authentication and Authorization
![Page 5: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/5.jpg)
.. provides identity, token, catalog, and policy services for use specifically by services in the OpenStack family
Features:● External authentication● Federation● Multi-factor authentication
Considerations:● Invalid login attempts ● No enforcement policies on password strength, expiration
Keystone
![Page 6: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/6.jpg)
● generated for authorization and access
● default value for expiry is one hour
● often passed within the structure of a larger context
● Token types:○ UUID○ PKI and PKIZ○ FERNET
Tokens
![Page 7: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/7.jpg)
● The Lightweight Directory Access Protocol (LDAP) - is an open, vendor-neutral, industry standard application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) network
● LDAP simplifies integration of Identity authentication
● Identity service MUST NOT be allowed to write to LDAP services
LDAP
![Page 8: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/8.jpg)
Domains
Keystone V2 Keystone V3
*Pictures found on Matt’s Dorn Blog - http://madorn.com/keystone-v3-api.html#.WwM2hXWUvgk
*
![Page 9: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/9.jpg)
● Keystone can now be executed in a web server like Apache HTTPD
● The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service
● Web server handle authentication is not exclusive
● Web server can provide support for X.509 or Kerberos authentication
● Keystone provides support for password authentication (with SQL or LDAP)
Kerberos
![Page 10: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/10.jpg)
Carnival Ticket Booth vs. Kerberos (server) Identity Service: Authentication Mechanisms
TGT● Grants a ticket called a
Ticket GrantingTicket (TGT) which says which services youhave access to. This ticket also has an expiration called a Time-To-Live (TTL).
● You present the TGT to the service you want access to and the service will use that to determine if you can access the service.
![Page 11: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/11.jpg)
● Consolidated repository● Simplified management● Ideal for multiple OpenStack Deployments with
shared resources● Performance of LDAP vs SQL● SSL/TLS● Back-end database flexibility● Well-defined client API● Decoupling of the roles
LDAP Benefits
![Page 12: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/12.jpg)
● Few people understand LDAP
● No writes allowed
● Is OpenStack a front end for your users?
● Very strict schema
● Performance considerations
LDAP Considerations
![Page 13: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/13.jpg)
● Token Performance● LDAP Size● Database● Horizon Dashboard● LDAP response
Issues and Bottlenecks
![Page 14: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/14.jpg)
● Enable Debug and Verbosity - Impact can be significant● Use slow hard drives for Controller’s OS and Databases● Use non-secured APIs, Dashboard, and LDAP queries
Generally a good idea NOT to...
![Page 15: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/15.jpg)
● The faster you can request tokens, the faster OpenStack responds● Fernet tokens offer superior performance over UUID
Tokens
![Page 16: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/16.jpg)
Who remembers the good old days when OpenStack started to crawl ?
# du -sh /var/lib/mysql/keystone149G /var/lib/mysql/keystone
MariaDB [keystone]> select count(*) from token;+----------+| count(*) |+----------+| 20717136 | Number of UUID tokens+----------+
Just say No to UUID
![Page 17: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/17.jpg)
● Active Directory DNS: Use Domain Nameurl = ldaps://example.com
Or obtain a list of Domain Controllers from DNS via SRV Recordsdig -t SRV _ldap.tcp.example.com
● LDAP: list multiple LDAP serversurl = ldaps://ldap1,ldaps://ldap2
Where do I query?
![Page 18: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/18.jpg)
● Do all 100,000+ users REALLY need access to OpenStack?● Keystone doesn’t always cleanup after itself
MariaDB [keystone]> select count(*) from id_mapping; +----------+ | count(*) | +----------+ | 129052 | Number of LDAP Accounts without filtering +----------+
MariaDB [keystone]> select count(*) from id_mapping; +----------+ | count(*) | +----------+ | 129052 | Number doesn’t decrease when adding filtering later +----------+
Limit number of LDAP records
![Page 19: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/19.jpg)
● Filtering users and groupsuser_filter = (&(|(memberOf=CN=OpenStack-Admins)(memberOf=CN=OpenStack-Users)))
group_filter = (&(objectClass=Group)(&(|(cn=OpenStack)(cn=OpenStack-Admins)(cn=OpenStack-Users))))
● Using AND / OR(&(|(memberOf=foo)(memberOf=bar)))(&(memberOf=for)(memberOf=bar))
● Active Directory Nested groups(memberOf:1.2.840.113556.1.4.1941:=cn=OpenStack)
LDAP - Filtering
![Page 20: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/20.jpg)
● Member of group1 OR group2ldapsearch -LLL -H ldap://example.com -E pr=2/noprompt -b 'dc=example,dc=com' -D 'example\USER' -w PASSWORD '(&(|(memberOf=CN=OpenStack-Admins)(memberOf=CN=OpenStack-Users)))'
● Member of group1 AND group2ldapsearch -LLL -H ldap://example.com -E pr=2/noprompt -b 'dc=example,dc=com' -D 'example\USER -w PASSWORD '(&(memberOf=CN=OpenStack-Admins)(memberOf=CN=OpenStack-Users))'
● Members of Active Directory Nested Groupldapsearch -LLL -H ldap://example.com -E pr=2/noprompt -b 'dc=example,dc=com' -D 'example\USER -w PASSWORD '(&(objectClass=organizationalPerson)(sAMAccountName=*)(memberOf: 1.2.840.113556.1.4.1941:=cn=OpenStack))'
ldapsearch
![Page 21: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/21.jpg)
Default value is 0 which disables paging
$ ldapsearch -LLL -H ldap://ldapserver.domain.com \ -b 'dc=domain,dc=com' -D 'DOMAIN\USERNAME' \ -w PASSWORD |grep sAMAccountName |wc -lSize limit exceeded (4)825
$ ldapsearch -LLL -H ldap://ldapserver.domain.com \ -E pr=1/noprompt -b 'dc=domain,dc=com' \ -D 'DOMAIN\USERNAME' \ -w PASSWORD |grep sAMAccountName |wc -l10052
LDAP Tuning - Pagination
![Page 22: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/22.jpg)
● View how keystone queries LDAP by enabling debugcrudini --set keystone.conf ldap debug_level 3crudini --set keystone.conf DEFAULT insecure_debug truecrudini --set keystone.conf DEFAULT debug true# restart keystone
● Watch the magic...tail -f /var/log/keystone/keystone.log | grep LDAP\ search
filterstr=(&(sAMAccountName=kholden)(&(|(memberOf=CN=OpenStack-Admins,OU=People,DC=lab,DC=lan)(memberOf=CN=OpenStack-Users,OU=People,DC=example,DC=com)))(objectClass=person)) attrs=['sAMAccountName', 'userPassword', 'userAccountControl', 'mail', 'description'] attrsonly=0
● Disable debug when done
LDAP Debug
![Page 23: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/23.jpg)
● Keystone Pooling○ Decrease load on Domain Controller especially with TLS○ Separate pool for User Authentication○ use_auth_pool requires use_pool enabled
crudini --set keystone.DOMAIN.conf ldap use_pool Truecrudini --set keystone.DOMAIN.conf ldap pool_size 200crudini --set keystone.DOMAIN.conf ldap pool_retry_max 20crudini --set keystone.DOMAIN.conf ldap pool_retry_delay 0.1crudini --set keystone.DOMAIN.conf ldap pool_connection_timeout -1crudini --set keystone.DOMAIN.conf ldap pool_connection_lifetime 600crudini --set keystone.DOMAIN.conf ldap use_auth_pool Truecrudini --set keystone.DOMAIN.conf ldap auth_pool_size 1000crudini --set keystone.DOMAIN.conf ldap auth_pool_connection_lifetime 60# restart keystone
Keystone Pooling
![Page 24: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/24.jpg)
● Enable Keystone Caching with memcached backendsystemctl enable memcached && systemctl start memcachedcrudini --set keystone.conf cache enabled truecrudini --set keystone.conf cache backend dogpile.cache.memcachedcrudini --set keystone.conf cache backend_argument url: LOCAL_IP:11211crudini --set keystone.conf catalog caching truecrudini --set keystone.conf domain_config caching truecrudini --set keystone.conf federation caching truecrudini --set keystone.conf revoke caching truecrudini --set keystone.conf role caching truecrudini --set keystone.conf token caching truecrudini --set keystone.conf token cache_on_issue truecrudini --set keystone.conf identity caching truecrudini --set keystone.conf identity cache_time 600# restart keystone
Keystone Caching
![Page 25: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/25.jpg)
Results
![Page 28: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/28.jpg)
● Challenges with Implementation○ Proxies in front of Domain Controllers○ Firewalled Domain Controllers○ Omitted Distinguished Names to save licensing costs ○ Service Account Password Change○ Certificate / CA change○ Changes require keystone restart○ Fernet Token rotation○ The Defiant Windows Admin
Meanwhile, in the real world...
![Page 29: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/29.jpg)
● Multiple OpenStack Environments with One LDAP○ Unique IDs for roles, Domains, and Projects are unique to each
site○ Determine Project, Domain and Role IDs from one site and
update other sites with same IDs (BACKUP DB FIRST)
select * from project where name='PROJECT_NAME';select * from project where name='DOMAIN_NAME';select * from project where name='ROLE_NAME';
update project set id='SITE1_PROJECT_ID' where name='PROJECT_NAME';update project set id='SITE1_DOMAIN_ID' where name='DOMAIN_NAME';update project set id='SITE1_ROLE_ID' where name='ROLE_NAME';
Multi-site
![Page 30: Integrating Keystone with Large-scale Centralized ... · The Kerberos ticket-granting ticket can be used to securely provide tickets for a given service Web server handle authentication](https://reader034.vdocuments.us/reader034/viewer/2022042315/5f03f9307e708231d40bb027/html5/thumbnails/30.jpg)