best practices for privileged access management …...“patterns” are methods we have seen that...

Best Practices for Privileged Access Management on

Elastic Infrastructure

BY GRAVITATIONAL

Abstract

In this guide we go through the basics of SSH access management and discuss the best practices for managing access to modern, elastic infrastructure environments.

Table of Contents

Introduction 3

Access Management Components Background 3 SSH 3 SSH Public Key Authentication 3 OpenSSH Certificates 6 OpenSSH Certificate authentication 6 LDAP 8 Identity Provider 8

PAM Production Patterns and Anti-Patterns 9 Anti Pattern: Host based authentication 9 Anti Pattern: Trust on first use 10 Anti Pattern: Using password based authentication 11 Anti-Pattern: Shared private key 12 Anti Pattern: Using LDAP with password authentication 13 Anti-Pattern: unknown source of user identity 13 Production Pattern: Using LDAP to setup trusted user and CA keys 13 Production pattern: Identity providers 14 Production pattern: Second Factor Auth with Identity providers 14 Anti Pattern: Local or poorly managed second factor 15 Production Pattern: Short lived user OpenSSH certificates 15 Production Pattern: Host OpenSSH Certificates 16 Anti-pattern: Forever certificates 18 Production Pattern: Certificate authority rotation 18 Production pattern: Centralized discovery 19 Production pattern: Role based access control 19 Production pattern: Centralized audit logging 20 Production pattern: Anomaly Detection and Alerting 21 Production pattern: Rate limiting and user locking 21 Anti-pattern: Intercepting interactive commands 22 Production pattern: Session capture 22 Anti-Pattern: Non-segmented, publicly accessible network 22 Production Pattern: Bastion servers 23 Anti-pattern: Source IP based access 23 Anti Pattern: Weak SSH and TLS setups 24

© Gravitational, Inc. All Rights Reserved [email protected] | 855-867-2538

1

Production pattern: Idle timeouts 24 Production pattern: outbound reverse tunnels 24 Production pattern: Inter-org trust 25

Conclusion 27


2

Introduction This guide addresses how to manage access to modern server fleets. Today, organizations are dealing with elastic infrastructure that includes thousands of servers with VMs that are launched and deleted every hour. In addition, the people that need to access the infrastructure may come and go in the organization and their roles may change while they are at the organization. This makes it difficult to implement a scalable system of Privileged Access Management (“PAM”) to the IT infrastructure. This guide does not attempt to be a complete overview of the PAM landscape and omits many topics such as Kerberos, SSSD and GSS-API. Instead, it focuses on patterns and anti-patterns that have we have seen implemented by system administrators building access management on top of OpenSSH systems, while trying to adopt to the new regulatory and scalability requirements. We adopted many of the SSH infrastructure patterns mentioned here while building Teleport, a modern SSH server which “manages privileged access to elastic infrastructure, without getting in the way.”

Background of Access Management Components Before we dive into the details of the PAM and SSH patterns and anti-patterns, let's cover key components that are used as building blocks of privileged access management and referenced throughout this guide. SSH SSH stands for Secure Shell - it's a cryptographic network protocol widely adopted in the internet to access servers using remote logins. Access is done via SSH client that dials in to SSH server. SSH Public Key Authentication SSH keys are based on Public Key Cryptography. The public key is distributed openly, and anyone holding the public key can encrypt data. However, only the owner of private key can decrypt it. With this method, a user owns and never shares the SSH private key and SSH servers are set up to trust the user identity of the users who can prove that the trusted private keys.


3

In addition to encryption, private keys can be used to sign any message that can be verified by using only the public key. Public key signatures are used in SSH public key authentication.

You’ll notice that it's not necessary to send the plaintext that was signed alongside the signature assuming both parties already know the plaintext that was pre-negotiated. Here are more detailed steps of public key authentication:


4

* Client sends SSH login username, public key corresponding to the public key and public key algorithm name. This is done to check if authentication method is supported by the server. * Client uses the private key to sign a message composed of a unique session id of the SSH connection and a public key and sends it as an authentication request. * Server looks up the public key in the `authorized_keys` file and verifies the signature based on the session id and public key of the client and, if the signature matches, accepts the authentication request. You can read more here . OpenSSH Certificates The OpenSSH certificate is special extension built by the OpenSSH team. OpenSSH certificate are built using public keys. The OpenSSH certificate authority is a special trusted party that holds its own public and private key pair. OpenSSH CA keys are not used to authenticate, but to sign SSH certificates. An OpenSSH certificate consists of a collection of fields signed by the certificate authority: * Nonce is a unique random string that is used to prevent signature collision attacks. * Public Key is public part of user or host key-pair that identifies a user or a host. * Type identifies user or host certificate. * Key ID identifies the user or a host in log messages. * Valid principals is a list of usernames or hostnames.


5

* Valid after and Valid before are timestamps that define starting time and expiration of the certificate. * Critical Options is a set of options requested by client that should be supported and turned on by the server. * Extensions is a set of optional SSH extensions that could be ignored by the server. * Signature key is a public key of OpenSSH certificate authority used to sign the certificate with its private key. OpenSSH Certificate authentication OpenSSH certificate authentication extends public-key based authentication and uses the same protocol messages:


6

* Client sends SSH login username, ssh certificate and public key algorithm name. This is done to check if authentication method is supported by the server. * Alice uses her private key to sign a message composed of a unique session id of the SSH connection and a certificate and sends it as an authentication request. * Server first checks that certificate is signed by the trusted CA by checking if public key of the certificate is in authorized_keys file. * Server then verifies the signature of the certificate sent by Alice to make sure the certificate was not tampered. * Server verifies that certificate is not expired and is not too early to use by checking expiration and starting dates against local server time. * If all checks above passed, server then can assume that this Alice's certificate and public keys are trusted and verifies the signature sent by Alice based on the session id and public key of Alice and, if the signature matches, accepts the authentication request. LDAP LDAP is a directory access protocol widely used in the enterprise server deployments. LDAP servers provide information about user and group membership and sometimes are used to store users passwords and SSH public keys. Identity Provider Identity provider (IdP) is a system that manages centralized identity for all principals (users or computers). Modern identity providers are used to set up a SSO (single sign on) service for many computer systems ranging from office software to SSH computer systems.


7

Identity providers are widely used in the enterprise and getting more traction in smaller teams because they allow organizations to provide internal authentication service and different applications can offload the process of authentication to the centralized service. Examples of providers are Okta , Auth0 , ADFS and there are many others. There are special protocols that allow applications to integrate between identity providers. In our experience, the most widely used one is SAML 2.0 . However, OIDC is getting more traction, as it provides easier authentication flows designed for mobile applications.

PAM Production Patterns and Anti-Patterns With those components as a background, we’ll now go through some best practices for PAM. “Patterns” are methods we have seen that are useful in production and sometimes necessary to pass compliance audits. “Anti-patterns” are generally things you want to avoid in production as they can lead to problems at scale. While some of the methods listed here as anti-patterns could be a good solution for small scale server deployments, they will quickly fall apart with larger clusters and organizations. Sometimes an anti-pattern should work well in theory, however the practical implementation is usually problematic.


8

Anti Pattern: Host based authentication To be honest, we have never encountered any cases of host based authentication, however here is a warning just in case - avoid using it in any environment. With this authentication method, the client sends signature and public key signed by the host, not the user. The achilles heel of this approach is the behavior of a compromised host: * A compromised host can impersonate other hosts in the system, so servers have to add extra checks over insecure network that source address and hostname matches, which in many cases, is impossible. * Users on compromised hosts can impersonate almost any other user allowed to login from this host. This authentication method should be disabled at all times for SSH servers. Anti Pattern: Trust on first use When the SSH connection is first established, the SSH server sends its public key, to identify itself to the user.


9

Client has to check if it is ok to trust the public key sent by the server. Usually, the client checks against the local database of `known_hosts` and finds the match of the server IP or hostname to the public key. If the public key changed, the user can decide that the host was compromised or altered and it is not safe to connect any more. However, if the public key is not found, the user has to make a choice whether to trust the host on the first use. The suggested solution is to use "trust on first use" - assume that host is trusted if it is the first time when it's encountered. This method assumes that ip or hostname are not changing and therefore the combination of hostname and public key can be trusted, resulting in this warning: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is 51:22:00:1c:1e:6f:ac:ac:de:f1:53:02:1c:7d:35:68. Please contact your system administrator. Add correct host key in /home/example.com/.ssh/known_hosts to get rid of this message. Offending RSA key in /home/example.com/.ssh/known_hosts:15 RSA host key for 192.168.1.1 has changed and you have requested strict checking. Host key verification failed. This leaves user with a choice, either alert security team and raise an incident or ignore the warning, by removing the old key. However, for cloud environments the same IP address and internal host name can be reused many times. That's why we recommend against using only host public keys for checking real host identity. We will return to this topic in the context of a production pattern: "SSH host certificates". Anti Pattern: Using password based authentication


10

* Client sends username and cleartext (non-hashed) password over encrypted SSH connection. * Server computes and checks password hash against local database and if hash matches, authenticates user. While theoretically there is nothing wrong with this approach for small scale servers, we recommend against it because it will likely break as an organization’s infrastructure footprint grows. Here are just some of the problems we have encountered: * Password authentication invites brute-force attacking bots that pollute the logs, so admins are forces to set up tools like fail2ban to prevent excessive scans. * Users user experience degrades as they have to re-enter the same password many times on every single server they log in to, forcing them to copy paste passwords or use very simple and short passwords. * In case of per-environment passwords, users tend to re-use the same password. If that's prohibited and checked, they come up with workarounds like using sticky notes, shared password for the team or overly simplified passwords. * When local passwords expire, some systems are configured in the way to prompt users to change the password to continue. This confuses many users, as they are not ready to come up with a good replacement password and tend to forget the replacement password. Many times, users are not sure if the have changed password for a single server or cluster wide as the system does not inform them about this. Anti-Pattern: Shared private key


11

When provisioning an instance, cloud providers like AWS ask for a pre-installed public key to set up as a key for first access to the server. Teams are tempted to use the same private shared key for every single instance. This is a bad practice, as too many people will have access to the same key, so security of the whole company relies on a single secret. Instead, it is easy to set up instances to boot with pre-installed trust to an external authentication method like an SSH user certificate authority (see the chapters below). We suggest using a shared key as a last resort, encrypt it and store in a safe place. Anti Pattern: Using LDAP with password authentication In addition to suffering from the general problems of password based authentication, there is another problem we have encountered with LDAP password based authentication: * LDAP setup becomes a single point of failure for the entire cluster. If LDAP server or servers become overloaded or go down, the whole cluster is locked down. In theory it is possible to set up highly available and scalable LDAP server (e.g. FreeIPA). However, in practice, we have seen this fail many times and we recommend against it for large server deployments. Anti-Pattern: unknown source of user identity When using public SSH keys authentication, we have encountered many times when it was not possible to determine why certain public SSH key was in authorized_keys or the origin of the key in the first place. This will be a problem during any security audit or any forensics analysis on the server. As a rule of thumb, we recommend a single source of truth of all public keys or certificates periodically synced on the hosts and prohibit any manual changes to the servers. It could be as simple as an S3 bucket with public keys or a github repository that is synced up by ansible job periodically. Production Pattern: Using LDAP to setup trusted user and CA keys LDAP can be used to store public user keys or SSH certificate authority keys. Every host has an agent that connects to LDAP server and syncs public keys on a regular basis. Here are advantages of this model compared to password based authentication:


12

* It does not require the LDAP server to be up and running at all times, as this method is only using periodic updates, and even if the server is down for some period of time, authentication works. * Only non-secret public keys are stored on the LDAP server, reducing the requirements of encryption of secrets at rest and simplifying deployments. * It provides a single source of truth, as users are managed through the identity provider. Keycloak and FreeIPA work well together to implement this pattern for distributed infrastructure. Production pattern: Identity providers Identity providers solve many problems for medium size and large organizations. They provide a Single Sign On service, so applications don't have to use local and custom authentication methods and they can rely on centralized service instead.

Identity providers serve another important goal. They consolidate and serve information about users’ and servers’ group memberships and permissions. Applications use this information to grant permissions. For example, an SSH server can decide if user can login as root based on the fact that that user is a member of group "admins" in the organization.


13

Production pattern: Second factor auth with identity providers While identity providers are recommended, using one also means that the security of the whole organization is only as secure as the password of a privileged user of the identity provider. This means that a single-password solution is no longer a secure option and strong second factor authentication is necessary. What are good strong second factor authentications? The ones that are hard to get without physical access to them. For example: FIDO U2F keys. An examples of a bad second factor is using SMS, because it is relatively easy to hack. Anti Pattern: Local or poorly managed second factor In attempts to protect the system, some administrators deploy a local second factor solution. For example, the second factor is enforced locally on a bastion server. This is a problem because, in these cases, the private material of the secret factor is usually not encrypted or stored securely, backed up and replicated. This leads to a confusing system for users who don't have a good way to manage their second factor and if server data gets lost, everyone will have to re-register with the system. This degrades the trust in the second factor because it becomes a nuisance and people start developing workarounds, such as having a webcam at home translating the values of RSA keys generated on the device. Production Pattern: Short lived user OpenSSH certificates OpenSSH certificates are very scalable because servers can check whether they trust the user based just on one signature of the trusted certificate authority. This means that servers do not need to have persistent connections to the active authentication server at all times. However, this also creates a problem. When an employee leaves the company, how do you revoke the access for an already issued certificate? It turns out there is a simple and elegant solution to this problem: OpenSSH Certificates include an optional expiry date that can be verified by the server in addition to the signature.


14

So if organization issues certificates that are issued for several hours (for example for a duration of the working day), all certificates will auto-expire without any additional action. What is a good duration for SSH certificate? From a security perspective, the shorter, the better. Ideally, certificates should be issued for the duration of the session. However, in practice, several hours may be used for convenience. Production Pattern: Host OpenSSH Certificates Just like with public key auth, when an SSH connection is established, the host can send a signed SSH certificate to the client to verify host's identity:


15

The host certificate is signed by the trusted certificate authority and includes information about the host name and expiration date. Alice has to check if it's ok to trust the certificate: * The client checks if the certificate authority that signed the certificate is trusted by checking `known_hosts` file for CA public key. * If CA public key is found, the client can verify the host signature. * If the signature check has passed, the client can verify if the host name she is connecting to is included in `ValidPrincipals` list. * If all above matches, the client assumes that host identity has been verified. There are several notable differences from public key: * The client does not need to store public keys for every host it connects to; only a list of trusted certificate authorities.


16

* There is additional information about valid principals that allows the client to check if hostname or ip matches the certificate, which makes it harder for the host to impersonate the another host. * If the signature check has failed or the CA is not trusted, this usually means that either some serious misconfiguration is happening or someone is attempting MITM (man-in-the-middle attack), so security team should be alerted. * Even if the public key of the host has been changed because hostname has been reused in the cloud environment during instance reprovisioning, the certificate will still match as there will be no conflict between different public keys. Anti-pattern: Forever certificates Imagine you have issued a certificate that is valid forever. How would one make the certificate invalid if the host has been compromised or an employee's laptop has been stolen? Web certificates provide a feature called Certificate Revocation Lists, however it is criticized by the security community for it's flaws. With OpenSSH certificates, there is no CRL infrastructure, so the only way to invalidate certificate issued by the system is to invalidate the certificate authority. In practice, this could be a very disruptive thing to do so we recommend against issuing certificates without expiry date. Production Pattern: Certificate authority rotation Imagine a case when there is an urgent need to lock down the system from using OpenSSH certificates as a way to access the cluster. A good way to do that without seriously disrupting work flow is to rotate all certificates in the system with a new certificate authority. In this model, a new certificate authority is created, and all principals in the system should receive new credentials. Production pattern: Centralized discovery Modern fleets of infrastructure are elastic, VMs and cloud instances are added and removed on an hourly basis, HA databases are performing automatic failover and changing cluster membership, so it becomes critical to have an up-to date view of the infrastructure at any given moment:


17

Modern access management tools should integrate with discovery services and help administrators and users to segment access based on the up-to date information about the services and servers. Users should not rely on scattered wiki-powered knowledge about servers as it becomes obsolete and can pose security risks, as well. Imagine a user trying to login by IP address recycled to another tenant in the cloud. Combined with trust of first use anti-pattern, this could result in a phishing attack by a bad actor. This is why we recommend to use a reliable service discovery service for production access. Production pattern: Role based access control Combined together, identity provider, and centralized discovery allow the implementation of role based access control (“RBAC”). An identity provider is responsible for issuing an OpenSSH certificate that includes the user identity and group membership information. An identity aware access proxy can use information about identity to allow or deny access to different parts of the infrastructure.


18

Instead of setting up access for every server for every user individually, users can be granted access to parts of the infrastructure, as in the example above. Dynamic discovery enables an even more powerful pattern - granting access based on the dynamic traits of the user and the server they need access to. For example, some users can be given access only to database replicas, wherever they are on the infrastructure. Production pattern: Centralized audit logging Imagine the following situation: There was a potential security breach on one of the servers in the cloud infrastructure. Now, the investigative team at the org has to assess the potential scope of the attack and whether to involve a third-party security team for an expensive full infrastructure audit or if the attack is localized and did not involve any production data and recycling the servers at risk is adequate. To do that, the security team would need to inspect every single server of the company to check for traits resembling the attacker, e.g. suspect source IP or all the access patterns in the same time and data.


19

In the absence of the centralized and structure security audit log, this becomes a several-week-task involving every single department manually collecting and submitting logs in a rushed environment. This does not necessarily require buying expensive commercial logging and security scanning software. A good first step might be collecting auditd logs to a centralized location with syslog. Production pattern: Anomaly Detection and Alerting Having centralized and structured events about failed and successful logins and operations performed on a server is helpful for security teams to easily detect anomalies on the infrastructure. For example, if audit data is combined with CPU and network usage data, it makes it easier to watch for attempts to extract data from the servers. Production pattern: Rate limiting and user locking When a proper authentication method used, brute force attacks are not going to produce any results. It is simply not possible with current technology to guess a private key used by the user. So why is rate limiting is necessary? Without any rate limiting, logs will be polluted by failed authentication attempts and will be harder to inspect. In addition to that, rate limiting serves as a deterrent for bots, as it gets too expensive to hold many connections on the wire. So this feature is not strictly necessary for the SSH level security, but is a good additional measure. However, rate limiting becomes more important for other components of the system that use password based authentication, for example SAML or OIDC identity providers. In addition to that, it's important to lock a SAML or OIDC user out after several incorrect login attempts to prevent brute-force attacks, especially if second factor authentication is failing, while password is succeeding. This could be a strong signal that an adversary has got access to passwords of the users. Anti-pattern: Intercepting interactive commands There are some system features that are called "security theater". Features that give a false sense of security, without adding anything significant to the security of the system. One of those features is special command detectors. When encountering a special command typed by the user, for example "reboot" or "rm", it locks the user out of the system.


20

These scanners are implemented on the protocol level and do simple pattern matching, so they are very easy to workaround... $(echo ZWNobyBoYWNrZWQK | base64 -d) hacked

...and only make life of the administrator harder. Instead, system administrators should rely on SELinux policies, linux user and group membership security. Production pattern: Session capture It is possible and beneficial to capture the SSH sessions of every user in the system. We don't think that it's a security feature though - it may be useful for forensic analysis of a breached system, but we don't think it adds anything to security because it is impossible to intercept and reliably understand the behavior of a live interactive session (see the anti-pattern "intercepting interactive commands"). However, we think this increases transparency in the team, allowing everyone to understand what the other team member is doing at any given moment of time. It also helps during "root cause analysis" sessions, and help train new employees. Anti-Pattern: Non-segmented, publicly accessible network In theory, if every server is secure, patched, all network communications are encrypted and secure networking protocols, DDOS protection is in place, etc. - then there is no need to partition network and build demilitarized zones. In practice, having a perfect security setup is extremely difficult to maintain. That's why we don't recommend exposing privileged access over the internet to the whole infrastructure. Production Pattern: Bastion servers Bastion servers are access endpoints set up as the only way to get into the rest of the cluster and provide a way to expose only a small part of the infrastructure.


21

Imagine an employee's laptop has been stolen and data could be extracted from the data center. Bastion servers allow a security team to quickly shut down all control plane traffic to the entire infrastructure, terminate all internet connections and focus on the inspection. Anti-pattern: Source IP based access Some companies limit ranges of source IPs that are allowed to connect to the infrastructure. In practice, we have seen this leading to nothing but confusion - folks adding their home IPs when working from home, clients blocked when NAT gateway IP has changed or entire office locked out because of misconfiguration on the firewall. Source IP is not a reliable way to identify the user or connecting party. That's why we recommend against it in production system access. Anti Pattern: Weak SSH and TLS setups The system is as secure as its weakest link. All of the actions taken above do not matter if SSH is set up with to use a cipher with a known vulnerability. We recommend using a set of trusted guides reference supported protocols, ciphersuites and key exchange algorithms.


22

Production pattern: Idle timeouts Bob has opened his laptop, entered as a root, forgot about it and went home. We can, of course, blame Bob for violating the security practices of the organization. However, it would be better to have a system that automatically protects users who are prone to mistakes, not punish people for their mistakes. In this case, the system should be set up to lock Bob's computer after a period of inactivity and terminate the idle connection after a period of time. A typical idle timeout is about 5 minutes because it does not disrupt the users’ daily routine. Production pattern: outbound reverse tunnels Sometimes it is not possible for a server to easily allow inbound internet connections. For example, in cases when there is no static publicly available address available or a firewall is set up to prevent inbound access. Reverse tunnels to the rescue:

* Servers can dial to the publicly available proxy and keep the connection - "reverse tunnel" * Clients can use this connection as a special tunnel to connect to the target server via publicly available proxy.


23

Production pattern: Inter-org trust There are cases when you need to set up access for an external party to your infrastructure (e.g. for security audit). The certificate authority approach makes this fairly easy, as long as access management layer supports "external" certificate authorities.

* In this case, the external auditor's organization will issue certificates for their users as usual.


24

* Users of the external org can use their credentials to access servers on the external infrastructure, as long as servers are set up to "trust" the key of the other org certificate authority.

Conclusion We hope this guide is useful as an overview of PAM best practices. Thanks for reading and feel free to send comments and feedback to [email protected]. The main motivation behind building Teleport, our modern SSH server for teams that manage elastic infrastructure, is to give you many of these best practices out of the box with minimal operational overhead. Feel free to reach out if you are interested in a demo.


25

best practices for privileged access management …...“patterns” are methods we have seen that...

Documents