DevOps DevSecOpsBuilding Security in to your DevOps Pipeline
Eric Johnson | Эрик Джонсон
• AWS Senior Developer Advocate - Serverless
• AWS Certified• Solutions Architect
• SysOps Administrator
• Developer
• Father of Five
• Musician | Drummer
• Avid Diet Dr. Pepper Drinker
• Lover of Pizza
@edjgeek
DefinitionsGetting Everyone on the Same Page
DevOps
Development Operations
Refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably.
De
vO
ps
DevSecOps
Development Operations
De
vO
ps
Wrapping the DevOps process with security best practices
DevOps ExamplesThings We See In The Wild
Deploying to Instances
• Deploying code to EC2 Instances
• Deploying to on premise instances
Deploying to Containers
• Amazon ECS
• Amazon EKS
Deploying to Serverless
• API Gateway
• Lambda
• DynamoDB
• SQS
• SNS
The ”DevOps” Part
• CodePipeline
• CodeBuild
• CodeCommit
• CloudFormation
• ECR
• OpsWorks
The Security RealityNot “If” but “When”
Everything fails all of the timeWerner Vogels – VP & CTO, AWS
“What If” Guy
Solid Security is Like Onions
It Has Layers
DevOps Layers of SecurityBuilding the Walls
The Account
Beta Staging Production
AWS
CloudFormation
AWS
CodeCommitAWS
CodeBuild
AWS
CodeDeploy
The Account
Beta Staging Production
The Account
Beta Staging Production
Master
The User
Root User
• Supports MFA
• Dashboard Access
• Programmatic Access
• Full Account Access
• CANNOT be locked out
IAM User
• Supports MFA
• Dashboard Access
• Programmatic Access
• Fine Grained Access
• CAN be locked out
The Root User
• Use a recoverable email address
• Remove programmable access
• Enable MFA
• Store in a VERY secure place
• Do not use
• LOCK IT UP!!
Best Practices
The IAM User
• Enable MFA
• Grant Least Privileges
• Disable Unused Users
• Employ user Groups
• Employ Managed Policies
• Use Groups to Assign Permissions
• Configure A Strong Password Policy
• Rotate Credentials Often
Best Practices
The Role & Policies
Roles
• An IAM entity that defines a set of permissions
• Can contain one or more policies
• Is assumable by trusted entities such as IAM Users or AWS Services
Policies
• Policies define permissions for an action
• Multiple types of policies• Identity
• Resource
• Access Control List (ACL)
The Role & Policies
• Break up policies by resource
• Use AWS Defined Policies When Possible
• Grant Least Privilege
• Use Policy Conditions for Extra Security
Best Practices
Role
Policy
Policy
SecretsKeeping Secrets… Secret
Common Patterns for Secrets
• Stored in code repo
• Stored on server
• Stored on developers machines
• Shared with other developers
• Maximum “blast radius”
• Available at rest and in transit
Hard Coded Values
Common Patterns for Secrets
• Where to store?
• Management nightmare
• Smaller blast radius
• Becomes problematic in managed services
• Passing in as parameters
• Exposed on dashboard
Environmental Variables
Common Patterns for Secrets
• Where to store?
• Smaller blast radius
• Hand rolled fetch and implementation
• Can add latency depending on location
External Central Config Files
Best Practices for Secrets
• Rotates Secrets Safely on Supported Services
• RDS
• Other Select Databases*
• Secure and Audit Secrets Centrally
• Manage Access with Fine-Grained Permissions
• Provides Code Examples!!
• Can store non-rotated strings as well
• Encryption and Decryption managed for you
• Minimal blast radius
• Accessible through CloudFormation*
AWS Secrets Manager
Best Practices for Secrets
1. User inserts secrets to AWS Secret Manager
2. AWS Secret Manager creates an AWS Lambda to manage rotation
3. Lambda rotates password in AWS Secrets Manager and the Amazon RDS Instance
4. User adds code to get secrets
5. Code calls AWS Secrets Manager to get secret DB User and Password
6. Code uses obtained credentials to query RDS database.
AWS Secrets Manager
AWS
Secrets Manager
AWS
Lambda
Amazon
RDS
Amazon
EC2
1
2
3
3
4 5
6
Best Practices for Secrets
1. User inserts secrets to Amazon EC2 Systems Manager Parameter Store
2. User adds code to get secrets
3. Code calls AWS Secrets Manager to get secret DB User and Password
4. Code uses obtained credentials to query RDS database.
Amazon SSM Parameter Store
Amazon
RDS
Amazon
EC2
1
2 3
4
Parameter
Store
Best Practices for Secrets
• Allow users to connect to RDS databases with an IAM Role instead of database credentials
• Apply IAM Roles to EC2 instances to allow permission to other services
• Allow authenticated users to assume roles using Cognito Federated Identities
• Provide fine-grained, row level access to DynamoDB tables through assumed roles
Don’t Use Them – Use IAM Access When Possible
AutomationAutomate all the things
“The first rule of any technology used in a
business is that automation applied to an efficient
operation will magnify the efficiency. The second
is that automation applied to an inefficient
operation will magnify the inefficiency.”
- Bill Gates
Why Automation
• Reproducible Processes
• Predictable Outcomes
• Minimize the Blast Radius
• Removing Human Error
• Reduce Wasted Man Hours
Humans = Security Risk
Not Automated
GIT
Bob the Builder
developers
Tammy the Tester
Debbie the Deployer
Automating Integration & DeploymentContinuous Integration and Continuous Delivery
AWS
CodeCommit
GitHub
Bitbucket
AWS
CodePipeline
AWS
CodeBuild
AWS
CodeDeploy
AWS
CloudFormation
Amazon ECR Amazon
ECS,
EKS, Fargate
EC2
Instance
Automating Configuration
• AWS CloudFormation
• AWS Systems Manager
• AWS OpsWorks
• AWS Service Catalog
Building and Maintaining Consistency
Automating Monitoring
• Amazon CloudWatch
• AWS CloudTrail
• AWS Config
Watching & Reacting
Ask MeI will be answering questions in the Ask Me Section
Thank You!Благодарю вас!
Eric Johnson @edjgeek