meetup fairfax lean agile engineering practices ci/cd continuous integration continuous delivery

Meetup

Fairfax Lean Agile Engineering Practices

CI/CDContinuous IntegrationContinuous Delivery

Ramakrishnan Srinivasan, AOL (devfactor@gmail.com)

Thursday, September 18, 2014

*Everything I say reflects my own opinion only.

CI/CD: We will cover Why CI/CD? The development process & its goals Industry & business demands Discuss DevOps, CI, CD and available software The CI/CD setup Opscode Chef terminology & basics Chef Project structure & show a few screens References Q & A

We will speak to Operations folks Quality Assurance analysts Business managers & analysts Software engineers

DevOps Its about increasing efficiencies in the

development/QA/deployment process Its about increasing agility and output of the total system Emphasizes collaboration of Dev, QA and Ops QA personnel need to learn some software development in

order to automate testing Operations folks will find their tasks to be much easier There is an increasing demand for these skills

Typical workflow in s/w shop Developer works on a story to satisfy specific requirements Code is checked in to the repository Developer initiates a build Upon a successful build, QA team is notified QA installs the build in their environment QA tests the delivered functionality Bugs are reported & resolved Once cleared, its installed in production

The problems & motivation Code breaks. Defects are inevitable. No code change, but an external system has changed -

breaking our functionality We forget to run our unit tests before checking in code We want to know about issues sooner rather than later We want to be faster to market, deliver rapidly and often We want to automate deployment in a configurable way

Business demand: Efficiency We are constantly striving to improve productivity and

efficiency Developers

Make functions generic Write unit tests Code with a view to make debugging easy Only checkin code when

you are actually done, or you are sure of not breaking existing functionality

Efficiency .. 2 Testing: Automate where possible Goals

Reduce costs Decrease number of defects Increase productivity Spend more time building the product and creating value for the

customer Spend less time on functions/process that can be automated Do more with less

CI: A solution to improve efficiency Write unit tests with sufficient coverage for the important

parts Trigger a code build on every checkin or maybe every 5

minutes Run unit tests as part of the build Write integration tests for both UI and backend Run integration tests frequently Tests must be high quality & cover major functions Detect issues early, and as they happen

CD: Continuous Deployment Code and tests must be deployed before it can run CD is really part of the CI solution Adap.tv does several deployments to production on a daily

basis

CI Practices – Martin Fowler Maintain a Single Source Repository Automate the Build Make Your Build Self-Testing Everyone Commits To the Mainline Every Day Every Commit Should Build the Mainline on an Integration Machine Keep the Build Fast Test in a Clone of the Production Environment Make it Easy for Anyone to Get the Latest Executable Everyone can see what's happening Automate Deployment Reference: http://martinfowler.com/articles/continuousIntegration.html

CI/CD: Benefits Technical

Continuous delivery Standardized environments Less complex problems since they are identified quickly Faster resolution

Business Faster delivery of features More stable, controlled, and predictable operating environments Focus on adding business value rather than fixing issues Increased process effectiveness Increased agility

CI/CD: Summary Automate testing as part of the build Automate deployment to staging, test or production Automate rollback Deploy continuously to production Incremental new features initially enabled for small set of

users, then turned on for a wider audience

Software alternatives for CI/CD Plenty of alternatives available Opscode Chef Puppet Ansible Salt Cobbler

Here are a few: http://alternativeto.net/software/chef/

Software used in my project need to install Ruby Berkshelf – used to manage dependencies in Chef Chef Server on the server chef-client Jenkins SonarQube git

Servers used in a CI/CD setup Build server Deployment server Chef Server for provisioning & deployment Chef Client on each participating node Jenkins for scheduling and coordinating automated build

pipelines SonarQube: continuously analyze and measure code quality A single server can be used for many of the above Of course, servers VMs can be started up on demand in the

cloud

Chef topology

Chef basics/terminology - I Environment: settings, properties maintained in each

environment like Dev, QA and Production Bootstrap: first time setup Server: hub for configuration data Client/Node: a computer/laptop/server that can run chef-

client Workstation: its configured to use the knife command knife command: Used to activate Chef functionality

Chef components

Chef basics/terminology - II Based on the Ruby language Attributes: properties, key-value pairs Resources: commands, e.g. to run a shell script Templates Recipes Cookbooks Databags: JSON data accessible in a Chef environment that

can be encrypted. Libraries Overview: http://docs.getchef.com/chef_overview.html

Chef Attributes key-value pairs default values can be set Some values can be overridden in specific environments e.g. Connecting to Salesforce Dev, QA and Production instance URLs can be overridden in

respective environments Reference: https://docs.getchef.com/

chef_overview_attributes.html

Attribute locations

Example project Java Web application Apache Tomcat MySQL or Oracle Dev, QA and Production environments CI/CD pipeline

CI/CD Pipeline build the project execute unit tests for each build, create new linux RPM package deploy to CI server run smoke tests automated backend integration tests automated UI testing jobs (Selenium and CasperJS for

automated UI testing) deploy to production if delivery criteria are met

Chef Resources Template Package Directory bash, csh Service Cron Deploy Reference: http://docs.getchef.com/chef/resources.html

Templates these are static text files

with variables embedded in them

Node attributes can be referenced

to dynamically generate environment-specific files like scripts and config files

Recipes & cookbooks basic Ruby syntax the programming part of Chef has standard programming constructs like loops, conditionals, functions,

etc. a Recipe resides in a Cookbook - collection of recipes that belong together uses resources (like bash and service) to generate and install software download new package versions only when available Just enough Ruby for Chef Reference: https://docs.getchef.com/essentials_cookbook_recipes.html

Data bags global variables Chef-wide stored as JSON data Can be encrypted Can be accessed by recipes Reference: http://docs.getchef.com/

essentials_data_bags.html

Libraries Contain Ruby functions Provides the ability to execute arbitrary Ruby code Use to define custom resources, connect to a database, etc. Use to define template helper modules Reference: https://docs.getchef.com/

essentials_cookbook_libraries.html

Chef Project structure Project

attributes environments

Ruby classes Contain Ruby hashes (KV pairs) like a JSON dictionary or associative array Different format than attributes

templates (static text, config files) data_bags (encryptable JSON data) libraries (any Ruby code, helper functions)

Chef Development lifecycle Setup project (one-time) Create recipes, organized inside cookbooks Test cookbooks locally using Test Kitchen (good option) or

Vagrant Run cookbooks/recipes Check the target server If, e.g. a file is not in place as expected, fix recipe and repeat Once its working, push to git (or other repository) Run a recipe:

chef-client -o recipe[my_cookbook::my_recipe]

References Learn Chef: http://learn.getchef.com/ Test Kitchen: https://github.com/test-kitchen/test-kitchen Vagrant: https://www.vagrantup.com/ ThoughtWorks Treehouse blog ThoughtWorks: ThoughtWorks: http://www.thoughtworks.com/continuous-

integration How DevOps is killing the Developer Etsy Their story Continuous delivery with product/feature flags:

Continuously deliver with confidence Etsy deploys over 50 times a day Adap.tv engineering blog

Thank you!!! Questions Discuss Feedback