how to utilise

Alexander Tolstikov Smile Open Source Solutions

How to utilise Open Source tools to create CI/CD & DevOps workflows

without vendor lock-in

Possible workflows:

- Create Kubernetes cluster- Build release artifact (composer install, yarn build, npm)

- Deliver artifact (ansistano, capistrano, etc)- Drush operations

- Automated functional / load / smoke testing - Anything else?

Workflow example:Kubernetes cluster creation on GCP

Evolution of the workflow

Iteration 1:Create k8s cluster manually with UI

ToolsGCP UI

Secrets/Access You need to be authenticated in the browser

Pros • Very fast • No need to maintain dependencies / tools on the local PC

Cons • Need to remember all command params • Need to do the same for all resources (VPC, subnets, DNS, etc) • Human error-prone • Only you know what you did

Congratulations! We just created a snowflake cluster!

Snowflakes are beautiful…

…but so different (and fragile)

“ “Martin Fowler

The true fragility of snowflakes, however, comes when you need to change them. Snowflakes soon become hard to understand and modify. You're not sure what parts of the configuration are important. Their fragility leads to long, stressful bouts of debugging.

Toolsgcloud (from Google Cloud SDK)

Secrets/Access Need to be authenticated on the local PC

Pros More “fixed” compared to manual UI operation

Cons • Need to remember all command params • Need to do the same for all other resources (VPC,

subnets, …) • Human error-prone • No centralised history / logs • Still “Snowflake” infrastructure

Iteration 2: Create k8s cluster manually with “gcloud” command

Tools • terraform Secrets • Need to use GCP key file stored on the local PC Principles / Concepts • Infrastructure as Code Pros • Logic and configuration are separated (different terraform files) • Can be (should be!) checked-in into Git repo Cons • Need to copy/paste files with variables for different environments manually • Configuration drift is still possible with multiple environments

Iteration 3:Create k8s cluster with terraform: Manual terraform commands execution

Iteration 4:PROD & TEST environments & single master config file

Need to introduce some tools!

“ “Unix philosophy

• Do one thing and do it well. • Make programs to work together. • Write programs to handle text streams,

because that is a universal interface.

Tool: uniconf (https://github.com/aroq/uniconf)Config processing (YAML)

Sorry, no documentation…

Input config Output config

OutputInput

Tool: uniconf (https://github.com/aroq/uniconf)Config processing (YAML)

Tool:jq (https://github.com/stedolan/jq)

Config processing (JSON)

Use to extract data from JSON

A lot of filters are available

Check the documentation!

Tool:yq (https://github.com/mikefarah/yq)

Config processing (YAML)

Same as jq tools but for YAML files

Basically is a wrapper over jq tool

uniconf | yq

Output

Tool:gomplate (https://github.com/hairyhenderson/gomplate)

Template rendering. Supports lots of local and remote datasources.

Check the documentation!

Output

Output

Template

Tool:variant (https://github.com/mumoshu/variant)

“One glue to glue them all!”

“Makefile on steroids”

Task definition

Task execution

Variant task execution

Plain command vs Variant task execution

Iteration 4:PROD & TEST environments & single master config file

Now you need to maintain more tools / dependencies:

terraform unicorn

jq / yq gomplate

variant

It can be an issue if we want to execute this workflow somewhere else because we’ll need to sync all software versions, dependencies, etc

Dependency hellPythonRuby

NodeJS…

How to fix the Dependency Hell issue: Put all tools into the container (Docker)

+ =love

Secrets management

Managing secrets

Git Secret (https://github.com/sobolevn/git-secret#git-secret)

Chamber (https://github.com/segmentio/chamber)

GPG (https://gpgtools.org/)

AWS Vault (https://github.com/99designs/aws-vault)

Execute the workflow in the CICD engine: Gitlab CI

Gitlab CI pipeline

Demo time!

Next steps:GitOps? Check Atlantis:

https://github.com/runatlantis/atlantis

Alexander Tolstikovtolstikov@gmail.com / alexander.tolstikov@smile.fr

You can find me on Drupal slack as well!

https://www.smile.eu/