ansible inside
TRANSCRIPT
WHY ANSIBLEwe will consider some facts:
ideato’s scenariodev needssysadm needs
IDEATO SCENARIOfocus on great software
development and good practices
50+ VM’s to provision, configure and maintain, no need a high
level of orchestration
DEV NEEDS• easy CM tool to setup their
environments
• time spent to debug CM tool error is waste
SYSADM NEEDS
• painless rolling updates
• going to mass production environments
VS
ANSIBLE ROLES ≈
PUPPET MODULES
LEARNING CURVE
YAML vs RubyDSL
i don’t want learn Ruby or other DSL…
- name: set up user user: name=alemazz shell=/bin/bash password={{ password}}
user{"$user": managehome=>true, ensure => present,
}
file{"/home/$user": ensure=>directory, mode=>755, require=>User["$user"],
}
file{"/home/$user/.ssh": ensure=>directory, require=>File["/home/$user"], }
Node specific information
Hiera
Node specific informationtemplate Jinja + ansible vars + ansible
vaultadd a yaml file on host_vars/ or group_vars for example:
— aws_access_key: AKIA aws_secret_key: ngxiw
and encrypt to AES: ansible-vault encrypt aws.yaml
Agentlessonly SSH/SFTP/SCP are required
no central server scalability
no need to update minions or puppet over your infrastructure
Inconsistency
• Ruby & PE• Puppetforge modules• Puppet skip everything
based on dep what just failed
• Rspec needed
TOWARD MASS PRODUCTION SYSTEM
DEMO: ELASTICSEARCH CLUSTER ON AWS
USING ANSIBLE
our demo will be on AWS multi AZ……
Why Elasticsearch is fit for CM management tools like Ansible?
Lot of sys adm configuration tips for a cluster environment
• java settings( jmx, mlockall….)• sysctl settings( swappiness, max_map,count..)• ulimit settings
Do I have to change these settings by hand repeated for n° instance times?
NO THANKS!
As a mention before Ansible has a plenty of sysadm modules:
- name: firewalld applying conf firewalld: service=elasticsearch permanent=true zone=public state=enabled
tags: - firewall
- name: sysctl configs sysctl: name=fs.file-max value=64000 state=present tags: - sysctl
Here’ s come AWSAWS provides a special plugin for discovery your ES
instances inside your cluster just by their security group!
discovery.type: ec2discovery.zen.ping.multicast.enabled: falsediscovery.ec2.groups: my_security_group
I don’t have to update the other node -1 configurations if i need to replace or add a new node!!
Create our instances--- hosts: localhost connection: local vars_files: - host_vars/el.yml vars: security_group: elsg instance_type: t2.medium image: ami-7cc4f661 region: eu-central-1 keypair: example.pem n_instances: "1" tasks: - name: Launch Instance to Frankfurt av 1 ec2: group: elsg instance_type: "{{ instance_type }}" image: ami-7cc4f661 wait: true region: eu-central-1 keypair: "{{ keypair }}" vpc_subnet_id: subnet-id count: "1" register: ec2 with_items: ec2_instances_fav1
… - name: Launch Instance to Frankfurt av 2 ec2: group: elsg instance_type: "{{ instance_type}}" image: ami-7cc4f661 wait: true region: eu-central-1 keypair: "{{ keypair }}" vpc_subnet_id: subnet-id2 count: "2" register: ec2 with_items: ec2_instances_fav2 remote_user: centos gather_facts: True sudo: false
ansible-playbook -i inventories/local/local el-aws_create-instance.yml
---- name: ensure pip is installed for curator yum: name=python-pip state=installed enablerepo=epel tags: - curator sudo: true
- stat: path=/opt/jre-8u45-linux-x64.rpm register: jre_exists tags: - jre
- name: Install Elasticsearch Curator and required dependencies. pip: "name={{ item }}" with_items: - elasticsearch-curator - argparse tags: - curator sudo: true
- name: download Oracle Java JRE Runtime command: 'wget -q -O /opt/jre-8u45-linux-x64.rpm --no-cookies --no-check-certificate —header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jre-8u45-linux-x64.rpm"' when: jre_exists.stat.exists == False
- name: install Oracle Java JRE Runtime yum: name="/opt/jre-8u45-linux-x64.rpm" state=present tags: - jre
Set up ES cluster(1/4)
- stat: path=/opt/elasticsearch-1.5.1.noarch.rpm register: el_exists tags: - elinstall
- name: download Elasticsearch command: 'wget -q -O /opt/elasticsearch-1.5.1.noarch.rpm https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.5.1.noarch.rpm' when: el_exists.stat.exists == False
- name: install Elasticsearch yum: name="/opt/elasticsearch-1.5.1.noarch.rpm" state=present tags: - elinstall
- name: install plugins command: "{{ item }} chdir=/usr/share/elasticsearch/bin/" with_items: - ./plugin -install elasticsearch/elasticsearch-cloud-aws/2.5.1 - ./plugin -install royrusso/elasticsearch-HQ ignore_errors: true tags: - plugin
Set up ES cluster(2/4)
- name: copy conf to mem limit unlimited copy: src=99-elastic-nproc.conf dest=/etc/security/limits.d/99-elastic-nproc.conf owner=root mode=0640 tags: - ulimit sudo: true- name: sysctl configs sysctl: name=vm.swappiness value=0 state=present tags: - sysctl sudo: true- name: sysctl configs sysctl: name=vm.max_map_count=262144 value=0 state=present tags: - sysctl sudo: true- name: sysctl configs sysctl: name=fs.file-max value=64000 state=present tags: - sysctl sudo: true- name: disable swap command: swapoff -a tags: - swap sudo: true
Set up ES cluster(3/4)
- name: set up elasticsearch.yaml template: src=elasticsearch.j2 dest=/etc/elasticsearch/elasticsearch.yml owner=root mode=0644 backup=yes tags: - elconf- name: ensure exists log directory and data directory file: path={{ item }} state=directory owner=elasticsearch with_items: - /var/data/elasticsearch - /var/log/elasticsearch tags: - directory sudo: true- name: start elastic service: name=elasticsearch state=restarted enabled=yes- name: copy json accounts copy: src=accounts.json dest=/home/centos owner=centos mode=0640 tags: - accounts sudo: true
Set up ES cluster(4/4)
Ansible provides a special plugin to find the running instances inside your EC2 account…
it’s called dynamic inventory
ansible-playbook -i inventories/dynamic/ec2.py el-aws_deploy-instance.yml
./ec2.py —list "eu-central-1b": [ “5*.2*.8*.4*”, “5*.2*.3*.9*”, “5*.2*.4*.3*” ],
Insert some datalet’s try to insert a sample bank dataset, here a small part of it:{ "account_number": 0, "balance": 16623, "firstname": "Bradshaw", "lastname": "Mckenzie", "age": 29, "gender": "F", "address": "244 Columbus Place", "employer": "Euron", "email": "[email protected]", "city": "Hobucken", "state": “CO"}
curl -XPOST 'localhost:9200/bank/account/_bulk?pretty' --data-binary @accounts.json
Let’s see the output
Elastic HQ
What we have achieved?• a mass production system without handy configuration• a fully reproducible environment• scalability• availability• exit staff proof• fully documentated by the code• reduced stress ……………………………………………………
state of the art &
current workflow
I’m almost a DevOps
Current workflowAssumptions:
dev environment = local environment
developers using Vagrant and Ansible to configure their environment
deploys are via Idephix or rsync
dev asks to sysadmins to provision staging & prod
sysadmins add their roles to production environment !!
roles repo is inside local network, remote dev can’t obtain that roles
we haven’t a single source of code for Ansible roles
we don’t share efforts on roles
Issues
we got rolling updates on all machines though Ansibleon newer machines we have some sysadmin roles like:
• distrib role• security role • s3 role• vpn role
but we haven’t any application oriented roles like webserver role or php role on stag/prod
easiest workflowsysadmin will provision staging and production using same roles that dev use•developers deploy app code•syasadmin deploy roles
2nd workflowdevelopers also deploy the infrastructural code
Can Idephix be also a provisioner ??
Resourceshttp://www.ansible.com/home
https://docs.ansible.com/playbooks_vault.htmlhttps://puppetlabs.com/
http://docs.puppetlabs.com/hiera/1/https://www.elastic.co/
https://github.com/elastic/elasticsearch-cloud-awshttps://github.com/ansible/ansible/blob/devel/plugins/inventory/ec2.pyhttp://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-
part-2-configuring-the-elasticsearch/https://github.com/royrusso/elasticsearch-HQ
http://getidephix.com/
Questions???