how docker accelerates continuous development at ironsource: containers #101 meetup
TRANSCRIPT
![Page 1: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/1.jpg)
Shimon Tolts General Manager, Data Solutions
AtomData Pipeline Processing 200B events
with Node.js And Docker On AWS
![Page 2: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/2.jpg)
About ironSource: Hypergrowth
People Reached Each Month
4200Apps Installed Every Minutewith the ironSource Platform
Registered & Analyzed Data EventsEvery Month
200B
800M
50B
0
100B
150B
200B
Jun 201
5
Jul 201
5
Aug 201
5
Sep 201
5
Oct 201
5
Nov 201
5
Dec 201
5
Jan 201
6
Feb 201
6
Mar 201
6
Apr 201
6
May 201
6
![Page 3: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/3.jpg)
We needed a way to manage this data:
Our Business Challenge
ProcessCollect Store
![Page 4: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/4.jpg)
![Page 5: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/5.jpg)
Collection
● Multi region layer - Latency based
routing
● Low latency from client to Atom servers
● High Availability - AWS regions does
fail!
● Storing raw data + headers upon
receiving
![Page 6: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/6.jpg)
Data Enrichment● Enrich data before storing in your Data
Lake and/or Warehouse○ IP to Country○ Currency conversion ○ Decrypt data○ User Agent parsing - OS, Browser, Device...
● Any custom logic you would like! - fully extendible
![Page 7: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/7.jpg)
Data Targets● Near real-time data insertion - 1
minute!● Stream data to Google Storage and/or
AWS S3● Smart insertion of data into AWS
Redshift○ Set the amount of parallel copys○ Configure priority on tables
● BigQuery - Streaming data using batch files import (saves 20% cost)
![Page 8: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/8.jpg)
![Page 9: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/9.jpg)
Micro-Services Architecture● Everything is a service● Decoupling● Distributed systems
Separate lifecycle● Communication using RESTful /
Queue / Streams
![Page 10: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/10.jpg)
Docker● Linux Container● Save provisioning time● Infrastructure as code● Dev-Test-Production - identical
container● Ship easily
![Page 11: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/11.jpg)
Cloud infrastructure● Pay as you go - (grow)● SaaS services ● Auto-scaling-groups● DynamoDB● RDS *SQL● Redshift data warehouse
![Page 12: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/12.jpg)
Continuous Integration● From commit to production● Jenkins commit hook● Git branching model● AWS dynamic slaves● Unit tests● Docker builds● Updating live environment
![Page 13: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/13.jpg)
Diagram
![Page 15: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/15.jpg)
Starting PointPre-baked images - AMIs
Supervisor
Nginx reverse proxy
Node.js * cpu-count
Provisioning time * instances
Bash provisioning scripts
![Page 16: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/16.jpg)
![Page 18: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/18.jpg)
Minimum Viable ProductInfrastructure as code
Nginx
Node.js * cpu-count
Supervisor
Docker Hub
No Bash scripts!
No provisioning time * instances
![Page 19: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/19.jpg)
https://github.com/ironSource/docker-config/blob/bb6be85b97132cbdd10084305ee1ee2f414b0b50/Dockerfile
![Page 20: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/20.jpg)
![Page 21: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/21.jpg)
![Page 22: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/22.jpg)
Interactive CycleNginx
Supervisor
Infrastructure as code
Node.js * cpu-count
Docker Hub
No Bash scripts!
No provisioning time * instances
![Page 23: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/23.jpg)
![Page 24: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/24.jpg)
https://github.com/ironSource/docker-config/blob/c4bbad11a323fd6e36ff31505c43e7c8dc51b1eb/Dockerfile-iojs-cluster
![Page 25: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/25.jpg)
![Page 26: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/26.jpg)
User Data
![Page 27: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/27.jpg)
https://github.com/ironSource/docker-config/blob/2f4ccc7c277850de928cc432f47b2fc58fb8732a/Dockerfile-nodejs-cluster
![Page 28: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/28.jpg)
docker-common.yml
docker-compose.yml
https://stash.ironsrc.com/projects/INFRA-IB/repos/ironbeastcompserter/browse/docker-compose.ymlDocker Compose Example #1 (Using ‘Extends):
![Page 29: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/29.jpg)
User Data
![Page 30: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/30.jpg)
Docker Compose Example #2 (Using ‘links’):
![Page 31: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/31.jpg)
![Page 32: How Docker Accelerates Continuous Development at ironSource: Containers #101 Meetup](https://reader035.vdocuments.us/reader035/viewer/2022062522/58e7d4b81a28ab3a578b4c33/html5/thumbnails/32.jpg)