![Page 1: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/1.jpg)
Continuous DeploymentPart 2 (technical)
Kari Systä, 27.09.2019
![Page 2: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/2.jpg)
Content
• Course matters• Recap (main points) of last week• Continuous Delivery – from technical point of view
![Page 3: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/3.jpg)
Numbers
• Sign-ups in POP: 96 -> 92 –> 87 -> 86• Sign-ups in plus: 71 -> 84 -> 90 -> 90• Responses to initial questionary: 53 -> 62 -> 71 -> 68• Responses to exercise 3 -> 51 -> 52
(by Thursday 26.09)• Responses to both 1 and 3. 48 -> 49• Final number 50?
26.9.2019 ssss 3
![Page 4: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/4.jpg)
Next exercise: orchestration in practice
• The task is use docker_compose to ”orchestrate” a system of two services.• The services are simple (naïve), but you need to implement them• You can use your favourite programming language and platform
• You have roughly two weeks time (starting last Monday)
![Page 5: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/5.jpg)
What is ”cloud orchestration”?
Two results of googling• Orchestration is the automated configuration, coordination, and
management of computer systems and software• Cloud orchestration is the use of programming technology to manage
the interconnections and interactions among workloads on public and private cloud infrastructure. It connects automated tasks into a cohesive workflow to accomplish a goal, with permissions oversight and policy enforcement.
![Page 6: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/6.jpg)
![Page 7: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/7.jpg)
Docker composeversion: '3’services:
web:build: .ports:
- "5000:5000”volumes:
- .:/code- logvolume01:/var/log
links:- redis
redis:image: redis
volumes:logvolume01: {}
$ docker-compose up –d$ ./run_tests$ docker-compose down
![Page 8: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/8.jpg)
Remember pets and cattle?
![Page 9: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/9.jpg)
YAML
• Wikipedia: YAML ("YAML Ain't Markup Language") is a human-readable data-serialization language. It is commonly usedfor configuration files• Spaces for indentation – have a syntactical meaning• https://www.tutorialspoint.com/yaml/yaml_basics.htm
![Page 10: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/10.jpg)
YAML -> JSONversion: '3’services:
web:build: .ports:
- "5000:5000”volumes:
- .:/code- logvolume01:/var/log
links:- redis
redis:image: redis
volumes:logvolume01: {}
{"version": "3","services": {"web": {"build": ".","ports": ["5000:5000"
],"volumes": [".:/code","logvolume01:/var/log"
],"links": ["redis"
]},"redis": {"image": "redis"
}},"volumes": {"logvolume01": {}
}}
![Page 11: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/11.jpg)
Nice looking tutorial
• https://www.baeldung.com/docker-compose
![Page 12: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/12.jpg)
Docker swarm - docker compose
Docker swarm
OrchestrationDocket composes
![Page 13: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/13.jpg)
Your task
• Service/application 1 should:• As a response to incoming Request 1
send a HTTP GET request to Service2• Compose a response from
• “Hello from “ + <Remote IP address and port of the incoming Request1>• “ to “ + <Local IP address and port of Service1>• Response of the above request to Service2
• Return the composed response• Service/application 2 should• As a response to incoming Request 2 compose a response from
• “Hello from “ + <Remote IP address and port of the incoming Request2>• “ to “ + <Local IP address and port of Service2>
• Return the composed response
![Page 14: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/14.jpg)
Your task
• You should write Dockerfiles for the both servicesand docker-compose.yaml to start both containersso that Service1 is exposed in port number 8001.The docker-compose should also create a privatenetwork that allows Services 1 and 2 to communicate.
• The service1 is assumed to be under development, so the image is rebuilt often (hint you may use ”build:” -primitive in docker-compose.yaml. Service2 is a reused service and you may pre-build the image. Image can be stored locally though.
• After the system is ready the student should return (details will be published later)• Content of Docker and docker-compose.yaml files• Response to Request 1 (that contains also response from Request 2)• Source codes of the applications
![Page 15: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/15.jpg)
Hints
• Remember to backup your application and docker and compose files – you will need them in the future. E.g. to gitlab.• It might be a good idea to create and test the applications first.• You may need to revisit parts 1-3 in https://docs.docker.com/get-
started/• Docker images are easy to access, if they are tagged when build• $ docker build --tag=pinger .• If Docker image is rebuilt, docker-compose should also be given a hint
that rebuilt should override the existing one• $ docker-compose up --build
![Page 16: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/16.jpg)
Programming language
• Whatever, but running and testing of the App should not requirelicense fees.• Python, JavaScript(nodejs), Golang, Java, Scala, Ruby, even C/C++ are
ok.
![Page 17: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/17.jpg)
Recap
![Page 18: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/18.jpg)
Continuous delivery and deployment(http://blog.crisp.se/2013/02/05/yassalsundman/continuous-delivery-vs-continuous-deployment)
05.03.2018 18
![Page 19: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/19.jpg)
Perceived benefits
• Improved delivery speed of software changes Improved speed in the development and deployment of software changes to production environment.• Improved productivity in operations work. Decreased communication problems,
bureaucracy, waiting overhead due to removal of manual deployment hand-offs and organisational boundaries; Lowered human error in deployment due to automation and making explicit knowledge of operation-related tasks to software development• Improvements in quality. Increased confidence in deployments and reduction of
deployment risk and stress; Improved code quality; Improved product value to customer resulting from production feedback about users and usage.• Improvements in organisational-wide culture and mind-set. Enrichment and
wider dissemination of DevOps in the company through discussions and dedicated training groups ‘communities of practice’
![Page 20: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/20.jpg)
Perceived challenges
• Insufficiencies in infrastructure automation• High demand for skills and knowledge• Project and resource constraints• Difficulties in monitoring, especially for microservice-based
applications and in determining useful metrics• Difficulties in determining a right balance between the speed
of new functionality and quality.
![Page 21: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/21.jpg)
Where was the beef?
Business Development Operation Use
Business Development Operation Use
![Page 22: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/22.jpg)
Some technical material
26.9.2019 22
![Page 23: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/23.jpg)
Deployment pipeline (a possible example)
26.9.2019 23
![Page 24: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/24.jpg)
Artifact repository
26.9.2019 24
![Page 25: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/25.jpg)
What does it really take to run CD?
14.3.2016 TIE-2210x/Kari Systä 25
Integration Test
”Build”
”Build”
”Build”
Test
Test
Test
Use
Set-up and operate
Testautomation
VM
VM VM
VM
Set-up and operate
Estimate & manage costs
![Page 26: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/26.jpg)
CI – essential practices(according to Humbley and Farley)
• Don’t check in on a broken code• Always run all commits tests locally before committing, or get your CI server
to do it for you• Wait for commit tests to pass before moving on• Never go home on a broken build• Always be prepared to revert to the previous revisions• Time-box fixing before reverting• Don’t comment out failing tests• Take responsible for all breakages that result from your changes• Test-driven development
26.9.2019 26
![Page 27: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/27.jpg)
Deployment essential pract.(according to Humbley and Farley)
• Only build your binaries once• Deploy the same way to every environment• Smoke-test your deployments• Deploy to copy of production• Each change should propagate through the pipeline instantly• If any part of pipeline fails, stop the line
26.9.2019 27
![Page 28: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/28.jpg)
Maturity models(https://developer.ibm.com/urbancode/docs/continuous-delivery-maturity-model/)
• Base: The base level is enough to “be on the model”. The team has left fully manual processes behind.• Beginner: At the beginner level, the team is trying to adopt some ECD practices in
earnest but is still performing them at a rudimentary level.• Intermediate: Practices are somewhat mature and are delivering fewer errors
and more efficiency. For many teams, Intermediate practices may be sufficient.• Advanced: The team is doing something well beyond what most of the rest of the
industry and is seeing a great deal of efficiency and error prevention as a result.• Extreme: Elements within the Extreme category are ones that are expensive to
achieve but for some teams should be their target. Put another way, most organizations would be crazy to implement them, while this minority would be crazy to not implement them.
![Page 29: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/29.jpg)
Another(https://www.infoq.com/articles/Continuous-Delivery-Maturity-Model/)
Base … started to prioritize work in backlogs, have some process defined which is rudimentarily documented and developers are practicing frequent commits into version control.Beginner … teams stabilize over projects and the organization has typically begun to remove boundaries by including test with development. Multiple backlogs are naturally consolidated into one per team and basic agile methods are adopted ….Intermediate … extended team collaboration when e.g. DBA, CM and Operations are beginning to be a part of the team or at least frequently consulted by the team. Multiple processes are consolidated and all changes, bugs, new features, emergency fixes, etc, follow the same path to production. Decisions are decentralized to the team and component ownership…Advanced … team will have the competence and confidence it needs to be responsible for changes all the way to production. Continuous improvement mechanisms are in place … releases of functionality can be disconnected from the actual deployment, which gives the projects a somewhat different role. A project can focus on producing requirements for one or multiple teams and when all or enough of those have been verified and deployed to production the project can plan and organize the actual release to users separately. Expert …some organizations choose to make a bigger effort and form complete cross functional teams that can be completely autonomous. With extremely short cycle time and a mature delivery pipeline, such organizations have the confidence to adopt a strict roll-forward only strategy to production failures.
![Page 30: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/30.jpg)
Another(https://www.infoq.com/articles/Continuous-Delivery-Maturity-Model/)
Base … one or more legacy systems of monolithic nature in terms of development, build and release. Many organizations at the base maturity level will have a diversified technology stack but have started to consolidate … to get best value from the effort spent on automation.Beginner … the monolithic structure of the system is addressed by splitting the system into modules … this will also naturally drive an API managed approach to describe internal dependencies and also influence applying a structured approach to manage 3rd party libraries … importance of applying version control to database changes will also reveal itself.Intermediate. … a solid architectural base for continuous delivery … feature hiding for the purpose of minimizing repository branching to enable true continuous integration. … modularization will evolve into identifying and breaking out modules into components that are self-contained and separately deployed. … start migrating scattered and ad-hoc managed application and runtime configuration into version control and treat it as part of the application just like any other code.Advanced. … split the entire system into self contained components and adopted a strict api-based approach to inter-communication so that each component can be deployed and released individually … every component is a self-contained releasable unit with business value, you can achieve small and frequent releases and extremely short release cycles..Expert … some organizations will evolve the component based architecture further and value the perfection of reducing as much shared infrastructure as possible by also treating infrastructure as code and tie it to application components. The result is a system that is totally reproducible from source control, from the O/S and all the way up to application. …
![Page 31: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/31.jpg)
Simplified pipeline
Develop& test
Build Pack Deploy OperateVMS Test
C++Python
![Page 32: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/32.jpg)
Build – which tools you know ?• Make• Old• Declarative• Hard to debug
• Ant• Designed for Java• Based on XML-based configuration language
• Maven
![Page 33: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/33.jpg)
https://developer.ibm.com/urbancode/docs/continuous-delivery-maturity-model/
BUILDING
![Page 34: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/34.jpg)
Testing• Automate, automate, automate• Know any tools?
AUTOMATED
(functionalacceptance tests)
Unit testsIntegration tests
System tests
AUTOMATED
MANUAL
ShowcaseUsability testing
Exploratory testingNonfunctonal
acceptance tests
MANUAL/AUTOM.
Business
Technology
Supportcoding
Critiqueproject
![Page 35: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/35.jpg)
https://developer.ibm.com/urbancode/docs/continuous-delivery-maturity-model/
TESTING
![Page 36: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/36.jpg)
When testing becomes serious business
• Running tests start to take time
ÞTest case selectionÞTest case prioritization
![Page 37: ContinuousDeployment Part2 (technical)Yourtask •You should write Dockerfilesfor the both services and docker-compose.yamlto start both containers so that Service1 is exposed in port](https://reader034.vdocuments.us/reader034/viewer/2022042310/5ed79fc99661ae43ff66a423/html5/thumbnails/37.jpg)
Rought plan – subject to change
• 30.08: Intro to the course and DevOps
• 06.09: Recap (from earlier courses) on virtualization technologies;
• 13.09: Cloud and scalability
• 20.09: Continuous Deployment –what & why
• 27.09: Continuous Deployment –tools and techniques
• 04.10: Security issues; recap on internet networking (from earlier courses)
• 11.10: Guest lecture on AWSAnna Ruokonen, Gofore
• 25.10: Issues on cloud-SW: isolation, dependency management etc,
• 01.11: Cloud-native architectures: microservices (featuring Davide Taibi)
• 08.11: Cloud-native architectures: FaaS/serverless TB104• 15.11: About automation: testing
and beyond
• 22.11: Hosting and deployment issues
• 29.11: Guest lecture on KubernetesRecap
26.9.2019 ssss 37