large-scale software developmenttdde06/lectures/tdde06-lecture2-2018.pdf · 1. single developer,...
TRANSCRIPT
Large-Scale Software DevelopmentTools and Methods
Scenarios - actions1. Single developer, multiple changes — Version control system 2. Many developers, multiple changes — Distributed version control system 3. Many groups of developers, multiple changes — Package management
system 4. Building artefacts based on multiple files with dependencies — build scripts 5. Conducting multiple actions with inter-dependencies on multiple files … —
Flexible build system 6. Automatically sensing changes and conducting such actions based on
changes — Continuous integration tools
2
Scenarios - configuration• Configuration files • Setup scripts • Container environments • Virtual Machines
3
Version control — Git
4
Scenarios - actions1. Single developer, multiple changes — Version control system 2. Many developers, multiple changes — Distributed version control system 3. Many groups of developers, multiple changes — Package management
system 4. Building artefacts based on multiple files with dependencies — build scripts 5. Conducting multiple actions with inter-dependencies on multiple files … —
Flexible build system 6. Automatically sensing changes and conducting such actions based on
changes — Continuous integration tools
5
Version control - GIT
6
Sample Disorganized Project• “Hey, Anders, could you send me a copy of those changes you
made last Tuesday?” • “Ola, this function doesn’t work anymore. Did you change
something?” • “Sorry, I can’t seem to find those old classes. I guess you’ll just
have to re-implement them.” • “OK, we’ve all been working hard for the last week. Now let’s
integrate everyone’s work together.”
7
What is version control?Basic functionality:
keep track of changes made to files (allows roll-backs) merge the contributions of multiple developers
Benefits: facilitates backups increased productivity (vs manual version control) encourages experimentation helps to identify/fix conflicts makes source readily available – less duplicated effort
8
Additional benefitsAccountability
who wrote the code? do we have the rights to it?
Support software engineering hooks for peer reviews
Software branches different versions of software need to be maintained, ensure bug fixes shared
Record Keeping Commit logs may tie to issue tracking system or be used to enforce guidelines
9
More BenefitsSupport Distribution of Work
Remote work, outsourcing, open-source projects Use in conjunction with “good communication habits” – via email etc.
Rapid Development (XP/Agile) Supports frequent refactoring Helps automate frequent system builds
10
The Big Picture
http://blog.mikepearce.net/2010/05/18/the-difference-between-git-pull-git-fetch-and-git-clone-and-git-rebase/
Branching
• Gitseescommitthisway…• Branchannotateswhichcommitweareworkingon
12
Branching
13
14
15
16
17
18
19
20
21
Merging
• Whatdowedowiththismess?– Mergethem
22
Merging
• Stepstomergetwobranch– Checkoutthebranchyouwanttomergeonto– Mergethebranchyouwanttomerge
23
24
25
26
27
28
29
Scenarios - actions1. Single developer, multiple changes — Version control system 2. Many developers, multiple changes — Distributed version control system 3. Many groups of developers, multiple changes — Package management
system 4. Building artefacts based on multiple files with dependencies — build scripts 5. Conducting multiple actions with inter-dependencies on multiple files … —
Flexible build system 6. Automatically sensing changes and conducting such actions based on
changes — Continuous integration tools
30
Package management systems
31
Dependency management
32
Dependency management issues• Is a request to modify the current software component graph
satisfiable? – Are additions compatible with other components? – Are deletions safe with respect to other dependencies?
• Given a component, determine versions of other components we can safely rely on
33
Dependency management as satisfiability
34
(a | b | c) & (d | e | f) ... = TRUE
(a | b | c) & (-c) & (-b | -a) ... = TRUE
A requires B provided by B1, B2, B3Rule: (-A | B1 | B2 | B3)
A conflicts with B provided by B1, B2, B33 Rules: (-A | -B1), (-A | -B2), (-A | -B3)
Rules
Dependency management issues• Y depends on X >= 1.8. X makes binary incompatible changes from
v. 1.9 to v. 2.0... • Can components be installed from local sources as well as from
remote? • Should OS-specific dependency management or language-specific
be used?
35
Software package management systems
36
Name Environment Format
NuGet .Net CLR XML
Gradle JVM XML
dpkg/APT Linux Ar archive
Rubygems Ruby Ruby
MSI Windows In-file DB
BSD Ports OS X/Linux/BSD Makefile
...
Maven
37
mvn -h
Lifecycles
Clean
Default
Site
validatecompiletestpackageverifyinstalldeploy
Maven example 38
Olas-MacBook-Pro:java-petclinic olale$ mvn graphwalker:test[INFO] Scanning for projects...[ … ][INFO] >>> graphwalker-maven-plugin:3.4.2:test (default-cli) > [graphwalker]test-compile @ java-petclinic >>>[INFO][INFO] --- graphwalker-maven-plugin:3.4.2:generate-sources (generate-sources) @ java-petclinic ---[INFO][INFO] --- graphwalker-maven-plugin:3.4.2:validate-models (default-cli) @ java-petclinic ---[INFO][INFO] --- graphwalker-maven-plugin:3.4.2:generate-sources (default-cli) @ java-petclinic ---[INFO][INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ java-petclinic ---[INFO] Using 'UTF-8' encoding to copy filtered resources.[INFO] Copying 10 resources[INFO][INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ java-petclinic ---[INFO] Nothing to compile - all classes are up to date[INFO][INFO] --- graphwalker-maven-plugin:3.4.2:validate-test-models (default-cli) @ java-petclinic ---
Maven — structure
39
mvn archetype:generate -DgroupId=com.mycompany.app -DartifactId=my-app -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false
Maven — Configuration
40
<parent> <groupId>org.graphwalker.example</groupId> <artifactId>graphwalker-example</artifactId> <version>3.4.2</version> </parent>
<artifactId>java-petclinic</artifactId>
<build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.1</version> <configuration> <source>1.7</source> <target>1.7</target> </configuration> </plugin> <plugin> <groupId>org.graphwalker</groupId> <artifactId>graphwalker-maven-plugin</artifactId> <version>${project.version}</version> <!-- Bind goals to the default lifecycle --> <executions> <execution> <id>generate-sources</id> <phase>generate-sources</phase> <goals> <goal>generate-sources</goal> </goals> </execution> </executions> </plugin> </plugins> </build>
Automation and languages for … development41
ASM
C
Java
Compiler
JVM
OS
Buildtools
Testautomation,packaging,…CIworkautomationMultipledevelopers/languages/components
Scenarios - actions1. Single developer, multiple changes — Version control system 2. Many developers, multiple changes — Distributed version control system 3. Many groups of developers, multiple changes — Package management
system 4. Building artefacts based on multiple files with dependencies — build scripts 5. Conducting multiple actions with inter-dependencies on multiple files … —
Flexible build system 6. Automatically sensing changes and conducting such actions based on
changes — Continuous integration tools
42
Jenkins
43
Jenkins 44
Workflowautomationtool
Jenkins 45
Workflowautomationtool-pipelines
node{//<1>stage('Build'){//<2>sh'make'//<3>}
stage('Test'){sh'makecheck'junit'reports/**/*.xml'//<4>}
stage('Deploy'){sh'makepublish'}}
Groovy(JVM-basedlanguage)
Travis CI 46
Automation and languages for … deployment
47
Singlefile,structurednon-programminglanguage.Requirementsinstalled,updated,startedmanually.Deploymentthroughmanualscripts
Programminglanguage Automationofsetup
Chef/Puppet/Ansible/Salt/Terraform
Isolationofcomponents
Xen/SolarisZones,Docker/Chroot
Chef/Puppet/Ansible/Salt/Terraform/…• Automates setup of environments for development/deployment • Introduces programming languages for your infrastructures
• Declarative (Terraform/Puppet) - What is the desired end-state? • Imperative (Chef/Ansible) - How do you achieve it?
48
49
Deployment
50
Market View: Evolution of IT1995 2015
Runningonanyavailablesetof
physicalresources(public/private/virtualized)
Assembledbydevelopersusingbestavailable
services
Thinapponmobile,tabletThick,client-serverapp
onthickclient
Well-definedstack:-O/S-Runtime-Middleware
MonolithicPhysical
Infrastructure
Challenges2015
Howtoensureservicesinteractconsistently,avoid
dependencyhell
Howtomigrate&scalequickly,ensurecompatibility
HowtoavoidnXndifferentconfigs
Runningonanyavailablesetof
physicalresources(public/private/virtualized)
Assembledbydevelopersusingbestavailable
services
Thinapponmobile,tablet
Static website
Web frontend
User DB
Queue Analytics DB
Background workers
API endpoint
nginx 1.5 + modsecurity + openssl + bootstrap 2
postgresql + pgv8 + v8
hadoop + hive + thrift + OpenJDK
Ruby + Rails + sass + Unicorn
Redis + redis-sentinel
Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs + phantomjs
Python 2.7 + Flask + pyredis + celery + psycopg + postgresql-client
Development VM
QA server
Public Cloud
Disaster recovery
Contributor’s laptopProduction Servers
The ChallengeMultip
licity
ofS
tacks
Multip
licity
of
hardware
environm
ents
Production Cluster
Customer Data Center
Doservicesandappsinteract
appropriately?
CanImigrate
smoothlyandquickly?
Results in NxN compatibility nightmareStatic website
Web frontend
Background workers
User DB
Analytics DB
Queue
Development VM QA Server Single Prod
ServerOnsite Cluster
Public Cloud
Contributor’s laptop
Customer Servers
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
Static website Web frontend User DB Queue Analytics DB
Development VM
QA server Public Cloud Contributor’s laptop
Placing code in movable boxes Multip
licity
ofStacks
Multip
licity
of
hardware
environm
ents
Production Cluster
Customer Data Center
Doservicesandappsinteract
appropriately?
CanImigrate
smoothlyandquickly
…that can be manipulated using standard operations and run consistently on virtually any hardware platform
An engine that enables any payload to be encapsulated as a lightweight, portable, self-sufficient container…
Static website Web frontend User DB Queue Analytics DB
Development VM
QA server Public Cloud Contributor’s laptop
Or…put more simplyMultip
licity
ofStacks
Multip
licity
of
hardware
environm
ents
Production Cluster
Customer Data Center
Doservicesandappsinteract
appropriately?
CanImigrate
smoothlyandquickly
Operator: Configure Once, Run Anything
Developer: Build Once, Run Anywhere (Finally)
Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
Development VM QA Server Single Prod
ServerOnsite Cluster
Public Cloud
Contributor’s laptop
Customer Servers
Containers tries solves the NxN problem
Two types of boxes
• Virtualization (Virtual machines)• Xen • KVM• QEMU • Virtual Box • UML • ….
• Containers • Docker (LXC)• LXD• CoreOS (rkt)
59
What is Docker?Docker is an open-source project that automates the deployment of applications inside software containers, by providing an additional layer of abstraction and automation of operating system–level virtualization on Linux.
[Source: en.wikipedia.org]
60
Docker: Namedocker [naut.]: der Dockarbeiter, der Hafenarbeiter
Source: leo.org
• Provide a uniformed wrapper around a software package: «Build, Ship and Run Any App, Anywhere» [www.docker.com]
– Similar to shipping containers: The container is always the same, regardless of the contents and thus fits on all trucks, cranes, ships, ...
[www.docker.com]
61
Docker vs. Virtual Machine
Source: https://www.docker.com/whatisdocker/
62
Docker Technology
• libvirt: Platform Virtualization • LXC (LinuX Containers): Multiple isolated
Linux systems (containers) on a single host • Layered File System
[Source: https://docs.docker.com/terms/layer/]
63
Run Platforms• Various Linux distributions (Ubuntu, Fedora, RHEL, Centos,
openSUSE, ...) • Cloud (Amazon EC2, Google Compute Engine, Rackspace,
Azure ) • Mac • Windows
64
Hello WorldSimple Command - Ad-Hoc Container • docker run ubuntu echo Hello World – docker images [-a]
– docker ps –a
65
Terminology - Image• Persisted snapshot that can be run
– images: List all local images – run: Create a container from an image and
execute a command in it – tag: Tag an image – pull: Download image from repository – rmi: Delete a local image
• This will also remove intermediate images if no longer used
66
Terminology - Container• Runnable instance of an image
– ps: List all running containers – ps –a: List all containers (incl. stopped) – top: Display processes of a container – start: Start a stopped container – stop: Stop a running container – pause: Pause all processes within a container – rm: Delete a container – commit: Create an image from a container
Container cid4
Container cid3
67
Image vs. ContainerBase Image ubuntu:latest
Container cid1
run
Container cid1
cmd à new state
New Image iid1
commit
base image
Container cid2
run
68
Dockerfile• Create images automatically using a build script: «Dockerfile» • Can be versioned in a version control system like Git or SVN,
along with all dependencies • Docker Hub can automatically build images based on
dockerfiles on Github
69
Dockerfile Example• Dockerfile:
– FROM ubuntu ENV DOCK_MESSAGE Hello My World ADD dir /files CMD ["bash", "someScript"]
• docker build [DockerFileDir]
• docker inspect [imageId]
70
FROM ubuntu
MAINTAINER Anders
RUN apt-get install -y python-software-properties python python-setuptools ruby rubygems
RUN add-apt-repository ppa:chris-lea/node.js
RUN echo "deb http://us.archive.ubuntu.com/ubuntu/ precise universe" >> /etc/apt/sources.list
RUN apt-get update
RUN apt-get install -y nodejs
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
RUN echo "deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen" | tee -a /etc/apt/sources.list.d/10gen.list
RUN apt-get -y update
RUN apt-get -y install mongodb-10gen
RUN easy_install supervisor
RUN echo_supervisord_conf > /etc/supervisord.conf
RUN printf "[include]\nfiles = /var/www/Supervisorfile\n" >> /etc/supervisord.conf
ADD . /var/www
RUN cd /var/www ; npm install
CMD ["/usr/local/bin/supervisord", "-n", "-c", "/etc/supervisord.conf"]
71
Mount Volumes• docker run –ti –v /hostLog:/log ubuntu
• Run second container: Volume can be shared – docker run –ti --volumes-from firstContainerName ubuntu
72
Publish Port• docker run –t –p 8080:80 ubuntu nc –l 80
– Map container port 80 to host port 8080 – Check on host: nc localhost 8080
• Link with other docker container – docker run -ti --link containerName:alias ubuntu
– See link info with set
73
Around Docker• Docker Images: Docker Hub • Vagrant (Also for VMs) • Automated Setup
– Puppet, Chef, Ansible, ...
74
Docker Hub• Public repository of Docker images
– https://hub.docker.com/ – docker search [term]
• Automated: Has been automatically built from Dockerfile – Source for build is available on GitHub