continuous integration and delivery of research applications
Post on 16-Jul-2015
204 Views
Preview:
TRANSCRIPT
Continuous Integration andDelivery of
Research Applications
Bruce Beckerbbecker@csir.co.za
Coordinator, AAROCMeraka Institute, CSIR
Fanie RiekertRiekertSJPK@ufs.ac.za
Senior Systems EngineerHPC Unit, UFS
Steps towards an Open e-Science Commons
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Outline● Compuational Infrastructure: It's a jungle out
there● How does research actually get done ? A
reminder of the scientific method in the 3rd and 4th paradigm
● How do applications get onto an infrastructure ? ● What are the issues ? ● What are we doing about it ? ● What can you do about it ?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
3
Functional Sites
Core Services
6 sites~2300 CPU cores~100 TB perm. Storage~17 permanent staff
Proposed extensions
So, we built this...
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
4
And others are too
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
5
Universite' Gaston Berger(Saint-Louis)
Kwame NkrumahUniversity of ScienceAnd Technology (Kumasi, Ghana)
Nsukka University(Nsukka)
http://cds.cern.ch/record/1476353
220 servers, 30 routers
And others are too
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
6
And it is part of this
blue: EGI council members and EGI-InSPIRE partners
green: integrated resource infrastructure providers
purple: peer resource infrastructure providershttp://www.egi.eu/infrastructure/
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
7
Great, right ?!
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
It's (apparently) a jungle out there.http://en.wikipedia.org/wiki/Jungle_computing
Seinstra, F. J., Maassen, J., van Nieuwpoort, R.~V., et al. 2011, Grids, Clouds and Virtualization, 167
Jungle Computing: Distributed Supercomputing beyond Clusters, Grids, and Clouds
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Where does your code run ?● On your laptop ?
● Congrats ! Uh oh, you just rand out of X● You want to do what !?
● Local cluster ?● hm. different setup... <hack hack>
hey it's working ! … kinda.● Permissions, environment,
data staging, etc.● CHPC ?
● There's more than one● ALL the Optimisations !!
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
10
Are you even talking about the same application ?
● Apart from performance, are you sure that you are getting the same results on different architectures and platforms ?
● Is it even worth the effort ?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
11
The problem
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
What users want
Amazing infrastructure
Some users want highly varied, modular
application selection
Vertically integratedHighly specialised
applications
Highly trained supportHighly trained support
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
What users get sometimes
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Current Methods
Are you a legitscientist ?
Ok, sounds goodwhat applications are you running ?
We can run almost anything !
Hi, I heard aboutAfricaGrid – sounds
grrrrreat ! Can I Use it ?
I've got beakers…
And glasses !
Awesome ! Well, first we do some preprocessing with RaNDo, then we run a massively parallel pipeline with XBOT, then blah and blah then blah
blah blah and blah, we also need blah blah and sometimes our colleagues from FarAwayU need to access the data with StrangeProtocolx1, and of
course blah blah, not to mention blah
…..
We'll see what we can do
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Actual workflow● Find the source code
● Unpack, ./configure && make … Error 1 missing x.h● … hack hack hack … hm. Missing dependencies
● Recursively* determine the dependencies**● *Yes, recursively means curse, curse then curse again.● **The actual dependencies, not the ones that were written on the
web page from 5 years ago.● … mutter mutter …● ???? Why are you compiling now !?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Actual communicationHey, how's it going ?
you done with my porting yet ?
You done yet ?
You done ?
…. bloody IT people !
Have you actually seen this code ?
I just... It looks like Cthulhu's beard....
Ok, I got it to compile, but only with if you
type with your left toes.
AAArrrgh, I'm a scientist, not an IT people !
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
How many applications are there ?
Natural Sciences
Other
Medical and Health SciencesSupport ActivitiesEngineering and TechnologyAgricultural SciencesHumanitiesSocial Sciences
http://appdb.egi.eu
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Way too many...(and these are just the ones we
know about)
Biological sciences
Bioinformatics
Biochemistry and molecular biology
Structural biologyBiophysicsGenetics and heredityCell biologyEvolutionary biology
Physical sciences
Astronomy
Astroparticle physics
Astrophysics
High energy physicsFusionPlasma physicsFluid MechanicsCondensed matter physicsMathematical physicsMedical physicsOpticsPhysicsAcoustics
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
There's gotta be a better way
Let's give researchers a ”better deal”
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
20
• Hypothesis 1: – It always comes down to an application
• At the end of all first-contact you get a list of applications that users want
– Let's focus on that, shall we ?
• Hypothesis 2: – No software is an island
• Everything has dependencies, even if it's just glibc or libgfortran
– Let's make those explicit, shall we ?
Compuatational InfrastructureWhat is it good for ?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
21
• Hypothesis 3:– Applications need an environment
• Compiler ? • Architecture ? • Dependencies ? • Middleware ?• Etc....
– Let's make those explicit shall we ?
Compuatational InfrastructureWhat is it good for ?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
22
• Hypothesis 4: – there is more than one environment.
• Nothing is going to ever be really useful if it can only run in one place.
• Nobody wants to be forced into a single environment (grid/HPC/cloud, etc)
– Let's try to simulate as many target environments (sites) as we can, shall we?
Compuatational InfrastructureWhat is it good for ?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
23
Compuatational InfrastructureWhat is it good for ?
● Hypothesis 5: ● Solutions decay
– The solution – as well as the dead ends – are part of the process.
– If it's not in a version-controlled repository, it's no good.– If it's not executable (in a wiki/document, etc), then you
have to convert it back into something a computer understands – why not just leave it in executable format ?
● Let's recognise that solutions decay and code for the future, shall we ?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
24
Humans need not apply(for certain tasks)
● Hypothesis 6: ● Most of the work of porting and supporting applications
can be automated– Reduce the load on the user and the operator, by automating
common tasks
● Let's:– Integrate the various software, middleware, service and
infrastructure components as far as possible, – Keep it Open
● … shall we ?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
25
Compuatational InfrastructureWhat is it good for ?
● Hypothesis 6 (Corollary) : You're in the way
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
26
Compuatational InfrastructureWhat is it good for ?
● Hypothesis 7: ● This stuff's not hard
– Continuous Integration has been around for a long time– If it can be done by hand, it can be scripted; if it can be
scripted, it can be automated properly. ● Let's
– adopt widely-used tools and methodologies, and – hey, how about we use the web to communciate
(you know, like we do with EVERYTHING else ?)
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Let's work on a better deal...● How do we lower the barrier to entry to the grid or
cloud infrastructure ?● How can the application expert prove to the resource
provider that the application will actually run on the execution environment of the site ?
● How can we manage the lifecycle of applications across multiple versions, architectures, configurations ?
● How can we ensure that once applications are ”certified”, they are actually available on as many sites as possible ?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Go to Bootcamp
http://software-carpentry.org/bootcamps
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
29
What are we actually doing ?● Put the ball in the developer/community/user's court
● You tell us how to build your application● Provide a simple interface to porting pipeline● Provide focussed, goal-oriented support when needed.● Communication has a context and is directly linked to the issues at
hand● The entry point is open
● Extensible and reproducible● Publish and preserve the entire toolchain
● It's a real collaboration now – with code !● Can we say the big S word ? (not the 4-letter word - SUSTAINABLE)
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
30
Tools of the Trade (1)
Waffle.ioProjectplanning
Slack.comTeam collab
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
31
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
32
What's the plan, Stan ?https://waffle.io/aaroc/devops
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Typical workflow and roles
Test
ing
mat
rix
Defines relevanttests in Jenkins
Writes code to pass required tests
Dev/Stage env.Application developer
Infrastructure expert
Reads descriptionof execution environment tests
Promote a buildto CVMFS
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
My Github brings all the hackers to the backyard
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Social coding
Identify yourself, create a branch
Jenkins will help you test your codeat every commit
Contribute back, open a pull request
Discuss contrbutions
https://guides.github.com/introduction/flow/index.html
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Get credit for your work
Bruce Becker, Marco Fargetta, pkoro, Pavlos Daoglou, Chris Lee, et al. (2014)
DevOps: AAROC Ansible LDAP role release.
Available:http://dx.doi.org/10.5281/zenodo.11914.
https://github.com/AAROC/DevOps/tree/v0.0.2
DOIDOI 10.5281/zenodo.1023310.5281/zenodo.10233
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Sounds great, show me the details
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
38
Application porting, testing, integration and deployment workflow
CreateBuild Script
Pull Request
Create Jenkins Job
Build
if first time
Invoke Test
if buildhas passed
Fail Build
Build Fails Build Passes
Follow upwith the user Fail Test
Tests Fail
Invoke Artifact
Tests Pass
Open Issueon Github
Stage toTesting Repo
Field Tests
Update RepoVersion
Lock CVMFSrepo
Field Tests Pass
Open Issueon Github
Field Tests Fail
DeployArtifact
Update RepoVersion
Tag sites
Application porting, testing, integration and deployment workflow
CreateBuild Script
Pull Request
Create Jenkins Job
Build
if first time
Invoke Test
if buildhas passed
Fail Build
Build Fails Build Passes
Follow upwith the user Fail Test
Tests Fail
Invoke Artifact
Tests Pass
Open Issueon Github
Stage toTesting Repo
Field Tests
Update RepoVersion
Lock CVMFSrepo
Field Tests Pass
Open Issueon Github
Field Tests Fail
DeployArtifact
Update RepoVersion
Tag sites
The new pipeline
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
39
Public View
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
40
Authentication / Authorisation
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
41
Authenticated View
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
42
Does it work !?
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
43
Yes, it works.
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
44
Let's get very explicit
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
45
Debugging
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
cat <<MODULE_FILE#%Module1.0## $NAME modulefile##proc ModulesHelp { } { puts stderr " This module does nothing but alert the user" puts stderr " that the [moduleinfo name] module is not available"}preqreq("gsl","fftw/2.1.5","hdf5")modulewhatis "$NAME $VERSION."setenv GSL_VERSION $VERSIONsetenv GSL_DIR /apprepo/$::env(SITE)/$::env(OS)/$::env(ARCH)/$NAME/$VERSIONprependpath LD_LIBRARY_PATH $::env(GSL_DIR)/libMODULE_FILE) > modules/$VERSION
cat <<MODULE_FILE#%Module1.0## $NAME modulefile##proc ModulesHelp { } { puts stderr " This module does nothing but alert the user" puts stderr " that the [moduleinfo name] module is not available"}preqreq("gsl","fftw/2.1.5","hdf5")modulewhatis "$NAME $VERSION."setenv GSL_VERSION $VERSIONsetenv GSL_DIR /apprepo/$::env(SITE)/$::env(OS)/$::env(ARCH)/$NAME/$VERSIONprependpath LD_LIBRARY_PATH $::env(GSL_DIR)/libMODULE_FILE) > modules/$VERSION
Generic build script.# GADGET requires HDF5 FFTW2 ZLIB and openmpimodule add cimodule add fftw/2.1.5module add hdf5module add openmpimodule add gsl
# GADGET requires HDF5 FFTW2 ZLIB and openmpimodule add cimodule add fftw/2.1.5module add hdf5module add openmpimodule add gsl
Set up theenvironment
make install DESTDIR=$WORKSPACE/buildmkdir p $REPO_DIRrm rf $REPO_DIR/*tar cvzf $REPO_DIR/build.tar.gz C $WORKSPACE/build apprepo
make install DESTDIR=$WORKSPACE/buildmkdir p $REPO_DIRrm rf $REPO_DIR/*tar cvzf $REPO_DIR/build.tar.gz C $WORKSPACE/build apprepoActually build...
Create the artifact
Create the modulefile
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
There's gotta be a better way !Literate/YAML/Templates...
---author: Bruce Beckerrepo: https://github.com/SAGridOps/SoftwareInstallationname: GCCtarget: - sites: - generic - mpi - mpi_infiniband - OS: - sl6 - u1404versions: - 4.6.1category: compilersdependencies: - [ name: mpfr, version: 3.1.2 ]url: http://gnu.org/path/to/gcc.tar.gzbuild_script: build.sh
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
48
Last Bit: Delivery
Executable Artifactsgo here
Every site can mountthe CVMFS repoEven your laptop...
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
49
Sound Familiar ?
Executable Artifactsgo here
This will take a lot of work, but there's a huge community behind the tools.
Deploy into linuxcontainers
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
50
Do you even remember the title of this talk ?
EVERYTHING=
CODE
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
51
What if we could automate infrastructure execution too ?
● We can and are, but there's no time to get into that right now.● It's a jungle out there – but we know most of the animals.● Configuration and integration are the main issues.● Turn it into … a zoo ? (kinda)
● See https://github.com/AAROC/DevOps● Yes, we need help
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
52
Summary● Making the best use of computational infrastructure comes
down to running applications● Maintaining and porting them is hard and tricky.
Communication is the hardest part.● We've built an automated porting system, which
● will deliver functional, tested, relevant software to your site.● Can ease the communication blocks in collaborative work
● Oh and it works :)● Come on in and help us build it.
We want an Open e-Science Commons
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
53
What can you do ?
● Join the orgs on Github● Propose applications
● Fork, branch, pull request.
● Help develop the workflow● Tell your friends
Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
54
Thanks !
● Join the organisations:● http://github.com/AAROC/● http://github.com/SAGridOps
● Let's talk : http://aaroc.github.com/ ● https://ciari.slack.com/ (ask for an invite)● @TheSAGrid● @brusisceddu● fb.com/SAGrid
top related