continuous integration and delivery of research applications

Continuous Integration andDelivery of

Research Applications

Bruce [email protected]

Coordinator, AAROCMeraka Institute, CSIR

Fanie [email protected]

Senior Systems EngineerHPC Unit, UFS

Steps towards an Open e-Science Commons

Bruce Becker: Coordinator, SAGrid | [email protected] | http://www.sagrid.ac.za

Outline● Compuational Infrastructure: It's a jungle out

there● How does research actually get done ? A

reminder of the scientific method in the 3rd and 4th paradigm

● How do applications get onto an infrastructure ? ● What are the issues ? ● What are we doing about it ? ● What can you do about it ?

mailto:[email protected]


3

Functional Sites

Core Services

6 sites~2300 CPU cores~100 TB perm. Storage~17 permanent staff

Proposed extensions

So, we built this...



4

And others are too



5

Universite' Gaston Berger(Saint-Louis)

Kwame NkrumahUniversity of ScienceAnd Technology (Kumasi, Ghana)

Nsukka University(Nsukka)

http://cds.cern.ch/record/1476353

220 servers, 30 routers

And others are too





6

And it is part of this

blue: EGI council members and EGI-InSPIRE partners

green: integrated resource infrastructure providers

purple: peer resource infrastructure providershttp://www.egi.eu/infrastructure/



7

Great, right ?!



It's (apparently) a jungle out there.http://en.wikipedia.org/wiki/Jungle_computing

Seinstra, F. J., Maassen, J., van Nieuwpoort, R.~V., et al. 2011, Grids, Clouds and Virtualization, 167

Jungle Computing: Distributed Supercomputing beyond Clusters, Grids, and Clouds



Where does your code run ?● On your laptop ?

● Congrats ! Uh oh, you just rand out of X● You want to do what !?

● Local cluster ?● hm. different setup... <hack hack>

hey it's working ! … kinda.● Permissions, environment,

data staging, etc.● CHPC ?

● There's more than one● ALL the Optimisations !!



10

Are you even talking about the same application ?

● Apart from performance, are you sure that you are getting the same results on different architectures and platforms ?

● Is it even worth the effort ?



11

The problem



What users want

Amazing infrastructure

Some users want highly varied, modular

application selection

Vertically integratedHighly specialised

applications

Highly trained supportHighly trained support



What users get sometimes



Current Methods

Are you a legitscientist ?

Ok, sounds goodwhat applications are you running ?

We can run almost anything !

Hi, I heard aboutAfricaGrid – sounds

grrrrreat ! Can I Use it ?

I've got beakers…

And glasses !

Awesome ! Well, first we do some preprocessing with RaNDo, then we run a massively parallel pipeline with XBOT, then blah and blah then blah

blah blah and blah, we also need blah blah and sometimes our colleagues from FarAwayU need to access the data with StrangeProtocolx1, and of

course blah blah, not to mention blah

…..

We'll see what we can do



Actual workflow● Find the source code

● Unpack, ./configure && make … Error 1 missing x.h● … hack hack hack … hm. Missing dependencies

● Recursively* determine the dependencies**● *Yes, recursively means curse, curse then curse again.● **The actual dependencies, not the ones that were written on the

web page from 5 years ago.● … mutter mutter …● ???? Why are you compiling now !?



Actual communicationHey, how's it going ?

you done with my porting yet ?

You done yet ?

You done ?

…. bloody IT people !

Have you actually seen this code ?

I just... It looks like Cthulhu's beard....

Ok, I got it to compile, but only with if you

type with your left toes.

AAArrrgh, I'm a scientist, not an IT people !



How many applications are there ?

Natural Sciences

Other

Medical and Health SciencesSupport ActivitiesEngineering and TechnologyAgricultural SciencesHumanitiesSocial Sciences

http://appdb.egi.eu



Way too many...(and these are just the ones we

know about)

Biological sciences

Bioinformatics

Biochemistry and molecular biology

Structural biologyBiophysicsGenetics and heredityCell biologyEvolutionary biology

Physical sciences

Astronomy

Astroparticle physics

Astrophysics

High energy physicsFusionPlasma physicsFluid MechanicsCondensed matter physicsMathematical physicsMedical physicsOpticsPhysicsAcoustics



There's gotta be a better way

Let's give researchers a ”better deal”



20

• Hypothesis 1: – It always comes down to an application

• At the end of all first-contact you get a list of applications that users want

– Let's focus on that, shall we ?

• Hypothesis 2: – No software is an island

• Everything has dependencies, even if it's just glibc or libgfortran

– Let's make those explicit, shall we ?

Compuatational InfrastructureWhat is it good for ?



21

• Hypothesis 3:– Applications need an environment

• Compiler ? • Architecture ? • Dependencies ? • Middleware ?• Etc....

– Let's make those explicit shall we ?




22

• Hypothesis 4: – there is more than one environment.

• Nothing is going to ever be really useful if it can only run in one place.

• Nobody wants to be forced into a single environment (grid/HPC/cloud, etc)

– Let's try to simulate as many target environments (sites) as we can, shall we?




23


● Hypothesis 5: ● Solutions decay

– The solution – as well as the dead ends – are part of the process.

– If it's not in a version-controlled repository, it's no good.– If it's not executable (in a wiki/document, etc), then you

have to convert it back into something a computer understands – why not just leave it in executable format ?

● Let's recognise that solutions decay and code for the future, shall we ?



24

Humans need not apply(for certain tasks)

● Hypothesis 6: ● Most of the work of porting and supporting applications

can be automated– Reduce the load on the user and the operator, by automating

common tasks

● Let's:– Integrate the various software, middleware, service and

infrastructure components as far as possible, – Keep it Open

● … shall we ?



25


● Hypothesis 6 (Corollary) : You're in the way



26


● Hypothesis 7: ● This stuff's not hard

– Continuous Integration has been around for a long time– If it can be done by hand, it can be scripted; if it can be

scripted, it can be automated properly. ● Let's

– adopt widely-used tools and methodologies, and – hey, how about we use the web to communciate

(you know, like we do with EVERYTHING else ?)



Let's work on a better deal...● How do we lower the barrier to entry to the grid or

cloud infrastructure ?● How can the application expert prove to the resource

provider that the application will actually run on the execution environment of the site ?

● How can we manage the lifecycle of applications across multiple versions, architectures, configurations ?

● How can we ensure that once applications are ”certified”, they are actually available on as many sites as possible ?



Go to Bootcamp

http://software-carpentry.org/bootcamps



29

What are we actually doing ?● Put the ball in the developer/community/user's court

● You tell us how to build your application● Provide a simple interface to porting pipeline● Provide focussed, goal-oriented support when needed.● Communication has a context and is directly linked to the issues at

hand● The entry point is open

● Extensible and reproducible● Publish and preserve the entire toolchain

● It's a real collaboration now – with code !● Can we say the big S word ? (not the 4-letter word - SUSTAINABLE)



30

Tools of the Trade (1)

Waffle.ioProjectplanning

Slack.comTeam collab


http://waffle.io/

http://sagrid.slack.com/


31



32

What's the plan, Stan ?https://waffle.io/aaroc/devops



Typical workflow and roles

Test

ing

mat

rix

Defines relevanttests in Jenkins

Writes code to pass required tests

Dev/Stage env.Application developer

Infrastructure expert

Reads descriptionof execution environment tests

Promote a buildto CVMFS



My Github brings all the hackers to the backyard



Social coding

Identify yourself, create a branch

Jenkins will help you test your codeat every commit

Contribute back, open a pull request

Discuss contrbutions

https://guides.github.com/introduction/flow/index.html


https://guides.github.com/introduction/flow/index.html


Get credit for your work

Bruce Becker, Marco Fargetta, pkoro, Pavlos Daoglou, Chris Lee, et al. (2014)

DevOps: AAROC Ansible LDAP role release.

Available:http://dx.doi.org/10.5281/zenodo.11914.

https://github.com/AAROC/DevOps/tree/v0.0.2

DOIDOI 10.5281/zenodo.1023310.5281/zenodo.10233


https://github.com/AAROC/DevOps/tree/v0.0.2


Sounds great, show me the details



38

Application porting, testing, integration and deployment workflow

CreateBuild Script

Pull Request

Create Jenkins Job

Build

if first time

Invoke Test

if buildhas passed

Fail Build

Build Fails Build Passes

Follow upwith the user Fail Test

Tests Fail

Invoke Artifact

Tests Pass

Open Issueon Github

Stage toTesting Repo

Field Tests

Update RepoVersion

Lock CVMFSrepo

Field Tests Pass

Open Issueon Github

Field Tests Fail

DeployArtifact

Update RepoVersion

Tag sites

Application porting, testing, integration and deployment workflow

CreateBuild Script

Pull Request

Create Jenkins Job

Build

if first time

Invoke Test

if buildhas passed

Fail Build

Build Fails Build Passes

Follow upwith the user Fail Test

Tests Fail

Invoke Artifact

Tests Pass

Open Issueon Github

Stage toTesting Repo

Field Tests

Update RepoVersion

Lock CVMFSrepo

Field Tests Pass

Open Issueon Github

Field Tests Fail

DeployArtifact

Update RepoVersion

Tag sites

The new pipeline



39

Public View



40

Authentication / Authorisation



41

Authenticated View



42

Does it work !?



43

Yes, it works.



44

Let's get very explicit



45

Debugging



cat <<MODULE_FILE#%Module1.0## $NAME modulefile##proc ModulesHelp { } { puts stderr " This module does nothing but alert the user" puts stderr " that the [moduleinfo name] module is not available"}preqreq("gsl","fftw/2.1.5","hdf5")modulewhatis "$NAME $VERSION."setenv GSL_VERSION $VERSIONsetenv GSL_DIR /apprepo/$::env(SITE)/$::env(OS)/$::env(ARCH)/$NAME/$VERSIONprependpath LD_LIBRARY_PATH $::env(GSL_DIR)/libMODULE_FILE) > modules/$VERSION

cat <<MODULE_FILE#%Module1.0## $NAME modulefile##proc ModulesHelp { } { puts stderr " This module does nothing but alert the user" puts stderr " that the [moduleinfo name] module is not available"}preqreq("gsl","fftw/2.1.5","hdf5")modulewhatis "$NAME $VERSION."setenv GSL_VERSION $VERSIONsetenv GSL_DIR /apprepo/$::env(SITE)/$::env(OS)/$::env(ARCH)/$NAME/$VERSIONprependpath LD_LIBRARY_PATH $::env(GSL_DIR)/libMODULE_FILE) > modules/$VERSION

Generic build script.# GADGET requires HDF5 FFTW2 ZLIB and openmpimodule add cimodule add fftw/2.1.5module add hdf5module add openmpimodule add gsl

# GADGET requires HDF5 FFTW2 ZLIB and openmpimodule add cimodule add fftw/2.1.5module add hdf5module add openmpimodule add gsl

Set up theenvironment

make install DESTDIR=$WORKSPACE/buildmkdir p $REPO_DIRrm rf $REPO_DIR/*tar cvzf $REPO_DIR/build.tar.gz C $WORKSPACE/build apprepo

make install DESTDIR=$WORKSPACE/buildmkdir p $REPO_DIRrm rf $REPO_DIR/*tar cvzf $REPO_DIR/build.tar.gz C $WORKSPACE/build apprepoActually build...

Create the artifact

Create the modulefile



There's gotta be a better way !Literate/YAML/Templates...

---author: Bruce Beckerrepo: https://github.com/SAGridOps/SoftwareInstallationname: GCCtarget: - sites: - generic - mpi - mpi_infiniband - OS: - sl6 - u1404versions: - 4.6.1category: compilersdependencies: - [ name: mpfr, version: 3.1.2 ]url: http://gnu.org/path/to/gcc.tar.gzbuild_script: build.sh



48

Last Bit: Delivery

Executable Artifactsgo here

Every site can mountthe CVMFS repoEven your laptop...



49

Sound Familiar ?

Executable Artifactsgo here

This will take a lot of work, but there's a huge community behind the tools.

Deploy into linuxcontainers



50

Do you even remember the title of this talk ?

EVERYTHING=

CODE



51

What if we could automate infrastructure execution too ?

● We can and are, but there's no time to get into that right now.● It's a jungle out there – but we know most of the animals.● Configuration and integration are the main issues.● Turn it into … a zoo ? (kinda)

● See https://github.com/AAROC/DevOps● Yes, we need help


https://github.com/AAROC/DevOps


52

Summary● Making the best use of computational infrastructure comes

down to running applications● Maintaining and porting them is hard and tricky.

Communication is the hardest part.● We've built an automated porting system, which

● will deliver functional, tested, relevant software to your site.● Can ease the communication blocks in collaborative work

● Oh and it works :)● Come on in and help us build it.

We want an Open e-Science Commons



53

What can you do ?

● Join the orgs on Github● Propose applications

● Fork, branch, pull request.

● Help develop the workflow● Tell your friends



54

Thanks !

● Join the organisations:● http://github.com/AAROC/● http://github.com/SAGridOps

● Let's talk : http://aaroc.github.com/ ● https://ciari.slack.com/ (ask for an invite)● @TheSAGrid● @brusisceddu● fb.com/SAGrid


http://github.com/AAROC/

http://github.com/SAGridOps

http://aaroc.github.com/

https://ciari.slack.com/

continuous integration and delivery of research applications

Science

toobruce becker

cloudsbruce becker

sometimesbruce becker

problembruce becker

compuational infrastructure

jungle computing

different architectures

different setup