docker and radio astronomy - containing fragile scientific software by gijs molenaar (university of...

Docker & radio astronomy

1

Who Am I

Gijs Molenaar

Scientific Software Engineer in NL

PhD student in SA

MSc Artificial Intelligence

2

@gijzelaerr http://pythonic.nl

3

4

5

6

7

8

The problem

9

Scientist Q wants to crunchData X with

software Y onPlatform Z

10

Scientist Q

Smart person

Self educated on the field of software engineer

Gets the job done

Good at duct tape programming

11

Data X

Really big data

Datasets of peta, exa bytes

LOFAR - 50 GB/s

SKA - 10 times global Internet traffic

12

Software Y

13

Complex math heavy software

Written during PhD of scientist Q

Often bad coding practices

(Unit) tests?

Version control?

‘Runs on my computer’

Platform Z

14

Solution until now

15

Solution until now

16

Compile everything by hand

Still common practice (!)

17

18

Canonical launchpad

19

Papino

20

https://github.com/ska-sa/papino

$ docker run skasa/papino ipython notebook

21

Rodrigues - telescope simulator

22

23

24

25

26

27

science vs scalable reliable web apps

Much less users

Much more data

High availability less important

Security less important

Performance very important

Durability very important

28

Lessons learned

Don’t build virtual machine images

use small minimal containers

use exec to examine problems inside container

29

Problems to solve / future work

GPU acceleration

Adaptation

‘Mixing’ containers (nixos?)

Container size (South Africa)

Container aware programming language / library?

30

Thank You.

31

@gijzelaerr - http://pythonic.nl