© 2014 Mohammad Fadli

Python for Earth

Mohammad Fadli

Days before python (back in college era)

• Using fortran for numerical programming

• Using matlab for matrix operational or data

plotting

• Using excel for data processing

• Using surfer or matlab for plotting

• Using arcmap or or for spatial data processing

• Using delphi for UI

• No database (all textfile matrix)

Create matrix

do j = 1, 3

do i = 1, 3

A(i,j) = real(i)/real(j)

blah.. Blah ..

read text file ..

write text file ..

enddo

enddo

Matlab

• Matrix Operation • a = [ 1 2 3 4 5; 2 3 4 5 6; 3 4 5 6 7; 4 5 6 7 8]

• a(2,5)

Courtesy of Mathworks Inc.

Plot stuff

Courtesy of Golden Software Inc.

Courtesy of Embarcadero Technologies, Inc.

Courtesy of ESRI, Inc.

Hydrodynamics model

Courtesy of TideTech, Inc.

Courtesy of Kementerian Kelautan dan Perikanan

Courtesy of Kementerian Kelautan dan Perikanan

Courtesy of Kementerian Kelautan dan Perikanan

Courtesy of Kementerian Kelautan dan Perikanan

Courtesy van oord nv

• Build model for

pollutant trajectory

Grid Analysis and Display System (GrADS) Courtesy of Weather & Climate Prediction Lab. ITB

Do something different!

• Install freeBSD and runs model on freeBSD

• Install mapserver on freeBSD

• Using grads for plotting

• Using fortran for data processing

• Using php for web (still)

• Parallel computing (not yet)

First time using python ArcGIS ArcPy

Courtesy of ESRI Inc.

Heboh Python (Python Fuss) :p

• A clear and readable syntax.

• Modules can be easily written in C, C++.

• It can be used on all major platforms (Windows,

Linux/Unix, Mac etc.)

• Easy to learn.

• And it is free.

• Python this, python that

Nowadays

• ANUGA for Hydrodynamics model

• EQRM for Earthquakes model

• Python for Climate Data Analysis Tools (CDAT)

• GDAL library for spatial data processing

• Django, Grok, Pylons, web2py (and other tons of

web library) for web

• Psycopg for postgre database connection

• PyMPI for Clusters computing and parallelization

Science with Python: NumPy, SciPy and Matplotlib Courtesy of Andrew Walker

Science with Python: NumPy, SciPy and Matplotlib Courtesy of Andrew Walker

SciPy: a collection of useful modules

Courtesy of Andrew Walker

• Python enables users to do more and better science.

• Python is a modern, interpreted, object-oriented, open-source

language used in all kinds of software engineering.

• Python is now a robust integration platform for all kinds of atmospheric

sciences work, from data analysis to distributed computing, and

graphical user interfaces to geographical information systems.

• Python has a concise but natural syntax for both arrays and nonarrays,

making programs exceedingly clear and easy to read; as the saying

goes, “Python is executable pseudocode.”

• Include visualization, numerical libraries, interconnection with compiled

and other languages, memory caching, Web services, mobile and

desktop graphical user interface programming, and others.

Another advantages

• Several packages exist that handle file formats used by

atmospheric and oceanic sciences users, including netCDF,

HDF4, HDF-EOS 2, and GRIB 1 and 2 (e.g., via the netCDF4-

Python, UV-CDAT, and PyNIO packages).

• Python can be used as a scripting language for

applications such as Unidata’s Integrated Data Viewer,

the NCAR Command Language (via PyNGL), GrADS (via

OpenGrADS), and Esri’s ArcGIS.

Disadvantages

• To be fair, Python has real disadvantages, including the fact that

pure Python code runs much slower than compiled code, there

are fewer scientific libraries compared to Fortran, and

documentation and support for new science users is relatively

sparse.

• There are tools to overcome the speed penalty, the collection of

scientific libraries is growing, and science support resources are

becoming more robust; nevertheless, these are real issues.

• For most Earth science applications, the strengths of Python

outweigh the weaknesses.

What we are doing with python now?

• Try to runs (and optimize) anuga on 128

core HPC

• Optimize and automate earthquake model

• Create our own spatial analysis framework

using GDAL lib

• Prove what John-Wei said

Flood Research 2013: Domain Pasar Minggu to Manggarai (Segment 1)

• Criteria:

– Major factor only from discharge and

local rainfall

– Homogenous Hydrologic characteristic 27

Manggarai Detail Domain

Katulampa

Courtesy of PT Reasuransi Maipark Indonesia 2013

Model Hidrodinamika 2D AnuGA

ANUGA adalah model hidrodinamika Free and Open Source (FOSS) yang dikembangkan oleh Australian National University dan Geoscience Australian; metode pemecahan persamaan menggunakan finite volume; dapat diunduh dari http://anuga.anu.edu.au/; Salah satu kelebihan ANUGA : dapat meng-input-kan hujan pada domain

Kenapa Anuga: 1. Public Domain 2. Open Source 3. Parallel 4. Finite Volume Shock Capturing

Capability Hydraulic Jump

Anuga for Jakarta Flood 2013 2013 flood simulation

Bukit Duri dan Kampung Melayu Kalibata

Courtesy of PT Reasuransi Maipark Indonesia 2014

Hasil Simulasi Banjir Jakarta

Simulasi Banjir Jakarta 2013 Hadi, dkk., 2013

Input Curah Hujan Spasial, Robert, 2013

Anuga dapat mengakomodasi simulasi banjir dengan input curah hujan spasial, simulasi paralel Dan input debit serta TMA

Courtesy of PT Reasuransi Maipark Indonesia 2014

Courtesy of LAPAN

Courtesy of LAPAN

Rendaman Max. vs Data UAV LAPAN

Courtesy of LAPAN, PT Reasuransi Maipark Indonesia

Courtesy of PT Reasuransi Maipark Indonesia

K.H. Abdullah Syafei Foto: Kantor Berita ANTARA (19/01/2014)

Courtesy of PT Reasuransi Maipark Indonesia

Depo, Bukit Duri (19/01/2014) Foto: Kantor Berita ANTARA

Bukit Duri

Bukit Duri (Depo & SMA 8)

Depo

SMA 8

Depo

Depo

Depo

SMA 8

SMA 8

(16/01/2013) Foto: Kantor Berita ANTARA

(19/01/2014) Foto diambil dari Heli PMI

Courtesy of PT Reasuransi Maipark Indonesia

K.H. Abdullah Syafei, (19/01/2014), Foto: ANTARA

Courtesy of PT Reasuransi Maipark Indonesia

Jembatan Layang Kalibata, 19/1/2014, Foto: ANTARA Courtesy of PT Reasuransi Maipark Indonesia

Jl. Jatinegara barat raya

Depo

Depo

Jatinegara Bukit Duri

Courtesy of PT Reasuransi Maipark Indonesia

Rawajati

Kampung Melayu

Diambil dari helikopter PMI, 16/1/2013

Thunder System

Summary Specification

Nodes 4 nodes

Cores 16 cores (4 cores/node)

Processors (per nodes)

AMD FX (tm) – 8350, 4 Ghz

Memory (per nodes)

8 GB DDR 3, 800 Mhz

Scenario: Spin up scenario for 16.67 hours Domain : Full Jakarta consist of Ciliwung River (Bogor-Pluit)

Even though faster

simulation could be

performed, we have to

notify that low

resolution mesh is used

in this case

Scenario: 15-18 January 2013 (3 days) Domain : Ciliwung River (Pasar Minggu-Manggarai)

“The computation

speed may increase as

the water volume inside

of the domain is

increasing”

Courtesy of WCPL-ITB

National Computational Infrastructure (NCI), Australia

Summary Specification

Nodes 3592 node

Cores 57,472 cores (16 cores/node)

Model Fujitsu Server Primergy Cx 250 S2

Processors 2 x intel Sandy E5-2600 (8-core, 2.6 GHz)

Memory 8 GB DDR 3, 1600 MHz 2395 nodes ->32 GB 1125 nodes -> 64 GB 72 nodes -> 128 GB TOTAL : 158 TB

Harddisk 500 GB 7200 rpm SATA

www.nci.org.au, www.fujitsu.com

Cluster Raijin, NCI

Domain 2 :Jakarta Approximately 10 rivers will be included in our scenario Approximately total mesh element

number: 3.300.940 triangles

Problem in computation time while using

large number of mesh elements could be

tackled by employing large core number

of machine used for simulation

NCI Building Australia National University

Contoh :

Reference

• http://www.johnny-lin.com/pyintro/

• http://jakevdp.github.io/blog/2013/03/23/matplotlib-and-the-future-of-

visualization-in-python/

• http://greenteapress.com/pythonhydro/pythonhydro.html

• http://resources.arcgis.com/en/communities/python/

• http://arcpy.wordpress.com/page/3/

• http://pyaos.johnny-lin.com/?p=1271

• http://www1.gly.bris.ac.uk/~walker/PythonEarthSci/

• http://conferences.ncl.ac.uk/sciprog/Presentations/python_processing_spyder_v1.0.2.

pdf

• http://numpy.sourceforge.net/numdoc/HTML/numdoc.htm

• http://arcpy.wordpress.com/

grads

• The Grid Analysis and Display System (GrADS) is an

interactive desktop tool that is used for easy access,

manipulation, and visualization of earth science

data.

• Support binary (stream or sequential), GRIB

(version 1 and 2), NetCDF, HDF (version 4 and 5),

and BUFR (for station data).

• Windows & Unix support