the language, libraries and culture of python in meteorologyscipy in py-art the language, libraries...
TRANSCRIPT
-
The Language, Libraries and Culture of Python in Meteorology
JonathanHelmus
ArgonneNa*onalLaboratory,USA
This presentation has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory (“Argonne”). Argonne, a U.S. Department of Energy Office of Science laboratory, is operated under Contract No. DE-AC02-06CH11357. This research was supported by the Office of Biological and Environmental Research of the U.S. Department of Energy as part of the Atmospheric Radiation Measurement Climate Research Facility.
-
Technology Powers Scientific Discovery
§ Scien*ficadvancementsaredrivenbystate-of-the-arttechnologies.
§ Newtechnologiesleadthewaytoscien*ficbreakthroughsandamorecompleteunderstandingoftheworldaroundus.
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
2
-
Computers Prime Technology for Science
§ Computersarethemostsignificanttechnologyavailabletoscien*ststoday.
§ Theabilitytoperformfastcomputa*onshasvastlychangedmeteorologyandwillcon*nuetochangethefieldinthefuture.
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
3
Source:R.G.Strauchelal,JTech,1984,1,37ARMCSAPRRadarhRp://www.arm.gov/instruments/csapr
-
Programming: Controlling Computers for Science
§ Thedailyopera*onofscienceinvolvescomputers.
§ Computersrequirespecificdirec*onsintheformofsoUwaretorealizetheirpoten*altopowerscien*ficresearchandopera*ons.
§ TheabilitytocreatesoUwarecustomizedtoaspecificscien*fictaskis
oUenneeded.
§ Wri*ngsoUwarecanbea*meconsumingandburdensometask.– Scien*stses*matethat35%oftheirresearch*meisspentdevelopingsoUware[1].– Mostscien*stslearntodevelopsoUwareinformallyfromtheirpeers[2].– Thechoiceofprogramminglanguageaffectsthe*merequiredfordevelopment.
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
4
[1]P.Prabhu,etal,ASurveyofthePrac*ceofComputa*onalScience,2011,10.1145/2063348.2063374[2]J.E.Hannay,etal,HowDoScien*stsDevelopandUseScien*ficSoUware,2009,10.1109/SECSE.2009.5069155.
-
Python is Ideal for Writing Scientific Software
§ Pythonistheidealprogramminglanguageforwri*ngsoUwaretomeetthecomputa*onalchallengesinmeteorologyandatmosphericscience.
§ DevelopingsoUwareinPythonissimpleandfastdueto:
– Thedesignandphilosophyofthelanguage.
– Theavailabilityofanumberofhigh-quality,efficientscien*ficPythonlibraries.
– Thewelcoming,vibrant,andgrowingPythoncommunity.
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
5
-
Python: the Language
Python,theprogramminglanguageis:
§ Ahigh-levellanguage.– Allowsonetofocusonthethetaskconceptsratherthanthedetailsofthecomputer.– Makesdevelopmentsimpleandquick.
§ Interpretedandinterac*ve.– Nocompila*onstep,speedsupdevelopmentcycle.– Interac*veshell(REPL)fortes*nganddebugging.
§ Generalpurpose.– CanbeusedtowritesoUwareacrossawiderangeofapplica*ons,notdomainspecific.– Largeruseranddeveloperbase.
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA6
-
Python: the Language
Pythonisanideallanguageforscien*ficprogrammingbecauseitis:§ Conciseandreadable
§ OpenSourceandFree
– NocompanycontrolsthesoUware,nolicensingcosts.
§ Hasalargenumberofthird-partylibraries.– PyPIcontainsmorethan72,000packages.– Manyscience-focusedlibrariesexistcoveringavarietyoffields.
§ Easytointerfacewithexis*ngC,C++andFortrancode.– ToolslikeCython,cffi,ctypes,andF2PYmakeinterfacingwithexis*ngcodesimple.
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
7
colors=['red','blue','green']forcolorincolors:print(color)
@colors=("red","blue","green");foreach(@colors){print"$_\n”;}
Python Perl
-
Python: the Challenges
Pythondoesfacesomechallenges:
§ Execu*onspeedisslowcomparedtosomeprogramminglanguages.– Manytasksarenotlimitedbyexecu*onspeed.– Op*onsexisttospeedupPythoncode.
§ Thelanguageisnotthebestatsometypesofparallelcomputa*ons.– TheGlobalInterpreterLock(GIL)poseslimitsofmul*-threadedperformance.– Handlingconcurrencyisnotasrobustassomelanguages(Go,Erlang,Clojure).– Supportforcomputa*ononGPUsandotherhighlyparallelarchitecturesislimitedand
requiresaddi*onallibraries(Numba,PyCUDA)§ Thelanguageiscurrentlyinflux.
– Python3isbackwardsincompa*blewithPython2.– SomelibrariesdonotsupportPython3.
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA8
-
Scientific Python Libraries: Introduction
TheScien*ficPythonEcosystemorSciPystackisacollec*onofopensourcesoUwareforscien*ficcompu*nginPython.Someofthecorepackagesare:§ NumPy:Providespowerful,efficientmul*-dimensionalarraysinPython§ SciPy:Fundamentalnumericalalgorithmsforcommontasksinscience.§ matplotlib:comprehensivepublica*on-quality2Dplolng
§ Jupyter/IPython:Rich,interac*veinterfacesforprocessingdataandtes*ngideas.§ pandas:Highperformance,easytousedatastructures.§ SymPy:Symbolicmathema*csandcomputeralgebra
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
9
-
The Python ARM Radar Toolkit: Py-ART
§ Py-ARTisamoduleforvisualizing,correc*ngandanalyzingweatherradardatausingpackagesfromthescien*ficPythonstack.
§ DevelopmentbegantoaddresstheneedsoftheARMprogramwiththeacquisi*onofmulPplescanningcloudandprecipitaPonradars.
§ Theprojecthassincebeenexpandedtoworkwithavarietyofweatherradars,includingNEXRAD,andhasawideuserbaseincludingradarresearchers,weatherenthusiastsandclimatemodelers.
§ AvailableonGitHubasopensourcesoQware,arm-doe.github.io/pyart/.§ Condapackagesareavailableatanaconda.org/jjhelmus
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
10
-
Scientific Python Libraries: NumPy
§ NumPyisthefundamentalpackageforscien*ficcomputa*oninPython.
§ NumPyprovides:– Powerful,efficient(fast)mul*-dimensionalarrayobject,thendarrayclass.– RobustmethodsformanipulaPngthesearrays.– Maskedandrecordarrayobjects.– Rou*nesforlinearalgebra,Fouriertransform,andrandomnumbers.– Comprehensive,well-wriRendocumenta*on,hRp://docs.scipy.org/doc/.
§ Otherscien*ficPythonpackagesbuildonNumPytoaddaddi*onalfeaturesandabili*es.
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
11
-
NumPy’s ndarray details
§ NumPy’scorefunc*onalityisprovidedbyitsndarrayobject.§ Andarrayisahomogeneous,stridedviewonaconPguousblockof
memory.
§ Althoughsimple,thendarrayisapowerfulconstructastheloca*onofunderlyingmemorycanbepassedtootherlanguages(C,C++,CythonandFortran)withouttheneedtocopydata.
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA12
4bytes 4bytes 4bytes
16bytes
16bytes
(3,4)ndarrayoffloat32strides(16,4)
-
ndarray views
§ SlicingaNumPyndarrayalmostalwayscreatesa“view”ofthedata.
§ Nocopyingofdataisneededwhenaccessingormodifyingviews.
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
13
4bytes 4bytes 4bytes
16bytes
16bytes
4bytes 4bytes 4bytes
8bytes
16bytes
a=np.zeros((3,4),np.float32)(3,4)ndarray
a[0,:](4,)ndarray
a[:2,::2](2,2)ndarray
-
NumPy in Py-ART
NumPyisusedextensivelythroughoutPy-ARTasndarraysaretheprimaryobjectsusedtostoreandmanipulatenumericaldata.
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
14
elifdata_type_name=='PHIDP2':out[:]=360.*(data.view('uint16')-1.)/65534.mask[data.view('uint16')==0]=Trueelifdata_type_name=='HCLASS2':out[:]=data.view('uint16')elifdata_type_name=='XHDR':#Herewereturnanarraywiththetimesinmilliseconds.returndata[...,:2].copy().view('i4')#onebytedatatypeselifdata_type_name[-1]!='2':#makeaviewoflefthalfofthedataasuint8,#thisistheactualraydatacollected,therighthalfisblank.nrays,nbin=data.shapendata=data.view('(2,)uint8').reshape(nrays,-1)[:,:nbin]
-
NumPy in Py-ART
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
15
#decoderunlengthencodingrle_size=radial_header['nbytes']*2rle=np.fromstring(buf2[pos:pos+rle_size],dtype='>u1')colors=np.bitwise_and(rle,0b00001111)runs=np.bitwise_and(rle,0b11110000)//16radial[:]=np.repeat(colors,runs)
-
Scientific Python Libraries: SciPy
SciPyisacollec*onofmathema*calalgorithmsandfunc*onswhichbuilduponNumPytoprovideefficientsolu*onstocommonnumericaltasks.SciPyisdividedintosubpackageswhichcoveranumberofscien*ficdomains:§ Imageprocessing§ Signalprocessing§ Interpola*on§ Spa*aldatastructuresandalgorithms§ Clustering§ Numericalintegra*on§ Differen*alequa*ons§ Sta*s*cs§ Sparsematrices
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
16
-
SciPy in Py-ART
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
17
SciPyfunc*onsforimageprocessing,numericalintegra*on,interpola*on,spa*alanalysisandsparsematrixstorageareallusedinPy-ART.
importscipy.ndimage…def_find_regions(vel,gfilter,limits):"""Findregionsofsimilarvelocity."""mask=~gfilterlabel=np.zeros(vel.shape,dtype=np.int32)nfeatures=0forlmin,lmaxinzip(limits[:-1],limits[1:]):#findconnectedregionswithinthelimitsinp=(lmin
-
Scientific Python Libraries: matplotlib
matplotlibisaplolnglibrarywhichworkswithNumPy.
§ Comprehensive2D,publica*onqualityplots.– Mul*pleplottypes:line,scaRer,image,contours,pseudocolor,…– Manyoutputformats:png,jpg,svg,ps,pdf,…
§ Alimitedsetof3Dplots.– Line,scaRer,wireframe,tri-surface,contour,polygon,…
§ Plotscanbeexaminedinterac*velyorembeddedinapplica*ons.
– ExploredatainaGUI– ARTView:GUIviewerbuiltontopofthePy-ARTwhichembedsmatplotlibplots
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
18
-
matplotlib in Py-ART
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
19
importmatplotlib.pyplotaspltimportpyartradar=pyart.io.read('sgpxsaprrhicmacI5.c0.20110524.015604_NC4.nc')fig=plt.figure(figsize=(12,3))display=pyart.graph.RadarDisplay(radar)display.plot('reflectivity_horizontal',vmin=-20,vmax=40,cmap='pyart_NWSRef',title='XSAPR',colorbar_label='Refl.(dBZ)')display.set_limits(ylim=(0,15),xlim=(-42,42))plt.tight_layout()plt.savefig('figure.png')
-
Scientific Python Libraries: Jupyter/IPython
ProjectJupyter(previouslyIPython)isasetofrich,interac*veinterfacesandtoolsforprocessingdataandtes*ngideasinPython.§ UserInterfaces
– JuypterConsole:Terminalbasedinterac*vePythonenvironment– JuypterNotebook:Webbasedplauormforauthoringrichdocuments– Bothhaveexcellentintegra*onwithmatplotlib
§ Kernels– IPython:interac*vecompu*nginPython– ipyparallel:Lightweightparallelcompu*ngwithnotebookintegra*on– IJulia,IRKernel,IRuby,IPerl,…
§ Manyotherinteres*ngtools:nbviewer,nbconvert,nbgrader,jupyterhub…
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
20
-
Jupyter/IPython in Py-ART
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
21
-
Scientific Python Libraries: Cython
Cython§ PythontoCcodetranslator.§ GeneratesaPythonextensionmodule.§ CanbeusedtospeedupPythoncodeby
addingsta*ctypeinforma*on.§ AlsocanbeusedtointeractwithC/C++
func*onandclassesinexternallibraries.
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
22
Cfile(.c)
CCompiler(gcc)
Externallibraries(*.so,.a)
Cythonfiles(.pyx,.pxd)
Cython
CompiledPythonmodule(.so)
-
Cython in Py-ART: wrapping libraries
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
23
cdefexternfrom"rsl.h":ctypedefstructRadar:Radar_headerhVolume**vctypedefstructRadar_header:intmonth,day,yearinthour,minutefloatsec...ctypedefstructVolume:Volume_headerhSweep**sweep...
...
Radar*RSL_anyformat_to_radar(char*infile)...
voidRSL_free_volume(Volume*v)voidRSL_free_radar(Radar*r)
cimport_rsl_hcdefclassRslFile:
cdef_rsl_h.Radar*_Radarcdef_rsl_h.Volume*_Volumedef__cinit__(self,filename):self._Radar=_rsl_h.RSL_anyformat_to_radar(filename)ifself._RadarisNULL:raiseIOError('filecannotberead.')def__dealloc__(self):_rsl_h.RSL_free_radar(self._Radar)defget_volume(self,intvolume_number):rslvolume=_RslVolume()rslvolume.load(self._Radar.v[volume_number])returnrslvolume...propertymonth:def__get__(self):returnself._Radar.h.monthdef__set__(self,intmonth):self._Radar.h.month=month
_rsl_h.pxd _rsl_interface.pyx
-
Cython in Py-ART: speeding up Python code
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
24
107secondsvs.0.234seconds,x450performanceimprovement.
@cython.boundscheck(False)@cython.wraparound(False)def_fast_edge_finder(int[:,::1]labels,float[:,::1]data,intrays_wrap_around,intmax_gap_x,intmax_gap_y,inttotal_nodes):"""Returnthegateindicesandvelocitiesofalledgesbetweenregions."""cdefintx_index,y_index,right,bottom,y_check,x_checkcdefintlabel,neighborcdeffloatvel,nvelcollector=_EdgeCollector(total_nodes)right=labels.shape[0]-1bottom=labels.shape[1]-1forx_indexinrange(labels.shape[0]):fory_indexinrange(labels.shape[1]):label=labels[x_index,y_index]iflabel==0:continuevel=data[x_index,y_index]#leftx_check=x_index-1ifx_check==-1andrays_wrap_around:x_check=right#wraparoundifx_check!=-1:neighbor=labels[x_check,y_index]nvel=data[x_check,y_index]...#addtheedgetothecollection(ifvalid)collector.add_edge(label,neighbor,vel,nvel)
-
More Scientific Python Libraries…
§ pandas:datastructuresandanalysis§ xray:labeledarrayanddatasets§ Iris:meteorology/climatedatamodel
§ basemap:plot2Ddataonmaps§ pyproj:cartographictransforma*ons§ Cartopy:cartographictoolsforPython
§ netCDF4-python:ReadandwriteNetCDFfiles.§ h5py:ReadandwriteHDF5files
§ scikit-learn:machinelearning§ scikit-image:imageprocessing§ Andmany,manymore…
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
25
-
Python Community: Online
ThePythoncommunityprovidesawelcoming,vibrantandhelpfulculture.Muchofthecommunityinterac*onsoccuronline:§ Websites
– Python.org:Documenta*on,tutorial,PyPIandawiki.– Otherwebsites,scipy.org,pyaos.johnny-lin.com
§ Socialmedia– Facebook,TwiRer,Google+,IRC,YouTube– Blogs:planetpython.organdplanet.scipy.org.– Podcasts:TalkPythontomeandpodcast.__init__
§ Mailinglists– NearlyalltheSciPystackpackageshavetheirownmailinglists– PyAOSmailinglist(pyaos.johnny-lin.com)
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
26
-
Python Community: in Person
ThePythoncommunityalsomeetsinperson.§ Conferences
– SciPy– PyCon– Localandspecializedconferences(PyData,AMS)– ConferencetalksoUenavailableatpyvideo.org
§ Localusergroups– wiki.python.org/moin/LocalUserGroups
§ Meetupsandhackathons.– python.meetup.com
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA27
-
Python Community: for Developers
§ Mailinglists– Manyprojectshavea–devmailinglist
§ Socialcodingsites– GitHub– Bitbucket
§ NumFocus
§ Scien*ficPythonfocusedcompanies– Con*nuumAnaly*cs– Enthought
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
28
-
My Own Path Through the Python Community
§ Undergraduate-ChemistryatMichiganTechnologicalUniversity
§ Ph.D.atTheOhioStateUniversity-ChemicalPhysics.– LearnedtoprograminPython– Wrotenmrglue,alibraryforworkingwithNMRdatainPython
§ Post-docatUConnHeathCenter– Con*nuedtoprograminPython–signalprocessingforNMR– ARendedfirstSciPyconference
§ AdvancedAlgorithmsEngineeratArgonneNa*onalLaboratory– DevelopmentleadofthewidelyusedopensourcePy-ARTproject.– Contribu*ngtootherlibrariesintheSciPystack.– ARendandat*mespresentatSciPy,PyCon,ChiPy,…
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA 29
-
Careers in Meteorology with Python Programming
Doyouenjoyprogramming?Solvingproblems?Python?§ HighPerformanceCompu*ng(HPC)§ Informa*onProcessingTechnologies§ Instrumenta*onandprocessing§ DataAssimila*on§ NumericalWeatherPredic*on§ Climateandatmosphericmodeling
Regardlessofthecareerpathyouchoose,learningPythonwillenhanceyourpoten*al.
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
30
-
Ques*ons?
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA 31
-
Resources for Learning Python
§ Websites– Python.orgtutorial– ScipyLectureNotes,www.scipy-lectures.org
§ Books– “ThinkPython:HowtoThinkLikeaComputerScien*st”byAllenB.Downey– “Effec*veComputa*oninPhysics:FieldGuidetoResearchwithPython”byAnthony
Scopatz,KathrynD.Huff
§ Inperson– Tutorialsatconference(PyCon,SciPy,AMS)– SoUwareCarpentry,soUware-carpentry.org
§ Videos– ManyPythonconferencesmakestheirtutorialsavailableonYouTubeorpyvideo.org
TheLanguage,LibrariesandCultureofPythoninMeteorology,J.Helmus,AMS2016,NewOrleans,LA
32