scripting languages - fast development, extensible programs

UNIVERSITY OF SCIENCE AND TECHNOLOGY OF CHINA SCHOOL OF SOFTWARE ENGINEERING OF USTC

Scripting LanguagesFast development, extensible programs

Devert AlexandreSchool of Software Engineering of USTC

November 30, 2012 — Slide 1/60


Table of Contents

1 Introduction

2 Dynamic languagesA Python tourFunctions as objectsMonkey patching

3 Languages comparison

4 Tools makingContent generation

5 Glue code

Devert Alexandre (School of Software Engineering of USTC) — Scripting Languages — Slide 2/60


Introduction

You probably heard about those programming languages

• Javascript

• Perl

• Python

• Lua

• Ruby

• . . .



Scripting languages

They are scripting languages, sharing common features

• Highly dynamic

• Automatic memory management

• Can compile to a virtual machine

• Can be embedded

• Can be extended with modules written in C or C++

• Cross-platform



Scripting languages

Scripting languages tends to help you to code faster

• less code to do the same things with other languages

• powerful default types

• very dynamic languages

• large, complete standard library



Usages

They are successful for a number of applications

• tool-making languages

• “glue” languages to write “glue code”

• building extensible, open-ended software

• . . .



Table of Contents

1 Introduction




5 Glue code



A Python tour

Let’s have a look at a scripting language, Python. Whatfollows is true for most other scripting languages.



Friendly syntax

List and dictionaries are parts of the language syntax

a = [ 1 , 2 , 3 , 4 , 5 , ” s i x ” , 7 ]

b = { ’ name ’ : ’ app l e ’ , ’ c o l o r ’ : ’ r ed ’ , ’ c o s t ’ : 0 .25 }

c = [ [ 1 . 5 , 3 . 0 ] , [−2.0 , 5 . 0 ] , [−9.7 , 5 . 2 ] ]

d = {’ f r u i t s ’ : [ ’ app l e ’ , ’ o range ’ , ’ peach ’ ] ,’ l o c a t i o n ’ : ’ Suzhou ’

}



Friendly syntax

List manipulations are part of the languages

a = [ 1 , 2 , 3 , 4 , 5 , 7 , 8 , 9 , 10 ]

p r i n t a [ 2 : 7 ]>>> [ 3 , 4 , 5 , 7 , 8 ]

p r i n t a [ 2 : ]>>> [ 3 , 4 , 5 , 7 , 8 , 9 , 10 ]

p r i n t a [ : 5 ]>>> [ 1 , 2 , 3 , 4 , 5 ]

p r i n t a [ 2 : 5 ] + a [ 7 : ]>>> [ 3 , 4 , 5 , 9 , 10 ]

p r i n t [ 2 , 3 ] ∗ 5>>> [ 2 , 3 , 2 , 3 , 2 , 3 , 2 , 3 , 2 , 3 ]



Friendly syntax

Iterators are nearly invisible

a = [ 1 , 2 , 3 , 4 , 5 ]

f o r v a l u e i n a :p r i n t v a l u e

Iterating over the elements of a list



Friendly syntax

Iterators are nearly invisible

myFi l e = open ( ” i npu t . t x t ” )

i = 0f o r l i n e i n myF i l e :

i += 1p r i n t i , l i n e

Iterating over the lines of a text file



Friendly syntax

You affect multiple variable in one statement

a , b , c = 1 , 2 , 3p r i n t a , b , c

de f myFunction ( x ) :r e t u r n x , x ∗ x

a , b = myFunction (3 )p r i n t a , b



Dynamic variable type

Variable types is not fixed, it changes

a = 42a = ” app l e ”a = [ ”wang” , ” f e i ” , ” yue ” ]a = (1 , 2 , 3 , ” f o u r ” , 5 . 0 )



Everything is object

Numbers are objects

a = 2p r i n t a + 3p r i n t a . a d d (3 )




Lists are objects

a = [ 1 , 2 , 3 , 4 , 5 ]p r i n t a [ 3 ]p r i n t a . g e t i t em (3)




Functions are objects

de f s q r ( x ) :r e t u r n x ∗ x

p r i n t s q r (3 )

c l a s s sqrFunc ( o b j e c t ) :d e f c a l l ( s e l f , x ) :

r e t u r n x ∗ x

sq r = sqrFunc ( )p r i n t s q r (3 )



Standard library

An extensive standard library

• OS-independent file and directory access

• data persistence

• XML, JSON, CSV, . . . parsing

• most common Internet protocols

• OS-independent GUI

• . . .



Standard library

Example: defining a command-line interface

impor t a r g p a r s e

cmdLine = a r gpa r s e . ArgumentParser ( d e s c r i p t i o n = ’ Read t h i n g s and do s t u f f s ’ )cmdLine . add argument ( ’ i nputPath ’ ,

a c t i o n = ’ s t o r e ’ ,h e l p = ’ path to i npu t f i l e ’ ,t ype = s t r )

cmdLine . add argument ( ’−s ’ , ’−−s i z e ’ ,a c t i o n = ’ s t o r e ’ ,d e s t = ’ f o n t S i z e ’ ,d e f a u l t = 16 ,h e l p = ’ s i z e o f r ead b u f f e r ’ ,t ype = i n t )

a r g s = cmdLine . p a r s e a r g s ( )



Standard library

Example: printing the content of a ZIP archive

impor t z i p f i l eimpor t a r g p a r s e

cmdLine = a r gpa r s e . ArgumentParser ( d e s c r i p t i o n = ’ L i s t the con t en t o f a ZIP a r c h i v e ’ )cmdLine . add argument ( ’ i nputPath ’ ,

a c t i o n = ’ s t o r e ’ ,h e l p = ’ path to i npu t ZIP a r c h i v e ’ ,t ype = s t r )

a r g s = cmdLine . p a r s e a r g s ( )

t r y :a r c h i v e = z i p f i l e . Z i p F i l e ( a r g s . inputPath , ’ r ’ )f o r i n f o i n a r c h i v e . i n f o l i s t ( ) :

p r i n t i n f o . f i l e name , i n f o . da te t ime , i n f o . f i l e s i z e , i n f o . c omp r e s s s i z eexcep t z i p f i l e . B a dZ i p f i l e :

p r i n t a r g s . inputPath , ” i s not a ZIP f i l e ”excep t IOEr ro r as ( e r rno , s t r e r r o r ) :

p r i n t s t r e r r o r



Standard library

Example: reading/writing JSON data

impor t j s o n

a = j s o n . l oad ( open ( ’ example . j s o n ’ ) )p r i n t a

impor t j s o n

a = [’ foo ’ ,{

’ bar ’ : ( ’ baz ’ , None , 1 . 0 , 2)}

]

j s o n . dumps ( a )



Fibonacci numbers

Fibonacci numbers are defined as follow

Fn = Fn−1 + Fn−2,F0 = 0,F1 = 1



Fibonacci numbers

Let’s code a function to compute Fibonacci numbers

de f f i b ( n ) :i f n < 2 :

r e t u r n 1r e t u r n f i b ( n − 1) + f i b ( n − 2)

p r i n t f i b (36)

On my computer, this takes about 11 sec. to compute



Memoization

To make it faster, we should store intermediate results.It’s called memoization

f ib memo = { }

de f f a s t F i b ( n ) :i f n < 2 :

r e t u r n 1

i f not f ib memo . ha s key ( n ) :f ib memo [ n ] = f a s t F i b ( n − 1) + f a s t F i b ( n − 2)

r e t u r n f ib memo [ n ]

p r i n t f a s t F i b (36)

Same function, but takes 80 msec. to compute !



Memoization

But we might not be so happy of that solution

• mix the idea and the implementation

• have to do by hand this for any expensive recursivefunction



Functions are objects

But remember, functions are objects ! ⇒ we can build afunction object which

• contains a function f and a dictionary m

• if a not in m, m[a] = f (a)

• returns m[a]



Memoizing functions

Python implementation for this (works for all arguments)

c l a s s Memoize :d e f i n i t ( s e l f , f ) :

s e l f . f = fs e l f .m = { }

de f c a l l ( s e l f , ∗a r g s ) :i f not s e l f .m. ha s key ( a r g s ) :

s e l f .m[ a r g s ] = s e l f . f (∗ a r g s )r e t u r n s e l f .m[ a r g s ]



Memoizing functions

And now, we have a general and clean way to domemoization

de f f i b ( n ) :i f n < 2 :

r e t u r n 1r e t u r n f i b ( n − 1) + f i b ( n − 2)

f i b = Memoize ( f i b )p r i n t f i b (36)



Monkey patching

Methods of objects are functions too

c l a s s Animal :d e f makeNoise ( s e l f ) :

p r i n t ’ not d e f i n e d ’

d e f doGroaaaar ( ) :p r i n t ’ g r oaaaa r ! ’

t i g e r = Animal ( )t i g e r . makeNoise = doGroaaaar

t i g e r . makeNoise ( )

Dynamic replacement of methods



Monkey patchingMonkey-patching allow to write very flexible and shortcode

impor t random

c l a s s Animal :d e f makeNoise ( s e l f ) :

p r i n t ’ not d e f i n e d ’

c l a s s RandomNoiseMaker :d e f i n i t ( s e l f , n o i s e L i s t ) :

s e l f . n o i s e L i s t = n o i s e L i s t

d e f c a l l ( s e l f ) :p r i n t random . cho i c e ( s e l f . n o i s e L i s t )

t i g e r = Animal ( )t i g e r . makeNoise = RandomNoiseMaker ( [ ’ g r oaaa r ! ’ , ’ g r r r r r ! ’ , ’ graaaaouuu ! ’ ] )

t i g e r . makeNoise ( )

Monkey patching with a dynamically defined functionDevert Alexandre (School of Software Engineering of USTC) — Scripting Languages — Slide 30/60


Table of Contents

1 Introduction




5 Glue code



Languages comparison

Let’s compare Python and Java to do common tasks




Printing “Hello, world !”

p u b l i c c l a s s He l l oWor ld {p u b l i c s t a t i c vo i d main ( S t r i n g [ ] a r g s ) {

System . out . p r i n t l n ( ” He l l o , wor ld ! ” ) ;}

}

p r i n t ” He l l o , wor ld ! ”




Opening a file

impor t j a v a . i o .∗ ;. . .

Bu f f e r edReade r myF i l e =new Bu f f e r edReade r (

new F i l eR e ad e r ( a rgF i l ename ) ) ;

myF i l e = open ( a rgF i l ename )




Add an integer to a list, get an other from the list

p u b l i c Vector<I n t e g e r> a L i s t =new Vector<I n t e g e r >;

p u b l i c i n t aNumber = 5 ;p u b l i c i n t anotherNumber ;

a L i s t . addElement ( aNumber ) ;anotherNumber = a L i s t . getE lement ( 0 ) ;

a L i s t = [ ]aNumber = 5

a L i s t . append ( aNumber )anotherNumber = a L i s t [ 0 ]




Print a list of numbers to a file

impor t j a v a . i o .∗ ;

p u b l i c c l a s s IOTest {p u b l i c s t a t i c vo i d main ( S t r i n g [ ] a r g s ) {

t r y {F i l e f = new F i l e ( ” out . t x t ” ) ;P r i n tW r i t e r ps =

new P r i n tW r i t e r ( new OutputStreamWriter( new F i l eOutputSt r eam ( f ) ) ) ;

f o r ( i n t i = 0 ; i < 1000000; i++)ps . p r i n t ( S t r i n g . va lueOf ( i ) ) ;

ps . c l o s e ( ) ;}ca tch ( IOExcept i on i o e ) {

i o e . p r i n t S t a c kT r a c e ( ) ;}

}}

f = open ( ’ out . t x t ’ , ’wb ’ )

f o r i i n x range (1000000) :f . w r i t e ( s t r ( i ) )

f . c l o s e ( )




Defining a simple class

p u b l i c c l a s s Employee {p r i v a t e S t r i n g myEmployeeName ;p r i v a t e i n t myTaxDeductions = 1 ;p r i v a t e S t r i n g myMar i t a lS ta tu s = ” s i n g l e ” ;

p u b l i c Employee ( S t r i n g EmployeName ) {t h i s ( employeeName , 1 ) ;

}

p u b l i c Employee ( S t r i n g EmployeName , i n t t axDeduc t i on s ) {t h i s ( employeeName , taxDeduct ions , ” s i n g l e ” ) ;

}

p u b l i c Employee ( S t r i n g EmployeName , i n t taxDeduct ions , S t r i n g ma r i t a l S t a t u s ) {t h i s . employeeName = employeeName ;t h i s . t a xDeduc t i on s = taxDeduc t i on s ;t h i s . m a r i t a l S t a t u s = ma r i t a l S t a t u s ;

}. . .




Defining a simple class

c l a s s Employee ( ) :d e f i n i t ( s e l f , employeeName , t axDeduc t i on s =1, ma r i t a l S t a t u s=” s i n g l e ” ) :

s e l f . employeeName = employeeNames e l f . t a xDeduc t i on s = taxDeduc t i on ss e l f . m a r i t a l S t a t u s = ma r i t a l S t a t u s

. . .



Table of Contents

1 Introduction




5 Glue code



Tools

It’s quite common to have to do code which will not bein the final product

• Build

• Test

• Content generation

• Source-code generation

Content generation often need project-specific tools



Texture atlas

In games, graphics are usually made from lot of pictures



Texture atlasSave speed & memory ⇒ pack all graphics in one texture

On mobile devices, it is essential



Texture atlas

A texture packing many graphics elements is called atexture atlas

Building a texture atlas is very device and projectdependent



Texture atlas

Automated texture atlas generation

• a big time saver

• artist can test many ideas

• does not need to be fast

• integration to build process



Texture atlas

Texture atlas generator should

1 read a list of pictures

2 read a list of rectangle coordinates

3 pack rectangles

4 generate rectangle coordinates

5 generate a picture



Texture atlas

Texture atlas generator as a script

• read/write pictures with one line of code

• read/write XML, JSON, plain text with very fewcode

• rectangle packing algorithm

Personal experience ⇒ 1 hour in Python, 12 hours in C



Tool making

Scripting languages are wonderful to build such tools



Table of Contents

1 Introduction




5 Glue code



Glue code

Software are often built by gluing together variouslibraries

• project specific code

• some specialized processing libraries

• a communication library

• a data storage library (database, XML, . . . )

• a GUI library



Glue code

Many important libraries have Python, Ruby, Lua or Perlbindings

• Sqlite, MySql, Berkeley DB ⇒ database

• wxWindow, QT, GTK ⇒ graphic users interfaces

• Numpy, Scipy ⇒ math and linear algebra

• Simple Direct Media Layer ⇒ basic graphic, soundand inputs handling

• Ogre3D ⇒ 3d graphic engine

• Cairo ⇒ 2d vector graphics

• . . .



Glue code

A way to build cross-platform software quickly ⇒ use ascripting language to glue libraries

• Glue code usually goes very well with very dynamiclanguages

• Fast prototyping

• Good performance (most bindings are coded in C orC++)



Success stories

Some successful applications using this approach

• Python ⇒ DropBox, Civilization IV, Mercurial,Django, Blender, BitTorrent

• Ruby ⇒ Ruby On Rails, Google Sketchup

• Erlang ⇒ Wings3D, CouchDB, Goldman Sachsrobot traders



Project specific code

You might want to create your own extensions for ascripting language

• using legacy code

• for the 10% code which takes 90% of the time



Project specific code

All scripting languages provides a way to create extensionin C/C++



Writing Python extensionsA “Say hello” Python extension

#i n c l u d e <Python . h>

s t a t i c PyObject∗s a y h e l l o ( PyObject∗ s e l f , PyObject∗ a r g s ) {

con s t cha r∗ name ;

i f ( ! PyArg ParseTup le ( args , ” s ” , &name ) )r e t u r n NULL ;

p r i n t f ( ” He l l o %s !\n” , name ) ;

Py RETURN NONE ;}

s t a t i c PyMethodDef He l loMethods [ ] = {{ ” s a y h e l l o ” , s a y h e l l o , METH VARARGS, ” Gree t somebody . ” } ,{ NULL , NULL , 0 , NULL }} ;

PyMODINIT FUNC

i n i t h e l l o ( vo i d ) {( vo i d ) Py In i tModu l e ( ” h e l l o ” , He l loMethods ) ;

}



Writing Python extensions

The “Say hello” Python extension setup

from d i s t u t i l s . c o r e impor t setup , Ex t en s i on

module1 = Ex t en s i on ( ’ h e l l o ’ , s o u r c e s = [ ’ h e l l omodu l e . c ’ ] )

s e tup (name = ’PackageName ’ ,v e r s i o n = ’ 1 .0 ’ ,d e s c r i p t i o n = ’ This i s a demo package ’ ,e x t modu l e s = [ module1 ] )




Using the ‘Say hello” extension

impor t h e l l o

h e l l o . s a y h e l l o ( ”World” )



Writing Python extensionsA Fibonacci numbers Python extension

#i n c l u d e <Python . h>

i n tf i b ( i n t n ) {i f ( n < 2)

r e t u r n n ;e l s e

r e t u r n f i b (n−1) + f i b (n−2);}

s t a t i c PyObject∗f i b ( PyObject∗ s e l f , PyObject∗ a r g s ) {

con s t cha r ∗command ;i n t n ;

i f ( ! PyArg ParseTup le ( args , ” i ” , &n ) )r e t u r n NULL ;

r e t u r n Py Bu i l dVa lue ( ” i ” , f i b ( n ) ) ;}

s t a t i c PyMethodDef FibMethods [ ] = {{” f i b ” , f i b , METH VARARGS, ” C a l c u l a t e the F i b ona c c i numbers . ”} ,{NULL , NULL , 0 , NULL}} ;

PyMODINIT FUNCi n i t f i b ( vo i d ) {

( vo i d ) Py In i tModu l e ( ” f i b ” , FibMethods ) ;} Devert Alexandre (School of Software Engineering of USTC) — Scripting Languages — Slide 58/60



The Fibonacci Python extension setup

from d i s t u t i l s . c o r e impor t setup , Ex t en s i on

module1 = Ex t en s i on ( ’ f i b ’ , s o u r c e s = [ ’ f i bmodu l e . c ’ ] )

s e tup (name = ’PackageName ’ ,v e r s i o n = ’ 1 .0 ’ ,d e s c r i p t i o n = ’ This i s a demo package ’ ,e x t modu l e s = [ module1 ] )




Using the Fibonacci extension

impor t f i b

p r i n t f i b . f i b (10)


scripting languages - fast development, extensible programs

Documents