Groovy On The Trading Desk

Jonathan FelchQuantitative Strategies

Volant Trading LLC

Overview● Motivations for Polyglot Software with Groovy● Introducing Groovy: Scripting on the VM

● Skills To Take With You: Learning Groovy● Groovy DSLs: Creating the syntax you need● Unlearning Java, C++, C#● Groovy Performance and Concurrency● Relearning Java: The Future of the VM ● Questions

Motivations

● High Speed OO Tools + Dynamic / Functional Tools

● Re-use existing assets

● Lazy Evaluation

● Externalize Logic and Business Rules

● Dynamic Evaluation

Why Groovy ?● Meet Requirements

– Lazy & Dynamic Evaluation, – Concise Synatx, Rapid Development, Iterative Prototyping– Supported But Did Not Require Functional Style, Support for

Closures / Lamda Expressions– Tool Support, Attractive Adoption Pattern– Human Readable / Machine Executable Data Interchange– Natural Data Integration Platform

● Strong and Focused Community● Familiar Syntax / Ease of Adoption● Expensible Syntax : DSLs

Scripting on the VM

Evolution of the VM: Towards Dynamic Programming

– Features Popular in Most Used 'Frameworks' Exploited Specific Benefits and Uses of Dynamic Programming for a general-purpose or 'cross-cutting' concern

● Hibernate, Spring Framework, Jboss AOP, AspectJ

– Introduction of Dynamic Programming for Testing,Configuration and Business Rules

● BeanShell, Jython, JRuby

– Next Generation Scripting: Groovy, Scala, Clojure● Specific to VM, not a straight port of another language● Emphasis on Functional Programming● Tight Integration of FP / DP with a high performance OO Language

Introducing Groovy● The Blue Pill: Syntactically compatible with Java

– Easy Adoption for Java Programmers– Virtually seamless / virtually flawless integration with Java– Natural Support for Multiple Styles– Compatibility allows for remarkable performance tuning

● The Red Pill: A Language On Its Own– Type-system: Dynamically / Optionally Strong Typing– Closures: Lambda Expressions / Functions as Objects– GDK: Object and Collection Enhancements– Dynam Eval: Code is Data, Data is Code– Dynam Syntax: Write your own syntax– Enhancements: New operators, data literals, new basic types

Job Trends / Adoption Hints for new JVM Languages

Groovy Use Cases: ● 'Super Glue': I/O and Integration

● 'Code as Data': Configuration, Process Scripting, Business Rules

● 'Live Analysis': Ad-Hoc Queries & Expressions

● 'Language Building': New Syntax and DSLs

● 'Half-Baked Ideas'

● 'House Elf': Cleanup and Maintenance

Learning Groovy

Im proving o n O b je c tdef abstract class OptionPxModel {

def vol, carry, strike, expiry, pc def abstract px (def spot)

} OptionPxModel model = new BlackScholesModel(strike : 100, pc : 'C' ) model.vol = 0.15 model.expiry = Date.parse('MM/dd/yyyy', '11/18/2010') println “The Strike 100 Nov 2010 Call is worth ${model.px(102.0)}”

C lo s ure s : def r = new Random() def normRand = { r.nextGaussian() } def stockPrice = { px, vol, rate, time → px * exp((rate – vol**2) * time + normRand() * vol * time) }

Learning Groovy: CollectionsL is tsdef list = [5, 6, 7, 8]assert list.size == list.size() assert list[2] == list.getAt(2) == list.get(2) == 7 assert [1,2,3,4,5][-1] == 5 && [1,2,3,4,5][-2] == 4assert list.find { it > 6 } == 7 && list.findAll { it > 6 } == [7, 8]list.each { println “$it ${it * 2}” }['a', 'b', 'c'].eachWithIndex{ it, i -> println "$i: $it" }

Multiple A s s ig nm e ntdef (int a, int b) = [1, 2](a, b) = [b, a]R a ng e sassert ('a'..'d') == ['a','b','c','d']assert [ 1, *3..5, 7, *9..<12 ] == [1,3,4,5,7,9,10,11]def oneWeekForward = (new Date() .. (new Date() + 7))

Ma ps[ 'id': 'FX-17', name: 'Turnip', 99: 123, (-97): 987, "tail's": true ].each { key, value → println “$key => $value” }

Groovy One (or 2, 3, 4) Liners AppsExecuting A Process: ”processMyFile ./output.txt $Destination”.execute()

Adding A She-Bang To All Groovy Files: groovy -i .bak -pe "if (count == 1) println '#!/usr/bin/groovy'" *.groovy

Print the 1st column and 2nd-to-last column from filegroovy -a -pe "split[0] split[-2]" some.filename // or set of files

Create and Shuffle a Deck of Cardsdef deck = [ [ *2..10, *('J Q K A'.split()) ], 'C D H S'.split() ]

.combinations().collect { it.join('') }; Collections.shuffle(deck)

Generate 21-Character OSI Symbol for Option def osi = { symbol, year, month, day, cp, strike -> sprintf('%1$s%2$2d%3$2d%4$2d%5$s%6$08.3f', [ symbol.padLeft(6), year as Integer, month as Integer, day as Integer, cp as String, strike as Double ] as Object[] )}

Learning Groovy: Operators'A s O pe ra to r'

"3.14159" as Double def readable = { it.put("12 34".reverse()); 5 } as ReadableAs O pe ra to r O n C lo s ure s + Ma ps Ob je c ts ? →

def map; map = [ i: 10, hasNext: { map.i > 0 }, next: { map.i-- } ] as Iterator

while ( map.hasNext() ) println map.next()

E lvis , S a fe Re fe re nc e , S pre a d & S pa c e s hip

def lang = data?.favoriteLanguage ?: 'Groovy' assert ['cat', 'elephant']*.size() == [3, 8] assert (3 <=> 4) == -1 && (3 <=> 3) == 0 && (5 <=> 3) == 1

Re g e x

def list = ["foo", "bar", "moo"] def pattern = ~/foo/

assert "foo" =~ list.find { it =~ pattern }assert ["foo", "moo"] == list.findAll { it ==~ /.*oo/ }

One Slide Apps: RSS Reader

def url ='http://www.groovyblogs.org/feed/rss'def items = new XmlParser().parse(url).channel.itemdef cols = 'pubDate title description'.tokenize()groovy.swing.SwingBuilder.build { frame(id:'f', title: 'Groovy RSS', visible :true) {

scrollPane {table {

tableModel(list: items) {cols.each { col →

closureColumn header: col,read: { it[col].text() }

} } } } }f.pack()

}.show()

Credit: Dierk König, GR8 Conference 2009, Copenhagen

One Slide Apps: Monte Carlo

def px = 100, rate = 0.05, vol = 0.15, time = 1.0def strikes = [ 8..12 ].collect { it * 10 }def w = RandomNumbers.getNormDist(1000,1000)def S = px * exp((rate-vol ** 2)* time + sqrt(time)* vol * w)strikes.each { K → def optionValue = max(0, S – K) def df = exp(-rate * time) println “${K} : ${df * optionValue as Number}”})

One Slide Apps: RESTful Web Serverimport com.sun.net.httpserver.*;

def isPrime = { Long N -> if (N < 2) return false; for (long i = 2; i*i < N; i++) if (N % i == 0) return false return true }

def pws = { HttpExchange exch -> exch.sendResponseHeaders(200,0); def pe = exch.requestURI.path.split('/') if (pe.size() == 3 && pe[1] == 'prime') { exch.responseBody.write("${isPrime(pe[2] as Long)}".bytes) } else exch.responseBody.write("${exch.requestURI.path}".bytes) exch.responseBody.close();} as HttpHandlerdef run = { handler, minutes -> def server = HttpServer.create(new InetSocketAddress(8888),0) server.createContext('/', handler) server.start()}run(pws,1)

Unlearning Java: Taking the Red Pill● Meta-Programming

● Design Patterns

● Compile Time Meta-Programming

● Power of Lazy Evaluation

Dynamic Language: Evaluation or Syntax

“In Lisp, you don’t just write your program down toward the language, you also build the language up toward your program” - Paul Graham

“When [Smalltalk] is used to describe an application system, the developer extends Smalltalk, creating a domain-specific language by adding a new vocabulary of language elements ...” - Adele Goldberg DSLs

Meta-Programming● Understanding The Type System

– Groovy has an optionally dynamic strongly typed type system– What does that mean ?

● Dynamic Method Invocation and GroovyObject– invokeMethod, get/setProperty

– What is a MetaClass ?● Meta Object Protocol

– def methodMissing(String name, args) { }– def propertyMissing(String name)– def propertyMissing(String name, value)

● Overloading Operators

GORM as a DSL// These Class Definitions Generate SQL and Database Tables

class Book {String titleDate releaseDateAuthor author

}

class Author {String name

}

// These Operation Generate SQL and Database Queries

def book = Book.findByTitle("The Stand")book = Book.findByTitleLike("Harry Pot%")book = Book.findByReleaseDateBetween( firstDate, secondDate )book = Book.findByReleaseDateGreaterThan( someDate )

book = Book.findByTitleLikeOrReleaseDateLessThan( "%Something%", someDate )

GORM as a DSLclass GORM {

def dynamicMethods = [ ]

def methodMissing(String name, args) {def method = dynamicMethods.find { it.match(name) }if(method) { GORM.metaClass."$name" = { Object[] varArgs →

method.invoke(delegate, name, varArgs)}return method.invoke(delegate,name, args)

}// GORM actually writes or creates a new method hereelse throw new MissingMethodException(

name, delegate, args )}

}

DSL: Tool Kit● Meta-Methods

– String.metaClass.swapCase= {delegate.collect{ c →

c in 'A'..'Z'? c.toLowerCase() :c.toUpperCase()}.join()}

assert "Foo".swapCase()== "fOO"

Number.metaClass.multiply= { Grid g g.multiple(delegate) }→

● MOP– propertyMissing, methodMissing, respondsTo,

hasProperty, listMethods, selectMethod

● Categories, ObjectBuilder, MetaBuilder

Function Eval / Visualizationnew SwingBuilder().frame(title:'Quant Modeller', size : [1100,600], // .. parameters here not shown {

tableLayout() {tr {

td { widget(gui) } td { widget(editor, border : titledBorder('Model Script',

preferredSize :[900,600])) } td {

panel(border : titledBorder('Model Calibration'), preferredSize :[200,600]) {tableLayout(cellpadding :15) { tr {

td { label(text : 'Params:') }td { button(text : 'Add', \* Edited Out *\ }

tr { td(colspan : 2, align : 'center') {

button(text : 'Execute', actionPerformed : {

shell.setVariable 'f', [:]shell.eval1uate(editor.text)def f = shell.getVariable('f')def obj = [ f : f] as Model3dgui.model = objgui.redraw()

} ) } } }

New AST Transforms

• @Singleton — okay, not really a pattern :-)

• @Immutable, @Lazy, @Delegate

• @Newify

• More DSL and Meta-Magic! @Category and @Mixin

• @PackageScope

• Swing’s @Bindable and @Vetoable

• Grape’s @Grab

The Evil Java Singleton

public class Evil {public static final Evil instance = new Evil();

private Evil() {}static Evil getInstance() { return instance; }

}

• In Groovy now:–@Singleton class Evil {}

• A lazy version also:–@Singleton(lazy = true) class Evil {}

Credit: Guillaume Laforge, “To Infinity and Beyond” & GDK 1.7 Release notes

DSLs → They Start Out As APIs// closure maddness// Before refactoring

def phrase = "The quick brown fox jumps over the lazy dog"

def result = phrase.toLowerCase().toList().findAll{ it in "aeiou".toList() }. // like WHERE …groupBy{ it }. // like GROUP BY …findAll{ it.value.size() > 1 }. // like HAVING …sort{ it.key}.reverse(). // like ORDER BY …collect{ "$it.key:${it.value.size()}" }.join(", ")

println result

Credit: Paul King, “Groovy DSL”

DSL: APIs That Get Refactored// Refactored helper closuresdef lowercaseLetters= phrase.toLowerCase()def vowels = { it in "aeiou".toList() }def occursMoreThanOnce= { it.value.size() > 1 }def byReverseKey= { a, b -> b.key<=> a.key}def self = { it }def entriesAsPrettyString= { "$it.key:${it.value.size()}" }def withCommaDelimiter= ", "

// Refactored main closure println lowercaseLetters.

findAll(vowels). groupBy(self).

findAll(occursMoreThanOnce).sort(byReverseKey).collect(entriesAsPrettyString).

join(withCommaDelimiter)Credit: Paul King, “Groovy DSL”

Putting All Together I● Drop Parenthesis

● Named Parameters

● Objects Graphs → Tree of 'Nouns' and 'Verbs'

● Human Readable / Machine Executable

● Specific Syntax

Putting All Together IIorder to buy 200.shares of GOOG {

limitPrice 500 allOrNone false at the value of { qty * unitPrice - 100 }

}

take 2.pills of chloroquinine after 6.hours

def "length of Spock's & his friends' names"() { expect: name.size() == length where:

name | length "Spock" | 5 "Kirk" | 4 "Scotty" | 6

} Credit: Paul King, “Groovy DSL”

Config SlurperLog4j {

appender.stdout = "org.apache.log4j.ConsoleAppender"appender."stdout.layout"="org.apache.log4j.PatternLayout"rootLogger="error,stdout"

logger {org.springframework="info,stdout"

}

additivity {org.springframework=false

}}

databaseURL : //

development {databaseURL : //

}

testing {databaseURL : //

}

Performance ● Dynamic Typing is Expensive (But Worth It)

– Hot Spot Opportunity

● Dynamic Method Invocation is expensive (But Getting Better)

– Da Vinci Project and invokeDynamic bytecode– Hot Spot Opportunity

● BigDecimal & Collections → Insanely Expensive For Serious Math

– People need to better learn to use IEEE floating point types–

Performance II ● Getting Too Groovy ???

– Abusing .each { tuple } → // for loops versus anon closures

● Simplicity --> faster– Groovy reduces your dependency on Java Enterprise– Gpath vs Xpath (Gpath almost always wins)– Use Visual VM: Groovy Memory Use Can Be Mysterious

● Too Many Collections ? – Is The Idea to 'Cache' The World As A Map

● Use Groovy-Aware Java for Faster Code– People Need to learn IEEE Floating Point Types

Actors

class Player extends AbstractPooledActor {String namedef random = new Random()

void act() {loop {

react {// player replies with a random movereply Move.values()[

random.nextInt(Move.values().length)]

}}

}}

Scaling the VM: Immutable Data Message + Services

Are Java & C# Becoming Groovy ?● Google Guava (formerly Collections)● Lots of new Libraries for FP, Enhanced

Collections, Tuples, Monads, etc...● Doug Lea's Fork-Join + Predicate Libraries● Project Lambda : Closures for Java (JDK 8 ?)● Da Vinci Project : invokeDynamic (JDK 7 ?)● Is Gpath LINQ for Java ? (Now ?)● Universal Low Level VM● Will Groovy or Groovy++ Continue ?

A few bad months for Java

● JavaTM By Oracle , ● The Java Community Process, Doug Lea● The Java Community Process, Apache● JavaOne, The Conference● Oracle vs Google● Oracle vs Apache● IBM and Apache● OpenJDK

Where is Goovy Going ?● Some Important JDK Improvements will be in

JDK 7 (as planned): invokeDynamic

● More explicitly FP approach to concurrency and state

– AST MP / GoF Patterns– @Immutable / @Singleton ( lazy : true) – GPars Actor-like syntax

● Groovy Server for Fast Scripting– Difficult on a cluster

Where is Goovy Going ?● Closures with Memory def cl = {a, b →

sleep(3000) // create a noticeable pause a + b

}def mem = cl.memoize()

// memorize() variants are // memorizeAtMost(n), memorizeAtLeast(n), // memorizeBetween(m, n)

● Groovy++– Not a fully compatible fork– An interesting argument – But no.

Groovy Gotchas!● Subclassing Map, Unexpected Results

● Map.class● the dot operator,

getAt(...), getXXX(), [ name ] literals

● Private isn't Private ● Inconvenient when you really want to hide implementation data

● Unexpected Weird Effects – Combining Static and Dynamic Numeric Types Across Java Code

and Groovy Code ● Be careful and deliberate with typing

– Method selection surprises ● identical static closures and static functions do not respond to

the same way Invoked dynamically

Best Practices● Explicitly Signatures and Types Where You Can

● Better GroovyDoc documentation● Faster method selection● More shareable (with Java / Scala / Ioke / etc..) Libraries

● Write Groovy Aware (GroovySupport) Java● This is where to build your high-speed / high-performance code

● Use ConfigSlurper to externalize environment / context data

● Use CliBuilder and Groovy Server if it fits your production model

● Add Performance Tests To Your Suite / Learn to use Visual VM– Testing is more important with dynamic and functional programming– Small tests – small in terms of code coverage – on big data sets

● Best place to find surprising performance enhancements– Automating 'Just Got Lucky' Into Your Development Cycle

Acknowledgments & Suggestions

● Derik Kognig, Author Groovy In Action ● Peter Bell, Systems Forge● Paul King, Asset Australia ● Guillaume Laforge, Groovy Project Manager● PLEAC, Project Euler, Groovy Zone

Contact Info● jonathan.felch@gmail.com ● jfelch@volanttrading.com● www.twitter.com/Jonathan.Felch

– Rare but regular contributions via Blog, GitHub, SlideShare

● 3 – 5 contributions a year, typically unfinished half-baked ideas● Really targeted at a dozen or so people that I have worked with

in the past or occasionally collaborate