clojure at datastax: the long road from python to clojure

CLOJURE AT DATASTAX: THE LONG ROAD FROM PYTHON TO CLOJURENick Bailey @nickmbailey

1 Introduction

2 Starting with Python

3 Clojure Enters the Ring

4 Clojure Everywhere!

5 Takeaways

1. Introduction

Me• DataStax • Clojure • Apache Cassandra • Austin

DataStax• DSE (distributed database) • Java

• OpsCenter • Python • Clojure

This talk• DataStax Clojure experience report • Python shop interested in Clojure? • Have a large python application you wish was written in Clojure?

2. Starting With Python

A long time ago in an galaxy far far away…

A long time ago in an office not far away…

(5 years ago)

Building a Database Monitoring Tool

Ok… but why?

Why Twisted?

• 2010 • 10ish person startup • 1 front-end dev • 1 back-end dev

Product Complexity Grows

• Single process isn’t enough • Need an agent to run on the cluster

Product Growth

Things go wrong

• The JVM doesn’t play nice • Python may not be the best choice for monitoring a JVM… • … but there is definitely a JVM available…

3. Clojure Enters the Ring

3. Clojure Enters the Ring (pun intended)

(still 5 years ago)

Replacement Needed

Responsibilities

• Gather and report metric data

Requirements

• Need to interact heavily with a JVM based database • Maintainability • Developer happiness

Contenders

• Java • Clojure • Scala • Jython

Why Clojure

• Isolated piece of the architecture • Immutability was a good fit • Great interoperability • An experiment worth taking a risk on

New Architecture

Experiment Results…

Smashing Success!

• Clojure fit all of our needs • Great recruitment tool • Clojure inspires passion • Picked up by new developers

On Learning Clojure

“…the most important transition will be opening your soul to Truth and Beauty, if you do so, Clojure will come naturally…”

—Philip Doctor

A Note on Hiring

• Datastax has a distributed engineering organization • Our clojure engineers span the globe

The rough edges

• Deployed on 1.2, immediate painful migration • JMX library unmaintained • Less mature libraries

Ok… why didn’t you just rewrite it all?

Hindsight is 20/20…

• Clojure was definitely considered an experiment/risk • Still a tiny startup • Bulk of code and functionality in the Twisted code

4. Clojure Everywhere!

Back to the future!

What’s changed?

• 2 person team to 20 person team • Clojure codebase has grown significantly • JVM usage has increased across the company • Sorry Python, Clojure is the favorite child

Time to Reevaluate

Reevaluating Twisted

• Scalability requirements have grown • GIL rears it’s ugly head • Overlapping responsibilities

Reevaluating Twisted

• Need a path the JVM • Need to address scalability concerns

Possible Solutions

• Rewrite completely • Multiple processes • Jython?

Rewrite completely

• Benefit of hindsight when starting from the ground up • 4+ years invested • Pause feature development • Inevitable rewrite bugs introduced

Multiple Processes

• Less time intensive than a re-write • Can gradually move things out of python • Multi-processing is difficult • Untangling Twisted is extremely difficult • Adding IPC • Additional customer burden

Port to Jython

• Theoretically completely behind the scenes • Shortest path to being completely on the JVM • No out of the box support for Twisted • Introduction of a “third language” • The road less traveled

Jython Wins!

The Present

Migrating to Jython1. Make Twisted run on Jython 2. Replace all broken python libraries 3. Fix all the Jython* bugs you found 4. Write Jython <-> Clojure translation layer

1. Make Twisted run on Jython

• Upgrade Twisted • Start application on twisted • Get a stack trace and fix the issue • Rinse • Repeat

2. Replace all broken python libraries

• C extensions won’t run on Jython • SSL, LDAP, sqlite, crypto • Jython doesn’t support python sub processing

3. Fix all the Jython* bugs you found

• Brace yourselves…

3. Fix all the Jython* bugs you found

• Socket leaks • Deadlocks (including during garbage collection) • Incorrect socket implementations • Lots of SSL bugs • *And also 1 bug in Netty

If you thought stack traces and heap dumps were bad in Clojure…

Write Jython <-> Clojure translation layer

• The fun part • Clojure: “What’s a PyFunction object!?” • Jython: “Shut up, what’s an IPersistentVector!?” • Luckily, everyone speaks Java • https://gist.github.com/nickmbailey/3f11300e944e17d92edd6d42da785ab3

The Present

Success?

• There were some bumps a long the road • Took a lot longer than hoped • It was definitely the right choice

Over the Rainbow

Takeaways

Should I use Clojure?

• Yes, open yourself to Truth and Beauty

Should I not use Python?

• No one said that! • But if you are mainly interacting with the JVM… probably not.

Should I migrate my Python app to Clojure via Jython?

• Maybe? • Do you need to be on the JVM? • How much C code will you need to replace? • Are you brave enough to venture into the Jython codebase itself?

Questions? nickmbailey@gmail.com

clojure at datastax: the long road from python to clojure

Software

clojure basics

clojure and altlaw - digital digressions by stuart...

clojure & functional programming? -...

intro clojure

the ideas of clojure - things i learn from clojure

datastax | data science with datastax enterprise (brian...

sql support in datastax enterprise€¦ · sql support in...

best practices in deploying and managing datastax...

clojure presentation

datastax odbc driver for apache cassandra and datastax ......

clojure concurrency

datastax: an introduction to datastax enterprise search

clojure noir

clojure 1a

clojure for business...

clojure 1.5

clojure values

storm and clojure - clojure meetup january 2015

(first '(clojure.))

mainstay datastax