Python and Clojure in cooperation?

[Webdev_Tory]

I work in academia, where Python has a strong grip (especially, it seems, in Computer and Data Science). Now, I don’t have any complaints with Python (other than that I believe Clojure’s syntax is miles beyond Python), but it lacks the sheer awesomeness of Clojure (that could be another post…). For community sake, though, I find myself wondering, are there any solutions to interoperability with Python? Has anyone made headway in Jython or anything else which could allow Clojure users to access Python libraries, perhaps? This comes to mind as all my colleagues are using Numpy, Gensimm, and NLTK, and as with all things Python, rave about the easiness of the process.

I realize this might be a misguided question, as Clojure is a hosted language and Python doesn’t seem capable of being a good host. It’s really just the eco-system and developer familiarity that could be desirable, since Clojure seems to have a tragic shyness about working on data science.

[alanmarazzi]

I’m very interested in this topic as well. I’m a data scientist, and Python is king among my colleagues. It would be very nice to be able to pipe stuff to scikit-learn or other popular packages, while keeping Clojure to perform I/O, ETL, keeping track of experiments, etc

[claudiu]

https://www.graalvm.org/docs/why-graal/

http://gigasquidsoftware.com/blog/2017/10/22/embedded-interop-between-clojure-r-and-python-with-graalvm/

Thomas Wuerthinger on Twitter

The graal project looks very interesting.

[jonasseglare]

I had a related problem, trying to call C++ code from Clojure. I ended up writing an abstraction layer that would translate my Clojure datastructures to something C++ would understand, and call C++ code from Clojure using JNA, with the C++ code being compiled into shared libraries. Unfortunately, I had issues with trying to reload the shared libraries in the REPL while editing Clojure and C++ code in parallel.

If I would do this again, I would probably try to use something like Apache Thrift (https://thrift.apache.org/) to glue them together. I could use Apache Thrift to generate Java code based on the protocol specification and have it automatically loaded in the REPL using the “lein-virgil” Leiningen plugin.

I believe you could try to use Apache Thrift to make your Clojure code talk to your Python code.

[Webdev_Tory]

Very interesting indeed! It looks tremendously promising. Too bad it can’t do the sophisticated Python libraries yet.

[baritonehands]

I instantly thought of Pixie but alas, it has its own VM. It doesn’t run on python or the jvm.

[baritonehands]

If interop directly within the same process is not possible, you could always do communication between two processes by writing an interface both understand. Since both languages are dynamic, some kind of RPC could be achieved.

For example, I did a proof of concept implementation of the Clojure nREPL in a Python shell: python-clojure-shell. All the python code will be converted to Clojure and evaluated inside the remote REPL session. Python data will be converted to EDN before passing to the REPL. This allows you to mix together remote Clojure data and local python data. Also, code is automatically converted from snake_case in python to camelCase in Clojure, since those are the preferred conventions.

>>> range_r = var('range')
>>> plus_r = var('+')
>>> reduce_r = var('reduce')
>>> reduce_r(plus_r, range_r(100))
4950

The above code is converted to:

(reduce + (range 100))

The Clojure syntax was simple enough to generate dynamically using python’s dynamic features. I bet you could do something similar in the other direction, where the Clojure code generates python to be evaluated in the other process.

[microamp]

If you don’t like Python’s syntax, then you may want to try Hy. There, you can use all of Python’s libraries without interoperability concerns.

[didibus]

Did they fix the missing let special form?

[draganrocks]

For those of you who are interested in implementing your own stuff, Clojure way, much faster than Python or Java, I’d suggest https://uncomplicate.org. It (still) does not provide ready-made black-box libraries like Pyhon does - someone has to implement that - but it does have a numpy-equivalent, that even works on the GPU out of the box AND enables easy-ish interactive OpenCL or CUDA development of highly efficient numerical algorithms for CPU and GPU…

Quite a few tutorials are available at https://dragan.rocks with more to come.

NOTE: This is primarily for implementing ML/Data Science algos and libs. It does not include the usual boilerplate and plotting stuff that the data science practitioner needs. But, having seen that some people were even ready to mix C++ into the story (ouch!) - the approach I’m suggesting is much easier and performant…

[Webdev_Tory]

Ah, yes – Uncomplicate, creator of Neanderthal. I’m a fan of their work! (although I’ve been using deeplearning4j and its constituent matrix stuff instead, atm)

EDIT: Just saw that Dragan, the creator, is the one who posted this You’ve done great work, and I look forward to working more in Neanderthal!