Clojure interoperability with the native world

Community Center Watercooler
1

It is not enough to interoperate with JVM, the adoption of Clojure is hindered by the lack of interoperability with the native world, which is primarily built on top of C. An EDN C library would hugely alleviate this problem. What do you think?

1 Like
2

I’m not sure how practical it is, but I’m pretty sure you can use Rust libs from C: GitHub - naomijub/edn-rs: Crate to parse and emit EDN

There is a “WIP postgres extension” that might be a good base for a full EDN C library: GitHub - Astronought/pg_edn: A postgres extension to support EDN as a datatype

1 Like
3

There’s a few good options for C interop already:

But that’s ways to call C code from Clojure. With EDN support, do you mean like where it could communicate to a C program using EDN?

1 Like
4

Correct. If the native world cannot consume whatever the Clojure world produces, they remain disconnected. People would not be able to see the value of what Clojure can provide for them. For that matter, even on JVM, other languages cannot easily consume Clojure’s output. So Clojure remains isolated.

To put it more uncharitably, it seems that Clojure only takes from others, but contributes nothing back. That’s not interoperability. An EDN C library would be a good first step in fixing that.

5

I’m not disagreeing, that said specifically with regards to C, I was under the impression it’s rare that the C code itself wants to consume from others? Normally others want to call C code.

Like I’m not sure how many C web server or daemon there are out there which would benefit from being able to call a Clojure API that returned EDN or read EDN files?

6

Here’s the list of known implementations: Implementations · edn-format/edn Wiki (github.com) – it includes a partial implementation in C which anyone could have contributed to over the last nine years… perhaps you could take that up and submit PRs to complete it?

7

C is the bridge. To interoperate, most languages have a C API. Java, Python, etc, they all go through that bridge.

8

Thanks for this one.

9

Probably I will do something along this line. So that Datalevin can become a C shared library that other languages can link to and use.

2 Likes
10

With GraalVM you can compile a “.so” and “.dll” file I believe which expose a C ABI and link to them. I think @borkdude played with this before. So it could be you can expose an ABI that way?

If I understand, Datalevin APIs take and return EDN? If so, I guess ya you’d also need to provide a C EDN reader/writer probably as a utility.

Edit: Found the post about it:

11

Right, that’s when I found this missing bridge. If there’s no C EDN reader/writer, there’s no point building a libdatalevin, for there’s no code can consume the results. It’s fine to take the queries and transactions in text form, like most databases do, but the results must be something C code can natively consume, i.e. a C API. I think this is a problem not only applicable for database software, any Clojure programs that produce machine consumable results have the same problem. Hence my post.

12

Frankly, I think you’re completely overplaying this: there are many data formats that Clojure can easily share with all other languages – JSON is far and away more popular than EDN (I have no idea how much EDN is even used “in the wild”). Clojure can interop with pretty much any database, just like any other language. Clojure can interop with message queues, APIs that traffic in any text-based format (and many binary formats), and any number of other interop points, just like any other language.

If you’d gone after Transit, you’d have a better case – there are fewer implementations of Transit than there are of EDN but even there many languages support Transit.

Clojure is a JVM language and is just as interoperable with C as any other JVM language.

4 Likes
13

If I understand correctly, he wants to use Datalevin as a library from C code, so EDN support would be the simplest way to do so (passing queries and receiving the results).

1 Like
14

Then hopefully he will contribute to the existing work that has been done on EDN interop in C already, as indicated above, and we’ll see yet another implementation of EDN out there in the wild.

1 Like
15

I am not talking about how Clojure can use other technologies. That’s just one side of interoperability, the side of taking. I am talking about the side of giving. Both are needed to be true “interoperability”.

Now in term of giving, obviously adopting the most popular format is going to gain the widest audience. However, it does very little to improve the acceptance of Clojure in the eyes of the outsiders.

Take the example of Storm, it was written in Clojure, but it tried its best to hide Clojure from the users. Despite it was once popular, it did very little to help the Clojure world. In the end, it got rewritten in Java, actually hurted the credibility of Clojure in the process.

I want the opposite. I want people to see Clojure code. I want Clojure to infect. I want people to see it often, and in the process, see it as normal. I like how Riemann did it. It made no apology that it is Clojure. It is proud of Clojure. It forces sysadmin to write Clojure. In my mind, that’s the best way to increase adoption.

That’s one of the reasons why I spent a lot effor to make Datalevin native compatible.

3 Likes
16

I suspect some of the things that make Clojure, and by extension, things that Datalevin et al. nice to use are the ergonomics it offers for manipulating data. Some time ago I was thinking about using EDN from C#, for which there is a library, but in the end dropped the idea, because generating and manipulating those EDN structures would’ve been a major pain (funnily enough, this was in the context of accessing Crux directly from C#).

17

copying @borkdude

Things are looking up on expanding the native front for clojure with espresso (jvm on truffle on graal, which enables dynamic JIT’d code and reflection for native-image).

I just got this working in jshell-espresso as demonstrated here, which is running clojure through espresso:

(require '[clojure.pprint :refer [pprint]])

(defn test-program [n]
  (dotimes [i n]
    (println [:hello i])))

(println [:testing-clojure-from-espresso!])

(test-program 10)

;;let's see if macros work...
(defmacro dumb-macro [n]
  `[~@(->> (range n) (map inc))])

(pprint (dumb-macro 10))

(defprotocol IBlah
  (blah [this]))
(defrecord rec [x]
  IBlah
  (blah [this] (pprint [:blah x])))

(blah (->rec {:hello {:world {:how {:are :you?}}}}))
:done!

I can’t get the Xmx options to work yet (I think there’s going to be some heavy GC for real programs), but still, it works. The jar is an earlier uberjar from a test, which in this case is just bundling clojure 1.10 and spec together.

joinr@DESKTOP-UIAPQD5:~/repos/graalvm-demos/espresso-jshell$ ./espresso-jshell -Xmx4g --class-path truffletest-0.1.0-SNAPSHOT-standalone.jar
|  Welcome to JShell -- Version 11.0.10
|  For an introduction type: /help intro

jshell> String [] args = {"-e", "(load-file \"script.clj\")"}
args ==> String[2] { "-e", "(load-file \"script.clj\")" }

jshell> clojure.main.main(args)
[:testing-clojure-from-espresso!]
[:hello 0]
[:hello 1]
[:hello 2]
[:hello 3]
[:hello 4]
[:hello 5]
[:hello 6]
[:hello 7]
[:hello 8]
[:hello 9]
[1 2 3 4 5 6 7 8 9 10]
[:blah {:hello {:world {:how {:are :you?}}}}]
:done!

jshell> /exit

This is normally impossible, but jshell-espresso has hooks to allow dynamic classloading (essential for clojure function definition and anything touching eval or generating bytecode at runtime) that is running on espresso at runtime. I’m also very very curious to see if (via espresso) one can capture image-based development akin to common lisp/small talk since we have the JVM state in memory. It may be possible to have the equivalent of sbcl’s save-lisp-and-die to enable fast-loading tooling at some point. Performance is supposed to improve rapidly. Having trouble with the default clojure.main repl (it’s not playing nice with jshell-espresso, probably due to clobbering io), but evaluating scripts at runtime works (with all of clojure). At the moment this is akin to shipping an entire jdk though (espresso-jshell is like 80mb; curious if that can be tree-shaked down)…but I can imagine tying this back into a bundled native-image somehow for a shared lib or smaller executable (exposing an api via JNA/JNI as well for “portability”). It’d be kind of odd though, since you would still end up with a model of multiple “jvms” per shared library, absent some additional infrastructure. I think it’s at least looking possible going forward.

If you can live with closed-world assumptions (no eval, no runtime codegen or class loading), then exposing a c ABI via JNA/JNI is probably acceptable as well. At this point though, clojure is just the implementation language hidden behind a portable C interface, so maybe not pushing clojure code up front so much as you indicated (aside from submitting queries to datalevin in EDN and receiving results in EDN maybe).

I think the libpython-clj (and libjulia, and libapl, and clojisr) are trying very hard to achieve the same bridging strategy (with an eye toward clojure “taking” at first, but also “giving” at some point). The “taking” strategy is more to provide a path for python/r users (much like clojure did for java/ruby and some python) to migrate toward clojure, and allow clojure users to tap into the disparate ecosystems (or wrap existing code).

5 Likes
18

I think a fast and complete C EDN reader/writer would be pretty neat to quickly add support for EDN to a lot of languages that can interrop with C.

I also agree with you, if you don’t mind missing on some potential users, keeping the interfaces Clojury is nice, I also do like that of Riemann, and keeping Datalevin an edn query, edn result, instead of switching to JSON or something else is in good spirit of staying committed to Clojure.

2 Likes
19

We opened sourced a C implementation of the EDN reader a few years ago, but it’s invisible because it’s integrated into a ruby gem. (We needed to consume EDN from legacy Rails code; this was ~100x faster than the pure Ruby gem that existed at the time.)

If you want to build your own C EDN reader library, you might want to start there. Note, though, that it’s unclear what the EDN data structures should map to in C because there are no universal versions of the container types in that world. For example, whatever set library you might choose/build won’t be everyone’s favorite. :man_shrugging:t2:

2 Likes
20

I think if we really want to share the benefits of Clojure with other runtimes, we need something like Flat Buffers for Clojure data structures. Not just EDN but chunked seqs and shared persistent data structures built right on top. Then, ideally, we could have different host environments in, for instance, Wasm, sharing Clojure persistent data structures and banging on that data in parallel, while still adhering to the concurrency semantics of Clojure. Then you could use Clojure-Rust to do some heavy crunching and then do zero-copy deserialization from CLJS in the same Wasm environment. It might work in a Truffle shared runtime too.