Deploying AOT compiled libraries?

What are the downsides of AOT compiling libraries? When it is ok to do that? Has a big impact on library load times in the REPL.

;; 0.3ms
(time (require '[clojure.spec.alpha]))

;; 928ms
(time (require '[schema.core]))
1 Like

Your dependencies become the fixed versions which must be used by any upstream consumers. I believe there’s a case where you now cannot depend on a different version of tools.reader than clojurescript. I think there’s some tracking at [CLJ-322] - JIRA to add support for compiling only your own library.

There is a gist that was put together by @hiredman which covers some issues of AOT, aot.org · GitHub. I think the notably applicable one here is:

The ABI is not stable.

Unstable abi means code compiled with Clojure version X is incompatible with code compiled with Clojure version Y
Once you AOT compile a library though, consumers can only reliable use the library with the same version of Clojure it was initially compiled with.

1 Like

The main problem is that AOT will compile your code and all its dependencies to .class files that cannot be separated from each other. So you cannot just ship your library AOT compiled as it would also contain at least clojure.core. Clojure AOT currently as far as I’m aware can not load .clj files from .class files. So AOT produces the .class files and will fail if one of the dependent classes is missing although the .clj file is present.

This leads to hard to track errors like if you lib includes a different version than may be specified in the projects dependencies as the .clj files will be of that version but only the AOT compiled versions that you shipped will be used.

2 Likes

Thanks for the replies. Could we solve this on the user side then? After the dependencies are compiled locally, next time the compiled classes would be used instead of recompiling them. Should work if no dependencies have been updated? At least with leiningen defaults, it doesn’t seem to work that way today.

Sure.

In lein you can set :aot [your.main] and lein will ensure that the class files are updated and used. This is a bit annoying during development since the extra checks if everything is up to date take some time and the AOT compilation does as well.

In my projects I just set :aot in the :uberjar profile so its only done for production uberjars.

1 Like

The significant downside of AOT’ed code is that it is inherently the product of a particular version of the Clojure compiler. While we do try pretty hard not to break the runtime interface (particularly stuff like the RT and Reflector APIs) we don’t exhaustively test for that.

In general, I would recommend NOT AOT compiling libraries (or at least not only distributing that). In a few cases we publish contrib libs under both source and AOT version (with “aot” classifier). Also, there are issues around loading related to having AOT libs that depend on non-AOT libs. If you want better performance at the app level, I think it’s pretty reasonable to AOT a final app uber jar as that avoids most of the issues with most of the benefits.

Rich and I have been kicking around some interesting ideas on a class compilation cache that would be integrated with core and that research is in scope for Clojure 1.10 but no guarantees on anything. Potentially it would combine the benefits of publishing source-only but the performance of AOT. It is inherently silly to recompile the same lib source file over and over when it’s not changing.

8 Likes

It is inherently silly to recompile the same lib source file over and over when it’s not changing.

Exactly. Looking forward to seeing the improved caching.

I’m curious if the class compilation cache feature being considered for Clojure has interesting ideas surrounding “impure” macros.

This causes an issue for a similar feature recently introduced in ClojureScript (see the the note in https://clojurescript.org/news/2018-03-28-shared-aot-cache), with no real good solution being devised.

One issue here, is a lib intended for consumption by both Java and Clojure, or just Java. You have to gen-class, and that will inherently mean some of your code and all transitive dependencies will also be compiled with it.

You can get around that by using runtime require/resolve so that AOT doesn’t “see” the namespaces brought in at runtime.

When I’ve produced a Clojure ns that needs to be compiled (because of gen-class), I’ve used runtime require calls so that only that one gen-classd namespace gets compiled (Clojure’s core is already compiled so using built-in functions makes no difference to what compiled code you end up with).

The other alternative for Java-calling-Clojure is to use Clojure’s Java API (perhaps wrapped in a Java class that you compile separately and include just its .class file(s) on your classpath for building the library).

Thanks @seancorfield for all the advice. I ended up adding a manual Jar packing build step which takes a list of gen-class names and only include their resulting .class in the Jar. So my Jars are full Clojure source + gen-class .classes only. And when I compile, I just AOT all.

Found this to be cleaner, my code can remain normal, no weird runtime require hacks, and I can still leverage gen-class instead of having to write Java wrappers with Clojure Java APIs by hand.

Any news on the class compilation cache @alexmiller? Empty REPL may start fast, but for example with Pedestal, as soon as one requires some library namespaces, core.async is required as one of the dependency and it takes ~3sec to compile on a new MacBook PRO.

user=> (time (require '[clojure.core.async]))
"Elapsed time: 2850.075649 msecs"

Nope, working on spec right now.

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.