Deploying AOT compiled libraries?


#1

What are the downsides of AOT compiling libraries? When it is ok to do that? Has a big impact on library load times in the REPL.

;; 0.3ms
(time (require '[clojure.spec.alpha]))

;; 928ms
(time (require '[schema.core]))

#2

Your dependencies become the fixed versions which must be used by any upstream consumers. I believe there’s a case where you now cannot depend on a different version of tools.reader than clojurescript. I think there’s some tracking at https://dev.clojure.org/jira/browse/CLJ-322 to add support for compiling only your own library.

There is a gist that was put together by @hiredman which covers some issues of AOT, https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e. I think the notably applicable one here is:

The ABI is not stable.

Unstable abi means code compiled with Clojure version X is incompatible with code compiled with Clojure version Y
Once you AOT compile a library though, consumers can only reliable use the library with the same version of Clojure it was initially compiled with.


#3

The main problem is that AOT will compile your code and all its dependencies to .class files that cannot be separated from each other. So you cannot just ship your library AOT compiled as it would also contain at least clojure.core. Clojure AOT currently as far as I’m aware can not load .clj files from .class files. So AOT produces the .class files and will fail if one of the dependent classes is missing although the .clj file is present.

This leads to hard to track errors like if you lib includes a different version than may be specified in the projects dependencies as the .clj files will be of that version but only the AOT compiled versions that you shipped will be used.


#4

Thanks for the replies. Could we solve this on the user side then? After the dependencies are compiled locally, next time the compiled classes would be used instead of recompiling them. Should work if no dependencies have been updated? At least with leiningen defaults, it doesn’t seem to work that way today.


#5

Sure.

In lein you can set :aot [your.main] and lein will ensure that the class files are updated and used. This is a bit annoying during development since the extra checks if everything is up to date take some time and the AOT compilation does as well.

In my projects I just set :aot in the :uberjar profile so its only done for production uberjars.


#6

The significant downside of AOT’ed code is that it is inherently the product of a particular version of the Clojure compiler. While we do try pretty hard not to break the runtime interface (particularly stuff like the RT and Reflector APIs) we don’t exhaustively test for that.

In general, I would recommend NOT AOT compiling libraries (or at least not only distributing that). In a few cases we publish contrib libs under both source and AOT version (with “aot” classifier). Also, there are issues around loading related to having AOT libs that depend on non-AOT libs. If you want better performance at the app level, I think it’s pretty reasonable to AOT a final app uber jar as that avoids most of the issues with most of the benefits.

Rich and I have been kicking around some interesting ideas on a class compilation cache that would be integrated with core and that research is in scope for Clojure 1.10 but no guarantees on anything. Potentially it would combine the benefits of publishing source-only but the performance of AOT. It is inherently silly to recompile the same lib source file over and over when it’s not changing.


#7

It is inherently silly to recompile the same lib source file over and over when it’s not changing.

Exactly. Looking forward to seeing the improved caching.


#8

I’m curious if the class compilation cache feature being considered for Clojure has interesting ideas surrounding “impure” macros.

This causes an issue for a similar feature recently introduced in ClojureScript (see the the note in https://clojurescript.org/news/2018-03-28-shared-aot-cache), with no real good solution being devised.