Dynamically adding to the classpath in a post Java 9 world. Help?

Questions & Help How to?
1

I’ve been trying to figure out how to dynamically add an entry to the classpath, in a way that works under Java 8 and Java 9, that works when running under Leiningen or through clj. I’ve gone down the rabbit hole trying to understand classloaders and the Java 9 changes, and I’m still utterly confused and without a working solution.

I’ve compared Pomegranate 1.0.0, Pomegranate 0.4.0, Dynapath, and the experimentel add-lib from tools.deps.alpha, and it seems the only thing that actually works in using Pomegranate 0.4.0 on Java 8. Here’s a repo with my experiments.

Before Java 9 Pomegranate/Dynapath would use Java reflection to change Java’s URLClassLoader to make it modifiable/dynamic. This is no longer possible in Java 9 (the dynapath README says it only results in a warning, but it still seems to break the existing behaviour).

The Dynapath README hints that you should implement your own dynamic class loaded. Clojure already implements one though, clojure.lang.DynamicClassLoader. You can try installing one of these

(let [thread        (Thread/currentThread)
      contextloader (.getContextClassLoader thread)
      classloader   (clojure.lang.DynamicClassLoader. contextloader)]
  (.setContextClassLoader thread classloader))

This seems to work at first, after this use any of the above libraries to add a directory to the classpath and e.g. (io/resource "...") will work, but a (require '...) will not. Clojure does not “see” this new entry.

I’d like to understand why this is so hard? How do you install a dynamic classloader that actually works? Why isn’t there a library that handles this? Has Java 9 doomed us to a static reality?

1 Like
2

I see two possible reasons for the behavior you are seeing.

  • When adding URLs to a DynamicClassLoader you might need to use the DynamicClassLoader that is the higher in the hierarchy to increase the chances for the ClassLoaders responsible for loading classes/namespaces/resources to see the new URLs. You can iterate the hierarchy using the .getParent method of a ClassLoader.

  • You should check if the DynamicClassLoader used to add new URLs is an ancestor of the ClassLoader bound to the var clojure.lang.Compiler/LOADER.
    clojure.main/repl creates and sets a new DynamicClassLoader (here), before the clojure.lang.Compiler/LOADER var is bound. Thus the DynamicClassLoader is an ancestor of clojure.lang.Compiler/LOADER. It might not be the case when starting the REPL with Leiningen.

3

I believe this is impossible using nREPL under Java 9 right now. There’s a fix in the new nREPL, but no-one uses that yet and it’ll be a while before we can rely on everyone in the wild using it.

Classloaders with Clojure are quite a mess. I’ve actually been meaning to write a blog about it, partly to actually understand it myself (I have to learn about it periodically when I need to but there are lots of subtleties I don’t get), and also to have something to refer to myself when I next need to learn it all again.

1 Like
4

I’ve been trying some more things, and something fairly unexpected is that with a CIDER/nREPL setup, each evaluation adds an extra classloader :see_no_evil:

Not sure yet where this is happening or under which condtions, but try this

(defn prn-classloaders []
  (do
    (println (apply str (repeat 120 \=)))
    (let [loaders (->> (.. java.lang.Thread currentThread getContextClassLoader)
                       (iterate #(.getParent %))
                       (take-while identity))]
      (doseq [cl loaders]
        (prn cl))
      (prn (count loaders)))))

On a freshly booted JVM this shows a hierarchy of three classloaders

========================================================================================================================
#object[clojure.lang.DynamicClassLoader 0x20f26043 "clojure.lang.DynamicClassLoader@20f26043"]
#object[sun.misc.Launcher$AppClassLoader 0x5c647e05 "sun.misc.Launcher$AppClassLoader@5c647e05"]
#object[sun.misc.Launcher$ExtClassLoader 0x200056a4 "sun.misc.Launcher$ExtClassLoader@200056a4"]
3

Then, each time you evaluate something, the chain gets longer.

I should note this is with “classic” nREPL, I haven’t tried it with the new 0.4 stuff.

1 Like
5

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.

7

I actually struggled with the same issue and ultimately managed to make it work. You actually just need to add DynamicClassLoader and assign it to clojure.lang.Compiler/LOADER
See for example here.
Took me quite a long time before realizing that things that do not work in REPL might work in standalone mode (I was kinda expecting it to be only the other way around).
So basically though my solution does not work in REPL (probably due to new classloaders being created and bound repeatedly), when run normally (via java -jar …) it actually works - I tested it on Java 9 - 11 and it works even when using the Pomegranate 1.1.0 with Dynapath 1.0.0. (I use them e.g. here ).
When I want to use/test in REPL I simply have a separate lein profile that uses Java 8 and Pomegranate 1.0.0 and Dynapath 0.4.0

1 Like
8

Thanks for bringing this up again @mikub. I managed to get it working in the end as well, here’s my solution: https://github.com/lambdaisland/kaocha/blob/master/src/kaocha/classpath.clj

It also binds Compiler/LOADER. After much poking around in the Clojure source I figured out that if that var is bound it takes precedence. Seems like a massive hack relying on deep implementation details, but hey it works.

The only downside is it doesn’t work for ClojureScript, which is why kaocha-cljs still has this issue.

1 Like