Statically expose pod dependencies — idea/proof of concept

As people discovered on various occasions Boot may download dependencies at any point in time to create isolated Clojure runtimes (“pods”). These are super useful to prevent dependency conflicts introduced by tooling helping you to avoid dependency hell as long as possible.

Not knowing what dependencies will be needed for various tasks can be a bit annoying though. In Continous Integration (CI) you usually want to download your dependencies once and cache them. Without knowing all dependencies up front that’s impossible to do. (The current workaround is to just run your tests during the setup stage but that’s not nice really.)

The deps.edn format that’s part of tools.deps.alpha sparked an idea that might fix this: tasks could expose their dependencies via deps.edn files. :bulb:

  • When packaging jars containing boot tasks which may use pods, include a file ending in .pod.deps.edn specifying all required dependencies
  • During the setup stage of your CI process, scan the classpath for these files and resolve/download all dependencies specified within
(require 'clojure.tools.deps.alpha.providers.maven)
(require '[clojure.tools.deps.alpha :as td]
         '[clojure.tools.deps.alpha.reader :as reader])
(require '[clojure.java.classpath :as jcp])

(defn pod-deps-edn-files []
  (->> (mapcat jcp/filenames-in-jar (jcp/classpath-jarfiles))
       (filter #(.endsWith % ".pod.deps.edn"))
       (map clojure.java.io/resource)))

(def f (first (pod-deps-edn-files)))

(defn download-deps [f]
  (td/resolve-deps
   (reader/merge-deps 
    [(reader/read-deps [(java.io.File. "/usr/local/Cellar/clojure/1.9.0.273/deps.edn")])
     (clojure.edn/read-string (slurp f))])
   {}))

Boot tasks could of course also use the information specified in this deps.edn file to construct the pod:

(defn make-ancient-pod []
  (let [deps-vec (->> (io/resource "org/martinklepsch/boot_deps.pod.deps.edn")
                      slurp clojure.edn/read-string :deps
                      (mapv (fn [[k v]] [k (:mvn/version v)])))]
    (pod/make-pod (assoc (boot/get-env) :dependencies deps-vec))))
;; org/martinklepsch/boot_deps.pod.deps.edn
{:deps {slingshot   {:mvn/version "0.12.2"}
        ancient-clj {:mvn/version "0.3.14"}}}

The deps.edn to dependency vector parsing is a stub at this point but I guess something like this should be possible, for most cases probably even without loading tools.deps proper.

Curious what you think! cc @juhoteperi @danielcompton

2 Likes

I’m a complete Boot novice, so I don’t understand all of the fine details here, but it seems like a good idea and would solve the problems I have. Leiningen plugins also have a similar issue, and this kind of approach could be applied there too.

Ultimately it would be great if this could become a standard and be built-in to Boot, but prototyping it outside seems like a good approach.

I like this, good job.

When I built boot-tools-deps one of things I felt was important to expose was the ability to leverage the aliases in deps.edn files to support overrides, defaults, and extras etc. Leveraging that tools.deps functionality consistently is something that I think all tools trying to build on deps.edn files should do.

Simply reading in a local deps.edn file and pulling the :deps out of it is misleading, in my opinion, and misses out on a lot of what makes tools.deps so powerful.

With some refactoring, boot-tools-deps could more easily support use within pods or even as a way to just display what the dependencies would be. It already has most of the logic in a function load-deps rather than locked into a task, so that it can form the basis of other Boot tooling. @martinklepsch opened an issue on GitHub to trigger that refactoring and a discussion about pods – feel free to post thoughts in there too.

As I told Martin, I’m a bit snowed at work through year end but happy to continue work on boot-tools-deps as and when I can.

I think I understand. And for a project’s deps.edn loader I totally agree.

Just pulling out :deps and ignoring the rest is a bit weird. That said I consider the usage of tools.deps more an implementation detail here. We might as well just use an edn file with a traditional dependency vector and some boot functions to download the deps (these would be in boot.aether I guess).
99% of the time when creating pods you just add some dependencies to a fresh or existing boot env.

Do you see any way in which deps.edn profiles could be useful for pods?

If deps.edn is packaged into the jars, similar to pom.xml, we could inspect those to check for deps with specific aliases.

For example boot-alt-http could spec have deps.edn like this:

{:deps
 {...}

 :aliases
 {:http-pod {:extra-deps {http-kit {:mvn/version "2.2.0"}
                          ring/ring-core {:mvn/version "1.5.1"}}
             :boot/pod? true}
  :another-pod {...
                :boot/pod? true}}}

Then when creating pods, the task would specify the alias and tools.deps would load given deps. To download these deps beforehand, we could find all the deps.edn files in classpath, under META-INF/maven/groupId/package/deps.edn or something, and load deps for all aliases tagged with :boot/pod?.

Not sure though if the packages are going to include the deps.edn file, or if aliases can contain extra data.

1 Like

Btw. similar system could work with current Maven dependencies. Boot tasks could add property into pom.xml:

<properties>
   <bootTask>true</bootTask>
</properties>

And when downloading dependencies, we could find any pom.xml in classpath with this property, and then read the dependencies from some part of the pom.xml. Maybe some scoped dependencies (like test), or maybe dependencies of from some specific Maven profiles. Boot pom task probably can’t currently write such pom.xml but that shouldn’t be to hard to extend.

Maybe something like:

  <profiles>
    <profile>
      <id>http-pod</id>
      <dependencies>
        <dependency>
           <groupId>http-kit</groupId>
           <artifactId>http-kit</artifactId>
           <version>2.2.0</version>
         </dependency>
      </dependencies>
      <properties>
        <bootPod>true</bootPod>
      </properties>
    </profile>
  </profiles>

One of the things I use pods for is to run tests with different versions of Clojure (or different versions of some other dependency) so, yes, being able to use the aliases to indicate which set of dependencies to use in the pod would be very valuable.

Putting it into a more standard location like META-INF is a good idea but introduces some additional complexities during development since you don’t have a jar at that point. (Same issue applies to putting it in pom.xml.)

I like the idea of having pod dependencies specified as aliases with a specific tag though.

I have mostly been thinking about pods inside boot-tasks which are opaque to the user of the task. What you describe seems like a situation where you create the pods yourself and thus are aware/in control what goes into them.

I do agree however that an API to make a pod from a deps.edn file (+ any parameters like profiles) would be very nice.

As Juho showed, the basic idea is to simply store the deps somewhere where it can be read afterwards. I personally would prefer edn over xml so the idea of using deps.edn appeals to me.
One of the things that I wanted to add to the list of requirements would be to dynamically modify the pod classpath. Maybe in phase two of this change, but many many times I have wondered if I could add a dependency to the pod while developing stuff (boot-figreload for instance).

Towards that goal if the we store each pod deps in an .edn in the jar, we should also think of a mechanisms to “override” those deps from a local file. But hey, we already have something like that in current tools.deps. If we represent pod deps as Juho wrote above, with aliases, then a “local” deps.edn could contain :override-deps for a certain pod key and voila’, your pod now reads it and loads the new dep.

I have been dreaming this many many times :slight_smile:

Pod dependencies can already be updated dynamically, Boot-cljs does this if user adds new dependencies to the project env: https://github.com/boot-clj/boot-cljs/blob/a30773d769b8b66cd228e7a7718bd363938979ec/src/adzerk/boot_cljs.clj#L155-L156