Failure to build a git dependency in production

I have a two projects.

One of which is a library that the other project uses.

Let’s call them “the-project” and “the-library”.

the-library was (still is) on Clojars, as a built Maven artefact.

the-library is built with leiningen.

the-project uses CLI / (with a deps.edn) file.

This setup has worked for me fine, BUT, now I’m doing some more work on, and adding new stuff to, the-library. It’s new stuff in the-library, but largely being written to serve new requirements in the-project.

So, I want to move the-project off dependency on the Clojars version of the-library, and on to something more frequently and dynamically updated.

I could point the-project’s deps.edn to the actual workspace on my machine where I’m building the-library.

But the-project is also public and up on github. So I don’t want to do that. Instead I want to make the-project depend on and use the version of the-library currently on github.

I update the github of the-library more frequently than the Clojars version.

So I’ve changed the-project’s deps.edn from getting the-library from Clojars to getting it from github.

And that seems to be working OK when I build the dev version of the-project. The latest version of the-library is now downloaded into my ~/.gitlib directory. And being successfully used in the-project.

However when I try to build a prod version of the-project (which is meant to go into an uberjar), the build blows up with the following error

Building main.js
Building uberjar
Compiling the-project.server
Syntax error macroexpanding at (library-wrapper.cljc:1:1).
Execution error (FileNotFoundException) at the-project.the-library/loading (library-wrapper.cljc:1).
Could not locate the-library/maths__init.class, the-library/maths.clj or the-library/maths.cljc on classpath.

This may be because the classpath (clj -Spath) doesn’t seem to include any references to .gitlib And it doesn’t seem like this attempt to build the code for prod knows anything about looking in ~/.gitlib

But doesn’t seem to have any way to tell the build process where to look. AFAICT it’s meant to “just work” if the deps.edn contains a reference to the git repo in the :app clause. (Which contains all the dependencies I believe that both :dev and :prod use)

But it doesn’t. Should I even be expecting it to? And, if so, what am I likely to be missing? Where should be I looking to solve this?

Here’s one thing I have done. Given the-library its own deps.edn file. It doesn’t really have many dependencies, and it’s normally built with lein, but on the grounds that will need to build it, I gave it one.

What does your actually do?

Do you have build.clj? What’s in that?

If you’re building an uberjar with, you typically compile-clj the main ns (or all nses) and that transitively compiles everything your code references – including any git deps because they’d be on the classpath from your deps.edn and hence in the basis you would create in build.clj.

Since clj -Spath doesn’t show your library, I wonder if your deps.edn is set up correctly.

TL;DR: show us, build.clj, and deps.edn


No. To be honest I don’t have a build.clj Nor have I been using compile-clj

I’ve just been building with

clj -A:prod:app

And that was working, as long as all my dependencies were from Clojars

I take it I should be doing something else then?

My deps.edn is pretty long.

Note my-library is actually Patterning. Which you’ll see me now trying to get from github.

{:paths ["src" "resources" ],
  {:extra-deps {clj-kondo/clj-kondo {:mvn/version "RELEASE"}}
   :main-opts ["-m" "clj-kondo.main"]}

   {orchestra/orchestra #:mvn{:version "2018.12.06-2"},
    expound/expound #:mvn{:version "0.7.2"},
    nightlight/nightlight #:mvn{:version "RELEASE"},
    com.bhauman/figwheel-main #:mvn{:version "0.2.0"}},
   :main-opts ["dev.clj"]},

  {:extra-paths ["test"]
    {:git/url ""
     :sha "209b64504cb3bd3b99ecfec7937b358a879f55c1"}
   :main-opts ["-m" "cognitect.test-runner"]}

   {leiningen/leiningen #:mvn{:version "2.9.0"},
    org.clojure/clojurescript #:mvn{:version "1.10.439"}},
   :main-opts ["prod.clj"]},

    markdown-clj/markdown-clj {:mvn/version "1.10.1"}
    instaparse/instaparse {
    org.clojure/core.logic {:mvn/version "0.8.11" }
    io.replikativ/hasch {:mvn/version "0.3.7"}

    remus/remus {:mvn/version "0.1.0"}

         {:git/url ""
          :sha "c22baf29cf33be1dd83dde3d0226b38ae96f77fa"}

    clj-rss/clj-rss {:mvn/version "0.2.5"}
    com.walmartlabs/lacinia {:mvn/version "0.36.0"}

    cljstache/cljstache {:mvn/version "2.0.6"}
    org.babashka/sci {:mvn/version "0.3.2"}

    org.clojure/core.memoize {:mvn/version "1.0.236"}

    org.clojure/data.json #:mvn{:version "0.2.6"},
    org.clojure/clojure #:mvn{:version "1.10.1"},
    reagent/reagent #:mvn{:version "0.8.0-alpha2"},
    org.clojure/tools.cli #:mvn{:version "0.3.5"},
    bidi/bidi #:mvn{:version "2.1.3"},
    com.h2database/h2 #:mvn{:version "1.4.196"},
    org.clojure/clojurescript #:mvn{:version "1.10.439"},

    hiccup/hiccup #:mvn {:version "1.0.5"}
    http-kit/http-kit #:mvn{:version "2.4.0-alpha6"},

    ring/ring #:mvn{:version "1.7.1"},
    {:mvn/version "1.6.0",
     :exclusions [org.bitbucket.daveyarwood/fluid-r3]},
    com.taoensso/sente #:mvn{:version "1.11.0"},
    org.clojure/java.jdbc #:mvn{:version "0.7.3"},
    org.clojure/tools.reader #:mvn{:version "1.3.2"},
    com.rpl/specter #:mvn{:version "1.0.4"},
    {:mvn/version "0.2.50",
    honeysql/honeysql #:mvn{:version "0.9.1"},
    ring/ring-core #:mvn{:version "1.7.1"},
    play-cljs/play-cljs #:mvn{:version "1.3.1"},
    org.clojure/core.async #:mvn{:version "0.4.490"}}}}}

So your “build” script is prod.clj?

I guess I have to ask what’s in that now…?

You mentioned – so I assumed you meant and therefore you were using build.clj which would be the standard CLI / deps.edn way to do things.

Oh! :face_with_open_eyes_and_hand_over_mouth:

I didn’t know about that : prod.clj

I don’t even remember writing it. I guess it got created when I created the project. (I used LightMod)

  '[clojure.string :as str]
  '[ :as api]
  '[leiningen.core.project :as p :refer [defproject]]
  '[leiningen.uberjar :refer [uberjar]]
  '[ :as io])

(defn read-project-clj []
  (-> "project.clj" load-file var-get))

(defn read-deps-edn [aliases-to-include]
  (let [{:keys [paths deps aliases]} (-> "deps.edn" slurp clojure.edn/read-string)
        deps (->> (select-keys aliases aliases-to-include)
                  (mapcat :extra-deps)
                  (into deps)
                  (map (fn parse-coord [coord]
                         (let [[artifact info] coord
                               s (str artifact)]
                           (if-let [i (str/index-of s "$")]
                             [(symbol (subs s 0 i))
                              (assoc info :classifier (subs s (inc i)))]
                    (fn [deps [artifact info]]
                      (if-let [version (:mvn/version info)]
                        (conj deps
                          (transduce cat conj [artifact version]
                            (select-keys info [:exclusions :classifier])))
        paths (->> (select-keys aliases aliases-to-include)
                   (mapcat :extra-paths)
                   (into paths))]
    {:dependencies deps
     :source-paths []
     :resource-paths paths}))

(def project (-> (read-project-clj)
                 (merge (read-deps-edn [:app]))

(defn delete-children-recursively! [f]
  (when (.isDirectory f)
    (doseq [f2 (.listFiles f)]
      (delete-children-recursively! f2)))
  (when (.exists f) (io/delete-file f)))

(def out-file "resources/clj_ts/main.js")
(def out-dir "resources/clj_ts/main.out")

(delete-children-recursively! (io/file out-dir))

(println "Building main.js")
(api/build "src" {:main          'clj-ts.client
                  :optimizations :advanced
                  :output-to     out-file
                  :output-dir    out-dir
                  :infer-externs true})

(delete-children-recursively! (io/file out-dir))

(println "Building uberjar")
(uberjar project)

(System/exit 0)

So, despite having a deps.edn, it’s actually using Leiningen under the hood to build the uberjar and Leiningen doesn’t understand git deps. So that’s your core problem.

It’s not even using to handle deps.edn and build a classpath etc – it’s treating it as just an EDN files containing dependencies, even tho’ deps.edn looks like a CLI project. That’s pretty far out there in left field so I can only assume this is quite an old project?

My advice would be to turn this into a proper CLI project, without project.clj and without Leiningen, and replace prod.clj with a build.clj file, based on – at least then you’ll be using standard tooling and there will be documentation and community you can rely on to help you.

1 Like

Thank you.

Right I’m taking a deep-breath. Grabbing an extra-strong cup of coffee, and sitting down to start the week by doing this : recreating the project from scratch.

Should I be following the steps in Clojure - Guide ? Would you say these the best, up-to-date instructions for how to do this?

What I really don’t understand about this new deps.edn way of doing things is where in the big map I’m constructing, should I put the dependencies.

I’m starting with a minimal deps.edn that looks like this

{:paths ["src"  "resources"] ;; project paths
 :deps {
    org.clojure/core.logic {:mvn/version "0.8.11" }

 {;; Run with clj -T:build function-in-build

  :build {:deps {io.github.clojure/ {:git/tag "v0.9.4" :git/sha "76b78fe"}}
          :ns-default build}

Which, at the moment, when I try to build, throws :

Execution error (FileNotFoundException) at clj-ts.logic/loading (logic.clj:1).
Could not locate clojure/core/logic__init.class, clojure/core/logic.clj or clojure/core/logic.cljc on classpath.

So I guess :deps at the top-level, alongside :paths etc. is wrong.

The old LightMod-created deps.edn had a pattern of {:aliases {:app {:extra-deps

I guess this is wrong, or was specific to that, and maybe some magic that its proj.clj performed. But where do I actually put the dependencies in deps.edn?

Clojure - Guide shows :deps within the :build clause, but I presume this is for libraries that the build process itself is dependent on, not the deps for my app itself.

So is deps.edn a file with a defined structure I should be following? Or is it a more-or-less free data-structure and I should be writing code to parse it in my build.clj?

If it does have a defined structure, where is that structure documented? Is there a page with useful examples? (Eg. how to organise it into dev vs prod builds etc?)

Actually, that’s weird.

I just added

    org.clojure/core.async #:mvn{:version "0.4.490"}

to the same :deps I had previously (alongside core.logic dependency) and now it seems to have moved on past that error.

Maybe this problem isn’t what I thought it was.

Possibly that top level :deps is correct, but some of the other interconnections between the dependencies I hadn’t satisfied yet are causing issues.

I think I’m going to close this, because the issues I now have have drifted a long way from the original topic of this thread.

At this point, you might want to join the Clojurians Slack – for self-signup – and join the #tools-build channel to get real-time, interactive help. CLI / deps.edn / tools.deps help can be found in the #tools-deps channel.

The Guide you linked to is a really good place to start. You may also find Clojure Guides: Building Projects: and the Clojure CLI ( helpful and, perhaps, the Clojure CLI - Practicalli Clojure guides.