Failure to build a git dependency in production

[interstar]

I have a two projects.

One of which is a library that the other project uses.

Let’s call them “the-project” and “the-library”.

the-library was (still is) on Clojars, as a built Maven artefact.

the-library is built with leiningen.

the-project uses CLI / build.tools (with a deps.edn) file.

This setup has worked for me fine, BUT, now I’m doing some more work on, and adding new stuff to, the-library. It’s new stuff in the-library, but largely being written to serve new requirements in the-project.

So, I want to move the-project off dependency on the Clojars version of the-library, and on to something more frequently and dynamically updated.

I could point the-project’s deps.edn to the actual workspace on my machine where I’m building the-library.

But the-project is also public and up on github. So I don’t want to do that. Instead I want to make the-project depend on and use the version of the-library currently on github.

I update the github of the-library more frequently than the Clojars version.

So I’ve changed the-project’s deps.edn from getting the-library from Clojars to getting it from github.

And that seems to be working OK when I build the dev version of the-project. The latest version of the-library is now downloaded into my ~/.gitlib directory. And being successfully used in the-project.

However when I try to build a prod version of the-project (which is meant to go into an uberjar), the build blows up with the following error

./build.sh 
Building main.js
Building uberjar
Compiling the-project.server
Syntax error macroexpanding at (library-wrapper.cljc:1:1).
Execution error (FileNotFoundException) at the-project.the-library/loading (library-wrapper.cljc:1).
Could not locate the-library/maths__init.class, the-library/maths.clj or the-library/maths.cljc on classpath.

This may be because the classpath (clj -Spath) doesn’t seem to include any references to .gitlib And it doesn’t seem like this attempt to build the code for prod knows anything about looking in ~/.gitlib

But build.tools doesn’t seem to have any way to tell the build process where to look. AFAICT it’s meant to “just work” if the deps.edn contains a reference to the git repo in the :app clause. (Which contains all the dependencies I believe that both :dev and :prod use)

But it doesn’t. Should I even be expecting it to? And, if so, what am I likely to be missing? Where should be I looking to solve this?

Here’s one thing I have done. Given the-library its own deps.edn file. It doesn’t really have many dependencies, and it’s normally built with lein, but on the grounds that build.tools will need to build it, I gave it one.

[seancorfield]

What does your build.sh actually do?

Do you have build.clj? What’s in that?

If you’re building an uberjar with tools.build, you typically compile-clj the main ns (or all nses) and that transitively compiles everything your code references – including any git deps because they’d be on the classpath from your deps.edn and hence in the basis you would create in build.clj.

Since clj -Spath doesn’t show your library, I wonder if your deps.edn is set up correctly.

TL;DR: show us build.sh, build.clj, and deps.edn

[interstar]

Thanks.

No. To be honest I don’t have a build.clj Nor have I been using compile-clj

I’ve just been building with

clj -A:prod:app

And that was working, as long as all my dependencies were from Clojars

I take it I should be doing something else then?

My deps.edn is pretty long.

Note my-library is actually Patterning. Which you’ll see me now trying to get from github.

{:paths ["src" "resources" ],
 :aliases
 {
  :clj-kondo
  {:extra-deps {clj-kondo/clj-kondo {:mvn/version "RELEASE"}}
   :main-opts ["-m" "clj-kondo.main"]}

  :dev
  {:extra-deps
   {orchestra/orchestra #:mvn{:version "2018.12.06-2"},
    expound/expound #:mvn{:version "0.7.2"},
    nightlight/nightlight #:mvn{:version "RELEASE"},
    com.bhauman/figwheel-main #:mvn{:version "0.2.0"}},
   :main-opts ["dev.clj"]},

  :test
  {:extra-paths ["test"]
   :extra-deps
   {
    com.cognitect/test-runner
    {:git/url "https://github.com/cognitect-labs/test-runner.git"
     :sha "209b64504cb3bd3b99ecfec7937b358a879f55c1"}
    }
   :main-opts ["-m" "cognitect.test-runner"]}

  :prod
  {:extra-deps
   {leiningen/leiningen #:mvn{:version "2.9.0"},
    org.clojure/clojurescript #:mvn{:version "1.10.439"}},
   :main-opts ["prod.clj"]},

  :app
  {:extra-deps
   {
    markdown-clj/markdown-clj {:mvn/version "1.10.1"}
    instaparse/instaparse {
                :mvn/version"1.4.10"
                }
    org.clojure/core.logic {:mvn/version "0.8.11" }
    io.replikativ/hasch {:mvn/version "0.3.7"}

    remus/remus {:mvn/version "0.1.0"}

    com.alchemyislands/patterning 
         {:git/url "https://github.com/interstar/Patterning-Core.git"
          :sha "c22baf29cf33be1dd83dde3d0226b38ae96f77fa"}

    clj-rss/clj-rss {:mvn/version "0.2.5"}
    com.walmartlabs/lacinia {:mvn/version "0.36.0"}

    cljstache/cljstache {:mvn/version "2.0.6"}
    org.babashka/sci {:mvn/version "0.3.2"}

    org.clojure/core.memoize {:mvn/version "1.0.236"}

    org.clojure/data.json #:mvn{:version "0.2.6"},
    org.clojure/clojure #:mvn{:version "1.10.1"},
    reagent/reagent #:mvn{:version "0.8.0-alpha2"},
    org.clojure/tools.cli #:mvn{:version "0.3.5"},
    bidi/bidi #:mvn{:version "2.1.3"},
    com.h2database/h2 #:mvn{:version "1.4.196"},
    org.clojure/clojurescript #:mvn{:version "1.10.439"},

    hiccup/hiccup #:mvn {:version "1.0.5"}
    http-kit/http-kit #:mvn{:version "2.4.0-alpha6"},

    ring/ring #:mvn{:version "1.7.1"},
    edna/edna
    {:mvn/version "1.6.0",
     :exclusions [org.bitbucket.daveyarwood/fluid-r3]},
    com.taoensso/sente #:mvn{:version "1.11.0"},
    org.clojure/java.jdbc #:mvn{:version "0.7.3"},
    org.clojure/tools.reader #:mvn{:version "1.3.2"},
    com.rpl/specter #:mvn{:version "1.0.4"},
    cljs-react-material-ui/cljs-react-material-ui
    {:mvn/version "0.2.50",
     :exclusions
     [org.clojure/clojure
      org.clojure/clojurescript
      cljsjs/react
      cljsjs/react-dom]},
    honeysql/honeysql #:mvn{:version "0.9.1"},
    ring/ring-core #:mvn{:version "1.7.1"},
    play-cljs/play-cljs #:mvn{:version "1.3.1"},
    org.clojure/core.async #:mvn{:version "0.4.490"}}}}}

[seancorfield]

So your “build” script is prod.clj?

I guess I have to ask what’s in that now…?

You mentioned build.tools – so I assumed you meant clojure.tools.build and therefore you were using build.clj which would be the standard CLI / deps.edn way to do things.

[interstar]

Oh!

I didn’t know about that : prod.clj

I don’t even remember writing it. I guess it got created when I created the project. (I used LightMod)

(require
  '[clojure.string :as str]
  '[cljs.build.api :as api]
  '[leiningen.core.project :as p :refer [defproject]]
  '[leiningen.uberjar :refer [uberjar]]
  '[clojure.java.io :as io])

(defn read-project-clj []
  (p/ensure-dynamic-classloader)
  (-> "project.clj" load-file var-get))

(defn read-deps-edn [aliases-to-include]
  (let [{:keys [paths deps aliases]} (-> "deps.edn" slurp clojure.edn/read-string)
        deps (->> (select-keys aliases aliases-to-include)
                  vals
                  (mapcat :extra-deps)
                  (into deps)
                  (map (fn parse-coord [coord]
                         (let [[artifact info] coord
                               s (str artifact)]
                           (if-let [i (str/index-of s "$")]
                             [(symbol (subs s 0 i))
                              (assoc info :classifier (subs s (inc i)))]
                             coord))))
                  (reduce
                    (fn [deps [artifact info]]
                      (if-let [version (:mvn/version info)]
                        (conj deps
                          (transduce cat conj [artifact version]
                            (select-keys info [:exclusions :classifier])))
                        deps))
                    []))
        paths (->> (select-keys aliases aliases-to-include)
                   vals
                   (mapcat :extra-paths)
                   (into paths))]
    {:dependencies deps
     :source-paths []
     :resource-paths paths}))

(def project (-> (read-project-clj)
                 (merge (read-deps-edn [:app]))
                 p/init-project))

(defn delete-children-recursively! [f]
  (when (.isDirectory f)
    (doseq [f2 (.listFiles f)]
      (delete-children-recursively! f2)))
  (when (.exists f) (io/delete-file f)))

(def out-file "resources/clj_ts/main.js")
(def out-dir "resources/clj_ts/main.out")

(delete-children-recursively! (io/file out-dir))

(println "Building main.js")
(api/build "src" {:main          'clj-ts.client
                  :optimizations :advanced
                  :output-to     out-file
                  :output-dir    out-dir
                  :infer-externs true})

(delete-children-recursively! (io/file out-dir))

(println "Building uberjar")
(uberjar project)

(System/exit 0)

[seancorfield]

So, despite having a deps.edn, it’s actually using Leiningen under the hood to build the uberjar and Leiningen doesn’t understand git deps. So that’s your core problem.

It’s not even using clojure.tools.deps to handle deps.edn and build a classpath etc – it’s treating it as just an EDN files containing dependencies, even tho’ deps.edn looks like a CLI project. That’s pretty far out there in left field so I can only assume this is quite an old project?

My advice would be to turn this into a proper CLI project, without project.clj and without Leiningen, and replace prod.clj with a build.clj file, based on clojure.tools.build – at least then you’ll be using standard tooling and there will be documentation and community you can rely on to help you.

[interstar]

@seancorfield

clojure.tools.build

Thank you.

Right I’m taking a deep-breath. Grabbing an extra-strong cup of coffee, and sitting down to start the week by doing this : recreating the project from scratch.

Should I be following the steps in Clojure - tools.build Guide ? Would you say these the best, up-to-date instructions for how to do this?

[interstar]

What I really don’t understand about this new deps.edn way of doing things is where in the big map I’m constructing, should I put the dependencies.

I’m starting with a minimal deps.edn that looks like this

{:paths ["src"  "resources"] ;; project paths
 :deps {
    org.clojure/core.logic {:mvn/version "0.8.11" }
        }

 :aliases
 {;; Run with clj -T:build function-in-build

  :build {:deps {io.github.clojure/tools.build {:git/tag "v0.9.4" :git/sha "76b78fe"}}
          :ns-default build}
  }}

Which, at the moment, when I try to build, throws :

Execution error (FileNotFoundException) at clj-ts.logic/loading (logic.clj:1).
Could not locate clojure/core/logic__init.class, clojure/core/logic.clj or clojure/core/logic.cljc on classpath.

So I guess :deps at the top-level, alongside :paths etc. is wrong.

The old LightMod-created deps.edn had a pattern of {:aliases {:app {:extra-deps

I guess this is wrong, or was specific to that, and maybe some magic that its proj.clj performed. But where do I actually put the dependencies in deps.edn?

Clojure - tools.build Guide shows :deps within the :build clause, but I presume this is for libraries that the build process itself is dependent on, not the deps for my app itself.

So is deps.edn a file with a defined structure I should be following? Or is it a more-or-less free data-structure and I should be writing code to parse it in my build.clj?

If it does have a defined structure, where is that structure documented? Is there a page with useful examples? (Eg. how to organise it into dev vs prod builds etc?)

[interstar]

Actually, that’s weird.

I just added

    org.clojure/core.async #:mvn{:version "0.4.490"}

to the same :deps I had previously (alongside core.logic dependency) and now it seems to have moved on past that error.

Maybe this problem isn’t what I thought it was.

Possibly that top level :deps is correct, but some of the other interconnections between the dependencies I hadn’t satisfied yet are causing issues.

[interstar]

I think I’m going to close this, because the issues I now have have drifted a long way from the original topic of this thread.

[seancorfield]

At this point, you might want to join the Clojurians Slack – http://clojurians.net for self-signup – and join the #tools-build channel to get real-time, interactive help. CLI / deps.edn / tools.deps help can be found in the #tools-deps channel.

The Guide you linked to is a really good place to start. You may also find Clojure Guides: Building Projects: tools.build and the Clojure CLI (clojure-doc.org) helpful and, perhaps, the Clojure CLI - Practicalli Clojure guides.