Making a Clojure / ClojureScript Project (In Oct 2024) : Round 2

Community Center Beginners
1

So, I’ve spent a few more hours this week wrestling with Making a Clojure / ClojureScript Project (Oct 2024 Edition)

@thheller’s guide to shadow was helpful but seemed to be taking me away from the core of my problem. So I went back to read more tutorials on the build tools and to try to fix the build I had half working.

Which was mainly about removing the vestiges of leiningen from the build setup I had from LightMod.

Here’s where I am

This is my deps.edn



{:paths ["src" "resources"],
 :aliases

 {
  :eastwood
      {:main-opts ["-m" "eastwood.lint" {:source-paths ["src"]}]
       :extra-deps {jonase/eastwood {:mvn/version "RELEASE"}}}


  :clj-kondo
  {:extra-deps {clj-kondo/clj-kondo {:mvn/version "RELEASE"}}
   :main-opts ["-m" "clj-kondo.main"]}



  :dev
  {:extra-deps
   {orchestra/orchestra #:mvn{:version "2018.12.06-2"},
    expound/expound #:mvn{:version "0.7.2"},
    nightlight/nightlight #:mvn{:version "RELEASE"},
    com.bhauman/figwheel-main #:mvn{:version "0.2.0"}},
   :main-opts ["dev.clj"]},



  :test
  {:extra-paths ["test"]
   :extra-deps
   {
    com.cognitect/test-runner
    {:git/url "https://github.com/cognitect-labs/test-runner.git"
     :sha "209b64504cb3bd3b99ecfec7937b358a879f55c1"}
    }
   :main-opts ["-m" "cognitect.test-runner"]}

  :build
  {
   :deps
   {
    io.github.clojure/tools.build {:git/tag "v0.8.1" :git/sha "7d40500"}
    org.clojure/clojurescript #:mvn{:version "1.10.439"}
    }
   :extra-paths ["src" ]
   :ns-default build
   }

  :prod
  {:extra-deps
   {leiningen/leiningen #:mvn{:version "2.9.0"},
    org.clojure/clojurescript #:mvn{:version "1.10.439"}},
   :main-opts ["prod.clj"]},

  :app
  {:extra-paths ["resources" "src/clj_ts"]
   :extra-deps
   {
    markdown-clj/markdown-clj {:mvn/version "1.10.1"}
    instaparse/instaparse {
                :mvn/version"1.4.10"
                }
    org.clojure/core.logic {:mvn/version "0.8.11" }
    io.replikativ/hasch {:mvn/version "0.3.7"}

    remus/remus {:mvn/version "0.1.0"}

    com.alchemyislands/patterning {:mvn/version "0.5.6-SNAPSHOT"}
    

    clj-rss/clj-rss {:mvn/version "0.2.5"}
    com.walmartlabs/lacinia {:mvn/version "0.36.0"}

    cljstache/cljstache {:mvn/version "2.0.6"}
    org.babashka/sci {:mvn/version "0.3.2"}

    org.clojure/core.memoize {:mvn/version "1.0.236"}

    org.clojure/data.json #:mvn{:version "0.2.6"},
    org.clojure/clojure #:mvn{:version "1.10.1"},
    reagent/reagent #:mvn{:version "0.8.0-alpha2"},
    org.clojure/tools.cli #:mvn{:version "0.3.5"},
    bidi/bidi #:mvn{:version "2.1.3"},
    com.h2database/h2 #:mvn{:version "1.4.196"},
    org.clojure/clojurescript #:mvn{:version "1.10.439"},


    hiccup/hiccup #:mvn {:version "1.0.5"}



                                        ;http-kit #:mvn{:version "2.3.0"},
    http-kit/http-kit #:mvn{:version "2.4.0-alpha6"},

    ring/ring #:mvn{:version "1.7.1"},
    edna/edna
    {:mvn/version "1.6.0",
     :exclusions [org.bitbucket.daveyarwood/fluid-r3]},
    com.taoensso/sente #:mvn{:version "1.11.0"},
    org.clojure/java.jdbc #:mvn{:version "0.7.3"},
    org.clojure/tools.reader #:mvn{:version "1.3.2"},
    com.rpl/specter #:mvn{:version "1.0.4"},
    cljs-react-material-ui/cljs-react-material-ui
    {:mvn/version "0.2.50",
     :exclusions
     [org.clojure/clojure
      org.clojure/clojurescript
      cljsjs/react
      cljsjs/react-dom]},
    honeysql/honeysql #:mvn{:version "0.9.1"},
    ring/ring-core #:mvn{:version "1.7.1"},
    play-cljs/play-cljs #:mvn{:version "1.3.1"},
    org.clojure/core.async #:mvn{:version "0.4.490"}}}}}

There’s some cruft that I don’t really need but I think it should be sufficient for understanding THIS question.

Here’s my new build.clj

(ns build
  (:require [clojure.tools.build.api :as b]
            [cljs.build.api :as cljs-api])) 

(def build-folder "target")

(def jar-content (str build-folder "/classes"))

(def basis (b/create-basis {:project "deps.edn" :aliases [:app]}))
(def version "0.9.0")
(def app-name "cardiganbay")
(def uber-file-name (format "%s/%s-%s-standalone.jar" build-folder app-name version))


(defn clean [_]
  (b/delete {:path build-folder})
  (println (format "Build folder \"%s\" removed" build-folder)))

(defn uber [_]
  (clean nil)
  (b/copy-dir {:src-dirs ["resources"]
               :target-dir jar-content})

  (println "Compiling Clojure Backend")
  (b/compile-clj {:basis     basis    
                  :src-dirs  ["src"]
                  :class-dir jar-content})
  
  (println "Compiling ClojureScript Client")
  
  (cljs-api/build
   "src"
   {:main 'clj-ts.client
    :src-dirs ["src/clj_ts"]
    :optimizations :none
    :output-to "resources/clj_ts/main.js"
    :output-dir "resources/clj_ts/main.out"
    :infer-externs true}
   )

  (println "Making Uberjar")
  (b/uber {:class-dir jar-content 
           :uber-file uber-file-name
           :basis     basis
           :main      'clj-ts.server})

  (println "Finished creating uber file")
  )

I’ve figured out how to invoke it properly (AFAICT)

clj -T:build uber

Now it seems to run fine building the clojure. And it would package the uberjar.

Where it’s failing is building the clojurescript

Compiling ClojureScript Client
WARNING: as-map at line 8 is being replaced at line 41 src/clj_ts/types.cljc
WARNING: report at line 8 is being replaced at line 41 src/clj_ts/types.cljc
Execution error (ExceptionInfo) at cljs.analyzer/error (analyzer.cljc:718).
No such namespace: instaparse.core, could not locate instaparse/core.cljs, instaparse/core.cljc, or JavaScript source providing "instaparse.core" in file src/clj_ts/command_line.cljc

Now, as far I can tell, the problem is simply, how do I get the list of dependencies into the clojurescript compiler?

The :build alias in the deps.edn doesn’t have them. But AFAICT I don’t need to have them there, because in the build.clj script I assemble that list using the

(def basis (b/create-basis {:project "deps.edn" :aliases [:app]}))

That successfully pulls the the list of dependencies from the :app alias into the basis data-structure. And the clojure compilation, for the server, does, indeed, see those dependencies.

However, the build function of cljs.build.api (cljs.build.api/build) doesn’t seem to take anything like basis as an argument.

(It’s hard to see because I can’t find any information about the list of “opts” it wants)

In fact although I can see lots of functions with the word “dependency” in their name in the cljs.build.api , I can’t really figure out how I would tell the compiler about dependencies when I invoke it.

So can anyone explain that? Or show me what’s missing to get the cljs compiler to compile with the appropriate dependencies?

Now, I know someone is going to suggest that I just shouldn’t use cljs.build.api and should use shadow or figwheel etc . Which may be good advice. But I’d like to point out that this rather kludgy build that came from LightMod was working for years to compile the cljs part of my project.


(require
  '[clojure.string :as str]
  '[clojure.pprint :as pp]
  '[cljs.build.api :as api]
  '[leiningen.core.project :as p :refer [defproject]]
  '[leiningen.uberjar :refer [uberjar]]
  '[clojure.java.io :as io])

(defn read-project-clj []
  (p/ensure-dynamic-classloader)
  (-> "project.clj" load-file var-get))

(defn read-deps-edn [aliases-to-include]
  (let [{:keys [paths deps aliases]} (-> "deps.edn" slurp clojure.edn/read-string)
        d0 (println "In read-deps-edn \nPATHS:: " paths " \nDEPS:: " deps "\nALIASES:: " aliases)
        deps (->> (select-keys aliases aliases-to-include)
                  vals
                  (mapcat :extra-deps)
                  (into deps)
                  (map (fn parse-coord [coord]
                         (let [[artifact info] coord
                               s (str artifact)]
                           (if-let [i (str/index-of s "$")]
                             [(symbol (subs s 0 i))
                              (assoc info :classifier (subs s (inc i)))]
                             coord))))
                  (reduce
                   (fn [deps [artifact info]]
                     (if-let [version (:mvn/version info)]
                       (conj deps
                             (transduce cat conj [artifact version]
                                        (select-keys info [:exclusions :classifier])))
                       deps))
                   []))
        d2 (println "NEW DEPS:: " deps)
        paths (->> (select-keys aliases aliases-to-include)
                   vals
                   (mapcat :extra-paths)
                   (into paths))
        d3 (println "NEW PATHS:: " paths )
        ]
    {:dependencies deps
     :source-paths []
     :resource-paths paths}))

(def project (-> (read-project-clj)
                 (merge (read-deps-edn [:app]))
                 p/init-project))

(println "Project is")
(pp/pprint project)

(defn delete-children-recursively! [f]
  (when (.isDirectory f)
    (doseq [f2 (.listFiles f)]
      (delete-children-recursively! f2)))
  (when (.exists f) (io/delete-file f)))

(def out-file "resources/clj_ts/main.js")
(def out-dir "resources/clj_ts/main.out")

(delete-children-recursively! (io/file out-dir))

(println "Building main.js")
(api/build "src" {:main          'clj-ts.client
                  :optimizations :advanced
                  :output-to     out-file
                  :output-dir    out-dir
                  :infer-externs true})

(delete-children-recursively! (io/file out-dir))

(println "Building uberjar")
(uberjar project)

(System/exit 0)

It’s broken now. For other reasons. And I want to move away from it, to a simpler, easier to understand, build script that doesn’t have the leiningen dependency.

Nevertheless, AFAICT, it’s doing exactly the same thing to build the cljs. So if the way I’m trying to build the cljs doesn’t work, why did that work for so long? Was it doing some other mysterious necromancy that I just haven’t grokked? Is it some magic to do with the leiningen dependencies? (I can’t see why it should be.)

Anyone who can shed any light on any of this? Or at least just explain how in my new build.clj I am meant to tell the cljs.build.api/build function which dependencies it ought to include when compiling my clojurescript?

2

tools.build is a set of functions to build Clojure code. Not ClojureScript.

cljs.build.api is entirely standalone and has absolutely nothing to do with tools.build. It is therefore unaware of what a “basis” is. The only thing it is concerned with is the available classpath when the API function is called.

The classpath is contructed by the command you run, so clj -T:build uber. That basically activates the :build alias, but not the :app alias where your dependency is specified.

In general CLJS, meaning cljs.build.api, figwheel and shadow-cljs, do not mix well into the the tools.build setup. It is generally advisable to either run them as separate steps before building the CLJ parts. Or using tools.build to run a new JVM instance and running that command for you, via java-command. Generally it is much easier to just to npx shadow-cljs release app && clj -T:build uber or using a Makefile or just a shell script.

Of course nothing stops you from making all CLJS related dependencies available when invoking the build task, but that is not the general idea behind tools.build.

1 Like
3

Thank you @thheller

So this ought to work then?

clj -T:build:app uber

(I mean, it seems not to be throwing the same error.)

This may be a stupid (or irritating, I’m sorry) question. But if cljs.build.api has nothing to do with tools.build, why does it rely on a classpath sent as an argument to a tools.build script, for the dependencies? Why aren’t these an explicit argument to the cljs.build.api/build function? In the same way that the compile-clj function takes the “basis” as an explicit list of dependencies?

But I guess, is the take-home message here - for slow people like me, and maybe all the other n00bs trying to get this stuff to work - that actually Clojure and ClojureScript are now two entirely distinct communities and projects who aren’t really coordinating or trying to make their things work together? Or give a consistent build experience across the two languages?

4

That is nonsense.

As already stated: The focus of tools.build is building Clojure projects. The focus of shadow-cljs, figwheel or CLJS directly is to build ClojureScript projects. Their requirements differ, so they work differently. Does there need to be one tool that does everything? I don’t think so.

Every project will have different requirements and there are many options. I, and others, already advised to use shadow-cljs for CLJS, but you seem set to go your own way. Which is fine, but then things might be a bit more complicated since that is not the common path.

clj is a tool to start Clojure programs. It provides a way to declare dependencies and making them available to the runtime. It does nothing else.

tools.build is a library, providing a couple helpful functions when build Clojure programs and nothing beyond that.

The :build alias in deps.edn basically just declares the dependency on the tools.build library and runs your build.clj code. It is just Clojure code, there is no “tools.build tool”. If you want to do “more” stuff than what the tools.build library provides, then that needs to come from somewhere.

cljs.build.api or even shadow-cljs are also just Clojure code that you need to run in some way

So, one option is using the aliases to make sure all required dependencies are available when the build.clj file runs.

I already linked the java-command alternate option, which does take a :basis argument and would let you invoke cljs.main, figwheel or shadow-cljs by starting a new JVM with its own classpath. This is also what the compile-clj function uses under the hood. So no difference there. I have never used it, so I cannot give you the exact command, but I’m 100% confident it will work. Heck, you could also just run the shell command via a tools.build process function.

I already made the recommendation to just treat these things separately. Much less complicated and much better documented. An uberjar is nothing but a zip archive full of files. There is no rule that says that all files going into that zip must come from the same JVM process.

5

I got curious and invested some time to figure out how to do this with tools.build.

Basically what I’d recommend is this. You create the below cljs-release function, and call it in the uber function, or separately if you want.

(defn cljs-release [& args]
  (let [basis
        (b/create-basis
          {:project "deps.edn"
           :aliases [:cljs]})

        res
        (-> (b/java-command
              {:basis basis
               :main 'clojure.main
               :main-args ["-m" "shadow.cljs.devtools.cli" "release" "frontend"]})
            (b/process))]

    (when-not (zero? (:exit res))
      (throw (ex-info "CLJS compilation failed!" {})))))

Basically this grabs the new basis with the :cljs alias activated as an example here. Could be adjusted when needed. Then it creates the needed java-command and executes it via process. Last it checks if the compilation fails, because shadow-cljs will return a non-zero exit code on errors. shadow-cljs will already have printed what went wrong, so just using the throw to terminate any further action, in case it is running as part of uber.

Then it is just clj -T:build uber without the need to add the extra aliases.

In essence this runs the same as running this command line directly.

clj -M:cljs -m shadow.cljs.devtools.cli release frontend

(Which would be the command used in reference to my prior blog post, frontend is the build id here, change accordingly if yours is different)

2 Likes
6

@thheller thank you for your blog posts about the basics of Clojure(script). We daily make implicit use of those facts, but the authoritative character of your post makes them findable in the spirit of FAIROS (findable, accessible, interoperable, reusable open source).

2 Likes