How to Effectively Use Deps & CLI?

Hi,
I’d like to leverage Deps & CLI moving forward for Clojure/ClojureScript development. While the official Deps & CLI guide was helpful I don’t feel I quite have as firm a grasp on it as I’d like. Does anyone know of follow-up guides and/or tutorials that delve further into (recommended) usage patterns/practices? I’m particularly curious how Deps & CLI may be used in ClojureScript related work. Thanks.

There is also the official reference documentation, which is different than the guide you linked: https://clojure.org/reference/deps_and_cli

I do not have follow-up guides, but I do have links to two deps.edn file that may interest you, as examples.

The first is Sean Corfield’s, which he often links to as an example of how to do many things. Sean does not yet use ClojureScript, so you won’t yet find any ClojureScript-specific stuff in there: https://github.com/seancorfield/dot-clojure/blob/master/deps.edn

I have recently been making changes to a fork of the core.rrb-vector library that supports both Clojure/Java and ClojureScript, but has no browser-specific code in it anywhere, so all REPL and tests are run only on Node.js. Still, you may find it useful as a working example: https://github.com/jafingerhut/core.rrb-vector/blob/proposed-fixes-for-4-issues/deps.edn I am fairly new to deps.edn and ClojureScript myself, so am open to suggestions on making that better.

There are a few other links I have collected on this page: https://github.com/jafingerhut/jafingerhut.github.com/blob/master/notes/clojure-development.md

1 Like

Hey Ari, welcome!

One thing to understand is that tools.deps and the Clojure CLI are not build tools.

tools.deps is a dependency manager and classpath generator. While Clojure CLI is a command line launcher for Clojure app which uses tools.deps to pull in dependencies and generate the required app classpath.

More simply speaking, you write some Clojure;

hello_world.clj

(ns hello-world)

(defn -main [& _]
  (println "Hello World"))

Now you want to run it? Not so easy. You would need to do the following:

  1. Manually download the Clojure Jar and save it next to your hello_world.clj file
  2. Manually download the Clojure spec Jar and save it next to your hello_world.clj
  3. Manually download the Clojure core spec Jar and save it next to your hello_world.clj
  4. Use java, the clojure, spec and core spec Jars in combination to run your code:
java -cp .:./clojure-1.10.1.jar:./spec.alpha-0.2.176.jar:./core.specs.alpha-0.2.44.jar clojure.main -m hello-world

We don’t want to do all that. So the core team has released two tools to make this easier: tools.deps and Clojure CLI.

With the Clojure CLI, we can do:

clj -Scp .:./clojure-1.10.1.jar:./spec.alpha-0.2.176.jar:./core.specs.alpha-0.2.44.jar -m hello-world

That’s not really better then when straight up using java. We only avoid having to call to clojure.main specifically, but we still need to download all Jars ourselves and put together the path to all of them and our code manually.

Luckily for us, it comes bundled with tools.deps. So now instead of manually downloading the three Jars and putting the classpath together ourselves we can simply create a file called deps.edn where we just declare the dependencies we want, and let the Clojure CLI download them for us, save them in a folder locally, and create the classpath to them:

deps.edn

{:paths ["."]
 :deps {org.clojure/clojure {:mvn/version "1.10.1"}}}

And now we can run it by simply doing:

clj -m hello-world

:paths is the path to our code we want added to the classpath, and :deps is the list of dependencies we want tools.deps to automatically download and add to our classpath for us.

So the steps become:

  1. Create a deps.edn file and specify the path to your code and the depencies it needs to run.
  2. Run clj -m your-main-namespace

That’s it! This is literally the only point of the Clojure CLI and tools.deps. To download the dependencies needed by a Clojure application, create the classpath to them, and launch the application.

You can create named groups of dependency sets and main namespaces called Alias as a convenience, and so you can then use the same deps.edn file to launch different Clojure app, by specifying which alias to run. Or launch the same Clojure app but with different settings, dependencies, entry point, etc.

Now Leiningen and Boot were tools which could also do all this, though using a slightly different mechanism.

What the Clojure CLI and tools.deps is not though, is everything else Leiningen and Boot can do, that is, it is not a build tool.

What does a build tool do? Well, it can run arbitrary tasks and create arbitrary chains of them. Normally, you use them to say compile your code ahead of time. Package your code as a library Jar, or full app Uberjar. Run a linter over your code. Run a test runner to check your tests against your code. Etc.

What the hell is a “task” though? In the abstract, that’s any piece of code. But to a build tool, a “task” is a way to bundle some arbitrary piece of code so that the build tool can execute it, give it some context, read its result, pass that result to the next task, etc.

Now when you think about it, the operating system already has exactly that, its called a process, and you can pipe input and output between them, and chain them. So what Clojure CLI and tools.deps do, is that if you build your “tasks” as Clojure programs, each task is a program which possibly takes input and returns output. And since Clojure CLI is a launcher for Clojure programs, well you got yourself a build tool! How cool is that!

Now Lein and Boot come with a bunch of useful common build related tasks, and Clojure CLI + tools.deps does not. But the community has started building tasks for it, which remember, are just normal Clojure apps.

Here’s a good list of them: https://github.com/clojure/tools.deps.alpha/wiki/Tools

The trick is you just create an alias for each one that you want, such as what @seancorfield does here: https://github.com/seancorfield/dot-clojure/blob/master/deps.edn

The only thing is that tools.deps and the Clojure CLI don’t allow a way to chain things together. So for now, you can only run one task at a time. So if you want to run multiple tasks. Like you can’t create a task of tasks. Also each one runs inside its own JVM, so they take longer to start, if you need to do a few of them back to back.

Regards!

10 Likes

In my opinion you can finished most of the work with tool like shadow-cljs with ClojureScript, why still using CLI tools.

Check the extensive docs for Figwheel Main. Here’s an example repo.

Thank you for this answer. I’ve been half-heartedly learning Clojure for a couple of years, using emacs, CIDER and Leiningen and I decided to try ClojureScript, so went back to the docs to discover the appropriate tooling. I was a bit shocked to discover Deps & CLI. I’ve come to clojureverse just to find out what is going on. :slight_smile: I also found that NREPL had been pulled out of Clojure into a separate repository, after looking at the one that was no longer being updated for a while.

It is obvious that the new tools are an easier starting point for a beginner but I’m still unclear what the path forward is. Will they eventually replace Leiningen? Will Leiningen adopt the deps.edn dependency format? Do we need duplicate dependency definitions for different tools? :-/
I guess I’m just looking for reassurance that someone is planning a route to reduce complexity of Clojure tooling rather than add more. I wrote this blog post of what I’ve discovered alone, earlier today https://andywootton.wordpress.com/2019/09/19/tooling-for-clojure-and-clojurescript/

I’d say this is the most likely outcome. Same for Boot. It would be the one I’d get behind as well.

There’s already plugins for both Boot and Lein to do so:

At the same time, there’s so much legacy use of lein and boot as dependency managers as well, that I don’t see their support for dependency management going away either.

1 Like

I’ll just note that boot-tools-deps is archived/deprecated (for reasons listed in the README).

At World Singles Networks, we started with Leiningen in 2011 and switched to Boot in 2015 as we needed easier “programmability” for custom tasks than Leiningen’s plugin system. Leiningen also didn’t seem to be a good fit for the monorepo approach we were trying to adopt.

We switched completely to CLI/deps.edn last year and we’ve been very happy with the simpler tool chain. Custom tasks – the reason we had switched to Boot – became just simple Clojure scripts that we ran with the CLI. We added a simple build shell script that could run multiple CLI commands in a “pipeline” (so one simple command line could be used to run a sequence of CLI command lines).

Was the motivation for this to incur startup cost once?

No, because it doesn’t address that.

clojure (clj) only executes at most one -main for each command you issue – so our build shell script is just shorthand for running multiple commands. For example:

build test api uberjar api deploy api

runs the following:

cd <path>/api
clojure -A:test:runner
clojure -A:uberjar api.jar
clojure -A:deploy api.jar <user>@<server>

So it knows to switch the subproject folder (we have a monorepo with about thirty subprojects) and it knows about certain additional command line arguments and aliases that are needed for the various shortcut commands.

2 Likes

I’m struggling with how to use deps in a monorepo. I know that I can force a repo managed top level deps file to replace the one in the user’s home directory but that feels wrong to me. I feel like there are a couple of problems that need to be solved for a mono-repo

  1. keeping paths to repo-level dependencies in sync (/apps/app1 depends on /libs/lib2). It’s troublesome to keep relative paths updated in multiple places. Putting these all in a top level deps file means I have to always run commands from the repo root.

  2. Running commands over multiple targets. This is something that lein-monrepo does well and can be managed with a bash script but that implies keeping the list of projects in bash instead of in the deps file

  3. Being able to chain together plugins. It’s currently possible to pass a complete deps file to clj, so you could generate a file and use xargs to pass that to another clj command but there doesn’t seem to be a way to generate a deps file and say “this is my user config, use it instead of whats in ~/.clojure/deps.edn”. If this were possible you could imagine a plugin that could resolve relative paths to local dependencies, set versions for globally managed dependencies and then run a repl or uberjar or whatever plugin.

I suppose you could write the deps to /tmp and then use that but it seems like a hack.

Rick

We use deps in a monorepo with about thirty subprojects and about 90K lines of code.

We have all the subprojects at the same relative level: <root>/clojure/<subproject>

There’s a master deps.edn in <root>/versions – and we use the CLJ_CONFIG env var to tell clojure about that. The build script described above does the following:

cd <root>/clojure/<subproject>
CLJ_CONFIG=../versions clojure -A:defaults:<other>:<aliases> <and> <arguments>

(I omitted the CLJ_CONFIG part before since it wasn’t relevant to the original thread – but it is relevant to your question about a monorepo)

How do you handle relative vs. absolute paths to other subprojects?

A typical subproject deps.edn file looks like:

{:deps
 {worldsingles {:local/root "../worldsingles"}
  worldsingles/lowlevel {:local/root "../lowlevel"}
  worldsingles/newrelic {:local/root "../newrelic"}
  cheshire {}
  clojure.java-time {}}}

The actual versions of non-local dependencies come from ../versions/deps.edn which has:

 :aliases
 {:defaults
  {:override-deps
   {cheshire {:mvn/version "5.8.1"}
    clojure.java-time {:mvn/version "0.3.2"}
    ,,,}}
   ,,,}

As I said above “all the subprojects [are] at the same relative level” which is key to making the relative paths work.The build shell script only has to know the absolute path of the repo root (which can be a convention or env var or whatever). Everything else is always relative.

Got it. And now I understand how to use override/default-deps with an empty map in the subproject :smile:

Right now our mono-repo isn’t very well organized for our clojure projects. This might be a good opportunity for some housekeeping.

I still feel like it would be great to be able to run a pre-processing step that generates deps to use in addition to the current directory, user’s home and the system deps files. Correct me if I’m wrong, but there’s no way with this setup for a user to define their own configurations.

Rick

The problem with including user deps.edn into the monorepo projects is that you may pick up unexpected versions of libraries which is bad from a reproducibility p.o.v. You really want “the project” to control what dependencies and aliases can go into anything you run/test/build.

What we’ve done at World Singles Networks is to agree as a team on what tooling is sufficiently useful for both “every developer” and for “any developer” and added standardized versions of those into our versions/deps.edn file, under a variety of aliases that support the team’s choices of editors etc.

We’ve also added a REPL-starter script (ws.dev.repl) under a couple of aliases with different dependencies and its -main function starts a combination of the following, depending on what libraries it finds on the classpath:

  • nREPL
  • Socket REPL
  • Cognitect REBL

In addition, we have another dev script that reads and merges :deps, :paths, :extra-deps, and :extra-paths (from :test-deps aliases) from all of the subproject deps.edn files into everything/deps.edn as a convenient way to start tooling with ALL monorepo dependencies in place, so we can start up a single REPL (of whatever flavor we prefer) and connect to it from our preferred editor/IDE/tooling and have the entire 90K line code base accessible during interactive dev/test.

1 Like

Thanks, these are all great ideas.