Moving from leiningen to deps.edn

Questions & Help How to?
1

There are a lot of compelling reasons to upgrade to deps.edn from Lein – decentralized package control, lighter footprint, better design (less conflation and “do it all” swiss-army-knife work, which makes for a more difficult system to understand and extend). But there are a number of commands I rely upon in leiningen and I cannot switch until I have working replacements for them. Now, most of these are covered in the excellent https://github.com/metabase/metabase/wiki/Migrating-from-Leiningen-to-tools.deps . But Sean Corfield, a major proponent of dep.edn, has mentioned that the metabase article is out-of-date for some of the commands and that there are more recent improvements. What are some of these?

For reference, here are the lein commands that are crucial to my work:

  • lein uberjar: clojure -T:build uberjar

  • lein test: clojure -X:dev:test

  • lein clean : (clears all caches and targets; no solution listed for deps.edn)

  • lein cljs build once: (builds the cljs file, useful for checking for clojurescript errors; no solution listed for deps.edn)

  • lein deploy: (not listed; this may not be relevant once using deps.edn, but possible it will still be desirable to make clojars artifacts)

    While there are others that I use, these are the first to mind as crucial. Are there improvements to the metabase recommendations for these?

1 Like
2
  • lein clean - depends on what you need exactly. When not using Lein, I have never had a need for such a command. If some specific cache is suspect, I just remove that particular cache without nuking everything
  • lein cljs build once - depends on your build tool. With shadow-cljs, I’d use npx shadow-cljs compile main (assuming :main is the build that you want). And that can be integrated into deps.edn since you can run shadow-cljs with clj as well, it’s just that I myself don’t do it
3

we generally make fullstack projects, in which we use lein clean to occasionally fix the problems somewhere in the stack, particularly in more opaque systems like the handlers and middleware, or for dealing with bad versions of the front-end.

Since we ore doing Full Stack we aren’t using Shadow for these projects, although I have very much enjoyed it on other standalone projects.

4

And that’s exactly my point. If you’re dealing with a bad NPM configuration, you just have to nuke node_modules and not the whole target along with classes and whatever else might be in the list of caches/artifacts that you have. If you’re dealing with a weird backend issue, it makes sense to clean the backend-specific caches to avoid having to rebuild the whole CLJS bundle just because you happen to have a rogue .class somewhere.

Since we ore doing Full Stack we aren’t using Shadow for these projects

I fail to see the causation. I am a full stack developer, all my projects are full stack, and I always use shadow-cljs.

5

lein deploy → clojure -T:build deploy with an appropriate deploy function in build.clj. See the next.jdbc example linked below and also the Automated Deployments section of the cookbook linked below.

As for lein clean, it really depends on why you need it. In theory, you should never need it – but primarily it just deletes the target folder and some lein-specific cache folders/data. You could add a clean function to your build.clj (that calls clojure.tools.build.api/delete with {:path "target"}).

You will find existing deps.edn and, especially, build.clj files in open source projects to be useful reading, e.g., next-jdbc/build.clj at develop · seancorfield/next-jdbc (github.com)

This cookbook will also probably help you: Clojure Guides: Building Projects: tools.build and the Clojure CLI (clojure-doc.org)

I think part of some people’s difficulty in migrating from Leiningen to deps.edn and build.clj is that they don’t really understand what the various lein tasks and plugins actually do under the hood – which I think is part of the problem with using such things in the first place :frowning:

3 Likes
6

We use lein clean to debug mysterious errors, which usually have to do with altered package versions (especially transitive dependencies). But that might itself be a leiningen-specific issue.

Yeah, you are right about lein ignorance – lein does lots of magic because it does SO MUCH, so devs like me get afraid to rock the boat.

7

It’s a “learning opportunity”! :slight_smile:

8

I get where OP is coming from. I really wish there was a cleaner offramp toward the emergent blessed solution (borkdude is trying, and you have some prior art that has filled gaps).

It’s a “learning opportunity”! :slight_smile:

I think that’s how they advertised the Great Leap Forward too :slight_smile:

I think part of some people’s difficulty in migrating from Leiningen to deps.edn and build.clj is that they don’t really understand what the various lein tasks and plugins actually do under the hood – which I think is part of the problem with using such things in the first place

For a decade I didn’t need to though. I still don’t if I stick with lein (for me at least around 99% of the time things just work; I confess to occasional debugging sessions in years past, but those days are mostly forgotten). lein and plugins did the lifting for me so I could focus on more meaningful problems.

Now under the (sole) approved tooling, I get the sausage making process exposed but I get to (or have to) make my own sausage directly so I should be grateful for the flexibility afforded. I can leverage third party tools to abstract some of the primitive sausage making parts away, which reverts toward delegating and not understanding what’s happening though, and places me closer to where I started with lein plugins and tasks in some respect.

I have yet to manifest the purported leverage provided by this alternate approach though (after about 4 years of dabbling); maybe in the coming year it will pay off with projects intentionally doing more esoteric and complicated builds.

2 Likes
9

lein works for me.

for people that have actually experienced the great leap forward, they don’t tend to speak out.

we should be thankful that we can.

10

The best part of deps is that it picks the newest version when there are dependency conflicts.

Otherwise I get that it lacks default tasks. That’s because it’s not a build tool. The build tool is. Called tools.build, and that’s more like a build library that makes it easier to implement your own tasks.

Think of tools.deps only as dependency manager/resolution. Paired with the Clojure CLI as an application launcher. And if you need build tasks, you can implement them yourself, and optionally use tools.build to help with the implementation.

Still, I reckon there’s two annoyances:

  1. Default tasks that works 99% of the time are still nice to have
  2. The Clojure CLI command invocations are hard to remember, cryptic and slow to type

The solution I’ve been using for those are two folds:

For problem #1

I recommend using Neil : GitHub - babashka/neil: A CLI to add common aliases and features to deps.edn-based projects

It’s not complete, it’s missing default tasks. But it’s a nice start to help.

For problem #2

I use Babashka and it’s BB tasks. This article talks about it near the ends: New Clojure project quickstart

This pretty much solves it, but you do have to copy/paste it to each project.

1 Like
11

Yeah, I’m familiar with all that, which is why I mentioned borkdude’s work in passing earlier. I am pretty sure we have had this conversation here or on reddit at some point (at least it seems like deja vu).

Neil is lein adjacent, and introduces a new (wider) task vocabulary to the api. It may also be superior; I haven’t used it in anger yet. Seems to be going in the right direction though.

I could envision a better bridging solution (or at least one conformant to legacy paths) that enables all of the good and none of the bad:

  • leverage deps.clj as an embedded dependency resolver [to enable what you see as an objectively better resolution scheme, instead of the current resolver],
  • map the legacy lein :dependencies vector onto a comparable deps.edn :dependencies map (trivial) to be fed transparently to deps.clj (keeps backwards compatibility with legacy projects),
  • extend the allowed format for project.clj’s :dependencies map to leverage the specs from deps.clj, so that deps-style dependencies work going forward
  • allow (as lein-tools-deps did) using a local deps.edn so we can accrete dependencies under the new blessed way
  • pull in tools.build support as a task and just delegate to it.
  • work in aliases maybe; there’s conflation with profiles so maybe ignore them initially.

Then you retain lein tasks, declarativeness where users want it, and opt in to the dependency resolution and build pieces. Eventually, if it’s useful enough, users have a migration path away from lein instead of the current wall (or cobbling multiple tools as you did).

I don’t know what the userbase for something like this would be (or if there are non-obvious infeasibilities there), but based on the last survey from 2023 there’s a decent overlap of people still on lein (I saw 60% for lein and 70% for tools.deps/cli/tools.build etc, guessing they allowed multiple selections since the totals exceed unity).

2 Likes
12

Ya, but honestly, I would just do a clean slate. Rebuilding lein tasks on tools.deps with tools.build is very easy. It’ll result in something much simpler.

All you need is someone to put a library over tools.build that comes with either its own config or just allows you to add some key/vals to deps.edn to configure it.

The problem is that, probably, tools.deps/build is easy enough to get the tasks you need setup, that no one is truly motivated to go and build what I’m describing.

Generally everything is a matter of ROI. Slightly more inconvenient setup is what tools.deps/build gives you. And that might not be enough to motivate someone to write a more convenient tool on top of it.

When Lein came about, it was very inconvenient to use Ant and Maven and all that. So the motivation and ROI of doing so was much greater.

2 Likes
13

Perhaps. It certainly excels for more complex setups. When we switched from lein to boot in 2015, it was because we’d “outgrown” what Leiningen could do out of the box and were already writing plugins but the plugin architecture was limiting (and added unnecessary complexity, IMO) so we also started writing ad hoc scripts (some Clojure, but mostly shell).

When we switched to Boot, we were able to do more in Clojure, writing “tasks” on top of Boot’s filesystem abstraction and running arbitrary processes in Boot’s “pods”. It was much simpler for us to build everything we needed in Clojure, in a more natural way than with Leiningen. But we started to run into bugs and limitations with Boot too: bugs in the pod system, a noticeable overhead in the filesystem abstraction (and sometimes it confused editor tooling on where a source file really was).

So we were ready to try something new in 2018, when the official CLI launched with deps.edn. We’d already decided to move our dependencies into EDN files with Boot (Boot was programmatic so it was simple to add support for declarative dependencies – going the other way is not so simple: adding programmatic support to a declarative system). Our Boot tasks became a number of small scripts that could be run with aliases. The conversion took about a week.

The only thing we missed was the orchestration of tasks so we wrote a fairly simple build.sh that would invoke the CLI multiple times in sequence to perform a series of tasks.

When tools.build and build.clj appeared, we were able to move that orchestration into Clojure in build.clj. Now we had all the benefits of Boot with none of the downsides.

We have over 170 subprojects, and over 20 buildable articles, each with their own deps.edn file (nearly 200 deps.edn files!). Our top-level deps.edn file is almost 600 lines and our build.clj is nearly 400 lines. Our CI/CD pipeline is some setup in shell and then a few clojure -T:build lines.

I think what I like most about working with the CLI, deps.edn, and build.clj is that is works the same way at any scale: create a bare bones app or lib project with deps-new (or clj-new) and you have a solid basic setup to run tests, build uberjar or JAR files, and deploy the latter, with an inherent CI-like task to build test+JAR – and that can easily be expanded to test against multiple versions of Clojure, run linters, CVE checkers, documentation testers; it can be expanded to do all of that across multiple subprojects, producing a variety of artifacts; it can run ad hoc commands and Java/Clojure processes; it can manage your git versioning and releasing process; and so on. It’s “all just Clojure code”.

2 Likes
14

I just migrated the Clojars build from lein to deps.edn if that is helpful to use as a reference. It’s a pretty simple build since it is only Clojure, no Clojurescript.

My primary motivation was to have better dependency management; the old project.clj had many exclusions and direct dependencies on things brought in transitively, mainly to address CVEs. The new build uses :override-deps to manage CVE overrides instead.

I used make to remove the need to remember all the various clojure invocations, and to have a single place for all build/dev tasks.

2 Likes
15

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.