Difference between deps.edn and leiningen wrt repositories

[pieterbreed]

I’m a long-time user of leiningen and going through the readjustment required to convert to deps.edn workflow. I’ve come across an unexpected difference between the two systems.

I’m doing a lot of confluent/kafka work. Confluent provides their own JVM libraries within a custom repository. In a lein project, I can do this:

(defproject blabla "0.1.0-SNAPSHOT"
  :repositories [["confluent" "https://packages.confluent.io/maven/"]]
  :dependencies [[org.clojure/clojure "1.10.3"]

                 [io.confluent/kafka-json-serializer "7.0.1"]
                 [io.confluent/kafka-json-schema-serializer "7.0.1"]
                 [io.confluent.ksql/ksqldb-api-client "7.0.1"]
                 [io.confluent/confluent-log4j "1.2.17-cp2"]
                 ])

With a clean ~/.m2/repository I can successfully start the REPL, do lein deps etc…

The most similar deps.edn config I can come up with is this:

{:deps {io.confluent/kafka-json-serializer        {:mvn/version "7.0.1"}
        io.confluent/kafka-json-schema-serializer {:mvn/version "7.0.1"}
        io.confluent.ksql/ksqldb-api-client       {:mvn/version "7.0.1"}
        io.confluent/confluent-log4j              {:mvn/version "1.2.17-cp2"}}
 :mvn/repos {"confluent" {:url "https://packages.confluent.io/maven/"}}}

Typing clj in the terminal then gives me a lot of Downloading ... from ... messages and ends with this error:

Error building classpath. Could not find artifact com.github.everit-org.json-schema:org.everit.json.schema:jar:1.12.2 in central (https://repo1.maven.org/maven2/)

I’ve looked around and I can fix this by adding a repository:

:mvn/repos {"confluent"  {:url "https://packages.confluent.io/maven/"}
             "jitpack"   {:url "https://jitpack.io"}}

My assumption is that the confluent project pom.xml files have this extra repository “baked in” and lein honors this, whereas with the deps.edn ecosystem I’m required to specify this repo in the client project. Is this correct? Why is this behaviour like this?

[mvarela]

I had that problem when bumping my Kafka clients deps recently. I solved it by adding the Mulesoft repo (I guess it’s also hosted on jitpack). I think that jar used to be on Maven Central, but the newer versions are no longer there. I’m not using leiningen anymore, so I don’t know if it does any “transitive lookups”, as it were, but the problem appeared (for me) when bumping versions of the confluent libs, not inherently because of using deps.edn.

[pieterbreed]

I tend to feel stupid when I’m using deps.edn . This is the 3rd time that I’ve seriously tried to use deps.edn to specify clj dependencies and I keep on feeling helpless and not achieving outcomes I want to rely on.

Maybe I’m old and grumpy and maybe lein is just familiar to me, but it does a few things in exactly the way I’m used to. I couldn’t figure out how to make deps.edn do the following:

• The equivalent of the lein checkouts system. With leiningen you can create an optional checkouts folder and symlink dependency libraries into it. Then when project.clj is loaded, if the library in checkouts/lib-name/project.clj matches a dep in the client project.clj, it will load that local project into the class path using that checkouts/lib-name folder. This allows me to depend on a github commit hash AND optionally a local checkout of the same library at the same time with a filesystem entry the only difference.
• depending on github commit hashes. It seems easy to do with deps.edn from the documentation, but there are subtle issues.
  • The dependency is on the maven.pom file and not on the project.clj file. If you have a lein project in github you want to commit-by-hash on, you have to remember to create the pom first.
  • This also means build config information like aot is forgotten. (I live in java interop land and sometimes need to provide pre-compiled classes in my distributables). With the deps.edn github commit hash concept, those files that have to be compiled as part of the dependency is not.
• I did not manage to find out how to smooth dependency version conflicts. In lein world, I use the stderr output of lein deps :tree to see transitive dependency conflicts. I then manually set/override the versions of the transitive dependencies in my project to eliminate the conflicts. I combine this with lein ancient and stay as up-to-date with dependency verlsions as I want to. This is fast and easy and useful, and I can’t see the incantation to perform with clj to achieve this outcome.

Maybe I’m just a bad user, but this is my feedback.

[mars0i]

I feel similarly, though about other functions of Leiningen. I was thinking lately that there should be a Leingingen-to-deps.edn/CLI translation guide. This seems to be an attempt to do that:

Migrating from Leiningen to tools.deps · metabase/metabase Wiki

(I am not suggesting that it answers your questions. Maybe not.)

I wonder whether there are other such guides.

Tools like deps.new should also help with new repos, but it’s hard to start a new repo using deps.edn if you don’t know how to make it do the things you are used to doing with Leiningen.