Should I use boot or tools.deps to build command line applications?


#1

I used to use clojure with leiningen years ago. Nowadays, I use only Haskell for writing tiny command line applications that I personally use. I also write shell scripts for really simple tasks.

I looked into clojure again. It seems clojure and clojurescript are not great for scripts. But, clojure can be used to build small command line applications that are not called repetitively.

It seems there are three major build systems.

  1. leiningen
  2. boot
  3. tools.dep with pack or depstar

leiningen is ok for small projects, but it is awkward for any kind of slightly complex build.
Boot can be great, but it seems to involve a non-trivial amount of coding for simple projects.
pack or depstar seems simple, but it seems awkward to use unless I specify options in a shell script. tools.dep approach doesn’t seem particularly well suited for building linux distribution packages, either.

What is the best way to build command line applications? Perhaps, shall I stick with haskell for command line programs?


#2

I use Clojure extensively for scripting, with clj. I ended up writing a catch-all tool to make my life easier when writing and self-documenting. You may want to have a look at https://github.com/l3nz/cli-matic

In general, I find that the performance is good enough for real-life usage, and that the availability of libraries makes me very productive with a few lines of code. Lately, I’m toying with planck http://planck-repl.org/guide.html for very quick startup and - on the opposite side - with GraalVM for all-included binaries: https://www.astrecipes.net/blog/2018/07/20/cmd-line-apps-with-clojure-and-graalvm/

Using Clojure for scripting is definitely feasible and fun. And, of course, as soon as your scripts exceed putting two or three commands in sequence, using a “real” programming language does help.


#3

To get started, here is how I run my own scripts:

  • I have everything under one git repo
  • I use clj, and have a set of scripts that I run. So for some common things, I just write clj foo.clj and run it. These scripts are very bare, just the minimum logic, because…
  • I keep modules under src/. Everything that is recyclable (database access, reading spreadsheets, API calls…) lives in its own module and is often referenced by multiple scripts.
  • I use deps.edn for dependencies, and at the same time I have a leiningen project.clj that references dependencies in deps.edn via
    :plugins [[lein-tools-deps "0.4.1"]] :middleware [lein-tools-deps.plugin/resolve-dependencies-with-deps-edn] - so I run stuff with clj, but when I need to do some development, run tests etc I use Cursive linking a Leiningen project.
  • I use cli-matic for argument passing, validation and help generation.

Works like a charm. :grin:


#4

Do you build command line appliations with lein and lein-tools-deps?


#5

Like, lein uberjar of the above? yes, it’s trivial when you have a working script.


#6

I’ll second everything @l3nz says about clj and deps.edn. At work, we started with Leiningen back in 2011, switched to Boot at the end of 2015, but now we do everything with clj and deps.edn – we build JARs, uberjars, run scripts, run tests. There’s really no need to build artifacts for local command line script usage: clj is a great way of running simple Clojure scripts without a lot of baggage.


#7

deps.edn applies to every invocation of clojure in the current directory. I wanted deps.edn to apply to only one clj file. deps.edn is a bit awkward with command line scripts unless I put every related script in a project folder structure.

I wanted each clj file in the same folder to have different dependencies.


#8

You can write shell scripts that use clj to run themselves, for example (courtesy of @dominicm as I recall?) :

#!/bin/sh

"exec" "clj" "-Sdeps" "{:deps,{markdown-clj,{:mvn/version,\"0.9.85\"}}}" "$0" "[email protected]"

(require '[markdown.core :refer :all])
...

The dependencies are self-contained for each script. I have not yet tried this myself tho’…


#9

Just tested this and it does indeed work on macOS:

(! 668)-> ex_clj.sh 
#object[java.time.Instant 0x45e9b12d 2018-12-28T01:13:11.799Z]

(! 669)-> cat `which ex_clj.sh`
#!/bin/sh

"exec" "clj" "-Sdeps" "{:deps,{clojure.java-time,{:mvn/version,\"RELEASE\"}}}" "$0" "[email protected]"

(require '[java-time :as jt])

(println (jt/instant))

#10

Yes, that can work although it would be awkward to write a list of dependencies in one line without automatic indentation support and automatic bracket closure.

It took ~4 seconds to execute that simple script. Such a script cannot be used in a large loop. The loop would have to go in the script.

I wish there were dedicated scripting facilities for clojure in the future. A few things for scripting.

  1. It is easy to edit dependencies in the clojure script.
  2. The script can quickly compile into a binary. After the binary is created in a (hidden) folder, whenever the script is executed, the binary is executed. Currently, GraalVM compilation time is not short, and it consumes a lot of memory.

#11

Yes, JVM startup combined with the overhead of loading and compiling clojure.core can take a few seconds. Part of the startup time with clj is building the classpath but that is cached so I suspect you’d find subsequent invocations of the script were faster (probably twice as fast).

If startup time is your biggest concern then you either need to look at something like lumo (which means ClojureScript and, I think, Node.js behind the scenes) if you’ve also ruled out GraalVM…

…but I think, realistically, you’re not going to find anything that satisfies you in the Clojure(Script) world for fast-starting scripts with everything else you also want.


#12

I can live with a few seconds of startup time for certain use cases. But, I want to be able to edit dependencies with ease. Is boot script any better? I tried to write a boot script, but I couldn’t get myself to write one that compiles.

Perhaps, I should just write dependencies in shebang lines for now if I want to write scripts in clojure.

Do you know any fine programming language for command line scripting other than python or shells?


#13

Boot doesn’t cache the classpath after construction – which clj does. Also, Boot creates a “shadow fileset” abstraction which also adds some overhead. Of Boot, Leiningen, and clj, the fastest repeated invocation is going to be clj.

As for alternative languages, I’m not sure how many of them I would consider “fine”. Things I’ve written scripts in before Clojure: lots of shell (which isn’t really so bad), some Python, Ruby, Perl… and a few obscure things that no one has heard of these days…

These days, my first choice would be shell for the simple stuff and Clojure via clj for anything more complicated.


#14

Right, there are better scripting languages.


#15

You can also shave off half a second or so on each invocation by setting clojure.spec.skip-macros=true.


#16

I am playing a bit with Planck in these holidays, and I see that repeated invocation of a script, after initial download of dependencies and compilation, is around 500 to 700ms per run. That is 100x what GraalVM takes, but very acceptable in most cases.


#17

lumo is even faster to start up.


#18

For the right type of command line app, Ferret might be the right tool, building tiny, native executable that have no VM overhead.

Be warned: Though it hass access to the full clojure ecosystem during compilation/macro expansion phase, the restricted environment at runtime requires a change in thinking - for instances macros that use str are fine, but functions that use str aren’t!