Babashka task runner

Dear fellow Clojurians,

I’m working on a new feature in babashka called tasks. The goal of this feature is to replace 80-90% of the Makefile/Justfile/npm scripts and bash wrapper scripts that are typically found in a Clojure project.

Here is a Youtube video showcasing the first preview version of babashka tasks. This preview is available in bb 0.3.5.

I would LOVE to have your feedback on this. As this feature is still in design, there is room to make some breaking changes if necessary. Maybe you could try to port an existing Makefile to this feature and see how it pans out.

Full documentation of the bb tasks feature as of today can be found here.

Thanks for taking a look!

19 Likes

Does it fit to replace tools like https://www.npmjs.com/package/foreman ?

If you mean that you can use bb tasks to run things in parallel, then yes. I should have mentioned that in the video, but at work I’m using it like this:

So define a couple of tasks. Then define one task which depends on all of those and run that task using the --parallel option.

Well, I mean running in parallel and constantly, sharing the console output.
So that I can start multiple development tools to work with the project and then stop them all at once with single key/command.

Yes, that is what happens in the link I shared.

I just converted our fairly simple Makefile to bb.edn without any problems. With babashka I was able to get rid of our bash files and now I can do the same with the Makefile. Thanks for all your awesome contributions @borkdude.

4 Likes

Is it possible to pass arguments to tasks? From the discussion looks like I should use (ednopt/parse *command-line-args*)

This is correct. You can use *command-line-args*. When writing {:task foo/bar} this is automatically expanded into (do (require 'foo) (apply foo/bar *command-line-args*)), if you want to make a function from your sources accessible as a task.

EDIT: if you want to define functions, you can do so in the :init hook.

Isn’t there a way to figure out what can be parallelized from the dependency tree?

That is what happens already. Every task should be able to run in parallel. But running them in parallel is optional and set using a command line option.

Maybe I’m confused, do you mean when you set --parallel it doesn’t just run all tasks in parallel, but instead only if it’s possible based on the dependencies? And it does so transitively?

a depends [b c d e]
c depends [b f g]

Now run a in parallel and what happens?

When you have task a which depends on b and c and you use the --parallel option, then b and c are run simultaneously but must both successfully terminate before a is executed.

With your example:

a depends [b c d e]
c depends [b f g]

what happens:

a waits for b and c.
c waits for b (and f and g).
so b must finish first, before a and c continues.
but c must also finish first before a continues.

So c and b never run in parallel in this example, but f and g will and d and e also will.

You can try this coffee brewing example to show how it works. Without parallel it takes about 11 seconds, with parallel it takes about 6 seconds.

Ok I see. I wanted to be sure that if there was a transitive dependency it be smart enough to serialize them, like how b and c shouldn’t be parallel because of c depending on b itself.

And so, to get f and g in parallel as well do you need to setup c to be --parallel as well, or setting a to be --parallel and running a is enough?

And last thing, in theory I think from the dependencies you can figure out that you can run d, e, f and g all in parallel, and then b, then c and then a. Is that what happens, or it’ll be staggered? Like f and g in parallel first, then b, then c, then d and e in parallel and then a?

@didibus --parallel is a global option, so all dependencies are ran in parallel if possible. This is similar to make -j4 foo.

Try this bb.edn:

{:tasks {:init (defn prn* [x]
                 (Thread/sleep (rand-int 100))
                 (prn x))
         a {:depends [b c d e]
            :task (prn* :a)}
         c {:depends [b f g]
            :task (prn* :c)}
         b (prn* :b)
         d (prn* :d)
         e (prn* :e)
         f (prn* :f)
         g (prn* :g)}}

Running this a few times with bb run --parallel a you can see that all tasks without dependencies can run in any order before tasks with dependencies on those.

3 Likes

I love it, bb and the Clojure CLI will be a powerful combination with this. Do it! :yum:

1 Like

I love the idea behind bb tasks. I would love to see a code equivalent since I want to have all the syntax highlight and testability of a normal source file. I have some bad experiences with coding in yaml/xml/json…

(ns tasks
  (:require [babashka.tasks :as task]))

(defn prn* [x]
  (Thread/sleep (rand-int 100))
  (prn x))

(task/deftask a
  "blah blah"
  [b c d e]
  (prn* :a))

(task/deftask c
  "this is for?"
  [b f g]
  (prn* :c))

(task/deftask b (prn* :b))
(task/deftask c (prn* :c))
(task/deftask d (prn* :d))
(task/deftask e (prn* :e))

(comment
  (task/run a :arg1 "skks"))

My current deployment/dev scripts are more or less similar to this example. I really want to port to babashka, but the current roadblocker would be aws-api :slightly_smiling_face:

1 Like

As for AWS api you can perhaps use GitHub - babashka/pod-babashka-aws: AWS pod wrapping the Cognitect aws-api library. It offers an almost compatible API with Cognitect’s AWS api.

As for the code equivalent of tasks: interesting thought, I will consider it.

2 Likes

I posted your comment in the bb tasks discussion over here: `bb tasks` feedback · Discussion #779 · babashka/babashka · GitHub

I have used mach as a Make replacement to good effect. It might be useful to gander at for design inspiration.

I took inspiration from that and have spoken with its creators. I hope Mach users who no longer use it, since it’s no longer maintained, can find a useful replacement in bb tasks.

3 Likes