Migrating from Boot back to Leiningen


#1

When I mentioned this in last week’s “what are you working on” thread I knew it would raise some eyebrows, and since someone asked, I figured I’d elaborate a bit on why we chose to go this way and how the experience has been.

Please no “you should have just” replies. This is presented as a data point, this is how we experienced it. Maybe if we had more boot experts on the team it could have gone differently. Maybe not.

The project is NextJournal. When I started helping them out some six months ago, they had been using boot for a while, and it seemed to be causing them a lot of pain. It’s hard to point at a single thing that caused this, it was a combination of factors, but it all led to team feeling like they weren’t in control. Debugging and trying to get the setup right was tedious and frustrating.

This is a project that includes three different ClojureScript builds: the web frontend, a web worker that evaluates JS and CLJS, and a renderer that runs on Node to generate the initial payload offline. For each of these we need dev, test, and production builds. Then there’s a Clojure based server component.

Further complications arise because of a large number of foreign js dependencies. Cljsjs works well for the browser, but many of the cljsjs packages are not suitable for use on Node, so we had to resort to some exotic hacks to prevent ClojureScript from using the cljsjs packages (even when a dependency asked for it, e.g. reagent -> cljsjs.react), and instead load them from node_modules directly.

I’d say the biggest trouble we ran into came from Boot’s “fileset” approach. Boot tries to turn the filesystem “immutable” by creating temporary directories, and having each build step write to a new tempdir. This is to prevent stale artifacts from messing up your build, but it also means your build can’t reuse any state from a previous build. What this meant in practice was that any change took close to a minute to recompile and show up in the browser. This was simply unacceptable.

There’s boot-figreload, but it’s not figwheel, it only borrows small parts of it, it does not in any way give you that smooth unified experience. We decided we wanted figwheel, and tried to use figwheel.sidecar from inside the app, but boot puts tmpdirs on the classpath Pedestal was serving up assets from there, whereas they should have been coming from the project directory.

Another huge source of frustration was just getting the ClojureScript builds right. I mentioned before the funky foreign-libs setup. The thing is that tools like lein-cljsbuild or figwheel pass there options more or less unchanged on to the ClojureScript compiler, whereas boot-cljs does a lot of “magic” under the covers that ended up getting in our way.

Boot itself was also a source of frustration. Tasks are supposed to compose, until they don’t. e.g. you can’t have a REPL and a watcher task in the same process. I spent a lot of time in the boot and boot-cljs codebases trying to understand what was going on, it did not seem like there was enough documentation for this. In particular I had the file with boot’s built-in taks almost constantly open.

We now just finished the migration to Leiningen, so it’s a bit hard to give any longer term perspective yet, but it feels like we’re in control again, and the development experience has improved a lot, with subsecond recompile times in Figwheel.

Leiningen isn’t as scriptable, but it’s more malleable than people think. We ended up pushing much more stuff into the system itself, or the other way into shell scripts, which also made everything behave more predictably. This is the approach I would generally recommend to people: don’t try to do everything with Leininge, do more inside your app instead.


Roadmap to 1.10?
Support for a "build matrix"
#2

Anything to compare with shadow-cljs?


#3

No idea about shadow-cljs. It would only solve part of our needs, we would still need leiningen for building an uberjar.


#4

Thanks for the write up. Definitely understandable pain points. :+1:


#5

Very interesting. Thank you. You definitely have a complex build setup there, wow!


#6

Thanks a lot for the interesting post!

There’s boot-figreload, but it’s not figwheel, it only borrows small parts of it, it does not in any way give you that smooth unified experience. We decided we wanted figwheel, and tried to use figwheel.sidecar from inside the app, but boot puts tmpdirs on the classpath Pedestal was serving up assets from there, whereas they should have been coming from the project directory.

As maintainer of boot-figreload I would be super interested in knowing what you have missed of the figwheel experience. There is always room for improvement!


#7

As author of both Cljsjs and Boot-cljs, I might be able to provide some insight into some of the mentioned points. As background, I work with projects using both Lein and Figwheel, and Boot and Boot-cljs stack, and I don’t have any plans on migrating all the projects to use either of tools. Both work and have their benefits.

Further complications arise because of a large number of foreign js dependencies. Cljsjs works well for the browser, but many of the cljsjs packages are not suitable for use on Node, so we had to resort to some exotic hacks to prevent ClojureScript from using the cljsjs packages (even when a dependency asked for it, e.g. reagent -> cljsjs.react), and instead load them from node_modules directly.

This is problematic in ClojureScript in general currently, and not really related to built tool. The Reagent alpha version currently requires React like this: [react :as react], and this will allow Reagent to use either Node Module or the updated CLJSJS package, which also provides react name and supports :global-exports which allows using :as and :refer with foreign libraries. Reagent is now tested against multiple environments: https://github.com/reagent-project/reagent/tree/master/test-environments So this should hopefully in future help with these kinds of problems.

I’d say the biggest trouble we ran into came from Boot’s “fileset” approach. Boot tries to turn the filesystem “immutable” by creating temporary directories, and having each build step write to a new tempdir. This is to prevent stale artifacts from messing up your build, but it also means your build can’t reuse any state from a previous build. What this meant in practice was that any change took close to a minute to recompile and show up in the browser. This was simply unacceptable.

Hmm, I think some parts of this are not completely correct. If I understand correctly, this refers to Figwheel style development workflow, and build means recompilation after saving a file? The initial compile will indeed be slow, as Boot-cljs doesn’t store state between Boot processes (it could), but the following compilations should be fast. Each task indeed creates a temp dir, but when task is ran again after a file change (with watch task), it will use the same temp dir as previously. For example Cljs task uses the same temp dir each time, so it should work the same as Lein-cljsbuild and Figwheel.

(The task function creates a closure and returns middleware function, which in turn returns handler function which modifies the fileset. The temp dir is created by task function and stored on the function closure. The following invocations of created handler fn use the same closure.)

There’s boot-figreload, but it’s not figwheel, it only borrows small parts of it, it does not in any way give you that smooth unified experience.

Boot-cljs and Boot-reload provide smooth experience, but the Node support is probably non-existent, so Figwheel is good choice for projects which require that.

whereas boot-cljs does a lot of “magic” under the covers that ended up getting in our way.

The last few releases improved this a bit. Boot-cljs now only provides default options in most cases, and in few cases modifies the value (e.g. to prepend the temp-dir path). All the options Boot-cljs has special logic for, are documented here: https://github.com/boot-clj/boot-cljs/blob/master/docs/compiler-options.md

it did not seem like there was enough documentation for this

True. And unfortunately, seeing the state of Boot development it doesn’t look very likely that this would improve a lot. But maybe it will help tiny bit if answer some questions here, instead of Slack, where the answers are lost :slight_smile:

This is the approach I would generally recommend to people: don’t try to do everything with Leininge, do more inside your app instead.

I can agree with this, and I follow the same principle with Boot projects :slight_smile:


#8

I started writing a reply but it went way off topic and talked about shadow-cljs too much.

In case you are interested you can find my take on this here:

The same conclusion as @plexus and @juhoteperi basically.


#9

I was recently trying to help someone get boot-figreload working with a Node project. It worked quite well, reloading worked mostly out of the box, by replacing it for boot-reload. The only feature missing was a ClojureScript REPL into the Node project, which we couldn’t get to work and it seems the figwheel cljs repl is tightly coupled with project.clj.


#10

I’m sorry to hear about your experience with boot!

To give this thread positive twist that could benefit both build tools:

Publish a minimal working leiningen project that reflects the app you’re talking about.
The challenge is to convert it to boot while maintaining these requirements.

I’ve done this two years ago when I was curious about boot and this blogpost was the result.

Maybe we could learn new things that way.

About watch and the REPL: we start nREPL from the app itself (not as part of the Component system). This way we can also inspect things in our staging and prod environments. But you can also run (repl :server true) in combination with (watch), this way it won’t start the client, which is blocking and preventing other tasks from running (credits to @juhoteperi for that piece of information I didn’t know myself yet).


#11

Some tips regarding figwheel and boot, figwheel-sidecar is a great tool to apprehend and build your own tasks. Here is a gist to get a browser connected repl to a node process


#13

I love this. Thanks for figuring this out.


#14

I wanted to give mach an honorable mention in the build tooling category. If you’ve been missing the Make experience from Clojure-land it is worth taking a look!


#15

I looked at the README Mach several times before. But still not sure what Mach is capable of.


#16

It is a build tool similar in principle to Make, though much improved. It allows you to define tasks for building artifacts, e.g. running a compiler to build an object file, where the tasks in turn depend on other tasks. The tool takes care of invoking tasks in the correct order. There are additional capabilities, but that’s the core of it.


#17

Do you have an example for building artifacts with Mach? Currently I use Boot but really want to replace it with something from ClojureScript side.


#18

This example builds asciidoc > xml > pdf. There is a fair amount of instruction to be had in the repo documentation and the examples/ directory. I believe there is a slide deck overview floating around somewhere as well.


#19

Looks cool. But you don’t use it to release jars on Clojars right?


#20

Not without a bit of work, I would expect. Hrmm, sounds likes an interesting puzzle to solve. I suppose you could invoke lein to deploy in a pinch.


#21