Unified Front and Backend Development Environments

[Kendall_Buchanan]

Can anyone recommend how to architect a frontend SPA (non-ClojureScript) to unify with their Clojure backend?

My team manages a Clojure monolith of ~40K lines; it’s in good shape, it has powerful tooling—all the things Clojure is known for. But, it works alongside a companion monolith built in JavaScript and Webpack, containing nearly 120K of JavaScript. The backend serves JS bundles over Ring and Optimus middleware.

The Clojure and JS toolchains exist as two separate entities, the frontend living physically in a different directory with a mocked, Express backend. The goal was to let frontend JS developers test on a lighter weight platform, unfettered by backend concerns.

Now, however, as the monolith has grown to five independent applications, each with mock data on both the front and backends, it’s painful:

• Handing off production bundles from the front end to be tested on the backend app (manual integration testing) is slow, and feels repetitive (testing in two places).
• Two sets of mock data for each application
• Communication errors between front and backend developers are common—there’s a tendency to “throw the code over the fence”; with neither developer understanding what the other should be testing for.

Going forward I would like to begin unifying these apps: to test JS code against a Clojure backend directly. But how does that work with reloadable patterns? (We use mount.) Do you rig up the (start) functions to load Webpack in the background?

Thanks for sticking with me… any recommendations?

[maxweber]

Maybe it is sufficient to make it as convenient as possible to start the backend in the JS development environment and vice versa?

I constantly try to improve the ‘developer experience’ of our tech stack. It went through many iterations. The latest version uses one Docker container that contains everything to run the backend, frontend and all development tools. The team members only need to install an IDE and babashka, the rest of the dev environment is shipped via the container.

I would also try to empower all developers to contribute to the backend and the frontend.

[Kendall_Buchanan]

@maxweber: Appreciate the insights. It’s probably time we investigate Docker (finally).

What baffles me is I feel like I’ve never been able to find any guidance on Google for marrying front and backend Clojure and JS. (CLJS is the obvious choice for most Clojure shops.)

[maxweber]

Is it more about the project setup or how the Clojure server should serve the files of the JS frontend?

[Kendall_Buchanan]

@maxweber: It’s really about harnessing the power of Webpack. Admittedly we’ve spent zero time trying to build tooling into our Clojure backend for working alongside Webpack, mostly cause I don’t know what I’m up against. Webpack is handling code splitting, assets, and a host of other things, and trying integrate its lifecycle into our reload patterns in Clojure sounds… like a big project, to say the least.

Anyway, mostly just curious if anyone has experience with these two environments in one place.

[pieterbreed]

I must admit I don’t fully understand what challenges you are facing. So I’m going to do a brave/stupid thing and just generally talk into this space based on what I think I understand and one or two things I’ve tried before. I hope it’s useful.

---

Generally, UI’s and backends are treated as “separate” projects. The UI (often React or Vue) are kept in separate source control repositories, apart from the backend project. The issue with this is that when it comes to run time, the UI and the backend are actually the same software stack. Ideally even, you want your backend to host the static assets of the frontend (ie have ring serve the .js, .html & .css files from resources). It’s not strictly speaking necessary to host the UI assets from the uberjar, but it makes the job of your devops team slightly easier.

Another thing is that I’ve found that the build tooling around frontend/javascript projects lives parallel to the build tooling of clojure projects. What I mean with this is that usually js projects build with npm or yarn (or shadow-cljs) or something; there are usually tools like PostCSS involved, you mention webpack etc. These are not integrated with lein or boot (and I don’t think they should be). Embrace and extend as they say…

Additionally, I’ve encountered the pattern a few times where the javascript build tooling for development depends on an HTTP dev-time server, that takes care of loading the assets from disk, reloads the frames when the assets are updated (eg you edit the javascript or the css, the postcss --watch -type processes kick off, new assets are generated. This dev-time HTTP server usually has a websocket or something that then notifies the UI and a reload happens. It’s super convenient.

So with all of this in mind, I’ve structured a few projects like below and it worked for me:

1. I host my UI and backend code in the same source repo, because I can symlink the output folder of the js build process into my clojure resources. I then have an HTTP route in clj that knows to serve those assets from there.
2. I structure my UI code to load a settings file from /settings.json or /settings.edn. This settings file contains high-level configuration, but usually at least the UI logging level and the protocol, host & port of the backend “API” service, eg http://localhost:8000. I create a static settings file for dev-time that contains the correct assumptions (port) about dev-time and then I have the clojure backend craft a settings file at runtime that points back to itself. Often one of the configuration values you need to give an HTTP service as the “public URL”. This is the value I put in the settings file served to the UI.
3. As mentioned above, I symlink the javascript project’s “output” folder into the clojure project’s resources. Resources are not often loaded from here during dev time, because you will lose the auto-reload, jacked-in websocket type behaviour I described. During prod-build time though, those assets once minified and compressed can live comfortably in the uberjar’s resources and can be served from there efficiently at run-time.

This setup allows me to start the clojure repl and service, start the js tooling, have all the niceties of hot-reloading for dev on the UI (and on the backend…) and still have the benefits of a self-contained deployable for prod/run-time.

There are other benefits too. Usually business stakeholders realise (or communicate) too late about whitelabeling user-interfaces. This setup with the settings.json file allows to statically deploy customized versions of the UI to separate UI domains, and have them all use the same backend.

Anyway - this is what came to mind when you asked about unifying frontend and backend work.

[Kendall_Buchanan]

@pieterbreed Thank you so much for your thoughts; this gave me TONS to think on, and helped clarify several ideas. I think most of all you’re giving me the courage to take on these problems by hand (previously fearing I was missing some clear pre-made path)!