The Further Adventures of Starting a new ClojureScript project in 2022

interstar · August 11, 2022, 1:05pm

So thanks to all the responses to my previous question : Starting a new ClojureScript project in 2022. Setup suggestions

It’s been very useful and informative. And I’m truly grateful.

But I’m continuing here because I think this experience, and the frustrations, are important for the community to think about.

I’m certainly no Clojure expert. (This is really bringing this home to me). But I have been using Clojure happily in my projects since 2014. I’m kind of surprised I’m having so much trouble with this now.

The key is that I’ve previously been using Lein to make my projects. And that has “just worked”. The one time I made a project with CLI tools I used GitHub - oakes/Lightmod: An all-in-one tool for full stack Clojure And that was fine too. But AFAICT that is more or less abandoned / not being updated. Plus it brings a lot of dependencies I don’t need, and none of the templates it comes with are quite what I want. It’s a shame because it was very useful.

So now I want to be able to start new projects in a “proper” and up-to-date way. The way I’m “meant” to do it. But it’s hard, partly because new things seem to be being invented all the time, and old things seem to be going out of date. And you have no idea whether random tutorials online are still relevant or broken.

So. What I want to do is setup a simple ClojureScript + Reagent + Figwheel project with tools.build / tools.deps so that I can incorporate a library which I am developing in parallel in a separate project.

What I want to do is the equivalent of

lein new reagent myproj +figwheel

as described on GitHub - reagent-project/reagent-template: A Leiningen template for projects using Reagent.

In fact, I know this is EXACTLY what I want to do, because right now, that’s what I’m doing. I’ve been able to set my project up like this and have started working on it.

But I STILL don’t know how to do the equivalent with CLI / tools.deps. And still can’t find an equivalently straightforward and comprehensive example online.

As I say, the only reason I don’t use Lein for this is because I want to pull from another development project on my machine. Which AFAICT Lein still can’t do. (If someone tells me Lein CAN now do this, I’ll stop worrying and just use Lein in a heartbeat)

But given that I can’t do that, I need to get to grips with CLI / tools.deps

So, of course. What I tried was @seancorfield’s GitHub - seancorfield/clj-new: Generate new projects based on clj, Boot, or Leiningen Templates!

clojure -Tclj-new create :template reagent :name myname/newproj

That, AFAICT is something that takes the Lein template and makes a tools.dep project from it, right?

Fine. It runs, I go into the directory. All I have to do is edit the deps.edn to add my new dependencies and …

Ah, so where’s the deps.edn file?

There isn’t one. I thought this was adapting the lein template to tools.deps? OK. Maybe not that straightforward.

Well let’s at least try to run the thing.

clj -X newproj/run

(based on Clojure - Deps and CLI Guide )

Namespace could not be loaded: newproj

OK. I should really go and read Sean’s page. So let’s try

clojure -M -m myname.newproj

Based on what he says there.

Hmmm …

Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate newproj__init.class, newproj.clj or newproj.cljc on classpath.

Full report at:
/tmp/clojure-2441616383675739567.edn

Shall we go and look in /tmp/clojure-2441616383675739567.edn ? I won’t bore you with the details. It’s just a variant on not being able to find any file it thinks is an executable entry point.

So this directory doesn’t have a deps.edn file, but it DOES still have a project.clj file for lein.

So can lein run it?

lein figwheel

Well … starts promising. Until I hit the

Exception in thread "Thread-10" java.lang.NullPointerException
	at java.base/java.util.regex.Matcher.getTextLength(Matcher.java:1770)
	at java.base/java.util.regex.Matcher.reset(Matcher.java:416)
	at java.base/java.util.regex.Matcher.<init>(Matcher.java:253)
	at java.base/java.util.regex.Pattern.matcher(Pattern.java:1133)
	at clojure.core$re_matcher.invokeStatic(core.clj:4881)
	at clojure.core$re_find.invokeStatic(core.clj:4923)
	at clojure.core$re_find.invoke(core.clj:4923)
	at cljs.analyzer$analyze_file.invokeStatic(analyzer.cljc:4850)
	at cljs.analyzer$analyze_file.invoke(analyzer.cljc:4826)
	at cljs.analyzer$analyze_file.invokeStatic(analyzer.cljc:4840)
	at cljs.analyzer$analyze_file.invoke(analyzer.cljc:4826)
	at cljs.analyzer$analyze_file.invokeStatic(analyzer.cljc:4836)
	at cljs.analyzer$analyze_file.invoke(analyzer.cljc:4826)
	at cljs.repl$repl_STAR_$fn__8064$fn__8065$fn__8068.invoke(repl.cljc:1181)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.core$apply.invokeStatic(core.clj:667)
	at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1990)
	at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1990)
	at clojure.lang.RestFn.invoke(RestFn.java:425)
	at clojure.lang.AFn.applyToHelper(AFn.java:156)
	at clojure.lang.RestFn.applyTo(RestFn.java:132)
	at clojure.core$apply.invokeStatic(core.clj:671)
	at clojure.core$bound_fn_STAR_$fn__5818.doInvoke(core.clj:2020)
	at clojure.lang.RestFn.invoke(RestFn.java:397)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.lang.Thread.run(Thread.java:829)
ClojureScript 1.11.54

Now, to be fair, this is probably not tools.dep nor Sean’s fault. There just must be something a bit weird / wrong with this template.

Though the lein project created with the same template works fine. So it’s not the template by itself that’s broken. But some mismatch between the template and the CLI environment?

Or maybe clj-new has done a perfectly good job of adapting the lein template, and this is a fully working project.

I just don’t have the faintest idea how the devil I’m meant to run it! Nor where I can find that information

Perhaps I’m getting this backwards. And it’s really about figwheel. This is a figwheel project after all, so how do I run that with CLI? Quick Google turns up figwheel-main | Figwheel Main provides tooling for developing ClojureScript applications which suggests I run

clj -M -m figwheel.main

Nah …

Execution error (FileNotFoundException) at clojure.main/main (main.java:40).
Could not locate figwheel/main__init.class, figwheel/main.clj or figwheel/main.cljc on classpath.

Full report at:
/tmp/clojure-15651618584841800008.edn

Well of course it doesn’t run. It probably needs a deps.edn file to tell it where to find the entry point to the app.

But that’s exactly what clj-new didn’t produce.

So I’m concluding that clj-new to adapt / convert the lein template is a non-starter. After all where would I add my dependencies without a deps.edn file?

And if anyone does know a magic formula for CLI tools.dep that does the same as

lein new reagent myproj +figwheel

then I’d be very grateful. Otherwise I’m off back to google to explore some other options.

cheers …

joinr · August 11, 2022, 1:40pm

I feel your pain. I still prefer/live in lein for the most part (that’s entirely valid). For both lein and the “official” CLI, I think the figwheel-main tutorial does a great job of answering your questions. It primarily focuses on the CLI.

Then there is the figwheel-main template that leverages clj-new to do what it looks like you were after.

The other option that is being popularized is to use shadow-cljs to handle everything. I am less familiar with the templating around that, but I assume the project setup problems have been solved.

WRT to lein, you can directly leverage other projects through checkouts (I stopped using them a long time ago though), adding them to source-paths (pretty trivial, allows hacking on several related projects simultaneously), some plugins like lein-git-down [I have not used it], and even using lein-tools-deps to outsource the dependency resolution to tools.deps (so you define a deps.edn and tap into everything the CLI does that way).

There are even more fragmentations these days with the emergence of tools.build. Sucks for newbs or temporal amnesiacs who are trying to get (re)started.

interstar · August 11, 2022, 2:05pm

Thanks.

I’d just discovered there is also a figwheel template for clj-new and was going to investigate that next.

Yep. I’ve seen shadow.js too. But as that seems to be all about node and npm etc. (which is yet another whole package management ecosystem I don’t want to get involved with) I’ve been trying to avoid that.

But possibly the most interesting idea is GitHub - RickMoynihan/lein-tools-deps: A leiningen plugin that lets you share tools.deps.alpha dependencies in your leiningen project if it works. Let’s see. I’ll try the figwheel route first and if that fails, look into that.

Cheers

Phil

joinr · August 11, 2022, 2:13pm

Yep. I’ve seen shadow.js too. But as that seems to be all about node and npm etc. (which is yet another whole package management ecosystem I don’t want to get involved with) I’ve been trying to avoid that.

It turns out, while it’s largely advertised as such, you can technically leverage all the shadowcljs stuff through lein or as a stand alone clojure lib/uberjar. Add shadow-cljs to existing Leiningen project - #5 by DrLjotsson

It is meant more as a cohesive replacement for cljsjs and to fill in some gaps and limitations the original cljs compiler had IIRC. I may be migrating to it in the future since I am using more js libs (many of which don’t have cljsjs presence) and I am more willing to entertain having npm around to get access to the ecosystem (aside from just including the js libs manually and wrapping them, which messes with advanced compilation). I think it’s possible to leverage shadow for its non-npm features, but you are cutting out a lot of the benefit there.

seancorfield · August 11, 2022, 2:58pm

clj-new is able to run Leiningen templates, Boot templates, and clj-templates.

In general, Leiningen templates produce Leiningen projects – the README for clj-new specifically notes this for the Luminus template (the re-frame template example is noted to produce a Shadow-cljs template ,without project.clj or deps.edn).

Boot templates produce Boot projects, and clj-templates produce CLI / deps.edn projects.

clj-new essentially includes lein new and boot new – so the convenience is a single tool to run all three types of templates with a consistent set of options. But templates produce whatever they’ve been written to produce.

When you generate the reagent template (using the command you showed), it tells you it is producing a Leiningen project:

Downloading: reagent/lein-template/maven-metadata.xml from clojars
Generating fresh 'lein new' Reagent project.

I’ll update the clj-new README to make it clearer that you need to follow the instructions for the template you’ve used as generated projects are all different.

interstar · August 11, 2022, 3:07pm

OK. Thanks for this Sean.

Sorry. I wasn’t intending to pick on clj-new in particular, just using it as an illustration of the general confusions I’ve been encountering when trying to find a solution online.

But I take it there’s no intention clj-new “translates” from a lein template to a tools.deps template then? Just that this one tool can work with all three?

seancorfield · August 11, 2022, 3:11pm

No worries – this helps improve the clj-new docs. It’s hard to know how much a potential user does or does not know about the Clojure ecosystem and I’d clearly made some assumptions in the README that don’t hold true for all users.

If I were starting a new ClojureScript project today, I’d start with re-frame and figwheel-main but everyone keeps telling me to use Shadow-cljs instead (but still re-frame). I just really want to avoid JavaScript and Node.js etc!

zcaudate · August 11, 2022, 3:23pm

I would absolutely recommend shadow-cljs for cljs projects. It was heads and shoulders the best experience for cljs 5 years ago when I last used it. It was a complete solution for build and live-repl and just worked with a single shadow.edn file. It was better than figwheel then, I’m sure it’s even better now. For a while, before deciding not to use cljs anymore, I wasn’t even using lein and was using shadowcljs for my clj projects as well. It’s that good.

You SHOULD care about the npm ecosystem if you work with JS devs. If you want write a cljs library, you will need a really good working knowledge of js. If you want to write a clic library, even more so. The npm integration is probably the best feature of shadow-cljs and will allow you to work with the latest version of a js library, instead of waiting for someone else to put it up on cljsjs for you.

The other library to try is GitHub - borkdude/cherry: Experimental ClojureScript to ES6 module compiler. It goes further by removing google closure. I’m not too sure of the build/live repl tooling so you’ll definitely be a guine pig - and it’s not a bad thing if you are interacting with the lead dev. cljs is quite different from jvm clojure in a lot of ways so the closer you can get to JS, the less of chance that your project will randomly break due to any number of upstream changes.

seancorfield · August 11, 2022, 4:02pm

My first time working with ClojureScript was in 2013 I think and tooling was pretty horrible back then. I built a prototype for work with Om, then rebuilt it with Reagent (which was brand new at the time) and liked that a lot better. We decided cljs wasn’t ready for prime time and set it aside.

I started to look at cljs again late last year and really liked the (new) figwheel-main experience, with re-frame at this point, and found it much simpler to work with than Shadow-cljs – and it let me avoid JS/Node.js which is the only reason I would even consider doing any frontend development (I toyed around with Elm for a while and that also let me pretty much ignore the JS ecosystem).

As someone who has designed programming languages and built many compilers, interpreters, source code analyzers, and runtime systems (including one of the first ANSI-validated C compilers, and also spent eight years on the ANSI C++ Standards Committee), I have fairly strong feelings about languages and JS is just a deal-breaker for me for any serious development

interstar · August 11, 2022, 4:11pm

OK.

I seem to have found a solution that works. Using clj-new and the figwheel-main template.

clojure -Tclj-new create :template figwheel-main :name myname/myapp :args '["+deps","–reagent"]'

Then run the app with

clojure -M:fig:build

I hope this helps anyone else with similar confusion to me.

Thanks again everyone.

joinr · August 11, 2022, 4:18pm

I’ve been on a similar route, particularly doing client-heavy SPAs designed as independent apps hosted on an intranet (but doing like heavy 3d, cartography, stuff with cesium and three.js). Went figwheel + reagent → figwheel-main + reagent. In the process, I started leveraging libs that weren’t available on cljsjs (or weren’t current) and discovered the joys of that circus.

The level of libraries (including stuff from c/c++/rust that has been cross compiled via ecmascript/wasm) that exist on NPM can’t really be understated. So if you want to tap into those libraries relatively transparently without having to deal with JS proper outside of interop, it looks like shadow-cljs provides a pretty compelling option. Its main benefit (over what [we] appear to be leveraging) is the bundling / advanced compilation support (it just works in theory), where otherwise you would have to handle everything or provide externs yourself. If you care about artifact size (I had some leeway since this was on an internal LAN, so big pages/bundles weren’t a huge deal and I could sidestep advanced compilation for a bit), then having something like shadow leverage NPM to sort out all the bundling /modulesand externs is a big advantage.

That being said, you still may end up messing with npm, webpack, maybe even babel, and a host of incidental fragments of a nutcase ecosystem that have nothing directly to do with writing code.

zcaudate · August 11, 2022, 4:19pm

I experimented alot with when it first came out (I think it would be 2013 as well).

I had to deal to a lot of breaking changes to both cljs and js at that and in the end, decided it wasn’t really worth it. After figwheel came out, I tried it, then tried shadow-cljs and found it to be so good. I didn’t realise that work on figwheel was on going - apologies. But at that time, shadow-cljs was just incredible to use.

We currently compile clojure forms to JS, but it’s a much different process to the CLJS approach, closer to that of what @borkdude is doing with cherry. I think that the approach we use produces more robust JS code than relying on the cljs ecosystem. So that’s why I’m more of a fan for integrating with npm, rather than going against it.

Fun story. If you visit the purnam repo, you can see it’s forked … but that’s because I accidentally deleted the original repo. The guy I forked it from used purnam it to build a bitcoin casino which he sold for a really good price.

zcaudate · August 11, 2022, 4:25pm

yeah agreed. webpack is awesome, babel is awesome. It pays to be familiar with them earlier and not when things are turning to shit because it probably will at some point on a cljs project.

qualifying the ‘turn to shit’ comment. What I meant is that when a project reaches a certain size and an error happens, it then requires a lot of js knowledge to understand what is going on and how to fix it.

mdiin · August 11, 2022, 5:20pm

I haven’t dug into Clojurescript for years, apart from the odd experiments in cljc files, so I have nothing to offer in this discussion but curiosity I’m afraid; how do you mean “turn to shit”?

I would really like to believe that Clojurescript is mature enough to build real apps with, but with no experience in this regard I lean on what others with that experience say. What parts will inevitably go wrong in a Clojurescript project, and how do I handle when that happens? I’m toying with the idea of using Clojurescript for something serious, and would like to understand the pitfalls.

zcaudate · August 11, 2022, 5:42pm

I don’t think there’s anything to be scared about cljs. Jump in, have a go. Debugging cljs errors is probably the best way to learn javascript… and if you don’t know javascript before your cljs project, you definitely will by the time you’ve built anything substantial. If you do know javascript, it’s worthwhile to go deeper.

I think the real difference is in the mentality for devs when clojurescript came out versus where we are now. When clojurescript first came out, it brought repl programming to the browser. At the time, JS was the browser. Node was really new and projects didn’t have the kind of tooling they have now. It was really exciting to be able to interact with the browser like you would with the JVM. It was really the first of it’s kind at the time and so even if things got messy, the promise of more maintainable code in the browser meant that I as a dev would want to persist in getting something working.

These days cljs projects are less about exploration and more about standards and best practices. Now, there are a lot more things to consider. There are other targets for JS besides the browser (node, qjs, react native, electron, etc). CLJS support for those targets are not as good. In these cases, shadow-cljs will be the best option because of it’s npm integration.

At that time, I had wanted to use React Native. RN is already complicated enough and could go wrong just because the compiler was having a bad day. Adding an additional CLJS compiler made things doubly worse - not to mention that the existing CLJS/RN tooling was really outdated and one would always be playing catchup as RN ecosystem changed relentlessly - as it still is today. I didn’t want to get into the cycle of breaking code due to any number of upstream changes as I had experienced with clojurescript in its early days.

The host interop on JS was also never as good as it was on the JVM. In a lot of cases, the immutable datastructures are more of a hinderance than a boon. Interop code becomes full of #js {:a #js ["a", "b"]} and by doing that, you’re not really writing clojure anymore, you’re writing clojurescript. Javascript now is less verbose.

And finally, whilst cljs has matured, it has lagged way behind the broader JS ecosystem in terms of tooling and my bet is that it will continue to lag further behind. That’s the main reason I don’t advocate for it because you are always going to be swimming against the current.

Anyways. don’t take my word for it. Give it a go and decide for yourself.

zcaudate · August 11, 2022, 6:20pm

JS objects are also much more flexible than Java. Java objects are closed, you cant really access them except through the specific of their interfaces. So having an intermediate representation through a more flexible datastructure was the reason why there are so many useful clj wrappers.

You don’t really need a cljs wrapper for native js libs because JS objects behave as a typical mutable hash map. The more useful cljs libraries are the ones for control flow semantics - ie, dealing with promise chaining and callbacks. I don’t find the control flow libraries as useful on JVM clojure as you can get a pretty good run just with CompletableFuture.

If you use immutable datastructures for your cljs app, chances are that you’ll also be using clj->js and js->clj calls a lot. I haven’t done the benchmarking for how much time it takes. but it just doesn’t really sit right with me, especially if its just marshalling data around on the SAME runtime. Not to mention that the clojure vars are all global. It’s a just a little bit dirty IMO.

So:

the pros are that it’s a lisp, there is a repl and there is an interop. Those are big pros
the cons are that immutability by default does not play well with native libraries, bringing in a whole new runtime, and the global vars.

There is core.async which unifies control flow semantics on both JVM and JS. This is the reason so many people use core.async and rave about transducers. Call me stupid but I have always found it impossible to debug anything to do with core.async. So for me, unification of the two runtimes is out and I treat CLJS and CLJ as two completely different languages.

I rely exclusively on promises for JS concurrency. Promises are not part of clojure.core so to me, even though the semantics are similar, they are two completely different languages.

zcaudate · August 11, 2022, 6:48pm

Another reason i stopped maintaining the repo was that cljs took quite a few new ideas i had in the purnam and put them into cljs, making it pretty much redundant.

At the time, cljs was really hard to use. It was impossible to create native objects in cljs. The #js macro didn’t exist. i created an issue on the clojurescript repo to see it some of the work i did on creating native objects could be merged. the reply was: Nah, it doesnt belong in the core.

About 8 months later, the #js macro was introduced and was marketed as the new awesome feature. It was pretty annoying.

mdiin · August 11, 2022, 6:56pm

Thank you for your insightful replies. I did evaluate Clojurescript when it came out for use in a product, but my team and I decided on plain JS React at the time, even though reagent and reframe were emerging.

What you write makes me less worried, as I remember the hoops you mention with the conversion to and from js data structures, as well as all the fun around cljsjs. Having spent quite a few years with both JavaScript and typescript, I definitely prefer the semantics of Clojure(script).

John_Newman · August 11, 2022, 6:57pm

Honestly, I’d try to get past depending on templates as quickly as possible. They’re useful for getting started, but you’re right that the template story is a little messy right now. And you’ll eventually have to learn how to construct projects by hand anyway, so there’s some utility in just biting the bullet and learning what these templates are doing to construct the project.

One thing I still bang my head against is the mismatch between figwheel.main and calva. Calva wants to inject its own piggieback and that doesn’t line up with most of the docs you’ll see on how to set up a figwheel project. I still run into difficulty reconciling these two different methods.

zcaudate · August 11, 2022, 11:30pm

Writing lisp is better than not writing lisp. Lisp code makes things a lot more maintainable. I just feel that it’s not worth bringing in a whole bunch of dependencies on top of the npm dependencies and then adding in that extra build step.

A mutable, clojure compatible lisp that compiled to JS is smaller and will allow simplified JS interop. It’s a shame that there haven’t been development in this area and we remain stuck with CLJS as the best available option.