[Poll] Reworking the Launcher

[thheller]

I’m reworking the “Launcher” for shadow-cljs and would be interested to hear what people think about my plans. I’m undecided on which way to go since there are several options each with their own pros/cons.

The launcher is used to download maven :dependencies and starting the JVM with the proper classpath. shadow-cljs basically consists of 3 separate parts:

1. Clojure Library doing all the work
2. The Standalone Launcher
3. The npm package providing the shadow-cljs CLI command

shadow-cljs supports 3 different launchers currently: lein via :lein+project.clj, tools.deps clojure via :deps+deps.edn and its standalone version. lein and tools.deps are not affected by this work and will stay exactly the same as they are now. You can even skip 2,3 entirely and just use the Clojure Library directly.

Issue #1

Previously the launcher was distributed via the shadow-cljs-jar npm package which the shadow-cljs npm package just depended on. This meant that the standalone launcher was always downloaded even when actually using the other launchers.

Issue #2

Due to the way npm operates the launcher would be downloaded for each project separately although it rarely changes and each project could use a shared version instead. 5 projects currently means 5 copies which is not the end of the world but could be improved.

Issue #3

The launcher was very limited: download dependencies and build a classpath. This is exactly the same way tools.deps and lein operate. A standalone “uberjar” is used to complete that job and then a secondary JVM is launched to run the actual application. This works well but means that sometimes 2 JVMs are started. I also want to get to the point where :dependencies can be added dynamically without restart which would require duplicated work in the Clojure Library + Launcher.

Plan A

Keep everything the way it is.

Plan B

Use the shadow-cljs JS script to download the launcher independently from npm. The launcher could be distributed via Github or so and shared between projects. It would only be downloaded when required, meaning lein or tools.deps users wouldn’t. This is also how tools.deps and lein work basically.

Plan C

Plan B + make the launcher a bit smarter to enable dynamic classpath additions and such.

Plan D

Actually bundle shadow-cljs with the launcher. Saves the first initial download step, can do more optimizations for startup. Could investigate GraalVM native-image for nearly instant startup. Downside is that shadow-cljs has far more frequent releases than the launcher. So “upgrading” would mean a larger download.

Bonus Level

Actually create a completely standalone executable. One that you could double-click to launch and even get an actual GUI application at some point. Terminal is so 20 years ago. I think we can do better and I personally want that. CLI version will stay for CI and such but developer experience would be much improved if you could click a thing instead of running 3 separate terminals or so. IMHO,YMMV. For those that remember there used to be cuttle for CLJS and the Vue world has a pretty badass UI too. There is also guppy which is fairly new but looks interesting too.

The final Question

I’m currently undecided whether to go with C or D and want the Bonus Level

Which would you prefer: A more frequent larger download (~40MB) or a less frequent ~20MB download plus a separate more frequent ~20MB download? Am I overthinking this?

If you use lein to deps nothing changes for you but you’ll miss out on some of the AOT optimizations (4sec startup vs 9sec startup). shadow-cljs will stay a Clojure Library primarily and this is just about finding the most optimal way to launch it.

• I use lein and don’t care
• I use tools.deps and don’t care
• Plan A - screw lein/tools.deps users
• Plan B - do what is known to work
• Plan C - optimize my bandwidth
• Plan D - I got the bandwidth
• Gimme the GUI already!

[daviwil]

I think that having the JS bits of shadow-cljs be shipped via npm and the .jar downloaded from the corresponding version’s GitHub release makes sense. I’d want the ability to lock my project to a specific version of the shadow-cljs compiler and that’d require using the same version of the .jar. You might need a folder in the user’s home directory that can have more than one version of the .jar installed side-by-side.

Regarding the standalone UI, I think that’s a “nice to have” but personally I don’t need it. I’m happy to just add the shadow-cljs dependency to my package.json file and get going with a hand-edited .edn config. Though I agree, it could be useful for other folks getting started, especially if they aren’t familiar with existing ClojureScript toolchains!

[lilactown]

First off, my priorities would be:

1. Ease of project maintenance
2. Faster startup
3. Dynamic dependency loading
4. Ease of setup

In roughly that order. I don’t really care that much about bandwidth TBH but it’s nice to optimize if it’s low hanging fruit.

I’m going to throw an idea out there and see if it’s worth anything: What if shadow-cljs started to adopt tools.deps as its default launcher (EDIT: while still maintaining the NPM launch script for ease of use)? I think that this would provide several benefits:

1. Potentially remove a bunch of code from shadow-cljs (downloading deps + doing classpath stuff that tools.deps already does)

   • Side note: git deps!

2. Dynamic classpath loading for free… eventually.

3. If it produces a deps.edn file, would serve as an entry point to the rest of the deps.edn / tools.deps ecosystem for JS developers. Support could also be routed to the larger tools.deps channels and take some of the strain off of you for the above tasks.

4. The only dependency now is the tools.deps CLI, which many people already have installed.

5. Could help us take advantage of other tooling that’s starting to coalesce around tools.deps and deps.edn.

Some downsides and risks I see:

1. Current lack of Windows support for tools.deps CLI

   • Probably makes it a non-starter in the immediate future, since I know you develop on Windows

2. We become beholden to the pace and feature set of tools.deps - dynamic class loading is not yet in master, there are bugs

3. There might be some impedance mismatch between tools.deps / deps.edn and what shadow-cljs / shadow-cljs.edn can/wants to do that we don’t yet know.

My personal philosophy is to always try and adopt the status-quo w.r.t. libraries and tooling when possible, which is why I think that adopting tools.deps is worth looking into. Take my opinion as you will

I think that shadow-cljs is a really great point of integration between the JS and CLJ(S) world, and could serve as a tool to provide better interoperability and bridge those two toolsets and ecosystems.

[thheller]

The launcher branch I have currently going in the shadow-cljs project does use the tools.deps.alpha library for dependency resolution. I replaced pomegranate to gain access to things such as git deps. It will however not use the tools.deps CLI. Mostly because I do not want to assume that it is already installed since JS devs won’t have that.

Dynamic classpath was possible with pomegranate as well so there is nothing that tools.deps adds in that department. The tricky part is dealing with classloader issues but since we are developing ClojureScript we can make certain assumptions which make this much less of an issue.

The problem with deps.edn or tools.deps in general is that they are optimized for Clojure JVM development. All the more complex options like aliases simply make little sense in pure ClojureScript projects since your build config already expresses all of the important bits.

All tooling around deps.edn was always accessible to shadow-cljs since all tools.deps does is build a classpath and call a main function. shadow-cljs could always do that. Any clj based tool that doesn’t rely on git-deps can be used today and git-deps will soon work too.

Since shadow-cljs must remain a Clojure Library anyways it pretty much is just about how much the shadow-cljs CLI should/could optimize the experience purely for CLJS development. The server-mode is one such optimization. The newer AOT bundle is another one. The standalone GUI could be something on top. Embedding will always be an option but just take the recent AOT addition as an example: It reduced startup time by more than 50%. When running in embedded more alongside a CLJ server the potential for conflicts due to AOT is incredibly high so I would never recommend doing it there.

The UI is going to be a Web UI anyways that you can just open in the browser. The standalone GUI would just open that Web UI in a Webview. The Web UI should also be easily embeddable in editors that supports webviews (eg. Cursive, Atom, VSCode, …).

I’m sort of getting increasingly convinced that embedding CLJS tools in a CLJ server development env is actually limiting the things we can actually do. CLJS gains nothing from running in your CLJ server and neither does CLJ from CLJS. Sure it is nice to have to only launch one JVM that does everything but it also means that you have to restart your CLJ server to add a CLJS dependency. Dynamic classpath is much harder in CLJ than it is for CLJS.

[arichiardi]

My 2c on this: I do not care of wasting bandwidth, disk space if this bring me:

• npm pinning - no globally installed tools that needs to be added in CI
• nRepl support for editors

Personally the webui would be good to me as long as the tool can talk to the editor - I would never ever sacrifice what you are doing now with nRepl for a webui. I would say I moved to shadow for npm sanity and nRepl support. I think you remember I even had a shadow mode for emacs that was talking to the socket REPL, but that was basically replicating a lot of what cider is doing.

My team is mixed though so I can tell you also that my colleagues that are using shadow in the (2) terminals with Intellij never ever complain.

About webui, one for testing like https://github.com/bhauman/cljs-test-display instead of a terminal would be actually very nice.

[thheller]

Not to worry: nREPL support will stay the way it is and will not be affected by the changes in any way. Socket REPL will also stay. In general all editor related things will remain or will be enhanced.

Running tests, browsing documentation, showing detailed build info and more stuff along those lines are the reasons I want the Web UI since its just too limited to do this with only text in the terminal (or emacs buffer).

Web UI is only complementary to the CLI since that must always be able to run all the important stuff. Automating build stuff, deployment, testing, CI must still be possible without being required to click something in a UI.

[jiyinyiyong]

The launcher could be distributed via Github or so and shared between projects.

As mentioned on Issues that downloading GitHub files from China can be really slow occasionally. I’m against this, but we would need a mirror server which has enough bandwidth from China.

Is there any statistics on the count Chinese users of shadow-cljs? We also don’t have Google Analytics on home page and the Manual.

Could investigate GraalVM native-image for nearly instant startup. Downside is that shadow-cljs has far more frequent releases than the launcher. So “upgrading” would mean a larger download.

Instant startup means more to me than size to download.

One that you could double-click to launch and even get an actual GUI application at some point. Terminal is so 20 years ago. I think we can do better and I personally want that.

Terminal is old, true. However I don’t have quite much interactions with watching server directly myself. I start shadow-cljs from npm command and just leave it running and throwing errors in HUB so I’m not really convinced a GUI would bring me better experiences. Would you give us a demo recording how you use a GUI and publish on Youtube?

[thheller]

@jiyinyiyong

downloading GitHub files from China can be really slow occasionally

That sucks. I could continue publishing the shadow-cljs-jar package that you could install manually on top to get the launcher from npm rather than github. Or we find a good Chinese mirror? Could you host the launcher file on shadow-cljs.org?

Don’t know how many Chinese users there are. Not using any tracking since I myself use an Ad Blocker and I always assume that most tech people do. So the stats would not be very accurate anyways. I do think that there are quite a few Chinese users hower, just based on the github stars, so I definitely don’t want anything to get worse for them.

Graal images will have to wait until a final release is available with Windows support. I’m also not sure if its even possible to use custom classloaders at all with native-image. Might not be.

@jiyinyiyong

Would you give us a demo recording how you use a GUI and publish on Youtube?

The GUI is just in my head and I only implemented a few experiments so far. I’ll probably start working on something more concrete soon.

[jiyinyiyong]

Or we find a good Chinese mirror? Could you host the launcher file on shadow-cljs.org ?

The server for shadow-cljs.org is located in Hongkong I think it’s fast enough accessing from China. We can use it at first. Maybe we can add a script detecting latest releases on GitHub and download them serving as files.

I have contact with on several shadow-cljs user in WeChat group. Not sure also. But I would definitely try recommending shadow-cljs and ClojureScript to more Web developers in China.

[thheller]

FWIW I realized that my outlined plans actually work incrementally.

Plan B is available as of 2.5.0. C just requires minor changes and adding a few API functions to actually load deps dynamically. D will be an opt-in option later on.

[thheller]

FWIW I just concluded a few experiments using appcds for shadow-cljs standalone and it does reduce startup time a bit but requires an addition 80MB(!!) download. So I do not think that its worth using just yet. 50MB for the shadow-cljs uberjar + 80MB for the appcds.cache is kind of ridiculous.

Results:

plain uberjar

$ time java -jar shadow-cljs-2.5.1-standalone.jar clj-eval 1
1

real    0m3.062s
user    0m7.906s
sys     0m0.938s

with appcds active

$ time java -XX:+UnlockCommercialFeatures -XX:+UseAppCDS -Xshare:on -XX:SharedArchiveFile=appcds.cache -jar shadow-cljs-2.5.1-standalone.jar clj-eval 1
1

real    0m1.778s
user    0m4.969s
sys     0m0.672s

Shoutout to this excellent post regarding clojure + appcds.

I may revisit this option in the future but for now it does not seem worth the effort.

[jiyinyiyong]

Surprise to JavaScript people, a Java CLI tool could be as large as 100+M.

[thheller]

JS people just aren’t aware of the size of their tools and don’t seem to care about anything since npm hides it well.

I just did a npx create-react-app dummy-cra and the created directory ended up with 117MB in size. I did not add any additional dependencies so this is the baseline.

$ du -hd 1 dummy-cra/
116M    dummy-cra/node_modules
8.0K    dummy-cra/public
12K     dummy-cra/src
117M    dummy-cra/

I would call this equivalent to the uberjar download size and compared the jar is actually small. Its just the appcds.cache that takes it too far.

[jiyinyiyong]

React workflows contains more than code, like images and CSS files. It’s in development mode. And it’s for building user interface. I don’t think a Node.js CLI tool without interfaces would get this huge.

[thheller]

Point was that a typical JS development environment will easily pass 100MB. Comparing a single CLI script is pointless since you must account for its dependencies as well. uberjar contains all of its deps, the normal shadow-cljs library is ~800KB.

Anyways, we can agree that file sizes are too big regardless. Unfortunately its the optimizations (AOT, uberjar) that makes it large and its a trade-off worth making IMHO.

[jiyinyiyong]

So uberjar is for development? I also compared it with websites, which is uglified and delivered over network. 10M is already too large for network transfering.

[chrisdavies]

I’m a fan of reproducible builds, and self-contained projects (e.g. I delete the project directory, and all of its dependencies go away). I prefer that to slightly improved disk space, etc.

I’ll second the vote to prioritize maintainability, and a close second would be to do what fits with the NPM / JS ecosystem unless it’s egregious, as that seems to be a key target of this project.

[thheller]

We may have a communication problem here. I’m absolutely lost to which file sizes you are referring to here or which problem you are referring to here.

When I say uberjar I’m referring to a standalone shadow-cljs distribution, which is basically shadow-cljs bundled with all its dependencies as a .jar file. This file would be 50MB or more. It only refers to the tool itself though and has absolutely no relation to the size of the builds it produces. Not a single bit of that will end up in your build.

I don’t understand where you are making the comparison to a website? Yes, 50MB would be way too large for a website but that is not what we are talking about here.

[jiyinyiyong]

Have to stop here. I’m not familiar with uberjars and the usages in Java world.