Generating ES Modules (Browser, Deno, ...)

ECMAScript Modules (also known as ES Modules or just ESM) were introduced quite a while ago but never were usable with CLJS in a convenient way. A new :target :esm in shadow-cljs attempts to change that by bundling the generated code into standard-compliant ESM files.

ESM is important since CommonJS <-> ESM interop is still horrible and hopefully all the JS world will move to that standard at some point as it fixes many of the ugly issues. Its not perfect either but miles better than all the older alternatives (eg. CommonJS, AMD, UMD, …). Support is good in most modern Browsers and even new engines like deno. This is a good thing. The sooner everything moves to ESM the better.

In shadow-cljs this is sort of a more modern version of the old :npm-module target and cleans up a bunch of issues that target had. I’ll keep :npm-module around but if you can you can consider it deprecated. It always had a bunch of issues that I hope to address with :target :esm.

Generating ESM with shadow-cljs

With 2.10.10 it is now possible to generate “almost” standards-complient ESM output. I say “almost” because it is technically cheating in development. release builds are fully :advanced optimized and compliant as they don’t leak any globals or do any other kinds of eval tricks.

In development builds however it still exports everything into the global scope via globalThis. It still uses the regular module-loading mechanism but no attempt is made to isolate the code to these modules. Basically you get the same result you get now via :browser builds where everything is global in development builds anyways. This is sort of required to keep our hot-reload or REPL functionality. There are still some limits to that but it works to some extent.

Config

The :target :esm outputs one or more ESM output files configure via :modules. Basically all your builds namespaces are grouped together in these modules and code-split if needed. You start by defining which :exports a module should have.

{...
 :builds
 {:app
  {:target :esm
   :output-dir "public/js"
   :modules {:demo {:exports {foo my.app/foo}}}}}  
(ns my.app)

(defn foo []
   (js/console.log "hello world from CLJS"))

(defn bar [] "bar")

This config generates a demo.js in the public/js output directory.

It can be loaded in modern browsers via

<script type="module">
  import { foo } from "/js/demo.js";
  foo();
</script>

Note that only the declared :exports in the build config are accessible. my.app/bar would not be accessible (and also removed by :advanced since it is dead code). The usual ^:export metadata hint is not yet supported and the build config is the only place to declare exports.

Note that often exports won’t be required at all if you just want to run code directly when it is loaded. For this you may use the :init-fn option as you would in a :browser build.

{...
 :builds
 {:app
  {:target :esm
   :output-dir "public/js"
   :modules {:demo {:init-fn my.app/foo}}}}  
<script type="module" src="/js/demo.js"></script>

This just runs (my.app/foo) when the file is loaded. You may also code-split similarly to the :browser target where each file can have its own :exports.

  {:target :esm
   :output-dir "public/js"
   :modules
   {:shared {}
    :a
    {:exports
     {default demo.esm.a/foo
      foo demo.esm.a/foo}
     :init-fn demo.esm.a/init
     :depends-on #{:shared}}
    :b
    {:exports
     {default demo.esm.b/bar}
     :depends-on #{:shared}}}}

Note that :depends-on is important as is controls how you code is split and how code is moved around in case it is used in more than one module. Just like :browser you can declared an “empty” :shared module that will act as a catch-all and group together shared namespaces (eg. cljs.core).

Importing Code

One of the most-important code aspects of ESM is the flexibility and strictness at the same time when it comes to accessing library code from elsewhere on the web or your server.

By default shadow-cljs will still load and bundle your code from npm but you can also completely remove npm dependencies by importing directly from the URL. In this case the code will not be bundled at all and will instead be loaded by the runtime.

The regular require will bundle code from node_modules and include them in the build output.

(ns my.app
  (:require ["preact" :as preact]))

If you instead use a full URL from a CDN

(ns my.app
   (:require ["https://cdn.pika.dev/preact@^10.0.0" :as preact]))

then shadow-cljs will make no attempt at bundling this and just keep the URL intact and let the runtime load it. While I wouldn’t recommend doing this for most things it is sometimes convenient to do.

You may also access other ESM modules from your own server like this by using "esm:/somewhere-else/foo.js". Just using /somewhere-else/foo.js would attempt to load that from from the classpath and bundle it at compile time.

Dynamic Import

import in ESM usually is static compile time info and cannot be used dynamically (much like the CLJS ns form) but there is also a dynamic import() function variant that can be called at runtime to load code. This can either be used to load your own :modules dynamically or any other source as long as its ESM.

Due to some limitations in the current Closure Compiler we cannot use (js/import "./foo.js") directly but I created simple enough helper ns that still lets us do this.

(ns my.app
  (:require [shadow.esm :refer (dynamic-import)]))

(defn foo []
   (-> (dynamic-import "https://cdn.pika.dev/preact@^10.0.0")
       (.then (fn [mod] ...))

shadow.esm/dynamic-import just takes one string argument and returns a Promise.

Deno

Deno is a “new” JS runtime/platform that aims to fix some of the bad decisions made in node. It uses ESM as the default and as such the generated code by :target :esm just works. Deno also uses URLs to access any other dependencies so that also just works.

shadow-cljs compile and shadow-cljs release already work fine. shadow-cljs watch however still needs some fixes so hot-reload and the REPL do not curently work with Deno. If there is enough interest I’ll see if I can sort that out.

Conclusion

I think this is an important step for keeping CLJS current in the JS world and this also provides an easy and convenient way to use “foreign” code directly. The output will be fully optimized by Closure :advanced optimizations and the declared fixed :exports ensure that the code still remains accessible from the outside.

Note that all of this is done strictly as a bundling step by shadow-cljs. I did not modify the CLJS compiler to emit ES6 code directly as that would be a lot more work. It would still be interesting to do this at some point. Doing it by “cheating” a little however ensures that all code we have today should pretty much just work. If :advanced worked before it’ll also work for this.

This should still be considered “alpha” since I just finished writing this today and I didn’t test too much yet. It appears to work fine Chrome, Firefox, Edge and Deno. Didn’t test anywhere else yet. Also no clue if this works if the output is then used and re-bundled by webpack or so. In theory it should just work but no clue how the globalThis abuse survives further bundling. Of course bundling things again is kinda pointless and should be avoided.

FWIW the somewhat recent release of snowpack renewed my interest in finally getting a good ESM support story for CLJS. Definitely worth a mention here.

Lots of interesting new possiblities with this overall. Excited to explore this further.

34 Likes

Oops, turns out 2.10.10 didn’t generate the output directory properly and crashed when trying to write files to the non-existant dir. Fixed in 2.10.11. Thanks @jiyinyiyong for the report.

And I added a quick demo for trying out :smiley: :

1 Like

Deno’s news is good news. :wink:

Maybe I can finally create isolated web components in cljs and using them in javascript projects (or any web project actually). But I still need to understand how I declare the Reagent/React dependency, every component would depend on it but it should load it only once.

You really don’t want to do that with CLJS and :esm is not meant to do this. The whole point of Google Closure :advanced optimizations is to optimize your whole program so it can throw away everything that isn’t used. So at the very least you must compile all CLJS together. Never include multiple different CLJS ouputs in the same page. They won’t be compatible with each other and each will include their version of cljs.core making everything bigger than it needs to be.

Taking react/react-dom or other JS deps out of the equation so it can be shared with JS is possible but not yet implemented cleanly. Basically you can (:require ["esm:react" :as react]) which would leave it as import * as ... from "react"; in the actual output and make no attempt at bundling it. With snowpack that would be (:require ["esm:/web_modules/react.js" :as react]).

It reminds me of the difference between ES6 modules and Closure Compiler. As today major platforms have added support for ES6 import/export, now we have chance to use browser native module system, which opens possibilities of “no bundling” deployment. And with help of HTTP2 it’s no longer a huge issue to have tens of small files. While there is still tradeoffs but at least it’s a considerable solution for JavaScript world today.

The difference in ClojureScript is we do need to bundle code, and then optimize. Just difference. However it brings confusions when someone attempts to combine two things together. And I’m don’t know if that could be a good idea.

And speaking of snowpack, I heard more about vite since the author shared quite some ideas on Twitter and in our WeChat group. Webpack is heavy. Vite provides a really fast environment for these people who want to play with lightweight web apps(or especially Vue apps),. No cold start compilation overhead, instant code replacement, makes it really quick.

I did follow vite and it is nice to see other JS bundlers besides webpack trying new ideas.

I do however not agree with the entire premise of its design. Sure fast startup is nice but doing everything to optimize for that is pointless. Assume you work on something more than 15 minutes, it doesn’t matter if its starts in 129ms or 3sec. Loading thousands of files separately on page load is too slow to be practical. ESM makes that a bit better but its still not great. Thats why I added :loader-mode :eval in shadow-cljs. There is a balance to be had here and the more files you have the less useful it is to keep them separate. The slow “bundling” step can be improved with better caching which shadow-cljs already does and webpack will do with v5 too.

I’m also absolutely horrified by the now “common” JS idiom and code layout of one file per function. This is ridiculous in my opinion and not how I want to write code. So if you write more code into one file you do want Closure style DCE to remove everything that wasn’t used. Of course this is all completely subjective and some people may prefer that style.

Ultimately you always want a final bundle step and ESM doesn’t change that.

What I like about snowpack is snowpack install. Couldn’t care less about the other stuff it now does. This is interesting because you bundle everything once, just like npm install and after that you can even keep the files in version control. No need to ever run it again until you change dependencies. shadow-cljs could even consume those files instead of npm directly and either load them via ESM or bundle normally.

I see snowpack as a “take random madness from npm and turn it into somewhat sane ESM”. Of course that becomes less useful once more packages actually ship as pure ESM but its a good step in that direction.

2 Likes

BTW, eval loader mode made hot-reload work while I was developing a plug-in in vscode - it’s the only way to make hot reload work on the webview, so thanks for that :smile:

Let me see if I got it, what you are telling me is that if I want to export web components, each one distributed as a singular package in npm, even if all of them have a dependency in cljs.core and react, these dependencies would be too heavy (because they wouldn’t pass by the compiler advanced mode) and the components wouldn’t work if I define two of them in the same page (maybe I didn’t understand the “compatible with each other” very well), is that right?

But I could create like a “library of components” declaring the common dependencies in the shared module, so they would pass by the compiler advanced mode, right?

I’m not sure I understand your question but as a general rule of thumb the JS users should only ever include output from a single CLJS build in their projects. You may use :modules to split that build into multiple files and allow the user to include single or multiple modules.

As soon as the JS users include output from multiple CLJS builds they will have duplicated code and things become too large to be practical quickly.

2 Likes

Hey Thomas! I was wondering what the current state of the :esm target is with shadow-cljs but maybe also generally with the ClojureScript compiler?

Snowpack seems quite interesting and the ESM target appears to be a nice way to bridge the divide between ClojureScript and existing JS tooling. (Or at least it looks like that from my superficial level of understanding. :slight_smile:)

1 Like

I’m waiting for some things in the Closure Compiler to land before making any other changes.

As far as shadow-cljs is concerned the support is mostly done but I haven’t used it in a real project so there might be issues I’m not aware of. As far as CLJS proper is concerned I don’t know. I’m not aware of anyone working on it.

Eventually I’d like to write a CLJS variant that actually just outputs “modern” JS (ES2018+ or so) instead of the old and “dead” Closure JS. That would however be a completely new compiler since we really can’t retrofit that into the current one. There are drawbacks to doing this though so this is very low priority for me.

2 Likes

Why do you say “dead”? Still seems alive and well maintained by Google. GitHub - google/closure-compiler: A JavaScript checker and optimizer. I see a commit just 1 hour ago from this writing.

The Closure Compiler is alive and well, that is not what I meant by Closure JS.

Closure JS is the “old” module format that used goog.provide and goog.require to do namespacing related things. That style is dead and the Closure Library is slowly migrating to the somewhat newer goog.module style or ESM directly. It is still well supported and not going anywhere but the Closure Compiler is at a point where the newer formats are preferred. The CLJS compiler currently emits that “old” style which is pretty much the reason we sometimes have issues integrating with other JS tools since most other JS tools (besides the Closure Compiler) don’t understand that format properly.

2 Likes

Oh I see, I didn’t know Closure had itself slowly migrated to ESM.

Interesting, are you aware of any further discussions/tickets weighing the pros and cons of emitting goog.module or ESM? Briefly searched Jira but didn’t find much.

It would be ESM only if anything. goog.module doesn’t buy us anything over what we currently have.

Not aware of any “archived” discussions about the topic, talked briefly in #cljs-dev some time ago. If we had pure ESM output basically any JS tool out there would work out of the box. Heck you could even load the output directly in the Browser without bundling. Everything else is pretty much downside since REPL and hot-reload get much harder but it might be worth for some interesting JS-based tools so we don’t have to re-invent everything all the time (eg. storybook).

Closure Compiler is still the best option because of :advanced so I’d never want to give that up but for some tools that just not needed.

1 Like

Hi @thheller,

I see the two Google Closure PRs you mentioned above have been merged and you have recently pushed things to shadow (at least https://github.com/thheller/shadow-cljs/commit/8b7f3964e732bc13a9981ba390908dd90f70f2e0 and https://github.com/thheller/shadow-cljs/commit/7575a7d8533be396cd6a630e6aa36848abebbf0d) related to ESM.

Would you recommend us giving this a spin or are you aware of blocking issues still?

Thank you for your great work as always.

1 Like

Doing what exactly? :browser is still the best option for everything browser based.

The changes that landed mostly don’t affect what shadow-cljs does, just maybe make things a little less hacky. Overall I would still only recommend ESM for targets where no better option exists (eg. Deno). For the browser or node the other targets are still way better.