Generating ES Modules (Browser, Deno, ...)

ECMAScript Modules (also known as ES Modules or just ESM) were introduced quite a while ago but never were usable with CLJS in a convenient way. A new :target :esm in shadow-cljs attempts to change that by bundling the generated code into standard-compliant ESM files.

ESM is important since CommonJS <-> ESM interop is still horrible and hopefully all the JS world will move to that standard at some point as it fixes many of the ugly issues. Its not perfect either but miles better than all the older alternatives (eg. CommonJS, AMD, UMD, …). Support is good in most modern Browsers and even new engines like deno. This is a good thing. The sooner everything moves to ESM the better.

In shadow-cljs this is sort of a more modern version of the old :npm-module target and cleans up a bunch of issues that target had. I’ll keep :npm-module around but if you can you can consider it deprecated. It always had a bunch of issues that I hope to address with :target :esm.

Generating ESM with shadow-cljs

With 2.10.10 it is now possible to generate “almost” standards-complient ESM output. I say “almost” because it is technically cheating in development. release builds are fully :advanced optimized and compliant as they don’t leak any globals or do any other kinds of eval tricks.

In development builds however it still exports everything into the global scope via globalThis. It still uses the regular module-loading mechanism but no attempt is made to isolate the code to these modules. Basically you get the same result you get now via :browser builds where everything is global in development builds anyways. This is sort of required to keep our hot-reload or REPL functionality. There are still some limits to that but it works to some extent.

Config

The :target :esm outputs one or more ESM output files configure via :modules. Basically all your builds namespaces are grouped together in these modules and code-split if needed. You start by defining which :exports a module should have.

{...
 :builds
 {:app
  {:target :esm
   :output-dir "public/js"
   :modules {:demo {:exports {foo my.app/foo}}}}}  
(ns my.app)

(defn foo []
   (js/console.log "hello world from CLJS"))

(defn bar [] "bar")

This config generates a demo.js in the public/js output directory.

It can be loaded in modern browsers via

<script type="module">
  import { foo } from "/js/demo.js";
  foo();
</script>

Note that only the declared :exports in the build config are accessible. my.app/bar would not be accessible (and also removed by :advanced since it is dead code). The usual ^:export metadata hint is not yet supported and the build config is the only place to declare exports.

Note that often exports won’t be required at all if you just want to run code directly when it is loaded. For this you may use the :init-fn option as you would in a :browser build.

{...
 :builds
 {:app
  {:target :esm
   :output-dir "public/js"
   :modules {:demo {:init-fn my.app/foo}}}}  
<script type="module" src="/js/demo.js"></script>

This just runs (my.app/foo) when the file is loaded. You may also code-split similarly to the :browser target where each file can have its own :exports.

  {:target :esm
   :output-dir "public/js"
   :modules
   {:shared {}
    :a
    {:exports
     {default demo.esm.a/foo
      foo demo.esm.a/foo}
     :init-fn demo.esm.a/init
     :depends-on #{:shared}}
    :b
    {:exports
     {default demo.esm.b/bar}
     :depends-on #{:shared}}}}

Note that :depends-on is important as is controls how you code is split and how code is moved around in case it is used in more than one module. Just like :browser you can declared an “empty” :shared module that will act as a catch-all and group together shared namespaces (eg. cljs.core).

Importing Code

One of the most-important code aspects of ESM is the flexibility and strictness at the same time when it comes to accessing library code from elsewhere on the web or your server.

By default shadow-cljs will still load and bundle your code from npm but you can also completely remove npm dependencies by importing directly from the URL. In this case the code will not be bundled at all and will instead be loaded by the runtime.

The regular require will bundle code from node_modules and include them in the build output.

(ns my.app
  (:require ["preact" :as preact]))

If you instead use a full URL from a CDN

(ns my.app
   (:require ["https://cdn.pika.dev/[email protected]^10.0.0" :as preact]))

then shadow-cljs will make no attempt at bundling this and just keep the URL intact and let the runtime load it. While I wouldn’t recommend doing this for most things it is sometimes convenient to do.

You may also access other ESM modules from your own server like this by using "esm:/somewhere-else/foo.js". Just using /somewhere-else/foo.js would attempt to load that from from the classpath and bundle it at compile time.

Dynamic Import

import in ESM usually is static compile time info and cannot be used dynamically (much like the CLJS ns form) but there is also a dynamic import() function variant that can be called at runtime to load code. This can either be used to load your own :modules dynamically or any other source as long as its ESM.

Due to some limitations in the current Closure Compiler we cannot use (js/import "./foo.js") directly but I created simple enough helper ns that still lets us do this.

(ns my.app
  (:require [shadow.esm :refer (dynamic-import)]))

(defn foo []
   (-> (dynamic-import "https://cdn.pika.dev/[email protected]^10.0.0")
       (.then (fn [mod] ...))

shadow.esm/dynamic-import just takes one string argument and returns a Promise.

Deno

Deno is a “new” JS runtime/platform that aims to fix some of the bad decisions made in node. It uses ESM as the default and as such the generated code by :target :esm just works. Deno also uses URLs to access any other dependencies so that also just works.

shadow-cljs compile and shadow-cljs release already work fine. shadow-cljs watch however still needs some fixes so hot-reload and the REPL do not curently work with Deno. If there is enough interest I’ll see if I can sort that out.

Conclusion

I think this is an important step for keeping CLJS current in the JS world and this also provides an easy and convenient way to use “foreign” code directly. The output will be fully optimized by Closure :advanced optimizations and the declared fixed :exports ensure that the code still remains accessible from the outside.

Note that all of this is done strictly as a bundling step by shadow-cljs. I did not modify the CLJS compiler to emit ES6 code directly as that would be a lot more work. It would still be interesting to do this at some point. Doing it by “cheating” a little however ensures that all code we have today should pretty much just work. If :advanced worked before it’ll also work for this.

This should still be considered “alpha” since I just finished writing this today and I didn’t test too much yet. It appears to work fine Chrome, Firefox, Edge and Deno. Didn’t test anywhere else yet. Also no clue if this works if the output is then used and re-bundled by webpack or so. In theory it should just work but no clue how the globalThis abuse survives further bundling. Of course bundling things again is kinda pointless and should be avoided.

FWIW the somewhat recent release of snowpack renewed my interest in finally getting a good ESM support story for CLJS. Definitely worth a mention here.

Lots of interesting new possiblities with this overall. Excited to explore this further.

24 Likes

Oops, turns out 2.10.10 didn’t generate the output directory properly and crashed when trying to write files to the non-existant dir. Fixed in 2.10.11. Thanks @jiyinyiyong for the report.

And I added a quick demo for trying out :smiley: :

Deno’s news is good news. :wink:

Maybe I can finally create isolated web components in cljs and using them in javascript projects (or any web project actually). But I still need to understand how I declare the Reagent/React dependency, every component would depend on it but it should load it only once.

You really don’t want to do that with CLJS and :esm is not meant to do this. The whole point of Google Closure :advanced optimizations is to optimize your whole program so it can throw away everything that isn’t used. So at the very least you must compile all CLJS together. Never include multiple different CLJS ouputs in the same page. They won’t be compatible with each other and each will include their version of cljs.core making everything bigger than it needs to be.

Taking react/react-dom or other JS deps out of the equation so it can be shared with JS is possible but not yet implemented cleanly. Basically you can (:require ["esm:react" :as react]) which would leave it as import * as ... from "react"; in the actual output and make no attempt at bundling it. With snowpack that would be (:require ["esm:/web_modules/react.js" :as react]).

It reminds me of the difference between ES6 modules and Closure Compiler. As today major platforms have added support for ES6 import/export, now we have chance to use browser native module system, which opens possibilities of “no bundling” deployment. And with help of HTTP2 it’s no longer a huge issue to have tens of small files. While there is still tradeoffs but at least it’s a considerable solution for JavaScript world today.

The difference in ClojureScript is we do need to bundle code, and then optimize. Just difference. However it brings confusions when someone attempts to combine two things together. And I’m don’t know if that could be a good idea.

And speaking of snowpack, I heard more about vite since the author shared quite some ideas on Twitter and in our WeChat group. Webpack is heavy. Vite provides a really fast environment for these people who want to play with lightweight web apps(or especially Vue apps),. No cold start compilation overhead, instant code replacement, makes it really quick.

I did follow vite and it is nice to see other JS bundlers besides webpack trying new ideas.

I do however not agree with the entire premise of its design. Sure fast startup is nice but doing everything to optimize for that is pointless. Assume you work on something more than 15 minutes, it doesn’t matter if its starts in 129ms or 3sec. Loading thousands of files separately on page load is too slow to be practical. ESM makes that a bit better but its still not great. Thats why I added :loader-mode :eval in shadow-cljs. There is a balance to be had here and the more files you have the less useful it is to keep them separate. The slow “bundling” step can be improved with better caching which shadow-cljs already does and webpack will do with v5 too.

I’m also absolutely horrified by the now “common” JS idiom and code layout of one file per function. This is ridiculous in my opinion and not how I want to write code. So if you write more code into one file you do want Closure style DCE to remove everything that wasn’t used. Of course this is all completely subjective and some people may prefer that style.

Ultimately you always want a final bundle step and ESM doesn’t change that.

What I like about snowpack is snowpack install. Couldn’t care less about the other stuff it now does. This is interesting because you bundle everything once, just like npm install and after that you can even keep the files in version control. No need to ever run it again until you change dependencies. shadow-cljs could even consume those files instead of npm directly and either load them via ESM or bundle normally.

I see snowpack as a “take random madness from npm and turn it into somewhat sane ESM”. Of course that becomes less useful once more packages actually ship as pure ESM but its a good step in that direction.

1 Like