Cannot import certain npm modules with shadow-cljs

[asiegf]

Hi everyone!
I am using shadow-cljs for my development workflow, and trying to import various npm librairies.

It works great for some like (loadash, shortid) but I get errors with others such as ecpair or crawlee.

for ecpair:

errors in file: .../node_modules/ecpair/src/ecpair.js
{:js-str-offsets [], :js-esm false, :js-imports [], :js-invalid-requires [], :goog-provides [], :js-language "es8", :goog-module nil, :js-warnings [], :resource-name "node_modules/ecpair/src/ecpair.js", :js-requires [], :js-errors [{:line 68, :column 7, :message "'(' expected"}], :goog-requires [], :tag :shadow.build.npm/errors, :uses-global-buffer false}
ExceptionInfo: errors in file: .../node_modules/ecpair/src/ecpair.js
	clojure.core/ex-info (core.clj:4739)
	clojure.core/ex-info (core.clj:4739)
	shadow.build.npm/get-file-info* (npm.clj:519)
	shadow.build.npm/get-file-info* (npm.clj:421)
	shadow.build.npm/get-file-info (npm.clj:556)
	shadow.build.npm/get-file-info (npm.clj:553)
	shadow.build.npm/find-resource (npm.clj:645)
	shadow.build.npm/find-resource (npm.clj:605)
	shadow.build.resolve/find-npm-resource (resolve.clj:110)
	shadow.build.resolve/find-npm-resource (resolve.clj:80)
	shadow.build.resolve/fn--11794 (resolve.clj:202)
	shadow.build.resolve/fn--11794 (resolve.clj:183)
	clojure.lang.MultiFn.invoke (MultiFn.java:243)
	shadow.build.resolve/resolve-string-require (resolve.clj:324)
	shadow.build.resolve/resolve-string-require (resolve.clj:307)
	shadow.build.resolve/resolve-require (resolve.clj:491)
	shadow.build.resolve/resolve-require (resolve.clj:484)
	shadow.build.resolve/resolve-deps/fn--11761 (resolve.clj:58)
	clojure.lang.PersistentVector.reduce (PersistentVector.java:341)
	clojure.core/reduce (core.clj:6747)
	clojure.core/reduce (core.clj:6730)
	shadow.cljs.util/reduce-> (util.clj:47)
	shadow.cljs.util/reduce-> (util.clj:46)
	shadow.build.resolve/resolve-deps (resolve.clj:56)
	shadow.build.resolve/resolve-deps (resolve.clj:40)
	shadow.build.resolve/resolve-string-require (resolve.clj:343)
	shadow.build.resolve/resolve-string-require (resolve.clj:307)
	shadow.build.resolve/resolve-require (resolve.clj:491)
	shadow.build.resolve/resolve-require (resolve.clj:484)
	shadow.build.resolve/resolve-deps/fn--11761 (resolve.clj:58)
	clojure.lang.PersistentVector.reduce (PersistentVector.java:341)
	clojure.core/reduce (core.clj:6747)
	clojure.core/reduce (core.clj:6730)
	shadow.cljs.util/reduce-> (util.clj:47)
	shadow.cljs.util/reduce-> (util.clj:46)
	shadow.build.resolve/resolve-deps (resolve.clj:56)
	shadow.build.resolve/resolve-deps (resolve.clj:40)
	shadow.build.resolve/resolve-repl (resolve.clj:528)
	shadow.build.resolve/resolve-repl (resolve.clj:514)
	shadow.cljs.repl/repl-ns (repl.clj:298)
	shadow.cljs.repl/repl-ns (repl.clj:291)
	shadow.cljs.repl/process-read-result (repl.clj:400)
	shadow.cljs.repl/process-read-result (repl.clj:380)
	shadow.cljs.devtools.server.worker.impl/fn--14362 (impl.clj:693)
	shadow.cljs.devtools.server.worker.impl/fn--14362 (impl.clj:657)
	clojure.lang.MultiFn.invoke (MultiFn.java:233)
	shadow.cljs.devtools.server.util/server-thread/fn--14076/fn--14077/fn--14085 (util.clj:294)
	shadow.cljs.devtools.server.util/server-thread/fn--14076/fn--14077 (util.clj:293)
	shadow.cljs.devtools.server.util/server-thread/fn--14076 (util.clj:266)
	java.lang.Thread.run (Thread.java:750)

for crawlee:

errors in file: .../node_modules/@crawlee/core/errors.js
{:js-str-offsets [], :js-esm false, :js-imports [], :js-invalid-requires [], :goog-provides [], :js-language "es3", :goog-module nil, :js-warnings [], :resource-name "node_modules/@crawlee/core/errors.js", :js-requires [], :js-errors [{:line 27, :column 24, :message "primary expression expected"}], :goog-requires [], :tag :shadow.build.npm/errors, :uses-global-buffer false}
ExceptionInfo: errors in file: .../node_modules/@crawlee/core/errors.js
	clojure.core/ex-info (core.clj:4739)
	clojure.core/ex-info (core.clj:4739)
	shadow.build.npm/get-file-info* (npm.clj:519)
	shadow.build.npm/get-file-info* (npm.clj:421)
	shadow.build.npm/get-file-info (npm.clj:556)
	shadow.build.npm/get-file-info (npm.clj:553)
	shadow.build.npm/find-resource (npm.clj:645)
	shadow.build.npm/find-resource (npm.clj:605)
	shadow.build.resolve/find-npm-resource (resolve.clj:110)
	shadow.build.resolve/find-npm-resource (resolve.clj:80)
	shadow.build.resolve/fn--11794 (resolve.clj:202)
	shadow.build.resolve/fn--11794 (resolve.clj:183)
	clojure.lang.MultiFn.invoke (MultiFn.java:243)
	shadow.build.resolve/resolve-string-require (resolve.clj:324)
	shadow.build.resolve/resolve-string-require (resolve.clj:307)
	shadow.build.resolve/resolve-require (resolve.clj:491)
	shadow.build.resolve/resolve-require (resolve.clj:484)
	shadow.build.resolve/resolve-deps/fn--11761 (resolve.clj:58)
	clojure.lang.PersistentVector.reduce (PersistentVector.java:341)
	clojure.core/reduce (core.clj:6747)
	clojure.core/reduce (core.clj:6730)
	shadow.cljs.util/reduce-> (util.clj:47)
	shadow.cljs.util/reduce-> (util.clj:46)
	shadow.build.resolve/resolve-deps (resolve.clj:56)
	shadow.build.resolve/resolve-deps (resolve.clj:40)
	shadow.build.resolve/resolve-string-require (resolve.clj:343)
	shadow.build.resolve/resolve-string-require (resolve.clj:307)
	shadow.build.resolve/resolve-require (resolve.clj:491)
	shadow.build.resolve/resolve-require (resolve.clj:484)
	shadow.build.resolve/resolve-deps/fn--11761 (resolve.clj:58)
	clojure.lang.PersistentVector.reduce (PersistentVector.java:341)
	clojure.core/reduce (core.clj:6747)
	clojure.core/reduce (core.clj:6730)
	shadow.cljs.util/reduce-> (util.clj:47)
	shadow.cljs.util/reduce-> (util.clj:46)
	shadow.build.resolve/resolve-deps (resolve.clj:56)
	shadow.build.resolve/resolve-deps (resolve.clj:40)
	shadow.build.resolve/resolve-string-require (resolve.clj:343)
	shadow.build.resolve/resolve-string-require (resolve.clj:307)
	shadow.build.resolve/resolve-require (resolve.clj:491)
	shadow.build.resolve/resolve-require (resolve.clj:484)
	shadow.build.resolve/resolve-deps/fn--11761 (resolve.clj:58)
	clojure.lang.PersistentVector.reduce (PersistentVector.java:341)
	clojure.core/reduce (core.clj:6747)
	clojure.core/reduce (core.clj:6730)
	shadow.cljs.util/reduce-> (util.clj:47)
	shadow.cljs.util/reduce-> (util.clj:46)
	shadow.build.resolve/resolve-deps (resolve.clj:56)
	shadow.build.resolve/resolve-deps (resolve.clj:40)
	shadow.build.resolve/resolve-symbol-require (resolve.clj:481)
	shadow.build.resolve/resolve-symbol-require (resolve.clj:433)
	shadow.build.resolve/resolve-require (resolve.clj:488)
	shadow.build.resolve/resolve-require (resolve.clj:484)
	shadow.build.resolve/resolve-entry (resolve.clj:498)
	shadow.build.resolve/resolve-entry (resolve.clj:497)
	clojure.lang.PersistentVector.reduce (PersistentVector.java:341)
	clojure.core/reduce (core.clj:6747)
	clojure.core/reduce (core.clj:6730)
	shadow.cljs.util/reduce-> (util.clj:47)
	shadow.cljs.util/reduce-> (util.clj:46)
	shadow.build.resolve/resolve-entries (resolve.clj:509)
	shadow.build.resolve/resolve-entries (resolve.clj:500)
	shadow.build.modules/analyze-module/fn--12569 (modules.clj:243)
	shadow.build.modules/analyze-module (modules.clj:239)
	shadow.build.modules/analyze-module (modules.clj:228)
	clojure.lang.PersistentVector.reduce (PersistentVector.java:341)
	clojure.core/reduce (core.clj:6747)
	clojure.core/reduce (core.clj:6730)
	shadow.build.modules/analyze-modules (modules.clj:259)
	shadow.build.modules/analyze-modules (modules.clj:258)
	shadow.build.modules/analyze (modules.clj:313)
	shadow.build.modules/analyze (modules.clj:304)
	shadow.build/resolve (build.clj:346)
	shadow.build/resolve (build.clj:340)
	shadow.build/compile (build.clj:354)
	shadow.build/compile (build.clj:348)
	shadow.cljs.devtools.server.worker.impl/build-compile (impl.clj:251)
	shadow.cljs.devtools.server.worker.impl/build-compile (impl.clj:237)
	shadow.cljs.devtools.server.worker.impl/do-resource-update (impl.clj:779)
	shadow.cljs.devtools.server.worker.impl/do-resource-update (impl.clj:742)

I am guessing this has to do with how shadow-cljs interprets those librairies, but I am not sure how to configure the shadow-cljs.edn in order to make the import.

Thanks for your help!

[p-himik]

Let’s go through ecpair first.
The error complains about this location: UNPKG - ecpair
As you can see, it’s a class field. And, unfortunately, GCC doesn’t support class fields yet: Add Support for Class Fields · Issue #2731 · google/closure-compiler · GitHub
I don’t know what the preferred way of fixing such an error is. Perhaps webpack could somehow be used. Or you might be able to vendor in the library. Or maybe use some older version of it that doesn’t use class fields - that was exactly what I did with ANTLR.

Regarding crawlee - that particular line is here: https://unpkg.com/browse/@crawlee/core@3.1.0/errors.js#L27
And the column points somewhere around ?? which is called “nullish coalescing operator”, which is supposed to be supported by GCC since v20200315. Is there a chance your CLJS version is out of date or you explicitly depend on an old GCC version? I just tried compiling a simple script that only requires ["crawlee"] and it worked just fine with shadow-cljs 2.20.4.

[asiegf]

Very helpful, thanks.

It looks like support for Class Fields won’t come before Q3-2023.

As for crawlee I have updated shadow-cljs, clojure and specified the latest version for clojurescript, but it still yields the same error. I am still trying to figure this out.

[thheller]

@asiegf

I am still trying to figure this out.

Make sure the restart shadow-cljs properly after changing versions/dependencies. And make sure you updated it in the correct place. If you use deps.edn or project.clj you need to update there, updating the npm package alone is not sufficient then.

As far as your general problem is concerned these are problems in the Closure Compiler. Unfortunately I cannot do anything on the shadow-cljs side to address these. You can however just use another bundler (eg. webpack) via the method described here, which just offloads the npm package processing to that bundler.

But crawlee at least seem like they it wants to run in node? Are you using a proper target (eg. :node-script) for that? shadow-cljs won’t process npm packages at all for those, so it should just work.

[asiegf]

Awesome! I will look into how to use another bundler, thanks.

You were on spot, with the proper target, the import in cljs works just fine.