Shadow-cljs and leinigen and NPM oh my!

clojurescript

#1

I’ve decided I want to really focus on this stack in my career going forward for several reasons. Particularly of interest is clojurescript, the notion of reagent as a client-side runtime is one of the most exciting and novel programming concepts I’ve worked with in years!

Enough about background though! I’m an old vim hacker who prefers to use “fireplace” as the IDE and hence lein as my automation tool.

Currently I am studying a model project from shadow-cljs and reagent. This means I have a “shadow-cljs.edn” file and, as I understand it, NPM commands are used for automation.

But I want to use vim-fireplace, or at least cursive, which both work with leinigen as a build-automation tool. So my goal is to make my shadow-cljs project fully built/executed vi lein

My project is small with, say ./src/pete/core.cljs as the source.

I’m writing this question because I cannot seem to find a de-facto “make leinigen work with shadow” example for beginners like myself. At the moment I’m trying to cobble together notes from both sites of documentation, but with not-so-successful results.

When I execute len run I get the error: ‘cant find pete.core as a class or clj for lein run’
This is the current error but there may be additional things wrong in my setup.

My shadow-cljs.edn

{
 :lein true
 :nrepl        {:port 3333}
 :builds {
   :app {:target :browser
        :output-dir "public/js"
        :asset-path "/js"

        :modules
        {:main
         {:entries [pete.core]}}

        :devtools
        {:after-load  pete.core/main
         :http-root   "public"
         :http-port   3000}}}}

My project.clj

(defproject pete "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "https://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "https://www.eclipse.org/legal/epl-v10.html"}
 :dependencies [[binaryage/devtools "0.9.10"]
                [proto-repl "0.3.1"]
                [reagent "0.8.0"]
                [org.clojure/clojure "1.9.0"]
                [org.clojure/clojurescript "1.10.339"]
                [thheller/shadow-cljs "2.2.8"]]
  :main ^:skip-aot pete.core
  ;;:target-path "target/%s"
  :source-paths ["src"]
  :profiles {
     :dev {}
  })

#2

lein run is for executing Clojure code on the JVM. It expects there to be a src/pete/core.clj file for it to run, based on what you’ve written in your project.clj.

You won’t be using that to run your ClojureScript code. Instead, we’ll use shadow-cljs to build it:

shadow-cljs watch app

This will start up shadow-cljs and begin compiling our ClojureScript application.

Now, I’m not sure what integrations fireplace (or Cursive) has with lein, but if you want to have some kind of leiningen command to start the shadow-cljs build process, you can setup a command to run shadow-cljs from a lein invocation:

;; in your project.clj
:aliases
  {"dev" ["do"
          ["npm" "install"]
          ["run" "-m" "shadow.cljs.devtools.cli" "--npm" "watch"
           "app"]]}

With CIDER, what I usually do is start the shadow-cljs build process, and then connect remotely to the nREPL server (in this case on port 3333) so that I can execute expressions, get autocomplete, etc.

Hope that helps!


#3

Thanks @lilactown
Question though: I add your snippet to my project.clj which seems plenty clear.
Then I run >lein dev? do i have that right? Because the output is:

'npm' is not a task. See 'lein help'.

Did you mean this?
         new
         pom 

So, perhaps npm needs to be added as a task type in dependencies?


#4

Ack! I copied that from a project that uses lein-npm - sorry about that. You can remove the line that says ["npm" "install"] to get it to work without installing that plugin.


#5

Alias is totally optional and you can skip the npm install.

:aliases
{"dev" ["run" "-m" "shadow.cljs.devtools.cli" "watch" "app"]}

Also remove :main entirely from your config as you are not going to run any Clojure classes.

Also please bump [thheller/shadow-cljs "2.2.8"] to 2.6.4. 2.2.8 is really old.

(defproject pete "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "https://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "https://www.eclipse.org/legal/epl-v10.html"}
  
  :dependencies
  [[binaryage/devtools "0.9.10"]
   [proto-repl "0.3.1"]
   [reagent "0.8.1"]
   [org.clojure/clojure "1.9.0"]
   [org.clojure/clojurescript "1.10.339"]
   [thheller/shadow-cljs "2.6.4"]]

  :source-paths ["src"]
  :profiles {:dev {}})

#6

@thheller thank you so much for sharing your wonderful shadow-cljs software.

I’m sorry to keep asking for responses, and I will post my successful configuration on this thread. However your suggestions bring forth a new error which as a beginner I think is somewhat impenetrable:

Caused by: clojure.lang.ExceptionInfo: Call to clojure.core/refer-clojure did not conform to spec:
In: [2 1] val: :as fails at: [:args :exclude :op :quoted-spec :spec] predicate: #{:exclude}
In: [2 1] val: :as fails at: [:args :only :op :quoted-spec :spec] predicate: #{:only}
In: [2 1] val: :as fails at: [:args :rename :op :quoted-spec :spec] predicate: #{:rename}
In: [2] val: (quote :as) fails at: [:args :exclude :op :spec] predicate: #{:exclude}
In: [2] val: (quote :as) fails at: [:args :only :op :spec] predicate: #{:only}
In: [2] val: (quote :as) fails at: [:args :rename :op :spec] predicate: #{:rename}
 {:clojure.spec.alpha/problems ({:path [:args :exclude :op :spec], :pred #{:exclude}, :val (quote :as), :via [], :in [2]} {:path [:args :exclude :op :quoted-spec :spec], :pred #{:exclude}, :val :as, :via [], :in [2 1]} {:path [:args :only :op :spec], :pred #{:only},
:val (quote :as), :via [], :in [2]} {:path [:args :only :op :quoted-spec :spec], :pred #{:only}, :val :as, :via [], :in [2 1]} {:path [:args :rename :op :spec], :pred #{:rename}, :val (quote :as), :via [], :in [2]} {:path [:args :rename :op :quoted-spec :spec], :pred #{:rename}, :val :as, :via [], :in [2 1]}), :clojure.spec.alpha/spec #object[clojure.spec.alpha$regex_spec_impl$reify__2436 0x6dfa915a "[email protected]"], :clojure.spec.alpha/value ((quote :exclude) (quote [reduce into merge map take partition partition-by]) (quote :as) (quote core)), :clojure.spec.alpha/args ((quote :exclude) (quote [reduce into merge map take partition partition-by]) (quote :as) (quote core))}
        at clojure.core$ex_info.invokeStatic(core.clj:4739)
        at clojure.core$ex_info.invoke(core.clj:4739)
        at clojure.spec.alpha$macroexpand_check.invokeStatic(alpha.clj:689)
        at clojure.spec.alpha$macroexpand_check.invoke(alpha.clj:681)
        at clojure.lang.AFn.applyToHelper(AFn.java:156)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.lang.Var.applyTo(Var.java:702)
        at clojure.lang.Compiler.checkSpecs(Compiler.java:6889)
        ... 125 more

#7

Thats equally confusing to me. Looks like you have a .clj file that is getting loaded with an invalid ns form?


#8

@thheller research indicates the dependency [proto-repl "0.3.1"] is the culprit. When I comment this out leiningen issues the commands :confused:

  1. Is this bad to comment out?
  2. I presume the nREPL is all the same whether it is served up via leiningen or via shadow, the only thing that matters is the .nrepl-port is correct and, [insert fancy dev tool here] can talk to that repl all the same?

#9

Depends if you want to use proto-repl or not. It is the Clojure support for the Atom Editor. If you don’t use it you can take it out.

nREPL sometimes requires some middleware to be available to work properly. I don’t know the vim-fireplace setup but I think it uses cider-nrepl? Cursive works without any further config.


#10

For posterity, to get lein to issue my shadow commands, I ended up with

project.clj

(defproject pete "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "https://example.com/FIXME"
  :license {:name "Eclipse Public License" :url "https://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[binaryage/devtools "0.9.10"]
                ;;[proto-repl "0.3.1"] <--this is commented out to avoid errors
                [reagent "0.8.1"]
                [org.clojure/clojure "1.9.0"]
                [org.clojure/clojurescript "1.10.339"]
                [thheller/shadow-cljs "2.6.4"]]
  :target-path "target/%s"
  :aliases {"dev" ["do" ["run" "-m" "shadow.cljs.devtools.cli" "--npm" "watch" "app"]]}
  :source-paths ["src"]
)

shadow-cljs.edn

{:source-paths ["src"]
 :dependencies [[binaryage/devtools "0.9.10"]
                ;;[proto-repl "0.3.1"] <--commented out to avoid errors
                [reagent "0.8.1"]]

 :nrepl        {:port 3333}

:builds
 {:app {:target :browser
        :output-dir "public/js"
        :asset-path "/js"

        :modules
        {:main
         {:entries [pete.core]}}

        :devtools
        {:after-load  pete.core/main
         :http-root   "public"
         :http-port   3000}}}}

My project structure is like:

./project.clj
./shadow-cljs.edn
./src/pete/core.cljs

I run lein with lein dev
And so far so good! Thanks guys for the wonderful assist! Youre the best


#11

You are setting the :source-paths and :dependencies in both the shadow-cljs.edn and the project.clj. This is not how it should be done. If you want to use lein then just make shadow-cljs use lein via :lein true.

{:lein true
 :nrepl {:port 3333}
 
 :builds
 {:app {:target :browser
        :output-dir "public/js"
        :asset-path "/js"

        :modules
        {:main
         {:entries [pete.core]}}

        :devtools
        {:after-load pete.core/main
         :http-root "public"
         :http-port 3000}}}}

That way if you run shadow-cljs watch app it will actually launch the JVM via lein. Otherwise you’d get different results when launching shadow-cljs watch app or lein dev. You only need to declare your dependencies/source-path in one place.

(defproject pete "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "https://example.com/FIXME"
  :license {:name "Eclipse Public License" :url "https://www.eclipse.org/legal/epl-v10.html"}
  
  :dependencies
  [[binaryage/devtools "0.9.10"]
   [reagent "0.8.1"]
   [org.clojure/clojure "1.9.0"]
   [org.clojure/clojurescript "1.10.339"]
   [thheller/shadow-cljs "2.6.4"]]
  
  :target-path "target/%s"
  :source-paths ["src"]
  )

Personally I would recommand dropping the alias completely and just calling shadow-cljs watch app. It’ll perform the same function as lein dev but with a few extra safeguards in place.

Why are you using lein at all though?

You mentioned Cursive and vim-fireplace. Which one are you actually using? Doesn’t vim-fireplace connect remotely anyways? All editors should be completely fine without lein as long as you can connect remotely. In case of Cursive you just need something to import to create your project which can either be a project.clj or a pom.xml.

I’ll try and enhance the Cursive section in the docs. I do not use vim however and can’t comment on what is required to set it up properly. It should require no setup at all if its capable of connecting to a remote nrepl socket.


#12

@thheller you’re more than likely right on all points.

Most of the examples in docs reference leiningen, but to your point: a REPL is a REPL. I learned that lesson when you pointed out proto-repl was some sort of REPL proxy for atom compatibility. I’m very new to the ecosystem; trying to get my bearings. So far my experience is that the language itself couldnt be more self-explainatory but the tooling is where I’m spending most of my time.

I’ll spend a few minutes wiring fireplace to shadow while taking lein out of the picture and report back to you.


#13

From the docs I think that you need to add [cider/cider-nrepl "0.18.0"] to the :dependencies. Then run the shadow-cljs watch app and in vim :Connect localhost 3333?

That is with or without lein. The only difference it where :dependencies and :source-paths is declared. If you do that in shadow-cljs.edn you can remove the project.clj (and :lein true).


#14

I just came across your message after I was trying something very similar!
I did factor lein and project.cljs out of the picture, and use only shadow-cljs watch app and this works well.
I didnt think of adding [cider/cider-nrepl] to :dependencies, but tried it out per your suggestion.

Your suggestion on how to use vim-fireplace is also roughly correct, as I installed the plugin, ensured I have a running REPL for my app, and I use :Connect command and this works successfully,

When I try to actually use this REPL however, exceptions are thrown. I needed to alter the vim-fireplace code a little bit to get the full error and stack trace but it looks like

Fireplace: class java. lang.AssertionError AssertionError Assert failed: (keyword? repl-env) cider. 
piggieback/cljs-repl (fake_piggieback04.clj cider. piggieback$cljs repl.invokeStatic :45) cider . piggieback$cljs repl . : 45) 
clojure. lang. RestFn . invoke(RestFn . java: 416) 
 clojure. lang. Var . invoke(Var. java: 381) 
 [email protected](Unknown Source) 
[email protected](Unknown Source) 
clojure . lang. Compiler . eval (Compiler . java: 7062) 
clojure . lang. Compiler . eval (Compiler . java: 7625) 
clojure.core$eval . invokestatic(core.clj :3206) 
clojure.core$eval . invoke(core.clj 
clojure.lang.RestFn.invoke(RestFn.java: 1523) 

My visceral reaction is the vimscript is attempting to treat this REPL connection as if it is a piggieback, yet shadow-cljs is not like leiningen REPL and doesnt need the piggieback. But that is just my best guess so far. I’m searching the vim code for more clues, it is well-written but unfamiliar to me. Also I submitted an issue on the vim-fireplace github project

Does anything about this error dump enlighten you? My primary suspect remains the vim-script but I dont trust myself (I am a clojure beginner, a sleazy hack at best)


#15

I think there is a :Piggieback vim command? That would need to be called like :Piggieback :app, since it looks like it otherwise calling with some other args thus the assertion error.

It is true that piggieback is not required but it is emulated so editors don’t get confused (which doesn’t seem to work too well though).


#16

Calling :Piggieback :app (verbatim from your example completes without error.

Calling any sort of REPL command thereafter yields the following error message:

No application has connected to the REPL server. Make sure your js environment has loaded your compiled clojurescript code.

I am confident my REPL is running on nrepl://localhost:3333 hence I think the error message suggests the connection failed silently.


#17

Did you open the generated code in the browser? That is what the message is telling you. You can’t eval anything in the REPL unless the JS code has been loaded by some JS engine (eg. browser).


#18

Now that actually works!! Cool!

Help me understand:

  • Not all REPLs are created equal. nREPLs only support TTY terminal-style interaction
  • :Connect command should connect to a REPL
  • :Piggieback [appname] takes a nREPL and converts it to REPL which is friendlier with IDE communication
  • shadow-cljs REPL doesn’t need piggieback however there is some sort of issue between the current-version of vim-fireplace and shadow-cljs
  • shadow-cljs also works with piggybacking which, in this case, works around whatever the issue is
  • In order for the REPL to be fully loaded in shadow-cljs it must be jitted, most easily by browsing to the http server site

Is my understanding correct?


#19

It’s close:

When you first connect to shadow-cljs’ nREPL server, you’re initially inside of a Clojure JVM REPL session. So you can execute Clojure commands in the JVM process that shadow-cljs runs in.

We want to interact with our ClojureScript application running in a browser, though, not the shadow-cljs application. We need to upgrade our REPL to a ClojureScript REPL. piggieback is usually used to upgrade our nREPL connection from a Clojure REPL to a ClojureScript REPL, though shadow-cljs comes with that functionality built-in. To not break editors, it also provides the cemerick.piggieback API as well.

Usually I connect to a the shadow-cljs nREPL connection and then immediately run (shadow/repl :app), which is the shadow-cljs-specific API. Running (cemerick.piggieback/cljs-repl :app) does the exact same thing. That’s what the vim command :Piggieback :app does: run the cemerick.piggieback/cljs-repl command, which is the same thing as running shadow/repl.

Finally, just like a Clojure REPL needs a JVM process to execute expressions in, so does a ClojureScript REPL need a JavaScript environment to execute expressions in. The difference between Clojure and ClojureScript is that usually the Clojure REPL server is running in the same process as the one we’re executing expressions in, but with ClojureScript we send expressions to the REPL server which then compiles it to JS and sends it to the JS process.

In order to start the JS process, we open a web browser to the page we’re running our compiled code in which will then connect to our REPL server and begin to receive compiled expressions.

To summarize the steps needed to begin developing:

  1. Start shadow-cljs shadow-cljs watch app
  2. Connect to the nREPL server (I’m not a Vim user, but something like :Connect ?)
  3. Upgrade our REPL to a CLJS REPL :Piggieback :app
  4. Open our development web page in a web browser

Hope that helps!


#20

No clue what you mean. It pretty much depends on what you editor supports.

It connects to a remote Clojure nREPL server.

It converts the Clojure REPL to a ClojureScript REPL.

piggieback provides the CLJS REPL support on top on nREPL usually. shadow-cljs has its own support and only emulates its API for editors that expect to find piggieback (eg. vim-fireplace).

Yes the compiled CLJS code must be loaded somewhere. You can however also just use shadow-cljs node-repl (then :Piggieback :node-repl) so start a standalone node repl that starts its own node process so you don’t have to.