Introducing shadow-cljs Inspect

Projects shadow-cljs
1

With shadow-cljs 2.8.67 I released the first preview of the shadow-cljs “Inspect” feature.

My first goal for this is to provide a console.log equivalent experience for all platforms. console.log in combination with something like cljs-devtools lets you log “large” values and inspect them in a user-friendly manner. It is basically an enhanced println but is limited to the Browser. It doesn’t scale to really large values but is definitely better than prn. It also doesn’t work for node, react-native or CLJ.

The shadow-cljs UI now has a “Inspect” section that will allow you to look at values that were sent by tap> on any of the supported platforms (eg. CLJ, CLJS in the Browser, node, react-native, etc). In future versions this will integrate more directly with the REPL itself (like REBL) but the first iteration is just about tap>. Basically once the runtime is loaded up it will start communicating with the UI and you can inspect all your taps there.

I made a quick video showing what this is about yesterday night (shouldn’t be making videos that late but I hope you get the idea).

The UI should be considered alpha and will change substantially but I wanted to get this to everyone now so I can collect some feedback and maybe get some UI suggestions. I will personally start using this for everything now and ban console.log or prn from my workflow. It has already proven useful for me by allowing me to dig into the shadow-cljs state itself as I showed in the video.

To get tap> support in your project you only need to include the appropriate namespace. This can either be done via :preloads or at the REPL if you just want to try it out occasionally.

  • shadow.remote.runtime.cljs.browser intended for the Browser but currently should also support react-native as it has native WebSocket support
  • shadow.remote.runtime.cljs.node intended for node targets and uses the ws package to communicate with the shadow-cljs relay

To test this really quickly run

shadow-cljs server

Then open the shadow-cljs web UI (usually http://localhost:9630) and select the Inspect menu option.

You then run

$ shadow-cljs node-repl

cljs.user=> (require 'shadow.remote.runtime.cljs.node)
cljs.user=> (tap> :hello)

or for a quick browser test

$ shadow-cljs browser-repl

cljs.user=> (require 'shadow.remote.runtime.cljs.browser)
cljs.user=> (tap> :hello)

or just include :devtools {:preloads [shadow.remote.runtime.cljs.browser]} in your build config.

I have many more ideas for this and what is available in 2.8.67 is only the beginning. This is currently based on the shadow.remote architecture which I described a little here. For now this is integrated with the shadow-cljs project but will become standalone if there is enough interest. The underlying “protocol” might be useful for tools and could potentially be used as a substitute for nREPL or maybe used via nREPL. Basically everything you see in the shadow-cljs UI is already accessible to other tools if you want it. You could build an entirely different UI. There is nothing shadow-cljs specific in the architecture.

I would appreciate help for the UI design since I’m a really horrible designer and the UX for this has to become better. I’m happy about suggestions on that front. Screenshots/Videos/Sketches are appreciated. For the most part this is inspired by REBL but I’m not really sold on the whole current/next paradigm and it IMHO requires a bit too much screen space. Ideally everything would fit into a “narrow” window somewhere on the sides. Don’t want to dedicate an entire monitor for it. :wink:

PS: Neither Inspect nor shadow.remote are great names … so these might change in the future. Naming is hard …

Edit: 2.8.68 moved the namespaces a bit. Updated to match the new ones.

37 Likes
2

Hello, Thomas!

Just wanted to say that this got me really excited. I really look forward to seeing where this goes!

I think learning how the REPL works, where you run your REPL server and where you evaluate your code from can be a great challenge when learning Clojure, and that you are making it simpler to understand what’s going on with Shadow-CLJS’ visual tooling. And that this is a great step in the direction of providing ability to explore the data without making the system harder to understand.

PS. Just got it working on my machine, this is fun!

Teodor

1 Like
3

This is fantastic @thheller! Thank you for building this.

How would I go about testing this in a regular Clojure library? Or is that not in scope yet?

I am not totally sold on the way REBL is set up either. It does take up a lot of screen space. It also is not extensible. Being able to provide a custom way to render a value is immensely valuable. The browser provides a rich ecosystem of visualization tools that you can use. I’ve also noticed that inspecting large values in REBL can completely freeze (and sometimes crash) the whole UI.

I am particularly interested in providing the ability to display values in a chart.

One piece of feedback on the UX: instead of loading items in the Browse UI when you mouse over them, load all the items in the table’s view window. As the user scrolls down, automatically load more the items that come into the view.

1 Like
4

Currently all the code lives in the shadow-cljs repo. I will eventually split the “runtime” part into a smaller separate library so it can be embedded more easily and be able to connect to a remote “relay”. It is probably a bit early to split this apart until all the fine details are settled. The “protocol” is pretty much done but I want this to be extensible on the runtime I haven’t quite worked out the details for that.

shadow-cljs however is just a library, so you could add it to your CLJ project and run it in the embedded form. It’ll provide its own UI but you’ll be able to use tap>. Just include the dependency and call

(require '[shadow.cljs.devtools.server :as server])
(server/start!)
(tap> :hello)

then open the web UI like normal. It should work without any config or npm stuff. It will bring in a ton of the shadow-cljs dependencies so this isn’t ideal but it should work.

That was the plan yes. Will also probably fetch in batch and allow for “paging” since a map/vector with thousands of entries will make the UI way too slow right now. The “rows” are created immediately, even if they aren’t visible or downloaded. React isn’t too great if it has to render thousands of DOM elements in a list. No one can look at thousands of entries anyways so regular paging with 100 (or so) items per page probably makes sense.

Definitely want to make the visualization extensible too but CLJS isn’t too great for this given :advanced optimizations don’t really allow it. Maybe I can just compile things on the fly … we’ll see.

1 Like
5

Very nice work Thomas!

This is something I was thinking about as well: https://github.com/binaryage/cljs-devtools/issues/11

1 Like
6

Is this the beginning of shadow-clj :yum: I could see myself using this for Clojure development as well.

3 Likes
7

Certainly not a full blown shadow-clj but a small library that connects to a remote “relay” (eg. shadow-cljs server) for sure.

3 Likes
8

Can I use this in a Clojure project?

9

Yes, in a “bad” form by embedding the full shadow-cljs server as describe in this response.

I’ll create a lighter remote adapter so that it can connect to a remote “relay” soon. Just need to sort out some of the runtime details first.

1 Like
10

Rewrote quite a bit of the UI so it doesn’t look super horrible anymore. Still not great though. Rewrote most of the “runtime” implementation as well. Also still not happy but getting closer to what I have planned.

Next release will look something like this. Still looking for UI design help. :wink:

5 Likes
11

Question, @thheller.

Would you be interested in integrating data visualization into shadow-cljs inspect? Like you hit some Vega data and is able to show it?

I’m not quite sure what this means yet, but I’m curious about the idea, and I’m not asking for commitment of any sort. I just came back from a walk in the woods, and this suddenly seemed like a good idea. And I’m not saying this should be a priority at this moment.

I feel that shadow-cljs Inspect + datavis might be a strong contender to notebooks, perhaps even better; encompassing the Clojure spirit.

Edit: I tried to see if I was able to mess around with it myself, but there were too many moving parts of shadow-cljs in whole that I didn’t understand. If you do decide to split Inspect out from the rest, I’ll happily do some testing.

As for the UI design help, I’m afraid I’m no better than you in that regard!

12

I do want this primarily as a development aid to make debugging (aka println) better and the overall runtime more inspectable. Much like re-frame-10x and fulcro-inspect let you inspect their respective state, just more general so each framework doesn’t have to build their own tooling.

The goal isn’t something like devcards or workspaces or other “notebooks”. At least not my goal.

I definitely want visualizations of some kind though. Sometimes thats just a better way to look at some data. Not sure how to do that in a generic way though. Might require some helpers on the runtime-side. We’ll see, still so much left to explore.

2 Likes
13

Just to be perfectly clear, I like the mechanism you’ve presented here better than notebooks. Reinventing notebooks isn’t too interesting to me. On the other hand, if you happen to hit a value with some metadata saying this data should be rendered as a Vega visualization, and having the actual visualization show, that would be great.

I don’t have a clear idea of how that would actually work, though, and I’m looking forward to seeing where the Inspect feature goes regardless!

1 Like
14

I do have a few ideas but nothing too certain yet. I know that I want some kind of “table” view. Suppose you have a vector of maps in Clojure. Each map respresents a certain “Entity”, lets use “Product” as an example.

The UI might display this as

[{:product-id 1 :product-name "Foo" ...}
 {:product-id 2 :product-name "Bar" ...}
 ...]

While this is decent for small amounts of data it becomes rather unwieldy quickly (even tens of rows). Given that hash-maps will also have keys in different order the data you are looking for might be hard to find and in different places for each row.

So instead the UI should be able to detect this kind of data and let you select the “columns” you want to see and then display the data in a better format while still allowing you to select specific values in more detail.

Excuse the bad ASCII art you can imagine how a table would look in an actual UI.

| Id | Name          |
----------------------
| 1  | Foo           |
| 2  | Bar           |

Sometimes you’ll want to configure this table ad-hoc in the UI itself. Sometimes it might be better to do this when submitting the value via some metadata or so. Sometimes this may require extra support from the runtime, the whole point of this after all is that this runs remote and never requires transmitting ALL data to the UI (eg. sort the table by :product-name). I have built variants of this in the past but they all either broke completely when trying to “inspect” larger values or became really slow and not fun to use.

I know basically nothing about visualizations though. I have built a few basic graphs in the past but thats about it. I’m really interested in example use-cases though. Maybe there are some standard cases that can be included by default.

1 Like
15

@thheller, offtopic:

why I can’t use all the readline (?) features in shadow-cljs repl?
Like, alt-arrow to navigate by word?

Basically all I can do is type a text and delete it back to the place where the typo started and re-type it

16

Most people (me included) use an editor-connected REPL so have little use for the CLI-driven REPL. You can use rlwrap shadow-cljs cljs-repl foo if you are on a platform that has rlwrap (linux/mac).

Please open a separate thread or github issue if you want to discuss ruther. No need to go further offtopic here.

1 Like
17

Thanks @thheller, great stuff as always :raised_hands: I‘d love to see inspection tools integrated with chromium devtools, similar to react/redux/angular/… devtools extensions. I know that‘d be far more inconvenient to maintain for a library author, but it’s just so frictionless (plus, it‘d be very welcoming for folks coming from other ecosystems).

18

Well, it is just a webpage.

So you can just write a simple Browser extension that queries the current page for the data and then open an iframe for the UI. You can find (and query) all the required info from current page via the extension eval. Check shadow.cljs.devtools.client.env in the browser console to see the settings. Assuming of course you are allowed to create iframes in the devtools, I haven’t checked that part.

The reason I haven’t built this as a Browser extension is simply that I want this to work in more environments than just a Browser (eg. node, react-native, etc).

19

Got curious if iframe was actually allowed. Seems so.

Might look into how to publish this thing when I find some time.

shadow-cljs-ui-via-extension
shadow-cljs-ui-via-extension1156×859 119 KB

3 Likes
20

Completely understandable, I suspected as much, just wanted to throw it on the table since I find the mentioned extensions pretty smooth to use :v: