Let's tap> with let> (a.k.a. My First Macro: taplet)

[PEZ]

I’ve made a macro! You will now find pez/taplet on Clojars.

OK. It s not my first macro. But it is my first public macro, and my first non-trivial one. It’s WIP and I would like some input on the names of the macros and also the API a bit.

What it does? There are two macros: let> and let>-l. The first one can be used as a drop-in replacement to let, only that it will also tap> the binding box. The second one takes a keyword as the first argument and will insert that first as a label in the tapped vector.

My use case if for debugging. I found myself doing some copy/paste and fancy regex search/replace to see what some let bindings actually evaluated to.

Example usage:

(require '[pez.taplet :refer [let> let>l]])

(let> [x 1
       y 2]
 [x y])

(let>l :foo
      [x 1
       y 2]
 [x y])

These will tap:

[:x 1
 :y 2]

[:foo
 :x 1
 :y 2]

respectively

Destructurings sort of works, a bit. For vector destructurings it is all just flattened:

(let> [[a [b [c d]]] [:foo [:bar [:baz :gaz]]]]
      [a b c d])

Taps: [:a :foo :b :bar :c :baz :d :gaz]

For map destructurings it is a bit rawer. I don’t really know how to pick out the bound symbols, so I just output the map as a string, and then the value is the same map, but with the values replacing the symbols (this is output as a map). Maybe it is easiest to show with my test for it:

(testing "Taps map destructuring bind-to as a string"
    (let [tapped (atom nil)
          save-tap (fn [v] (reset! tapped v))]
      (add-tap save-tap)
      (is (= [2 {:x 2}]
             (sut/let> [{:keys [x] :as y} {:x 2}]
               [x y])))
      (is (= ["{:keys [x], :as y}" {:keys [2] :as {:x 2}}]
             @tapped))
      (remove-tap save-tap)))

I tried pretty hard to output the vector using symbols instead of keywords for the ”bind-to”, but my macro-fu did not suffice. Same with the string representation of the destructuring map. I would want a map instead, but couldn’t figure out how to do it. If you peeps know how to do this, please tell me!

Do you think the names for the macros are any good? Have other suggestions? I have deployed it only as a -SNAPSHOT release for reasons.

You find the repo here. Issues and PRs welcome, of course.

PEZ/taplet

[bsless]

Pretty cool
Regarding maps, this version works:

(defmacro let>l
  "Like `let>`, adding a label first in the tapped vector"
  [label bindings & body]
  (assert (or (nil? label)
              (keyword? label))
          "`label` is not a keyword")
  (let [bindings (destructure bindings)
        taps (as-> bindings $
               (map first (partition 2 $))
               (mapcat vector (map keyword $) $)
               (into (if label [label] []) $))]
    `(let [~@bindings]
       (tap> ~taps)
       ~@body)))

Will take a crack at symbols, too
Edit:
for symbols replace keyword with

(fn [sym] `(quote ~sym))

Edit 2: I don’t have a good idea regarding the gensym-ed bindings, but I can see why keeping them would be convenient

[PEZ]

Thanks!

I get a lot of clues from this, even if it is not exactly what I am after. It gets a bit less readable with this. Consider the case of a map destructure:

(let> [{:keys [x] :as y} {:x 2}]
       [x y])

What is tapped now:

["{:keys [x], :as y}"
 {:keys [2] :as {:x 2}}]

I’m reasonably fine with that, but tried (and failed) to get this:

[{:keys [x], :as y}
 {:keys [2] :as {:x 2}}]

(Not quite sure what that means, but I sort of can read it… )

What I get with your changes:

[:map__14605
 {:x 2}
 :map__14605
 {:x 2}
 :y
 {:x 2}
 :x
 2]

I then think maybe a mix of the two is what would be most useful?

[{:keys [x], :as y}
 :y
 {:x 2}
 :x
 2]

My head explodes a bit trying to figure out how to achieve it though…

Maybe this would be best though, come to think of it like this:

[:y
 {:x 2}
 :x
 2]

Vector destructurings:

(let> [[a [b [c d]]] [:foo [:bar [:baz :gaz]]]]
      [a b c d])

What I now have:

[:a :foo
 :b :bar
 :c :baz
 :d :gaz]

Which is quite ok, at least for my use case where I want to quickly see what each symbol gets bound to.

With your changes I get this:

[:vec__14610
 [:foo
  [:bar
   [:baz :gaz]]]
 :a
 :foo
 :vec__14613
 [:bar
  [:baz :gaz]]
 :b
 :bar
 :vec__14616
 [:baz :gaz]
 :c
 :baz
 :d
 :gaz]

Which takes some time for me to unpack. (Even if it does make sense in some way.)

EDITED, I pressed some keys that submitted it prematurely. =)

[PEZ]

Taking inspiration from your version @bsless , I now have this:

(defmacro let>l
  "Like `let>`, adding a label first in the tapped vector"
  [label bindings & body]
  (assert (or (nil? label)
              (keyword? label))
          "`label` is not a keyword")
  (let [symbolize (fn [sym] `(quote ~sym))
        taps (as-> bindings $
                  (flatten (partition 1 2 $))
                  (mapv (fn [b]
                          [(symbolize b) b])
                        $)
                  (into (if label [label] []) $))]
    `(let ~(destructure bindings)
       (tap> ~taps)
       ~@body)))

It taps almost exactly what I had first aimed for. But now I get greedy and would want to skip outputting the map destructions like that and instead just report the bound variables, like I do with simple bindings and vectors. I think.I might need to use a combo of the raw bindings and the destructured ones. And even more of a combo of yours and mine versions.

[bsless]

You might consider it a hack but you can just regexp filter gensymed bindings:

(defmacro let>l
  "Like `let>`, adding a label first in the tapped vector"
  [label bindings & body]
  (assert (or (nil? label)
              (keyword? label))
          "`label` is not a keyword")
  (let [bindings (destructure bindings)
        symbolize (fn [sym] `(quote ~sym))
        gensymed? (fn [sym] (re-matches #"(map|vec)__\d+" (name sym)))
        taps (as-> bindings $
               (map first (partition 2 $))
               (map vector (map symbolize $) $)
               (remove (fn [[_ s]] (gensymed? s)) $)
               (into (if label [label] []) $))]
    `(let [~@bindings]
       (tap> ~taps)
       ~@body)))

Edit:

In case you find as-> inelegant:

taps (into (if label [label] [])
           (comp (partition-all 2)
                 (map first)
                 (remove gensymed?)
                 (mapcat (fn [x] [(symbolize x) x])))
           bindings)

[PEZ]

Oh, that’s perfect! I guess it is a bit hackish, but it also is straight forward.

(let> [x 1
       {:keys [z] :as y} {:z 2}
       [a [b {:keys [c d]}]] [:foo [:bar {:c :baz :d :gaz}]]]
      [x z y a b c d])

=> [1 2 {:z 2} :foo :bar :baz :gaz]

tap> [[x 1]
      [y
       {:z 2}]
      [z 2]
      [a :foo]
      [b :bar]
      [c :baz]
      [d :gaz]]

Thanks! I’ll run with this.

@bsless

In case you find as-> inelegant:

I am quite the fan of as->, actually. I can appreciate the elegance of the comp version, and it is educational to see these side by side like this. But it still takes me a bit too long to unpack the comp, while the at-> is easy for me to read.

[thheller]

Have you considered a different approach of just tapping the locals? That seems to be what you are after? Rewriting let is always going to be trickier but accessing the locals from a macro is easy.

If you are running shadow-cljs try shadow-cljs clj-repl then open the http://localhost:9630/inspect-latest UI and run this in the clj-repl

(require '[shadow.debug :as dbg])
(let [{:keys [x] :as y} {:x 1}] (dbg/locals))

The macro for this is rather simple and you could tweak it to your needs without ever getting too complex. Works for CLJS too.

[PEZ]

Version 0.1.1 now released. I’m holding off 1.0.0 until I’ve had a bit more exposure to the macro names. I still think I should be able to come up with something better than let>l.

[PEZ]

Yes, actually that is pretty often what I need. Just tried it with the shadow-cljs inspector. I’ll be using that often. Thanks!

[PEZ]

Trying the shadow dbg/locals I also notice that I was a bit too quick to celebrate and release the updated taplet macro. Map destructurings do not work in CLJS:

(let> [{:keys [x] :as y} {:x 1}]
  [x y])

------ WARNING - :undeclared-ns ------------------------------------------------
Resource: <eval>:1:1
 No such namespace: clojure.lang.PersistentHashMap, could not locate
clojure/lang/PersistentHashMap.cljs, clojure/lang/PersistentHashMap.cljc, 
or JavaScript source providing "clojure.lang.PersistentHashMap"

Hmmm.

[thheller]

Yeah you call destructure (from clojure.core) but for CLJS you’d need to call cljs.core/destructure so it doesn’t end up using stuff from CLJ.

[PEZ]

@thheller

cljs.core/destructure

Trying this:

(def destr #?(:clj destructure
              :cljs cljs.core/destructure))

I get this.

Use of undeclared Var cljs.core/destructure

Though if I just use it directly (for testing and not caring that it won’t work for CLJ) it seems to work…

[thheller]

Its a function supposed to be called from macros so it is written in CLJ and should be called from CLJ. reader-literals won’t work here. You need to check in the macro if thats supposed to generate CLJ or CLJS code. The macro I linked that that by checking (:ns &env) which only CLJS has so you can use it to decide which desctructure variant to use.

[PEZ]

Ah, yes, of course.

[seancorfield]

Another approach to dealing with locals: commons/extensions.clj at master · worldsingles/commons (github.com) that might be useful here.

[p1mps]

I always wanted to create something similar but I’m a very noob with macros so thank you very much, I will for sure try it out!

[PEZ]

Thanks, fellow macro noob.

I can totally recommend writing some macros when you see the need for them. Generally I prefer data oriented solutions over macros, so I have not felt it being so important to explore. But here was something that clearly is the job for a macro, and I must say it was an eye opener of huge proportions for me to try implement it.

I wrote the macro for two reasons, the first being that I wanted this macro, the other that I am writing a beginners guide to Clojure. Even though the guide is not going to touch writing macros it felt a bit awkward to write about macros without having really tried to write something non-trivial myself. Now I am super happy that I tried it. I mean, on a superficial level I knew what Clojure being homoiconic menans, but after the macro fact I realize that I didn’t really know. OMG! I am actually manipulating Clojure code using the full expressivity of the Clojure language itself. I’m still a bit dizzy from it.

The macro should now even work in CLJS, btw.

[PEZ]

Thanks to the wonderfully helpful community I’d say my macro got pretty decent. I have already had good use for it myself.

I probably should make a write-up about the different aspects of the process. We’ll see if I find the time. But anyway if you hear of someone who want some head start to creating a library for both Clojure and ClojureScript, with unit tests for both that run in Github Actions CI, by all means I think taplet is small and to the point enough to work as a starting point.

In case I won’t find the time for a full write-up, let me share what I found was the biggest hurdle to overcome: testing what is tap>ed by the macro. @seancorfield pointed me at how I can add a tap and then check there.

After a bit of experimenting with different ways of tapping/reading, I turned to core/async sliding buffers:

(def ^:private tapped (a/chan (a/sliding-buffer 1)))
(def ^:private  save-tap (fn [v] (a/offer! tapped v)))
(defn- read-tapped [] (a/poll! tapped))

(defn fixture [f]
  (add-tap save-tap)
  (f)
  (remove-tap save-tap))

(use-fixtures :once fixture)

To me the models of the tap and the sliding buffer channels mix well in my head. The fixture makes the tests have very little of the boilerplate I had a while.

Then I still had some problems with making it predictable. And also CLJ and CLJS behaved totally different in the testing, even when the macro seemed to work the same on both.

My symptoms for the CLJ brittleness of the tests where that of my 5 tests, 0, or 1, or 2 of any of them sometimes failed. A while I thought the problem was that the tap queue was full when I tried to tap>, but it turned out I was a bit to eager to check what had been tapped. I am now using core async to pause a millisecond between calling the macro and checking the tap.

(a/<!! (a/timeout 1))

That seems to be enough to make the tests predictable.

For CLJS my reading of the tap were more predictable, they always returned nil. @thheller pointed out it was that in ClojureScript the tap> is happening async, (in a js/setTimeout). But the function for tapping is a dynamic var so I could replace it. I let this be enough for that:

(set! *exec-tap-fn* (fn [f] (f)))

Super predictable tests after that.

Putting it all together, my tests now look like this:

PEZ/taplet/blob/824ad8766f72de4aed4b75face06ada7c6411076/test/pez/taplet_test.cljc#L1-L17

PEZ/taplet/blob/824ad8766f72de4aed4b75face06ada7c6411076/test/pez/taplet_test.cljc#LL25-33

I am pretty happy with this now. Thanks to everyone who helped me!