Callbacks as Clojure Records: an approach to generic Reagent components

Someone probably had the idea before, but since I haven’t seen it posted I thought I’d share it here.

In summary: the idea is to parameterize generic components with Clojure Records, which combine both the “pure values” benefits of Clojure data (immutability, transparency, smart equality for caching) and the “behavioural expressiveness” qualities of callback functions.

These ideas also apply to re-frame, Rum, Quiescent, etc.

The problem: generic components want callbacks

For making highly reusable components (e.g a generic date-picker input), you want to give the caller high expressive power. Typically, this is done by parameterizing the component with callback functions: for example, in re-com, the datepicker component accepts callback functions at keys :on-change and :selectable-fn.

One problem with that is that function-typed arguments aren’t very friendly to Reagent components. Reagent rewards calling components with immutable data structures that permit smart equality checks, to avoid re-rendering a component when its inputs haven’t changed. As soon as you call a Reagent component with a callback function (typically re-created by the caller at every render), you’re very likely to lose this benefit.

Compared to data structures, another downside of callbacks is that they’re opaque (e.g not practical to inspect at the REPL). Even if you’re able to decipher the origin of the callback from its name, you won’t typically be able to see the values it closes over, and so its behaviour will remain mysterious.

Proposal: callbacks as Clojure Records

I suggest tackling this problem by parameterizing the generic component not with callback functions, but with a Clojure Protocol, e.g:

(ns my.generic-components.date)

(defprotocol DatepickerCallbacks
  (on-change [this new-value] "will be called when the user has picked a date.")
  (selectable-date? [this d] "must return whether `d` is a valid date to pick."))

(defn datepicker
  [opts callbacks current-date]
  [:div
   ...
   [:div
    (for [d displayed-dates
          :when (selectable-date? callbacks d)]
      [:div {:on-click #(on-change callbacks d)}
       ...])]
   ...])

On the caller side, you define a Clojure Record implementing that protocol, and pass an instance of it to the generic component:

(ns my.specific.ui.order-form 
    (:require [re-frame.core :as rf]
              [my.generic-components.date]
              [cljs-time.core :as time]))
  
  (defrecord DeliveryDatepickerCallbacks
    [cart-id today-date]
    my.generic-components.date/DatepickerCallbacks
    (on-change [this new-delivery-date]
      (rf/dispatch [:myapp.orders.events/set-order-delivery-date cart-id new-delivery-date]))
    (selectable-date? [this d]
      (time/after? 
        (time/floor d time/day)
        today-date)))
  
  (defn delivery-date-field 
    [order-form]
    [:div
     [:p "Please choose a delivery date:"]
     [my.generic-components.date/datepicker {}
      (->DeliveryDatepickerCallbacks 
        (:myapp.cart/id order-form)
        (time/floor (time/now) time/day))
      (:myapp.order-form/date order-form)]])

Benefits

The record instance is an easily-inspected value, can be easily manipulated for experimentation, and will benefit from the caching facilities of Reagent.

I also find that the (defprotocol ...) expression is a convenient way to document the expected behaviour of the callbacks, so it can help with code clarity.

Drawbacks

One issue with the proposed approach is that you’re forced to define both a Protocol and a Record, and in particular you have to give them names. I think that’s not too much of an issue in this case, and if you really want an anonymous implementation at some point, there’s always reify.

Another issue is partial implementation. A lot of the operations defined in the protocols might stem from optional customizability. Yet when the callee receives an instance of the protocol, it can’t know which operations have been left unimplemented by the caller; that means the caller will need to provide that operation elsewhere, e.g in an options map.

One pitfall of this approach is to let the Protocol grow too big and complect many aspects of the component. Don’t hesitate making several Protocols to avoid that.

Abstract thoughts

Let’s take a step back: in essence, how do we want to invoke our generic component?

  1. we prefer to use immutable, transparent data structures
  2. we want to customize our generic component by telling it how to perform various operations ; i.e, we want multi-operation polymorphism

It turns out that this is exactly what Clojure records are designed for ; that’s one reason to make me feel the approach is a natural fit for this problem.

Now, you might object that if this were such a good idea, people would be doing it already. Maybe. But I suspect that ClojureScript users don’t often think of using Records and Protocols: they’re a relatively advanced part of Clojure, and so far the ecosystem hasn’t given much exposure to these constructs. So maybe the opportunity has been missed simply for lack of attention.

What do you think?

4 Likes

Interesting. What do you think about reagent.core/partial? Seems like it would solve some of the same problems without needing a protocol.

Btw, another downside I found out about recently is that defprotocol and reify generates an extreme amount of javascript, so bundle size suffers.

1 Like

Yeah all fair points. One issue with partial is that it does not output a transparent value.

yeah I find several cons w/ the record approach, namely:

  • proliferation of names
  • bundle size increase
  • composition complexity
  • verbosity

It’s all of the same problems you have with Java projects that require instances of over-specific classes or interfaces to be passed in, rather than something general like a callback or map.

Having a record that contains the closed-over values is IMO an anti-pattern, since now you have to change code in at least 2 places in order to close over a new value. You end up spreading your specific component’s behavior across multiple different structures (some in the component body, some in the record), making it more difficult to understand IME.

At the end of the day, we do have a real problem with combining memoization (which reagent does by default) with closing over values in a callback inside of a render. This is IME best solved using React Hooks, which were specifically designed to address this. E.g.:

(defn datepicker
  [{:keys [on-change selectable-date? ,,,]}]
  ,,,)

;; elsewhere
(defn delivery-date-field
  [{:keys [order-form]}]
  (let [cart-id (:myapp.cart/id order-form)
        on-date-change (React/useCallback
                        (fn [new-delivery-date]
                          (rf/dispatch [:myapp.orders.events/set-order-delivery-date cart-id new-delivery-date]))
                        #js [cart-id])

        today (time/floor (time/now) time/day) ;; capture the current day this was rendered
        selectable-date? (React/useCallback
                          (fn [d]
                            (time/after? (time/floor d time/day) today))
                          #js [today])]
    ,,,))

React/useCallback will hook into React’s lifecycle and ensure that the referenced returned by it only changes if the values in the dependency array change. This means that memoizing datepicker is very efficient, because we can do a simple reference equality rather than relying on a records equality semantics.

It also means that we claw back the ability to close over values and compose functions without an explosion of names, and changing code in multiple different places in order to change component behavior. For an example of what I mean, imagine that delivery-date-field also wanted to allow an on-change behavior that proxied to the datepicker's.

Obviously the above is also pretty verbose. Helix’s job is to make that less verbose. In the current version, it can be re-written to:

(defnc delivery-date-field
  [{:keys [order-form]}]
  (let [cart-id (:myapp.cart/id order-form)
        on-date-change (hooks/use-callback
                        :auto-deps
                        (fn [new-delivery-date]
                          (rf/dispatch [:myapp.orders.events/set-order-delivery-date cart-id new-delivery-date])))

        today (time/floor (time/now) time/day) ;; capture the current day this was rendered
        selectable-date? (hooks/use-callback
                          :auto-deps
                          (fn [d]
                            (time/after? (time/floor d time/day) today)))]
    ,,,))

And with the enabling of an experimental feature flag (which will later be enabled by default), even shorter:

(defnc delivery-date-field
  [{:keys [order-form]}]
  (let [cart-id (:myapp.cart/id order-form)
        on-date-change ^:callback (fn [new-delivery-date]
                                    (rf/dispatch [:myapp.orders.events/set-order-delivery-date cart-id new-delivery-date])))

        today (time/floor (time/now) time/day) ;; capture the current day this was rendered
        selectable-date? ^:callback (fn [d]
                                      (time/after?
                                       (time/floor d time/day)
                                       today)))]
    ,,,))
6 Likes

Very interesting thoughts. Thanks. I too struggle from time to time with generic components. There are just so many ways you can do things with Reagent and in my case re-frame. Also it is not always completely obvious how generic a component will be in practice.

This seems like an original solution, will try :slight_smile:

I think this is a very important question: how generic can we reasonably expect to make a component?

From what I’ve seen, web UIs tend to be (by essence) very irregular, and as a consequence the potential for generalization is usually small. To our frustration, we have to accept duplicating code much more often than factoring it.

2 Likes

Hyperfiddle deals with this by further extending reagent.core/partial mechanic using these utilities

Here is where your code ends up if you try to muscle through these issues code link

Obviously it is not a good time. Even with the perfect closure prop equality, the derefs (side effects) are unpredictable and thrash reagent’s rendering queue like hell and actual performance is terrible and opaque to reason about. We are leaving Reagent over this and deeply regret trying to muscle our way through these issues, which set the project back a full year at least.

1 Like

So my web-apps are of the ‘forms and save buttons’ type, mostly. Administrative software mostly.
So in most UIs I build, perf is not really an issue, and in fact, most stuff @vvvvalvalval mentions, wouldn’t be really manifest itself in sluggish behaviour. In fact, I consider my re-frame UI as pretty snappy indeed :slight_smile:

I’m curious, what kind of components do you have in your system that make you hit these perf troubles @dustingetz?

Hyperfiddle is in essence metadata driven UI, where the UI reconfigures itself in response to live changing metadata and this reactive reconfiguration needs to happen in milliseconds. The fundamental problem with Reagent (as described in this thread) is that you don’t get to program with closures. You’ll note in the code link i posted that there is not a single closure in view code, there are only six occurrences of (fn [...]) in 1000+ LOC and those six closures are e.g. contained within reduce (if the closure escapes and ends up passing through hiccup it would be a prop equality bug). Hundreds maybe thousands of devs are using Reagent without understanding this, they get away with it due to a very specific symmetry: in the value/onChange pattern the closure travels alongside a corresponding value, and if the onChange closure was rebuilt then the value also was likely changed, so the unstable closure ref is actually not harming anything because the component was about to re-render anyway due to the value changing. Also as Val hints at, views are hard to generalize and therefore most people just write boilerplate with very little usage of closures other than change handlers so they don’t hit this problem in practice. TLDR reagent is easy but not simple.

4 Likes

My team’s consensus right now is that reactive computations should be expressed through a proper FRP construct (that implements the right category theory interfaces – Applicative, Functor etc), and the reactive stream’s endpoint can be bound to React via a hook. React is thus a low-level rendering primitive that is a separate layer from the reactive computations. Such an approach we hope will bring back our ability to program with higher order functions without muscling through the tarpit.

Very interesting! Let’s continue down this road, with the caveat that this is still in the brainstorming stage.

The problem is you want something like a function that also has decent hashing and equality semantics like we see with hash maps.

So, my question is: why define a new protocol? It seems like the IFn protocol is ready-made for this! Unfortunately, IFn is not set up to allow metadata extension (which would be really handy for this problem!).

Most of the time, you’re passing in an equivalent function but with potentially different closed-over state. That is, the free variables might have different values on each re-rendering, but the code itself is the same. The ideal, then, would be to capture those free variables as part of the identity of the value so that it would trigger a difference if one of those values changed. The fn that actually does the work could be part of the metadata. Here’s an example (not sure if this would run):

(defrecord Handler [values]
  IFn
  (-invoke [this & rest] (-> this meta :callback (apply rest))))

(defn ->callback [f values]
  (with-meta (->Handler values) {:callback f}))

You could imagine a macro called fn** that figured out the free variables and called the ->callback constructor.

Anyway, sounds pretty cumbersome, but potentially could save a lot of re-rendering.

@ericnormand as mentioned by @isaksky, I think that’s exactly what reagent.core/partial does?

Also note that the rationale for using a Protocol in this case is to have multi-operation polymorphism (plus some other documentation affordances).

@ericnormand Note the symbolic AST `(foo ~a ~b) has value semantics because we have named the function (the symbol is resolved by eval in the namespaces). Could compile that AST into some sort of data bindings like Svelte. Does this accomplish more or less what you are thinking?

Here is reagent.core/partial btw which is only stable if it’s inputs are stable

1 Like

Everytime this comes up I can’t seem to remember what the problem is again? I’m not an active reagent user.

So… If I recall the problem is that React chooses which component to re-render by doing a diff, and in this case, it means that lambdas always diff unequal because every time the function is created under a different address and the equality check uses the address of the function?

And what would be the correct behavior again? That only two functions closing over different variables should be consider unequal? Or is it that two functions closing over the same variables but with different values that should be unequal?

Here is a summary

(= (fn [& args] x) 
   (fn [& args] x)) ; false bc closures are reference types

This messes with React render tree memoization

; different f each render, will never memoize
[:div {:f (fn [] x)}]

Reagent compares that prop check with cljs.core/= which is extensible with IEquiv, so there are a couple paths to hack it:

  1. Make a fake closure as a value object (defrecord F [x] IFn (-invoke [& args] x)) (= (->F 42) (->F 42)) ; equal records are equal
  2. Name the function with a var and reuse the var e.g. (defn f [] 42) (= f f) ; obviously true
  3. Various protocol gymnastics that push the complexity around without solving it

What all these hacks have in common is they all rely on an extrinsic stable name. Anonymous closure is forbidden forever, you can never write (fn []) ever again

More instances of the same problem:

(= (partial identity 42) (partial identity 2)) ; false
(= (constantly 42) (constantly 42))            ; false
(= (comp inc inc) (comp inc inc))              ; false

reagent.core/partial is a stabilized analog of cljs.core/partial using the record technique. r/constantly and r/comp can be implemented the same record technique (Reagent doesn’t provide this, but imo should, the utility ns i provided upthread implements this and more)

Advanced Reagent programming involves using reagent.core/track to derive reactions from other reactions, which is just an awkwardly shaped equivalent to fmap – reaction is a Functor and you can apply a function through it to get a new reaction out.

(= (fmap r inc) (fmap r inc)) ; false !!!

r/track is a stable version of this that can be used to implement r/fmap. But the herein reference equality problems poison your entire view layer and you basically drown in this.

3 Likes

This is what I came up with:

It includes a macro that makes any expr yielding a function at runtime comparable on the basis of value semantics, where a function’s value is the union of its source expr and the locals it closes over (I just use the whole &env in this gist but I have done work towards this goal in the past). Also I directly implemented equals rather than equiv but it’s just a sketch anyway.

Another alternative would be to retain this info (source + locals) for every fn very early in the bootstrap process so that any function in the game, including core functions could benefit from it. Then a patch to reagent would adapt the way it checks equality so that it would use this data when comparing functions.

This way, comp and any other function-returning function would return functions reagent could make sense of.

1 Like

Will this work in ClojureScript though?

1 Like

What about this kind of hack?

(defn get-fn-name [f]
 (second
  (re-find #"\$+(.*)\(" (str f))))

(extend-type
 function
 IEquiv
 (-equiv [this other]
  (or
   (identical? this other)
   (= (get-fn-name this) (get-fn-name other)))))        

With that (granted I’m trying on self-hosted ClojureScript) you can just name your anonymous functions, and equal names will be equal.

Seems var-args breaks on my get-fn-name function, but someone could spend more time on it.

Now, I don’t know, this rely on the fact that “str” returns the JavaScript compiled code for the function. Maybe this is only a self-hosted behavior?

Instead of defining a protocol you can make your records extend IFn — I know this can be done in Clojure, not sure about ClojureScript.