Callbacks as Clojure Records: an approach to generic Reagent components

Hyperfiddle deals with this by further extending reagent.core/partial mechanic using these utilities

Here is where your code ends up if you try to muscle through these issues code link

Obviously it is not a good time. Even with the perfect closure prop equality, the derefs (side effects) are unpredictable and thrash reagent’s rendering queue like hell and actual performance is terrible and opaque to reason about. We are leaving Reagent over this and deeply regret trying to muscle our way through these issues, which set the project back a full year at least.

1 Like

So my web-apps are of the ‘forms and save buttons’ type, mostly. Administrative software mostly.
So in most UIs I build, perf is not really an issue, and in fact, most stuff @vvvvalvalval mentions, wouldn’t be really manifest itself in sluggish behaviour. In fact, I consider my re-frame UI as pretty snappy indeed :slight_smile:

I’m curious, what kind of components do you have in your system that make you hit these perf troubles @dustingetz?

Hyperfiddle is in essence metadata driven UI, where the UI reconfigures itself in response to live changing metadata and this reactive reconfiguration needs to happen in milliseconds. The fundamental problem with Reagent (as described in this thread) is that you don’t get to program with closures. You’ll note in the code link i posted that there is not a single closure in view code, there are only six occurrences of (fn [...]) in 1000+ LOC and those six closures are e.g. contained within reduce (if the closure escapes and ends up passing through hiccup it would be a prop equality bug). Hundreds maybe thousands of devs are using Reagent without understanding this, they get away with it due to a very specific symmetry: in the value/onChange pattern the closure travels alongside a corresponding value, and if the onChange closure was rebuilt then the value also was likely changed, so the unstable closure ref is actually not harming anything because the component was about to re-render anyway due to the value changing. Also as Val hints at, views are hard to generalize and therefore most people just write boilerplate with very little usage of closures other than change handlers so they don’t hit this problem in practice. TLDR reagent is easy but not simple.

4 Likes

My team’s consensus right now is that reactive computations should be expressed through a proper FRP construct (that implements the right category theory interfaces – Applicative, Functor etc), and the reactive stream’s endpoint can be bound to React via a hook. React is thus a low-level rendering primitive that is a separate layer from the reactive computations. Such an approach we hope will bring back our ability to program with higher order functions without muscling through the tarpit.

Very interesting! Let’s continue down this road, with the caveat that this is still in the brainstorming stage.

The problem is you want something like a function that also has decent hashing and equality semantics like we see with hash maps.

So, my question is: why define a new protocol? It seems like the IFn protocol is ready-made for this! Unfortunately, IFn is not set up to allow metadata extension (which would be really handy for this problem!).

Most of the time, you’re passing in an equivalent function but with potentially different closed-over state. That is, the free variables might have different values on each re-rendering, but the code itself is the same. The ideal, then, would be to capture those free variables as part of the identity of the value so that it would trigger a difference if one of those values changed. The fn that actually does the work could be part of the metadata. Here’s an example (not sure if this would run):

(defrecord Handler [values]
  IFn
  (-invoke [this & rest] (-> this meta :callback (apply rest))))

(defn ->callback [f values]
  (with-meta (->Handler values) {:callback f}))

You could imagine a macro called fn** that figured out the free variables and called the ->callback constructor.

Anyway, sounds pretty cumbersome, but potentially could save a lot of re-rendering.

@ericnormand as mentioned by @isaksky, I think that’s exactly what reagent.core/partial does?

Also note that the rationale for using a Protocol in this case is to have multi-operation polymorphism (plus some other documentation affordances).

@ericnormand Note the symbolic AST `(foo ~a ~b) has value semantics because we have named the function (the symbol is resolved by eval in the namespaces). Could compile that AST into some sort of data bindings like Svelte. Does this accomplish more or less what you are thinking?

Here is reagent.core/partial btw which is only stable if it’s inputs are stable

1 Like

Everytime this comes up I can’t seem to remember what the problem is again? I’m not an active reagent user.

So… If I recall the problem is that React chooses which component to re-render by doing a diff, and in this case, it means that lambdas always diff unequal because every time the function is created under a different address and the equality check uses the address of the function?

And what would be the correct behavior again? That only two functions closing over different variables should be consider unequal? Or is it that two functions closing over the same variables but with different values that should be unequal?

Here is a summary

(= (fn [& args] x) 
   (fn [& args] x)) ; false bc closures are reference types

This messes with React render tree memoization

; different f each render, will never memoize
[:div {:f (fn [] x)}]

Reagent compares that prop check with cljs.core/= which is extensible with IEquiv, so there are a couple paths to hack it:

  1. Make a fake closure as a value object (defrecord F [x] IFn (-invoke [& args] x)) (= (->F 42) (->F 42)) ; equal records are equal
  2. Name the function with a var and reuse the var e.g. (defn f [] 42) (= f f) ; obviously true
  3. Various protocol gymnastics that push the complexity around without solving it

What all these hacks have in common is they all rely on an extrinsic stable name. Anonymous closure is forbidden forever, you can never write (fn []) ever again

More instances of the same problem:

(= (partial identity 42) (partial identity 2)) ; false
(= (constantly 42) (constantly 42))            ; false
(= (comp inc inc) (comp inc inc))              ; false

reagent.core/partial is a stabilized analog of cljs.core/partial using the record technique. r/constantly and r/comp can be implemented the same record technique (Reagent doesn’t provide this, but imo should, the utility ns i provided upthread implements this and more)

Advanced Reagent programming involves using reagent.core/track to derive reactions from other reactions, which is just an awkwardly shaped equivalent to fmap – reaction is a Functor and you can apply a function through it to get a new reaction out.

(= (fmap r inc) (fmap r inc)) ; false !!!

r/track is a stable version of this that can be used to implement r/fmap. But the herein reference equality problems poison your entire view layer and you basically drown in this.

3 Likes

This is what I came up with:

It includes a macro that makes any expr yielding a function at runtime comparable on the basis of value semantics, where a function’s value is the union of its source expr and the locals it closes over (I just use the whole &env in this gist but I have done work towards this goal in the past). Also I directly implemented equals rather than equiv but it’s just a sketch anyway.

Another alternative would be to retain this info (source + locals) for every fn very early in the bootstrap process so that any function in the game, including core functions could benefit from it. Then a patch to reagent would adapt the way it checks equality so that it would use this data when comparing functions.

This way, comp and any other function-returning function would return functions reagent could make sense of.

1 Like

Will this work in ClojureScript though?

1 Like

What about this kind of hack?

(defn get-fn-name [f]
 (second
  (re-find #"\$+(.*)\(" (str f))))

(extend-type
 function
 IEquiv
 (-equiv [this other]
  (or
   (identical? this other)
   (= (get-fn-name this) (get-fn-name other)))))        

With that (granted I’m trying on self-hosted ClojureScript) you can just name your anonymous functions, and equal names will be equal.

Seems var-args breaks on my get-fn-name function, but someone could spend more time on it.

Now, I don’t know, this rely on the fact that “str” returns the JavaScript compiled code for the function. Maybe this is only a self-hosted behavior?

Instead of defining a protocol you can make your records extend IFn — I know this can be done in Clojure, not sure about ClojureScript.

Yes see my reply to Eric above

@didibus you can name the closure like

(let [x (rand-nth [42 43])] 
  [:div {:f (fn foo-explicit-name [] x)}])

(and then use the name as a source of stability) but now non-equivalent closures have the same name, which breaks React by memoizing things that are not the same … you still need to capture the locals which means you need r/partial, r/comp, r/constantly, etc. And all of these “stabilized” combinators are only as stable as their input parameters, so the problem is recursive and making a single mistake breaks your React app. @TristeFigure 's CrystalFn (which is awesome) appears to automate this boilerplate in a very cool way as suggested by @ericnormand , but also suffers the problem that it can only be as stable as the locals it closes over and the problem is recursive. You are never free from thinking about this due to Reagent coupling reactive programming to React’s evaluation rules.

In my test, I saw two problems. Two functions arn’t equal even without closures:

(=
 (fn [a b] (+ a b))
 (fn [a b] (+ a b)))

:=> false

So I guess this is the first issue.

And the second issue you bring up is that if there are closures, which are really just additional inputs to the function, they should be part of the memoization. Using a name can only solve the first issue, since the second requires value equality.

And the other you bring up, just so I understand, is related to if you call another function which also closes over something? Now that too is indirect input that the memoization isn’t aware off.

I think @TristeFigure code should be recursive already no? If the locals are also of their custom Fn type, when they get compared they will also be compared using the proper equality semantics.

In general you probably want fast reference equality w/ functions, so I would not want to mess with redefining its IEquiv method.

The OP & replies are very interesting in exploring different ways to handle problems of equality in ClojureScript. However, I would again recommend people to use react/useCallback or react/useMemo for these purposes as it’s exactly what they’re made for.

example:

(react/useCallback
  (fn on-change [ev]
    (do-thing-with localA localB ev))
  #js [localA localB])

When used inside the body of a component, this expression will return the callback passed to it. The reference will only change if localA or localB change. This is precisely what you want when rendering components which are memoized (like reagent’s are by default) or other memoized values.

The way they work is by storing the callback & the array of values on the component instance. If you imagine you have a simple component tree like:

            app
           /   \
 componentA     componentB
     |              |
    input         button
                    |
                "click me"

When instantiated, this is represented roughly like (psuedo-code ahead):

{:type app
 :state {,,,}
 :props {,,,}
 :children [{:type componentA
             :state {,,,}
             :props {:on-change #object[Function on-change] ,,,}
             :children [,,,]}
            {:type componentB
             ,,,}]}

Now let’s say that app uses useCallback to memoize the on-change prop passed to componentA:

(defn app [{:keys [localA localB]}]
  (let [on-change (react/useCallback
                    (fn on-change [ev] ,,,)
                    #js [localA localB])]
    ,,,))

What this does is inform React to store the (fn on-change [ev] ,,,) value and the array [localA localB] on first render on the instance, something like:

{:type app
 :state {:hooks [{:type useCallback
                  :fn #object[Function on-change]
                  :deps #js [localA localB]}]}

And on each subsequent render, it will do a comparison between the values of the array in :deps to determine whether it should re-use the one stored on the component instance, or use the freshly created one.

Note that this strategy also allows us to ignore values or even pass in values that aren’t used, but semantically should bust the reference identity, which in practice can be quite helpful when tuning render behavior & performance.

2 Likes

@lilactown i agree with this but to get parity with reagent reactions i believe we’d need to layer in an event-streams or FRP library on top of hooks. Also, i believe render behavior and performance should not need to be tuned, it must be simply correct & optimal always, otherwise we are in the tar pit.

@didibus yes – suppose closure A closes over another closure B; closure B also needs to be stable for A to be stable.

You can easily use reagent reactions with React hooks. A use-reaction hook is a dozen lines of code, maybe.

i believe render behavior and performance should not need to be tuned, it must be simply correct & optimal always, otherwise we are in the tar pit.

This made me laugh out loud :joy: yes, we are all in the tar pit and always will be. There’s no escaping the complexity of trying to describe our desired behavior perfectly to our computers.