Creating "async" subscriptions in re-frame

[DrLjotsson]

I’m creating a report generator using re-frame. It processes a reasonably large collection of data items in several steps, where each step needs some computation and input from the user.

Schematic overview

Data [D0] → Step 1 computation → New data [D1] → Render view → Collect user input [I1]
[D0 D1 I1] → Step 2 computation → New data [D2] → Render view → Collect user input [I2]
[D0 D1 I1 D2 I2] → Step 3 computation → New data [D3] → Render view → Collect user input [I3]
[D0 D1 I1 D2 I2 D3 I3] → Step 4 computation and so on

My solution

The three views are all shown on the same screen which makes the UI laggy. Although each step’s computation only takes a few hundred ms, each click from the user starts a cascade of computations, especially if I1 is edited. I have solved this in the following way:

• Each computation as a reducing function. The function is run for 20 ms before it is suspended and control is given back to the browser (inspired by Solve the CPU hog problem).
• A state machine has one state for each step and controls the progression between the steps.
• When the computation for a step is completed, data is saved to app-db.
• An interceptor monitors if any of the input data is changed. Change in D0 moves the state machine to Step 1, change in I1 moves it to Step 2, etc.
• The UI disables user input for views that edit data for steps that have not been completed.

This solution works very well from a user experience perspective. There is no lag and the increase in computation time is inconsequential, the state machine progresses through the steps much faster than the user anyway.

However, the solution is not perfect. The steps are actually not serially dependent as described above, there is a bifurcation, and I need to use some of the computations in other views also. I could create multiple dependent state machines to solve this but it quickly gets messy.

My ideal solution

It would be much better to leverage re-frame’s subscription model. Each view would subscribe to the data it needs for user input and the subscriptions would use previous computations as signals. Then the data flows through the subscriptions to the views. A subscription to an incomplete computation would return some value the indicates that the computation is still ongoing (ideally a progress indication, which my solution above does), which the view would need to handle.

I’ve tried to make a wrapper for reg-sub that achieves this but I gave up. It just became too complicated and involved caching of signals that I don’t know how to dispose of when the subscription is disposed, emitting events in the computation function, and other complicated stuff. I think I would need at least a new reg-sub function and potentially a new subscribe function.

This is my idea on how a call to my-reg-sub would look for step 2

(my-reg-sub
  :d2 ; Returns data computed for step 2
  (fn [_]
    (subscribe [:d0])    ; A regular subscription
    (my-subscribe [:d1]) ; A (potentially special) subscription to :d1,
                         ; which is an async computation
    (subscribe [:i1]))   ; A regular subscription
  (fn [[d0 d1 i1]]
    ; Return a reducer that produces d2 by reducing d1, 
    ; also using data from d0 and i1
    ))

• The computation restarts if :i1 changes
• The computation also restarts if :d0 changes, however, change in :d0 also leads to recomputing of :d1 and I hope I could leverage logic in re-frame so that upstream subscriptions run before downstream subscriptions (I guess that re-frame orders computing of subscriptions based on their inter-dependence?).
• If :d1 is being computed, the computation function for :d2 is not run and (my-subscribe [:d2]) returns “a special value” indicating that the computation is pending.
• When the computation of :d1 has been completed, the computation function for :d2 runs. It returns a reducing function that is applied in chunks of 20 ms (or so). While this is ongoing, (my-subscribe [:d2]) returns “a special value” indicating that the data is being computed.
• When the :d2 computation has been completed, (my-subscribe [:d2]) returns the computed data for :d2.

I have no idea if this is possible by just adding one (my-reg-sub) or two (+ my-subscribe) functions that leverage what is already in re-frame or if I need to completely re-invent the wheel. And I don’t know if it is a good idea either.

I would love to hear your thoughts on this!

[DrLjotsson]

So I was able to implement “My ideal solution” as I specified it, using reg-sub-raw and it works as I expected. Basically, I used track! to monitor the progress the computations (stored in an “ratom”, which was derefed by the reaction returned by reg-sub-raw). And the computations were restarted as soon has upstream subscriptions changed. However, actually using this “async” subscription in components became a bit too cumbersome and I will stick to my original solution.

It was a good learning experience tho

[greinseth]

I was meaning to take on this problem, but time is flying so fast. One alternative could be to offload the computations to a webworker using effects. Then the rest of the re-frame modelling should be straight forward.

[DrLjotsson]

Yes, I considered using webworkers. I have no experience of webworkers but I figured that having to serialize (and unserialize on return) a fairly large data structure and sending it to the webworker as soon as the user edits an input would lead to UI lag. My goal is that the user should not feel the cascading data processing that e.g., checking a checkbox leads to. But I have not actually tried it - and I guess that if only the input changes, the data structure that is filtered through the user’s input could be cached on the web worker’s side.